[celery] Queue-based task prioritization (672c81bf)

This commit is contained in:
2026-08-08 12:11:08 -04:00
parent d5a753146f
commit f7087ebe4e
11 changed files with 192 additions and 13 deletions

View File

@ -117,6 +117,34 @@ Import gym sessions and routines from a Flexify backup.
cover body-composition metrics on Task scrobbles and `bodyweight_kg` on
Workout scrobbles.
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
:PROPERTIES:
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
:END:
Split Celery work into four queues so time-sensitive tasks run promptly even
when long batch jobs are queued:
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
high-concurrency worker (`--concurrency=4 --prefetch-multiplier=1`).
- `charts` — per-scrobble chart updates (`update_charts_for_timestamp`) stay on
their own queue, no longer blocked behind scheduled chart rebuilds (those
moved to `background`).
- `background` — heavy/scheduled jobs (db backup, trend computation, sentiment
backfill, chart rebuilds, twitch VOD check, historical Last.fm dispatch) run
on a dedicated low-concurrency worker (`--concurrency=1`).
- `default` — everything else (imports, notifications, archivebox pushes,
board-game expansion fetches). `CELERY_TASK_DEFAULT_QUEUE` is now set
explicitly to `default`; previously un-routed tasks were published to
Celery's default `celery` queue, which no worker consumed.
Workers run as three FreeBSD daemons (`vrobbler_celery` on `default,charts`,
plus new `vrobbler_celery_priority` and `vrobbler_celery_background` rc.d
scripts with distinct nodenames). Procfile, justfile, Makefile, Drone and
Gitea workflows updated accordingly.
* Version 64.5 [1/1]
** DONE [#B] Clean up issues with org-mode notes :orgmode:notes:scrobbles:
:PROPERTIES: