[celery] Queue-based task prioritization (672c81bf)
This commit is contained in:
30
PROJECT.org
30
PROJECT.org
@ -18,7 +18,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
|
||||
to be sometimes cathartic and sometimes functional as I try to remember when I
|
||||
did a thing.
|
||||
|
||||
* Backlog [0/24] :vrobbler:project:personal:
|
||||
* Backlog [2/26] :vrobbler:project:personal:
|
||||
** TODO [#C] After transition to linux add curl_cffi as webpage scrapper again :webpages:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: d3cce1a7-d540-4d66-bf66-e75378e4eac7
|
||||
@ -556,4 +556,32 @@ Added a `workouts` Django app so we can scrobble gym sessions.
|
||||
- Templates, admin, DRF viewsets (`exercises`, `workout-routines`), MCP tools,
|
||||
and `tests/workouts_tests/` (unit conversion, form round-trips, importer,
|
||||
imperial POST) included.
|
||||
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
|
||||
:PROPERTIES:
|
||||
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Split Celery work into four queues so time-sensitive tasks run fast even when
|
||||
long batch jobs are queued up:
|
||||
|
||||
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
|
||||
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
|
||||
high-concurrency worker with `--prefetch-multiplier=1`.
|
||||
- `charts` — per-scrobble chart updates stay on their own queue, no longer
|
||||
blocked by scheduled chart rebuilds.
|
||||
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill,
|
||||
chart rebuilds, historical imports, twitch VOD check) run on a dedicated
|
||||
low-concurrency worker so they can't starve the others.
|
||||
- `default` — everything else (imports, notifications, archivebox, expansions).
|
||||
|
||||
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks
|
||||
were published to Celery's default `celery` queue, which no worker consumed.
|
||||
New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background`
|
||||
added alongside the existing `vrobbler_celery` (now `default,charts`).
|
||||
|
||||
** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm:
|
||||
:PROPERTIES:
|
||||
:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b
|
||||
:END:
|
||||
|
||||
Reference in New Issue
Block a user