From 2adc7f6f6b91450157ab94918b8e843fa3170edd Mon Sep 17 00:00:00 2001 From: Colin Powell Date: Sat, 8 Aug 2026 20:40:45 -0400 Subject: [PATCH] [release] Bump to version 65.1 - Add an exception list of artists as a constant that are exempted from splitting - Fix celery task prioritization especially for agent sessions - Before enriching anything, trust the POST data - Fix lastfm rate limiting dropping scrobbles --- CHANGELOG.org | 101 ++++++++++++++++++++++++++++++++++++++++++++++++ PROJECT.org | 102 +------------------------------------------------ pyproject.toml | 2 +- 3 files changed, 103 insertions(+), 102 deletions(-) diff --git a/CHANGELOG.org b/CHANGELOG.org index e0aaf51..0cdae07 100644 --- a/CHANGELOG.org +++ b/CHANGELOG.org @@ -1,5 +1,106 @@ #+title: CHANGELOG +* Version 65.1 [4/4] +** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata: +:PROPERTIES: +:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8 +:END: + +- Note taken on [2026-08-08 Sat 20:37] + Instead of a hardcoded exception list, we resolve artist identity before + splitting on " & ": + + - `resolve_artist_names()` in music/utils.py keeps an artist string literal + when a MusicBrainz artist id is provided (webhooks), when the full string + already exists as an Artist, or when it matches a single MusicBrainz + artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a + last resort for genuine collaborations. + - `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name + instead of always splitting. + - `reconcile_split_artists` management command repairs tracks already split + by the old logic, using the raw artist data stored in scrobble logs, and + deletes orphaned fragment artists. + +*** Description + +Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up +tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like +this put into it and then we stop trying to pull the artist apart when we run into it. + +** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents: +:PROPERTIES: +:ID: 672c81bf-bba9-a963-e8ac-52246d976cea +:END: + +*** Description + +Split Celery work into four queues so time-sensitive tasks run fast even when +long batch jobs are queued up: + +- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy + queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated + high-concurrency worker with `--prefetch-multiplier=1`. +- `charts` — per-scrobble chart updates stay on their own queue, no longer + blocked by scheduled chart rebuilds. +- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill, + chart rebuilds, historical imports, twitch VOD check) run on a dedicated + low-concurrency worker so they can't starve the others. +- `default` — everything else (imports, notifications, archivebox, expansions). + +Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks +were published to Celery's default `celery` queue, which no worker consumed. +New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background` +added alongside the existing `vrobbler_celery` (now `default,charts`). + +** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata: +:PROPERTIES: +:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0 +:END: + +*** Description + +Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks. + +In most cases, we should be able to trust this data to created music tracks or videos rather +than going to third-party services to enrich. Thus, for tracks and videos we should search in +the local database for imdb_id or musicbrainz_id for the specific content and, if found, not +enrich further. + +If not found, tracks and videos from mopidy and jellyfin should be created as completely as +possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only" +and start the scrobble. A separate celery task should be kicked off to enrich the track or video +async with the POST data stored in the log["raw_data"] and used by the celery enrichment task +to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed" +log a warning and move on. + +*** Implementation + +Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now: + +- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data + with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id` + are reused without re-enriching; otherwise the track is stamped with the POST-provided + artist/album/track mbids and run time and tagged `musicbrainz-provider`. +- The scrobble is tagged `webhook-metadata-only` and a new background celery task + `scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the + POST payload in `log["raw_data"]`. +- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run + time, featured artists, album linkage). If resolution matches an existing track by + musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and + deletes the webhook-created duplicate (skipping any in-progress conflicts). +- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a + warning. Notifications still blast out with the raw-data track at scrobble time; a later + repoint to a canonical track may show in a subsequent stop notification, which is accepted. + +Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` → +`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future +`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path. + +** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm: +:PROPERTIES: +:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b +:END: + * Version 65.0 [7/7] ** DONE [#B] Investigate how historice lastfm imports work :importers:music: :PROPERTIES: diff --git a/PROJECT.org b/PROJECT.org index 17b3dcc..d79cc0c 100644 --- a/PROJECT.org +++ b/PROJECT.org @@ -18,7 +18,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out to be sometimes cathartic and sometimes functional as I try to remember when I did a thing. -* Backlog [3/26] :vrobbler:project:personal: +* Backlog [0/36] :vrobbler:project:personal: ** TODO [#C] Configure IMAP folder/start in user profile :imap:settings: *** Description @@ -612,33 +612,6 @@ favorited media objects. *** Description As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction -** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata: -:PROPERTIES: -:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8 -:END: - -*** Description - -Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up -tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like -this put into it and then we stop trying to pull the artist apart when we run into it. - -*** Notes - -Instead of a hardcoded exception list, we resolve artist identity before -splitting on " & ": - - - `resolve_artist_names()` in music/utils.py keeps an artist string literal - when a MusicBrainz artist id is provided (webhooks), when the full string - already exists as an Artist, or when it matches a single MusicBrainz - artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a - last resort for genuine collaborations. - - `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name - instead of always splitting. - - `reconcile_split_artists` management command repairs tracks already split - by the old logic, using the raw artist data stored in scrobble logs, and - deletes orphaned fragment artists. - ** TODO [#A] Update how board game scrobbles work :boardgames: *** Description @@ -662,76 +635,3 @@ The Edit log form should have from top to bottom: There should be a tool that can look at the track associated with a scrobble and the actual raw_data from mopidy or jellyfin, and flag all tracks where the artist or track name or both differ. -** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata: -:PROPERTIES: -:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0 -:END: - -*** Description - -Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks. - -In most cases, we should be able to trust this data to created music tracks or videos rather -than going to third-party services to enrich. Thus, for tracks and videos we should search in -the local database for imdb_id or musicbrainz_id for the specific content and, if found, not -enrich further. - -If not found, tracks and videos from mopidy and jellyfin should be created as completely as -possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only" -and start the scrobble. A separate celery task should be kicked off to enrich the track or video -async with the POST data stored in the log["raw_data"] and used by the celery enrichment task -to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed" -log a warning and move on. - -*** Implementation - -Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now: - -- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data - with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id` - are reused without re-enriching; otherwise the track is stamped with the POST-provided - artist/album/track mbids and run time and tagged `musicbrainz-provider`. -- The scrobble is tagged `webhook-metadata-only` and a new background celery task - `scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the - POST payload in `log["raw_data"]`. -- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run - time, featured artists, album linkage). If resolution matches an existing track by - musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and - deletes the webhook-created duplicate (skipping any in-progress conflicts). -- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a - warning. Notifications still blast out with the raw-data track at scrobble time; a later - repoint to a canonical track may show in a subsequent stop notification, which is accepted. - -Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` → -`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future -`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path. - -** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents: -:PROPERTIES: -:ID: 672c81bf-bba9-a963-e8ac-52246d976cea -:END: - -*** Description - -Split Celery work into four queues so time-sensitive tasks run fast even when -long batch jobs are queued up: - -- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy - queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated - high-concurrency worker with `--prefetch-multiplier=1`. -- `charts` — per-scrobble chart updates stay on their own queue, no longer - blocked by scheduled chart rebuilds. -- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill, - chart rebuilds, historical imports, twitch VOD check) run on a dedicated - low-concurrency worker so they can't starve the others. -- `default` — everything else (imports, notifications, archivebox, expansions). - -Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks -were published to Celery's default `celery` queue, which no worker consumed. -New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background` -added alongside the existing `vrobbler_celery` (now `default,charts`). - -** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm: -:PROPERTIES: -:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b -:END: diff --git a/pyproject.toml b/pyproject.toml index 6f9bfff..00c0d36 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "vrobbler" -version = "65.0" +version = "65.1" description = "" authors = ["Colin Powell "]