[release] Bump to version 65.1
- Add an exception list of artists as a constant that are exempted from splitting - Fix celery task prioritization especially for agent sessions - Before enriching anything, trust the POST data - Fix lastfm rate limiting dropping scrobbles
This commit is contained in:
101
CHANGELOG.org
101
CHANGELOG.org
@ -1,5 +1,106 @@
|
||||
#+title: CHANGELOG
|
||||
|
||||
* Version 65.1 [4/4]
|
||||
** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||
:END:
|
||||
|
||||
- Note taken on [2026-08-08 Sat 20:37]
|
||||
Instead of a hardcoded exception list, we resolve artist identity before
|
||||
splitting on " & ":
|
||||
|
||||
- `resolve_artist_names()` in music/utils.py keeps an artist string literal
|
||||
when a MusicBrainz artist id is provided (webhooks), when the full string
|
||||
already exists as an Artist, or when it matches a single MusicBrainz
|
||||
artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a
|
||||
last resort for genuine collaborations.
|
||||
- `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name
|
||||
instead of always splitting.
|
||||
- `reconcile_split_artists` management command repairs tracks already split
|
||||
by the old logic, using the raw artist data stored in scrobble logs, and
|
||||
deletes orphaned fragment artists.
|
||||
|
||||
*** Description
|
||||
|
||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||
|
||||
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
|
||||
:PROPERTIES:
|
||||
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Split Celery work into four queues so time-sensitive tasks run fast even when
|
||||
long batch jobs are queued up:
|
||||
|
||||
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
|
||||
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
|
||||
high-concurrency worker with `--prefetch-multiplier=1`.
|
||||
- `charts` — per-scrobble chart updates stay on their own queue, no longer
|
||||
blocked by scheduled chart rebuilds.
|
||||
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill,
|
||||
chart rebuilds, historical imports, twitch VOD check) run on a dedicated
|
||||
low-concurrency worker so they can't starve the others.
|
||||
- `default` — everything else (imports, notifications, archivebox, expansions).
|
||||
|
||||
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks
|
||||
were published to Celery's default `celery` queue, which no worker consumed.
|
||||
New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background`
|
||||
added alongside the existing `vrobbler_celery` (now `default,charts`).
|
||||
|
||||
** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||
|
||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||
enrich further.
|
||||
|
||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||
log a warning and move on.
|
||||
|
||||
*** Implementation
|
||||
|
||||
Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now:
|
||||
|
||||
- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data
|
||||
with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id`
|
||||
are reused without re-enriching; otherwise the track is stamped with the POST-provided
|
||||
artist/album/track mbids and run time and tagged `musicbrainz-provider`.
|
||||
- The scrobble is tagged `webhook-metadata-only` and a new background celery task
|
||||
`scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the
|
||||
POST payload in `log["raw_data"]`.
|
||||
- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run
|
||||
time, featured artists, album linkage). If resolution matches an existing track by
|
||||
musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and
|
||||
deletes the webhook-created duplicate (skipping any in-progress conflicts).
|
||||
- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a
|
||||
warning. Notifications still blast out with the raw-data track at scrobble time; a later
|
||||
repoint to a canonical track may show in a subsequent stop notification, which is accepted.
|
||||
|
||||
Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` →
|
||||
`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future
|
||||
`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path.
|
||||
|
||||
** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm:
|
||||
:PROPERTIES:
|
||||
:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b
|
||||
:END:
|
||||
|
||||
* Version 65.0 [7/7]
|
||||
** DONE [#B] Investigate how historice lastfm imports work :importers:music:
|
||||
:PROPERTIES:
|
||||
|
||||
102
PROJECT.org
102
PROJECT.org
@ -18,7 +18,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
|
||||
to be sometimes cathartic and sometimes functional as I try to remember when I
|
||||
did a thing.
|
||||
|
||||
* Backlog [3/26] :vrobbler:project:personal:
|
||||
* Backlog [0/36] :vrobbler:project:personal:
|
||||
** TODO [#C] Configure IMAP folder/start in user profile :imap:settings:
|
||||
*** Description
|
||||
|
||||
@ -612,33 +612,6 @@ favorited media objects.
|
||||
*** Description
|
||||
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
||||
|
||||
** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||
|
||||
*** Notes
|
||||
|
||||
Instead of a hardcoded exception list, we resolve artist identity before
|
||||
splitting on " & ":
|
||||
|
||||
- `resolve_artist_names()` in music/utils.py keeps an artist string literal
|
||||
when a MusicBrainz artist id is provided (webhooks), when the full string
|
||||
already exists as an Artist, or when it matches a single MusicBrainz
|
||||
artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a
|
||||
last resort for genuine collaborations.
|
||||
- `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name
|
||||
instead of always splitting.
|
||||
- `reconcile_split_artists` management command repairs tracks already split
|
||||
by the old logic, using the raw artist data stored in scrobble logs, and
|
||||
deletes orphaned fragment artists.
|
||||
|
||||
** TODO [#A] Update how board game scrobbles work :boardgames:
|
||||
|
||||
*** Description
|
||||
@ -662,76 +635,3 @@ The Edit log form should have from top to bottom:
|
||||
There should be a tool that can look at the track associated with a scrobble and
|
||||
the actual raw_data from mopidy or jellyfin, and flag all tracks where the
|
||||
artist or track name or both differ.
|
||||
** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||
|
||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||
enrich further.
|
||||
|
||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||
log a warning and move on.
|
||||
|
||||
*** Implementation
|
||||
|
||||
Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now:
|
||||
|
||||
- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data
|
||||
with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id`
|
||||
are reused without re-enriching; otherwise the track is stamped with the POST-provided
|
||||
artist/album/track mbids and run time and tagged `musicbrainz-provider`.
|
||||
- The scrobble is tagged `webhook-metadata-only` and a new background celery task
|
||||
`scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the
|
||||
POST payload in `log["raw_data"]`.
|
||||
- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run
|
||||
time, featured artists, album linkage). If resolution matches an existing track by
|
||||
musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and
|
||||
deletes the webhook-created duplicate (skipping any in-progress conflicts).
|
||||
- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a
|
||||
warning. Notifications still blast out with the raw-data track at scrobble time; a later
|
||||
repoint to a canonical track may show in a subsequent stop notification, which is accepted.
|
||||
|
||||
Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` →
|
||||
`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future
|
||||
`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path.
|
||||
|
||||
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
|
||||
:PROPERTIES:
|
||||
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Split Celery work into four queues so time-sensitive tasks run fast even when
|
||||
long batch jobs are queued up:
|
||||
|
||||
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
|
||||
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
|
||||
high-concurrency worker with `--prefetch-multiplier=1`.
|
||||
- `charts` — per-scrobble chart updates stay on their own queue, no longer
|
||||
blocked by scheduled chart rebuilds.
|
||||
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill,
|
||||
chart rebuilds, historical imports, twitch VOD check) run on a dedicated
|
||||
low-concurrency worker so they can't starve the others.
|
||||
- `default` — everything else (imports, notifications, archivebox, expansions).
|
||||
|
||||
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks
|
||||
were published to Celery's default `celery` queue, which no worker consumed.
|
||||
New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background`
|
||||
added alongside the existing `vrobbler_celery` (now `default,charts`).
|
||||
|
||||
** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm:
|
||||
:PROPERTIES:
|
||||
:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b
|
||||
:END:
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "vrobbler"
|
||||
version = "65.0"
|
||||
version = "65.1"
|
||||
description = ""
|
||||
authors = ["Colin Powell <colin@unbl.ink>"]
|
||||
|
||||
|
||||
Reference in New Issue
Block a user