[release] Bump to version 65.1
- Add an exception list of artists as a constant that are exempted from splitting - Fix celery task prioritization especially for agent sessions - Before enriching anything, trust the POST data - Fix lastfm rate limiting dropping scrobbles
This commit is contained in:
101
CHANGELOG.org
101
CHANGELOG.org
@ -1,5 +1,106 @@
|
|||||||
#+title: CHANGELOG
|
#+title: CHANGELOG
|
||||||
|
|
||||||
|
* Version 65.1 [4/4]
|
||||||
|
** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||||
|
:END:
|
||||||
|
|
||||||
|
- Note taken on [2026-08-08 Sat 20:37]
|
||||||
|
Instead of a hardcoded exception list, we resolve artist identity before
|
||||||
|
splitting on " & ":
|
||||||
|
|
||||||
|
- `resolve_artist_names()` in music/utils.py keeps an artist string literal
|
||||||
|
when a MusicBrainz artist id is provided (webhooks), when the full string
|
||||||
|
already exists as an Artist, or when it matches a single MusicBrainz
|
||||||
|
artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a
|
||||||
|
last resort for genuine collaborations.
|
||||||
|
- `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name
|
||||||
|
instead of always splitting.
|
||||||
|
- `reconcile_split_artists` management command repairs tracks already split
|
||||||
|
by the old logic, using the raw artist data stored in scrobble logs, and
|
||||||
|
deletes orphaned fragment artists.
|
||||||
|
|
||||||
|
*** Description
|
||||||
|
|
||||||
|
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||||
|
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||||
|
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||||
|
|
||||||
|
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
|
||||||
|
:END:
|
||||||
|
|
||||||
|
*** Description
|
||||||
|
|
||||||
|
Split Celery work into four queues so time-sensitive tasks run fast even when
|
||||||
|
long batch jobs are queued up:
|
||||||
|
|
||||||
|
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
|
||||||
|
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
|
||||||
|
high-concurrency worker with `--prefetch-multiplier=1`.
|
||||||
|
- `charts` — per-scrobble chart updates stay on their own queue, no longer
|
||||||
|
blocked by scheduled chart rebuilds.
|
||||||
|
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill,
|
||||||
|
chart rebuilds, historical imports, twitch VOD check) run on a dedicated
|
||||||
|
low-concurrency worker so they can't starve the others.
|
||||||
|
- `default` — everything else (imports, notifications, archivebox, expansions).
|
||||||
|
|
||||||
|
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks
|
||||||
|
were published to Celery's default `celery` queue, which no worker consumed.
|
||||||
|
New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background`
|
||||||
|
added alongside the existing `vrobbler_celery` (now `default,charts`).
|
||||||
|
|
||||||
|
** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||||
|
:END:
|
||||||
|
|
||||||
|
*** Description
|
||||||
|
|
||||||
|
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||||
|
|
||||||
|
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||||
|
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||||
|
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||||
|
enrich further.
|
||||||
|
|
||||||
|
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||||
|
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||||
|
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||||
|
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||||
|
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||||
|
log a warning and move on.
|
||||||
|
|
||||||
|
*** Implementation
|
||||||
|
|
||||||
|
Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now:
|
||||||
|
|
||||||
|
- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data
|
||||||
|
with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id`
|
||||||
|
are reused without re-enriching; otherwise the track is stamped with the POST-provided
|
||||||
|
artist/album/track mbids and run time and tagged `musicbrainz-provider`.
|
||||||
|
- The scrobble is tagged `webhook-metadata-only` and a new background celery task
|
||||||
|
`scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the
|
||||||
|
POST payload in `log["raw_data"]`.
|
||||||
|
- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run
|
||||||
|
time, featured artists, album linkage). If resolution matches an existing track by
|
||||||
|
musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and
|
||||||
|
deletes the webhook-created duplicate (skipping any in-progress conflicts).
|
||||||
|
- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a
|
||||||
|
warning. Notifications still blast out with the raw-data track at scrobble time; a later
|
||||||
|
repoint to a canonical track may show in a subsequent stop notification, which is accepted.
|
||||||
|
|
||||||
|
Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` →
|
||||||
|
`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future
|
||||||
|
`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path.
|
||||||
|
|
||||||
|
** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm:
|
||||||
|
:PROPERTIES:
|
||||||
|
:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b
|
||||||
|
:END:
|
||||||
|
|
||||||
* Version 65.0 [7/7]
|
* Version 65.0 [7/7]
|
||||||
** DONE [#B] Investigate how historice lastfm imports work :importers:music:
|
** DONE [#B] Investigate how historice lastfm imports work :importers:music:
|
||||||
:PROPERTIES:
|
:PROPERTIES:
|
||||||
|
|||||||
102
PROJECT.org
102
PROJECT.org
@ -18,7 +18,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
|
|||||||
to be sometimes cathartic and sometimes functional as I try to remember when I
|
to be sometimes cathartic and sometimes functional as I try to remember when I
|
||||||
did a thing.
|
did a thing.
|
||||||
|
|
||||||
* Backlog [3/26] :vrobbler:project:personal:
|
* Backlog [0/36] :vrobbler:project:personal:
|
||||||
** TODO [#C] Configure IMAP folder/start in user profile :imap:settings:
|
** TODO [#C] Configure IMAP folder/start in user profile :imap:settings:
|
||||||
*** Description
|
*** Description
|
||||||
|
|
||||||
@ -612,33 +612,6 @@ favorited media objects.
|
|||||||
*** Description
|
*** Description
|
||||||
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
||||||
|
|
||||||
** DONE [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
|
||||||
:PROPERTIES:
|
|
||||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
|
||||||
:END:
|
|
||||||
|
|
||||||
*** Description
|
|
||||||
|
|
||||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
|
||||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
|
||||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
|
||||||
|
|
||||||
*** Notes
|
|
||||||
|
|
||||||
Instead of a hardcoded exception list, we resolve artist identity before
|
|
||||||
splitting on " & ":
|
|
||||||
|
|
||||||
- `resolve_artist_names()` in music/utils.py keeps an artist string literal
|
|
||||||
when a MusicBrainz artist id is provided (webhooks), when the full string
|
|
||||||
already exists as an Artist, or when it matches a single MusicBrainz
|
|
||||||
artist (e.g. "Simon & Garfunkel" is a Group). Splitting on " & " is now a
|
|
||||||
last resort for genuine collaborations.
|
|
||||||
- `Track.find_or_create()` and `Track.fix_metadata()` use the resolved name
|
|
||||||
instead of always splitting.
|
|
||||||
- `reconcile_split_artists` management command repairs tracks already split
|
|
||||||
by the old logic, using the raw artist data stored in scrobble logs, and
|
|
||||||
deletes orphaned fragment artists.
|
|
||||||
|
|
||||||
** TODO [#A] Update how board game scrobbles work :boardgames:
|
** TODO [#A] Update how board game scrobbles work :boardgames:
|
||||||
|
|
||||||
*** Description
|
*** Description
|
||||||
@ -662,76 +635,3 @@ The Edit log form should have from top to bottom:
|
|||||||
There should be a tool that can look at the track associated with a scrobble and
|
There should be a tool that can look at the track associated with a scrobble and
|
||||||
the actual raw_data from mopidy or jellyfin, and flag all tracks where the
|
the actual raw_data from mopidy or jellyfin, and flag all tracks where the
|
||||||
artist or track name or both differ.
|
artist or track name or both differ.
|
||||||
** DONE [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
|
||||||
:PROPERTIES:
|
|
||||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
|
||||||
:END:
|
|
||||||
|
|
||||||
*** Description
|
|
||||||
|
|
||||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
|
||||||
|
|
||||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
|
||||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
|
||||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
|
||||||
enrich further.
|
|
||||||
|
|
||||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
|
||||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
|
||||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
|
||||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
|
||||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
|
||||||
log a warning and move on.
|
|
||||||
|
|
||||||
*** Implementation
|
|
||||||
|
|
||||||
Implemented for tracks (videos deferred for a follow-up). Jellyfin/Mopidy webhook scrobbles now:
|
|
||||||
|
|
||||||
- `Track.find_or_create(..., trust_webhook_data=True)` builds the track purely from POST data
|
|
||||||
with zero MusicBrainz lookups in the webhook path. Existing tracks found by `musicbrainz_id`
|
|
||||||
are reused without re-enriching; otherwise the track is stamped with the POST-provided
|
|
||||||
artist/album/track mbids and run time and tagged `musicbrainz-provider`.
|
|
||||||
- The scrobble is tagged `webhook-metadata-only` and a new background celery task
|
|
||||||
`scrobbles.tasks.enrich_media_from_webhook` is dispatched to enrich asynchronously using the
|
|
||||||
POST payload in `log["raw_data"]`.
|
|
||||||
- The async task runs the newly-implemented `Track.fix_metadata()` (MusicBrainz resolution, run
|
|
||||||
time, featured artists, album linkage). If resolution matches an existing track by
|
|
||||||
musicbrainz_id, `Track.merge()` repoints scrobbles/favorites onto the canonical track and
|
|
||||||
deletes the webhook-created duplicate (skipping any in-progress conflicts).
|
|
||||||
- Success tags the scrobble `webhook-enriched`; failure tags it `enrichment-failed` and logs a
|
|
||||||
warning. Notifications still blast out with the raw-data track at scrobble time; a later
|
|
||||||
repoint to a canonical track may show in a subsequent stop notification, which is accepted.
|
|
||||||
|
|
||||||
Also fixed a typo in `MOPIDY_POST_KEYS["ARTIST_MB_ID"]` (`muscibrainz_artist_id` →
|
|
||||||
`musicbrainz_artist_id`). The slow, subtler matching rules (e.g. a future
|
|
||||||
`LITERAL_ARTIST_TITLES` exception list) can now live in `fix_metadata` off the request path.
|
|
||||||
|
|
||||||
** DONE [#A] Fix celery task prioritization especially for agent sessions :celery:tasks:agents:
|
|
||||||
:PROPERTIES:
|
|
||||||
:ID: 672c81bf-bba9-a963-e8ac-52246d976cea
|
|
||||||
:END:
|
|
||||||
|
|
||||||
*** Description
|
|
||||||
|
|
||||||
Split Celery work into four queues so time-sensitive tasks run fast even when
|
|
||||||
long batch jobs are queued up:
|
|
||||||
|
|
||||||
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy
|
|
||||||
queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated
|
|
||||||
high-concurrency worker with `--prefetch-multiplier=1`.
|
|
||||||
- `charts` — per-scrobble chart updates stay on their own queue, no longer
|
|
||||||
blocked by scheduled chart rebuilds.
|
|
||||||
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill,
|
|
||||||
chart rebuilds, historical imports, twitch VOD check) run on a dedicated
|
|
||||||
low-concurrency worker so they can't starve the others.
|
|
||||||
- `default` — everything else (imports, notifications, archivebox, expansions).
|
|
||||||
|
|
||||||
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks
|
|
||||||
were published to Celery's default `celery` queue, which no worker consumed.
|
|
||||||
New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background`
|
|
||||||
added alongside the existing `vrobbler_celery` (now `default,charts`).
|
|
||||||
|
|
||||||
** DONE [#A] Fix lastfm rate limiting dropping scrobbles :importers:scrobbles:lastfm:
|
|
||||||
:PROPERTIES:
|
|
||||||
:ID: e6718364-8f9e-d156-3a79-2e89b2ac697b
|
|
||||||
:END:
|
|
||||||
|
|||||||
@ -1,6 +1,6 @@
|
|||||||
[tool.poetry]
|
[tool.poetry]
|
||||||
name = "vrobbler"
|
name = "vrobbler"
|
||||||
version = "65.0"
|
version = "65.1"
|
||||||
description = ""
|
description = ""
|
||||||
authors = ["Colin Powell <colin@unbl.ink>"]
|
authors = ["Colin Powell <colin@unbl.ink>"]
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user