22 KiB
Vrobbler Project
- Overview
- Backlog
[2/26]- After transition to linux add curl_cffi as webpage scrapper again
- Create small utility to clean up tracks scrobbled with wonky playback times
- Move to using more robust mopidy-webhooks pacakge form pypi
- Allow auto trail tracking via email with Garmin LiveTrack URLs
- Fix exporting so it works reliably
- Add AllTrails as a source for Trail data
- Add `garmin_activity_id` to the TrailLogData class
- Fix how we show notes and descriptions from scrobbles to users
- Add CSV endpoint for book scrobbles that LibraryThing can ingest
- Make IMAP and WebDAV configurable
- Add an exception list of artists as a constant that are exempted from splitting
- Before enriching anything, trust the POST data
- Allow browing a user's favorited media
- Find page numbers for comic books from ComicVine
- Implement loguru into project
- Scrape ComicBookRoundUp ratings for comic book metadata
- Make podcast date format configurable in settings
- Extract zombie scrobble query into custom manager
- Allow profile to set start of week
- Add constants for data dictionary keys (multiple files)
- Is there way to create unique slugs for media instances
- Update how board game scrobbles work
- Auto sync board game scrobbles to BGG
- Add trends tests for concurrent trends
- Fix celery task prioritization especially for agent sessions
- Fix lastfm rate limiting dropping scrobbles
We should convert this PROJECT file to put tickets in a subdirectory, tickets, with each ticket having it's own shortid_title.org
Overview
Vrobbler began humbly enough as a way to use Jellyfin's webhook to keep track of the shows and movies I was watching. More specifically, I broke my ankle a few days after Christmas in 2022 and spent the next four months very slowly recovering after surgical repair. So once I had the webhook working, and scrobbling videos, it was only a matter of time till I expaned it to mopidy to replicate LastFM. Then I added board games, books via KoReader, sports events, podcasts … it just keeps going. Vrobbler is now a sort of Frankenstein's monster of scrobbling an entire life.
I am still unconvinced I can keep this going, but being able to scrobble org tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out to be sometimes cathartic and sometimes functional as I try to remember when I did a thing.
Backlog [2/26] vrobbler project personal
TODO [C] After transition to linux add curl_cffi as webpage scrapper again webpages metadata
TODO [C] Create small utility to clean up tracks scrobbled with wonky playback times bug music scrobbles
TODO [C] Move to using more robust mopidy-webhooks pacakge form pypi utility improvement
Example payloads from mopidy-webhooks
Podcast playback ended
{
"type": "event",
"event": "track_playback_ended",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
},
"time_position": 3290
}
}Podcast playback state changes
{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "paused",
"new_state": "playing"
}
}{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "stopped",
"new_state": "playing"
}
}Podcast playback started
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
}
}
}Podcast playback paused
{
"type": "status",
"data": {
"state": "paused",
"current_track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
},
"time_position": 2350
}
}Track playback started
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
}
}
}Track playback in progress
{
"type": "status",
"data": {
"state": "playing",
"current_track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
},
"time_position": 17031
}
}Track event playback paused
{
"type": "event",
"event": "track_playback_paused",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
},
"time_position": 67578
}
}TODO [C] Allow auto trail tracking via email with Garmin LiveTrack URLs trails feature
TODO [C] Fix exporting so it works reliably exporting feature
Description
The existing export function is very naieve. It runs in the web process, takes too long and just dumps tracks. We should make it more robust by creating one CSV file per scrobble media type and writing them into a zip file that gets placed in the media directory:
`/media/exports/user_<user_id>/<timestamp>-export.zip`
And this should all be done in a celery task that is just kicked off by the "Export" button on the frontend
TODO [B] Add AllTrails as a source for Trail data trails feature
Description
Pretty clear, I would love to make trails more useful. Historically I wasn't hiking a lot, which made the source for this a bit silly. But it's clear that AllTrails is the best source, though having TrailForks is nice to.
TODO [B] Add `garmin_activity_id` to the TrailLogData class trails feature
Description
Would be nice to have some loose connection to the actual event in my Garmin profile.
TODO [B] Fix how we show notes and descriptions from scrobbles to users metadata notes tasks
Description
Currently the display of notes leaves something to be desired. The biggest issue is that they don't look good on mobile and are probably trying to be too cute. Rather than post-it note style, we should just put notes in a list under the description, above the Edit Log toggle, with timestamps for when they were added.
They should also probably support markdown formatting and that should be displayed in the template.
TODO [B] Add CSV endpoint for book scrobbles that LibraryThing can ingest books feature export
TODO [B] Make IMAP and WebDAV configurable webdav feature imap importers
Description
Currently we have webdav able to import post types of file-based incoming data, usually in the form of CSVs but also gpx files, bgstats json files, and audioscrobbler TSV files.
What if the user could specify via their profile (settings) which imports they wanted to use IMAP for and which ones they wanted to use WebDAV for.
Then we'd have two celery tasks that would be kicked off periodically via celerybeat, one for IMAP imports every 12 minutes and one for WebDAV every 3 minutes. Both would be responsible for checking if a user has an configured imports of their type, check if an import needs to run, and dispatch the needed import celery task. This is how the WebDAV celery task currently works.
This would also be an opporunity to clean up the code around WebDAV imports and make them more re-usable for other import services.
TODO [A] Add an exception list of artists as a constant that are exempted from splitting music artists metadata
Description
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like this put into it and then we stop trying to pull the artist apart when we run into it.
TODO [A] Before enriching anything, trust the POST data feature scrobbles metadata
Description
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
In most cases, we should be able to trust this data to created music tracks or videos rather than going to third-party services to enrich. Thus, for tracks and videos we should search in the local database for imdb_id or musicbrainz_id for the specific content and, if found, not enrich further.
If not found, tracks and videos from mopidy and jellyfin should be created as completely as possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only" and start the scrobble. A separate celery task should be kicked off to enrich the track or video async with the POST data stored in the log["raw_data"] and used by the celery enrichment task to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed" log a warning and move on.
TODO [B] Allow browing a user's favorited media favorites feature
Description
We should have a global view `/favorites/` that shows the logged in users's favorited media objects.
TODO [B] Find page numbers for comic books from ComicVine feature books
TODO [C] Implement loguru into project feature loguru logging
Description
Would be great to formalize how we log so we can search for errors and such more easily. And our exposure to PII is really low at this point in the project, so we can probably use backtrace=True and diagnose=True to help us root cause bugs faster.
TODO [B] Scrape ComicBookRoundUp ratings for comic book metadata books feature comicbook
Description
TODO [C] Make podcast date format configurable in settings podcasts configuration
Description
PODCAST_DATE_FORMAT is hardcoded to "YYYY-MM-DD". Should be in Django settings or environment variables for deploy-specific configuration.
File: vrobbler/apps/podcasts/utils.py (line 13)
TODO [C] Extract zombie scrobble query into custom manager refactoring manager
Description
The zombie scrobble cleanup query lives in a utility function. Should be a
custom model manager method (e.g. Scrobble.objects.zombies()).
File: vrobbler/apps/scrobbles/utils.py (line 204)
TODO [C] Allow profile to set start of week profiles configuration
Description
start_of_week() and end_of_week() use Monday as default. Should be a user
profile setting for different cultural week start conventions.
File: vrobbler/apps/profiles/utils.py (lines 39, 44)
TODO [C] Add constants for data dictionary keys (multiple files) refactoring constants
Description
Multiple files use magic string literals for dict keys. Should be extracted to named constants for maintainability.
-
Files:
vrobbler/apps/locations/models.py(line 63) –"lat","lon"etc.vrobbler/apps/webpages/models.py(line 290) –"url"vrobbler/apps/scrobbles/importers/tsv.py(line 55) –"S"completion status
TODO [B] Is there way to create unique slugs for media instances media_types
TODO [A] Update how board game scrobbles work boardgames
Description
When we scrobble a board game from a BGG URL, instead of going to the media detail page, we should go to the scrobble detail page, with the Edit Log form expanded by default.
The Edit log form should have from top to bottom:
- Board/Variant (one or many BoardGameVariant in a multi-select widget)
- People (which should be similar to the Bird widget on BirdLocation and allow setting per user score, win true/false, rank, new true/false, seat_ordrer)
- Expansion ids (which should a multi-select widget of expansions for this game)
- Location (which should be a drop down of BoardGameLocations for this user)
TODO [B] Auto sync board game scrobbles to BGG boardgames
Description
TODO [A] Add trends tests for concurrent trends trends tests concurrent
Description
Added a `workouts` Django app so we can scrobble gym sessions.
- New `Exercise` model with the wrkout/exercises.json catalog (873 exercises), imported via `python manage.py import_wrkout_exercises`. Image files come from inside the tarball and are stored via ImageKit specs.
- New `WorkoutRoutine` model (ScrobblableMixin, media_type_label "Workout"), scrobbled through the existing Scrobble system with a `workout_routine` FK and `WorkoutLogData` storing sets/reps/weight per exercise plus duration, bodyweight and RPE.
- Weights are stored canonically in kg; the profile's `weigh_in_units` setting drives form input and display units. A generic `apply_media_unit_conversions` helper in scrobbles/utils.py handles both drinks (`size_ml`) and workout weights, fixing the old GET-only drinks conversion so imperial POSTs store kg.
- Templates, admin, DRF viewsets (`exercises`, `workout-routines`), MCP tools, and `tests/workouts_tests/` (unit conversion, form round-trips, importer, imperial POST) included.
DONE [A] Fix celery task prioritization especially for agent sessions celery tasks agents
Description
Split Celery work into four queues so time-sensitive tasks run fast even when long batch jobs are queued up:
- `priority` — interactive/user-facing tasks (agent session prompts, Mopidy queue/playlist adds, favorite toggles, reverse geocoding) get a dedicated high-concurrency worker with `–prefetch-multiplier=1`.
- `charts` — per-scrobble chart updates stay on their own queue, no longer blocked by scheduled chart rebuilds.
- `background` — heavy/scheduled jobs (db backup, trends, sentiment backfill, chart rebuilds, historical imports, twitch VOD check) run on a dedicated low-concurrency worker so they can't starve the others.
- `default` — everything else (imports, notifications, archivebox, expansions).
Also set `CELERY_TASK_DEFAULT_QUEUE = "default"` — previously un-routed tasks were published to Celery's default `celery` queue, which no worker consumed. New rc.d scripts `vrobbler_celery_priority` and `vrobbler_celery_background` added alongside the existing `vrobbler_celery` (now `default,charts`).