Compare commits

...

117 Commits

Author SHA1 Message Date
d5830f5cd1 [release] Bump to version 51.2
All checks were successful
build / test (push) Successful in 2m14s
deploy / test (push) Successful in 2m15s
deploy / build-and-deploy (push) Successful in 1m27s
- Fix bug where last page of book gets separate scrobble
- Fix metadata scraping for books
2026-06-12 09:52:56 -04:00
c71b51fdb8 [books] Fix one-extra-scrobble bug on Koreader imports
Some checks failed
build / test (push) Has been cancelled
2026-06-12 09:52:33 -04:00
935d059a20 [books] Fix metadata scrapping
All checks were successful
build / test (push) Successful in 2m30s
It's not perfect, but at least we get covers now
2026-06-12 09:36:21 -04:00
25776eb495 [release] Bump to version 51.1
All checks were successful
build / test (push) Successful in 2m55s
deploy / test (push) Successful in 2m38s
deploy / build-and-deploy (push) Successful in 44s
- Fix scrobbling comic books
2026-06-11 19:07:00 -04:00
5ac4625af9 [books] Fix bug in scrobbling comic books
Some checks failed
build / test (push) Has been cancelled
2026-06-11 19:06:37 -04:00
a731427f6e [release] Bump to version 51.0
All checks were successful
build / test (push) Successful in 2m52s
deploy / test (push) Successful in 2m25s
deploy / build-and-deploy (push) Successful in 49s
- Fix koreader scrobble imports to use DST properly
- Fix book scrobbles where page_data is a list
- Lichess imports do not set default visbility
2026-06-11 18:39:31 -04:00
410da163fe [books] Fix koreader imports, maybe forever 2026-06-11 18:38:51 -04:00
a171192a6f [books] Fix list page_data for old book scrobbles
All checks were successful
build / test (push) Successful in 2m2s
2026-06-11 11:00:04 -04:00
c16b61db40 [importers] Fix strange bug where celery version mismatches
All checks were successful
build / test (push) Successful in 2m9s
2026-06-11 10:48:38 -04:00
29cb6a4991 [books] Really fix the subtitle bug 2026-06-11 10:48:21 -04:00
25c28e8335 [books] Fix bug in subtitle gen
All checks were successful
build / test (push) Successful in 2m5s
2026-06-11 10:34:20 -04:00
25626be3b6 [boardgames] Add visibility to lichess imports 2026-06-11 10:02:37 -04:00
0a880a2f2f [release] Bump to version 50.2
All checks were successful
build / test (push) Successful in 2m6s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Successful in 35s
- Koreader imports only import single-page scrobbles the next day
- Fix bugs in celery tasks causing imports to fail
2026-06-11 09:58:56 -04:00
248d3f2d3e [settings] Check webdav every two minuts
All checks were successful
build / test (push) Successful in 2m13s
2026-06-11 09:41:50 -04:00
e243fec679 [books] Try fixing the one-off import issue 2026-06-11 09:41:19 -04:00
de9b4ee9c1 [release] Bump to version 50.1
All checks were successful
build / test (push) Successful in 2m2s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 32s
- Fix bug in charts where only #1 is displayed
2026-06-09 22:08:48 -04:00
bf9a6a9679 [charts] Fix only seeing first top media instance 2026-06-09 22:08:15 -04:00
709fed5cfe [release] Bump to version 50.0
All checks were successful
build / test (push) Successful in 2m7s
deploy / test (push) Successful in 2m6s
deploy / build-and-deploy (push) Successful in 31s
- Allow updating all a user's scrobble visibility at once
- Replace columsn of Top Artists, Tracks and Series with Maloja widget
2026-06-09 17:19:10 -04:00
b7df6299d0 [sharing] Add bulk scrobble share management 2026-06-09 17:18:51 -04:00
be16d513ef [charts] Add better chart views per Maloja
All checks were successful
build / test (push) Successful in 2m3s
2026-06-09 17:04:59 -04:00
15d27f6d94 [release] Bump to version 49.1
All checks were successful
build / test (push) Successful in 2m7s
deploy / test (push) Successful in 2m3s
deploy / build-and-deploy (push) Successful in 33s
- Fix bug with missing default visbility for scrobbles
2026-06-09 13:16:53 -04:00
c8292d1c06 [scrobbles] Back fill visibility field
Some checks failed
build / test (push) Has been cancelled
2026-06-09 13:15:57 -04:00
68f821fce1 [release] Bump to version 49.0
Some checks failed
build / test (push) Successful in 1m59s
deploy / test (push) Successful in 2m9s
deploy / build-and-deploy (push) Failing after 2m20s
- Fix broken tests with new sharing and add tests
2026-06-09 12:48:22 -04:00
ed2ed59f65 [scrobbles] Fix tests around visbility 2026-06-09 12:48:01 -04:00
17a7bb52fa [release] Bump to version 48.3
Some checks failed
build / test (push) Failing after 1m39s
deploy / test (push) Failing after 1m39s
deploy / build-and-deploy (push) Has been skipped
- Fix bug in missing sqids dep
2026-06-09 12:37:40 -04:00
bbac142b40 [deps] Add sqids
Some checks failed
build / test (push) Has been cancelled
2026-06-09 12:37:19 -04:00
5f55ec557f [release] Bump to version 48.2
Some checks failed
build / test (push) Failing after 1m21s
deploy / test (push) Failing after 1m18s
deploy / build-and-deploy (push) Has been skipped
- Lock down scrobbles and use sqids to share them
2026-06-09 12:33:50 -04:00
7f3076608f [scrobbles] Add sharing of scrobbles
Some checks failed
build / test (push) Has been cancelled
2026-06-09 12:33:25 -04:00
568772a0e6 [release] Bump to version 48.1
All checks were successful
deploy / test (push) Successful in 2m0s
build / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 35s
- Generate a report of tracks with mistmatched metadata
- Date parsing failing in eBird imports
2026-06-08 11:18:08 -04:00
91c3376256 [music] Add mgmt command to see mismatched metadata
Some checks failed
build / test (push) Has been cancelled
2026-06-08 11:17:36 -04:00
58639c6fc1 [importers] Add error_log to surface import errors
All checks were successful
build / test (push) Successful in 2m2s
2026-06-08 10:58:02 -04:00
228441ddc5 [importers] Fix parsing of dates in ebird files 2026-06-08 10:52:50 -04:00
6341075f07 [agent] Add some notes about how we track tasks 2026-06-08 10:52:32 -04:00
a135b9f5f2 [project] Update format for PROJECT file
All checks were successful
build / test (push) Successful in 1m58s
2026-06-07 11:10:30 -04:00
9088412d1e [release] Bump to version 48.0
All checks were successful
build / test (push) Successful in 2m1s
deploy / test (push) Successful in 2m2s
deploy / build-and-deploy (push) Successful in 33s
- Show team or player images on sport detail and scrobble detail
- Add fix_metadta method to Video instances
2026-06-07 11:06:58 -04:00
c7339fbe31 [templates] Clean up how str and subtitles work 2026-06-07 11:06:30 -04:00
4ce3dc03c5 [videos] Add fix_metadata for videos 2026-06-07 10:13:55 -04:00
5a4ef678a8 [release] Bump to version 47.2
All checks were successful
build / test (push) Successful in 1m58s
deploy / test (push) Successful in 1m58s
deploy / build-and-deploy (push) Successful in 53s
- Add OMDB source as backup when TMDB returns nothing
2026-06-07 09:59:05 -04:00
5ca22efeaa [videos] Fix full metadata from OMDB
We were missing the series and episode number info.
2026-06-07 09:58:28 -04:00
912ea8bfac [videos] Add OMDB enrichment when TMDb fails 2026-06-07 09:47:52 -04:00
b541e1084d [release] Bump to version 47.1
All checks were successful
build / test (push) Successful in 2m0s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 32s
- Untangle the sports migrations errors
2026-06-06 23:47:25 -04:00
c9b9da4abc [sports] Fix migrations 2026-06-06 23:47:10 -04:00
8236f43026 [release] Bump to version 47.0
Some checks failed
build / test (push) Successful in 2m4s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Failing after 46s
- Change sports scrobbling a bit
2026-06-06 23:35:30 -04:00
ea1b43d1b8 [sports] Big sports structure revamp
Some checks failed
build / test (push) Has been cancelled
This should make scrobbling sports more like tasks.

The root scrobbled items are a little more generic, but
it's easier to see viewing patterns.
2026-06-06 23:32:21 -04:00
4bf22c96e9 [release] Bump to version 46.0
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 2m9s
deploy / build-and-deploy (push) Successful in 43s
- Add sentiment parsing for Scrobbles with notes
2026-06-05 19:42:21 -04:00
dec7a79509 [scrobbles] Add basic sentiment analysis
All checks were successful
build / test (push) Successful in 2m4s
2026-06-05 19:35:45 -04:00
371e1d654c [tooling] Release in one step
All checks were successful
build / test (push) Successful in 1m57s
2026-06-05 14:58:35 -04:00
bef7e683c5 [release] Bump to version 45.1
All checks were successful
build / test (push) Successful in 1m58s
deploy / test (push) Successful in 2m2s
deploy / build-and-deploy (push) Successful in 31s
- Mopidy favorites or monthly playlist adds should look at all scrobbles
2026-06-05 14:42:11 -04:00
ec219ef3ea [tracks] Fix adding tracks without mopidy_uri 2026-06-05 14:41:37 -04:00
dcc7229e90 [tooling] Just release does it all now
All checks were successful
build / test (push) Successful in 1m57s
2026-06-05 14:06:45 -04:00
73665ef19e [release] Bump to version 45.0
All checks were successful
build / test (push) Successful in 2m2s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Successful in 33s
- Add ability to add mopidy tracks to Monthly playlists
2026-06-05 13:57:06 -04:00
2536e330af [tracks] Use todays date for creating monthly playlists
All checks were successful
build / test (push) Successful in 2m3s
2026-06-05 13:41:30 -04:00
99c056adeb [tracks] Allow adding tracks to monthly playlists 2026-06-05 13:29:49 -04:00
7a504e45de [release] Bump to version 44.0
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 44s
- Add favorite feature for scrobbles
2026-06-05 11:26:39 -04:00
7618d0ba30 [tooling] Add full push back to justfile
Some checks failed
build / test (push) Has been cancelled
2026-06-05 11:25:48 -04:00
ce4dc40033 [favorites] Add ability to favorite and add to Mopidy
Some checks failed
build / test (push) Has been cancelled
2026-06-05 11:24:26 -04:00
b0b22b79dc [release] Bump to version 43.0
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 1m56s
deploy / build-and-deploy (push) Successful in 33s
- Can we show a graph of all past Weigh-in tasks
- When viewing scrobbles by tag, sum the total time
- Orgmode tasks are not updated if in progress
- Ignore tag 'inprogress' for Tasks
- Deploys are now throwing an unknown version error
2026-06-04 22:13:36 -04:00
6471413681 [tasks] Add weigh in graph 2026-06-04 22:13:14 -04:00
50b10689fc [scrobbles] Add total time to tag views 2026-06-04 22:01:35 -04:00
85bddb6cba [tasks] Better updating of org mode tasks 2026-06-04 15:16:44 -04:00
c285b0d3b3 [tasks] Exclude inpgrogress tag, they're always inprogress
All checks were successful
build / test (push) Successful in 2m14s
2026-06-04 14:45:10 -04:00
671fe8d86f [tooling] Fix releases once and for all 2026-06-04 14:44:19 -04:00
89817110de [release] Bump to version 42.0
Some checks failed
deploy / test (push) Successful in 2m6s
build / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Failing after 26s
- Add ability to add track to current Mopidy queue
2026-06-04 13:13:21 -04:00
ee01e3d8df [tracks] Add mopidy queue button
All checks were successful
build / test (push) Successful in 2m2s
2026-06-04 13:11:21 -04:00
a70343d6f3 [release] Bump to version 41.0
Some checks failed
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 1m58s
deploy / build-and-deploy (push) Failing after 34s
- For any scrobble detail page with notes display them better
- Imports should send notifications
- Board game imports send duplicate ntfy message
- Too many geolocation notifications go out
- Fix bug where Weigh-in imports do not set title
2026-06-04 11:22:48 -04:00
3e72042c24 [justfile] push for release should only push tags
Some checks failed
build / test (push) Has been cancelled
2026-06-04 11:22:14 -04:00
087c7775ae [templates] Fix note parsing
Some checks failed
build / test (push) Has been cancelled
2026-06-04 11:15:58 -04:00
3f71065ad6 [notifications] Send ntfy on more imports
All checks were successful
build / test (push) Successful in 2m6s
2026-06-04 10:50:32 -04:00
801672124f [notifications] Fix duplicate ntfy for board games
All checks were successful
build / test (push) Successful in 1m58s
2026-06-04 10:47:22 -04:00
811e9c1ce9 [notfications] Dont send ntfy on non-titled geolocations
All checks were successful
build / test (push) Successful in 2m7s
2026-06-04 10:44:21 -04:00
415b32bdc7 [tasks] Scale task sets title properly
All checks were successful
build / test (push) Successful in 1m57s
2026-06-03 16:49:11 -04:00
22319c807a [tooling] Fix deploys for realz
All checks were successful
build / test (push) Successful in 2m0s
2026-06-03 16:38:10 -04:00
f9ba6fec14 [release] Bump to version 40.2
Some checks failed
build / test (push) Successful in 2m0s
deploy / test (push) Successful in 1m59s
deploy / build-and-deploy (push) Failing after 25s
- Try fixing deploy bugs again
2026-06-03 16:32:09 -04:00
5f55163147 [tooling] Try fixing deploys one more time 2026-06-03 16:31:09 -04:00
a6ef34623e [release] Bump to version 40.1
Some checks failed
deploy / test (push) Successful in 2m5s
build / test (push) Successful in 2m6s
deploy / build-and-deploy (push) Failing after 28s
- Releases are still broken
- Fix bug on chart pages where trail titles missing
2026-06-03 16:13:54 -04:00
7cb48d20f6 [tooling] Try to fix deploy issues
Some checks failed
build / test (push) Has been cancelled
2026-06-03 16:13:16 -04:00
445103a878 [trails] Fix display of trails in charts
All checks were successful
build / test (push) Successful in 2m7s
2026-06-03 15:47:56 -04:00
579da8c44e [release] Bump to version 40.0
Some checks failed
build / test (push) Successful in 1m59s
deploy / test (push) Successful in 2m3s
deploy / build-and-deploy (push) Failing after 19s
- Fix error in org-mode task sync
- Adjust how similar artists are shown
2026-06-01 11:12:31 -04:00
daabd2f37f [tasks] Fix bug in syncing orgmode tasks without notes
All checks were successful
build / test (push) Successful in 1m50s
2026-06-01 11:02:56 -04:00
039c58cf89 [tooling] Try fixing deploys again
All checks were successful
build / test (push) Successful in 1m53s
2026-06-01 10:50:38 -04:00
410c033f12 [music] Fix slow loading artist page 2026-06-01 10:50:22 -04:00
ce302e4d45 [release] Bump to version 39.3
Some checks failed
build / test (push) Successful in 1m53s
deploy / test (push) Successful in 1m55s
deploy / build-and-deploy (push) Failing after 19s
- Issue found when doing a release
- Fix deploy actions running twice
2026-06-01 10:35:51 -04:00
19589c9463 [tooling] Fix failing deploys 2026-06-01 10:35:10 -04:00
3d9506b14e [tooling] Actulaly fix the release problems
All checks were successful
build / test (push) Successful in 2m1s
Turns out we need build and deploy in separate files to trigger at
different times. Now we build on all pushes, but only deploy on tag pushes.
2026-06-01 10:21:56 -04:00
23b87278b2 [tooling] Stop running release task twice 2026-06-01 10:12:00 -04:00
0b8e027c30 [release] Bump to version 39.2
Some checks failed
build & deploy / test (push) Successful in 1m59s
build & deploy / build-and-deploy (push) Failing after 24s
- Releases do not pin commit to the repo for display
- Fix the way timestamps are stored for notes on tasks
2026-06-01 10:11:02 -04:00
1bd9f0d942 [tooling] Fix commit stamping on release
All checks were successful
build & deploy / test (push) Successful in 1m56s
build & deploy / build-and-deploy (push) Has been skipped
2026-06-01 09:58:59 -04:00
fa7890cb21 [tasks] Fix timestamps for notes and syncing from org-mode (32973bb3)
All checks were successful
build & deploy / test (push) Successful in 1m54s
build & deploy / build-and-deploy (push) Has been skipped
2026-06-01 09:37:41 -04:00
957c32e3a7 [release] Bump to version 39.1
All checks were successful
build & deploy / test (push) Successful in 1m56s
build & deploy / build-and-deploy (push) Has been skipped
- Fix bug in tests for notes saving
2026-05-31 23:25:34 -04:00
8d069df9d1 [scrobbles] Fix log saving tests 2026-05-31 23:24:22 -04:00
96d1d7ac6b [release] Bump to version 39.0
Some checks failed
build & deploy / test (push) Failing after 1m43s
build & deploy / build-and-deploy (push) Has been skipped
- Clean up org-mode tasks metadata
- Actually push branches up and add a just command to do it
- Try to fix deploy failing with bad release plan
2026-05-31 23:17:52 -04:00
009b2ba243 [scrobbles] Big update for notes in tasks and boardgames
Some checks failed
build & deploy / build-and-deploy (push) Has been cancelled
build & deploy / test (push) Has been cancelled
2026-05-31 23:17:16 -04:00
4f051ae250 [tooling] Add push command to just
All checks were successful
build & deploy / test (push) Successful in 2m7s
build & deploy / build-and-deploy (push) Has been skipped
2026-05-31 13:56:57 -04:00
7e9fbb1bf6 [tooling] Try to fix CI script
All checks were successful
build & deploy / test (push) Successful in 1m58s
build & deploy / build-and-deploy (push) Has been skipped
2026-05-31 13:10:31 -04:00
ec1a54f623 [release] Bump to version 38.0
All checks were successful
build & deploy / test (push) Successful in 2m41s
build & deploy / build-and-deploy (push) Successful in 21s
- Fix release flow to be easier to trigger
- Move imported retroarch lrtl files to processed/ directory on WebDAV
- Add listenbrainz support for similar tracks
- Consolidate albums in the same musicbrainz_releasegroup_id
- Clean up metadata on music tracks
- Make artists_m2m field source of artist truth for albums
- Fix various artist album problem with Superwolves (track with multiple artists)
- Move imported eBird CSV files to processed/ directory on WebDAV
- Move imported Board Game CSV files to processed/ directory on WebDAV
- Move imported Scale CSV files to processed/ directory on WebDAV
- Allow special parameter to re-import already processed GPX files
- Move imported GPX files to processed/ directory on WebDAV
- Add CSS Grid calendar view for scrobbles
- Come up with a possible flow using WebDAV and super-productivity for tasks
- Fix PuzzleLogData has no attribute form
- Add PuzzleLogData class with with_people and completed
- Add weather lookup to the mood check-in flow
- Add importing of openScale CSV files to Tasks
- Add ability to track Birding sessions via BirdingLocation scrobbles
- List only the last 20 scrobbles per category on the home page
- Fix display of notes so they look like stickies
- Add searching to scrobbles
- Fix uniqueness of imdb_id messing up youtube videos
- Fix genearting chart records
- Save raw scrobble request data to every scrobble log
- Clean up follow up notifications for if you're still scrobbling
- Fix lookup of music tracks from Musicbrainz
- Check opencode about a way to present stats like movies per month
- Fix bug in Jellyfin audio track playback
- Auto calc duration if no playback time seconds present
- Fix bug in video find_or_create
- Update admin page to be easier to use
- Fix migrations and update repo
- Add recipe parsing for food lookups
- Videos are scrobbling duplicates again
- Fix board games not saving BGG id on lookup
- Fix board game lookup with name like Unmatched Game System
- Fix raw text webpage title not truncating to 254 chars
2026-05-31 12:58:31 -04:00
b502667ca6 [importers] Retroarch webdav imports oldest first 2026-05-31 12:35:39 -04:00
263874288a [importers] WebDAV for retroarch should stash in procesesd now
All checks were successful
build & deploy / test (push) Successful in 2m18s
build & deploy / build-and-deploy (push) Successful in 49s
2026-05-31 12:09:56 -04:00
bca90c97ae [music] Add listenbrainz support 2026-05-31 11:21:11 -04:00
12f49a6cee [project] Update tasks
All checks were successful
build & deploy / test (push) Successful in 2m0s
build & deploy / build-and-deploy (push) Successful in 29s
2026-05-29 19:47:52 -04:00
034c7cb413 [music] Add better track fetching 2026-05-29 19:47:09 -04:00
e737870733 [music] Add metadata cleanup command
All checks were successful
build & deploy / test (push) Successful in 2m2s
build & deploy / build-and-deploy (push) Successful in 31s
2026-05-29 17:58:23 -04:00
95757650f6 [project] Add metadata clean up tasks 2026-05-29 08:37:10 -04:00
1928acf8a6 [music] Make artists_m2m field source of artist truth (05a24455)
All checks were successful
build & deploy / test (push) Successful in 2m0s
build & deploy / build-and-deploy (push) Successful in 32s
2026-05-29 08:26:30 -04:00
22956c7c7f [music] Tracks can have multiple artists
All checks were successful
build & deploy / test (push) Successful in 1m54s
build & deploy / build-and-deploy (push) Successful in 33s
2026-05-28 23:37:46 -04:00
17aed1191d [videos] Remove unused source
All checks were successful
build & deploy / test (push) Successful in 2m2s
build & deploy / build-and-deploy (push) Successful in 30s
2026-05-28 23:26:49 -04:00
0b1fb8667c [deps] Pin lxml version and upgrade python
Some checks failed
build & deploy / test (push) Successful in 1m52s
build & deploy / build-and-deploy (push) Failing after 2m20s
2026-05-28 23:16:27 -04:00
13dd5b67d0 [importers] Add processed stashing to eBird imports
All checks were successful
build & deploy / test (push) Successful in 1m55s
build & deploy / build-and-deploy (push) Successful in 28s
2026-05-28 17:40:19 -04:00
20c7874466 [importers] Add processed directory flow to board games
All checks were successful
build & deploy / test (push) Successful in 1m56s
build & deploy / build-and-deploy (push) Successful in 30s
2026-05-28 17:22:56 -04:00
62d8ffd794 [importers] Add flag to reimport already processed files
All checks were successful
build & deploy / test (push) Successful in 2m1s
build & deploy / build-and-deploy (push) Successful in 29s
2026-05-28 17:19:08 -04:00
bea2b2d187 [importers] Fix scale file checker 2026-05-28 17:05:48 -04:00
034cb99c77 [charts] Add timemachine links
All checks were successful
build & deploy / test (push) Successful in 2m4s
build & deploy / build-and-deploy (push) Successful in 35s
2026-05-28 16:57:57 -04:00
37187f33dd [settings] WebDAV is fast again, back to 3 minutes
All checks were successful
build & deploy / test (push) Successful in 1m58s
build & deploy / build-and-deploy (push) Successful in 1m3s
2026-05-28 10:00:11 -04:00
d7a23c3832 [charts] Clean up some chart views
All checks were successful
build & deploy / test (push) Successful in 1m58s
build & deploy / build-and-deploy (push) Successful in 29s
2026-05-27 22:08:58 -04:00
6d45571e75 [templates] Have some fun with CSS
All checks were successful
build & deploy / test (push) Successful in 2m14s
build & deploy / build-and-deploy (push) Successful in 32s
2026-05-27 21:53:03 -04:00
88fd0ed7f8 [moods] Checkin notification should use correct link 2026-05-27 21:52:40 -04:00
2100cedc1a [calendar] Fix no cal scroll
All checks were successful
build & deploy / test (push) Successful in 1m52s
build & deploy / build-and-deploy (push) Successful in 30s
2026-05-27 14:58:10 -04:00
2b17a92c6c [calendar] Fix bug in query
All checks were successful
build & deploy / test (push) Successful in 1m57s
build & deploy / build-and-deploy (push) Successful in 27s
2026-05-27 14:46:04 -04:00
141 changed files with 8661 additions and 1101 deletions

View File

@ -0,0 +1,68 @@
name: build
on:
push:
branches: ["**"]
pull_request:
jobs:
test:
runs-on: ubuntu-latest
env:
VROBBLER_DATABASE_URL: sqlite:///test.db
VROBBLER_USDA_API_KEY: ${{ vars.VROBBLER_USDA_API_KEY }}
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Setup Python
uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Cache pip/poetry
uses: actions/cache@v4
with:
path: |
~/.cache/pip
~/.cache/pypoetry
key: ${{ runner.os }}-py311-${{ hashFiles('**/poetry.lock') }}
restore-keys: |
${{ runner.os }}-py311-
- name: Install Poetry
run: |
python -m pip install --upgrade pip
pip install poetry
- name: Install deps
run: |
cp vrobbler.conf.test vrobbler.conf
poetry install --with test
- name: Pytest with coverage
run: |
poetry run pytest -n 5 --cov-report term:skip-covered --cov=vrobbler tests
- name: Notify success (ntfy)
if: success()
run: |
curl -fsS \
-H "Title: vrobbler CI success" \
-H "Priority: low" \
-H "Tags: success,vrobbler" \
-H "Actions: view, Changes, ${{ gitea.server_url }}/${{ gitea.repository }}/commit/${{ gitea.sha }}; view, Build, ${{ gitea.server_url }}/${{ gitea.repository }}/actions/runs/${{ gitea.run_id }}" \
-d "✅ Build succeeded: ${{ gitea.repository }} @ ${{ gitea.sha }}" \
https://ntfy.unbl.ink/drone
- name: Notify failure (ntfy)
if: failure()
run: |
curl -fsS \
-H "Title: vrobbler CI failure" \
-H "Priority: high" \
-H "Tags: failure,vrobbler" \
-H "Actions: view, Changes, ${{ gitea.server_url }}/${{ gitea.repository }}/commit/${{ gitea.sha }}; view, Build, ${{ gitea.server_url }}/${{ gitea.repository }}/actions/runs/${{ gitea.run_id }}" \
-d "❌ Build failed: ${{ gitea.repository }} @ ${{ gitea.sha }}" \
https://ntfy.unbl.ink/drone

View File

@ -1,10 +1,8 @@
name: build & deploy
name: deploy
on:
push:
branches: ["**"]
tags: ["*"]
pull_request:
jobs:
test:
@ -22,7 +20,6 @@ jobs:
with:
python-version: "3.11"
# Cache pip + Poetry caches (rough equivalent to your mounted pip_cache)
- name: Cache pip/poetry
uses: actions/cache@v4
with:
@ -47,7 +44,6 @@ jobs:
run: |
poetry run pytest -n 5 --cov-report term:skip-covered --cov=vrobbler tests
# Notifications (success/failure) for the test job
- name: Notify success (ntfy)
if: success()
run: |
@ -71,8 +67,6 @@ jobs:
https://ntfy.unbl.ink/drone
build-and-deploy:
# Only deploy on tags (equivalent to Drone when: ref: refs/tags/*)
# if: startsWith(gitea.ref, 'refs/tags/')
needs: [test]
runs-on: ubuntu-latest
@ -97,12 +91,19 @@ jobs:
- name: Build package with commit info
run: |
# Write commit to _commit.py before build
echo "commit = '$(echo ${{ gitea.sha }} | cut -c1-8)'" > vrobbler/_commit.py
poetry build
# Restore original _commit.py
git checkout vrobbler/_commit.py
- name: Clean old wheels from server
uses: appleboy/ssh-action@v1.0.3
with:
host: vrobbler.service
username: root
key: ${{ secrets.JAIL_KEY }}
script: |
rm -f /var/lib/vrobbler/dist/*.whl
- name: Copy wheel to server and deploy
uses: appleboy/scp-action@v1.0.0
with:
@ -120,10 +121,13 @@ jobs:
key: ${{ secrets.JAIL_KEY }}
command_timeout: 2m
script: |
set -e
mkdir -p /var/lib/vrobbler
echo "${{ gitea.sha }}" | cut -c1-8 > /var/lib/vrobbler/commit.txt
pip uninstall -y vrobbler
pip install /var/lib/vrobbler/dist/*.whl
rm -f /var/lib/vrobbler/dist/*.whl
python3 -c "import vrobbler; print(f'vrobbler {vrobbler.__version__} installed OK')"
vrobbler migrate
vrobbler collectstatic --noinput
immortalctl restart vrobbler-celery && immortalctl restart vrobbler-celerybeat && immortalctl restart vrobbler

View File

@ -14,3 +14,11 @@ ro class method should call the utility function.
Be sure to check pyproject.toml for project defaults. Specifically for black and
isort expectations.
All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
In local development, environment variables for various sensitive values live in a .envrc file
The .envrc file can be loaded into a shell environment to allow access to most third party services
Care should be taken when using .envrc that we do not spam services we use in production with requests

File diff suppressed because it is too large Load Diff

View File

@ -14,3 +14,12 @@ celery:
celery-beat:
poetry run celery -A vrobbler beat -l info
push:
git push && git push gitea
git push --tags && git push --tags gitea
release kind="minor":
poetry run python scripts/release.py {{kind}}
just push

302
poetry.lock generated
View File

@ -2509,144 +2509,146 @@ zookeeper = ["kazoo (>=2.8.0)"]
[[package]]
name = "lxml"
version = "5.4.0"
version = "6.1.1"
description = "Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API."
optional = false
python-versions = ">=3.6"
python-versions = ">=3.8"
groups = ["main"]
files = [
{file = "lxml-5.4.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:e7bc6df34d42322c5289e37e9971d6ed114e3776b45fa879f734bded9d1fea9c"},
{file = "lxml-5.4.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6854f8bd8a1536f8a1d9a3655e6354faa6406621cf857dc27b681b69860645c7"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:696ea9e87442467819ac22394ca36cb3d01848dad1be6fac3fb612d3bd5a12cf"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:6ef80aeac414f33c24b3815ecd560cee272786c3adfa5f31316d8b349bfade28"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:3b9c2754cef6963f3408ab381ea55f47dabc6f78f4b8ebb0f0b25cf1ac1f7609"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:7a62cc23d754bb449d63ff35334acc9f5c02e6dae830d78dab4dd12b78a524f4"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:8f82125bc7203c5ae8633a7d5d20bcfdff0ba33e436e4ab0abc026a53a8960b7"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:b67319b4aef1a6c56576ff544b67a2a6fbd7eaee485b241cabf53115e8908b8f"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_ppc64le.whl", hash = "sha256:a8ef956fce64c8551221f395ba21d0724fed6b9b6242ca4f2f7beb4ce2f41997"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_s390x.whl", hash = "sha256:0a01ce7d8479dce84fc03324e3b0c9c90b1ece9a9bb6a1b6c9025e7e4520e78c"},
{file = "lxml-5.4.0-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:91505d3ddebf268bb1588eb0f63821f738d20e1e7f05d3c647a5ca900288760b"},
{file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:a3bcdde35d82ff385f4ede021df801b5c4a5bcdfb61ea87caabcebfc4945dc1b"},
{file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_ppc64le.whl", hash = "sha256:aea7c06667b987787c7d1f5e1dfcd70419b711cdb47d6b4bb4ad4b76777a0563"},
{file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_s390x.whl", hash = "sha256:a7fb111eef4d05909b82152721a59c1b14d0f365e2be4c742a473c5d7372f4f5"},
{file = "lxml-5.4.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:43d549b876ce64aa18b2328faff70f5877f8c6dede415f80a2f799d31644d776"},
{file = "lxml-5.4.0-cp310-cp310-win32.whl", hash = "sha256:75133890e40d229d6c5837b0312abbe5bac1c342452cf0e12523477cd3aa21e7"},
{file = "lxml-5.4.0-cp310-cp310-win_amd64.whl", hash = "sha256:de5b4e1088523e2b6f730d0509a9a813355b7f5659d70eb4f319c76beea2e250"},
{file = "lxml-5.4.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:98a3912194c079ef37e716ed228ae0dcb960992100461b704aea4e93af6b0bb9"},
{file = "lxml-5.4.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:0ea0252b51d296a75f6118ed0d8696888e7403408ad42345d7dfd0d1e93309a7"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:b92b69441d1bd39f4940f9eadfa417a25862242ca2c396b406f9272ef09cdcaa"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:20e16c08254b9b6466526bc1828d9370ee6c0d60a4b64836bc3ac2917d1e16df"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7605c1c32c3d6e8c990dd28a0970a3cbbf1429d5b92279e37fda05fb0c92190e"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ecf4c4b83f1ab3d5a7ace10bafcb6f11df6156857a3c418244cef41ca9fa3e44"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:0cef4feae82709eed352cd7e97ae062ef6ae9c7b5dbe3663f104cd2c0e8d94ba"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:df53330a3bff250f10472ce96a9af28628ff1f4efc51ccba351a8820bca2a8ba"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_ppc64le.whl", hash = "sha256:aefe1a7cb852fa61150fcb21a8c8fcea7b58c4cb11fbe59c97a0a4b31cae3c8c"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_s390x.whl", hash = "sha256:ef5a7178fcc73b7d8c07229e89f8eb45b2908a9238eb90dcfc46571ccf0383b8"},
{file = "lxml-5.4.0-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:d2ed1b3cb9ff1c10e6e8b00941bb2e5bb568b307bfc6b17dffbbe8be5eecba86"},
{file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:72ac9762a9f8ce74c9eed4a4e74306f2f18613a6b71fa065495a67ac227b3056"},
{file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:f5cb182f6396706dc6cc1896dd02b1c889d644c081b0cdec38747573db88a7d7"},
{file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_s390x.whl", hash = "sha256:3a3178b4873df8ef9457a4875703488eb1622632a9cee6d76464b60e90adbfcd"},
{file = "lxml-5.4.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:e094ec83694b59d263802ed03a8384594fcce477ce484b0cbcd0008a211ca751"},
{file = "lxml-5.4.0-cp311-cp311-win32.whl", hash = "sha256:4329422de653cdb2b72afa39b0aa04252fca9071550044904b2e7036d9d97fe4"},
{file = "lxml-5.4.0-cp311-cp311-win_amd64.whl", hash = "sha256:fd3be6481ef54b8cfd0e1e953323b7aa9d9789b94842d0e5b142ef4bb7999539"},
{file = "lxml-5.4.0-cp312-cp312-macosx_10_9_universal2.whl", hash = "sha256:b5aff6f3e818e6bdbbb38e5967520f174b18f539c2b9de867b1e7fde6f8d95a4"},
{file = "lxml-5.4.0-cp312-cp312-macosx_10_9_x86_64.whl", hash = "sha256:942a5d73f739ad7c452bf739a62a0f83e2578afd6b8e5406308731f4ce78b16d"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:460508a4b07364d6abf53acaa0a90b6d370fafde5693ef37602566613a9b0779"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:529024ab3a505fed78fe3cc5ddc079464e709f6c892733e3f5842007cec8ac6e"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7ca56ebc2c474e8f3d5761debfd9283b8b18c76c4fc0967b74aeafba1f5647f9"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:a81e1196f0a5b4167a8dafe3a66aa67c4addac1b22dc47947abd5d5c7a3f24b5"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:00b8686694423ddae324cf614e1b9659c2edb754de617703c3d29ff568448df5"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:c5681160758d3f6ac5b4fea370495c48aac0989d6a0f01bb9a72ad8ef5ab75c4"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_ppc64le.whl", hash = "sha256:2dc191e60425ad70e75a68c9fd90ab284df64d9cd410ba8d2b641c0c45bc006e"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_s390x.whl", hash = "sha256:67f779374c6b9753ae0a0195a892a1c234ce8416e4448fe1e9f34746482070a7"},
{file = "lxml-5.4.0-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:79d5bfa9c1b455336f52343130b2067164040604e41f6dc4d8313867ed540079"},
{file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:3d3c30ba1c9b48c68489dc1829a6eede9873f52edca1dda900066542528d6b20"},
{file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:1af80c6316ae68aded77e91cd9d80648f7dd40406cef73df841aa3c36f6907c8"},
{file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_s390x.whl", hash = "sha256:4d885698f5019abe0de3d352caf9466d5de2baded00a06ef3f1216c1a58ae78f"},
{file = "lxml-5.4.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aea53d51859b6c64e7c51d522c03cc2c48b9b5d6172126854cc7f01aa11f52bc"},
{file = "lxml-5.4.0-cp312-cp312-win32.whl", hash = "sha256:d90b729fd2732df28130c064aac9bb8aff14ba20baa4aee7bd0795ff1187545f"},
{file = "lxml-5.4.0-cp312-cp312-win_amd64.whl", hash = "sha256:1dc4ca99e89c335a7ed47d38964abcb36c5910790f9bd106f2a8fa2ee0b909d2"},
{file = "lxml-5.4.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:773e27b62920199c6197130632c18fb7ead3257fce1ffb7d286912e56ddb79e0"},
{file = "lxml-5.4.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:ce9c671845de9699904b1e9df95acfe8dfc183f2310f163cdaa91a3535af95de"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:9454b8d8200ec99a224df8854786262b1bd6461f4280064c807303c642c05e76"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:cccd007d5c95279e529c146d095f1d39ac05139de26c098166c4beb9374b0f4d"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0fce1294a0497edb034cb416ad3e77ecc89b313cff7adbee5334e4dc0d11f422"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:24974f774f3a78ac12b95e3a20ef0931795ff04dbb16db81a90c37f589819551"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:497cab4d8254c2a90bf988f162ace2ddbfdd806fce3bda3f581b9d24c852e03c"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:e794f698ae4c5084414efea0f5cc9f4ac562ec02d66e1484ff822ef97c2cadff"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_ppc64le.whl", hash = "sha256:2c62891b1ea3094bb12097822b3d44b93fc6c325f2043c4d2736a8ff09e65f60"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_s390x.whl", hash = "sha256:142accb3e4d1edae4b392bd165a9abdee8a3c432a2cca193df995bc3886249c8"},
{file = "lxml-5.4.0-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:1a42b3a19346e5601d1b8296ff6ef3d76038058f311902edd574461e9c036982"},
{file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4291d3c409a17febf817259cb37bc62cb7eb398bcc95c1356947e2871911ae61"},
{file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:4f5322cf38fe0e21c2d73901abf68e6329dc02a4994e483adbcf92b568a09a54"},
{file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_s390x.whl", hash = "sha256:0be91891bdb06ebe65122aa6bf3fc94489960cf7e03033c6f83a90863b23c58b"},
{file = "lxml-5.4.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:15a665ad90054a3d4f397bc40f73948d48e36e4c09f9bcffc7d90c87410e478a"},
{file = "lxml-5.4.0-cp313-cp313-win32.whl", hash = "sha256:d5663bc1b471c79f5c833cffbc9b87d7bf13f87e055a5c86c363ccd2348d7e82"},
{file = "lxml-5.4.0-cp313-cp313-win_amd64.whl", hash = "sha256:bcb7a1096b4b6b24ce1ac24d4942ad98f983cd3810f9711bcd0293f43a9d8b9f"},
{file = "lxml-5.4.0-cp36-cp36m-macosx_10_9_x86_64.whl", hash = "sha256:7be701c24e7f843e6788353c055d806e8bd8466b52907bafe5d13ec6a6dbaecd"},
{file = "lxml-5.4.0-cp36-cp36m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:fb54f7c6bafaa808f27166569b1511fc42701a7713858dddc08afdde9746849e"},
{file = "lxml-5.4.0-cp36-cp36m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:97dac543661e84a284502e0cf8a67b5c711b0ad5fb661d1bd505c02f8cf716d7"},
{file = "lxml-5.4.0-cp36-cp36m-manylinux_2_28_x86_64.whl", hash = "sha256:c70e93fba207106cb16bf852e421c37bbded92acd5964390aad07cb50d60f5cf"},
{file = "lxml-5.4.0-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.whl", hash = "sha256:9c886b481aefdf818ad44846145f6eaf373a20d200b5ce1a5c8e1bc2d8745410"},
{file = "lxml-5.4.0-cp36-cp36m-musllinux_1_2_x86_64.whl", hash = "sha256:fa0e294046de09acd6146be0ed6727d1f42ded4ce3ea1e9a19c11b6774eea27c"},
{file = "lxml-5.4.0-cp36-cp36m-win32.whl", hash = "sha256:61c7bbf432f09ee44b1ccaa24896d21075e533cd01477966a5ff5a71d88b2f56"},
{file = "lxml-5.4.0-cp36-cp36m-win_amd64.whl", hash = "sha256:7ce1a171ec325192c6a636b64c94418e71a1964f56d002cc28122fceff0b6121"},
{file = "lxml-5.4.0-cp37-cp37m-macosx_10_9_x86_64.whl", hash = "sha256:795f61bcaf8770e1b37eec24edf9771b307df3af74d1d6f27d812e15a9ff3872"},
{file = "lxml-5.4.0-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:29f451a4b614a7b5b6c2e043d7b64a15bd8304d7e767055e8ab68387a8cacf4e"},
{file = "lxml-5.4.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:891f7f991a68d20c75cb13c5c9142b2a3f9eb161f1f12a9489c82172d1f133c0"},
{file = "lxml-5.4.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4aa412a82e460571fad592d0f93ce9935a20090029ba08eca05c614f99b0cc92"},
{file = "lxml-5.4.0-cp37-cp37m-manylinux_2_28_aarch64.whl", hash = "sha256:ac7ba71f9561cd7d7b55e1ea5511543c0282e2b6450f122672a2694621d63b7e"},
{file = "lxml-5.4.0-cp37-cp37m-manylinux_2_28_x86_64.whl", hash = "sha256:c5d32f5284012deaccd37da1e2cd42f081feaa76981f0eaa474351b68df813c5"},
{file = "lxml-5.4.0-cp37-cp37m-musllinux_1_2_aarch64.whl", hash = "sha256:ce31158630a6ac85bddd6b830cffd46085ff90498b397bd0a259f59d27a12188"},
{file = "lxml-5.4.0-cp37-cp37m-musllinux_1_2_x86_64.whl", hash = "sha256:31e63621e073e04697c1b2d23fcb89991790eef370ec37ce4d5d469f40924ed6"},
{file = "lxml-5.4.0-cp37-cp37m-win32.whl", hash = "sha256:be2ba4c3c5b7900246a8f866580700ef0d538f2ca32535e991027bdaba944063"},
{file = "lxml-5.4.0-cp37-cp37m-win_amd64.whl", hash = "sha256:09846782b1ef650b321484ad429217f5154da4d6e786636c38e434fa32e94e49"},
{file = "lxml-5.4.0-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:eaf24066ad0b30917186420d51e2e3edf4b0e2ea68d8cd885b14dc8afdcf6556"},
{file = "lxml-5.4.0-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:2b31a3a77501d86d8ade128abb01082724c0dfd9524f542f2f07d693c9f1175f"},
{file = "lxml-5.4.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0e108352e203c7afd0eb91d782582f00a0b16a948d204d4dec8565024fafeea5"},
{file = "lxml-5.4.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:a11a96c3b3f7551c8a8109aa65e8594e551d5a84c76bf950da33d0fb6dfafab7"},
{file = "lxml-5.4.0-cp38-cp38-manylinux_2_28_aarch64.whl", hash = "sha256:ca755eebf0d9e62d6cb013f1261e510317a41bf4650f22963474a663fdfe02aa"},
{file = "lxml-5.4.0-cp38-cp38-manylinux_2_28_x86_64.whl", hash = "sha256:4cd915c0fb1bed47b5e6d6edd424ac25856252f09120e3e8ba5154b6b921860e"},
{file = "lxml-5.4.0-cp38-cp38-musllinux_1_2_aarch64.whl", hash = "sha256:226046e386556a45ebc787871d6d2467b32c37ce76c2680f5c608e25823ffc84"},
{file = "lxml-5.4.0-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:b108134b9667bcd71236c5a02aad5ddd073e372fb5d48ea74853e009fe38acb6"},
{file = "lxml-5.4.0-cp38-cp38-win32.whl", hash = "sha256:1320091caa89805df7dcb9e908add28166113dcd062590668514dbd510798c88"},
{file = "lxml-5.4.0-cp38-cp38-win_amd64.whl", hash = "sha256:073eb6dcdf1f587d9b88c8c93528b57eccda40209cf9be549d469b942b41d70b"},
{file = "lxml-5.4.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:bda3ea44c39eb74e2488297bb39d47186ed01342f0022c8ff407c250ac3f498e"},
{file = "lxml-5.4.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:9ceaf423b50ecfc23ca00b7f50b64baba85fb3fb91c53e2c9d00bc86150c7e40"},
{file = "lxml-5.4.0-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl", hash = "sha256:664cdc733bc87449fe781dbb1f309090966c11cc0c0cd7b84af956a02a8a4729"},
{file = "lxml-5.4.0-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:67ed8a40665b84d161bae3181aa2763beea3747f748bca5874b4af4d75998f87"},
{file = "lxml-5.4.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:9b4a3bd174cc9cdaa1afbc4620c049038b441d6ba07629d89a83b408e54c35cd"},
{file = "lxml-5.4.0-cp39-cp39-manylinux_2_28_aarch64.whl", hash = "sha256:b0989737a3ba6cf2a16efb857fb0dfa20bc5c542737fddb6d893fde48be45433"},
{file = "lxml-5.4.0-cp39-cp39-manylinux_2_28_x86_64.whl", hash = "sha256:dc0af80267edc68adf85f2a5d9be1cdf062f973db6790c1d065e45025fa26140"},
{file = "lxml-5.4.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:639978bccb04c42677db43c79bdaa23785dc7f9b83bfd87570da8207872f1ce5"},
{file = "lxml-5.4.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:5a99d86351f9c15e4a901fc56404b485b1462039db59288b203f8c629260a142"},
{file = "lxml-5.4.0-cp39-cp39-win32.whl", hash = "sha256:3e6d5557989cdc3ebb5302bbdc42b439733a841891762ded9514e74f60319ad6"},
{file = "lxml-5.4.0-cp39-cp39-win_amd64.whl", hash = "sha256:a8c9b7f16b63e65bbba889acb436a1034a82d34fa09752d754f88d708eca80e1"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:1b717b00a71b901b4667226bba282dd462c42ccf618ade12f9ba3674e1fabc55"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:27a9ded0f0b52098ff89dd4c418325b987feed2ea5cc86e8860b0f844285d740"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:4b7ce10634113651d6f383aa712a194179dcd496bd8c41e191cec2099fa09de5"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:53370c26500d22b45182f98847243efb518d268374a9570409d2e2276232fd37"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:c6364038c519dffdbe07e3cf42e6a7f8b90c275d4d1617a69bb59734c1a2d571"},
{file = "lxml-5.4.0-pp310-pypy310_pp73-win_amd64.whl", hash = "sha256:b12cb6527599808ada9eb2cd6e0e7d3d8f13fe7bbb01c6311255a15ded4c7ab4"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-macosx_10_9_x86_64.whl", hash = "sha256:5f11a1526ebd0dee85e7b1e39e39a0cc0d9d03fb527f56d8457f6df48a10dc0c"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:48b4afaf38bf79109bb060d9016fad014a9a48fb244e11b94f74ae366a64d252"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:de6f6bb8a7840c7bf216fb83eec4e2f79f7325eca8858167b68708b929ab2172"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:5cca36a194a4eb4e2ed6be36923d3cffd03dcdf477515dea687185506583d4c9"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:b7c86884ad23d61b025989d99bfdd92a7351de956e01c61307cb87035960bcb1"},
{file = "lxml-5.4.0-pp37-pypy37_pp73-win_amd64.whl", hash = "sha256:53d9469ab5460402c19553b56c3648746774ecd0681b1b27ea74d5d8a3ef5590"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-macosx_10_9_x86_64.whl", hash = "sha256:56dbdbab0551532bb26c19c914848d7251d73edb507c3079d6805fa8bba5b706"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:14479c2ad1cb08b62bb941ba8e0e05938524ee3c3114644df905d2331c76cd57"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:32697d2ea994e0db19c1df9e40275ffe84973e4232b5c274f47e7c1ec9763cdd"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:24f6df5f24fc3385f622c0c9d63fe34604893bc1a5bdbb2dbf5870f85f9a404a"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:151d6c40bc9db11e960619d2bf2ec5829f0aaffb10b41dcf6ad2ce0f3c0b2325"},
{file = "lxml-5.4.0-pp38-pypy38_pp73-win_amd64.whl", hash = "sha256:4025bf2884ac4370a3243c5aa8d66d3cb9e15d3ddd0af2d796eccc5f0244390e"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:9459e6892f59ecea2e2584ee1058f5d8f629446eab52ba2305ae13a32a059530"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:47fb24cc0f052f0576ea382872b3fc7e1f7e3028e53299ea751839418ade92a6"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:50441c9de951a153c698b9b99992e806b71c1f36d14b154592580ff4a9d0d877"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_28_aarch64.whl", hash = "sha256:ab339536aa798b1e17750733663d272038bf28069761d5be57cb4a9b0137b4f8"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-manylinux_2_28_x86_64.whl", hash = "sha256:9776af1aad5a4b4a1317242ee2bea51da54b2a7b7b48674be736d463c999f37d"},
{file = "lxml-5.4.0-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:63e7968ff83da2eb6fdda967483a7a023aa497d85ad8f05c3ad9b1f2e8c84987"},
{file = "lxml-5.4.0.tar.gz", hash = "sha256:d12832e1dbea4be280b22fd0ea7c9b87f0d8fc51ba06e92dc62d52f804f78ebd"},
{file = "lxml-6.1.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:09dd5b7075dc2f7709654a46543ba1ea3c2e217b2ed8fbd413a8a945a0f40f60"},
{file = "lxml-6.1.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:f6ac4ef4d82dff54670227a69c67782ae0b811b5cf6b17954f1e8f7502fc0d1d"},
{file = "lxml-6.1.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:556e94a63c9b04716f8e4de2abb65775061f846e89331b6c5be79183a24f98ea"},
{file = "lxml-6.1.1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5c6bf403fbb3b3e348a561a5f4f0b9961835657981c802a1df03653eef8a9074"},
{file = "lxml-6.1.1-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1dde6131244bba38a17c745836ba190bc753fd73c9291666287fd0a3fa3dcf30"},
{file = "lxml-6.1.1-cp310-cp310-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:98fc784c2c1440667aeedf8465bdfe10208acf0ead656a2c68627299f546b315"},
{file = "lxml-6.1.1-cp310-cp310-manylinux_2_28_i686.whl", hash = "sha256:add8cf6ddf9a65116119a28ece0f7886e30af27ba724a7594305f1d1b58a92a1"},
{file = "lxml-6.1.1-cp310-cp310-manylinux_2_31_armv7l.whl", hash = "sha256:cf9d57306d848218f3601fee7601fab1a327c942d56e2e97610583cb4dd74206"},
{file = "lxml-6.1.1-cp310-cp310-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:88136950da4d13c318bde414ce10219931937851327f44328f2df4d2c4614067"},
{file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:cecdd5dfdc87b1fd87dbf81d4b037a544f47f4c744200a67013771682d67686a"},
{file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_armv7l.whl", hash = "sha256:cd312b9692e831d2ffcad61eab31d91d4b4655a962e61de8fb410472cbcd37aa"},
{file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:5b7328b46d49fc9477d91ae8f6d55340347d827b7734ba3ea33faae0efef1383"},
{file = "lxml-6.1.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:37a58976370f36d9329d118ad0b953c5aeb9119ac9c6a4e258942a225d0573a1"},
{file = "lxml-6.1.1-cp310-cp310-win32.whl", hash = "sha256:cea3f4c1af79af13cdb2da0c028111d8f8522d4f22a000c82385535f24e5cf3a"},
{file = "lxml-6.1.1-cp310-cp310-win_amd64.whl", hash = "sha256:3abf332af33a74288675d936fe861fd4344da0dd6622193fbc4f2bfbb35536b5"},
{file = "lxml-6.1.1-cp310-cp310-win_arm64.whl", hash = "sha256:8dadbe5b217ff35b6a8d16610dd710219b59b76d13f0e3f0d9f36786206e4485"},
{file = "lxml-6.1.1-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:53b7d2b7a10b1c35c0a5e21e9224accf60c1bbfba523990732e521b2b73adef2"},
{file = "lxml-6.1.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ff3f333630ab480244a1bff72043e511a91eb22e7595dead8653ee5612dd8f3d"},
{file = "lxml-6.1.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:a4bbea04c97f6d78a48e3fbc1cb9116d2780b1b39e03a23f6eb9b603fd61f510"},
{file = "lxml-6.1.1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:db1d75f6617a49c1c01bc7023713e0ff59ab32c9579ae62a7674c0e34f3b0b0a"},
{file = "lxml-6.1.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3a12689be69a28ddaa0ab99a5a1137da2afd5f8f16df7b5680b66f616d3eda1d"},
{file = "lxml-6.1.1-cp311-cp311-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:18b73c339ae29b90fd2d06e58ebd555a751bde9cd6bbd36cc0281b9a2c94e9d8"},
{file = "lxml-6.1.1-cp311-cp311-manylinux_2_28_i686.whl", hash = "sha256:752d3bbfe874715ccd0aec7f88d7fc623c0f1fd7aa7b3238a084e017bad2a009"},
{file = "lxml-6.1.1-cp311-cp311-manylinux_2_31_armv7l.whl", hash = "sha256:6b1761fbf9ec984e2e9d9c589ef5f5fd684b7c19f92aadd567a26c5224958db6"},
{file = "lxml-6.1.1-cp311-cp311-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d680fbcb768404c601ecb43519ecd8461f6954cb11c06a78962f666832ccfca8"},
{file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:162af1091cd785f2f27e62d3547ae9bc58ec5c86dd314d67021fd02463708d83"},
{file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_armv7l.whl", hash = "sha256:e9308ff8241c532df3f3e570f9a5aeed6c853f888512ba4b75638d7c11c95ef6"},
{file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:5f6994074ebae6ffb04447268e37dc16edc304f9859cf91acb86e0af6c1b395c"},
{file = "lxml-6.1.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:80c2dfadb855da477cf73373ad29a333535dedb9b12bad02c9814c8e2b43bf08"},
{file = "lxml-6.1.1-cp311-cp311-win32.whl", hash = "sha256:30a89d3ac8faec007453fb541f3f46807eeec88edd5826f6e3fe001752a2c621"},
{file = "lxml-6.1.1-cp311-cp311-win_amd64.whl", hash = "sha256:abbefa31eee84842140f67acef1c828e28bba8bbf0c3bc6e5492a9af88152c28"},
{file = "lxml-6.1.1-cp311-cp311-win_arm64.whl", hash = "sha256:dcb292aa7fe485ceff7af4f92e46c5af397daec5dff64871a528f0fc47a3cc5b"},
{file = "lxml-6.1.1-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:104c09bda8d2a562824c0e319d0768ce26a779b7601e0931d33b09b53c392ef7"},
{file = "lxml-6.1.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:25c6997a9a534e016695a0ba06b2f07945de682731ff01065b6d5a4474179da1"},
{file = "lxml-6.1.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c921ba5c51e4e9f63b8b00267d06566e1f63407408a0496da2d1d0bfc819c7fc"},
{file = "lxml-6.1.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:54a7f95e4de5fb94e2f9f4b9055c6ba33bf3d628fd77a1d647c5923caa2cdcdc"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:96f2ec43df44b1f76249ee0a615334f9b5b060e1c8bd90e706dad2d14d02f383"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:70ef8a7e102a1508f8121aae5b0867abd663f72c14f0a9c937e6554cb4587b7b"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ebe6af670449830d6d9b752c256a983291c766a1365ba5d5460048f9e33a7818"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_28_i686.whl", hash = "sha256:27acc820660aaffa4f7c087f29120e12980f7779d56d8492d263170111284740"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_31_armv7l.whl", hash = "sha256:1db753c9115ec7100d073b744d17e25e88a8f90f5c39b2f5dd878149af59671f"},
{file = "lxml-6.1.1-cp312-cp312-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:c4f469aebd783bb741c2ecb2a681008fd26bfe5c16a9a72ed5467f834e810df2"},
{file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:766b010012d59470072c1816b5b6c69f1d243e5db36ea5968e94accf430a4635"},
{file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_armv7l.whl", hash = "sha256:b8d812c6011c08b8111a15e54dd990b8923692d80adf35488bee34026c35accf"},
{file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:fe0306bd29505a9177aac19f1877174b0e7422c222a59f70b2cd41633448c3dc"},
{file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:5ba186ad207446c65d3bb3d3e0412b032b1d9f595e59861e2354798c5703d955"},
{file = "lxml-6.1.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:aa366a1e55b8ebfe8ca8ddc3cfe75c8ebade181aeb0f661d0cb05986b647f72a"},
{file = "lxml-6.1.1-cp312-cp312-win32.whl", hash = "sha256:126c93f7f56f0eda92f6d8c619edc463a4f23d9252f1c9d0405a76f25fa9f11a"},
{file = "lxml-6.1.1-cp312-cp312-win_amd64.whl", hash = "sha256:26e6eda8d38c1fcab1090dd196ee87cbd13788e531937610e2589085de074e77"},
{file = "lxml-6.1.1-cp312-cp312-win_arm64.whl", hash = "sha256:6540377fbd53fe1b629172288c464fb18db11ce1fa7dc15891da10aa9dcc3e7f"},
{file = "lxml-6.1.1-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:68a9198d0fc122d14bb76837de9aa80cf84caed990b5b237f532ed87d3706736"},
{file = "lxml-6.1.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7d47866cb32fb503450b6edc9df355d10dc49836af2e89901bd6ac6b0896d9d9"},
{file = "lxml-6.1.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:eb7c9811bfaa8b1ed5ed319f5d370dfbcaa59d52ea64be2a5a85e18195930354"},
{file = "lxml-6.1.1-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:762ff394d5bd56da0cf034a23dcce4e13923f15321a2adfa2ac00201dc6d3fca"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a088f287f7d8275a33c07f2cac6c50b9319309a0200a39e7e75d80c707723099"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:e902da4b04e6b52e5893900d4b8ab46068f75f3561f01bf1080957f9fd932ed6"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1d4962d4c66bf830a7e59ed6cfc17d148149898a3aefa8ec6e59763e6e3ed085"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_28_i686.whl", hash = "sha256:581d4c8ae690a6609e64862dd6b7c2489635c2d13907fc2b20f2bc200ff1d21e"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_31_armv7l.whl", hash = "sha256:876e1ff5930ed8bf295ec5ef9a8155e9b6b1876bbf1deed8b3a8069311875a8f"},
{file = "lxml-6.1.1-cp313-cp313-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9eb9b5a968f6e0f6d640092a567e14529ff8cea2e29d00da6f78a79fa49f013c"},
{file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:aa49e06d94aba782c6a02eecb7e507969e7e7a41b267f1b359bb35585f295d5b"},
{file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_armv7l.whl", hash = "sha256:70cdfd80589d59e43e18005dd7244e8895e93db8ab6a620b7e23df5445a4e3d2"},
{file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:aad9aa39483ed8ec44d6d2e59e5b98a0d80676ef0d92f44bfc374836111f62f5"},
{file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:d49514be2f28d895c38cf9d2b72d7b9a07d00314519f456c0b50b53cfcf4c785"},
{file = "lxml-6.1.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:47402e62c52ff5988c1e8c6c63177f5708bccf48e366dea4e3dcf1e645e04947"},
{file = "lxml-6.1.1-cp313-cp313-win32.whl", hash = "sha256:3483644525531e1d5762b0c44a8e18b6efba321b6dcf8a8952de10b037618bca"},
{file = "lxml-6.1.1-cp313-cp313-win_amd64.whl", hash = "sha256:a10bd2fd62e8ce916ececb342f348f190724a098c1faa056fdfb2a22ad5e8660"},
{file = "lxml-6.1.1-cp313-cp313-win_arm64.whl", hash = "sha256:424aa57aca0897eb922aef34395bd1289b3b6f04e6bae20ea123c0c7e333cffc"},
{file = "lxml-6.1.1-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:19b7ab10b210b0b3ad7985d9ac4eb66ab09a90b20fe6e2f7ba55d01a234345d0"},
{file = "lxml-6.1.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:c08e5c694306507275f2290073350c4f32e383db15213b2c69e7ff39c1193840"},
{file = "lxml-6.1.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:74a9717fd0d82effef5c2854f0d917231d5324b5a3eb7275c43ac9fa32f97a14"},
{file = "lxml-6.1.1-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:efe0374196335f93b53269acd811b944f2e6bdc88e8894f214bd636455484909"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ac931cdc9442c1763b8a8f6cd62c0c938737eafc5be75eff88df55fc73bc0d00"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:aee395f5d0927f947758b4ec119fd5fc8ec71f07a1c5c52077b30b04c0fa6955"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9395002973c827b3ed67db77e6ec09f092919a587022174554096a269378fb13"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_28_i686.whl", hash = "sha256:73bc2086f141224ebddb7fc5c6a36ca58b31b94b561e1dfe8e073e3270fad1e7"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_31_armv7l.whl", hash = "sha256:3779def59032b81e44a5f70096ef6bf2082f8d901937dca354474ba09782e245"},
{file = "lxml-6.1.1-cp314-cp314-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:86c89b9d55ebf820ad7c90bc533410f0d098054f293351f10603c0c46ff598f5"},
{file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:19607c6bbff2a44cf3fe8250abccd20942d3462473e0a721d01d379ed017e462"},
{file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_armv7l.whl", hash = "sha256:c6ed5141a5c7507cf3ee76bd363b0d6f801e3321adc35b5d825a23115faa5465"},
{file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:62aeb7e85b5d60320b9d77eef2e773994e2c0ce10121b277e0a19804e1654a5a"},
{file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:b1b963fd8f5caa68e99dfae060d54de1fe9cba899b8718b44a00cdca53c3e590"},
{file = "lxml-6.1.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:63876be28efefa04a1df615b46770e82042cce445cfdce55160522f57b231ccb"},
{file = "lxml-6.1.1-cp314-cp314-win32.whl", hash = "sha256:7f7a92e8583f06b1fd49d01158143b8461cfcd135dcb10ec807270a3051bd603"},
{file = "lxml-6.1.1-cp314-cp314-win_amd64.whl", hash = "sha256:b2d444f2e66624d68e9c6b211e28a76e22fff5fcabcfff4deac18b529b7d4137"},
{file = "lxml-6.1.1-cp314-cp314-win_arm64.whl", hash = "sha256:3fd9728a2735fda14f4e8235830c86b539e9661e849665bf926d3f867943b4bf"},
{file = "lxml-6.1.1-cp314-cp314t-macosx_10_15_universal2.whl", hash = "sha256:787b2496d0dbe8cd180984e8d29e3a6f76e7ea34db781cb3bd55e4ba1ef8b4ee"},
{file = "lxml-6.1.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:2c8daa471358dc2d6fcf02165e80ec68f77871a286df95bc5cc3816153b0fd2c"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:acd7d70b64c0aae0c7922cca83d288a16f5f6da523637697872253415269baef"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4f0dd2f01f9f8a89f565d000e03abcf0a13d692a346c8d22f628d49af098777a"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b7e8a14c8634bf6f7a568634cb395305a6d964aeb5b7ee32248094bed3a7e2c"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:86281fbdd6a8162756f8d603f37e3435bfa38043adb79c6dc6a2dfee065e7525"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5d7152ec39ca7c402d8fb9bad86140a15b9503bd0c54484e3f1bbe3dd37ceca"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_28_i686.whl", hash = "sha256:88d8cb75b9d82858497a5393e3c63cfbf03035225e4b35a49ed7ccb151e4dc0e"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_31_armv7l.whl", hash = "sha256:f64ec5397ea6a41fc1b4af0380d79b44a755b5531dcaccd9940fb260dca93038"},
{file = "lxml-6.1.1-cp314-cp314t-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:d34bbf07dbc7ca5970671b1512e928991fb5e9d95365636c9b2d8b4f53af405e"},
{file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:17e0e18d4ad8adbd0399291bc44845b69d9dd68439a3cdebdf35ff902ec05072"},
{file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_armv7l.whl", hash = "sha256:3ab541146f1f6968c462d6c2ac495148e8cdba2f8347700b2141b6ec5a75bf52"},
{file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_ppc64le.whl", hash = "sha256:2a0217714657e023ef4293500f65aa20fce6164c8fd6b08fa5bd4a859fb14b9b"},
{file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:05a82eb6e1530a64f26225b55cbd178113bd0b5af1c2b625f25e5296742c26d2"},
{file = "lxml-6.1.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:9e36f163528fc50cbef305f02a5fd66d404edf7049cdaff211dbc2cba5a7013e"},
{file = "lxml-6.1.1-cp314-cp314t-win32.whl", hash = "sha256:649dda677cf3bd6ac9ae14007ba0c824ded8ce5808b53fc7431d9140399118c1"},
{file = "lxml-6.1.1-cp314-cp314t-win_amd64.whl", hash = "sha256:793033d6c5cdf33a573f910d9bea14ef8f5771820411d118da8e1182edb53d5e"},
{file = "lxml-6.1.1-cp314-cp314t-win_arm64.whl", hash = "sha256:58bb955caba94e467d2a96da17660d2d704e0675894cba21ab8a775b8621fd1c"},
{file = "lxml-6.1.1-cp38-cp38-macosx_10_9_x86_64.whl", hash = "sha256:6689e828a94eee4f139408c337bb198e014724bb8a8c26d3cfac49d119ed69a6"},
{file = "lxml-6.1.1-cp38-cp38-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:bdebcc8a75d38c7598dfb2c9ed852d7a9eb4a10d6e2d0764b919b802bf32ac88"},
{file = "lxml-6.1.1-cp38-cp38-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8be8ad51249698103d24b0571df35a10990fbe93dd043b6c024172189485f5e3"},
{file = "lxml-6.1.1-cp38-cp38-manylinux_2_28_i686.whl", hash = "sha256:76447f65250ed2501ead1a1552f5ce8edff159a86f308348e6a9c4acb5e1f1b4"},
{file = "lxml-6.1.1-cp38-cp38-musllinux_1_2_x86_64.whl", hash = "sha256:ffecec8eb889b58ba9be5b95fb1cc78e22ea8eedea38e8736a1568fe1979250e"},
{file = "lxml-6.1.1-cp38-cp38-win32.whl", hash = "sha256:c674693f055fa2495de12292cb45e9944199d8eaef5a2dec45175c7c61cb73e3"},
{file = "lxml-6.1.1-cp38-cp38-win_amd64.whl", hash = "sha256:55b03549819867ea141c0202242c4816c82e52ec36e7e648db9d8da5a3dc3ed6"},
{file = "lxml-6.1.1-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:c9f79d5325907f13e1be0b3e4dacc1049d1dffc4aeee3c995284bea5fe0fab7d"},
{file = "lxml-6.1.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:83b6b30eb131da7a75b601f28c5d6971e6ed3e887919bf6b6a1ad3c2df289080"},
{file = "lxml-6.1.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:441dd227fa0690eb9fc81edabc63cdcefc212bba99b906dcf6e32cc1a9d3e533"},
{file = "lxml-6.1.1-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:e07c65f443c887bbcf31cc1771d932ecc192a5273943589b3c7572b749f1ffb2"},
{file = "lxml-6.1.1-cp39-cp39-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5bec7d03d78d853597d6107854c2310ce3f761fd218fe9fe91d5101fcf6c2efe"},
{file = "lxml-6.1.1-cp39-cp39-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f76acfb5f68ba982635a53fd985a8044be98a35b43232c2a1ee235ffab3e1dd"},
{file = "lxml-6.1.1-cp39-cp39-manylinux_2_28_i686.whl", hash = "sha256:8d43ca737b20e106e4aebc42b2f3ae19f00ba63d7eb731698ee083d72d15646f"},
{file = "lxml-6.1.1-cp39-cp39-manylinux_2_31_armv7l.whl", hash = "sha256:32ab449a5486f6c758e849bb86710d0e45edc24a04e250c01555f8f5653958f8"},
{file = "lxml-6.1.1-cp39-cp39-manylinux_2_38_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:53c909b62a0532183542fed00c5a7218258c56292d409bc789886fe1cb04c438"},
{file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:640f97d43d867bcb9c75b3af013b64850756b746cb6bce8ace83b70da3abba9d"},
{file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_armv7l.whl", hash = "sha256:469e3618338bd7ab5beb412d2439825479fcf0dab99e394ca563dbc4eaf6c834"},
{file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_riscv64.whl", hash = "sha256:aae97dfdb60715c164419ac2532a76d013c3918a665eb6cb7288098b5f349aaf"},
{file = "lxml-6.1.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:c9a4b821dc7055bf9e05ff5719e18ec501f75c0f0bbfabd573b277559780833d"},
{file = "lxml-6.1.1-cp39-cp39-win32.whl", hash = "sha256:639f6c857d91d9be29bd7502348d6736dab168b54b5158cd899abf11684dc186"},
{file = "lxml-6.1.1-cp39-cp39-win_amd64.whl", hash = "sha256:34c2d737beabfe35baada43941ed519251e9a12e779031496bcd5d539fcfd730"},
{file = "lxml-6.1.1-cp39-cp39-win_arm64.whl", hash = "sha256:07a4a68e286ee7a1ed7dfb8af83e615757c0ccfe9f18c6b4ea6771388d9ba8c9"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-macosx_10_15_x86_64.whl", hash = "sha256:31033dc34636ea6b7d5cc11b1ddbda78a14de858ba9d3e1ed4b69a3085bc521e"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:3893c14c4b6ac5b2d54ba8cf03e99fe5104e592de491f19bd6b82756c09f8004"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c07da4cebf6889f03ebac8d238f62318e29f495de0aa18a51ea14e61ae907e2e"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:f6f0ce10945fab9c4c06ce14e22af9059d1a87493a9af4501a5b0b9187e21cf2"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-manylinux_2_26_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8844cd288697c6425c9beba919302241e3278871dc6519515e72b04e987abcf"},
{file = "lxml-6.1.1-pp311-pypy311_pp73-win_amd64.whl", hash = "sha256:ed21202aec73cda4d55d1ce57b389aadb90ffb044e6cd1080b8347efe1b1ec84"},
{file = "lxml-6.1.1.tar.gz", hash = "sha256:ba96ae44888e0185281e937633a743ea90d5a196c6000f82565ebb0580012d40"},
]
[package.dependencies]
@ -2657,7 +2659,6 @@ cssselect = ["cssselect (>=0.7)"]
html-clean = ["lxml_html_clean"]
html5 = ["html5lib"]
htmlsoup = ["BeautifulSoup4"]
source = ["Cython (>=3.0.11,<3.1.0)"]
[[package]]
name = "lxml-html-clean"
@ -4965,6 +4966,18 @@ files = [
{file = "soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349"},
]
[[package]]
name = "sqids"
version = "0.5.2"
description = "Generate YouTube-like ids from numbers."
optional = false
python-versions = ">=3.6"
groups = ["main"]
files = [
{file = "sqids-0.5.2-py3-none-any.whl", hash = "sha256:0089ba823e21fd44290c7225f02fb0b5140c36e41959c04d86d3f6f2513799be"},
{file = "sqids-0.5.2.tar.gz", hash = "sha256:5ac08f0c5c9b6814bc2e7c79ee5931e0849d25d95c50e415771b022a44f58af9"},
]
[[package]]
name = "sqlparse"
version = "0.5.5"
@ -5438,6 +5451,21 @@ brotli = ["brotli (==1.0.9) ; os_name != \"nt\" and python_version < \"3\" and p
secure = ["certifi", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "ipaddress ; python_version == \"2.7\"", "pyOpenSSL (>=0.14)", "urllib3-secure-extra"]
socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
[[package]]
name = "vadersentiment"
version = "3.3.2"
description = "VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains."
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "vaderSentiment-3.3.2-py2.py3-none-any.whl", hash = "sha256:3bf1d243b98b1afad575b9f22bc2cb1e212b94ff89ca74f8a23a588d024ea311"},
{file = "vaderSentiment-3.3.2.tar.gz", hash = "sha256:5d7c06e027fc8b99238edb0d53d970cf97066ef97654009890b83703849632f9"},
]
[package.dependencies]
requests = "*"
[[package]]
name = "vine"
version = "5.1.0"
@ -6003,5 +6031,5 @@ cffi = ["cffi (>=1.17,<2.0) ; platform_python_implementation != \"PyPy\" and pyt
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<3.14"
content-hash = "395bc627d9e2fa0434082fc574693e349cf0c604c47e8554b9a50a28bdd273d0"
python-versions = ">=3.11,<3.15"
content-hash = "cc5b3b44071d6b0ab4f05189580232cc129b4ed694ab3f0673c3d838c3af0f8a"

View File

@ -1,11 +1,11 @@
[tool.poetry]
name = "vrobbler"
version = "0.16.1"
version = "51.2"
description = ""
authors = ["Colin Powell <colin@unbl.ink>"]
[tool.poetry.dependencies]
python = ">=3.11,<3.14"
python = ">=3.11,<3.15"
Django = "^4.0.3"
django-extensions = "^3.1.5"
python-dateutil = "^2.8.2"
@ -61,6 +61,9 @@ bgg-api = "^1.1.13"
recipe-scrapers = "^15.11.0"
gpxpy = "^1.6.2"
fitparse = "^1.2.0"
lxml = ">=5.5.0"
vaderSentiment = "^3.3.2"
sqids = "^0.5.2"
[tool.poetry.group.test]
optional = true
@ -106,6 +109,8 @@ exclude_dirs = ["*/tests/*", "*/migrations/*"]
[tool.poetry.scripts]
vrobbler = "vrobbler.cli:main"
[tool.poetry_bumpversion.file."vrobbler/__init__.py"]
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

217
scripts/release.py Executable file
View File

@ -0,0 +1,217 @@
#!/usr/bin/env python3
"""Cut a new release: collect DONE items from Backlog into a new Version section.
Usage:
poetry run python scripts/release.py major
poetry run python scripts/release.py minor
"""
import re
import subprocess
import sys
from pathlib import Path
PROJECT_FILE = Path("PROJECT.org")
PYPROJECT_FILE = Path("pyproject.toml")
BACKLOG_RE = re.compile(r"^\* Backlog\s+\[(\d+)/(\d+)\](.*)$")
VERSION_RE = re.compile(r"^\* Version\s+(\d+\.\d+)\s+\[\d+/\d+\]")
DONE_HEADER_RE = re.compile(r"^(\*\* DONE\s+)(.*)$")
ITEM_HEADER_RE = re.compile(r"^\*\* ")
def parse_done_line(line):
"""Extract a clean title from a ** DONE line, stripping priority and tags."""
rest = line[8:].strip() # remove "** DONE "
# strip priority marker like [#A]
rest = re.sub(r"^\[#[A-C]\]\s+", "", rest, count=1)
# strip org-mode tags at end (space-colon-tags)
rest = re.sub(r"\s+:\S.*:\s*$", "", rest)
return rest
def bump_version(current_major, current_minor, kind):
if kind == "major":
return current_major + 1, 0
elif kind == "minor":
return current_major, current_minor + 1
else:
raise ValueError(f"Unknown bump kind: {kind}")
def main():
if len(sys.argv) < 2 or sys.argv[1] not in ("major", "minor"):
print(f"Usage: {sys.argv[0]} <major|minor>", file=sys.stderr)
sys.exit(1)
kind = sys.argv[1]
lines = PROJECT_FILE.read_text().splitlines(keepends=True)
# ---------------------------------------------------------------
# 1. Identify top-level sections
# ---------------------------------------------------------------
section_starts = []
for i, line in enumerate(lines):
if line.startswith("* ") and not line.startswith("** "):
section_starts.append(i)
section_starts.append(len(lines))
backlog_idx = None
version_idx = None
for idx, start in enumerate(section_starts[:-1]):
header = lines[start].strip()
if header.startswith("* Backlog"):
backlog_idx = idx
if header.startswith("* Version"):
version_idx = idx # last occurrence wins
if backlog_idx is None:
print("ERROR: no Backlog section found", file=sys.stderr)
sys.exit(1)
if version_idx is None:
print("ERROR: no Version section found", file=sys.stderr)
sys.exit(1)
backlog_start = section_starts[backlog_idx]
backlog_end = section_starts[backlog_idx + 1]
# Find the newest Version section (first after Backlog) that matches
# our expected format (e.g. "37.0" not "0.11.4").
version_start = None
for idx in range(backlog_idx + 1, version_idx + 1):
header = lines[section_starts[idx]].strip()
if VERSION_RE.match(header):
version_start = section_starts[idx]
break
if version_start is None:
print("ERROR: no parseable Version header found", file=sys.stderr)
sys.exit(1)
version_header = lines[version_start].strip()
# ---------------------------------------------------------------
# 2. Parse current version from the newest * Version header
# ---------------------------------------------------------------
vm = VERSION_RE.match(version_header)
current_version = vm.group(1)
major_str, minor_str = current_version.split(".")
current_major = int(major_str)
current_minor = int(minor_str)
new_major, new_minor = bump_version(current_major, current_minor, kind)
new_version = f"{new_major}.{new_minor}"
# ---------------------------------------------------------------
# 3. Collect ** DONE items from the Backlog section
# ---------------------------------------------------------------
backlog_lines = lines[backlog_start:backlog_end]
# Split Backlog into items at each ** line (skip the section header)
items = [] # list of (start_idx, end_idx, is_done)
item_start = None
for i in range(1, len(backlog_lines)):
if ITEM_HEADER_RE.match(backlog_lines[i]):
if item_start is not None:
items.append((item_start, i, backlog_lines[item_start].startswith("** DONE")))
item_start = i
if item_start is not None:
items.append((item_start, len(backlog_lines), backlog_lines[item_start].startswith("** DONE")))
done_items = [(s, e) for s, e, is_done in items if is_done]
kept_items = [(s, e) for s, e, is_done in items if not is_done]
if not done_items:
print("No DONE items found in Backlog — nothing to release.")
sys.exit(1)
# ---------------------------------------------------------------
# 4. Build the new Version section text
# ---------------------------------------------------------------
version_section_lines = [f"* Version {new_version} [{len(done_items)}/{len(done_items)}]\n"]
for s, e in done_items:
version_section_lines.extend(backlog_lines[s:e])
# ---------------------------------------------------------------
# 5. Build updated Backlog section
# ---------------------------------------------------------------
backlog_header_line = backlog_lines[0]
bm = BACKLOG_RE.match(backlog_header_line.strip())
if not bm:
print(f"ERROR: could not parse backlog header: {backlog_header_line!r}", file=sys.stderr)
sys.exit(1)
done_count = int(bm.group(1))
total_count = int(bm.group(2))
tags = bm.group(3)
new_done = done_count - len(done_items)
new_total = total_count - len(done_items)
new_backlog_header = f"* Backlog [{new_done}/{new_total}]{tags}\n"
backlog_body = []
for s, e in kept_items:
backlog_body.extend(backlog_lines[s:e])
# ---------------------------------------------------------------
# 6. Assemble the new file
# ---------------------------------------------------------------
before_backlog = lines[:backlog_start]
after_backlog = lines[backlog_end:version_start]
# Everything from the first Version section onwards
from_version = lines[version_start:]
output = (
before_backlog
+ [new_backlog_header]
+ backlog_body
+ version_section_lines
+ ["\n"]
+ after_backlog
+ from_version
)
# ---------------------------------------------------------------
# 7. Update pyproject.toml
# ---------------------------------------------------------------
pyproject = PYPROJECT_FILE.read_text()
pyproject = re.sub(
r'^version = "[\d.]+"',
f'version = "{new_version}"',
pyproject,
count=1,
flags=re.MULTILINE,
)
# ---------------------------------------------------------------
# 8. Write files
# ---------------------------------------------------------------
PROJECT_FILE.write_text("".join(output))
PYPROJECT_FILE.write_text(pyproject)
# ---------------------------------------------------------------
# 9. Build commit body from done item titles
# ---------------------------------------------------------------
commit_lines = []
for s, e in done_items:
title = parse_done_line(backlog_lines[s])
if title:
commit_lines.append(f"- {title}")
commit_body = "\n".join(commit_lines)
commit_message = f"[release] Bump to version {new_version}\n\n{commit_body}"
# ---------------------------------------------------------------
# 10. Git commit + tag
# ---------------------------------------------------------------
subprocess.run(["git", "add", str(PROJECT_FILE), str(PYPROJECT_FILE)], check=True)
subprocess.run(["git", "commit", "-m", commit_message], check=True)
subprocess.run(["git", "tag", new_version], check=True)
print(f"\nReleased v{new_version} — tag {new_version} created.")
print(f"Moved {len(done_items)} DONE item(s) from Backlog to Version section.")
if __name__ == "__main__":
main()

View File

@ -1,3 +1,4 @@
import tempfile
from datetime import timedelta
import pytest
@ -128,6 +129,23 @@ class TestBirdingCSVImportModel:
assert imp.import_type == "Birding CSV"
assert "Birding" in str(imp)
def test_record_error(self, db, user):
imp = BirdingCSVImport.objects.create(user=user)
assert imp.error_log is None
imp.record_error("test error")
imp.refresh_from_db()
assert imp.error_log is not None
assert "test error" in imp.error_log
def test_record_error_appends(self, db, user):
imp = BirdingCSVImport.objects.create(user=user)
imp.record_error("first error")
imp.record_error("second error")
imp.refresh_from_db()
assert imp.error_log.count("\n") == 1
assert "first error" in imp.error_log
assert "second error" in imp.error_log
@pytest.mark.django_db(transaction=True)
def test_process_via_model(self, user, birding_csv_file):
imp = BirdingCSVImport.objects.create(user=user)
@ -137,3 +155,35 @@ class TestBirdingCSVImportModel:
imp.refresh_from_db()
assert imp.process_count == 1
assert imp.processed_finished is not None
def test_record_error_on_bad_csv(self, user, db):
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
Canada Goose,6,Test Park,Stationary,"Bad Date",4:15 PM,9 minute(s),,,4,true,4 species,
"""
with tempfile.NamedTemporaryFile(
mode="w", suffix=".csv", delete=False, encoding="utf-8"
) as f:
f.write(content)
file_path = f.name
errors = []
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
assert len(scrobbles) == 0
assert len(errors) == 1
assert "Could not parse date/time" in errors[0]
def test_record_error_on_bad_location(self, user, db):
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
Canada Goose,6,,Stationary,"May 10, 2026",4:15 PM,9 minute(s),,,4,true,4 species,
"""
with tempfile.NamedTemporaryFile(
mode="w", suffix=".csv", delete=False, encoding="utf-8"
) as f:
f.write(content)
file_path = f.name
errors = []
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
assert len(scrobbles) == 0
assert len(errors) == 1
assert "Skipping rows with no location" in errors[0]

View File

@ -8,7 +8,8 @@ from django.urls import reverse
from django.utils import timezone
from music.models import Album, Artist, Track
from podcasts.models import PodcastEpisode
from scrobbles.models import Scrobble
from scrobbles.models import Scrobble, ShareViewLog
from scrobbles.sqids import encode_scrobble_share
from tasks.models import Task
@ -143,9 +144,10 @@ def test_mopidy_track_webhook_creates_track_and_scrobble(
album = Album.objects.create(name="Sublime", album_artist=artist)
track = Track.objects.create(
title="Same in the End",
artist=artist,
artist_fk=artist,
base_run_time_seconds=60,
)
track.artists.add(artist)
track.albums.add(album)
mock_artist_fc.return_value = artist
@ -181,9 +183,10 @@ def test_jellyfin_track_webhook_creates_track_and_scrobble(
album = Album.objects.create(name="Emotion", album_artist=artist)
track = Track.objects.create(
title="Emotion",
artist=artist,
artist_fk=artist,
base_run_time_seconds=60,
)
track.artists.add(artist)
track.albums.add(album)
mock_artist_fc.return_value = artist
@ -225,9 +228,10 @@ def test_mopidy_track_webhook_stores_raw_data(
album = Album.objects.create(name="Sublime", album_artist=artist)
track = Track.objects.create(
title="Same in the End",
artist=artist,
artist_fk=artist,
base_run_time_seconds=60,
)
track.artists.add(artist)
track.albums.add(album)
mock_artist_fc.return_value = artist
@ -265,10 +269,11 @@ def test_mopidy_track_webhook_stores_album_id(
album = Album.objects.create(name="Sublime", album_artist=artist)
track = Track.objects.create(
title="Same in the End",
artist=artist,
artist_fk=artist,
album=album,
base_run_time_seconds=60,
)
track.artists.add(artist)
mock_artist_fc.return_value = artist
mock_track_fc.return_value = track
@ -303,9 +308,10 @@ def test_jellyfin_track_webhook_stores_raw_data(
album = Album.objects.create(name="Emotion", album_artist=artist)
track = Track.objects.create(
title="Emotion",
artist=artist,
artist_fk=artist,
base_run_time_seconds=60,
)
track.artists.add(artist)
track.albums.add(album)
mock_artist_fc.return_value = artist
@ -347,11 +353,11 @@ def test_jellyfin_track_webhook_stores_album_id(
album = Album.objects.create(name="Emotion", album_artist=artist)
track = Track.objects.create(
title="Emotion",
artist=artist,
artist_fk=artist,
album=album,
base_run_time_seconds=60,
)
track.albums.add(album)
track.artists.add(artist)
mock_artist_fc.return_value = artist
mock_track_fc.return_value = track
@ -507,6 +513,7 @@ def test_scrobble_detail_view_with_notes_as_flat_list(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": ["First note", "Second note"],
"description": "Test description",
@ -529,6 +536,7 @@ def test_scrobble_detail_view_with_notes_as_dict_timestamps(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": [
{"2024-01-01 10:00:00": "Note at first timestamp"},
@ -557,6 +565,7 @@ def test_scrobble_detail_view_with_notes_and_labels(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": [
{"2024-01-01 10:00:00": "Note with label"},
@ -602,7 +611,8 @@ def test_scrobble_detail_view_post_updates_log(client):
scrobble.refresh_from_db()
assert scrobble.log["description"] == "Updated description"
assert scrobble.log["notes"] == ["Updated note"]
assert isinstance(scrobble.log["notes"], dict)
assert list(scrobble.log["notes"].values()) == ["Updated note"]
@pytest.mark.skip("Need to refactor")
@ -733,3 +743,293 @@ def test_gps_webhook_creates_location(client, valid_auth_token):
)
assert response.status_code == 200
assert "scrobble_id" in response.data
@pytest.mark.django_db
def test_share_view_shared_visibility(client):
user = get_user_model().objects.create_user(
username="shareuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_view_public_visibility(client):
user = get_user_model().objects.create_user(
username="shareuser2", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="public",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_view_private_visibility_returns_404(client):
user = get_user_model().objects.create_user(
username="shareuser3", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_invalid_sqid_returns_404(client):
url = reverse("scrobbles:shared-detail", kwargs={"sqid": "InvalidSqid123"})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_expired_token_returns_404(client):
user = get_user_model().objects.create_user(
username="shareuser4", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
scrobble.regenerate_share_token()
url = reverse("scrobbles:shared-detail", kwargs={"sqid": old_sqid})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_increments_count_and_logs_view(client):
user = get_user_model().objects.create_user(
username="shareuser5", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
assert scrobble.share_view_count == 0
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
scrobble.refresh_from_db()
assert scrobble.share_view_count == 1
assert ShareViewLog.objects.filter(scrobble=scrobble).count() == 1
log_entry = ShareViewLog.objects.filter(scrobble=scrobble).first()
assert log_entry.ip_address == "127.0.0.1"
assert log_entry.user_agent == ""
assert log_entry.referrer == ""
@pytest.mark.django_db
def test_explore_view_shows_only_public_scrobbles(client):
user = get_user_model().objects.create_user(
username="exploreuser", password="testpass"
)
public_task = Task.objects.create(title="Public Task Title")
shared_task = Task.objects.create(title="Shared Task Title")
private_task = Task.objects.create(title="Private Task Title")
ts = timezone.now()
public_scrobble = Scrobble.objects.create(
task=public_task, media_type="Task", user=user, visibility="public",
timestamp=ts,
)
Scrobble.objects.create(
task=shared_task, media_type="Task", user=user, visibility="shared",
timestamp=ts,
)
Scrobble.objects.create(
task=private_task, media_type="Task", user=user, visibility="private",
timestamp=ts,
)
url = reverse("scrobbles:explore")
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
assert "Public Task Title" in content
assert "Shared Task Title" not in content
assert "Private Task Title" not in content
@pytest.mark.django_db
def test_change_visibility_owner_can_change(client):
user = get_user_model().objects.create_user(
username="visuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
client.force_login(user)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
scrobble.refresh_from_db()
assert scrobble.visibility == "shared"
@pytest.mark.django_db
def test_change_visibility_non_owner_gets_404(client):
owner = get_user_model().objects.create_user(
username="owner", password="testpass"
)
other = get_user_model().objects.create_user(
username="other", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=owner, visibility="private",
timestamp=timezone.now(),
)
client.force_login(other)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 404
scrobble.refresh_from_db()
assert scrobble.visibility == "private"
@pytest.mark.django_db
def test_change_visibility_anonymous_redirects_to_login(client):
user = get_user_model().objects.create_user(
username="anontest", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
assert "/login/" in response.url
@pytest.mark.django_db
def test_regenerate_share_token_invalidates_old_sqid(client):
user = get_user_model().objects.create_user(
username="regentest", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
client.force_login(user)
url = reverse("scrobbles:regenerate-share-token", kwargs={"uuid": scrobble.uuid})
response = client.post(url)
assert response.status_code == 302
scrobble.refresh_from_db()
assert scrobble.share_token_version == 1
old_url = reverse("scrobbles:shared-detail", kwargs={"sqid": old_sqid})
old_response = client.get(old_url)
assert old_response.status_code == 404
new_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
new_url = reverse("scrobbles:shared-detail", kwargs={"sqid": new_sqid})
new_response = client.get(new_url)
assert new_response.status_code == 200
@pytest.mark.django_db
def test_share_analytics_owner_can_view(client):
user = get_user_model().objects.create_user(
username="analyticsuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_analytics_non_owner_gets_404(client):
owner = get_user_model().objects.create_user(
username="analyticsowner", password="testpass"
)
other = get_user_model().objects.create_user(
username="analyticsother", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=owner, visibility="shared",
timestamp=timezone.now(),
)
client.force_login(other)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_analytics_shows_view_logs(client):
user = get_user_model().objects.create_user(
username="analyticsviews", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
share_url = reverse("scrobbles:shared-detail", kwargs={"sqid": sqid})
client.get(share_url)
client.get(share_url)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
assert "127.0.0.1" in content
assert ShareViewLog.objects.filter(scrobble=scrobble).count() == 2

View File

@ -2,4 +2,5 @@
# Django starts so that shared_task will use this app.
from .celery import app as celery_app
__all__ = ("celery_app",)
__version__ = "42.0"
__all__ = ("celery_app", "__version__")

View File

@ -26,5 +26,5 @@ class BirdingLocationAdmin(admin.ModelAdmin):
@admin.register(BirdingCSVImport)
class BirdingCSVImportAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("uuid", "process_count", "processed_finished", "processing_started")
list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
ordering = ("-created",)

View File

@ -2,12 +2,15 @@ import csv
import logging
import re
from collections import defaultdict
from datetime import datetime, timedelta
from datetime import timedelta
from dateutil import parser
from django.contrib.auth import get_user_model
from birds.models import Bird, BirdSightingEntry, BirdSightingLogData, BirdingLocation
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
logger = logging.getLogger(__name__)
User = get_user_model()
@ -34,11 +37,12 @@ def parse_coords(location_str):
def parse_timestamp(date_str, time_str):
try:
dt = datetime.strptime(f"{date_str} {time_str}", "%B %d, %Y %I:%M %p")
dt_str = f"{date_str} {time_str}".strip()
dt = parser.parse(dt_str)
return dt
except (ValueError, TypeError):
try:
dt = datetime.strptime(date_str, "%B %d, %Y")
dt = parser.parse(date_str)
return dt
except (ValueError, TypeError):
logger.warning(f"Could not parse date/time: {date_str} {time_str}")
@ -60,7 +64,7 @@ def parse_int(value):
return None
def import_birding_csv(file_path, user_id):
def import_birding_csv(file_path, user_id, record_error=None):
user = User.objects.get(id=user_id)
new_scrobbles = []
@ -79,11 +83,17 @@ def import_birding_csv(file_path, user_id):
for (location_str, date_str, time_str), sighting_rows in groups.items():
if not location_str:
logger.warning("Skipping rows with no location")
msg = "Skipping rows with no location"
logger.warning(msg)
if record_error:
record_error(msg)
continue
timestamp = parse_timestamp(date_str, time_str)
if not timestamp:
msg = f"Could not parse date/time: {date_str} {time_str}"
if record_error:
record_error(msg)
continue
timestamp = user.profile.get_timestamp_with_tz(timestamp)
@ -183,4 +193,6 @@ def import_birding_csv(file_path, user_id):
created = Scrobble.objects.bulk_create(new_scrobbles)
logger.info(f"Created {len(created)} birding scrobbles")
for scrobble in created:
ScrobbleNtfyNotification(scrobble).send()
return created

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.29 on 2026-06-08 14:57
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("birds", "0002_birdingcsvimport"),
]
operations = [
migrations.AddField(
model_name="birdingcsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -61,9 +61,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
@cached_property
def bird_list(self) -> str:
if self.birds:
return ", ".join(
[BirdSightingEntry(**b).__str__() for b in self.birds]
)
return ", ".join([BirdSightingEntry(**b).__str__() for b in self.birds])
return ""
def as_html(self) -> str:
@ -80,9 +78,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
)
if self.area:
html_parts.append(
f'<div class="birding-area">Area: {self.area}</div>'
)
html_parts.append(f'<div class="birding-area">Area: {self.area}</div>')
if self.party_size:
html_parts.append(
@ -105,9 +101,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
)
if self.guide:
html_parts.append(
f'<div class="birding-guide">Guide: {self.guide}</div>'
)
html_parts.append(f'<div class="birding-guide">Guide: {self.guide}</div>')
if self.duration_minutes:
html_parts.append(
@ -183,9 +177,7 @@ class Bird(TimeStampedModel):
class BirdingLocation(ScrobblableMixin):
description = models.TextField(**BNULL)
geo_location = models.ForeignKey(
GeoLocation, **BNULL, on_delete=models.DO_NOTHING
)
geo_location = models.ForeignKey(GeoLocation, **BNULL, on_delete=models.DO_NOTHING)
ebird_hotspot_id = models.CharField(max_length=255, **BNULL)
def get_absolute_url(self):
@ -193,7 +185,7 @@ class BirdingLocation(ScrobblableMixin):
@property
def subtitle(self):
return ""
return self.geo_location
@property
def strings(self) -> ScrobblableConstants:
@ -224,6 +216,7 @@ class BirdingCSVImport(TimeStampedModel):
processed_finished = models.DateTimeField(**BNULL)
process_log = models.TextField(**BNULL)
process_count = models.IntegerField(**BNULL)
error_log = models.TextField(**BNULL)
csv_file = models.FileField(upload_to="birding-csv-uploads/", **BNULL)
class Meta:
@ -269,9 +262,7 @@ class BirdingCSVImport(TimeStampedModel):
self.save(update_fields=["process_log", "process_count"])
return
for count, scrobble in enumerate(scrobbles):
scrobble_str = (
f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
)
scrobble_str = f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
log_line = f"{scrobble_str}"
if count > 0:
log_line = "\n" + log_line
@ -279,6 +270,14 @@ class BirdingCSVImport(TimeStampedModel):
self.process_count = len(scrobbles)
self.save(update_fields=["process_log", "process_count"])
def record_error(self, error_message):
log_line = f"{timezone.now().isoformat()}: {error_message}"
if self.error_log:
self.error_log += "\n" + log_line
else:
self.error_log = log_line
self.save(update_fields=["error_log"])
def scrobbles(self):
from scrobbles.models import Scrobble
@ -297,6 +296,13 @@ class BirdingCSVImport(TimeStampedModel):
from birds.importer import import_birding_csv
self.mark_started()
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_birding_csv(
self.upload_file_path, self.user_id, record_error=self.record_error
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()

View File

@ -90,6 +90,26 @@ class BoardGameLogData(BaseLogData, LongPlayLogData):
"variant",
}
@classmethod
def override_fields(cls) -> dict:
from scrobbles.forms import NotesDictField
fields = {}
for base in cls.mro()[1:]:
if hasattr(base, "override_fields"):
base_fields = base.override_fields()
fields.update(base_fields)
custom_fields = {
"notes": NotesDictField(required=False),
"location_id": forms.ModelChoiceField(
queryset=BoardGameLocation.objects.all(),
required=False,
widget=forms.Select(),
),
}
fields.update(custom_fields)
return fields
@cached_property
def location(self):
if not self.location_id:
@ -133,23 +153,6 @@ class BoardGameLogData(BaseLogData, LongPlayLogData):
return "".join(html_parts)
@classmethod
def override_fields(cls) -> dict:
fields = {}
for base in cls.mro()[1:]:
if hasattr(base, "override_fields"):
base_fields = base.override_fields()
fields.update(base_fields)
custom_fields = {
"location_id": forms.ModelChoiceField(
queryset=BoardGameLocation.objects.all(),
required=False,
widget=forms.Select(),
)
}
fields.update(custom_fields)
return fields
class BoardGamePublisher(TimeStampedModel):
name = models.CharField(max_length=255)
@ -263,8 +266,9 @@ class BoardGame(ScrobblableMixin):
"self", **BNULL, on_delete=models.DO_NOTHING
)
def __str__(self):
return self.title
@property
def subtitle(self) -> str:
return self.publisher
def get_absolute_url(self):
return reverse("boardgames:boardgame_detail", kwargs={"slug": self.uuid})

View File

@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
"source": "Lichess",
"timezone": user.profile.timezone,
"log": log_data,
"visibility": "private",
}
if commit:
Scrobble.objects.create(**scrobble_dict)

View File

@ -1,9 +1,8 @@
import logging
import re
import sqlite3
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from enum import Enum
from zoneinfo import ZoneInfo
import requests
from books.constants import BOOKS_TITLES_TO_IGNORE
@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
return book_id_map
def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
def build_page_data(page_rows: list, book_map: dict) -> dict:
"""Given rows of page data from KoReader, parse each row and build
scrobbles for our user, loading the page data into the page_data
field on the scrobble instance.
@ -187,18 +186,20 @@ def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
book_ids_not_found.append(koreader_book_id)
continue
if "pages" not in book_map[koreader_book_id].keys():
book_map[koreader_book_id]["pages"] = {}
book_map[koreader_book_id].setdefault("pages", [])
page_number = page_row[KoReaderPageStatColumn.PAGE.value]
duration = page_row[KoReaderPageStatColumn.DURATION.value]
start_ts = page_row[KoReaderPageStatColumn.START_TIME.value]
book_map[koreader_book_id]["pages"][page_number] = {
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
book_map[koreader_book_id]["pages"].append(
{
"page_number": page_number,
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
)
if book_ids_not_found:
logger.info(f"Found pages for books not in file: {set(book_ids_not_found)}")
return book_map
@ -216,7 +217,6 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_not_found.append(book_id)
continue
should_create_scrobble = False
scrobble_page_data = {}
playback_position_seconds = 0
prev_page_stats = {}
@ -225,11 +225,12 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_processed = 0
total_pages_read = len(book_map[koreader_book_id]["pages"])
ordered_pages = sorted(
book_map[koreader_book_id]["pages"].items(),
key=lambda x: x[1]["start_ts"],
book_map[koreader_book_id]["pages"],
key=lambda x: x["start_ts"],
)
for cur_page_number, stats in ordered_pages:
for stats in ordered_pages:
page_number = stats["page_number"]
pages_processed += 1
seconds_from_last_page = 0
@ -243,46 +244,52 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
)
end_of_reading = pages_processed == total_pages_read
big_jump_to_this_page = (cur_page_number - last_page_number) > 10
big_jump_to_this_page = (page_number - last_page_number) > 10
is_session_gap = seconds_from_last_page > SESSION_GAP_SECONDS
if (is_session_gap and not big_jump_to_this_page) or end_of_reading:
should_create_scrobble = True
should_create_scrobble = (
is_session_gap and not big_jump_to_this_page
) or end_of_reading
# Always accumulate the current page first
scrobble_page_data[page_number] = stats
if should_create_scrobble:
# For end-of-reading, the current page is already accumulated
# and belongs in this scrobble. For a session gap, remove the
# current page from this scrobble — it starts a new session.
if is_session_gap and not end_of_reading:
del scrobble_page_data[page_number]
scrobble_page_data = dict(
sorted(
scrobble_page_data.items(),
key=lambda x: x[1]["start_ts"],
)
)
try:
first_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[0]
)
last_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[-1]
)
except IndexError:
if not scrobble_page_data:
logger.error(
"Could not process book, no page data found",
extra={"scrobble_page_data": scrobble_page_data},
)
continue
first_page = next(iter(scrobble_page_data.values()))
last_page = scrobble_page_data[
list(scrobble_page_data.keys())[-1]
]
timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(first_page.get("start_ts")))
datetime.fromtimestamp(
int(first_page.get("start_ts"))
)
)
stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(last_page.get("end_ts")))
datetime.fromtimestamp(
int(last_page.get("end_ts"))
)
)
# Adjust for Daylight Saving Time
# if timestamp.dst() == timedelta(
# 0
# ) or stop_timestamp.dst() == timedelta(0):
# timestamp = timestamp - timedelta(hours=1)
# stop_timestamp = stop_timestamp - timedelta(hours=1)
scrobble = Scrobble.objects.filter(
timestamp=timestamp,
book_id=book_id,
@ -291,7 +298,7 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
if not scrobble:
logger.info(
f"Queueing scrobble for {book_id}, page {cur_page_number}"
f"Queueing scrobble for {book_id}, page {page_number}"
)
log_data = {
"koreader_hash": book_dict.get("hash"),
@ -318,16 +325,18 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
timezone=tz,
)
)
# Then start over
should_create_scrobble = False
# Then start over for the next session
playback_position_seconds = 0
scrobble_page_data = {}
# We accumulate pages for the scrobble until we should create a new one
scrobble_page_data[cur_page_number] = stats
# For session gaps, re-add the current page as the
# beginning of the next session's accumulation
if is_session_gap and not end_of_reading:
scrobble_page_data[page_number] = stats
last_page_number = cur_page_number
last_page_number = page_number
prev_page_stats = stats
if pages_not_found:
logger.info(f"Pages not found for books: {set(pages_not_found)}")
return scrobbles_to_create
@ -357,9 +366,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
new_scrobbles = []
user = User.objects.filter(id=user_id).first()
tz = ZoneInfo("UTC")
if user:
tz = user.profile.tzinfo
is_os_file = "https://" not in file_path
if is_os_file:
@ -375,7 +381,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
book_map = build_page_data(
cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
book_map,
tz,
)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
else:
@ -392,7 +397,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
_sqlite_bytes(file_path), max_buffer_size=1_048_576
):
if table_name == "page_stat_data":
book_map = build_page_data(rows, book_map, tz)
book_map = build_page_data(rows, book_map)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
logger.info(f"Creating {len(new_scrobbles)} new scrobbles")

View File

@ -0,0 +1,282 @@
import logging
import time
from django.core.management.base import BaseCommand
from django.db import transaction
from books.constants import READCOMICSONLINE_URL
logger = logging.getLogger(__name__)
MISSING_ALL = [
"cover",
"summary",
"isbn",
"pages",
"language",
"publisher",
"publish_year",
]
MISSING_GROUPS = {
"cover": lambda b: not bool(b.cover),
"summary": lambda b: not b.summary,
"isbn": lambda b: not b.isbn_13 and not b.isbn_10,
"pages": lambda b: b.pages is None,
"language": lambda b: not b.language,
"publisher": lambda b: not b.publisher,
"publish_year": lambda b: b.first_publish_year is None,
}
def _book_matches(book, flags):
if not flags:
return False
for flag in flags:
fn = MISSING_GROUPS.get(flag)
if fn and fn(book):
return True
return False
class Command(BaseCommand):
help = "Backfill missing metadata on books from Google Books, OpenLibrary, and ComicVine"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of books to process per batch (default: 100)",
)
parser.add_argument(
"--sleep",
type=float,
default=0.5,
help="Seconds to sleep between API calls (default: 0.5)",
)
for flag in MISSING_ALL:
parser.add_argument(
f"--missing-{flag}",
dest="missing_flags",
action="append_const",
const=flag,
help=f"Process books missing {flag}",
)
parser.add_argument(
"--comics-only",
action="store_true",
help="Only process books with a readcomicsonline.ru URL",
)
parser.add_argument(
"--all",
action="store_true",
dest="all_missing",
help="Process books missing any metadata field",
)
def handle(self, *args, **options):
from books.models import Book
commit = options["commit"]
batch_size = options["batch_size"]
sleep_secs = options["sleep"]
flags = options.get("missing_flags") or []
comics_only = options["comics_only"]
all_missing = options["all_missing"]
if all_missing:
flags = MISSING_ALL
if not flags and not comics_only:
self.stdout.write(
"No filters specified. Use --all, --missing-*, or --comics-only."
)
return
qs = Book.objects.all()
if comics_only:
qs = qs.filter(readcomics_url__isnull=False)
if flags:
qs = [b for b in qs.iterator() if _book_matches(b, flags)]
else:
qs = list(qs)
total = len(qs)
self.stdout.write(f"Found {total} books to process")
if not commit:
self.stdout.write(
"Dry run — no API calls will be made. Use --commit to run lookups."
)
return
enriched = 0
skipped = 0
stats = {
"cover_fixed": 0,
"summary_fixed": 0,
"isbn_fixed": 0,
"pages_fixed": 0,
"language_fixed": 0,
"publisher_fixed": 0,
"publish_year_fixed": 0,
}
for batch_num, offset in enumerate(range(0, len(qs), batch_size)):
batch = qs[offset : offset + batch_size]
for book in batch:
result = self._enrich_book(book, sleep_secs)
if result:
enriched += 1
for key in stats:
if result.get(key):
stats[key] += 1
else:
skipped += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch)}/{total}"
f"enriched: {enriched}, skipped: {skipped}"
)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Books enriched: {enriched}\n"
f" Books skipped: {skipped}\n"
f" Covers fixed: {stats['cover_fixed']}\n"
f" Summaries fixed:{stats['summary_fixed']}\n"
f" ISBNs fixed: {stats['isbn_fixed']}\n"
f" Pages fixed: {stats['pages_fixed']}\n"
f" Languages fixed:{stats['language_fixed']}\n"
f" Publishers fixed:{stats['publisher_fixed']}\n"
f" Publish yrs fixed: {stats['publish_year_fixed']}"
)
def _enrich_book(self, book, sleep_secs):
from books.sources.comicvine import lookup_comic_from_comicvine
from books.sources.google import lookup_book_from_google
from books.sources.openlibrary import lookup_book_from_openlibrary as lookup_book_from_ol
title = book.original_title or book.title
author_name = book.author.name if book.author else None
book_dict = {}
is_comic = bool(book.readcomics_url) or (
book.issue_number is not None or book.volume_number is not None
)
if is_comic and READCOMICSONLINE_URL in (book.readcomics_url or ""):
cv_data = lookup_comic_from_comicvine(title)
if cv_data:
book_dict.update(cv_data)
ol_data = lookup_book_from_ol(title, author=author_name)
time.sleep(sleep_secs)
google_data = lookup_book_from_google(title)
if ol_data:
for k, v in ol_data.items():
book_dict.setdefault(k, v)
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
if not book_dict:
return None
changed = self._apply(book, book_dict, title)
return changed
def _apply(self, book, data, title):
changed = {
"cover_fixed": False,
"summary_fixed": False,
"isbn_fixed": False,
"pages_fixed": False,
"language_fixed": False,
"publisher_fixed": False,
"publish_year_fixed": False,
}
update_fields = []
cover_url = data.pop("cover_url", "")
if data.get("summary") and not book.summary:
book.summary = data["summary"]
update_fields.append("summary")
changed["summary_fixed"] = True
if data.get("isbn_13") and not book.isbn_13:
book.isbn_13 = data["isbn_13"]
update_fields.append("isbn_13")
changed["isbn_fixed"] = True
if data.get("isbn_10") and not book.isbn_10:
book.isbn_10 = data["isbn_10"]
update_fields.append("isbn_10")
changed["isbn_fixed"] = True
if data.get("pages") and book.pages is None:
book.pages = data["pages"]
update_fields.append("pages")
changed["pages_fixed"] = True
if data.get("language") and not book.language:
book.language = data["language"]
update_fields.append("language")
changed["language_fixed"] = True
if data.get("publisher") and not book.publisher:
book.publisher = data["publisher"]
update_fields.append("publisher")
changed["publisher_fixed"] = True
if data.get("first_publish_year") and book.first_publish_year is None:
book.first_publish_year = data["first_publish_year"]
update_fields.append("first_publish_year")
changed["publish_year_fixed"] = True
if data.get("openlibrary_id") and not book.openlibrary_id:
book.openlibrary_id = data["openlibrary_id"]
update_fields.append("openlibrary_id")
if data.get("comicvine_id") and not book.comicvine_id:
book.comicvine_id = data["comicvine_id"]
update_fields.append("comicvine_id")
if data.get("issue_number") and book.issue_number is None:
book.issue_number = data["issue_number"]
update_fields.append("issue_number")
if data.get("volume_number") and book.volume_number is None:
book.volume_number = data["volume_number"]
update_fields.append("volume_number")
if update_fields:
book.save(update_fields=update_fields)
self.stdout.write(f" [ENRICHED] {book}{', '.join(update_fields)}")
if cover_url and not book.cover:
book.save_image_from_url(cover_url)
if book.cover:
changed["cover_fixed"] = True
self.stdout.write(f" [COVER] {book} — cover saved from source")
genres = data.pop("genres", data.pop("generes", []))
if genres:
existing = set(book.genre.names())
new_genres = [g for g in genres if g not in existing]
if new_genres:
book.genre.add(*new_genres)
self.stdout.write(f" [GENRES] {book} — added {len(new_genres)} genres")
return changed if any(changed.values()) else None

View File

@ -0,0 +1,130 @@
import logging
from datetime import timedelta
from django.core.management.base import BaseCommand
from django.db import transaction
from books.koreader import SESSION_GAP_SECONDS, fix_long_play_stats_for_scrobbles
logger = logging.getLogger(__name__)
SESSION_GAP = timedelta(seconds=SESSION_GAP_SECONDS)
def _page_data_keys(pages):
return sorted(int(k) for k in (pages or {}))
class Command(BaseCommand):
help = "Merge orphaned 1-page KOReader scrobbles into the preceding scrobble"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
def handle(self, *args, **options):
from scrobbles.models import Scrobble
commit = options["commit"]
qs = Scrobble.objects.filter(
media_type="Book", source="KOReader"
).order_by("book_id", "timestamp")
if not qs.exists():
self.stdout.write("No KOReader book scrobbles found.")
return
merged = 0
affected_books = set()
# Group by book_id manually since we're iterating in order
book_scrobbles = {}
for s in qs:
book_scrobbles.setdefault(s.book_id, []).append(s)
if not commit:
self.stdout.write("Dry run — no changes will be saved. Use --commit to apply.")
for book_id, scrobbles in book_scrobbles.items():
batch_merged = 0
i = 0
while i < len(scrobbles) - 1:
current = scrobbles[i]
orphan = scrobbles[i + 1]
orphan_pages = orphan.logdata.page_data if orphan.logdata else {}
orphan_keys = _page_data_keys(orphan_pages)
if len(orphan_keys) != 1:
i += 1
continue
current_pages = current.logdata.page_data if current.logdata else {}
current_keys = _page_data_keys(current_pages)
if not current_keys:
i += 1
continue
orphan_page_num = orphan_keys[0]
current_last_page = current_keys[-1]
if orphan_page_num != current_last_page + 1:
i += 1
continue
# Check that the orphan is close enough in time
gap = orphan.timestamp - current.stop_timestamp
if gap > SESSION_GAP:
i += 1
continue
# Merge orphan into current
current_pages[str(orphan_page_num)] = orphan_pages[str(orphan_page_num)]
current.log["page_data"] = current_pages
current.log["pages_read"] = len(current_pages)
current.stop_timestamp = orphan.stop_timestamp
current.playback_position_seconds += orphan.playback_position_seconds
affected_books.add(book_id)
if commit:
with transaction.atomic():
current.save(
update_fields=[
"log",
"stop_timestamp",
"playback_position_seconds",
]
)
orphan.delete()
merged += 1
batch_merged += 1
scrobbles.pop(i + 1)
if batch_merged:
self.stdout.write(
f" Book {book_id}: merged {batch_merged} orphan scrobble(s)"
)
self.stdout.write(f"\nTotal orphans merged: {merged}")
if commit and affected_books:
self.stdout.write("Recalculating long_play_stats for affected books...")
for book_id in affected_books:
scrobbles_to_fix = (
Scrobble.objects.filter(book_id=book_id, source="KOReader")
.order_by("timestamp")
)
fix_long_play_stats_for_scrobbles(list(scrobbles_to_fix))
self.stdout.write(f"Fixed stats for {len(affected_books)} books.")
if not commit:
self.stdout.write(
f"\nWould merge {merged} orphan scrobble(s) across "
f"{len(affected_books)} book(s)."
)

View File

@ -4,6 +4,7 @@ from dataclasses import dataclass
from datetime import datetime
from typing import Optional
from uuid import uuid4
from zoneinfo import ZoneInfo
import requests
from books.constants import MediaSourceTag, READCOMICSONLINE_URL
@ -71,7 +72,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
_excluded_fields = {"koreader_hash", "page_data"}
def avg_seconds_per_page(self):
if self.page_data:
if self.page_data and isinstance(self.page_data, dict):
total_duration = 0
for page_num, stats in self.page_data.items():
total_duration += stats.get("duration", 0)
@ -173,11 +174,10 @@ class Book(LongPlayScrobblableMixin):
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
def __str__(self) -> str:
if self.issue_number and "Issue" not in str(self.title):
return f"{self.title} - Issue {self.issue_number}"
if self.volume_number and "Volume" not in str(self.title):
return f"{self.title} - Volume {self.volume_number}"
return f"{self.title}"
if not self.subtitle:
return self.title
return f"{self.title} - {self.subtitle}"
def save(self, *args, **kwargs):
if self.pages:
@ -188,7 +188,18 @@ class Book(LongPlayScrobblableMixin):
@property
def subtitle(self):
return f" by {self.author}"
subtitle_parts = []
if self.author:
subtitle_parts.append(self.author.name)
if self.issue_number and "Issue" not in str(self.title):
subtitle_parts.append(f"Issue {self.issue_number}")
if self.volume_number and "Volume" not in str(self.title):
subtitle_parts.append(f"Volume {self.volume_number}")
if len(subtitle_parts) > 1:
return " / ".join(subtitle_parts)
if len(subtitle_parts) == 1:
return subtitle_parts[0]
return ""
@property
def strings(self) -> ScrobblableConstants:
@ -218,36 +229,20 @@ class Book(LongPlayScrobblableMixin):
) -> "Book":
book, created = cls.objects.get_or_create(title=title)
if not created:
if not created and not overwrite:
return book
book_dict = lookup_comic_from_comicvine(title)
if not book_dict:
return book
if created or overwrite:
author_list = []
author_dicts = book_dict.pop("author_dicts")
if author_dicts:
for author_dict in author_dicts:
if author_dict.get("authorId"):
author, a_created = Author.objects.get_or_create(
semantic_id=author_dict.get("authorId")
)
author_list.append(author)
if a_created:
author.name = author_dict.get("name")
author.save()
# TODO enrich author?
...
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
if author_list:
book.authors.add(*author_list)
genres = book_dict.pop("genres", [])
if genres:
book.genre.add(*genres)
genres = book_dict.get("genres", [])
if genres:
book.genre.add(*genres)
return book
@classmethod
@ -285,20 +280,27 @@ class Book(LongPlayScrobblableMixin):
book_dict = lookup_comic_from_comicvine(title)
if book_dict:
source_tag = MediaSourceTag.COMICVINE
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
if not book_dict:
book_dict = lookup_book_from_ol(title, author=author)
if book_dict:
book_dict = {}
ol_data = lookup_book_from_ol(title, author=author)
google_data = lookup_book_from_google(title)
if ol_data:
book_dict.update(ol_data)
source_tag = MediaSourceTag.OPENLIBRARY
if not book_dict:
book_dict = lookup_book_from_google(title)
if book_dict:
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
source_tag = MediaSourceTag.GOOGLE_BOOKS
if ol_data and ol_data.get("cover_url"):
book_dict["cover_url"] = ol_data["cover_url"]
if not book_dict:
logger.warning(
@ -367,7 +369,6 @@ class Book(LongPlayScrobblableMixin):
if not data and COMICVINE_API_KEY:
logger.warn(f"Checking ComicVine for {self.title}")
cv_client = ComicVineClient(api_key=COMICVINE_API_KEY)
data = lookup_comic_from_comicvine(str(self.title))
if not data:
@ -462,8 +463,11 @@ class Book(LongPlayScrobblableMixin):
if scrobble.logdata.page_data:
for page, data in scrobble.logdata.page_data.items():
if convert_timestamps:
data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
tz = None
if scrobble.timezone:
tz = ZoneInfo(scrobble.timezone)
data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
pages[page] = data
sorted_pages = OrderedDict(
sorted(pages.items(), key=lambda x: x[1]["start_ts"])

View File

@ -18,7 +18,7 @@ class ComicVineClient(object):
"""
# All API requests made by this client will be made to this URL.
API_URL = "https://www.comicvine.com/api/search/"
API_URL = "https://comicvine.gamespot.com/api/search/"
# A valid User-Agent header must be set in order for our API requests to
# be accepted, otherwise our request will be rejected with a
@ -41,15 +41,12 @@ class ComicVineClient(object):
"volume",
}
def __init__(self, api_key, expire_after=300):
def __init__(self, api_key):
"""
Store the API key in a class variable, and install the requests cache,
configuring it using the ``expire_after`` parameter.
Store the API key in a class variable.
:param api_key: Your personal ComicVine API key.
:type api_key: str
:param expire_after: The number of seconds to retain an entry in cache.
:type expire_after: int or None
"""
self.api_key = api_key
@ -109,14 +106,17 @@ class ComicVineClient(object):
:rtype: dict
"""
return {
params = {
"api_key": self.api_key,
"format": "json",
"limit": min(10, limit), # hard limit of 10
"offset": max(0, offset), # cannot provide negative offset
"query": query,
"resources": self._validate_resources(resources),
}
validated = self._validate_resources(resources)
if validated:
params["resources"] = validated
return params
def _validate_resources(self, resources):
"""
@ -141,33 +141,35 @@ class ComicVineClient(object):
def _query_api(self, params):
"""
Query the ComicVine API's ``search`` resource, providing the required
headers and parameters with the request. Optionally allow the caller
of the function to disable the request cache.
headers and parameters with the request.
If an error occurs during the request, handle it accordingly. Upon
success, return the JSON from the response.
:param params: Parameters to include with the request.
:type params: dict
:param use_cache: Toggle the use of requests_cache.
:type use_cache: bool
:return: The JSON contained in the response.
:rtype: dict
"""
# Since we're performing the identical action regardless of whether
# or not the request cache is to be used, store the procedure in a
# local function to avoid repetition.
def __httpget():
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
if not response.ok:
self._handle_http_error(response)
return response.json()
json_data = response.json()
return __httpget()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data
def _handle_http_error(self, response):
"""
@ -200,10 +202,8 @@ def lookup_comic_from_comicvine(title: str) -> dict:
original_title = title
issue_number = None
volume_nubmer = None
resource_type = "issue"
if "Issue " in title:
resource_type = "issue"
issue_number = title.split("Issue ")[1]
volume_number = None
if "Volume " in title:
@ -215,48 +215,49 @@ def lookup_comic_from_comicvine(title: str) -> dict:
logger.warning("No ComicVine API key configured, not looking anything up")
return {}
client = ComicVineClient(api_key=getattr(settings, "COMICVINE_API_KEY", None))
client = ComicVineClient(api_key=api_key)
raw_results = client.search(title).get("results")
results = [r for r in raw_results if r.get("resource_type") == resource_type]
raw_results = client.search(title)
if not raw_results:
return {}
results = raw_results.get("results", [])
results = [r for r in results if r.get("resource_type") == resource_type]
if not results:
logger.warning("No comic found on ComicVine")
return {}
found_result = None
for result in results:
if result.get("issue_number") == str(issue_number):
if issue_number is not None and result.get("issue_number") == str(issue_number):
found_result = result
break
if result.get("volume_number") == str(volume_number):
if volume_number is not None and result.get("volume_number") == str(volume_number):
found_result = result
break
if not found_result:
found_result = results[0]
logger.info("ComicVine results", extra={"results": results})
if not found_result:
logger.warning("No matches found on ComicVine")
return {}
title = found_result.get("name")
if found_result.get("volume"):
title = found_result.get("volume").get("name")
cover_url = None
if found_result.get("image"):
cover_url = found_result["image"].get("original_url")
data_dict = {
"title": title,
"original_title": original_title,
"issue_number": found_result.get("issue_number"),
"volume_number": found_result.get("volume_number"),
"cover_url": found_result.get("image").get("original_url"),
"cover_url": cover_url,
"comicvine_id": found_result.get("id"),
"comicvine_data": found_result,
"summary": found_result.get("description"),
"publish_date": found_result.get("cover_date"),
"first_publish_year": found_result.get("cover_date", "")[:4],
"first_publish_year": (found_result.get("cover_date") or "")[:4],
}
return data_dict

View File

@ -26,8 +26,6 @@ def lookup_book_from_google(title: str) -> dict:
if not google_result:
return {}
publish_date = pendulum.parse(google_result.get("publishedDate"))
isbn_13 = ""
isbn_10 = ""
for ident in google_result.get("industryIdentifiers", []):
@ -35,25 +33,25 @@ def lookup_book_from_google(title: str) -> dict:
isbn_13 = ident.get("identifier")
if ident.get("type") == "ISBN_10":
isbn_10 = ident.get("identifier")
# TODO this may lead to issues with the first get if Google changes our title
# book_metadata.title = google_result.get("title")
# if google_result.get("subtitle"):
# book_metadata["title"] = ": ".join(
# [google_result.get("title"), google_result.get("subtitle")]
# )
# book_dict["subtitle"] = google_result.get("subtitle")
book_dict["authors"] = google_result.get("authors")
book_dict["publisher"] = google_result.get("publisher")
book_dict["first_publish_year"] = publish_date.year
book_dict["pages"] = google_result.get("pageCount")
book_dict["isbn_13"] = isbn_13
book_dict["isbn_10"] = isbn_10
book_dict["publish_date"] = google_result.get("publishedDate")
if len(book_dict["publish_date"]) == 4:
book_dict["publish_date"] = f"{book_dict['publish_date']}-1-1"
book_dict["language"] = google_result.get("language")
book_dict["summary"] = google_result.get("description")
book_dict["genres"] = google_result.get("categories")
raw_date = google_result.get("publishedDate")
if raw_date:
try:
publish_date = pendulum.parse(raw_date)
book_dict["first_publish_year"] = publish_date.year
except Exception:
pass
book_dict["publish_date"] = raw_date
if len(raw_date) == 4:
book_dict["publish_date"] = f"{raw_date}-1-1"
book_dict["cover_url"] = (
google_result.get("imageLinks", {})
.get("thumbnail", "")

View File

@ -32,8 +32,6 @@ class KoReaderBookRows:
DEFAULT_STR = "N/A"
DEFAULT_INT = 0
DEFAULT_TIME = 1703800469
BOOK_ROWS = []
PAGE_STATS_ROWS = []
def _gen_random_row(self, i):
wiggle = random.randrange(15)
@ -110,6 +108,8 @@ class KoReaderBookRows:
end_session = True
def __init__(self, book_count=0, **kwargs):
self.BOOK_ROWS = []
self.PAGE_STATS_ROWS = []
self._generate_random_book_rows(book_count)
self._generate_custom_book_row(**kwargs)
self._generate_random_page_stats_rows()

View File

@ -44,7 +44,10 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
# The test data generator adds the session-gap 3600s AFTER the trigger page
# (not before), so the first session includes 21 pages (1-21), and each
# subsequent session has 20 until the last. The last page is now included
# in the final scrobble instead of being orphaned.
expected_scrobbles = 6 * len(book_map.keys())
assert len(scrobbles) == expected_scrobbles
assert len(scrobbles[0].logdata.page_data.keys()) == 21
@ -52,7 +55,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
assert len(scrobbles[2].logdata.page_data.keys()) == 20
assert len(scrobbles[3].logdata.page_data.keys()) == 20
assert len(scrobbles[4].logdata.page_data.keys()) == 20
assert len(scrobbles[5].logdata.page_data.keys()) == 18
assert len(scrobbles[5].logdata.page_data.keys()) == 19
def test_get_author_str_from_row():

View File

@ -83,6 +83,12 @@
{% endfor %}
<a href="#" class="btn btn-xs btn-outline-secondary" style="padding:1px 3px;font-size:9px;" onclick="var w=prompt('Week number (1-52):');if(w)this.href='/charts/?date={{ year }}-W'+w.padStart(2,'0');return false;">+</a>
</div>
<div class="row mb-3">
<div class="col-12">
<a href="{% url 'charts:spotify-tracks' %}" class="btn btn-sm btn-outline-secondary">🎵 Spotify Tracks</a>
<a href="{% url 'charts:bandcamp-tracks' %}" class="btn btn-sm btn-outline-secondary">🎵 Bandcamp Tracks</a>
</div>
</div>
{% if chart_type == "maloja" %}
{% include "scrobbles/_top_charts.html" %}
{% else %}

View File

@ -7,7 +7,10 @@ register = template.Library()
def get_item(dictionary, key):
if isinstance(dictionary, dict):
return dictionary.get(key)
return None
try:
return dictionary[int(key)]
except (IndexError, KeyError, TypeError, ValueError):
return None
@register.filter

View File

@ -1,4 +1,10 @@
from charts.views import BirdsChartView, ChartDetailView, ChartRecordView, SpotifyTracksView
from charts.views import (
BandcampTracksView,
BirdsChartView,
ChartDetailView,
ChartRecordView,
SpotifyTracksView,
)
from django.urls import path
app_name = "charts"
@ -6,6 +12,7 @@ app_name = "charts"
urlpatterns = [
path("charts/", ChartRecordView.as_view(), name="charts-home"),
path("charts/spotify/", SpotifyTracksView.as_view(), name="spotify-tracks"),
path("charts/bandcamp/", BandcampTracksView.as_view(), name="bandcamp-tracks"),
path("charts/birds/", BirdsChartView.as_view(), name="birds-chart"),
path("charts/<slug:media_type>/", ChartDetailView.as_view(), name="chart-detail"),
]

View File

@ -104,8 +104,8 @@ def build_charts(
media_config = {
"artist": {
"filter": Q(track__isnull=False) & Q(track__artist__isnull=False),
"values": "track__artist",
"filter": Q(track__isnull=False) & Q(track__artists__isnull=False),
"values": "track__artists",
"annotate": Count("id", distinct=True),
},
"album": {

View File

@ -114,113 +114,112 @@ class ChartRecordView(TemplateView):
context["current_week"] = current_week
context["current_day"] = current_day
if chart_type == "maloja":
context["chart_keys"] = {
"today": "Today",
"week": "This Week",
"month": "This Month",
"year": "This Year",
"all": "All Time",
}
context["chart_keys"] = {
"today": "Today",
"week": "This Week",
"month": "This Month",
"year": "This Year",
"all": "All Time",
}
context["maloja_charts"] = {
"artist": {
"today": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "artist", year=current_year)
),
"all": list(self.get_charts_for_period(user, "artist")),
},
"track": {
"today": list(
self.get_charts_for_period(
user,
"track",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user, "track", year=current_year, week=current_week
)
),
"month": list(
self.get_charts_for_period(
user,
"track",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "track", year=current_year)
),
"all": list(self.get_charts_for_period(user, "track")),
},
"tv_series": {
"today": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "tv_series", year=current_year)
),
"all": list(self.get_charts_for_period(user, "tv_series")),
},
}
return context
context["maloja_charts"] = {
"artist": {
"today": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "artist", year=current_year)
),
"all": list(self.get_charts_for_period(user, "artist")),
},
"album": {
"today": list(
self.get_charts_for_period(
user,
"album",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user, "album", year=current_year, week=current_week
)
),
"month": list(
self.get_charts_for_period(
user,
"album",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "album", year=current_year)
),
"all": list(self.get_charts_for_period(user, "album")),
},
"tv_series": {
"today": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "tv_series", year=current_year)
),
"all": list(self.get_charts_for_period(user, "tv_series")),
},
}
if not date_param:
context["period"] = "current"
context["year"] = current_year
context["month"] = current_month
context["month_name"] = calendar.month_name[current_month]
context["week"] = current_week
context["day"] = current_day
@ -301,6 +300,7 @@ class ChartRecordView(TemplateView):
context["period"] = "historical"
context["year"] = year
context["month"] = month
context["month_name"] = calendar.month_name[month] if month else None
context["week"] = week
context["day"] = day
@ -604,9 +604,9 @@ class ChartRecordView(TemplateView):
if bird_id:
bird_counts[bird_id] = bird_counts.get(bird_id, 0) + quantity
sorted_birds = sorted(
bird_counts.items(), key=lambda x: x[1], reverse=True
)[:limit]
sorted_birds = sorted(bird_counts.items(), key=lambda x: x[1], reverse=True)[
:limit
]
bird_ids = [bid for bid, _ in sorted_birds]
birds = Bird.objects.filter(id__in=bird_ids)
@ -727,12 +727,22 @@ class SpotifyTracksView(TemplateView):
template_name = "charts/spotify_tracks.html"
def get_spotify_tracks(self, user, limit=50):
non_spotify_mopidy_tracks = (
Scrobble.objects.filter(
user=user,
source="Mopidy",
track__isnull=False,
)
.exclude(log__mopidy_source="spotify")
.values("track")
)
track_ids = (
Scrobble.objects.filter(
user=user,
track__isnull=False,
)
.filter(Q(source="Last.fm") | Q(log__mopidy_source="spotify"))
.exclude(track__in=non_spotify_mopidy_tracks)
.values("track")
.annotate(count=Count("id"))
.order_by("-count")[:limit]
@ -760,6 +770,53 @@ class SpotifyTracksView(TemplateView):
return context
class BandcampTracksView(TemplateView):
template_name = "charts/bandcamp_tracks.html"
def get_bandcamp_tracks(self, user, limit=50):
non_bandcamp_mopidy_tracks = (
Scrobble.objects.filter(
user=user,
source="Mopidy",
track__isnull=False,
)
.exclude(log__mopidy_source="bandcamp")
.values("track")
)
track_ids = (
Scrobble.objects.filter(
user=user,
track__isnull=False,
log__mopidy_source="bandcamp",
)
.exclude(track__in=non_bandcamp_mopidy_tracks)
.values("track")
.annotate(count=Count("id"))
.order_by("-count")[:limit]
)
from music.models import Track
track_id_list = [item["track"] for item in track_ids]
tracks = Track.objects.filter(id__in=track_id_list)
track_map = {t.id: t for t in tracks}
return [
{
"track": track_map[tid],
"count": next(
(item["count"] for item in track_ids if item["track"] == tid), 0
),
}
for tid in track_id_list
if tid in track_map
]
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
user = self.request.user
context["bandcamp_tracks"] = self.get_bandcamp_tracks(user)
return context
class BirdsChartView(TemplateView):
template_name = "charts/birds_chart.html"
@ -792,9 +849,9 @@ class BirdsChartView(TemplateView):
if bird_id:
bird_counts[bird_id] = bird_counts.get(bird_id, 0) + quantity
sorted_birds = sorted(
bird_counts.items(), key=lambda x: x[1], reverse=True
)[:limit]
sorted_birds = sorted(bird_counts.items(), key=lambda x: x[1], reverse=True)[
:limit
]
bird_ids = [bid for bid, _ in sorted_birds]
birds = Bird.objects.filter(id__in=bird_ids)

View File

@ -1,7 +1,5 @@
from django.contrib import admin
from music.models import Artist, Album, Track
from music.models import Album, Artist, Track
from scrobbles.admin import ScrobbleInline
@ -11,7 +9,7 @@ class AlbumAdmin(admin.ModelAdmin):
list_display = (
"name",
"year",
"album_artist",
"artist",
"theaudiodb_genre",
"theaudiodb_mood",
"musicbrainz_id",
@ -54,12 +52,12 @@ class TrackAdmin(admin.ModelAdmin):
"artist",
"musicbrainz_id",
)
raw_id_fields = ("artist", "albums", "album")
list_filter = ("album", "artist")
raw_id_fields = ("artist_fk", "artists", "albums", "album")
search_fields = ("title",)
ordering = ("-created",)
filter_horizontal = [
"albums",
"artists",
]
inlines = [
ScrobbleInline,

View File

@ -149,7 +149,7 @@ def live_charts(
def artist_scrobble_count(artist_id: int, filter: str = "today") -> int:
return Scrobble.objects.filter(track__artist=artist_id).count()
return Scrobble.objects.filter(track__artists=artist_id).count()
def live_tv_charts(
@ -158,7 +158,7 @@ def live_tv_charts(
limit: int = 15,
) -> QuerySet:
from django.db.models import OuterRef, Subquery
from videos.models import Video, Series
from videos.models import Series, Video
now = timezone.now()
tzinfo = now.tzinfo
@ -215,7 +215,7 @@ def live_youtube_channel_charts(
limit: int = 15,
) -> QuerySet:
from django.db.models import OuterRef, Subquery
from videos.models import Video, Channel
from videos.models import Channel, Video
now = timezone.now()
tzinfo = now.tzinfo

View File

@ -1,3 +1,5 @@
from taggit.serializers import TaggitSerializer, TagListSerializerField
from music.models import Album, Artist, Track
from rest_framework import serializers
@ -14,7 +16,25 @@ class AlbumSerializer(serializers.HyperlinkedModelSerializer):
fields = "__all__"
class TrackSerializer(serializers.HyperlinkedModelSerializer):
class TrackSerializer(TaggitSerializer, serializers.HyperlinkedModelSerializer):
tags = TagListSerializerField()
genre = TagListSerializerField()
class Meta:
model = Track
fields = "__all__"
fields = [
"url",
"id",
"uuid",
"title",
"artist_fk",
"artists",
"album",
"albums",
"musicbrainz_id",
"genre",
"tags",
"base_run_time_seconds",
"created",
"modified",
]

View File

@ -0,0 +1,90 @@
import logging
import requests
logger = logging.getLogger(__name__)
LB_LABS_API = "https://labs.api.listenbrainz.org"
STANDARD_ALGORITHM = "session_based_days_180_session_300_contribution_5_threshold_15_limit_50_skip_30"
MLHD_ALGORITHM = "session_based_mlhd_session_300_contribution_5_threshold_15_limit_50_skip_30"
SIMILAR_ARTISTS_ALGORITHM = "session_based_days_1825_session_300_contribution_3_threshold_10_limit_100_filter_True_skip_30"
def get_similar_recordings(recording_mbid: str) -> list[dict]:
"""Fetch similar recordings from ListenBrainz.
Tries the standard dataset first, falls back to MLHD dataset.
Returns [] on error or no data.
"""
if not recording_mbid:
return []
try:
resp = requests.post(
f"{LB_LABS_API}/similar-recordings/json",
json=[
{
"recording_mbids": [recording_mbid],
"algorithm": STANDARD_ALGORITHM,
}
],
timeout=10,
)
resp.raise_for_status()
data = resp.json()
if data:
return data
resp = requests.post(
f"{LB_LABS_API}/mlhd-similar-recordings/json",
json=[
{
"recording_mbids": [recording_mbid],
"algorithm": MLHD_ALGORITHM,
}
],
timeout=10,
)
resp.raise_for_status()
return resp.json()
except requests.RequestException as e:
logger.warning(
"ListenBrainz similar recordings error",
extra={
"recording_mbid": recording_mbid,
"error": str(e),
},
)
return []
def get_similar_artists(artist_mbid: str) -> list[dict]:
"""Fetch similar artists from ListenBrainz.
Returns [] on error or no data.
"""
if not artist_mbid:
return []
try:
resp = requests.post(
f"{LB_LABS_API}/similar-artists/json",
json=[
{
"artist_mbids": [artist_mbid],
"algorithm": SIMILAR_ARTISTS_ALGORITHM,
}
],
timeout=10,
)
resp.raise_for_status()
return resp.json()
except requests.RequestException as e:
logger.warning(
"ListenBrainz similar artists error",
extra={
"artist_mbid": artist_mbid,
"error": str(e),
},
)
return []

View File

@ -0,0 +1,249 @@
import logging
import time
import musicbrainzngs
from django.core.management.base import BaseCommand
from django.db import transaction
from music.musicbrainz import resolve_track, get_track_metadata_with_artist, extract_featured_artists
logger = logging.getLogger(__name__)
VARIOUS_ARTISTS_NAMES = ["various artists", "va", "various"]
def _is_various_artists(artist_name: str) -> bool:
return artist_name.strip().casefold() in VARIOUS_ARTISTS_NAMES
def _get_artist_from_scrobble_logs(track_id: int):
from scrobbles.models import Scrobble
scrobbles = Scrobble.objects.filter(track_id=track_id)[:10]
for scrobble in scrobbles:
raw = scrobble.log.get("raw_data", {})
if not raw:
continue
provider_mbid = raw.get("Provider_musicbrainztrack")
if provider_mbid:
return {"artist_name": raw.get("Artist", ""), "recording_mbid": provider_mbid}
artist = raw.get("Artist", "")
if artist:
return {"artist_name": artist, "recording_mbid": None}
return None
def _lookup_recording_by_mbid(mbid: str):
try:
result = musicbrainzngs.get_recording_by_id(mbid, includes=["artists"])
recording = result["recording"]
length_ms = recording.get("length")
artist_credit = recording.get("artist-credit", [{}])
primary_artist = artist_credit[0].get("artist", {}) if artist_credit else {}
return {
"recording_mbid": mbid,
"length_ms": int(length_ms) if length_ms else None,
"artist_name": primary_artist.get("name", ""),
"artist_mbid": primary_artist.get("id", ""),
}
except (musicbrainzngs.WebServiceError, musicbrainzngs.ResponseError) as e:
logger.warning(f"MusicBrainz error looking up {mbid}: {e}")
return None
def _lookup_length_by_mbid(mbid: str):
try:
result = musicbrainzngs.get_recording_by_id(mbid)
recording = result["recording"]
length_ms = recording.get("length")
if length_ms:
return int(length_ms)
return None
except (musicbrainzngs.WebServiceError, musicbrainzngs.ResponseError) as e:
logger.warning(f"MusicBrainz error looking up {mbid}: {e}")
return None
class Command(BaseCommand):
help = "Clean up metadata on music tracks from MusicBrainz"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of tracks to process per batch (default: 100)",
)
parser.add_argument(
"--only-missing-mbid",
action="store_true",
help="Only process tracks missing musicbrainz_id",
)
parser.add_argument(
"--only-missing-length",
action="store_true",
help="Only process tracks missing base_run_time_seconds",
)
def handle(self, *args, **options):
from music.models import Track
commit = options["commit"]
batch_size = options["batch_size"]
if not commit:
self.stdout.write("Dry run — no changes will be saved. Use --commit to apply.")
musicbrainzngs.set_useragent("vrobbler", "0.3.0")
qs = Track.objects.all()
if options["only_missing_mbid"]:
qs = qs.filter(musicbrainz_id__isnull=True)
if options["only_missing_length"]:
qs = qs.filter(base_run_time_seconds__isnull=True)
total = qs.count()
self.stdout.write(f"Processing {total} tracks")
missing_mbid_qs = qs.filter(musicbrainz_id__isnull=True)
missing_length_qs = qs.filter(
musicbrainz_id__isnull=False, base_run_time_seconds__isnull=True
)
self.stdout.write(
f" {missing_mbid_qs.count()} tracks without musicbrainz_id"
)
self.stdout.write(
f" {missing_length_qs.count()} tracks with mbid but no run time"
)
if not commit:
self.stdout.write("\nSkipping API lookups in dry-run mode. Use --commit to run against MusicBrainz.")
return
found_count = 0
not_found_count = 0
length_fixed_count = 0
va_fixed_count = 0
track_ids = list(missing_mbid_qs.values_list("pk", flat=True))
i = 0
for batch_num, offset in enumerate(range(0, len(track_ids), batch_size)):
batch_pks = track_ids[offset : offset + batch_size]
with transaction.atomic():
for track in Track.objects.filter(pk__in=batch_pks).iterator():
i += 1
artist_name = ""
artist_obj = track.artist
if artist_obj:
artist_name = artist_obj.name
is_va = _is_various_artists(artist_name)
result = None
method = ""
if is_va:
scrobble_info = _get_artist_from_scrobble_logs(track.id)
if scrobble_info:
if scrobble_info["recording_mbid"]:
result = _lookup_recording_by_mbid(
scrobble_info["recording_mbid"]
)
if result:
method = "provider"
self.stdout.write(
f" VA track '{track.title}' matched via provider MBID "
f"{scrobble_info['recording_mbid']} (artist: {scrobble_info['artist_name']})"
)
else:
result, method = resolve_track(
track.title, scrobble_info["artist_name"]
)
if result:
self.stdout.write(
f" VA track '{track.title}' matched via log artist "
f"'{scrobble_info['artist_name']}' as {result['recording_mbid']} ({method})"
)
if not result:
result, method = resolve_track(track.title)
if result:
self.stdout.write(
f" VA track '{track.title}' matched by title alone "
f"as {result['recording_mbid']} ({method})"
)
if not result:
result, method = resolve_track(track.title, artist_name)
if result and result.get("recording_mbid"):
track.musicbrainz_id = result["recording_mbid"]
length_ms = result.get("length_ms")
if length_ms and not track.base_run_time_seconds:
track.base_run_time_seconds = int(int(length_ms) / 1000)
track.save(update_fields=["musicbrainz_id", "base_run_time_seconds"])
method_tag = f"musicbrainz-{method}" if method else "musicbrainz-enriched"
track.tags.add(method_tag, "musicbrainz-enriched")
from music.models import Artist
cleaned_title, featured_names = extract_featured_artists(track.title)
for feat_name in featured_names:
artist = Artist.find_or_create(feat_name, track_name=track.title)
if artist:
track.artists.add(artist)
self.stdout.write(
f" [{i}] FOUND {track} — mbid={track.musicbrainz_id} ({method})"
)
found_count += 1
else:
track.tags.add("musicbrainz-notfound")
self.stdout.write(
f" [{i}] NOTFOUND {track}"
)
not_found_count += 1
if is_va and result:
va_fixed_count += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch_pks)}/{total} processed, "
f"found: {found_count}, not found: {not_found_count}"
)
time.sleep(1)
length_ids = list(missing_length_qs.values_list("pk", flat=True))
j = 0
for batch_num, offset in enumerate(range(0, len(length_ids), batch_size)):
batch_pks = length_ids[offset : offset + batch_size]
with transaction.atomic():
for track in Track.objects.filter(pk__in=batch_pks).iterator():
j += 1
length_ms = _lookup_length_by_mbid(track.musicbrainz_id)
if length_ms:
track.base_run_time_seconds = int(int(length_ms) / 1000)
track.tags.add("musicbrainz-enriched")
track.save(update_fields=["base_run_time_seconds"])
self.stdout.write(
f" [{j}] LENGTH {track}{track.base_run_time_seconds}s"
)
length_fixed_count += 1
self.stdout.write(
f" Batch {batch_num + 1}: {j}/{len(length_ids)} length lookups"
)
time.sleep(1)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Found on MusicBrainz: {found_count}\n"
f" Not found / tagged musicbrainz-notfound: {not_found_count}\n"
f" Run times filled from MBID: {length_fixed_count}\n"
f" Various Artists tracks resolved: {va_fixed_count}"
)

View File

@ -0,0 +1,123 @@
import logging
import time
from django.core.management.base import BaseCommand
from music.listenbrainz import get_similar_artists
logger = logging.getLogger(__name__)
class Command(BaseCommand):
help = "Fetch similar artists from ListenBrainz for artists with a musicbrainz_id"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of artists to process per batch (default: 100)",
)
parser.add_argument(
"--artist-id",
type=str,
default="",
help="Only process the artist with this musicbrainz_id",
)
parser.add_argument(
"--overwrite",
action="store_true",
help="Re-fetch similar artists even if already populated",
)
def handle(self, *args, **options):
from music.models import Artist
commit = options["commit"]
batch_size = options["batch_size"]
artist_id = options["artist_id"]
overwrite = options["overwrite"]
if not commit:
self.stdout.write(
"Dry run — no changes will be saved. Use --commit to apply."
)
qs = Artist.objects.exclude(musicbrainz_id__isnull=True).exclude(
musicbrainz_id=""
)
if artist_id:
qs = qs.filter(musicbrainz_id=artist_id)
self.stdout.write(f"Filtering to artist with musicbrainz_id: {artist_id}")
elif not overwrite:
qs = qs.filter(similar_artists__isnull=True)
else:
qs = qs.all()
total = qs.count()
if total == 0:
self.stdout.write("No artists to process.")
return
self.stdout.write(
f"Found {total} artists with musicbrainz_id"
+ (" (overwrite mode)" if overwrite else "")
+ (" (--artist-id filter)" if artist_id else "")
)
if not commit:
self.stdout.write(
"\nSkipping API lookups in dry-run mode. Use --commit to run against ListenBrainz."
)
return
found_count = 0
empty_count = 0
error_count = 0
artist_ids = list(qs.values_list("pk", flat=True))
i = 0
for batch_num, offset in enumerate(
range(0, len(artist_ids), batch_size)
):
batch_pks = artist_ids[offset : offset + batch_size]
for artist in Artist.objects.filter(pk__in=batch_pks).iterator():
i += 1
self.stdout.write(
f" [{i}/{total}] Fetching similar artists for {artist}...",
ending="",
)
try:
similar = get_similar_artists(artist.musicbrainz_id)
if similar:
artist.similar_artists = similar
artist.save(update_fields=["similar_artists"])
self.stdout.write(f" {len(similar)} similar artists found")
found_count += 1
else:
artist.similar_artists = []
artist.save(update_fields=["similar_artists"])
self.stdout.write(" none found")
empty_count += 1
except Exception as e:
self.stdout.write(f" error: {e}")
error_count += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch_pks)}/{total} processed, "
f"found: {found_count}, empty: {empty_count}, errors: {error_count}"
)
time.sleep(1)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Similar artists found: {found_count}\n"
f" No similar artists: {empty_count}\n"
f" Errors: {error_count}"
)

View File

@ -0,0 +1,125 @@
import logging
import time
from django.core.management.base import BaseCommand
from music.listenbrainz import get_similar_recordings
logger = logging.getLogger(__name__)
class Command(BaseCommand):
help = "Fetch similar recordings from ListenBrainz for tracks with a musicbrainz_id"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of tracks to process per batch (default: 100)",
)
parser.add_argument(
"--track-id",
type=str,
default="",
help="Only process the track with this musicbrainz_id",
)
parser.add_argument(
"--overwrite",
action="store_true",
help="Re-fetch similar recordings even if already populated",
)
def handle(self, *args, **options):
from music.models import Track
commit = options["commit"]
batch_size = options["batch_size"]
track_id = options["track_id"]
overwrite = options["overwrite"]
if not commit:
self.stdout.write(
"Dry run — no changes will be saved. Use --commit to apply."
)
qs = Track.objects.exclude(musicbrainz_id__isnull=True).exclude(
musicbrainz_id=""
)
if track_id:
qs = qs.filter(musicbrainz_id=track_id)
self.stdout.write(f"Filtering to track with musicbrainz_id: {track_id}")
elif not overwrite:
qs = qs.filter(similar_recordings__isnull=True)
else:
qs = qs.all()
total = qs.count()
if total == 0:
self.stdout.write("No tracks to process.")
return
self.stdout.write(
f"Found {total} tracks with musicbrainz_id"
+ (" (overwrite mode)" if overwrite else "")
+ (" (--track-id filter)" if track_id else "")
)
if not commit:
self.stdout.write(
"\nSkipping API lookups in dry-run mode. Use --commit to run against ListenBrainz."
)
return
found_count = 0
empty_count = 0
error_count = 0
track_ids = list(qs.values_list("pk", flat=True))
i = 0
for batch_num, offset in enumerate(
range(0, len(track_ids), batch_size)
):
batch_pks = track_ids[offset : offset + batch_size]
for track in Track.objects.filter(pk__in=batch_pks).iterator():
i += 1
self.stdout.write(
f" [{i}/{total}] Fetching similar recordings for {track}...",
ending="",
)
try:
similar = get_similar_recordings(track.musicbrainz_id)
if similar:
track.similar_recordings = similar
track.save(update_fields=["similar_recordings"])
self.stdout.write(
f" {len(similar)} similar recordings found"
)
found_count += 1
else:
track.similar_recordings = []
track.save(update_fields=["similar_recordings"])
self.stdout.write(" none found")
empty_count += 1
except Exception as e:
self.stdout.write(f" error: {e}")
error_count += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch_pks)}/{total} processed, "
f"found: {found_count}, empty: {empty_count}, errors: {error_count}"
)
time.sleep(1)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Similar recordings found: {found_count}\n"
f" No similar recordings: {empty_count}\n"
f" Errors: {error_count}"
)

View File

@ -0,0 +1,127 @@
import csv
import logging
from django.core.management.base import BaseCommand
logger = logging.getLogger(__name__)
def _get_source(raw_data):
if "Artist" in raw_data:
return "Jellyfin"
if "artist" in raw_data:
return "Mopidy"
return None
def _get_raw_values(raw_data, source):
if source == "Jellyfin":
return raw_data.get("Artist", ""), raw_data.get("Album", "")
return raw_data.get("artist", ""), raw_data.get("album", "")
def _normalize(name):
return name.strip().casefold()
def _artist_mismatch(raw_artist, track_artist_names):
if not raw_artist or not track_artist_names:
return False
track_names = [_normalize(n) for n in track_artist_names.split(" / ")]
raw = _normalize(raw_artist)
if raw in track_names:
return False
if raw == _normalize(track_artist_names):
return False
return True
def _album_mismatch(raw_album, track_album_name):
if not raw_album or not track_album_name:
return False
return _normalize(raw_album) != _normalize(track_album_name)
class Command(BaseCommand):
help = (
"Outputs a CSV of track IDs where raw metadata from scrobble logs "
"does not match the track's stored artists or album"
)
def add_arguments(self, parser):
parser.add_argument(
"--file-path",
type=str,
default="/tmp/metadata-report.csv",
help="Output CSV file path (default: /tmp/metadata-report.csv)",
)
def handle(self, *args, **options):
from scrobbles.models import Scrobble
file_path = options["file_path"]
qs = (
Scrobble.objects.filter(media_type=Scrobble.MediaType.TRACK)
.exclude(log__isnull=True)
.exclude(log={})
.select_related("track__album")
.prefetch_related("track__artists")
.iterator()
)
rows = []
for scrobble in qs:
track = scrobble.track
if not track:
continue
raw_data = scrobble.log.get("raw_data")
if not raw_data:
continue
source = _get_source(raw_data)
if not source:
continue
raw_artist, raw_album = _get_raw_values(raw_data, source)
if not raw_artist and not raw_album:
continue
track_artist_names = " / ".join(
track.artists.all().values_list("name", flat=True)
)
track_album_name = track.album.name if track.album else ""
if _artist_mismatch(raw_artist, track_artist_names) or _album_mismatch(
raw_album, track_album_name
):
rows.append(
{
"track_id": track.id,
"track_artist_name": track_artist_names,
"track_album_name": track_album_name,
"raw_artist": raw_artist,
"raw_album": raw_album,
"source": source,
}
)
fieldnames = [
"track_id",
"track_artist_name",
"track_album_name",
"raw_artist",
"raw_album",
"source",
]
with open(file_path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
self.stdout.write(
self.style.SUCCESS(
f"Wrote {len(rows)} mismatched track(s) to {file_path}"
)
)

View File

@ -0,0 +1,32 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0031_alter_track_genre"),
]
operations = [
migrations.RenameField(
model_name="track",
old_name="artist",
new_name="artist_fk",
),
migrations.AlterField(
model_name="track",
name="artist_fk",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=models.DO_NOTHING,
related_name="+",
to="music.artist",
),
),
migrations.AddField(
model_name="track",
name="artists",
field=models.ManyToManyField(to="music.artist"),
),
]

View File

@ -0,0 +1,27 @@
from django.db import migrations
def backfill_artist_fk_to_artists(apps, schema_editor):
Track = apps.get_model("music", "Track")
TrackArtist = Track.artists.through
TrackArtist.objects.bulk_create(
[
TrackArtist(track_id=r["id"], artist_id=r["artist_fk_id"])
for r in Track.objects.filter(artist_fk__isnull=False).values("id", "artist_fk_id")
],
ignore_conflicts=True,
)
class Migration(migrations.Migration):
dependencies = [
("music", "0032_track_artist_fk_and_artists"),
]
operations = [
migrations.RunPython(
backfill_artist_fk_to_artists,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,29 @@
from django.db import migrations
def backfill_album_artist_to_artists(apps, schema_editor):
Album = apps.get_model("music", "Album")
AlbumArtist = Album.artists.through
AlbumArtist.objects.bulk_create(
[
AlbumArtist(album_id=r["id"], artist_id=r["album_artist_id"])
for r in Album.objects.filter(album_artist__isnull=False).values(
"id", "album_artist_id"
)
],
ignore_conflicts=True,
)
class Migration(migrations.Migration):
dependencies = [
("music", "0033_backfill_track_artists"),
]
operations = [
migrations.RunPython(
backfill_album_artist_to_artists,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.30 on 2026-05-29 23:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0034_backfill_album_artists"),
]
operations = [
migrations.AddField(
model_name="track",
name="similar_recordings",
field=models.JSONField(blank=True, null=True),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.30 on 2026-05-30 00:00
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0035_track_similar_recordings"),
]
operations = [
migrations.AddField(
model_name="artist",
name="similar_artists",
field=models.JSONField(blank=True, null=True),
),
]

View File

@ -16,9 +16,11 @@ from imagekit.processors import ResizeToFit
from music.allmusic import get_allmusic_slug, scrape_data_from_allmusic
from music.bandcamp import get_bandcamp_slug
from music.musicbrainz import (
extract_featured_artists,
get_album_metadata_with_artist,
get_recording_mbid_exact,
get_track_metadata_with_artist,
resolve_track,
)
from music.theaudiodb import lookup_album_from_tadb, lookup_artist_from_tadb
from music.utils import clean_artist_name
@ -54,6 +56,7 @@ class Artist(TimeStampedModel):
theaudiodb_genre = models.CharField(max_length=255, **BNULL)
theaudiodb_mood = models.CharField(max_length=255, **BNULL)
musicbrainz_id = models.CharField(max_length=255, **BNULL)
similar_artists = models.JSONField(**BNULL)
allmusic_id = models.CharField(max_length=100, **BNULL)
bandcamp_id = models.CharField(max_length=100, **BNULL)
thumbnail = models.ImageField(upload_to="artist/", **BNULL)
@ -124,7 +127,7 @@ class Artist(TimeStampedModel):
def charts(self):
from scrobbles.models import ChartRecord
return ChartRecord.objects.filter(track__artist=self).order_by("-year")
return ChartRecord.objects.filter(track__artists=self).order_by("-year")
def scrape_allmusic(self, force=False) -> None:
if not self.allmusic_id or force:
@ -294,7 +297,7 @@ class Album(TimeStampedModel):
alt_names = models.TextField(**BNULL)
def __str__(self) -> str:
return "{} by {}".format(self.name, self.album_artist)
return "{} by {}".format(self.name, self.artist)
def get_absolute_url(self):
return reverse("music:album_detail", kwargs={"slug": self.uuid})
@ -320,6 +323,10 @@ class Album(TimeStampedModel):
.order_by("-scrobble_count")
)
@property
def artist(self):
return self.artists.first() or self.album_artist
def fix_album_artist(self):
from music.utils import get_or_create_various_artists
@ -338,10 +345,10 @@ class Album(TimeStampedModel):
)
return
if self.album_artist and (not self.allmusic_id or force):
slug = get_allmusic_slug(self.album_artist.name, self.name)
if self.artist and (not self.allmusic_id or force):
slug = get_allmusic_slug(self.artist.name, self.name)
if not slug:
logger.info(f"No allmsuic link for {self} by {self.album_artist}")
logger.info(f"No allmsuic link for {self} by {self.artist}")
return
self.allmusic_id = slug
self.save(update_fields=["allmusic_id"])
@ -351,7 +358,7 @@ class Album(TimeStampedModel):
allmusic_data = scrape_data_from_allmusic(self.allmusic_link)
if not allmusic_data:
logger.info(f"No allmsuic data for {self} by {self.album_artist}")
logger.info(f"No allmsuic data for {self} by {self.artist}")
return
self.allmusic_review = allmusic_data["review"]
@ -360,8 +367,8 @@ class Album(TimeStampedModel):
def scrape_theaudiodb(self) -> None:
artist = "Various Artists"
if self.album_artist:
artist = self.album_artist.name
if self.artist:
artist = self.artist.name
album_data = lookup_album_from_tadb(self.name, artist)
if not album_data.get("theaudiodb_id"):
logger.info(f"No data for {self} found in TheAudioDB")
@ -374,7 +381,7 @@ class Album(TimeStampedModel):
def scrape_bandcamp(self, force=False) -> None:
if not self.bandcamp_id or force:
slug = get_bandcamp_slug(self.album_artist.name, self.name)
slug = get_bandcamp_slug(self.artist.name, self.name)
if not slug:
logger.info(f"No bandcamp link for {self}")
return
@ -422,7 +429,8 @@ class Album(TimeStampedModel):
self.artists.add(new_artist)
if not new_artist:
for t in self.track_set.all():
self.artists.add(t.artist)
for a in t.artists.all():
self.artists.add(a)
if not self.cover_image or self.cover_image == "default-image-replace-me":
self.fetch_artwork()
self.fix_album_artist()
@ -484,19 +492,19 @@ class Album(TimeStampedModel):
@property
def rym_link(self):
artist_slug = self.album_artist.name.lower().replace(" ", "-")
artist_slug = self.artist.name.lower().replace(" ", "-")
album_slug = self.name.lower().replace(" ", "-")
return f"https://rateyourmusic.com/release/album/{artist_slug}/{album_slug}/"
@property
def bandcamp_link(self):
if self.bandcamp_id and self.album_artist.bandcamp_id:
return f"https://{self.album_artist.bandcamp_id}.bandcamp.com/album/{self.bandcamp_id}"
if self.bandcamp_id and self.artist.bandcamp_id:
return f"https://{self.artist.bandcamp_id}.bandcamp.com/album/{self.bandcamp_id}"
return ""
@property
def bandcamp_search_link(self):
artist = self.album_artist.name.lower()
artist = self.artist.name.lower()
album = self.name.lower()
return f"https://bandcamp.com/search?q={album} {artist}&item_type=a"
@ -564,7 +572,7 @@ class Album(TimeStampedModel):
album_artist=artist,
alt_names=alt_name,
)
album.artists.add(*extra_artists)
album.artists.add(artist, *extra_artists)
album.fetch_artwork()
return album
@ -573,14 +581,22 @@ class Album(TimeStampedModel):
class Track(ScrobblableMixin):
COMPLETION_PERCENT = getattr(settings, "MUSIC_COMPLETION_PERCENT", 100)
artist = models.ForeignKey(Artist, on_delete=models.DO_NOTHING)
artist_fk = models.ForeignKey(
Artist, on_delete=models.DO_NOTHING, **BNULL, related_name="+"
)
artists = models.ManyToManyField(Artist)
albums = models.ManyToManyField(Album, related_name="tracks")
album = models.ForeignKey(Album, on_delete=models.DO_NOTHING, **BNULL)
musicbrainz_id = models.CharField(max_length=255, **BNULL)
similar_recordings = models.JSONField(**BNULL)
class Meta:
unique_together = [["album", "musicbrainz_id"]]
@property
def artist(self):
return self.artists.first() or self.artist_fk
def __str__(self):
return f"{self.title} by {self.artist}"
@ -615,8 +631,9 @@ class Track(ScrobblableMixin):
@property
def primary_image_url(self) -> str:
url = ""
if self.artist.thumbnail:
url = self.artist.thumbnail_medium.url
primary_artist = self.artist
if primary_artist and primary_artist.thumbnail:
url = primary_artist.thumbnail_medium.url
if self.primary_album and self.primary_album.cover_image:
url = self.primary_album.cover_image_medium.url
return url
@ -625,6 +642,7 @@ class Track(ScrobblableMixin):
def find_or_create(
cls,
title: str = "",
artist_names: list[str] | None = None,
artist_name: str = "",
album_name: str = "",
run_time_seconds: int | None = None,
@ -638,25 +656,40 @@ class Track(ScrobblableMixin):
name
Optionally, we can update any found artists with overwrite."""
from music.utils import parse_artist_names
if artist_names is None and artist_name:
artist_names = parse_artist_names(artist_name)
if not artist_names:
artist_names = []
album = None
if album_name:
logger.info(f"Looking up album for: {album_name}")
album = Album.find_or_create(name=album_name, artist_name=artist_name)
artist = album.album_artist
else:
artist = Artist.find_or_create(artist_name, track_name=title)
if not artist:
artist = Artist.find_or_create(artist_name)
first_name = artist_names[0] if artist_names else ""
album = Album.find_or_create(name=album_name, artist_name=first_name)
artist_objs = []
for name in artist_names:
artist = Artist.find_or_create(name, track_name=title)
if artist:
artist_objs.append(artist)
track = None
if artist_objs:
track = cls.objects.filter(title=title, artists__in=artist_objs).first()
lookup_keys = {"title": title, "artist": artist}
if run_time_seconds:
lookup_keys["base_run_time_seconds"] = run_time_seconds
logger.info(f"Looking up track using: {lookup_keys}")
track = cls.objects.filter(**lookup_keys).first()
if not track:
track = cls.objects.filter(title=title, artist=artist).first()
if not track:
track, _ = cls.objects.get_or_create(title=title, artist=artist)
track = cls.objects.filter(title=title).first()
if not track:
track = cls(title=title)
track.save()
track.refresh_from_db()
if artist_objs:
track.artists.add(*artist_objs)
if album:
track.albums.add(album)
@ -665,29 +698,37 @@ class Track(ScrobblableMixin):
if mbid and run_time_seconds:
track.base_run_time_seconds = run_time_seconds
track.musicbrainz_id = mbid
track.tags.add("musicbrainz-provider", "musicbrainz-enriched")
else:
artist_name_str = " & ".join(artist_names) if artist_names else ""
logger.info(
f"Enriching track {track}",
extra={
"title": title,
"artist_name": artist_name,
"artist_name": artist_name_str,
"track_id": track.id,
},
)
try:
mbid, length = get_recording_mbid_exact(
title, artist_name, album_name
)
except Exception:
result, method = resolve_track(title, artist_name_str, album_name)
if result and result.get("recording_mbid"):
track.musicbrainz_id = result["recording_mbid"]
length_ms = result.get("length_ms")
if length_ms and not track.base_run_time_seconds:
track.base_run_time_seconds = int(int(length_ms) / 1000)
method_tag = f"musicbrainz-{method}" if method else ""
track.tags.add(method_tag, "musicbrainz-enriched")
cleaned_title, featured_names = extract_featured_artists(title)
for feat_name in featured_names:
artist = Artist.find_or_create(feat_name, track_name=title)
if artist:
track.artists.add(artist)
else:
print("No musicbrainz result found, cannot enrich")
track.tags.add("musicbrainz-notfound")
return track
track.base_run_time_seconds = run_time_seconds or int(length / 1000)
track.musicbrainz_id = mbid
if commit:
track.save()
return track
def fix_metadata(self, force_update=False):
...
def fix_metadata(self, force_update=False): ...

View File

@ -1,11 +1,155 @@
from datetime import datetime
import logging
import re
from datetime import datetime
import musicbrainzngs
from dateutil.parser import parse
logger = logging.getLogger(__name__)
TRACK_SUFFIXES = [
r" - Bonus Track",
r" - Bonus",
r" - Remix",
r" - Live",
r" - Radio Edit",
r" - Extended Mix",
r" - Instrumental",
r" - Acoustic",
r" - Mix",
r" - Edit",
r" \[Explicit\]",
]
FEATURED_PATTERNS = [
(r"\(feat\. (.+?)\)", re.IGNORECASE),
(r"\(ft\. (.+?)\)", re.IGNORECASE),
(r"featuring (.+)", re.IGNORECASE),
(r"feat\. (.+)", re.IGNORECASE),
(r"ft\. (.+)", re.IGNORECASE),
]
def strip_track_suffixes(title: str) -> str:
for suffix in TRACK_SUFFIXES:
title = re.sub(suffix, "", title, flags=re.IGNORECASE)
return title.strip()
def extract_featured_artists(title: str) -> tuple[str, list[str]]:
featured = []
cleaned = title
for pattern, flags in FEATURED_PATTERNS:
m = re.search(pattern, cleaned, flags)
if m:
featured.append(m.group(1).strip())
cleaned = re.sub(pattern, "", cleaned, flags=flags).strip()
return cleaned, featured
def search_recordings(title: str, artist: str = "") -> list[dict]:
kwargs: dict = {"recording": title, "limit": 10}
if artist:
kwargs["artist"] = artist
try:
return musicbrainzngs.search_recordings(**kwargs).get("recording-list", [])
except (musicbrainzngs.WebServiceError, musicbrainzngs.ResponseError) as e:
logger.warning(f"MusicBrainz search error: {e}")
return []
def pick_best_recording(
recordings: list[dict], title: str, artist: str = ""
) -> tuple[dict | None, str]:
query_title = title.strip().casefold()
query_artist = artist.strip().casefold() if artist else ""
best_score: tuple[dict | None, str] = (None, "")
for rec in recordings:
rec_title = rec.get("title", "").strip()
score = int(rec.get("ext:score", 0))
if rec_title.casefold() != query_title:
if score >= 85 and not best_score[0]:
best_score = (rec, "score")
continue
if query_artist:
artist_credit = rec.get("artist-credit", [])
rec_artist = (
artist_credit[0]["artist"]["name"].strip()
if artist_credit
else ""
)
if rec_artist.casefold() != query_artist:
if score >= 85 and not best_score[0]:
best_score = (rec, "score")
continue
return (rec, "exact")
return best_score
def resolve_track(
title: str, artist: str = "", album: str = ""
) -> tuple[dict | None, str]:
if not title:
return (None, "")
titles_to_try: list[tuple[str, str]] = [(title, "")]
stripped = strip_track_suffixes(title)
if stripped != title:
titles_to_try.append((stripped, "stripped-"))
for attempt_title, prefix in titles_to_try:
if album and not prefix:
try:
mbid, length = get_recording_mbid_exact(
attempt_title, artist, album
)
return (
{
"recording_mbid": mbid,
"length_ms": length,
},
"exact",
)
except Exception:
pass
if artist:
result = get_track_metadata_with_artist(attempt_title, artist)
if result and result.get("recording_mbid"):
return (result, f"{prefix}exact")
recordings = search_recordings(attempt_title, artist)
result, tag = pick_best_recording(recordings, attempt_title, artist)
if result:
length_ms = result.get("length")
return (
{
"recording_mbid": result["id"],
"length_ms": int(length_ms) if length_ms else None,
},
f"{prefix}{tag}",
)
recordings = search_recordings(attempt_title)
result, tag = pick_best_recording(recordings, attempt_title)
if result:
tag = "title-only" if not prefix else f"{prefix}title-only"
length_ms = result.get("length")
return (
{
"recording_mbid": result["id"],
"length_ms": int(length_ms) if length_ms else None,
},
tag,
)
return (None, "")
musicbrainzngs.set_useragent("Vrobbler", "1.0", "help@unbl.ink")

View File

@ -15,12 +15,26 @@ def clean_artist_name(name: str) -> str:
name = re.split(" w. ", name, flags=re.IGNORECASE)[0].strip()
if " featuring " in name.lower():
name = re.split(" featuring ", name, flags=re.IGNORECASE)[0].strip()
# if " & " in name.lower() and "of the wand" not in name.lower():
# name = re.split("&", name, flags=re.IGNORECASE)[0].strip()
return name
def parse_artist_names(name: str) -> list[str]:
"""Split a combined artist string (e.g. 'A & B') into individual names.
First strips feat./featuring/w. prefixes, then splits on ' & ' to
support collaboration-style credits like 'Matt Sweeney & Bonnie Prince Billy'.
"""
name = clean_artist_name(name)
if " & " in name.lower():
return [
part.strip()
for part in re.split(r"\s+&\s+", name, flags=re.IGNORECASE)
if part.strip()
]
return [name]
def get_or_create_various_artists() -> "Artist":
from music.models import Artist
@ -35,19 +49,19 @@ def deduplicate_tracks(commit=False) -> int:
from music.models import Track
duplicates = (
Track.objects.values("artist", "title")
Track.objects.values("title")
.annotate(dup_count=models.Count("id"))
.filter(dup_count__gt=1)
)
query = models.Q()
for dup in duplicates:
query |= models.Q(artist=dup["artist"], title=dup["title"])
query |= models.Q(title=dup["title"])
duplicate_tracks = Track.objects.filter(query)
for b in duplicate_tracks:
tracks = Track.objects.filter(artist=b.artist, title=b.title)
tracks = Track.objects.filter(title=b.title)
first = tracks.first()
for other in tracks.exclude(id=first.id):
print("Moving scrobbles for", other.id, " to ", first.id)
@ -74,7 +88,7 @@ def condense_albums(commit: bool = False):
for track in Track.objects.all():
albums_to_add = []
duplicates = (
Track.objects.filter(title=track.title, artist=track.artist)
Track.objects.filter(title=track.title)
.exclude(id=track.id)
.exclude(id__in=processed_ids)
)
@ -88,7 +102,7 @@ def condense_albums(commit: bool = False):
track.albums.add(dup_track.album)
# Find out if this track appears more than once
duplicates = Track.objects.filter(title=track.title, artist=track.artist)
duplicates = Track.objects.filter(title=track.title)
if duplicates.count() > 1:
logger.info(f"Track appears more than once, condensing: {track}")

View File

@ -1,10 +1,9 @@
from charts.models import ChartRecord
from django.db.models import Count
from django.views import generic
from music.models import Album, Artist, Track
from charts.models import ChartRecord
from scrobbles.stats import get_scrobble_count_qs
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
from scrobbles.views import ScrobbleableDetailView, ScrobbleableListView
class TrackListView(ScrobbleableListView):
@ -20,16 +19,24 @@ class ArtistListView(generic.ListView):
paginate_by = 100
def get_queryset(self):
return (
super()
.get_queryset()
.annotate(scrobble_count=Count("track__scrobble"))
.order_by("-scrobble_count")
)
qs = super().get_queryset().annotate(scrobble_count=Count("track__scrobble"))
genre = self.request.GET.get("genre")
if genre:
qs = qs.filter(theaudiodb_genre=genre)
mood = self.request.GET.get("mood")
if mood:
qs = qs.filter(theaudiodb_mood=mood)
return qs.order_by("-scrobble_count")
def get_context_data(self, *, object_list=None, **kwargs):
context_data = super().get_context_data(object_list=object_list, **kwargs)
context_data["view"] = self.request.GET.get("view")
genre = self.request.GET.get("genre")
if genre:
context_data["active_filter"] = f"genre: {genre}"
mood = self.request.GET.get("mood")
if mood:
context_data["active_filter"] = f"mood: {mood}"
return context_data
@ -64,11 +71,38 @@ class ArtistDetailView(generic.DetailView):
from scrobbles.models import Scrobble
context_data["recent_scrobbles"] = (
Scrobble.objects.filter(track__artist=artist)
Scrobble.objects.filter(track__artists=artist)
.select_related("track", "track__album")
.order_by("-timestamp")[:100]
)
similar = artist.similar_artists or []
if similar:
top = similar[:10]
mbids = [sa["artist_mbid"] for sa in top if sa.get("artist_mbid")]
local_artists = {
a.musicbrainz_id: a
for a in Artist.objects.filter(musicbrainz_id__in=mbids)
}
for sa in top:
local = local_artists.get(sa.get("artist_mbid"))
sa["local_url"] = local.get_absolute_url() if local else None
sa["musicbrainz_url"] = (
f"https://musicbrainz.org/artist/{sa['artist_mbid']}"
if sa.get("artist_mbid")
else None
)
context_data["similar_artists"] = top
if artist.theaudiodb_genre:
context_data["genre_count"] = Artist.objects.filter(
theaudiodb_genre=artist.theaudiodb_genre
).count()
if artist.theaudiodb_mood:
context_data["mood_count"] = Artist.objects.filter(
theaudiodb_mood=artist.theaudiodb_mood
).count()
return context_data

View File

@ -1,6 +1,8 @@
from django import forms
from profiles.models import UserProfile
from scrobbles.constants import Visibility
from scrobbles.models import Scrobble
class UserProfileForm(forms.ModelForm):
@ -31,6 +33,9 @@ class UserProfileForm(forms.ModelForm):
"webdav_auto_import",
"ntfy_url",
"ntfy_enabled",
"mopidy_api_url",
"favorites_mopidy_playlist",
"monthly_mopidy_playlist_pattern",
"redirect_to_webpage",
"enable_public_widgets",
"widget_custom_css",
@ -42,3 +47,51 @@ class UserProfileForm(forms.ModelForm):
"archivebox_password": forms.PasswordInput(render_value=True),
"webdav_pass": forms.PasswordInput(render_value=True),
}
MEDIA_TYPE_LABELS = {
mt.value: mt.label for mt in Scrobble.MediaType
}
INHERIT = ""
class BulkVisibilityForm(forms.Form):
bulk_action = forms.ChoiceField(
choices=[
(Visibility.PUBLIC, "Public"),
(Visibility.PRIVATE, "Private"),
],
widget=forms.RadioSelect,
required=False,
label="Set all non-shared scrobbles to",
)
def __init__(self, *args, **kwargs):
self.profile = kwargs.pop("profile")
super().__init__(*args, **kwargs)
media_types = Scrobble.MediaType.values
choices = [
(Visibility.PUBLIC, "Public"),
(Visibility.SHARED, "Shared"),
(Visibility.PRIVATE, "Private"),
]
existing_overrides = self.profile.media_type_visibility or {}
for mt in sorted(media_types):
label = MEDIA_TYPE_LABELS.get(mt, mt)
self.fields[f"media_type_{mt}"] = forms.ChoiceField(
choices=choices,
required=False,
label=label,
initial=existing_overrides.get(mt, Visibility.PRIVATE),
)
def clean(self):
cleaned = super().clean()
overrides = {}
for mt in Scrobble.MediaType.values:
val = cleaned.get(f"media_type_{mt}")
if val:
overrides[mt] = val
cleaned["media_type_visibility"] = overrides
return cleaned

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.30 on 2026-06-04 16:27
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0032_userprofile_weigh_in_units"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="mopidy_api_url",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -0,0 +1,23 @@
# Generated by Django 4.2.30 on 2026-06-05 14:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0033_userprofile_mopidy_api_url"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="favorites_mopidy_playlist",
field=models.CharField(
blank=True,
help_text="Playlist name (e.g. 'Favorites'). Will map to m3u:Favorites.m3u8",
max_length=255,
null=True,
),
),
]

View File

@ -0,0 +1,23 @@
# Generated by Django 4.2.30 on 2026-06-05 17:22
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0034_userprofile_favorites_mopidy_playlist"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="monthly_mopidy_playlist_pattern",
field=models.CharField(
blank=True,
help_text="Django date format pattern for monthly playlists (e.g. 'Y F')",
max_length=255,
null=True,
),
),
]

View File

@ -0,0 +1,26 @@
# Generated by Django 4.2.29 on 2026-06-09 15:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0035_userprofile_monthly_mopidy_playlist_pattern"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="default_scrobble_visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
default="shared",
max_length=10,
),
),
]

View File

@ -0,0 +1,26 @@
# Generated by Django 4.2.29 on 2026-06-09 16:25
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0036_userprofile_default_scrobble_visibility"),
]
operations = [
migrations.AlterField(
model_name="userprofile",
name="default_scrobble_visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
default="private",
max_length=10,
),
),
]

View File

@ -0,0 +1,20 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0037_alter_userprofile_default_scrobble_visibility"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="media_type_visibility",
field=models.JSONField(
blank=True,
default=dict,
help_text='Per-media-type visibility overrides, e.g. {"Video": "public", "Track": "private"}',
),
),
]

View File

@ -9,6 +9,11 @@ from django.utils.functional import cached_property
from django_extensions.db.models import TimeStampedModel
from encrypted_field import EncryptedField
from profiles.constants import PRETTY_TIMEZONE_CHOICES
VISIBILITY_CHOICES = (
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
)
User = get_user_model()
BNULL = {"blank": True, "null": True}
@ -64,11 +69,33 @@ class UserProfile(TimeStampedModel):
ntfy_url = models.CharField(max_length=255, **BNULL)
ntfy_enabled = models.BooleanField(default=False)
mopidy_api_url = models.CharField(max_length=255, **BNULL)
favorites_mopidy_playlist = models.CharField(
max_length=255, **BNULL,
help_text="Playlist name (e.g. 'Favorites'). Will map to m3u:Favorites.m3u8",
)
monthly_mopidy_playlist_pattern = models.CharField(
max_length=255, **BNULL,
help_text="Django date format pattern for monthly playlists (e.g. 'Y F')",
)
redirect_to_webpage = models.BooleanField(default=True)
enable_public_widgets = models.BooleanField(default=False)
widget_custom_css = models.TextField(**BNULL)
default_scrobble_visibility = models.CharField(
max_length=10,
choices=VISIBILITY_CHOICES,
default="private",
)
media_type_visibility = models.JSONField(
default=dict,
blank=True,
help_text="Per-media-type visibility overrides, e.g. {\"Video\": \"public\", \"Track\": \"private\"}",
)
home_scrobble_limit = models.IntegerField(default=20)
weigh_in_units = models.CharField(
@ -113,6 +140,11 @@ class UserProfile(TimeStampedModel):
return history
def get_timestamp_with_tz(self, timestamp):
from django.conf import settings
server_tz = ZoneInfo(settings.TIME_ZONE)
ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
timezone = self.tzinfo
if self.timezone_change_log:
change_list = self.historic_timezone_changes
@ -123,13 +155,13 @@ class UserProfile(TimeStampedModel):
end = None
if end:
if start <= timestamp.replace(tzinfo=end.timezone) <= end:
if start <= ref_dt <= end:
timezone = start.timezone
else:
if start <= timestamp.replace(tzinfo=start.timezone):
if start <= ref_dt:
timezone = start.timezone
return timestamp.replace(tzinfo=timezone)
return ref_dt.astimezone(timezone)
def adjust_timezone_of_scrobbles(self, commit=False):
current_dt = None

View File

@ -6,4 +6,9 @@ app_name = "profiles"
urlpatterns = [
path("settings/", views.ProfileFormView.as_view(), name="profile_settings"),
path(
"settings/visibility/",
views.BulkVisibilityView.as_view(),
name="bulk_visibility",
),
]

View File

@ -1,8 +1,12 @@
from django.contrib import messages
from django.contrib.auth.mixins import LoginRequiredMixin
from django.db.models import Count, Q
from django.http.response import HttpResponseBadRequest
from django.urls import reverse_lazy
from django.views.generic import FormView
from profiles.forms import UserProfileForm
from profiles.forms import BulkVisibilityForm, UserProfileForm
from scrobbles.constants import Visibility
from scrobbles.models import Scrobble
from tasks.todoist import generate_todoist_oauth_url
@ -30,3 +34,46 @@ class ProfileFormView(LoginRequiredMixin, FormView):
context["profile"] = self.request.user.profile
context["todoist_oauth_url"] = generate_todoist_oauth_url(self.request.user.id)
return context
class BulkVisibilityView(LoginRequiredMixin, FormView):
template_name = "profiles/visibility_settings.html"
form_class = BulkVisibilityForm
success_url = reverse_lazy("profiles:bulk_visibility")
def get_form_kwargs(self):
kwargs = super().get_form_kwargs()
kwargs["profile"] = self.request.user.profile
return kwargs
def form_valid(self, form):
request = self.request
profile = request.user.profile
bulk_action = form.cleaned_data.get("bulk_action")
if bulk_action:
qs = Scrobble.objects.filter(
user=request.user,
).exclude(visibility=Visibility.SHARED)
total = qs.count()
qs.update(visibility=bulk_action)
messages.success(
request,
f"Updated {total} scrobble(s) to {bulk_action}.",
)
profile.media_type_visibility = form.cleaned_data["media_type_visibility"]
profile.save(update_fields=["media_type_visibility"])
messages.success(request, "Per-media-type visibility overrides saved.")
return super().form_valid(form)
def get_context_data(self, **kwargs):
ctx = super().get_context_data(**kwargs)
profile = self.request.user.profile
qs = Scrobble.objects.filter(user=self.request.user)
ctx["scrobble_count"] = qs.count()
ctx["visibility_counts"] = qs.values("visibility").annotate(
count=Count("id")
)
ctx["profile"] = profile
return ctx

View File

@ -4,11 +4,13 @@ from scrobbles.models import (
AudioScrobblerTSVImport,
BGStatsImport,
EBirdCSVImport,
FavoriteMedia,
KoReaderImport,
LastFmImport,
RetroarchImport,
ScaleCSVImport,
Scrobble,
ShareViewLog,
TrailGPXImport,
)
from scrobbles.mixins import Genre
@ -19,6 +21,7 @@ class ScrobbleInline(admin.TabularInline):
extra = 0
raw_id_fields = (
"video",
"channel",
"podcast_episode",
"track",
"video_game",
@ -29,6 +32,7 @@ class ScrobbleInline(admin.TabularInline):
"board_game",
"geo_location",
"task",
"puzzle",
"mood",
"brick_set",
"trail",
@ -53,52 +57,44 @@ class ImportBaseAdmin(admin.ModelAdmin):
"process_count",
"processed_finished",
"processing_started",
"error_log",
)
ordering = ("-created",)
@admin.register(AudioScrobblerTSVImport)
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin):
...
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin): ...
@admin.register(LastFmImport)
class LastFmImportAdmin(ImportBaseAdmin):
...
class LastFmImportAdmin(ImportBaseAdmin): ...
@admin.register(KoReaderImport)
class KoReaderImportAdmin(ImportBaseAdmin):
...
class KoReaderImportAdmin(ImportBaseAdmin): ...
@admin.register(RetroarchImport)
class RetroarchImportAdmin(ImportBaseAdmin):
...
class RetroarchImportAdmin(ImportBaseAdmin): ...
class RetroarchImportAdmin(ImportBaseAdmin):
...
class RetroarchImportAdmin(ImportBaseAdmin): ...
@admin.register(BGStatsImport)
class BGStatsImportAdmin(ImportBaseAdmin):
...
class BGStatsImportAdmin(ImportBaseAdmin): ...
@admin.register(EBirdCSVImport)
class EBirdCSVImportAdmin(ImportBaseAdmin):
...
class EBirdCSVImportAdmin(ImportBaseAdmin): ...
@admin.register(ScaleCSVImport)
class ScaleCSVImportAdmin(ImportBaseAdmin):
...
class ScaleCSVImportAdmin(ImportBaseAdmin): ...
@admin.register(TrailGPXImport)
class TrailGPXImportAdmin(ImportBaseAdmin):
...
class TrailGPXImportAdmin(ImportBaseAdmin): ...
@admin.register(Genre)
@ -121,6 +117,7 @@ class ScrobbleAdmin(admin.ModelAdmin):
"in_progress",
"is_paused",
"played_to_completion",
"visibility",
"user",
)
raw_id_fields = (
@ -148,6 +145,7 @@ class ScrobbleAdmin(admin.ModelAdmin):
"is_paused",
"in_progress",
"media_type",
"visibility",
"long_play_complete",
"source",
"timezone",
@ -164,3 +162,36 @@ class ScrobbleAdmin(admin.ModelAdmin):
def get_queryset(self, request):
qs = super().get_queryset(request).exclude(timestamp__year=None)
return qs
@admin.register(ShareViewLog)
class ShareViewLogAdmin(admin.ModelAdmin):
list_display = ("scrobble", "ip_address", "created")
list_filter = ("created",)
date_hierarchy = "created"
raw_id_fields = ("scrobble",)
@admin.register(FavoriteMedia)
class FavoriteMediaAdmin(admin.ModelAdmin):
list_display = ("user", "media_type", "sent_to_mopidy", "created")
list_filter = ("media_type", "sent_to_mopidy", "user")
date_hierarchy = "created"
raw_id_fields = (
"video",
"track",
"podcast_episode",
"sport_event",
"book",
"video_game",
"board_game",
"geo_location",
"task",
"mood",
"brick_set",
"trail",
"beer",
"web_page",
"life_event",
"birding_location",
)

View File

@ -1,6 +1,7 @@
import re
from rest_framework import serializers
from scrobbles.constants import Visibility
from scrobbles.models import (
AudioScrobblerTSVImport,
KoReaderImport,

View File

@ -1,6 +1,8 @@
from logging import getLogger
from rest_framework import permissions, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
from scrobbles.api.serializers import (
AudioScrobblerTSVImportSerializer,
KoReaderImportSerializer,
@ -26,6 +28,12 @@ class ScrobbleViewSet(viewsets.ModelViewSet):
def get_queryset(self):
return super().get_queryset().filter(user=self.request.user)
@action(detail=True, methods=["post"])
def regenerate_share_token(self, request, uuid=None):
scrobble = self.get_object()
scrobble.regenerate_share_token()
return Response({"share_url": scrobble.get_share_url()})
class KoReaderImportViewSet(viewsets.ModelViewSet):
queryset = KoReaderImport.objects.all().order_by("-created")

View File

@ -1,6 +1,12 @@
from django.db import models
from enum import Enum
JELLYFIN_VIDEO_ITEM_TYPES = ["Episode", "Movie"]
class Visibility(models.TextChoices):
PUBLIC = "public", "Public"
SHARED = "shared", "Shared"
PRIVATE = "private", "Private"
JELLYFIN_AUDIO_ITEM_TYPES = ["Audio"]
LONG_PLAY_MEDIA = {

View File

@ -4,6 +4,26 @@ from django.utils import timezone
from scrobbles.constants import EXCLUDE_FROM_NOW_PLAYING
from scrobbles.models import Scrobble
MONTH_COLORS = [
"#db7a7a", # Jan
"#db847a", # Feb
"#b0db7a", # Mar
"#7adb82", # Apr
"#7adbb3", # May
"#7ab6db", # Jun
"#7a8edb", # Jul
"#977adb", # Aug
"#c47adb", # Sep
"#db7ac5", # Oct
"#db7a90", # Nov
"#db7a7a", # Dec
]
def month_color(request):
from datetime import date
return {"month_color": MONTH_COLORS[(date.today().month - 1) % 12]}
def now_playing(request):
user = request.user

View File

@ -57,38 +57,152 @@ class BaseLogData(JSONDataclass):
if not self.notes:
return ""
if isinstance(self.notes, dict):
lines = []
for ts, text in self.notes.items():
note_text = " ".join(text.strip().split())
lines.append(f"{ts}: {note_text}")
return separator.join(lines)
if isinstance(self.notes, str):
return html.escape(re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", self.notes))
if isinstance(self.notes, list):
cleaned_notes = []
for note in self.notes:
cleaned = html.escape(re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", note))
cleaned_notes.append(cleaned)
if isinstance(note, dict):
timestamp, note_text = next(iter(note.items()))
note_text = " ".join(note_text.strip().split())
cleaned_notes.append(f"{timestamp}: {note_text}")
elif isinstance(note, str):
cleaned = html.escape(
re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", note)
)
cleaned_notes.append(cleaned)
return separator.join(cleaned_notes)
return ""
@staticmethod
def _format_timestamp(ts: str) -> str:
from datetime import datetime, timezone
import re
cleaned = ts.strip().strip("[]")
dt = None
if cleaned.isdigit():
try:
seconds = int(cleaned)
dt = datetime.fromtimestamp(seconds, tz=timezone.utc)
except (ValueError, OSError):
pass
if dt is None:
for fmt in [
"%Y-%m-%dT%H:%M:%S.%fZ",
"%Y-%m-%dT%H:%M:%SZ",
"%Y-%m-%dT%H:%M:%S.%f",
"%Y-%m-%dT%H:%M:%S",
]:
try:
dt = datetime.strptime(cleaned, fmt)
break
except ValueError:
continue
if dt is None:
m = re.match(r"(\d{4}-\d{2}-\d{2})\s+\w{3}\s+(\d{2}:\d{2})", cleaned)
if m:
try:
dt = datetime.strptime(
f"{m.group(1)} {m.group(2)}", "%Y-%m-%d %H:%M"
)
except ValueError:
pass
if dt is None:
try:
dt = datetime.fromisoformat(cleaned)
except ValueError:
pass
if dt:
return dt.strftime("%b %-d, %Y %-I:%M %p")
return ts
def notes_as_html(self) -> str:
import html
import bleach
import markdown
from django.utils.safestring import mark_safe
if not self.notes:
return ""
notes_list = []
if isinstance(self.notes, str):
notes_list = [self.notes]
elif isinstance(self.notes, list):
notes_list = self.notes
md = markdown.Markdown(extensions=["extra"])
allowed_tags = [
"p",
"br",
"strong",
"em",
"a",
"ul",
"ol",
"li",
"code",
"pre",
"blockquote",
"h1",
"h2",
"h3",
"h4",
"h5",
"h6",
"hr",
"img",
]
html_notes = []
for note in notes_list:
for line in note.split("\n"):
note_items = []
if isinstance(self.notes, dict):
for ts, text in sorted(self.notes.items(), reverse=True):
note_items.append((ts, text.strip()))
elif isinstance(self.notes, str):
for line in self.notes.split("\n"):
if line.strip():
escaped_line = html.escape(line)
html_notes.append(f'<div class="sticky-note">{escaped_line}</div>')
return mark_safe("".join(html_notes))
note_items.append((None, line.strip()))
elif isinstance(self.notes, list):
for note in self.notes:
if isinstance(note, dict):
timestamp, note_text = next(iter(note.items()))
note_items.append((timestamp, note_text.strip()))
elif isinstance(note, str):
for line in note.split("\n"):
if line.strip():
note_items.append((None, line.strip()))
html_parts = []
for i, (ts, text) in enumerate(note_items):
if i > 0:
html_parts.append('<hr class="note-divider">')
ts_html = ""
if ts:
ts_html = (
f'<h5 class="note-timestamp">{self._format_timestamp(ts)}</h5>'
)
content_html = bleach.clean(
md.convert(text),
tags=allowed_tags,
strip=True,
)
html_parts.append(
f'<div class="note-item">{ts_html}<div class="note-content">{content_html}</div></div>'
)
return mark_safe("".join(html_parts))
@dataclass
@ -96,6 +210,14 @@ class LongPlayLogData(JSONDataclass):
long_play_complete: bool = False
@dataclass
class SportEventLogData(BaseLogData):
thesportsdb_id: Optional[str] = None
start: Optional[str] = None
round_name: Optional[str] = None
season_name: Optional[str] = None
@dataclass
class WithPeopleLogData(JSONDataclass):
with_people_ids: Optional[list[int]] = None

View File

@ -1,8 +1,8 @@
import csv
import tempfile
from scrobbles.models import Scrobble
from django.db.models import Q
from scrobbles.models import Scrobble
def export_scrobbles(start_date=None, end_date=None, format="AS"):
@ -50,7 +50,7 @@ def export_scrobbles(start_date=None, end_date=None, format="AS"):
track = scrobble.track
track_number = 0 # TODO Add track number
track_rating = "S" # TODO implement ratings?
track_artist = track.artist or track.album.album_artist
track_artist = track.artist or track.album.artist
row = [
track_artist,
track.album.name,

View File

@ -1,3 +1,4 @@
import json
from dataclasses import fields
from typing import Union, get_args, get_origin
@ -89,7 +90,7 @@ def form_from_dataclass(dataclass):
form_cls = type(f"{dataclass.__name__}Form", (forms.Form,), form_fields)
if "notes" in form_cls.base_fields:
if "notes" in form_cls.base_fields and "notes" not in override_fields:
form_cls.base_fields["notes"] = forms.CharField(
required=False,
widget=forms.Textarea(attrs={"rows": 4}),
@ -101,3 +102,54 @@ def form_from_dataclass(dataclass):
form_cls.clean_notes = clean_notes
return form_cls
class NotesDictWidget(forms.Widget):
template_name = "tasks/task_notes_widget.html"
class Media:
js = ("tasks/task_notes.js",)
def value_from_datadict(self, data, files, name):
timestamps = data.getlist(f"{name}_timestamps")
if timestamps:
contents = data.getlist(f"{name}_contents")
result = {}
for i, ts in enumerate(timestamps):
if i < len(contents) and ts:
result[ts] = contents[i]
return result if result else ""
return data.get(name, "")
def get_context(self, name, value, attrs):
context = super().get_context(name, value, attrs)
notes = {}
if value:
if isinstance(value, str):
try:
notes = json.loads(value)
except (json.JSONDecodeError, TypeError):
notes = {}
elif isinstance(value, dict):
notes = value
context["widget"]["notes"] = notes
return context
class NotesDictField(forms.Field):
widget = NotesDictWidget
def clean(self, value):
if not value:
return {}
if isinstance(value, str):
if value.strip():
from scrobbles.utils import make_note_timestamp
return {make_note_timestamp(): value.strip()}
return {}
if isinstance(value, dict):
return value
return {}

View File

@ -76,6 +76,7 @@ class LastFM:
in_progress=False,
media_type=Scrobble.MediaType.TRACK,
timezone=tz_timestamp.tzinfo.name,
visibility="private",
)
# Vrobbler scrobbles on finish, LastFM scrobbles on start
seconds_eariler = timestamp - timedelta(seconds=20)

View File

@ -5,6 +5,7 @@ from datetime import datetime, timedelta
from django.contrib.auth import get_user_model
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
from tasks.models import Task
logger = logging.getLogger(__name__)
@ -85,6 +86,11 @@ def import_scale_csv(file_path, user_id):
log_dict["unit_type"] = user.profile.weigh_in_units
weight_val = log_dict.get("weight")
if weight_val is not None:
unit = "kg" if user.profile.weigh_in_units == "metric" else "lbs"
log_dict["title"] = f"{weight_val} {unit}"
existing = Scrobble.objects.filter(
timestamp=start,
task=weigh_in,
@ -105,9 +111,12 @@ def import_scale_csv(file_path, user_id):
played_to_completion=True,
in_progress=False,
media_type=Scrobble.MediaType.TASK,
visibility="private",
)
new_scrobbles.append(new_scrobble)
created = Scrobble.objects.bulk_create(new_scrobbles)
logger.info(f"Created {len(created)} weigh-in scrobbles")
for scrobble in created:
ScrobbleNtfyNotification(scrobble).send()
return created

View File

@ -12,6 +12,7 @@ from django.core.files import File
from locations.models import GeoLocation
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
from trails.models import Trail, TrailLogData
logger = logging.getLogger(__name__)
@ -304,6 +305,7 @@ def import_trail_gpx(file_path, user_id, original_filename=None):
played_to_completion=True,
in_progress=False,
media_type=Scrobble.MediaType.TRAIL,
visibility="private",
)
_, ext = os.path.splitext(file_path)
@ -328,4 +330,6 @@ def import_trail_gpx(file_path, user_id, original_filename=None):
created = Scrobble.objects.bulk_create(new_scrobbles)
logger.info(f"Created {len(created)} trail scrobbles")
for scrobble in created:
ScrobbleNtfyNotification(scrobble).send()
return created

View File

@ -80,6 +80,7 @@ def import_audioscrobbler_tsv_file(file_path, user_id):
in_progress=False,
media_type=Scrobble.MediaType.TRACK,
timezone=timestamp.tzinfo.name,
visibility="private",
)
existing = Scrobble.objects.filter(
timestamp=timestamp, track=track, user=user

View File

@ -22,7 +22,10 @@ DEFAULT_SCALE_PATH = "var/scale/"
def import_from_webdav_for_all_users(
restart=False, update_retroarch_hash=False, update_koreader_etag=False
restart=False,
update_retroarch_hash=False,
update_koreader_etag=False,
include_processed=False,
):
"""Iterate all WebDAV-enabled users, scanning each media-type directory."""
webdav_enabled_user_ids = UserProfile.objects.filter(
@ -31,6 +34,12 @@ def import_from_webdav_for_all_users(
webdav_pass__isnull=False,
webdav_auto_import=True,
).values_list("user_id", flat=True)
if include_processed:
logger.warning(
"Re-importing previously-processed files from processed/ subdirectories "
"— this may create duplicate scrobbles."
)
logger.info(f"Start import of {webdav_enabled_user_ids.count()} webdav accounts")
ko_count = 0
@ -49,15 +58,24 @@ def import_from_webdav_for_all_users(
client, user_id, restart, update_etag_only=update_koreader_etag
)
logger.info("Scanning WebDAV gpx for user %s", user_id)
gpx_count += scan_webdav_for_gpx(client, user_id)
gpx_count += scan_webdav_for_gpx(
client, user_id, include_processed=include_processed
)
logger.info("Scanning WebDAV retroarch for user %s", user_id)
retro_count += scan_webdav_for_retroarch(
client, user_id, update_hash_only=update_retroarch_hash
client,
user_id,
update_hash_only=update_retroarch_hash,
include_processed=include_processed,
)
logger.info("Scanning WebDAV bgstats for user %s", user_id)
bgstats_count += scan_webdav_for_bgstats(client, user_id)
bgstats_count += scan_webdav_for_bgstats(
client, user_id, include_processed=include_processed
)
logger.info("Scanning WebDAV ebird for user %s", user_id)
ebird_count += scan_webdav_for_ebird(client, user_id)
ebird_count += scan_webdav_for_ebird(
client, user_id, include_processed=include_processed
)
logger.info("Scanning WebDAV scale for user %s", user_id)
scale_count += scan_webdav_for_scale(client, user_id)
@ -135,7 +153,7 @@ def scan_webdav_for_koreader(
)
if update_etag_only:
if last_import and remote_etag:
if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
last_import.webdav_etag = remote_etag
last_import.save(update_fields=["webdav_etag"])
logger.info(
@ -145,7 +163,7 @@ def scan_webdav_for_koreader(
)
return 0
if last_import and last_import.webdav_etag and remote_etag:
if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
if last_import.webdav_etag == remote_etag:
logger.info(
"koreader stats file unchanged (ETag match)",
@ -200,7 +218,7 @@ def scan_webdav_for_koreader(
return 1
def scan_webdav_for_gpx(webdav_client, user_id):
def scan_webdav_for_gpx(webdav_client, user_id, include_processed=False):
gpx_path = DEFAULT_GPX_PATH # TODO allow this to be configured in user settings
try:
webdav_client.info(DEFAULT_GPX_PATH)
@ -259,10 +277,47 @@ def scan_webdav_for_gpx(webdav_client, user_id):
finally:
os.unlink(tmp.name)
if include_processed:
try:
processed_files = webdav_client.list(processed_dir)
except Exception as e:
logger.warning(
"Could not list var/gpx/processed/",
extra={"user_id": user_id, "error": str(e)},
)
return new_imports
for fname in processed_files:
fname = os.path.basename(fname)
if not fname.lower().endswith(gpx_extensions):
continue
remote_path = f"{processed_dir}{fname}"
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=remote_path, local_path=tmp.name
)
imp = TrailGPXImport.objects.create(
user_id=user_id,
original_filename=fname,
)
with open(tmp.name, "rb") as f:
imp.gpx_file.save(fname, f, save=True)
process_trail_gpx_import.delay(imp.id)
new_imports += 1
except Exception as e:
logger.error(f"Failed to import processed GPX file {fname}: {e}")
finally:
os.unlink(tmp.name)
return new_imports
def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
def scan_webdav_for_retroarch(
webdav_client, user_id, update_hash_only=False, include_processed=False
):
"""Check for new .lrtl files on WebDAV, download+import if changed.
Uses ETags from a single PROPFIND to detect changes without downloading
@ -271,16 +326,24 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
When *update_hash_only* is True, mismatched hashes update the last import's
stored hash without re-downloading — useful when migrating to a new hash
scheme to avoid a one-time re-import.
When *include_processed* is True, also imports files from the processed/
subdirectory. These are sorted chronologically by their YYYYMMDDHHMM-
timestamp prefix so that the importer always sees the earliest snapshot
first, which is critical for correct incremental scrobbling since
retroarch overwrites the same .lrtl filename.
"""
import hashlib
import shutil
import re
import zipfile
from datetime import datetime
from scrobbles.models import RetroarchImport
from scrobbles.tasks import process_retroarch_import
retroarch_path = DEFAULT_RETROARCH_PATH + "playlogs/"
retroarch_path = DEFAULT_RETROARCH_PATH
processed_dir = f"{retroarch_path}processed/"
try:
webdav_client.info(retroarch_path)
except:
@ -288,7 +351,13 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
return 0
try:
files = webdav_client.list(retroarch_path, get_info=True)
webdav_client.mkdir(processed_dir, recursive=True)
except Exception:
pass
# Collect files from root directory
try:
root_files = webdav_client.list(retroarch_path, get_info=True)
except Exception as e:
logger.warning(
"Could not list var/retroarch/",
@ -296,14 +365,51 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
)
return 0
lrtl_files = sorted(
[f for f in files if (f.get("name") or "").lower().endswith(".lrtl")],
key=lambda x: (x.get("name") or ""),
)
# Extract basename from path since the webdav adapter returns name=None
for f in root_files:
if f.get("path") and not f.get("name"):
f["name"] = os.path.basename(f["path"])
lrtl_files = [
f for f in root_files
if (f.get("name") or "").lower().endswith(".lrtl")
]
# Optionally include historical files from processed/
if include_processed:
try:
processed_files = webdav_client.list(processed_dir, get_info=True)
except Exception as e:
logger.warning(
"Could not list var/retroarch/processed/",
extra={"user_id": user_id, "error": str(e)},
)
else:
for f in processed_files:
if f.get("path") and not f.get("name"):
f["name"] = os.path.basename(f["path"])
lrtl_files.extend([
f for f in processed_files
if (f.get("name") or "").lower().endswith(".lrtl")
])
if not lrtl_files:
logger.info("No .lrtl files found on webdav", extra={"user_id": user_id})
return 0
# Sort chronologically: processed files (timestamp prefixed) first,
# current root files last.
ts_pattern = re.compile(r"^(\d{12})-")
def sort_key(f):
name = f.get("name") or ""
m = ts_pattern.match(name)
if m:
return m.group(1)
# Root files (no timestamp) go after all historical snapshots
return "999999999999"
lrtl_files.sort(key=sort_key)
# Don't queue if one is still being processed
if RetroarchImport.objects.filter(
user_id=user_id, processed_finished__isnull=True
@ -321,7 +427,6 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
hasher.update((f.get("etag") or f.get("modified") or "").encode())
content_hash = hasher.hexdigest()
# Skip if the last completed import already has this hash
last_import = (
RetroarchImport.objects.filter(
user_id=user_id, processed_finished__isnull=False
@ -329,7 +434,9 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
.order_by("-processed_finished")
.first()
)
if last_import and last_import.files_hash == content_hash:
# Skip unchanged root files only, never skip when re-processing history
if not include_processed and last_import and last_import.files_hash == content_hash:
logger.info(
"Retroarch lrtl files unchanged for user, skipping",
extra={"user_id": user_id},
@ -350,15 +457,32 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
download_dir = tempfile.mkdtemp()
try:
downloaded = []
imported_filenames = []
for f in lrtl_files:
basename = f.get("name")
# Determine source directory: processed files already live under processed/
is_processed = ts_pattern.match(basename)
source_dir = processed_dir if is_processed else retroarch_path
remote_path = f"{source_dir}{basename}"
dst = os.path.join(download_dir, basename)
try:
webdav_client.download_sync(
remote_path=f.get("path"),
remote_path=remote_path,
local_path=dst,
)
downloaded.append(basename)
if is_processed:
imported_filenames.append(basename)
else:
ts = datetime.now().strftime("%Y%m%d%H%M%S")
processed_name = f"{ts}-{basename}"
imported_filenames.append(processed_name)
webdav_client.move(
remote_path,
f"{processed_dir}{processed_name}",
)
except Exception as e:
logger.error("Failed to download retroarch %s: %s", basename, e)
@ -375,7 +499,7 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
imp = RetroarchImport.objects.create(
user_id=user_id,
original_filename=f"batch-{len(downloaded)}-files",
original_filename=", ".join(imported_filenames),
files_hash=content_hash,
)
with open(zip_path, "rb") as f:
@ -392,8 +516,12 @@ def scan_webdav_for_retroarch(webdav_client, user_id, update_hash_only=False):
shutil.rmtree(download_dir, ignore_errors=True)
def scan_webdav_for_bgstats(webdav_client, user_id):
"""Download .bgsplay files from WebDAV and queue imports for new files."""
def scan_webdav_for_bgstats(webdav_client, user_id, include_processed=False):
"""Download .bgsplay files from WebDAV and queue imports for new files.
After importing, files are moved to var/bgstats/processed/ so they are
not re-imported on subsequent scans unless *include_processed* is True.
"""
from scrobbles.models import BGStatsImport
from scrobbles.tasks import process_bgstats_import
@ -413,6 +541,12 @@ def scan_webdav_for_bgstats(webdav_client, user_id):
)
return 0
processed_dir = f"{bgstats_path}processed/"
try:
webdav_client.mkdir(processed_dir, recursive=True)
except Exception:
pass
new_imports = 0
already_imported = set(
BGStatsImport.objects.filter(user_id=user_id).values_list(
@ -424,6 +558,8 @@ def scan_webdav_for_bgstats(webdav_client, user_id):
fname = os.path.basename(fname)
if not fname.lower().endswith(".bgsplay"):
continue
if fname == "processed":
continue
if fname in already_imported:
logger.debug(f"Skipping already-imported {fname}")
continue
@ -431,7 +567,7 @@ def scan_webdav_for_bgstats(webdav_client, user_id):
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{bgstats_path}/{fname}", local_path=tmp.name
remote_path=f"{bgstats_path}{fname}", local_path=tmp.name
)
imp = BGStatsImport.objects.create(
user_id=user_id,
@ -439,6 +575,14 @@ def scan_webdav_for_bgstats(webdav_client, user_id):
)
with open(tmp.name, "rb") as f:
imp.bgsplay_file.save(fname, f, save=True)
stem, ext = os.path.splitext(fname)
ts = datetime.now().strftime("%Y%m%dT%H%M%S")
webdav_client.move(
f"{bgstats_path}{fname}",
f"{processed_dir}{stem}_{ts}{ext}",
)
process_bgstats_import.delay(imp.id)
new_imports += 1
except Exception as e:
@ -446,11 +590,48 @@ def scan_webdav_for_bgstats(webdav_client, user_id):
finally:
os.unlink(tmp.name)
if include_processed:
try:
processed_files = webdav_client.list(processed_dir)
except Exception as e:
logger.warning(
"Could not list var/bgstats/processed/",
extra={"user_id": user_id, "error": str(e)},
)
return new_imports
for fname in processed_files:
fname = os.path.basename(fname)
if not fname.lower().endswith(".bgsplay"):
continue
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{processed_dir}{fname}", local_path=tmp.name
)
imp = BGStatsImport.objects.create(
user_id=user_id,
original_filename=fname,
)
with open(tmp.name, "rb") as f:
imp.bgsplay_file.save(fname, f, save=True)
process_bgstats_import.delay(imp.id)
new_imports += 1
except Exception as e:
logger.error(f"Failed to import processed BG Stats file {fname}: {e}")
finally:
os.unlink(tmp.name)
return new_imports
def scan_webdav_for_ebird(webdav_client, user_id):
"""Download .csv files from WebDAV var/ebird/ and queue imports for new files."""
def scan_webdav_for_ebird(webdav_client, user_id, include_processed=False):
"""Download .csv files from WebDAV var/ebird/ and queue imports for new files.
After importing, files are moved to var/ebird/processed/ so they are
not re-imported on subsequent scans unless *include_processed* is True.
"""
from scrobbles.models import EBirdCSVImport
from scrobbles.tasks import process_ebird_csv_import
@ -470,6 +651,12 @@ def scan_webdav_for_ebird(webdav_client, user_id):
)
return 0
processed_dir = f"{ebird_path}processed/"
try:
webdav_client.mkdir(processed_dir, recursive=True)
except Exception:
pass
new_imports = 0
already_imported = set(
EBirdCSVImport.objects.filter(user_id=user_id).values_list(
@ -481,6 +668,8 @@ def scan_webdav_for_ebird(webdav_client, user_id):
fname = os.path.basename(fname)
if not fname.lower().endswith(".csv"):
continue
if fname == "processed":
continue
if fname in already_imported:
logger.debug(f"Skipping already-imported {fname}")
continue
@ -488,7 +677,7 @@ def scan_webdav_for_ebird(webdav_client, user_id):
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{ebird_path}/{fname}", local_path=tmp.name
remote_path=f"{ebird_path}{fname}", local_path=tmp.name
)
imp = EBirdCSVImport.objects.create(
user_id=user_id,
@ -496,6 +685,14 @@ def scan_webdav_for_ebird(webdav_client, user_id):
)
with open(tmp.name, "rb") as f:
imp.csv_file.save(fname, f, save=True)
stem, ext = os.path.splitext(fname)
ts = datetime.now().strftime("%Y%m%dT%H%M%S")
webdav_client.move(
f"{ebird_path}{fname}",
f"{processed_dir}{stem}_{ts}{ext}",
)
process_ebird_csv_import.delay(imp.id)
new_imports += 1
except Exception as e:
@ -503,11 +700,49 @@ def scan_webdav_for_ebird(webdav_client, user_id):
finally:
os.unlink(tmp.name)
if include_processed:
try:
processed_files = webdav_client.list(processed_dir)
except Exception as e:
logger.warning(
"Could not list var/ebird/processed/",
extra={"user_id": user_id, "error": str(e)},
)
return new_imports
for fname in processed_files:
fname = os.path.basename(fname)
if not fname.lower().endswith(".csv"):
continue
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{processed_dir}{fname}", local_path=tmp.name
)
imp = EBirdCSVImport.objects.create(
user_id=user_id,
original_filename=fname,
)
with open(tmp.name, "rb") as f:
imp.csv_file.save(fname, f, save=True)
process_ebird_csv_import.delay(imp.id)
new_imports += 1
except Exception as e:
logger.error(f"Failed to import processed eBird CSV file {fname}: {e}")
finally:
os.unlink(tmp.name)
return new_imports
def scan_webdav_for_scale(webdav_client, user_id):
"""Download .csv files from WebDAV var/scale/ and queue imports for new files."""
"""Download .csv files from WebDAV var/scale/ and queue imports for new files.
Because the scale CSV always has the same filename but grows by appending
rows, we detect changes by downloading the file and comparing its content
hash against the last completed import, rather than checking the filename.
"""
from scrobbles.models import ScaleCSVImport
from scrobbles.tasks import process_scale_csv_import
@ -530,28 +765,41 @@ def scan_webdav_for_scale(webdav_client, user_id):
return 0
new_imports = 0
already_imported = set(
ScaleCSVImport.objects.filter(user_id=user_id).values_list(
"original_filename", flat=True
)
)
for fname in files:
fname = os.path.basename(fname)
if not fname.lower().endswith(".csv"):
continue
if fname in already_imported:
logger.debug(f"Skipping already-imported {fname}")
continue
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{scale_path}/{fname}", local_path=tmp.name
)
new_hash = get_file_md5_hash(tmp.name)
last_import = (
ScaleCSVImport.objects.filter(
user_id=user_id,
original_filename=fname,
processed_finished__isnull=False,
)
.order_by("-processed_finished")
.first()
)
if last_import and last_import.file_hash == new_hash:
logger.debug(
"Scale CSV %s unchanged (hash match), skipping",
fname,
extra={"user_id": user_id, "hash": new_hash},
)
continue
imp = ScaleCSVImport.objects.create(
user_id=user_id,
original_filename=fname,
file_hash=new_hash,
)
with open(tmp.name, "rb") as f:
imp.csv_file.save(fname, f, save=True)

View File

@ -0,0 +1,68 @@
from django.core.management.base import BaseCommand
from django.db import models
from scrobbles.models import Scrobble
from scrobbles.utils import analyze_scrobble_sentiment
class Command(BaseCommand):
help = "Backfill VADER sentiment analysis for scrobbles with notes"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Actually update scrobble logs with sentiment data",
)
parser.add_argument(
"--overwrite",
action="store_true",
help="Re-analyze scrobbles that already have sentiment data",
)
def handle(self, *args, **options):
commit = options["commit"]
overwrite = options["overwrite"]
qs = Scrobble.objects.filter(
~models.Q(log__notes__isnull=True)
& ~models.Q(log__notes=[])
& ~models.Q(log__notes={})
)
if not overwrite:
qs = qs.filter(log__sentiment__isnull=True)
total = qs.count()
analyzed_count = 0
skipped_count = 0
self.stdout.write(f"Found {total} scrobbles to process")
for scrobble in qs.iterator():
if commit:
analyzed = analyze_scrobble_sentiment(scrobble, overwrite=overwrite)
else:
notes_str = ""
if scrobble.logdata:
notes_str = scrobble.logdata.notes_as_str()
analyzed = bool(notes_str)
if analyzed:
analyzed_count += 1
if commit:
scores = (scrobble.log or {}).get("sentiment", {})
self.stdout.write(
f" Updated scrobble {scrobble.id}: compound={scores.get('compound', 'N/A')}"
)
else:
self.stdout.write(
f" [DRY RUN] Would analyze scrobble {scrobble.id}"
)
else:
skipped_count += 1
self.stdout.write(
f"\nAnalyzed {analyzed_count} scrobbles, skipped {skipped_count}"
)
if not commit:
self.stdout.write("Run with --commit to persist changes")

View File

@ -1,9 +1,7 @@
from django.core.management.base import BaseCommand
from vrobbler.apps.tasks.utils import (
convert_notes_to_dict,
convert_old_boardgame_log_to_new,
convert_old_orgmode_log_to_new,
convert_old_todoist_log_to_new,
convert_tasks_notes_list_to_dict,
)
@ -21,7 +19,5 @@ class Command(BaseCommand):
commit = True
else:
print("No changes will be saved, use --commit to save")
convert_old_orgmode_log_to_new(commit)
convert_old_todoist_log_to_new(commit)
convert_notes_to_dict(commit)
convert_tasks_notes_list_to_dict(commit)
convert_old_boardgame_log_to_new(commit)

View File

@ -1,4 +1,5 @@
from django.core.management.base import BaseCommand
from vrobbler.apps.scrobbles.importers import webdav
@ -21,6 +22,12 @@ class Command(BaseCommand):
help="Store current WebDAV ETag on last KoReader import "
"without re-importing (migration helper)",
)
parser.add_argument(
"--include-processed",
action="store_true",
help="Also import files already moved to processed/ subdirectories "
"(may produce duplicate scrobbles; use with care on production)",
)
def handle(self, *args, **options):
restart = False
@ -28,8 +35,10 @@ class Command(BaseCommand):
restart = True
update_hash = options.get("update_retroarch_hash", False)
update_etag = options.get("update_koreader_etag", False)
include_processed = options.get("include_processed", False)
webdav.import_from_webdav_for_all_users(
restart=restart,
update_retroarch_hash=update_hash,
update_koreader_etag=update_etag,
include_processed=include_processed,
)

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.29 on 2026-05-28
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0087_koreaderimport_webdav_etag"),
]
operations = [
migrations.AddField(
model_name="scalecsvimport",
name="file_hash",
field=models.CharField(blank=True, max_length=32, null=True),
),
]

View File

@ -0,0 +1,287 @@
# Generated by Django 4.2.30 on 2026-06-05 14:55
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
import uuid
class Migration(migrations.Migration):
dependencies = [
("boardgames", "0015_alter_boardgame_genre"),
("trails", "0009_trail_route_waypoint"),
("moods", "0008_alter_mood_genre"),
("birds", "0002_birdingcsvimport"),
("bricksets", "0005_alter_brickset_genre"),
("music", "0036_artist_similar_artists"),
("puzzles", "0006_alter_puzzle_genre"),
("videogames", "0015_alter_videogame_genre"),
("lifeevents", "0005_alter_lifeevent_genre"),
("beers", "0008_alter_beer_genre"),
("foods", "0007_alter_food_genre"),
("tasks", "0007_alter_task_genre"),
("books", "0036_alter_book_genre_alter_paper_genre"),
("videos", "0030_alter_channel_genre_alter_series_genre_and_more"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
("podcasts", "0020_alter_podcast_genre_alter_podcastepisode_genre"),
("webpages", "0009_alter_webpage_genre"),
("sports", "0018_alter_sportevent_genre"),
("locations", "0010_clean_start"),
("scrobbles", "0088_scalecsvimport_file_hash"),
]
operations = [
migrations.CreateModel(
name="FavoriteMedia",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True, default=uuid.uuid4, editable=False, null=True
),
),
(
"media_type",
models.CharField(
choices=[
("Video", "Video"),
("Track", "Track"),
("PodcastEpisode", "Podcast episode"),
("SportEvent", "Sport event"),
("Book", "Book"),
("Paper", "Paper"),
("VideoGame", "Video game"),
("BoardGame", "Board game"),
("GeoLocation", "GeoLocation"),
("Trail", "Trail"),
("Beer", "Beer"),
("Puzzle", "Puzzle"),
("Food", "Food"),
("Task", "Task"),
("WebPage", "Web Page"),
("LifeEvent", "Life event"),
("Mood", "Mood"),
("BrickSet", "Brick set"),
("Channel", "Channel"),
("BirdingLocation", "Birding location"),
],
max_length=20,
),
),
("sent_to_mopidy", models.BooleanField(default=False)),
(
"beer",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="beers.beer",
),
),
(
"birding_location",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="birds.birdinglocation",
),
),
(
"board_game",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="boardgames.boardgame",
),
),
(
"book",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="books.book",
),
),
(
"brick_set",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="bricksets.brickset",
),
),
(
"channel",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="videos.channel",
),
),
(
"food",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="foods.food",
),
),
(
"geo_location",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="locations.geolocation",
),
),
(
"life_event",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="lifeevents.lifeevent",
),
),
(
"mood",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="moods.mood",
),
),
(
"paper",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="books.paper",
),
),
(
"podcast_episode",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="podcasts.podcastepisode",
),
),
(
"puzzle",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="puzzles.puzzle",
),
),
(
"sport_event",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="sports.sportevent",
),
),
(
"task",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="tasks.task",
),
),
(
"track",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="music.track",
),
),
(
"trail",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="trails.trail",
),
),
(
"user",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL,
),
),
(
"video",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="videos.video",
),
),
(
"video_game",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="videogames.videogame",
),
),
(
"web_page",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="webpages.webpage",
),
),
],
options={
"ordering": ["-created"],
},
),
]

View File

@ -0,0 +1,53 @@
# Generated by Django 4.2.29 on 2026-06-08 14:57
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0089_favoritemedia"),
]
operations = [
migrations.AddField(
model_name="audioscrobblertsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="bgstatsimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="ebirdcsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="koreaderimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="lastfmimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="retroarchimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="scalecsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="trailgpximport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -0,0 +1,32 @@
# Generated by Django 4.2.29 on 2026-06-09 15:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0090_audioscrobblertsvimport_error_log_and_more"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="share_token",
field=models.UUIDField(blank=True, editable=False, null=True, unique=True),
),
migrations.AddField(
model_name="scrobble",
name="visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
db_index=True,
default="shared",
max_length=10,
),
),
]

View File

@ -0,0 +1,29 @@
from uuid import uuid4
from django.db import migrations
def backfill_share_token(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
batch = []
for scrobble in Scrobble.objects.filter(share_token__isnull=True).iterator(
chunk_size=500
):
scrobble.share_token = uuid4()
batch.append(scrobble)
if batch:
Scrobble.objects.bulk_update(batch, ["share_token"], batch_size=500)
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0091_scrobble_share_token_scrobble_visibility"),
]
operations = [
migrations.RunPython(
backfill_share_token,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,22 @@
# Generated by Django 4.2.29 on 2026-06-09 16:05
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0092_backfill_visibility_and_share_token"),
]
operations = [
migrations.RemoveField(
model_name="scrobble",
name="share_token",
),
migrations.AddField(
model_name="scrobble",
name="share_token_version",
field=models.PositiveIntegerField(default=0),
),
]

View File

@ -0,0 +1,75 @@
# Generated by Django 4.2.29 on 2026-06-09 16:24
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0093_remove_scrobble_share_token_and_more"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="share_view_count",
field=models.PositiveIntegerField(default=0),
),
migrations.AlterField(
model_name="scrobble",
name="visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
db_index=True,
default="private",
max_length=10,
),
),
migrations.CreateModel(
name="ShareViewLog",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("ip_address", models.GenericIPAddressField(blank=True, null=True)),
("user_agent", models.TextField(blank=True, null=True)),
("referrer", models.URLField(blank=True, max_length=2048, null=True)),
(
"scrobble",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="share_views",
to="scrobbles.scrobble",
),
),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
]

View File

@ -0,0 +1,20 @@
from django.db import migrations
def backfill_null_visibility(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
Scrobble.objects.filter(visibility__isnull=True).update(visibility="private")
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0094_scrobble_share_view_count_alter_scrobble_visibility_and_more"),
]
operations = [
migrations.RunPython(
backfill_null_visibility,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,35 @@
from django.db import migrations
def convert_page_data_to_dict(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
log__page_data=None
):
page_data = scrobble.log.get("page_data")
if isinstance(page_data, list):
new_page_data = {}
for entry in page_data:
page_num = entry.get("page_number")
if page_num is not None:
try:
page_num = int(page_num)
except (ValueError, TypeError):
continue
new_page_data[page_num] = entry
scrobble.log["page_data"] = new_page_data
scrobble.save(update_fields=["log"])
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0095_backfill_null_visibility"),
]
operations = [
migrations.RunPython(
convert_page_data_to_dict,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -65,6 +65,9 @@ class ScrobblableMixin(TimeStampedModel):
class Meta:
abstract = True
def __str__(self) -> str:
return f"{self.title} - {self.subtitle}"
@property
def run_time_seconds(self) -> int:
run_time = 900

View File

@ -18,6 +18,8 @@ from bricksets.models import BrickSet
from charts.utils import build_charts
from dataclass_wizard.errors import ParseError
from django.conf import settings
from scrobbles.constants import Visibility
from scrobbles.sqids import encode_scrobble_share
from django.contrib.auth import get_user_model
from django.core.files import File
from django.db import models
@ -74,6 +76,7 @@ class BaseFileImportMixin(TimeStampedModel):
processed_finished = models.DateTimeField(**BNULL)
process_log = models.TextField(**BNULL)
process_count = models.IntegerField(**BNULL)
error_log = models.TextField(**BNULL)
class Meta:
abstract = True
@ -158,6 +161,14 @@ class BaseFileImportMixin(TimeStampedModel):
self.process_count = len(scrobbles)
self.save(update_fields=["process_log", "process_count"])
def record_error(self, error_message):
log_line = f"{timezone.now().isoformat()}: {error_message}"
if self.error_log:
self.error_log += "\n" + log_line
else:
self.error_log = log_line
self.save(update_fields=["error_log"])
@property
def upload_file_path(self):
raise NotImplementedError
@ -213,9 +224,16 @@ class KoReaderImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = process_koreader_sqlite_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class AudioScrobblerTSVImport(BaseFileImportMixin):
@ -255,10 +273,16 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_audioscrobbler_tsv_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class ScaleCSVImport(BaseFileImportMixin):
@ -287,6 +311,7 @@ class ScaleCSVImport(BaseFileImportMixin):
csv_file = models.FileField(upload_to=get_path, **BNULL)
original_filename = models.CharField(max_length=255, **BNULL)
file_hash = models.CharField(max_length=32, **BNULL)
def process(self, force=False):
from scrobbles.importers.scale import import_scale_csv
@ -296,9 +321,14 @@ class ScaleCSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class TrailGPXImport(BaseFileImportMixin):
@ -336,11 +366,16 @@ class TrailGPXImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_trail_gpx(
self.upload_file_path, self.user.id, self.original_filename
)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_trail_gpx(
self.upload_file_path, self.user.id, self.original_filename
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class LastFmImport(BaseFileImportMixin):
@ -390,11 +425,14 @@ class LastFmImport(BaseFileImportMixin):
last_processed = last_import.processed_finished
self.mark_started()
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class RetroarchImport(BaseFileImportMixin):
@ -425,43 +463,48 @@ class RetroarchImport(BaseFileImportMixin):
logger.info(f"You told me to force import from Retroarch")
self.mark_started()
scrobbles = None
try:
if self.lrtl_file:
import os
import tempfile
import zipfile
if self.lrtl_file:
import os
import tempfile
import zipfile
tmpdir = tempfile.mkdtemp()
try:
zip_path = os.path.join(tmpdir, "archive.zip")
with open(zip_path, "wb") as f:
f.write(self.lrtl_file.read())
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(tmpdir)
os.unlink(zip_path)
scrobbles = retroarch.import_retroarch_lrtl_files(
tmpdir + "/",
self.user.id,
)
finally:
import shutil
shutil.rmtree(tmpdir, ignore_errors=True)
else:
if not self.user.profile.retroarch_path:
logger.info(
"Trying to import Retroarch logs, but user has no retroarch_path configured"
)
self.mark_finished()
return
tmpdir = tempfile.mkdtemp()
try:
zip_path = os.path.join(tmpdir, "archive.zip")
with open(zip_path, "wb") as f:
f.write(self.lrtl_file.read())
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(tmpdir)
os.unlink(zip_path)
scrobbles = retroarch.import_retroarch_lrtl_files(
tmpdir + "/",
self.user.profile.retroarch_path,
self.user.id,
)
finally:
import shutil
shutil.rmtree(tmpdir, ignore_errors=True)
else:
if not self.user.profile.retroarch_path:
logger.info(
"Tying to import Retroarch logs, but user has no retroarch_path configured"
)
self.mark_finished()
return
scrobbles = retroarch.import_retroarch_lrtl_files(
self.user.profile.retroarch_path,
self.user.id,
)
self.record_log(scrobbles)
self.mark_finished()
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class BGStatsImport(BaseFileImportMixin):
@ -498,17 +541,21 @@ class BGStatsImport(BaseFileImportMixin):
return
self.mark_started()
try:
import json
import json
from scrobbles.scrobblers import email_scrobble_board_game
from scrobbles.scrobblers import email_scrobble_board_game
with open(self.upload_file_path, "r", encoding="utf-8") as f:
parsed_json = json.load(f)
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
with open(self.upload_file_path, "r", encoding="utf-8") as f:
parsed_json = json.load(f)
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class EBirdCSVImport(BaseFileImportMixin):
@ -553,9 +600,16 @@ class EBirdCSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_birding_csv(
self.upload_file_path, self.user_id, record_error=self.record_error
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class ScrobbleQuerySet(models.QuerySet):
@ -563,7 +617,7 @@ class ScrobbleQuerySet(models.QuerySet):
return self.select_related("user").prefetch_related(
"video",
"track",
"track__artist",
"track__artist_fk",
"podcast_episode",
"podcast_episode__podcast",
"sport_event",
@ -585,6 +639,18 @@ class ScrobbleQuerySet(models.QuerySet):
)
class ShareViewLog(TimeStampedModel):
scrobble = models.ForeignKey(
"Scrobble", on_delete=models.CASCADE, related_name="share_views"
)
ip_address = models.GenericIPAddressField(**BNULL)
user_agent = models.TextField(**BNULL)
referrer = models.URLField(max_length=2048, **BNULL)
def __str__(self):
return f"View of {self.scrobble} at {self.created}"
class Scrobble(TimeStampedModel):
"""A scrobble tracks played media items by a user."""
@ -646,6 +712,14 @@ class Scrobble(TimeStampedModel):
media_type = models.CharField(
max_length=20, choices=MediaType.choices, default=MediaType.VIDEO
)
visibility = models.CharField(
max_length=10,
choices=Visibility.choices,
default=Visibility.PRIVATE,
db_index=True,
)
share_token_version = models.PositiveIntegerField(default=0)
share_view_count = models.PositiveIntegerField(default=0)
user = models.ForeignKey(User, blank=True, null=True, on_delete=models.DO_NOTHING)
# Time keeping
@ -827,6 +901,16 @@ class Scrobble(TimeStampedModel):
self.save(update_fields=["uuid"])
return reverse("scrobbles:detail", kwargs={"uuid": self.uuid})
def get_share_url(self):
if self.visibility == Visibility.PRIVATE:
return None
sqid = encode_scrobble_share(self.id, self.share_token_version)
return reverse("scrobbles:shared-detail", kwargs={"sqid": sqid})
def regenerate_share_token(self):
self.share_token_version += 1
self.save(update_fields=["share_token_version"])
def push_to_archivebox(self):
pushable_media = hasattr(self.media_obj, "push_to_archivebox") and callable(
self.media_obj.push_to_archivebox
@ -856,6 +940,8 @@ class Scrobble(TimeStampedModel):
logdata_cls = logdata.BaseLogData
log_dict = self.log
if isinstance(log_dict, logdata.BaseLogData):
log_dict = log_dict.asdict
if isinstance(self.log, str):
# There's nothing stopping django from saving a string in a JSONField :(
logger.warning(
@ -881,8 +967,13 @@ class Scrobble(TimeStampedModel):
logger.warning("Log data could not be loaded", e)
return logdata_cls()
# Strip log-only keys (stored in JSONField but not part of LogData dataclass)
logdata_kwargs = {
k: v for k, v in log_dict.items() if k in logdata_cls().__dataclass_fields__
}
try:
return logdata_cls(**log_dict)
return logdata_cls(**logdata_kwargs)
except ParseError as e:
logger.warning(
"Could not parse log data",
@ -1327,20 +1418,18 @@ class Scrobble(TimeStampedModel):
or scrobble_data["playback_status"] == "stopped"
):
if read_log_page:
page_list = scrobble.log.get("page_data", [])
if page_list:
for page in page_list:
page_data = scrobble.log.get("page_data", {})
if page_data:
for page_num, page in page_data.items():
if not page.get("end_ts", None):
page["end_ts"] = int(timezone.now().timestamp())
page["duration"] = page["end_ts"] - page.get("start_ts")
page_list.append(
BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
page_data[read_log_page] = BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
scrobble.log["page_data"] = page_list
scrobble.log["page_data"] = page_data
scrobble.save(update_fields=["log"])
elif "log" in scrobble_data.keys() and scrobble.log:
scrobble_data["log"] = scrobble.log | scrobble_data["log"]
@ -1351,13 +1440,13 @@ class Scrobble(TimeStampedModel):
if read_log_page:
scrobble_data["log"] = BookLogData(
page_data=[
BookPageLogData(
page_data={
read_log_page: BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
]
)
}
).asdict
logger.info(
f"[scrobbling] creating new scrobble",
@ -1372,7 +1461,7 @@ class Scrobble(TimeStampedModel):
"calories", None
):
if media.calories:
scrobble_data["log"] = FoodLogData(calories=media.calories)
scrobble_data["log"] = FoodLogData(calories=media.calories).asdict
scrobble = cls.create(scrobble_data)
return scrobble
@ -1527,8 +1616,25 @@ class Scrobble(TimeStampedModel):
cls,
scrobble_data: dict,
) -> "Scrobble":
if "visibility" not in scrobble_data:
user = scrobble_data.get("user")
media_type = scrobble_data.get("media_type")
override = None
if user and media_type:
try:
profile = user.profile
overrides = profile.media_type_visibility or {}
override = overrides.get(media_type)
except user.__class__.profile.RelatedObjectDoesNotExist:
pass
scrobble_data["visibility"] = override or Visibility.PRIVATE
scrobble = cls.objects.create(**scrobble_data)
ScrobbleNtfyNotification(scrobble).send()
if not (
scrobble.media_type == cls.MediaType.GEO_LOCATION
and scrobble.media_obj
and not scrobble.media_obj.title
):
ScrobbleNtfyNotification(scrobble).send()
return scrobble
def stop(self, timestamp=None, force_finish=False) -> None:
@ -1678,33 +1784,27 @@ class Scrobble(TimeStampedModel):
return False
def calculate_reading_stats(self, commit=True):
page_data = self.log.get("page_data")
page_data = self.logdata.page_data
if page_data:
# --- Sort safely by numeric page_number ---
def safe_page_number(entry):
try:
return int(getattr("page_number", entry), 0)
except (ValueError, TypeError):
return float("inf")
if isinstance(page_data, dict):
logger.warning("Page data is dict, migrate koreader data")
return
valid_pages = sorted(int(k) for k in page_data.keys())
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
elif isinstance(page_data, list):
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
page_data.sort(key=safe_page_number)
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
else:
page_start = self.log.get("page_start")
page_end = self.log.get("page_end")
@ -1713,3 +1813,129 @@ class Scrobble(TimeStampedModel):
if commit and "pages_read" in self.log:
self.save(update_fields=["log"])
class FavoriteMedia(TimeStampedModel):
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
user = models.ForeignKey(User, on_delete=models.CASCADE)
video = models.ForeignKey(Video, on_delete=models.CASCADE, **BNULL)
channel = models.ForeignKey("videos.Channel", on_delete=models.CASCADE, **BNULL)
track = models.ForeignKey(Track, on_delete=models.CASCADE, **BNULL)
podcast_episode = models.ForeignKey(
PodcastEpisode, on_delete=models.CASCADE, **BNULL
)
sport_event = models.ForeignKey(SportEvent, on_delete=models.CASCADE, **BNULL)
book = models.ForeignKey(Book, on_delete=models.CASCADE, **BNULL)
paper = models.ForeignKey(Paper, on_delete=models.CASCADE, **BNULL)
video_game = models.ForeignKey(VideoGame, on_delete=models.CASCADE, **BNULL)
board_game = models.ForeignKey(BoardGame, on_delete=models.CASCADE, **BNULL)
geo_location = models.ForeignKey(GeoLocation, on_delete=models.CASCADE, **BNULL)
beer = models.ForeignKey(Beer, on_delete=models.CASCADE, **BNULL)
puzzle = models.ForeignKey(Puzzle, on_delete=models.CASCADE, **BNULL)
food = models.ForeignKey(Food, on_delete=models.CASCADE, **BNULL)
trail = models.ForeignKey(Trail, on_delete=models.CASCADE, **BNULL)
task = models.ForeignKey(Task, on_delete=models.CASCADE, **BNULL)
web_page = models.ForeignKey(WebPage, on_delete=models.CASCADE, **BNULL)
life_event = models.ForeignKey(LifeEvent, on_delete=models.CASCADE, **BNULL)
mood = models.ForeignKey(Mood, on_delete=models.CASCADE, **BNULL)
brick_set = models.ForeignKey(BrickSet, on_delete=models.CASCADE, **BNULL)
birding_location = models.ForeignKey(
BirdingLocation, on_delete=models.CASCADE, **BNULL
)
media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
sent_to_mopidy = models.BooleanField(default=False)
class Meta:
ordering = ["-created"]
def __str__(self):
return f"{self.user} favorites {self.media_obj}"
@property
def media_obj(self):
media_obj = None
if self.video:
media_obj = self.video
if self.track:
media_obj = self.track
if self.podcast_episode:
media_obj = self.podcast_episode
if self.sport_event:
media_obj = self.sport_event
if self.book:
media_obj = self.book
if self.video_game:
media_obj = self.video_game
if self.board_game:
media_obj = self.board_game
if self.geo_location:
media_obj = self.geo_location
if self.web_page:
media_obj = self.web_page
if self.life_event:
media_obj = self.life_event
if self.mood:
media_obj = self.mood
if self.brick_set:
media_obj = self.brick_set
if self.trail:
media_obj = self.trail
if self.beer:
media_obj = self.beer
if self.puzzle:
media_obj = self.puzzle
if self.task:
media_obj = self.task
if self.food:
media_obj = self.food
if self.channel:
media_obj = self.channel
if self.birding_location:
media_obj = self.birding_location
return media_obj
@classmethod
def toggle(cls, media_obj, user):
media_type = media_obj.__class__.__name__
if media_type not in Scrobble.MediaType.list():
raise ValueError(f"Unknown media type: {media_type}")
fk_map = {
"Video": "video",
"Channel": "channel",
"Track": "track",
"PodcastEpisode": "podcast_episode",
"SportEvent": "sport_event",
"Book": "book",
"Paper": "paper",
"VideoGame": "video_game",
"BoardGame": "board_game",
"GeoLocation": "geo_location",
"Beer": "beer",
"Puzzle": "puzzle",
"Food": "food",
"Trail": "trail",
"Task": "task",
"WebPage": "web_page",
"LifeEvent": "life_event",
"Mood": "mood",
"BrickSet": "brick_set",
"BirdingLocation": "birding_location",
}
fk = fk_map.get(media_type)
if not fk:
raise ValueError(f"No FK mapping for media type: {media_type}")
existing = cls.objects.filter(user=user, **{fk: media_obj}).first()
if existing:
existing.delete()
return None
return cls.objects.create(
user=user,
media_type=media_type,
**{fk: media_obj},
)

View File

@ -83,7 +83,7 @@ class MoodNtfyNotification(BasicNtfyNotification):
def __init__(self, profile, **kwargs):
super().__init__(profile)
self.ntfy_str: str = "Would you like to check in about your mood?"
self.click_url = self.url_tmpl.format(path=reverse("moods:mood_list"))
self.click_url = self.url_tmpl.format(path=reverse("moods:checkin"))
self.title = "Mood Check-in!"
def send(self):

View File

@ -35,6 +35,7 @@ from scrobbles.notifications import ScrobbleNtfyNotification
from scrobbles.utils import (
convert_to_seconds,
extract_domain,
make_note_timestamp,
next_url_if_exists,
remove_last_part,
)
@ -241,12 +242,13 @@ def manual_scrobble_event(
):
data_dict = lookup_event_from_thesportsdb(thesportsdb_id)
event = SportEvent.find_or_create(data_dict)
event, logdata = SportEvent.find_or_create(data_dict)
scrobble_dict = {
"user_id": user_id,
"timestamp": timezone.now(),
"playback_position_seconds": 0,
"source": "TheSportsDB",
"log": logdata,
}
return Scrobble.create_or_update(event, user_id, scrobble_dict)
@ -355,10 +357,7 @@ def manual_scrobble_book(
if action == "stop":
if url:
if isinstance(scrobble.log, "BookLogData"):
scrobble.log.resume_url = next_url_if_exists(url)
else:
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.save(update_fields=["log"])
scrobble.stop(force_finish=True)
@ -450,11 +449,11 @@ def email_scrobble_board_game(
locations[location_dict.get("id")] = location
scrobbles_created = []
second = 0
for play_dict in bgstat_data.get("plays", []):
hour = None
minute = None
second = None
comments = None
if "comments" in play_dict.keys():
for line in play_dict.get("comments", "").split("\n"):
if "Learning to play" in line:
@ -469,7 +468,7 @@ def email_scrobble_board_game(
except IndexError:
second = 0
log_data["notes"] = [play_dict.get("comments")]
comments = play_dict.get("comments")
log_data["expansion_ids"] = []
try:
base_game = base_games[play_dict.get("gameRefId")]
@ -527,6 +526,9 @@ def email_scrobble_board_game(
duration_seconds = base_game.run_time_seconds
stop_timestamp = timestamp + timedelta(seconds=duration_seconds)
if comments:
log_data["notes"] = {make_note_timestamp(stop_timestamp): comments}
logger.info(f"Creating scrobble for {base_game} at {timestamp}")
log_data["raw_data"] = bgstat_data
scrobble_dict = {
@ -535,6 +537,7 @@ def email_scrobble_board_game(
"playback_position_seconds": duration_seconds,
"source": "BG Stats",
"log": log_data,
"visibility": "private",
}
scrobble = None
@ -558,7 +561,6 @@ def email_scrobble_board_game(
scrobble.played_to_completion = True
scrobble.save()
scrobbles_created.append(scrobble)
ScrobbleNtfyNotification(scrobble).send()
return scrobbles_created
@ -685,9 +687,10 @@ def todoist_scrobble_update_task(
)
return
timestamp = todoist_note.get("posted_at") or make_note_timestamp()
if not scrobble.log.get("notes"):
scrobble.log["notes"] = []
scrobble.log["notes"].append(todoist_note.get("notes"))
scrobble.log["notes"] = {}
scrobble.log["notes"][timestamp] = todoist_note.get("notes")
scrobble.save(update_fields=["log"])
logger.info(
"[todoist_scrobble_update_task] todoist note added",
@ -757,7 +760,10 @@ def todoist_scrobble_task(
todoist_task["title"] = todoist_task.pop("description")
todoist_task["description"] = todoist_task.pop("details")
todoist_task["labels"] = todoist_task.pop("todoist_label_list", [])
labels = todoist_task.pop("todoist_label_list", [])
todoist_task["labels"] = [
l for l in labels if l.lower() != "inprogress"
]
todoist_task.pop("todoist_type")
todoist_task.pop("todoist_event")
@ -782,9 +788,37 @@ def todoist_scrobble_task(
return scrobble
ORG_HEADING_RE = re.compile(r"^(\*+\s+.*)$", re.MULTILINE)
NOTE_CONTENT_SPLIT = re.compile(r"^\*{3,}\s", re.MULTILINE)
def _truncate_at_org_header(text: str) -> str:
"""Truncate text at the first org-mode heading (*** or more)."""
parts = NOTE_CONTENT_SPLIT.split(text, maxsplit=1)
return parts[0].strip()
def _extract_org_section(body: str | None, heading: str) -> str | None:
"""Extract content under a specific org-mode sub-heading (e.g. '*** Description')."""
if not body:
return None
sections = ORG_HEADING_RE.split(body)
# sections alternates: [prefix, heading1, content1, heading2, content2, ...]
for i, section in enumerate(sections):
if section.strip().startswith(heading):
if i + 1 < len(sections):
return sections[i + 1].strip()
return ""
return None
def emacs_scrobble_update_task(
emacs_id: str, emacs_notes: dict, user_id: int
emacs_id: str,
emacs_notes: list,
user_id: int,
description: Optional[str] = None,
) -> Optional[Scrobble]:
description = _extract_org_section(description, "*** Description")
scrobble = Scrobble.objects.filter(
in_progress=True,
user_id=user_id,
@ -803,27 +837,35 @@ def emacs_scrobble_update_task(
)
return
notes_updated = False
log_updated = False
if not scrobble.log.get("notes"):
scrobble.log["notes"] = {}
for note in emacs_notes:
try:
existing_note_ts = [
n.get("timestamp") for n in scrobble.log.get("notes", [])
]
except AttributeError:
existing_note_ts = []
if not scrobble.log.get('notes"'):
scrobble.log["notes"] = []
if note.get("timestamp") not in existing_note_ts:
scrobble.log["notes"].append({note.get("timestamp"): note.get("content")})
notes_updated = True
timestamp = note.get("timestamp")
content = note.get("content")
if not content:
continue
content = _truncate_at_org_header(content)
if not content:
continue
if timestamp:
existing = scrobble.log["notes"].get(timestamp)
if existing != content:
scrobble.log["notes"][timestamp] = content
log_updated = True
if notes_updated:
if description is not None and scrobble.log.get("description") != description:
scrobble.log["description"] = description
log_updated = True
if log_updated:
scrobble.save(update_fields=["log"])
logger.info(
"[emacs_scrobble_update_task] emacs note added",
"[emacs_scrobble_update_task] emacs scrobble updated",
extra={
"emacs_note": emacs_notes,
"emacs_id": emacs_id,
"user_id": user_id,
"media_type": Scrobble.MediaType.TASK,
},
@ -865,7 +907,7 @@ def emacs_scrobble_task(
logger.info(
"[emacs_scrobble_task] cannot start already started task",
extra={
"ormode_id": orgmode_id,
"orgmode_id": orgmode_id,
},
)
return in_progress_scrobble
@ -884,11 +926,9 @@ def emacs_scrobble_task(
if in_progress_scrobble:
return in_progress_scrobble
notes = task_data.pop("notes")
if notes:
task_data["notes"] = [note.get("content") for note in notes]
task_data.pop("notes", None)
task_data["title"] = task_data.pop("description")
task_data["description"] = task_data.pop("body")
task_data["description"] = _extract_org_section(task_data.pop("body"), "*** Description")
task_data["labels"] = task_data.pop("labels")
task_data["orgmode_id"] = task_data.pop("source_id")
@ -1110,7 +1150,7 @@ def web_scrobbler_scrobble_video_or_song(
artist_name = data_dict.get("artist")
track_name = data_dict.get("track")
tracks = Track.objects.filter(
artist__name=data_dict.get("artist"), title=data_dict.get("track")
artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
)
if tracks.count() > 1:
logger.warning(

View File

@ -4,8 +4,14 @@ from django.db.models.signals import post_delete, post_save
from django.dispatch import receiver
from django.utils import timezone
from scrobbles.models import Scrobble
from scrobbles.tasks import CHARTABLE_MEDIA_TYPES, SCROBBLES_WITHOUT_CHARTS, update_charts_for_timestamp
from scrobbles.models import FavoriteMedia, Scrobble
from scrobbles.tasks import (
add_favorite_to_mopidy_playlist,
CHARTABLE_MEDIA_TYPES,
remove_favorite_from_mopidy_playlist,
SCROBBLES_WITHOUT_CHARTS,
update_charts_for_timestamp,
)
from scrobbles.utils import tokenize_title_to_tags
logger = logging.getLogger(__name__)
@ -72,3 +78,28 @@ def add_tags_from_task_title(sender, instance, **kwargs):
for tag in new_tags:
if tag not in existing_tags:
instance.tags.add(tag)
@receiver(post_save, sender=FavoriteMedia)
def add_to_mopidy_playlist_on_favorite(sender, instance, created, **kwargs):
if not created:
return
if instance.media_type != Scrobble.MediaType.TRACK:
return
if not instance.track:
return
add_favorite_to_mopidy_playlist.delay(instance.id)
@receiver(post_delete, sender=FavoriteMedia)
def remove_from_mopidy_playlist_on_unfavorite(sender, instance, **kwargs):
if instance.media_type != Scrobble.MediaType.TRACK:
return
if not instance.track_id:
return
remove_favorite_from_mopidy_playlist.delay(
user_id=instance.user_id,
track_id=instance.track_id,
)

View File

@ -0,0 +1,36 @@
from sqids import Sqids
_sqids = None
def _make_alphabet() -> str:
import hashlib
from django.conf import settings
digest = hashlib.sha256(settings.SECRET_KEY.encode()).hexdigest()
base = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
seed = int(digest[:16], 16)
shuffled = list(base)
for i in range(len(shuffled) - 1, 0, -1):
seed = (seed * 1103515245 + 12345) & 0x7FFFFFFF
j = seed % (i + 1)
shuffled[i], shuffled[j] = shuffled[j], shuffled[i]
return "".join(shuffled)
def get_sqids() -> Sqids:
global _sqids
if _sqids is None:
_sqids = Sqids(
alphabet=_make_alphabet(),
min_length=6,
)
return _sqids
def encode_scrobble_share(scrobble_id: int, version: int) -> str:
return get_sqids().encode([scrobble_id, version])
def decode_scrobble_share(sqid: str) -> list[int] | None:
return get_sqids().decode(sqid)

View File

@ -12,6 +12,7 @@ from charts.utils import (
from django.apps import apps
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import models
from django.utils import timezone
logger = logging.getLogger(__name__)
@ -542,3 +543,149 @@ def send_mood_checkin():
from vrobbler.apps.scrobbles.utils import send_mood_checkin_reminders
send_mood_checkin_reminders()
@shared_task
def backfill_scrobble_sentiment():
"""Backfill VADER sentiment for scrobbles with notes (replaces @hourly cron)."""
from scrobbles.models import Scrobble
from scrobbles.utils import analyze_scrobble_sentiment
qs = Scrobble.objects.filter(
models.Q(log__notes__isnull=False)
& ~models.Q(log__notes=[])
& ~models.Q(log__notes={})
& models.Q(log__sentiment__isnull=True)
)
count = 0
for scrobble in qs.iterator():
if analyze_scrobble_sentiment(scrobble):
count += 1
logger.info(
"Backfilled sentiment for %d scrobbles",
count,
)
@shared_task
def add_favorite_to_mopidy_playlist(favorite_id):
from scrobbles.models import FavoriteMedia
from scrobbles.utils import add_track_to_mopidy_favorites_playlist
favorite = FavoriteMedia.objects.filter(id=favorite_id).first()
if not favorite:
return
add_track_to_mopidy_favorites_playlist(favorite)
@shared_task
def remove_favorite_from_mopidy_playlist(user_id, track_id):
from music.models import Track
from scrobbles.utils import remove_track_from_mopidy_favorites_playlist
User = get_user_model()
try:
user = User.objects.get(id=user_id)
track = Track.objects.get(id=track_id)
except (User.DoesNotExist, Track.DoesNotExist):
return
import types
proxy = types.SimpleNamespace(
media_type="Track",
user=user,
user_id=user.id,
track=track,
)
remove_track_from_mopidy_favorites_playlist(proxy)
@shared_task
def add_scrobble_to_mopidy_queue(scrobble_id):
from scrobbles.models import Scrobble
scrobble = Scrobble.objects.filter(id=scrobble_id).first()
if not scrobble:
return
profile = scrobble.user.profile
mopidy_url = profile.mopidy_api_url
if not mopidy_url:
return
mopidy_uri = (scrobble.log or {}).get("raw_data", {}).get("mopidy_uri")
track = scrobble.track if scrobble.media_type == "Track" else None
if not mopidy_uri and track:
sibling = (
Scrobble.objects.filter(track=track, user=scrobble.user)
.order_by("-timestamp")
.iterator()
)
for s in sibling:
uri = (s.log or {}).get("raw_data", {}).get("mopidy_uri")
if uri:
mopidy_uri = uri
break
if not mopidy_uri:
logger.warning(
"No Mopidy URI found for scrobble",
extra={"scrobble_id": scrobble_id, "user_id": scrobble.user_id},
)
return
import requests
rpc_url = mopidy_url.rstrip("/") + "/rpc"
payload = {
"jsonrpc": "2.0",
"id": 1,
"method": "core.tracklist.add",
"params": {"uris": [mopidy_uri]},
}
try:
resp = requests.post(rpc_url, json=payload, timeout=10)
resp.raise_for_status()
rpc_result = resp.json()
if rpc_result.get("error"):
logger.error(
"Mopidy error adding to queue",
extra={"error": rpc_result["error"], "scrobble_id": scrobble_id},
)
else:
logger.info(
"Added track to Mopidy queue",
extra={"scrobble_id": scrobble_id, "mopidy_uri": mopidy_uri},
)
except requests.RequestException as e:
logger.error(
"Failed to add track to Mopidy queue",
extra={"scrobble_id": scrobble_id, "error": str(e)},
)
@shared_task
def add_scrobble_to_mopidy_monthly_playlist(scrobble_id):
from scrobbles.models import Scrobble
from scrobbles.utils import add_track_to_mopidy_monthly_playlist
scrobble = Scrobble.objects.filter(id=scrobble_id).first()
if not scrobble:
return
track = scrobble.track if scrobble.media_type == "Track" else None
if track:
sibling = (
Scrobble.objects.filter(track=track, user=scrobble.user)
.order_by("-timestamp")
.iterator()
)
for s in sibling:
if (s.log or {}).get("raw_data", {}).get("mopidy_uri"):
scrobble = s
break
add_track_to_mopidy_monthly_playlist(scrobble)

View File

@ -153,12 +153,48 @@ urlpatterns = [
name="long-plays",
),
path("scrobbles/", views.ScrobbleListView.as_view(), name="scrobble-list"),
path("explore/", views.ScrobbleExploreView.as_view(), name="explore"),
path(
"shared/<str:sqid>/",
views.ScrobbleShareView.as_view(),
name="shared-detail",
),
path(
"scrobbles/<slug:uuid>/",
views.ScrobbleDetailView.as_view(),
name="detail",
),
path(
"scrobbles/<slug:uuid>/regenerate-share-token/",
views.RegenerateShareTokenView.as_view(),
name="regenerate-share-token",
),
path(
"scrobbles/<slug:uuid>/change-visibility/",
views.ChangeVisibilityView.as_view(),
name="change-visibility",
),
path(
"scrobbles/<slug:uuid>/share-analytics/",
views.ScrobbleShareAnalyticsView.as_view(),
name="share-analytics",
),
path(
"scrobbles/<slug:uuid>/add-to-mopidy-queue/",
views.add_to_mopidy_queue,
name="add-to-mopidy-queue",
),
path(
"scrobbles/<slug:uuid>/add-to-mopidy-monthly-playlist/",
views.add_to_mopidy_monthly_playlist,
name="add-to-mopidy-monthly-playlist",
),
path("scrobbles/<slug:uuid>/start/", views.scrobble_start, name="start"),
path("scrobbles/<slug:uuid>/finish/", views.scrobble_finish, name="finish"),
path("scrobbles/<slug:uuid>/cancel/", views.scrobble_cancel, name="cancel"),
path(
"favorite/<str:media_type>/<int:object_id>/toggle/",
views.toggle_favorite,
name="toggle-favorite",
),
]

View File

@ -15,6 +15,7 @@ from django.db import models
from django.db.models.fields.json import KeyTextTransform
from django.db.models.functions import Cast, TruncDate
from django.utils import timezone
from django.utils.dateformat import DateFormat
from profiles.models import UserProfile
from profiles.utils import now_user_timezone
from scrobbles.constants import LONG_PLAY_MEDIA
@ -32,6 +33,16 @@ from webdav.client import get_webdav_client
if TYPE_CHECKING:
from scrobbles.models import Scrobble
NOTE_TIMESTAMP_FORMAT = "%Y-%m-%dT%H:%M:%S.%fZ"
def make_note_timestamp(dt: datetime | None = None) -> str:
if dt is None:
dt = timezone.now()
return dt.strftime(NOTE_TIMESTAMP_FORMAT)
logger = logging.getLogger(__name__)
User = get_user_model()
@ -77,8 +88,7 @@ def get_scrobbles_for_media(media_obj, user: User) -> models.QuerySet:
return Scrobble.objects.filter(media_query, user=user)
def get_recently_played_board_games(user: User) -> dict:
...
def get_recently_played_board_games(user: User) -> dict: ...
def get_long_plays_in_progress(user: User) -> dict:
@ -413,6 +423,288 @@ def get_daily_calorie_dict_for_user(user_id: int) -> dict[date, int]:
return {entry["day"]: entry["total_calories"] for entry in qs}
def _mopidy_rpc(profile, method, params=None):
rpc_url = profile.mopidy_api_url.rstrip("/") + "/rpc"
payload = {
"jsonrpc": "2.0",
"id": 1,
"method": method,
}
if params:
payload["params"] = params
resp = requests.post(rpc_url, json=payload, timeout=10)
resp.raise_for_status()
result = resp.json()
if result.get("error"):
raise RuntimeError(f'Mopidy error: {result["error"]}')
return result.get("result")
def _ensure_mopidy_playlist(profile):
playlist_name = profile.favorites_mopidy_playlist
# Strip any m3u: prefix and .m3u8 suffix the user may have included
playlist_name = playlist_name.removeprefix("m3u:").removesuffix(".m3u8")
try:
playlists = _mopidy_rpc(profile, "core.playlists.as_list") or []
for pl in playlists:
if pl.get("name") == playlist_name:
existing = _mopidy_rpc(
profile, "core.playlists.lookup", {"uri": pl["uri"]}
)
if existing:
return existing
except (requests.RequestException, RuntimeError):
pass
result = _mopidy_rpc(
profile, "core.playlists.create",
{"name": playlist_name, "uri_scheme": "m3u"},
)
logger.info(
"Created Mopidy favorites playlist",
extra={"uri": result.get("uri") if result else playlist_name},
)
return result
def _scrobble_with_mopidy_uri(track, user):
"""Find a scrobble for this track+user that has a mopidy_uri in its log."""
from scrobbles.models import Scrobble
for scrobble in (
Scrobble.objects.filter(track=track, user=user)
.order_by("-timestamp")
.iterator()
):
raw_data = scrobble.log.get("raw_data") or {}
if raw_data.get("mopidy_uri"):
return scrobble
return None
def add_track_to_mopidy_favorites_playlist(favorite):
if favorite.media_type != "Track" or not favorite.track:
return
profile = favorite.user.profile
if not profile.favorites_mopidy_playlist or not profile.mopidy_api_url:
return
track = favorite.track
scrobble = _scrobble_with_mopidy_uri(track, favorite.user)
if not scrobble:
logger.warning(
"No Mopidy URI found for track",
extra={"track_id": track.id, "user_id": favorite.user_id},
)
return
mopidy_uri = (scrobble.log or {}).get("raw_data", {}).get("mopidy_uri")
try:
playlist = _ensure_mopidy_playlist(profile)
if playlist and playlist.get("uri"):
existing_tracks = playlist.get("tracks") or []
track_uris = [t["uri"] for t in existing_tracks if isinstance(t, dict)]
if mopidy_uri in track_uris:
logger.info(
"Track already in Mopidy favorites playlist",
extra={"track_id": track.id, "mopidy_uri": mopidy_uri},
)
favorite.sent_to_mopidy = True
favorite.save(update_fields=["sent_to_mopidy"])
return
new_track = {"__model__": "Track", "uri": mopidy_uri}
existing_tracks.append(new_track)
_mopidy_rpc(
profile,
"core.playlists.save",
{
"playlist": {
"__model__": "Playlist",
"uri": playlist["uri"],
"name": playlist.get("name", "Favorites"),
"tracks": existing_tracks,
"last_modified": playlist.get("last_modified"),
},
},
)
else:
_mopidy_rpc(profile, "core.tracklist.add", {"uris": [mopidy_uri]})
favorite.sent_to_mopidy = True
favorite.save(update_fields=["sent_to_mopidy"])
logger.info(
"Added track to Mopidy favorites playlist",
extra={"track_id": track.id, "user_id": favorite.user_id},
)
except (requests.RequestException, RuntimeError) as e:
logger.debug(e)
logger.error(
"Failed to add track to Mopidy favorites playlist",
extra={"track_id": track.id, "user_id": favorite.user_id, "error": str(e)},
)
def resubmit_favorites_to_mopidy(user):
from scrobbles.models import FavoriteMedia
favorites = FavoriteMedia.objects.filter(
user=user,
media_type="Track",
track__isnull=False,
)
for favorite in favorites:
add_track_to_mopidy_favorites_playlist(favorite)
def remove_track_from_mopidy_favorites_playlist(favorite):
if favorite.media_type != "Track" or not favorite.track:
return
profile = favorite.user.profile
if not profile.favorites_mopidy_playlist or not profile.mopidy_api_url:
return
track = favorite.track
scrobble = _scrobble_with_mopidy_uri(track, favorite.user)
if not scrobble:
logger.warning(
"No Mopidy URI found for track",
extra={"track_id": track.id, "user_id": favorite.user_id},
)
return
mopidy_uri = (scrobble.log or {}).get("raw_data", {}).get("mopidy_uri")
try:
playlist = _ensure_mopidy_playlist(profile)
if playlist and playlist.get("uri"):
existing_tracks = playlist.get("tracks") or []
filtered = [
t for t in existing_tracks
if not (isinstance(t, dict) and t.get("uri") == mopidy_uri)
]
if len(filtered) == len(existing_tracks):
logger.info(
"Track not found in Mopidy favorites playlist",
extra={"track_id": track.id, "mopidy_uri": mopidy_uri},
)
return
_mopidy_rpc(
profile,
"core.playlists.save",
{
"playlist": {
"__model__": "Playlist",
"uri": playlist["uri"],
"name": playlist.get("name", "Favorites"),
"tracks": filtered,
"last_modified": playlist.get("last_modified"),
},
},
)
logger.info(
"Removed track from Mopidy favorites playlist",
extra={"track_id": track.id, "user_id": favorite.user_id},
)
except (requests.RequestException, RuntimeError) as e:
logger.debug(e)
logger.error(
"Failed to remove track from Mopidy favorites playlist",
extra={"track_id": track.id, "user_id": favorite.user_id, "error": str(e)},
)
def _ensure_mopidy_playlist_by_name(profile, playlist_name):
"""Find or create a Mopidy playlist by name (without m3u: prefix handling)."""
playlist_name = playlist_name.removeprefix("m3u:").removesuffix(".m3u8")
try:
playlists = _mopidy_rpc(profile, "core.playlists.as_list") or []
for pl in playlists:
if pl.get("name") == playlist_name:
existing = _mopidy_rpc(
profile, "core.playlists.lookup", {"uri": pl["uri"]}
)
if existing:
return existing
except (requests.RequestException, RuntimeError):
pass
result = _mopidy_rpc(
profile, "core.playlists.create",
{"name": playlist_name, "uri_scheme": "m3u"},
)
return result
def add_track_to_mopidy_monthly_playlist(scrobble):
"""Add a scrobbled track to a monthly Mopidy playlist based on the user's pattern."""
profile = scrobble.user.profile
pattern = profile.monthly_mopidy_playlist_pattern
if not pattern or not profile.mopidy_api_url:
return
mopidy_uri = (scrobble.log or {}).get("raw_data", {}).get("mopidy_uri")
if not mopidy_uri:
return
now = now_user_timezone(profile)
playlist_name = DateFormat(now).format(pattern)
if not playlist_name:
return
try:
playlist = _ensure_mopidy_playlist_by_name(profile, playlist_name)
if playlist and playlist.get("uri"):
existing_tracks = playlist.get("tracks") or []
track_uris = [t["uri"] for t in existing_tracks if isinstance(t, dict)]
if mopidy_uri in track_uris:
logger.info(
"Track already in monthly Mopidy playlist",
extra={"playlist": playlist_name, "mopidy_uri": mopidy_uri},
)
return
new_track = {"__model__": "Track", "uri": mopidy_uri}
existing_tracks.append(new_track)
_mopidy_rpc(
profile,
"core.playlists.save",
{
"playlist": {
"__model__": "Playlist",
"uri": playlist["uri"],
"name": playlist.get("name", playlist_name),
"tracks": existing_tracks,
"last_modified": playlist.get("last_modified"),
},
},
)
else:
_mopidy_rpc(profile, "core.tracklist.add", {"uris": [mopidy_uri]})
logger.info(
"Added track to monthly Mopidy playlist",
extra={
"playlist": playlist_name,
"track_id": scrobble.media_obj.id,
"user_id": scrobble.user_id,
},
)
except (requests.RequestException, RuntimeError) as e:
logger.debug(e)
logger.error(
"Failed to add track to monthly Mopidy playlist",
extra={"playlist": playlist_name, "error": str(e)},
)
def remove_last_part(url: str) -> str:
url = url.rstrip("/")
if "/" not in url:
@ -507,8 +799,35 @@ def tokenize_title_to_tags(title: str) -> list[str]:
cleaned = re.sub(r"[^\w\s]", "", cleaned)
words = [
w.lower()
for w in cleaned.split()
if w.lower() not in STOPWORDS and len(w) > 2
w.lower() for w in cleaned.split() if w.lower() not in STOPWORDS and len(w) > 2
]
return words
def analyze_scrobble_sentiment(scrobble, overwrite=False) -> bool:
"""Run VADER sentiment analysis on a scrobble's notes.
Stores result in log["sentiment"] as a dict with keys:
neg, neu, pos, compound.
Returns True if analyzed, False if skipped (no notes or already done).
"""
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
log = scrobble.log or {}
if not overwrite and log.get("sentiment") is not None:
return False
notes_str = ""
if scrobble.logdata:
notes_str = scrobble.logdata.notes_as_str()
if not notes_str:
return False
analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores(notes_str)
log["sentiment"] = scores
scrobble.log = log
scrobble.save(update_fields=["log"])
return True

View File

@ -4,6 +4,8 @@ import logging
from datetime import datetime, timedelta
from uuid import uuid4
import requests
import pendulum
import pytz
from dateutil.relativedelta import relativedelta
@ -12,7 +14,7 @@ from django.contrib import messages
from django.contrib.auth import get_user_model
from django.contrib.auth.mixins import LoginRequiredMixin
from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
from django.db.models import Count, Max, Q
from django.db.models import Count, F, Max, Q, Sum
from django.db.models.query import QuerySet
from rest_framework.authentication import TokenAuthentication
from rest_framework.authtoken.models import Token
@ -30,11 +32,13 @@ from django.http import (
HttpResponseRedirect,
JsonResponse,
)
from django.shortcuts import redirect
from django.shortcuts import get_object_or_404, redirect
from django.urls import reverse_lazy
from django.utils import timezone
from django.utils.dateformat import DateFormat
from django.views.decorators.csrf import csrf_exempt
from django.views.generic import DetailView, FormView, TemplateView
from django.views.decorators.http import require_POST
from django.views.generic import DetailView, FormView, TemplateView, View
from django.views.generic.edit import CreateView
from django.views.generic.list import ListView
from moods.models import Mood
@ -68,16 +72,20 @@ from scrobbles.constants import (
)
from scrobbles.export import export_scrobbles
from scrobbles.forms import ExportScrobbleForm, ScrobbleForm
from scrobbles.constants import Visibility
from scrobbles.sqids import decode_scrobble_share
from scrobbles.models import (
AudioScrobblerTSVImport,
BGStatsImport,
EBirdCSVImport,
FavoriteMedia,
KoReaderImport,
LastFmImport,
RetroarchImport,
ScaleCSVImport,
Scrobble,
ScrobbleQuerySet,
ShareViewLog,
TrailGPXImport,
)
from scrobbles.scrobblers import *
@ -179,7 +187,7 @@ class ScrobbleableDetailView(ChartContextMixin, DetailView):
def get_context_data(self, **kwargs):
context_data = super().get_context_data(**kwargs)
scrobbles = []
if not self.request.user.is_anonymous:
if not self.request.user.is_anonymous and hasattr(self.object, "scrobble_set"):
scrobbles = self.object.scrobble_set.filter(
user=self.request.user
).order_by("-timestamp")
@ -226,7 +234,7 @@ class RecentScrobbleList(ListView):
# Get user's home scrobble limit (default 20)
home_limit = 20
if hasattr(user, 'profile') and user.profile.home_scrobble_limit:
if hasattr(user, "profile") and user.profile.home_scrobble_limit:
home_limit = user.profile.home_scrobble_limit
today = timezone.localtime(timezone.now())
@ -363,7 +371,11 @@ class ScrobbleListView(LoginRequiredMixin, ListView):
qs = qs.filter(id__in=matching_ids).distinct()
else:
tag_list = []
visibility_param = self.request.GET.get("visibility", "")
if visibility_param in ("public", "shared", "private"):
qs = qs.filter(visibility=visibility_param)
self.tag_list = tag_list
self._full_queryset = qs
return qs
def _compute_overlap_groups(self, scrobbles):
@ -430,6 +442,12 @@ class ScrobbleListView(LoginRequiredMixin, ListView):
ctx["tag_list"] = getattr(self, "tag_list", [])
scrobbles = list(ctx.get("object_list", []))
ctx["overlap_map"] = self._compute_overlap_groups(scrobbles)
full_qs = getattr(self, "_full_queryset", None)
if full_qs is not None and getattr(self, "tag_list", []):
total = (
full_qs.aggregate(total=Sum("playback_position_seconds"))["total"] or 0
)
ctx["total_time_seconds"] = total
return ctx
@ -612,9 +630,7 @@ class KoReaderImportCreateView(LoginRequiredMixin, JsonableResponseMixin, Create
return HttpResponseRedirect(self.request.META.get("HTTP_REFERER"))
class ScaleCSVImportCreateView(
LoginRequiredMixin, JsonableResponseMixin, CreateView
):
class ScaleCSVImportCreateView(LoginRequiredMixin, JsonableResponseMixin, CreateView):
model = ScaleCSVImport
fields = ["csv_file"]
template_name = "scrobbles/upload_form.html"
@ -623,17 +639,13 @@ class ScaleCSVImportCreateView(
def form_valid(self, form):
self.object = form.save(commit=False)
self.object.user = self.request.user
self.object.original_filename = (
form.cleaned_data["csv_file"].name
)
self.object.original_filename = form.cleaned_data["csv_file"].name
self.object.save()
self.object.process()
return HttpResponseRedirect(self.request.META.get("HTTP_REFERER"))
class TrailGPXImportCreateView(
LoginRequiredMixin, JsonableResponseMixin, CreateView
):
class TrailGPXImportCreateView(LoginRequiredMixin, JsonableResponseMixin, CreateView):
model = TrailGPXImport
fields = ["gpx_file"]
template_name = "scrobbles/upload_form.html"
@ -852,7 +864,9 @@ def scrobble_start(request, uuid):
book=media_obj,
user_id=user_id,
)
.filter(Q(long_play_complete=False) | Q(long_play_complete__isnull=True))
.filter(
Q(long_play_complete=False) | Q(long_play_complete__isnull=True)
)
.filter(log__page_end__isnull=False)
.order_by("-timestamp")
.first()
@ -967,6 +981,113 @@ def scrobble_cancel(request, uuid):
return HttpResponseRedirect(request.META.get("HTTP_REFERER"))
@require_POST
def add_to_mopidy_queue(request, uuid):
if not request.user.is_authenticated:
return redirect("scrobbles:detail", uuid=uuid)
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
mopidy_url = request.user.profile.mopidy_api_url
if not mopidy_url:
messages.add_message(
request,
messages.ERROR,
"Mopidy API URL not configured in your profile settings.",
)
return redirect("scrobbles:detail", uuid=uuid)
from scrobbles.tasks import add_scrobble_to_mopidy_queue as task
task.delay(scrobble.id)
msg = f'Adding "{scrobble.media_obj}" to Mopidy queue.'
messages.add_message(request, messages.SUCCESS, msg)
return redirect("scrobbles:detail", uuid=uuid)
@require_POST
def add_to_mopidy_monthly_playlist(request, uuid):
if not request.user.is_authenticated:
return redirect("scrobbles:detail", uuid=uuid)
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
profile = request.user.profile
pattern = profile.monthly_mopidy_playlist_pattern
if not pattern or not profile.mopidy_api_url:
messages.add_message(
request,
messages.ERROR,
"Monthly playlist pattern or Mopidy API URL not configured in your profile.",
)
return redirect("scrobbles:detail", uuid=uuid)
now = now_user_timezone(profile)
playlist_name = DateFormat(now).format(pattern)
from scrobbles.tasks import add_scrobble_to_mopidy_monthly_playlist as task
task.delay(scrobble.id)
messages.add_message(
request,
messages.SUCCESS,
f'Adding "{scrobble.media_obj}" to monthly playlist "{playlist_name}".',
)
return redirect("scrobbles:detail", uuid=uuid)
@require_POST
def toggle_favorite(request, media_type, object_id):
if not request.user.is_authenticated:
return HttpResponseRedirect(request.META.get("HTTP_REFERER", "/"))
app_model_map = {
"Video": ("videos", "Video"),
"Channel": ("videos", "Channel"),
"Track": ("music", "Track"),
"PodcastEpisode": ("podcasts", "PodcastEpisode"),
"SportEvent": ("sports", "SportEvent"),
"Book": ("books", "Book"),
"Paper": ("books", "Paper"),
"VideoGame": ("videogames", "VideoGame"),
"BoardGame": ("boardgames", "BoardGame"),
"GeoLocation": ("locations", "GeoLocation"),
"Beer": ("beers", "Beer"),
"Puzzle": ("puzzles", "Puzzle"),
"Food": ("foods", "Food"),
"Trail": ("trails", "Trail"),
"Task": ("tasks", "Task"),
"WebPage": ("webpages", "WebPage"),
"LifeEvent": ("lifeevents", "LifeEvent"),
"Mood": ("moods", "Mood"),
"BrickSet": ("bricksets", "BrickSet"),
"BirdingLocation": ("birds", "BirdingLocation"),
}
app_label, model_name = app_model_map.get(media_type, (None, None))
if not app_label:
messages.add_message(
request, messages.ERROR, f"Unknown media type: {media_type}"
)
return HttpResponseRedirect(request.META.get("HTTP_REFERER", "/"))
model = apps.get_model(app_label, model_name)
media_obj = get_object_or_404(model, id=object_id)
result = FavoriteMedia.toggle(media_obj, request.user)
is_favorited = result is not None
if not is_favorited:
msg = f'Removed "{media_obj}" from favorites.'
else:
msg = f'Added "{media_obj}" to favorites.'
if request.headers.get("x-requested-with") == "XMLHttpRequest":
return JsonResponse({"is_favorited": is_favorited, "message": msg})
messages.add_message(request, messages.SUCCESS, msg)
return HttpResponseRedirect(request.META.get("HTTP_REFERER", "/"))
@api_view(["GET"])
@permission_classes([IsAuthenticated])
def export(request):
@ -1021,6 +1142,15 @@ class ScrobbleDetailView(DetailView):
slug_url_kwarg = "uuid"
paginate_by = 100
def get_object(self, queryset=None):
scrobble = super().get_object(queryset=queryset)
user = self.request.user
if scrobble.visibility == Visibility.PUBLIC:
return scrobble
if user.is_authenticated and scrobble.user == user:
return scrobble
raise Http404
def get_form_class(self):
return self.object.media_obj.logdata_cls().form()
@ -1028,7 +1158,11 @@ class ScrobbleDetailView(DetailView):
FormClass = self.get_form_class()
log = self.object.log or {}
log["notes"] = self.object.logdata.notes_as_str(separator="\n")
notes = log.get("notes")
if isinstance(notes, dict):
log["notes"] = notes
else:
log["notes"] = self.object.logdata.notes_as_str(separator="\n")
return FormClass(initial=log)
@ -1113,6 +1247,110 @@ class ScrobbleDetailView(DetailView):
except EmptyPage:
context["related_scrobbles"] = paginator.page(paginator.num_pages)
if self.request.user.is_authenticated:
fk_field = self.MEDIA_FK_MAP.get(media_type)
if fk_field and media_obj:
context["is_favorited"] = FavoriteMedia.objects.filter(
user=self.request.user, **{fk_field: media_obj}
).exists()
if media_type == "Track" and media_obj:
scrobbles = Scrobble.objects.filter(
track=media_obj, user=self.object.user
).order_by("-timestamp")[:20]
context["has_mopidy_uri"] = any(
(s.log or {}).get("raw_data", {}).get("mopidy_uri") for s in scrobbles
)
else:
context["has_mopidy_uri"] = False
return context
class ScrobbleShareView(TemplateView):
template_name = "scrobbles/scrobble_share.html"
def get_object(self):
sqid = self.kwargs.get("sqid")
decoded = decode_scrobble_share(sqid)
if not decoded or len(decoded) != 2:
raise Http404
scrobble_id, version = decoded
scrobble = get_object_or_404(Scrobble, id=scrobble_id)
if scrobble.share_token_version != version:
raise Http404
if scrobble.visibility not in (Visibility.PUBLIC, Visibility.SHARED):
raise Http404
Scrobble.objects.filter(id=scrobble.id).update(
share_view_count=F("share_view_count") + 1
)
scrobble.refresh_from_db(fields=["share_view_count"])
ShareViewLog.objects.create(
scrobble=scrobble,
ip_address=self.request.META.get("REMOTE_ADDR"),
user_agent=self.request.META.get("HTTP_USER_AGENT", "")[:500],
referrer=self.request.META.get("HTTP_REFERER", ""),
)
return scrobble
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
scrobble = self.get_object()
context["object"] = scrobble
context["log_form"] = None
context["related_scrobbles"] = Scrobble.objects.none()
context["has_mopidy_uri"] = False
if self.request.user.is_authenticated:
media_type = scrobble.media_type
fk_field = ScrobbleDetailView.MEDIA_FK_MAP.get(media_type)
media_obj = scrobble.media_obj
if fk_field and media_obj:
context["is_favorited"] = FavoriteMedia.objects.filter(
user=self.request.user, **{fk_field: media_obj}
).exists()
return context
class ScrobbleExploreView(ListView):
model = Scrobble
paginate_by = 100
template_name = "scrobbles/scrobble_explore.html"
queryset = Scrobble.objects.filter(visibility=Visibility.PUBLIC).order_by(
"-timestamp"
)
class RegenerateShareTokenView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
scrobble.regenerate_share_token()
return redirect(scrobble.get_absolute_url())
class ChangeVisibilityView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
visibility = request.POST.get("visibility")
if visibility not in (Visibility.PUBLIC, Visibility.SHARED, Visibility.PRIVATE):
return redirect(scrobble.get_absolute_url())
scrobble.visibility = visibility
scrobble.save(update_fields=["visibility"])
return redirect(scrobble.get_absolute_url())
class ScrobbleShareAnalyticsView(LoginRequiredMixin, DetailView):
model = Scrobble
slug_field = "uuid"
slug_url_kwarg = "uuid"
template_name = "scrobbles/scrobble_share_analytics.html"
def get_queryset(self):
return Scrobble.objects.filter(user=self.request.user)
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
scrobble = self.object
context["share_views"] = scrobble.share_views.order_by("-created")[:50]
return context
@ -1328,6 +1566,7 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
def _day_color(self, month_index, day_num, total_days):
import colorsys
hue = (month_index - 1) * 30 / 360
lightness = 0.80 + (day_num / total_days) * 0.15
r, g, b = colorsys.hls_to_rgb(hue, lightness, 0.5)
@ -1355,7 +1594,9 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
if media_type_filter and media_type_filter in self.CALENDAR_MEDIA_TYPES:
active_types = [media_type_filter]
else:
active_types = [t for t in self.CALENDAR_MEDIA_TYPES if t not in self.DEFAULT_EXCLUDE]
active_types = [
t for t in self.CALENDAR_MEDIA_TYPES if t not in self.DEFAULT_EXCLUDE
]
scrobbles = (
Scrobble.objects.filter(
@ -1364,20 +1605,35 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
timestamp__date__lte=month_end,
media_type__in=active_types,
)
.select_related("task", "birding_location", "food", "trail", "video_game", "book", "mood", "video", "board_game")
.select_related(
"task",
"birding_location",
"food",
"trail",
"video_game",
"book",
"mood",
"video",
"board_game",
)
.order_by("timestamp")
)
from django.db.models import Count
from django.db.models import Count, Q
from django.db.models.functions import TruncDate
total_by_day = dict(
Scrobble.objects.filter(
user=self.request.user,
timestamp__date__gte=month_start,
timestamp__date__lte=month_end,
)
.exclude(media_type="GeoLocation", title__isnull=True)
.annotate(local_date=TruncDate("timestamp", tzinfo=timezone.get_current_timezone()))
.exclude(Q(media_type="GeoLocation") & Q(geo_location__title__isnull=True))
.annotate(
local_date=TruncDate(
"timestamp", tzinfo=timezone.get_current_timezone()
)
)
.values("local_date")
.annotate(count=Count("id"))
.values_list("local_date", "count")
@ -1403,17 +1659,27 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
{
"uuid": scrobble.uuid,
"emoji": self.MEDIA_EMOJI.get(scrobble.media_type, "📌"),
"title": str(scrobble.media_obj) if scrobble.media_obj else scrobble.media_type,
"title": (
str(scrobble.media_obj)
if scrobble.media_obj
else scrobble.media_type
),
"media_type": scrobble.media_type,
}
)
calendar_days.append({
"day": day_num,
"scrobbles": day_scrobbles,
"total_count": total_by_day.get(datetime(year, month, day_num).date(), 0),
"is_today": year == today.year and month == today.month and day_num == today.day,
"color": self._day_color(month, day_num, total_days),
})
calendar_days.append(
{
"day": day_num,
"scrobbles": day_scrobbles,
"total_count": total_by_day.get(
datetime(year, month, day_num).date(), 0
),
"is_today": year == today.year
and month == today.month
and day_num == today.day,
"color": self._day_color(month, day_num, total_days),
}
)
ctx.update(
{
@ -1430,7 +1696,10 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
"day_names": ["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"],
"month_color": month_color,
"active_filter": media_type_filter or "",
"media_types": [{"name": mt, "emoji": self.MEDIA_EMOJI.get(mt, "📌")} for mt in self.CALENDAR_MEDIA_TYPES],
"media_types": [
{"name": mt, "emoji": self.MEDIA_EMOJI.get(mt, "📌")}
for mt in self.CALENDAR_MEDIA_TYPES
],
}
)
return ctx
@ -1441,7 +1710,7 @@ class ScrobbleSearchView(LoginRequiredMixin, TemplateView):
MEDIA_FIELDS = {
"Video": ["video__title", "video__overview"],
"Track": ["track__title", "track__artist__name", "track__album__name"],
"Track": ["track__title", "track__artists__name", "track__album__name"],
"PodcastEpisode": ["podcast_episode__title", None],
"Book": ["book__title", "book__summary"],
"Paper": ["paper__title", None],

View File

@ -59,19 +59,27 @@ class SportEventAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
"league",
"event_type",
"start",
"comp_str",
"round",
)
list_filter = ("round__season", "home_team", "away_team")
list_filter = ("league", "event_type")
ordering = ("-created",)
inlines = [
ScrobbleInline,
]
def comp_str(self, obj):
if obj.home_team:
return f"{obj.away_team} @ {obj.home_team}"
if obj.player_one:
return f"{obj.player_one} v {obj.player_two}"
teams = list(obj.teams.all())
if len(teams) >= 2:
return f"{teams[1]} v {teams[0]}"
players = list(obj.players.all())
if len(players) >= 2:
return f"{players[0]} v {players[1]}"
if len(players) == 1:
return str(players[0])
if len(teams) == 1:
return str(teams[0])

View File

@ -0,0 +1,36 @@
from django.core.management.base import BaseCommand
from sports.models import League, Team
from sports.thesportsdb import enrich_league_logo, enrich_team_logo, has_logo
class Command(BaseCommand):
help = "Fetch missing league and team logos from TheSportsDB"
def add_arguments(self, parser):
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be done without making changes",
)
def handle(self, *args, **options):
dry_run = options["dry_run"]
leagues = League.objects.filter(thesportsdb_id__isnull=False)
for league in leagues:
if has_logo(league):
continue
if dry_run:
self.stdout.write(f"Would enrich logo for league: {league.name} ({league.thesportsdb_id})")
else:
enrich_league_logo(league)
teams = Team.objects.filter(thesportsdb_id__isnull=False)
for team in teams:
if has_logo(team):
continue
if dry_run:
self.stdout.write(f"Would enrich logo for team: {team.name} ({team.thesportsdb_id})")
else:
enrich_team_logo(team)

View File

@ -0,0 +1,24 @@
# Generated by Django 4.2.30 on 2026-06-06 15:29
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("sports", "0018_alter_sportevent_genre"),
]
operations = [
migrations.AddField(
model_name="sportevent",
name="league",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="sports.league",
),
),
]

View File

@ -0,0 +1,111 @@
import json
from django.db import migrations, models
import django.db.models.deletion
def canonical_key(event):
if event.home_team_id and event.away_team_id:
return ("teams", event.league_id, event.home_team_id, event.away_team_id)
if event.player_one_id and event.player_two_id:
return ("players", event.league_id, event.player_one_id, event.player_two_id)
return ("title", event.league_id, event.event_type, (event.title or "").strip())
def build_logdata(event):
logdata = {}
if event.thesportsdb_id:
logdata["thesportsdb_id"] = event.thesportsdb_id
if event.start:
logdata["start"] = (
event.start.isoformat()
if hasattr(event.start, "isoformat")
else str(event.start)
)
if event.round:
logdata["round_name"] = event.round.name or str(event.round)
if event.round.season:
logdata["season_name"] = event.round.season.name or str(event.round.season)
return logdata
def merge_scrobble_logs(scrobble, logdata):
existing_log = scrobble.log or {}
if isinstance(existing_log, str):
existing_log = json.loads(existing_log)
existing_log.update(logdata)
scrobble.log = existing_log
scrobble.save(update_fields=["log"])
def populate_league(event):
if event.league:
return
if event.round and event.round.season and event.round.season.league:
event.league = event.round.season.league
event.save(update_fields=["league"])
def populate_m2m(event):
if event.home_team_id:
event.teams.add(event.home_team_id)
if event.away_team_id:
event.teams.add(event.away_team_id)
if event.player_one_id:
event.players.add(event.player_one_id)
if event.player_two_id:
event.players.add(event.player_two_id)
def migrate_sport_event_data(apps, schema_editor):
SportEvent = apps.get_model("sports", "SportEvent")
Scrobble = apps.get_model("scrobbles", "Scrobble")
db_alias = schema_editor.connection.alias
canonical_events = {}
for event in SportEvent.objects.using(db_alias).iterator():
populate_league(event)
key = canonical_key(event)
canonical = canonical_events.get(key)
if not canonical:
canonical_events[key] = event
populate_m2m(event)
logdata = build_logdata(event)
for scrobble in (
Scrobble.objects.using(db_alias).filter(sport_event=event).iterator()
):
merge_scrobble_logs(scrobble, logdata)
else:
logdata = build_logdata(event)
for scrobble in (
Scrobble.objects.using(db_alias).filter(sport_event=event).iterator()
):
scrobble.sport_event = canonical
merge_scrobble_logs(scrobble, logdata)
scrobble.save(update_fields=["sport_event", "log"])
event.delete()
class Migration(migrations.Migration):
dependencies = [
("sports", "0019_sportevent_league_alter_sportevent_away_team_and_more"),
]
operations = [
migrations.AddField(
model_name="sportevent",
name="teams",
field=models.ManyToManyField(blank=True, to="sports.team"),
),
migrations.AddField(
model_name="sportevent",
name="players",
field=models.ManyToManyField(blank=True, to="sports.player"),
),
migrations.RunPython(
migrate_sport_event_data, reverse_code=migrations.RunPython.noop
),
]

Some files were not shown because too many files have changed in this diff Show More