Compare commits

...

351 Commits
0.15.0 ... 19.0

Author SHA1 Message Date
a681b4d63b [notifications] Fix a few typos 2025-07-30 18:34:22 -04:00
c452ac24e0 [notifications] Send mood check-in 2025-07-30 18:30:18 -04:00
ae889bff7d [tasks] Fix bug in note str method 2025-07-30 17:50:59 -04:00
99dc86dc27 [moods] Fix mood list view 2025-07-30 16:05:48 -04:00
8eefcb8290 [tasks] Fix emacs metadata 2025-07-30 16:05:34 -04:00
ad0f9a54d0 [tasks] Fix dataclass models 2025-07-30 15:46:18 -04:00
1531b77b5c [tests] Fix metadata test 2025-07-30 13:59:11 -04:00
9437fdba60 [scrobbles] Fix log data parsing for tasks and boardgames
Add pagination to task and board game detail pages
2025-07-30 11:37:57 -04:00
a7551ef162 [music] Weird hack to get timezone for LFM scrobbles
Last.fm seems to send timestamps for scrobbles with a timezone of UTC
but the actual timezone is already localized. But that means we can't
extract the timezone we want, even though the timestamp is already in
the right timezone for storage.
2025-07-28 10:52:02 -04:00
c20204a6ea [music] Turns out lastfm already has our timeszone 2025-07-28 09:14:25 -04:00
685de842ea [views] Fix showing only a users scrobbles 2025-07-26 21:31:44 -04:00
7d13967708 [scrobbles] Fix admin filtering 2025-07-26 20:57:23 -04:00
109697a746 [project] Bump version 2025-07-26 10:19:34 -04:00
dde28f4aff [importers] Fix setting timezones before all imports 2025-07-26 10:18:43 -04:00
2f6ed3770f [books] Fix bad import after moving webdav to importers 2025-07-26 01:49:37 -04:00
e3d1cfb838 [books] Fix webdav importer 2025-07-25 23:40:39 -04:00
1821ac0d7b [project] Update tasks 2025-07-25 22:55:33 -04:00
4eb8289e55 [scrobbles] LastFM only creates import if there are imports 2025-07-25 22:53:31 -04:00
66e805542c [scrobbles] Add notification to board game imports 2025-07-25 21:28:08 -04:00
f91b127a2c [scrobbles] Allow skipping checks for existing scrobbles 2025-07-25 17:36:19 -04:00
b2077678e2 [music] Fix timezones on lastfm imports 2025-07-25 17:35:56 -04:00
5427198185 [profiles] Just black 2025-07-25 17:35:10 -04:00
2bdba14cd6 [boardgames] Fix import from imap for timezones 2025-07-25 17:34:57 -04:00
95d8c4e4d6 [profiles] Clean up timezone stuff 2025-07-25 17:33:59 -04:00
6ab7745151 [music] Use found name to look it up 2025-07-25 10:50:00 -04:00
8b062a6c1d [music] Add tracks migration 2025-07-25 10:44:06 -04:00
cd48e7a402 [boardgames] Fix migration path 2025-07-25 10:40:53 -04:00
22830b0cea [profiles] Fix extra newline in one off 2025-07-25 10:24:33 -04:00
fd36034f6d [templates] Fix album use and local_timestamp 2025-07-25 10:21:20 -04:00
edf9fbd9c1 [music] Reorganize importer and fix lookups 2025-07-25 10:20:49 -04:00
e8e989bb63 [music] Add albums to tracks and utility to condense tracks 2025-07-20 22:00:06 -04:00
69401d11c8 [importers] Reorganize importers a little 2025-07-20 16:43:50 -04:00
759caef45d [books] Move one off creator to profile utils 2025-07-20 16:31:20 -04:00
9514861b32 [tests] Skip failing tests 2025-07-19 21:09:51 -04:00
aa644aa9cf [project] Start adding features and update todos 2025-07-19 01:56:23 -04:00
94820b1d9c [scrobbles] Exclude geolocs from stop notifications 2025-07-19 01:56:03 -04:00
4db8793d5c [books] Allow timezone changes when importing from KOReader
Turns out you need a city-based timezone for DST stuff to work properly.
The US/Eastern timezone doesn't mess with DST because it can be so wonky
in different regions. So while we fix timezone defaulting to a
DST-friendly timezone too.
2025-07-19 01:54:27 -04:00
7c6e895ae4 [books] Start cleaning up get_from_google method 2025-07-09 13:46:06 -04:00
b1b67528bf [music] Fix find_or_create for tracks 2025-07-09 13:39:10 -04:00
dd54a33159 [profiles] Remove paswords from admin 2025-07-06 11:26:38 -04:00
92c4f91e5a [boardgames] Add check for learning plays from BG stats 2025-07-06 10:02:14 -04:00
838b19e996 [boardgames] Remove expansion_ids key if not needed 2025-07-06 00:01:01 -04:00
3808277025 [boardgames] Add comments and bgstats_id to scrobbles 2025-07-05 23:55:00 -04:00
f64863f2bc [scrobblers] Connect designers to board games 2025-07-03 22:02:27 -04:00
2c199c0e93 [boardgames] Check if scrobble exists first 2025-07-03 20:19:30 -04:00
4924ef316f [scrobbles] Add a check for Garmin emails 2025-07-03 16:26:29 -04:00
64cb17e91f [scrobbles] Add management command for imap 2025-07-03 14:59:33 -04:00
1fd325823b [boardgames] Clean up email parser to work with many plays 2025-07-03 00:34:51 -04:00
1590ce5f18 [boardgames] Adding email scrobbler for BG Stats 2025-07-02 23:01:13 -04:00
3548c29f97 [people] Add a more general people app 2025-07-02 11:00:06 -04:00
0fa831fa42 [boardgames] Start adding email scrobbling for board games 2025-07-02 10:55:24 -04:00
a2f64a98c3 [project] Update tasks 2025-07-02 10:29:13 -04:00
872ca17432 [tasks] Hackety hack 2025-07-02 09:29:15 -04:00
224c165d72 [tasks] Fix bad matching in label titles 2025-06-27 11:54:14 -04:00
bf7d2514f2 [tasks] Clean up param names and loggin 2025-06-27 11:50:30 -04:00
4e37bc5ab9 [tasks] Fix bug in looking up user profile 2025-06-27 11:42:37 -04:00
125da84f4e [tasks] Fix emacs scrobbling of tasks 2025-06-27 11:32:09 -04:00
36ceb4c7fe [release] 17.1 2025-06-27 10:52:50 -04:00
88a3831975 [tasks] Make tasks use user profile string 2025-06-27 10:48:05 -04:00
63361964ca [tasks] Add optional user context labels to get title 2025-06-27 10:37:09 -04:00
40b54b27f4 [tasks] Actually add the new utils file 2025-06-26 14:50:11 -04:00
a7eca4b9a7 [tasks] Actually get title consistently 2025-06-26 14:09:17 -04:00
d152412e99 [books] Add optional details key to log data for books 2025-06-25 10:21:26 -04:00
3ba6c6b6e4 [project] Add new bug in tasks
Though honestly, it will probably be a config setting in the Emacs hook,
but we'll see. Actually, maybe the emacs hook code should end up in this
repo somewhere 🤔
2025-06-24 14:25:20 -04:00
bffbf47c2f [release] 17.0 2025-06-24 10:56:14 -04:00
f4e81da533 [project] Completing task for label bug 2025-06-24 10:55:01 -04:00
4b7f5459be [project] Add new task for TV series bug 2025-06-24 10:53:57 -04:00
c68b0e9d7e [tasks] Fix bug in emacs labels 2025-06-24 10:52:47 -04:00
32ec65116b [tasks] Move to single context for task titles 2025-06-22 21:26:20 -04:00
da8d26fcd9 [trails] Add alltrails and gaiagps ids to model 2025-06-22 20:26:37 -04:00
d33954e494 [project] Tidy up todos 2025-06-17 10:53:26 -04:00
1b306d6493 [project] Move todos.org file to PROJECT.org 2025-06-17 10:50:30 -04:00
c881143e1b [todos] Update backlog with tasks from Todoist 2025-06-17 10:18:48 -04:00
141700fcb3 [music] Fixes RYM slug including commas 2025-06-16 09:38:19 -04:00
7357b5bfec [todos] Update todo formatting 2025-06-16 09:35:30 -04:00
99cabd0007 [todos] Finish another task! 2025-06-13 12:35:24 -04:00
cf77e12cc3 [tasks] Add description of task to the template 2025-06-13 12:34:55 -04:00
3f2cbbb34a [videos] Fix TMDB lookup for movies 2025-06-13 12:10:43 -04:00
650ecf12c6 Merge branch 'main' into develop 2025-06-13 11:55:04 -04:00
d7cc009d07 [ci] Make deploys on happen on tags 2025-06-13 11:52:14 -04:00
a872cf3611 [templates] Use TMDB rating ID if available 2025-06-13 11:47:42 -04:00
1f9713312b [videos] Fix lookup for mevie posters 2025-06-13 11:47:04 -04:00
159e555d7c [templates] Remove global counts from home page 2025-06-13 11:33:21 -04:00
981f4f9c9a [videos] Quick fix 2025-06-13 11:23:37 -04:00
ddd5ce1392 Merge branch 'develop' 2025-06-13 11:21:07 -04:00
7a75b31b56 [todos] Update task for metadata fixing 2025-06-13 11:20:34 -04:00
24ac545f55 [videos] Switch to TMDB for scraping videos 2025-06-13 11:19:15 -04:00
d5da8ae701 [todos] Update priority task 2025-06-13 09:43:08 -04:00
53a04d064d [videos] At least stop IMDB from breaking 2025-06-13 09:15:48 -04:00
c0871a3b9e [scrobbles] Fix failing tests 2025-06-12 09:57:32 -04:00
d917dd8b2c [tasks] Fix source checking for emacs/orgmode tasks when updating 2025-06-12 09:54:16 -04:00
6fc8084f2d [scrobbles] Fix log of media type 2025-06-11 12:01:02 -04:00
41d6fe8ff6 [todos] Add new task 2025-06-10 11:27:37 -04:00
73838312cd [tasks] Clean up emacs webhook flow 2025-06-10 11:06:47 -04:00
eb2dd4c839 [todos] Clean up todos and add new milestone 17 2025-06-10 10:05:40 -04:00
1a0c4f69f0 Merge branch 'develop' 2025-06-09 11:26:25 -04:00
356e579558 [brickset] Little fix to scrobble of bricksets 2025-06-09 10:40:12 -04:00
e980e3c5c9 [brickset] Fix manual scrobbling and admin 2025-06-09 10:36:13 -04:00
8773542099 [tasks] Fix source being orgmode not emacs 2025-06-08 03:05:43 -04:00
70378c9968 [tasks] Fix updating and stopping tasks from Emacs 2025-06-08 02:40:07 -04:00
c871087496 [tasks] First try at adding an emacs webhook 2025-06-08 01:09:03 -04:00
059d7780a0 [templates] Add counts and durations to new tables 2025-06-07 20:02:21 -04:00
db36329011 Merge branch 'develop' 2025-06-07 18:11:08 -04:00
69aa80e6c1 [templates] Allow going back and forward in time 2025-06-07 15:17:56 -04:00
99a6e5107b [scrobbles] Add orgparse and start of emacs webhook 2025-06-06 17:25:04 -04:00
39e2fdce27 [music] Fix allmusic lookup 2025-06-06 17:24:41 -04:00
b2f98d780b [middleware] Add auto timezone setting 2025-06-06 09:49:53 -04:00
07b5dc6a2c [templates] Clean up dashboard 2025-06-06 09:49:08 -04:00
7207ca385e [scrobbles] Add two new methods to Scrobble model 2025-06-06 09:48:15 -04:00
2a254e28d0 [templates] Start redesigning the dashboard 2025-05-18 23:20:47 -04:00
d4377e49ac [music] Fix getting album metadata 2025-05-18 23:19:17 -04:00
9dc0a818ff [templates] Fix chart loading 2025-05-18 23:19:05 -04:00
5e672fc9ed [webpages] Add title to webpage reading 2025-05-18 09:37:53 -04:00
6d194d227e Merge branch 'develop' 2025-05-11 01:43:55 -04:00
ed217cbad2 [beers] Blacken and clean up imports 2025-05-11 01:42:30 -04:00
3cca30dc70 [puzzles] Add puzzle lookup via IPDB 2025-05-11 01:41:43 -04:00
d3a15d5e7b Merge branch 'develop' 2025-05-11 00:03:48 -04:00
deeaa0af4b [puzzles] Fix misnamed ipdb field 2025-05-10 23:58:44 -04:00
159459e1b9 [puzzles] Add puzzle model and hooks 2025-05-10 23:48:40 -04:00
89b6d8de06 [importers] Fixes TSV comma missing 2025-05-10 23:48:40 -04:00
3c725de2ac [webpages] Fixes 500 errors when webpage lookup fails 2025-05-10 23:48:40 -04:00
dd71bdd38c [puzzles] Add puzzle model and hooks 2025-05-10 23:36:03 -04:00
d066a98282 [importers] Fixes TSV comma missing 2025-04-15 16:32:14 -04:00
73bc4a1cd1 [webpages] Fixes 500 errors when webpage lookup fails 2025-04-15 16:18:26 -04:00
79d58e6390 Merge branch 'develop' 2025-04-07 17:45:16 -04:00
69e9caf477 [release] Bump version to 0.16.1 2025-04-07 17:43:36 -04:00
776e839ca4 [ci] Fixes wrong public key 2025-04-07 17:42:31 -04:00
97792898df [release] Bump version to 0.16.0 2025-04-07 17:34:00 -04:00
9986163d20 [notifications] Fixes missing title 2025-04-07 17:34:00 -04:00
d1229585a1 [release] Bump version to 0.16.0 2025-04-07 17:31:50 -04:00
b8c5c3f3e9 Merge branch 'main' into develop 2025-04-07 16:57:54 -04:00
6d9e237f9b [notifications] Fixes missing title 2025-04-07 16:53:30 -04:00
e65a2d300d [ci] Fixes reference to wrong ssh key 2025-04-07 14:38:46 -04:00
f45541de6d [deps] Fixes premature bump to py312
Turns out our deployment uses python 3.11, so we need to roll back
the pillow and pendulum upgrades so they work with python 3.11 ... how
irritating to have all this version bump nonesense.
2025-04-07 14:27:28 -04:00
391f0cc335 [tests] Trying to parallel tests 2025-04-07 13:55:42 -04:00
4b5281bdd8 [cleaning] Adding migrations that are past due 2025-04-07 13:47:21 -04:00
484be0a64e [tests] Skip openlibrary lookup tests 2025-04-07 13:45:13 -04:00
257e6899d3 [tests] Clean up podcast tests 2025-04-07 13:40:32 -04:00
4767cc7e52 [podcasts] Fixes enrichment of podcasts with podcastindex 2025-04-07 13:32:03 -04:00
bcc3f46806 [tests] We dont differentiate on albums anymore 2025-04-07 12:39:57 -04:00
6c461ed55f [tests] Simplify view tests to not mess with time 2025-04-07 12:30:54 -04:00
28cf57c6dd [tests] Try fixing CI breaking with small seconds 2025-04-07 09:56:10 -04:00
0bb874f1db [scrobblers] Fixes test with wiggly second 2025-04-07 00:12:29 -04:00
27f50baf5d [music] Fixes missing jellyfin ts converter 2025-04-06 23:39:32 -04:00
499c3d6859 [music] Fix bug in scobblers for tracks 2025-04-06 22:40:33 -04:00
b0e9f13e11 [music] Attempts to fix bad lookups from LastFM and Jellyfin
Broader issue was creating tracks without albums that were duplicates of
existing tracks because sometimes Jellyfin and LastFM do not have albums
sent with them.
2025-04-06 22:28:32 -04:00
b2ee79b3ea [project] Update TODOs file 2025-04-04 11:00:28 -04:00
3ddd3b1684 [deps] Upgrades pillow to work with Python 3.11 2025-04-04 10:23:41 -04:00
4f8a359ab9 [deps] Upgrades pendulum to work with python 3.13 2025-04-02 22:14:24 -04:00
48114aee5e [scrobblers] Fix missing r in regex string 2025-04-02 22:14:11 -04:00
0cc87a2dbe [profiles] Add user profile context override 2025-04-02 22:13:43 -04:00
23d3e19db9 [scrobbles] Fixes typo in transaction import 2025-03-29 13:56:20 -04:00
29f5e2b940 [scrobbles] Add action parsing to scrobble url parsing
This also elaborates on the web scrobbler endpoint, though it does not
actually work at this time. But I'm tired of carrying this around in
stashes, so we'll push it and fix it later.
2025-03-29 13:52:46 -04:00
3208a32ffe [scrobbles] Fixes dedup utility, adding a transaction
Deleting tracks needs to be in a transaction so we don't try to delete a
track that may still have scrobbles associated with it.
2025-03-29 13:49:54 -04:00
99da9b62bf [music] Fixes bug where lastfm created new tracks
The issue here was that if we couldn't find a track from a musicbrainz
ID (which was almost never), we would fall back to just creating a new
track with a blank album. So we'd spam the track table with identical
tracks just no albums.

Now we do a quick check for the track where the title and artist match
and use that if it's found. This will result in some tracks being
associated with the wrong album, but I think that's better than the
current behavior.
2025-03-29 13:47:45 -04:00
36048d9a0a [music] Cleans up last.fm import logging
We weren't tracking import info very well in the original flow, this
should provide better insight into what happened after a run.
2025-03-29 13:44:48 -04:00
16091c9053 [tests] Skips the podcast scrobble tests, no Google Podcast 2025-03-27 16:50:42 -04:00
eec00ce658 [urls] Removes accidentaly committed URL updates
The modern_ui app is not ready yet, still sitting on a dev machine.

These url paths should not have been committed, so we'll just comment
them out for now.
2025-03-27 16:38:34 -04:00
e02010e409 [videos] Fix skip for youtube tests 2025-03-27 16:35:23 -04:00
498712e531 [drone] Update drone to install with test, rather than dev 2025-03-27 16:32:33 -04:00
14e4432495 [poetry] Update pyproject to only install test deps when needed
We'll also need to fast follow this with a new line in the drone
file otherwise CI will fail because pytest wont be installed by default
any longer.
2025-03-27 16:31:44 -04:00
676c40176c [notifications] Add a class for notifications 2025-03-25 09:50:14 -04:00
0a50bca622 [notifications] Add stop notification utility 2025-03-24 13:23:33 -04:00
ac9fc315b1 [books] Fix date lookup and literal string title search 2025-03-19 01:02:39 -04:00
50d1a4a2bd [videos] Fix youtube videos longer than an hour 2025-03-19 00:17:21 -04:00
444562235f [webpages] Handle title truncation better 2025-03-07 14:36:35 -05:00
760575e41d [webpages] Bail if we can't get text 2025-03-07 14:33:02 -05:00
42699f84d2 [tasks] Finish task based on timestamp, not now 2025-03-07 09:02:29 -05:00
b660e47bc2 [videos] Just a little type hinting 2025-02-25 21:09:02 -05:00
06b4ba8bcc [foods] Fix image bug in foods 2025-02-25 21:08:34 -05:00
1f67207f81 [books] Fix two bugs in looking up books 2025-02-25 21:07:45 -05:00
baa8dbee46 [boardgames] Modularize the Lichess imports 2025-02-25 11:09:29 -05:00
3f9cdcac65 [books] Clean up old page model and bug in page calc 2025-02-25 10:34:08 -05:00
71874510a4 [profiles] Actually make the form work 2025-02-25 01:17:12 -05:00
8c600d6b4b [books] Fix no result for detail lookup 2025-02-23 23:08:47 -05:00
e95b6f50dc [tests] Fix views and comment out youtube 2025-02-23 22:49:44 -05:00
93c16d80ec [profiles] Add settings form 2025-02-23 22:49:18 -05:00
b03da9ab37 [middleware] Add health check middleware 2025-02-21 10:40:44 -05:00
09ca05cb4f [make] Move deploys back to homelab 2025-02-20 23:27:47 -05:00
d6e02d241c [settings] Fix test settings 2025-02-20 23:27:31 -05:00
8dd94e2fc4 [books] Fix book admin 2025-02-20 23:27:22 -05:00
9e3f714c61 [books] Add papers as a data model 2025-02-20 23:07:56 -05:00
e2b0decd83 [utils] Add commit option to the dedupper 2025-02-17 17:36:43 -05:00
e08db5e3ad [github] Change when actions run 2025-02-10 09:33:38 -05:00
1460b9ba77 Fix py version for github actions 2025-02-10 09:32:07 -05:00
a41e0ffa5d Create django.yml for Github Actions 2025-02-10 09:30:34 -05:00
c51c75b6a6 [books] Handle authors from Google 2025-02-09 22:44:11 -05:00
041435bc93 [books] Use google and title to get book 2025-02-09 21:58:03 -05:00
15f27b73a5 [videos] Fix error where we add tt to the IMDB id 2025-02-03 00:32:22 -05:00
25900a9911 [boardgames] Send notifications on chess imports 2025-01-29 00:23:46 -05:00
4277e355e0 [boardgames] Few more little tweaks to imports 2025-01-29 00:19:40 -05:00
855f59b83f [boardgames] Fix small bug in black username 2025-01-29 00:08:55 -05:00
9c115c0b65 [boardgames] Add lichess importing 2025-01-29 00:04:49 -05:00
fd726a125f [books] Remove meta_yt references 2025-01-28 22:20:04 -05:00
6685669b29 [videos] Fix missing defaults for no tags 2025-01-27 14:58:47 -05:00
3fa43f02d0 [tasks] Add exercise and lifeevent to constants 2025-01-27 14:52:15 -05:00
66c34942e6 [videos] Youtube using the actual API 2025-01-27 14:51:56 -05:00
ef1fcd4026 [videos] Fix video type reference 2025-01-27 01:37:21 -05:00
a36efa3b1d [videos] Add check if youtube fails 2025-01-27 01:30:41 -05:00
300a2ae6aa [scrobblers] Add ability to stop via bookmarklet 2025-01-27 01:14:20 -05:00
1d5b91b6e2 [videos] Fix splitting youtube IDs 2025-01-27 00:41:03 -05:00
a5c67a7fe1 [videos] Clean up video tools 2025-01-26 23:41:41 -05:00
b16d0b1864 [vrobbler] Update poetry deps 2025-01-26 23:39:18 -05:00
f90a3b84a8 [books] Add google as a source and clean up data model 2025-01-26 23:39:02 -05:00
25a14ed9e7 [videos] Clean imdb getter up and fix series 2025-01-21 23:20:30 -05:00
84790c805c [migrations] Add default value to run time seconds 2025-01-21 23:20:09 -05:00
a9499f0463 [video] Fix youtube redirects 2025-01-21 22:49:15 -05:00
c109ed79eb [video] Update how we get video metadata for YT add 2025-01-20 09:13:18 -05:00
89e5455b29 [deps] Add meta-yt and updates 2024-11-25 13:55:30 -05:00
647762f201 [beers] Fix missing url path 2024-11-24 17:26:42 -05:00
b788cc65d1 [notifications] Fix finish url being on media 2024-11-24 09:59:18 -05:00
50ec82213c [foods] Fix migrations 2024-11-24 09:45:56 -05:00
51c1acd677 [notifications] Add click to finish 2024-11-24 09:39:44 -05:00
fd38046113 [foods] Add Food scrobbling 2024-11-20 14:16:44 -05:00
d294f2ecd1 [koreader] Fix list access bug 2024-11-19 20:46:25 -05:00
3c7940c6c6 [lastfm] Fix importing tracks 2024-11-18 15:12:20 -05:00
56c000154c [scrobbles] Simplify notifications and add to books 2024-11-18 14:40:05 -05:00
8157836b42 [lastfm] Fix issue with no media obj 2024-11-18 14:19:41 -05:00
af0e76b29c [utils] Add crud deduplicator 2024-11-17 22:02:19 -05:00
39004aac0c [make] Restart celery too 2024-11-17 21:01:43 -05:00
8cbd746681 [tasks] Fix import in utils 2024-11-17 21:00:27 -05:00
dee04a47cb [config] No need to load pypi pass everytime 2024-11-17 20:49:14 -05:00
1304a27408 [books] Add webdav koreader importer 2024-11-17 20:49:14 -05:00
2327b1f622 [notifications] Clean up emojis and fix priority 2024-11-08 10:51:44 -05:00
94fed8ae38 [tasks] Add research as a tag 2024-11-08 08:59:17 -05:00
042a26f148 [make] Add migrations to deploy 2024-11-06 14:50:15 -05:00
e606f0de01 [trails] Add trail and default activity type 2024-11-06 14:48:56 -05:00
ed2253cb6b [videos] Clean up admin stuff 2024-11-06 11:07:27 -05:00
9a1508b7a6 [videos] Add channel admin access 2024-11-06 11:03:16 -05:00
39f3a31847 [videos] Update youtubes stuff 2024-11-06 10:50:39 -05:00
f0b32961c1 [scrobbles] Can't just use subtitle for notifications 2024-11-06 09:00:21 -05:00
a08574b359 [videos] Add subtitle to notifications 2024-11-06 00:18:16 -05:00
26ebb4108d [videos] Try to fix auth on webhooks 2024-11-05 23:54:21 -05:00
1b95706f70 [videos] Add youtube scrobbling 2024-11-05 23:32:03 -05:00
b65ebbe397 [books] Fix koreader import with nul strings 2024-11-05 21:13:37 -05:00
0679af3029 [scrobbles] Fix if log does not exist 2024-11-05 18:47:34 -05:00
8888b42adf [scrobbles] Actually use notify str 2024-11-05 16:50:35 -05:00
b8d68739cf [scrobbles] Kidding, we use description 2024-11-05 16:46:14 -05:00
4c2b838d7b [scrobbles] Add details if we have them 2024-11-05 16:43:40 -05:00
cdb3c29844 [music] Fix timestamp setting in lastfm importer 2024-11-05 16:41:12 -05:00
22c07bdb82 [music] Fix lastfm importing 2024-11-05 16:33:28 -05:00
66513c5758 [scrobbles] Memos 2024-11-05 16:28:58 -05:00
f3c0d20268 [scrobbles] Fix ntfy formatting 2024-11-05 16:27:16 -05:00
622a30899f [music] Fix how we create tracks from LastFM 2024-11-05 14:47:30 -05:00
2c1e8c08ae [scrobbles] Add ntfy config to user profiles 2024-11-05 14:38:01 -05:00
cc52e00d15 [videos] Fix utils 2024-11-04 20:40:49 -05:00
e762658082 [deps] Update howlongtobeatpy 2024-11-04 15:30:50 -05:00
5111cee14b [videogames] Fix minor metadata look up bug 2024-11-04 15:30:00 -05:00
68a6d58339 [tsv] Try fixing our lookup when titla and MB id are none 2024-11-04 10:18:52 -05:00
b91c8b27d7 [tsv] Fix lookup to use new dict method 2024-11-04 09:55:38 -05:00
3a91aa5903 [tsv] Fix missing comma error 2024-11-04 09:45:13 -05:00
bfd6331be3 [vidoes] default run time seconds to 1800 2024-11-03 11:31:13 -05:00
38ba474c1f [beers] Fix abv lookup 2024-11-02 18:59:11 -04:00
76272b7e39 [beers] Fix rating error 2024-11-01 22:42:42 -04:00
1f0c950b17 [beers] Fix missing abv error 2024-11-01 22:39:46 -04:00
f2998205e1 [tasks] Remove data class until we fix it 2024-11-01 16:03:20 -04:00
38e108c1ae [tasks] Trying to fix logdata 2024-11-01 15:58:16 -04:00
dec100f8ff [make] Add basic makefile 2024-11-01 15:55:21 -04:00
e89eb332d3 [task] Try to fix logdata for tasks 2024-11-01 15:55:07 -04:00
cb5b279300 [templates] Fix missing logdata model 2024-11-01 15:42:47 -04:00
59d681dc00 [task] Ooops, add task id 2024-11-01 13:10:47 -04:00
82dcad569a [task] Log when we don't find a task for notes 2024-11-01 12:48:59 -04:00
d8aaf3bf55 [task] Little bug 2024-11-01 12:43:18 -04:00
29b92d89b2 [task] Try to fix scrobbling 2024-11-01 12:35:11 -04:00
86bcdef13d [task] Fix small bug 2024-11-01 12:12:30 -04:00
20e6ae7421 [tasks] Update checking for inprogress changed 2024-11-01 12:08:06 -04:00
4ed5117900 [tasks] Stop ignoring notes 2024-11-01 11:16:24 -04:00
3388471685 [tasks] Don't stop tasks when modified while in progress 2024-10-31 15:28:06 -04:00
58a957c98a [tasks] Add ability to save comments as notes on tasks 2024-10-25 15:10:35 -04:00
2ad626cd59 [beers] Add subtitles 2024-10-23 12:59:18 -04:00
6ce0257dc0 [beers] Cleaning up 2024-10-23 12:57:45 -04:00
43dbd3b28b [beers] Fix description tag typo 2024-10-23 12:55:37 -04:00
7671644e87 [beers] Allow scrobbling from URLS 2024-10-23 12:48:55 -04:00
f555a49746 [beers] Fix untappd and beeradvocate urls 2024-10-23 10:58:07 -04:00
9fe474978a [beers] Add manual scrobbling from URLs 2024-10-23 10:51:35 -04:00
9f8465d364 [beers] Fix a few more display stuffs 2024-10-22 17:58:39 -04:00
ddfddc33f5 [beers] Actually give producers names 2024-10-22 17:52:18 -04:00
0ec7ed3a18 [beers] Finish out the model 2024-10-22 17:47:38 -04:00
0bda3f6fd8 [beers] Fix error in cover reference, add adminf or beer producers 2024-10-22 17:44:46 -04:00
59765b14ca [beers] Connect beers with producers 2024-10-22 17:34:20 -04:00
08b48371bc [beers] Add beer scrobbling 2024-10-22 17:26:55 -04:00
d3d5b088cd [webdav] Add basic client 2024-10-20 16:22:56 -04:00
083e931a78 [webdav] Add crednetials to user profile 2024-10-20 16:06:24 -04:00
0f0fb7cceb [scrobbles] Add lookup of last serial scrobble 2024-10-20 14:35:42 -04:00
f2bbb7f5d0 [scrobblers] Use udpated_at key for Todoist scrobbles 2024-10-20 14:29:17 -04:00
218c68dee0 [books] Fix importing scrobbles from existing books 2024-10-18 14:50:46 -04:00
202cf24722 [books] Clean up koreader imports and data storage 2024-10-18 12:49:59 -04:00
1ddacd4454 [scrobbles] Improve webhook logging 2024-10-17 22:51:10 -04:00
2dbb091d61 [books] Fix importing to not use OL by default for now 2024-10-17 16:40:07 -04:00
dbaf189628 [scrobbles] Fix scrobbling duplicate tasks 2024-10-15 21:29:50 -04:00
4a0bac5b87 [scrobbles] Fix double scrobbling tasks 2024-10-15 19:34:27 -04:00
59b7e3dada [scrobbles] Add description to current task 2024-10-15 15:29:31 -04:00
24223ebe13 [scrobbles] Add working to status page 2024-10-15 15:27:48 -04:00
de6e9ce2d6 [tasks] We only care about in-progress for scrobbling 2024-10-15 15:23:15 -04:00
470eb0778a [tasks] Get last task, not first 2024-10-15 15:14:37 -04:00
f075492554 [tasks] Add Bug as suffix type 2024-10-15 15:12:58 -04:00
8fb2fac47f [tasks] Remove in-progress tag to stop 2024-10-15 15:11:03 -04:00
b1eac1454b [tasks] Don't create new scrobbles for in progress tasks 2024-10-15 15:07:43 -04:00
84737e0c3b [tasks] Add habit to suffix type 2024-10-15 15:02:01 -04:00
c4359a2331 [tasks] Fix catch for no scrobble available 2024-10-15 14:58:34 -04:00
8fa538dbee [tasks] Fix use of source_id 2024-10-15 14:56:26 -04:00
26d82518fa [tasks] Temporarily don't use updated_at 2024-10-15 14:52:58 -04:00
04a7ba51e4 [tasks] Add user id to Profile 2024-10-15 14:49:43 -04:00
ccf14c51bf [tasks] Fix webhook creating duplicates 2024-10-15 14:41:40 -04:00
b97aa8936e [scrobbles] Fix if past seconds is zero 2024-10-15 14:40:05 -04:00
98924e362e [tasks] Fix loading todoist data from webhook data 2024-10-15 14:37:15 -04:00
0384f72cbd [tasks] Fix webhook for Todoist 2024-10-15 14:25:15 -04:00
0c8a486b6a [tasks] Implement todoist webhooks #8479861260 2024-10-15 14:21:30 -04:00
7954765b73 [tasks] Add oauth flow for Todoist 2024-10-14 22:15:29 -04:00
7604327ca9 [oauth] Adding oauth pattern 2024-10-14 18:13:27 -04:00
20542ac7e9 [scrobbles] Fix tasks overwriting tasks 2024-10-11 18:34:13 -04:00
34137af815 [videos] Fix series lookup from IDMB too 2024-10-09 21:08:04 -04:00
0f2570e51b [videos] Fix imdb dict lookups 2024-10-09 20:31:43 -04:00
ed917e16fc [scrobbles] Fix when scrobble has not dict 2024-10-07 16:50:54 -04:00
164510b7b7 [tasks] Add media url link 2024-10-05 17:59:32 -04:00
6764023016 [tasks] Fix capitalization call 2024-10-05 16:07:19 -04:00
7c6c1cee6d [tasks] Fix scrobbling tasks 2024-10-05 15:49:22 -04:00
c251c5f413 [tasks] Allow scrobbling tasks from URLs 2024-10-05 15:08:38 -04:00
342e86d7fb [videos] Don't lookup video when we know what it is 2024-10-05 14:33:25 -04:00
176b698f6e [templates] Clean up tables 2024-10-02 17:51:02 -04:00
ddf2ca5630 [scrobbling] Actually fix typo in mopidy uri 2024-10-02 17:51:00 -04:00
d52061f6d8 [scrobbling] Actually fix typo in mopidy uri 2024-10-02 15:40:16 -04:00
3c0a75755b [tasks] Fix source url generation 2024-09-30 17:10:28 -04:00
183469ebe5 [tasks] Add tasks app 2024-09-30 15:05:07 -04:00
bbe8149e6c [scrobbles] Fix scrobbling from IMDB urls 2024-09-26 22:12:58 -04:00
f876caabe1 [scrobblers] Add mopidy source to scrobbler 2024-09-26 22:07:01 -04:00
87c078f47d [scrobbles] Allow scrobbling any content via URLs 2024-09-26 21:58:40 -04:00
5a9292e10a [videos] Fix looking up on IMDB 2024-09-26 21:58:39 -04:00
a5630022f5 [status] Add pocast playing to listening status 2024-09-16 12:42:39 -04:00
babc2aeb9d [boardgames] Fix missing default None for serial scrobble 2024-09-14 09:34:07 -04:00
875b0f98a0 [scrobblers] Fix manual vidoe lookup and simplify Sports 2024-09-12 20:16:05 -04:00
5d1edc71d7 [scrobbles] Add gpx file for trails 2024-09-11 13:31:48 -04:00
e4738e464f [trails] Finish hooking things up for trails 2024-09-09 17:20:10 -04:00
85c4963619 [templates] Fix trail template loading 2024-09-09 17:07:13 -04:00
8d6707db95 [templates] Clean up redundent templates and fix main menu 2024-09-09 16:58:33 -04:00
0df3dd728d [release] Bump to version 0.15.4 2024-09-09 13:05:17 -04:00
1fe8d8aa51 [boardgames] Add better list template and fix URLs (also for moods) 2024-09-09 12:56:42 -04:00
8f3c7beffa [trails] Make trail admin raw ID field 2024-09-09 12:12:39 -04:00
aac0efbb14 [misc] Cleaning up imports and urls for webpages 2024-09-09 12:12:08 -04:00
b0d4dd0899 [trails] Add new trails app 2024-09-09 12:11:39 -04:00
16db67ea84 [bgg] Add rudimentary bgg push scrobble script 2024-09-09 11:56:42 -04:00
7a747268a1 [lifevents] Remove redundant completion percentage 2024-09-09 11:56:20 -04:00
2037ffc67a [deps] Add webdav client library and fix boto3 2024-09-09 11:55:58 -04:00
2136c1562a [books] Fix calculation of current page 2024-09-08 14:13:40 -04:00
eae169aff7 [version] Bump to version 0.15.3 2024-09-08 00:53:19 -04:00
6c0bd2e409 [scrobbles] Clean up references to book_page_data 2024-09-08 00:52:50 -04:00
d69d7311d5 [version] Bump to 0.15.2 2024-09-08 00:45:36 -04:00
c484dab210 [books] Fix importing order of page data 2024-09-08 00:44:55 -04:00
3d62a6a227 [videos] Fix error looking up TV shows 2024-09-07 23:07:39 -04:00
1a5d4a6717 [scrobbles] Clean up dataclass inheritence 2024-09-07 11:36:36 -04:00
8f0491a90b [pyproject] Upgrade version to 0.15.0 2024-09-07 11:04:01 -04:00
262 changed files with 17567 additions and 4245 deletions

View File

@ -14,9 +14,9 @@ steps:
# Install dependencies
- cp vrobbler.conf.test vrobbler.conf
- pip install poetry
- poetry install --with dev
- poetry install --with test
# Start with a fresh database (which is already running as a service from Drone)
- poetry run pytest --cov-report term:skip-covered --cov=vrobbler tests
- poetry run pytest -n 5 --cov-report term:skip-covered --cov=vrobbler tests
environment:
VROBBLER_DATABASE_URL: sqlite:///test.db
volumes:
@ -39,8 +39,8 @@ steps:
- vrobbler collectstatic --noinput
- immortalctl restart celery && immortalctl restart vrobbler
when:
branch:
- main
ref:
- refs/tags/*
- name: build success notification
image: parrazam/drone-ntfy:0.3-linux-amd64
when:

31
.github/workflows/django.yml vendored Normal file
View File

@ -0,0 +1,31 @@
name: Django CI
on:
push:
branches: [ "develop" ]
pull_request:
branches: [ "main" ]
jobs:
build:
runs-on: ubuntu-latest
strategy:
max-parallel: 4
matrix:
python-version: [3.9, 3.11, 3.12]
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install Dependencies
run: |
python -m pip install --upgrade pip
python -m pip install poetry
poetry install
- name: Run Tests
run: |
pytest

6
Makefile Normal file
View File

@ -0,0 +1,6 @@
deploy:
ssh vrobbler.service "rm -rf /usr/local/lib/python3.11/site-packages/vrobbler-0.15.4.dist-info/ && pip install git+https://code.unbl.ink/secstate/vrobbler.git@develop && immortalctl restart vrobbler && immortalctl restart vrobbler-celery && vrobbler migrate"
logs:
ssh life.unbl.ink tail -n 100 -f /var/log/vrobbler.json
test:
pytest vrobbler

847
PROJECT.org Normal file
View File

@ -0,0 +1,847 @@
#+title: Vrobbler Project
* Overview
* Features
** Beer
*** Triggers
**** Bookmarklet
**** Manual
*** Metadata sources
**** Untappd
** Book
*** Triggers
**** Webdav via KoReader
**** Manual
*** Metadata sources
**** Google Books
This is the preferred method at this time. Also, the Book model implements a
`find_or_create` classmethod which is an example of an interface we can use for
other data models to get metadata in a way that provides easy testing, bulk
fetching and simple saving.
**** OpenLibrary
**** ComicVine
** Board Game
*** Triggers
**** IMAP import
**** Bookmarklet
**** Manual
** Location
*** Triggers
**** GPSLogger (Android)
*** Metadata sources
**** User input
** Music
*** Triggers
**** Last.FM
**** Rockbox files
**** Mopidy
**** Jellyfin
*** Metadata sources
**** Musicbrainz
** Podcast
*** Triggers
**** Mopidy
*** Metadata sources
**** Google Podcasts
**** PodcastIndex
** Sport
*** Triggers
**** Bookmarklet
**** Manual
*** Metadata sources
**** Thes Sports DB
** Task
*** Triggers
**** Todoist
**** Org-mode
*** Metadata sources
**** User profile
** Trails
** Video
*** Triggers
**** Jellyfin
**** Bookmarklet
**** Manual
*** Metadata sources
**** IMDB
**** Youtube
** Web Page
*** Triggers
**** Bookmarklet
*** Metadata sources
**** Scraper
* Release history
* Chores
** DONE Document various vrobbler features :chore:personal:project:vrobbler:documentation:
:PROPERTIES:
:ID: 514e9285-96f1-265f-56df-118c12f60918
:END:
:LOGBOOK:
CLOCK: [2025-07-09 Wed 09:55]--[2025-07-09 Wed 10:15] => 0:20
:END:
* Backlog [1/22]
** TODO [#A] Add classmethod for metadata fetching to tracks :vrobbler:feature:music:personal:project:
:PROPERTIES:
:ID: bc4b45e5-4c65-13c5-ab7b-1937d3fbf5c2
:END:
:LOGBOOK:
CLOCK: [2025-07-09 Wed 10:15]
:END:
** TODO Look in comments for a timestamp for start from BG stats if the time is missing :vrobbler:feature:boardgames:project:personal:
** TODO Add importer class for IMAP imports :vrobbler:feature:imap:importers:project:personal:
** TODO Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
** TODO [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
** TODO [#A] Fix small bug in views for TV series where next episode is now None :vrobbler:bug:personal:videos:
#+begin_src python
ERROR django.request:241 log_response Internal Server Error: /series/c24100d1-da45-4abe-86bf-27cfce9b1f89/
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
return self.dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
return super().dispatch(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
return handler(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/views/generic/detail.py", line 109, in get
context = self.get_context_data(object=self.object)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/videos/views.py", line 33, in get_context_data
context_data["next_episode_id"] = "tt" + next_episode_id
~~~~~^~~~~~~~~~~~~~~~~
TypeError: can only concatenate str (not "NoneType") to str
#+end_src
** TODO [#A] Send periodic check notifications for board games and beer that ask if you're still scrobbling :vrobbler:personal:project:beers:boardgames:notifications:feature:
** TODO [#A] Fix raw text webpage title not truncating to 254 chars :vrobbler:personal:bug:webpages:
** TODO [#A] Fix koreader scrobble imports to use DST properly :vrobbler:personal:bug:books:imports:
The page data has the canonical date something was read in it, but it seems to be an hour off. I traced this back to being off during DST, so we just need the importer to be aware of whether a user is using DST or not and roll back an hour for part of the year.
Also, we'd need to adjust any old scrobbles that took place with DST off to roll them back by an hour.
** TODO [#A] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
Pretty clear, I would love to make trails more useful. Historically I wasn't
hiking a lot, which made the source for this a bit silly. But it's clear that
AllTrails is the best source, though having TrailForks is nice to.
** TODO [#B] Add `garmin_activity_id` to the TrailMetadataLog class :vrobbler:trails:feature:personal:project:
Would be nice to have some loose connection to the actual event in my Garmin profile.
** TODO [#B] Explore a way to add metadata editing to scrobbles after saving :vrobbler:spike:scrobbling:personal:project:
Could be as simple as a JSON form on the scrobble detail page (do I have have one of those yet?).
** TODO [#B] Explore a good way to show notes and descriptions from scrobbles to users :personal:project:scrobbling:vrobbler:spike:
** TODO [#B] Add webdav syncing to retroarch imports :vrobbler:videogames:webdav:feature:project:personal:
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
** TODO [#C] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
Rather than pick up an existing Podcast using the podcast title in the mopidy
file name, Vrobbler creates a new podcast with no enriched data. Not a big deal
for my use as the volume of podcasts I listen to makes manual fixes easy. But
it's annoying.
** TODO [#C] Move to using more robust mopidy-webhooks pacakge form pypi :utility:improvement:
:PROPERTIES:
:ID: ab31fdc3-359c-1b1d-6b9d-546b476021ba
:END:
*** Example payloads from mopidy-webhooks
**** Podcast playback ended
#+begin_src json
{
"type": "event",
"event": "track_playback_ended",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
},
"time_position": 3290
}
}
#+end_src
**** Podcast playback state changes
#+begin_src json
{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "paused",
"new_state": "playing"
}
}
#+end_src
#+begin_src json
{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "stopped",
"new_state": "playing"
}
}
#+end_src
**** Podcast playback started
#+begin_src json
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
}
}
}
#+end_src
**** Podcast playback paused
#+begin_src json
{
"type": "status",
"data": {
"state": "paused",
"current_track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
},
"time_position": 2350
}
}
#+end_src
**** Track playback started
#+begin_src json
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
}
}
}
#+end_src
**** Track playback in progress
#+begin_src json
{
"type": "status",
"data": {
"state": "playing",
"current_track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
},
"time_position": 17031
}
}
#+end_src
**** Track event playback paused
#+begin_src json
{
"type": "event",
"event": "track_playback_paused",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
},
"time_position": 67578
}
}
#+end_src
** TODO [#C] User should be able to enable auto trail tracking via amail reader with Garmin LiveTrack URLs :vrobbler:trails:project:feature:personal:
** TODO [#C] Allow users to see tasks on calendar view :vrobbler:personal:project:templates:feature:
https://codepen.io/oliviale/pen/QYqybo
** TODO [#C] Come up with a possible flow using WebDAV and super-productivity for tasks :personal:feature:project:vrobbler:tasks:
* Version 19.0
** DONE Add periodic check for mood :vrobbler:feature:moods:personal:project:
:PROPERTIES:
:ID: 55404488-c69f-0dd5-838e-1d1e15c873eb
:END:
* Version 18.7
** DONE Use the timezone history log to fix old Scrobbles that fall into those timezone blocks :vrobbler:chore:scrobbles:project:personal:
:PROPERTIES:
:ID: 9d055ac1-584b-20c8-7ad9-9ce36b329dc7
:END:
* Version 18.4
** DONE Track timezone changes for profiles :vrobbler:feature:profiles:personal:project:
:PROPERTIES:
:ID: 89ec867f-29fd-82f1-be17-b49dddc30c78
:END:
[2025-07-11 14:23]
** DONE Only create a LastFM import if there are files to import :vrobbler:feature:lastfm:importers:project:personal:
:PROPERTIES:
:ID: 7a1456af-c5f1-6385-b34f-0be24a6b65b0
:END:
- Note taken on [2025-07-20 Sun 16:21]
This thing is kicking my butt. As it stands it works, but the scrobbles are not assigned to the tracks properly.
* Version 18.3
** DONE Add timezone awarness to IMAP importer :personal:project:vrobbler:feature:importer:imap:timezones:
:PROPERTIES:
:ID: 05837b7c-96aa-6190-3678-e2ae7c7cac75
:END:
* Version 18
** DONE Condense tracks of the same title by the same artist with multiple albums :vrobbler:feature:music:project:personal:
:PROPERTIES:
:ID: b39fcec8-59fd-eab0-5809-b8144c7d2708
:END:
** DONE Import from BG stats a "learning" log field when "Learning to play" is in the comment :vrobbler:feature:boardgames:project:personal:
:PROPERTIES:
:ID: fda59fab-4349-e99e-54c6-9f1392a1c474
:END:
** DONE [#A] Add email importer for BG stats file uploads :vrobbler:feature:boardgames:personal:project:
:PROPERTIES:
:ID: 116fe738-7966-615c-d195-ccff0337b101
:END:
#+begin_src json example of a file
{
"about": "This is a Play file that can be read by Board Game Stats. If you see this text, try to use a share, export or open-in function to open it with Board Game Stats.",
"players": [
{
"uuid": "31f8b92e-11d8-4162-88b1-fd9c79eea249",
"id": 2,
"name": "Colin",
"isAnonymous": false,
"modificationDate": "2025-07-01 18:10:32",
"metaData": "{\"isNpc\":0}"
},
{
"uuid": "00074700-cf4e-4ad3-b334-d35805bb0d90",
"id": 4,
"name": "Asa Sewell",
"isAnonymous": false,
"modificationDate": "2025-07-01 18:03:37"
}
],
"locations": [
{
"uuid": "14f7389c-767f-4725-9b35-906c407b293c",
"id": 3,
"name": "Timberwyck Farm",
"modificationDate": "2025-07-01 18:03:38"
}
],
"games": [
{
"uuid": "043a2851-f201-467a-a60c-0b0a7e9c33d2",
"id": 333,
"name": "Ghost Fightin' Treasure Hunters: Anniversary Edition",
"modificationDate": "2025-07-02 01:37:14",
"cooperative": true,
"highestWins": true,
"noPoints": false,
"usesTeams": false,
"urlThumb": "https://cf.geekdo-images.com/DHA-mcH3zzw_OjfDxOPj1A__thumb/img/UhaIm4KIDIiraUc44QIvSAbMUXI=/fit-in/200x150/filters:strip_icc()/pic8266874.jpg",
"urlImage": "https://cf.geekdo-images.com/DHA-mcH3zzw_OjfDxOPj1A__original/img/2-Lb6nLePhn0I0Hh2j1pOtbO4rg=/0x0/filters:format(jpeg)/pic8266874.jpg",
"bggName": "Ghost Fightin' Treasure Hunters: Anniversary Edition",
"bggYear": 2024,
"bggId": 422668,
"designers": "Brian Yu",
"isBaseGame": 1,
"isExpansion": 0,
"rating": 75,
"minPlayerCount": 2,
"maxPlayerCount": 5,
"minPlayTime": 30,
"maxPlayTime": 0,
"minAge": 8
}
],
"plays": [
{
"uuid": "bae3f29e-5e1e-45d8-b409-47a665c8d5b5",
"modificationDate": "2025-07-02 01:37:59",
"entryDate": "2025-07-02 01:31:38",
"playDate": "2025-07-02 01:31:38",
"usesTeams": false,
"durationMin": 23,
"ignored": false,
"manualWinner": true,
"rounds": 3,
"scoresheet": "{\"bggId\":244711,\"version\":1,\"langCode\":\"en\",\"scoreType\":\"bestTotalWins\",\"groups\":[{\"templateId\":\"1\",\"maxRepeat\":-1,\"repetition\":1,\"hasSubTotal\":false,\"hideSingleGroupLabel\":false,\"isExtra\":false,\"rows\":[{\"templateId\":\"vptrack\",\"label\":\"VP track\",\"repetition\":1,\"repeatable\":false,\"negative\":false,\"isExtra\":false,\"scores\":{}},{\"templateId\":\"objectives\",\"label\":\"Objectives\",\"repetition\":1,\"repeatable\":false,\"negative\":false,\"isExtra\":false,\"scores\":{}},{\"templateId\":\"mastercards\",\"label\":\"Master cards\",\"repetition\":1,\"repeatable\":false,\"negative\":false,\"isExtra\":false,\"scores\":{}}]}]}",
"locationRefId": 3,
"gameRefId": 333,
"board": "",
"scoringSetting": 4,
"metaData": "{\"playUsedGameCopy\":2}",
"playerScores": [
{
"score": "",
"winner": true,
"newPlayer": true,
"startPlayer": false,
"playerRefId": 4,
"role": "",
"rank": 0,
"seatOrder": 0,
"metaData": "{\"scoreUuid\":\"00074700-cf4e-4ad3-b334-d35805bb0d90\"}"
},
{
"score": "",
"winner": true,
"newPlayer": true,
"startPlayer": false,
"playerRefId": 2,
"role": "",
"rank": 0,
"seatOrder": 0,
"metaData": "{\"scoreUuid\":\"31f8b92e-11d8-4162-88b1-fd9c79eea249\"}"
}
],
"expansionPlays": []
}
],
"userInfo": {
"meRefId": 2
}
}
#+end_src
** DONE [#B] Fix task app to only use one tag for the context a task was done in and allow configurable contexts by user profile :personal:vrobbler:feature:tasks:project:
:PROPERTIES:
:ID: 23f485e3-988c-6198-c79d-91fdf92f001c
:END:
* Version 17.0
** DONE [#A] Fix bug in new task label lookup for Emacs/Org-mode :vrobbler:bug:tasks:
:PROPERTIES:
:ID: 683fb109-dfc4-85e4-80f0-ea618434f61e
:END:
** DONE [#C] Replace commas in the bandcamp URL for artists with nothing :vrobbler:music:bug:personal:
:PROPERTIES:
:ID: 9b30d67b-91f0-a480-dfaa-5d9dc090e76c
:END:
- Note taken on [2025-06-16 Mon 09:36]
This firt appeared with Black Country, New Road, where the RYM slug generator
leaves commas in and ends up sending you to a 404. I suspect this wont be the
first tweak we'll need to this, as the RYM link creator is really just
guessing based on the artist name at the path.
** DONE [#A] Investigate new source of video metadata :personal:project:video:imdb:
:PROPERTIES:
:ID: df2b486c-1170-5199-c312-9bc87760d962
:END:
Cinemagoer broke and I probably should find a more reilable source of video data.
- Note taken on [2025-06-13 Fri 11:19]
TMDB is much more reliable, but does require an API key. That's all setup now,
so hopefully this breaking IMDB crap is over.
** DONE [#A] IMDB video lookups are failing :personal:bug:video:imdb:
:PROPERTIES:
:ID: 38f1081f-37b4-f4f2-79aa-c1e87eca4b69
:END:
<2025-06-13 Fri>
- Note taken on [2025-06-13 Fri 08:24]
Looks like Cinemagoer is broken: https://github.com/cinemagoer/cinemagoer/issues/537
** DONE [#A] Emacs is not syncing notes :personal:scrobbling:emacs:bug:
:PROPERTIES:
:ID: c79cd491-b30f-0945-d84b-b8cac7562791
:END:
<2025-06-12 Thu 9:30>
Not sure if the problem is in my Emacs hook sending or Vrobbler itself.
- Note taken on [2025-06-12 Thu 09:47]
Adding a quick note to check on it
- Note taken on [2025-06-12 Thu 09:50]
Ah ha. All the messing about with the source field meant that I was looking
for `emacs` as a source but the hook was initially setting sources to
`orgmode` I think I prefer `orgmode` as the source, so updating it thusly.
Fixed in `490d60cbbb1f8bf90b5fc47d8685b15bdc1d485b`
** DONE [#A] Show the description of a task in the string rep for a scrobble of a Task :personal:project:scrobbling:vrobbler:feature:
:PROPERTIES:
:ID: df58f8d0-fa4a-2037-c7d7-e5388c239042
:END:
* Version 0.16.0
** DONE [#A] Jellyfin, bandcamp tracks from Mopidy create duplicate music tracks :bug:scrobbling:music:
:PROPERTIES:
:ID: 670e8634-49b5-dce9-1684-14f2ffb797f1
:END:
Effectively, any track that comes in without a MusicBrainz ID does some funky
lookup where it doesn't find a track without an MB id and the track title /
artist combination and creates a new track every time. This has to be cleaned up
by condensing the duplicated tracks into the original proper track.
But it opens a bigger question about how much MB id should the drive the app
lookup. If it can't be depended on to exist from all sources, it really can't be
canonical. Instead, the combination of track title / artist is really the best
we can do. Last.fm also has this problem, where it doesn't know about albums and
definitely does not know or care about MB ids.
** DONE Add a user profile page with ability to change settings :profiles:improvement:
- Note taken on [2025-04-04 Fri 10:51]
[[orgit-rev:~/src/code.unbl.ink/secstate/vrobbler/::93c16d80ecff4cd1663cf9ec40fbe6d8f58c3e44][~/src/code.unbl.ink/secstate/vrobbler/ (magit-rev 93c16d8)]]
https://code.unbl.ink/secstate/vrobbler/commit/93c16d80ecff4cd1663cf9ec40fbe6d8f58c3e44
** DONE What to do with Youtube videos from LastFM and web-scrobbler :bug:source:lastfm:
- Note taken on [2025-04-04 Fri 10:46]
Nothing. Over the last few months I built out a youtube model in videos and
use a bookmarklet scrobbling pattern. Now web-scrobbler is just disabled for
Youtube.
May want to revisit this at some point and only scrobble tracks from Youtube,
because many people use YT for music listening.
** DONE [#C] Consider a purge command for duplicated and stuck in-progress scrobbles :utililty:improvement:
CLOSED: [2023-04-06 Thu 14:09]
** DONE Add a "stop_timestamp" so we don't rely on content length :improvement:scrobbling:
CLOSED: [2023-04-02 Sun 23:58]
Essentially, we currently have the timestamp as when the content began
scrobbling and then calculate the finish time from the length of the content.
This works pretty well because we know how long most things are.
But in some cases, sports events or long podcasts, we may start mid-way through
an event or finish halfway through but still want to mark it as done. In these
cases, knowing the finish time could be useful, especially when interfacing with
other scrobblers which may have different definitions of when a scrobble
finishes or started.
** DONE Fix bug with Various Artist albums being labeled with first artist as album artist :scrobbling:bug:music:
CLOSED: [2023-03-27 Mon 20:18]
:LOGBOOK:
CLOCK: [2023-03-26 Sun 22:01]--[2023-03-27 Mon 01:07] => 3:06
:END:
** DONE Fix bug with weekly aggregator being blank on Sundays :aggregators:music:bug:
CLOSED: [2023-03-26 Sun 13:52]
** DONE Fix KoReader scrobbling to use pages rather than time of last read :scrobbling:books:improvement:
CLOSED: [2023-03-26 Sun 13:51]
:LOGBOOK:
CLOCK: [2023-03-26 Sun 13:11]--[2023-03-26 Sun 13:51] => 0:40
:END:
** DONE [#A] Add django-storage to store files on S3 :settings:improvement:
CLOSED: [2023-03-24 Fri 14:46]
:LOGBOOK:
CLOCK: [2023-03-24 Fri 10:47]--[2023-03-24 Fri 14:46] => 3:59
CLOCK: [2023-03-24 Fri 10:36]--[2023-03-24 Fri 10:40] => 0:04
:END:
** DONE Fix vrobbler settings not using booleans :settings:bug:
CLOSED: [2023-03-24 Fri 10:45]
:LOGBOOK:
CLOCK: [2023-03-24 Fri 10:40]--[2023-03-24 Fri 10:46] => 0:06
:END:
** DONE Update weekly live chart to be 7-day continuous rather than weekly :views:bug:
CLOSED: [2023-03-24 Fri 00:31]
The live view will be blank every Monday, no reason to tie it to a day of the
week. It should be "the last 7 days"
** DONE [#B] Implement a detail view for TV shows :improvement:views:
CLOSED: [2023-03-22 Wed 17:05]
** DONE [#B] Implement a detail view for Movies :improvement:views:
CLOSED: [2023-03-22 Wed 17:05]
** DONE Add "service provider" to TV Series, and use that for source when available :bug:scrobbling:
CLOSED: [2023-03-22 Wed 17:04]
** DONE Add view for long-play content (books, video games) to restart them :views:improvement:
CLOSED: [2023-03-22 Wed 17:01]
** DONE Add live chart view like Maloja :improvement:views:
CLOSED: [2023-03-07 Tue 11:13]
** DONE [#C] Figure out how to add to web-scrobbler :improvement:scrobbling:
CLOSED: [2023-03-22 Wed 17:06]
An example:
https://github.com/web-scrobbler/web-scrobbler/blob/master/src/core/background/scrobbler/maloja-scrobbler.js
This is actually going to be moot because we can import from LastFM, and
web-scrobbler integrates well with LastFM. The only thing to think through here
now is what to do with all the garbage web-scrobbler sometimes pushes to LastFM
from Youtube (all videos get pushed, sigh).
** DONE Add Amazon scraper to look up books when OL fails :books:improvement:
This turned out to be a non-starter ... Amazon is aggressive at disallowing
scraping quality. And all the OSS tools out there are stuck in an arms race
trying to keep them from breaking.
That said, Google Books actually has a decent API (for now), and I've built this
out using that.
** DONE Fix bug in Jellyfin scrobbles that spam more scrobbles after completion :scrobbling:videos:bug:
This was fixed a while ago, but there's a new manifested bug. Going to create a
separate bug tracking ticket for that.
* Version 0.11.4
** DONE Add rudimentary video game scrobbling :improvement:content:videogames:
CLOSED: [2023-03-07 Tue 11:11]
** DONE Add ability to scrobble from KOReader statistics files :improvement:books:content:
CLOSED: [2023-03-07 Tue 11:11]
** DONE [#A] Fix fetching artwork without release group :bug:
CLOSED: [2023-01-29 Sun 14:27]
When we get artwork from Musicbrianz, and it's not found, we should check for
release groups as well. This will stop issues with missing artwork because of
obscure MB release matches.
** DONE [#A] Fix Jellyfin music scrobbling N+1 past 90 completion percent :bug:
CLOSED: [2023-01-30 Mon 18:31]
:LOGBOOK:
CLOCK: [2023-01-30 Mon 18:00]--[2023-01-30 Mon 18:31] => 0:31
:END:
If we play music from Jellyfin and the track reaches 90% completion, the
scrobbling goes crazy and starts creating new scrobbles with every update.
The cause is pretty simple, but the solution is hard. We want to mark a scrobble
as complete for the following conditions:
- Play stopped and percent played beyond 90%
- Play completely finished
But if we keep listening beyond 90, we should basically ignore updates (or just
update the existing scrobble)
** DONE [#A] Add support for Audioscrobbler tab-separated file uploads :improvement:
CLOSED: [2023-02-03 Fri 16:52]
An example of the format:
#+begin_src csv
,
#AUDIOSCROBBLER/1.1
#TZ/UNKNOWN
#CLIENT/Rockbox sansaclipplus $Revision$
75 Dollar Bill I Was Real I Was Real 4 1015 S 1740494944 64ff5f53-d187-4512-827e-7606c69e66ff
75 Dollar Bill I Was Real I Was Real 4 1015 S 1740494990 64ff5f53-d187-4512-827e-7606c69e66ff
311 311 Down 1 173 S 1740495003 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Down 1 173 L 1740495049 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Down 1 173 L 1740495113 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Random 2 187 S 1740495190 530c09f3-46fe-4d90-b11f-7b63bcb4b373
311 311 Random 2 187 L 1740495194 530c09f3-46fe-4d90-b11f-7b63bcb4b373
311 311 Jackolanterns Weather 3 204 L 1740495382 cc3b2dec-5d99-47ea-8930-20bf258be4ea
311 311 All Mixed Up 4 182 L 1740495586 980a78b5-5bdd-4f50-9e3a-e13261e2817b
311 311 Hive 5 179 L 1740495768 18f6dc98-d3a2-4f81-b967-97359d14c68c
311 311 Guns (Are for Pussies) 6 137 L 1740495948 5e97ed9f-c8cc-4282-9cbe-f8e17aee5128
311 311 Misdirected Hostility 7 179 S 1740496085 61ff2c1a-fc9c-44c3-8da1-5e50a44245af
,
#+end_src
** DONE [#B] Allow scrobbling music without MB IDs by grabbing them before scrobble :improvement:
CLOSED: [2023-02-17 Fri 00:10]
This would allow a few nice flows. One, you'd be able to record the play of an
entire album by just dropping the muscibrainz_id in. This could be helpful for
offline listening. It would also mean bad metadata from mopidy would not break
scrobbling.
** DONE When updating musicbrainz IDs, clear and run fetch artwrok :improvement:
CLOSED: [2023-02-17 Fri 00:11]
** DONE [#A] Add ability to manually scrobble albums or tracks from MB :improvement:
CLOSED: [2023-03-07 Tue 11:09]
Given a UUID from musicbrainz, we should be able to scrobble an album or
individual track.
** DONE [#C] Implement keeping track of week/month/year chart-toppers :improvement:
CLOSED: [2023-03-07 Tue 11:10]
:LOGBOOK:
CLOCK: [2023-01-30 Mon 16:30]--[2023-01-30 Mon 18:00] => 1:30
:END:
Maloja does this cool thing where artists and tracks get recorded as the top
track of a given week, month or year. They get gold, silver or bronze stars for
their place in the time period.
I could see this being implemented as a separate Chart table which gets
populated at the end of a time period and has a start and end date that defines
a period, along with a one, two, three instance.
Of course, it could also be a data model without a table, where it runs some fun
calculations, stores it's values in Redis as a long-term lookup table and just
has to re-populate when the server restarts.

File diff suppressed because one or more lines are too long

View File

@ -1,4 +1,4 @@
export ENV_PATH=$(poetry env info --path)
source "${ENV_PATH}/bin/activate"
export PYPI_PASSWORD="$(pass personal/apikey/pypi)"
#export PYPI_PASSWORD="$(pass personal/apikey/pypi)"

4724
poetry.lock generated

File diff suppressed because it is too large Load Diff

View File

@ -1,17 +1,18 @@
[tool.poetry]
name = "vrobbler"
version = "0.11.12"
version = "0.16.1"
description = ""
authors = ["Colin Powell <colin@unbl.ink>"]
[tool.poetry.dependencies]
python = ">=3.9,<4.0"
python = ">=3.9,<3.12"
Django = "^4.0.3"
django-extensions = "^3.1.5"
python-dateutil = "^2.8.2"
python-dotenv = "^0.20.0"
python-json-logger = "^2.0.2"
colorlog = "^6.6.0"
httpx = "<=0.27.2"
djangorestframework = "^3.13.1"
Markdown = "^3.3.6"
django-filter = "^21.1"
@ -38,7 +39,6 @@ honcho = "^1.1.0"
howlongtobeatpy = "^1.0.5"
beautifulsoup4 = "^4.11.2"
django-storages = "^1.13.2"
boto3 = "^1.26.98"
stream-sqlite = "^0.0.41"
ipython = "^8.14.0"
pendulum = "^2.1.2"
@ -46,11 +46,21 @@ trafilatura = "^1.6.3"
django-imagekit = "^5.0.0"
thefuzz = "^0.22.1"
dataclass-wizard = "0.22.0"
webdavclient3 = "^3.14.6"
boto3 = "^1.35.37"
urllib3 = "<2"
django-oauth-toolkit = "^3.0.1"
meta-yt = "^0.1.9"
berserk = "^0.13.2"
poetry-bumpversion = "^0.3.3"
orgparse = "^0.4.20250520"
tmdbv3api = "^1.9.0"
themoviedb = "^1.0.2"
[tool.poetry.group.dev]
[tool.poetry.group.test]
optional = true
[tool.poetry.group.dev.dependencies]
[tool.poetry.group.test.dependencies]
Werkzeug = "2.0.3"
black = "^22.3"
coverage = "^7.0.5"
@ -59,6 +69,7 @@ pytest = "^7.1"
pytest-black = "^0.3.12"
pytest-cov = "^3.0"
pytest-django = "^4.5.2"
pytest-xdist= "^1.0.0"
pytest-flake8 = "^1.1"
pytest-isort = "^3.0"
pytest-runner = "^6.0"

View File

@ -4,29 +4,40 @@ import pytest
from django.contrib.auth import get_user_model
from rest_framework.authtoken.models import Token
from scrobbles.models import Scrobble
from boardgames.models import BoardGame
from music.models import Track, Artist
from scrobbles.models import Scrobble
from people.models import Person
User = get_user_model()
@pytest.fixture
def boardgame_scrobble():
user = User.objects.create(
email="test@exmaple.com", first_name="Test", last_name="User"
)
first = Person.objects.create(name="First Player")
second = Person.objects.create(name="Second Player")
return Scrobble.objects.create(
board_game=BoardGame.objects.create(title="Test Board Game"),
media_type="BoardGame",
played_to_completion=True,
log={
"players": [
{"user_id": user.id, "win": True, "score": 30, "color": "Blue"}
]
{"person_id": first.id, "win": True, "score": 30, "color": "Blue"},
{"person_id": second.id, "win": False, "score": 28, "color": "Red"}
],
},
)
@pytest.fixture
def test_track():
Track.objects.create(
title="Emotion",
artist=Artist.objects.create(name="Carly Rae Jepsen"),
run_time_seconds=60,
)
class MopidyRequest:
name = "Same in the End"
artist = "Sublime"
@ -98,9 +109,11 @@ def mopidy_track_diff_album_request_data(**kwargs):
@pytest.fixture
def mopidy_podcast():
def mopidy_podcast_request_data():
mopidy_uri = "local:podcast:Up%20First/2022-01-01%20Up%20First.mp3"
return MopidyRequest(mopidy_uri=mopidy_uri)
return MopidyRequest(
mopidy_uri=mopidy_uri, artist="NPR", album="Up First"
).request_json
class JellyfinTrackRequest:

View File

@ -3,14 +3,13 @@ import pytest
from scrobbles.dataclasses import BoardGameLogData, BoardGameScoreLogData
@pytest.mark.skip("Need to get local tests running working again")
@pytest.mark.django_db
def test_boardgame_log_data(boardgame_scrobble):
assert not boardgame_scrobble.geo_location
assert boardgame_scrobble.logdata == BoardGameLogData(
players=[
BoardGameScoreLogData(
user_id=1,
name_str="",
person_id=1,
bgg_username="",
color="Blue",
character=None,
@ -18,10 +17,24 @@ def test_boardgame_log_data(boardgame_scrobble):
score=30,
win=True,
new=None,
)
rank=None,
seat_order=None,
role=None
),
BoardGameScoreLogData(
person_id=2,
bgg_username="",
color="Red",
character=None,
team=None,
score=28,
win=False,
new=None,
rank=None,
seat_order=None,
role=None
),
],
location=None,
geo_location_id=None,
difficulty=None,
solo=None,
two_handed=None,

View File

@ -30,55 +30,26 @@ def test_bad_mopidy_request_data(client, valid_auth_token):
)
@pytest.mark.parametrize(
"seconds, expected_percent_played, expected_scrobble_id",
[
(1, 1, 1),
(58, 96, 1),
(59, 98, 1),
(60, 100, 1),
(1, 1, 2),
(1, 1, 3),
],
@pytest.mark.skip("Need to refactor")
@pytest.mark.django_db
@patch("music.utils.lookup_artist_from_mb", return_value={})
@patch(
"music.utils.lookup_album_dict_from_mb",
return_value={"year": "1999", "mb_group_id": 1},
)
@pytest.mark.django_db
def test_scrobble_mopidy_track(
client,
mopidy_track,
valid_auth_token,
seconds,
expected_percent_played,
expected_scrobble_id,
):
url = reverse("scrobbles:mopidy-webhook")
headers = {"Authorization": f"Token {valid_auth_token}"}
# Start new scrobble
minutes = 0
calc_seconds = seconds
if seconds >= 60:
minutes = 1
calc_seconds = calc_seconds % 10
with time_machine.travel(datetime(2024, 1, 14, 12, minutes, calc_seconds)):
mopidy_track.request_data["playback_time_ticks"] = seconds * 1000
response = client.post(
url,
mopidy_track.request_json,
content_type="application/json",
headers=headers,
)
assert response.status_code == 200
assert response.data == {"scrobble_id": expected_scrobble_id}
scrobble = Scrobble.objects.get(id=1)
assert scrobble.percent_played == expected_percent_played
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Same in the End"
@pytest.mark.skip(reason="Allmusic API is unstable")
@pytest.mark.django_db
@patch("music.utils.lookup_track_from_mb", return_value={})
@patch("music.models.lookup_artist_from_tadb", return_value={})
@patch("music.models.lookup_album_from_tadb", return_value={"year": "1999"})
@patch("music.models.Album.fetch_artwork", return_value=None)
@patch("music.models.Album.scrape_allmusic", return_value=None)
def test_scrobble_mopidy_same_track_different_album(
mock_lookup_artist,
mock_lookup_album,
mock_lookup_track,
mock_lookup_artist_tadb,
mock_lookup_album_tadb,
mock_fetch_artwork,
mock_scrape_allmusic,
client,
mopidy_track,
mopidy_track_diff_album_request_data,
@ -107,13 +78,17 @@ def test_scrobble_mopidy_same_track_different_album(
assert response.data == {"scrobble_id": 2}
scrobble = Scrobble.objects.last()
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.album.name == "Gold"
assert scrobble.media_obj.album.name == "Sublime"
assert scrobble.media_obj.title == "Same in the End"
@pytest.mark.django_db
@patch(
"podcasts.sources.podcastindex.lookup_podcast_from_podcastindex",
return_value={},
)
def test_scrobble_mopidy_podcast(
client, mopidy_podcast_request_data, valid_auth_token
mock_lookup_podcast, client, mopidy_podcast_request_data, valid_auth_token
):
url = reverse("scrobbles:mopidy-webhook")
headers = {"Authorization": f"Token {valid_auth_token}"}
@ -131,6 +106,7 @@ def test_scrobble_mopidy_podcast(
assert scrobble.media_obj.title == "Up First"
@pytest.mark.skip("Need to refactor")
@pytest.mark.django_db
@patch("music.utils.lookup_artist_from_mb", return_value={})
@patch(
@ -174,38 +150,103 @@ def test_scrobble_jellyfin_track(
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Emotion"
with time_machine.travel(datetime(2024, 1, 14, 12, 0, 58)):
jellyfin_track.request_data["UtcTimestamp"] = timezone.now().strftime(
"%Y-%m-%d %H:%M:%S"
)
response = client.post(
url,
jellyfin_track.request_json,
content_type="application/json",
headers=headers,
)
assert response.status_code == 200
assert response.data == {"scrobble_id": 1}
@pytest.mark.skip("Need to refactor")
@pytest.mark.django_db
@patch("music.utils.lookup_artist_from_mb", return_value={})
@patch(
"music.utils.lookup_album_dict_from_mb",
return_value={"year": "1999", "mb_group_id": 1},
)
@patch("music.utils.lookup_track_from_mb", return_value={})
@patch("music.models.lookup_artist_from_tadb", return_value={})
@patch("music.models.lookup_album_from_tadb", return_value={"year": "1999"})
@patch("music.models.Album.fetch_artwork", return_value=None)
@patch("music.models.Album.scrape_allmusic", return_value=None)
def test_scrobble_jellyfin_track_update(
mock_lookup_artist,
mock_lookup_album,
mock_lookup_track,
mock_lookup_artist_tadb,
mock_lookup_album_tadb,
mock_fetch_artwork,
mock_scrape_allmusic,
test_track,
client,
jellyfin_track,
valid_auth_token,
):
Scrobble.objects.create(
timestamp=timezone.now() - timedelta(minutes=0.5),
track=Track.objects.first(),
user_id=1,
)
url = reverse("scrobbles:jellyfin-webhook")
headers = {"Authorization": f"Token {valid_auth_token}"}
scrobble = Scrobble.objects.get(id=1)
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Emotion"
jellyfin_track.request_data["UtcTimestamp"] = timezone.now().strftime(
"%Y-%m-%d %H:%M:%S"
)
response = client.post(
url,
jellyfin_track.request_json,
content_type="application/json",
headers=headers,
)
with time_machine.travel(datetime(2024, 1, 14, 12, 1, 1)):
jellyfin_track.request_data["UtcTimestamp"] = timezone.now().strftime(
"%Y-%m-%d %H:%M:%S"
)
response = client.post(
url,
jellyfin_track.request_json,
content_type="application/json",
headers=headers,
)
assert response.status_code == 200
assert response.data == {"scrobble_id": 1}
assert response.status_code == 200
assert response.data == {"scrobble_id": 2}
scrobble = Scrobble.objects.get(id=1)
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Emotion"
scrobble = Scrobble.objects.get(id=1)
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Emotion"
@pytest.mark.skip("Need to refactor")
@pytest.mark.django_db
@patch("music.utils.lookup_artist_from_mb", return_value={})
@patch(
"music.utils.lookup_album_dict_from_mb",
return_value={"year": "1999", "mb_group_id": 1},
)
@patch("music.utils.lookup_track_from_mb", return_value={})
@patch("music.models.lookup_artist_from_tadb", return_value={})
@patch("music.models.lookup_album_from_tadb", return_value={"year": "1999"})
@patch("music.models.Album.fetch_artwork", return_value=None)
@patch("music.models.Album.scrape_allmusic", return_value=None)
def test_scrobble_jellyfin_track_create_new(
mock_lookup_artist,
mock_lookup_album,
mock_lookup_track,
mock_lookup_artist_tadb,
mock_lookup_album_tadb,
mock_fetch_artwork,
mock_scrape_allmusic,
test_track,
client,
jellyfin_track,
valid_auth_token,
):
url = reverse("scrobbles:jellyfin-webhook")
headers = {"Authorization": f"Token {valid_auth_token}"}
Scrobble.objects.create(
timestamp=timezone.now() - timedelta(minutes=1),
track=Track.objects.first(),
user_id=1,
)
jellyfin_track.request_data["UtcTimestamp"] = timezone.now().strftime(
"%Y-%m-%d %H:%M:%S"
)
response = client.post(
url,
jellyfin_track.request_json,
content_type="application/json",
headers=headers,
)
assert response.status_code == 200
assert response.data == {"scrobble_id": 2}
scrobble = Scrobble.objects.get(id=1)
assert scrobble.media_obj.__class__ == Track
assert scrobble.media_obj.title == "Emotion"

View File

@ -1,11 +1,6 @@
import pytest
from videos.imdb import lookup_video_from_imdb
from videos.sources.imdb import lookup_video_from_imdb
@pytest.mark.skip(reason="Need to sort out third party API testing")
def test_lookup_imdb_bad_id(caplog):
data = lookup_video_from_imdb("3409324")
assert data is None
assert caplog.records[0].levelname == "WARNING"
assert caplog.records[0].msg == "IMDB ID should begin with 'tt' 3409324"
def test_lookup_imdb():
metadata = lookup_video_from_imdb("8946378")
assert metadata.title == "Knives Out"

View File

@ -0,0 +1,9 @@
import pytest
from videos.sources.youtube import lookup_video_from_youtube
@pytest.mark.skip(reason="Need to configure Youtube API stuffs in CI")
@pytest.mark.django_db
def test_lookup_youtube_id():
metadata = lookup_video_from_youtube("RZxs9pAv99Y")
assert metadata.title == "No Pun Included's Board Game of the Year 2024"

449
todos.org
View File

@ -1,449 +0,0 @@
#+title: TODOs
A fun way to keep track of things in the project to fix or improve.
* Version 1.0.0
** TODO What to do with Youtube videos from LastFM and web-scrobbler :bug:source:lastfm:
** TODO Add a user profile page with ability to change settings :profiles:improvement:
** DONE [#C] Consider a purge command for duplicated and stuck in-progress scrobbles :utililty:improvement:
CLOSED: [2023-04-06 Thu 14:09]
** DONE Add a "stop_timestamp" so we don't rely on content length :improvement:scrobbling:
CLOSED: [2023-04-02 Sun 23:58]
Essentially, we currently have the timestamp as when the content began
scrobbling and then calculate the finish time from the length of the content.
This works pretty well because we know how long most things are.
But in some cases, sports events or long podcasts, we may start mid-way through
an event or finish halfway through but still want to mark it as done. In these
cases, knowing the finish time could be useful, especially when interfacing with
other scrobblers which may have different definitions of when a scrobble
finishes or started.
** DONE Fix bug with Various Artist albums being labeled with first artist as album artist :scrobbling:bug:music:
CLOSED: [2023-03-27 Mon 20:18]
:LOGBOOK:
CLOCK: [2023-03-26 Sun 22:01]--[2023-03-27 Mon 01:07] => 3:06
:END:
** DONE Fix bug with weekly aggregator being blank on Sundays :aggregators:music:bug:
CLOSED: [2023-03-26 Sun 13:52]
** DONE Fix KoReader scrobbling to use pages rather than time of last read :scrobbling:books:improvement:
CLOSED: [2023-03-26 Sun 13:51]
:LOGBOOK:
CLOCK: [2023-03-26 Sun 13:11]--[2023-03-26 Sun 13:51] => 0:40
:END:
** DONE [#A] Add django-storage to store files on S3 :settings:improvement:
CLOSED: [2023-03-24 Fri 14:46]
:LOGBOOK:
CLOCK: [2023-03-24 Fri 10:47]--[2023-03-24 Fri 14:46] => 3:59
CLOCK: [2023-03-24 Fri 10:36]--[2023-03-24 Fri 10:40] => 0:04
:END:
** DONE Fix vrobbler settings not using booleans :settings:bug:
CLOSED: [2023-03-24 Fri 10:45]
:LOGBOOK:
CLOCK: [2023-03-24 Fri 10:40]--[2023-03-24 Fri 10:46] => 0:06
:END:
** DONE Update weekly live chart to be 7-day continuous rather than weekly :views:bug:
CLOSED: [2023-03-24 Fri 00:31]
The live view will be blank every Monday, no reason to tie it to a day of the
week. It should be "the last 7 days"
** DONE [#B] Implement a detail view for TV shows :improvement:views:
CLOSED: [2023-03-22 Wed 17:05]
** DONE [#B] Implement a detail view for Movies :improvement:views:
CLOSED: [2023-03-22 Wed 17:05]
** DONE Add "service provider" to TV Series, and use that for source when available :bug:scrobbling:
CLOSED: [2023-03-22 Wed 17:04]
** DONE Add view for long-play content (books, video games) to restart them :views:improvement:
CLOSED: [2023-03-22 Wed 17:01]
** DONE Add live chart view like Maloja :improvement:views:
CLOSED: [2023-03-07 Tue 11:13]
** DONE [#C] Figure out how to add to web-scrobbler :improvement:scrobbling:
CLOSED: [2023-03-22 Wed 17:06]
An example:
https://github.com/web-scrobbler/web-scrobbler/blob/master/src/core/background/scrobbler/maloja-scrobbler.js
This is actually going to be moot because we can import from LastFM, and
web-scrobbler integrates well with LastFM. The only thing to think through here
now is what to do with all the garbage web-scrobbler sometimes pushes to LastFM
from Youtube (all videos get pushed, sigh).
* Version 0.11.4
** DONE Add rudimentary video game scrobbling :improvement:content:videogames:
CLOSED: [2023-03-07 Tue 11:11]
** DONE Add ability to scrobble from KOReader statistics files :improvement:books:content:
CLOSED: [2023-03-07 Tue 11:11]
** DONE [#A] Fix fetching artwork without release group :bug:
CLOSED: [2023-01-29 Sun 14:27]
When we get artwork from Musicbrianz, and it's not found, we should check for
release groups as well. This will stop issues with missing artwork because of
obscure MB release matches.
** DONE [#A] Fix Jellyfin music scrobbling N+1 past 90 completion percent :bug:
CLOSED: [2023-01-30 Mon 18:31]
:LOGBOOK:
CLOCK: [2023-01-30 Mon 18:00]--[2023-01-30 Mon 18:31] => 0:31
:END:
If we play music from Jellyfin and the track reaches 90% completion, the
scrobbling goes crazy and starts creating new scrobbles with every update.
The cause is pretty simple, but the solution is hard. We want to mark a scrobble
as complete for the following conditions:
- Play stopped and percent played beyond 90%
- Play completely finished
But if we keep listening beyond 90, we should basically ignore updates (or just
update the existing scrobble)
** DONE [#A] Add support for Audioscrobbler tab-separated file uploads :improvement:
CLOSED: [2023-02-03 Fri 16:52]
An example of the format:
#+begin_src csv
,
#AUDIOSCROBBLER/1.1
#TZ/UNKNOWN
#CLIENT/Rockbox sansaclipplus $Revision$
75 Dollar Bill I Was Real I Was Real 4 1015 S 1740494944 64ff5f53-d187-4512-827e-7606c69e66ff
75 Dollar Bill I Was Real I Was Real 4 1015 S 1740494990 64ff5f53-d187-4512-827e-7606c69e66ff
311 311 Down 1 173 S 1740495003 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Down 1 173 L 1740495049 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Down 1 173 L 1740495113 00476c23-fd9e-464b-9b27-a62d69f3d4f4
311 311 Random 2 187 S 1740495190 530c09f3-46fe-4d90-b11f-7b63bcb4b373
311 311 Random 2 187 L 1740495194 530c09f3-46fe-4d90-b11f-7b63bcb4b373
311 311 Jackolanterns Weather 3 204 L 1740495382 cc3b2dec-5d99-47ea-8930-20bf258be4ea
311 311 All Mixed Up 4 182 L 1740495586 980a78b5-5bdd-4f50-9e3a-e13261e2817b
311 311 Hive 5 179 L 1740495768 18f6dc98-d3a2-4f81-b967-97359d14c68c
311 311 Guns (Are for Pussies) 6 137 L 1740495948 5e97ed9f-c8cc-4282-9cbe-f8e17aee5128
311 311 Misdirected Hostility 7 179 S 1740496085 61ff2c1a-fc9c-44c3-8da1-5e50a44245af
,
#+end_src
** DONE [#B] Allow scrobbling music without MB IDs by grabbing them before scrobble :improvement:
CLOSED: [2023-02-17 Fri 00:10]
This would allow a few nice flows. One, you'd be able to record the play of an
entire album by just dropping the muscibrainz_id in. This could be helpful for
offline listening. It would also mean bad metadata from mopidy would not break
scrobbling.
** DONE When updating musicbrainz IDs, clear and run fetch artwrok :improvement:
CLOSED: [2023-02-17 Fri 00:11]
** DONE [#A] Add ability to manually scrobble albums or tracks from MB :improvement:
CLOSED: [2023-03-07 Tue 11:09]
Given a UUID from musicbrainz, we should be able to scrobble an album or
individual track.
** DONE [#C] Implement keeping track of week/month/year chart-toppers :improvement:
CLOSED: [2023-03-07 Tue 11:10]
:LOGBOOK:
CLOCK: [2023-01-30 Mon 16:30]--[2023-01-30 Mon 18:00] => 1:30
:END:
Maloja does this cool thing where artists and tracks get recorded as the top
track of a given week, month or year. They get gold, silver or bronze stars for
their place in the time period.
I could see this being implemented as a separate Chart table which gets
populated at the end of a time period and has a start and end date that defines
a period, along with a one, two, three instance.
Of course, it could also be a data model without a table, where it runs some fun
calculations, stores it's values in Redis as a long-term lookup table and just
has to re-populate when the server restarts.
* Backlog
** TODO Add Amazon scraper to look up books when OL fails :books:improvement:
** TODO [#C] Move to using more robust mopidy-webhooks pacakge form pypi :utility:improvement:
*** Example payloads from mopidy-webhooks
**** Podcast playback ended
#+begin_src json
{
"type": "event",
"event": "track_playback_ended",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
},
"time_position": 3290
}
}
#+end_src
**** Podcast playback state changes
#+begin_src json
{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "paused",
"new_state": "playing"
}
}
#+end_src
#+begin_src json
{
"type": "event",
"event": "playback_state_changed",
"data": {
"old_state": "stopped",
"new_state": "playing"
}
}
#+end_src
**** Podcast playback started
#+begin_src json
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 13,
"track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
}
}
}
}
#+end_src
**** Podcast playback paused
#+begin_src json
{
"type": "status",
"data": {
"state": "paused",
"current_track": {
"__model__": "Track",
"uri": "file:///var/lib/mopidy/media/podcasts/The%20Prince/2022-09-28-Wolf-warriors.mp3",
"name": "Wolf warriors",
"artists": [
{
"__model__": "Artist",
"name": "The Economist"
}
],
"album": {
"__model__": "Album",
"name": "The Prince",
"date": "2022"
},
"genre": "Blues",
"date": "2022",
"length": 2437778,
"bitrate": 127988
},
"time_position": 2350
}
}
#+end_src
**** Track playback started
#+begin_src json
{
"type": "event",
"event": "track_playback_started",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
}
}
}
#+end_src
**** Track playback in progress
#+begin_src json
{
"type": "status",
"data": {
"state": "playing",
"current_track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
},
"time_position": 17031
}
}
#+end_src
**** Track event playback paused
#+begin_src json
{
"type": "event",
"event": "track_playback_paused",
"data": {
"tl_track": {
"__model__": "TlTrack",
"tlid": 14,
"track": {
"__model__": "Track",
"uri": "local:track:Various%20Artists%20-%202008%20-%20Twilight%20OST/01-muse-supermassive_black_hole.mp3",
"name": "Supermassive Black Hole",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:250dd6551b66a58a6b4897aa697f200c",
"name": "Muse",
"musicbrainz_id": "9c9f1380-2516-4fc9-a3e6-f9f61941d090"
}
],
"album": {
"__model__": "Album",
"uri": "local:album:md5:455343d54cdd89cb5a3b5ad537ea99d0",
"name": "Twilight: Original Motion Picture Soundtrack",
"artists": [
{
"__model__": "Artist",
"uri": "local:artist:md5:54e4db2d5624f80b0cc290346e696756",
"name": "Various Artists",
"musicbrainz_id": "89ad4ac3-39f7-470e-963a-56509c546377"
}
],
"num_tracks": 12,
"num_discs": 1,
"date": "2008-11-04",
"musicbrainz_id": "b4889eaf-d9f4-434c-a68d-69227b12b6a4"
},
"composers": [
{
"__model__": "Artist",
"uri": "local:artist:md5:4d49cbca0b347e0a89047bb019d2779d",
"name": "Matt Bellamy"
}
],
"genre": "Rock",
"track_no": 1,
"disc_no": 1,
"date": "2008-11-04",
"length": 211121,
"musicbrainz_id": "ff1e3e1a-f6e8-4692-b426-355880383bb6",
"last_modified": 1672712949510
}
},
"time_position": 67578
}
}
#+end_src
** TODO Fix bug in Jellyfin scrobbles that spam more scrobbles after completion :scrobbling:videos:bug:
** TODO Fix bug in podcast scrobbling where a second scrobble is created after completion :scrobbling:podcasts:bug:

View File

@ -21,6 +21,10 @@ VROBBLER_THEAUDIODB_API_KEY="<key>"
VROBBLER_IGDB_CLIENT_ID="<id>"
VROBBLER_IGDB_CLIENT_SECRET="<key>"
VROBBLER_COMICVINE_API_KEY="<key>"
VROBBLER_TODOIST_CLIENT_ID="<id>"
VROBBLER_TODOIST_CLIENT_SECRET="<key>"
VROBBLER_GOOGLE_API_KEY="<key>"
VROBBLER_LICHESS_API_KEY = "<key>"
# Storages
# VROBBLER_DATABASE_URL="postgres://USER:PASSWORD@HOST:PORT/NAME"

View File

@ -0,0 +1,34 @@
from beers.models import Beer, BeerProducer, BeerStyle
from django.contrib import admin
from scrobbles.admin import ScrobbleInline
class BeerInline(admin.TabularInline):
model = Beer
extra = 0
@admin.register(BeerStyle)
class BeerStyle(admin.ModelAdmin):
date_hierarchy = "created"
search_fields = ("name",)
@admin.register(BeerProducer)
class BeerProducer(admin.ModelAdmin):
date_hierarchy = "created"
search_fields = ("name",)
@admin.register(Beer)
class BeerAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"uuid",
"title",
)
ordering = ("-created",)
search_fields = ("title",)
inlines = [
ScrobbleInline,
]

View File

@ -0,0 +1,5 @@
from django.apps import AppConfig
class BeersConfig(AppConfig):
name = "beers"

View File

@ -0,0 +1,133 @@
# Generated by Django 4.2.16 on 2024-10-22 21:26
from django.db import migrations, models
import django_extensions.db.fields
import taggit.managers
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
("scrobbles", "0065_alter_scrobble_log"),
]
operations = [
migrations.CreateModel(
name="BeerProducer",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("description", models.TextField(blank=True, null=True)),
(
"location",
models.CharField(blank=True, max_length=255, null=True),
),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
migrations.CreateModel(
name="Beer",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
(
"title",
models.CharField(blank=True, max_length=255, null=True),
),
(
"run_time_seconds",
models.IntegerField(blank=True, null=True),
),
(
"run_time_ticks",
models.PositiveBigIntegerField(blank=True, null=True),
),
("description", models.TextField(blank=True, null=True)),
("ibu", models.SmallIntegerField(blank=True, null=True)),
("abv", models.FloatField(blank=True, null=True)),
(
"style",
models.CharField(blank=True, max_length=100, null=True),
),
("non_alcoholic", models.BooleanField(default=False)),
(
"beeradvocate_id",
models.CharField(blank=True, max_length=255, null=True),
),
(
"beeradvocate_score",
models.SmallIntegerField(blank=True, null=True),
),
(
"untappd_id",
models.CharField(blank=True, max_length=255, null=True),
),
(
"genre",
taggit.managers.TaggableManager(
blank=True,
help_text="A comma-separated list of tags.",
through="scrobbles.ObjectWithGenres",
to="scrobbles.Genre",
verbose_name="Tags",
),
),
],
options={
"abstract": False,
},
),
]

View File

@ -0,0 +1,36 @@
# Generated by Django 4.2.16 on 2024-10-22 21:34
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("beers", "0001_initial"),
]
operations = [
migrations.AddField(
model_name="beer",
name="beeradvocate_image",
field=models.ImageField(
blank=True, null=True, upload_to="beers/beeradvcoate/"
),
),
migrations.AddField(
model_name="beer",
name="producer",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="beers.beerproducer",
),
),
migrations.AddField(
model_name="beerproducer",
name="beeradvocate_id",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -0,0 +1,75 @@
# Generated by Django 4.2.16 on 2024-10-22 21:47
from django.db import migrations, models
import django_extensions.db.fields
class Migration(migrations.Migration):
dependencies = [
("beers", "0002_beer_beeradvocate_image_beer_producer_and_more"),
]
operations = [
migrations.CreateModel(
name="BeerStyle",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("description", models.TextField(blank=True, null=True)),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
migrations.RemoveField(
model_name="beer",
name="beeradvocate_image",
),
migrations.RemoveField(
model_name="beer",
name="style",
),
migrations.AddField(
model_name="beer",
name="untappd_image",
field=models.ImageField(
blank=True, null=True, upload_to="beers/untappd/"
),
),
migrations.AddField(
model_name="beer",
name="untappd_rating",
field=models.FloatField(blank=True, null=True),
),
migrations.AddField(
model_name="beerproducer",
name="untappd_id",
field=models.CharField(blank=True, max_length=255, null=True),
),
migrations.AddField(
model_name="beer",
name="styles",
field=models.ManyToManyField(to="beers.beerstyle"),
),
]

View File

@ -0,0 +1,47 @@
# Generated by Django 4.2.16 on 2024-10-22 21:52
from django.db import migrations, models
import uuid
class Migration(migrations.Migration):
dependencies = [
("beers", "0003_beerstyle_remove_beer_beeradvocate_image_and_more"),
]
operations = [
migrations.AddField(
model_name="beerproducer",
name="name",
field=models.CharField(default="Untitled", max_length=255),
preserve_default=False,
),
migrations.AddField(
model_name="beerproducer",
name="uuid",
field=models.UUIDField(
blank=True, default=uuid.uuid4, editable=False, null=True
),
),
migrations.AddField(
model_name="beerstyle",
name="name",
field=models.CharField(default="Untitled", max_length=255),
preserve_default=False,
),
migrations.AddField(
model_name="beerstyle",
name="uuid",
field=models.UUIDField(
blank=True, default=uuid.uuid4, editable=False, null=True
),
),
migrations.AlterField(
model_name="beer",
name="styles",
field=models.ManyToManyField(
related_name="styles", to="beers.beerstyle"
),
),
]

View File

@ -0,0 +1,21 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
(
"beers",
"0004_beerproducer_name_beerproducer_uuid_beerstyle_name_and_more",
),
]
operations = [
migrations.AlterField(
model_name="beer",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,140 @@
from uuid import uuid4
from beers.untappd import get_beer_from_untappd_id, get_rating_from_soup
from django.apps import apps
from django.db import models
from django.urls import reverse
from django_extensions.db.models import TimeStampedModel
from imagekit.models import ImageSpecField
from imagekit.processors import ResizeToFit
from scrobbles.dataclasses import BeerLogData
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
BNULL = {"blank": True, "null": True}
class BeerStyle(TimeStampedModel):
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
name = models.CharField(max_length=255)
description = models.TextField(**BNULL)
def __str__(self):
return self.name
class BeerProducer(TimeStampedModel):
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
name = models.CharField(max_length=255)
description = models.TextField(**BNULL)
location = models.CharField(max_length=255, **BNULL)
beeradvocate_id = models.CharField(max_length=255, **BNULL)
untappd_id = models.CharField(max_length=255, **BNULL)
def find_or_create(cls, title: str) -> "BeerProducer":
return cls.objects.filter(title=title).first()
def __str__(self):
return self.name
class Beer(ScrobblableMixin):
description = models.TextField(**BNULL)
ibu = models.SmallIntegerField(**BNULL)
abv = models.FloatField(**BNULL)
styles = models.ManyToManyField(BeerStyle, related_name="styles")
non_alcoholic = models.BooleanField(default=False)
beeradvocate_id = models.CharField(max_length=255, **BNULL)
beeradvocate_score = models.SmallIntegerField(**BNULL)
untappd_image = models.ImageField(upload_to="beers/untappd/", **BNULL)
untappd_image_small = ImageSpecField(
source="untappd_image",
processors=[ResizeToFit(100, 100)],
format="JPEG",
options={"quality": 60},
)
untappd_image_medium = ImageSpecField(
source="untappd_image",
processors=[ResizeToFit(300, 300)],
format="JPEG",
options={"quality": 75},
)
untappd_id = models.CharField(max_length=255, **BNULL)
untappd_rating = models.FloatField(**BNULL)
producer = models.ForeignKey(
BeerProducer, on_delete=models.DO_NOTHING, **BNULL
)
def get_absolute_url(self) -> str:
return reverse("beers:beer_detail", kwargs={"slug": self.uuid})
def __str__(self):
return f"{self.title} by {self.producer}"
@property
def subtitle(self):
return self.producer.name
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Drinking", tags="beer")
@property
def beeradvocate_link(self) -> str:
link = ""
if self.producer and self.beeradvocate_id:
if self.beeradvocate_id:
link = f"https://www.beeradvocate.com/beer/profile/{self.producer.beeradvocate_id}/{self.beeradvocate_id}/"
return link
@property
def untappd_link(self) -> str:
link = ""
if self.untappd_id:
link = f"https://www.untappd.com/beer/{self.untappd_id}/"
return link
@property
def primary_image_url(self) -> str:
url = ""
if self.untappd_image:
url = self.untappd_image.url
return url
@property
def logdata_cls(self):
return BeerLogData
@classmethod
def find_or_create(cls, untappd_id: str) -> "Beer":
beer = cls.objects.filter(untappd_id=untappd_id).first()
if not beer:
beer_dict = get_beer_from_untappd_id(untappd_id)
producer_dict = {}
style_ids = []
for key in list(beer_dict.keys()):
if "producer__" in key:
pkey = key.replace("producer__", "")
producer_dict[pkey] = beer_dict.pop(key)
if "styles" in key:
for style in beer_dict.pop("styles"):
style_inst, created = BeerStyle.objects.get_or_create(
name=style
)
style_ids.append(style_inst.id)
producer, _created = BeerProducer.objects.get_or_create(
**producer_dict
)
beer_dict["producer_id"] = producer.id
beer = Beer.objects.create(**beer_dict)
for style_id in style_ids:
beer.styles.add(style_id)
return beer
def scrobbles(self, user_id):
Scrobble = apps.get_model("scrobbles", "Scrobble")
return Scrobble.objects.filter(user_id=user_id, beer=self).order_by(
"-timestamp"
)

View File

@ -0,0 +1,142 @@
import logging
from typing import Optional
import requests
from bs4 import BeautifulSoup
logger = logging.getLogger(__name__)
UNTAPPD_URL = "https://untappd.com/beer/{id}"
def get_first(key: str, result: dict) -> str:
obj = ""
if obj_list := result.get(key):
obj = obj_list[0]
return obj
def get_title_from_soup(soup) -> str:
title = ""
try:
title = soup.find("h1").get_text()
except AttributeError:
pass
except ValueError:
pass
return title
def get_description_from_soup(soup) -> str:
desc = ""
try:
desc = (
soup.find(class_="beer-descrption-read-less")
.get_text()
.replace("Show Less", "")
.strip()
)
except AttributeError:
pass
except ValueError:
pass
return desc
def get_styles_from_soup(soup) -> list[str]:
styles = []
try:
styles = soup.find("p", class_="style").get_text().split(" - ")
except AttributeError:
pass
except ValueError:
pass
return styles
def get_abv_from_soup(soup) -> Optional[float]:
abv = None
try:
abv = soup.find(class_="abv").get_text()
if abv:
abv = float(abv.strip("\n").strip("% ABV").strip())
except AttributeError:
pass
except ValueError:
pass
except TypeError:
pass
return abv
def get_ibu_from_soup(soup) -> Optional[int]:
ibu = None
try:
ibu = soup.find(class_="ibu").get_text()
if ibu:
ibu = int(ibu.strip("\n").strip(" IBU").strip())
except AttributeError:
pass
except ValueError:
ibu = None
return ibu
def get_rating_from_soup(soup) -> str:
rating = ""
try:
rating = float(
soup.find(class_="num").get_text().strip("(").strip(")")
)
except AttributeError:
rating = None
except ValueError:
rating = None
return rating
def get_producer_id_from_soup(soup) -> str:
id = ""
try:
id = soup.find(class_="brewery").find("a")["href"].strip("/")
except ValueError:
pass
except IndexError:
pass
return id
def get_producer_name_from_soup(soup) -> str:
name = ""
try:
name = soup.find(class_="brewery").find("a").get_text()
except AttributeError:
pass
except ValueError:
pass
return name
def get_beer_from_untappd_id(untappd_id: str) -> dict:
beer_url = UNTAPPD_URL.format(id=untappd_id)
headers = {"User-Agent": "Vrobbler 0.11.12"}
response = requests.get(beer_url, headers=headers)
beer_dict = {"untappd_id": untappd_id}
if response.status_code != 200:
logger.warn(
"Bad response from untappd.com", extra={"response": response}
)
return beer_dict
soup = BeautifulSoup(response.text, "html.parser")
beer_dict["title"] = get_title_from_soup(soup)
beer_dict["description"] = get_description_from_soup(soup)
beer_dict["styles"] = get_styles_from_soup(soup)
beer_dict["abv"] = get_abv_from_soup(soup)
beer_dict["ibu"] = get_ibu_from_soup(soup)
beer_dict["untappd_rating"] = get_rating_from_soup(soup)
beer_dict["producer__untappd_id"] = get_producer_id_from_soup(soup)
beer_dict["producer__name"] = get_producer_name_from_soup(soup)
return beer_dict

View File

@ -0,0 +1,14 @@
from django.urls import path
from beers import views
app_name = "beers"
urlpatterns = [
path("beers/", views.BeerListView.as_view(), name="beer_list"),
path(
"beers/<slug:slug>/",
views.BeerDetailView.as_view(),
name="beer_detail",
),
]

View File

@ -0,0 +1,11 @@
from beers.models import Beer
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
class BeerListView(ScrobbleableListView):
model = Beer
class BeerDetailView(ScrobbleableDetailView):
model = Beer

View File

@ -1,6 +1,11 @@
from django.contrib import admin
from boardgames.models import BoardGame, BoardGamePublisher
from boardgames.models import (
BoardGame,
BoardGameLocation,
BoardGamePublisher,
BoardGameDesigner,
)
from scrobbles.admin import ScrobbleInline
@ -15,13 +20,34 @@ class BoardGamePublisherAdmin(admin.ModelAdmin):
ordering = ("-created",)
@admin.register(BoardGameDesigner)
class BoardGameDesignerAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"name",
"uuid",
)
ordering = ("-created",)
@admin.register(BoardGameLocation)
class BoardGameLocationAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"name",
"uuid",
"geo_location",
)
ordering = ("-created",)
@admin.register(BoardGame)
class GameAdmin(admin.ModelAdmin):
class BoardGameAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"bggeek_id",
"title",
"published_date",
"published_year",
)
search_fields = ("title",)
ordering = ("-created",)

View File

@ -1,8 +1,15 @@
import csv
import json
import logging
from typing import Optional
from typing import TYPE_CHECKING, Optional
import requests
from bs4 import BeautifulSoup
from django.contrib.auth import get_user_model
User = get_user_model()
if TYPE_CHECKING:
from scrobbles.models import Scrobble
logger = logging.getLogger(__name__)
@ -90,3 +97,60 @@ def lookup_boardgame_from_bgg(lookup_id: str) -> dict:
}
return game_dict
def push_scrobble_to_bgg(scrobble: "Scrobble", user: User) -> Optional[bool]:
bgg_username = "secstate" # user.profile.bgg_username
bgg_password = "yYFCKnfo8AK89lc68q0S"
if not bgg_username or bgg_password:
return
login_payload = {
"credentials": {"username": bgg_username, "password": bgg_password}
}
headers = {"content-type": "application/json"}
# TODO Look up past plays for scrobble.media_obj.bggeek_id, and make sure we haven't scrobbled this before
with requests.Session() as s:
p = s.post(
"https://boardgamegeek.com/login/api/v1",
data=json.dumps(login_payload),
headers=headers,
)
print(p)
players = []
if scrobble.log:
for player in scrobble.log.get("players"):
player_person = Person.objects.filter(
id=player.get("person_id")
).first()
if player_person.get("bgg_username"):
player["username"] = player_person.get("bgg_username")
player["name"] = player_person.get("name")
player["win"] = player.get("win")
# player["role"] = player.get("role")
player["new"] = player.get("new")
player["score"] = player.get("score")
players.append(player)
play_payload = {
"playdate": scrobble.timestamp.date.strftime("%Y-%m-%d"),
"length": scrobble.playback_position_seconds / 60,
"comments": "Uploaded from Vrobbler",
"location": scrobble.log.location or None,
"objectid": scrobble.media_obj.bggeek_id,
"quantity": "1",
"action": "save",
"players": players,
"objecttype": "thing",
"ajax": 1,
}
r = s.post(
"https://boardgamegeek.com/geekplay.php",
data=json.dumps(play_payload),
headers=headers,
)
print(r)

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("boardgames", "0006_alter_boardgame_genre"),
]
operations = [
migrations.AlterField(
model_name="boardgame",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,167 @@
# Generated by Django 4.2.19 on 2025-07-03 01:57
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
import uuid
class Migration(migrations.Migration):
dependencies = [
("locations", "0007_alter_geolocation_run_time_seconds"),
("boardgames", "0007_alter_boardgame_run_time_seconds"),
]
operations = [
migrations.CreateModel(
name="BoardGameDesigner",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("name", models.CharField(max_length=255)),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
("bgg_id", models.IntegerField(blank=True, null=True)),
("bio", models.TextField(blank=True, null=True)),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
migrations.RenameField(
model_name="boardgamepublisher",
old_name="igdb_id",
new_name="bgg_id",
),
migrations.AddField(
model_name="boardgame",
name="bgstats_id",
field=models.UUIDField(blank=True, null=True),
),
migrations.AddField(
model_name="boardgame",
name="cooperative",
field=models.BooleanField(default=False),
),
migrations.AddField(
model_name="boardgame",
name="expansion_for_boardgame",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="boardgames.boardgame",
),
),
migrations.AddField(
model_name="boardgame",
name="highest_wins",
field=models.BooleanField(default=True),
),
migrations.AddField(
model_name="boardgame",
name="max_play_time",
field=models.IntegerField(blank=True, null=True),
),
migrations.AddField(
model_name="boardgame",
name="min_play_time",
field=models.IntegerField(blank=True, null=True),
),
migrations.AddField(
model_name="boardgame",
name="no_points",
field=models.BooleanField(default=False),
),
migrations.AddField(
model_name="boardgame",
name="uses_teams",
field=models.BooleanField(default=False),
),
migrations.CreateModel(
name="BoardGameLocation",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("name", models.CharField(max_length=255)),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
("bgstats_id", models.UUIDField(blank=True, null=True)),
("description", models.TextField(blank=True, null=True)),
(
"geo_location",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="locations.geolocation",
),
),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
migrations.AddField(
model_name="boardgame",
name="designers",
field=models.ManyToManyField(
related_name="board_games", to="boardgames.boardgamedesigner"
),
),
]

View File

@ -0,0 +1,33 @@
# Generated by Django 4.2.19 on 2025-07-03 02:21
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("boardgames", "0008_boardgamedesigner_and_more"),
]
operations = [
migrations.AlterField(
model_name="boardgame",
name="cooperative",
field=models.BooleanField(blank=True, default=False, null=True),
),
migrations.AlterField(
model_name="boardgame",
name="highest_wins",
field=models.BooleanField(blank=True, default=True, null=True),
),
migrations.AlterField(
model_name="boardgame",
name="no_points",
field=models.BooleanField(blank=True, default=False, null=True),
),
migrations.AlterField(
model_name="boardgame",
name="uses_teams",
field=models.BooleanField(blank=True, default=False, null=True),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.19 on 2025-07-03 04:05
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("boardgames", "0009_alter_boardgame_cooperative_and_more"),
]
operations = [
migrations.AddField(
model_name="boardgame",
name="published_year",
field=models.IntegerField(blank=True, null=True),
),
]

View File

@ -13,7 +13,8 @@ from django_extensions.db.models import TimeStampedModel
from imagekit.models import ImageSpecField
from imagekit.processors import ResizeToFit
from scrobbles.dataclasses import BoardGameLogData
from scrobbles.mixins import ScrobblableMixin
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
from locations.models import GeoLocation
logger = logging.getLogger(__name__)
BNULL = {"blank": True, "null": True}
@ -23,7 +24,7 @@ class BoardGamePublisher(TimeStampedModel):
name = models.CharField(max_length=255)
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
logo = models.ImageField(upload_to="games/platform-logos/", **BNULL)
igdb_id = models.IntegerField(**BNULL)
bgg_id = models.IntegerField(**BNULL)
def __str__(self):
return self.name
@ -34,6 +35,39 @@ class BoardGamePublisher(TimeStampedModel):
)
class BoardGameDesigner(TimeStampedModel):
name = models.CharField(max_length=255)
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
bgg_id = models.IntegerField(**BNULL)
bio = models.TextField(**BNULL)
def __str__(self) -> str:
return str(self.name)
def get_absolute_url(self):
return reverse(
"boardgames:designer_detail", kwargs={"slug": self.uuid}
)
class BoardGameLocation(TimeStampedModel):
name = models.CharField(max_length=255)
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
bgstats_id = models.UUIDField(**BNULL)
description = models.TextField(**BNULL)
geo_location = models.ForeignKey(
GeoLocation, **BNULL, on_delete=models.DO_NOTHING
)
def __str__(self) -> str:
return str(self.name)
def get_absolute_url(self):
return reverse(
"boardgames:location_detail", kwargs={"slug": self.uuid}
)
class BoardGame(ScrobblableMixin):
COMPLETION_PERCENT = getattr(
settings, "BOARD_GAME_COMPLETION_PERCENT", 100
@ -53,6 +87,10 @@ class BoardGame(ScrobblableMixin):
publisher = models.ForeignKey(
BoardGamePublisher, **BNULL, on_delete=models.DO_NOTHING
)
designers = models.ManyToManyField(
BoardGameDesigner,
related_name="board_games",
)
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
description = models.TextField(**BNULL)
cover = models.ImageField(upload_to="boardgames/covers/", **BNULL)
@ -85,8 +123,19 @@ class BoardGame(ScrobblableMixin):
max_players = models.PositiveSmallIntegerField(**BNULL)
min_players = models.PositiveSmallIntegerField(**BNULL)
published_date = models.DateField(**BNULL)
published_year = models.IntegerField(**BNULL)
recommended_age = models.PositiveSmallIntegerField(**BNULL)
bggeek_id = models.CharField(max_length=255, **BNULL)
bgstats_id = models.UUIDField(**BNULL)
uses_teams = models.BooleanField(default=False, **BNULL)
cooperative = models.BooleanField(default=False, **BNULL)
highest_wins = models.BooleanField(default=True, **BNULL)
no_points = models.BooleanField(default=False, **BNULL)
min_play_time = models.IntegerField(**BNULL)
max_play_time = models.IntegerField(**BNULL)
expansion_for_boardgame = models.ForeignKey(
"self", **BNULL, on_delete=models.DO_NOTHING
)
def __str__(self):
return self.title
@ -100,6 +149,10 @@ class BoardGame(ScrobblableMixin):
def logdata_cls(self):
return BoardGameLogData
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Playing", tags="game_die")
def primary_image_url(self) -> str:
url = ""
if self.cover:
@ -112,9 +165,6 @@ class BoardGame(ScrobblableMixin):
link = f"https://boardgamegeek.com/boardgame/{self.bggeek_id}"
return link
def get_start_url(self):
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
def fix_metadata(self, data: dict = {}, force_update=False) -> None:
if not self.published_date or force_update:
@ -127,7 +177,12 @@ class BoardGame(ScrobblableMixin):
publisher_name = data.pop("publisher_name")
if year:
data["published_date"] = datetime(int(year), 1, 1)
data["published_year"] = int(year)
if not data["min_players"]:
data.pop("min_players")
if not data["min_players"]:
data.pop("max_players")
# Fun trick for updating all fields at once
BoardGame.objects.filter(pk=self.id).update(**data)

View File

@ -0,0 +1,128 @@
import berserk
from boardgames.models import BoardGame
from django.conf import settings
from django.contrib.auth import get_user_model
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
User = get_user_model()
def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
user = User.objects.get(id=user_id)
client = berserk.Client(
session=berserk.TokenSession(settings.LICHESS_API_KEY)
)
games = client.games.export_by_player(user.profile.lichess_username)
for game_dict in games:
chess, created = BoardGame.objects.get_or_create(title="Chess")
if created:
chess.run_time_seconds = 1800
chess.bggeek_id = 171
chess.save(update_fields=["run_time_seconds", "bggeek_id"])
scrobble = Scrobble.objects.filter(
user_id=user.id,
timestamp=game_dict.get("createdAt"),
board_game_id=chess.id,
).first()
if scrobble:
continue
log_data = {
"variant": game_dict.get("variant"),
"lichess_id": game_dict.get("id"),
"rated": game_dict.get("rated"),
"speed": game_dict.get("speed"),
"moves": game_dict.get("moves"),
"players": [],
}
winner = game_dict.get("winner")
black_player = game_dict.get("players", {}).get("black", {})
white_player = game_dict.get("players", {}).get("white", {})
user_player = {
"user_id": user.id,
"lichess_username": user.profile.lichess_username,
"bgg_username": user.profile.bgg_username,
"color": "",
"win": False,
}
other_player = {"name_str": "", "color": "", "win": False}
if (
black_player.get("user", {}).get("name", "")
== user.profile.lichess_username
):
user_player["color"] = "black"
if "aiLevel" in white_player.keys():
other_player["name_str"] = "aiLevel_" + str(
white_player.get("aiLevel", "")
)
else:
other_player["name_str"] = white_player.get("user", {}).get(
"name", ""
)
other_player["lichess_username"] = other_player["name_str"]
other_player["color"] = "white"
if winner == "black":
user_player["win"] = True
else:
other_player["win"] = True
if (
white_player.get("user", {}).get("name", "")
== user.profile.lichess_username
):
user_player["color"] = "white"
if "aiLevel" in black_player.keys():
other_player["name_str"] = "aiLevel_" + str(
black_player.get("aiLevel", "")
)
else:
other_player["name_str"] = black_player.get("user", {}).get(
"name", ""
)
other_player["lichess_username"] = other_player["name_str"]
other_player["color"] = "black"
if winner == "white":
user_player["win"] = True
else:
other_player["win"] = True
log_data["players"].append(user_player)
log_data["players"].append(other_player)
scrobble_dict = {
"media_type": Scrobble.MediaType.BOARD_GAME,
"user_id": user.id,
"playback_position_seconds": (
game_dict.get("lastMoveAt") - game_dict.get("createdAt")
).seconds,
"in_progress": False,
"played_to_completion": True,
"timestamp": game_dict.get("createdAt"),
"stop_timestamp": game_dict.get("lastMoveAt"),
"board_game_id": chess.id,
"source": "Lichess",
"timezone": user.profile.timezone,
"log": log_data,
}
if commit:
Scrobble.objects.create(**scrobble_dict)
return scrobble_dict
def import_chess_games_for_all_users():
scrobbles_to_create = []
for user in User.objects.filter(profile__lichess_username__isnull=False):
scrobble_dict = import_chess_games_for_user_id(user.id)
scrobbles_to_create.append(Scrobble(**scrobble_dict))
if scrobbles_to_create:
created = Scrobble.objects.bulk_create(scrobbles_to_create)
for scrobble in created:
ScrobbleNtfyNotification(scrobble).send()
return scrobbles_to_create

View File

@ -6,10 +6,12 @@ app_name = "boardgames"
urlpatterns = [
path(
"board-game/", views.BoardGameListView.as_view(), name="boardgame_list"
"board-games/",
views.BoardGameListView.as_view(),
name="boardgame_list",
),
path(
"board-game/<slug:slug>/",
"board-games/<slug:slug>/",
views.BoardGameDetailView.as_view(),
name="boardgame_detail",
),

View File

@ -1,15 +1,14 @@
from django.views import generic
from boardgames.models import BoardGame, BoardGamePublisher
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
class BoardGameListView(generic.ListView):
class BoardGameListView(ScrobbleableListView):
model = BoardGame
paginate_by = 20
class BoardGameDetailView(generic.DetailView):
class BoardGameDetailView(ScrobbleableDetailView):
model = BoardGame
slug_field = "uuid"
class BoardGamePublisherDetailView(generic.DetailView):

View File

@ -1,7 +1,5 @@
from books.models import Author, Book, Paper
from django.contrib import admin
from books.models import Author, Book, Page
from scrobbles.admin import ScrobbleInline
@ -18,22 +16,32 @@ class AuthorAdmin(admin.ModelAdmin):
search_fields = ("name",)
@admin.register(Page)
class PageAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_filter = ("book",)
ordering = ("book", "number")
@admin.register(Book)
class BookAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
"isbn",
"subtitle",
"isbn_13",
"first_publish_year",
"pages",
)
search_fields = ("name",)
ordering = ("-created",)
inlines = [
ScrobbleInline,
]
@admin.register(Paper)
class BookAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
"subtitle",
"arxiv_id",
"first_publish_year",
"pages",
"openlibrary_id",
)
search_fields = ("name",)
ordering = ("-created",)

View File

@ -0,0 +1,7 @@
#!/usr/bin/env python3
BOOKS_TITLES_TO_IGNORE = [
"KOReader Quickstart Guide",
"zb2rhkSwygt9vjkAEBj7tP5KVgFqejJqsJ2W3bYsrgiiKK8XL",
"zb2rhchGpo7P27mofV9hYjT63d9ZaQnbQ6LSfzmkvsYzvARif",
]

View File

@ -1,17 +1,17 @@
from collections import OrderedDict
import logging
import re
import sqlite3
from datetime import datetime, timedelta
from enum import Enum
from zoneinfo import ZoneInfo
import pytz
import requests
from books.models import Author, Book
from books.openlibrary import get_author_openlibrary_id
from books.constants import BOOKS_TITLES_TO_IGNORE
from django.apps import apps
from django.contrib.auth import get_user_model
from scrobbles.notifications import ScrobbleNtfyNotification
from stream_sqlite import stream_sqlite
from webdav.client import get_webdav_client
logger = logging.getLogger(__name__)
User = get_user_model()
@ -62,6 +62,8 @@ def lookup_or_create_authors_from_author_str(ko_author_str: str) -> list:
"""Takes a string of authors from KoReader and returns a list
of Authors from our database
"""
from books.models import Author
author_str_list = ko_author_str.split(", ")
author_list = []
for author_str in author_str_list:
@ -73,36 +75,49 @@ def lookup_or_create_authors_from_author_str(ko_author_str: str) -> list:
author = Author.objects.filter(name=author_str).first()
if not author:
author = Author.objects.create(
name=author_str,
openlibrary_id=get_author_openlibrary_id(author_str),
)
author.fix_metadata()
author = Author.objects.create(name=author_str)
# TODO Move these to async processes after importing
# author.fix_metadata()
logger.debug(f"Created author {author}")
author_list.append(author)
return author_list
def create_book_from_row(row: list):
from books.models import Book
# No KoReader book yet, create it
author_str = get_author_str_from_row(row)
author_str = get_author_str_from_row(row).replace("\x00", "")
total_pages = row[KoReaderBookColumn.PAGES.value]
run_time = total_pages * Book.AVG_PAGE_READING_SECONDS
book_title = row[KoReaderBookColumn.TITLE.value].replace("\x00", "")
if " - " in book_title:
split_title = book_title.split(" - ")
book_title = split_title[0]
if (not author_str or author_str == "N/A") and len(split_title) > 1:
author_str = split_title[1].split("_")[0]
clean_row = []
for value in row:
if isinstance(value, str):
value = value.replace("\x00", "")
clean_row.append(value)
book = Book.objects.create(
title=row[KoReaderBookColumn.TITLE.value],
title=book_title.replace("_", ":"),
pages=total_pages,
koreader_data_by_hash={
row[KoReaderBookColumn.MD5.value]: {
"title": row[KoReaderBookColumn.TITLE.value],
str(row[KoReaderBookColumn.MD5.value]): {
"title": book_title,
"author_str": author_str,
"book_id": row[KoReaderBookColumn.ID.value],
"pages": total_pages,
"raw_row_data": clean_row,
}
},
run_time_seconds=run_time,
)
book.fix_metadata()
# TODO Move these to async processes after importing
# book.fix_metadata()
# Add authors
author_list = lookup_or_create_authors_from_author_str(author_str)
@ -119,15 +134,16 @@ def build_book_map(rows) -> dict:
primary key IDs for page creation.
"""
from books.models import Book
book_id_map = {}
for book_row in rows:
if (
book_row[KoReaderBookColumn.TITLE.value]
== "KOReader Quickstart Guide"
):
if book_row[KoReaderBookColumn.TITLE.value] in BOOKS_TITLES_TO_IGNORE:
logger.info(
"Ignoring the KOReader quickstart guide. No on wants that."
"[build_book_map] Ignoring book title that is likely garbage",
extra={"book_row": book_row, "media_type": "Book"},
)
continue
book = Book.objects.filter(
@ -136,6 +152,15 @@ def build_book_map(rows) -> dict:
]
).first()
if not book:
title = (
book_row[KoReaderBookColumn.TITLE.value]
.split(" - ")[0]
.lower()
.replace("\x00", "")
)
book = Book.objects.filter(title=title).first()
if not book:
book = create_book_from_row(book_row)
@ -233,7 +258,7 @@ def build_scrobbles_from_book_map(
should_create_scrobble = True
if should_create_scrobble:
scrobble_page_data = OrderedDict(
scrobble_page_data = dict(
sorted(
scrobble_page_data.items(),
key=lambda x: x[1]["start_ts"],
@ -253,35 +278,21 @@ def build_scrobbles_from_book_map(
)
continue
timezone = user.profile.timezone
timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(first_page.get("start_ts")))
)
stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(last_page.get("end_ts")))
)
timestamp = datetime.fromtimestamp(
int(first_page.get("start_ts"))
).replace(tzinfo=pytz.timezone(timezone))
# Add a shim here temporarily to fix imports while we were in France
# if date is between 10/15 and 12/15, cast it to Europe/Central
if (
datetime(2023, 10, 15).replace(
tzinfo=pytz.timezone("Europe/Paris")
)
<= timestamp
<= datetime(2023, 12, 15).replace(
tzinfo=pytz.timezone("Europe/Paris")
)
):
timezone = "Europe/Paris"
stop_timestamp = datetime.fromtimestamp(
int(last_page.get("end_ts"))
).replace(tzinfo=pytz.timezone(timezone))
if (
timestamp.tzinfo._dst.seconds == 0
or stop_timestamp.tzinfo._dst.seconds == 0
):
# Adjust for Daylight Saving Time
if timestamp.dst() == timedelta(
0
) or stop_timestamp.dst() == timedelta(0):
timestamp = timestamp - timedelta(hours=1)
stop_timestamp = stop_timestamp - timedelta(hours=1)
else:
print("In DST! ", timestamp)
scrobble = Scrobble.objects.filter(
timestamp=timestamp,
@ -296,7 +307,7 @@ def build_scrobbles_from_book_map(
log_data = {
"koreader_hash": book_dict.get("hash"),
"page_data": scrobble_page_data,
"pages_read": cur_page_number,
"pages_read": len(scrobble_page_data.keys()),
}
scrobbles_to_create.append(
Scrobble(
@ -311,13 +322,13 @@ def build_scrobbles_from_book_map(
in_progress=False,
played_to_completion=True,
long_play_complete=False,
timezone=timezone,
timezone=timestamp.tzinfo.name,
)
)
# Then start over
should_create_scrobble = False
playback_position_seconds = 0
scrobble_page_data = {}
# Then start over
should_create_scrobble = False
playback_position_seconds = 0
scrobble_page_data = {}
# We accumulate pages for the scrobble until we should create a new one
scrobble_page_data[cur_page_number] = stats
@ -353,12 +364,13 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
new_scrobbles = []
user = User.objects.filter(id=user_id).first()
tz = pytz.utc
tz = ZoneInfo("UTC")
if user:
tz = user.profile.timezone
tz = user.profile.tzinfo
is_os_file = "https://" not in file_path
if is_os_file:
# Loading sqlite file from local filesystem
con = sqlite3.connect(file_path)
cur = con.cursor()
try:
@ -376,6 +388,8 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
else:
# Streaming the sqlite file off S3
book_map = {}
for table_name, pragma_table_info, rows in stream_sqlite(
_sqlite_bytes(file_path), max_buffer_size=1_048_576
):
@ -383,6 +397,9 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
if table_name == "book":
book_map = build_book_map(rows)
for table_name, pragma_table_info, rows in stream_sqlite(
_sqlite_bytes(file_path), max_buffer_size=1_048_576
):
if table_name == "page_stat_data":
book_map = build_page_data(rows, book_map, tz)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
@ -391,9 +408,27 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
created = []
if new_scrobbles:
created = Scrobble.objects.bulk_create(new_scrobbles)
if created:
ScrobbleNtfyNotification(created[-1]).send()
fix_long_play_stats_for_scrobbles(created)
logger.info(
f"Created {len(created)} scrobbles",
extra={"created_scrobbles": created},
)
return created
def fetch_file_from_webdav(user_id: int) -> str:
file_path = f"/tmp/{user_id}-koreader-import.sqlite3"
client = get_webdav_client(user_id)
if not client:
logger.warning("could not get webdav client for user")
# TODO maybe we raise an exception here?
return ""
client.download_sync(
remote_path="var/koreader/statistics.sqlite3",
local_path=file_path,
)
return file_path

View File

@ -1,6 +1,5 @@
#!/usr/bin/env python3
from enum import Enum
from typing import Optional
from bs4 import BeautifulSoup
import requests

View File

@ -0,0 +1,40 @@
from enum import Enum
from typing import Optional
import pendulum
YOUTUBE_VIDEO_URL = "https://www.youtube.com/watch?v="
IMDB_VIDEO_URL = "https://www.imdb.com/title/tt"
class BookType:
...
class BookMetadata:
title: str
run_time_seconds: Optional[int]
authors = Optional[str]
goodreads_id = Optional[str]
koreader_data_by_hash = Optional[dict]
isbn = Optional[str]
# isbn_13 = Optional[str]
# isbn_10 = Optional[str]
pages = Optional[int]
language = Optional[str]
first_publish_year = Optional[int]
summary = Optional[str]
# General
cover_url: Optional[str]
genres: list[str]
def __init__(self, title: Optional[str] = ""):
self.title = title
def as_dict_with_authors_cover_and_genres(self) -> tuple:
book_dict = vars(self)
authors = book_dict.pop("authors")
cover = book_dict.pop("cover_url")
genres = book_dict.pop("genres")
return book_dict, authors, cover, genres

View File

@ -0,0 +1,25 @@
# Generated by Django 4.2.16 on 2024-10-17 20:08
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("books", "0020_author_comicvine_data_book_comicvine_data_and_more"),
]
operations = [
migrations.RemoveField(
model_name="book",
name="koreader_authors",
),
migrations.RemoveField(
model_name="book",
name="koreader_id",
),
migrations.RemoveField(
model_name="book",
name="koreader_md5",
),
]

View File

@ -0,0 +1,21 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
(
"books",
"0021_remove_book_koreader_authors_remove_book_koreader_id_and_more",
),
]
operations = [
migrations.AlterField(
model_name="book",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,40 @@
# Generated by Django 4.2.18 on 2025-01-27 04:20
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("books", "0022_alter_book_run_time_seconds"),
]
operations = [
migrations.RenameField(
model_name="book",
old_name="isbn",
new_name="isbn_13",
),
migrations.RemoveField(
model_name="book",
name="comicvine_data",
),
migrations.RemoveField(
model_name="book",
name="goodreads_id",
),
migrations.RemoveField(
model_name="book",
name="locg_slug",
),
migrations.AddField(
model_name="book",
name="isbn_10",
field=models.CharField(blank=True, max_length=255, null=True),
),
migrations.AddField(
model_name="book",
name="publish_date",
field=models.DateField(blank=True, null=True),
),
]

View File

@ -0,0 +1,21 @@
# Generated by Django 4.2.18 on 2025-01-27 04:31
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
(
"books",
"0023_rename_isbn_book_isbn_13_remove_book_comicvine_data_and_more",
),
]
operations = [
migrations.AddField(
model_name="book",
name="publisher",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -0,0 +1,136 @@
# Generated by Django 4.2.19 on 2025-02-18 05:11
from django.db import migrations, models
import django_extensions.db.fields
import taggit.managers
import uuid
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0067_scrobble_food_alter_scrobble_media_type"),
("books", "0024_book_publisher"),
]
operations = [
migrations.RemoveField(
model_name="author",
name="amazon_id",
),
migrations.RemoveField(
model_name="author",
name="librarything_id",
),
migrations.RemoveField(
model_name="author",
name="locg_slug",
),
migrations.AddField(
model_name="author",
name="semantic_id",
field=models.CharField(blank=True, max_length=50, null=True),
),
migrations.CreateModel(
name="Paper",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
("run_time_seconds", models.IntegerField(default=900)),
(
"run_time_ticks",
models.PositiveBigIntegerField(blank=True, null=True),
),
("title", models.CharField(max_length=255)),
("semantic_title", models.CharField(max_length=255)),
(
"koreader_data_by_hash",
models.JSONField(blank=True, null=True),
),
(
"semantic_id",
models.CharField(blank=True, max_length=50, null=True),
),
(
"arxiv_id",
models.CharField(blank=True, max_length=50, null=True),
),
(
"corpus_id",
models.CharField(blank=True, max_length=50, null=True),
),
(
"doi_id",
models.CharField(blank=True, max_length=50, null=True),
),
("pages", models.IntegerField(blank=True, null=True)),
(
"language",
models.CharField(blank=True, max_length=4, null=True),
),
(
"first_publish_year",
models.IntegerField(blank=True, null=True),
),
("publish_date", models.DateField(blank=True, null=True)),
(
"journal",
models.CharField(blank=True, max_length=255, null=True),
),
(
"journal_volume",
models.CharField(blank=True, max_length=50, null=True),
),
("abstract", models.TextField(blank=True, null=True)),
("num_citations", models.IntegerField(blank=True, null=True)),
(
"openaccess_pdf_url",
models.CharField(blank=True, max_length=255, null=True),
),
(
"authors",
models.ManyToManyField(blank=True, to="books.author"),
),
(
"genre",
taggit.managers.TaggableManager(
help_text="A comma-separated list of tags.",
through="scrobbles.ObjectWithGenres",
to="scrobbles.Genre",
verbose_name="Tags",
),
),
],
options={
"abstract": False,
},
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.19 on 2025-02-18 05:28
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("books", "0025_remove_author_amazon_id_and_more"),
]
operations = [
migrations.AlterField(
model_name="paper",
name="semantic_title",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -0,0 +1,22 @@
# Generated by Django 4.2.19 on 2025-02-18 05:33
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("books", "0026_alter_paper_semantic_title"),
]
operations = [
migrations.RemoveField(
model_name="paper",
name="num_citations",
),
migrations.AddField(
model_name="paper",
name="tldr",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -0,0 +1,16 @@
# Generated by Django 4.2.19 on 2025-04-07 17:16
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("books", "0027_remove_paper_num_citations_paper_tldr"),
]
operations = [
migrations.DeleteModel(
name="Page",
),
]

View File

@ -19,6 +19,7 @@ from imagekit.processors import ResizeToFit
from scrobbles.mixins import (
LongPlayScrobblableMixin,
ObjectWithGenres,
ScrobblableConstants,
ScrobblableMixin,
)
from scrobbles.utils import get_scrobbles_for_media
@ -34,6 +35,9 @@ from vrobbler.apps.books.locg import (
lookup_comic_from_locg,
lookup_comic_writer_by_locg_slug,
)
from vrobbler.apps.books.sources.google import lookup_book_from_google
from vrobbler.apps.books.sources.semantic import lookup_paper_from_semantic
from vrobbler.apps.scrobbles.dataclasses import BookLogData
COMICVINE_API_KEY = getattr(settings, "COMICVINE_API_KEY", "")
@ -61,22 +65,27 @@ class Author(TimeStampedModel):
)
bio = models.TextField(**BNULL)
wikipedia_url = models.CharField(max_length=255, **BNULL)
isni = models.CharField(max_length=255, **BNULL)
locg_slug = models.CharField(max_length=255, **BNULL)
wikidata_id = models.CharField(max_length=255, **BNULL)
isni = models.CharField(max_length=255, **BNULL)
goodreads_id = models.CharField(max_length=255, **BNULL)
librarything_id = models.CharField(max_length=255, **BNULL)
comicvine_data = models.JSONField(**BNULL)
amazon_id = models.CharField(max_length=255, **BNULL)
semantic_id = models.CharField(max_length=50, **BNULL)
def __str__(self):
return f"{self.name}"
def fix_metadata(self, data_dict: dict = {}):
if not data_dict and self.openlibrary_id:
data_dict = lookup_author_from_openlibrary(self.openlibrary_id)
def enrich_from_semantic(self, overwrite=False):
...
def enrich_from_google_books(self, overwrite=False):
...
def enrich_from_openlibrary(self, overwrite=False):
data_dict = lookup_author_from_openlibrary(self.openlibrary_id)
if not data_dict or not data_dict.get("name"):
logger.warning("Could not find author on openlibrary")
return
headshot_url = data_dict.pop("author_headshot_url", "")
@ -99,20 +108,16 @@ class Book(LongPlayScrobblableMixin):
title = models.CharField(max_length=255)
authors = models.ManyToManyField(Author, blank=True)
goodreads_id = models.CharField(max_length=255, **BNULL)
# All individual koreader fields are deprecated
koreader_id = models.IntegerField(**BNULL)
koreader_authors = models.CharField(max_length=255, **BNULL)
koreader_md5 = models.CharField(max_length=255, **BNULL)
koreader_data_by_hash = models.JSONField(**BNULL)
isbn = models.CharField(max_length=255, **BNULL)
isbn_13 = models.CharField(max_length=255, **BNULL)
isbn_10 = models.CharField(max_length=255, **BNULL)
pages = models.IntegerField(**BNULL)
language = models.CharField(max_length=4, **BNULL)
first_publish_year = models.IntegerField(**BNULL)
publish_date = models.DateField(**BNULL)
publisher = models.CharField(max_length=255, **BNULL)
first_sentence = models.TextField(**BNULL)
openlibrary_id = models.CharField(max_length=255, **BNULL)
locg_slug = models.CharField(max_length=255, **BNULL)
comicvine_data = models.JSONField(**BNULL)
cover = models.ImageField(upload_to="books/covers/", **BNULL)
cover_small = ImageSpecField(
source="cover",
@ -137,6 +142,14 @@ class Book(LongPlayScrobblableMixin):
def subtitle(self):
return f" by {self.author}"
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Reading", tags="book")
@property
def logdata_cls(self):
return BookLogData
@property
def primary_image_url(self) -> str:
url = ""
@ -144,12 +157,68 @@ class Book(LongPlayScrobblableMixin):
url = self.cover_medium.url
return url
def get_start_url(self):
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
def get_absolute_url(self):
return reverse("books:book_detail", kwargs={"slug": self.uuid})
@classmethod
def find_or_create(cls, title: str, enrich: bool = False, commit: bool = True):
"""Given a title, get a Book instance.
If the book is not already in our database, or overwrite is True,
this method will enrich the Book with data from Google.
By default this method will also save the data back to the model. If you'd
like to batch create, use commit=False and you'll get an unsaved but enriched
instance back which you can then save at your convenience."""
# TODO use either a Google Books id identifier or author name like for tracks
book, created = cls.objects.get_or_create(title=title)
if not created:
logger.info("Found exact match for book by title", extra={"title": title})
if not enrich:
logger.info("Found book by title, but not enriching", extra={"title": title})
return book
book_dict = lookup_book_from_google(title)
author_list = []
authors = book_dict.pop("authors")
cover_url = book_dict.pop("cover_url")
try:
genres = book_dict.pop("generes")
except:
genres = []
if authors:
for author_str in authors:
if author_str:
author, a_created = Author.objects.get_or_create(
name=author_str
)
author_list.append(author)
if a_created:
# TODO enrich author
...
for k, v in book_dict.items():
setattr(book, k, v)
if commit:
book.save()
book.save_image_from_url(cover_url)
book.genre.add(*genres)
book.authors.add(*author_list)
return book
def save_image_from_url(self, url: str, force_update: bool = False):
if not self.cover or (force_update and url):
r = requests.get(url)
if r.status_code == 200:
fname = f"{self.title}_{self.uuid}.jpg"
self.cover.save(fname, ContentFile(r.content), save=True)
def fix_metadata(self, data: dict = {}, force_update=False):
if (not self.openlibrary_id or not self.locg_slug) or force_update:
author_name = ""
@ -278,19 +347,24 @@ class Book(LongPlayScrobblableMixin):
author.headshot.save(fname, ContentFile(r.content), save=True)
self.authors.add(author)
def page_data_for_user(self, user_id: int, convert_timestamps: bool=True) -> dict:
def page_data_for_user(
self, user_id: int, convert_timestamps: bool = True
) -> dict:
scrobbles = self.scrobble_set.filter(user=user_id)
pages = {}
for scrobble in scrobbles:
if scrobble.book_page_data:
for page, data in scrobble.book_page_data.items():
if scrobble.logdata.page_data:
for page, data in scrobble.logdata.page_data.items():
if convert_timestamps:
data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
data["start_ts"] = datetime.fromtimestamp(
data["start_ts"]
)
data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
pages[page] = data
sorted_pages = OrderedDict(sorted(pages.items(), key=lambda x: x[1]["start_ts"]))
sorted_pages = OrderedDict(
sorted(pages.items(), key=lambda x: x[1]["start_ts"])
)
return sorted_pages
@ -319,112 +393,67 @@ class Book(LongPlayScrobblableMixin):
progress = int((last_scrobble.last_page_read / self.pages) * 100)
return progress
class Paper(LongPlayScrobblableMixin):
"""Keeps track of Academic Papers"""
COMPLETION_PERCENT = getattr(settings, "PAPER_COMPLETION_PERCENT", 60)
AVG_PAGE_READING_SECONDS = getattr(
settings, "AVERAGE_PAGE_READING_SECONDS", 60
)
title = models.CharField(max_length=255)
semantic_title = models.CharField(max_length=255, **BNULL)
authors = models.ManyToManyField(Author, blank=True)
koreader_data_by_hash = models.JSONField(**BNULL)
arxiv_id = models.CharField(max_length=50, **BNULL)
semantic_id = models.CharField(max_length=50, **BNULL)
arxiv_id = models.CharField(max_length=50, **BNULL)
corpus_id = models.CharField(max_length=50, **BNULL)
doi_id = models.CharField(max_length=50, **BNULL)
pages = models.IntegerField(**BNULL)
language = models.CharField(max_length=4, **BNULL)
first_publish_year = models.IntegerField(**BNULL)
publish_date = models.DateField(**BNULL)
journal = models.CharField(max_length=255, **BNULL)
journal_volume = models.CharField(max_length=50, **BNULL)
abstract = models.TextField(**BNULL)
tldr = models.CharField(max_length=255, **BNULL)
openaccess_pdf_url = models.CharField(max_length=255, **BNULL)
genre = TaggableManager(through=ObjectWithGenres)
@classmethod
def find_or_create(cls, lookup_id: str, author: str = "") -> "Book":
book = cls.objects.filter(openlibrary_id=lookup_id).first()
def get_from_semantic(cls, title: str, overwrite: bool = False) -> "Paper":
paper, created = cls.objects.get_or_create(title=title)
if not created and not overwrite:
return paper
if not book:
data = lookup_book_from_openlibrary(lookup_id, author)
paper_dict = lookup_paper_from_semantic(title)
if not data:
logger.error(
f"No book found on openlibrary, or in our database for {lookup_id}"
)
return book
if created or overwrite:
author_list = []
author_dicts = paper_dict.pop("author_dicts")
if author_dicts:
for author_dict in author_dicts:
if author_dict.get("authorId"):
author, a_created = Author.objects.get_or_create(
semantic_id=author_dict.get("authorId")
)
author_list.append(author)
if a_created:
author.name = author_dict.get("name")
author.save()
# TODO enrich author?
...
book, book_created = cls.objects.get_or_create(isbn=data["isbn"])
if book_created:
book.fix_metadata(data=data)
for k, v in paper_dict.items():
setattr(paper, k, v)
paper.save()
return book
class Page(TimeStampedModel):
"""DEPRECATED, we need to migrate pages into page_data on scrobbles and move on"""
book = models.ForeignKey(Book, on_delete=models.CASCADE)
number = models.IntegerField()
user = models.ForeignKey(User, on_delete=models.CASCADE)
start_time = models.DateTimeField(**BNULL)
end_time = models.DateTimeField(**BNULL)
duration_seconds = models.IntegerField(**BNULL)
class Meta:
unique_together = (
"book",
"number",
)
def __str__(self):
return f"Page {self.number} of {self.book.pages} in {self.book.title}"
def save(self, *args, **kwargs):
if not self.end_time and self.duration_seconds:
self._set_end_time()
return super(Page, self).save(*args, **kwargs)
@property
def next(self):
page = self.book.page_set.filter(number=self.number + 1).first()
if not page:
page = (
self.book.page_set.filter(created__gt=self.created)
.order_by("created")
.first()
)
return page
@property
def previous(self):
page = self.book.page_set.filter(number=self.number - 1).first()
if not page:
page = (
self.book.page_set.filter(created__lt=self.created)
.order_by("-created")
.first()
)
return page
@property
def seconds_to_next_page(self) -> int:
seconds = 999999 # Effectively infnity time as we have no next
if not self.end_time:
self._set_end_time()
if self.next:
seconds = (self.next.start_time - self.end_time).seconds
return seconds
@property
def is_scrobblable(self) -> bool:
"""A page defines the start of a scrobble if the seconds to next page
are greater than an hour, or 3600 seconds, and it's not a single page,
so the next seconds to next_page is less than an hour as well.
As a special case, the first recorded page is a scrobble, so we establish
when the book was started.
"""
is_scrobblable = False
over_an_hour_since_last_page = False
if not self.previous:
is_scrobblable = True
if self.previous:
over_an_hour_since_last_page = (
self.previous.seconds_to_next_page >= 3600
)
blip = self.seconds_to_next_page >= 3600
if over_an_hour_since_last_page and not blip:
is_scrobblable = True
return is_scrobblable
def _set_end_time(self) -> None:
if self.end_time:
return
self.end_time = self.start_time + timedelta(
seconds=self.duration_seconds
)
self.save(update_fields=["end_time"])
if author_list:
paper.authors.add(*author_list)
genres = paper_dict.pop("genres", [])
if genres:
paper.genre.add(*genres)
return paper

View File

View File

@ -0,0 +1,71 @@
import json
import logging
import pendulum
import requests
from django.conf import settings
API_KEY = settings.GOOGLE_API_KEY
GOOGLE_BOOKS_URL = (
"https://www.googleapis.com/books/v1/volumes?q=\"{title}\"&key={key}"
)
logger = logging.getLogger(__name__)
def lookup_book_from_google(title: str) -> dict:
book_dict = {"title": title}
url = GOOGLE_BOOKS_URL.format(title=title, key=API_KEY)
headers = {"User-Agent": "Vrobbler 0.11.12"}
response = requests.get(url, headers=headers)
if response.status_code != 200:
logger.warning(
"Bad response from Google", extra={"response": response}
)
return book_dict
google_result = (
json.loads(response.content).get("items", [{}])[0].get("volumeInfo")
)
publish_date = pendulum.parse(google_result.get("publishedDate"))
isbn_13 = ""
isbn_10 = ""
for ident in google_result.get("industryIdentifiers", []):
if ident.get("type") == "ISBN_13":
isbn_13 = ident.get("identifier")
if ident.get("type") == "ISBN_10":
isbn_10 = ident.get("identifier")
# TODO this may lead to issues with the first get if Google changes our title
# book_metadata.title = google_result.get("title")
# if google_result.get("subtitle"):
# book_metadata["title"] = ": ".join(
# [google_result.get("title"), google_result.get("subtitle")]
# )
# book_dict["subtitle"] = google_result.get("subtitle")
book_dict["authors"] = google_result.get("authors")
book_dict["publisher"] = google_result.get("publisher")
book_dict["first_publish_year"] = publish_date.year
book_dict["pages"] = google_result.get("pageCount")
book_dict["isbn_13"] = isbn_13
book_dict["isbn_10"] = isbn_10
book_dict["publish_date"] = google_result.get("publishedDate")
if len(book_dict["publish_date"]) == 4:
book_dict["publish_date"] = f"{book_dict['publish_date']}-1-1"
book_dict["language"] = google_result.get("language")
book_dict["summary"] = google_result.get("description")
book_dict["genres"] = google_result.get("categories")
book_dict["cover_url"] = (
google_result.get("imageLinks", {})
.get("thumbnail")
.replace("zoom=1", "zoom=15")
.replace("&edge=curl", "")
)
book_dict["run_time_seconds"] = book_dict.get("pages", 10) * getattr(
settings, "AVERAGE_PAGE_READING_SECONDS", 60
)
return book_dict

View File

@ -0,0 +1,76 @@
import json
import logging
from datetime import datetime
import requests
from django.conf import settings
PAPER_SEARCH_URL = (
"https://api.semanticscholar.org/graph/v1/paper/search/match?query={}"
)
PAPER_DETAIL_URL = "https://api.semanticscholar.org/graph/v1/paper/{}?fields=title,authors,url,year,abstract,externalIds,citationCount,referenceCount,journal,fieldsOfStudy,publicationDate,openAccessPdf"
logger = logging.getLogger(__name__)
def get_api_result(url):
headers = {"User-Agent": "Vrobbler 0.11.12"}
response = requests.get(url, headers=headers)
if response.status_code != 200:
logger.warning(
"Bad response from Semantic", extra={"response": response}
)
return None
return response
def lookup_paper_from_semantic(title: str) -> dict:
paper_dict = {"title": title}
response = get_api_result(PAPER_SEARCH_URL.format(title))
if not response:
return paper_dict
semantic_id = json.loads(response.content).get("data")[0].get("paperId")
response = get_api_result(PAPER_DETAIL_URL.format(semantic_id))
result = json.loads(response.content)
if not result:
return paper_dict
page_str = result.get("journal", {}).get("pages")
if page_str:
try:
start_page = page_str.split(" - ")[0]
end_page = page_str.split(" - ")[1]
paper_dict["pages"] = int(end_page) - int(start_page)
except IndexError:
pass
paper_dict["semantic_id"] = result.get("paperId")
paper_dict["doi_id"] = result.get("externalIds", {}).get("DOI")
paper_dict["arxiv_id"] = result.get("externalIds", {}).get("ArXiv")
paper_dict["pubmed_id"] = result.get("externalIds", {}).get("PubMed")
paper_dict["corpus_id"] = result.get("externalIds", {}).get("CorpusId")
paper_dict["semantic_title"] = result.get("title")
paper_dict["first_publish_year"] = result.get("year")
paper_dict["publish_date"] = datetime.strptime(
result.get("publicationDate", "1950-01-01"), "%Y-%m-%d"
)
paper_dict["abstract"] = result.get("abstract")
paper_dict["tldr"] = result.get("bib", {}).get("abstract")
paper_dict["journal"] = result.get("journal", {}).get("name")
paper_dict["journal_volume"] = result.get("journal", {}).get("volume")
paper_dict["openaccess_pdf_url"] = result.get("openAccessPdf", {}).get(
"url"
)
paper_dict["run_time_seconds"] = paper_dict.get("pages", 10) * getattr(
settings, "AVERAGE_PAGE_READING_SECONDS", 60
)
paper_dict["author_dicts"] = result.get("authors")
paper_dict["genres"] = result.get("fieldsOfStudy")
return paper_dict

View File

@ -48,7 +48,7 @@ class KoReaderBookRows:
300 + wiggle,
self.DEFAULT_STR,
self.DEFAULT_STR,
hashlib.md5(title.encode()),
hashlib.md5(title.encode()).hexdigest(),
i * wiggle * 20,
120,
]

View File

@ -21,9 +21,10 @@ def test_build_book_map(get_mock, koreader_rows, valid_response):
@mock.patch("requests.get")
def test_load_page_data_to_map(get_mock, koreader_rows, valid_response):
get_mock.return_value = valid_response
book_map = build_book_map(koreader_rows.BOOK_ROWS)
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
book_map = build_page_data(
koreader_rows.PAGE_STATS_ROWS,
build_book_map(koreader_rows.BOOK_ROWS),
)
assert (
len(book_map[1]["pages"])
== koreader_rows.BOOK_ROWS[0][
@ -45,9 +46,9 @@ def test_build_scrobbles_from_pages(
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
expected_scrobbles = 6 * len(book_map.keys())
assert len(scrobbles) == expected_scrobbles
assert len(scrobbles[0].book_page_data.keys()) == 22
assert len(scrobbles[1].book_page_data.keys()) == 20
assert len(scrobbles[2].book_page_data.keys()) == 20
assert len(scrobbles[3].book_page_data.keys()) == 20
assert len(scrobbles[4].book_page_data.keys()) == 20
assert len(scrobbles[5].book_page_data.keys()) == 18
assert len(scrobbles[0].logdata.page_data.keys()) == 21
assert len(scrobbles[1].logdata.page_data.keys()) == 20
assert len(scrobbles[2].logdata.page_data.keys()) == 20
assert len(scrobbles[3].logdata.page_data.keys()) == 20
assert len(scrobbles[4].logdata.page_data.keys()) == 20
assert len(scrobbles[5].logdata.page_data.keys()) == 18

View File

@ -5,6 +5,7 @@ import pytest
from books.openlibrary import lookup_book_from_openlibrary
@pytest.mark.skip()
def test_lookup_modern_book():
book = lookup_book_from_openlibrary("Matrix", "Lauren Groff")
assert book.get("title") == "Matrix"
@ -12,6 +13,7 @@ def test_lookup_modern_book():
assert book.get("ol_author_id") == "OL3675729A"
@pytest.mark.skip()
def test_lookup_classic_book():
book = lookup_book_from_openlibrary(
"The Life of Castruccio Castracani", "Machiavelli"
@ -21,6 +23,7 @@ def test_lookup_classic_book():
assert book.get("ol_author_id") == "OL23135A"
@pytest.mark.skip()
def test_lookup_foreign_book():
book = lookup_book_from_openlibrary("Ravagé", "René Barjavel")
assert book.get("title") == "Ravage"

View File

@ -5,14 +5,14 @@ app_name = "books"
urlpatterns = [
path("book/", views.BookListView.as_view(), name="book_list"),
path("books/", views.BookListView.as_view(), name="book_list"),
path(
"book/<slug:slug>/",
"books/<slug:slug>/",
views.BookDetailView.as_view(),
name="book_detail",
),
path(
"author/<slug:slug>/",
"authors/<slug:slug>/",
views.AuthorDetailView.as_view(),
name="author_detail",
),

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("bricksets", "0001_initial"),
]
operations = [
migrations.AlterField(
model_name="brickset",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -16,7 +16,9 @@ class BrickSet(LongPlayScrobblableMixin):
number = models.CharField(max_length=10, **BNULL)
release_year = models.IntegerField(**BNULL)
piece_count = models.IntegerField(**BNULL)
brickset_rating = models.DecimalField(max_digits=3, decimal_places=1, **BNULL)
brickset_rating = models.DecimalField(
max_digits=3, decimal_places=1, **BNULL
)
lego_item_number = models.CharField(max_length=10, **BNULL)
box_image = models.ImageField(upload_to="brickset/boxes/", **BNULL)
box_image_small = ImageSpecField(
@ -48,13 +50,23 @@ class BrickSet(LongPlayScrobblableMixin):
def get_absolute_url(self):
return reverse("bricksets:brickset_detail", kwargs={"slug": self.uuid})
def __str__(self) -> str:
name = str(self.title)
if not self.title:
name = str(self.number)
return name
@property
def logdata_cls(self):
return BrickSetLogData
@classmethod
def find_or_create(cls, title: str) -> "BrickSet":
return cls.objects.filter(title=title).first()
def find_or_create(cls, brickset_id: str) -> "BrickSet":
brickset = cls.objects.filter(number=brickset_id).first()
if not brickset:
# TODO: enrich this from the website
brickset = cls.objects.create(number=brickset_id)
return brickset
@property
def primary_image_url(self) -> str:

View File

@ -0,0 +1,28 @@
from foods.models import Food, FoodCategory
from django.contrib import admin
from scrobbles.admin import ScrobbleInline
class FoodInline(admin.TabularInline):
model = Food
extra = 0
@admin.register(FoodCategory)
class FoodCategoryAdmin(admin.ModelAdmin):
date_hierarchy = "created"
search_fields = ("name",)
@admin.register(Food)
class FoodAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"uuid",
"title",
)
ordering = ("-created",)
search_fields = ("title",)
inlines = [
ScrobbleInline,
]

View File

@ -0,0 +1,143 @@
import json
import logging
import urllib
from typing import Optional
import requests
from bs4 import BeautifulSoup
logger = logging.getLogger(__name__)
ALLRECIPE_URL = "https://allrecipe.com/{id}"
def get_first(key: str, result: dict) -> str:
obj = ""
if obj_list := result.get(key):
obj = obj_list[0]
return obj
def get_title_from_soup(soup) -> str:
title = ""
try:
title = soup.find("h1").get_text()
except AttributeError:
pass
except ValueError:
pass
return title
def get_description_from_soup(soup) -> str:
desc = ""
try:
desc = (
soup.find(class_="beer-descrption-read-less")
.get_text()
.replace("Show Less", "")
.strip()
)
except AttributeError:
pass
except ValueError:
pass
return desc
def get_styles_from_soup(soup) -> list[str]:
styles = []
try:
styles = soup.find("p", class_="style").get_text().split(" - ")
except AttributeError:
pass
except ValueError:
pass
return styles
def get_abv_from_soup(soup) -> Optional[float]:
abv = None
try:
abv = soup.find(class_="abv").get_text()
if abv:
abv = float(abv.strip("\n").strip("% ABV").strip())
except AttributeError:
pass
except ValueError:
pass
except TypeError:
pass
return abv
def get_ibu_from_soup(soup) -> Optional[int]:
ibu = None
try:
ibu = soup.find(class_="ibu").get_text()
if ibu:
ibu = int(ibu.strip("\n").strip(" IBU").strip())
except AttributeError:
pass
except ValueError:
ibu = None
return ibu
def get_rating_from_soup(soup) -> str:
rating = ""
try:
rating = float(
soup.find(class_="num").get_text().strip("(").strip(")")
)
except AttributeError:
rating = None
except ValueError:
rating = None
return rating
def get_producer_id_from_soup(soup) -> str:
id = ""
try:
id = soup.find(class_="brewery").find("a")["href"].strip("/")
except ValueError:
pass
except IndexError:
pass
return id
def get_producer_name_from_soup(soup) -> str:
name = ""
try:
name = soup.find(class_="brewery").find("a").get_text()
except AttributeError:
pass
except ValueError:
pass
return name
def get_food_from_allrecipe_id(allrecipe_id: str) -> dict:
url = ALLRECIPE_URL.format(id=allrecipe_id)
headers = {"User-Agent": "Vrobbler 0.11.12"}
response = requests.get(url, headers=headers)
food_dict = {"allrecipe_id": allrecipe_id}
if response.status_code != 200:
logger.warn(
"Bad response from allrecipe", extra={"response": response}
)
return food_dict
import pdb
pdb.set_trace()
soup = BeautifulSoup(response.text, "html.parser")
food_dict["title"] = get_title_from_soup(soup)
food_dict["description"] = get_description_from_soup(soup)
food_dict["allrecipe_rating"] = get_rating_from_soup(soup)
return food_dict

View File

@ -0,0 +1,5 @@
from django.apps import AppConfig
class FoodsConfig(AppConfig):
name = "foods"

View File

@ -0,0 +1,151 @@
# Generated by Django 4.2.16 on 2024-11-24 14:45
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
import taggit.managers
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
("scrobbles", "0066_scrobble_beer_alter_scrobble_media_type"),
]
operations = [
migrations.CreateModel(
name="FoodCategory",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
("name", models.CharField(max_length=255)),
(
"allrecipe_image",
models.ImageField(
blank=True, null=True, upload_to="food/recipe/"
),
),
(
"allrecipe_id",
models.CharField(blank=True, max_length=255, null=True),
),
("description", models.TextField(blank=True, null=True)),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
migrations.CreateModel(
name="Food",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True,
default=uuid.uuid4,
editable=False,
null=True,
),
),
(
"title",
models.CharField(blank=True, max_length=255, null=True),
),
(
"run_time_seconds",
models.IntegerField(blank=True, null=True),
),
(
"run_time_ticks",
models.PositiveBigIntegerField(blank=True, null=True),
),
("description", models.TextField(blank=True, null=True)),
(
"allrecipe_image",
models.ImageField(
blank=True, null=True, upload_to="food/recipe/"
),
),
(
"allrecipe_id",
models.CharField(blank=True, max_length=255, null=True),
),
("allrecipe_rating", models.FloatField(blank=True, null=True)),
(
"category",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="foods.foodcategory",
),
),
(
"genre",
taggit.managers.TaggableManager(
blank=True,
help_text="A comma-separated list of tags.",
through="scrobbles.ObjectWithGenres",
to="scrobbles.Genre",
verbose_name="Tags",
),
),
],
options={
"abstract": False,
},
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("foods", "0001_initial"),
]
operations = [
migrations.AlterField(
model_name="food",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,113 @@
from uuid import uuid4
from django.apps import apps
from django.db import models
from django.urls import reverse
from django_extensions.db.models import TimeStampedModel
from imagekit.models import ImageSpecField
from imagekit.processors import ResizeToFit
from scrobbles.dataclasses import FoodLogData
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
BNULL = {"blank": True, "null": True}
class FoodCategory(TimeStampedModel):
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
name = models.CharField(max_length=255)
allrecipe_image = models.ImageField(upload_to="food/recipe/", **BNULL)
allrecipe_image_small = ImageSpecField(
source="recipe_image",
processors=[ResizeToFit(100, 100)],
format="JPEG",
options={"quality": 60},
)
allrecipe_image_medium = ImageSpecField(
source="recipe_image",
processors=[ResizeToFit(300, 300)],
format="JPEG",
options={"quality": 75},
)
allrecipe_id = models.CharField(max_length=255, **BNULL)
description = models.TextField(**BNULL)
def find_or_create(cls, title: str) -> "FoodCategory":
return cls.objects.filter(title=title).first()
def __str__(self):
return self.name
class Food(ScrobblableMixin):
description = models.TextField(**BNULL)
allrecipe_image = models.ImageField(upload_to="food/recipe/", **BNULL)
allrecipe_image_small = ImageSpecField(
source="allrecipe_image",
processors=[ResizeToFit(100, 100)],
format="JPEG",
options={"quality": 60},
)
allrecipe_image_medium = ImageSpecField(
source="allrecipe_image",
processors=[ResizeToFit(300, 300)],
format="JPEG",
options={"quality": 75},
)
allrecipe_id = models.CharField(max_length=255, **BNULL)
allrecipe_rating = models.FloatField(**BNULL)
category = models.ForeignKey(
FoodCategory, on_delete=models.DO_NOTHING, **BNULL
)
def get_absolute_url(self) -> str:
return reverse("foods:food_detail", kwargs={"slug": self.uuid})
@property
def subtitle(self):
return self.category.name
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Eating", tags="food")
@property
def allrecipe_link(self) -> str:
link = ""
if self.producer and self.allrecipe_id:
if self.allrecipe_id:
link = f"https://www.allrecipe.com/{self.allrecipe_id}/"
return link
@property
def primary_image_url(self) -> str:
url = ""
if self.allrecipe_image:
url = self.allrecipe_image.url
return url
@property
def logdata_cls(self):
return FoodLogData
@classmethod
def find_or_create(cls, allrecipe_id: str) -> "Food":
food = cls.objects.filter(allrecipe_id=allrecipe_id).first()
if not food:
food_dict = get_food_from_allrecipe_id(allrecipe_id)
# category_dict = {}
# category, _created = FoodCategory.objects.get_or_create(
# **category_dict
# )
food = Food.objects.create(**food_dict)
# for category_id in category_ids:
# food.category.add(category_id)
return food
def scrobbles(self, user_id):
Scrobble = apps.get_model("scrobbles", "Scrobble")
return Scrobble.objects.filter(user_id=user_id, food=self).order_by(
"-timestamp"
)

View File

@ -0,0 +1,14 @@
from django.urls import path
from foods import views
app_name = "foods"
urlpatterns = [
path("foods/", views.FoodListView.as_view(), name="food_list"),
path(
"foods/<slug:slug>/",
views.FoodDetailView.as_view(),
name="food_detail",
),
]

View File

@ -0,0 +1,11 @@
from foods.models import Food
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
class FoodListView(ScrobbleableListView):
model = Food
class FoodDetailView(ScrobbleableDetailView):
model = Food

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("lifeevents", "0001_initial"),
]
operations = [
migrations.AlterField(
model_name="lifeevent",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -1,16 +1,13 @@
from django.apps import apps
from django.db import models
from django.urls import reverse
import pendulum
from scrobbles.dataclasses import LifeEventLogData
from scrobbles.mixins import ScrobblableMixin
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
BNULL = {"blank": True, "null": True}
class LifeEvent(ScrobblableMixin):
COMPLETION_PERCENT = 100
description = models.TextField(**BNULL)
def __str__(self):
@ -25,6 +22,10 @@ class LifeEvent(ScrobblableMixin):
def logdata_cls(self):
return LifeEventLogData
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Experiencing", tags="camping")
@classmethod
def find_or_create(cls, title: str) -> "LifeEvent":
return cls.objects.filter(title=title).first()

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("locations", "0006_delete_rawgeolocation"),
]
operations = [
migrations.AlterField(
model_name="geolocation",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -1,14 +1,12 @@
from decimal import Decimal, getcontext
import logging
from typing import Dict
from uuid import uuid4
from django.contrib.auth import get_user_model
from django.conf import settings
from django.db import models
from django.urls import reverse
from django_extensions.db.models import TimeStampedModel
from scrobbles.mixins import ScrobblableMixin
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
logger = logging.getLogger(__name__)
BNULL = {"blank": True, "null": True}
@ -90,6 +88,12 @@ class GeoLocation(ScrobblableMixin):
return f"{self.lat} x {self.lon}"
return ""
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(
verb="Going", tags="world_map", priority="low"
)
def loc_diff(self, old_lat_lon: tuple) -> tuple:
return (
abs(Decimal(old_lat_lon[0]) - Decimal(self.lat)),

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("moods", "0002_mood_image"),
]
operations = [
migrations.AlterField(
model_name="mood",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -5,7 +5,7 @@ from django.db import models
from django.urls import reverse
from imagekit.models import ImageSpecField
from imagekit.processors import ResizeToFit
from scrobbles.mixins import ScrobblableMixin
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
from vrobbler.apps.scrobbles.dataclasses import MoodLogData
@ -38,13 +38,14 @@ class Mood(ScrobblableMixin):
def get_absolute_url(self):
return reverse("moods:mood-detail", kwargs={"slug": self.uuid})
def get_start_url(self):
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
@property
def subtitle(self) -> str:
return ""
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Feeling", tags="thinking")
@property
def logdata_cls(self):
return MoodLogData

View File

@ -5,9 +5,9 @@ app_name = "moods"
urlpatterns = [
path("mood/", views.MoodListView.as_view(), name="mood-list"),
path("moods/", views.MoodListView.as_view(), name="mood-list"),
path(
"mood/<slug:slug>/",
"moods/<slug:slug>/",
views.MoodDetailView.as_view(),
name="mood-detail",
),

View File

@ -16,6 +16,7 @@ class AlbumAdmin(admin.ModelAdmin):
"theaudiodb_mood",
"musicbrainz_id",
)
raw_id_fields = ("album_artist",)
list_filter = (
"theaudiodb_score",
"theaudiodb_genre",
@ -49,13 +50,17 @@ class TrackAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
"album",
"primary_album",
"artist",
"musicbrainz_id",
)
raw_id_fields = ("artist", "albums", "album")
list_filter = ("album", "artist")
search_fields = ("title",)
ordering = ("-created",)
filter_horizontal = [
"albums",
]
inlines = [
ScrobbleInline,
]

View File

@ -1,153 +0,0 @@
import logging
import time
from datetime import datetime, timedelta
import pylast
import pytz
from django.conf import settings
from django.utils import timezone
from music.utils import (
get_or_create_album,
get_or_create_artist,
get_or_create_track,
)
logger = logging.getLogger(__name__)
PYLAST_ERRORS = tuple(
getattr(pylast, exc_name)
for exc_name in (
"ScrobblingError",
"NetworkError",
"MalformedResponseError",
"WSError",
)
if hasattr(pylast, exc_name)
)
class LastFM:
def __init__(self, user):
try:
self.client = pylast.LastFMNetwork(
api_key=getattr(settings, "LASTFM_API_KEY"),
api_secret=getattr(settings, "LASTFM_SECRET_KEY"),
username=user.profile.lastfm_username,
password_hash=pylast.md5(user.profile.lastfm_password),
)
self.user = self.client.get_user(user.profile.lastfm_username)
self.vrobbler_user = user
except PYLAST_ERRORS as e:
logger.error(f"Error during Last.fm setup: {e}")
def import_from_lastfm(self, last_processed=None):
"""Given a last processed time, import all scrobbles from LastFM since then"""
from scrobbles.models import Scrobble
new_scrobbles = []
source = "Last.fm"
lastfm_scrobbles = self.get_last_scrobbles(time_from=last_processed)
for lfm_scrobble in lastfm_scrobbles:
timestamp = lfm_scrobble.pop("timestamp")
artist = get_or_create_artist(lfm_scrobble.pop("artist"))
album = get_or_create_album(lfm_scrobble.pop("album"), artist)
lfm_scrobble["artist"] = artist
if album:
lfm_scrobble["album"] = album
track = get_or_create_track(**lfm_scrobble)
timezone = settings.TIME_ZONE
if self.vrobbler_user.profile:
timezone = self.vrobbler_user.profile.timezone
new_scrobble = Scrobble(
user=self.vrobbler_user,
timestamp=timestamp,
source=source,
track=track,
timezone=timezone,
played_to_completion=True,
in_progress=False,
media_type=Scrobble.MediaType.TRACK,
)
# Vrobbler scrobbles on finish, LastFM scrobbles on start
seconds_eariler = timestamp - timedelta(seconds=20)
seconds_later = timestamp + timedelta(seconds=20)
existing = Scrobble.objects.filter(
created__gte=seconds_eariler,
created__lte=seconds_later,
track=track,
).first()
if existing:
logger.debug(f"Skipping existing scrobble {new_scrobble}")
continue
logger.debug(f"Queued scrobble {new_scrobble} for creation")
new_scrobbles.append(new_scrobble)
created = Scrobble.objects.bulk_create(new_scrobbles)
logger.info(
f"Created {len(created)} scrobbles",
extra={"created_scrobbles": created},
)
return created
def get_last_scrobbles(self, time_from=None, time_to=None):
"""Given a user, Last.fm api key, and secret key, grab a list of scrobbled
tracks"""
lfm_params = {}
scrobbles = []
if time_from:
lfm_params["time_from"] = int(time_from.timestamp())
if time_to:
lfm_params["time_to"] = int(time_to.timestamp())
# if not time_from and not time_to:
lfm_params["limit"] = None
found_scrobbles = self.user.get_recent_tracks(**lfm_params)
# TOOD spin this out into a celery task over certain threshold of found scrobbles?
for scrobble in found_scrobbles:
logger.debug(f"Processing {scrobble}")
run_time = None
mbid = None
artist = None
try:
run_time = int(scrobble.track.get_duration() / 1000)
mbid = scrobble.track.get_mbid()
artist = scrobble.track.get_artist().name
except pylast.MalformedResponseError as e:
logger.warn(e)
except pylast.WSError as e:
logger.warn(
"LastFM barfed trying to get the track for {scrobble.track}"
)
except pylast.NetworkError as e:
logger.warn(
"LastFM barfed trying to get the track for {scrobble.track}"
)
if not artist:
logger.warn(f"Silly LastFM, no artist found for {scrobble}")
continue
timestamp = datetime.utcfromtimestamp(
int(scrobble.timestamp)
).replace(tzinfo=pytz.utc)
logger.info(f"{artist},{scrobble.track.title},{timestamp}")
scrobbles.append(
{
"artist": artist,
"album": scrobble.album,
"title": scrobble.track.title,
"mbid": mbid,
"run_time_seconds": run_time,
"timestamp": timestamp,
}
)
return scrobbles

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0023_alter_track_genre"),
]
operations = [
migrations.AlterField(
model_name="track",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.19 on 2025-04-07 00:23
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0024_alter_track_run_time_seconds"),
]
operations = [
migrations.AddField(
model_name="artist",
name="alt_names",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.19 on 2025-04-07 00:34
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0025_artist_alt_names"),
]
operations = [
migrations.AddField(
model_name="album",
name="alt_names",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -0,0 +1,20 @@
# Generated by Django 4.2.19 on 2025-07-20 20:54
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0026_album_alt_names"),
]
operations = [
migrations.AddField(
model_name="track",
name="albums",
field=models.ManyToManyField(
blank=True, null=True, related_name="tracks", to="music.album"
),
),
]

View File

@ -0,0 +1,20 @@
# Generated by Django 4.2.19 on 2025-07-25 14:41
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("music", "0027_track_albums"),
]
operations = [
migrations.AlterField(
model_name="track",
name="albums",
field=models.ManyToManyField(
related_name="tracks", to="music.album"
),
),
]

View File

@ -1,13 +1,11 @@
import logging
from tempfile import NamedTemporaryFile
from typing import Dict, Optional
from urllib.request import urlopen
from uuid import uuid4
import musicbrainzngs
import requests
from django.conf import settings
from django.core.files.base import ContentFile, File
from django.core.files.base import ContentFile
from django.db import models
from django.urls import reverse
from django.utils.translation import gettext_lazy as _
@ -16,14 +14,36 @@ from imagekit.models import ImageSpecField
from imagekit.processors import ResizeToFit
from music.allmusic import get_allmusic_slug, scrape_data_from_allmusic
from music.bandcamp import get_bandcamp_slug
from music.musicbrainz import (
get_album_metadata,
get_album_metadata_with_artist,
get_artist_metadata_extended,
get_recording_mbid_exact,
get_track_metadata_with_artist,
lookup_album_dict_from_mb,
lookup_album_from_mb,
lookup_track_from_mb,
lookup_artist_from_mb,
)
from music.theaudiodb import lookup_album_from_tadb, lookup_artist_from_tadb
from scrobbles.mixins import ScrobblableMixin
from music.utils import clean_artist_name
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
logger = logging.getLogger(__name__)
BNULL = {"blank": True, "null": True}
class Artist(TimeStampedModel):
"""Represents a music artist.
# Lookup or create by title alone
>>> Artist.find_or_create(name="Bon Iver")
# Lookup or create by MB id alone
>>> Artist.find_or_create(musicbrainz_id="0307edfc-437c-4b48-8700-80680e66a228")
"""
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
name = models.CharField(max_length=255)
biography = models.TextField(**BNULL)
@ -46,6 +66,7 @@ class Artist(TimeStampedModel):
format="JPEG",
options={"quality": 75},
)
alt_names = models.TextField(**BNULL)
class Meta:
unique_together = [["name", "musicbrainz_id"]]
@ -62,8 +83,10 @@ class Artist(TimeStampedModel):
return ""
@property
def mb_link(self):
return f"https://musicbrainz.org/artist/{self.musicbrainz_id}"
def mb_link(self) -> str:
if self.musicbrainz_id:
return f"https://musicbrainz.org/artist/{self.musicbrainz_id}"
return ""
@property
def allmusic_link(self):
@ -104,7 +127,9 @@ class Artist(TimeStampedModel):
if not self.allmusic_id or force:
slug = get_allmusic_slug(self.name)
if not slug:
logger.info(f"No allmsuic link for {self}")
logger.info(
"No allmusic link found", extra={"track_id": self.id}
)
return
self.allmusic_id = slug
self.save(update_fields=["allmusic_id"])
@ -113,7 +138,9 @@ class Artist(TimeStampedModel):
if not self.bandcamp_id or force:
slug = get_bandcamp_slug(self.name)
if not slug:
logger.info(f"No bandcamp link for {self}")
logger.info(
"No bandcamp link found", extra={"track_id": self.id}
)
return
self.bandcamp_id = slug
self.save(update_fields=["bandcamp_id"])
@ -145,7 +172,7 @@ class Artist(TimeStampedModel):
@property
def rym_link(self):
artist_slug = self.name.lower().replace(" ", "-")
artist_slug = self.name.lower().replace(" ", "-").replace(",", "")
return f"https://rateyourmusic.com/artist/{artist_slug}/"
@property
@ -153,6 +180,79 @@ class Artist(TimeStampedModel):
artist = self.name.lower()
return f"https://bandcamp.com/search?q={artist}&item_type=b"
@classmethod
def find_or_create(
cls, name: str, album_name: str = "", track_name: str = ""
) -> "Artist":
"""The biggest challenge to finding artists is that the search often
fails miserably unless you can look it up along with an album or a track name.
Thus, when we find or create an artist, we should always provide an optional
album name or track name, but probably not both."""
if album_name:
logger.info(
f"Looking for artist with name {name} and album {album_name}"
)
if track_name:
logger.info(
f"Looking for artist with name {name} and track {track_name}"
)
keys = {}
name = clean_artist_name(name)
keys["name"] = name
artist = cls.objects.filter(name=name).first()
if artist:
return artist
# alt_name = None
artist_dict = {}
if album_name:
album_dict = get_album_metadata_with_artist(album_name, name)
if album_dict:
artist_dict = album_dict.get("primary_artist")
if track_name:
track_dict = get_track_metadata_with_artist(track_name, name)
if track_dict:
artist_dict = track_dict.get("primary_artist")
if not artist_dict:
artist, created = cls.objects.get_or_create(name=name)
if created:
artist.fix_metadata()
return artist
musicbrainz_id = artist_dict.get("mbid")
found_name = artist_dict.get("name", name)
if found_name and name != found_name:
alt_name = found_name
artist = cls.objects.filter(
name=found_name, musicbrainz_id=musicbrainz_id
).first()
if not artist:
artist = cls.objects.create(
name=found_name,
musicbrainz_id=musicbrainz_id,
)
artist.fix_metadata()
# TODO: See if this alt_names stuff actually works or causes hard to debug problems
# If we did find our artist, but the found name is slightly differnt, record that
# if artist and alt_name:
# if not artist.alt_names:
# artist.alt_names = alt_name
# else:
# artist.alt_names += f"\\{alt_name}"
# logger.info(
# f"Add alt_name {alt_name} to artist {artist}",
# extra={"alt_name": alt_name, "artist_id": artist.id},
# )
# artist.save(update_fields=["alt_names"])
return artist
class Album(TimeStampedModel):
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
@ -196,9 +296,10 @@ class Album(TimeStampedModel):
wikipedia_slug = models.CharField(max_length=255, **BNULL)
discogs_id = models.CharField(max_length=255, **BNULL)
wikidata_id = models.CharField(max_length=255, **BNULL)
alt_names = models.TextField(**BNULL)
def __str__(self):
return self.name
def __str__(self) -> str:
return "{} by {}".format(self.name, self.album_artist)
def get_absolute_url(self):
return reverse("music:album_detail", kwargs={"slug": self.uuid})
@ -235,8 +336,15 @@ class Album(TimeStampedModel):
self.save(update_fields=["album_artist"])
def scrape_allmusic(self, force=False) -> None:
if not self.allmusic_id or force:
slug = get_allmusic_slug(self.name, self.album_artist.name)
if not self.name:
logger.warning(
"Album without a name cannot be scraped",
extra={"album_id": self.id},
)
return
if self.album_artist and (not self.allmusic_id or force):
slug = get_allmusic_slug(self.album_artist.name, self.name)
if not slug:
logger.info(
f"No allmsuic link for {self} by {self.album_artist}"
@ -245,7 +353,9 @@ class Album(TimeStampedModel):
self.allmusic_id = slug
self.save(update_fields=["allmusic_id"])
allmusic_data = scrape_data_from_allmusic(self.allmusic_link)
allmusic_data = None
if self.allmusic_link:
allmusic_data = scrape_data_from_allmusic(self.allmusic_link)
if not allmusic_data:
logger.info(f"No allmsuic data for {self} by {self.album_artist}")
@ -264,7 +374,12 @@ class Album(TimeStampedModel):
logger.info(f"No data for {self} found in TheAudioDB")
return
Album.objects.filter(pk=self.pk).update(**album_data)
try:
Album.objects.filter(pk=self.pk).update(**album_data)
except:
logger.info(
f"Could not save info for album {self} with data {album_data}"
)
def scrape_bandcamp(self, force=False) -> None:
if not self.bandcamp_id or force:
@ -402,16 +517,85 @@ class Album(TimeStampedModel):
album = self.name.lower()
return f"https://bandcamp.com/search?q={album} {artist}&item_type=a"
@classmethod
def find_or_create(cls, name: str, artist_name: str) -> "Album":
logger.info(
f"Looking for album with name {name} and artist_name {artist_name}"
)
artist = Artist.find_or_create(artist_name, album_name=name)
album_dict = get_album_metadata_with_artist(name, artist.name)
if not album_dict:
logger.info(
f"Could not find album {name} with artist {artist.name} on musicbrainz"
)
album, created = Album.objects.get_or_create(
name=name,
)
if created:
# album.fix_metadata()
# album.fetch_artwork()
...
return album
if not artist:
artist_dict = album_dict.get("primary_artist", {})
if artist_dict:
artist = Artist.objects.filter(
musicbrainz_id=artist_dict.get("mbid"),
).first()
if not artist:
artist = Artist.objects.create(
musicbrainz_id=artist_dict.get("mbid"),
)
extra_artists = []
if not artist and len(album_dict.get("all_artists")) > 1:
artist = Artist.objects.filter(name="Various Artists").first()
extra_artists.append(artist)
if not artist:
raise Exception("No album artist found, and not a compliation")
album = cls.objects.filter(
models.Q(name=name) | models.Q(alt_names__icontains=name),
album_artist=artist,
).first()
alt_name = None
found_name = album_dict.get("album_title", name)
if found_name and name != found_name:
alt_name = name
album = Album.objects.filter(
name=found_name, musicbrainz_id=album_dict.get("mbid")
).first()
if not album:
year = None
if album_dict.get("release_date"):
year = album_dict.get("release_date", "").split("-")[0]
album = Album.objects.create(
name=found_name,
musicbrainz_id=album_dict.get("mbid"),
musicbrainz_releasegroup_id=album_dict.get(
"release_group_mbid"
),
year=year,
album_artist=artist,
alt_names=alt_name,
)
album.artists.add(*extra_artists)
album.fetch_artwork()
return album
class Track(ScrobblableMixin):
COMPLETION_PERCENT = getattr(settings, "MUSIC_COMPLETION_PERCENT", 100)
class Opinion(models.IntegerChoices):
DOWN = -1, "Thumbs down"
NEUTRAL = 0, "No opinion"
UP = 1, "Thumbs up"
artist = models.ForeignKey(Artist, on_delete=models.DO_NOTHING)
albums = models.ManyToManyField(Album, related_name="tracks")
album = models.ForeignKey(Album, on_delete=models.DO_NOTHING, **BNULL)
musicbrainz_id = models.CharField(max_length=255, **BNULL)
@ -421,12 +605,22 @@ class Track(ScrobblableMixin):
def __str__(self):
return f"{self.title} by {self.artist}"
@property
def primary_album(self):
if self.album:
return self.album
return self.albums.order_by("year").first()
def get_absolute_url(self):
return reverse("music:track_detail", kwargs={"slug": self.uuid})
@property
def subtitle(self):
return self.artist
def subtitle(self) -> str:
return str(self.artist)
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Listening", tags="notes")
@property
def mb_link(self):
@ -441,37 +635,85 @@ class Track(ScrobblableMixin):
url = ""
if self.artist.thumbnail:
url = self.artist.thumbnail_medium.url
if self.album and self.album.cover_image:
url = self.album.cover_image_medium.url
if self.primary_album and self.primary_album.cover_image:
url = self.primary_album.cover_image_medium.url
return url
@classmethod
def find_or_create(
cls, artist_dict: Dict, album_dict: Dict, track_dict: Dict
) -> Optional["Track"]:
"""Given a data dict from Jellyfin, does the heavy lifting of looking up
the video and, if need, TV Series, creating both if they don't yet
exist.
cls,
title: str = "",
artist_name: str = "",
album_name: str = "",
run_time_seconds: int | None = None,
enrich: bool = False,
commit: bool = True,
) -> "Track":
"""Given a name, try to find the track by the artist from Musicbrainz.
"""
if not artist_dict.get("name") or not artist_dict.get(
"musicbrainz_id"
):
logger.warning(
f"No artist or artist musicbrainz ID found in message from source, not scrobbling"
As a basic conceit we trust the source for giving us the track and artist
name
Optionally, we can update any found artists with overwrite."""
album = None
if album_name:
logger.info("Looking up album for: {album_name}")
album = Album.find_or_create(
name=album_name, artist_name=artist_name
)
return
artist = album.album_artist
else:
artist = Artist.find_or_create(artist_name, track_name=title)
if not artist:
artist = Artist.find_or_create(artist_name)
artist, artist_created = Artist.objects.get_or_create(**artist_dict)
album, album_created = Album.objects.get_or_create(**album_dict)
lookup_keys = {"title": title, "artist": artist}
if run_time_seconds:
lookup_keys["run_time_seconds"] = run_time_seconds
logger.info(f"Looking up track using: {lookup_keys}")
track = cls.objects.filter(**lookup_keys).first()
if track:
logger.info(
"Found match for track by name and artist, not going to musicbrainz ",
extra={
"track_id": track.id,
"title": title,
"artist_name": artist_name,
"run_time_seconds": run_time_seconds,
},
)
return track
album.fix_metadata()
if not album.cover_image:
album.fetch_artwork()
track = cls.objects.filter(title=title, artist=artist).first()
if not track:
track, _ = cls.objects.get_or_create(title=title, artist=artist)
track_dict["album_id"] = getattr(album, "id", None)
track_dict["artist_id"] = artist.id
if album:
track.albums.add(album)
track, created = cls.objects.get_or_create(**track_dict)
if enrich or not track.run_time_seconds:
logger.info(
f"Enriching track {track}",
extra={
"title": title,
"artist_name": artist_name,
"track_id": track.id,
},
)
try:
mbid, length = get_recording_mbid_exact(
title, artist_name, album_name
)
except Exception:
print("No musicbrainz result found, cannot enrich")
return track
track.run_time_seconds = run_time_seconds or int(length / 1000)
track.musicbrainz_id = mbid
if commit:
track.save()
return track
def fix_metadata(self, force_update=False):
...

View File

@ -1,16 +1,17 @@
from datetime import datetime
import logging
from typing import Iterable
import musicbrainzngs
from dateutil.parser import parse
logger = logging.getLogger(__name__)
musicbrainzngs.set_useragent("Vrobbler", "1.0", "help@unbl.ink")
def lookup_album_from_mb(musicbrainz_id: str) -> dict:
release_dict = {}
musicbrainzngs.set_useragent("vrobbler", "0.3.0")
release_data = musicbrainzngs.get_release_by_id(
musicbrainz_id,
includes=["artists", "release-groups", "recordings"],
@ -51,7 +52,6 @@ def lookup_album_from_mb(musicbrainz_id: str) -> dict:
def lookup_album_dict_from_mb(release_name: str, artist_name: str) -> dict:
musicbrainzngs.set_useragent("vrobbler", "0.3.0")
top_result = {}
@ -84,7 +84,6 @@ def lookup_album_dict_from_mb(release_name: str, artist_name: str) -> dict:
def lookup_artist_from_mb(artist_name: str) -> dict:
musicbrainzngs.set_useragent("vrobbler", "0.3.0")
try:
top_result = musicbrainzngs.search_artists(artist=artist_name)[
@ -104,7 +103,7 @@ def lookup_artist_from_mb(artist_name: str) -> dict:
def lookup_track_from_mb(
track_name: str, artist_mb_id: str, album_mb_id: str
track_name: str, artist_mb_id: str, album_mb_id: str = ""
) -> dict:
logger.info(
"[lookup_track_from_mb] called",
@ -114,7 +113,6 @@ def lookup_track_from_mb(
"album_mb_id": album_mb_id,
},
)
musicbrainzngs.set_useragent("vrobbler", "0.3.0")
try:
results = musicbrainzngs.search_recordings(
@ -138,3 +136,352 @@ def lookup_track_from_mb(
return {}
return top_result
def get_album_metadata(album_name, artist_name, strict=True) -> dict:
"""
Get detailed metadata for an album from MusicBrainz.
:param album_name: Name of the album
:param artist_name: Name of the artist
:param strict: If True, only exact matches on album and artist (case-insensitive)
:return: dict with album metadata, or None if not found
"""
try:
result = musicbrainzngs.search_releases(
release=album_name, artist=artist_name, limit=5
)
for release in result.get("release-list", []):
title = release["title"]
primary_artist = release["artist-credit"][0]["artist"]["name"]
title_match = title.lower() == album_name.lower()
artist_match = primary_artist.lower() == artist_name.lower()
if not strict or (title_match and artist_match):
all_artists = [
ac["artist"]["name"]
for ac in release["artist-credit"]
if isinstance(ac, dict) and "artist" in ac
]
return {
"album_title": title,
"primary_artist": primary_artist,
"all_artists": all_artists,
"mbid": release["id"],
"release_date": release.get(
"date"
), # May be partial (e.g., just year)
"release_group_mbid": release["release-group"]["id"],
}
return {}
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error:", e)
return {}
def get_recording_mbid_exact(
track_title: str, artist_name: str, album_name: str
) -> tuple[str, int]:
try:
result = musicbrainzngs.search_releases(
artist=artist_name, release=album_name, limit=1
)
releases = result.get("release-list", [])
if not releases:
raise Exception("No releases found")
release_id = releases[0]["id"]
release_data = musicbrainzngs.get_release_by_id(
release_id, includes=["recordings"]
)
tracks = release_data["release"]["medium-list"][0]["track-list"]
for track in tracks:
if track["recording"]["title"].lower() == track_title.lower():
return track["recording"]["id"], int(
track["recording"]["length"]
)
raise Exception("No recording found")
except musicbrainzngs.WebServiceError as e:
print(f"MusicBrainz error: {e}")
raise Exception(e)
def get_artist_metadata_extended(artist_name, strict=True):
"""
Fetch artist metadata including MBID, name, origin, tags, and description.
:param artist_name: The artist's name
:param strict: If True, only return exact name match
:return: dict with metadata, or None if not found
"""
try:
# Step 1: Search for artist
search_results = musicbrainzngs.search_artists(
artist=artist_name, limit=5
)
for artist in search_results.get("artist-list", []):
if not strict or artist["name"].lower() == artist_name.lower():
mbid = artist["id"]
# Step 2: Get detailed info about the artist
details = musicbrainzngs.get_artist_by_id(
mbid, includes=["tags", "url-rels"]
)["artist"]
begin_date = details.get("life-span", {}).get("begin")
area = details.get("area", {}).get("name")
disambiguation = details.get("disambiguation")
tags = [t["name"] for t in details.get("tag-list", [])]
# Step 3: Try to find a Wikipedia or Wikidata link
description_url = None
for rel in details.get("url-relation-list", []):
if rel["type"] == "wikipedia":
description_url = rel["target"]
break
elif rel["type"] == "wikidata":
description_url = rel["target"]
return {
"mbid": mbid,
"name": details["name"],
"disambiguation": disambiguation,
"begin_date": begin_date,
"area": area,
"tags": tags,
"description_url": description_url, # user can fetch summary if needed
}
return None
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error:", e)
return None
def get_artist_metadata_brief(artist_id):
"""Fetch basic artist metadata by MBID."""
try:
details = musicbrainzngs.get_artist_by_id(
artist_id, includes=["tags", "aliases", "url-rels"]
)["artist"]
begin_date = details.get("life-span", {}).get("begin")
area = details.get("area", {}).get("name")
disambiguation = details.get("disambiguation")
tags = [t["name"] for t in details.get("tag-list", [])]
description_url = None
for rel in details.get("url-relation-list", []):
if rel["type"] == "wikipedia":
description_url = rel["target"]
break
elif rel["type"] == "wikidata" and not description_url:
description_url = rel["target"]
return {
"mbid": artist_id,
"name": details["name"],
"disambiguation": disambiguation,
"begin_date": begin_date,
"area": area,
"tags": tags,
"description_url": description_url,
}
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error (artist lookup):", e)
return None
def parse_date(date_str):
"""Parse MusicBrainz date format into sortable datetime object."""
if not date_str:
return None
for fmt in ("%Y-%m-%d", "%Y-%m", "%Y"):
try:
return datetime.strptime(date_str, fmt)
except ValueError:
continue
return None
def get_album_metadata_with_artist(album_name, artist_name, strict=True):
"""
Get metadata for the earliest release of an album and its primary artist.
:param album_name: Album title
:param artist_name: Name of the artist
:param strict: If True, enforce exact match for album and artist
:return: dict with album and primary artist metadata
"""
try:
result = musicbrainzngs.search_releases(
release=album_name, artist=artist_name, limit=100
)
query_album = album_name.strip().casefold()
query_artist = artist_name.strip().casefold()
valid_releases = []
for release in result.get("release-list", []):
release_title = release["title"].strip()
primary_artist = release["artist-credit"][0]["artist"]
artist_name_actual = primary_artist["name"].strip()
if strict:
if release_title.casefold() != query_album:
continue
if artist_name_actual.casefold() != query_artist:
continue
release_date = parse_date(release.get("date"))
valid_releases.append((release, release_date))
if not valid_releases:
return None
# Sort releases by earliest release date
valid_releases.sort(key=lambda x: x[1] or datetime.max)
release, _ = valid_releases[0]
primary_artist = release["artist-credit"][0]["artist"]
all_artists = [
ac["artist"]["name"]
for ac in release["artist-credit"]
if "artist" in ac
]
artist_metadata = get_artist_metadata_brief(primary_artist["id"])
return {
"album_title": release["title"],
"primary_artist_name": primary_artist["name"],
"all_artists": all_artists,
"mbid": release["id"],
"release_group_mbid": release["release-group"]["id"],
"release_date": release.get("date"),
"primary_artist": artist_metadata,
}
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error (album lookup):", e)
return None
def get_artist_metadata_brief(artist_id):
try:
details = musicbrainzngs.get_artist_by_id(
artist_id, includes=["tags", "aliases", "url-rels"]
)["artist"]
begin_date = details.get("life-span", {}).get("begin")
area = details.get("area", {}).get("name")
disambiguation = details.get("disambiguation")
tags = [t["name"] for t in details.get("tag-list", [])]
description_url = None
for rel in details.get("url-relation-list", []):
if rel["type"] == "wikipedia":
description_url = rel["target"]
break
elif rel["type"] == "wikidata" and not description_url:
description_url = rel["target"]
return {
"mbid": artist_id,
"name": details["name"],
"disambiguation": disambiguation,
"begin_date": begin_date,
"area": area,
"tags": tags,
"description_url": description_url,
}
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error (artist lookup):", e)
return None
def get_track_metadata_with_artist(track_title, artist_name, strict=True):
"""
Get metadata for the earliest-known recording of a track, including artist info.
:param track_title: Track title
:param artist_name: Artist name
:param strict: If True, match exactly (case-insensitive)
:return: dict with track + release + artist metadata
"""
try:
result = musicbrainzngs.search_recordings(
recording=track_title, artist=artist_name, limit=100
)
query_track = track_title.strip().casefold()
query_artist = artist_name.strip().casefold()
valid_candidates = []
for recording in result.get("recording-list", []):
rec_title = recording["title"].strip()
artist_credit = recording["artist-credit"][0]["artist"]
artist_name_actual = artist_credit["name"].strip()
if strict:
if rec_title.casefold() != query_track:
continue
if artist_name_actual.casefold() != query_artist:
continue
if "release-list" not in recording:
continue
for release in recording["release-list"]:
release_date = parse_date(release.get("date"))
if release_date:
valid_candidates.append(
(recording["id"], release, release_date)
)
if not valid_candidates:
return None
# Pick the earliest release
valid_candidates.sort(key=lambda x: x[2])
recording_id, release, _ = valid_candidates[0]
# Fetch full recording info
full_recording = musicbrainzngs.get_recording_by_id(
recording_id, includes=["artists", "releases"]
)["recording"]
primary_artist = full_recording["artist-credit"][0]["artist"]
all_artists = [
ac["artist"]["name"]
for ac in full_recording["artist-credit"]
if "artist" in ac
]
artist_metadata = get_artist_metadata_brief(primary_artist["id"])
return {
"track_title": full_recording["title"],
"length_ms": full_recording.get("length"),
"recording_mbid": recording_id,
"release_title": release["title"],
"release_date": release.get("date"),
"release_group_mbid": release["release-group"]["id"],
"primary_artist_name": primary_artist["name"],
"all_artists": all_artists,
"primary_artist": artist_metadata,
}
except musicbrainzngs.WebServiceError as e:
print("MusicBrainz error (track lookup):", e)
return None

View File

@ -1,150 +1,113 @@
import logging
import re
from typing import Optional
from music.musicbrainz import (
lookup_album_dict_from_mb,
lookup_artist_from_mb,
lookup_track_from_mb,
)
from django.db import IntegrityError, models, transaction
from music.constants import VARIOUS_ARTIST_DICT
from scrobbles.utils import convert_to_seconds
logger = logging.getLogger(__name__)
from music.models import Album, Artist, Track
def get_or_create_artist(name: str, mbid: str = None) -> Artist:
artist = None
if "feat." in name.lower():
def clean_artist_name(name: str) -> str:
"""Remove featured names from artist string."""
if " feat. " in name.lower():
name = re.split("feat.", name, flags=re.IGNORECASE)[0].strip()
if "featuring" in name.lower():
if " w. " in name.lower():
name = re.split("feat.", name, flags=re.IGNORECASE)[0].strip()
if " featuring " in name.lower():
name = re.split("featuring", name, flags=re.IGNORECASE)[0].strip()
if "&" in name.lower():
name = re.split("&", name, flags=re.IGNORECASE)[0].strip()
# if " & " in name.lower() and "of the wand" not in name.lower():
# name = re.split("&", name, flags=re.IGNORECASE)[0].strip()
artist_dict = lookup_artist_from_mb(name)
mbid = mbid or artist_dict.get("id", None)
if mbid:
artist = Artist.objects.filter(musicbrainz_id=mbid).first()
if not artist:
artist = Artist.objects.create(name=name, musicbrainz_id=mbid)
artist.fix_metadata()
return artist
return name
def get_or_create_album(
name: str, artist: Artist, mbid: str = None
) -> Optional[Album]:
album = None
album_dict = lookup_album_dict_from_mb(name, artist_name=artist.name)
def get_or_create_various_artists() -> "Artist":
from music.models import Artist
name = name or album_dict.get("title", None)
if not name:
logger.debug(
f"Cannot get or create album by {artist} with no name ({name})"
)
return
album = Album.objects.filter(
musicbrainz_id=mbid, name=name, artists__in=[artist]
).first()
if not album:
mbid_group = album_dict.get("mb_group_id")
album = Album.objects.filter(
musicbrainz_releasegroup_id=mbid_group
).first()
if not album and name:
mbid = mbid or album_dict["mb_id"]
album, album_created = Album.objects.get_or_create(musicbrainz_id=mbid)
if album_created:
album.name = name
album.year = album_dict["year"]
album.musicbrainz_releasegroup_id = album_dict["mb_group_id"]
album.musicbrainz_albumartist_id = artist.musicbrainz_id
album.save(
update_fields=[
"name",
"musicbrainz_id",
"year",
"musicbrainz_releasegroup_id",
"musicbrainz_albumartist_id",
]
)
album.artists.add(artist)
album.fix_album_artist()
album.fetch_artwork()
album.scrape_allmusic()
if not album:
logger.warn(f"No album found for {name} and {mbid}")
album.fix_album_artist()
return album
def get_or_create_track(post_data: dict, post_keys: dict) -> Track:
try:
track_run_time_seconds = int(
post_data.get(post_keys.get("RUN_TIME"), 0)
)
except ValueError: # Sometimes we get run time as a string like "01:35"
track_run_time_seconds = convert_to_seconds(
post_data.get(post_keys.get("RUN_TIME"), 0)
)
artist_name = post_data.get(post_keys.get("ARTIST_NAME"), "")
artist_mb_id = post_data.get(post_keys.get("ARTIST_MB_ID"), "")
album_title = post_data.get(post_keys.get("ALBUM_NAME"), "")
album_mb_id = post_data.get(post_keys.get("ALBUM_MB_ID"), "")
track_title = post_data.get(post_keys.get("TRACK_TITLE"), "")
track_mb_id = post_data.get(post_keys.get("TRACK_MB_ID"), "")
artist = get_or_create_artist(
artist_name,
mbid=artist_mb_id,
)
album = get_or_create_album(
album_title,
artist=artist,
mbid=album_mb_id,
)
track = None
if not track_mb_id and album:
try:
track_mb_id = lookup_track_from_mb(
track_title,
artist.musicbrainz_id,
album.musicbrainz_id,
).get("id", 0)
except TypeError:
pass
if track_mb_id:
track = Track.objects.filter(musicbrainz_id=track_mb_id).first()
if not track:
track = Track.objects.create(
title=track_title,
artist=artist,
album=album,
musicbrainz_id=track_mb_id,
run_time_seconds=track_run_time_seconds,
)
return track
def get_or_create_various_artists():
artist = Artist.objects.filter(name="Various Artists").first()
if not artist:
artist = Artist.objects.create(**VARIOUS_ARTIST_DICT)
logger.info("Created Various Artists placeholder")
return artist
def deduplicate_tracks(commit=False) -> int:
from music.models import Track
duplicates = (
Track.objects.values("artist", "title")
.annotate(dup_count=models.Count("id"))
.filter(dup_count__gt=1)
)
query = models.Q()
for dup in duplicates:
query |= models.Q(artist=dup["artist"], title=dup["title"])
duplicate_tracks = Track.objects.filter(query)
for b in duplicate_tracks:
tracks = Track.objects.filter(artist=b.artist, title=b.title)
first = tracks.first()
for other in tracks.exclude(id=first.id):
print("Moving scrobbles for", other.id, " to ", first.id)
if commit:
with transaction.atomic():
other.scrobble_set.update(track=first)
print("deleting ", other.id, " - ", other)
try:
other.delete()
except IntegrityError as e:
print(
"could not delete ",
other.id,
f": IntegrityError {e}",
)
return len(duplicate_tracks)
def condense_albums(commit: bool = False):
from music.models import Track
from scrobbles.models import Scrobble
processed_ids = []
for track in Track.objects.all():
albums_to_add = []
duplicates = (
Track.objects.filter(title=track.title, artist=track.artist)
.exclude(id=track.id)
.exclude(id__in=processed_ids)
)
if commit and track.album:
albums_to_add.append(track.album)
for dup_track in duplicates:
logger.info(f"Adding {dup_track.album} to {track} albums")
if commit and dup_track.album:
track.albums.add(dup_track.album)
# Find out if this track appears more than once
duplicates = Track.objects.filter(
title=track.title, artist=track.artist
)
if duplicates.count() > 1:
logger.info(f"Track appears more than once, condensing: {track}")
albums_to_add.extend([d.album for d in duplicates])
# Find all scrobbles
duplicate_ids = duplicates.values_list("id", flat=True)
scrobbles = Scrobble.objects.filter(track_id__in=duplicate_ids)
logger.info(
f"Found {scrobbles.count()} scrobbles to merge onto {track}"
)
if commit:
scrobbles.update(track=track)
track.albums.add(*list(set(albums_to_add)))
processed_ids.extend(duplicate_ids)
if commit:
Track.objects.filter(scrobble__isnull=True).delete()
return len(set(processed_ids))

View File

View File

@ -0,0 +1,10 @@
from django.contrib import admin
from people.models import Person
@admin.register(Person)
class PersonAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "bgg_username", "bgstats_id")
ordering = ("-created",)
search_fields = ("name",)

View File

@ -0,0 +1,74 @@
# Generated by Django 4.2.19 on 2025-07-02 14:59
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name="Person",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"name",
models.CharField(blank=True, max_length=100, null=True),
),
(
"bgstat_id",
models.CharField(blank=True, max_length=100, null=True),
),
(
"bgg_username",
models.CharField(blank=True, max_length=100, null=True),
),
(
"lichess_username",
models.CharField(blank=True, max_length=100, null=True),
),
("bio", models.TextField(blank=True, null=True)),
(
"user",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to=settings.AUTH_USER_MODEL,
),
),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
]

View File

@ -0,0 +1,22 @@
# Generated by Django 4.2.19 on 2025-07-03 02:25
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("people", "0001_initial"),
]
operations = [
migrations.RemoveField(
model_name="person",
name="bgstat_id",
),
migrations.AddField(
model_name="person",
name="bgstats_id",
field=models.UUIDField(blank=True, null=True),
),
]

View File

@ -0,0 +1,17 @@
from django.contrib.auth import get_user_model
from django.db import models
from django_extensions.db.models import TimeStampedModel
User = get_user_model()
BNULL = {"blank": True, "null": True}
class Person(TimeStampedModel):
"""A non-system user model that can be optionally associated with a User."""
user = models.ForeignKey(User, on_delete=models.DO_NOTHING, **BNULL)
name = models.CharField(max_length=100, **BNULL)
bgstats_id = models.UUIDField(**BNULL)
bgg_username = models.CharField(max_length=100, **BNULL)
lichess_username = models.CharField(max_length=100, **BNULL)
bio = models.TextField(**BNULL)

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.16 on 2025-01-22 03:55
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("podcasts", "0013_rename_episode_podcastepisode"),
]
operations = [
migrations.AlterField(
model_name="podcastepisode",
name="run_time_seconds",
field=models.IntegerField(default=900),
),
]

View File

@ -0,0 +1,40 @@
# Generated by Django 4.2.19 on 2025-04-07 17:16
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("podcasts", "0014_alter_podcastepisode_run_time_seconds"),
]
operations = [
migrations.RemoveField(
model_name="podcast",
name="google_podcasts_url",
),
migrations.AddField(
model_name="podcast",
name="dead_date",
field=models.DateField(blank=True, null=True),
),
migrations.AddField(
model_name="podcast",
name="itunes_id",
field=models.TextField(blank=True, max_length=15, null=True),
),
migrations.AddField(
model_name="podcast",
name="null",
field=models.CharField(
default="", max_length=150, verbose_name="blank"
),
preserve_default=False,
),
migrations.AddField(
model_name="podcast",
name="site_link",
field=models.URLField(blank=True, null=True),
),
]

View File

@ -0,0 +1,28 @@
# Generated by Django 4.2.19 on 2025-04-07 17:18
from django.db import migrations
import taggit.managers
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0068_scrobble_paper_alter_scrobble_media_type"),
(
"podcasts",
"0015_remove_podcast_google_podcasts_url_podcast_dead_date_and_more",
),
]
operations = [
migrations.AddField(
model_name="podcast",
name="genre",
field=taggit.managers.TaggableManager(
help_text="A comma-separated list of tags.",
through="scrobbles.ObjectWithGenres",
to="scrobbles.Genre",
verbose_name="Tags",
),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.19 on 2025-04-07 17:29
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("podcasts", "0016_podcast_genre"),
]
operations = [
migrations.AddField(
model_name="podcast",
name="podcastindex_id",
field=models.CharField(blank=True, max_length=100, null=True),
),
]

View File

@ -1,5 +1,4 @@
import logging
from typing import Dict, Optional
from uuid import uuid4
import requests
@ -10,8 +9,13 @@ from django.db import models
from django.urls import reverse
from django.utils.translation import gettext_lazy as _
from django_extensions.db.models import TimeStampedModel
from podcasts.scrapers import scrape_data_from_google_podcasts
from scrobbles.mixins import ScrobblableMixin
from podcasts.sources.podcastindex import lookup_podcast_from_podcastindex
from scrobbles.mixins import (
ObjectWithGenres,
ScrobblableConstants,
ScrobblableMixin,
)
from taggit.managers import TaggableManager
logger = logging.getLogger(__name__)
BNULL = {"blank": True, "null": True}
@ -24,6 +28,13 @@ class Producer(TimeStampedModel):
def __str__(self):
return f"{self.name}"
@classmethod
def find_or_create(cls, name):
producer = cls.objects.filter(name__iexact=name).first()
if not producer:
producer = cls.objects.create(name=name)
return producer
class Podcast(TimeStampedModel):
name = models.CharField(max_length=255)
@ -31,11 +42,17 @@ class Podcast(TimeStampedModel):
producer = models.ForeignKey(
Producer, on_delete=models.DO_NOTHING, **BNULL
)
podcastindex_id = models.CharField(max_length=100, **BNULL)
owner = models.CharField(max_length=150, *BNULL)
description = models.TextField(**BNULL)
active = models.BooleanField(default=True)
feed_url = models.URLField(**BNULL)
google_podcasts_url = models.URLField(**BNULL)
site_link = models.URLField(**BNULL)
description = models.TextField(**BNULL)
cover_image = models.ImageField(upload_to="podcasts/covers/", **BNULL)
itunes_id = models.TextField(max_length=15, **BNULL)
dead_date = models.DateField(**BNULL)
genre = TaggableManager(through=ObjectWithGenres)
def __str__(self):
return f"{self.name}"
@ -49,32 +66,43 @@ class Podcast(TimeStampedModel):
user=user, podcast_episode__podcast=self
).order_by("-timestamp")
def scrape_google_podcasts(self, force=False):
podcast_dict = {}
if not self.cover_image or force:
podcast_dict = scrape_data_from_google_podcasts(self.name)
if podcast_dict:
if not self.producer:
self.producer, created = Producer.objects.get_or_create(
name=podcast_dict["producer"]
)
self.description = podcast_dict.get("description")
self.google_podcasts_url = podcast_dict.get("google_url")
self.save(
update_fields=[
"description",
"producer",
"google_podcasts_url",
]
)
@property
def itunes_link(self) -> str:
if not self.itunes_id:
return ""
return f"https://podcasts.apple.com/us/podcast/id{self.itunes_id}"
def fix_metadata(self, force=False):
if self.podcastindex_id and not force:
logger.warning(
"Podcast already has PodcastIndex ID, use force=True to overwrite"
)
return
podcast_dict = lookup_podcast_from_podcastindex(self.name)
if not podcast_dict:
logger.info(
"No podcast data found from PodcastIndex. Are credentials setup?"
)
return
genres = podcast_dict.pop("genres")
if genres:
self.genre.add(*genres)
cover_url = podcast_dict.pop("image_url")
cover_url = podcast_dict.get("image_url")
if (not self.cover_image or force) and cover_url:
r = requests.get(cover_url)
if r.status_code == 200:
fname = f"{self.name}_{self.uuid}.jpg"
self.cover_image.save(fname, ContentFile(r.content), save=True)
for attr, value in podcast_dict.items():
setattr(self, attr, value)
self.save()
class PodcastEpisode(ScrobblableMixin):
COMPLETION_PERCENT = getattr(settings, "PODCAST_COMPLETION_PERCENT", 90)
@ -84,6 +112,11 @@ class PodcastEpisode(ScrobblableMixin):
pub_date = models.DateField(**BNULL)
mopidy_uri = models.CharField(max_length=255, **BNULL)
def get_absolute_url(self):
return reverse(
"podcasts:podcast_detail", kwargs={"slug": self.podcast.uuid}
)
def __str__(self):
return f"{self.title}"
@ -91,6 +124,10 @@ class PodcastEpisode(ScrobblableMixin):
def subtitle(self):
return self.podcast
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Listening", tags="microphone")
@property
def info_link(self):
return ""
@ -104,42 +141,45 @@ class PodcastEpisode(ScrobblableMixin):
@classmethod
def find_or_create(
cls, podcast_dict: Dict, producer_dict: Dict, episode_dict: Dict
) -> Optional["Episode"]:
cls,
title: str,
podcast_name: str,
pub_date: str,
number: int = 0,
mopidy_uri: str = "",
producer_name: str = "",
run_time_seconds: int = 1800,
enrich: bool = True,
) -> "PodcastEpisode":
"""Given a data dict from Mopidy, finds or creates a podcast and
producer before saving the epsiode so it can be scrobbled.
"""
if not podcast_dict.get("name"):
logger.warning(f"No name from source for podcast, not scrobbling")
return
producer = None
if producer_dict.get("name"):
producer, producer_created = Producer.objects.get_or_create(
**producer_dict
if producer_name:
producer = Producer.find_or_create(producer_name)
podcast = Podcast.objects.filter(
name__iexact=podcast_name,
).first()
if not podcast:
podcast = Podcast.objects.create(
name=podcast_name, producer=producer
)
if producer_created:
logger.debug(f"Created new producer {producer}")
else:
logger.debug(f"Found producer {producer}")
if enrich:
podcast.fix_metadata()
if producer:
podcast_dict["producer_id"] = producer.id
podcast, podcast_created = Podcast.objects.get_or_create(
**podcast_dict
)
if podcast_created:
logger.debug(f"Created new podcast {podcast}")
else:
logger.debug(f"Found podcast {podcast}")
episode_dict["podcast_id"] = podcast.id
episode, created = cls.objects.get_or_create(**episode_dict)
if created:
logger.debug(f"Created new episode: {episode}")
else:
logger.debug(f"Found episode {episode}")
episode = cls.objects.filter(
title__iexact=title, podcast=podcast
).first()
if not episode:
episode = cls.objects.create(
title=title,
podcast=podcast,
run_time_seconds=run_time_seconds,
number=number,
pub_date=pub_date,
mopidy_uri=mopidy_uri,
)
return episode

View File

@ -0,0 +1,75 @@
import hashlib
import time
import pytz
import requests
from django.conf import settings
from django.utils import timezone
from scrobbles.utils import timestamp_user_tz_to_utc
PODCASTINDEX_API_KEY = getattr(settings, "PODCASTINDEX_API_KEY")
PODCASTINDEX_API_SECRET = getattr(settings, "PODCASTINDEX_API_SECRET")
def get_auth_headers():
now = int(time.time())
hash_data = hashlib.sha1(
(PODCASTINDEX_API_KEY + PODCASTINDEX_API_SECRET + str(now)).encode(
"utf-8"
)
).hexdigest()
return {
"User-Agent": "MyPodcastApp/1.0",
"X-Auth-Date": str(now),
"X-Auth-Key": PODCASTINDEX_API_KEY,
"Authorization": hash_data,
"Content-Type": "application/json",
}
def lookup_podcast_from_podcastindex(
podcast_name: str, dump_raw_response: bool = False
) -> dict:
url = "https://api.podcastindex.org/api/1.0/search/byterm"
headers = get_auth_headers()
params = {"q": podcast_name}
response = requests.get(url, headers=headers, params=params)
if response.status_code == 200:
data = response.json()
if dump_raw_response:
return data.get("feeds")
if data.get("feeds"):
try:
top_feed_dict = data["feeds"][0]
newest_episode_date = timestamp_user_tz_to_utc(
top_feed_dict.get("newestItemPubdate"), pytz.UTC
)
days_since_last_episode = ()
dead_date = None
if (timezone.now() - newest_episode_date).days > 180:
dead_date = newest_episode_date
return {
"podcastindex_id": top_feed_dict.get("id"),
"title": top_feed_dict.get("title"),
"site_link": top_feed_dict.get("link"),
"description": top_feed_dict.get("description"),
"owner": top_feed_dict.get("ownerName"),
"image_url": top_feed_dict.get("artwork"),
"feed_url": top_feed_dict.get("url"),
"itunes_id": top_feed_dict.get("itunesId"),
"genres": list(top_feed_dict.get("categories").values()),
"dead_date": dead_date,
}
except IndexError:
return {}
else:
print("No podcasts found.")
return {}
else:
print("Failed to fetch data:", response.status_code, response.text)
return {}

View File

@ -7,9 +7,14 @@ from profiles.models import UserProfile
class UserProfileAdmin(admin.ModelAdmin):
date_hierarchy = "created"
ordering = ("-created",)
readonly_fields = ("timezone_change_log",)
exclude = (
"twitch_token",
"twitch_client_secret",
"lastfm_password",
"webdav_pass",
"imap_pass",
"archivebox_password",
"todoist_auth_key",
"todoist_state",
)

View File

@ -0,0 +1,44 @@
from django import forms
from profiles.models import UserProfile
class UserProfileForm(forms.ModelForm):
def __init__(self, *args, **kwargs):
self.request = kwargs.pop("request")
self.profile = UserProfile.objects.filter(
user=self.request.user
).first()
if not self.profile:
raise Exception
super(UserProfileForm, self).__init__(
*args, **kwargs, instance=self.profile
)
class Meta:
model = UserProfile
fields = [
"timezone",
"lastfm_username",
"lastfm_password",
"lastfm_auto_import",
"retroarch_path",
"retroarch_auto_import",
"archivebox_username",
"archivebox_password",
"archivebox_url",
"bgg_username",
"lichess_username",
"webdav_url",
"webdav_user",
"webdav_pass",
"webdav_auto_import",
"ntfy_url",
"ntfy_enabled",
"redirect_to_webpage",
]
widgets = {
"lastfm_password": forms.PasswordInput(render_value=True),
"archivebox_password": forms.PasswordInput(render_value=True),
"webdav_pass": forms.PasswordInput(render_value=True),
}

Some files were not shown because too many files have changed in this diff Show More