Compare commits

...

84 Commits
50.0 ... 56.0

Author SHA1 Message Date
cb01781615 [release] Bump to version 56.0
Some checks failed
build / test (push) Failing after 1m33s
deploy / test (push) Failing after 1m35s
deploy / build-and-deploy (push) Has been skipped
- Add DiscGolf as a scrobbleable media
2026-06-20 00:37:42 -04:00
1f5fada8b1 [discgolf] Add new scrobble type
Some checks failed
build / test (push) Has been cancelled
2026-06-20 00:37:18 -04:00
31888a85cb [release] Bump to version 55.6
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 1m54s
deploy / build-and-deploy (push) Successful in 30s
- Figure out why historical Lastfm imports don't work
2026-06-19 14:12:44 -04:00
22d8b0787e [importers] Allow starting full import for new user
Some checks failed
build / test (push) Has been cancelled
2026-06-19 14:12:09 -04:00
8cc559752b [release] Bump to version 55.5
All checks were successful
build / test (push) Successful in 2m5s
deploy / test (push) Successful in 3m12s
deploy / build-and-deploy (push) Successful in 31s
- Fix bug in lastfm import for new users
2026-06-19 14:06:10 -04:00
db3f9696fa [importers] Fix smol bug in lastfm importer
Some checks failed
build / test (push) Has been cancelled
2026-06-19 14:05:46 -04:00
407d570c82 [release] Bump to version 55.4
All checks were successful
build / test (push) Successful in 2m12s
deploy / test (push) Successful in 2m4s
deploy / build-and-deploy (push) Successful in 1m22s
- Tighten up the speed of startup and first request
2026-06-19 13:41:23 -04:00
033239260f [perf] Add caching and lock protections
All checks were successful
build / test (push) Successful in 1m57s
2026-06-19 13:37:30 -04:00
9f854dc735 [release] Bump to version 55.3
All checks were successful
build / test (push) Successful in 2m6s
deploy / test (push) Successful in 2m3s
deploy / build-and-deploy (push) Successful in 39s
- =alt_names= feature for artists (commented out / dead code)
- Put chart rebuilds in a lower priority task queue
- Check for existing book scrobble and update page count
2026-06-19 01:14:39 -04:00
f29272a853 [music] Clean up dead code
Some checks failed
build / test (push) Has been cancelled
2026-06-19 01:13:28 -04:00
4e56d9420a [settings] Put chart rebuilds in their own queue 2026-06-19 01:12:50 -04:00
852a257159 [scrobbles] Clean up a TODO already done
All checks were successful
build / test (push) Successful in 1m59s
2026-06-18 16:59:03 -04:00
68ff230f13 [release] Bump to version 55.2
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 2m0s
deploy / build-and-deploy (push) Successful in 30s
- Fix bug in scrobble id in calendar view
- Video game cleanup script should clear out broken images
2026-06-18 15:27:29 -04:00
57a952a6d1 [templates] Fix bug in calendar view
Some checks failed
build / test (push) Has been cancelled
2026-06-18 15:27:09 -04:00
718fcf7392 [videogames] Fix broken images in cleanup
All checks were successful
build / test (push) Successful in 2m0s
2026-06-18 15:24:47 -04:00
52adcf83c7 [release] Bump to version 55.1
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 30s
- Clean up metadata scrapping for video games
2026-06-18 15:08:46 -04:00
0061623f7e [videogames] Fix metadata scrapping for video games
Some checks failed
build / test (push) Has been cancelled
2026-06-18 15:08:23 -04:00
ec73e5151e [release] Bump to version 55.0
All checks were successful
build / test (push) Successful in 1m58s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 37s
- Use pk ID for scrobble detail view, not uuid
- Display videogame screenshots on scrobble detail if they exist
- Add autotagging to webpages based on domain, title
2026-06-18 12:15:16 -04:00
2c90dd38b5 [project] Add todos 2026-06-18 12:14:58 -04:00
c6b1e42d7a [scrobbles] Use IDs not UUIDs in URLs
All checks were successful
build / test (push) Successful in 1m59s
2026-06-18 11:25:57 -04:00
fcf86d5b3f [scrobbles] Add screenshots to templates
All checks were successful
build / test (push) Successful in 2m6s
2026-06-18 10:54:28 -04:00
6fde9ec8d2 [webpages] Add autotagging to webpages
All checks were successful
build / test (push) Successful in 1m59s
2026-06-18 10:43:21 -04:00
0f1882b21f [release] Bump to version 54.5
All checks were successful
build / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Has been skipped
deploy / test (push) Successful in 1m58s
- Fix bug in generating mood trends
2026-06-17 21:46:21 -04:00
e819a2db0d [trends] Fix bug in mood trend generation 2026-06-17 21:45:45 -04:00
e03cf6c9b1 [release] Bump to version 54.4
All checks were successful
build / test (push) Successful in 2m0s
deploy / test (push) Successful in 1m58s
deploy / build-and-deploy (push) Successful in 29s
- Remove all-time trends
- Add a trend around moods
2026-06-17 17:09:30 -04:00
471e70ff7f [trends] Remove all time trends
Some checks failed
build / test (push) Has been cancelled
2026-06-17 17:09:11 -04:00
255e335d7a [trends] Add some mood related trends 2026-06-17 17:07:16 -04:00
c8cf80b513 [release] Bump to version 54.3
All checks were successful
deploy / test (push) Successful in 2m1s
build / test (push) Successful in 1m52s
deploy / build-and-deploy (push) Successful in 29s
- Fix bug in series metadata cleanup script
2026-06-17 12:05:50 -04:00
b4180afbed [videos] Fix metadata for series script
Some checks failed
build / test (push) Has been cancelled
2026-06-17 12:05:22 -04:00
37112babbb [release] Bump to version 54.2
All checks were successful
build / test (push) Successful in 1m59s
deploy / test (push) Successful in 1m56s
deploy / build-and-deploy (push) Successful in 30s
- Add script to clean up TV series metadata
- Update youtube video detail pages with links to channel
- Concurrent reading trend does not consolidate on single book
- Trends dont seem to look very far back
2026-06-17 11:06:11 -04:00
fb775f2f58 [videos] Add cmd to cleanup series metadata
Some checks failed
build / test (push) Has been cancelled
2026-06-17 11:05:38 -04:00
b26470c279 [videos] Fix channel templates
All checks were successful
build / test (push) Successful in 2m0s
2026-06-17 10:59:42 -04:00
d3b9ec815b [trends] Fix concurrent reading trend
All checks were successful
build / test (push) Successful in 2m14s
2026-06-17 10:50:59 -04:00
19f2b5e801 [trends] Add time periods 2026-06-17 10:50:16 -04:00
9e3288a5ff [release] Bump to version 54.1
All checks were successful
build / test (push) Successful in 2m0s
deploy / test (push) Successful in 1m58s
deploy / build-and-deploy (push) Successful in 35s
- Concurrent listening trend is inefficient and should be disabled
2026-06-17 09:21:20 -04:00
06465919dd [trends] Disable concurrent listening
Some checks failed
build / test (push) Has been cancelled
2026-06-17 09:20:29 -04:00
253e58eb48 [release] Bump to version 54.0
All checks were successful
build / test (push) Successful in 1m50s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 49s
- Add peak hour, weekly rhythm and activity dist trends
- Implement YouTube channel info scraping
- Fix Amazon book scraper
2026-06-16 16:42:44 -04:00
5393996e47 [trends] Add peak hours, weekly rhtyhms and activity dist trends
Some checks failed
build / test (push) Has been cancelled
2026-06-16 16:42:14 -04:00
1624f01e11 [videos] Increase metdata for Youtube
All checks were successful
build / test (push) Successful in 2m1s
2026-06-16 15:54:16 -04:00
535dead7e8 [videos] Clean up channel metadata
All checks were successful
build / test (push) Successful in 1m54s
2026-06-16 13:49:32 -04:00
3b97d49227 [books] Fix amazon scrapper to use actual AWS endpoints
All checks were successful
build / test (push) Successful in 2m2s
2026-06-16 13:18:20 -04:00
ea7b0946bb [release] Bump to version 53.1
All checks were successful
deploy / test (push) Successful in 1m58s
build / test (push) Successful in 1m56s
deploy / build-and-deploy (push) Successful in 34s
- Error with loading logdict
2026-06-16 11:31:14 -04:00
b8384166de [music] Fix logdata lookup for music
Some checks failed
build / test (push) Has been cancelled
2026-06-16 11:30:54 -04:00
d2705758c6 [release] Bump to version 53.0
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 1m59s
deploy / build-and-deploy (push) Successful in 52s
- Add a /trends/ page that shows trends based on scrobble data
- Notify users when Last.fm import completes
- Cleaner =GeoLocationLogData= deserialization
- Webpage scrobbles should diff existing webpages content
- Make ArchiveBox push asynchronous
2026-06-16 11:13:07 -04:00
f4368c31f3 [project] Update todos
Some checks failed
build / test (push) Has been cancelled
2026-06-16 11:12:50 -04:00
57f273b0cc [trends] Initial trends app
Some checks failed
build / test (push) Has been cancelled
2026-06-16 11:12:13 -04:00
ac82292200 [importers] Add last.fm import notifications 2026-06-16 10:26:08 -04:00
6a8432c08f [locations] Handle log data cleaner
All checks were successful
build / test (push) Successful in 1m54s
2026-06-16 10:12:58 -04:00
5a2c41155c [webpages] Add historical extract stashing
All checks were successful
build / test (push) Successful in 1m50s
2026-06-16 09:52:56 -04:00
83a046111b [webpages] Async pushing to archivebox
All checks were successful
build / test (push) Successful in 1m52s
2026-06-16 09:27:40 -04:00
ab10758f40 [release] Bump to version 52.2
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 1m56s
deploy / build-and-deploy (push) Successful in 50s
- Fix bug in recomputing long play seconds taking forever
2026-06-15 17:27:38 -04:00
88f16f0aaa [longplay] Fix recompute script
All checks were successful
build / test (push) Successful in 2m0s
2026-06-15 17:20:34 -04:00
c1744fab37 [release] Bump to version 52.1
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 29s
- Show time per scrobble in long play lists and total time playing
2026-06-15 15:27:10 -04:00
042a3eb737 [templates] Add aggregate data
Some checks failed
build / test (push) Has been cancelled
2026-06-15 15:26:46 -04:00
01d25e1b55 [release] Bump to version 52.0
All checks were successful
build / test (push) Successful in 1m55s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 31s
- Allow marking media as long play complete from detail page
- Fix how long play scrobbles are tracked
- Paginate or limite scrobbles on media admin pages
- Clean up books admin
- Clean up favorites admin
2026-06-15 14:56:27 -04:00
c0be131e3d [longplay] Add ability to undo finishes
Some checks failed
build / test (push) Has been cancelled
2026-06-15 14:55:58 -04:00
7d3f615ed7 [longplay] Make sure they're marked status is correct
All checks were successful
build / test (push) Successful in 2m2s
2026-06-15 14:37:20 -04:00
c2138b3ac6 [longplay] Add finish long play button
All checks were successful
build / test (push) Successful in 1m53s
2026-06-15 14:15:34 -04:00
947713d44a [longplay] Fix how we store long plays
All checks were successful
build / test (push) Successful in 2m3s
2026-06-15 14:12:21 -04:00
12b76837a3 [project] Update todos
All checks were successful
build / test (push) Successful in 2m11s
2026-06-15 13:43:58 -04:00
102494ede7 [admin] Use raw ids where possible and simplify scrobble inlines
All checks were successful
build / test (push) Successful in 1m55s
2026-06-15 13:25:10 -04:00
96bda8d4ad [data] Add example data 2026-06-15 12:38:21 -04:00
46956d06d8 [books] Clean up admin a little 2026-06-15 12:37:32 -04:00
8a28d0675b [release] Bump to version 51.4
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 38s
- Clean up metadata comicbook enrichment
2026-06-15 12:20:18 -04:00
5f6e75b14e [books] Fix comic book metadata importing
Some checks failed
build / test (push) Has been cancelled
2026-06-15 12:19:54 -04:00
a96a42cdbf [release] Bump to version 51.3
All checks were successful
build / test (push) Successful in 2m12s
deploy / test (push) Successful in 2m6s
deploy / build-and-deploy (push) Successful in 1m5s
- Improve speed of index and chart pages
2026-06-12 13:35:01 -04:00
c7f5d7d384 [charts] Add index to speed things up 2026-06-12 13:34:41 -04:00
d5830f5cd1 [release] Bump to version 51.2
All checks were successful
build / test (push) Successful in 2m14s
deploy / test (push) Successful in 2m15s
deploy / build-and-deploy (push) Successful in 1m27s
- Fix bug where last page of book gets separate scrobble
- Fix metadata scraping for books
2026-06-12 09:52:56 -04:00
c71b51fdb8 [books] Fix one-extra-scrobble bug on Koreader imports
Some checks failed
build / test (push) Has been cancelled
2026-06-12 09:52:33 -04:00
935d059a20 [books] Fix metadata scrapping
All checks were successful
build / test (push) Successful in 2m30s
It's not perfect, but at least we get covers now
2026-06-12 09:36:21 -04:00
25776eb495 [release] Bump to version 51.1
All checks were successful
build / test (push) Successful in 2m55s
deploy / test (push) Successful in 2m38s
deploy / build-and-deploy (push) Successful in 44s
- Fix scrobbling comic books
2026-06-11 19:07:00 -04:00
5ac4625af9 [books] Fix bug in scrobbling comic books
Some checks failed
build / test (push) Has been cancelled
2026-06-11 19:06:37 -04:00
a731427f6e [release] Bump to version 51.0
All checks were successful
build / test (push) Successful in 2m52s
deploy / test (push) Successful in 2m25s
deploy / build-and-deploy (push) Successful in 49s
- Fix koreader scrobble imports to use DST properly
- Fix book scrobbles where page_data is a list
- Lichess imports do not set default visbility
2026-06-11 18:39:31 -04:00
410da163fe [books] Fix koreader imports, maybe forever 2026-06-11 18:38:51 -04:00
a171192a6f [books] Fix list page_data for old book scrobbles
All checks were successful
build / test (push) Successful in 2m2s
2026-06-11 11:00:04 -04:00
c16b61db40 [importers] Fix strange bug where celery version mismatches
All checks were successful
build / test (push) Successful in 2m9s
2026-06-11 10:48:38 -04:00
29cb6a4991 [books] Really fix the subtitle bug 2026-06-11 10:48:21 -04:00
25c28e8335 [books] Fix bug in subtitle gen
All checks were successful
build / test (push) Successful in 2m5s
2026-06-11 10:34:20 -04:00
25626be3b6 [boardgames] Add visibility to lichess imports 2026-06-11 10:02:37 -04:00
0a880a2f2f [release] Bump to version 50.2
All checks were successful
build / test (push) Successful in 2m6s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Successful in 35s
- Koreader imports only import single-page scrobbles the next day
- Fix bugs in celery tasks causing imports to fail
2026-06-11 09:58:56 -04:00
248d3f2d3e [settings] Check webdav every two minuts
All checks were successful
build / test (push) Successful in 2m13s
2026-06-11 09:41:50 -04:00
e243fec679 [books] Try fixing the one-off import issue 2026-06-11 09:41:19 -04:00
de9b4ee9c1 [release] Bump to version 50.1
All checks were successful
build / test (push) Successful in 2m2s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 32s
- Fix bug in charts where only #1 is displayed
2026-06-09 22:08:48 -04:00
bf9a6a9679 [charts] Fix only seeing first top media instance 2026-06-09 22:08:15 -04:00
161 changed files with 7352 additions and 761 deletions

View File

@ -15,6 +15,8 @@ ro class method should call the utility function.
Be sure to check pyproject.toml for project defaults. Specifically for black and
isort expectations.
Imports in python files should always be top level if possible.
All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
In local development, environment variables for various sensitive values live in a .envrc file

View File

@ -88,8 +88,8 @@ fetching and simple saving.
*** Metadata sources
**** Scraper
* Backlog [0/15] :vrobbler:project:personal:
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
* Backlog [0/22] :vrobbler:project:personal:
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :bug:music:scrobbles:
:PROPERTIES:
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
:END:
@ -387,11 +387,11 @@ fetching and simple saving.
}
}
#+end_src
** TODO [#C] User should be able to enable auto trail tracking via amail reader with Garmin LiveTrack URLs :vrobbler:trails:project:feature:personal:
** TODO [#C] Allow auto trail tracking via email with Garmin LiveTrack URLs :trails:feature:
:PROPERTIES:
:ID: 133bcf71-078f-4efa-a029-1eae4b4d146d
:END:
** TODO [#C] Fix exporting so it works reliably :exporting:project:feature:
** TODO [#C] Fix exporting so it works reliably :exporting:feature:
*** Description
@ -405,8 +405,7 @@ placed in the media directory:
And this should all be done in a celery task that is just kicked off by the
"Export" button on the frontend
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
** TODO [#B] Add AllTrails as a source for Trail data :trails:feature:
:PROPERTIES:
:ID: 39313362-cdfe-46e7-bbd4-9139a65c0b3c
:END:
@ -416,7 +415,7 @@ Pretty clear, I would love to make trails more useful. Historically I wasn't
hiking a lot, which made the source for this a bit silly. But it's clear that
AllTrails is the best source, though having TrailForks is nice to.
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:personal:project:
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:
:PROPERTIES:
:ID: 5a4fb0f8-0555-40ec-b06f-93c26bd686f4
:END:
@ -440,43 +439,8 @@ added.
They should also probably support markdown formatting and that should be
displayed in the template.
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :books:feature:export:
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :vrobbler:books:feature:comicbook:personal:project:
- Note taken on [2025-09-25 Thu 10:51]
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
** TODO [#B] Find page numbers for comic books from ComicVine :feature:books:
:PROPERTIES:
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
:END:
** TODO [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
- Note taken on [2025-09-25 Thu 10:37] \\
This may already be fixed ... need to check.
- Note taken on [2025-02-25 12:34] \\
The page data has the canonical date something was read in it, but it seems
to be an hour off. I traced this back to being off during DST, so we just need
the importer to be aware of whether a user is using DST or not and roll back
an hour for part of the year. Also, we'd need to adjust any old scrobbles that
took place with DST off to roll them back by an hour.
*** Description
This is a long-standing problem when daylight saving time takes effect. Time is
manually set on a KoReader device (or at least, always saved in local time). So
whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
:PROPERTIES:
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
@ -500,7 +464,6 @@ needed import celery task. This is how the WebDAV celery task currently works.
This would also be an opporunity to clean up the code around WebDAV imports
and make them more re-usable for other import services.
** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
:PROPERTIES:
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
@ -512,7 +475,6 @@ Certain artists like "Simon & Garfunkel" are actually one artist. While we don't
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
this put into it and then we stop trying to pull the artist apart when we run into it.
** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
:PROPERTIES:
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
@ -544,6 +506,649 @@ log a warning and move on.
We should have a global view `/favorites/` that shows the logged in users's
favorited media objects.
** TODO [#B] Find page numbers for comic books from ComicVine :feature:books:
:PROPERTIES:
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
:END:
** TODO [#C] Implement loguru into project :feature:loguru:logging:
:PROPERTIES:
:ID: efcd0c0a-db81-4518-9c23-5505d59e8ef5
:END:
*** Description
Would be great to formalize how we log so we can search for errors and such more
easily. And our exposure to PII is really low at this point in the project,
so we can probably use backtrace=True and diagnose=True to help us root cause
bugs faster.
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :books:feature:comicbook:
:PROPERTIES:
:ID: b3cc57ca-3d2c-468d-ab7c-c47f1120309b
:END:
*** Description
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
** TODO [#C] Make podcast date format configurable in settings :podcasts:configuration:
:PROPERTIES:
:ID: b01a94f8-328f-41ed-a62e-8b99c755b82d
:END:
*** Description
=PODCAST_DATE_FORMAT= is hardcoded to ="YYYY-MM-DD"=. Should be in Django settings or environment variables for deploy-specific configuration.
File: ~vrobbler/apps/podcasts/utils.py~ (line 13)
** TODO [#C] Extract zombie scrobble query into custom manager :refactoring:manager:
:PROPERTIES:
:ID: 79c874e1-ca6f-4bce-9259-e3eebdca8a41
:END:
*** Description
The zombie scrobble cleanup query lives in a utility function. Should be a
custom model manager method (e.g. =Scrobble.objects.zombies()=).
File: ~vrobbler/apps/scrobbles/utils.py~ (line 204)
** TODO [#C] Allow profile to set start of week :profiles:configuration:
:PROPERTIES:
:ID: 0449279a-9550-430e-be0c-816df7273080
:END:
*** Description
=start_of_week()= and =end_of_week()= use Monday as default. Should be a user
profile setting for different cultural week start conventions.
File: ~vrobbler/apps/profiles/utils.py~ (lines 39, 44)
** TODO [#C] Add constants for data dictionary keys (multiple files) :refactoring:constants:
:PROPERTIES:
:ID: d4415f9b-620a-4be7-925d-fa71c02ba1d1
:END:
*** Description
Multiple files use magic string literals for dict keys. Should be extracted to
named constants for maintainability.
- Files:
- ~vrobbler/apps/locations/models.py~ (line 63) -- ="lat"=, ="lon"= etc.
- ~vrobbler/apps/webpages/models.py~ (line 290) -- ="url"=
- ~vrobbler/apps/scrobbles/importers/tsv.py~ (line 55) -- ="S"= completion status
** TODO [#A] Deduplicate BGG plays before posting :boardgames:bgg:duplication:
:PROPERTIES:
:ID: e9b842bf-0049-42e7-a060-f3ebd0067d2f
:END:
*** Description
No check for existing BGG plays before posting, which can create duplicates.
Should look up past plays by =bggeek_id= first.
File: ~vrobbler/apps/boardgames/bgg.py~ (line 117)
** TODO [#C] Clean up naming of =bgsplay= parsing :importers:refactoring:
:PROPERTIES:
:ID: c751dbbc-464a-4e63-9fe3-e034303f7b54
:END:
*** Description
We should rename `email_scrobble_board_game` to reflect the fact that it's just
a helper method to create board game scrobbles given a json blob. It's
independent of the email flow it was originally creatdd for
** TODO [#B] Is there way to create unique slugs for media instances :media_types:
* Version 56.0 [1/1]
** DONE [#B] Add DiscGolf as a scrobbleable media :discgolf:
:PROPERTIES:
:ID: 8cdde5d3-0ae5-7d5a-99d2-200c86afae03
:END:
*** Description
I have a csv file fro the uDisc disc golf scoring app that looks like:
Singles round. Note second row is the par for the course
#+begin_src csv
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
Par,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,27,,,3,3,3,3,3,3,3,3,3
Colin Powell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,30,3,,2,3,4,4,3,4,3,3,4
Asa Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,5,4,4,8,5,5,4,4,5
Emma Sweet,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,4,5,6,3,4,3,5,6
Jane Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,4,5,5,5,5,5,4,6,5
Nabby Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,59,32,,6,6,7,7,6,7,6,6,8
Silas Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,5,4,5,3,5,4,4,6`
#+end_src
Teams of two or more persons. Note second row is the par for the course
#+begin_src csv
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
Par,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,27,,,3,3,3,3,3,3,3,3,3
Colin Powell + Asa Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,29,2,,3,4,2,3,2,3,5,3,4
Emma Sweet + Jane Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,28,1,,4,3,4,2,3,4,3,3,2
#+end_src
We should add a new app called discgolf that has the following data models:
- DiscGolfRound - scrobblable media + course_id, round_type (Singles, Teams)
- DiscGolfCourse - name, layoutname, number_of_holes
And the logdata for a DiscGolfOuting scrobble should have:
+ {person: {hole_number: score}, total: int}
+ {team: {name: "", people: [person, person], hole_number: score}, total: int}
+ weather
+ fun_factor (miserable, not great, so-so, good, excellent, party time)
* Version 55.6 [1/1]
** DONE [#A] Figure out why historical Lastfm imports don't work :importers:lastfm:music:
:PROPERTIES:
:ID: 71b18c1b-de96-6d93-20fa-de2ec0df1288
:END:
* Version 55.5 [1/1]
** DONE [#B] Fix bug in lastfm import for new users :importers:lastfm:music:
:PROPERTIES:
:ID: d034966d-0c7f-e512-4cf8-7329c9026b6f
:END:
* Version 55.4 [1/1]
** DONE [#A] Tighten up the speed of startup and first request :perf:
:PROPERTIES:
:ID: 9ee8834c-6be2-d04b-df6d-56375504083f
:END:
* Version 55.3 [3/3]
** DONE [#C] =alt_names= feature for artists (commented out / dead code) :music:dead-code:
:PROPERTIES:
:ID: e22060a2-5f7a-4f33-9056-309ecd27159c
:END:
*** Description
File: ~vrobbler/apps/music/models.py~ (line 236)
An entire block of code for tracking alternate artist names is commented
out. The TODO questions whether it even works. Review: either implement
properly or remove the dead code.
** DONE [#A] Put chart rebuilds in a lower priority task queue :charts:tasks:
:PROPERTIES:
:ID: 43c90de0-fc1c-1139-dac7-9b7c82006b2e
:END:
** DONE [#A] Check for existing book scrobble and update page count :books:scrobbling:
:PROPERTIES:
:ID: 1a0609bc-6b16-4da4-96c1-59588229e4b4
:END:
*** Description
File: ~vrobbler/apps/scrobbles/scrobblers.py~ (line 330)
When scrobbling a book (comic), the code doesn't check for prior scrobbles to
update reading progress. Needed for proper page-count tracking.
* Version 55.2 [2/2]
** DONE [#A] Fix bug in scrobble id in calendar view :templates:
:PROPERTIES:
:ID: 8cb34852-b18f-e794-cd9b-fb1ecad70a0d
:END:
** DONE [#A] Video game cleanup script should clear out broken images :metadata:videogames:
:PROPERTIES:
:ID: ca1f1ea9-0f79-082c-5ff7-867671faff4b
:END:
* Version 55.1 [1/1]
** DONE [#A] Clean up metadata scrapping for video games :metadata:videogames:
:PROPERTIES:
:ID: fbc421b5-21a3-4aed-9062-c59192ead065
:END:
* Version 55.0 [3/3]
** DONE [#B] Use pk ID for scrobble detail view, not uuid :scrobbles:
:PROPERTIES:
:ID: 9cc3b285-e478-041e-394b-3d550aefbe1d
:END:
** DONE [#B] Display videogame screenshots on scrobble detail if they exist :videogames:templates:
:PROPERTIES:
:ID: 0406d082-20f6-0d12-76e2-f281c4801468
:END:
** DONE [#B] Add autotagging to webpages based on domain, title :webpages:metadata:
:PROPERTIES:
:ID: f658435b-f7a0-42e6-b9f6-226678a77a55
:END:
*** Description
For easier filtering, like we do with tasks, we should auto tag WebPage instances
based on the domain name split part by periods (so news.ycombinator.com tags: news, ycombinator, com)
And also based on the nouns in the title.
* Version 54.5 [1/1]
** DONE Fix bug in generating mood trends :trends:
:PROPERTIES:
:ID: 8e75abfa-8e70-d85b-00a4-a4813bbce879
:END:
* Version 54.4 [2/2]
** DONE [#A] Remove all-time trends :trends:
:PROPERTIES:
:ID: 53b231d1-7677-8cd3-1d88-dae110aba1e6
:END:
*** Description
All time trends take forever to calculate and don't provide too much data
** DONE [#B] Add a trend around moods :moods:trends:
:PROPERTIES:
:ID: fba3f4ae-8f97-ee0b-e762-31630884518a
:END:
* Version 54.3 [1/1]
** DONE [#B] Fix bug in series metadata cleanup script :videos:metadta:
:PROPERTIES:
:ID: 85448702-907c-5d63-f5af-7795661d7c46
:END:
* Version 54.2 [4/4]
** DONE [#B] Add script to clean up TV series metadata :videos:metadata:
:PROPERTIES:
:ID: a468b328-59d9-f84b-9ddb-087216783453
:END:
** DONE [#A] Update youtube video detail pages with links to channel :videos:templates:
:PROPERTIES:
:ID: 8b87cb42-09e5-a3f5-136f-182f967fa81f
:END:
** DONE [#A] Concurrent reading trend does not consolidate on single book :trends:reading:
:PROPERTIES:
:ID: fe220f55-7e0d-2a17-2477-a5aa7c4a1f2c
:END:
** DONE [#B] Trends dont seem to look very far back :trends:
:PROPERTIES:
:ID: ffcfba3f-5a93-9ee0-9680-666e6eccd684
:END:
*** Description
Specificially, looking at reading-pace when run on prod, it claims that I've
only had one reading session without music. Which may be true, but perhaps we
need to indicate what the time frame we're looking at is (month, week, year)
and provide a way to jump back and forward through time, same as charts.
* Version 54.1 [1/1]
** DONE [#A] Concurrent listening trend is inefficient and should be disabled :trends:scrobbles:
:PROPERTIES:
:ID: 4aa3b719-6b22-cae9-85f0-fac67b4fc753
:END:
* Version 54.0 [3/3]
** DONE [#B] Add peak hour, weekly rhythm and activity dist trends :trends:scrobbles:
:PROPERTIES:
:ID: 5fa52fac-d5f0-4369-bcaa-589c886b07d3
:END:
** DONE [#A] Implement YouTube channel info scraping :videos:youtube:stub:
:PROPERTIES:
:ID: 1d3beafd-62cb-4735-a465-edb37bf885db
:END:
*** Description
File: ~vrobbler/apps/videos/models.py~ (line 140)
=Video.fix_metadata()= is a stub that logs "Not implemented yet" and returns.
Needs actual implementation to scrape channel metadata from YouTube.
** DONE [#A] Fix Amazon book scraper :amazon:scraper:broken:
:PROPERTIES:
:ID: c38aba25-0171-49ab-a9f3-acf2003da429
:END:
*** Description
File: ~vrobbler/apps/books/amazon.py~ (line 56)
The =scrape_data_from_amazon()= function is likely broken due to Amazon blocking
scrapers and changing HTML structure. Needs rewrite or replacement with a proper
API.
* Version 53.1 [1/1]
** DONE [#A] Error with loading logdict :scrobbles:bug:logdata:
:PROPERTIES:
:ID: 92d4fa16-4b90-47e0-95ae-472bdca582ce
:END:
* Version 53.0 [5/5]
** DONE [#B] Add a /trends/ page that shows trends based on scrobble data :feature:trends:scrobbles:
:PROPERTIES:
:ID: 03e9fe30-2bc6-4062-bb24-e95b98daf05b
:END:
*** Description
This project is a bit invovled. But we should add a top level URL `trends` that shows
various trends as defined either in a static settings file, or dynamically via a database table.
Trends could be things like doing multiple things at the same time, like while driving, what
did we listen to this week, or while running, what were listening to this week?
Or more complicated trends like, how time per page changes based on the book I was reading, or if I was doing something else (music or sport event) while reading.
** DONE [#B] Notify users when Last.fm import completes :importers:notifications:
:PROPERTIES:
:ID: 92846b36-54c5-4b78-9c57-bdc401045fbe
:END:
*** Description
After a bulk import from Last.fm, users receive no confirmation. Should add a notification (in-app, email, or similar).
File: ~vrobbler/apps/scrobbles/importers/lastfm.py~ (line 96)
** DONE [#C] Cleaner =GeoLocationLogData= deserialization :models:refactoring:
:PROPERTIES:
:ID: 85465dbf-69b3-48cb-9df0-cd076c4470ab
:END:
*** Description
Currently special-cases =GeoLocationLogData= by reaching into a nested ="movement_detection"= key. Should be handled at the LogData dataclass level.
File: ~vrobbler/apps/scrobbles/models.py~ (line 977)
** DONE [#B] Webpage scrobbles should diff existing webpages content :webpages:metadata:
:PROPERTIES:
:ID: 25576197-258f-48d6-bfe9-e4172a0a1898
:END:
*** Description
Webpages change content between scrobbles. The current model stores the webpage content once, the
first time it's scrobbled. When a page has been seen before, we should move the existing content
to a new model HistoricalWebPage with the following fields:
webpage_id -> FK to WebPage
date -> date from existing WebPage content
domain -> same as existing WebPage content
extract -> copy of existing WebPage content
Once the HistoricalWebPage instance is successfully created, the new extract data
should be saved into the WebPage instance.
** DONE [#B] Make ArchiveBox push asynchronous :archivebox:async:
:PROPERTIES:
:ID: 17c116a7-5952-db37-e56c-2987c2fc456b
:END:
*** Description
=push_to_archivebox()= runs synchronously during the request. Should be moved to a
Celery task or similar background worker.
File: ~vrobbler/apps/webpages/models.py~ (line 133)
* Version 52.2 [1/1]
** DONE [#A] Fix bug in recomputing long play seconds taking forever :bug:longplay:commands:
:PROPERTIES:
:ID: 0a813cf9-17fb-dbd7-b5a7-7410d9bd4d8c
:END:
* Version 52.1 [1/1]
** DONE [#C] Show time per scrobble in long play lists and total time playing :templates:longplay:scrobbles:
:PROPERTIES:
:ID: b3d16230-8ec5-46db-b166-59e98d0ee06c
:END:
*** Description
Long play time should be show in the table of scrobbles on a media detail page.
The total time spent in a long play that's either no completed yet or completed
should be displayed as well. If completed, the date finished should be shown as
well.
* Version 52.0 [5/5]
** DONE [#B] Allow marking media as long play complete from detail page :templates:scrobbles:longplay:
:PROPERTIES:
:ID: 2c314768-be97-9b10-d13c-9cfd0f38a64e
:END:
** DONE [#A] Fix how long play scrobbles are tracked :scrobbles:longplay:serial:
:PROPERTIES:
:ID: 908b0493-cabf-40c1-825f-cd59a8ad0f7a
:END:
*** Description
Currently we have this idea of "long_play" scrobbles but there's a lot missing
to tie it together.
What we'd prefer is that when a new scrobble is added for a media_type that
`is_long_play` the most recent scrobble finished is added as the
`last_serial_scrobble` to the log data. But all the other long play stuff exsits
as data model fields. We should add `long_play_last_scrobble` as a FK to this
scrobble when creating a new longplay scrobble.
Additionally, `long_play_seconds` we should have a recompute management command
to walk backward from `long_play_last_scrobble` until a `long_play_complete`
scrobble is found (exclusive) and save the time.
We should also ony use `long_play_complete` field on the scrobble ... some
logdatas have a similar field, but we should make sure that we always use the
model field to determine if a long play is finished.
This should include a command to clean up long play data to consolidate around
the `long_play_complete` field.
** DONE [#B] Paginate or limite scrobbles on media admin pages :admin:scrobbles:media:
:PROPERTIES:
:ID: f02e487b-d7ed-4834-838a-303560f2ad3b
:END:
** DONE [#B] Clean up books admin :admin:books:bug:
:PROPERTIES:
:ID: 7539bee6-0a52-26f6-ebc6-5554ac49a716
:END:
** DONE [#B] Clean up favorites admin :admin:favorites:scrobbles:
:PROPERTIES:
:ID: f2be0c69-1bf8-b5a3-5269-9c8ea873361d
:END:
*** Description
Some FK lookups in admin should be raw_id_fields.
* Version 51.4 [1/1]
** DONE [#A] Clean up metadata comicbook enrichment :bug:comics:books:metadata:
:PROPERTIES:
:ID: cd875450-7117-78ca-8be4-9c8b73037dba
:END:
*** Description
Still getting wonky results with some comicbooks. Would be nice to be able to
tag a Book as a comicbook, and also gather volume information. I also noticed
that some books that are found in OL never get their comicvine_id populated. We
should make sure we always have comicvine_ids if available.
* Version 51.3 [1/1]
** DONE [#A] Improve speed of index and chart pages :bug:scrobbles:perf:
:PROPERTIES:
:ID: 031a23f8-7c02-4926-9884-6654ceca16c2
:END:
*** Description
Over the last few releases, the home page and charts pages have gotten really
slow.
We should look into what's causing the slowness and maybe do more agressive
query optimization or caching.
* Version 51.2 [2/2]
** DONE [#A] Fix bug where last page of book gets separate scrobble :bug:books:importers:koreader:
:PROPERTIES:
:ID: e13e0b4c-461e-e5a9-c685-b972f4e262e5
:END:
*** Description
The new KoReader code is working great to import books with the correct timezone
and what-not. But it has a weird artifact of creating one extra scrobble for the
last page read. Need to button that up.
** DONE [#B] Fix metadata scraping for books :books:metadata:
:PROPERTIES:
:ID: ea416a69-a8a8-4d05-b7d4-0a3470820e34
:END:
* Version 51.1 [1/1]
** DONE [#A] Fix scrobbling comic books :books:scrobbles:bug:
:PROPERTIES:
:ID: 8dfbff19-3fa4-f3b8-21c7-7a416498000c
:END:
*** Description
At some point logdata and log got confused, and now when you try
to scrobble a comic book, it just throws errors. We should look
into where the confusion happened and fix it.
* Version 51.0 [3/3]
** DONE [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
- Note taken on [2025-09-25 Thu 10:37] \\
This may already be fixed ... need to check.
- Note taken on [2025-02-25 12:34] \\
The page data has the canonical date something was read in it, but it seems
to be an hour off. I traced this back to being off during DST, so we just need
the importer to be aware of whether a user is using DST or not and roll back
an hour for part of the year. Also, we'd need to adjust any old scrobbles that
took place with DST off to roll them back by an hour.
*** Description
This is a long-standing problem when daylight saving time takes effect. Time is
manually set on a KoReader device (or at least, always saved in local time). So
whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** DONE [#A] Fix book scrobbles where page_data is a list :bug:books:scrobbles:
:PROPERTIES:
:ID: 35b323fa-ccc0-4009-b227-8a0f12bbd469
:END:
*** Description
Comic scrobbling is currently kind of janky. Most of the problems boil down to them
storing saved page data in a list of dicts rather than a dict keyed off of the page number that
was read.
We need to adjust comic scrobbling to use a dict of pages keyed off the page number, and also
write a migration script that runs as a data migration to update any book scrobbles that may
have page_data as a list.
*** Example data
#+begin_src python
{"notes": null, "page_end": 27, "page_data": [{"notes": null, "end_ts": 1771815895, "duration": 14, "start_ts": 1771815881, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815908, "duration": 13, "start_ts": 1771815895, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815913, "duration": 5, "start_ts": 1771815908, "description": null, "page_number": "2"}, {"notes": null, "end_ts": 1771815933, "duration": 20, "start_ts": 1771815913, "description": null, "page_number": "3"}, {"notes": null, "end_ts": 1771815945, "duration": 12, "start_ts": 1771815933, "description": null, "page_number": "4"}, {"notes": null, "end_ts": 1771815983, "duration": 38, "start_ts": 1771815945, "description": null, "page_number": "5"}, {"notes": null, "end_ts": 1771816007, "duration": 24, "start_ts": 1771815983, "description": null, "page_number": "6"}, {"notes": null, "end_ts": 1771816011, "duration": 4, "start_ts": 1771816007, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816013, "duration": 2, "start_ts": 1771816011, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816052, "duration": 39, "start_ts": 1771816013, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816127, "duration": 75, "start_ts": 1771816052, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816134, "duration": 7, "start_ts": 1771816127, "description": null, "page_number": "9"}, {"notes": null, "end_ts": 1771816196, "duration": 62, "start_ts": 1771816134, "description": null, "page_number": "10"}, {"notes": null, "end_ts": 1771816262, "duration": 66, "start_ts": 1771816196, "description": null, "page_number": "11"}, {"notes": null, "end_ts": 1771816293, "duration": 31, "start_ts": 1771816262, "description": null, "page_number": "12"}, {"notes": null, "end_ts": 1771816322, "duration": 29, "start_ts": 1771816293, "description": null, "page_number": "13"}, {"notes": null, "end_ts": 1771816330, "duration": 8, "start_ts": 1771816322, "description": null, "page_number": "14"}, {"notes": null, "end_ts": 1771816368, "duration": 38, "start_ts": 1771816330, "description": null, "page_number": "15"}, {"notes": null, "end_ts": 1771816388, "duration": 20, "start_ts": 1771816368, "description": null, "page_number": "16"}, {"notes": null, "end_ts": 1771816482, "duration": 94, "start_ts": 1771816388, "description": null, "page_number": "17"}, {"notes": null, "end_ts": 1771816550, "duration": 68, "start_ts": 1771816482, "description": null, "page_number": "18"}, {"notes": null, "end_ts": 1771816567, "duration": 17, "start_ts": 1771816550, "description": null, "page_number": "19"}, {"notes": null, "end_ts": 1771816586, "duration": 19, "start_ts": 1771816567, "description": null, "page_number": "20"}, {"notes": null, "end_ts": 1771816597, "duration": 11, "start_ts": 1771816586, "description": null, "page_number": "21"}, {"notes": null, "end_ts": 1771816616, "duration": 19, "start_ts": 1771816597, "description": null, "page_number": "22"}, {"notes": null, "end_ts": 1771816640, "duration": 24, "start_ts": 1771816616, "description": null, "page_number": "23"}, {"notes": null, "end_ts": 1771816690, "duration": 50, "start_ts": 1771816640, "description": null, "page_number": "24"}, {"notes": null, "end_ts": 1771816702, "duration": 12, "start_ts": 1771816690, "description": null, "page_number": "25"}, {"notes": null, "end_ts": 1771816823, "duration": 121, "start_ts": 1771816702, "description": null, "page_number": "26"}, {"notes": null, "end_ts": null, "duration": null, "start_ts": 1771816823, "description": null, "page_number": "27"}], "page_start": 1, "pages_read": 27, "resume_url": null, "description": null, "koreader_hash": null, "long_play_complete": false}
#+end_src
** DONE [#A] Lichess imports do not set default visbility :boardgames:bug:importers:lichess:
:PROPERTIES:
:ID: a78f7c72-a20a-8db2-cde0-d92a731d4fba
:END:
* Version 50.2 [2/2]
** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers:
:PROPERTIES:
:ID: b50141fd-cda6-4a3a-afd3-cd8499e7523e
:END:
*** Description
When you read a single page in a book in Koreader and try to import it, the scrobble is only
created the day after, not on the day of the reading.
** DONE [#A] Fix bugs in celery tasks causing imports to fail :bug:celery:tasks:
:PROPERTIES:
:ID: d1171cb0-6413-44b8-a68a-019a4d2fb285
:END:
*** Description
Seems like all celery tasks are failing for different reasons except the chart
updates.
*** Errors
**** scrobbles.tasks.send_notification_for_in_progress
#+begin_src bash
KeyError: 'track'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedColumn: column music_track.artist_id does not exist
LINE 1: ..."."title", "music_track"."base_run_time_seconds", "music_tra...
^
HINT: Perhaps you meant to reference the column "music_track.artist_fk_id".
#+end_src
**** scrobbles.tasks.import_from_webdav_all_users
#+begin_src bash
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/importers/webdav.py", line 166, in scan_webdav_for_koreader
if last_import and last_import.webdav_etag and remote_etag:
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'KoReaderImport' object has no attribute 'webdav_etag'
#+end_src
**** scrobbles.tasks.process_bgstats_import
#+begin_src bash
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.NotNullViolation: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
DETAIL: Failing row contains (374463, 2026-06-11 13:27:06.528319+00, 2026-06-11 13:27:06.52834+00, 2026-06-11 13:17:34+00, 180, f, f, BG Stats, 1, null, t, {"players": [{"new": false, "win": false, "rank": 0, "role": "",..., null, null, null, 8e73ceec-b731-4623-9637-712bbf9f76ce, null, null, null, null, , null, BoardGame, 324, null, null, America/New_York, null, null, , null, null, , null, null, null, null, null, null, null, null, null, null, null).
#+end_src
* Version 50.1 [1/1]
** DONE [#B] Fix bug in charts where only #1 is displayed :charts:templates:
:PROPERTIES:
:ID: 7136dffb-e6b7-184b-48ac-bb09bae0b0f0
:END:
* Version 50.0 [2/2]
** DONE [#A] Allow updating all a user's scrobble visibility at once :scrobbles:sharing:feature:
:PROPERTIES:

View File

@ -1,2 +1,2 @@
web: python manage.py runserver 0.0.0.0:8014
worker: celery -A vrobbler worker -l DEBUG
worker: celery -A vrobbler worker -Q default,charts -l DEBUG

96
data/play-example.json Normal file
View File

@ -0,0 +1,96 @@
{
"about": "This is a Play file that can be read by Board Game Stats. If you see this text, try to use a share, export or open-in function to open it with Board Game Stats.",
"players": [
{
"uuid": "31f8b92e-11d8-4162-88b1-fd9c79eea249",
"id": 2,
"name": "Colin Powell",
"isAnonymous": false,
"modificationDate": "2025-10-18 08:32:40",
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"AnnikaHeller_Capybara.webp\",\"shape\":[1,1,1,1],\"color\":[0.6,0.20000000298023224,0.9803921580314636]}}"
},
{
"uuid": "dd2d1881-84ab-474c-a6b2-3045d034dc40",
"id": 3,
"name": "Silas Sewell",
"isAnonymous": false,
"modificationDate": "2026-01-18 12:27:12",
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"RoryMuldoon_07.webp\",\"shape\":[1,1,1,1],\"color\":[0,0,0.1835034190722739]}}"
}
],
"locations": [
{
"uuid": "14f7389c-767f-4725-9b35-906c407b293c",
"id": 3,
"name": "Timberwyck Farm",
"modificationDate": "2025-07-01 18:03:38"
}
],
"games": [
{
"uuid": "9e431cdd-b325-4061-a875-d415d46342c0",
"id": 1046,
"name": "Sweet Takes",
"modificationDate": "2026-04-11 16:25:35",
"cooperative": false,
"highestWins": true,
"noPoints": false,
"usesTeams": false,
"urlThumb": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__small/img/gmGqlmwe9fniqYpniGwhg5RUgVQ=/fit-in/200x150/filters:strip_icc()/pic8784202.jpg",
"urlImage": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__original/img/b4IU8WIEWRpacbXp0FHG9HfFRpw=/0x0/filters:format(jpeg)/pic8784202.jpg",
"bggName": "Sweet Takes",
"bggYear": 2023,
"bggId": 407581,
"designers": "Hisashi Hayashi",
"isBaseGame": 1,
"isExpansion": 0,
"rating": 67,
"minPlayerCount": 2,
"maxPlayerCount": 5,
"minPlayTime": 15,
"maxPlayTime": 15,
"minAge": 8
}
],
"plays": [
{
"uuid": "7b2fd583-e8f2-40fe-9565-90178390b87e",
"modificationDate": "2026-04-16 20:18:03",
"entryDate": "2026-04-16 20:13:33",
"playDate": "2026-04-16 20:13:33",
"usesTeams": false,
"durationMin": 4,
"ignored": false,
"manualWinner": false,
"rounds": 0,
"locationRefId": 3,
"gameRefId": 1046,
"board": "",
"scoringSetting": 1,
"metaData": "{\"playerRefId\":2,\"playGameBggVersion\":\"{\\\"versionId\\\":0,\\\"versionName\\\":\\\"\\\",\\\"imageUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"thumbUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"yearPublished\\\":0}\",\"playUsedGameCopy\":2}",
"playerScores": [
{
"score": "27",
"winner": false,
"newPlayer": false,
"startPlayer": false,
"playerRefId": 2,
"role": "",
"rank": 0,
"seatOrder": 0
},
{
"score": "36",
"winner": true,
"newPlayer": true,
"startPlayer": false,
"playerRefId": 3,
"rank": 0,
"seatOrder": 0
}
],
"expansionPlays": []
}
],
"userInfo": { "meRefId": 2 }
}

BIN
data/statistics.sqlite3 Normal file

Binary file not shown.

25
poetry.lock generated
View File

@ -4270,6 +4270,29 @@ six = "*"
[package.extras]
testing = ["filelock"]
[[package]]
name = "python-amazon-paapi"
version = "6.3.0"
description = "Amazon Product Advertising API 5.0 wrapper for Python"
optional = false
python-versions = ">=3.9"
groups = ["main"]
files = [
{file = "python_amazon_paapi-6.3.0-py3-none-any.whl", hash = "sha256:b7cd852084a49d53c3ba2195531fccbc8c7f4124b2e82e2fda02b53d3b8de521"},
{file = "python_amazon_paapi-6.3.0.tar.gz", hash = "sha256:e525d69efcbe4f9566ec2b9b43fa3183c484d166d3852edb38b4df9c0b19cf1f"},
]
[package.dependencies]
certifi = ">=2023.0.0"
pydantic = ">=2.0.0"
python-dateutil = ">=2.8.0"
requests = ">=2.28.0"
six = ">=1.16.0"
urllib3 = ">=1.26.0,<3"
[package.extras]
async = ["httpx (>=0.27.0)", "typing-extensions (>=4.15.0)"]
[[package]]
name = "python-dateutil"
version = "2.9.0.post0"
@ -6032,4 +6055,4 @@ cffi = ["cffi (>=1.17,<2.0) ; platform_python_implementation != \"PyPy\" and pyt
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<3.15"
content-hash = "cc5b3b44071d6b0ab4f05189580232cc129b4ed694ab3f0673c3d838c3af0f8a"
content-hash = "aafab54d3c3d674b917782bf449b7d6324ca2259fb58bff13a08caabe110c342"

View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "vrobbler"
version = "50.0"
version = "56.0"
description = ""
authors = ["Colin Powell <colin@unbl.ink>"]
@ -64,6 +64,7 @@ fitparse = "^1.2.0"
lxml = ">=5.5.0"
vaderSentiment = "^3.3.2"
sqids = "^0.5.2"
python-amazon-paapi = "^6.3.0"
[tool.poetry.group.test]
optional = true

View File

@ -519,7 +519,7 @@ def test_scrobble_detail_view_with_notes_as_flat_list(client):
"description": "Test description",
},
)
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 200
assert "First note" in response.content.decode()
@ -545,7 +545,7 @@ def test_scrobble_detail_view_with_notes_as_dict_timestamps(client):
"description": "Test description",
},
)
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
@ -574,7 +574,7 @@ def test_scrobble_detail_view_with_notes_and_labels(client):
"description": "Test description",
},
)
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
@ -597,7 +597,7 @@ def test_scrobble_detail_view_post_updates_log(client):
"description": "Original description",
},
)
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
client.force_login(user)
response = client.post(
@ -896,7 +896,7 @@ def test_change_visibility_owner_can_change(client):
)
client.force_login(user)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
@ -919,7 +919,7 @@ def test_change_visibility_non_owner_gets_404(client):
)
client.force_login(other)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 404
@ -937,7 +937,7 @@ def test_change_visibility_anonymous_redirects_to_login(client):
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
assert "/login/" in response.url
@ -956,7 +956,7 @@ def test_regenerate_share_token_invalidates_old_sqid(client):
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
client.force_login(user)
url = reverse("scrobbles:regenerate-share-token", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:regenerate-share-token", kwargs={"pk": scrobble.id})
response = client.post(url)
assert response.status_code == 302
@ -985,7 +985,7 @@ def test_share_analytics_owner_can_view(client):
)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 200
@ -1005,7 +1005,7 @@ def test_share_analytics_non_owner_gets_404(client):
)
client.force_login(other)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 404
@ -1027,7 +1027,7 @@ def test_share_analytics_shows_view_logs(client):
client.get(share_url)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()

View File

@ -4,9 +4,15 @@ from unittest.mock import MagicMock, patch
import pytest
from vrobbler import context_processors
from vrobbler.context_processors import version_info
@pytest.fixture(autouse=True)
def reset_git_cache():
context_processors._GIT_COMMIT = None
@pytest.fixture
def mock_request():
return MagicMock()

View File

@ -27,6 +27,7 @@ class BeerAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("styles", "producer")
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -27,4 +27,5 @@ class BirdingLocationAdmin(admin.ModelAdmin):
class BirdingCSVImportAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
raw_id_fields = ("user",)
ordering = ("-created",)

View File

@ -38,6 +38,7 @@ class BoardGameLocationAdmin(admin.ModelAdmin):
"uuid",
"geo_location",
)
raw_id_fields = ("geo_location",)
ordering = ("-created",)
@ -49,6 +50,7 @@ class BoardGameAdmin(admin.ModelAdmin):
"title",
"published_year",
)
raw_id_fields = ("publisher", "publishers", "designers", "expansion_for_boardgame")
search_fields = ("title",)
ordering = ("-created",)
inlines = [

View File

@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
"source": "Lichess",
"timezone": user.profile.timezone,
"log": log_data,
"visibility": "private",
}
if commit:
Scrobble.objects.create(**scrobble_dict)

View File

@ -27,6 +27,7 @@ class BookAdmin(admin.ModelAdmin):
"first_publish_year",
"pages",
)
raw_id_fields = ("authors",)
search_fields = ("name",)
ordering = ("-created",)
inlines = [
@ -34,11 +35,11 @@ class BookAdmin(admin.ModelAdmin):
]
def issue_or_volume(self, obj):
return obj.issue_number or obj.volume_number
return obj.subtitle
@admin.register(Paper)
class BookAdmin(admin.ModelAdmin):
class PaperAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
@ -47,6 +48,7 @@ class BookAdmin(admin.ModelAdmin):
"first_publish_year",
"pages",
)
raw_id_fields = ("authors",)
search_fields = ("name",)
ordering = ("-created",)
inlines = [

View File

@ -1,128 +0,0 @@
from enum import Enum
from typing import Optional
from bs4 import BeautifulSoup
import requests
import logging
logger = logging.getLogger(__name__)
USER_AGENT = "Mozilla/5.0 (Android 4.4; Mobile; rv:41.0) Gecko/41.0 Firefox/41.0"
AMAZON_SEARCH_URL = "https://www.amazon.com/s?k={amazon_id}"
class AmazonAttribute(Enum):
SERIES = 0
PAGES = 1
LANGUAGE = 2
PUBLISHER = 3
PUB_DATE = 4
DIMENSIONS = 5
ISBN_10 = 6
ISBN_13 = 7
def strip_and_clean(text):
return text.strip("\n").rstrip().lstrip()
def get_rating_from_soup(soup) -> Optional[int]:
rating = None
try:
potential_rating = soup.find("div", class_="allmusic-rating")
if potential_rating:
rating = int(strip_and_clean(potential_rating.get_text()))
except ValueError:
pass
return rating
def get_review_from_soup(soup) -> str:
review = ""
try:
potential_text = soup.find("div", class_="text")
if potential_text:
review = strip_and_clean(potential_text.get_text())
except ValueError:
pass
return review
def scrape_data_from_amazon(url) -> dict:
data_dict = {}
headers = {"User-Agent": USER_AGENT}
r = requests.get(url, headers=headers)
if r.status_code == 200:
soup = BeautifulSoup(r.text, "html.parser")
# TODO Fix this scraper
data_dict["rating"] = get_rating_from_soup(soup)
data_dict["review"] = get_review_from_soup(soup)
return data_dict
def get_amazon_product_dict(amazon_id: str) -> dict:
data_dict = {}
url = ""
search_url = AMAZON_SEARCH_URL.format(amazon_id=amazon_id)
headers = {
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
"accept-language": "en-GB,en;q=0.9",
}
response = requests.get(search_url, headers=headers)
if response.status_code != 200:
logger.info(f"Bad http response from Amazon {response}")
return data_dict
soup = BeautifulSoup(response.text, "html.parser")
results = soup.find("a", class_="a-link-normal")
if not results:
logger.info(f"No search results for {amazon_id}")
return data_dict
product_url = "https://www.amazon.com" + str(results.get("href", ""))
data_dict = {}
response = requests.get(product_url, headers=headers)
if response.status_code != 200:
logger.info(f"Bad http response from Amazon {response}")
return data_dict
soup = BeautifulSoup(response.text, "html.parser")
try:
data_dict["title"] = soup.findAll("span", class_="celwidget")[1].text.strip()
data_dict["cover_url"] = soup.find("img", class_="frontImage").get("src")
data_dict["summary"] = soup.findAll("div", class_="a-expander-content")[1].text
meta = soup.findAll("div", class_="rpi-attribute-value")
data_dict["isbn"] = meta[AmazonAttribute.ISBN_10.value].text.strip()
pages = meta[AmazonAttribute.PAGES.value].text
if "pages" in pages:
data_dict["pages"] = (
meta[AmazonAttribute.PAGES.value].text.split("pages")[0].strip()
)
except IndexError as e:
logger.error(f"Amazon lookup is failing for this product {amazon_id}: {e}")
except AttributeError as e:
logger.error(f"Amazon lookup is failing for this product {amazon_id}: {e}")
return data_dict
def lookup_book_from_amazon(amazon_id: str) -> dict:
top = {}
return {
"title": top.get("title"),
"isbn": isbn,
"openlibrary_id": ol_id,
"goodreads_id": get_first("id_goodreads", top),
"first_publish_year": top.get("first_publish_year"),
"first_sentence": first_sentence,
"pages": top.get("number_of_pages_median", None),
"cover_url": COVER_URL.format(id=ol_id),
"ol_author_id": ol_author_id,
"subject_key_list": top.get("subject_key", []),
}

View File

@ -17,6 +17,7 @@ class MediaSourceTag(str, Enum):
LOCG = "source_locg"
KOREADER = "source_koreader"
SEMANTIC_SCHOLAR = "source_semantic_scholar"
AMAZON = "source_amazon"
@classmethod
def choices(cls):

View File

@ -1,9 +1,8 @@
import logging
import re
import sqlite3
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from enum import Enum
from zoneinfo import ZoneInfo
import requests
from books.constants import BOOKS_TITLES_TO_IGNORE
@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
return book_id_map
def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
def build_page_data(page_rows: list, book_map: dict) -> dict:
"""Given rows of page data from KoReader, parse each row and build
scrobbles for our user, loading the page data into the page_data
field on the scrobble instance.
@ -187,18 +186,20 @@ def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
book_ids_not_found.append(koreader_book_id)
continue
if "pages" not in book_map[koreader_book_id].keys():
book_map[koreader_book_id]["pages"] = {}
book_map[koreader_book_id].setdefault("pages", [])
page_number = page_row[KoReaderPageStatColumn.PAGE.value]
duration = page_row[KoReaderPageStatColumn.DURATION.value]
start_ts = page_row[KoReaderPageStatColumn.START_TIME.value]
book_map[koreader_book_id]["pages"][page_number] = {
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
book_map[koreader_book_id]["pages"].append(
{
"page_number": page_number,
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
)
if book_ids_not_found:
logger.info(f"Found pages for books not in file: {set(book_ids_not_found)}")
return book_map
@ -216,7 +217,6 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_not_found.append(book_id)
continue
should_create_scrobble = False
scrobble_page_data = {}
playback_position_seconds = 0
prev_page_stats = {}
@ -225,11 +225,12 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_processed = 0
total_pages_read = len(book_map[koreader_book_id]["pages"])
ordered_pages = sorted(
book_map[koreader_book_id]["pages"].items(),
key=lambda x: x[1]["start_ts"],
book_map[koreader_book_id]["pages"],
key=lambda x: x["start_ts"],
)
for cur_page_number, stats in ordered_pages:
for stats in ordered_pages:
page_number = stats["page_number"]
pages_processed += 1
seconds_from_last_page = 0
@ -243,46 +244,52 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
)
end_of_reading = pages_processed == total_pages_read
big_jump_to_this_page = (cur_page_number - last_page_number) > 10
big_jump_to_this_page = (page_number - last_page_number) > 10
is_session_gap = seconds_from_last_page > SESSION_GAP_SECONDS
if (is_session_gap and not big_jump_to_this_page) or end_of_reading:
should_create_scrobble = True
should_create_scrobble = (
is_session_gap and not big_jump_to_this_page
) or end_of_reading
# Always accumulate the current page first
scrobble_page_data[page_number] = stats
if should_create_scrobble:
# For end-of-reading, the current page is already accumulated
# and belongs in this scrobble. For a session gap, remove the
# current page from this scrobble — it starts a new session.
if is_session_gap and not end_of_reading:
del scrobble_page_data[page_number]
scrobble_page_data = dict(
sorted(
scrobble_page_data.items(),
key=lambda x: x[1]["start_ts"],
)
)
try:
first_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[0]
)
last_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[-1]
)
except IndexError:
if not scrobble_page_data:
logger.error(
"Could not process book, no page data found",
extra={"scrobble_page_data": scrobble_page_data},
)
continue
first_page = next(iter(scrobble_page_data.values()))
last_page = scrobble_page_data[
list(scrobble_page_data.keys())[-1]
]
timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(first_page.get("start_ts")))
datetime.fromtimestamp(
int(first_page.get("start_ts"))
)
)
stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(last_page.get("end_ts")))
datetime.fromtimestamp(
int(last_page.get("end_ts"))
)
)
# Adjust for Daylight Saving Time
# if timestamp.dst() == timedelta(
# 0
# ) or stop_timestamp.dst() == timedelta(0):
# timestamp = timestamp - timedelta(hours=1)
# stop_timestamp = stop_timestamp - timedelta(hours=1)
scrobble = Scrobble.objects.filter(
timestamp=timestamp,
book_id=book_id,
@ -291,7 +298,7 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
if not scrobble:
logger.info(
f"Queueing scrobble for {book_id}, page {cur_page_number}"
f"Queueing scrobble for {book_id}, page {page_number}"
)
log_data = {
"koreader_hash": book_dict.get("hash"),
@ -318,16 +325,18 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
timezone=tz,
)
)
# Then start over
should_create_scrobble = False
# Then start over for the next session
playback_position_seconds = 0
scrobble_page_data = {}
# We accumulate pages for the scrobble until we should create a new one
scrobble_page_data[cur_page_number] = stats
# For session gaps, re-add the current page as the
# beginning of the next session's accumulation
if is_session_gap and not end_of_reading:
scrobble_page_data[page_number] = stats
last_page_number = cur_page_number
last_page_number = page_number
prev_page_stats = stats
if pages_not_found:
logger.info(f"Pages not found for books: {set(pages_not_found)}")
return scrobbles_to_create
@ -335,13 +344,16 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
def fix_long_play_stats_for_scrobbles(scrobbles: list) -> None:
"""Given a list of scrobbles, update pages read, long play seconds and check
for media completion"""
for media completion.
Uses the long_play_last_scrobble FK chain to accumulate time.
Consider using the recompute_long_play_seconds management command instead.
"""
for scrobble in scrobbles:
# But if there's a next scrobble, set pages read to their starting page
if scrobble.previous and not scrobble.previous.long_play_complete:
if scrobble.long_play_last_scrobble and not scrobble.long_play_last_scrobble.long_play_complete:
scrobble.long_play_seconds = scrobble.playback_position_seconds + (
scrobble.previous.long_play_seconds or 0
scrobble.long_play_last_scrobble.long_play_seconds or 0
)
else:
scrobble.long_play_seconds = scrobble.playback_position_seconds
@ -357,9 +369,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
new_scrobbles = []
user = User.objects.filter(id=user_id).first()
tz = ZoneInfo("UTC")
if user:
tz = user.profile.tzinfo
is_os_file = "https://" not in file_path
if is_os_file:
@ -375,7 +384,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
book_map = build_page_data(
cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
book_map,
tz,
)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
else:
@ -392,7 +400,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
_sqlite_bytes(file_path), max_buffer_size=1_048_576
):
if table_name == "page_stat_data":
book_map = build_page_data(rows, book_map, tz)
book_map = build_page_data(rows, book_map)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
logger.info(f"Creating {len(new_scrobbles)} new scrobbles")

View File

@ -0,0 +1,301 @@
import logging
import time
from django.core.management.base import BaseCommand
from django.db import transaction
from books.constants import READCOMICSONLINE_URL
logger = logging.getLogger(__name__)
MISSING_ALL = [
"cover",
"summary",
"isbn",
"pages",
"language",
"publisher",
"publish_year",
]
MISSING_GROUPS = {
"cover": lambda b: not bool(b.cover),
"summary": lambda b: not b.summary,
"isbn": lambda b: not b.isbn_13 and not b.isbn_10,
"pages": lambda b: b.pages is None,
"language": lambda b: not b.language,
"publisher": lambda b: not b.publisher,
"publish_year": lambda b: b.first_publish_year is None,
}
def _book_matches(book, flags):
if not flags:
return False
for flag in flags:
fn = MISSING_GROUPS.get(flag)
if fn and fn(book):
return True
return False
class Command(BaseCommand):
help = "Backfill missing metadata on books from Google Books, OpenLibrary, and ComicVine"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of books to process per batch (default: 100)",
)
parser.add_argument(
"--sleep",
type=float,
default=0.5,
help="Seconds to sleep between API calls (default: 0.5)",
)
for flag in MISSING_ALL:
parser.add_argument(
f"--missing-{flag}",
dest="missing_flags",
action="append_const",
const=flag,
help=f"Process books missing {flag}",
)
parser.add_argument(
"--comics-only",
action="store_true",
help="Only process books with a readcomicsonline.ru URL",
)
parser.add_argument(
"--all",
action="store_true",
dest="all_missing",
help="Process books missing any metadata field",
)
def handle(self, *args, **options):
from books.models import Book
commit = options["commit"]
batch_size = options["batch_size"]
sleep_secs = options["sleep"]
flags = options.get("missing_flags") or []
comics_only = options["comics_only"]
all_missing = options["all_missing"]
if all_missing:
flags = MISSING_ALL
if not flags and not comics_only:
self.stdout.write(
"No filters specified. Use --all, --missing-*, or --comics-only."
)
return
qs = Book.objects.all()
if comics_only:
qs = qs.filter(readcomics_url__isnull=False)
if flags:
qs = [b for b in qs.iterator() if _book_matches(b, flags)]
else:
qs = list(qs)
total = len(qs)
self.stdout.write(f"Found {total} books to process")
if not commit:
self.stdout.write(
"Dry run — no API calls will be made. Use --commit to run lookups."
)
return
enriched = 0
skipped = 0
stats = {
"cover_fixed": 0,
"summary_fixed": 0,
"isbn_fixed": 0,
"pages_fixed": 0,
"language_fixed": 0,
"publisher_fixed": 0,
"publish_year_fixed": 0,
}
for batch_num, offset in enumerate(range(0, len(qs), batch_size)):
batch = qs[offset : offset + batch_size]
for book in batch:
result = self._enrich_book(book, sleep_secs)
if result:
enriched += 1
for key in stats:
if result.get(key):
stats[key] += 1
else:
skipped += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch)}/{total}"
f"enriched: {enriched}, skipped: {skipped}"
)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Books enriched: {enriched}\n"
f" Books skipped: {skipped}\n"
f" Covers fixed: {stats['cover_fixed']}\n"
f" Summaries fixed:{stats['summary_fixed']}\n"
f" ISBNs fixed: {stats['isbn_fixed']}\n"
f" Pages fixed: {stats['pages_fixed']}\n"
f" Languages fixed:{stats['language_fixed']}\n"
f" Publishers fixed:{stats['publisher_fixed']}\n"
f" Publish yrs fixed: {stats['publish_year_fixed']}"
)
def _enrich_book(self, book, sleep_secs):
from books.sources.comicvine import (
lookup_comic_from_comicvine,
lookup_issue_by_comicvine_id,
)
from books.sources.google import lookup_book_from_google
from books.sources.openlibrary import lookup_book_from_openlibrary as lookup_book_from_ol
title = book.original_title or book.title
author_name = book.author.name if book.author else None
book_dict = {}
cv_data = None
if book.comicvine_id:
cv_data = lookup_issue_by_comicvine_id(str(book.comicvine_id))
if not cv_data:
cv_data = lookup_comic_from_comicvine(title)
if cv_data:
book_dict.update(cv_data)
ol_data = lookup_book_from_ol(title, author=author_name)
time.sleep(sleep_secs)
google_data = lookup_book_from_google(title)
if ol_data:
for k, v in ol_data.items():
book_dict.setdefault(k, v)
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
if not book_dict:
return None
changed = self._apply(book, book_dict, title)
return changed
def _apply(self, book, data, title):
changed = {
"cover_fixed": False,
"summary_fixed": False,
"isbn_fixed": False,
"pages_fixed": False,
"language_fixed": False,
"publisher_fixed": False,
"publish_year_fixed": False,
}
update_fields = []
cover_url = data.pop("cover_url", "")
if data.get("summary") and not book.summary:
book.summary = data["summary"]
update_fields.append("summary")
changed["summary_fixed"] = True
if data.get("isbn_13") and not book.isbn_13:
book.isbn_13 = data["isbn_13"]
update_fields.append("isbn_13")
changed["isbn_fixed"] = True
if data.get("isbn_10") and not book.isbn_10:
book.isbn_10 = data["isbn_10"]
update_fields.append("isbn_10")
changed["isbn_fixed"] = True
if data.get("pages") and book.pages is None:
book.pages = data["pages"]
update_fields.append("pages")
changed["pages_fixed"] = True
if data.get("language") and not book.language:
book.language = data["language"]
update_fields.append("language")
changed["language_fixed"] = True
if data.get("publisher") and not book.publisher:
book.publisher = data["publisher"]
update_fields.append("publisher")
changed["publisher_fixed"] = True
if data.get("first_publish_year") and book.first_publish_year is None:
book.first_publish_year = data["first_publish_year"]
update_fields.append("first_publish_year")
changed["publish_year_fixed"] = True
if data.get("openlibrary_id") and not book.openlibrary_id:
book.openlibrary_id = data["openlibrary_id"]
update_fields.append("openlibrary_id")
if data.get("comicvine_id") and not book.comicvine_id:
book.comicvine_id = data["comicvine_id"]
update_fields.append("comicvine_id")
if data.get("issue_number") and book.issue_number is None:
book.issue_number = data["issue_number"]
update_fields.append("issue_number")
if data.get("volume_number") and book.volume_number is None:
book.volume_number = data["volume_number"]
update_fields.append("volume_number")
if data.get("volume") and not book.volume:
book.volume = data["volume"]
update_fields.append("volume")
if data.get("volume_comicvine_id") and not book.volume_comicvine_id:
book.volume_comicvine_id = data["volume_comicvine_id"]
update_fields.append("volume_comicvine_id")
if update_fields:
book.save(update_fields=update_fields)
self.stdout.write(f" [ENRICHED] {book}{', '.join(update_fields)}")
if cover_url and not book.cover:
book.save_image_from_url(cover_url)
if book.cover:
changed["cover_fixed"] = True
self.stdout.write(f" [COVER] {book} — cover saved from source")
genres = data.pop("genres", data.pop("generes", []))
if genres:
existing = set(book.genre.names())
new_genres = [g for g in genres if g not in existing]
if new_genres:
book.genre.add(*new_genres)
self.stdout.write(f" [GENRES] {book} — added {len(new_genres)} genres")
tags = data.pop("tags", [])
if tags:
existing_tags = set(book.tags.names())
new_tags = [t for t in tags if t not in existing_tags]
if new_tags:
book.tags.add(*new_tags)
self.stdout.write(f" [TAGS] {book} — added {', '.join(new_tags)}")
return changed if any(changed.values()) else None

View File

@ -0,0 +1,130 @@
import logging
from datetime import timedelta
from django.core.management.base import BaseCommand
from django.db import transaction
from books.koreader import SESSION_GAP_SECONDS, fix_long_play_stats_for_scrobbles
logger = logging.getLogger(__name__)
SESSION_GAP = timedelta(seconds=SESSION_GAP_SECONDS)
def _page_data_keys(pages):
return sorted(int(k) for k in (pages or {}))
class Command(BaseCommand):
help = "Merge orphaned 1-page KOReader scrobbles into the preceding scrobble"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
def handle(self, *args, **options):
from scrobbles.models import Scrobble
commit = options["commit"]
qs = Scrobble.objects.filter(
media_type="Book", source="KOReader"
).order_by("book_id", "timestamp")
if not qs.exists():
self.stdout.write("No KOReader book scrobbles found.")
return
merged = 0
affected_books = set()
# Group by book_id manually since we're iterating in order
book_scrobbles = {}
for s in qs:
book_scrobbles.setdefault(s.book_id, []).append(s)
if not commit:
self.stdout.write("Dry run — no changes will be saved. Use --commit to apply.")
for book_id, scrobbles in book_scrobbles.items():
batch_merged = 0
i = 0
while i < len(scrobbles) - 1:
current = scrobbles[i]
orphan = scrobbles[i + 1]
orphan_pages = orphan.logdata.page_data if orphan.logdata else {}
orphan_keys = _page_data_keys(orphan_pages)
if len(orphan_keys) != 1:
i += 1
continue
current_pages = current.logdata.page_data if current.logdata else {}
current_keys = _page_data_keys(current_pages)
if not current_keys:
i += 1
continue
orphan_page_num = orphan_keys[0]
current_last_page = current_keys[-1]
if orphan_page_num != current_last_page + 1:
i += 1
continue
# Check that the orphan is close enough in time
gap = orphan.timestamp - current.stop_timestamp
if gap > SESSION_GAP:
i += 1
continue
# Merge orphan into current
current_pages[str(orphan_page_num)] = orphan_pages[str(orphan_page_num)]
current.log["page_data"] = current_pages
current.log["pages_read"] = len(current_pages)
current.stop_timestamp = orphan.stop_timestamp
current.playback_position_seconds += orphan.playback_position_seconds
affected_books.add(book_id)
if commit:
with transaction.atomic():
current.save(
update_fields=[
"log",
"stop_timestamp",
"playback_position_seconds",
]
)
orphan.delete()
merged += 1
batch_merged += 1
scrobbles.pop(i + 1)
if batch_merged:
self.stdout.write(
f" Book {book_id}: merged {batch_merged} orphan scrobble(s)"
)
self.stdout.write(f"\nTotal orphans merged: {merged}")
if commit and affected_books:
self.stdout.write("Recalculating long_play_stats for affected books...")
for book_id in affected_books:
scrobbles_to_fix = (
Scrobble.objects.filter(book_id=book_id, source="KOReader")
.order_by("timestamp")
)
fix_long_play_stats_for_scrobbles(list(scrobbles_to_fix))
self.stdout.write(f"Fixed stats for {len(affected_books)} books.")
if not commit:
self.stdout.write(
f"\nWould merge {merged} orphan scrobble(s) across "
f"{len(affected_books)} book(s)."
)

View File

@ -0,0 +1,23 @@
# Generated by Django 4.2.29 on 2026-06-15 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("books", "0036_alter_book_genre_alter_paper_genre"),
]
operations = [
migrations.AddField(
model_name="book",
name="volume",
field=models.CharField(blank=True, max_length=255, null=True),
),
migrations.AddField(
model_name="book",
name="volume_comicvine_id",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -4,6 +4,7 @@ from dataclasses import dataclass
from datetime import datetime
from typing import Optional
from uuid import uuid4
from zoneinfo import ZoneInfo
import requests
from books.constants import MediaSourceTag, READCOMICSONLINE_URL
@ -19,8 +20,10 @@ from books.openlibrary import (
from books.sources.comicvine import (
ComicVineClient,
lookup_comic_from_comicvine,
lookup_issue_by_comicvine_id,
)
from books.sources.google import lookup_book_from_google
from books.sources.amazon import lookup_book_from_amazon
from books.sources.openlibrary import (
lookup_book_from_openlibrary as lookup_book_from_ol,
)
@ -71,7 +74,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
_excluded_fields = {"koreader_hash", "page_data"}
def avg_seconds_per_page(self):
if self.page_data:
if self.page_data and isinstance(self.page_data, dict):
total_duration = 0
for page_num, stats in self.page_data.items():
total_duration += stats.get("duration", 0)
@ -149,6 +152,8 @@ class Book(LongPlayScrobblableMixin):
first_sentence = models.TextField(**BNULL)
# ComicVine
comicvine_id = models.CharField(max_length=255, **BNULL)
volume = models.CharField(max_length=255, **BNULL)
volume_comicvine_id = models.CharField(max_length=255, **BNULL)
readcomics_url = models.CharField(max_length=255, **BNULL)
next_readcomics_url = models.CharField(max_length=255, **BNULL)
issue_number = models.IntegerField(**BNULL)
@ -173,6 +178,9 @@ class Book(LongPlayScrobblableMixin):
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
def __str__(self) -> str:
if not self.subtitle:
return self.title
return f"{self.title} - {self.subtitle}"
def save(self, *args, **kwargs):
@ -184,12 +192,18 @@ class Book(LongPlayScrobblableMixin):
@property
def subtitle(self):
subtitle = self.author
subtitle_parts = []
if self.author:
subtitle_parts.append(self.author.name)
if self.issue_number and "Issue" not in str(self.title):
subtitle += " - Issue {self.issue_number}"
subtitle_parts.append(f"Issue {self.issue_number}")
if self.volume_number and "Volume" not in str(self.title):
subtitle += " - Volume {self.volume_number}"
return subtitle
subtitle_parts.append(f"Volume {self.volume_number}")
if len(subtitle_parts) > 1:
return " / ".join(subtitle_parts)
if len(subtitle_parts) == 1:
return subtitle_parts[0]
return ""
@property
def strings(self) -> ScrobblableConstants:
@ -211,7 +225,7 @@ class Book(LongPlayScrobblableMixin):
@property
def resume_start_url(self):
return reverse("scrobbles:start", kwargs={"uuid": self.uuid}) + "?resume=1"
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid}) + "?resume=1"
@classmethod
def get_from_comicvine(
@ -219,36 +233,24 @@ class Book(LongPlayScrobblableMixin):
) -> "Book":
book, created = cls.objects.get_or_create(title=title)
if not created:
if not created and not overwrite:
return book
book_dict = lookup_comic_from_comicvine(title)
if not book_dict:
return book
if created or overwrite:
author_list = []
author_dicts = book_dict.pop("author_dicts")
if author_dicts:
for author_dict in author_dicts:
if author_dict.get("authorId"):
author, a_created = Author.objects.get_or_create(
semantic_id=author_dict.get("authorId")
)
author_list.append(author)
if a_created:
author.name = author_dict.get("name")
author.save()
# TODO enrich author?
...
tags = book_dict.pop("tags", [])
genres = book_dict.pop("genres", book_dict.pop("generes", []))
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
if author_list:
book.authors.add(*author_list)
genres = book_dict.pop("genres", [])
if genres:
book.genre.add(*genres)
if genres:
book.genre.add(*genres)
if tags:
book.tags.add(*tags)
return book
@classmethod
@ -259,6 +261,7 @@ class Book(LongPlayScrobblableMixin):
url: str = "",
enrich: bool = True,
commit: bool = True,
amazon_id: str | None = None,
):
"""Given a title, get a Book instance.
@ -282,24 +285,50 @@ class Book(LongPlayScrobblableMixin):
book_dict = None
source_tag = None
tried_comicvine = False
if READCOMICSONLINE_URL in url:
book_dict = lookup_comic_from_comicvine(title)
tried_comicvine = True
if book_dict:
source_tag = MediaSourceTag.COMICVINE
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
if not book_dict:
book_dict = lookup_book_from_ol(title, author=author)
if book_dict:
book_dict = {}
ol_data = lookup_book_from_ol(title, author=author)
google_data = lookup_book_from_google(title)
if ol_data:
book_dict.update(ol_data)
source_tag = MediaSourceTag.OPENLIBRARY
if not book_dict:
book_dict = lookup_book_from_google(title)
if book_dict:
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
source_tag = MediaSourceTag.GOOGLE_BOOKS
if ol_data and ol_data.get("cover_url"):
book_dict["cover_url"] = ol_data["cover_url"]
# Always try ComicVine as a fallback — it may recognize books that
# OL/Google don't flag as comics
if not tried_comicvine:
cv_data = lookup_comic_from_comicvine(title)
if cv_data:
for k, v in cv_data.items():
if v:
book_dict.setdefault(k, v)
source_tag = MediaSourceTag.COMICVINE
# Try Amazon PAAPI as a fallback when given an ASIN
if amazon_id and not book_dict:
amazon_data = lookup_book_from_amazon(amazon_id)
if amazon_data:
book_dict.update(amazon_data)
source_tag = MediaSourceTag.AMAZON
if not book_dict:
logger.warning(
@ -311,6 +340,7 @@ class Book(LongPlayScrobblableMixin):
authors = book_dict.pop("authors", [])
cover_url = book_dict.pop("cover_url", "")
genres = book_dict.pop("genres", book_dict.pop("generes", []))
tags = book_dict.pop("tags", [])
if authors:
for author_str in authors:
@ -330,6 +360,8 @@ class Book(LongPlayScrobblableMixin):
book.save_image_from_url(cover_url)
if genres:
book.genre.add(*genres)
if tags:
book.tags.add(*tags)
book.authors.add(*author_list)
if source_tag:
book.tags.add(source_tag.value)
@ -367,9 +399,14 @@ class Book(LongPlayScrobblableMixin):
data = lookup_comic_from_locg(str(self.title))
if not data and COMICVINE_API_KEY:
logger.warn(f"Checking ComicVine for {self.title}")
cv_client = ComicVineClient(api_key=COMICVINE_API_KEY)
data = lookup_comic_from_comicvine(str(self.title))
if self.comicvine_id:
logger.warn(
f"Checking ComicVine by ID for {self.title}"
)
data = lookup_issue_by_comicvine_id(str(self.comicvine_id))
if not data:
logger.warn(f"Checking ComicVine for {self.title}")
data = lookup_comic_from_comicvine(str(self.title))
if not data:
logger.warn(f"Book not found in any sources: {self.title}")
@ -407,10 +444,10 @@ class Book(LongPlayScrobblableMixin):
)
data.pop("pages")
# Pop this, so we can look it up later
# Pop these so they don't get passed to update()
cover_url = data.pop("cover_url", "")
subject_key_list = data.pop("subject_key_list", [])
tags = data.pop("tags", [])
# Fun trick for updating all fields at once
Book.objects.filter(pk=self.id).update(**data)
@ -418,6 +455,8 @@ class Book(LongPlayScrobblableMixin):
if subject_key_list:
self.genre.add(*subject_key_list)
if tags:
self.tags.add(*tags)
if cover_url:
r = requests.get(cover_url)
@ -463,8 +502,11 @@ class Book(LongPlayScrobblableMixin):
if scrobble.logdata.page_data:
for page, data in scrobble.logdata.page_data.items():
if convert_timestamps:
data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
tz = None
if scrobble.timezone:
tz = ZoneInfo(scrobble.timezone)
data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
pages[page] = data
sorted_pages = OrderedDict(
sorted(pages.items(), key=lambda x: x[1]["start_ts"])

View File

@ -0,0 +1,123 @@
import logging
import warnings
from django.conf import settings
with warnings.catch_warnings():
warnings.simplefilter("ignore", DeprecationWarning)
from amazon_paapi import AmazonApi
logger = logging.getLogger(__name__)
_amazon_client = None
def _get_client() -> AmazonApi | None:
global _amazon_client
if _amazon_client is not None:
return _amazon_client
key = settings.AMAZON_PAAPI_ACCESS_KEY
secret = settings.AMAZON_PAAPI_SECRET_KEY
tag = settings.AMAZON_PAAPI_ASSOCIATE_TAG
country = settings.AMAZON_PAAPI_COUNTRY
if not all([key, secret, tag]):
logger.warning("Amazon PAAPI credentials not configured")
return None
_amazon_client = AmazonApi(key, secret, tag, country)
return _amazon_client
def lookup_book_from_amazon(asin: str) -> dict:
book_dict: dict = {}
client = _get_client()
if not client:
return book_dict
try:
items = client.get_items(
items=[asin],
Condition="New",
LanguagesOfPreference=["en_US"],
)
except Exception as e:
logger.warning(f"Amazon PAAPI lookup failed for {asin}: {e}")
return book_dict
if not items:
logger.info(f"No Amazon item found for {asin}")
return book_dict
item = items[0]
raw = item.to_dict()
item_info = raw.get("item_info", {}) or {}
book_dict["title"] = _get_nested(item_info, "title", "display_value")
if not book_dict.get("title"):
book_dict["title"] = _get_nested(item_info, "title", "value")
contributors = _get_nested(item_info, "by_line_info", "contributors") or []
authors = [
c["name"]
for c in contributors
if c.get("role", "").lower() in ("author", "artist", "writer")
]
if authors:
book_dict["authors"] = authors
publisher = _get_nested(item_info, "by_line_info", "manufacturer")
if publisher:
book_dict["publisher"] = publisher
isb_ns = _get_nested(item_info, "external_ids", "isb_ns")
if isb_ns and isinstance(isb_ns, list):
for isb in isb_ns:
if isinstance(isb, dict):
if isb.get("type") == "ISBN_13":
book_dict["isbn_13"] = isb.get("value")
elif isb.get("type") == "ISBN_10":
book_dict["isbn_10"] = isb.get("value")
pages_count = _get_nested(item_info, "content_info", "pages_count")
if pages_count and isinstance(pages_count, dict):
book_dict["pages"] = pages_count.get("value") or pages_count.get("display_value")
languages = _get_nested(item_info, "content_info", "languages") or []
if languages and isinstance(languages, list):
lang = languages[0]
if isinstance(lang, dict):
book_dict["language"] = lang.get("display_value") or lang.get("value")
pub_date = _get_nested(item_info, "content_info", "publication_date")
if not pub_date:
pub_date = _get_nested(item_info, "product_info", "release_date")
if pub_date and isinstance(pub_date, dict):
book_dict["publish_date"] = pub_date.get("display_value") or pub_date.get("value")
features = item_info.get("features") or []
if features and isinstance(features, list):
book_dict["summary"] = " ".join(features[:5])
images = raw.get("images", {}) or {}
primary = images.get("primary", {}) or {}
for size in ("large", "hi_res", "medium"):
candidate = primary.get(size, {}) or {}
url = candidate.get("url")
if url:
book_dict["cover_url"] = url
break
book_dict["detail_page_url"] = raw.get("detail_page_url")
return book_dict
def _get_nested(d: dict, *keys):
for key in keys:
if not isinstance(d, dict):
return None
d = d.get(key)
return d

View File

@ -17,8 +17,10 @@ class ComicVineClient(object):
account on https://comicvine.gamespot.com/ in order to obtain an API key.
"""
# All API requests made by this client will be made to this URL.
API_URL = "https://www.comicvine.com/api/search/"
# All API requests made by this client will be made to these URLs.
API_URL = "https://comicvine.gamespot.com/api/search/"
ISSUE_API_URL = "https://comicvine.gamespot.com/api/issue/4000-{issue_id}/"
VOLUME_API_URL = "https://comicvine.gamespot.com/api/volume/4050-{volume_id}/"
# A valid User-Agent header must be set in order for our API requests to
# be accepted, otherwise our request will be rejected with a
@ -41,15 +43,12 @@ class ComicVineClient(object):
"volume",
}
def __init__(self, api_key, expire_after=300):
def __init__(self, api_key):
"""
Store the API key in a class variable, and install the requests cache,
configuring it using the ``expire_after`` parameter.
Store the API key in a class variable.
:param api_key: Your personal ComicVine API key.
:type api_key: str
:param expire_after: The number of seconds to retain an entry in cache.
:type expire_after: int or None
"""
self.api_key = api_key
@ -109,14 +108,17 @@ class ComicVineClient(object):
:rtype: dict
"""
return {
params = {
"api_key": self.api_key,
"format": "json",
"limit": min(10, limit), # hard limit of 10
"offset": max(0, offset), # cannot provide negative offset
"query": query,
"resources": self._validate_resources(resources),
}
validated = self._validate_resources(resources)
if validated:
params["resources"] = validated
return params
def _validate_resources(self, resources):
"""
@ -141,33 +143,35 @@ class ComicVineClient(object):
def _query_api(self, params):
"""
Query the ComicVine API's ``search`` resource, providing the required
headers and parameters with the request. Optionally allow the caller
of the function to disable the request cache.
headers and parameters with the request.
If an error occurs during the request, handle it accordingly. Upon
success, return the JSON from the response.
:param params: Parameters to include with the request.
:type params: dict
:param use_cache: Toggle the use of requests_cache.
:type use_cache: bool
:return: The JSON contained in the response.
:rtype: dict
"""
# Since we're performing the identical action regardless of whether
# or not the request cache is to be used, store the procedure in a
# local function to avoid repetition.
def __httpget():
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
if not response.ok:
self._handle_http_error(response)
return response.json()
json_data = response.json()
return __httpget()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data
def _handle_http_error(self, response):
"""
@ -195,15 +199,81 @@ class ComicVineClient(object):
raise exception(message)
def get_issue(self, issue_id: str) -> dict:
"""
Fetch a single issue by its ComicVine ID directly from the issue detail
endpoint, which returns richer data than the search endpoint.
:param issue_id: The ComicVine numeric ID for the issue (e.g. "538480")
:type issue_id: str
:return: The full JSON response for the issue, or empty dict on failure.
:rtype: dict
"""
params = {
"api_key": self.api_key,
"format": "json",
}
url = self.ISSUE_API_URL.format(issue_id=issue_id)
response = requests.get(url, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
json_data = response.json()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data.get("results", {})
def get_volume(self, volume_id: str) -> dict:
"""
Fetch a single volume by its ComicVine ID from the volume detail
endpoint. Used to get publisher info and other volume-level metadata.
:param volume_id: The ComicVine numeric ID for the volume (e.g. "91273")
:type volume_id: str
:return: The full JSON response for the volume, or empty dict on failure.
:rtype: dict
"""
params = {
"api_key": self.api_key,
"format": "json",
}
url = self.VOLUME_API_URL.format(volume_id=volume_id)
response = requests.get(url, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
json_data = response.json()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data.get("results", {})
def lookup_comic_from_comicvine(title: str) -> dict:
original_title = title
issue_number = None
volume_nubmer = None
resource_type = "issue"
if "Issue " in title:
resource_type = "issue"
issue_number = title.split("Issue ")[1]
volume_number = None
if "Volume " in title:
@ -215,48 +285,136 @@ def lookup_comic_from_comicvine(title: str) -> dict:
logger.warning("No ComicVine API key configured, not looking anything up")
return {}
client = ComicVineClient(api_key=getattr(settings, "COMICVINE_API_KEY", None))
client = ComicVineClient(api_key=api_key)
raw_results = client.search(title).get("results")
results = [r for r in raw_results if r.get("resource_type") == resource_type]
raw_results = client.search(title)
if not raw_results:
return {}
results = raw_results.get("results", [])
results = [r for r in results if r.get("resource_type") == resource_type]
if not results:
logger.warning("No comic found on ComicVine")
return {}
found_result = None
for result in results:
if result.get("issue_number") == str(issue_number):
if issue_number is not None and result.get("issue_number") == str(issue_number):
found_result = result
break
if result.get("volume_number") == str(volume_number):
if volume_number is not None and result.get("volume_number") == str(volume_number):
found_result = result
break
if not found_result:
found_result = results[0]
logger.info("ComicVine results", extra={"results": results})
data_dict = _build_data_dict_from_issue(found_result, original_title)
_enrich_with_volume_data(client, data_dict)
return data_dict
if not found_result:
logger.warning("No matches found on ComicVine")
def lookup_issue_by_comicvine_id(comicvine_id: str) -> dict:
"""
Look up an issue directly by its ComicVine ID using the issue detail
endpoint. Returns richer data than the search-based lookup.
:param comicvine_id: The ComicVine numeric ID for the issue (e.g. "538480")
:type comicvine_id: str
:return: A dict of extracted book metadata, or empty dict on failure.
:rtype: dict
"""
if not comicvine_id:
return {}
title = found_result.get("name")
api_key = getattr(settings, "COMICVINE_API_KEY", "")
if not api_key:
logger.warning("No ComicVine API key configured, not looking anything up")
return {}
if found_result.get("volume"):
title = found_result.get("volume").get("name")
client = ComicVineClient(api_key=api_key)
issue_data = client.get_issue(comicvine_id)
if not issue_data:
logger.warning("No issue found on ComicVine for ID %s", comicvine_id)
return {}
data_dict = _build_data_dict_from_issue(issue_data, issue_data.get("name", ""))
_enrich_with_volume_data(client, data_dict)
return data_dict
def _build_data_dict_from_issue(issue_data: dict, original_title: str = "") -> dict:
"""
Build a book metadata dict from a ComicVine issue resource (either from
search results or issue detail endpoint). Both return the same shape of
issue data.
:param issue_data: The issue resource dict from ComicVine.
:param original_title: The original search term, if any.
:return: A dict of extracted book metadata.
:rtype: dict
"""
title = issue_data.get("name")
if issue_data.get("volume"):
title = issue_data.get("volume").get("name")
cover_url = None
if issue_data.get("image"):
cover_url = issue_data["image"].get("original_url")
volume_name = None
volume_cv_id = None
publisher_name = None
volume_data = issue_data.get("volume")
if volume_data:
volume_name = volume_data.get("name")
volume_cv_id = volume_data.get("id")
publisher_data = volume_data.get("publisher")
if publisher_data:
publisher_name = publisher_data.get("name")
data_dict = {
"title": title,
"original_title": original_title,
"issue_number": found_result.get("issue_number"),
"volume_number": found_result.get("volume_number"),
"cover_url": found_result.get("image").get("original_url"),
"comicvine_id": found_result.get("id"),
"comicvine_data": found_result,
"summary": found_result.get("description"),
"publish_date": found_result.get("cover_date"),
"first_publish_year": found_result.get("cover_date", "")[:4],
"issue_number": issue_data.get("issue_number"),
"volume_number": issue_data.get("volume_number"),
"volume": volume_name,
"volume_comicvine_id": volume_cv_id,
"publisher": publisher_name,
"cover_url": cover_url,
"comicvine_id": issue_data.get("id"),
"summary": issue_data.get("description"),
"publish_date": issue_data.get("cover_date"),
"first_publish_year": (issue_data.get("cover_date") or "")[:4],
"tags": ["comicbook"],
}
return data_dict
def _enrich_with_volume_data(client: ComicVineClient, data_dict: dict) -> None:
"""
Follow-up a successful issue lookup by fetching the volume detail and
filling in publisher and other volume-level metadata that the issue
endpoint doesn't provide.
:param client: An initialised ComicVineClient instance.
:param data_dict: The data dict from an issue lookup (mutated in place).
"""
volume_cv_id = data_dict.get("volume_comicvine_id")
if not volume_cv_id:
return
volume_data = client.get_volume(str(volume_cv_id))
if not volume_data:
return
publisher_data = volume_data.get("publisher")
if publisher_data:
publisher_name = publisher_data.get("name")
if publisher_name and not data_dict.get("publisher"):
data_dict["publisher"] = publisher_name
if not data_dict.get("volume"):
data_dict["volume"] = volume_data.get("name")

View File

@ -26,8 +26,6 @@ def lookup_book_from_google(title: str) -> dict:
if not google_result:
return {}
publish_date = pendulum.parse(google_result.get("publishedDate"))
isbn_13 = ""
isbn_10 = ""
for ident in google_result.get("industryIdentifiers", []):
@ -35,25 +33,25 @@ def lookup_book_from_google(title: str) -> dict:
isbn_13 = ident.get("identifier")
if ident.get("type") == "ISBN_10":
isbn_10 = ident.get("identifier")
# TODO this may lead to issues with the first get if Google changes our title
# book_metadata.title = google_result.get("title")
# if google_result.get("subtitle"):
# book_metadata["title"] = ": ".join(
# [google_result.get("title"), google_result.get("subtitle")]
# )
# book_dict["subtitle"] = google_result.get("subtitle")
book_dict["authors"] = google_result.get("authors")
book_dict["publisher"] = google_result.get("publisher")
book_dict["first_publish_year"] = publish_date.year
book_dict["pages"] = google_result.get("pageCount")
book_dict["isbn_13"] = isbn_13
book_dict["isbn_10"] = isbn_10
book_dict["publish_date"] = google_result.get("publishedDate")
if len(book_dict["publish_date"]) == 4:
book_dict["publish_date"] = f"{book_dict['publish_date']}-1-1"
book_dict["language"] = google_result.get("language")
book_dict["summary"] = google_result.get("description")
book_dict["genres"] = google_result.get("categories")
raw_date = google_result.get("publishedDate")
if raw_date:
try:
publish_date = pendulum.parse(raw_date)
book_dict["first_publish_year"] = publish_date.year
except Exception:
pass
book_dict["publish_date"] = raw_date
if len(raw_date) == 4:
book_dict["publish_date"] = f"{raw_date}-1-1"
book_dict["cover_url"] = (
google_result.get("imageLinks", {})
.get("thumbnail", "")

View File

@ -32,8 +32,6 @@ class KoReaderBookRows:
DEFAULT_STR = "N/A"
DEFAULT_INT = 0
DEFAULT_TIME = 1703800469
BOOK_ROWS = []
PAGE_STATS_ROWS = []
def _gen_random_row(self, i):
wiggle = random.randrange(15)
@ -110,6 +108,8 @@ class KoReaderBookRows:
end_session = True
def __init__(self, book_count=0, **kwargs):
self.BOOK_ROWS = []
self.PAGE_STATS_ROWS = []
self._generate_random_book_rows(book_count)
self._generate_custom_book_row(**kwargs)
self._generate_random_page_stats_rows()

View File

@ -44,7 +44,10 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
# The test data generator adds the session-gap 3600s AFTER the trigger page
# (not before), so the first session includes 21 pages (1-21), and each
# subsequent session has 20 until the last. The last page is now included
# in the final scrobble instead of being orphaned.
expected_scrobbles = 6 * len(book_map.keys())
assert len(scrobbles) == expected_scrobbles
assert len(scrobbles[0].logdata.page_data.keys()) == 21
@ -52,7 +55,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
assert len(scrobbles[2].logdata.page_data.keys()) == 20
assert len(scrobbles[3].logdata.page_data.keys()) == 20
assert len(scrobbles[4].logdata.page_data.keys()) == 20
assert len(scrobbles[5].logdata.page_data.keys()) == 18
assert len(scrobbles[5].logdata.page_data.keys()) == 19
def test_get_author_str_from_row():

View File

@ -0,0 +1,146 @@
# Generated by Django 4.2.29 on 2026-06-12 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("charts", "0001_initial"),
]
operations = [
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "album", "rank"],
name="charts_char_user_id_1adcde_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "track", "rank"],
name="charts_char_user_id_d18aab_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "video", "rank"],
name="charts_char_user_id_de9f0a_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "board_game", "rank"],
name="charts_char_user_id_d5d58f_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "book", "rank"],
name="charts_char_user_id_e877cf_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "food", "rank"],
name="charts_char_user_id_a0ad71_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "podcast", "rank"],
name="charts_char_user_id_846b80_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "trail", "rank"],
name="charts_char_user_id_54feba_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "album", "rank"],
name="charts_char_user_id_a3dc49_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "track", "rank"],
name="charts_char_user_id_4b01ab_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "video", "rank"],
name="charts_char_user_id_2ac9d2_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "board_game", "rank"],
name="charts_char_user_id_ba968a_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "book", "rank"],
name="charts_char_user_id_e66751_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "food", "rank"],
name="charts_char_user_id_d23f06_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "podcast", "rank"],
name="charts_char_user_id_be8122_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "trail", "rank"],
name="charts_char_user_id_b94ea9_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "artist", "rank"],
name="charts_char_user_id_406e0e_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "album", "rank"],
name="charts_char_user_id_322b0d_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "tv_series", "rank"],
name="charts_char_user_id_aa44b7_idx",
),
),
]

View File

@ -0,0 +1,109 @@
# Generated by Django 4.2.29 on 2026-06-19 16:25
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("charts", "0002_chartrecord_charts_char_user_id_1adcde_idx_and_more"),
]
operations = [
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("artist__isnull", False)),
fields=("user", "year", "month", "week", "day", "artist"),
name="unique_chart_artist_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("album__isnull", False)),
fields=("user", "year", "month", "week", "day", "album"),
name="unique_chart_album_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("track__isnull", False)),
fields=("user", "year", "month", "week", "day", "track"),
name="unique_chart_track_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("tv_series__isnull", False)),
fields=("user", "year", "month", "week", "day", "tv_series"),
name="unique_chart_tv_series_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("video__isnull", False)),
fields=("user", "year", "month", "week", "day", "video"),
name="unique_chart_video_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("podcast__isnull", False)),
fields=("user", "year", "month", "week", "day", "podcast"),
name="unique_chart_podcast_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("podcast_episode__isnull", False)),
fields=("user", "year", "month", "week", "day", "podcast_episode"),
name="unique_chart_podcast_episode_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("board_game__isnull", False)),
fields=("user", "year", "month", "week", "day", "board_game"),
name="unique_chart_board_game_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("trail__isnull", False)),
fields=("user", "year", "month", "week", "day", "trail"),
name="unique_chart_trail_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("geo_location__isnull", False)),
fields=("user", "year", "month", "week", "day", "geo_location"),
name="unique_chart_geo_location_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("food__isnull", False)),
fields=("user", "year", "month", "week", "day", "food"),
name="unique_chart_food_period",
),
),
migrations.AddConstraint(
model_name="chartrecord",
constraint=models.UniqueConstraint(
condition=models.Q(("book__isnull", False)),
fields=("user", "year", "month", "week", "day", "book"),
name="unique_chart_book_period",
),
),
]

View File

@ -2,6 +2,7 @@ import calendar
from django.contrib.auth import get_user_model
from django.db import models
from django.db.models import Q
from django.urls import reverse
from django_extensions.db.models import TimeStampedModel
@ -60,10 +61,91 @@ class ChartRecord(TimeStampedModel):
models.Index(fields=["user", "year", "geo_location", "rank"]),
models.Index(fields=["user", "year", "food", "rank"]),
models.Index(fields=["user", "year", "book", "rank"]),
models.Index(fields=["user", "year", "week", "artist", "rank"]),
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
models.Index(fields=["user", "year", "month", "artist", "rank"]),
models.Index(fields=["user", "year", "month", "album", "rank"]),
models.Index(fields=["user", "year", "month", "track", "rank"]),
models.Index(fields=["user", "year", "month", "tv_series", "rank"]),
models.Index(fields=["user", "year", "month", "video", "rank"]),
models.Index(fields=["user", "year", "month", "board_game", "rank"]),
models.Index(fields=["user", "year", "month", "book", "rank"]),
models.Index(fields=["user", "year", "month", "food", "rank"]),
models.Index(fields=["user", "year", "month", "podcast", "rank"]),
models.Index(fields=["user", "year", "month", "trail", "rank"]),
models.Index(fields=["user", "year", "week", "artist", "rank"]),
models.Index(fields=["user", "year", "week", "album", "rank"]),
models.Index(fields=["user", "year", "week", "track", "rank"]),
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
models.Index(fields=["user", "year", "week", "video", "rank"]),
models.Index(fields=["user", "year", "week", "board_game", "rank"]),
models.Index(fields=["user", "year", "week", "book", "rank"]),
models.Index(fields=["user", "year", "week", "food", "rank"]),
models.Index(fields=["user", "year", "week", "podcast", "rank"]),
models.Index(fields=["user", "year", "week", "trail", "rank"]),
models.Index(fields=["user", "year", "month", "day", "artist", "rank"]),
models.Index(fields=["user", "year", "month", "day", "album", "rank"]),
models.Index(fields=["user", "year", "month", "day", "tv_series", "rank"]),
]
constraints = [
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "artist"],
condition=Q(artist__isnull=False),
name="unique_chart_artist_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "album"],
condition=Q(album__isnull=False),
name="unique_chart_album_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "track"],
condition=Q(track__isnull=False),
name="unique_chart_track_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "tv_series"],
condition=Q(tv_series__isnull=False),
name="unique_chart_tv_series_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "video"],
condition=Q(video__isnull=False),
name="unique_chart_video_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "podcast"],
condition=Q(podcast__isnull=False),
name="unique_chart_podcast_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "podcast_episode"],
condition=Q(podcast_episode__isnull=False),
name="unique_chart_podcast_episode_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "board_game"],
condition=Q(board_game__isnull=False),
name="unique_chart_board_game_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "trail"],
condition=Q(trail__isnull=False),
name="unique_chart_trail_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "geo_location"],
condition=Q(geo_location__isnull=False),
name="unique_chart_geo_location_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "food"],
condition=Q(food__isnull=False),
name="unique_chart_food_period",
),
models.UniqueConstraint(
fields=["user", "year", "month", "week", "day", "book"],
condition=Q(book__isnull=False),
name="unique_chart_book_period",
),
]
@property

View File

@ -7,7 +7,10 @@ register = template.Library()
def get_item(dictionary, key):
if isinstance(dictionary, dict):
return dictionary.get(key)
return None
try:
return dictionary[int(key)]
except (IndexError, KeyError, TypeError, ValueError):
return None
@register.filter

View File

@ -6,6 +6,7 @@ from typing import Optional
import pytz
from django.apps import apps
from django.conf import settings
from django.db import transaction
from django.db.models import Count, Q
from django.utils import timezone
@ -186,60 +187,64 @@ def build_charts(
ranks = {count: rank for rank, count in enumerate(unique_counts, start=1)}
media_field = f"{media_type}_id"
records_to_create = []
records_to_update = []
existing = ChartRecord.objects.filter(
period_filter, user=user, **{media_field + "__isnull": False}
)
existing_by_media_id = {getattr(r, media_field): r for r in existing}
found_media_ids = set()
with transaction.atomic():
records_to_create = []
records_to_update = []
for result in results:
media_id = result[config["values"]]
if media_id is None:
continue
found_media_ids.add(media_id)
chart_record_data = {
"user_id": user.id,
"year": year,
"month": month,
"week": week,
"day": day,
"rank": ranks[result["scrobble_count"]],
"count": result["scrobble_count"],
}
chart_record_data[media_field] = media_id
if media_id in existing_by_media_id:
existing_record = existing_by_media_id[media_id]
existing_record.rank = chart_record_data["rank"]
existing_record.count = chart_record_data["count"]
records_to_update.append(existing_record)
else:
records_to_create.append(ChartRecord(**chart_record_data))
ids_to_delete = [
r.id for r in existing if getattr(r, media_field) not in found_media_ids
]
if ids_to_delete:
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
if records_to_update:
ChartRecord.objects.bulk_update(
records_to_update, ["rank", "count"], batch_size=500
existing = ChartRecord.objects.select_for_update().filter(
period_filter, user=user, **{media_field + "__isnull": False}
)
existing_by_media_id = {getattr(r, media_field): r for r in existing}
found_media_ids = set()
if records_to_create:
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
for result in results:
media_id = result[config["values"]]
if media_id is None:
continue
logger.info(
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
f"chart records for {media_type}, period "
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
)
found_media_ids.add(media_id)
chart_record_data = {
"user_id": user.id,
"year": year,
"month": month,
"week": week,
"day": day,
"rank": ranks[result["scrobble_count"]],
"count": result["scrobble_count"],
}
chart_record_data[media_field] = media_id
if media_id in existing_by_media_id:
existing_record = existing_by_media_id[media_id]
existing_record.rank = chart_record_data["rank"]
existing_record.count = chart_record_data["count"]
records_to_update.append(existing_record)
else:
records_to_create.append(ChartRecord(**chart_record_data))
ids_to_delete = [
r.id
for r in existing
if getattr(r, media_field) not in found_media_ids
]
if ids_to_delete:
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
if records_to_update:
ChartRecord.objects.bulk_update(
records_to_update, ["rank", "count"], batch_size=500
)
if records_to_create:
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
logger.info(
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
f"chart records for {media_type}, period "
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
)
def build_yesterdays_charts(user, media_types: Optional[list] = None) -> None:

View File

@ -438,12 +438,14 @@ class ChartRecordView(TemplateView):
return context
def get_available_years(self, user):
return list(
ChartRecord.objects.filter(user=user)
.values_list("year", flat=True)
.distinct()
.order_by("-year")
)
if not hasattr(self, "_available_years"):
self._available_years = list(
ChartRecord.objects.filter(user=user)
.values_list("year", flat=True)
.distinct()
.order_by("-year")
)
return self._available_years
def get_period_type(self):
date_param = self.request.GET.get("date")

View File

View File

@ -0,0 +1,14 @@
from discgolf.models import DiscGolfCourse
from django.contrib import admin
from scrobbles.admin import ScrobbleInline
@admin.register(DiscGolfCourse)
class DiscGolfCourseAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("title", "layout_name", "number_of_holes", "par_total")
raw_id_fields = ("trail",)
search_fields = ("title", "layout_name")
inlines = [
ScrobbleInline,
]

View File

View File

@ -0,0 +1,11 @@
from discgolf import models
from rest_framework import serializers
class DiscGolfCourseSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = models.DiscGolfCourse
fields = "__all__"

View File

@ -0,0 +1,13 @@
from rest_framework import permissions, viewsets
from discgolf.api import serializers
from discgolf import models
class DiscGolfCourseViewSet(viewsets.ModelViewSet):
queryset = models.DiscGolfCourse.objects.all().order_by("-created")
serializer_class = serializers.DiscGolfCourseSerializer
permission_classes = [permissions.IsAuthenticated]

View File

@ -0,0 +1,6 @@
from django.apps import AppConfig
class DiscgolfConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "discgolf"

View File

@ -0,0 +1,83 @@
# Generated by Django 4.2.29 on 2026-06-20 04:19
from django.db import migrations, models
import django_extensions.db.fields
import taggit.managers
import uuid
class Migration(migrations.Migration):
initial = True
dependencies = [
("taggit", "0004_alter_taggeditem_content_type_alter_taggeditem_tag"),
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
]
operations = [
migrations.CreateModel(
name="DiscGolfCourse",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
(
"uuid",
models.UUIDField(
blank=True, default=uuid.uuid4, editable=False, null=True
),
),
("title", models.CharField(blank=True, max_length=255, null=True)),
("base_run_time_seconds", models.IntegerField(blank=True, null=True)),
("description", models.TextField(blank=True, null=True)),
(
"layout_name",
models.CharField(blank=True, max_length=255, null=True),
),
("number_of_holes", models.IntegerField(blank=True, null=True)),
("par_total", models.IntegerField(blank=True, null=True)),
(
"genre",
taggit.managers.TaggableManager(
blank=True,
help_text="A comma-separated list of tags.",
through="scrobbles.ObjectWithGenres",
to="scrobbles.Genre",
verbose_name="Genre",
),
),
(
"tags",
taggit.managers.TaggableManager(
blank=True,
help_text="A comma-separated list of tags.",
through="taggit.TaggedItem",
to="taggit.Tag",
verbose_name="Tags",
),
),
],
options={
"abstract": False,
},
),
]

View File

@ -0,0 +1,25 @@
# Generated by Django 4.2.29 on 2026-06-20 04:29
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("trails", "0009_trail_route_waypoint"),
("discgolf", "0001_initial"),
]
operations = [
migrations.AddField(
model_name="discgolfcourse",
name="trail",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="trails.trail",
),
),
]

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.29 on 2026-06-20 04:35
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("discgolf", "0002_discgolfcourse_trail"),
]
operations = [
migrations.AddField(
model_name="discgolfcourse",
name="par_per_hole",
field=models.JSONField(blank=True, null=True),
),
]

View File

@ -0,0 +1,76 @@
from dataclasses import dataclass
from typing import Optional, TypedDict
from django.apps import apps
from django.db import models
from django.urls import reverse
from scrobbles.dataclasses import BaseLogData, WithPeopleLogData
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
BNULL = {"blank": True, "null": True}
class DiscGolfSingleScores(TypedDict, total=False):
person_id: int
total: int
class DiscGolfTeamScores(TypedDict, total=False):
person_ids: list[int]
total: int
@dataclass
class DiscGolfLogData(BaseLogData, WithPeopleLogData):
scores: Optional[dict[str, DiscGolfSingleScores | DiscGolfTeamScores]] = None
weather: Optional[str] = None
fun_factor: Optional[str] = None
course_name: Optional[str] = None
par: Optional[int] = None
round_type: Optional[str] = None
class DiscGolfCourse(ScrobblableMixin):
description = models.TextField(**BNULL)
layout_name = models.CharField(max_length=255, **BNULL)
number_of_holes = models.IntegerField(**BNULL)
par_total = models.IntegerField(**BNULL)
par_per_hole = models.JSONField(**BNULL)
trail = models.ForeignKey(
"trails.Trail", on_delete=models.DO_NOTHING, **BNULL
)
def get_absolute_url(self) -> str:
return reverse("discgolf:course_detail", kwargs={"slug": self.uuid})
def __str__(self):
return f"{self.title} ({self.layout_name or 'Default'})"
@property
def logdata_cls(self):
return DiscGolfLogData
@property
def subtitle(self):
return self.layout_name or ""
@property
def strings(self) -> ScrobblableConstants:
return ScrobblableConstants(verb="Playing", tags="golf")
@property
def primary_image_url(self) -> str:
return ""
@classmethod
def find_or_create(cls, name: str, **defaults) -> "DiscGolfCourse":
course = cls.objects.filter(title=name).first()
if not course:
course = cls.objects.create(title=name, **defaults)
return course
def scrobbles(self, user_id):
Scrobble = apps.get_model("scrobbles", "Scrobble")
return Scrobble.objects.filter(user_id=user_id, disc_golf=self).order_by(
"-timestamp"
)

View File

@ -0,0 +1,14 @@
from django.urls import path
from discgolf import views
app_name = "discgolf"
urlpatterns = [
path("disc-golf/", views.DiscGolfCourseListView.as_view(), name="course_list"),
path(
"disc-golf/<slug:slug>/",
views.DiscGolfCourseDetailView.as_view(),
name="course_detail",
),
]

View File

@ -0,0 +1,126 @@
import csv
import logging
from datetime import datetime
from dateutil.parser import parse as parse_datetime
from django.utils import timezone
from people.models import Person
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
logger = logging.getLogger(__name__)
def _parse_udisc_datetime(raw: str) -> datetime:
return parse_datetime(raw)
def _resolve_player(name: str) -> Person:
person, _ = Person.objects.get_or_create(name=name.strip())
return person
def import_udisc_csv(
file_path: str, user_id: int, record_error=None
) -> list[Scrobble]:
from discgolf.models import DiscGolfCourse
new_scrobbles = []
with open(file_path, newline="", encoding="utf-8-sig") as f:
reader = csv.DictReader(f)
rows = list(reader)
if not rows:
return []
par_row = None
player_rows = []
for row in rows:
name = row.get("PlayerName", "").strip()
if name.lower() == "par":
par_row = row
else:
player_rows.append(row)
if not par_row:
return []
course_name = par_row.get("CourseName", "").strip()
layout_name = par_row.get("LayoutName", "").strip()
start_date_raw = par_row.get("StartDate", "").strip()
start_dt = _parse_udisc_datetime(start_date_raw) if start_date_raw else timezone.now()
number_of_holes = sum(1 for k in par_row if k.startswith("Hole") and k[4:].isdigit())
par_total_str = par_row.get("Total", "").strip()
par_total = int(par_total_str) if par_total_str.isdigit() else None
par_per_hole = {}
for k, v in par_row.items():
if k.startswith("Hole") and k[4:].isdigit() and v:
hole_num = int(k[4:])
try:
par_per_hole[f"hole_{hole_num}"] = int(v)
except (ValueError, TypeError):
pass
course, _ = DiscGolfCourse.objects.get_or_create(
title=course_name,
defaults={
"layout_name": layout_name,
"number_of_holes": number_of_holes,
"par_total": par_total,
"par_per_hole": par_per_hole or None,
},
)
is_teams = "+" in player_rows[0].get("PlayerName", "") if player_rows else False
round_type = "Teams" if is_teams else "Singles"
scores = {}
for row in player_rows:
player_name = row.get("PlayerName", "").strip()
hole_scores = {}
for k, v in row.items():
if k.startswith("Hole") and k[4:].isdigit() and v:
hole_num = int(k[4:])
try:
hole_scores[f"hole_{hole_num}"] = int(v)
except (ValueError, TypeError):
pass
total_str = row.get("Total", "").strip()
total = int(total_str) if total_str.isdigit() else None
if total is not None:
hole_scores["total"] = total
if is_teams:
people = player_name.split("+")
person_ids = [_resolve_player(p.strip()).id for p in people]
hole_scores["person_ids"] = person_ids
else:
person = _resolve_player(player_name)
hole_scores["person_id"] = person.id
scores[player_name] = hole_scores
log = {
"scores": scores,
"course_name": course_name,
"par": par_total,
"round_type": round_type,
}
scrobble_dict = {
"user_id": user_id,
"timestamp": start_dt,
"source": "uDisc",
"playback_position_seconds": 0,
"log": log,
}
scrobble = Scrobble.create_or_update(course, user_id, scrobble_dict)
if scrobble:
new_scrobbles.append(scrobble)
ScrobbleNtfyNotification(scrobble).send()
return new_scrobbles

View File

@ -0,0 +1,15 @@
from discgolf.models import DiscGolfCourse
from scrobbles.views import (
ScrobbleableListView,
ScrobbleableDetailView,
ChartContextMixin,
)
class DiscGolfCourseListView(ScrobbleableListView):
model = DiscGolfCourse
class DiscGolfCourseDetailView(ScrobbleableDetailView, ChartContextMixin):
model = DiscGolfCourse

View File

@ -21,6 +21,7 @@ class FoodAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("category",)
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -28,6 +28,14 @@ class GeoLocationLogData(BaseLogData, WithPeopleLogData):
activity: str = ""
detected_at: str = ""
@classmethod
def from_log_dict(cls, log_dict: dict) -> dict:
instance_data = log_dict.get("movement_detection", {}).copy()
for field_name in ["description", "notes", "with_people_ids"]:
if field_name in log_dict:
instance_data[field_name] = log_dict[field_name]
return instance_data
class GeoLocation(ScrobblableMixin):
COMPLETION_PERCENT = getattr(settings, "LOCATION_COMPLETION_PERCENT", 100)

View File

@ -1,8 +1,22 @@
from django.core.cache import cache
from music.models import Artist, Album
CACHE_TTL = 300
def music_lists(request):
artist_list = cache.get("music_lists_artist_list")
if artist_list is None:
artist_list = list(Artist.objects.all().only("id", "name"))
cache.set("music_lists_artist_list", artist_list, CACHE_TTL)
album_list = cache.get("music_lists_album_list")
if album_list is None:
album_list = list(Album.objects.all().only("id", "name"))
cache.set("music_lists_album_list", album_list, CACHE_TTL)
return {
"artist_list": Artist.objects.all(),
"album_list": Album.objects.all(),
"artist_list": artist_list,
"album_list": album_list,
}

View File

@ -236,19 +236,6 @@ class Artist(TimeStampedModel):
)
artist.fix_metadata()
# TODO: See if this alt_names stuff actually works or causes hard to debug problems
# If we did find our artist, but the found name is slightly differnt, record that
# if artist and alt_name:
# if not artist.alt_names:
# artist.alt_names = alt_name
# else:
# artist.alt_names += f"\\{alt_name}"
# logger.info(
# f"Add alt_name {alt_name} to artist {artist}",
# extra={"alt_name": alt_name, "artist_id": artist.id},
# )
# artist.save(update_fields=["alt_names"])
return artist
@ -600,8 +587,9 @@ class Track(ScrobblableMixin):
def __str__(self):
return f"{self.title} by {self.artist}"
@property
def logdata_cls(self):
return TrackLogData()
return TrackLogData
@property
def primary_album(self):

View File

@ -6,6 +6,7 @@ from people.models import Person, PersonScrobble
class PersonAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "bgg_username", "bgstats_id")
raw_id_fields = ("user", "created_by")
ordering = ("-created",)
search_fields = ("name",)

View File

@ -18,6 +18,7 @@ class PodcastAdmin(admin.ModelAdmin):
"producer",
"active",
)
raw_id_fields = ("producer",)
ordering = ("name",)
@ -28,6 +29,7 @@ class PodcastEpisodeAdmin(admin.ModelAdmin):
"title",
"podcast",
)
raw_id_fields = ("podcast",)
list_filter = ("podcast",)
ordering = ("-created",)
inlines = [

View File

@ -6,6 +6,7 @@ from profiles.models import UserProfile
@admin.register(UserProfile)
class UserProfileAdmin(admin.ModelAdmin):
date_hierarchy = "created"
raw_id_fields = ("user",)
ordering = ("-created",)
readonly_fields = ("timezone_change_log",)
exclude = (

View File

@ -140,6 +140,11 @@ class UserProfile(TimeStampedModel):
return history
def get_timestamp_with_tz(self, timestamp):
from django.conf import settings
server_tz = ZoneInfo(settings.TIME_ZONE)
ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
timezone = self.tzinfo
if self.timezone_change_log:
change_list = self.historic_timezone_changes
@ -150,13 +155,13 @@ class UserProfile(TimeStampedModel):
end = None
if end:
if start <= timestamp.replace(tzinfo=end.timezone) <= end:
if start <= ref_dt <= end:
timezone = start.timezone
else:
if start <= timestamp.replace(tzinfo=start.timezone):
if start <= ref_dt:
timezone = start.timezone
return timestamp.replace(tzinfo=timezone)
return ref_dt.astimezone(timezone)
def adjust_timezone_of_scrobbles(self, commit=False):
current_dt = None

View File

@ -21,6 +21,7 @@ class PuzzleAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("manufacturer",)
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -12,6 +12,7 @@ from scrobbles.models import (
Scrobble,
ShareViewLog,
TrailGPXImport,
UDiscCSVImport,
)
from scrobbles.mixins import Genre
@ -19,35 +20,11 @@ from scrobbles.mixins import Genre
class ScrobbleInline(admin.TabularInline):
model = Scrobble
extra = 0
raw_id_fields = (
"video",
"channel",
"podcast_episode",
"track",
"video_game",
"book",
"paper",
"sport_event",
"food",
"board_game",
"geo_location",
"task",
"puzzle",
"mood",
"brick_set",
"trail",
"beer",
"web_page",
"life_event",
"birding_location",
"user",
)
exclude = (
"scrobble_log",
"timezone",
"videogame_save_data",
"screenshot",
)
per_page = 15
ordering = ("-timestamp",)
show_change_link = True
fields = ("timestamp", "media_type", "source", "in_progress")
readonly_fields = fields
class ImportBaseAdmin(admin.ModelAdmin):
@ -97,6 +74,10 @@ class ScaleCSVImportAdmin(ImportBaseAdmin): ...
class TrailGPXImportAdmin(ImportBaseAdmin): ...
@admin.register(UDiscCSVImport)
class UDiscCSVImportAdmin(ImportBaseAdmin): ...
@admin.register(Genre)
class GenreAdmin(admin.ModelAdmin):
list_display = (
@ -121,7 +102,9 @@ class ScrobbleAdmin(admin.ModelAdmin):
"user",
)
raw_id_fields = (
"user",
"video",
"channel",
"podcast_episode",
"track",
"sport_event",
@ -140,6 +123,8 @@ class ScrobbleAdmin(admin.ModelAdmin):
"web_page",
"life_event",
"birding_location",
"disc_golf",
"long_play_last_scrobble",
)
list_filter = (
"is_paused",
@ -152,6 +137,7 @@ class ScrobbleAdmin(admin.ModelAdmin):
"user",
)
ordering = ("-timestamp",)
readonly_fields = ("share_token_version", "share_view_count")
def media_name(self, obj):
return obj.media_obj
@ -178,14 +164,19 @@ class FavoriteMediaAdmin(admin.ModelAdmin):
list_filter = ("media_type", "sent_to_mopidy", "user")
date_hierarchy = "created"
raw_id_fields = (
"user",
"video",
"channel",
"track",
"podcast_episode",
"sport_event",
"book",
"paper",
"video_game",
"board_game",
"geo_location",
"puzzle",
"food",
"task",
"mood",
"brick_set",
@ -194,4 +185,5 @@ class FavoriteMediaAdmin(admin.ModelAdmin):
"web_page",
"life_event",
"birding_location",
"disc_golf",
)

View File

@ -36,6 +36,7 @@ PLAY_AGAIN_MEDIA = {
"locations": "GeoLocation",
"videos": "Video",
"birds": "BirdingLocation",
"discgolf": "DiscGolf",
}
MEDIA_END_PADDING_SECONDS = {
@ -73,6 +74,7 @@ MANUAL_SCROBBLE_FNS = {
"-c": "manual_scrobble_book",
"-f": "manual_scrobble_food",
"-h": "manual_scrobble_twitch_channel",
"-dg": "manual_scrobble_discgolf",
}

View File

@ -1,4 +1,5 @@
import pytz
from django.core.cache import cache
from django.utils import timezone
from scrobbles.constants import EXCLUDE_FROM_NOW_PLAYING
@ -19,6 +20,8 @@ MONTH_COLORS = [
"#db7a7a", # Dec
]
CACHE_TTL = 60
def month_color(request):
from datetime import date
@ -27,15 +30,25 @@ def month_color(request):
def now_playing(request):
user = request.user
now = timezone.now()
if not user.is_authenticated:
return {}
return {
"now_playing_list": Scrobble.objects.filter(
in_progress=True,
is_paused=False,
user=user,
).exclude(
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
cache_key = f"now_playing_list_{user.id}"
now_playing_list = cache.get(cache_key)
if now_playing_list is None:
now_playing_list = list(
Scrobble.objects.filter(
in_progress=True,
is_paused=False,
user=user,
)
.exclude(
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
)
.select_related("track", "video", "podcast_episode")
)
cache.set(cache_key, now_playing_list, CACHE_TTL)
return {
"now_playing_list": now_playing_list,
}

View File

@ -50,6 +50,18 @@ class BaseLogData(JSONDataclass):
def override_fields(cls) -> dict:
return {}
@classmethod
def from_log_dict(cls, log_dict: dict) -> dict:
"""Extract LogData keyword arguments from a stored log dict.
Override in subclasses to handle custom nesting/structure.
"""
return {
k: v
for k, v in log_dict.items()
if k in cls.__dataclass_fields__
}
def notes_as_str(self, separator: str = " | ") -> str:
import html
import re
@ -207,7 +219,7 @@ class BaseLogData(JSONDataclass):
@dataclass
class LongPlayLogData(JSONDataclass):
long_play_complete: bool = False
pass
@dataclass

View File

@ -93,7 +93,6 @@ class LastFM:
new_scrobbles.append(new_scrobble)
created = Scrobble.objects.bulk_create(new_scrobbles)
# TODO Add a notification for users that their import is complete
logger.info(
f"Last.fm import fnished",
extra={

View File

@ -19,6 +19,7 @@ DEFAULT_RETROARCH_PATH = "var/retroarch/"
DEFAULT_BGSTATS_PATH = "var/bgstats/"
DEFAULT_EBIRD_PATH = "var/ebird/"
DEFAULT_SCALE_PATH = "var/scale/"
DEFAULT_UDISC_PATH = "var/udisc/"
def import_from_webdav_for_all_users(
@ -48,6 +49,7 @@ def import_from_webdav_for_all_users(
bgstats_count = 0
ebird_count = 0
scale_count = 0
udisc_count = 0
for user_id in webdav_enabled_user_ids:
client = get_webdav_client(user_id)
@ -78,9 +80,13 @@ def import_from_webdav_for_all_users(
)
logger.info("Scanning WebDAV scale for user %s", user_id)
scale_count += scan_webdav_for_scale(client, user_id)
logger.info("Scanning WebDAV udisc for user %s", user_id)
udisc_count += scan_webdav_for_udisc(
client, user_id, include_processed=include_processed
)
logger.info(
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale WebDAV imports",
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale, %d uDisc WebDAV imports",
ko_count,
gpx_count,
retro_count,
@ -94,9 +100,10 @@ def import_from_webdav_for_all_users(
"bgstats": bgstats_count,
"ebird": ebird_count,
"scale": scale_count,
"udisc": udisc_count,
},
)
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count, udisc_count
def scan_webdav_for_koreader(
@ -153,7 +160,7 @@ def scan_webdav_for_koreader(
)
if update_etag_only:
if last_import and remote_etag:
if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
last_import.webdav_etag = remote_etag
last_import.save(update_fields=["webdav_etag"])
logger.info(
@ -163,7 +170,7 @@ def scan_webdav_for_koreader(
)
return 0
if last_import and last_import.webdav_etag and remote_etag:
if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
if last_import.webdav_etag == remote_etag:
logger.info(
"koreader stats file unchanged (ETag match)",
@ -811,3 +818,115 @@ def scan_webdav_for_scale(webdav_client, user_id):
os.unlink(tmp.name)
return new_imports
def scan_webdav_for_udisc(webdav_client, user_id, include_processed=False):
"""Download .csv files from WebDAV var/udisc/ and queue imports for new files.
After importing, files are moved to var/udisc/processed/ so they are
not re-imported on subsequent scans unless *include_processed* is True.
"""
from scrobbles.models import UDiscCSVImport
from scrobbles.tasks import process_udisc_csv_import
udisc_path = DEFAULT_UDISC_PATH
try:
webdav_client.info(udisc_path)
except:
logger.info("No var/udisc/ directory on webdav", extra={"user_id": user_id})
return 0
try:
files = webdav_client.list(udisc_path)
except Exception as e:
logger.warning(
"Could not list var/udisc/",
extra={"user_id": user_id, "error": str(e)},
)
return 0
processed_dir = f"{udisc_path}processed/"
try:
webdav_client.mkdir(processed_dir, recursive=True)
except Exception:
pass
new_imports = 0
already_imported = set(
UDiscCSVImport.objects.filter(user_id=user_id).values_list(
"original_filename", flat=True
)
)
for fname in files:
fname = os.path.basename(fname)
if not fname.lower().endswith(".csv"):
continue
if fname == "processed":
continue
if fname in already_imported:
logger.debug(f"Skipping already-imported {fname}")
continue
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{udisc_path}{fname}", local_path=tmp.name
)
imp = UDiscCSVImport.objects.create(
user_id=user_id,
original_filename=fname,
)
with open(tmp.name, "rb") as f:
imp.csv_file.save(fname, f, save=True)
stem, ext = os.path.splitext(fname)
ts = datetime.now().strftime("%Y%m%dT%H%M%S")
webdav_client.move(
f"{udisc_path}{fname}",
f"{processed_dir}{stem}_{ts}{ext}",
)
process_udisc_csv_import.delay(imp.id)
new_imports += 1
except Exception as e:
logger.error(f"Failed to import uDisc CSV file {fname}: {e}")
finally:
os.unlink(tmp.name)
if include_processed:
try:
processed_files = webdav_client.list(processed_dir)
except Exception as e:
logger.warning(
"Could not list var/udisc/processed/",
extra={"user_id": user_id, "error": str(e)},
)
return new_imports
for fname in processed_files:
fname = os.path.basename(fname)
if not fname.lower().endswith(".csv"):
continue
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
try:
webdav_client.download_sync(
remote_path=f"{processed_dir}{fname}", local_path=tmp.name
)
imp = UDiscCSVImport.objects.create(
user_id=user_id,
original_filename=fname,
)
with open(tmp.name, "rb") as f:
imp.csv_file.save(fname, f, save=True)
process_udisc_csv_import.delay(imp.id)
new_imports += 1
except Exception as e:
logger.error(
f"Failed to import processed uDisc CSV file {fname}: {e}"
)
finally:
os.unlink(tmp.name)
return new_imports

View File

@ -0,0 +1,174 @@
from django.core.management.base import BaseCommand
from django.db import connection
from scrobbles.constants import LONG_PLAY_MEDIA
from scrobbles.models import Scrobble
BATCH_SIZE = 1000
class Command(BaseCommand):
help = (
"Backfill long_play_last_scrobble FK chains, then recompute "
"long_play_seconds by walking forward through scrobbles in "
"timestamp order with a running accumulator."
)
def add_arguments(self, parser):
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be changed without making changes",
)
parser.add_argument(
"--media-type",
type=str,
help="Only process a specific media type (e.g., Book, VideoGame)",
)
def handle(self, *args, **options):
dry_run = options["dry_run"]
media_type = options.get("media_type")
media_types = list(LONG_PLAY_MEDIA.values())
if media_type:
if media_type not in media_types:
self.stdout.write(
self.style.ERROR(
f"Invalid media type '{media_type}'. "
f"Valid: {', '.join(media_types)}"
)
)
return
media_types = [media_type]
# Step 1: backfill long_play_last_scrobble
self.stdout.write("Step 1: Backfilling long_play_last_scrobble chains...")
total_backfilled = 0
for mt in media_types:
n = self._backfill_chain(mt, dry_run)
total_backfilled += n
self.stdout.write(f" {mt}: {n} scrobbles linked")
if dry_run:
self.stdout.write(
self.style.WARNING(
f"Would backfill {total_backfilled} scrobbles total. "
"Run without --dry-run to apply."
)
)
else:
self.stdout.write(
self.style.SUCCESS(f"Backfilled {total_backfilled} scrobbles")
)
# Step 2: recompute long_play_seconds
self.stdout.write(
"\nStep 2: Recomputing long_play_seconds in timestamp order..."
)
total_updated = 0
for mt in media_types:
n = self._recompute_for_media_type(mt, dry_run)
total_updated += n
self.stdout.write(f" {mt}: {n} scrobbles updated")
if dry_run:
self.stdout.write(
self.style.WARNING(
f"Dry run: would update {total_updated} scrobbles total. "
"Use without --dry-run to apply."
)
)
else:
self.stdout.write(
self.style.SUCCESS(f"Updated {total_updated} scrobbles")
)
def _recompute_for_media_type(self, media_type: str, dry_run: bool) -> int:
"""Process scrobbles for a single media type in timestamp order with a
running accumulator, avoiding O(n2) FK chain walks."""
fk = _media_type_to_fk(media_type)
fk_id = f"{fk}_id"
scrobbles = Scrobble.objects.filter(
media_type=media_type,
**{f"{fk}__isnull": False},
playback_position_seconds__isnull=False,
).order_by(fk_id, "user_id", "timestamp")
total = scrobbles.count()
if not total:
return 0
updated = 0
batch = []
last_key = None
running_total = 0
for scrobble in scrobbles.iterator():
key = (getattr(scrobble, fk_id), scrobble.user_id)
if key != last_key:
running_total = 0
last_key = key
running_total += scrobble.playback_position_seconds or 0
if scrobble.long_play_seconds != running_total:
updated += 1
if not dry_run:
scrobble.long_play_seconds = running_total
batch.append(scrobble)
if len(batch) >= BATCH_SIZE:
Scrobble.objects.bulk_update(batch, ["long_play_seconds"])
batch = []
if scrobble.long_play_complete:
running_total = 0
if batch:
Scrobble.objects.bulk_update(batch, ["long_play_seconds"])
return updated
def _backfill_chain(self, media_type: str, dry_run: bool) -> int:
"""Set long_play_last_scrobble on each scrobble to the previous
scrobble for the same media+user using a single UPDATE with a
correlated subquery."""
fk = _media_type_to_fk(media_type)
table = Scrobble._meta.db_table
if dry_run:
with connection.cursor() as cursor:
cursor.execute(
f"SELECT COUNT(*) FROM {table} "
f"WHERE long_play_last_scrobble_id IS NULL "
f"AND {fk}_id IS NOT NULL"
)
return cursor.fetchone()[0]
with connection.cursor() as cursor:
cursor.execute(
f"UPDATE {table} "
f"SET long_play_last_scrobble_id = ("
f" SELECT id FROM {table} AS prev "
f" WHERE prev.{fk}_id = {table}.{fk}_id "
f" AND prev.user_id = {table}.user_id "
f" AND prev.timestamp < {table}.timestamp "
f" ORDER BY prev.timestamp DESC LIMIT 1"
f") "
f"WHERE long_play_last_scrobble_id IS NULL "
f"AND {fk}_id IS NOT NULL"
)
return cursor.rowcount
def _media_type_to_fk(media_type):
mapping = {
"VideoGame": "video_game",
"Book": "book",
"BrickSet": "brick_set",
"Task": "task",
}
return mapping.get(media_type)

View File

@ -0,0 +1,35 @@
from django.db import migrations
def convert_page_data_to_dict(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
log__page_data=None
):
page_data = scrobble.log.get("page_data")
if isinstance(page_data, list):
new_page_data = {}
for entry in page_data:
page_num = entry.get("page_number")
if page_num is not None:
try:
page_num = int(page_num)
except (ValueError, TypeError):
continue
new_page_data[page_num] = entry
scrobble.log["page_data"] = new_page_data
scrobble.save(update_fields=["log"])
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0095_backfill_null_visibility"),
]
operations = [
migrations.RunPython(
convert_page_data_to_dict,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,19 @@
# Generated by Django 4.2.29 on 2026-06-12 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0096_convert_book_page_data_to_dict"),
]
operations = [
migrations.AddIndex(
model_name="scrobble",
index=models.Index(
fields=["user", "-timestamp"], name="scrobbles_s_user_id_d367a7_idx"
),
),
]

View File

@ -0,0 +1,25 @@
# Generated by Django 4.2.29 on 2026-06-15 17:48
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0097_scrobble_scrobbles_s_user_id_d367a7_idx"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="long_play_last_scrobble",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="next_long_play_scrobbles",
to="scrobbles.scrobble",
),
),
]

View File

@ -0,0 +1,156 @@
# Generated by Django 4.2.29 on 2026-06-20 04:19
from django.conf import settings
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
import scrobbles.models
import uuid
class Migration(migrations.Migration):
dependencies = [
("discgolf", "0001_initial"),
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
]
operations = [
migrations.AddField(
model_name="favoritemedia",
name="disc_golf",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.CASCADE,
to="discgolf.discgolfcourse",
),
),
migrations.AddField(
model_name="scrobble",
name="disc_golf",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="discgolf.discgolfcourse",
),
),
migrations.AlterField(
model_name="favoritemedia",
name="media_type",
field=models.CharField(
choices=[
("Video", "Video"),
("Track", "Track"),
("PodcastEpisode", "Podcast episode"),
("SportEvent", "Sport event"),
("Book", "Book"),
("Paper", "Paper"),
("VideoGame", "Video game"),
("BoardGame", "Board game"),
("GeoLocation", "GeoLocation"),
("Trail", "Trail"),
("Beer", "Beer"),
("Puzzle", "Puzzle"),
("Food", "Food"),
("Task", "Task"),
("WebPage", "Web Page"),
("LifeEvent", "Life event"),
("Mood", "Mood"),
("BrickSet", "Brick set"),
("Channel", "Channel"),
("DiscGolf", "Disc golf"),
],
max_length=20,
),
),
migrations.AlterField(
model_name="scrobble",
name="media_type",
field=models.CharField(
choices=[
("Video", "Video"),
("Track", "Track"),
("PodcastEpisode", "Podcast episode"),
("SportEvent", "Sport event"),
("Book", "Book"),
("Paper", "Paper"),
("VideoGame", "Video game"),
("BoardGame", "Board game"),
("GeoLocation", "GeoLocation"),
("Trail", "Trail"),
("Beer", "Beer"),
("Puzzle", "Puzzle"),
("Food", "Food"),
("Task", "Task"),
("WebPage", "Web Page"),
("LifeEvent", "Life event"),
("Mood", "Mood"),
("BrickSet", "Brick set"),
("Channel", "Channel"),
("DiscGolf", "Disc golf"),
],
default="Video",
max_length=20,
),
),
migrations.CreateModel(
name="UDiscCSVImport",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("uuid", models.UUIDField(default=uuid.uuid4, editable=False)),
("processing_started", models.DateTimeField(blank=True, null=True)),
("processed_finished", models.DateTimeField(blank=True, null=True)),
("process_log", models.TextField(blank=True, null=True)),
("process_count", models.IntegerField(blank=True, null=True)),
("error_log", models.TextField(blank=True, null=True)),
(
"csv_file",
models.FileField(
blank=True,
null=True,
upload_to=scrobbles.models.UDiscCSVImport.get_path,
),
),
(
"original_filename",
models.CharField(blank=True, max_length=255, null=True),
),
(
"user",
models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
related_name="scrobbles_udisccsvimport_set",
to=settings.AUTH_USER_MODEL,
),
),
],
options={
"verbose_name": "uDisc CSV Import",
},
),
]

View File

@ -114,7 +114,7 @@ class ScrobblableMixin(TimeStampedModel):
@property
def start_url(self):
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid})
@property
def strings(self) -> ScrobblableConstants:
@ -162,26 +162,21 @@ class LongPlayScrobblableMixin(ScrobblableMixin):
return False
def get_longplay_finish_url(self):
return reverse("scrobbles:longplay-finish", kwargs={"uuid": self.uuid})
return reverse("scrobbles:longplay-finish", kwargs={"media_uuid": self.uuid})
def first_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
return (
get_scrobbles_for_media(self, user)
.filter(
log__long_play_complete=False,
log__serial_scrobble_id__isnull=True,
)
.order_by("timestamp")
.first()
)
last = self.last_long_play_scrobble_for_user(user)
if not last:
return None
current = last
while current.long_play_last_scrobble and not current.long_play_last_scrobble.long_play_complete:
current = current.long_play_last_scrobble
return current
def last_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
return (
get_scrobbles_for_media(self, user)
.filter(
log__long_play_complete=False,
log__serial_scrobble_id__isnull=False,
)
.order_by("timestamp")
.last()
.filter(long_play_complete=False)
.order_by("-timestamp")
.first()
)

View File

@ -11,6 +11,7 @@ import pendulum
import pytz
from beers.models import Beer
from birds.models import BirdingLocation
from discgolf.models import DiscGolfCourse
from boardgames.models import BoardGame
from books.koreader import process_koreader_sqlite_file
from books.models import Book, BookLogData, BookPageLogData, Paper
@ -51,7 +52,10 @@ from scrobbles.constants import (
MEDIA_END_PADDING_SECONDS,
)
from scrobbles.importers.lastfm import LastFM
from scrobbles.notifications import ScrobbleNtfyNotification
from scrobbles.notifications import (
LastFmImportNtfyNotification,
ScrobbleNtfyNotification,
)
from scrobbles.utils import get_file_md5_hash, media_class_to_foreign_key
from sports.models import SportEvent
from taggit.managers import TaggableManager
@ -225,7 +229,9 @@ class KoReaderImport(BaseFileImportMixin):
self.mark_started()
try:
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
scrobbles = process_koreader_sqlite_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
@ -272,7 +278,9 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
self.mark_started()
try:
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
scrobbles = import_audioscrobbler_tsv_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
@ -424,6 +432,8 @@ class LastFmImport(BaseFileImportMixin):
try:
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
self.record_log(scrobbles)
if scrobbles:
LastFmImportNtfyNotification(self, len(scrobbles)).send()
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
@ -608,6 +618,85 @@ class EBirdCSVImport(BaseFileImportMixin):
self.mark_finished()
class UDiscCSVImport(BaseFileImportMixin):
class Meta:
verbose_name = "uDisc CSV Import"
user = models.ForeignKey(
User,
on_delete=models.DO_NOTHING,
**BNULL,
related_name="scrobbles_udisccsvimport_set",
)
@property
def import_type(self) -> str:
return "uDisc"
def get_absolute_url(self):
return reverse("scrobbles:udisc-csv-import-detail", kwargs={"slug": self.uuid})
def get_path(instance, filename):
extension = filename.split(".")[-1]
uuid = instance.uuid
return f"udisc-csv-uploads/{uuid}.{extension}"
@property
def upload_file_path(self):
if getattr(settings, "USE_S3_STORAGE"):
path = self.csv_file.url
else:
path = self.csv_file.path
return path
csv_file = models.FileField(upload_to=get_path, **BNULL)
original_filename = models.CharField(max_length=255, **BNULL)
def process(self, force=False):
from discgolf.utils import import_udisc_csv
if self.processed_finished and not force:
logger.info(f"{self} already processed on {self.processed_finished}")
return
self.mark_started()
try:
scrobbles = import_udisc_csv(
self.upload_file_path, self.user_id, record_error=self.record_error
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
TYPE_FK_PREFETCHES: dict[str, tuple[str, ...]] = {
"Video": ("video",),
"Track": ("track", "track__artist_fk"),
"PodcastEpisode": ("podcast_episode", "podcast_episode__podcast"),
"SportEvent": ("sport_event",),
"Book": ("book",),
"Paper": ("paper",),
"VideoGame": ("video_game",),
"BoardGame": ("board_game",),
"GeoLocation": ("geo_location",),
"Trail": ("trail",),
"Beer": ("beer",),
"Puzzle": ("puzzle",),
"Food": ("food",),
"Task": ("task",),
"WebPage": ("web_page",),
"LifeEvent": ("life_event",),
"Mood": ("mood",),
"BrickSet": ("brick_set",),
"Channel": ("channel",),
"BirdingLocation": ("birding_location",),
"DiscGolf": ("disc_golf",),
}
class ScrobbleQuerySet(models.QuerySet):
def with_related(self):
return self.select_related("user").prefetch_related(
@ -632,8 +721,16 @@ class ScrobbleQuerySet(models.QuerySet):
"mood",
"brick_set",
"birding_location",
"disc_golf",
)
def with_related_for_types(self, media_types: list[str]):
prefetches = []
for t in media_types:
if t in TYPE_FK_PREFETCHES:
prefetches.extend(TYPE_FK_PREFETCHES[t])
return self.select_related("user").prefetch_related(*prefetches)
class ShareViewLog(TimeStampedModel):
scrobble = models.ForeignKey(
@ -674,7 +771,7 @@ class Scrobble(TimeStampedModel):
MOOD = "Mood", "Mood"
BRICKSET = "BrickSet", "Brick set"
CHANNEL = "Channel", "Channel"
BIRDING_LOCATION = "BirdingLocation", "Birding location"
DISC_GOLF = "DiscGolf", "Disc golf"
@classmethod
def list(cls):
@ -705,6 +802,9 @@ class Scrobble(TimeStampedModel):
birding_location = models.ForeignKey(
BirdingLocation, on_delete=models.DO_NOTHING, **BNULL
)
disc_golf = models.ForeignKey(
DiscGolfCourse, on_delete=models.DO_NOTHING, **BNULL
)
media_type = models.CharField(
max_length=20, choices=MediaType.choices, default=MediaType.VIDEO
)
@ -758,8 +858,20 @@ class Scrobble(TimeStampedModel):
format="JPEG",
options={"quality": 75},
)
screenshot_large = ImageSpecField(
source="screenshot",
processors=[ResizeToFit(800, 800)],
format="JPEG",
options={"quality": 85},
)
long_play_seconds = models.BigIntegerField(**BNULL)
long_play_complete = models.BooleanField(**BNULL)
long_play_last_scrobble = models.ForeignKey(
"self",
**BNULL,
on_delete=models.SET_NULL,
related_name="next_long_play_scrobbles",
)
class Meta:
indexes = [
@ -774,6 +886,7 @@ class Scrobble(TimeStampedModel):
"is_paused",
]
),
models.Index(fields=["user", "-timestamp"]),
]
@classmethod
@ -806,11 +919,14 @@ class Scrobble(TimeStampedModel):
@classmethod
def as_dict_by_type(cls, scrobble_qs: models.QuerySet) -> dict:
scrobbles_by_type = defaultdict(list)
scrobbles = (
scrobble_qs.with_related()
if hasattr(scrobble_qs, "with_related")
else scrobble_qs
)
if hasattr(scrobble_qs, "with_related"):
media_types_present = list(
scrobble_qs.values_list("media_type", flat=True).distinct()
)
scrobbles = scrobble_qs.with_related_for_types(media_types_present)
else:
scrobbles = scrobble_qs
for scrobble in scrobbles:
scrobbles_by_type[scrobble.media_type].append(scrobble)
@ -825,7 +941,7 @@ class Scrobble(TimeStampedModel):
# Remove any locations without titles
if "GeoLocation" in scrobbles_by_type.keys():
for loc_scrobble in scrobbles_by_type["GeoLocation"]:
for loc_scrobble in list(scrobbles_by_type["GeoLocation"]):
if not loc_scrobble.media_obj.title:
scrobbles_by_type["GeoLocation"].remove(loc_scrobble)
scrobbles_by_type["GeoLocation_count"] -= 1
@ -850,7 +966,7 @@ class Scrobble(TimeStampedModel):
@property
def finish_url(self) -> str:
return reverse("scrobbles:finish", kwargs={"uuid": self.uuid})
return reverse("scrobbles:finish", kwargs={"pk": self.pk})
def save(self, *args, **kwargs):
class_name = self.media_obj.__class__.__name__
@ -892,10 +1008,7 @@ class Scrobble(TimeStampedModel):
return super(Scrobble, self).save(*args, **kwargs)
def get_absolute_url(self):
if not self.uuid:
self.uuid = uuid4()
self.save(update_fields=["uuid"])
return reverse("scrobbles:detail", kwargs={"uuid": self.uuid})
return reverse("scrobbles:detail", kwargs={"pk": self.pk})
def get_share_url(self):
if self.visibility == Visibility.PRIVATE:
@ -907,27 +1020,6 @@ class Scrobble(TimeStampedModel):
self.share_token_version += 1
self.save(update_fields=["share_token_version"])
def push_to_archivebox(self):
pushable_media = hasattr(self.media_obj, "push_to_archivebox") and callable(
self.media_obj.push_to_archivebox
)
if pushable_media and self.user.profile.archivebox_url:
try:
self.media_obj.push_to_archivebox(
url=self.user.profile.archivebox_url,
username=self.user.profile.archivebox_username,
password=self.user.profile.archivebox_password,
)
except Exception:
logger.info(
"Failed to push URL to archivebox",
extra={
"archivebox_url": self.user.profile.archivebox_url,
"archivebox_username": self.user.profile.archivebox_username,
},
)
@property
def logdata(self) -> Optional[logdata.BaseLogData]:
if self.media_obj:
@ -936,6 +1028,8 @@ class Scrobble(TimeStampedModel):
logdata_cls = logdata.BaseLogData
log_dict = self.log
if isinstance(log_dict, logdata.BaseLogData):
log_dict = log_dict.asdict
if isinstance(self.log, str):
# There's nothing stopping django from saving a string in a JSONField :(
logger.warning(
@ -947,24 +1041,8 @@ class Scrobble(TimeStampedModel):
if not log_dict:
log_dict = {}
# Special handling for GeoLocationLogData - data is nested under 'movement_detection'
# TODO there's a better way to fix this this at the LogData level
if logdata_cls.__name__ == "GeoLocationLogData":
instance_data = log_dict.get("movement_detection", {}).copy()
# Add top-level fields that GeoLocationLogData expects from BaseLogData/WithPeopleLogData
for field_name in ["description", "notes", "with_people_ids"]:
if field_name in log_dict:
instance_data[field_name] = log_dict[field_name]
try:
return logdata_cls(**instance_data)
except Exception as e:
logger.warning("Log data could not be loaded", e)
return logdata_cls()
# Strip log-only keys (stored in JSONField but not part of LogData dataclass)
logdata_kwargs = {
k: v for k, v in log_dict.items() if k in logdata_cls().__dataclass_fields__
}
# Use LogData's from_log_dict to handle any custom nesting/structure
logdata_kwargs = logdata_cls.from_log_dict(log_dict)
try:
return logdata_cls(**logdata_kwargs)
@ -1141,8 +1219,8 @@ class Scrobble(TimeStampedModel):
if self.is_long_play:
long_play_secs = 0
if self.previous and not self.previous.long_play_complete:
long_play_secs = self.previous.long_play_seconds or 0
if self.long_play_last_scrobble and not self.long_play_last_scrobble.long_play_complete:
long_play_secs = self.long_play_last_scrobble.long_play_seconds or 0
percent = int(((playback_seconds + long_play_secs) / run_time_secs) * 100)
return percent
@ -1298,6 +1376,8 @@ class Scrobble(TimeStampedModel):
media_obj = self.channel
if self.birding_location:
media_obj = self.birding_location
if self.disc_golf:
media_obj = self.disc_golf
return media_obj
def __str__(self):
@ -1412,20 +1492,18 @@ class Scrobble(TimeStampedModel):
or scrobble_data["playback_status"] == "stopped"
):
if read_log_page:
page_list = scrobble.log.get("page_data", [])
if page_list:
for page in page_list:
page_data = scrobble.log.get("page_data", {})
if page_data:
for page_num, page in page_data.items():
if not page.get("end_ts", None):
page["end_ts"] = int(timezone.now().timestamp())
page["duration"] = page["end_ts"] - page.get("start_ts")
page_list.append(
BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
page_data[read_log_page] = BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
scrobble.log["page_data"] = page_list
scrobble.log["page_data"] = page_data
scrobble.save(update_fields=["log"])
elif "log" in scrobble_data.keys() and scrobble.log:
scrobble_data["log"] = scrobble.log | scrobble_data["log"]
@ -1436,13 +1514,13 @@ class Scrobble(TimeStampedModel):
if read_log_page:
scrobble_data["log"] = BookLogData(
page_data=[
BookPageLogData(
page_data={
read_log_page: BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
]
)
}
).asdict
logger.info(
f"[scrobbling] creating new scrobble",
@ -1457,9 +1535,27 @@ class Scrobble(TimeStampedModel):
"calories", None
):
if media.calories:
scrobble_data["log"] = FoodLogData(calories=media.calories)
scrobble_data["log"] = FoodLogData(calories=media.calories).asdict
if mtype not in LONG_PLAY_MEDIA.values():
scrobble_data.pop("long_play_complete", None)
scrobble = cls.create(scrobble_data)
if mtype in LONG_PLAY_MEDIA.values():
last_finished = (
cls.objects.filter(
models.Q(**{key: media}),
user_id=user_id,
timestamp__lt=scrobble.timestamp,
)
.order_by("-timestamp")
.first()
)
if last_finished:
scrobble.long_play_last_scrobble = last_finished
scrobble.save(update_fields=["long_play_last_scrobble"])
return scrobble
@classmethod
@ -1718,8 +1814,8 @@ class Scrobble(TimeStampedModel):
# Set our playback seconds, and calc long play seconds
self.playback_position_seconds = seconds_elapsed
if self.previous:
past_seconds = self.previous.long_play_seconds or 0
if self.long_play_last_scrobble:
past_seconds = self.long_play_last_scrobble.long_play_seconds or 0
self.long_play_seconds = past_seconds + seconds_elapsed
@ -1780,33 +1876,27 @@ class Scrobble(TimeStampedModel):
return False
def calculate_reading_stats(self, commit=True):
page_data = self.log.get("page_data")
page_data = self.logdata.page_data
if page_data:
# --- Sort safely by numeric page_number ---
def safe_page_number(entry):
try:
return int(getattr("page_number", entry), 0)
except (ValueError, TypeError):
return float("inf")
if isinstance(page_data, dict):
logger.warning("Page data is dict, migrate koreader data")
return
valid_pages = sorted(int(k) for k in page_data.keys())
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
elif isinstance(page_data, list):
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
page_data.sort(key=safe_page_number)
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
else:
page_start = self.log.get("page_start")
page_end = self.log.get("page_end")
@ -1846,9 +1936,10 @@ class FavoriteMedia(TimeStampedModel):
birding_location = models.ForeignKey(
BirdingLocation, on_delete=models.CASCADE, **BNULL
)
media_type = models.CharField(
max_length=20, choices=Scrobble.MediaType.choices
disc_golf = models.ForeignKey(
DiscGolfCourse, on_delete=models.CASCADE, **BNULL
)
media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
sent_to_mopidy = models.BooleanField(default=False)
class Meta:
@ -1898,6 +1989,8 @@ class FavoriteMedia(TimeStampedModel):
media_obj = self.channel
if self.birding_location:
media_obj = self.birding_location
if self.disc_golf:
media_obj = self.disc_golf
return media_obj
@classmethod
@ -1927,6 +2020,7 @@ class FavoriteMedia(TimeStampedModel):
"Mood": "mood",
"BrickSet": "brick_set",
"BirdingLocation": "birding_location",
"DiscGolf": "disc_golf",
}
fk = fk_map.get(media_type)

View File

@ -79,6 +79,27 @@ class ScrobbleNtfyNotification(ScrobbleNotification):
)
class LastFmImportNtfyNotification(BasicNtfyNotification):
def __init__(self, lfm_import, scrobble_count):
super().__init__(lfm_import.user.profile)
self.ntfy_str = f"Imported {scrobble_count} scrobble(s) from Last.fm"
self.click_url = lfm_import.get_absolute_url()
self.title = "Last.fm Import Complete"
def send(self):
if self.profile and self.profile.ntfy_enabled and self.profile.ntfy_url:
requests.post(
self.profile.ntfy_url,
data=self.ntfy_str.encode(encoding="utf-8"),
headers={
"Title": self.title,
"Priority": "default",
"Tags": "musical_note",
"Click": self.click_url,
},
)
class MoodNtfyNotification(BasicNtfyNotification):
def __init__(self, profile, **kwargs):
super().__init__(profile)

View File

@ -13,6 +13,7 @@ from books.models import Book, BookLogData, BookPageLogData
from books.utils import parse_readcomicsonline_uri
from bricksets.models import BrickSet
from dateutil.parser import parse
from discgolf.models import DiscGolfCourse
from django.utils import timezone
from foods.models import Food
from foods.sources.rscraper import RecipeScraperService
@ -32,6 +33,7 @@ from scrobbles.constants import (
)
from scrobbles.models import Scrobble
from scrobbles.notifications import ScrobbleNtfyNotification
from scrobbles.tasks import push_scrobble_to_archivebox
from scrobbles.utils import (
convert_to_seconds,
extract_domain,
@ -329,8 +331,6 @@ def manual_scrobble_book(
source = READCOMICSONLINE_URL.replace("https://", "")
# TODO: Check for scrobble of this book already and if so, update the page count
book = Book.find_or_create(title, url=url, enrich=True)
scrobble_dict = {
@ -357,10 +357,7 @@ def manual_scrobble_book(
if action == "stop":
if url:
if isinstance(scrobble.log, "BookLogData"):
scrobble.log.resume_url = next_url_if_exists(url)
else:
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.save(update_fields=["log"])
scrobble.stop(force_finish=True)
@ -540,6 +537,7 @@ def email_scrobble_board_game(
"playback_position_seconds": duration_seconds,
"source": "BG Stats",
"log": log_data,
"visibility": "private",
}
scrobble = None
@ -1030,8 +1028,7 @@ def manual_scrobble_webpage(
if action == "stop":
scrobble.stop(force_finish=True)
else:
# possibly async this?
scrobble.push_to_archivebox()
push_scrobble_to_archivebox.delay(scrobble.id)
return scrobble
@ -1152,7 +1149,7 @@ def web_scrobbler_scrobble_video_or_song(
artist_name = data_dict.get("artist")
track_name = data_dict.get("track")
tracks = Track.objects.filter(
artist__name=data_dict.get("artist"), title=data_dict.get("track")
artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
)
if tracks.count() > 1:
logger.warning(
@ -1331,3 +1328,32 @@ def manual_scrobble_food(
)
return Scrobble.create_or_update(food, user_id, scrobble_dict)
def manual_scrobble_discgolf(
item_id: str,
user_id: int,
source: str = "Vrobbler",
action: Optional[str] = None,
):
from discgolf.models import DiscGolfCourse
course, _ = DiscGolfCourse.objects.get_or_create(title=item_id.strip())
scrobble_dict = {
"user_id": user_id,
"timestamp": timezone.now(),
"playback_position_seconds": 0,
"source": source,
}
logger.info(
"[scrobblers] manual disc golf scrobble request received",
extra={
"course_id": course.id,
"user_id": user_id,
"scrobble_dict": scrobble_dict,
},
)
return Scrobble.create_or_update(course, user_id, scrobble_dict)

View File

@ -1,5 +1,6 @@
import logging
from django.core.cache import cache
from django.db.models.signals import post_delete, post_save
from django.dispatch import receiver
from django.utils import timezone
@ -52,6 +53,11 @@ def _update_charts_for_timestamp(user, ts):
if ts is None:
return
lock_key = f"chart_update_{user.id}"
if not cache.add(lock_key, "locked", timeout=30):
logger.info(f"Chart update already queued for user {user.id}, skipping")
return
if timezone.is_naive(ts):
ts = timezone.make_aware(ts)

View File

@ -12,6 +12,7 @@ from charts.utils import (
from django.apps import apps
from django.conf import settings
from django.contrib.auth import get_user_model
from django.core.cache import cache
from django.db import models
from django.utils import timezone
@ -170,6 +171,16 @@ def process_ebird_csv_import(import_id):
birding_import.process()
@shared_task
def process_udisc_csv_import(import_id):
UDiscCSVImport = apps.get_model("scrobbles", "UDiscCSVImport")
udisc_import = UDiscCSVImport.objects.filter(id=import_id).first()
if not udisc_import:
logger.warn(f"UDiscCSVImport not found with id {import_id}")
return
udisc_import.process()
@shared_task
def process_scale_csv_import(import_id):
ScaleCSVImport = apps.get_model("scrobbles", "ScaleCSVImport")
@ -241,6 +252,11 @@ def update_charts_for_timestamp(user_id, year, month, day, week):
logger.error(f"User with id {user_id} not found")
return
lock_key = f"chart_update_running_{user_id}"
if not cache.add(lock_key, "locked", timeout=300):
logger.info(f"Chart update already running for user {user_id}, skipping")
return
try:
build_daily_charts(user, year, month, day, CHARTABLE_MEDIA_TYPES)
build_weekly_charts(user, year, week, CHARTABLE_MEDIA_TYPES)
@ -250,6 +266,27 @@ def update_charts_for_timestamp(user_id, year, month, day, week):
logger.info(f"[charts] Updated charts for {user} on {date_str}")
except Exception as e:
logger.error(f"[charts] Failed to update charts: {e}")
finally:
cache.delete(lock_key)
@shared_task
def push_scrobble_to_archivebox(scrobble_id):
from scrobbles.models import Scrobble
scrobble = Scrobble.objects.filter(id=scrobble_id).first()
if not scrobble:
logger.warning(
"Scrobble %s not found for archivebox push", scrobble_id
)
return
webpage = scrobble.web_page
if not webpage:
logger.warning(
"Scrobble %s has no web_page for archivebox push", scrobble_id
)
return
webpage.push_to_archivebox(scrobble.user)
# ── Crontab replacements ──────────────────────────────────────────────────────

View File

@ -44,7 +44,7 @@ urlpatterns = [
name="lookup-manual-scrobble",
),
path(
"long-play-finish/<slug:uuid>/",
"long-play-finish/<slug:media_uuid>/",
views.scrobble_longplay_finish,
name="longplay-finish",
),
@ -147,6 +147,11 @@ urlpatterns = [
views.ScrobbleBirdingCSVImportDetailView.as_view(),
name="ebird-csv-import-detail",
),
path(
"imports/udisc-csv/<slug:slug>/",
views.ScrobbleUDiscCSVImportDetailView.as_view(),
name="udisc-csv-import-detail",
),
path(
"long-plays/",
views.ScrobbleLongPlaysView.as_view(),
@ -160,38 +165,38 @@ urlpatterns = [
name="shared-detail",
),
path(
"scrobbles/<slug:uuid>/",
"scrobbles/<int:pk>/",
views.ScrobbleDetailView.as_view(),
name="detail",
),
path(
"scrobbles/<slug:uuid>/regenerate-share-token/",
"scrobbles/<int:pk>/regenerate-share-token/",
views.RegenerateShareTokenView.as_view(),
name="regenerate-share-token",
),
path(
"scrobbles/<slug:uuid>/change-visibility/",
"scrobbles/<int:pk>/change-visibility/",
views.ChangeVisibilityView.as_view(),
name="change-visibility",
),
path(
"scrobbles/<slug:uuid>/share-analytics/",
"scrobbles/<int:pk>/share-analytics/",
views.ScrobbleShareAnalyticsView.as_view(),
name="share-analytics",
),
path(
"scrobbles/<slug:uuid>/add-to-mopidy-queue/",
"scrobbles/<int:pk>/add-to-mopidy-queue/",
views.add_to_mopidy_queue,
name="add-to-mopidy-queue",
),
path(
"scrobbles/<slug:uuid>/add-to-mopidy-monthly-playlist/",
"scrobbles/<int:pk>/add-to-mopidy-monthly-playlist/",
views.add_to_mopidy_monthly_playlist,
name="add-to-mopidy-monthly-playlist",
),
path("scrobbles/<slug:uuid>/start/", views.scrobble_start, name="start"),
path("scrobbles/<slug:uuid>/finish/", views.scrobble_finish, name="finish"),
path("scrobbles/<slug:uuid>/cancel/", views.scrobble_cancel, name="cancel"),
path("scrobbles/<slug:media_uuid>/start/", views.scrobble_start, name="start"),
path("scrobbles/<int:pk>/finish/", views.scrobble_finish, name="finish"),
path("scrobbles/<int:pk>/cancel/", views.scrobble_cancel, name="cancel"),
path(
"favorite/<str:media_type>/<int:object_id>/toggle/",
views.toggle_favorite,

View File

@ -1,4 +1,5 @@
import hashlib
import html
import logging
import requests
import re
@ -153,10 +154,11 @@ def import_lastfm_for_all_users(restart=False):
last_processed = lfm_import.processed_finished
else:
logger.info(
f"Not resuming failed LastFM import {lfm_import.id} for user {user_id}, use restart=True to restart"
"No existing LastFM import, we should start a monthly parsing of lastFm for this user going back to 2002"
"No existing LastFM import for user %s, "
"starting a full parse",
user_id,
)
continue
last_processed = None
lfm_client = LastFM(user=get_user_model().objects.filter(id=user_id).first())
@ -795,6 +797,7 @@ def tokenize_title_to_tags(title: str) -> list[str]:
if not title:
return []
title = html.unescape(title)
cleaned = re.sub(r"[\(\)\[\]\{\}]", "", title)
cleaned = re.sub(r"[^\w\s]", "", cleaned)

View File

@ -87,6 +87,7 @@ from scrobbles.models import (
ScrobbleQuerySet,
ShareViewLog,
TrailGPXImport,
UDiscCSVImport,
)
from scrobbles.scrobblers import *
from scrobbles.tasks import (
@ -206,6 +207,19 @@ class ScrobbleableDetailView(ChartContextMixin, DetailView):
context_data["scrobbles"] = page_obj.object_list
context_data["is_paginated"] = paginator.num_pages > 1
media = self.object
if hasattr(media, "is_long_play_media") and media.is_long_play_media():
qs = media.scrobble_set.filter(user=self.request.user)
completed = qs.filter(long_play_complete=True).order_by("-timestamp").first()
if completed and completed.long_play_seconds:
context_data["long_play_total_seconds"] = completed.long_play_seconds
context_data["long_play_finished_date"] = completed.timestamp
else:
latest_finished = qs.filter(played_to_completion=True).order_by("-timestamp").first()
if latest_finished and latest_finished.long_play_seconds:
context_data["long_play_total_seconds"] = latest_finished.long_play_seconds
context_data["long_play_finished_date"] = None
return context_data
@ -522,6 +536,8 @@ class BaseScrobbleImportDetailView(DetailView):
title = "Scale CSV Import"
if self.model == TrailGPXImport:
title = "Trail GPX Import"
if self.model == UDiscCSVImport:
title = "uDisc CSV Import"
context_data["title"] = title
return context_data
@ -558,6 +574,10 @@ class ScrobbleBirdingCSVImportDetailView(BaseScrobbleImportDetailView):
model = EBirdCSVImport
class ScrobbleUDiscCSVImportDetailView(BaseScrobbleImportDetailView):
model = UDiscCSVImport
class ManualScrobbleView(FormView):
form_class = ScrobbleForm
template_name = "scrobbles/manual_form.html"
@ -826,10 +846,10 @@ def import_audioscrobbler_file(request):
@api_view(["GET"])
@permission_classes([IsAuthenticated])
def scrobble_start(request, uuid):
def scrobble_start(request, media_uuid):
logger.info(
"[scrobble_start] called",
extra={"request": request, "uuid": uuid},
extra={"request": request, "media_uuid": media_uuid},
)
user = request.user
success_url = request.META.get("HTTP_REFERER")
@ -840,14 +860,14 @@ def scrobble_start(request, uuid):
media_obj = None
for app, model in PLAY_AGAIN_MEDIA.items():
media_model = apps.get_model(app_label=app, model_name=model)
media_obj = media_model.objects.filter(uuid=uuid).first()
media_obj = media_model.objects.filter(uuid=media_uuid).first()
if media_obj:
break
if not media_obj:
logger.info(
"[scrobble_start] media object not found",
extra={"uuid": uuid, "user_id": user.id},
extra={"media_uuid": media_uuid, "user_id": user.id},
)
raise Exception("No media object provided to scrobble")
@ -884,7 +904,7 @@ def scrobble_start(request, uuid):
)
else:
messages.add_message(
request, messages.ERROR, f"Media with uuid {uuid} not found."
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
)
if (
@ -902,25 +922,58 @@ def scrobble_start(request, uuid):
@api_view(["GET"])
def scrobble_longplay_finish(request, uuid):
def scrobble_longplay_finish(request, media_uuid):
user = request.user
success_url = request.META.get("HTTP_REFERER")
if not user.is_authenticated:
return HttpResponseRedirect(success_url)
# Try scrobble UUID first
scrobble = Scrobble.objects.filter(uuid=media_uuid, user=user).first()
if scrobble:
if scrobble.long_play_complete == True:
scrobble.long_play_complete = None
scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.INFO,
f"Long play of {scrobble.media_obj} marked as not complete.",
)
else:
scrobble.long_play_complete = True
scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.SUCCESS,
f"Long play of {scrobble.media_obj} finished.",
)
return HttpResponseRedirect(success_url)
# Fall back to media UUID (existing behavior)
media_obj = None
for app, model in LONG_PLAY_MEDIA.items():
media_model = apps.get_model(app_label=app, model_name=model)
media_obj = media_model.objects.filter(uuid=uuid).first()
media_obj = media_model.objects.filter(uuid=media_uuid).first()
if media_obj:
break
if not media_obj:
return
messages.add_message(
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
)
return HttpResponseRedirect(success_url)
last_scrobble = media_obj.last_long_play_scrobble_for_user(user)
if last_scrobble and last_scrobble.long_play_complete == False:
if last_scrobble and last_scrobble.long_play_complete == True:
last_scrobble.long_play_complete = None
last_scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.INFO,
f"Long play of {media_obj} marked as not complete.",
)
elif last_scrobble:
last_scrobble.long_play_complete = True
last_scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
@ -930,14 +983,14 @@ def scrobble_longplay_finish(request, uuid):
)
else:
messages.add_message(
request, messages.ERROR, f"Media with uuid {uuid} not found."
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
)
return HttpResponseRedirect(success_url)
@api_view(["GET"])
@permission_classes([IsAuthenticated])
def scrobble_finish(request, uuid):
def scrobble_finish(request, pk):
user = request.user
success_url = request.META.get("HTTP_REFERER")
if not success_url:
@ -946,7 +999,7 @@ def scrobble_finish(request, uuid):
if not user.is_authenticated:
return HttpResponseRedirect(success_url)
scrobble = Scrobble.objects.filter(user=user, uuid=uuid).first()
scrobble = Scrobble.objects.filter(user=user, pk=pk).first()
if scrobble:
scrobble.stop(force_finish=True)
messages.add_message(
@ -961,14 +1014,14 @@ def scrobble_finish(request, uuid):
@api_view(["GET"])
@permission_classes([IsAuthenticated])
def scrobble_cancel(request, uuid):
def scrobble_cancel(request, pk):
user = request.user
success_url = reverse_lazy("vrobbler-home")
if not user.is_authenticated:
return HttpResponseRedirect(success_url)
scrobble = Scrobble.objects.filter(user=user, uuid=uuid).first()
scrobble = Scrobble.objects.filter(user=user, pk=pk).first()
if scrobble:
scrobble.cancel()
messages.add_message(
@ -982,11 +1035,11 @@ def scrobble_cancel(request, uuid):
@require_POST
def add_to_mopidy_queue(request, uuid):
def add_to_mopidy_queue(request, pk):
if not request.user.is_authenticated:
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
mopidy_url = request.user.profile.mopidy_api_url
if not mopidy_url:
@ -995,22 +1048,22 @@ def add_to_mopidy_queue(request, uuid):
messages.ERROR,
"Mopidy API URL not configured in your profile settings.",
)
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
from scrobbles.tasks import add_scrobble_to_mopidy_queue as task
task.delay(scrobble.id)
msg = f'Adding "{scrobble.media_obj}" to Mopidy queue.'
messages.add_message(request, messages.SUCCESS, msg)
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
@require_POST
def add_to_mopidy_monthly_playlist(request, uuid):
def add_to_mopidy_monthly_playlist(request, pk):
if not request.user.is_authenticated:
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
profile = request.user.profile
pattern = profile.monthly_mopidy_playlist_pattern
@ -1020,7 +1073,7 @@ def add_to_mopidy_monthly_playlist(request, uuid):
messages.ERROR,
"Monthly playlist pattern or Mopidy API URL not configured in your profile.",
)
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
now = now_user_timezone(profile)
playlist_name = DateFormat(now).format(pattern)
@ -1033,7 +1086,7 @@ def add_to_mopidy_monthly_playlist(request, uuid):
messages.SUCCESS,
f'Adding "{scrobble.media_obj}" to monthly playlist "{playlist_name}".',
)
return redirect("scrobbles:detail", uuid=uuid)
return redirect("scrobbles:detail", pk=pk)
@require_POST
@ -1062,6 +1115,7 @@ def toggle_favorite(request, media_type, object_id):
"Mood": ("moods", "Mood"),
"BrickSet": ("bricksets", "BrickSet"),
"BirdingLocation": ("birds", "BirdingLocation"),
"DiscGolf": ("discgolf", "DiscGolfCourse"),
}
app_label, model_name = app_model_map.get(media_type, (None, None))
@ -1138,8 +1192,6 @@ class ScrobbleStatusView(LoginRequiredMixin, TemplateView):
class ScrobbleDetailView(DetailView):
model = Scrobble
slug_field = "uuid"
slug_url_kwarg = "uuid"
paginate_by = 100
def get_object(self, queryset=None):
@ -1264,6 +1316,24 @@ class ScrobbleDetailView(DetailView):
else:
context["has_mopidy_uri"] = False
if self.object.is_long_play and fk_field:
all_scrobbles = Scrobble.objects.filter(
user=user, **{fk_field: media_obj}
)
completed = all_scrobbles.filter(
long_play_complete=True
).order_by("-timestamp").first()
if completed and completed.long_play_seconds:
context["long_play_total_seconds"] = completed.long_play_seconds
context["long_play_finished_date"] = completed.timestamp
else:
latest_finished = all_scrobbles.filter(
played_to_completion=True
).order_by("-timestamp").first()
if latest_finished and latest_finished.long_play_seconds:
context["long_play_total_seconds"] = latest_finished.long_play_seconds
context["long_play_finished_date"] = None
return context
@ -1321,15 +1391,15 @@ class ScrobbleExploreView(ListView):
class RegenerateShareTokenView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
def post(self, request, pk):
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
scrobble.regenerate_share_token()
return redirect(scrobble.get_absolute_url())
class ChangeVisibilityView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
def post(self, request, pk):
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
visibility = request.POST.get("visibility")
if visibility not in (Visibility.PUBLIC, Visibility.SHARED, Visibility.PRIVATE):
return redirect(scrobble.get_absolute_url())
@ -1340,8 +1410,6 @@ class ChangeVisibilityView(LoginRequiredMixin, View):
class ScrobbleShareAnalyticsView(LoginRequiredMixin, DetailView):
model = Scrobble
slug_field = "uuid"
slug_url_kwarg = "uuid"
template_name = "scrobbles/scrobble_share_analytics.html"
def get_queryset(self):
@ -1657,6 +1725,7 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
for scrobble in day_map[day_num]:
day_scrobbles.append(
{
"id": scrobble.pk,
"uuid": scrobble.uuid,
"emoji": self.MEDIA_EMOJI.get(scrobble.media_type, "📌"),
"title": (

View File

@ -23,6 +23,7 @@ class SportAdmin(admin.ModelAdmin):
class LeagueAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "abbreviation_str")
raw_id_fields = ("sport",)
ordering = ("name",)
@ -30,6 +31,7 @@ class LeagueAdmin(admin.ModelAdmin):
class PlayerAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league", "team")
raw_id_fields = ("league", "team")
ordering = ("name",)
@ -37,6 +39,7 @@ class PlayerAdmin(admin.ModelAdmin):
class SeasonAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league")
raw_id_fields = ("league",)
ordering = ("name",)
@ -44,6 +47,7 @@ class SeasonAdmin(admin.ModelAdmin):
class RoundAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "season")
raw_id_fields = ("season",)
ordering = ("name",)
@ -51,6 +55,7 @@ class RoundAdmin(admin.ModelAdmin):
class TeamAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league")
raw_id_fields = ("league",)
ordering = ("name",)
@ -63,6 +68,7 @@ class SportEventAdmin(admin.ModelAdmin):
"event_type",
"comp_str",
)
raw_id_fields = ("league", "teams", "players", "round")
list_filter = ("league", "event_type")
ordering = ("-created",)
inlines = [

View File

View File

@ -0,0 +1,9 @@
from django.contrib import admin
from trends.models import TrendResult
@admin.register(TrendResult)
class TrendResultAdmin(admin.ModelAdmin):
list_display = ("user", "trend_slug", "computed_at", "created")
list_filter = ("user", "trend_slug")
ordering = ("-computed_at",)

View File

@ -0,0 +1,6 @@
from django.apps import AppConfig
class TrendsConfig(AppConfig):
default_auto_field = "django.db.models.BigAutoField"
name = "trends"

View File

@ -0,0 +1,82 @@
import logging
from django.contrib.auth import get_user_model
from django.core.management.base import BaseCommand
from django.utils import timezone
from trends.trends import TREND_REGISTRY
from trends.utils import compute_and_save_trend, get_supported_periods
logger = logging.getLogger(__name__)
User = get_user_model()
class Command(BaseCommand):
help = "Compute trends for all users (or a specific user)"
def add_arguments(self, parser):
parser.add_argument(
"--user-id",
type=int,
help="Compute trends for a specific user only",
)
def handle(self, *args, **options):
if options["user_id"]:
user = User.objects.filter(id=options["user_id"]).first()
if not user:
self.stderr.write(
self.style.ERROR(f"User with id {options['user_id']} not found")
)
return
users = [user]
else:
users = User.objects.filter(is_active=True)
total_users = len(users)
self.stdout.write(f"Computing trends for {total_users} user(s)...")
overall_start = timezone.now()
ok_count = 0
fail_count = 0
for user in users:
total_trends = len(TREND_REGISTRY)
self.stdout.write(f" {user} ({user.id}): {total_trends} trends...")
user_start = timezone.now()
user_ok = 0
user_fail = 0
for idx, (slug, _) in enumerate(TREND_REGISTRY.items(), start=1):
periods = get_supported_periods(slug)
self.stdout.write(f" [{idx}/{total_trends}] {slug}...\n")
for period in periods:
trend_start = timezone.now()
self.stdout.write(f" {period}... ", ending="")
try:
elapsed = compute_and_save_trend(user, slug, period)
self.stdout.write(self.style.SUCCESS(f"OK ({elapsed:.1f}s)"))
user_ok += 1
except Exception as e:
elapsed = (timezone.now() - trend_start).total_seconds()
self.stdout.write(
self.style.ERROR(f"FAILED after {elapsed:.1f}s: {e}")
)
user_fail += 1
user_elapsed = (timezone.now() - user_start).total_seconds()
self.stdout.write(
self.style.SUCCESS(
f" {user}: {user_ok} OK, {user_fail} failed "
f"({user_elapsed:.1f}s total)"
)
)
ok_count += user_ok
fail_count += user_fail
overall_elapsed = (timezone.now() - overall_start).total_seconds()
self.stdout.write(
self.style.SUCCESS(
f"Done! {ok_count} OK, {fail_count} failed "
f"({overall_elapsed:.1f}s across {total_users} user(s))"
)
)

View File

@ -0,0 +1,57 @@
# Generated by Django 4.2.29 on 2026-06-16 14:52
import django.db.models.deletion
import django_extensions.db.fields
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
initial = True
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
]
operations = [
migrations.CreateModel(
name="TrendResult",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("trend_slug", models.CharField(db_index=True, max_length=100)),
("computed_at", models.DateTimeField(auto_now_add=True)),
("data", models.JSONField(default=dict)),
(
"user",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
to=settings.AUTH_USER_MODEL,
),
),
],
options={
"unique_together": {("user", "trend_slug")},
},
),
]

View File

@ -0,0 +1,37 @@
# Generated by Django 4.2.29 on 2026-06-17 14:32
from django.conf import settings
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
("trends", "0001_initial"),
]
operations = [
migrations.AlterUniqueTogether(
name="trendresult",
unique_together=set(),
),
migrations.AddField(
model_name="trendresult",
name="period",
field=models.CharField(
choices=[
("last_30", "Last 30 days"),
("last_90", "Last 90 days"),
("last_year", "Last year"),
("all_time", "All time"),
],
default="all_time",
max_length=20,
),
),
migrations.AlterUniqueTogether(
name="trendresult",
unique_together={("user", "trend_slug", "period")},
),
]

View File

@ -0,0 +1,29 @@
from django.contrib.auth import get_user_model
from django.db import models
from django_extensions.db.models import TimeStampedModel
User = get_user_model()
PERIOD_CHOICES = [
("last_30", "Last 30 days"),
("last_90", "Last 90 days"),
("last_year", "Last year"),
]
class TrendResult(TimeStampedModel):
user = models.ForeignKey(User, on_delete=models.CASCADE)
trend_slug = models.CharField(max_length=100, db_index=True)
period = models.CharField(
max_length=20,
choices=PERIOD_CHOICES,
default="last_30",
)
computed_at = models.DateTimeField(auto_now_add=True)
data = models.JSONField(default=dict)
class Meta:
unique_together = ["user", "trend_slug", "period"]
def __str__(self):
return f"{self.user} - {self.trend_slug} ({self.period})"

View File

@ -0,0 +1,69 @@
import logging
from celery import shared_task
from django.contrib.auth import get_user_model
from django.utils import timezone
from trends.trends import TREND_REGISTRY
from trends.utils import compute_and_save_trend, get_supported_periods
logger = logging.getLogger(__name__)
User = get_user_model()
@shared_task
def compute_all_trends():
user_ids = list(User.objects.filter(is_active=True).values_list("id", flat=True))
logger.info("Dispatching trend computation for %d users", len(user_ids))
for uid in user_ids:
compute_user_trends.delay(uid)
@shared_task
def compute_user_trends(user_id):
try:
user = User.objects.get(id=user_id)
except User.DoesNotExist:
logger.warning("User %s not found, skipping trends", user_id)
return
total = len(TREND_REGISTRY)
logger.info(
"Computing %d trends for user %s (%d)",
total,
user,
user_id,
)
for idx, (slug, _) in enumerate(TREND_REGISTRY.items(), start=1):
compute_single_trend.delay(user_id, slug)
logger.info("Dispatched all %d trends for user %s (%d)", total, user, user_id)
@shared_task
def compute_single_trend(user_id, slug):
try:
user = User.objects.get(id=user_id)
except User.DoesNotExist:
logger.warning("User %d not found for trend '%s', skipping", user_id, slug)
return
if slug not in TREND_REGISTRY:
logger.warning("Unknown trend slug '%s' for user %d", slug, user_id)
return
periods = get_supported_periods(slug)
for period in periods:
logger.info("[%s/%s] Computing for user %d...", slug, period, user_id)
try:
elapsed = compute_and_save_trend(user, slug, period)
logger.info(
"[%s/%s] Completed for user %d in %.1fs",
slug,
period,
user_id,
elapsed,
)
except Exception:
logger.exception("[%s/%s] Failed for user %d", slug, period, user_id)

View File

@ -0,0 +1,47 @@
<div class="row">
<div class="col-12">
{% if data.distribution %}
<p class="text-muted mb-3">
Total scrobbles{% if current_period_label %} ({{ current_period_label }}){% endif %}: <strong>{{ data.total_count }}</strong>
</p>
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Media Type</th>
<th class="text-end">Total</th>
<th class="text-end">Completed</th>
<th class="text-end">%</th>
<th>Distribution</th>
</tr>
</thead>
<tbody>
{% with max=data.distribution.0.count %}
{% for entry in data.distribution %}
<tr>
<td>{{ entry.media_type }}</td>
<td class="text-end">{{ entry.count }}</td>
<td class="text-end">{{ entry.completed }}</td>
<td class="text-end">{{ entry.pct }}%</td>
<td style="width: 30%;">
{% if max > 0 %}
<div class="progress" style="height: 12px;">
<div class="progress-bar" role="progressbar"
style="width: {{ entry.pct }}%;"
aria-valuenow="{{ entry.count }}"
aria-valuemin="0" aria-valuemax="{{ max }}">
</div>
</div>
{% endif %}
</td>
</tr>
{% endfor %}
{% endwith %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No activity data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,89 @@
<div class="row">
{% if data.trails %}
<div class="col-md-6 mb-3">
<h4>🥾 While on Trails</h4>
{% for trail in data.trails %}
<div class="card mb-2">
<div class="card-body py-2">
<h6 class="card-title mb-1">
{% if trail.uuid %}
<a href="{% url 'trails:trail_detail' trail.uuid %}">{{ trail.name }}</a>
{% else %}
{{ trail.name }}
{% endif %}
<small class="text-muted">({{ trail.total_sessions }} sessions)</small>
</h6>
{% if trail.tracks %}
<table class="table table-sm table-borderless mb-0">
<thead>
<tr>
<th>Track</th>
<th>Artist</th>
<th class="text-end">Plays</th>
</tr>
</thead>
<tbody>
{% for t in trail.tracks %}
<tr>
<td>{% if t.track_uuid %}<a href="{% url 'music:track_detail' t.track_uuid %}">{{ t.track_name }}</a>{% else %}{{ t.track_name }}{% endif %}</td>
<td>{{ t.artist_name }}</td>
<td class="text-end">{{ t.count }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% endif %}
</div>
</div>
{% empty %}
<p class="text-muted">No concurrent listening data for trails.</p>
{% endfor %}
</div>
{% endif %}
{% if data.locations %}
<div class="col-md-6 mb-3">
<h4>📍 While at Locations</h4>
{% for loc in data.locations %}
<div class="card mb-2">
<div class="card-body py-2">
<h6 class="card-title mb-1">
{% if loc.uuid %}
<a href="{% url 'locations:geolocation_detail' loc.uuid %}">{{ loc.name }}</a>
{% else %}
{{ loc.name }}
{% endif %}
<small class="text-muted">({{ loc.total_sessions }} sessions)</small>
</h6>
{% if loc.tracks %}
<table class="table table-sm table-borderless mb-0">
<thead>
<tr>
<th>Track</th>
<th>Artist</th>
<th class="text-end">Plays</th>
</tr>
</thead>
<tbody>
{% for t in loc.tracks %}
<tr>
<td>{% if t.track_uuid %}<a href="{% url 'music:track_detail' t.track_uuid %}">{{ t.track_name }}</a>{% else %}{{ t.track_name }}{% endif %}</td>
<td>{{ t.artist_name }}</td>
<td class="text-end">{{ t.count }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% endif %}
</div>
</div>
{% empty %}
<p class="text-muted">No concurrent listening data for locations.</p>
{% endfor %}
</div>
{% endif %}
</div>
{% if not data.trails and not data.locations %}
<p class="text-muted">No concurrent listening data found.</p>
{% endif %}

View File

@ -0,0 +1,42 @@
<div class="row">
{% if data.books %}
{% for book in data.books %}
<div class="col-md-6 mb-3">
<div class="card">
<div class="card-body py-2">
<h6 class="card-title mb-1">
{% if book.book_uuid %}
<a href="{% url 'books:book_detail' book.book_uuid %}">{{ book.book_title }}</a>
{% else %}
{{ book.book_title }}
{% endif %}
<small class="text-muted">({{ book.total_sessions }} listening sessions)</small>
</h6>
{% if book.tracks %}
<table class="table table-sm table-borderless mb-0">
<thead>
<tr>
<th>Track</th>
<th>Artist</th>
<th class="text-end">Plays</th>
</tr>
</thead>
<tbody>
{% for t in book.tracks %}
<tr>
<td>{% if t.track_uuid %}<a href="{% url 'music:track_detail' t.track_uuid %}">{{ t.track_name }}</a>{% else %}{{ t.track_name }}{% endif %}</td>
<td>{{ t.artist_name }}</td>
<td class="text-end">{{ t.count }}</td>
</tr>
{% endfor %}
</tbody>
</table>
{% endif %}
</div>
</div>
</div>
{% endfor %}
{% else %}
<p class="text-muted">No concurrent reading data found.</p>
{% endif %}
</div>

View File

@ -0,0 +1,78 @@
<div class="row">
<div class="col-md-6">
<h5 class="mb-3">By Hour of Day</h5>
{% if data.hours %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Hour</th>
<th class="text-end">Avg Quality</th>
<th class="text-end">Check-ins</th>
</tr>
</thead>
<tbody>
{% for entry in data.hours %}
{% if entry.count > 0 %}
<tr>
<td>
{% if entry.hour == 0 %}
12 AM
{% elif entry.hour < 12 %}
{{ entry.hour }} AM
{% elif entry.hour == 12 %}
12 PM
{% else %}
{{ entry.hour|add:"-12" }} PM
{% endif %}
</td>
<td class="text-end">
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
{{ entry.avg_quality }}
</span>
</td>
<td class="text-end">{{ entry.count }}</td>
</tr>
{% endif %}
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No hourly data.</p>
{% endif %}
</div>
<div class="col-md-6">
<h5 class="mb-3">By Day of Week</h5>
{% if data.days %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Day</th>
<th class="text-end">Avg Quality</th>
<th class="text-end">Check-ins</th>
</tr>
</thead>
<tbody>
{% for entry in data.days %}
{% if entry.count > 0 %}
<tr>
<td>{{ entry.day_name }}</td>
<td class="text-end">
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
{{ entry.avg_quality }}
</span>
</td>
<td class="text-end">{{ entry.count }}</td>
</tr>
{% endif %}
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No daily data.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,45 @@
<div class="row">
<div class="col-12">
{% if data.moods %}
<p class="text-muted mb-3">
Total mood check-ins{% if current_period_label %} ({{ current_period_label }}){% endif %}: <strong>{{ data.total }}</strong>
&middot; Positive: <strong>{{ data.positive_count }}</strong>
&middot; Negative: <strong>{{ data.negative_count }}</strong>
</p>
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Mood</th>
<th class="text-end">Count</th>
<th>Distribution</th>
</tr>
</thead>
<tbody>
{% with max=data.moods.0.count %}
{% for entry in data.moods %}
<tr>
<td>{{ entry.mood }}</td>
<td class="text-end">{{ entry.count }}</td>
<td style="width: 40%;">
{% if max > 0 %}
<div class="progress" style="height: 12px;">
<div class="progress-bar" role="progressbar"
style="width: {% widthratio entry.count max 100 %}%;"
aria-valuenow="{{ entry.count }}"
aria-valuemin="0" aria-valuemax="{{ max }}">
</div>
</div>
{% endif %}
</td>
</tr>
{% endfor %}
{% endwith %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No mood distribution data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,47 @@
<div class="row">
<div class="col-12">
{% if data.current_streak %}
<div class="alert alert-info">
<strong>Current streak:</strong>
{{ data.current_streak.length }} consecutive
<span class="{% if data.current_streak.mood_type == 'positive' %}text-success{% else %}text-danger{% endif %}">
{{ data.current_streak.mood_type }}
</span>
check-ins since {{ data.current_streak.start_date }}.
</div>
{% endif %}
{% if data.streaks %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>#</th>
<th>Mood Type</th>
<th class="text-end">Length</th>
<th>Start</th>
<th>End</th>
</tr>
</thead>
<tbody>
{% for streak in data.streaks %}
<tr>
<td>{{ forloop.counter }}</td>
<td>
<span class="{% if streak.mood_type == 'positive' %}text-success{% elif streak.mood_type == 'negative' %}text-danger{% endif %}">
{{ streak.mood_type|title }}
</span>
</td>
<td class="text-end">{{ streak.length }}</td>
<td>{{ streak.start_date }}</td>
<td>{{ streak.end_date }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No streak data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,39 @@
<div class="row">
<div class="col-12">
{% if data.trajectory %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Date</th>
<th class="text-end">Avg Quality</th>
<th class="text-end">Check-ins</th>
<th>Mood Bar</th>
</tr>
</thead>
<tbody>
{% for entry in data.trajectory %}
<tr>
<td>{{ entry.date }}</td>
<td class="text-end">{{ entry.avg_quality }}</td>
<td class="text-end">{{ entry.count }}</td>
<td style="width: 40%;">
<div class="progress" style="height: 16px;">
<div class="progress-bar {% if entry.avg_quality >= 5 %}bg-success{% elif entry.avg_quality >= 4 %}bg-info{% elif entry.avg_quality >= 3 %}bg-warning{% else %}bg-danger{% endif %}"
role="progressbar"
style="width: {% widthratio entry.avg_quality 7 100 %}%;"
aria-valuenow="{{ entry.avg_quality }}"
aria-valuemin="1" aria-valuemax="7">
</div>
</div>
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No mood check-in data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,64 @@
<div class="row">
<div class="col-md-6">
<h5 class="mb-3">By Weather Condition</h5>
{% if data.conditions %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Condition</th>
<th class="text-end">Avg Quality</th>
<th class="text-end">Check-ins</th>
</tr>
</thead>
<tbody>
{% for entry in data.conditions %}
<tr>
<td>{{ entry.condition }}</td>
<td class="text-end">
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
{{ entry.avg_quality }}
</span>
</td>
<td class="text-end">{{ entry.count }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No weather-linked mood data found.</p>
{% endif %}
</div>
<div class="col-md-6">
<h5 class="mb-3">By Temperature Range</h5>
{% if data.temp_ranges %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Temp Range</th>
<th class="text-end">Avg Quality</th>
<th class="text-end">Check-ins</th>
</tr>
</thead>
<tbody>
{% for entry in data.temp_ranges %}
<tr>
<td>{{ entry.range }}</td>
<td class="text-end">
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
{{ entry.avg_quality }}
</span>
</td>
<td class="text-end">{{ entry.count }}</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No temperature-linked mood data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,52 @@
<div class="row">
<div class="col-12">
{% if data.hours %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Hour</th>
<th class="text-end">Scrobbles</th>
<th>Distribution</th>
</tr>
</thead>
<tbody>
{% with total=data.hours|dictsortreversed:"count"|first %}
{% with max_count=total.count %}
{% for entry in data.hours %}
<tr>
<td>
{% if entry.hour == 0 %}
12 AM
{% elif entry.hour < 12 %}
{{ entry.hour }} AM
{% elif entry.hour == 12 %}
12 PM
{% else %}
{{ entry.hour|add:"-12" }} PM
{% endif %}
</td>
<td class="text-end">{{ entry.count }}</td>
<td style="width: 40%;">
{% if max_count > 0 %}
<div class="progress" style="height: 12px;">
<div class="progress-bar" role="progressbar"
style="width: {{ entry.count|floatformat:0 }}%;"
aria-valuenow="{{ entry.count }}"
aria-valuemin="0" aria-valuemax="{{ max_count }}">
</div>
</div>
{% endif %}
</td>
</tr>
{% endfor %}
{% endwith %}
{% endwith %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No activity data found.</p>
{% endif %}
</div>
</div>

View File

@ -0,0 +1,57 @@
<div class="row">
{% if current_period_label %}
<div class="col-12 mb-2">
<small class="text-muted">Period: {{ current_period_label }}</small>
</div>
{% endif %}
<div class="col-md-6 mb-3">
<div class="card">
<div class="card-body">
<h5 class="card-title">🎵 Reading with Music</h5>
{% if data.with_music %}
<table class="table table-sm mb-0">
<tr>
<th>Avg session duration</th>
<td>{{ data.with_music.avg_seconds }} seconds</td>
</tr>
<tr>
<th>Total reading time</th>
<td>{{ data.with_music.total_seconds }} seconds</td>
</tr>
<tr>
<th>Reading sessions</th>
<td>{{ data.with_music.sessions_count }}</td>
</tr>
</table>
{% else %}
<p class="text-muted mb-0">No data.</p>
{% endif %}
</div>
</div>
</div>
<div class="col-md-6 mb-3">
<div class="card">
<div class="card-body">
<h5 class="card-title">🔇 Reading without Music</h5>
{% if data.without_music %}
<table class="table table-sm mb-0">
<tr>
<th>Avg session duration</th>
<td>{{ data.without_music.avg_seconds }} seconds</td>
</tr>
<tr>
<th>Total reading time</th>
<td>{{ data.without_music.total_seconds }} seconds</td>
</tr>
<tr>
<th>Reading sessions</th>
<td>{{ data.without_music.sessions_count }}</td>
</tr>
</table>
{% else %}
<p class="text-muted mb-0">No data.</p>
{% endif %}
</div>
</div>
</div>
</div>

View File

@ -0,0 +1,38 @@
<div class="row">
<div class="col-12">
{% if data %}
<div class="table-responsive">
<table class="table table-striped table-sm">
<thead>
<tr>
<th>Media Type</th>
<th class="text-end">Recent ({{ current_period_label }})</th>
<th class="text-end">Previous ({{ current_period_label }})</th>
<th class="text-end">Change</th>
</tr>
</thead>
<tbody>
{% for mt, info in data.items %}
<tr>
<td>{{ mt }}</td>
<td class="text-end">{{ info.recent }}</td>
<td class="text-end">{{ info.previous }}</td>
<td class="text-end">
{% if info.change_pct > 0 %}
<span class="text-success">+{{ info.change_pct }}%</span>
{% elif info.change_pct < 0 %}
<span class="text-danger">{{ info.change_pct }}%</span>
{% else %}
<span class="text-muted">0%</span>
{% endif %}
</td>
</tr>
{% endfor %}
</tbody>
</table>
</div>
{% else %}
<p class="text-muted">No trending data found.</p>
{% endif %}
</div>
</div>

Some files were not shown because too many files have changed in this diff Show More