Compare commits

..

60 Commits
45.1 ... 52.0

Author SHA1 Message Date
01d25e1b55 [release] Bump to version 52.0
All checks were successful
build / test (push) Successful in 1m55s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 31s
- Allow marking media as long play complete from detail page
- Fix how long play scrobbles are tracked
- Paginate or limite scrobbles on media admin pages
- Clean up books admin
- Clean up favorites admin
2026-06-15 14:56:27 -04:00
c0be131e3d [longplay] Add ability to undo finishes
Some checks failed
build / test (push) Has been cancelled
2026-06-15 14:55:58 -04:00
7d3f615ed7 [longplay] Make sure they're marked status is correct
All checks were successful
build / test (push) Successful in 2m2s
2026-06-15 14:37:20 -04:00
c2138b3ac6 [longplay] Add finish long play button
All checks were successful
build / test (push) Successful in 1m53s
2026-06-15 14:15:34 -04:00
947713d44a [longplay] Fix how we store long plays
All checks were successful
build / test (push) Successful in 2m3s
2026-06-15 14:12:21 -04:00
12b76837a3 [project] Update todos
All checks were successful
build / test (push) Successful in 2m11s
2026-06-15 13:43:58 -04:00
102494ede7 [admin] Use raw ids where possible and simplify scrobble inlines
All checks were successful
build / test (push) Successful in 1m55s
2026-06-15 13:25:10 -04:00
96bda8d4ad [data] Add example data 2026-06-15 12:38:21 -04:00
46956d06d8 [books] Clean up admin a little 2026-06-15 12:37:32 -04:00
8a28d0675b [release] Bump to version 51.4
All checks were successful
build / test (push) Successful in 1m56s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 38s
- Clean up metadata comicbook enrichment
2026-06-15 12:20:18 -04:00
5f6e75b14e [books] Fix comic book metadata importing
Some checks failed
build / test (push) Has been cancelled
2026-06-15 12:19:54 -04:00
a96a42cdbf [release] Bump to version 51.3
All checks were successful
build / test (push) Successful in 2m12s
deploy / test (push) Successful in 2m6s
deploy / build-and-deploy (push) Successful in 1m5s
- Improve speed of index and chart pages
2026-06-12 13:35:01 -04:00
c7f5d7d384 [charts] Add index to speed things up 2026-06-12 13:34:41 -04:00
d5830f5cd1 [release] Bump to version 51.2
All checks were successful
build / test (push) Successful in 2m14s
deploy / test (push) Successful in 2m15s
deploy / build-and-deploy (push) Successful in 1m27s
- Fix bug where last page of book gets separate scrobble
- Fix metadata scraping for books
2026-06-12 09:52:56 -04:00
c71b51fdb8 [books] Fix one-extra-scrobble bug on Koreader imports
Some checks failed
build / test (push) Has been cancelled
2026-06-12 09:52:33 -04:00
935d059a20 [books] Fix metadata scrapping
All checks were successful
build / test (push) Successful in 2m30s
It's not perfect, but at least we get covers now
2026-06-12 09:36:21 -04:00
25776eb495 [release] Bump to version 51.1
All checks were successful
build / test (push) Successful in 2m55s
deploy / test (push) Successful in 2m38s
deploy / build-and-deploy (push) Successful in 44s
- Fix scrobbling comic books
2026-06-11 19:07:00 -04:00
5ac4625af9 [books] Fix bug in scrobbling comic books
Some checks failed
build / test (push) Has been cancelled
2026-06-11 19:06:37 -04:00
a731427f6e [release] Bump to version 51.0
All checks were successful
build / test (push) Successful in 2m52s
deploy / test (push) Successful in 2m25s
deploy / build-and-deploy (push) Successful in 49s
- Fix koreader scrobble imports to use DST properly
- Fix book scrobbles where page_data is a list
- Lichess imports do not set default visbility
2026-06-11 18:39:31 -04:00
410da163fe [books] Fix koreader imports, maybe forever 2026-06-11 18:38:51 -04:00
a171192a6f [books] Fix list page_data for old book scrobbles
All checks were successful
build / test (push) Successful in 2m2s
2026-06-11 11:00:04 -04:00
c16b61db40 [importers] Fix strange bug where celery version mismatches
All checks were successful
build / test (push) Successful in 2m9s
2026-06-11 10:48:38 -04:00
29cb6a4991 [books] Really fix the subtitle bug 2026-06-11 10:48:21 -04:00
25c28e8335 [books] Fix bug in subtitle gen
All checks were successful
build / test (push) Successful in 2m5s
2026-06-11 10:34:20 -04:00
25626be3b6 [boardgames] Add visibility to lichess imports 2026-06-11 10:02:37 -04:00
0a880a2f2f [release] Bump to version 50.2
All checks were successful
build / test (push) Successful in 2m6s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Successful in 35s
- Koreader imports only import single-page scrobbles the next day
- Fix bugs in celery tasks causing imports to fail
2026-06-11 09:58:56 -04:00
248d3f2d3e [settings] Check webdav every two minuts
All checks were successful
build / test (push) Successful in 2m13s
2026-06-11 09:41:50 -04:00
e243fec679 [books] Try fixing the one-off import issue 2026-06-11 09:41:19 -04:00
de9b4ee9c1 [release] Bump to version 50.1
All checks were successful
build / test (push) Successful in 2m2s
deploy / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 32s
- Fix bug in charts where only #1 is displayed
2026-06-09 22:08:48 -04:00
bf9a6a9679 [charts] Fix only seeing first top media instance 2026-06-09 22:08:15 -04:00
709fed5cfe [release] Bump to version 50.0
All checks were successful
build / test (push) Successful in 2m7s
deploy / test (push) Successful in 2m6s
deploy / build-and-deploy (push) Successful in 31s
- Allow updating all a user's scrobble visibility at once
- Replace columsn of Top Artists, Tracks and Series with Maloja widget
2026-06-09 17:19:10 -04:00
b7df6299d0 [sharing] Add bulk scrobble share management 2026-06-09 17:18:51 -04:00
be16d513ef [charts] Add better chart views per Maloja
All checks were successful
build / test (push) Successful in 2m3s
2026-06-09 17:04:59 -04:00
15d27f6d94 [release] Bump to version 49.1
All checks were successful
build / test (push) Successful in 2m7s
deploy / test (push) Successful in 2m3s
deploy / build-and-deploy (push) Successful in 33s
- Fix bug with missing default visbility for scrobbles
2026-06-09 13:16:53 -04:00
c8292d1c06 [scrobbles] Back fill visibility field
Some checks failed
build / test (push) Has been cancelled
2026-06-09 13:15:57 -04:00
68f821fce1 [release] Bump to version 49.0
Some checks failed
build / test (push) Successful in 1m59s
deploy / test (push) Successful in 2m9s
deploy / build-and-deploy (push) Failing after 2m20s
- Fix broken tests with new sharing and add tests
2026-06-09 12:48:22 -04:00
ed2ed59f65 [scrobbles] Fix tests around visbility 2026-06-09 12:48:01 -04:00
17a7bb52fa [release] Bump to version 48.3
Some checks failed
build / test (push) Failing after 1m39s
deploy / test (push) Failing after 1m39s
deploy / build-and-deploy (push) Has been skipped
- Fix bug in missing sqids dep
2026-06-09 12:37:40 -04:00
bbac142b40 [deps] Add sqids
Some checks failed
build / test (push) Has been cancelled
2026-06-09 12:37:19 -04:00
5f55ec557f [release] Bump to version 48.2
Some checks failed
build / test (push) Failing after 1m21s
deploy / test (push) Failing after 1m18s
deploy / build-and-deploy (push) Has been skipped
- Lock down scrobbles and use sqids to share them
2026-06-09 12:33:50 -04:00
7f3076608f [scrobbles] Add sharing of scrobbles
Some checks failed
build / test (push) Has been cancelled
2026-06-09 12:33:25 -04:00
568772a0e6 [release] Bump to version 48.1
All checks were successful
deploy / test (push) Successful in 2m0s
build / test (push) Successful in 1m57s
deploy / build-and-deploy (push) Successful in 35s
- Generate a report of tracks with mistmatched metadata
- Date parsing failing in eBird imports
2026-06-08 11:18:08 -04:00
91c3376256 [music] Add mgmt command to see mismatched metadata
Some checks failed
build / test (push) Has been cancelled
2026-06-08 11:17:36 -04:00
58639c6fc1 [importers] Add error_log to surface import errors
All checks were successful
build / test (push) Successful in 2m2s
2026-06-08 10:58:02 -04:00
228441ddc5 [importers] Fix parsing of dates in ebird files 2026-06-08 10:52:50 -04:00
6341075f07 [agent] Add some notes about how we track tasks 2026-06-08 10:52:32 -04:00
a135b9f5f2 [project] Update format for PROJECT file
All checks were successful
build / test (push) Successful in 1m58s
2026-06-07 11:10:30 -04:00
9088412d1e [release] Bump to version 48.0
All checks were successful
build / test (push) Successful in 2m1s
deploy / test (push) Successful in 2m2s
deploy / build-and-deploy (push) Successful in 33s
- Show team or player images on sport detail and scrobble detail
- Add fix_metadta method to Video instances
2026-06-07 11:06:58 -04:00
c7339fbe31 [templates] Clean up how str and subtitles work 2026-06-07 11:06:30 -04:00
4ce3dc03c5 [videos] Add fix_metadata for videos 2026-06-07 10:13:55 -04:00
5a4ef678a8 [release] Bump to version 47.2
All checks were successful
build / test (push) Successful in 1m58s
deploy / test (push) Successful in 1m58s
deploy / build-and-deploy (push) Successful in 53s
- Add OMDB source as backup when TMDB returns nothing
2026-06-07 09:59:05 -04:00
5ca22efeaa [videos] Fix full metadata from OMDB
We were missing the series and episode number info.
2026-06-07 09:58:28 -04:00
912ea8bfac [videos] Add OMDB enrichment when TMDb fails 2026-06-07 09:47:52 -04:00
b541e1084d [release] Bump to version 47.1
All checks were successful
build / test (push) Successful in 2m0s
deploy / test (push) Successful in 2m1s
deploy / build-and-deploy (push) Successful in 32s
- Untangle the sports migrations errors
2026-06-06 23:47:25 -04:00
c9b9da4abc [sports] Fix migrations 2026-06-06 23:47:10 -04:00
8236f43026 [release] Bump to version 47.0
Some checks failed
build / test (push) Successful in 2m4s
deploy / test (push) Successful in 2m7s
deploy / build-and-deploy (push) Failing after 46s
- Change sports scrobbling a bit
2026-06-06 23:35:30 -04:00
ea1b43d1b8 [sports] Big sports structure revamp
Some checks failed
build / test (push) Has been cancelled
This should make scrobbling sports more like tasks.

The root scrobbled items are a little more generic, but
it's easier to see viewing patterns.
2026-06-06 23:32:21 -04:00
4bf22c96e9 [release] Bump to version 46.0
All checks were successful
build / test (push) Successful in 1m57s
deploy / test (push) Successful in 2m9s
deploy / build-and-deploy (push) Successful in 43s
- Add sentiment parsing for Scrobbles with notes
2026-06-05 19:42:21 -04:00
dec7a79509 [scrobbles] Add basic sentiment analysis
All checks were successful
build / test (push) Successful in 2m4s
2026-06-05 19:35:45 -04:00
371e1d654c [tooling] Release in one step
All checks were successful
build / test (push) Successful in 1m57s
2026-06-05 14:58:35 -04:00
111 changed files with 5167 additions and 738 deletions

View File

@ -14,3 +14,11 @@ ro class method should call the utility function.
Be sure to check pyproject.toml for project defaults. Specifically for black and
isort expectations.
All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
In local development, environment variables for various sensitive values live in a .envrc file
The .envrc file can be loaded into a shell environment to allow access to most third party services
Care should be taken when using .envrc that we do not spam services we use in production with requests

View File

@ -2,6 +2,7 @@
We should convert this PROJECT file to put tickets in a subdirectory, tickets, with each ticket having it's own shortid_title.org
* Overview
Vrobbler began humbly enough as a way to use Jellyfin's webhook to keep track of
the shows and movies I was watching. More specifically, I broke my ankle a few
days after Christmas in 2022 and spent the next four months very slowly
@ -16,6 +17,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
to be sometimes cathartic and sometimes functional as I try to remember when I
did a thing.
* Features
** Beer
*** Triggers
@ -85,20 +87,9 @@ fetching and simple saving.
**** Bookmarklet
*** Metadata sources
**** Scraper
* Chores
** DONE Document various vrobbler features :chore:personal:project:vrobbler:documentation:
:PROPERTIES:
:ID: 514e9285-96f1-265f-56df-118c12f60918
:END:
:LOGBOOK:
CLOCK: [2025-07-09 Wed 09:55]--[2025-07-09 Wed 10:15] => 0:20
:END:
* Backlog [0/13] :vrobbler:project:personal:
** TODO [#C] Add sentiment parsing for Scrobbles with notes :vrobbler:project:scrobbles:sentiment:
:PROPERTIES:
:ID: 37781d6a-f3b0-48b2-bf98-33c2c791cf85
:END:
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
* Backlog [0/20] :vrobbler:project:personal:
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :bug:music:scrobbles:
:PROPERTIES:
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
:END:
@ -396,11 +387,11 @@ CLOCK: [2025-07-09 Wed 09:55]--[2025-07-09 Wed 10:15] => 0:20
}
}
#+end_src
** TODO [#C] User should be able to enable auto trail tracking via amail reader with Garmin LiveTrack URLs :vrobbler:trails:project:feature:personal:
** TODO [#C] Allow auto trail tracking via email with Garmin LiveTrack URLs :trails:feature:
:PROPERTIES:
:ID: 133bcf71-078f-4efa-a029-1eae4b4d146d
:END:
** TODO [#C] Fix exporting so it works reliably :exporting:project:feature:
** TODO [#C] Fix exporting so it works reliably :exporting:feature:
*** Description
@ -414,8 +405,7 @@ placed in the media directory:
And this should all be done in a celery task that is just kicked off by the
"Export" button on the frontend
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
** TODO [#B] Add AllTrails as a source for Trail data :trails:feature:
:PROPERTIES:
:ID: 39313362-cdfe-46e7-bbd4-9139a65c0b3c
:END:
@ -425,7 +415,7 @@ Pretty clear, I would love to make trails more useful. Historically I wasn't
hiking a lot, which made the source for this a bit silly. But it's clear that
AllTrails is the best source, though having TrailForks is nice to.
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:personal:project:
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:
:PROPERTIES:
:ID: 5a4fb0f8-0555-40ec-b06f-93c26bd686f4
:END:
@ -449,18 +439,278 @@ added.
They should also probably support markdown formatting and that should be
displayed in the template.
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :books:feature:export:
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :vrobbler:books:feature:comicbook:personal:project:
** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
:PROPERTIES:
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
:END:
- Note taken on [2025-09-25 Thu 10:51]
*** Description
Currently we have webdav able to import post types of file-based incoming data,
usually in the form of CSVs but also gpx files, bgstats json files, and
audioscrobbler TSV files.
What if the user could specify via their profile (settings) which imports they
wanted to use IMAP for and which ones they wanted to use WebDAV for.
Then we'd have two celery tasks that would be kicked off periodically via
celerybeat, one for IMAP imports every 12 minutes and one for WebDAV every 3
minutes. Both would be responsible for checking if a user has an configured
imports of their type, check if an import needs to run, and dispatch the
needed import celery task. This is how the WebDAV celery task currently works.
This would also be an opporunity to clean up the code around WebDAV imports
and make them more re-usable for other import services.
** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
:PROPERTIES:
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
:END:
*** Description
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
this put into it and then we stop trying to pull the artist apart when we run into it.
** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
:PROPERTIES:
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
:END:
*** Description
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
In most cases, we should be able to trust this data to created music tracks or videos rather
than going to third-party services to enrich. Thus, for tracks and videos we should search in
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
enrich further.
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
log a warning and move on.
** TODO [#B] Allow browing a user's favorited media :favorites:feature:
:PROPERTIES:
:ID: 5c2cf004-d01f-4576-9bbb-974235e7408a
:END:
*** Description
We should have a global view `/favorites/` that shows the logged in users's
favorited media objects.
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
** TODO [#B] Find page numbers for comic books from ComicVine :feature:books:
:PROPERTIES:
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
:END:
** TODO [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
** TODO [#C] Implement loguru into project :feature:loguru:logging:
:PROPERTIES:
:ID: efcd0c0a-db81-4518-9c23-5505d59e8ef5
:END:
*** Description
Would be great to formalize how we log so we can search for errors and such more
easily. And our exposure to PII is really low at this point in the project,
so we can probably use backtrace=True and diagnose=True to help us root cause
bugs faster.
** TODO [#B] Add a /trends/ page that shows trends based on scrobble data :feature:trends:scrobbles:
*** Description
This project is a bit invovled. But we should add a top level URL `trends` that shows
various trends as defined either in a static settings file, or dynamically via a database table.
Examples of trends:
- How often does the user:
+ watch sports while doing a task?
+ do a task while watching a video?
* how often do I do
- trail_scrobble__average_heartrate per trail
- ...
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :books:feature:comicbook:
:PROPERTIES:
:ID: b3cc57ca-3d2c-468d-ab7c-c47f1120309b
:END:
*** Description
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
** TODO [#C] Make podcast date format configurable in settings :podcasts:configuration:
:PROPERTIES:
:ID: b01a94f8-328f-41ed-a62e-8b99c755b82d
:END:
*** Description
=PODCAST_DATE_FORMAT= is hardcoded to ="YYYY-MM-DD"=. Should be in Django settings or environment variables for deploy-specific configuration.
File: ~vrobbler/apps/podcasts/utils.py~ (line 13)
** TODO [#C] Extract zombie scrobble query into custom manager :refactoring:manager:
:PROPERTIES:
:ID: 79c874e1-ca6f-4bce-9259-e3eebdca8a41
:END:
*** Description
The zombie scrobble cleanup query lives in a utility function. Should be a
custom model manager method (e.g. =Scrobble.objects.zombies()=).
File: ~vrobbler/apps/scrobbles/utils.py~ (line 204)
** TODO [#C] Allow profile to set start of week :profiles:configuration:
:PROPERTIES:
:ID: 0449279a-9550-430e-be0c-816df7273080
:END:
*** Description
=start_of_week()= and =end_of_week()= use Monday as default. Should be a user
profile setting for different cultural week start conventions.
File: ~vrobbler/apps/profiles/utils.py~ (lines 39, 44)
** TODO [#C] Add constants for data dictionary keys (multiple files) :refactoring:constants:
:PROPERTIES:
:ID: d4415f9b-620a-4be7-925d-fa71c02ba1d1
:END:
*** Description
Multiple files use magic string literals for dict keys. Should be extracted to
named constants for maintainability.
- Files:
- ~vrobbler/apps/locations/models.py~ (line 63) -- ="lat"=, ="lon"= etc.
- ~vrobbler/apps/webpages/models.py~ (line 290) -- ="url"=
- ~vrobbler/apps/scrobbles/importers/tsv.py~ (line 55) -- ="S"= completion status
** TODO [#C] Show time per scrobble in long play lists and total time playing :templates:longplay:scrobbles:
* Version 52.0 [5/5]
** DONE [#B] Allow marking media as long play complete from detail page :templates:scrobbles:longplay:
:PROPERTIES:
:ID: 2c314768-be97-9b10-d13c-9cfd0f38a64e
:END:
** DONE [#A] Fix how long play scrobbles are tracked :scrobbles:longplay:serial:
:PROPERTIES:
:ID: 908b0493-cabf-40c1-825f-cd59a8ad0f7a
:END:
*** Description
Currently we have this idea of "long_play" scrobbles but there's a lot missing
to tie it together.
What we'd prefer is that when a new scrobble is added for a media_type that
`is_long_play` the most recent scrobble finished is added as the
`last_serial_scrobble` to the log data. But all the other long play stuff exsits
as data model fields. We should add `long_play_last_scrobble` as a FK to this
scrobble when creating a new longplay scrobble.
Additionally, `long_play_seconds` we should have a recompute management command
to walk backward from `long_play_last_scrobble` until a `long_play_complete`
scrobble is found (exclusive) and save the time.
We should also ony use `long_play_complete` field on the scrobble ... some
logdatas have a similar field, but we should make sure that we always use the
model field to determine if a long play is finished.
This should include a command to clean up long play data to consolidate around
the `long_play_complete` field.
** DONE [#B] Paginate or limite scrobbles on media admin pages :admin:scrobbles:media:
:PROPERTIES:
:ID: f02e487b-d7ed-4834-838a-303560f2ad3b
:END:
** DONE [#B] Clean up books admin :admin:books:bug:
:PROPERTIES:
:ID: 7539bee6-0a52-26f6-ebc6-5554ac49a716
:END:
** DONE [#B] Clean up favorites admin :admin:favorites:scrobbles:
:PROPERTIES:
:ID: f2be0c69-1bf8-b5a3-5269-9c8ea873361d
:END:
*** Description
Some FK lookups in admin should be raw_id_fields.
* Version 51.4 [1/1]
** DONE [#A] Clean up metadata comicbook enrichment :bug:comics:books:metadata:
:PROPERTIES:
:ID: cd875450-7117-78ca-8be4-9c8b73037dba
:END:
*** Description
Still getting wonky results with some comicbooks. Would be nice to be able to
tag a Book as a comicbook, and also gather volume information. I also noticed
that some books that are found in OL never get their comicvine_id populated. We
should make sure we always have comicvine_ids if available.
* Version 51.3 [1/1]
** DONE [#A] Improve speed of index and chart pages :bug:scrobbles:perf:
:PROPERTIES:
:ID: 031a23f8-7c02-4926-9884-6654ceca16c2
:END:
*** Description
Over the last few releases, the home page and charts pages have gotten really
slow.
We should look into what's causing the slowness and maybe do more agressive
query optimization or caching.
* Version 51.2 [2/2]
** DONE [#A] Fix bug where last page of book gets separate scrobble :bug:books:importers:koreader:
:PROPERTIES:
:ID: e13e0b4c-461e-e5a9-c685-b972f4e262e5
:END:
*** Description
The new KoReader code is working great to import books with the correct timezone
and what-not. But it has a weird artifact of creating one extra scrobble for the
last page read. Need to button that up.
** DONE [#B] Fix metadata scraping for books :books:metadata:
:PROPERTIES:
:ID: ea416a69-a8a8-4d05-b7d4-0a3470820e34
:END:
* Version 51.1 [1/1]
** DONE [#A] Fix scrobbling comic books :books:scrobbles:bug:
:PROPERTIES:
:ID: 8dfbff19-3fa4-f3b8-21c7-7a416498000c
:END:
*** Description
At some point logdata and log got confused, and now when you try
to scrobble a comic book, it just throws errors. We should look
into where the confusion happened and fix it.
* Version 51.0 [3/3]
** DONE [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
@ -486,28 +736,409 @@ whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** TODO [#A] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
** DONE [#A] Fix book scrobbles where page_data is a list :bug:books:scrobbles:
:PROPERTIES:
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
:ID: 35b323fa-ccc0-4009-b227-8a0f12bbd469
:END:
*** Description
Currently we have webdav able to import post types of file-based incoming data,
usually in the form of CSVs but also gpx files, bgstats json files, and
audioscrobbler TSV files.
Comic scrobbling is currently kind of janky. Most of the problems boil down to them
storing saved page data in a list of dicts rather than a dict keyed off of the page number that
was read.
What if the user could specify via their profile (settings) which imports they
wanted to use IMAP for and which ones they wanted to use WebDAV for.
We need to adjust comic scrobbling to use a dict of pages keyed off the page number, and also
write a migration script that runs as a data migration to update any book scrobbles that may
have page_data as a list.
Then we'd have two celery tasks that would be kicked off periodically via
celerybeat, one for IMAP imports every 12 minutes and one for WebDAV every 3
minutes. Both would be responsible for checking if a user has an configured
imports of their type, check if an import needs to run, and dispatch the
needed import celery task. This is how the WebDAV celery task currently works.
*** Example data
#+begin_src python
{"notes": null, "page_end": 27, "page_data": [{"notes": null, "end_ts": 1771815895, "duration": 14, "start_ts": 1771815881, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815908, "duration": 13, "start_ts": 1771815895, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815913, "duration": 5, "start_ts": 1771815908, "description": null, "page_number": "2"}, {"notes": null, "end_ts": 1771815933, "duration": 20, "start_ts": 1771815913, "description": null, "page_number": "3"}, {"notes": null, "end_ts": 1771815945, "duration": 12, "start_ts": 1771815933, "description": null, "page_number": "4"}, {"notes": null, "end_ts": 1771815983, "duration": 38, "start_ts": 1771815945, "description": null, "page_number": "5"}, {"notes": null, "end_ts": 1771816007, "duration": 24, "start_ts": 1771815983, "description": null, "page_number": "6"}, {"notes": null, "end_ts": 1771816011, "duration": 4, "start_ts": 1771816007, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816013, "duration": 2, "start_ts": 1771816011, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816052, "duration": 39, "start_ts": 1771816013, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816127, "duration": 75, "start_ts": 1771816052, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816134, "duration": 7, "start_ts": 1771816127, "description": null, "page_number": "9"}, {"notes": null, "end_ts": 1771816196, "duration": 62, "start_ts": 1771816134, "description": null, "page_number": "10"}, {"notes": null, "end_ts": 1771816262, "duration": 66, "start_ts": 1771816196, "description": null, "page_number": "11"}, {"notes": null, "end_ts": 1771816293, "duration": 31, "start_ts": 1771816262, "description": null, "page_number": "12"}, {"notes": null, "end_ts": 1771816322, "duration": 29, "start_ts": 1771816293, "description": null, "page_number": "13"}, {"notes": null, "end_ts": 1771816330, "duration": 8, "start_ts": 1771816322, "description": null, "page_number": "14"}, {"notes": null, "end_ts": 1771816368, "duration": 38, "start_ts": 1771816330, "description": null, "page_number": "15"}, {"notes": null, "end_ts": 1771816388, "duration": 20, "start_ts": 1771816368, "description": null, "page_number": "16"}, {"notes": null, "end_ts": 1771816482, "duration": 94, "start_ts": 1771816388, "description": null, "page_number": "17"}, {"notes": null, "end_ts": 1771816550, "duration": 68, "start_ts": 1771816482, "description": null, "page_number": "18"}, {"notes": null, "end_ts": 1771816567, "duration": 17, "start_ts": 1771816550, "description": null, "page_number": "19"}, {"notes": null, "end_ts": 1771816586, "duration": 19, "start_ts": 1771816567, "description": null, "page_number": "20"}, {"notes": null, "end_ts": 1771816597, "duration": 11, "start_ts": 1771816586, "description": null, "page_number": "21"}, {"notes": null, "end_ts": 1771816616, "duration": 19, "start_ts": 1771816597, "description": null, "page_number": "22"}, {"notes": null, "end_ts": 1771816640, "duration": 24, "start_ts": 1771816616, "description": null, "page_number": "23"}, {"notes": null, "end_ts": 1771816690, "duration": 50, "start_ts": 1771816640, "description": null, "page_number": "24"}, {"notes": null, "end_ts": 1771816702, "duration": 12, "start_ts": 1771816690, "description": null, "page_number": "25"}, {"notes": null, "end_ts": 1771816823, "duration": 121, "start_ts": 1771816702, "description": null, "page_number": "26"}, {"notes": null, "end_ts": null, "duration": null, "start_ts": 1771816823, "description": null, "page_number": "27"}], "page_start": 1, "pages_read": 27, "resume_url": null, "description": null, "koreader_hash": null, "long_play_complete": false}
#+end_src
** DONE [#A] Lichess imports do not set default visbility :boardgames:bug:importers:lichess:
:PROPERTIES:
:ID: a78f7c72-a20a-8db2-cde0-d92a731d4fba
:END:
* Version 50.2 [2/2]
** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers:
:PROPERTIES:
:ID: b50141fd-cda6-4a3a-afd3-cd8499e7523e
:END:
*** Description
When you read a single page in a book in Koreader and try to import it, the scrobble is only
created the day after, not on the day of the reading.
** DONE [#A] Fix bugs in celery tasks causing imports to fail :bug:celery:tasks:
:PROPERTIES:
:ID: d1171cb0-6413-44b8-a68a-019a4d2fb285
:END:
*** Description
Seems like all celery tasks are failing for different reasons except the chart
updates.
*** Errors
**** scrobbles.tasks.send_notification_for_in_progress
#+begin_src bash
KeyError: 'track'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedColumn: column music_track.artist_id does not exist
LINE 1: ..."."title", "music_track"."base_run_time_seconds", "music_tra...
^
HINT: Perhaps you meant to reference the column "music_track.artist_fk_id".
#+end_src
**** scrobbles.tasks.import_from_webdav_all_users
#+begin_src bash
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/importers/webdav.py", line 166, in scan_webdav_for_koreader
if last_import and last_import.webdav_etag and remote_etag:
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'KoReaderImport' object has no attribute 'webdav_etag'
#+end_src
**** scrobbles.tasks.process_bgstats_import
#+begin_src bash
Traceback (most recent call last):
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.NotNullViolation: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
DETAIL: Failing row contains (374463, 2026-06-11 13:27:06.528319+00, 2026-06-11 13:27:06.52834+00, 2026-06-11 13:17:34+00, 180, f, f, BG Stats, 1, null, t, {"players": [{"new": false, "win": false, "rank": 0, "role": "",..., null, null, null, 8e73ceec-b731-4623-9637-712bbf9f76ce, null, null, null, null, , null, BoardGame, 324, null, null, America/New_York, null, null, , null, null, , null, null, null, null, null, null, null, null, null, null, null).
#+end_src
* Version 50.1 [1/1]
** DONE [#B] Fix bug in charts where only #1 is displayed :charts:templates:
:PROPERTIES:
:ID: 7136dffb-e6b7-184b-48ac-bb09bae0b0f0
:END:
* Version 50.0 [2/2]
** DONE [#A] Allow updating all a user's scrobble visibility at once :scrobbles:sharing:feature:
:PROPERTIES:
:ID: 9ed2ec65-bf69-4300-965c-6a7d3ef7ea03
:END:
*** Description
We now have the ability to share or unshare scrobbles and create private links.
We should add a toggle in the user's settings that will bulk make all their
scrobbles public or private, so that a user can either share everything, or lock
their account down.
This should not affect scrobbles that are in the "Shared" visibility state.
And users should be able to also control whether all scrobbles of a specific
type are shared or not. Maybe this could be a JSONField in profile that contains
a media_type key with a visibility type for a value, and if it's not present,
sharing defaults to private?
Additionally, users's should have links in their settings to see what scrobbles
are either public, shared or private. Probably this could be done with a
?visbility=<> filter on the /scrobbles/ page.
*** Changes
- Added `media_type_visibility` JSONField to UserProfile (migration 0038)
- Created `BulkVisibilityView` at `/settings/visibility/` with:
- Radio toggle to make all non-shared scrobbles Public or Private
- Per-media-type dropdown for each of the 20 media types (inherit/public/shared/private)
- Created `BulkVisibilityForm` with dynamic media_type fields
- Created `profiles/visibility_settings.html` template with visibility stats + filter links
- Added link from main settings page to visibility settings
- Added `?visibility=` filter support to `ScrobbleListView` (public/shared/private)
- Added filter indicator to `scrobble_all_list.html`
- Updated `Scrobble.create()` to check `user.profile.media_type_visibility` for media-type-specific defaults before falling back to PRIVATE
** DONE [#A] Replace columsn of Top Artists, Tracks and Series with Maloja widget :templates:charts:
:PROPERTIES:
:ID: 3946afb1-932c-46fe-a188-f4c9add1a491
:END:
*** Description
The tables are fine, but Maloja widgets are better. We should drop the top track table, add top albums
and replace top artists and top tv series with the Maloja style widgets.
* Version 49.1 [1/1]
** DONE [#A] Fix bug with missing default visbility for scrobbles :bug:scrobbles:sharing:
:PROPERTIES:
:ID: 20843992-6453-9a9a-cde6-2c2b6677db23
:END:
*** Description
We can't scrobble anything now because visbility is not null, but has no default
value.
*** Notes
- Note taken on [2026-06-09 Tue 13:14]
The full stack trace:
#+begin_src sh
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 1430, in create_or_update
elif "log" in scrobble_data.keys() and scrobble.log:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 1583, in create
)
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
obj.save(force_insert=True, using=self.db)
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 870, in save
if self.media_obj:
^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_extensions/db/models.py", line 22, in save
super().save(**kwargs)
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
self.save_base(
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
updated = self._save_table(
^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
results = self._do_insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
return manager._insert(
^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
django.db.utils.IntegrityError: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
DETAIL: Failing row contains (373813, 2026-06-09 17:13:38.11355+00, 2026-06-09 17:13:38.113566+00, 2026-06-09 17:13:36+00, 0, f, f, Todoist, 1, null, t, {"title": "Animal chores", "labels": ["chore", "farm"], "todoist..., null, null, null, 68680dbf-f9a9-476c-b1c7-adbd231bbab6, null, null, null, null, , null, Task, null, null, null, America/New_York, null, null, , null, null, , 72, null, null, null, null, null, null, null, null, null, null).
#+end_src
* Version 49.0 [1/1]
** DONE [#A] Fix broken tests with new sharing and add tests :scrobbles:sharing:tests:
:PROPERTIES:
:ID: 10ecd169-eaee-8554-d4ee-f1d34bfad99f
:END:
* Version 48.3 [1/1]
** DONE [#A] Fix bug in missing sqids dep :dependencies:project:
:PROPERTIES:
:ID: 0b619837-729a-cd74-7a97-6fa2a148b27d
:END:
* Version 48.2 [1/1]
** DONE [#A] Lock down scrobbles and use sqids to share them :feature:sharing:scrobbles:
:PROPERTIES:
:ID: a6e869f7-8012-7e83-8f68-d0a0ed4c3c6a
:END:
*** Description
Currently all scrobbles are public. Anyone with the uuid can view any other
scrobbles. We should use SQIDs to allow shareable links to scrobbles and then
make all scrobbles hidden by default.
* Version 48.1 [2/2]
** DONE [#A] Generate a report of tracks with mistmatched metadata :music:tracks:metadata:
:PROPERTIES:
:ID: 684b8cd2-a3c1-4995-ba9e-7abdb02c37f2
:END:
*** Description
We should have a management command that outputs a CSV file of track IDs where
the log["raw_data"]["Artist"] (for Jellyfin) or log["raw_data"]["artist"]
(mopidy) value does not match the Track.artists names.
And we should see the same thing for albums (log["raw_data"]["Album"] or
log["raw_data]["album"]).
It should output the fields "track_id", "track_artist_name", "track_album_name",
"raw_artist", "raw_album", "source"
Where source is either Jellyfin or Mopidy based on the keys.
Put the file /tmp/metadata-report.csv by default and overwrite exsiting reports.
The command should also accept a file-path to overide this default.
** DONE [#A] Date parsing failing in eBird imports :birds:ebird:importers:
:PROPERTIES:
:ID: 72e0254f-39d8-4843-9857-623e0362d77e
:END:
*** Description
On line 45 in the apps/birds/importer.py file, the import is thorowing this
error:
#+begin_src python
ValueError: time data 'Jun 7, 2026, 5:15 PM' does not match format '%B %d, %Y %I:%M %p'
#+end_src
Historically other files starting on May 24 worked, so I suspect this is a
problem of the date formatter expecting Long month names and zero-padded days
and the only sample we had was a three-letter month (May) and days with two
digites(24 through the 31)
We should also add a "error_log" to the importers so that errors tha occur are
surfaced, even when 0 successful files were processed. And we should make sure
all importers do this as well.
* Version 48.0 [2/2]
** DONE [#B] Show team or player images on sport detail and scrobble detail :sports:templates:
:PROPERTIES:
:ID: 68c17383-ee6e-4b5f-b3f5-1b637a0a3ea8
:END:
*** Description
On the sport event detail page, we should show the images of the teams or
players invovled.
Also, those images for the sport event should be shown on the scrobble detail
page for sport event scrobble details.
** DONE [#B] Add fix_metadta method to Video instances :videos:metadata:
:PROPERTIES:
:ID: 9df5404d-1b60-4eee-b7cf-1f7e6dfade65
:END:
*** Description
Turns out we don't have a fix_metadata method for videos. We should add that using
the basic logic from find_or_create on the Video model.
* Version 47.2 [1/1]
** DONE [#B] Add OMDB source as backup when TMDB returns nothing :videos:metadata:imdb:
:PROPERTIES:
:ID: 20195445-7fdd-49be-9767-103b12da0bfb
:END:
*** Description
TMDb works great for most cases. There are some edge cases, though where it does
not import videos, when TV shows are split up differently in TMDb than in IMDB.
One example I stumbled on is the 2020 reboot of Animaniacs. TMDb splits the
epiodes up in three parts, while they were always broadcast three-in-one, and
that's how IMDB lists them. Thus, the IMDB ID means nothing, and the videos end
up unenriched.
* Version 47.1 [1/1]
** DONE [#A] Untangle the sports migrations errors :sports:bug:migrations:
:PROPERTIES:
:ID: 4d50ca2e-f45b-dde8-e3c9-cd84f353b349
:END:
* Version 47.0 [1/1]
** DONE [#B] Change sports scrobbling a bit :feature:sports:scrobbles:
:PROPERTIES:
:ID: cd27d683-c847-4251-b3d1-8243f45c01ca
:END:
*** Description
Currently, the way we scrobble sports means that basically the same event will
never be scrobbled again. I will likely never watch the 2025 Monaco Grand Prix
again, but I will watch the Monaco Grand Prix again. But I also wont watch one
specific game between Arsenal and Man City twice, but I may watch those two
teams play multiple times.
What if instead of scrobbling a specific sports event on a specific date, we
make the unique Scrobblable item the players or teams in the event?
That would not work for races where the unique item would have to be the name of
the event.
Maybe that means SportEvent is too generic, and we'd need the event type to be
scrobble items.
A race, the Indy 500 or Coke 600, or Boston Marathon would be scrobblable, while
for games, the teams would be unique, so a game between Arsenal and Man City
would be unique (with extra logdata context for who's home and who's away, and
the location, could even have the weather per scrobble).
And finally, for Tennis, the title would be the round of the event, Roland
Garros Women's Semifinal, US Open Men's Final, Miami Invitational Round of 32,
with two players, or two teams and a start datetime, which is similar to what we
have now. The round becomes not a foreign key, but just a string, and we'd need
a FK to an organizer field which would replace league, and would be like "ATP
Tour" or "PGA Tour". Season would also need to be a string, and would be
something like: 2026 or 2024-2025.
Examples:
- Super Bowl
- Sochaux v Concarneau
- French Open Final
- Carlos Alcaraz v Jannik Sinner
We'd also want a script to reorganize existing sports events and move scrobbles
to the right place as best as we're able, and to flag sportsevents and scrobbles
that could not automatically be migrated with a unique tag like
"migration-failed"
Ultimately I think what we need is to greatly simplify the SportEvent to be just a placeholder
for a sport event type for a given league, then each scrobble holds the details of teams, players
start, thesportsdb_id, round and season.
Thus, I've already simplified that model, but what we need is a migration script that will move
existing complex SportEvent instances into very basic ones, and updating any scrobbles for those
events with a new SportEventLogData structure with all the specific details in it. We also need to
move the obj.round.season.league into the FK for the given event.
* Version 46.0 [1/1]
** DONE [#C] Add sentiment parsing for Scrobbles with notes :scrobbles:sentiment:
:PROPERTIES:
:ID: 37781d6a-f3b0-48b2-bf98-33c2c791cf85
:END:
*** Description
Not sure how useful this would be, but I wonder if we can add a `sentiment` JSONField
on each scrobble that can store the output of VADER over the notes in a scrobble with notes.
I'm not sure that the value prop here is worth the storage and processing time.
But if we do add it, it should be a process that scans for scrobbles with both notes and no
sentiment field value (unless --overwrite is used) and just run periodically.
This would also be an opporunity to clean up the code around WebDAV imports
and make them more re-usable for other import services.
* Version 45.1 [1/1]
** DONE [#B] Mopidy favorites or monthly playlist adds should look at all scrobbles :bug:mopidy:favorites:tracks:
@ -849,6 +1480,7 @@ Format should be: "%Y-%m-%dT%H:%M:%S.%fZ"
:ID: 68a011b2-bb6f-3ba8-2312-5947c41db9ac
:END:
* Version 39.0 [3/3]
** DONE [#B] Clean up org-mode tasks metadata :bug:tasks:metadata:
:PROPERTIES:
@ -903,6 +1535,7 @@ If the task is completed, don't touch it.
:ID: 63dc633c-4382-e6a5-e663-b01871ce86ce
:END:
* Version 38.0 [38/38]
** DONE [#A] Fix release flow to be easier to trigger :pyproject:release:tooling:
:PROPERTIES:
@ -1294,6 +1927,7 @@ urllib.error.HTTPError: HTTP Error 500: Internal Server Error
This may have already been resolved ... need to just confirm it.
* Version 37.0 [4/4]
** DONE [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
:PROPERTIES:
@ -1312,11 +1946,13 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
:PROPERTIES:
:ID: e3e49a9a-67d2-8ad8-1114-6f05effee9b7
:END:
* Version 36.0 [1/1]
** DONE [#A] Refactor how videos are scrobbled :vrobbler:vidoes:feature:personal:project:
:PROPERTIES:
:ID: 6034a11d-5376-994d-9a4b-e1640e258cfa
:END:
* Version 35.0 [3/3]
** DONE [#B] Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
:PROPERTIES:
@ -1330,6 +1966,7 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
:PROPERTIES:
:ID: d1ba1ca1-509b-13a9-1307-b2dc94a2eafe
:END:
* Version 34.0 [4/4]
** DONE [#A] Use bgg-api for BoardGameGeek lookups :vrobbler:feature:boardgames:personal:project:
:PROPERTIES:
@ -1389,6 +2026,7 @@ TypeError: can only concatenate str (not "NoneType") to str
A good lesson in using constants for things.
* Version 33.0 [3/3]
** DONE [#A] Fix bug where scrobble is_stale only uses seconds not total_seconds :vrobbler:bug:scrobbles:personal:project:
:PROPERTIES:
@ -1403,6 +2041,7 @@ TypeError: can only concatenate str (not "NoneType") to str
:ID: 4955cc34-0882-50db-92f7-f36a95bf57a4
:END:
<2025-10-28 Tue>
* Version 32.0 [2/2]
** DONE [#B] Save path to reading source on book scrobbles and show it on the detail page :vrobbler:feature:books:personal:project:
:PROPERTIES:
@ -1412,6 +2051,7 @@ TypeError: can only concatenate str (not "NoneType") to str
:PROPERTIES:
:ID: 9fe09567-11a3-7083-53c7-07458a9591d0
:END:
* Version 31.0 [3/3]
** DONE [#A] Stop comic book webpage scrobbles from overwriting old scrobbles :vrobbler:personal:bug:books:scrobbling:
:PROPERTIES:
@ -1425,6 +2065,7 @@ TypeError: can only concatenate str (not "NoneType") to str
:PROPERTIES:
:ID: 9a870c05-6d20-0803-d35d-c03fbe1d0ee1
:END:
* Version 30.0 [3/3]
** DONE [#A] Fix readcomicsonline browsing to update pages :vrobbler:books:feature:comicbook:personal:project:scrobbling:
:PROPERTIES:
@ -1439,10 +2080,13 @@ TypeError: can only concatenate str (not "NoneType") to str
:PROPERTIES:
:ID: d22cec3f-117f-f203-33a5-efbefa8a5cee
:END:
* Version 29.0 [1/1]
** DONE HOTFIX podcast lookups, final
* Version 28.0 [1/1]
** DONE HOTFIX podcast lookups
* Version 27.0 [3/3]
** DONE [#A] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
:PROPERTIES:
@ -1483,6 +2127,7 @@ it's annoying.
actually be much more reliable than the current state of the podcast lookup
which depends on the file to be name properly.
* Version 26.0 [3/3]
** DONE Clean up templates for scrobble details :vrobbler:personal:bug:templates:
:PROPERTIES:
@ -1496,6 +2141,7 @@ it's annoying.
:PROPERTIES:
:ID: c03a38ce-b337-f4fa-adba-aee08d4329f5
:END:
* Version 25.0 [3/3]
** DONE Add basic food templates and fix urls :food:vrobbler:personal:project:bug:urls:
:PROPERTIES:
@ -1509,6 +2155,7 @@ it's annoying.
:PROPERTIES:
:ID: 7debfbaf-cdd8-f49b-57ff-804bfe7c9236
:END:
* Version 24.0 [2/2]
** DONE Clean up logdata for various media :personal:feature:project:vrobbler:logdata:
:PROPERTIES:
@ -1519,6 +2166,7 @@ it's annoying.
:ID: 1a1c0aa6-0313-c8be-1676-5d6adddef0a4
:END:
* Version 23.0 [3/3]
** DONE Add dynamic forms for LogData classes :personal:feature:vrobbler:project:forms:logdata:
:PROPERTIES:
@ -1529,16 +2177,19 @@ it's annoying.
:PROPERTIES:
:ID: 99f6bd77-dc8f-6ed1-0321-32a52c944264
:END:
* Version 19.0 [1/1]
** DONE Add periodic check for mood :vrobbler:feature:moods:personal:project:
:PROPERTIES:
:ID: 55404488-c69f-0dd5-838e-1d1e15c873eb
:END:
* Version 18.7 [1/1]
** DONE Use the timezone history log to fix old Scrobbles that fall into those timezone blocks :vrobbler:chore:scrobbles:project:personal:
:PROPERTIES:
:ID: 9d055ac1-584b-20c8-7ad9-9ce36b329dc7
:END:
* Version 18.4 [2/2]
** DONE Track timezone changes for profiles :vrobbler:feature:profiles:personal:project:
:PROPERTIES:
@ -1552,11 +2203,13 @@ it's annoying.
- Note taken on [2025-07-20 Sun 16:21]
This thing is kicking my butt. As it stands it works, but the scrobbles are not assigned to the tracks properly.
* Version 18.3 [1/1]
** DONE Add timezone awarness to IMAP importer :personal:project:vrobbler:feature:importer:imap:timezones:
:PROPERTIES:
:ID: 05837b7c-96aa-6190-3678-e2ae7c7cac75
:END:
* Version 18 [4/4]
** DONE Condense tracks of the same title by the same artist with multiple albums :vrobbler:feature:music:project:personal:
:PROPERTIES:
@ -1678,6 +2331,7 @@ it's annoying.
:PROPERTIES:
:ID: 23f485e3-988c-6198-c79d-91fdf92f001c
:END:
* Version 17.0 [6/6]
** DONE [#A] Fix bug in new task label lookup for Emacs/Org-mode :vrobbler:bug:tasks:
:PROPERTIES:
@ -1741,6 +2395,7 @@ Not sure if the problem is in my Emacs hook sending or Vrobbler itself.
:PROPERTIES:
:ID: df58f8d0-fa4a-2037-c7d7-e5388c239042
:END:
* Version 0.16.0 [19/19]
** DONE [#A] Jellyfin, bandcamp tracks from Mopidy create duplicate music tracks :bug:scrobbling:music:
:PROPERTIES:
@ -1846,6 +2501,7 @@ out using that.
** DONE Fix bug in Jellyfin scrobbles that spam more scrobbles after completion :scrobbling:videos:bug:
This was fixed a while ago, but there's a new manifested bug. Going to create a
separate bug tracking ticket for that.
* Version 0.11.4 [9/9]
** DONE Add rudimentary video game scrobbling :improvement:content:videogames:
CLOSED: [2023-03-07 Tue 11:11]

96
data/play-example.json Normal file
View File

@ -0,0 +1,96 @@
{
"about": "This is a Play file that can be read by Board Game Stats. If you see this text, try to use a share, export or open-in function to open it with Board Game Stats.",
"players": [
{
"uuid": "31f8b92e-11d8-4162-88b1-fd9c79eea249",
"id": 2,
"name": "Colin Powell",
"isAnonymous": false,
"modificationDate": "2025-10-18 08:32:40",
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"AnnikaHeller_Capybara.webp\",\"shape\":[1,1,1,1],\"color\":[0.6,0.20000000298023224,0.9803921580314636]}}"
},
{
"uuid": "dd2d1881-84ab-474c-a6b2-3045d034dc40",
"id": 3,
"name": "Silas Sewell",
"isAnonymous": false,
"modificationDate": "2026-01-18 12:27:12",
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"RoryMuldoon_07.webp\",\"shape\":[1,1,1,1],\"color\":[0,0,0.1835034190722739]}}"
}
],
"locations": [
{
"uuid": "14f7389c-767f-4725-9b35-906c407b293c",
"id": 3,
"name": "Timberwyck Farm",
"modificationDate": "2025-07-01 18:03:38"
}
],
"games": [
{
"uuid": "9e431cdd-b325-4061-a875-d415d46342c0",
"id": 1046,
"name": "Sweet Takes",
"modificationDate": "2026-04-11 16:25:35",
"cooperative": false,
"highestWins": true,
"noPoints": false,
"usesTeams": false,
"urlThumb": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__small/img/gmGqlmwe9fniqYpniGwhg5RUgVQ=/fit-in/200x150/filters:strip_icc()/pic8784202.jpg",
"urlImage": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__original/img/b4IU8WIEWRpacbXp0FHG9HfFRpw=/0x0/filters:format(jpeg)/pic8784202.jpg",
"bggName": "Sweet Takes",
"bggYear": 2023,
"bggId": 407581,
"designers": "Hisashi Hayashi",
"isBaseGame": 1,
"isExpansion": 0,
"rating": 67,
"minPlayerCount": 2,
"maxPlayerCount": 5,
"minPlayTime": 15,
"maxPlayTime": 15,
"minAge": 8
}
],
"plays": [
{
"uuid": "7b2fd583-e8f2-40fe-9565-90178390b87e",
"modificationDate": "2026-04-16 20:18:03",
"entryDate": "2026-04-16 20:13:33",
"playDate": "2026-04-16 20:13:33",
"usesTeams": false,
"durationMin": 4,
"ignored": false,
"manualWinner": false,
"rounds": 0,
"locationRefId": 3,
"gameRefId": 1046,
"board": "",
"scoringSetting": 1,
"metaData": "{\"playerRefId\":2,\"playGameBggVersion\":\"{\\\"versionId\\\":0,\\\"versionName\\\":\\\"\\\",\\\"imageUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"thumbUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"yearPublished\\\":0}\",\"playUsedGameCopy\":2}",
"playerScores": [
{
"score": "27",
"winner": false,
"newPlayer": false,
"startPlayer": false,
"playerRefId": 2,
"role": "",
"rank": 0,
"seatOrder": 0
},
{
"score": "36",
"winner": true,
"newPlayer": true,
"startPlayer": false,
"playerRefId": 3,
"rank": 0,
"seatOrder": 0
}
],
"expansionPlays": []
}
],
"userInfo": { "meRefId": 2 }
}

BIN
data/statistics.sqlite3 Normal file

Binary file not shown.

View File

@ -21,5 +21,5 @@ push:
release kind="minor":
poetry run python scripts/release.py {{kind}}
@push
just push

29
poetry.lock generated
View File

@ -4966,6 +4966,18 @@ files = [
{file = "soupsieve-2.8.3.tar.gz", hash = "sha256:3267f1eeea4251fb42728b6dfb746edc9acaffc4a45b27e19450b676586e8349"},
]
[[package]]
name = "sqids"
version = "0.5.2"
description = "Generate YouTube-like ids from numbers."
optional = false
python-versions = ">=3.6"
groups = ["main"]
files = [
{file = "sqids-0.5.2-py3-none-any.whl", hash = "sha256:0089ba823e21fd44290c7225f02fb0b5140c36e41959c04d86d3f6f2513799be"},
{file = "sqids-0.5.2.tar.gz", hash = "sha256:5ac08f0c5c9b6814bc2e7c79ee5931e0849d25d95c50e415771b022a44f58af9"},
]
[[package]]
name = "sqlparse"
version = "0.5.5"
@ -5439,6 +5451,21 @@ brotli = ["brotli (==1.0.9) ; os_name != \"nt\" and python_version < \"3\" and p
secure = ["certifi", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "ipaddress ; python_version == \"2.7\"", "pyOpenSSL (>=0.14)", "urllib3-secure-extra"]
socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
[[package]]
name = "vadersentiment"
version = "3.3.2"
description = "VADER Sentiment Analysis. VADER (Valence Aware Dictionary and sEntiment Reasoner) is a lexicon and rule-based sentiment analysis tool that is specifically attuned to sentiments expressed in social media, and works well on texts from other domains."
optional = false
python-versions = "*"
groups = ["main"]
files = [
{file = "vaderSentiment-3.3.2-py2.py3-none-any.whl", hash = "sha256:3bf1d243b98b1afad575b9f22bc2cb1e212b94ff89ca74f8a23a588d024ea311"},
{file = "vaderSentiment-3.3.2.tar.gz", hash = "sha256:5d7c06e027fc8b99238edb0d53d970cf97066ef97654009890b83703849632f9"},
]
[package.dependencies]
requests = "*"
[[package]]
name = "vine"
version = "5.1.0"
@ -6005,4 +6032,4 @@ cffi = ["cffi (>=1.17,<2.0) ; platform_python_implementation != \"PyPy\" and pyt
[metadata]
lock-version = "2.1"
python-versions = ">=3.11,<3.15"
content-hash = "bd3f14a9cfce403db426af98774f1e3c41b97283aa43f4bd80f84594ee0dd726"
content-hash = "cc5b3b44071d6b0ab4f05189580232cc129b4ed694ab3f0673c3d838c3af0f8a"

View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "vrobbler"
version = "45.1"
version = "52.0"
description = ""
authors = ["Colin Powell <colin@unbl.ink>"]
@ -62,6 +62,8 @@ recipe-scrapers = "^15.11.0"
gpxpy = "^1.6.2"
fitparse = "^1.2.0"
lxml = ">=5.5.0"
vaderSentiment = "^3.3.2"
sqids = "^0.5.2"
[tool.poetry.group.test]
optional = true

View File

@ -124,7 +124,7 @@ def main():
if not done_items:
print("No DONE items found in Backlog — nothing to release.")
sys.exit(0)
sys.exit(1)
# ---------------------------------------------------------------
# 4. Build the new Version section text

View File

@ -1,3 +1,4 @@
import tempfile
from datetime import timedelta
import pytest
@ -128,6 +129,23 @@ class TestBirdingCSVImportModel:
assert imp.import_type == "Birding CSV"
assert "Birding" in str(imp)
def test_record_error(self, db, user):
imp = BirdingCSVImport.objects.create(user=user)
assert imp.error_log is None
imp.record_error("test error")
imp.refresh_from_db()
assert imp.error_log is not None
assert "test error" in imp.error_log
def test_record_error_appends(self, db, user):
imp = BirdingCSVImport.objects.create(user=user)
imp.record_error("first error")
imp.record_error("second error")
imp.refresh_from_db()
assert imp.error_log.count("\n") == 1
assert "first error" in imp.error_log
assert "second error" in imp.error_log
@pytest.mark.django_db(transaction=True)
def test_process_via_model(self, user, birding_csv_file):
imp = BirdingCSVImport.objects.create(user=user)
@ -137,3 +155,35 @@ class TestBirdingCSVImportModel:
imp.refresh_from_db()
assert imp.process_count == 1
assert imp.processed_finished is not None
def test_record_error_on_bad_csv(self, user, db):
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
Canada Goose,6,Test Park,Stationary,"Bad Date",4:15 PM,9 minute(s),,,4,true,4 species,
"""
with tempfile.NamedTemporaryFile(
mode="w", suffix=".csv", delete=False, encoding="utf-8"
) as f:
f.write(content)
file_path = f.name
errors = []
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
assert len(scrobbles) == 0
assert len(errors) == 1
assert "Could not parse date/time" in errors[0]
def test_record_error_on_bad_location(self, user, db):
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
Canada Goose,6,,Stationary,"May 10, 2026",4:15 PM,9 minute(s),,,4,true,4 species,
"""
with tempfile.NamedTemporaryFile(
mode="w", suffix=".csv", delete=False, encoding="utf-8"
) as f:
f.write(content)
file_path = f.name
errors = []
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
assert len(scrobbles) == 0
assert len(errors) == 1
assert "Skipping rows with no location" in errors[0]

View File

@ -8,7 +8,8 @@ from django.urls import reverse
from django.utils import timezone
from music.models import Album, Artist, Track
from podcasts.models import PodcastEpisode
from scrobbles.models import Scrobble
from scrobbles.models import Scrobble, ShareViewLog
from scrobbles.sqids import encode_scrobble_share
from tasks.models import Task
@ -512,6 +513,7 @@ def test_scrobble_detail_view_with_notes_as_flat_list(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": ["First note", "Second note"],
"description": "Test description",
@ -534,6 +536,7 @@ def test_scrobble_detail_view_with_notes_as_dict_timestamps(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": [
{"2024-01-01 10:00:00": "Note at first timestamp"},
@ -562,6 +565,7 @@ def test_scrobble_detail_view_with_notes_and_labels(client):
task=task,
media_type="Task",
user=user,
visibility="public",
log={
"notes": [
{"2024-01-01 10:00:00": "Note with label"},
@ -739,3 +743,293 @@ def test_gps_webhook_creates_location(client, valid_auth_token):
)
assert response.status_code == 200
assert "scrobble_id" in response.data
@pytest.mark.django_db
def test_share_view_shared_visibility(client):
user = get_user_model().objects.create_user(
username="shareuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_view_public_visibility(client):
user = get_user_model().objects.create_user(
username="shareuser2", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="public",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_view_private_visibility_returns_404(client):
user = get_user_model().objects.create_user(
username="shareuser3", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_invalid_sqid_returns_404(client):
url = reverse("scrobbles:shared-detail", kwargs={"sqid": "InvalidSqid123"})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_expired_token_returns_404(client):
user = get_user_model().objects.create_user(
username="shareuser4", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
scrobble.regenerate_share_token()
url = reverse("scrobbles:shared-detail", kwargs={"sqid": old_sqid})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_view_increments_count_and_logs_view(client):
user = get_user_model().objects.create_user(
username="shareuser5", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
assert scrobble.share_view_count == 0
url = reverse(
"scrobbles:shared-detail",
kwargs={"sqid": encode_scrobble_share(scrobble.id, scrobble.share_token_version)},
)
response = client.get(url)
assert response.status_code == 200
scrobble.refresh_from_db()
assert scrobble.share_view_count == 1
assert ShareViewLog.objects.filter(scrobble=scrobble).count() == 1
log_entry = ShareViewLog.objects.filter(scrobble=scrobble).first()
assert log_entry.ip_address == "127.0.0.1"
assert log_entry.user_agent == ""
assert log_entry.referrer == ""
@pytest.mark.django_db
def test_explore_view_shows_only_public_scrobbles(client):
user = get_user_model().objects.create_user(
username="exploreuser", password="testpass"
)
public_task = Task.objects.create(title="Public Task Title")
shared_task = Task.objects.create(title="Shared Task Title")
private_task = Task.objects.create(title="Private Task Title")
ts = timezone.now()
public_scrobble = Scrobble.objects.create(
task=public_task, media_type="Task", user=user, visibility="public",
timestamp=ts,
)
Scrobble.objects.create(
task=shared_task, media_type="Task", user=user, visibility="shared",
timestamp=ts,
)
Scrobble.objects.create(
task=private_task, media_type="Task", user=user, visibility="private",
timestamp=ts,
)
url = reverse("scrobbles:explore")
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
assert "Public Task Title" in content
assert "Shared Task Title" not in content
assert "Private Task Title" not in content
@pytest.mark.django_db
def test_change_visibility_owner_can_change(client):
user = get_user_model().objects.create_user(
username="visuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
client.force_login(user)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
scrobble.refresh_from_db()
assert scrobble.visibility == "shared"
@pytest.mark.django_db
def test_change_visibility_non_owner_gets_404(client):
owner = get_user_model().objects.create_user(
username="owner", password="testpass"
)
other = get_user_model().objects.create_user(
username="other", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=owner, visibility="private",
timestamp=timezone.now(),
)
client.force_login(other)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 404
scrobble.refresh_from_db()
assert scrobble.visibility == "private"
@pytest.mark.django_db
def test_change_visibility_anonymous_redirects_to_login(client):
user = get_user_model().objects.create_user(
username="anontest", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="private",
timestamp=timezone.now(),
)
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
response = client.post(url, {"visibility": "shared"})
assert response.status_code == 302
assert "/login/" in response.url
@pytest.mark.django_db
def test_regenerate_share_token_invalidates_old_sqid(client):
user = get_user_model().objects.create_user(
username="regentest", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
client.force_login(user)
url = reverse("scrobbles:regenerate-share-token", kwargs={"uuid": scrobble.uuid})
response = client.post(url)
assert response.status_code == 302
scrobble.refresh_from_db()
assert scrobble.share_token_version == 1
old_url = reverse("scrobbles:shared-detail", kwargs={"sqid": old_sqid})
old_response = client.get(old_url)
assert old_response.status_code == 404
new_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
new_url = reverse("scrobbles:shared-detail", kwargs={"sqid": new_sqid})
new_response = client.get(new_url)
assert new_response.status_code == 200
@pytest.mark.django_db
def test_share_analytics_owner_can_view(client):
user = get_user_model().objects.create_user(
username="analyticsuser", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 200
@pytest.mark.django_db
def test_share_analytics_non_owner_gets_404(client):
owner = get_user_model().objects.create_user(
username="analyticsowner", password="testpass"
)
other = get_user_model().objects.create_user(
username="analyticsother", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=owner, visibility="shared",
timestamp=timezone.now(),
)
client.force_login(other)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 404
@pytest.mark.django_db
def test_share_analytics_shows_view_logs(client):
user = get_user_model().objects.create_user(
username="analyticsviews", password="testpass"
)
task = Task.objects.create(title="Test Task")
scrobble = Scrobble.objects.create(
task=task, media_type="Task", user=user, visibility="shared",
timestamp=timezone.now(),
)
sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
share_url = reverse("scrobbles:shared-detail", kwargs={"sqid": sqid})
client.get(share_url)
client.get(share_url)
client.force_login(user)
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
response = client.get(url)
assert response.status_code == 200
content = response.content.decode()
assert "127.0.0.1" in content
assert ShareViewLog.objects.filter(scrobble=scrobble).count() == 2

View File

@ -27,6 +27,7 @@ class BeerAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("styles", "producer")
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -26,5 +26,6 @@ class BirdingLocationAdmin(admin.ModelAdmin):
@admin.register(BirdingCSVImport)
class BirdingCSVImportAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("uuid", "process_count", "processed_finished", "processing_started")
list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
raw_id_fields = ("user",)
ordering = ("-created",)

View File

@ -2,7 +2,9 @@ import csv
import logging
import re
from collections import defaultdict
from datetime import datetime, timedelta
from datetime import timedelta
from dateutil import parser
from django.contrib.auth import get_user_model
@ -35,11 +37,12 @@ def parse_coords(location_str):
def parse_timestamp(date_str, time_str):
try:
dt = datetime.strptime(f"{date_str} {time_str}", "%B %d, %Y %I:%M %p")
dt_str = f"{date_str} {time_str}".strip()
dt = parser.parse(dt_str)
return dt
except (ValueError, TypeError):
try:
dt = datetime.strptime(date_str, "%B %d, %Y")
dt = parser.parse(date_str)
return dt
except (ValueError, TypeError):
logger.warning(f"Could not parse date/time: {date_str} {time_str}")
@ -61,7 +64,7 @@ def parse_int(value):
return None
def import_birding_csv(file_path, user_id):
def import_birding_csv(file_path, user_id, record_error=None):
user = User.objects.get(id=user_id)
new_scrobbles = []
@ -80,11 +83,17 @@ def import_birding_csv(file_path, user_id):
for (location_str, date_str, time_str), sighting_rows in groups.items():
if not location_str:
logger.warning("Skipping rows with no location")
msg = "Skipping rows with no location"
logger.warning(msg)
if record_error:
record_error(msg)
continue
timestamp = parse_timestamp(date_str, time_str)
if not timestamp:
msg = f"Could not parse date/time: {date_str} {time_str}"
if record_error:
record_error(msg)
continue
timestamp = user.profile.get_timestamp_with_tz(timestamp)

View File

@ -0,0 +1,18 @@
# Generated by Django 4.2.29 on 2026-06-08 14:57
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("birds", "0002_birdingcsvimport"),
]
operations = [
migrations.AddField(
model_name="birdingcsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -61,9 +61,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
@cached_property
def bird_list(self) -> str:
if self.birds:
return ", ".join(
[BirdSightingEntry(**b).__str__() for b in self.birds]
)
return ", ".join([BirdSightingEntry(**b).__str__() for b in self.birds])
return ""
def as_html(self) -> str:
@ -80,9 +78,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
)
if self.area:
html_parts.append(
f'<div class="birding-area">Area: {self.area}</div>'
)
html_parts.append(f'<div class="birding-area">Area: {self.area}</div>')
if self.party_size:
html_parts.append(
@ -105,9 +101,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
)
if self.guide:
html_parts.append(
f'<div class="birding-guide">Guide: {self.guide}</div>'
)
html_parts.append(f'<div class="birding-guide">Guide: {self.guide}</div>')
if self.duration_minutes:
html_parts.append(
@ -183,9 +177,7 @@ class Bird(TimeStampedModel):
class BirdingLocation(ScrobblableMixin):
description = models.TextField(**BNULL)
geo_location = models.ForeignKey(
GeoLocation, **BNULL, on_delete=models.DO_NOTHING
)
geo_location = models.ForeignKey(GeoLocation, **BNULL, on_delete=models.DO_NOTHING)
ebird_hotspot_id = models.CharField(max_length=255, **BNULL)
def get_absolute_url(self):
@ -193,7 +185,7 @@ class BirdingLocation(ScrobblableMixin):
@property
def subtitle(self):
return ""
return self.geo_location
@property
def strings(self) -> ScrobblableConstants:
@ -224,6 +216,7 @@ class BirdingCSVImport(TimeStampedModel):
processed_finished = models.DateTimeField(**BNULL)
process_log = models.TextField(**BNULL)
process_count = models.IntegerField(**BNULL)
error_log = models.TextField(**BNULL)
csv_file = models.FileField(upload_to="birding-csv-uploads/", **BNULL)
class Meta:
@ -269,9 +262,7 @@ class BirdingCSVImport(TimeStampedModel):
self.save(update_fields=["process_log", "process_count"])
return
for count, scrobble in enumerate(scrobbles):
scrobble_str = (
f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
)
scrobble_str = f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
log_line = f"{scrobble_str}"
if count > 0:
log_line = "\n" + log_line
@ -279,6 +270,14 @@ class BirdingCSVImport(TimeStampedModel):
self.process_count = len(scrobbles)
self.save(update_fields=["process_log", "process_count"])
def record_error(self, error_message):
log_line = f"{timezone.now().isoformat()}: {error_message}"
if self.error_log:
self.error_log += "\n" + log_line
else:
self.error_log = log_line
self.save(update_fields=["error_log"])
def scrobbles(self):
from scrobbles.models import Scrobble
@ -297,6 +296,13 @@ class BirdingCSVImport(TimeStampedModel):
from birds.importer import import_birding_csv
self.mark_started()
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_birding_csv(
self.upload_file_path, self.user_id, record_error=self.record_error
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()

View File

@ -38,6 +38,7 @@ class BoardGameLocationAdmin(admin.ModelAdmin):
"uuid",
"geo_location",
)
raw_id_fields = ("geo_location",)
ordering = ("-created",)
@ -49,6 +50,7 @@ class BoardGameAdmin(admin.ModelAdmin):
"title",
"published_year",
)
raw_id_fields = ("publisher", "publishers", "designers", "expansion_for_boardgame")
search_fields = ("title",)
ordering = ("-created",)
inlines = [

View File

@ -266,8 +266,9 @@ class BoardGame(ScrobblableMixin):
"self", **BNULL, on_delete=models.DO_NOTHING
)
def __str__(self):
return self.title
@property
def subtitle(self) -> str:
return self.publisher
def get_absolute_url(self):
return reverse("boardgames:boardgame_detail", kwargs={"slug": self.uuid})

View File

@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
"source": "Lichess",
"timezone": user.profile.timezone,
"log": log_data,
"visibility": "private",
}
if commit:
Scrobble.objects.create(**scrobble_dict)

View File

@ -27,6 +27,7 @@ class BookAdmin(admin.ModelAdmin):
"first_publish_year",
"pages",
)
raw_id_fields = ("authors",)
search_fields = ("name",)
ordering = ("-created",)
inlines = [
@ -34,11 +35,11 @@ class BookAdmin(admin.ModelAdmin):
]
def issue_or_volume(self, obj):
return obj.issue_number or obj.volume_number
return obj.subtitle
@admin.register(Paper)
class BookAdmin(admin.ModelAdmin):
class PaperAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
@ -47,6 +48,7 @@ class BookAdmin(admin.ModelAdmin):
"first_publish_year",
"pages",
)
raw_id_fields = ("authors",)
search_fields = ("name",)
ordering = ("-created",)
inlines = [

View File

@ -1,9 +1,8 @@
import logging
import re
import sqlite3
from datetime import datetime, timedelta
from datetime import datetime, timedelta, timezone
from enum import Enum
from zoneinfo import ZoneInfo
import requests
from books.constants import BOOKS_TITLES_TO_IGNORE
@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
return book_id_map
def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
def build_page_data(page_rows: list, book_map: dict) -> dict:
"""Given rows of page data from KoReader, parse each row and build
scrobbles for our user, loading the page data into the page_data
field on the scrobble instance.
@ -187,18 +186,20 @@ def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
book_ids_not_found.append(koreader_book_id)
continue
if "pages" not in book_map[koreader_book_id].keys():
book_map[koreader_book_id]["pages"] = {}
book_map[koreader_book_id].setdefault("pages", [])
page_number = page_row[KoReaderPageStatColumn.PAGE.value]
duration = page_row[KoReaderPageStatColumn.DURATION.value]
start_ts = page_row[KoReaderPageStatColumn.START_TIME.value]
book_map[koreader_book_id]["pages"][page_number] = {
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
book_map[koreader_book_id]["pages"].append(
{
"page_number": page_number,
"duration": duration,
"start_ts": start_ts,
"end_ts": start_ts + duration,
}
)
if book_ids_not_found:
logger.info(f"Found pages for books not in file: {set(book_ids_not_found)}")
return book_map
@ -216,7 +217,6 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_not_found.append(book_id)
continue
should_create_scrobble = False
scrobble_page_data = {}
playback_position_seconds = 0
prev_page_stats = {}
@ -225,11 +225,12 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
pages_processed = 0
total_pages_read = len(book_map[koreader_book_id]["pages"])
ordered_pages = sorted(
book_map[koreader_book_id]["pages"].items(),
key=lambda x: x[1]["start_ts"],
book_map[koreader_book_id]["pages"],
key=lambda x: x["start_ts"],
)
for cur_page_number, stats in ordered_pages:
for stats in ordered_pages:
page_number = stats["page_number"]
pages_processed += 1
seconds_from_last_page = 0
@ -243,46 +244,52 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
)
end_of_reading = pages_processed == total_pages_read
big_jump_to_this_page = (cur_page_number - last_page_number) > 10
big_jump_to_this_page = (page_number - last_page_number) > 10
is_session_gap = seconds_from_last_page > SESSION_GAP_SECONDS
if (is_session_gap and not big_jump_to_this_page) or end_of_reading:
should_create_scrobble = True
should_create_scrobble = (
is_session_gap and not big_jump_to_this_page
) or end_of_reading
# Always accumulate the current page first
scrobble_page_data[page_number] = stats
if should_create_scrobble:
# For end-of-reading, the current page is already accumulated
# and belongs in this scrobble. For a session gap, remove the
# current page from this scrobble — it starts a new session.
if is_session_gap and not end_of_reading:
del scrobble_page_data[page_number]
scrobble_page_data = dict(
sorted(
scrobble_page_data.items(),
key=lambda x: x[1]["start_ts"],
)
)
try:
first_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[0]
)
last_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[-1]
)
except IndexError:
if not scrobble_page_data:
logger.error(
"Could not process book, no page data found",
extra={"scrobble_page_data": scrobble_page_data},
)
continue
first_page = next(iter(scrobble_page_data.values()))
last_page = scrobble_page_data[
list(scrobble_page_data.keys())[-1]
]
timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(first_page.get("start_ts")))
datetime.fromtimestamp(
int(first_page.get("start_ts"))
)
)
stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(last_page.get("end_ts")))
datetime.fromtimestamp(
int(last_page.get("end_ts"))
)
)
# Adjust for Daylight Saving Time
# if timestamp.dst() == timedelta(
# 0
# ) or stop_timestamp.dst() == timedelta(0):
# timestamp = timestamp - timedelta(hours=1)
# stop_timestamp = stop_timestamp - timedelta(hours=1)
scrobble = Scrobble.objects.filter(
timestamp=timestamp,
book_id=book_id,
@ -291,7 +298,7 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
if not scrobble:
logger.info(
f"Queueing scrobble for {book_id}, page {cur_page_number}"
f"Queueing scrobble for {book_id}, page {page_number}"
)
log_data = {
"koreader_hash": book_dict.get("hash"),
@ -318,16 +325,18 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
timezone=tz,
)
)
# Then start over
should_create_scrobble = False
# Then start over for the next session
playback_position_seconds = 0
scrobble_page_data = {}
# We accumulate pages for the scrobble until we should create a new one
scrobble_page_data[cur_page_number] = stats
# For session gaps, re-add the current page as the
# beginning of the next session's accumulation
if is_session_gap and not end_of_reading:
scrobble_page_data[page_number] = stats
last_page_number = cur_page_number
last_page_number = page_number
prev_page_stats = stats
if pages_not_found:
logger.info(f"Pages not found for books: {set(pages_not_found)}")
return scrobbles_to_create
@ -335,13 +344,16 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
def fix_long_play_stats_for_scrobbles(scrobbles: list) -> None:
"""Given a list of scrobbles, update pages read, long play seconds and check
for media completion"""
for media completion.
Uses the long_play_last_scrobble FK chain to accumulate time.
Consider using the recompute_long_play_seconds management command instead.
"""
for scrobble in scrobbles:
# But if there's a next scrobble, set pages read to their starting page
if scrobble.previous and not scrobble.previous.long_play_complete:
if scrobble.long_play_last_scrobble and not scrobble.long_play_last_scrobble.long_play_complete:
scrobble.long_play_seconds = scrobble.playback_position_seconds + (
scrobble.previous.long_play_seconds or 0
scrobble.long_play_last_scrobble.long_play_seconds or 0
)
else:
scrobble.long_play_seconds = scrobble.playback_position_seconds
@ -357,9 +369,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
new_scrobbles = []
user = User.objects.filter(id=user_id).first()
tz = ZoneInfo("UTC")
if user:
tz = user.profile.tzinfo
is_os_file = "https://" not in file_path
if is_os_file:
@ -375,7 +384,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
book_map = build_page_data(
cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
book_map,
tz,
)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
else:
@ -392,7 +400,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
_sqlite_bytes(file_path), max_buffer_size=1_048_576
):
if table_name == "page_stat_data":
book_map = build_page_data(rows, book_map, tz)
book_map = build_page_data(rows, book_map)
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
logger.info(f"Creating {len(new_scrobbles)} new scrobbles")

View File

@ -0,0 +1,301 @@
import logging
import time
from django.core.management.base import BaseCommand
from django.db import transaction
from books.constants import READCOMICSONLINE_URL
logger = logging.getLogger(__name__)
MISSING_ALL = [
"cover",
"summary",
"isbn",
"pages",
"language",
"publisher",
"publish_year",
]
MISSING_GROUPS = {
"cover": lambda b: not bool(b.cover),
"summary": lambda b: not b.summary,
"isbn": lambda b: not b.isbn_13 and not b.isbn_10,
"pages": lambda b: b.pages is None,
"language": lambda b: not b.language,
"publisher": lambda b: not b.publisher,
"publish_year": lambda b: b.first_publish_year is None,
}
def _book_matches(book, flags):
if not flags:
return False
for flag in flags:
fn = MISSING_GROUPS.get(flag)
if fn and fn(book):
return True
return False
class Command(BaseCommand):
help = "Backfill missing metadata on books from Google Books, OpenLibrary, and ComicVine"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
parser.add_argument(
"--batch-size",
type=int,
default=100,
help="Number of books to process per batch (default: 100)",
)
parser.add_argument(
"--sleep",
type=float,
default=0.5,
help="Seconds to sleep between API calls (default: 0.5)",
)
for flag in MISSING_ALL:
parser.add_argument(
f"--missing-{flag}",
dest="missing_flags",
action="append_const",
const=flag,
help=f"Process books missing {flag}",
)
parser.add_argument(
"--comics-only",
action="store_true",
help="Only process books with a readcomicsonline.ru URL",
)
parser.add_argument(
"--all",
action="store_true",
dest="all_missing",
help="Process books missing any metadata field",
)
def handle(self, *args, **options):
from books.models import Book
commit = options["commit"]
batch_size = options["batch_size"]
sleep_secs = options["sleep"]
flags = options.get("missing_flags") or []
comics_only = options["comics_only"]
all_missing = options["all_missing"]
if all_missing:
flags = MISSING_ALL
if not flags and not comics_only:
self.stdout.write(
"No filters specified. Use --all, --missing-*, or --comics-only."
)
return
qs = Book.objects.all()
if comics_only:
qs = qs.filter(readcomics_url__isnull=False)
if flags:
qs = [b for b in qs.iterator() if _book_matches(b, flags)]
else:
qs = list(qs)
total = len(qs)
self.stdout.write(f"Found {total} books to process")
if not commit:
self.stdout.write(
"Dry run — no API calls will be made. Use --commit to run lookups."
)
return
enriched = 0
skipped = 0
stats = {
"cover_fixed": 0,
"summary_fixed": 0,
"isbn_fixed": 0,
"pages_fixed": 0,
"language_fixed": 0,
"publisher_fixed": 0,
"publish_year_fixed": 0,
}
for batch_num, offset in enumerate(range(0, len(qs), batch_size)):
batch = qs[offset : offset + batch_size]
for book in batch:
result = self._enrich_book(book, sleep_secs)
if result:
enriched += 1
for key in stats:
if result.get(key):
stats[key] += 1
else:
skipped += 1
self.stdout.write(
f" Batch {batch_num + 1}: {offset + len(batch)}/{total}"
f"enriched: {enriched}, skipped: {skipped}"
)
self.stdout.write(
f"\nResults (commit={commit}):\n"
f" Books enriched: {enriched}\n"
f" Books skipped: {skipped}\n"
f" Covers fixed: {stats['cover_fixed']}\n"
f" Summaries fixed:{stats['summary_fixed']}\n"
f" ISBNs fixed: {stats['isbn_fixed']}\n"
f" Pages fixed: {stats['pages_fixed']}\n"
f" Languages fixed:{stats['language_fixed']}\n"
f" Publishers fixed:{stats['publisher_fixed']}\n"
f" Publish yrs fixed: {stats['publish_year_fixed']}"
)
def _enrich_book(self, book, sleep_secs):
from books.sources.comicvine import (
lookup_comic_from_comicvine,
lookup_issue_by_comicvine_id,
)
from books.sources.google import lookup_book_from_google
from books.sources.openlibrary import lookup_book_from_openlibrary as lookup_book_from_ol
title = book.original_title or book.title
author_name = book.author.name if book.author else None
book_dict = {}
cv_data = None
if book.comicvine_id:
cv_data = lookup_issue_by_comicvine_id(str(book.comicvine_id))
if not cv_data:
cv_data = lookup_comic_from_comicvine(title)
if cv_data:
book_dict.update(cv_data)
ol_data = lookup_book_from_ol(title, author=author_name)
time.sleep(sleep_secs)
google_data = lookup_book_from_google(title)
if ol_data:
for k, v in ol_data.items():
book_dict.setdefault(k, v)
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
if not book_dict:
return None
changed = self._apply(book, book_dict, title)
return changed
def _apply(self, book, data, title):
changed = {
"cover_fixed": False,
"summary_fixed": False,
"isbn_fixed": False,
"pages_fixed": False,
"language_fixed": False,
"publisher_fixed": False,
"publish_year_fixed": False,
}
update_fields = []
cover_url = data.pop("cover_url", "")
if data.get("summary") and not book.summary:
book.summary = data["summary"]
update_fields.append("summary")
changed["summary_fixed"] = True
if data.get("isbn_13") and not book.isbn_13:
book.isbn_13 = data["isbn_13"]
update_fields.append("isbn_13")
changed["isbn_fixed"] = True
if data.get("isbn_10") and not book.isbn_10:
book.isbn_10 = data["isbn_10"]
update_fields.append("isbn_10")
changed["isbn_fixed"] = True
if data.get("pages") and book.pages is None:
book.pages = data["pages"]
update_fields.append("pages")
changed["pages_fixed"] = True
if data.get("language") and not book.language:
book.language = data["language"]
update_fields.append("language")
changed["language_fixed"] = True
if data.get("publisher") and not book.publisher:
book.publisher = data["publisher"]
update_fields.append("publisher")
changed["publisher_fixed"] = True
if data.get("first_publish_year") and book.first_publish_year is None:
book.first_publish_year = data["first_publish_year"]
update_fields.append("first_publish_year")
changed["publish_year_fixed"] = True
if data.get("openlibrary_id") and not book.openlibrary_id:
book.openlibrary_id = data["openlibrary_id"]
update_fields.append("openlibrary_id")
if data.get("comicvine_id") and not book.comicvine_id:
book.comicvine_id = data["comicvine_id"]
update_fields.append("comicvine_id")
if data.get("issue_number") and book.issue_number is None:
book.issue_number = data["issue_number"]
update_fields.append("issue_number")
if data.get("volume_number") and book.volume_number is None:
book.volume_number = data["volume_number"]
update_fields.append("volume_number")
if data.get("volume") and not book.volume:
book.volume = data["volume"]
update_fields.append("volume")
if data.get("volume_comicvine_id") and not book.volume_comicvine_id:
book.volume_comicvine_id = data["volume_comicvine_id"]
update_fields.append("volume_comicvine_id")
if update_fields:
book.save(update_fields=update_fields)
self.stdout.write(f" [ENRICHED] {book}{', '.join(update_fields)}")
if cover_url and not book.cover:
book.save_image_from_url(cover_url)
if book.cover:
changed["cover_fixed"] = True
self.stdout.write(f" [COVER] {book} — cover saved from source")
genres = data.pop("genres", data.pop("generes", []))
if genres:
existing = set(book.genre.names())
new_genres = [g for g in genres if g not in existing]
if new_genres:
book.genre.add(*new_genres)
self.stdout.write(f" [GENRES] {book} — added {len(new_genres)} genres")
tags = data.pop("tags", [])
if tags:
existing_tags = set(book.tags.names())
new_tags = [t for t in tags if t not in existing_tags]
if new_tags:
book.tags.add(*new_tags)
self.stdout.write(f" [TAGS] {book} — added {', '.join(new_tags)}")
return changed if any(changed.values()) else None

View File

@ -0,0 +1,130 @@
import logging
from datetime import timedelta
from django.core.management.base import BaseCommand
from django.db import transaction
from books.koreader import SESSION_GAP_SECONDS, fix_long_play_stats_for_scrobbles
logger = logging.getLogger(__name__)
SESSION_GAP = timedelta(seconds=SESSION_GAP_SECONDS)
def _page_data_keys(pages):
return sorted(int(k) for k in (pages or {}))
class Command(BaseCommand):
help = "Merge orphaned 1-page KOReader scrobbles into the preceding scrobble"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Commit changes to the database",
)
def handle(self, *args, **options):
from scrobbles.models import Scrobble
commit = options["commit"]
qs = Scrobble.objects.filter(
media_type="Book", source="KOReader"
).order_by("book_id", "timestamp")
if not qs.exists():
self.stdout.write("No KOReader book scrobbles found.")
return
merged = 0
affected_books = set()
# Group by book_id manually since we're iterating in order
book_scrobbles = {}
for s in qs:
book_scrobbles.setdefault(s.book_id, []).append(s)
if not commit:
self.stdout.write("Dry run — no changes will be saved. Use --commit to apply.")
for book_id, scrobbles in book_scrobbles.items():
batch_merged = 0
i = 0
while i < len(scrobbles) - 1:
current = scrobbles[i]
orphan = scrobbles[i + 1]
orphan_pages = orphan.logdata.page_data if orphan.logdata else {}
orphan_keys = _page_data_keys(orphan_pages)
if len(orphan_keys) != 1:
i += 1
continue
current_pages = current.logdata.page_data if current.logdata else {}
current_keys = _page_data_keys(current_pages)
if not current_keys:
i += 1
continue
orphan_page_num = orphan_keys[0]
current_last_page = current_keys[-1]
if orphan_page_num != current_last_page + 1:
i += 1
continue
# Check that the orphan is close enough in time
gap = orphan.timestamp - current.stop_timestamp
if gap > SESSION_GAP:
i += 1
continue
# Merge orphan into current
current_pages[str(orphan_page_num)] = orphan_pages[str(orphan_page_num)]
current.log["page_data"] = current_pages
current.log["pages_read"] = len(current_pages)
current.stop_timestamp = orphan.stop_timestamp
current.playback_position_seconds += orphan.playback_position_seconds
affected_books.add(book_id)
if commit:
with transaction.atomic():
current.save(
update_fields=[
"log",
"stop_timestamp",
"playback_position_seconds",
]
)
orphan.delete()
merged += 1
batch_merged += 1
scrobbles.pop(i + 1)
if batch_merged:
self.stdout.write(
f" Book {book_id}: merged {batch_merged} orphan scrobble(s)"
)
self.stdout.write(f"\nTotal orphans merged: {merged}")
if commit and affected_books:
self.stdout.write("Recalculating long_play_stats for affected books...")
for book_id in affected_books:
scrobbles_to_fix = (
Scrobble.objects.filter(book_id=book_id, source="KOReader")
.order_by("timestamp")
)
fix_long_play_stats_for_scrobbles(list(scrobbles_to_fix))
self.stdout.write(f"Fixed stats for {len(affected_books)} books.")
if not commit:
self.stdout.write(
f"\nWould merge {merged} orphan scrobble(s) across "
f"{len(affected_books)} book(s)."
)

View File

@ -0,0 +1,23 @@
# Generated by Django 4.2.29 on 2026-06-15 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("books", "0036_alter_book_genre_alter_paper_genre"),
]
operations = [
migrations.AddField(
model_name="book",
name="volume",
field=models.CharField(blank=True, max_length=255, null=True),
),
migrations.AddField(
model_name="book",
name="volume_comicvine_id",
field=models.CharField(blank=True, max_length=255, null=True),
),
]

View File

@ -4,6 +4,7 @@ from dataclasses import dataclass
from datetime import datetime
from typing import Optional
from uuid import uuid4
from zoneinfo import ZoneInfo
import requests
from books.constants import MediaSourceTag, READCOMICSONLINE_URL
@ -19,6 +20,7 @@ from books.openlibrary import (
from books.sources.comicvine import (
ComicVineClient,
lookup_comic_from_comicvine,
lookup_issue_by_comicvine_id,
)
from books.sources.google import lookup_book_from_google
from books.sources.openlibrary import (
@ -71,7 +73,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
_excluded_fields = {"koreader_hash", "page_data"}
def avg_seconds_per_page(self):
if self.page_data:
if self.page_data and isinstance(self.page_data, dict):
total_duration = 0
for page_num, stats in self.page_data.items():
total_duration += stats.get("duration", 0)
@ -149,6 +151,8 @@ class Book(LongPlayScrobblableMixin):
first_sentence = models.TextField(**BNULL)
# ComicVine
comicvine_id = models.CharField(max_length=255, **BNULL)
volume = models.CharField(max_length=255, **BNULL)
volume_comicvine_id = models.CharField(max_length=255, **BNULL)
readcomics_url = models.CharField(max_length=255, **BNULL)
next_readcomics_url = models.CharField(max_length=255, **BNULL)
issue_number = models.IntegerField(**BNULL)
@ -173,11 +177,10 @@ class Book(LongPlayScrobblableMixin):
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
def __str__(self) -> str:
if self.issue_number and "Issue" not in str(self.title):
return f"{self.title} - Issue {self.issue_number}"
if self.volume_number and "Volume" not in str(self.title):
return f"{self.title} - Volume {self.volume_number}"
return f"{self.title}"
if not self.subtitle:
return self.title
return f"{self.title} - {self.subtitle}"
def save(self, *args, **kwargs):
if self.pages:
@ -188,7 +191,18 @@ class Book(LongPlayScrobblableMixin):
@property
def subtitle(self):
return f" by {self.author}"
subtitle_parts = []
if self.author:
subtitle_parts.append(self.author.name)
if self.issue_number and "Issue" not in str(self.title):
subtitle_parts.append(f"Issue {self.issue_number}")
if self.volume_number and "Volume" not in str(self.title):
subtitle_parts.append(f"Volume {self.volume_number}")
if len(subtitle_parts) > 1:
return " / ".join(subtitle_parts)
if len(subtitle_parts) == 1:
return subtitle_parts[0]
return ""
@property
def strings(self) -> ScrobblableConstants:
@ -218,36 +232,24 @@ class Book(LongPlayScrobblableMixin):
) -> "Book":
book, created = cls.objects.get_or_create(title=title)
if not created:
if not created and not overwrite:
return book
book_dict = lookup_comic_from_comicvine(title)
if not book_dict:
return book
if created or overwrite:
author_list = []
author_dicts = book_dict.pop("author_dicts")
if author_dicts:
for author_dict in author_dicts:
if author_dict.get("authorId"):
author, a_created = Author.objects.get_or_create(
semantic_id=author_dict.get("authorId")
)
author_list.append(author)
if a_created:
author.name = author_dict.get("name")
author.save()
# TODO enrich author?
...
tags = book_dict.pop("tags", [])
genres = book_dict.pop("genres", book_dict.pop("generes", []))
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
for k, v in book_dict.items():
setattr(book, k, v)
book.save()
if author_list:
book.authors.add(*author_list)
genres = book_dict.pop("genres", [])
if genres:
book.genre.add(*genres)
if genres:
book.genre.add(*genres)
if tags:
book.tags.add(*tags)
return book
@classmethod
@ -281,24 +283,43 @@ class Book(LongPlayScrobblableMixin):
book_dict = None
source_tag = None
tried_comicvine = False
if READCOMICSONLINE_URL in url:
book_dict = lookup_comic_from_comicvine(title)
tried_comicvine = True
if book_dict:
source_tag = MediaSourceTag.COMICVINE
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
book_dict["readcomics_url"] = get_comic_issue_url(url)
book_dict["next_readcomics_url"] = next_url_if_exists(
book_dict["readcomics_url"]
)
if not book_dict:
book_dict = lookup_book_from_ol(title, author=author)
if book_dict:
book_dict = {}
ol_data = lookup_book_from_ol(title, author=author)
google_data = lookup_book_from_google(title)
if ol_data:
book_dict.update(ol_data)
source_tag = MediaSourceTag.OPENLIBRARY
if not book_dict:
book_dict = lookup_book_from_google(title)
if book_dict:
if google_data:
for k, v in google_data.items():
if v:
book_dict.setdefault(k, v)
source_tag = MediaSourceTag.GOOGLE_BOOKS
if ol_data and ol_data.get("cover_url"):
book_dict["cover_url"] = ol_data["cover_url"]
# Always try ComicVine as a fallback — it may recognize books that
# OL/Google don't flag as comics
if not tried_comicvine:
cv_data = lookup_comic_from_comicvine(title)
if cv_data:
for k, v in cv_data.items():
if v:
book_dict.setdefault(k, v)
source_tag = MediaSourceTag.COMICVINE
if not book_dict:
logger.warning(
@ -310,6 +331,7 @@ class Book(LongPlayScrobblableMixin):
authors = book_dict.pop("authors", [])
cover_url = book_dict.pop("cover_url", "")
genres = book_dict.pop("genres", book_dict.pop("generes", []))
tags = book_dict.pop("tags", [])
if authors:
for author_str in authors:
@ -329,6 +351,8 @@ class Book(LongPlayScrobblableMixin):
book.save_image_from_url(cover_url)
if genres:
book.genre.add(*genres)
if tags:
book.tags.add(*tags)
book.authors.add(*author_list)
if source_tag:
book.tags.add(source_tag.value)
@ -366,9 +390,14 @@ class Book(LongPlayScrobblableMixin):
data = lookup_comic_from_locg(str(self.title))
if not data and COMICVINE_API_KEY:
logger.warn(f"Checking ComicVine for {self.title}")
cv_client = ComicVineClient(api_key=COMICVINE_API_KEY)
data = lookup_comic_from_comicvine(str(self.title))
if self.comicvine_id:
logger.warn(
f"Checking ComicVine by ID for {self.title}"
)
data = lookup_issue_by_comicvine_id(str(self.comicvine_id))
if not data:
logger.warn(f"Checking ComicVine for {self.title}")
data = lookup_comic_from_comicvine(str(self.title))
if not data:
logger.warn(f"Book not found in any sources: {self.title}")
@ -406,10 +435,10 @@ class Book(LongPlayScrobblableMixin):
)
data.pop("pages")
# Pop this, so we can look it up later
# Pop these so they don't get passed to update()
cover_url = data.pop("cover_url", "")
subject_key_list = data.pop("subject_key_list", [])
tags = data.pop("tags", [])
# Fun trick for updating all fields at once
Book.objects.filter(pk=self.id).update(**data)
@ -417,6 +446,8 @@ class Book(LongPlayScrobblableMixin):
if subject_key_list:
self.genre.add(*subject_key_list)
if tags:
self.tags.add(*tags)
if cover_url:
r = requests.get(cover_url)
@ -462,8 +493,11 @@ class Book(LongPlayScrobblableMixin):
if scrobble.logdata.page_data:
for page, data in scrobble.logdata.page_data.items():
if convert_timestamps:
data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
tz = None
if scrobble.timezone:
tz = ZoneInfo(scrobble.timezone)
data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
pages[page] = data
sorted_pages = OrderedDict(
sorted(pages.items(), key=lambda x: x[1]["start_ts"])

View File

@ -17,8 +17,10 @@ class ComicVineClient(object):
account on https://comicvine.gamespot.com/ in order to obtain an API key.
"""
# All API requests made by this client will be made to this URL.
API_URL = "https://www.comicvine.com/api/search/"
# All API requests made by this client will be made to these URLs.
API_URL = "https://comicvine.gamespot.com/api/search/"
ISSUE_API_URL = "https://comicvine.gamespot.com/api/issue/4000-{issue_id}/"
VOLUME_API_URL = "https://comicvine.gamespot.com/api/volume/4050-{volume_id}/"
# A valid User-Agent header must be set in order for our API requests to
# be accepted, otherwise our request will be rejected with a
@ -41,15 +43,12 @@ class ComicVineClient(object):
"volume",
}
def __init__(self, api_key, expire_after=300):
def __init__(self, api_key):
"""
Store the API key in a class variable, and install the requests cache,
configuring it using the ``expire_after`` parameter.
Store the API key in a class variable.
:param api_key: Your personal ComicVine API key.
:type api_key: str
:param expire_after: The number of seconds to retain an entry in cache.
:type expire_after: int or None
"""
self.api_key = api_key
@ -109,14 +108,17 @@ class ComicVineClient(object):
:rtype: dict
"""
return {
params = {
"api_key": self.api_key,
"format": "json",
"limit": min(10, limit), # hard limit of 10
"offset": max(0, offset), # cannot provide negative offset
"query": query,
"resources": self._validate_resources(resources),
}
validated = self._validate_resources(resources)
if validated:
params["resources"] = validated
return params
def _validate_resources(self, resources):
"""
@ -141,33 +143,35 @@ class ComicVineClient(object):
def _query_api(self, params):
"""
Query the ComicVine API's ``search`` resource, providing the required
headers and parameters with the request. Optionally allow the caller
of the function to disable the request cache.
headers and parameters with the request.
If an error occurs during the request, handle it accordingly. Upon
success, return the JSON from the response.
:param params: Parameters to include with the request.
:type params: dict
:param use_cache: Toggle the use of requests_cache.
:type use_cache: bool
:return: The JSON contained in the response.
:rtype: dict
"""
# Since we're performing the identical action regardless of whether
# or not the request cache is to be used, store the procedure in a
# local function to avoid repetition.
def __httpget():
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
if not response.ok:
self._handle_http_error(response)
return response.json()
json_data = response.json()
return __httpget()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data
def _handle_http_error(self, response):
"""
@ -195,15 +199,81 @@ class ComicVineClient(object):
raise exception(message)
def get_issue(self, issue_id: str) -> dict:
"""
Fetch a single issue by its ComicVine ID directly from the issue detail
endpoint, which returns richer data than the search endpoint.
:param issue_id: The ComicVine numeric ID for the issue (e.g. "538480")
:type issue_id: str
:return: The full JSON response for the issue, or empty dict on failure.
:rtype: dict
"""
params = {
"api_key": self.api_key,
"format": "json",
}
url = self.ISSUE_API_URL.format(issue_id=issue_id)
response = requests.get(url, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
json_data = response.json()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data.get("results", {})
def get_volume(self, volume_id: str) -> dict:
"""
Fetch a single volume by its ComicVine ID from the volume detail
endpoint. Used to get publisher info and other volume-level metadata.
:param volume_id: The ComicVine numeric ID for the volume (e.g. "91273")
:type volume_id: str
:return: The full JSON response for the volume, or empty dict on failure.
:rtype: dict
"""
params = {
"api_key": self.api_key,
"format": "json",
}
url = self.VOLUME_API_URL.format(volume_id=volume_id)
response = requests.get(url, headers=self.HEADERS, params=params)
if not response.ok:
self._handle_http_error(response)
json_data = response.json()
if json_data.get("status_code") != 1:
error_msg = json_data.get("error", "Unknown ComicVine API error")
logger.error(
"ComicVine API returned status_code %s: %s",
json_data.get("status_code"),
error_msg,
)
return {}
return json_data.get("results", {})
def lookup_comic_from_comicvine(title: str) -> dict:
original_title = title
issue_number = None
volume_nubmer = None
resource_type = "issue"
if "Issue " in title:
resource_type = "issue"
issue_number = title.split("Issue ")[1]
volume_number = None
if "Volume " in title:
@ -215,48 +285,136 @@ def lookup_comic_from_comicvine(title: str) -> dict:
logger.warning("No ComicVine API key configured, not looking anything up")
return {}
client = ComicVineClient(api_key=getattr(settings, "COMICVINE_API_KEY", None))
client = ComicVineClient(api_key=api_key)
raw_results = client.search(title).get("results")
results = [r for r in raw_results if r.get("resource_type") == resource_type]
raw_results = client.search(title)
if not raw_results:
return {}
results = raw_results.get("results", [])
results = [r for r in results if r.get("resource_type") == resource_type]
if not results:
logger.warning("No comic found on ComicVine")
return {}
found_result = None
for result in results:
if result.get("issue_number") == str(issue_number):
if issue_number is not None and result.get("issue_number") == str(issue_number):
found_result = result
break
if result.get("volume_number") == str(volume_number):
if volume_number is not None and result.get("volume_number") == str(volume_number):
found_result = result
break
if not found_result:
found_result = results[0]
logger.info("ComicVine results", extra={"results": results})
data_dict = _build_data_dict_from_issue(found_result, original_title)
_enrich_with_volume_data(client, data_dict)
return data_dict
if not found_result:
logger.warning("No matches found on ComicVine")
def lookup_issue_by_comicvine_id(comicvine_id: str) -> dict:
"""
Look up an issue directly by its ComicVine ID using the issue detail
endpoint. Returns richer data than the search-based lookup.
:param comicvine_id: The ComicVine numeric ID for the issue (e.g. "538480")
:type comicvine_id: str
:return: A dict of extracted book metadata, or empty dict on failure.
:rtype: dict
"""
if not comicvine_id:
return {}
title = found_result.get("name")
api_key = getattr(settings, "COMICVINE_API_KEY", "")
if not api_key:
logger.warning("No ComicVine API key configured, not looking anything up")
return {}
if found_result.get("volume"):
title = found_result.get("volume").get("name")
client = ComicVineClient(api_key=api_key)
issue_data = client.get_issue(comicvine_id)
if not issue_data:
logger.warning("No issue found on ComicVine for ID %s", comicvine_id)
return {}
data_dict = _build_data_dict_from_issue(issue_data, issue_data.get("name", ""))
_enrich_with_volume_data(client, data_dict)
return data_dict
def _build_data_dict_from_issue(issue_data: dict, original_title: str = "") -> dict:
"""
Build a book metadata dict from a ComicVine issue resource (either from
search results or issue detail endpoint). Both return the same shape of
issue data.
:param issue_data: The issue resource dict from ComicVine.
:param original_title: The original search term, if any.
:return: A dict of extracted book metadata.
:rtype: dict
"""
title = issue_data.get("name")
if issue_data.get("volume"):
title = issue_data.get("volume").get("name")
cover_url = None
if issue_data.get("image"):
cover_url = issue_data["image"].get("original_url")
volume_name = None
volume_cv_id = None
publisher_name = None
volume_data = issue_data.get("volume")
if volume_data:
volume_name = volume_data.get("name")
volume_cv_id = volume_data.get("id")
publisher_data = volume_data.get("publisher")
if publisher_data:
publisher_name = publisher_data.get("name")
data_dict = {
"title": title,
"original_title": original_title,
"issue_number": found_result.get("issue_number"),
"volume_number": found_result.get("volume_number"),
"cover_url": found_result.get("image").get("original_url"),
"comicvine_id": found_result.get("id"),
"comicvine_data": found_result,
"summary": found_result.get("description"),
"publish_date": found_result.get("cover_date"),
"first_publish_year": found_result.get("cover_date", "")[:4],
"issue_number": issue_data.get("issue_number"),
"volume_number": issue_data.get("volume_number"),
"volume": volume_name,
"volume_comicvine_id": volume_cv_id,
"publisher": publisher_name,
"cover_url": cover_url,
"comicvine_id": issue_data.get("id"),
"summary": issue_data.get("description"),
"publish_date": issue_data.get("cover_date"),
"first_publish_year": (issue_data.get("cover_date") or "")[:4],
"tags": ["comicbook"],
}
return data_dict
def _enrich_with_volume_data(client: ComicVineClient, data_dict: dict) -> None:
"""
Follow-up a successful issue lookup by fetching the volume detail and
filling in publisher and other volume-level metadata that the issue
endpoint doesn't provide.
:param client: An initialised ComicVineClient instance.
:param data_dict: The data dict from an issue lookup (mutated in place).
"""
volume_cv_id = data_dict.get("volume_comicvine_id")
if not volume_cv_id:
return
volume_data = client.get_volume(str(volume_cv_id))
if not volume_data:
return
publisher_data = volume_data.get("publisher")
if publisher_data:
publisher_name = publisher_data.get("name")
if publisher_name and not data_dict.get("publisher"):
data_dict["publisher"] = publisher_name
if not data_dict.get("volume"):
data_dict["volume"] = volume_data.get("name")

View File

@ -26,8 +26,6 @@ def lookup_book_from_google(title: str) -> dict:
if not google_result:
return {}
publish_date = pendulum.parse(google_result.get("publishedDate"))
isbn_13 = ""
isbn_10 = ""
for ident in google_result.get("industryIdentifiers", []):
@ -35,25 +33,25 @@ def lookup_book_from_google(title: str) -> dict:
isbn_13 = ident.get("identifier")
if ident.get("type") == "ISBN_10":
isbn_10 = ident.get("identifier")
# TODO this may lead to issues with the first get if Google changes our title
# book_metadata.title = google_result.get("title")
# if google_result.get("subtitle"):
# book_metadata["title"] = ": ".join(
# [google_result.get("title"), google_result.get("subtitle")]
# )
# book_dict["subtitle"] = google_result.get("subtitle")
book_dict["authors"] = google_result.get("authors")
book_dict["publisher"] = google_result.get("publisher")
book_dict["first_publish_year"] = publish_date.year
book_dict["pages"] = google_result.get("pageCount")
book_dict["isbn_13"] = isbn_13
book_dict["isbn_10"] = isbn_10
book_dict["publish_date"] = google_result.get("publishedDate")
if len(book_dict["publish_date"]) == 4:
book_dict["publish_date"] = f"{book_dict['publish_date']}-1-1"
book_dict["language"] = google_result.get("language")
book_dict["summary"] = google_result.get("description")
book_dict["genres"] = google_result.get("categories")
raw_date = google_result.get("publishedDate")
if raw_date:
try:
publish_date = pendulum.parse(raw_date)
book_dict["first_publish_year"] = publish_date.year
except Exception:
pass
book_dict["publish_date"] = raw_date
if len(raw_date) == 4:
book_dict["publish_date"] = f"{raw_date}-1-1"
book_dict["cover_url"] = (
google_result.get("imageLinks", {})
.get("thumbnail", "")

View File

@ -32,8 +32,6 @@ class KoReaderBookRows:
DEFAULT_STR = "N/A"
DEFAULT_INT = 0
DEFAULT_TIME = 1703800469
BOOK_ROWS = []
PAGE_STATS_ROWS = []
def _gen_random_row(self, i):
wiggle = random.randrange(15)
@ -110,6 +108,8 @@ class KoReaderBookRows:
end_session = True
def __init__(self, book_count=0, **kwargs):
self.BOOK_ROWS = []
self.PAGE_STATS_ROWS = []
self._generate_random_book_rows(book_count)
self._generate_custom_book_row(**kwargs)
self._generate_random_page_stats_rows()

View File

@ -44,7 +44,10 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
# The test data generator adds the session-gap 3600s AFTER the trigger page
# (not before), so the first session includes 21 pages (1-21), and each
# subsequent session has 20 until the last. The last page is now included
# in the final scrobble instead of being orphaned.
expected_scrobbles = 6 * len(book_map.keys())
assert len(scrobbles) == expected_scrobbles
assert len(scrobbles[0].logdata.page_data.keys()) == 21
@ -52,7 +55,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
assert len(scrobbles[2].logdata.page_data.keys()) == 20
assert len(scrobbles[3].logdata.page_data.keys()) == 20
assert len(scrobbles[4].logdata.page_data.keys()) == 20
assert len(scrobbles[5].logdata.page_data.keys()) == 18
assert len(scrobbles[5].logdata.page_data.keys()) == 19
def test_get_author_str_from_row():

View File

@ -0,0 +1,146 @@
# Generated by Django 4.2.29 on 2026-06-12 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("charts", "0001_initial"),
]
operations = [
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "album", "rank"],
name="charts_char_user_id_1adcde_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "track", "rank"],
name="charts_char_user_id_d18aab_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "video", "rank"],
name="charts_char_user_id_de9f0a_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "board_game", "rank"],
name="charts_char_user_id_d5d58f_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "book", "rank"],
name="charts_char_user_id_e877cf_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "food", "rank"],
name="charts_char_user_id_a0ad71_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "podcast", "rank"],
name="charts_char_user_id_846b80_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "trail", "rank"],
name="charts_char_user_id_54feba_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "album", "rank"],
name="charts_char_user_id_a3dc49_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "track", "rank"],
name="charts_char_user_id_4b01ab_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "video", "rank"],
name="charts_char_user_id_2ac9d2_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "board_game", "rank"],
name="charts_char_user_id_ba968a_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "book", "rank"],
name="charts_char_user_id_e66751_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "food", "rank"],
name="charts_char_user_id_d23f06_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "podcast", "rank"],
name="charts_char_user_id_be8122_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "week", "trail", "rank"],
name="charts_char_user_id_b94ea9_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "artist", "rank"],
name="charts_char_user_id_406e0e_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "album", "rank"],
name="charts_char_user_id_322b0d_idx",
),
),
migrations.AddIndex(
model_name="chartrecord",
index=models.Index(
fields=["user", "year", "month", "day", "tv_series", "rank"],
name="charts_char_user_id_aa44b7_idx",
),
),
]

View File

@ -60,10 +60,29 @@ class ChartRecord(TimeStampedModel):
models.Index(fields=["user", "year", "geo_location", "rank"]),
models.Index(fields=["user", "year", "food", "rank"]),
models.Index(fields=["user", "year", "book", "rank"]),
models.Index(fields=["user", "year", "week", "artist", "rank"]),
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
models.Index(fields=["user", "year", "month", "artist", "rank"]),
models.Index(fields=["user", "year", "month", "album", "rank"]),
models.Index(fields=["user", "year", "month", "track", "rank"]),
models.Index(fields=["user", "year", "month", "tv_series", "rank"]),
models.Index(fields=["user", "year", "month", "video", "rank"]),
models.Index(fields=["user", "year", "month", "board_game", "rank"]),
models.Index(fields=["user", "year", "month", "book", "rank"]),
models.Index(fields=["user", "year", "month", "food", "rank"]),
models.Index(fields=["user", "year", "month", "podcast", "rank"]),
models.Index(fields=["user", "year", "month", "trail", "rank"]),
models.Index(fields=["user", "year", "week", "artist", "rank"]),
models.Index(fields=["user", "year", "week", "album", "rank"]),
models.Index(fields=["user", "year", "week", "track", "rank"]),
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
models.Index(fields=["user", "year", "week", "video", "rank"]),
models.Index(fields=["user", "year", "week", "board_game", "rank"]),
models.Index(fields=["user", "year", "week", "book", "rank"]),
models.Index(fields=["user", "year", "week", "food", "rank"]),
models.Index(fields=["user", "year", "week", "podcast", "rank"]),
models.Index(fields=["user", "year", "week", "trail", "rank"]),
models.Index(fields=["user", "year", "month", "day", "artist", "rank"]),
models.Index(fields=["user", "year", "month", "day", "album", "rank"]),
models.Index(fields=["user", "year", "month", "day", "tv_series", "rank"]),
]
@property

View File

@ -7,7 +7,10 @@ register = template.Library()
def get_item(dictionary, key):
if isinstance(dictionary, dict):
return dictionary.get(key)
return None
try:
return dictionary[int(key)]
except (IndexError, KeyError, TypeError, ValueError):
return None
@register.filter

View File

@ -114,108 +114,106 @@ class ChartRecordView(TemplateView):
context["current_week"] = current_week
context["current_day"] = current_day
if chart_type == "maloja":
context["chart_keys"] = {
"today": "Today",
"week": "This Week",
"month": "This Month",
"year": "This Year",
"all": "All Time",
}
context["chart_keys"] = {
"today": "Today",
"week": "This Week",
"month": "This Month",
"year": "This Year",
"all": "All Time",
}
context["maloja_charts"] = {
"artist": {
"today": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "artist", year=current_year)
),
"all": list(self.get_charts_for_period(user, "artist")),
},
"track": {
"today": list(
self.get_charts_for_period(
user,
"track",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user, "track", year=current_year, week=current_week
)
),
"month": list(
self.get_charts_for_period(
user,
"track",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "track", year=current_year)
),
"all": list(self.get_charts_for_period(user, "track")),
},
"tv_series": {
"today": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "tv_series", year=current_year)
),
"all": list(self.get_charts_for_period(user, "tv_series")),
},
}
return context
context["maloja_charts"] = {
"artist": {
"today": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"artist",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "artist", year=current_year)
),
"all": list(self.get_charts_for_period(user, "artist")),
},
"album": {
"today": list(
self.get_charts_for_period(
user,
"album",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user, "album", year=current_year, week=current_week
)
),
"month": list(
self.get_charts_for_period(
user,
"album",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "album", year=current_year)
),
"all": list(self.get_charts_for_period(user, "album")),
},
"tv_series": {
"today": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
day=current_day,
)
),
"week": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
week=current_week,
)
),
"month": list(
self.get_charts_for_period(
user,
"tv_series",
year=current_year,
month=current_month,
)
),
"year": list(
self.get_charts_for_period(user, "tv_series", year=current_year)
),
"all": list(self.get_charts_for_period(user, "tv_series")),
},
}
if not date_param:
context["period"] = "current"
@ -440,12 +438,14 @@ class ChartRecordView(TemplateView):
return context
def get_available_years(self, user):
return list(
ChartRecord.objects.filter(user=user)
.values_list("year", flat=True)
.distinct()
.order_by("-year")
)
if not hasattr(self, "_available_years"):
self._available_years = list(
ChartRecord.objects.filter(user=user)
.values_list("year", flat=True)
.distinct()
.order_by("-year")
)
return self._available_years
def get_period_type(self):
date_param = self.request.GET.get("date")

View File

@ -21,6 +21,7 @@ class FoodAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("category",)
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -0,0 +1,127 @@
import csv
import logging
from django.core.management.base import BaseCommand
logger = logging.getLogger(__name__)
def _get_source(raw_data):
if "Artist" in raw_data:
return "Jellyfin"
if "artist" in raw_data:
return "Mopidy"
return None
def _get_raw_values(raw_data, source):
if source == "Jellyfin":
return raw_data.get("Artist", ""), raw_data.get("Album", "")
return raw_data.get("artist", ""), raw_data.get("album", "")
def _normalize(name):
return name.strip().casefold()
def _artist_mismatch(raw_artist, track_artist_names):
if not raw_artist or not track_artist_names:
return False
track_names = [_normalize(n) for n in track_artist_names.split(" / ")]
raw = _normalize(raw_artist)
if raw in track_names:
return False
if raw == _normalize(track_artist_names):
return False
return True
def _album_mismatch(raw_album, track_album_name):
if not raw_album or not track_album_name:
return False
return _normalize(raw_album) != _normalize(track_album_name)
class Command(BaseCommand):
help = (
"Outputs a CSV of track IDs where raw metadata from scrobble logs "
"does not match the track's stored artists or album"
)
def add_arguments(self, parser):
parser.add_argument(
"--file-path",
type=str,
default="/tmp/metadata-report.csv",
help="Output CSV file path (default: /tmp/metadata-report.csv)",
)
def handle(self, *args, **options):
from scrobbles.models import Scrobble
file_path = options["file_path"]
qs = (
Scrobble.objects.filter(media_type=Scrobble.MediaType.TRACK)
.exclude(log__isnull=True)
.exclude(log={})
.select_related("track__album")
.prefetch_related("track__artists")
.iterator()
)
rows = []
for scrobble in qs:
track = scrobble.track
if not track:
continue
raw_data = scrobble.log.get("raw_data")
if not raw_data:
continue
source = _get_source(raw_data)
if not source:
continue
raw_artist, raw_album = _get_raw_values(raw_data, source)
if not raw_artist and not raw_album:
continue
track_artist_names = " / ".join(
track.artists.all().values_list("name", flat=True)
)
track_album_name = track.album.name if track.album else ""
if _artist_mismatch(raw_artist, track_artist_names) or _album_mismatch(
raw_album, track_album_name
):
rows.append(
{
"track_id": track.id,
"track_artist_name": track_artist_names,
"track_album_name": track_album_name,
"raw_artist": raw_artist,
"raw_album": raw_album,
"source": source,
}
)
fieldnames = [
"track_id",
"track_artist_name",
"track_album_name",
"raw_artist",
"raw_album",
"source",
]
with open(file_path, "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=fieldnames)
writer.writeheader()
writer.writerows(rows)
self.stdout.write(
self.style.SUCCESS(
f"Wrote {len(rows)} mismatched track(s) to {file_path}"
)
)

View File

@ -6,6 +6,7 @@ from people.models import Person, PersonScrobble
class PersonAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "bgg_username", "bgstats_id")
raw_id_fields = ("user", "created_by")
ordering = ("-created",)
search_fields = ("name",)

View File

@ -18,6 +18,7 @@ class PodcastAdmin(admin.ModelAdmin):
"producer",
"active",
)
raw_id_fields = ("producer",)
ordering = ("name",)
@ -28,6 +29,7 @@ class PodcastEpisodeAdmin(admin.ModelAdmin):
"title",
"podcast",
)
raw_id_fields = ("podcast",)
list_filter = ("podcast",)
ordering = ("-created",)
inlines = [

View File

@ -6,6 +6,7 @@ from profiles.models import UserProfile
@admin.register(UserProfile)
class UserProfileAdmin(admin.ModelAdmin):
date_hierarchy = "created"
raw_id_fields = ("user",)
ordering = ("-created",)
readonly_fields = ("timezone_change_log",)
exclude = (

View File

@ -1,6 +1,8 @@
from django import forms
from profiles.models import UserProfile
from scrobbles.constants import Visibility
from scrobbles.models import Scrobble
class UserProfileForm(forms.ModelForm):
@ -45,3 +47,51 @@ class UserProfileForm(forms.ModelForm):
"archivebox_password": forms.PasswordInput(render_value=True),
"webdav_pass": forms.PasswordInput(render_value=True),
}
MEDIA_TYPE_LABELS = {
mt.value: mt.label for mt in Scrobble.MediaType
}
INHERIT = ""
class BulkVisibilityForm(forms.Form):
bulk_action = forms.ChoiceField(
choices=[
(Visibility.PUBLIC, "Public"),
(Visibility.PRIVATE, "Private"),
],
widget=forms.RadioSelect,
required=False,
label="Set all non-shared scrobbles to",
)
def __init__(self, *args, **kwargs):
self.profile = kwargs.pop("profile")
super().__init__(*args, **kwargs)
media_types = Scrobble.MediaType.values
choices = [
(Visibility.PUBLIC, "Public"),
(Visibility.SHARED, "Shared"),
(Visibility.PRIVATE, "Private"),
]
existing_overrides = self.profile.media_type_visibility or {}
for mt in sorted(media_types):
label = MEDIA_TYPE_LABELS.get(mt, mt)
self.fields[f"media_type_{mt}"] = forms.ChoiceField(
choices=choices,
required=False,
label=label,
initial=existing_overrides.get(mt, Visibility.PRIVATE),
)
def clean(self):
cleaned = super().clean()
overrides = {}
for mt in Scrobble.MediaType.values:
val = cleaned.get(f"media_type_{mt}")
if val:
overrides[mt] = val
cleaned["media_type_visibility"] = overrides
return cleaned

View File

@ -0,0 +1,26 @@
# Generated by Django 4.2.29 on 2026-06-09 15:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0035_userprofile_monthly_mopidy_playlist_pattern"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="default_scrobble_visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
default="shared",
max_length=10,
),
),
]

View File

@ -0,0 +1,26 @@
# Generated by Django 4.2.29 on 2026-06-09 16:25
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0036_userprofile_default_scrobble_visibility"),
]
operations = [
migrations.AlterField(
model_name="userprofile",
name="default_scrobble_visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
default="private",
max_length=10,
),
),
]

View File

@ -0,0 +1,20 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("profiles", "0037_alter_userprofile_default_scrobble_visibility"),
]
operations = [
migrations.AddField(
model_name="userprofile",
name="media_type_visibility",
field=models.JSONField(
blank=True,
default=dict,
help_text='Per-media-type visibility overrides, e.g. {"Video": "public", "Track": "private"}',
),
),
]

View File

@ -9,6 +9,11 @@ from django.utils.functional import cached_property
from django_extensions.db.models import TimeStampedModel
from encrypted_field import EncryptedField
from profiles.constants import PRETTY_TIMEZONE_CHOICES
VISIBILITY_CHOICES = (
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
)
User = get_user_model()
BNULL = {"blank": True, "null": True}
@ -79,6 +84,18 @@ class UserProfile(TimeStampedModel):
enable_public_widgets = models.BooleanField(default=False)
widget_custom_css = models.TextField(**BNULL)
default_scrobble_visibility = models.CharField(
max_length=10,
choices=VISIBILITY_CHOICES,
default="private",
)
media_type_visibility = models.JSONField(
default=dict,
blank=True,
help_text="Per-media-type visibility overrides, e.g. {\"Video\": \"public\", \"Track\": \"private\"}",
)
home_scrobble_limit = models.IntegerField(default=20)
weigh_in_units = models.CharField(
@ -123,6 +140,11 @@ class UserProfile(TimeStampedModel):
return history
def get_timestamp_with_tz(self, timestamp):
from django.conf import settings
server_tz = ZoneInfo(settings.TIME_ZONE)
ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
timezone = self.tzinfo
if self.timezone_change_log:
change_list = self.historic_timezone_changes
@ -133,13 +155,13 @@ class UserProfile(TimeStampedModel):
end = None
if end:
if start <= timestamp.replace(tzinfo=end.timezone) <= end:
if start <= ref_dt <= end:
timezone = start.timezone
else:
if start <= timestamp.replace(tzinfo=start.timezone):
if start <= ref_dt:
timezone = start.timezone
return timestamp.replace(tzinfo=timezone)
return ref_dt.astimezone(timezone)
def adjust_timezone_of_scrobbles(self, commit=False):
current_dt = None

View File

@ -6,4 +6,9 @@ app_name = "profiles"
urlpatterns = [
path("settings/", views.ProfileFormView.as_view(), name="profile_settings"),
path(
"settings/visibility/",
views.BulkVisibilityView.as_view(),
name="bulk_visibility",
),
]

View File

@ -1,8 +1,12 @@
from django.contrib import messages
from django.contrib.auth.mixins import LoginRequiredMixin
from django.db.models import Count, Q
from django.http.response import HttpResponseBadRequest
from django.urls import reverse_lazy
from django.views.generic import FormView
from profiles.forms import UserProfileForm
from profiles.forms import BulkVisibilityForm, UserProfileForm
from scrobbles.constants import Visibility
from scrobbles.models import Scrobble
from tasks.todoist import generate_todoist_oauth_url
@ -30,3 +34,46 @@ class ProfileFormView(LoginRequiredMixin, FormView):
context["profile"] = self.request.user.profile
context["todoist_oauth_url"] = generate_todoist_oauth_url(self.request.user.id)
return context
class BulkVisibilityView(LoginRequiredMixin, FormView):
template_name = "profiles/visibility_settings.html"
form_class = BulkVisibilityForm
success_url = reverse_lazy("profiles:bulk_visibility")
def get_form_kwargs(self):
kwargs = super().get_form_kwargs()
kwargs["profile"] = self.request.user.profile
return kwargs
def form_valid(self, form):
request = self.request
profile = request.user.profile
bulk_action = form.cleaned_data.get("bulk_action")
if bulk_action:
qs = Scrobble.objects.filter(
user=request.user,
).exclude(visibility=Visibility.SHARED)
total = qs.count()
qs.update(visibility=bulk_action)
messages.success(
request,
f"Updated {total} scrobble(s) to {bulk_action}.",
)
profile.media_type_visibility = form.cleaned_data["media_type_visibility"]
profile.save(update_fields=["media_type_visibility"])
messages.success(request, "Per-media-type visibility overrides saved.")
return super().form_valid(form)
def get_context_data(self, **kwargs):
ctx = super().get_context_data(**kwargs)
profile = self.request.user.profile
qs = Scrobble.objects.filter(user=self.request.user)
ctx["scrobble_count"] = qs.count()
ctx["visibility_counts"] = qs.values("visibility").annotate(
count=Count("id")
)
ctx["profile"] = profile
return ctx

View File

@ -21,6 +21,7 @@ class PuzzleAdmin(admin.ModelAdmin):
"uuid",
"title",
)
raw_id_fields = ("manufacturer",)
ordering = ("-created",)
search_fields = ("title",)
inlines = [

View File

@ -10,6 +10,7 @@ from scrobbles.models import (
RetroarchImport,
ScaleCSVImport,
Scrobble,
ShareViewLog,
TrailGPXImport,
)
from scrobbles.mixins import Genre
@ -18,33 +19,11 @@ from scrobbles.mixins import Genre
class ScrobbleInline(admin.TabularInline):
model = Scrobble
extra = 0
raw_id_fields = (
"video",
"podcast_episode",
"track",
"video_game",
"book",
"paper",
"sport_event",
"food",
"board_game",
"geo_location",
"task",
"mood",
"brick_set",
"trail",
"beer",
"web_page",
"life_event",
"birding_location",
"user",
)
exclude = (
"scrobble_log",
"timezone",
"videogame_save_data",
"screenshot",
)
per_page = 15
ordering = ("-timestamp",)
show_change_link = True
fields = ("timestamp", "media_type", "source", "in_progress")
readonly_fields = fields
class ImportBaseAdmin(admin.ModelAdmin):
@ -54,52 +33,44 @@ class ImportBaseAdmin(admin.ModelAdmin):
"process_count",
"processed_finished",
"processing_started",
"error_log",
)
ordering = ("-created",)
@admin.register(AudioScrobblerTSVImport)
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin):
...
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin): ...
@admin.register(LastFmImport)
class LastFmImportAdmin(ImportBaseAdmin):
...
class LastFmImportAdmin(ImportBaseAdmin): ...
@admin.register(KoReaderImport)
class KoReaderImportAdmin(ImportBaseAdmin):
...
class KoReaderImportAdmin(ImportBaseAdmin): ...
@admin.register(RetroarchImport)
class RetroarchImportAdmin(ImportBaseAdmin):
...
class RetroarchImportAdmin(ImportBaseAdmin): ...
class RetroarchImportAdmin(ImportBaseAdmin):
...
class RetroarchImportAdmin(ImportBaseAdmin): ...
@admin.register(BGStatsImport)
class BGStatsImportAdmin(ImportBaseAdmin):
...
class BGStatsImportAdmin(ImportBaseAdmin): ...
@admin.register(EBirdCSVImport)
class EBirdCSVImportAdmin(ImportBaseAdmin):
...
class EBirdCSVImportAdmin(ImportBaseAdmin): ...
@admin.register(ScaleCSVImport)
class ScaleCSVImportAdmin(ImportBaseAdmin):
...
class ScaleCSVImportAdmin(ImportBaseAdmin): ...
@admin.register(TrailGPXImport)
class TrailGPXImportAdmin(ImportBaseAdmin):
...
class TrailGPXImportAdmin(ImportBaseAdmin): ...
@admin.register(Genre)
@ -122,10 +93,13 @@ class ScrobbleAdmin(admin.ModelAdmin):
"in_progress",
"is_paused",
"played_to_completion",
"visibility",
"user",
)
raw_id_fields = (
"user",
"video",
"channel",
"podcast_episode",
"track",
"sport_event",
@ -144,17 +118,20 @@ class ScrobbleAdmin(admin.ModelAdmin):
"web_page",
"life_event",
"birding_location",
"long_play_last_scrobble",
)
list_filter = (
"is_paused",
"in_progress",
"media_type",
"visibility",
"long_play_complete",
"source",
"timezone",
"user",
)
ordering = ("-timestamp",)
readonly_fields = ("share_token_version", "share_view_count")
def media_name(self, obj):
return obj.media_obj
@ -167,20 +144,33 @@ class ScrobbleAdmin(admin.ModelAdmin):
return qs
@admin.register(ShareViewLog)
class ShareViewLogAdmin(admin.ModelAdmin):
list_display = ("scrobble", "ip_address", "created")
list_filter = ("created",)
date_hierarchy = "created"
raw_id_fields = ("scrobble",)
@admin.register(FavoriteMedia)
class FavoriteMediaAdmin(admin.ModelAdmin):
list_display = ("user", "media_type", "sent_to_mopidy", "created")
list_filter = ("media_type", "sent_to_mopidy", "user")
date_hierarchy = "created"
raw_id_fields = (
"user",
"video",
"channel",
"track",
"podcast_episode",
"sport_event",
"book",
"paper",
"video_game",
"board_game",
"geo_location",
"puzzle",
"food",
"task",
"mood",
"brick_set",

View File

@ -1,6 +1,7 @@
import re
from rest_framework import serializers
from scrobbles.constants import Visibility
from scrobbles.models import (
AudioScrobblerTSVImport,
KoReaderImport,

View File

@ -1,6 +1,8 @@
from logging import getLogger
from rest_framework import permissions, viewsets
from rest_framework.decorators import action
from rest_framework.response import Response
from scrobbles.api.serializers import (
AudioScrobblerTSVImportSerializer,
KoReaderImportSerializer,
@ -26,6 +28,12 @@ class ScrobbleViewSet(viewsets.ModelViewSet):
def get_queryset(self):
return super().get_queryset().filter(user=self.request.user)
@action(detail=True, methods=["post"])
def regenerate_share_token(self, request, uuid=None):
scrobble = self.get_object()
scrobble.regenerate_share_token()
return Response({"share_url": scrobble.get_share_url()})
class KoReaderImportViewSet(viewsets.ModelViewSet):
queryset = KoReaderImport.objects.all().order_by("-created")

View File

@ -1,6 +1,12 @@
from django.db import models
from enum import Enum
JELLYFIN_VIDEO_ITEM_TYPES = ["Episode", "Movie"]
class Visibility(models.TextChoices):
PUBLIC = "public", "Public"
SHARED = "shared", "Shared"
PRIVATE = "private", "Private"
JELLYFIN_AUDIO_ITEM_TYPES = ["Audio"]
LONG_PLAY_MEDIA = {

View File

@ -112,9 +112,7 @@ class BaseLogData(JSONDataclass):
continue
if dt is None:
m = re.match(
r"(\d{4}-\d{2}-\d{2})\s+\w{3}\s+(\d{2}:\d{2})", cleaned
)
m = re.match(r"(\d{4}-\d{2}-\d{2})\s+\w{3}\s+(\d{2}:\d{2})", cleaned)
if m:
try:
dt = datetime.strptime(
@ -143,9 +141,25 @@ class BaseLogData(JSONDataclass):
md = markdown.Markdown(extensions=["extra"])
allowed_tags = [
"p", "br", "strong", "em", "a", "ul", "ol", "li",
"code", "pre", "blockquote", "h1", "h2", "h3", "h4", "h5", "h6",
"hr", "img",
"p",
"br",
"strong",
"em",
"a",
"ul",
"ol",
"li",
"code",
"pre",
"blockquote",
"h1",
"h2",
"h3",
"h4",
"h5",
"h6",
"hr",
"img",
]
note_items = []
@ -174,7 +188,9 @@ class BaseLogData(JSONDataclass):
ts_html = ""
if ts:
ts_html = f'<h5 class="note-timestamp">{self._format_timestamp(ts)}</h5>'
ts_html = (
f'<h5 class="note-timestamp">{self._format_timestamp(ts)}</h5>'
)
content_html = bleach.clean(
md.convert(text),
@ -191,7 +207,15 @@ class BaseLogData(JSONDataclass):
@dataclass
class LongPlayLogData(JSONDataclass):
long_play_complete: bool = False
pass
@dataclass
class SportEventLogData(BaseLogData):
thesportsdb_id: Optional[str] = None
start: Optional[str] = None
round_name: Optional[str] = None
season_name: Optional[str] = None
@dataclass

View File

@ -76,6 +76,7 @@ class LastFM:
in_progress=False,
media_type=Scrobble.MediaType.TRACK,
timezone=tz_timestamp.tzinfo.name,
visibility="private",
)
# Vrobbler scrobbles on finish, LastFM scrobbles on start
seconds_eariler = timestamp - timedelta(seconds=20)

View File

@ -111,6 +111,7 @@ def import_scale_csv(file_path, user_id):
played_to_completion=True,
in_progress=False,
media_type=Scrobble.MediaType.TASK,
visibility="private",
)
new_scrobbles.append(new_scrobble)

View File

@ -305,6 +305,7 @@ def import_trail_gpx(file_path, user_id, original_filename=None):
played_to_completion=True,
in_progress=False,
media_type=Scrobble.MediaType.TRAIL,
visibility="private",
)
_, ext = os.path.splitext(file_path)

View File

@ -80,6 +80,7 @@ def import_audioscrobbler_tsv_file(file_path, user_id):
in_progress=False,
media_type=Scrobble.MediaType.TRACK,
timezone=timestamp.tzinfo.name,
visibility="private",
)
existing = Scrobble.objects.filter(
timestamp=timestamp, track=track, user=user

View File

@ -153,7 +153,7 @@ def scan_webdav_for_koreader(
)
if update_etag_only:
if last_import and remote_etag:
if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
last_import.webdav_etag = remote_etag
last_import.save(update_fields=["webdav_etag"])
logger.info(
@ -163,7 +163,7 @@ def scan_webdav_for_koreader(
)
return 0
if last_import and last_import.webdav_etag and remote_etag:
if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
if last_import.webdav_etag == remote_etag:
logger.info(
"koreader stats file unchanged (ETag match)",

View File

@ -0,0 +1,68 @@
from django.core.management.base import BaseCommand
from django.db import models
from scrobbles.models import Scrobble
from scrobbles.utils import analyze_scrobble_sentiment
class Command(BaseCommand):
help = "Backfill VADER sentiment analysis for scrobbles with notes"
def add_arguments(self, parser):
parser.add_argument(
"--commit",
action="store_true",
help="Actually update scrobble logs with sentiment data",
)
parser.add_argument(
"--overwrite",
action="store_true",
help="Re-analyze scrobbles that already have sentiment data",
)
def handle(self, *args, **options):
commit = options["commit"]
overwrite = options["overwrite"]
qs = Scrobble.objects.filter(
~models.Q(log__notes__isnull=True)
& ~models.Q(log__notes=[])
& ~models.Q(log__notes={})
)
if not overwrite:
qs = qs.filter(log__sentiment__isnull=True)
total = qs.count()
analyzed_count = 0
skipped_count = 0
self.stdout.write(f"Found {total} scrobbles to process")
for scrobble in qs.iterator():
if commit:
analyzed = analyze_scrobble_sentiment(scrobble, overwrite=overwrite)
else:
notes_str = ""
if scrobble.logdata:
notes_str = scrobble.logdata.notes_as_str()
analyzed = bool(notes_str)
if analyzed:
analyzed_count += 1
if commit:
scores = (scrobble.log or {}).get("sentiment", {})
self.stdout.write(
f" Updated scrobble {scrobble.id}: compound={scores.get('compound', 'N/A')}"
)
else:
self.stdout.write(
f" [DRY RUN] Would analyze scrobble {scrobble.id}"
)
else:
skipped_count += 1
self.stdout.write(
f"\nAnalyzed {analyzed_count} scrobbles, skipped {skipped_count}"
)
if not commit:
self.stdout.write("Run with --commit to persist changes")

View File

@ -0,0 +1,146 @@
from django.core.management.base import BaseCommand
from django.db import connection
from scrobbles.constants import LONG_PLAY_MEDIA
from scrobbles.models import Scrobble
class Command(BaseCommand):
help = (
"Backfill long_play_last_scrobble FK chains, then recompute "
"long_play_seconds by walking backward through the chain."
)
def add_arguments(self, parser):
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be changed without making changes",
)
parser.add_argument(
"--media-type",
type=str,
help="Only process a specific media type (e.g., Book, VideoGame)",
)
def handle(self, *args, **options):
dry_run = options["dry_run"]
media_type = options.get("media_type")
media_types = list(LONG_PLAY_MEDIA.values())
if media_type:
if media_type not in media_types:
self.stdout.write(
self.style.ERROR(
f"Invalid media type '{media_type}'. "
f"Valid: {', '.join(media_types)}"
)
)
return
media_types = [media_type]
# Step 1: backfill long_play_last_scrobble
self.stdout.write("Step 1: Backfilling long_play_last_scrobble chains...")
total_backfilled = 0
for mt in media_types:
n = self._backfill_chain(mt, dry_run)
total_backfilled += n
self.stdout.write(f" {mt}: {n} scrobbles linked")
if dry_run:
self.stdout.write(
self.style.WARNING(
f"Would backfill {total_backfilled} scrobbles total. "
"Run without --dry-run to apply."
)
)
else:
self.stdout.write(
self.style.SUCCESS(f"Backfilled {total_backfilled} scrobbles")
)
# Step 2: recompute long_play_seconds
self.stdout.write(
"\nStep 2: Recomputing long_play_seconds via FK chain..."
)
scrobbles = Scrobble.objects.filter(
media_type__in=media_types,
playback_position_seconds__isnull=False,
).order_by("-timestamp")
total = scrobbles.count()
self.stdout.write(f" Found {total} long play scrobbles to process")
to_update = []
for scrobble in scrobbles.iterator():
accumulated = scrobble.playback_position_seconds or 0
current = scrobble.long_play_last_scrobble
while current and not current.long_play_complete:
accumulated += current.playback_position_seconds or 0
current = current.long_play_last_scrobble
if scrobble.long_play_seconds != accumulated:
self.stdout.write(
f" Scrobble {scrobble.id} ({scrobble.media_type}): "
f"{scrobble.long_play_seconds or 0} -> {accumulated}"
)
if not dry_run:
scrobble.long_play_seconds = accumulated
to_update.append(scrobble)
if to_update:
Scrobble.objects.bulk_update(to_update, ["long_play_seconds"])
if dry_run:
self.stdout.write(
self.style.WARNING(
f"Dry run: would update {len(to_update)} scrobbles. "
"Use without --dry-run to apply."
)
)
else:
self.stdout.write(
self.style.SUCCESS(f"Updated {len(to_update)} scrobbles")
)
def _backfill_chain(self, media_type: str, dry_run: bool) -> int:
"""Set long_play_last_scrobble on each scrobble to the previous
scrobble for the same media+user using a single UPDATE with a
correlated subquery."""
fk = _media_type_to_fk(media_type)
table = Scrobble._meta.db_table
if dry_run:
with connection.cursor() as cursor:
cursor.execute(
f"SELECT COUNT(*) FROM {table} "
f"WHERE long_play_last_scrobble_id IS NULL "
f"AND {fk}_id IS NOT NULL"
)
return cursor.fetchone()[0]
with connection.cursor() as cursor:
cursor.execute(
f"UPDATE {table} "
f"SET long_play_last_scrobble_id = ("
f" SELECT id FROM {table} AS prev "
f" WHERE prev.{fk}_id = {table}.{fk}_id "
f" AND prev.user_id = {table}.user_id "
f" AND prev.timestamp < {table}.timestamp "
f" ORDER BY prev.timestamp DESC LIMIT 1"
f") "
f"WHERE long_play_last_scrobble_id IS NULL "
f"AND {fk}_id IS NOT NULL"
)
return cursor.rowcount
def _media_type_to_fk(media_type):
mapping = {
"VideoGame": "video_game",
"Book": "book",
"BrickSet": "brick_set",
"Task": "task",
}
return mapping.get(media_type)

View File

@ -0,0 +1,53 @@
# Generated by Django 4.2.29 on 2026-06-08 14:57
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0089_favoritemedia"),
]
operations = [
migrations.AddField(
model_name="audioscrobblertsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="bgstatsimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="ebirdcsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="koreaderimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="lastfmimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="retroarchimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="scalecsvimport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
migrations.AddField(
model_name="trailgpximport",
name="error_log",
field=models.TextField(blank=True, null=True),
),
]

View File

@ -0,0 +1,32 @@
# Generated by Django 4.2.29 on 2026-06-09 15:52
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0090_audioscrobblertsvimport_error_log_and_more"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="share_token",
field=models.UUIDField(blank=True, editable=False, null=True, unique=True),
),
migrations.AddField(
model_name="scrobble",
name="visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
db_index=True,
default="shared",
max_length=10,
),
),
]

View File

@ -0,0 +1,29 @@
from uuid import uuid4
from django.db import migrations
def backfill_share_token(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
batch = []
for scrobble in Scrobble.objects.filter(share_token__isnull=True).iterator(
chunk_size=500
):
scrobble.share_token = uuid4()
batch.append(scrobble)
if batch:
Scrobble.objects.bulk_update(batch, ["share_token"], batch_size=500)
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0091_scrobble_share_token_scrobble_visibility"),
]
operations = [
migrations.RunPython(
backfill_share_token,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,22 @@
# Generated by Django 4.2.29 on 2026-06-09 16:05
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0092_backfill_visibility_and_share_token"),
]
operations = [
migrations.RemoveField(
model_name="scrobble",
name="share_token",
),
migrations.AddField(
model_name="scrobble",
name="share_token_version",
field=models.PositiveIntegerField(default=0),
),
]

View File

@ -0,0 +1,75 @@
# Generated by Django 4.2.29 on 2026-06-09 16:24
from django.db import migrations, models
import django.db.models.deletion
import django_extensions.db.fields
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0093_remove_scrobble_share_token_and_more"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="share_view_count",
field=models.PositiveIntegerField(default=0),
),
migrations.AlterField(
model_name="scrobble",
name="visibility",
field=models.CharField(
choices=[
("public", "Public"),
("shared", "Shared"),
("private", "Private"),
],
db_index=True,
default="private",
max_length=10,
),
),
migrations.CreateModel(
name="ShareViewLog",
fields=[
(
"id",
models.BigAutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created",
django_extensions.db.fields.CreationDateTimeField(
auto_now_add=True, verbose_name="created"
),
),
(
"modified",
django_extensions.db.fields.ModificationDateTimeField(
auto_now=True, verbose_name="modified"
),
),
("ip_address", models.GenericIPAddressField(blank=True, null=True)),
("user_agent", models.TextField(blank=True, null=True)),
("referrer", models.URLField(blank=True, max_length=2048, null=True)),
(
"scrobble",
models.ForeignKey(
on_delete=django.db.models.deletion.CASCADE,
related_name="share_views",
to="scrobbles.scrobble",
),
),
],
options={
"get_latest_by": "modified",
"abstract": False,
},
),
]

View File

@ -0,0 +1,20 @@
from django.db import migrations
def backfill_null_visibility(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
Scrobble.objects.filter(visibility__isnull=True).update(visibility="private")
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0094_scrobble_share_view_count_alter_scrobble_visibility_and_more"),
]
operations = [
migrations.RunPython(
backfill_null_visibility,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,35 @@
from django.db import migrations
def convert_page_data_to_dict(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
log__page_data=None
):
page_data = scrobble.log.get("page_data")
if isinstance(page_data, list):
new_page_data = {}
for entry in page_data:
page_num = entry.get("page_number")
if page_num is not None:
try:
page_num = int(page_num)
except (ValueError, TypeError):
continue
new_page_data[page_num] = entry
scrobble.log["page_data"] = new_page_data
scrobble.save(update_fields=["log"])
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0095_backfill_null_visibility"),
]
operations = [
migrations.RunPython(
convert_page_data_to_dict,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -0,0 +1,19 @@
# Generated by Django 4.2.29 on 2026-06-12 16:09
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0096_convert_book_page_data_to_dict"),
]
operations = [
migrations.AddIndex(
model_name="scrobble",
index=models.Index(
fields=["user", "-timestamp"], name="scrobbles_s_user_id_d367a7_idx"
),
),
]

View File

@ -0,0 +1,25 @@
# Generated by Django 4.2.29 on 2026-06-15 17:48
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0097_scrobble_scrobbles_s_user_id_d367a7_idx"),
]
operations = [
migrations.AddField(
model_name="scrobble",
name="long_play_last_scrobble",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="next_long_play_scrobbles",
to="scrobbles.scrobble",
),
),
]

View File

@ -65,6 +65,9 @@ class ScrobblableMixin(TimeStampedModel):
class Meta:
abstract = True
def __str__(self) -> str:
return f"{self.title} - {self.subtitle}"
@property
def run_time_seconds(self) -> int:
run_time = 900
@ -162,23 +165,18 @@ class LongPlayScrobblableMixin(ScrobblableMixin):
return reverse("scrobbles:longplay-finish", kwargs={"uuid": self.uuid})
def first_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
return (
get_scrobbles_for_media(self, user)
.filter(
log__long_play_complete=False,
log__serial_scrobble_id__isnull=True,
)
.order_by("timestamp")
.first()
)
last = self.last_long_play_scrobble_for_user(user)
if not last:
return None
current = last
while current.long_play_last_scrobble and not current.long_play_last_scrobble.long_play_complete:
current = current.long_play_last_scrobble
return current
def last_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
return (
get_scrobbles_for_media(self, user)
.filter(
log__long_play_complete=False,
log__serial_scrobble_id__isnull=False,
)
.order_by("timestamp")
.last()
.filter(long_play_complete=False)
.order_by("-timestamp")
.first()
)

View File

@ -18,6 +18,8 @@ from bricksets.models import BrickSet
from charts.utils import build_charts
from dataclass_wizard.errors import ParseError
from django.conf import settings
from scrobbles.constants import Visibility
from scrobbles.sqids import encode_scrobble_share
from django.contrib.auth import get_user_model
from django.core.files import File
from django.db import models
@ -74,6 +76,7 @@ class BaseFileImportMixin(TimeStampedModel):
processed_finished = models.DateTimeField(**BNULL)
process_log = models.TextField(**BNULL)
process_count = models.IntegerField(**BNULL)
error_log = models.TextField(**BNULL)
class Meta:
abstract = True
@ -158,6 +161,14 @@ class BaseFileImportMixin(TimeStampedModel):
self.process_count = len(scrobbles)
self.save(update_fields=["process_log", "process_count"])
def record_error(self, error_message):
log_line = f"{timezone.now().isoformat()}: {error_message}"
if self.error_log:
self.error_log += "\n" + log_line
else:
self.error_log = log_line
self.save(update_fields=["error_log"])
@property
def upload_file_path(self):
raise NotImplementedError
@ -213,9 +224,16 @@ class KoReaderImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = process_koreader_sqlite_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class AudioScrobblerTSVImport(BaseFileImportMixin):
@ -255,10 +273,16 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_audioscrobbler_tsv_file(
self.upload_file_path, self.user.id
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class ScaleCSVImport(BaseFileImportMixin):
@ -297,9 +321,14 @@ class ScaleCSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class TrailGPXImport(BaseFileImportMixin):
@ -337,11 +366,16 @@ class TrailGPXImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_trail_gpx(
self.upload_file_path, self.user.id, self.original_filename
)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_trail_gpx(
self.upload_file_path, self.user.id, self.original_filename
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class LastFmImport(BaseFileImportMixin):
@ -391,11 +425,14 @@ class LastFmImport(BaseFileImportMixin):
last_processed = last_import.processed_finished
self.mark_started()
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class RetroarchImport(BaseFileImportMixin):
@ -426,43 +463,48 @@ class RetroarchImport(BaseFileImportMixin):
logger.info(f"You told me to force import from Retroarch")
self.mark_started()
scrobbles = None
try:
if self.lrtl_file:
import os
import tempfile
import zipfile
if self.lrtl_file:
import os
import tempfile
import zipfile
tmpdir = tempfile.mkdtemp()
try:
zip_path = os.path.join(tmpdir, "archive.zip")
with open(zip_path, "wb") as f:
f.write(self.lrtl_file.read())
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(tmpdir)
os.unlink(zip_path)
scrobbles = retroarch.import_retroarch_lrtl_files(
tmpdir + "/",
self.user.id,
)
finally:
import shutil
shutil.rmtree(tmpdir, ignore_errors=True)
else:
if not self.user.profile.retroarch_path:
logger.info(
"Trying to import Retroarch logs, but user has no retroarch_path configured"
)
self.mark_finished()
return
tmpdir = tempfile.mkdtemp()
try:
zip_path = os.path.join(tmpdir, "archive.zip")
with open(zip_path, "wb") as f:
f.write(self.lrtl_file.read())
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(tmpdir)
os.unlink(zip_path)
scrobbles = retroarch.import_retroarch_lrtl_files(
tmpdir + "/",
self.user.profile.retroarch_path,
self.user.id,
)
finally:
import shutil
shutil.rmtree(tmpdir, ignore_errors=True)
else:
if not self.user.profile.retroarch_path:
logger.info(
"Tying to import Retroarch logs, but user has no retroarch_path configured"
)
self.mark_finished()
return
scrobbles = retroarch.import_retroarch_lrtl_files(
self.user.profile.retroarch_path,
self.user.id,
)
self.record_log(scrobbles)
self.mark_finished()
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class BGStatsImport(BaseFileImportMixin):
@ -499,17 +541,21 @@ class BGStatsImport(BaseFileImportMixin):
return
self.mark_started()
try:
import json
import json
from scrobbles.scrobblers import email_scrobble_board_game
from scrobbles.scrobblers import email_scrobble_board_game
with open(self.upload_file_path, "r", encoding="utf-8") as f:
parsed_json = json.load(f)
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
with open(self.upload_file_path, "r", encoding="utf-8") as f:
parsed_json = json.load(f)
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
class EBirdCSVImport(BaseFileImportMixin):
@ -554,9 +600,40 @@ class EBirdCSVImport(BaseFileImportMixin):
return
self.mark_started()
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
self.record_log(scrobbles)
self.mark_finished()
try:
scrobbles = import_birding_csv(
self.upload_file_path, self.user_id, record_error=self.record_error
)
self.record_log(scrobbles)
except Exception as e:
self.record_error(f"Import failed: {e}")
logger.exception(f"Import failed for {self}")
finally:
self.mark_finished()
TYPE_FK_PREFETCHES: dict[str, tuple[str, ...]] = {
"Video": ("video",),
"Track": ("track", "track__artist_fk"),
"PodcastEpisode": ("podcast_episode", "podcast_episode__podcast"),
"SportEvent": ("sport_event",),
"Book": ("book",),
"Paper": ("paper",),
"VideoGame": ("video_game",),
"BoardGame": ("board_game",),
"GeoLocation": ("geo_location",),
"Trail": ("trail",),
"Beer": ("beer",),
"Puzzle": ("puzzle",),
"Food": ("food",),
"Task": ("task",),
"WebPage": ("web_page",),
"LifeEvent": ("life_event",),
"Mood": ("mood",),
"BrickSet": ("brick_set",),
"Channel": ("channel",),
"BirdingLocation": ("birding_location",),
}
class ScrobbleQuerySet(models.QuerySet):
@ -585,6 +662,25 @@ class ScrobbleQuerySet(models.QuerySet):
"birding_location",
)
def with_related_for_types(self, media_types: list[str]):
prefetches = []
for t in media_types:
if t in TYPE_FK_PREFETCHES:
prefetches.extend(TYPE_FK_PREFETCHES[t])
return self.select_related("user").prefetch_related(*prefetches)
class ShareViewLog(TimeStampedModel):
scrobble = models.ForeignKey(
"Scrobble", on_delete=models.CASCADE, related_name="share_views"
)
ip_address = models.GenericIPAddressField(**BNULL)
user_agent = models.TextField(**BNULL)
referrer = models.URLField(max_length=2048, **BNULL)
def __str__(self):
return f"View of {self.scrobble} at {self.created}"
class Scrobble(TimeStampedModel):
"""A scrobble tracks played media items by a user."""
@ -647,6 +743,14 @@ class Scrobble(TimeStampedModel):
media_type = models.CharField(
max_length=20, choices=MediaType.choices, default=MediaType.VIDEO
)
visibility = models.CharField(
max_length=10,
choices=Visibility.choices,
default=Visibility.PRIVATE,
db_index=True,
)
share_token_version = models.PositiveIntegerField(default=0)
share_view_count = models.PositiveIntegerField(default=0)
user = models.ForeignKey(User, blank=True, null=True, on_delete=models.DO_NOTHING)
# Time keeping
@ -691,6 +795,12 @@ class Scrobble(TimeStampedModel):
)
long_play_seconds = models.BigIntegerField(**BNULL)
long_play_complete = models.BooleanField(**BNULL)
long_play_last_scrobble = models.ForeignKey(
"self",
**BNULL,
on_delete=models.SET_NULL,
related_name="next_long_play_scrobbles",
)
class Meta:
indexes = [
@ -705,6 +815,7 @@ class Scrobble(TimeStampedModel):
"is_paused",
]
),
models.Index(fields=["user", "-timestamp"]),
]
@classmethod
@ -737,11 +848,14 @@ class Scrobble(TimeStampedModel):
@classmethod
def as_dict_by_type(cls, scrobble_qs: models.QuerySet) -> dict:
scrobbles_by_type = defaultdict(list)
scrobbles = (
scrobble_qs.with_related()
if hasattr(scrobble_qs, "with_related")
else scrobble_qs
)
if hasattr(scrobble_qs, "with_related"):
media_types_present = list(
scrobble_qs.values_list("media_type", flat=True).distinct()
)
scrobbles = scrobble_qs.with_related_for_types(media_types_present)
else:
scrobbles = scrobble_qs
for scrobble in scrobbles:
scrobbles_by_type[scrobble.media_type].append(scrobble)
@ -756,7 +870,7 @@ class Scrobble(TimeStampedModel):
# Remove any locations without titles
if "GeoLocation" in scrobbles_by_type.keys():
for loc_scrobble in scrobbles_by_type["GeoLocation"]:
for loc_scrobble in list(scrobbles_by_type["GeoLocation"]):
if not loc_scrobble.media_obj.title:
scrobbles_by_type["GeoLocation"].remove(loc_scrobble)
scrobbles_by_type["GeoLocation_count"] -= 1
@ -828,6 +942,16 @@ class Scrobble(TimeStampedModel):
self.save(update_fields=["uuid"])
return reverse("scrobbles:detail", kwargs={"uuid": self.uuid})
def get_share_url(self):
if self.visibility == Visibility.PRIVATE:
return None
sqid = encode_scrobble_share(self.id, self.share_token_version)
return reverse("scrobbles:shared-detail", kwargs={"sqid": sqid})
def regenerate_share_token(self):
self.share_token_version += 1
self.save(update_fields=["share_token_version"])
def push_to_archivebox(self):
pushable_media = hasattr(self.media_obj, "push_to_archivebox") and callable(
self.media_obj.push_to_archivebox
@ -857,6 +981,8 @@ class Scrobble(TimeStampedModel):
logdata_cls = logdata.BaseLogData
log_dict = self.log
if isinstance(log_dict, logdata.BaseLogData):
log_dict = log_dict.asdict
if isinstance(self.log, str):
# There's nothing stopping django from saving a string in a JSONField :(
logger.warning(
@ -882,8 +1008,13 @@ class Scrobble(TimeStampedModel):
logger.warning("Log data could not be loaded", e)
return logdata_cls()
# Strip log-only keys (stored in JSONField but not part of LogData dataclass)
logdata_kwargs = {
k: v for k, v in log_dict.items() if k in logdata_cls().__dataclass_fields__
}
try:
return logdata_cls(**log_dict)
return logdata_cls(**logdata_kwargs)
except ParseError as e:
logger.warning(
"Could not parse log data",
@ -1057,8 +1188,8 @@ class Scrobble(TimeStampedModel):
if self.is_long_play:
long_play_secs = 0
if self.previous and not self.previous.long_play_complete:
long_play_secs = self.previous.long_play_seconds or 0
if self.long_play_last_scrobble and not self.long_play_last_scrobble.long_play_complete:
long_play_secs = self.long_play_last_scrobble.long_play_seconds or 0
percent = int(((playback_seconds + long_play_secs) / run_time_secs) * 100)
return percent
@ -1328,20 +1459,18 @@ class Scrobble(TimeStampedModel):
or scrobble_data["playback_status"] == "stopped"
):
if read_log_page:
page_list = scrobble.log.get("page_data", [])
if page_list:
for page in page_list:
page_data = scrobble.log.get("page_data", {})
if page_data:
for page_num, page in page_data.items():
if not page.get("end_ts", None):
page["end_ts"] = int(timezone.now().timestamp())
page["duration"] = page["end_ts"] - page.get("start_ts")
page_list.append(
BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
page_data[read_log_page] = BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
scrobble.log["page_data"] = page_list
scrobble.log["page_data"] = page_data
scrobble.save(update_fields=["log"])
elif "log" in scrobble_data.keys() and scrobble.log:
scrobble_data["log"] = scrobble.log | scrobble_data["log"]
@ -1352,13 +1481,13 @@ class Scrobble(TimeStampedModel):
if read_log_page:
scrobble_data["log"] = BookLogData(
page_data=[
BookPageLogData(
page_data={
read_log_page: BookPageLogData(
page_number=read_log_page,
start_ts=int(timezone.now().timestamp()),
)
]
)
}
).asdict
logger.info(
f"[scrobbling] creating new scrobble",
@ -1373,9 +1502,27 @@ class Scrobble(TimeStampedModel):
"calories", None
):
if media.calories:
scrobble_data["log"] = FoodLogData(calories=media.calories)
scrobble_data["log"] = FoodLogData(calories=media.calories).asdict
if mtype not in LONG_PLAY_MEDIA.values():
scrobble_data.pop("long_play_complete", None)
scrobble = cls.create(scrobble_data)
if mtype in LONG_PLAY_MEDIA.values():
last_finished = (
cls.objects.filter(
models.Q(**{key: media}),
user_id=user_id,
timestamp__lt=scrobble.timestamp,
)
.order_by("-timestamp")
.first()
)
if last_finished:
scrobble.long_play_last_scrobble = last_finished
scrobble.save(update_fields=["long_play_last_scrobble"])
return scrobble
@classmethod
@ -1528,6 +1675,18 @@ class Scrobble(TimeStampedModel):
cls,
scrobble_data: dict,
) -> "Scrobble":
if "visibility" not in scrobble_data:
user = scrobble_data.get("user")
media_type = scrobble_data.get("media_type")
override = None
if user and media_type:
try:
profile = user.profile
overrides = profile.media_type_visibility or {}
override = overrides.get(media_type)
except user.__class__.profile.RelatedObjectDoesNotExist:
pass
scrobble_data["visibility"] = override or Visibility.PRIVATE
scrobble = cls.objects.create(**scrobble_data)
if not (
scrobble.media_type == cls.MediaType.GEO_LOCATION
@ -1622,8 +1781,8 @@ class Scrobble(TimeStampedModel):
# Set our playback seconds, and calc long play seconds
self.playback_position_seconds = seconds_elapsed
if self.previous:
past_seconds = self.previous.long_play_seconds or 0
if self.long_play_last_scrobble:
past_seconds = self.long_play_last_scrobble.long_play_seconds or 0
self.long_play_seconds = past_seconds + seconds_elapsed
@ -1684,33 +1843,27 @@ class Scrobble(TimeStampedModel):
return False
def calculate_reading_stats(self, commit=True):
page_data = self.log.get("page_data")
page_data = self.logdata.page_data
if page_data:
# --- Sort safely by numeric page_number ---
def safe_page_number(entry):
try:
return int(getattr("page_number", entry), 0)
except (ValueError, TypeError):
return float("inf")
if isinstance(page_data, dict):
logger.warning("Page data is dict, migrate koreader data")
return
valid_pages = sorted(int(k) for k in page_data.keys())
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
elif isinstance(page_data, list):
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
page_data.sort(key=safe_page_number)
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
else:
page_start = self.log.get("page_start")
page_end = self.log.get("page_end")
@ -1750,9 +1903,7 @@ class FavoriteMedia(TimeStampedModel):
birding_location = models.ForeignKey(
BirdingLocation, on_delete=models.CASCADE, **BNULL
)
media_type = models.CharField(
max_length=20, choices=Scrobble.MediaType.choices
)
media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
sent_to_mopidy = models.BooleanField(default=False)
class Meta:

View File

@ -242,12 +242,13 @@ def manual_scrobble_event(
):
data_dict = lookup_event_from_thesportsdb(thesportsdb_id)
event = SportEvent.find_or_create(data_dict)
event, logdata = SportEvent.find_or_create(data_dict)
scrobble_dict = {
"user_id": user_id,
"timestamp": timezone.now(),
"playback_position_seconds": 0,
"source": "TheSportsDB",
"log": logdata,
}
return Scrobble.create_or_update(event, user_id, scrobble_dict)
@ -356,10 +357,7 @@ def manual_scrobble_book(
if action == "stop":
if url:
if isinstance(scrobble.log, "BookLogData"):
scrobble.log.resume_url = next_url_if_exists(url)
else:
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.log["resume_url"] = next_url_if_exists(url)
scrobble.save(update_fields=["log"])
scrobble.stop(force_finish=True)
@ -539,6 +537,7 @@ def email_scrobble_board_game(
"playback_position_seconds": duration_seconds,
"source": "BG Stats",
"log": log_data,
"visibility": "private",
}
scrobble = None
@ -1151,7 +1150,7 @@ def web_scrobbler_scrobble_video_or_song(
artist_name = data_dict.get("artist")
track_name = data_dict.get("track")
tracks = Track.objects.filter(
artist__name=data_dict.get("artist"), title=data_dict.get("track")
artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
)
if tracks.count() > 1:
logger.warning(

View File

@ -0,0 +1,36 @@
from sqids import Sqids
_sqids = None
def _make_alphabet() -> str:
import hashlib
from django.conf import settings
digest = hashlib.sha256(settings.SECRET_KEY.encode()).hexdigest()
base = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789"
seed = int(digest[:16], 16)
shuffled = list(base)
for i in range(len(shuffled) - 1, 0, -1):
seed = (seed * 1103515245 + 12345) & 0x7FFFFFFF
j = seed % (i + 1)
shuffled[i], shuffled[j] = shuffled[j], shuffled[i]
return "".join(shuffled)
def get_sqids() -> Sqids:
global _sqids
if _sqids is None:
_sqids = Sqids(
alphabet=_make_alphabet(),
min_length=6,
)
return _sqids
def encode_scrobble_share(scrobble_id: int, version: int) -> str:
return get_sqids().encode([scrobble_id, version])
def decode_scrobble_share(sqid: str) -> list[int] | None:
return get_sqids().decode(sqid)

View File

@ -12,6 +12,7 @@ from charts.utils import (
from django.apps import apps
from django.conf import settings
from django.contrib.auth import get_user_model
from django.db import models
from django.utils import timezone
logger = logging.getLogger(__name__)
@ -544,6 +545,30 @@ def send_mood_checkin():
send_mood_checkin_reminders()
@shared_task
def backfill_scrobble_sentiment():
"""Backfill VADER sentiment for scrobbles with notes (replaces @hourly cron)."""
from scrobbles.models import Scrobble
from scrobbles.utils import analyze_scrobble_sentiment
qs = Scrobble.objects.filter(
models.Q(log__notes__isnull=False)
& ~models.Q(log__notes=[])
& ~models.Q(log__notes={})
& models.Q(log__sentiment__isnull=True)
)
count = 0
for scrobble in qs.iterator():
if analyze_scrobble_sentiment(scrobble):
count += 1
logger.info(
"Backfilled sentiment for %d scrobbles",
count,
)
@shared_task
def add_favorite_to_mopidy_playlist(favorite_id):
from scrobbles.models import FavoriteMedia

View File

@ -153,11 +153,32 @@ urlpatterns = [
name="long-plays",
),
path("scrobbles/", views.ScrobbleListView.as_view(), name="scrobble-list"),
path("explore/", views.ScrobbleExploreView.as_view(), name="explore"),
path(
"shared/<str:sqid>/",
views.ScrobbleShareView.as_view(),
name="shared-detail",
),
path(
"scrobbles/<slug:uuid>/",
views.ScrobbleDetailView.as_view(),
name="detail",
),
path(
"scrobbles/<slug:uuid>/regenerate-share-token/",
views.RegenerateShareTokenView.as_view(),
name="regenerate-share-token",
),
path(
"scrobbles/<slug:uuid>/change-visibility/",
views.ChangeVisibilityView.as_view(),
name="change-visibility",
),
path(
"scrobbles/<slug:uuid>/share-analytics/",
views.ScrobbleShareAnalyticsView.as_view(),
name="share-analytics",
),
path(
"scrobbles/<slug:uuid>/add-to-mopidy-queue/",
views.add_to_mopidy_queue,

View File

@ -802,3 +802,32 @@ def tokenize_title_to_tags(title: str) -> list[str]:
w.lower() for w in cleaned.split() if w.lower() not in STOPWORDS and len(w) > 2
]
return words
def analyze_scrobble_sentiment(scrobble, overwrite=False) -> bool:
"""Run VADER sentiment analysis on a scrobble's notes.
Stores result in log["sentiment"] as a dict with keys:
neg, neu, pos, compound.
Returns True if analyzed, False if skipped (no notes or already done).
"""
from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer
log = scrobble.log or {}
if not overwrite and log.get("sentiment") is not None:
return False
notes_str = ""
if scrobble.logdata:
notes_str = scrobble.logdata.notes_as_str()
if not notes_str:
return False
analyzer = SentimentIntensityAnalyzer()
scores = analyzer.polarity_scores(notes_str)
log["sentiment"] = scores
scrobble.log = log
scrobble.save(update_fields=["log"])
return True

View File

@ -14,7 +14,7 @@ from django.contrib import messages
from django.contrib.auth import get_user_model
from django.contrib.auth.mixins import LoginRequiredMixin
from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
from django.db.models import Count, Max, Q, Sum
from django.db.models import Count, F, Max, Q, Sum
from django.db.models.query import QuerySet
from rest_framework.authentication import TokenAuthentication
from rest_framework.authtoken.models import Token
@ -38,7 +38,7 @@ from django.utils import timezone
from django.utils.dateformat import DateFormat
from django.views.decorators.csrf import csrf_exempt
from django.views.decorators.http import require_POST
from django.views.generic import DetailView, FormView, TemplateView
from django.views.generic import DetailView, FormView, TemplateView, View
from django.views.generic.edit import CreateView
from django.views.generic.list import ListView
from moods.models import Mood
@ -72,6 +72,8 @@ from scrobbles.constants import (
)
from scrobbles.export import export_scrobbles
from scrobbles.forms import ExportScrobbleForm, ScrobbleForm
from scrobbles.constants import Visibility
from scrobbles.sqids import decode_scrobble_share
from scrobbles.models import (
AudioScrobblerTSVImport,
BGStatsImport,
@ -83,6 +85,7 @@ from scrobbles.models import (
ScaleCSVImport,
Scrobble,
ScrobbleQuerySet,
ShareViewLog,
TrailGPXImport,
)
from scrobbles.scrobblers import *
@ -184,7 +187,7 @@ class ScrobbleableDetailView(ChartContextMixin, DetailView):
def get_context_data(self, **kwargs):
context_data = super().get_context_data(**kwargs)
scrobbles = []
if not self.request.user.is_anonymous:
if not self.request.user.is_anonymous and hasattr(self.object, "scrobble_set"):
scrobbles = self.object.scrobble_set.filter(
user=self.request.user
).order_by("-timestamp")
@ -368,6 +371,9 @@ class ScrobbleListView(LoginRequiredMixin, ListView):
qs = qs.filter(id__in=matching_ids).distinct()
else:
tag_list = []
visibility_param = self.request.GET.get("visibility", "")
if visibility_param in ("public", "shared", "private"):
qs = qs.filter(visibility=visibility_param)
self.tag_list = tag_list
self._full_queryset = qs
return qs
@ -439,8 +445,7 @@ class ScrobbleListView(LoginRequiredMixin, ListView):
full_qs = getattr(self, "_full_queryset", None)
if full_qs is not None and getattr(self, "tag_list", []):
total = (
full_qs.aggregate(total=Sum("playback_position_seconds"))["total"]
or 0
full_qs.aggregate(total=Sum("playback_position_seconds"))["total"] or 0
)
ctx["total_time_seconds"] = total
return ctx
@ -904,6 +909,28 @@ def scrobble_longplay_finish(request, uuid):
if not user.is_authenticated:
return HttpResponseRedirect(success_url)
# Try scrobble UUID first
scrobble = Scrobble.objects.filter(uuid=uuid, user=user).first()
if scrobble:
if scrobble.long_play_complete == True:
scrobble.long_play_complete = None
scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.INFO,
f"Long play of {scrobble.media_obj} marked as not complete.",
)
else:
scrobble.long_play_complete = True
scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.SUCCESS,
f"Long play of {scrobble.media_obj} finished.",
)
return HttpResponseRedirect(success_url)
# Fall back to media UUID (existing behavior)
media_obj = None
for app, model in LONG_PLAY_MEDIA.items():
media_model = apps.get_model(app_label=app, model_name=model)
@ -912,10 +939,21 @@ def scrobble_longplay_finish(request, uuid):
break
if not media_obj:
return
messages.add_message(
request, messages.ERROR, f"Media with uuid {uuid} not found."
)
return HttpResponseRedirect(success_url)
last_scrobble = media_obj.last_long_play_scrobble_for_user(user)
if last_scrobble and last_scrobble.long_play_complete == False:
if last_scrobble and last_scrobble.long_play_complete == True:
last_scrobble.long_play_complete = None
last_scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
request,
messages.INFO,
f"Long play of {media_obj} marked as not complete.",
)
elif last_scrobble:
last_scrobble.long_play_complete = True
last_scrobble.save(update_fields=["long_play_complete"])
messages.add_message(
@ -1137,6 +1175,15 @@ class ScrobbleDetailView(DetailView):
slug_url_kwarg = "uuid"
paginate_by = 100
def get_object(self, queryset=None):
scrobble = super().get_object(queryset=queryset)
user = self.request.user
if scrobble.visibility == Visibility.PUBLIC:
return scrobble
if user.is_authenticated and scrobble.user == user:
return scrobble
raise Http404
def get_form_class(self):
return self.object.media_obj.logdata_cls().form()
@ -1245,8 +1292,7 @@ class ScrobbleDetailView(DetailView):
track=media_obj, user=self.object.user
).order_by("-timestamp")[:20]
context["has_mopidy_uri"] = any(
(s.log or {}).get("raw_data", {}).get("mopidy_uri")
for s in scrobbles
(s.log or {}).get("raw_data", {}).get("mopidy_uri") for s in scrobbles
)
else:
context["has_mopidy_uri"] = False
@ -1254,6 +1300,93 @@ class ScrobbleDetailView(DetailView):
return context
class ScrobbleShareView(TemplateView):
template_name = "scrobbles/scrobble_share.html"
def get_object(self):
sqid = self.kwargs.get("sqid")
decoded = decode_scrobble_share(sqid)
if not decoded or len(decoded) != 2:
raise Http404
scrobble_id, version = decoded
scrobble = get_object_or_404(Scrobble, id=scrobble_id)
if scrobble.share_token_version != version:
raise Http404
if scrobble.visibility not in (Visibility.PUBLIC, Visibility.SHARED):
raise Http404
Scrobble.objects.filter(id=scrobble.id).update(
share_view_count=F("share_view_count") + 1
)
scrobble.refresh_from_db(fields=["share_view_count"])
ShareViewLog.objects.create(
scrobble=scrobble,
ip_address=self.request.META.get("REMOTE_ADDR"),
user_agent=self.request.META.get("HTTP_USER_AGENT", "")[:500],
referrer=self.request.META.get("HTTP_REFERER", ""),
)
return scrobble
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
scrobble = self.get_object()
context["object"] = scrobble
context["log_form"] = None
context["related_scrobbles"] = Scrobble.objects.none()
context["has_mopidy_uri"] = False
if self.request.user.is_authenticated:
media_type = scrobble.media_type
fk_field = ScrobbleDetailView.MEDIA_FK_MAP.get(media_type)
media_obj = scrobble.media_obj
if fk_field and media_obj:
context["is_favorited"] = FavoriteMedia.objects.filter(
user=self.request.user, **{fk_field: media_obj}
).exists()
return context
class ScrobbleExploreView(ListView):
model = Scrobble
paginate_by = 100
template_name = "scrobbles/scrobble_explore.html"
queryset = Scrobble.objects.filter(visibility=Visibility.PUBLIC).order_by(
"-timestamp"
)
class RegenerateShareTokenView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
scrobble.regenerate_share_token()
return redirect(scrobble.get_absolute_url())
class ChangeVisibilityView(LoginRequiredMixin, View):
def post(self, request, uuid):
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
visibility = request.POST.get("visibility")
if visibility not in (Visibility.PUBLIC, Visibility.SHARED, Visibility.PRIVATE):
return redirect(scrobble.get_absolute_url())
scrobble.visibility = visibility
scrobble.save(update_fields=["visibility"])
return redirect(scrobble.get_absolute_url())
class ScrobbleShareAnalyticsView(LoginRequiredMixin, DetailView):
model = Scrobble
slug_field = "uuid"
slug_url_kwarg = "uuid"
template_name = "scrobbles/scrobble_share_analytics.html"
def get_queryset(self):
return Scrobble.objects.filter(user=self.request.user)
def get_context_data(self, **kwargs):
context = super().get_context_data(**kwargs)
scrobble = self.object
context["share_views"] = scrobble.share_views.order_by("-created")[:50]
return context
class BaseEmbeddableWidget(TemplateView):
template_name = "scrobbles/embeddable_top_media.html"

View File

@ -23,6 +23,7 @@ class SportAdmin(admin.ModelAdmin):
class LeagueAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "abbreviation_str")
raw_id_fields = ("sport",)
ordering = ("name",)
@ -30,6 +31,7 @@ class LeagueAdmin(admin.ModelAdmin):
class PlayerAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league", "team")
raw_id_fields = ("league", "team")
ordering = ("name",)
@ -37,6 +39,7 @@ class PlayerAdmin(admin.ModelAdmin):
class SeasonAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league")
raw_id_fields = ("league",)
ordering = ("name",)
@ -44,6 +47,7 @@ class SeasonAdmin(admin.ModelAdmin):
class RoundAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "season")
raw_id_fields = ("season",)
ordering = ("name",)
@ -51,6 +55,7 @@ class RoundAdmin(admin.ModelAdmin):
class TeamAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = ("name", "league")
raw_id_fields = ("league",)
ordering = ("name",)
@ -59,19 +64,28 @@ class SportEventAdmin(admin.ModelAdmin):
date_hierarchy = "created"
list_display = (
"title",
"league",
"event_type",
"start",
"comp_str",
"round",
)
list_filter = ("round__season", "home_team", "away_team")
raw_id_fields = ("league", "teams", "players", "round")
list_filter = ("league", "event_type")
ordering = ("-created",)
inlines = [
ScrobbleInline,
]
def comp_str(self, obj):
if obj.home_team:
return f"{obj.away_team} @ {obj.home_team}"
if obj.player_one:
return f"{obj.player_one} v {obj.player_two}"
teams = list(obj.teams.all())
if len(teams) >= 2:
return f"{teams[1]} v {teams[0]}"
players = list(obj.players.all())
if len(players) >= 2:
return f"{players[0]} v {players[1]}"
if len(players) == 1:
return str(players[0])
if len(teams) == 1:
return str(teams[0])

View File

@ -0,0 +1,36 @@
from django.core.management.base import BaseCommand
from sports.models import League, Team
from sports.thesportsdb import enrich_league_logo, enrich_team_logo, has_logo
class Command(BaseCommand):
help = "Fetch missing league and team logos from TheSportsDB"
def add_arguments(self, parser):
parser.add_argument(
"--dry-run",
action="store_true",
help="Show what would be done without making changes",
)
def handle(self, *args, **options):
dry_run = options["dry_run"]
leagues = League.objects.filter(thesportsdb_id__isnull=False)
for league in leagues:
if has_logo(league):
continue
if dry_run:
self.stdout.write(f"Would enrich logo for league: {league.name} ({league.thesportsdb_id})")
else:
enrich_league_logo(league)
teams = Team.objects.filter(thesportsdb_id__isnull=False)
for team in teams:
if has_logo(team):
continue
if dry_run:
self.stdout.write(f"Would enrich logo for team: {team.name} ({team.thesportsdb_id})")
else:
enrich_team_logo(team)

View File

@ -0,0 +1,24 @@
# Generated by Django 4.2.30 on 2026-06-06 15:29
from django.db import migrations, models
import django.db.models.deletion
class Migration(migrations.Migration):
dependencies = [
("sports", "0018_alter_sportevent_genre"),
]
operations = [
migrations.AddField(
model_name="sportevent",
name="league",
field=models.ForeignKey(
blank=True,
null=True,
on_delete=django.db.models.deletion.DO_NOTHING,
to="sports.league",
),
),
]

View File

@ -0,0 +1,111 @@
import json
from django.db import migrations, models
import django.db.models.deletion
def canonical_key(event):
if event.home_team_id and event.away_team_id:
return ("teams", event.league_id, event.home_team_id, event.away_team_id)
if event.player_one_id and event.player_two_id:
return ("players", event.league_id, event.player_one_id, event.player_two_id)
return ("title", event.league_id, event.event_type, (event.title or "").strip())
def build_logdata(event):
logdata = {}
if event.thesportsdb_id:
logdata["thesportsdb_id"] = event.thesportsdb_id
if event.start:
logdata["start"] = (
event.start.isoformat()
if hasattr(event.start, "isoformat")
else str(event.start)
)
if event.round:
logdata["round_name"] = event.round.name or str(event.round)
if event.round.season:
logdata["season_name"] = event.round.season.name or str(event.round.season)
return logdata
def merge_scrobble_logs(scrobble, logdata):
existing_log = scrobble.log or {}
if isinstance(existing_log, str):
existing_log = json.loads(existing_log)
existing_log.update(logdata)
scrobble.log = existing_log
scrobble.save(update_fields=["log"])
def populate_league(event):
if event.league:
return
if event.round and event.round.season and event.round.season.league:
event.league = event.round.season.league
event.save(update_fields=["league"])
def populate_m2m(event):
if event.home_team_id:
event.teams.add(event.home_team_id)
if event.away_team_id:
event.teams.add(event.away_team_id)
if event.player_one_id:
event.players.add(event.player_one_id)
if event.player_two_id:
event.players.add(event.player_two_id)
def migrate_sport_event_data(apps, schema_editor):
SportEvent = apps.get_model("sports", "SportEvent")
Scrobble = apps.get_model("scrobbles", "Scrobble")
db_alias = schema_editor.connection.alias
canonical_events = {}
for event in SportEvent.objects.using(db_alias).iterator():
populate_league(event)
key = canonical_key(event)
canonical = canonical_events.get(key)
if not canonical:
canonical_events[key] = event
populate_m2m(event)
logdata = build_logdata(event)
for scrobble in (
Scrobble.objects.using(db_alias).filter(sport_event=event).iterator()
):
merge_scrobble_logs(scrobble, logdata)
else:
logdata = build_logdata(event)
for scrobble in (
Scrobble.objects.using(db_alias).filter(sport_event=event).iterator()
):
scrobble.sport_event = canonical
merge_scrobble_logs(scrobble, logdata)
scrobble.save(update_fields=["sport_event", "log"])
event.delete()
class Migration(migrations.Migration):
dependencies = [
("sports", "0019_sportevent_league_alter_sportevent_away_team_and_more"),
]
operations = [
migrations.AddField(
model_name="sportevent",
name="teams",
field=models.ManyToManyField(blank=True, to="sports.team"),
),
migrations.AddField(
model_name="sportevent",
name="players",
field=models.ManyToManyField(blank=True, to="sports.player"),
),
migrations.RunPython(
migrate_sport_event_data, reverse_code=migrations.RunPython.noop
),
]

View File

@ -0,0 +1,20 @@
# Generated by Django 4.2.29 on 2026-06-07 03:24
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
("sports", "0020_migrate_sport_event_data_to_logdata"),
]
operations = [
migrations.AddField(
model_name="team",
name="logo",
field=models.ImageField(
blank=True, null=True, upload_to="sports/team-logos/"
),
),
]

View File

@ -0,0 +1,27 @@
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
("sports", "0021_team_logo"),
]
operations = [
migrations.RemoveField(
model_name="sportevent",
name="home_team",
),
migrations.RemoveField(
model_name="sportevent",
name="away_team",
),
migrations.RemoveField(
model_name="sportevent",
name="player_one",
),
migrations.RemoveField(
model_name="sportevent",
name="player_two",
),
]

View File

@ -68,6 +68,7 @@ class Season(TheSportsDbMixin):
class Team(TheSportsDbMixin):
league = models.ForeignKey(League, on_delete=models.DO_NOTHING, **BNULL)
logo = models.ImageField(upload_to="sports/team-logos/", **BNULL)
class Player(TheSportsDbMixin):
@ -88,50 +89,61 @@ class Round(TheSportsDbMixin):
class SportEvent(ScrobblableMixin):
COMPLETION_PERCENT = getattr(settings, "SPORT_COMPLETION_PERCENT", 90)
thesportsdb_id = models.CharField(max_length=255, **BNULL)
event_type = models.CharField(
max_length=2,
choices=SportEventType.choices,
default=SportEventType.UNKNOWN,
)
round = models.ForeignKey(Round, on_delete=models.DO_NOTHING, **BNULL)
start = models.DateTimeField(**BNULL)
home_team = models.ForeignKey(
Team,
on_delete=models.DO_NOTHING,
related_name="home_event_set",
**BNULL,
)
away_team = models.ForeignKey(
Team,
on_delete=models.DO_NOTHING,
related_name="away_event_set",
**BNULL,
)
player_one = models.ForeignKey(
Player,
on_delete=models.DO_NOTHING,
related_name="player_one_set",
**BNULL,
)
player_two = models.ForeignKey(
Player,
on_delete=models.DO_NOTHING,
related_name="player_two_set",
**BNULL,
)
league = models.ForeignKey(League, on_delete=models.DO_NOTHING, **BNULL)
def __str__(self):
return (
f"{self.start.date()} - {self.round} - {self.home_team} v {self.away_team}"
)
@property
def logdata_cls(self):
from scrobbles.dataclasses import SportEventLogData
return SportEventLogData
teams = models.ManyToManyField(Team, blank=True)
players = models.ManyToManyField(Player, blank=True)
# Deprecated - data migrated to scrobble.log via SportEventLogData
thesportsdb_id = models.CharField(max_length=255, **BNULL)
start = models.DateTimeField(**BNULL)
round = models.ForeignKey(Round, on_delete=models.DO_NOTHING, **BNULL)
def save(self, *args, **kwargs):
old_instance = None
try:
old_instance = UserProfile.objects.get(pk=self.pk)
except:
pass
if not self.title or (old_instance and old_instance.title != self.title):
self.title = self.comp_str
super(SportEvent, self).save(*args, **kwargs)
def __str__(self) -> str:
return f"{self.title} - {self.subtitle}"
def get_absolute_url(self):
return reverse("sports:event_detail", kwargs={"slug": self.uuid})
@property
def subtitle(self):
return self.round.season.league
def subtitle(self) -> str:
league = self.league
if self.league and self.league.abbreviation_str:
league = self.league.abbreviation_str
return f"{league} {self.get_event_type_display()}"
@property
def comp_str(self) -> str:
if self.players.exists():
return " v ".join(str(p) for p in self.players.all())
if self.teams.exists():
return " v ".join(str(t) for t in self.teams.all())
return ""
@property
def strings(self) -> ScrobblableConstants:
@ -147,16 +159,17 @@ class SportEvent(ScrobblableMixin):
@property
def primary_image_url(self) -> str:
url = ""
if self.round.season.league.logo:
url = self.round.season.league.logo.url
return url
if self.league and self.league.logo:
return self.league.logo.url
return ""
@classmethod
def find_or_create(cls, data_dict: Dict) -> "Event":
"""Given a data dict from Jellyfin, does the heavy lifting of looking up
the video and, if need, TV Series, creating both if they don't yet
exist.
def find_or_create(cls, data_dict: Dict) -> tuple["Event", dict]:
"""Given a data dict from TheSportsDB, finds or creates a canonical
SportEvent by teams, players or title, and returns (event, logdata).
The logdata dict contains per-scrobble details (thesportsdb_id, start,
round/season names) that should be stored in the scrobble's log field.
"""
# Find or create our Sport
@ -187,32 +200,29 @@ class SportEvent(ScrobblableMixin):
# Find or create our Round
rid = data_dict.get("RoundId")
round, r_created = Round.objects.get_or_create(
round_obj, r_created = Round.objects.get_or_create(
thesportsdb_id=rid,
season=season,
name=rid,
)
if r_created:
round.season = season
round.save(update_fields=["season"])
round_obj.season = season
round_obj.save(update_fields=["season"])
# Set some special data for Tennis
player_one = None
player_two = None
if data_dict.get("Sport") == "Tennis":
event_name = data_dict.get("Name", "")
if not round.name:
round.name = get_round_name_from_event(event_name)
round.save(update_fields=["name"])
players_list = get_players_from_event(event_name)
player_one = Player.objects.filter(name__icontains=players_list[0]).first()
if not player_one:
player_one = Player.objects.create(name=players_list[0])
player_two = Player.objects.filter(name__icontains=players_list[1]).first()
if not player_two:
player_two = Player.objects.create(name=players_list[1])
# Build logdata with per-scrobble details
logdata = {}
logdata["thesportsdb_id"] = data_dict.get("EventId")
start = data_dict.get("Start")
if start:
logdata["start"] = (
start.isoformat() if hasattr(start, "isoformat") else str(start)
)
if round_obj:
logdata["round_name"] = round_obj.name or str(round_obj)
if round_obj.season:
logdata["season_name"] = round_obj.season.name or str(round_obj.season)
# Look up or create teams/players
home_team = None
away_team = None
if data_dict.get("HomeTeamName"):
@ -221,27 +231,73 @@ class SportEvent(ScrobblableMixin):
"thesportsdb_id": data_dict.get("HomeTeamId", ""),
"league": league,
}
home_team, _created = Team.objects.get_or_create(**home_team_dict)
home_team, ht_created = Team.objects.get_or_create(**home_team_dict)
if ht_created:
from sports.thesportsdb import enrich_team_logo
enrich_team_logo(home_team)
away_team_dict = {
"name": data_dict.get("AwayTeamName", ""),
"thesportsdb_id": data_dict.get("AwayTeamId", ""),
"league": league,
}
away_team, _created = Team.objects.get_or_create(**away_team_dict)
away_team, at_created = Team.objects.get_or_create(**away_team_dict)
if at_created:
from sports.thesportsdb import enrich_team_logo
event_dict = {
"thesportsdb_id": data_dict.get("EventId"),
"title": data_dict.get("Name"),
"event_type": sport.default_event_type,
"home_team": home_team,
"away_team": away_team,
"player_one": player_one,
"player_two": player_two,
"start": data_dict.get("Start"),
"round": round,
"base_run_time_seconds": data_dict.get("RunTime"),
}
event, _created = cls.objects.get_or_create(**event_dict)
enrich_team_logo(away_team)
return event
players_list = []
if data_dict.get("Sport") == "Tennis":
event_name = data_dict.get("Name", "")
if not round_obj.name:
round_obj.name = get_round_name_from_event(event_name)
round_obj.save(update_fields=["name"])
players_list = get_players_from_event(event_name)
# Find existing canonical event by teams, players, or title
event = None
if home_team and away_team:
event = (
cls.objects.filter(league=league, teams=home_team)
.filter(teams=away_team)
.first()
)
if not event and players_list:
player_objs = []
for player_name in players_list:
player = Player.objects.filter(name__icontains=player_name).first()
if not player:
player = Player.objects.create(name=player_name)
player_objs.append(player)
qs = cls.objects.filter(league=league, players=player_objs[0])
for player in player_objs[1:]:
qs = qs.filter(players=player)
event = qs.first()
if not event:
title = data_dict.get("Name", "").strip()
if title:
event = cls.objects.filter(league=league, title=title).first()
if not event:
event = cls.objects.create(
title=data_dict.get("Name"),
event_type=sport.default_event_type,
league=league,
base_run_time_seconds=data_dict.get("RunTime"),
)
# Ensure M2M is populated on the canonical event
if home_team and not event.teams.filter(id=home_team.id).exists():
event.teams.add(home_team)
if away_team and not event.teams.filter(id=away_team.id).exists():
event.teams.add(away_team)
for player_name in players_list:
player = Player.objects.filter(name__icontains=player_name).first()
if player and not event.players.filter(id=player.id).exists():
event.players.add(player)
return event, logdata

View File

@ -1,10 +1,12 @@
import logging
import requests
from dateutil.parser import parse
from django.conf import settings
from django.core.files.base import ContentFile
from django.utils import timezone
from pysportsdb import TheSportsDbClient
from sports.models import Sport
from sports.models import League, Sport, Team
logger = logging.getLogger(__name__)
@ -12,6 +14,84 @@ API_KEY = getattr(settings, "THESPORTSDB_API_KEY", "2")
client = TheSportsDbClient(api_key=API_KEY)
def has_logo(league_or_team) -> bool:
"""Check if a model instance has a logo (handles both NULL and empty string)."""
return bool(league_or_team.logo and league_or_team.logo.name)
def enrich_league_logo(league: League) -> None:
"""Fetch the league badge from TheSportsDB and save it as the league logo."""
if not league.thesportsdb_id or has_logo(league):
return
url = (
f"https://www.thesportsdb.com/api/v1/json/{API_KEY}"
f"/lookupleague.php?id={league.thesportsdb_id}"
)
try:
resp = requests.get(url, timeout=10)
data = resp.json()
leagues = data.get("leagues", [])
if not leagues:
return
badge_url = leagues[0].get("strBadge")
if badge_url:
r = requests.get(badge_url, timeout=10)
if r.status_code == 200:
fname = f"{league.uuid or league.thesportsdb_id}.png"
league.logo.save(fname, ContentFile(r.content), save=True)
logger.info(
"Saved league logo from TheSportsDB",
extra={"league_id": league.id, "league_name": league.name},
)
except Exception as e:
logger.warning(
"Failed to fetch league logo from TheSportsDB",
extra={"league_id": league.id, "error": str(e)},
)
def enrich_team_logo(team: Team) -> None:
"""Fetch the team badge from TheSportsDB and save it as the team logo."""
if not team.thesportsdb_id or has_logo(team):
return
try:
badge_url = None
# Try direct lookup by thesportsdb_id first (more reliable)
url = (
f"https://www.thesportsdb.com/api/v1/json/{API_KEY}"
f"/lookupteam.php?id={team.thesportsdb_id}"
)
resp = requests.get(url, timeout=10)
data = resp.json()
api_teams = data.get("teams", [])
if api_teams:
badge_url = api_teams[0].get("strBadge")
else:
# Fall back to name search
result = client.search_teams(team.name) or {}
api_teams = result.get("teams", [])
if api_teams:
badge_url = api_teams[0].get("strBadge")
if badge_url:
r = requests.get(badge_url, timeout=10)
if r.status_code == 200:
fname = f"{team.uuid or team.thesportsdb_id}.png"
team.logo.save(fname, ContentFile(r.content), save=True)
logger.info(
"Saved team logo from TheSportsDB",
extra={"team_id": team.id, "team_name": team.name},
)
except Exception as e:
logger.warning(
"Failed to fetch team logo from TheSportsDB",
extra={"team_id": team.id, "error": str(e)},
)
def lookup_event_from_thesportsdb(event_id: str) -> dict:
try:
@ -23,6 +103,18 @@ def lookup_event_from_thesportsdb(event_id: str) -> dict:
return {}
sport, _created = Sport.objects.get_or_create(thesportsdb_id=event.get("strSport"))
# Find or create the league and optionally enrich its logo
lid = event.get("idLeague")
league, l_created = League.objects.get_or_create(
thesportsdb_id=lid,
defaults={"name": event.get("strLeague", "")},
)
if l_created:
league.name = event.get("strLeague", "")
league.sport = sport
league.save(update_fields=["name", "sport"])
enrich_league_logo(league)
try:
start = parse(event.get("strTimestamp"))
except:
@ -38,7 +130,7 @@ def lookup_event_from_thesportsdb(event_id: str) -> dict:
"RunTime": sport.default_event_run_time_seconds,
"Sport": event.get("strSport"),
"Season": event.get("strSeason"),
"LeagueId": event.get("idLeague"),
"LeagueId": lid,
"LeagueName": event.get("strLeague"),
"HomeTeamId": event.get("idHomeTeam"),
"HomeTeamName": event.get("strHomeTeam"),

View File

@ -1,12 +1,12 @@
from django.views import generic
from sports.models import SportEvent
from vrobbler.apps.scrobbles.views import ScrobbleableDetailView, ScrobbleableListView
class SportEventListView(generic.ListView):
class SportEventListView(ScrobbleableListView):
model = SportEvent
paginate_by = 50
class SportEventDetailView(generic.DetailView):
class SportEventDetailView(ScrobbleableDetailView):
model = SportEvent
slug_field = "uuid"

View File

@ -26,6 +26,7 @@ class GameAdmin(admin.ModelAdmin):
"main_story_time",
"release_year",
)
raw_id_fields = ("platforms",)
search_fields = (
"title",
"alternative_name",

View File

@ -163,12 +163,9 @@ class VideoGame(LongPlayScrobblableMixin):
platforms = models.ManyToManyField(VideoGamePlatform)
retroarch_name = models.CharField(max_length=255, **BNULL)
def __str__(self):
return self.title
@property
def subtitle(self):
return f" On {self.platforms.first()}"
return f"{self.platforms.first()}"
@property
def strings(self) -> ScrobblableConstants:

View File

@ -71,6 +71,7 @@ class VideoMetadata:
twitch_id: Optional[str] = "",
base_run_time_seconds: int = 900,
):
self.title = ""
self.imdb_id = imdb_id
self.youtube_id = youtube_id
self.twitch_id = twitch_id

View File

@ -20,6 +20,7 @@ from scrobbles.mixins import (
)
from taggit.managers import TaggableManager
from videos.metadata import VideoMetadata
from videos.sources.omdb import lookup_video_from_omdb
from videos.sources.tmdb import lookup_video_from_tmdb
from videos.sources.youtube import lookup_video_from_youtube
@ -224,6 +225,8 @@ class Series(TimeStampedModel):
def is_episode_playing(self, user_id: int) -> bool:
last_scrobble = self.scrobbles_for_user(user_id, include_playing=True).first()
if not last_scrobble:
return False
return not last_scrobble.played_to_completion
def fix_metadata(self, force_update=False):
@ -255,9 +258,20 @@ class Series(TimeStampedModel):
logger.info("Series not created and overwrite=False, returning")
return series
vdict, _, cover, genres = lookup_video_from_tmdb(
imdb_id
).as_dict_with_cover_and_genres()
try:
metadata = lookup_video_from_tmdb(imdb_id)
except Exception as e:
logger.warning(f"TMDB lookup failed for series {imdb_id}: {e}")
metadata = None
if not metadata or not metadata.title:
metadata = lookup_video_from_omdb(imdb_id)
if not metadata or not metadata.title:
logger.warning(f"No metadata found for series {imdb_id} from TMDB or OMDB")
return series
vdict, _, cover, genres = metadata.as_dict_with_cover_and_genres()
vdict.pop("video_type")
vdict["name"] = vdict.pop("title")
@ -406,6 +420,84 @@ class Video(ScrobblableMixin):
fname = f"{self.title}_{self.uuid}.jpg"
self.cover_image.save(fname, ContentFile(r.content), save=True)
def fix_metadata(self, force_update: bool = False) -> None:
if self.video_type == self.VideoType.YOUTUBE and self.youtube_id:
vdict, _, cover, genres = lookup_video_from_youtube(
self.youtube_id
).as_dict_with_cover_and_genres()
for k, v in vdict.items():
setattr(self, k, v)
self.save()
if cover:
self.save_image_from_url(cover)
if genres:
self.genre.add(*genres)
return
if self.video_type == self.VideoType.TWITCH and self.twitch_id:
from videos.sources.twitch import lookup_video_from_twitch
metadata = lookup_video_from_twitch(self.twitch_id)
self.title = metadata.title or f"Twitch VOD {self.twitch_id}"
self.overview = metadata.overview
self.base_run_time_seconds = metadata.base_run_time_seconds
if metadata.upload_date:
self.upload_date = metadata.upload_date
if metadata.year:
self.year = metadata.year
self.video_type = Video.VideoType.TWITCH
if metadata.channel_id:
from videos.models import Channel
self.channel = Channel.objects.filter(
id=metadata.channel_id
).first()
self.save()
if metadata.cover_url:
self.save_image_from_url(metadata.cover_url)
return
if self.imdb_id:
try:
metadata = lookup_video_from_tmdb(self.imdb_id)
except Exception as e:
logger.warning(f"TMDB lookup failed for {self.imdb_id}: {e}")
metadata = None
if not metadata or not metadata.title:
metadata = lookup_video_from_omdb(self.imdb_id)
if not metadata or not metadata.title:
logger.warning(f"No metadata found for {self} from TMDB or OMDB")
return
vdict, series_id, cover, genres = (
metadata.as_dict_with_cover_and_genres()
)
for k, v in vdict.items():
setattr(self, k, v)
if series_id:
self.tv_series = Series.find_or_create(imdb_id=series_id)
self.save()
if cover:
self.save_image_from_url(cover)
if genres:
self.genre.add(*genres)
return
logger.warning(
f"No metadata source available for {self} (type={self.video_type})"
)
@classmethod
def get_from_youtube_id(cls, youtube_id: str, overwrite: bool = False) -> "Video":
video, created = cls.objects.get_or_create(youtube_id=youtube_id)
@ -432,9 +524,20 @@ class Video(ScrobblableMixin):
if not created and not overwrite:
return video
vdict, series_id, cover, genres = lookup_video_from_tmdb(
imdb_id
).as_dict_with_cover_and_genres()
try:
metadata = lookup_video_from_tmdb(imdb_id)
except Exception as e:
logger.warning(f"TMDB lookup failed for {imdb_id}: {e}")
metadata = None
if not metadata or not metadata.title:
metadata = lookup_video_from_omdb(imdb_id)
if not metadata or not metadata.title:
logger.warning(f"No metadata found for {imdb_id} from TMDB or OMDB")
return video
vdict, series_id, cover, genres = metadata.as_dict_with_cover_and_genres()
if created or overwrite:
for k, v in vdict.items():

View File

@ -0,0 +1,83 @@
import logging
import re
from typing import Optional
import requests
from django.conf import settings
from videos.metadata import VideoMetadata, VideoType
logger = logging.getLogger(__name__)
OMDB_API_KEY = getattr(settings, "OMDB_API_KEY", "")
OMDB_URL = "https://www.omdbapi.com/"
RUNTIME_RE = re.compile(r"(\d+)\s*min")
def lookup_video_from_omdb(imdb_id: str) -> Optional[VideoMetadata]:
if not imdb_id.startswith("tt"):
imdb_id = f"tt{imdb_id}"
if not OMDB_API_KEY:
logger.warning("No OMDB API key configured")
return None
params = {"apikey": OMDB_API_KEY, "i": imdb_id, "plot": "full"}
try:
response = requests.get(OMDB_URL, params=params)
response.raise_for_status()
data = response.json()
except requests.exceptions.RequestException as e:
logger.error(f"OMDB API error for {imdb_id}: {e}")
return None
if data.get("Response") == "False":
logger.info(f"OMDB no result for {imdb_id}: {data.get('Error')}")
return None
metadata = VideoMetadata(imdb_id=imdb_id)
metadata.title = data.get("Title")
metadata.plot = data.get("Plot")
metadata.overview = data.get("Plot")
raw_year = data.get("Year")
if raw_year and raw_year.isdigit():
metadata.year = int(raw_year)
raw_rating = data.get("imdbRating")
if raw_rating and raw_rating != "N/A":
metadata.imdb_rating = raw_rating
raw_cover = data.get("Poster")
if raw_cover and raw_cover != "N/A":
metadata.cover_url = raw_cover
raw_runtime = data.get("Runtime")
if raw_runtime:
match = RUNTIME_RE.match(raw_runtime)
if match:
metadata.base_run_time_seconds = int(match.group(1)) * 60
media_type = data.get("Type")
if media_type == "movie":
metadata.video_type = VideoType.MOVIE.value
elif media_type in ("series", "episode"):
metadata.video_type = VideoType.TV_EPISODE.value
if media_type == "episode":
raw_season = data.get("Season")
if raw_season and raw_season != "N/A":
metadata.season_number = int(raw_season)
raw_episode = data.get("Episode")
if raw_episode and raw_episode != "N/A":
metadata.episode_number = int(raw_episode)
series_imdb_id = data.get("seriesID")
if series_imdb_id and series_imdb_id != "N/A":
metadata.tv_series_imdb_id = series_imdb_id
raw_genres = data.get("Genre")
if raw_genres:
metadata.genres = [g.strip() for g in raw_genres.split(",") if g.strip()]
return metadata

View File

@ -3,14 +3,11 @@ import logging
import pendulum
from django.conf import settings
from themoviedb import TMDb
from tmdbv3api import TV, Movie, TMDb as TMDb_direct
from tmdbv3api import TV, Movie
from videos.metadata import VideoMetadata, VideoType
TMDB_KEY = getattr(settings, "TMDB_API_KEY", "")
tmdb_direct = TMDb_direct()
tmdb_direct.api_key = TMDB_KEY
tmdb = TMDb(key=TMDB_KEY, language="en-US", region="US")
TMDB_IMAGE_URL = "https://image.tmdb.org/t/p/original"
@ -36,7 +33,7 @@ def lookup_video_from_tmdb(name_or_id: str, kind: str = "movie") -> VideoMetadat
video_metadata = VideoMetadata(imdb_id=imdb_id)
media = None
show = None
show_data = None
if len(tmdb_result.movie_results) > 0:
media = Movie().details(tmdb_result.movie_results[0].id)
video_metadata.video_type = VideoType.MOVIE.value

View File

@ -32,7 +32,11 @@ class SeriesDetailView(LoginRequiredMixin, ChartContextMixin, generic.DetailView
context_data = super().get_context_data(**kwargs)
context_data["scrobbles"] = self.object.scrobbles_for_user(user_id)
next_episode_id = self.object.last_scrobbled_episode(user_id).next_imdb_id or ""
next_episode_id = ""
if self.object.last_scrobbled_episode(user_id):
next_episode_id = (
self.object.last_scrobbled_episode(user_id).next_imdb_id or ""
)
if self.object.is_episode_playing(user_id):
next_episode_id = ""
if next_episode_id:

View File

@ -60,6 +60,7 @@ THEAUDIODB_API_KEY = os.getenv("VROBBLER_THEAUDIODB_API_KEY", "2")
PODCASTINDEX_API_KEY = os.getenv("VROBBLER_PODCASTINDEX_API_KEY", "")
PODCASTINDEX_API_SECRET = os.getenv("VROBBLER_PODCASTINDEX_API_SECRET", "")
TMDB_API_KEY = os.getenv("VROBBLER_TMDB_API_KEY", "")
OMDB_API_KEY = os.getenv("VROBBLER_OMDB_API_KEY", "")
LASTFM_API_KEY = os.getenv("VROBBLER_LASTFM_API_KEY")
LASTFM_SECRET_KEY = os.getenv("VROBBLER_LASTFM_SECRET_KEY")
IGDB_CLIENT_ID = os.getenv("VROBBLER_IGDB_CLIENT_ID")
@ -144,7 +145,7 @@ CELERY_BEAT_SCHEDULE = {
},
"import-from-webdav": {
"task": "scrobbles.tasks.import_from_webdav_all_users",
"schedule": crontab(minute="*/3"),
"schedule": crontab(minute="*/2"),
},
# Deprecated: BG Stats files now picked up from WebDAV var/bgstats/
# "import-from-imap": {
@ -163,6 +164,10 @@ CELERY_BEAT_SCHEDULE = {
"task": "scrobbles.tasks.send_mood_checkin",
"schedule": crontab(hour="*/4", minute=0),
},
"backfill-scrobble-sentiment": {
"task": "scrobbles.tasks.backfill_scrobble_sentiment",
"schedule": crontab(minute="0"),
},
}
INSTALLED_APPS = [

View File

@ -9,6 +9,7 @@
<th scope="col">Scrobbles</th>
<th scope="col">Complete</th>
<th scope="col">Start</th>
<th scope="col">Finish</th>
</tr>
</thead>
<tbody>
@ -19,8 +20,9 @@
<td><a href="{{obj.get_absolute_url}}">{{obj}}</a></td>
{% if request.user.is_authenticated %}
<td>{{obj.scrobble_count}}</td>
<td>{% if obj.scrobble_set.last.logdata.long_play_complete == True %}Yes{% endif %}</td>
<td>{% if obj.scrobble_set.last.long_play_complete == True %}Yes{% else %}No{% endif %}</td>
<td><a type="button" class="btn btn-sm btn-primary" href="{{obj.start_url}}">Scrobble</a></td>
<td><a type="button" class="btn btn-sm btn-warning" href="{{obj.get_longplay_finish_url}}">Finish</a></td>
{% endif %}
</tr>
{% endif %}

View File

@ -315,8 +315,7 @@
{% for scrobble in now_playing_list %}
<div class="now-playing">
{% if scrobble.media_obj.primary_image_url %}<div style="float:left;padding-right:10px;padding-bottom:10px;"><img src="{{scrobble.media_obj.primary_image_url}}" /></div>{% endif %}
<p><a href="{{scrobble.media_obj.get_absolute_url}}">{{scrobble.media_obj.title}}</a></p>
{% if scrobble.media_obj.subtitle %}<p><em><a href="{{scrobble.media_obj.subtitle.get_absolute_url}}">{{scrobble.media_obj.subtitle}}</a></em></p>{% endif %}
<p><a href="{{scrobble.get_absolute_url}}">{{scrobble.media_obj}}</a></p>
{% if scrobble.logdata %}{% if scrobble.logdata.title%}<p><em>{{scrobble.logdata.title}}</em></p>{% endif %}{% endif %}
<p><small>{{scrobble.local_timestamp|naturaltime}} from {{scrobble.source}}</small></p>
<div class="progress-bar" style="margin-right:5px;">

View File

@ -15,6 +15,7 @@
<a href="{% url 'charts:charts-home' %}" class="btn btn-sm btn-outline-secondary">Charts</a>
</div>
{% endif %}
{% block grid_view_button %}
<div class="btn-group me-2">
{% if view == 'grid' %}
<button type="button" class="btn btn-sm btn-outline-secondary"><a href="?{% urlreplace view='list' %}">List View</a>
@ -22,6 +23,7 @@
<button type="button" class="btn btn-sm btn-outline-secondary"><a href="?{% urlreplace view='grid' %}">Grid View</a>
{% endif %}
</div>
{% endblock %}
{% block charts_button %}{% endblock %}
</div>
</div>

View File

@ -61,7 +61,7 @@
{% for scrobble in scrobbles.all|dictsortreversed:"timestamp" %}
<tr>
<td><a href="{{scrobble.get_absolute_url}}">{{scrobble.local_timestamp}}</a></td>
<td>{% if scrobble.logdata.long_play_complete == True %}Yes{% endif %}</td>
<td>{% if scrobble.long_play_complete == True %}Yes <small><a href="{% url 'scrobbles:longplay-finish' uuid=scrobble.uuid %}">(not complete?)</a></small>{% else %}<a href="{% url 'scrobbles:longplay-finish' uuid=scrobble.uuid %}">No</a>{% endif %}</td>
<td>{% if scrobble.in_progress %}Now reading{% else %}{{scrobble.session_pages_read}}{% endif %}</td>
<td>{% for author in scrobble.book.authors.all %}<a href="{{author.get_absolute_url}}">{{author}}</a>{% if not forloop.last %}, {% endif %}{% endfor %}</td>
</tr>

View File

@ -44,7 +44,7 @@
{% for scrobble in scrobbles.all|dictsortreversed:"timestamp" %}
<tr>
<td><a href="{{scrobble.get_absolute_url}}">{{scrobble.local_timestamp}}</a></td>
<td>{% if scrobble.logdata.long_play_complete == True %}Yes{% endif %}</td>
<td>{% if scrobble.long_play_complete == True %}Yes <small><a href="{% url 'scrobbles:longplay-finish' uuid=scrobble.uuid %}">(not complete?)</a></small>{% else %}<a href="{% url 'scrobbles:longplay-finish' uuid=scrobble.uuid %}">No</a>{% endif %}</td>
<td>{% if scrobble.in_progress %}Now reading{% else %}{{scrobble.session_pages_read}}{% endif %}</td>
</tr>
{% endfor %}

View File

@ -1,4 +1,5 @@
{% extends "base_list.html" %}
{% load static %}
{% block title %}Charts{% if period_str %} - {{ period_str }}{% endif %}{% endblock %}
@ -12,11 +13,46 @@
.nav-tabs {
cursor: pointer;
}
.image-wrapper {
contain: content;
}
.image-wrapper :hover {
background: rgba(0,0,0,0.3);
}
.caption {
position: fixed;
top: 5px;
left: 5px;
padding: 3px;
font-size: 90%;
color: white;
background: rgba(0,0,0,0.4);
}
.caption-medium {
position: fixed;
top: 5px;
left: 5px;
padding: 3px;
font-size: 75%;
color: white;
background: rgba(0,0,0,0.4);
}
.caption-small {
position: fixed;
top: 5px;
left: 5px;
padding: 3px;
font-size: 60%;
color: white;
background: rgba(0,0,0,0.4);
}
</style>
{% endblock %}
{% block lists %}
{% block grid_view_button %}{% endblock %}
<div class="row mb-3">
<div class="col-12">
<a href="{% url 'charts:spotify-tracks' %}" class="btn btn-sm btn-outline-secondary">🎵 Spotify Tracks</a>
@ -24,10 +60,6 @@
</div>
</div>
{% if chart_type == "maloja" %}
{% include "scrobbles/_top_charts.html" %}
{% else %}
<div class="row">
<div class="col-12">
<div class="btn-group mb-3" role="group">
@ -88,43 +120,9 @@
</div>
{% endif %}
{% include "scrobbles/_top_charts.html" %}
<div class="row">
{% if charts.artist %}
<div class="col-md-6 col-lg-4 chart-section">
<h3>🎤 Top Artists</h3>
<ul class="list-group">
{% for chart in charts.artist|slice:":20" %}
<li class="list-group-item d-flex justify-content-between align-items-center">
<span class="me-2"><strong>#{{chart.rank}}</strong></span>
<a href="{{chart.artist.get_absolute_url}}">{{chart.artist.name}}</a>
<span class="badge bg-primary rounded-pill">{{chart.count}}</span>
</li>
{% endfor %}
<li class="list-group-item">
<a href="{% url 'charts:chart-detail' 'artist' %}">View all &raquo;</a>
</li>
</ul>
</div>
{% endif %}
{% if charts.album %}
<div class="col-md-6 col-lg-4 chart-section">
<h3>💿 Top Albums</h3>
<ul class="list-group">
{% for chart in charts.album|slice:":20" %}
<li class="list-group-item d-flex justify-content-between align-items-center">
<span class="me-2"><strong>#{{chart.rank}}</strong></span>
<a href="{{chart.album.get_absolute_url}}">{{chart.album.name}}</a>
<span class="badge bg-primary rounded-pill">{{chart.count}}</span>
</li>
{% endfor %}
<li class="list-group-item">
<a href="{% url 'charts:chart-detail' 'album' %}">View all &raquo;</a>
</li>
</ul>
</div>
{% endif %}
{% if charts.track %}
<div class="col-md-6 col-lg-4 chart-section">
<h3>🎵 Top Tracks</h3>
@ -143,24 +141,6 @@
</div>
{% endif %}
{% if charts.tv_series %}
<div class="col-md-6 col-lg-4 chart-section">
<h3>📺 Top TV Series</h3>
<ul class="list-group">
{% for chart in charts.tv_series|slice:":20" %}
<li class="list-group-item d-flex justify-content-between align-items-center">
<span class="me-2"><strong>#{{chart.rank}}</strong></span>
<a href="{{chart.tv_series.get_absolute_url}}">{{chart.tv_series.name}}</a>
<span class="badge bg-primary rounded-pill">{{chart.count}}</span>
</li>
{% endfor %}
<li class="list-group-item">
<a href="{% url 'charts:chart-detail' 'tv_series' %}">View all &raquo;</a>
</li>
</ul>
</div>
{% endif %}
{% if charts.video %}
<div class="col-md-6 col-lg-4 chart-section">
<h3>🎬 Top Videos</h3>
@ -314,6 +294,4 @@
</div>
{% endif %}
{% endif %}
{% endblock %}

View File

@ -106,6 +106,7 @@
{% block title %}Settings{% endblock %}
{% block details %}
<p class="settings-link"><a href="{% url 'people:person_form' %}">Manage People</a></p>
<p class="settings-link"><a href="{% url 'profiles:bulk_visibility' %}">Scrobble Visibility Settings</a></p>
<form method="post" class="settings-form">{% csrf_token %}
{% for field in form %}
{% if field.name == "enable_public_widgets" %}

Some files were not shown because too many files have changed in this diff Show More