Compare commits
109 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 04f9e00c9c | |||
| c2dabd1dac | |||
| 7a0cb8b9d0 | |||
| 1c2c570c4b | |||
| 0671ab432f | |||
| 893867419a | |||
| d9dfec81aa | |||
| 948fbc19bf | |||
| 7d708ad8a6 | |||
| e0505cb82c | |||
| ab6459e4b0 | |||
| c001248d1b | |||
| f1c777d4ef | |||
| 931488c288 | |||
| ab897fd848 | |||
| 4f189b4d66 | |||
| 1487504318 | |||
| 0655363a0d | |||
| dccc80c615 | |||
| 4f91d5b40b | |||
| cb01781615 | |||
| 1f5fada8b1 | |||
| 31888a85cb | |||
| 22d8b0787e | |||
| 8cc559752b | |||
| db3f9696fa | |||
| 407d570c82 | |||
| 033239260f | |||
| 9f854dc735 | |||
| f29272a853 | |||
| 4e56d9420a | |||
| 852a257159 | |||
| 68ff230f13 | |||
| 57a952a6d1 | |||
| 718fcf7392 | |||
| 52adcf83c7 | |||
| 0061623f7e | |||
| ec73e5151e | |||
| 2c90dd38b5 | |||
| c6b1e42d7a | |||
| fcf86d5b3f | |||
| 6fde9ec8d2 | |||
| 0f1882b21f | |||
| e819a2db0d | |||
| e03cf6c9b1 | |||
| 471e70ff7f | |||
| 255e335d7a | |||
| c8cf80b513 | |||
| b4180afbed | |||
| 37112babbb | |||
| fb775f2f58 | |||
| b26470c279 | |||
| d3b9ec815b | |||
| 19f2b5e801 | |||
| 9e3288a5ff | |||
| 06465919dd | |||
| 253e58eb48 | |||
| 5393996e47 | |||
| 1624f01e11 | |||
| 535dead7e8 | |||
| 3b97d49227 | |||
| ea7b0946bb | |||
| b8384166de | |||
| d2705758c6 | |||
| f4368c31f3 | |||
| 57f273b0cc | |||
| ac82292200 | |||
| 6a8432c08f | |||
| 5a2c41155c | |||
| 83a046111b | |||
| ab10758f40 | |||
| 88f16f0aaa | |||
| c1744fab37 | |||
| 042a3eb737 | |||
| 01d25e1b55 | |||
| c0be131e3d | |||
| 7d3f615ed7 | |||
| c2138b3ac6 | |||
| 947713d44a | |||
| 12b76837a3 | |||
| 102494ede7 | |||
| 96bda8d4ad | |||
| 46956d06d8 | |||
| 8a28d0675b | |||
| 5f6e75b14e | |||
| a96a42cdbf | |||
| c7f5d7d384 | |||
| d5830f5cd1 | |||
| c71b51fdb8 | |||
| 935d059a20 | |||
| 25776eb495 | |||
| 5ac4625af9 | |||
| a731427f6e | |||
| 410da163fe | |||
| a171192a6f | |||
| c16b61db40 | |||
| 29cb6a4991 | |||
| 25c28e8335 | |||
| 25626be3b6 | |||
| 0a880a2f2f | |||
| 248d3f2d3e | |||
| e243fec679 | |||
| de9b4ee9c1 | |||
| bf9a6a9679 | |||
| 709fed5cfe | |||
| b7df6299d0 | |||
| be16d513ef | |||
| 15d27f6d94 | |||
| c8292d1c06 |
@ -15,6 +15,8 @@ ro class method should call the utility function.
|
||||
Be sure to check pyproject.toml for project defaults. Specifically for black and
|
||||
isort expectations.
|
||||
|
||||
Imports in python files should always be top level if possible.
|
||||
|
||||
All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
|
||||
|
||||
In local development, environment variables for various sensitive values live in a .envrc file
|
||||
|
||||
884
PROJECT.org
884
PROJECT.org
@ -88,8 +88,8 @@ fetching and simple saving.
|
||||
*** Metadata sources
|
||||
**** Scraper
|
||||
|
||||
* Backlog [0/15] :vrobbler:project:personal:
|
||||
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
|
||||
* Backlog [0/22] :vrobbler:project:personal:
|
||||
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :bug:music:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
|
||||
:END:
|
||||
@ -387,11 +387,11 @@ fetching and simple saving.
|
||||
}
|
||||
}
|
||||
#+end_src
|
||||
** TODO [#C] User should be able to enable auto trail tracking via amail reader with Garmin LiveTrack URLs :vrobbler:trails:project:feature:personal:
|
||||
** TODO [#C] Allow auto trail tracking via email with Garmin LiveTrack URLs :trails:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 133bcf71-078f-4efa-a029-1eae4b4d146d
|
||||
:END:
|
||||
** TODO [#C] Fix exporting so it works reliably :exporting:project:feature:
|
||||
** TODO [#C] Fix exporting so it works reliably :exporting:feature:
|
||||
|
||||
*** Description
|
||||
|
||||
@ -405,8 +405,7 @@ placed in the media directory:
|
||||
And this should all be done in a celery task that is just kicked off by the
|
||||
"Export" button on the frontend
|
||||
|
||||
|
||||
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
|
||||
** TODO [#B] Add AllTrails as a source for Trail data :trails:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 39313362-cdfe-46e7-bbd4-9139a65c0b3c
|
||||
:END:
|
||||
@ -416,7 +415,7 @@ Pretty clear, I would love to make trails more useful. Historically I wasn't
|
||||
hiking a lot, which made the source for this a bit silly. But it's clear that
|
||||
AllTrails is the best source, though having TrailForks is nice to.
|
||||
|
||||
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:personal:project:
|
||||
** TODO [#B] Add `garmin_activity_id` to the TrailLogData class :trails:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 5a4fb0f8-0555-40ec-b06f-93c26bd686f4
|
||||
:END:
|
||||
@ -440,18 +439,669 @@ added.
|
||||
They should also probably support markdown formatting and that should be
|
||||
displayed in the template.
|
||||
|
||||
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
|
||||
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :books:feature:export:
|
||||
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
|
||||
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :vrobbler:books:feature:comicbook:personal:project:
|
||||
** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
|
||||
:PROPERTIES:
|
||||
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
|
||||
:END:
|
||||
|
||||
- Note taken on [2025-09-25 Thu 10:51]
|
||||
*** Description
|
||||
|
||||
Currently we have webdav able to import post types of file-based incoming data,
|
||||
usually in the form of CSVs but also gpx files, bgstats json files, and
|
||||
audioscrobbler TSV files.
|
||||
|
||||
What if the user could specify via their profile (settings) which imports they
|
||||
wanted to use IMAP for and which ones they wanted to use WebDAV for.
|
||||
|
||||
Then we'd have two celery tasks that would be kicked off periodically via
|
||||
celerybeat, one for IMAP imports every 12 minutes and one for WebDAV every 3
|
||||
minutes. Both would be responsible for checking if a user has an configured
|
||||
imports of their type, check if an import needs to run, and dispatch the
|
||||
needed import celery task. This is how the WebDAV celery task currently works.
|
||||
|
||||
This would also be an opporunity to clean up the code around WebDAV imports
|
||||
and make them more re-usable for other import services.
|
||||
|
||||
** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||
|
||||
** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||
|
||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||
enrich further.
|
||||
|
||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||
log a warning and move on.
|
||||
|
||||
** TODO [#B] Allow browing a user's favorited media :favorites:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 5c2cf004-d01f-4576-9bbb-974235e7408a
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
We should have a global view `/favorites/` that shows the logged in users's
|
||||
favorited media objects.
|
||||
|
||||
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
||||
** TODO [#B] Find page numbers for comic books from ComicVine :feature:books:
|
||||
:PROPERTIES:
|
||||
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
|
||||
:END:
|
||||
** TODO [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
|
||||
** TODO [#C] Implement loguru into project :feature:loguru:logging:
|
||||
:PROPERTIES:
|
||||
:ID: efcd0c0a-db81-4518-9c23-5505d59e8ef5
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Would be great to formalize how we log so we can search for errors and such more
|
||||
easily. And our exposure to PII is really low at this point in the project,
|
||||
so we can probably use backtrace=True and diagnose=True to help us root cause
|
||||
bugs faster.
|
||||
|
||||
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :books:feature:comicbook:
|
||||
:PROPERTIES:
|
||||
:ID: b3cc57ca-3d2c-468d-ab7c-c47f1120309b
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
||||
|
||||
** TODO [#C] Make podcast date format configurable in settings :podcasts:configuration:
|
||||
:PROPERTIES:
|
||||
:ID: b01a94f8-328f-41ed-a62e-8b99c755b82d
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
=PODCAST_DATE_FORMAT= is hardcoded to ="YYYY-MM-DD"=. Should be in Django settings or environment variables for deploy-specific configuration.
|
||||
|
||||
File: ~vrobbler/apps/podcasts/utils.py~ (line 13)
|
||||
|
||||
** TODO [#C] Extract zombie scrobble query into custom manager :refactoring:manager:
|
||||
:PROPERTIES:
|
||||
:ID: 79c874e1-ca6f-4bce-9259-e3eebdca8a41
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
The zombie scrobble cleanup query lives in a utility function. Should be a
|
||||
custom model manager method (e.g. =Scrobble.objects.zombies()=).
|
||||
|
||||
File: ~vrobbler/apps/scrobbles/utils.py~ (line 204)
|
||||
|
||||
** TODO [#C] Allow profile to set start of week :profiles:configuration:
|
||||
:PROPERTIES:
|
||||
:ID: 0449279a-9550-430e-be0c-816df7273080
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
=start_of_week()= and =end_of_week()= use Monday as default. Should be a user
|
||||
profile setting for different cultural week start conventions.
|
||||
|
||||
File: ~vrobbler/apps/profiles/utils.py~ (lines 39, 44)
|
||||
|
||||
** TODO [#C] Add constants for data dictionary keys (multiple files) :refactoring:constants:
|
||||
:PROPERTIES:
|
||||
:ID: d4415f9b-620a-4be7-925d-fa71c02ba1d1
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
Multiple files use magic string literals for dict keys. Should be extracted to
|
||||
named constants for maintainability.
|
||||
|
||||
- Files:
|
||||
- ~vrobbler/apps/locations/models.py~ (line 63) -- ="lat"=, ="lon"= etc.
|
||||
- ~vrobbler/apps/webpages/models.py~ (line 290) -- ="url"=
|
||||
- ~vrobbler/apps/scrobbles/importers/tsv.py~ (line 55) -- ="S"= completion status
|
||||
|
||||
|
||||
** TODO [#A] Deduplicate BGG plays before posting :boardgames:bgg:duplication:
|
||||
:PROPERTIES:
|
||||
:ID: e9b842bf-0049-42e7-a060-f3ebd0067d2f
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
No check for existing BGG plays before posting, which can create duplicates.
|
||||
Should look up past plays by =bggeek_id= first.
|
||||
|
||||
File: ~vrobbler/apps/boardgames/bgg.py~ (line 117)
|
||||
|
||||
** TODO [#C] Clean up naming of =bgsplay= parsing :importers:refactoring:
|
||||
:PROPERTIES:
|
||||
:ID: c751dbbc-464a-4e63-9fe3-e034303f7b54
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
We should rename `email_scrobble_board_game` to reflect the fact that it's just
|
||||
a helper method to create board game scrobbles given a json blob. It's
|
||||
independent of the email flow it was originally creatdd for
|
||||
|
||||
** TODO [#B] Is there way to create unique slugs for media instances :media_types:
|
||||
|
||||
* Version 58.0 [1/1]
|
||||
** DONE [#B] Add scrobbling of Papers via webpages with doi.org links in them :feature:papers:
|
||||
:PROPERTIES:
|
||||
:ID: d30bb8aa-eefd-002c-38d5-3f2fcef345f2
|
||||
:END:
|
||||
|
||||
* Version 57.1 [1/1]
|
||||
** DONE [#A] Write poetry lock file :bug:deps:
|
||||
:PROPERTIES:
|
||||
:ID: 0f5a6f4b-a486-ba7e-bbce-f7581274398c
|
||||
:END:
|
||||
|
||||
* Version 57.0 [5/5]
|
||||
** DONE [#A] Scrobble button on some media list pages dont work :bug:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: a3a5c707-2e3d-a6b1-0f7f-4c6f7433aa1f
|
||||
:END:
|
||||
** DONE [#B] Use HTMx to update the Now Playing widget :feature:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 5f5631fc-9ee1-d5a5-d0f8-94fea6fbbfa4
|
||||
:END:
|
||||
** DONE [#B] Add a live page that updates the scrobble list via JS polling :feature:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 58790d76-dc6e-8aa5-2dc0-e64fe786fbf1
|
||||
:END:
|
||||
** DONE [#A] Turns out we cant cache the now playing widget :bug:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 9ce669ea-c000-cdfe-a634-ad5cdaeae81c
|
||||
:END:
|
||||
** DONE [#C] What would it look like to add an MCP server to expose scrobbles and media items? :mcpserver:feature:
|
||||
:PROPERTIES:
|
||||
:ID: c5fca159-c7e0-5795-7c05-bbc48f539650
|
||||
:END:
|
||||
|
||||
* Version 56.4 [3/3]
|
||||
** DONE [#B] Add ability to do reverse address lookup on lat-long pairs :geolocations:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 86c071ff-7638-41ba-6b65-1382df1cb5aa
|
||||
:END:
|
||||
** DONE [#B] Add address fields to GeoLocation :addresses:geolocation:
|
||||
:PROPERTIES:
|
||||
:ID: a55ae508-07ab-ccdd-e453-846bd3fca6fb
|
||||
:END:
|
||||
** DONE [#B] Add better detail template for Disc Golf Courses :discgolf:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 12cee67c-f723-0fa3-848a-cbc6e4d65fc3
|
||||
:END:
|
||||
|
||||
* Version 56.3 [1/1]
|
||||
** DONE [#A] Fix bug in importer script from discgolf being added :bug:
|
||||
:PROPERTIES:
|
||||
:ID: c3733f96-18f1-eef8-f5d9-edaf97e35623
|
||||
:END:
|
||||
|
||||
* Version 56.2 [1/1]
|
||||
** DONE [#A] Fix bug in creating people when importing course plays :discgolf:bug:
|
||||
:PROPERTIES:
|
||||
:ID: 255e9886-098b-39ae-1077-25e43223660e
|
||||
:END:
|
||||
|
||||
* Version 56.1 [1/1]
|
||||
** DONE [#A] Add tests to discgolf app :discgolf:tests:
|
||||
:PROPERTIES:
|
||||
:ID: 28e8344e-c3cf-19af-ce1c-cb821d4fcb5f
|
||||
:END:
|
||||
|
||||
* Version 56.0 [1/1]
|
||||
** DONE [#B] Add DiscGolf as a scrobbleable media :discgolf:
|
||||
:PROPERTIES:
|
||||
:ID: 8cdde5d3-0ae5-7d5a-99d2-200c86afae03
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
I have a csv file fro the uDisc disc golf scoring app that looks like:
|
||||
|
||||
Singles round. Note second row is the par for the course
|
||||
#+begin_src csv
|
||||
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
|
||||
Par,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,27,,,3,3,3,3,3,3,3,3,3
|
||||
Colin Powell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,30,3,,2,3,4,4,3,4,3,3,4
|
||||
Asa Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,5,4,4,8,5,5,4,4,5
|
||||
Emma Sweet,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,4,5,6,3,4,3,5,6
|
||||
Jane Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,4,5,5,5,5,5,4,6,5
|
||||
Nabby Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,59,32,,6,6,7,7,6,7,6,6,8
|
||||
Silas Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,5,4,5,3,5,4,4,6`
|
||||
#+end_src
|
||||
|
||||
Teams of two or more persons. Note second row is the par for the course
|
||||
#+begin_src csv
|
||||
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
|
||||
Par,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,27,,,3,3,3,3,3,3,3,3,3
|
||||
Colin Powell + Asa Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,29,2,,3,4,2,3,2,3,5,3,4
|
||||
Emma Sweet + Jane Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,28,1,,4,3,4,2,3,4,3,3,2
|
||||
#+end_src
|
||||
|
||||
We should add a new app called discgolf that has the following data models:
|
||||
|
||||
- DiscGolfRound - scrobblable media + course_id, round_type (Singles, Teams)
|
||||
- DiscGolfCourse - name, layoutname, number_of_holes
|
||||
|
||||
And the logdata for a DiscGolfOuting scrobble should have:
|
||||
+ {person: {hole_number: score}, total: int}
|
||||
+ {team: {name: "", people: [person, person], hole_number: score}, total: int}
|
||||
+ weather
|
||||
+ fun_factor (miserable, not great, so-so, good, excellent, party time)
|
||||
|
||||
|
||||
* Version 55.6 [1/1]
|
||||
** DONE [#A] Figure out why historical Lastfm imports don't work :importers:lastfm:music:
|
||||
:PROPERTIES:
|
||||
:ID: 71b18c1b-de96-6d93-20fa-de2ec0df1288
|
||||
:END:
|
||||
|
||||
* Version 55.5 [1/1]
|
||||
** DONE [#B] Fix bug in lastfm import for new users :importers:lastfm:music:
|
||||
:PROPERTIES:
|
||||
:ID: d034966d-0c7f-e512-4cf8-7329c9026b6f
|
||||
:END:
|
||||
|
||||
* Version 55.4 [1/1]
|
||||
** DONE [#A] Tighten up the speed of startup and first request :perf:
|
||||
:PROPERTIES:
|
||||
:ID: 9ee8834c-6be2-d04b-df6d-56375504083f
|
||||
:END:
|
||||
|
||||
|
||||
* Version 55.3 [3/3]
|
||||
** DONE [#C] =alt_names= feature for artists (commented out / dead code) :music:dead-code:
|
||||
:PROPERTIES:
|
||||
:ID: e22060a2-5f7a-4f33-9056-309ecd27159c
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
File: ~vrobbler/apps/music/models.py~ (line 236)
|
||||
|
||||
An entire block of code for tracking alternate artist names is commented
|
||||
out. The TODO questions whether it even works. Review: either implement
|
||||
properly or remove the dead code.
|
||||
|
||||
** DONE [#A] Put chart rebuilds in a lower priority task queue :charts:tasks:
|
||||
:PROPERTIES:
|
||||
:ID: 43c90de0-fc1c-1139-dac7-9b7c82006b2e
|
||||
:END:
|
||||
** DONE [#A] Check for existing book scrobble and update page count :books:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 1a0609bc-6b16-4da4-96c1-59588229e4b4
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
File: ~vrobbler/apps/scrobbles/scrobblers.py~ (line 330)
|
||||
|
||||
When scrobbling a book (comic), the code doesn't check for prior scrobbles to
|
||||
update reading progress. Needed for proper page-count tracking.
|
||||
|
||||
|
||||
* Version 55.2 [2/2]
|
||||
** DONE [#A] Fix bug in scrobble id in calendar view :templates:
|
||||
:PROPERTIES:
|
||||
:ID: 8cb34852-b18f-e794-cd9b-fb1ecad70a0d
|
||||
:END:
|
||||
** DONE [#A] Video game cleanup script should clear out broken images :metadata:videogames:
|
||||
:PROPERTIES:
|
||||
:ID: ca1f1ea9-0f79-082c-5ff7-867671faff4b
|
||||
:END:
|
||||
|
||||
* Version 55.1 [1/1]
|
||||
** DONE [#A] Clean up metadata scrapping for video games :metadata:videogames:
|
||||
:PROPERTIES:
|
||||
:ID: fbc421b5-21a3-4aed-9062-c59192ead065
|
||||
:END:
|
||||
|
||||
* Version 55.0 [3/3]
|
||||
** DONE [#B] Use pk ID for scrobble detail view, not uuid :scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 9cc3b285-e478-041e-394b-3d550aefbe1d
|
||||
:END:
|
||||
** DONE [#B] Display videogame screenshots on scrobble detail if they exist :videogames:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 0406d082-20f6-0d12-76e2-f281c4801468
|
||||
:END:
|
||||
** DONE [#B] Add autotagging to webpages based on domain, title :webpages:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: f658435b-f7a0-42e6-b9f6-226678a77a55
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
For easier filtering, like we do with tasks, we should auto tag WebPage instances
|
||||
based on the domain name split part by periods (so news.ycombinator.com tags: news, ycombinator, com)
|
||||
|
||||
And also based on the nouns in the title.
|
||||
|
||||
|
||||
* Version 54.5 [1/1]
|
||||
** DONE Fix bug in generating mood trends :trends:
|
||||
:PROPERTIES:
|
||||
:ID: 8e75abfa-8e70-d85b-00a4-a4813bbce879
|
||||
:END:
|
||||
|
||||
* Version 54.4 [2/2]
|
||||
** DONE [#A] Remove all-time trends :trends:
|
||||
:PROPERTIES:
|
||||
:ID: 53b231d1-7677-8cd3-1d88-dae110aba1e6
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
All time trends take forever to calculate and don't provide too much data
|
||||
|
||||
** DONE [#B] Add a trend around moods :moods:trends:
|
||||
:PROPERTIES:
|
||||
:ID: fba3f4ae-8f97-ee0b-e762-31630884518a
|
||||
:END:
|
||||
|
||||
* Version 54.3 [1/1]
|
||||
** DONE [#B] Fix bug in series metadata cleanup script :videos:metadta:
|
||||
:PROPERTIES:
|
||||
:ID: 85448702-907c-5d63-f5af-7795661d7c46
|
||||
:END:
|
||||
|
||||
* Version 54.2 [4/4]
|
||||
** DONE [#B] Add script to clean up TV series metadata :videos:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: a468b328-59d9-f84b-9ddb-087216783453
|
||||
:END:
|
||||
** DONE [#A] Update youtube video detail pages with links to channel :videos:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 8b87cb42-09e5-a3f5-136f-182f967fa81f
|
||||
:END:
|
||||
** DONE [#A] Concurrent reading trend does not consolidate on single book :trends:reading:
|
||||
:PROPERTIES:
|
||||
:ID: fe220f55-7e0d-2a17-2477-a5aa7c4a1f2c
|
||||
:END:
|
||||
** DONE [#B] Trends dont seem to look very far back :trends:
|
||||
:PROPERTIES:
|
||||
:ID: ffcfba3f-5a93-9ee0-9680-666e6eccd684
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Specificially, looking at reading-pace when run on prod, it claims that I've
|
||||
only had one reading session without music. Which may be true, but perhaps we
|
||||
need to indicate what the time frame we're looking at is (month, week, year)
|
||||
and provide a way to jump back and forward through time, same as charts.
|
||||
|
||||
|
||||
* Version 54.1 [1/1]
|
||||
** DONE [#A] Concurrent listening trend is inefficient and should be disabled :trends:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 4aa3b719-6b22-cae9-85f0-fac67b4fc753
|
||||
:END:
|
||||
|
||||
* Version 54.0 [3/3]
|
||||
** DONE [#B] Add peak hour, weekly rhythm and activity dist trends :trends:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 5fa52fac-d5f0-4369-bcaa-589c886b07d3
|
||||
:END:
|
||||
|
||||
** DONE [#A] Implement YouTube channel info scraping :videos:youtube:stub:
|
||||
:PROPERTIES:
|
||||
:ID: 1d3beafd-62cb-4735-a465-edb37bf885db
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
File: ~vrobbler/apps/videos/models.py~ (line 140)
|
||||
|
||||
=Video.fix_metadata()= is a stub that logs "Not implemented yet" and returns.
|
||||
Needs actual implementation to scrape channel metadata from YouTube.
|
||||
|
||||
** DONE [#A] Fix Amazon book scraper :amazon:scraper:broken:
|
||||
:PROPERTIES:
|
||||
:ID: c38aba25-0171-49ab-a9f3-acf2003da429
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
File: ~vrobbler/apps/books/amazon.py~ (line 56)
|
||||
|
||||
The =scrape_data_from_amazon()= function is likely broken due to Amazon blocking
|
||||
scrapers and changing HTML structure. Needs rewrite or replacement with a proper
|
||||
API.
|
||||
|
||||
|
||||
* Version 53.1 [1/1]
|
||||
** DONE [#A] Error with loading logdict :scrobbles:bug:logdata:
|
||||
:PROPERTIES:
|
||||
:ID: 92d4fa16-4b90-47e0-95ae-472bdca582ce
|
||||
:END:
|
||||
|
||||
|
||||
* Version 53.0 [5/5]
|
||||
** DONE [#B] Add a /trends/ page that shows trends based on scrobble data :feature:trends:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 03e9fe30-2bc6-4062-bb24-e95b98daf05b
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
This project is a bit invovled. But we should add a top level URL `trends` that shows
|
||||
various trends as defined either in a static settings file, or dynamically via a database table.
|
||||
|
||||
Trends could be things like doing multiple things at the same time, like while driving, what
|
||||
did we listen to this week, or while running, what were listening to this week?
|
||||
|
||||
Or more complicated trends like, how time per page changes based on the book I was reading, or if I was doing something else (music or sport event) while reading.
|
||||
|
||||
** DONE [#B] Notify users when Last.fm import completes :importers:notifications:
|
||||
:PROPERTIES:
|
||||
:ID: 92846b36-54c5-4b78-9c57-bdc401045fbe
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
After a bulk import from Last.fm, users receive no confirmation. Should add a notification (in-app, email, or similar).
|
||||
|
||||
File: ~vrobbler/apps/scrobbles/importers/lastfm.py~ (line 96)
|
||||
|
||||
** DONE [#C] Cleaner =GeoLocationLogData= deserialization :models:refactoring:
|
||||
:PROPERTIES:
|
||||
:ID: 85465dbf-69b3-48cb-9df0-cd076c4470ab
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
Currently special-cases =GeoLocationLogData= by reaching into a nested ="movement_detection"= key. Should be handled at the LogData dataclass level.
|
||||
|
||||
File: ~vrobbler/apps/scrobbles/models.py~ (line 977)
|
||||
|
||||
** DONE [#B] Webpage scrobbles should diff existing webpages content :webpages:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: 25576197-258f-48d6-bfe9-e4172a0a1898
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Webpages change content between scrobbles. The current model stores the webpage content once, the
|
||||
first time it's scrobbled. When a page has been seen before, we should move the existing content
|
||||
to a new model HistoricalWebPage with the following fields:
|
||||
|
||||
webpage_id -> FK to WebPage
|
||||
date -> date from existing WebPage content
|
||||
domain -> same as existing WebPage content
|
||||
extract -> copy of existing WebPage content
|
||||
|
||||
Once the HistoricalWebPage instance is successfully created, the new extract data
|
||||
should be saved into the WebPage instance.
|
||||
|
||||
|
||||
** DONE [#B] Make ArchiveBox push asynchronous :archivebox:async:
|
||||
:PROPERTIES:
|
||||
:ID: 17c116a7-5952-db37-e56c-2987c2fc456b
|
||||
:END:
|
||||
*** Description
|
||||
|
||||
=push_to_archivebox()= runs synchronously during the request. Should be moved to a
|
||||
Celery task or similar background worker.
|
||||
|
||||
File: ~vrobbler/apps/webpages/models.py~ (line 133)
|
||||
|
||||
|
||||
* Version 52.2 [1/1]
|
||||
** DONE [#A] Fix bug in recomputing long play seconds taking forever :bug:longplay:commands:
|
||||
:PROPERTIES:
|
||||
:ID: 0a813cf9-17fb-dbd7-b5a7-7410d9bd4d8c
|
||||
:END:
|
||||
|
||||
* Version 52.1 [1/1]
|
||||
** DONE [#C] Show time per scrobble in long play lists and total time playing :templates:longplay:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: b3d16230-8ec5-46db-b166-59e98d0ee06c
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Long play time should be show in the table of scrobbles on a media detail page.
|
||||
The total time spent in a long play that's either no completed yet or completed
|
||||
should be displayed as well. If completed, the date finished should be shown as
|
||||
well.
|
||||
|
||||
|
||||
* Version 52.0 [5/5]
|
||||
** DONE [#B] Allow marking media as long play complete from detail page :templates:scrobbles:longplay:
|
||||
:PROPERTIES:
|
||||
:ID: 2c314768-be97-9b10-d13c-9cfd0f38a64e
|
||||
:END:
|
||||
** DONE [#A] Fix how long play scrobbles are tracked :scrobbles:longplay:serial:
|
||||
:PROPERTIES:
|
||||
:ID: 908b0493-cabf-40c1-825f-cd59a8ad0f7a
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Currently we have this idea of "long_play" scrobbles but there's a lot missing
|
||||
to tie it together.
|
||||
|
||||
What we'd prefer is that when a new scrobble is added for a media_type that
|
||||
`is_long_play` the most recent scrobble finished is added as the
|
||||
`last_serial_scrobble` to the log data. But all the other long play stuff exsits
|
||||
as data model fields. We should add `long_play_last_scrobble` as a FK to this
|
||||
scrobble when creating a new longplay scrobble.
|
||||
|
||||
Additionally, `long_play_seconds` we should have a recompute management command
|
||||
to walk backward from `long_play_last_scrobble` until a `long_play_complete`
|
||||
scrobble is found (exclusive) and save the time.
|
||||
|
||||
We should also ony use `long_play_complete` field on the scrobble ... some
|
||||
logdatas have a similar field, but we should make sure that we always use the
|
||||
model field to determine if a long play is finished.
|
||||
|
||||
This should include a command to clean up long play data to consolidate around
|
||||
the `long_play_complete` field.
|
||||
|
||||
|
||||
** DONE [#B] Paginate or limite scrobbles on media admin pages :admin:scrobbles:media:
|
||||
:PROPERTIES:
|
||||
:ID: f02e487b-d7ed-4834-838a-303560f2ad3b
|
||||
:END:
|
||||
|
||||
** DONE [#B] Clean up books admin :admin:books:bug:
|
||||
:PROPERTIES:
|
||||
:ID: 7539bee6-0a52-26f6-ebc6-5554ac49a716
|
||||
:END:
|
||||
** DONE [#B] Clean up favorites admin :admin:favorites:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: f2be0c69-1bf8-b5a3-5269-9c8ea873361d
|
||||
:END:
|
||||
|
||||
|
||||
*** Description
|
||||
|
||||
Some FK lookups in admin should be raw_id_fields.
|
||||
|
||||
|
||||
|
||||
* Version 51.4 [1/1]
|
||||
** DONE [#A] Clean up metadata comicbook enrichment :bug:comics:books:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: cd875450-7117-78ca-8be4-9c8b73037dba
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Still getting wonky results with some comicbooks. Would be nice to be able to
|
||||
tag a Book as a comicbook, and also gather volume information. I also noticed
|
||||
that some books that are found in OL never get their comicvine_id populated. We
|
||||
should make sure we always have comicvine_ids if available.
|
||||
|
||||
|
||||
* Version 51.3 [1/1]
|
||||
** DONE [#A] Improve speed of index and chart pages :bug:scrobbles:perf:
|
||||
:PROPERTIES:
|
||||
:ID: 031a23f8-7c02-4926-9884-6654ceca16c2
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Over the last few releases, the home page and charts pages have gotten really
|
||||
slow.
|
||||
|
||||
We should look into what's causing the slowness and maybe do more agressive
|
||||
query optimization or caching.
|
||||
|
||||
|
||||
* Version 51.2 [2/2]
|
||||
** DONE [#A] Fix bug where last page of book gets separate scrobble :bug:books:importers:koreader:
|
||||
:PROPERTIES:
|
||||
:ID: e13e0b4c-461e-e5a9-c685-b972f4e262e5
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
The new KoReader code is working great to import books with the correct timezone
|
||||
and what-not. But it has a weird artifact of creating one extra scrobble for the
|
||||
last page read. Need to button that up.
|
||||
|
||||
** DONE [#B] Fix metadata scraping for books :books:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: ea416a69-a8a8-4d05-b7d4-0a3470820e34
|
||||
:END:
|
||||
|
||||
|
||||
* Version 51.1 [1/1]
|
||||
** DONE [#A] Fix scrobbling comic books :books:scrobbles:bug:
|
||||
:PROPERTIES:
|
||||
:ID: 8dfbff19-3fa4-f3b8-21c7-7a416498000c
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
At some point logdata and log got confused, and now when you try
|
||||
to scrobble a comic book, it just throws errors. We should look
|
||||
into where the confusion happened and fix it.
|
||||
|
||||
|
||||
* Version 51.0 [3/3]
|
||||
** DONE [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
|
||||
:PROPERTIES:
|
||||
:ID: 79758cba-a440-48b6-a637-efb88827acf2
|
||||
:END:
|
||||
@ -477,80 +1127,212 @@ whatever time KoReader reports, we need to know, given the date and the user
|
||||
profile's historic timezone, how many hours to adjust the KoReader time to get
|
||||
to GMT to save it in the database.
|
||||
|
||||
** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
|
||||
** DONE [#A] Fix book scrobbles where page_data is a list :bug:books:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
|
||||
:ID: 35b323fa-ccc0-4009-b227-8a0f12bbd469
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Currently we have webdav able to import post types of file-based incoming data,
|
||||
usually in the form of CSVs but also gpx files, bgstats json files, and
|
||||
audioscrobbler TSV files.
|
||||
Comic scrobbling is currently kind of janky. Most of the problems boil down to them
|
||||
storing saved page data in a list of dicts rather than a dict keyed off of the page number that
|
||||
was read.
|
||||
|
||||
What if the user could specify via their profile (settings) which imports they
|
||||
wanted to use IMAP for and which ones they wanted to use WebDAV for.
|
||||
We need to adjust comic scrobbling to use a dict of pages keyed off the page number, and also
|
||||
write a migration script that runs as a data migration to update any book scrobbles that may
|
||||
have page_data as a list.
|
||||
|
||||
Then we'd have two celery tasks that would be kicked off periodically via
|
||||
celerybeat, one for IMAP imports every 12 minutes and one for WebDAV every 3
|
||||
minutes. Both would be responsible for checking if a user has an configured
|
||||
imports of their type, check if an import needs to run, and dispatch the
|
||||
needed import celery task. This is how the WebDAV celery task currently works.
|
||||
*** Example data
|
||||
#+begin_src python
|
||||
{"notes": null, "page_end": 27, "page_data": [{"notes": null, "end_ts": 1771815895, "duration": 14, "start_ts": 1771815881, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815908, "duration": 13, "start_ts": 1771815895, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815913, "duration": 5, "start_ts": 1771815908, "description": null, "page_number": "2"}, {"notes": null, "end_ts": 1771815933, "duration": 20, "start_ts": 1771815913, "description": null, "page_number": "3"}, {"notes": null, "end_ts": 1771815945, "duration": 12, "start_ts": 1771815933, "description": null, "page_number": "4"}, {"notes": null, "end_ts": 1771815983, "duration": 38, "start_ts": 1771815945, "description": null, "page_number": "5"}, {"notes": null, "end_ts": 1771816007, "duration": 24, "start_ts": 1771815983, "description": null, "page_number": "6"}, {"notes": null, "end_ts": 1771816011, "duration": 4, "start_ts": 1771816007, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816013, "duration": 2, "start_ts": 1771816011, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816052, "duration": 39, "start_ts": 1771816013, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816127, "duration": 75, "start_ts": 1771816052, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816134, "duration": 7, "start_ts": 1771816127, "description": null, "page_number": "9"}, {"notes": null, "end_ts": 1771816196, "duration": 62, "start_ts": 1771816134, "description": null, "page_number": "10"}, {"notes": null, "end_ts": 1771816262, "duration": 66, "start_ts": 1771816196, "description": null, "page_number": "11"}, {"notes": null, "end_ts": 1771816293, "duration": 31, "start_ts": 1771816262, "description": null, "page_number": "12"}, {"notes": null, "end_ts": 1771816322, "duration": 29, "start_ts": 1771816293, "description": null, "page_number": "13"}, {"notes": null, "end_ts": 1771816330, "duration": 8, "start_ts": 1771816322, "description": null, "page_number": "14"}, {"notes": null, "end_ts": 1771816368, "duration": 38, "start_ts": 1771816330, "description": null, "page_number": "15"}, {"notes": null, "end_ts": 1771816388, "duration": 20, "start_ts": 1771816368, "description": null, "page_number": "16"}, {"notes": null, "end_ts": 1771816482, "duration": 94, "start_ts": 1771816388, "description": null, "page_number": "17"}, {"notes": null, "end_ts": 1771816550, "duration": 68, "start_ts": 1771816482, "description": null, "page_number": "18"}, {"notes": null, "end_ts": 1771816567, "duration": 17, "start_ts": 1771816550, "description": null, "page_number": "19"}, {"notes": null, "end_ts": 1771816586, "duration": 19, "start_ts": 1771816567, "description": null, "page_number": "20"}, {"notes": null, "end_ts": 1771816597, "duration": 11, "start_ts": 1771816586, "description": null, "page_number": "21"}, {"notes": null, "end_ts": 1771816616, "duration": 19, "start_ts": 1771816597, "description": null, "page_number": "22"}, {"notes": null, "end_ts": 1771816640, "duration": 24, "start_ts": 1771816616, "description": null, "page_number": "23"}, {"notes": null, "end_ts": 1771816690, "duration": 50, "start_ts": 1771816640, "description": null, "page_number": "24"}, {"notes": null, "end_ts": 1771816702, "duration": 12, "start_ts": 1771816690, "description": null, "page_number": "25"}, {"notes": null, "end_ts": 1771816823, "duration": 121, "start_ts": 1771816702, "description": null, "page_number": "26"}, {"notes": null, "end_ts": null, "duration": null, "start_ts": 1771816823, "description": null, "page_number": "27"}], "page_start": 1, "pages_read": 27, "resume_url": null, "description": null, "koreader_hash": null, "long_play_complete": false}
|
||||
|
||||
This would also be an opporunity to clean up the code around WebDAV imports
|
||||
and make them more re-usable for other import services.
|
||||
|
||||
|
||||
** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||
#+end_src
|
||||
** DONE [#A] Lichess imports do not set default visbility :boardgames:bug:importers:lichess:
|
||||
:PROPERTIES:
|
||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||
:ID: a78f7c72-a20a-8db2-cde0-d92a731d4fba
|
||||
:END:
|
||||
|
||||
* Version 50.2 [2/2]
|
||||
** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers:
|
||||
:PROPERTIES:
|
||||
:ID: b50141fd-cda6-4a3a-afd3-cd8499e7523e
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||
When you read a single page in a book in Koreader and try to import it, the scrobble is only
|
||||
created the day after, not on the day of the reading.
|
||||
|
||||
|
||||
** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||
** DONE [#A] Fix bugs in celery tasks causing imports to fail :bug:celery:tasks:
|
||||
:PROPERTIES:
|
||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||
:ID: d1171cb0-6413-44b8-a68a-019a4d2fb285
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||
Seems like all celery tasks are failing for different reasons except the chart
|
||||
updates.
|
||||
|
||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||
enrich further.
|
||||
*** Errors
|
||||
**** scrobbles.tasks.send_notification_for_in_progress
|
||||
#+begin_src bash
|
||||
KeyError: 'track'
|
||||
|
||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||
log a warning and move on.
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
** TODO [#A] Allow updating all a user's scrobble visibility at once :scrobbles:sharing:feature:
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
|
||||
return self.cursor.execute(sql, params)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
psycopg2.errors.UndefinedColumn: column music_track.artist_id does not exist
|
||||
LINE 1: ..."."title", "music_track"."base_run_time_seconds", "music_tra...
|
||||
^
|
||||
HINT: Perhaps you meant to reference the column "music_track.artist_fk_id".
|
||||
#+end_src
|
||||
|
||||
**** scrobbles.tasks.import_from_webdav_all_users
|
||||
#+begin_src bash
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/importers/webdav.py", line 166, in scan_webdav_for_koreader
|
||||
if last_import and last_import.webdav_etag and remote_etag:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^
|
||||
AttributeError: 'KoReaderImport' object has no attribute 'webdav_etag'
|
||||
#+end_src
|
||||
|
||||
**** scrobbles.tasks.process_bgstats_import
|
||||
#+begin_src bash
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
|
||||
return self.cursor.execute(sql, params)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
psycopg2.errors.NotNullViolation: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
|
||||
DETAIL: Failing row contains (374463, 2026-06-11 13:27:06.528319+00, 2026-06-11 13:27:06.52834+00, 2026-06-11 13:17:34+00, 180, f, f, BG Stats, 1, null, t, {"players": [{"new": false, "win": false, "rank": 0, "role": "",..., null, null, null, 8e73ceec-b731-4623-9637-712bbf9f76ce, null, null, null, null, , null, BoardGame, 324, null, null, America/New_York, null, null, , null, null, , null, null, null, null, null, null, null, null, null, null, null).
|
||||
#+end_src
|
||||
|
||||
* Version 50.1 [1/1]
|
||||
** DONE [#B] Fix bug in charts where only #1 is displayed :charts:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 7136dffb-e6b7-184b-48ac-bb09bae0b0f0
|
||||
:END:
|
||||
|
||||
* Version 50.0 [2/2]
|
||||
** DONE [#A] Allow updating all a user's scrobble visibility at once :scrobbles:sharing:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 9ed2ec65-bf69-4300-965c-6a7d3ef7ea03
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
We now have the ability to share or unshare scrobbles and create private links. We should add a toggle
|
||||
in the user's settings that will bulk make all their scrobbles public or private, so that a user
|
||||
can either share everything, or lock their account down.
|
||||
We now have the ability to share or unshare scrobbles and create private links.
|
||||
We should add a toggle in the user's settings that will bulk make all their
|
||||
scrobbles public or private, so that a user can either share everything, or lock
|
||||
their account down.
|
||||
|
||||
This should not affect scrobbles that are in the "Shared" visibility state.
|
||||
|
||||
And users should be able to also control whether all scrobbles of a specific
|
||||
type are shared or not. Maybe this could be a JSONField in profile that contains
|
||||
a media_type key with a visibility type for a value, and if it's not present,
|
||||
sharing defaults to private?
|
||||
|
||||
Additionally, users's should have links in their settings to see what scrobbles
|
||||
are either public, shared or private. Probably this could be done with a
|
||||
?visbility=<> filter on the /scrobbles/ page.
|
||||
|
||||
*** Changes
|
||||
|
||||
- Added `media_type_visibility` JSONField to UserProfile (migration 0038)
|
||||
- Created `BulkVisibilityView` at `/settings/visibility/` with:
|
||||
- Radio toggle to make all non-shared scrobbles Public or Private
|
||||
- Per-media-type dropdown for each of the 20 media types (inherit/public/shared/private)
|
||||
- Created `BulkVisibilityForm` with dynamic media_type fields
|
||||
- Created `profiles/visibility_settings.html` template with visibility stats + filter links
|
||||
- Added link from main settings page to visibility settings
|
||||
- Added `?visibility=` filter support to `ScrobbleListView` (public/shared/private)
|
||||
- Added filter indicator to `scrobble_all_list.html`
|
||||
- Updated `Scrobble.create()` to check `user.profile.media_type_visibility` for media-type-specific defaults before falling back to PRIVATE
|
||||
|
||||
|
||||
|
||||
** DONE [#A] Replace columsn of Top Artists, Tracks and Series with Maloja widget :templates:charts:
|
||||
:PROPERTIES:
|
||||
:ID: 3946afb1-932c-46fe-a188-f4c9add1a491
|
||||
:END:
|
||||
|
||||
|
||||
*** Description
|
||||
|
||||
The tables are fine, but Maloja widgets are better. We should drop the top track table, add top albums
|
||||
and replace top artists and top tv series with the Maloja style widgets.
|
||||
|
||||
|
||||
* Version 49.1 [1/1]
|
||||
** DONE [#A] Fix bug with missing default visbility for scrobbles :bug:scrobbles:sharing:
|
||||
:PROPERTIES:
|
||||
:ID: 20843992-6453-9a9a-cde6-2c2b6677db23
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
We can't scrobble anything now because visbility is not null, but has no default
|
||||
value.
|
||||
|
||||
*** Notes
|
||||
- Note taken on [2026-06-09 Tue 13:14]
|
||||
The full stack trace:
|
||||
#+begin_src sh
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 1430, in create_or_update
|
||||
elif "log" in scrobble_data.keys() and scrobble.log:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 1583, in create
|
||||
)
|
||||
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
|
||||
return getattr(self.get_queryset(), name)(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 658, in create
|
||||
obj.save(force_insert=True, using=self.db)
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/models.py", line 870, in save
|
||||
if self.media_obj:
|
||||
^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django_extensions/db/models.py", line 22, in save
|
||||
super().save(**kwargs)
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 814, in save
|
||||
self.save_base(
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 877, in save_base
|
||||
updated = self._save_table(
|
||||
^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1020, in _save_table
|
||||
results = self._do_insert(
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/base.py", line 1061, in _do_insert
|
||||
return manager._insert(
|
||||
^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/manager.py", line 87, in manager_method
|
||||
return getattr(self.get_queryset(), name)(*args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1805, in _insert
|
||||
return query.get_compiler(using=using).execute_sql(returning_fields)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1822, in execute_sql
|
||||
cursor.execute(sql, params)
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
|
||||
return self._execute_with_wrappers(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
|
||||
return executor(sql, params, many, context)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
|
||||
with self.db.wrap_database_errors:
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
|
||||
raise dj_exc_value.with_traceback(traceback) from exc_value
|
||||
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
|
||||
return self.cursor.execute(sql, params)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
django.db.utils.IntegrityError: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
|
||||
DETAIL: Failing row contains (373813, 2026-06-09 17:13:38.11355+00, 2026-06-09 17:13:38.113566+00, 2026-06-09 17:13:36+00, 0, f, f, Todoist, 1, null, t, {"title": "Animal chores", "labels": ["chore", "farm"], "todoist..., null, null, null, 68680dbf-f9a9-476c-b1c7-adbd231bbab6, null, null, null, null, , null, Task, null, null, null, America/New_York, null, null, , null, null, , 72, null, null, null, null, null, null, null, null, null, null).
|
||||
#+end_src
|
||||
|
||||
* Version 49.0 [1/1]
|
||||
** DONE [#A] Fix broken tests with new sharing and add tests :scrobbles:sharing:tests:
|
||||
:PROPERTIES:
|
||||
|
||||
2
Procfile
2
Procfile
@ -1,2 +1,2 @@
|
||||
web: python manage.py runserver 0.0.0.0:8014
|
||||
worker: celery -A vrobbler worker -l DEBUG
|
||||
worker: celery -A vrobbler worker -Q default,charts -l DEBUG
|
||||
|
||||
96
data/play-example.json
Normal file
96
data/play-example.json
Normal file
@ -0,0 +1,96 @@
|
||||
{
|
||||
"about": "This is a Play file that can be read by Board Game Stats. If you see this text, try to use a share, export or open-in function to open it with Board Game Stats.",
|
||||
"players": [
|
||||
{
|
||||
"uuid": "31f8b92e-11d8-4162-88b1-fd9c79eea249",
|
||||
"id": 2,
|
||||
"name": "Colin Powell",
|
||||
"isAnonymous": false,
|
||||
"modificationDate": "2025-10-18 08:32:40",
|
||||
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"AnnikaHeller_Capybara.webp\",\"shape\":[1,1,1,1],\"color\":[0.6,0.20000000298023224,0.9803921580314636]}}"
|
||||
},
|
||||
{
|
||||
"uuid": "dd2d1881-84ab-474c-a6b2-3045d034dc40",
|
||||
"id": 3,
|
||||
"name": "Silas Sewell",
|
||||
"isAnonymous": false,
|
||||
"modificationDate": "2026-01-18 12:27:12",
|
||||
"metaData": "{\"isNpc\":0,\"playerAvatar\":{\"image\":\"RoryMuldoon_07.webp\",\"shape\":[1,1,1,1],\"color\":[0,0,0.1835034190722739]}}"
|
||||
}
|
||||
],
|
||||
"locations": [
|
||||
{
|
||||
"uuid": "14f7389c-767f-4725-9b35-906c407b293c",
|
||||
"id": 3,
|
||||
"name": "Timberwyck Farm",
|
||||
"modificationDate": "2025-07-01 18:03:38"
|
||||
}
|
||||
],
|
||||
"games": [
|
||||
{
|
||||
"uuid": "9e431cdd-b325-4061-a875-d415d46342c0",
|
||||
"id": 1046,
|
||||
"name": "Sweet Takes",
|
||||
"modificationDate": "2026-04-11 16:25:35",
|
||||
"cooperative": false,
|
||||
"highestWins": true,
|
||||
"noPoints": false,
|
||||
"usesTeams": false,
|
||||
"urlThumb": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__small/img/gmGqlmwe9fniqYpniGwhg5RUgVQ=/fit-in/200x150/filters:strip_icc()/pic8784202.jpg",
|
||||
"urlImage": "https://cf.geekdo-images.com/l4HILZn5iLbceQeDph4G5A__original/img/b4IU8WIEWRpacbXp0FHG9HfFRpw=/0x0/filters:format(jpeg)/pic8784202.jpg",
|
||||
"bggName": "Sweet Takes",
|
||||
"bggYear": 2023,
|
||||
"bggId": 407581,
|
||||
"designers": "Hisashi Hayashi",
|
||||
"isBaseGame": 1,
|
||||
"isExpansion": 0,
|
||||
"rating": 67,
|
||||
"minPlayerCount": 2,
|
||||
"maxPlayerCount": 5,
|
||||
"minPlayTime": 15,
|
||||
"maxPlayTime": 15,
|
||||
"minAge": 8
|
||||
}
|
||||
],
|
||||
"plays": [
|
||||
{
|
||||
"uuid": "7b2fd583-e8f2-40fe-9565-90178390b87e",
|
||||
"modificationDate": "2026-04-16 20:18:03",
|
||||
"entryDate": "2026-04-16 20:13:33",
|
||||
"playDate": "2026-04-16 20:13:33",
|
||||
"usesTeams": false,
|
||||
"durationMin": 4,
|
||||
"ignored": false,
|
||||
"manualWinner": false,
|
||||
"rounds": 0,
|
||||
"locationRefId": 3,
|
||||
"gameRefId": 1046,
|
||||
"board": "",
|
||||
"scoringSetting": 1,
|
||||
"metaData": "{\"playerRefId\":2,\"playGameBggVersion\":\"{\\\"versionId\\\":0,\\\"versionName\\\":\\\"\\\",\\\"imageUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"thumbUrl\\\":\\\"https:\\\\\\/\\\\\\/cf.geekdo-images.com\\\\\\/l4HILZn5iLbceQeDph4G5A__small\\\\\\/img\\\\\\/gmGqlmwe9fniqYpniGwhg5RUgVQ=\\\\\\/fit-in\\\\\\/200x150\\\\\\/filters:strip_icc()\\\\\\/pic8784202.jpg\\\",\\\"yearPublished\\\":0}\",\"playUsedGameCopy\":2}",
|
||||
"playerScores": [
|
||||
{
|
||||
"score": "27",
|
||||
"winner": false,
|
||||
"newPlayer": false,
|
||||
"startPlayer": false,
|
||||
"playerRefId": 2,
|
||||
"role": "",
|
||||
"rank": 0,
|
||||
"seatOrder": 0
|
||||
},
|
||||
{
|
||||
"score": "36",
|
||||
"winner": true,
|
||||
"newPlayer": true,
|
||||
"startPlayer": false,
|
||||
"playerRefId": 3,
|
||||
"rank": 0,
|
||||
"seatOrder": 0
|
||||
}
|
||||
],
|
||||
"expansionPlays": []
|
||||
}
|
||||
],
|
||||
"userInfo": { "meRefId": 2 }
|
||||
}
|
||||
BIN
data/statistics.sqlite3
Normal file
BIN
data/statistics.sqlite3
Normal file
Binary file not shown.
454
poetry.lock
generated
454
poetry.lock
generated
@ -1512,6 +1512,25 @@ files = [
|
||||
{file = "django_mathfilters-1.0.0-py3-none-any.whl", hash = "sha256:64200a21bb249fbf27be601d4bbb788779e09c6e063170c097cd82c4d18ebb83"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "django-mcp-server"
|
||||
version = "0.5.7"
|
||||
description = "Django MCP Server is a Django extensions to easily enable AI Agents to interact with Django Apps through the Model Context Protocol it works equally well on WSGI and ASGI"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "django_mcp_server-0.5.7-py3-none-any.whl", hash = "sha256:04b58bf02623aaee59708c3661ffe17981acd4532587c38b6cfe2c9e7090c6d3"},
|
||||
{file = "django_mcp_server-0.5.7.tar.gz", hash = "sha256:5077f8fabf5fb621b5ce490afd0db60f21e57b3a451ed14a9f44aef545ea4eee"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
django = ">=4.0"
|
||||
djangorestframework = ">=3.15.0"
|
||||
inflection = ">=0.5.1,<0.6.0"
|
||||
mcp = ">=1.8.0"
|
||||
uritemplate = ">=4.1.1,<5.0.0"
|
||||
|
||||
[[package]]
|
||||
name = "django-oauth-toolkit"
|
||||
version = "3.2.0"
|
||||
@ -2156,6 +2175,18 @@ http2 = ["h2 (>=3,<5)"]
|
||||
socks = ["socksio (==1.*)"]
|
||||
zstd = ["zstandard (>=0.18.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "httpx-sse"
|
||||
version = "0.4.3"
|
||||
description = "Consume Server-Sent Event (SSE) messages with HTTPX."
|
||||
optional = false
|
||||
python-versions = ">=3.9"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "httpx_sse-0.4.3-py3-none-any.whl", hash = "sha256:0ac1c9fe3c0afad2e0ebb25a934a59f4c7823b60792691f779fad2c5568830fc"},
|
||||
{file = "httpx_sse-0.4.3.tar.gz", hash = "sha256:9b1ed0127459a66014aec3c56bebd93da3c1bc8bb6618c8082039a44889a755d"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "idna"
|
||||
version = "3.11"
|
||||
@ -2196,6 +2227,18 @@ perf = ["ipython"]
|
||||
test = ["packaging", "pyfakefs", "pytest (>=6,!=8.1.*)", "pytest-perf (>=0.9.2)"]
|
||||
type = ["pytest-mypy (>=1.0.1) ; platform_python_implementation != \"PyPy\""]
|
||||
|
||||
[[package]]
|
||||
name = "inflection"
|
||||
version = "0.5.1"
|
||||
description = "A port of Ruby on Rails inflector to Python"
|
||||
optional = false
|
||||
python-versions = ">=3.5"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "inflection-0.5.1-py2.py3-none-any.whl", hash = "sha256:f38b2b640938a4f35ade69ac3d053042959b62a0f1076a5bbaa1b9526605a8a2"},
|
||||
{file = "inflection-0.5.1.tar.gz", hash = "sha256:1a29730d366e996aaacffb2f1f1cb9593dc38e2ddd30c91250c6dde09ea9b417"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "iniconfig"
|
||||
version = "2.3.0"
|
||||
@ -2399,6 +2442,43 @@ files = [
|
||||
{file = "jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "jsonschema"
|
||||
version = "4.26.0"
|
||||
description = "An implementation of JSON Schema validation for Python"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "jsonschema-4.26.0-py3-none-any.whl", hash = "sha256:d489f15263b8d200f8387e64b4c3a75f06629559fb73deb8fdfb525f2dab50ce"},
|
||||
{file = "jsonschema-4.26.0.tar.gz", hash = "sha256:0c26707e2efad8aa1bfc5b7ce170f3fccc2e4918ff85989ba9ffa9facb2be326"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
attrs = ">=22.2.0"
|
||||
jsonschema-specifications = ">=2023.03.6"
|
||||
referencing = ">=0.28.4"
|
||||
rpds-py = ">=0.25.0"
|
||||
|
||||
[package.extras]
|
||||
format = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3987", "uri-template", "webcolors (>=1.11)"]
|
||||
format-nongpl = ["fqdn", "idna", "isoduration", "jsonpointer (>1.13)", "rfc3339-validator", "rfc3986-validator (>0.1.0)", "rfc3987-syntax (>=1.1.0)", "uri-template", "webcolors (>=24.6.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "jsonschema-specifications"
|
||||
version = "2025.9.1"
|
||||
description = "The JSON Schema meta-schemas and vocabularies, exposed as a Registry"
|
||||
optional = false
|
||||
python-versions = ">=3.9"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "jsonschema_specifications-2025.9.1-py3-none-any.whl", hash = "sha256:98802fee3a11ee76ecaca44429fda8a41bff98b00a0f2838151b113f210cc6fe"},
|
||||
{file = "jsonschema_specifications-2025.9.1.tar.gz", hash = "sha256:b540987f239e745613c7a9176f3edb72b832a4ac465cf02712288397832b5e8d"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
referencing = ">=0.31.0"
|
||||
|
||||
[[package]]
|
||||
name = "jstyleson"
|
||||
version = "0.0.2"
|
||||
@ -2745,6 +2825,48 @@ files = [
|
||||
{file = "mccabe-0.7.0.tar.gz", hash = "sha256:348e0240c33b60bbdf4e523192ef919f28cb2c3d7d5c7794f74009290f236325"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "mcp"
|
||||
version = "1.28.0"
|
||||
description = "Model Context Protocol SDK"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "mcp-1.28.0-py3-none-any.whl", hash = "sha256:9c1e7cf3a9125557e418ecd4fed8e9adddce81b0dfdae4d6601d700f5beb71a4"},
|
||||
{file = "mcp-1.28.0.tar.gz", hash = "sha256:559d3f9943674cafbe5744c5d3794f3237e8b47f9bbc58e20c0fad680d8487c2"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
anyio = ">=4.5"
|
||||
httpx = ">=0.27.1,<1.0.0"
|
||||
httpx-sse = ">=0.4"
|
||||
jsonschema = ">=4.20.0"
|
||||
pydantic = [
|
||||
{version = ">=2.11.0,<3.0.0", markers = "python_version < \"3.14\""},
|
||||
{version = ">=2.12.0,<3.0.0", markers = "python_version >= \"3.14\""},
|
||||
]
|
||||
pydantic-settings = ">=2.5.2"
|
||||
pyjwt = {version = ">=2.10.1", extras = ["crypto"]}
|
||||
python-multipart = ">=0.0.9"
|
||||
pywin32 = [
|
||||
{version = ">=310", markers = "sys_platform == \"win32\" and python_version < \"3.14\""},
|
||||
{version = ">=311", markers = "sys_platform == \"win32\" and python_version >= \"3.14\""},
|
||||
]
|
||||
sse-starlette = ">=1.6.1"
|
||||
starlette = [
|
||||
{version = ">=0.27", markers = "python_version < \"3.14\""},
|
||||
{version = ">=0.48.0", markers = "python_version >= \"3.14\""},
|
||||
]
|
||||
typing-extensions = ">=4.9.0"
|
||||
typing-inspection = ">=0.4.1"
|
||||
uvicorn = {version = ">=0.31.1", markers = "sys_platform != \"emscripten\""}
|
||||
|
||||
[package.extras]
|
||||
cli = ["python-dotenv (>=1.0.0)", "typer (>=0.16.0)"]
|
||||
rich = ["rich (>=13.9.4)"]
|
||||
ws = ["websockets (>=15.0.1)"]
|
||||
|
||||
[[package]]
|
||||
name = "mdurl"
|
||||
version = "0.1.2"
|
||||
@ -3980,6 +4102,30 @@ files = [
|
||||
[package.dependencies]
|
||||
typing-extensions = ">=4.14.1"
|
||||
|
||||
[[package]]
|
||||
name = "pydantic-settings"
|
||||
version = "2.14.2"
|
||||
description = "Settings management using Pydantic"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "pydantic_settings-2.14.2-py3-none-any.whl", hash = "sha256:a20c97b37910b6550d5ea50fbcc2d4187defe58cd57070b73863d069419c9440"},
|
||||
{file = "pydantic_settings-2.14.2.tar.gz", hash = "sha256:c19dd64b19097f1de80184f0cc7b0272a13ae6e170cbf240a3e27e381ed14a5f"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
pydantic = ">=2.7.0"
|
||||
python-dotenv = ">=0.21.0"
|
||||
typing-inspection = ">=0.4.0"
|
||||
|
||||
[package.extras]
|
||||
aws-secrets-manager = ["boto3 (>=1.35.0)", "types-boto3[secretsmanager]"]
|
||||
azure-key-vault = ["azure-identity (>=1.16.0)", "azure-keyvault-secrets (>=4.8.0)"]
|
||||
gcp-secret-manager = ["google-cloud-secret-manager (>=2.23.1)"]
|
||||
toml = ["tomli (>=2.0.1)"]
|
||||
yaml = ["pyyaml (>=6.0.1)"]
|
||||
|
||||
[[package]]
|
||||
name = "pyflakes"
|
||||
version = "3.4.0"
|
||||
@ -4270,6 +4416,29 @@ six = "*"
|
||||
[package.extras]
|
||||
testing = ["filelock"]
|
||||
|
||||
[[package]]
|
||||
name = "python-amazon-paapi"
|
||||
version = "6.3.0"
|
||||
description = "Amazon Product Advertising API 5.0 wrapper for Python"
|
||||
optional = false
|
||||
python-versions = ">=3.9"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "python_amazon_paapi-6.3.0-py3-none-any.whl", hash = "sha256:b7cd852084a49d53c3ba2195531fccbc8c7f4124b2e82e2fda02b53d3b8de521"},
|
||||
{file = "python_amazon_paapi-6.3.0.tar.gz", hash = "sha256:e525d69efcbe4f9566ec2b9b43fa3183c484d166d3852edb38b4df9c0b19cf1f"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
certifi = ">=2023.0.0"
|
||||
pydantic = ">=2.0.0"
|
||||
python-dateutil = ">=2.8.0"
|
||||
requests = ">=2.28.0"
|
||||
six = ">=1.16.0"
|
||||
urllib3 = ">=1.26.0,<3"
|
||||
|
||||
[package.extras]
|
||||
async = ["httpx (>=0.27.0)", "typing-extensions (>=4.15.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "python-dateutil"
|
||||
version = "2.9.0.post0"
|
||||
@ -4307,14 +4476,14 @@ testing = ["covdefaults (>=2.3)", "coverage (>=7.5.4)", "pytest (>=8.3.5)", "pyt
|
||||
|
||||
[[package]]
|
||||
name = "python-dotenv"
|
||||
version = "0.20.0"
|
||||
version = "1.2.2"
|
||||
description = "Read key-value pairs from a .env file and set them as environment variables"
|
||||
optional = false
|
||||
python-versions = ">=3.5"
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "python-dotenv-0.20.0.tar.gz", hash = "sha256:b7e3b04a59693c42c36f9ab1cc2acc46fa5df8c78e178fc33a8d4cd05c8d498f"},
|
||||
{file = "python_dotenv-0.20.0-py3-none-any.whl", hash = "sha256:d92a187be61fe482e4fd675b6d52200e7be63a12b724abbf931a40ce4fa92938"},
|
||||
{file = "python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a"},
|
||||
{file = "python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3"},
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
@ -4332,6 +4501,18 @@ files = [
|
||||
{file = "python_json_logger-2.0.7-py3-none-any.whl", hash = "sha256:f380b826a991ebbe3de4d897aeec42760035ac760345e57b812938dc8b35e2bd"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "python-multipart"
|
||||
version = "0.0.32"
|
||||
description = "A streaming multipart parser for Python"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "python_multipart-0.0.32-py3-none-any.whl", hash = "sha256:ff6d3f776f16878c894e52e107296ffc890e913c611b1a4ec6c44e2821fe2e23"},
|
||||
{file = "python_multipart-0.0.32.tar.gz", hash = "sha256:be54b7f3fa167bb83e4fcd936b887b708f4e57fe75911c02aebf53efaf8d938e"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "python3-openid"
|
||||
version = "3.2.0"
|
||||
@ -4363,6 +4544,38 @@ files = [
|
||||
{file = "pytz-2022.7.1.tar.gz", hash = "sha256:01a0681c4b9684a28304615eba55d1ab31ae00bf68ec157ec3708a8182dbbcd0"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pywin32"
|
||||
version = "312"
|
||||
description = "Python for Windows Extensions"
|
||||
optional = false
|
||||
python-versions = ">=3.9"
|
||||
groups = ["main"]
|
||||
markers = "sys_platform == \"win32\""
|
||||
files = [
|
||||
{file = "pywin32-312-cp310-cp310-win32.whl", hash = "sha256:772235332b5d1024c696f11cea1ae4be7930f0a8b894bb43db14e3f435f1ff7e"},
|
||||
{file = "pywin32-312-cp310-cp310-win_amd64.whl", hash = "sha256:5dbc35d2b5320dc07f25fa31269cfb767471002b17de5eb067d03da68c7cb2db"},
|
||||
{file = "pywin32-312-cp310-cp310-win_arm64.whl", hash = "sha256:3020656e34f1cf7faeb7bccd2b84653a607c6ff0c55ada85e6487d61716deabd"},
|
||||
{file = "pywin32-312-cp311-cp311-win32.whl", hash = "sha256:17948aeadbdb091f0ced6ef0841620794e68327b94ee415571c1203594b7215c"},
|
||||
{file = "pywin32-312-cp311-cp311-win_amd64.whl", hash = "sha256:d11417d84412f859b722fad0841b3614459ed0047f7542d8362e77884f6b6e8a"},
|
||||
{file = "pywin32-312-cp311-cp311-win_arm64.whl", hash = "sha256:b2200a054ca6d6625c4842fc56a4976a4b47f96b73dbe5538c3f813a80359f47"},
|
||||
{file = "pywin32-312-cp312-cp312-win32.whl", hash = "sha256:dab4f65ac9c4e48400a2a0530c46c3c579cd5905ecd11b80692373915269208b"},
|
||||
{file = "pywin32-312-cp312-cp312-win_amd64.whl", hash = "sha256:b457f6d628a47e8a7346ce22acb7e1a46a4a78b52e1d17e1af56871bd19a93bc"},
|
||||
{file = "pywin32-312-cp312-cp312-win_arm64.whl", hash = "sha256:6017c58e12f6809fbb0555b75df144c2922a9ffd18e4b9b5afa863b6c1a9d950"},
|
||||
{file = "pywin32-312-cp313-cp313-win32.whl", hash = "sha256:7a27df850933d16a8eabfbaeb73d52b273e2da667f80d70b01a89d1f6828d02c"},
|
||||
{file = "pywin32-312-cp313-cp313-win_amd64.whl", hash = "sha256:c53e878d15a1c44788082bfe712a905433473aa38f86375b7cf8b45e3acbaaf9"},
|
||||
{file = "pywin32-312-cp313-cp313-win_arm64.whl", hash = "sha256:59aba5d5940842075343a5ddc6b11f1cdf0d1567fe745290359dfbcc7c2eb831"},
|
||||
{file = "pywin32-312-cp314-cp314-win32.whl", hash = "sha256:a77a90fbb6881238d2ca9c6fd797b25817f3768fe78d214a90137ff055a75f5b"},
|
||||
{file = "pywin32-312-cp314-cp314-win_amd64.whl", hash = "sha256:a4dd3a848290ef724347b19f301045831d8e802fa4464f491b98b1e0a081432e"},
|
||||
{file = "pywin32-312-cp314-cp314-win_arm64.whl", hash = "sha256:9fce94568364e0155e6dfb781ac5d95903be8baf28670632beab1b523f300daa"},
|
||||
{file = "pywin32-312-cp315-cp315-win32.whl", hash = "sha256:5c1fbe4a937a73ae9297384a3da38518cbc694c68ad8a809b2e19acd350f03ed"},
|
||||
{file = "pywin32-312-cp315-cp315-win_amd64.whl", hash = "sha256:c2f03a0f73f804a13c2735b99392b0cd426bb4f2c4d0178e5ac966a0f21618d5"},
|
||||
{file = "pywin32-312-cp315-cp315-win_arm64.whl", hash = "sha256:a8597d28f267b39074aef51fa593530082b39cbe5a074226096857b1fed2dfb9"},
|
||||
{file = "pywin32-312-cp39-cp39-win32.whl", hash = "sha256:d620900033cc7531e50727c3c8333091df5dd3ffe6d68cdca38c03f5821408d5"},
|
||||
{file = "pywin32-312-cp39-cp39-win_amd64.whl", hash = "sha256:dc90147579a905b8635e1b0ec6514967dcb07e6e0d9c42f1477feef14cac23bb"},
|
||||
{file = "pywin32-312-cp39-cp39-win_arm64.whl", hash = "sha256:02ebca0f0242b75292e218065004310d6a477407c09fa449bfe4f6022bc0c0fc"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "pywin32-ctypes"
|
||||
version = "0.2.3"
|
||||
@ -4623,6 +4836,23 @@ async-timeout = {version = ">=4.0.2", markers = "python_full_version <= \"3.11.2
|
||||
hiredis = ["hiredis (>=1.0.0)"]
|
||||
ocsp = ["cryptography (>=36.0.1)", "pyopenssl (==20.0.1)", "requests (>=2.26.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "referencing"
|
||||
version = "0.37.0"
|
||||
description = "JSON Referencing + Python"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "referencing-0.37.0-py3-none-any.whl", hash = "sha256:381329a9f99628c9069361716891d34ad94af76e461dcb0335825aecc7692231"},
|
||||
{file = "referencing-0.37.0.tar.gz", hash = "sha256:44aefc3142c5b842538163acb373e24cce6632bd54bdb01b21ad5863489f50d8"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
attrs = ">=22.2.0"
|
||||
rpds-py = ">=0.7.0"
|
||||
typing-extensions = {version = ">=4.4.0", markers = "python_version < \"3.13\""}
|
||||
|
||||
[[package]]
|
||||
name = "regex"
|
||||
version = "2026.2.28"
|
||||
@ -4851,6 +5081,146 @@ pygments = ">=2.13.0,<3.0.0"
|
||||
[package.extras]
|
||||
jupyter = ["ipywidgets (>=7.5.1,<9)"]
|
||||
|
||||
[[package]]
|
||||
name = "rpds-py"
|
||||
version = "2026.5.1"
|
||||
description = "Python bindings to Rust's persistent data structures (rpds)"
|
||||
optional = false
|
||||
python-versions = ">=3.11"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-macosx_10_12_x86_64.whl", hash = "sha256:3397a5ed7174dc2786bb214030232fc36fe8e5584fec43a9952cc542b1a12036"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:99ab6ba7bfa2cb0f96a04e3652355bf04e3f51aceb1e943b8541dab7ba4828cc"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:d0efbe45632665e53e3db8fe1e5692db58fc5cb9bab4459d570b83efefe11164"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:01d17b29c0c23d82b1f4751147ec49cf451f1fc2554eb9ef5f957e55d2656ead"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:7559f72b94ae52659086c595dfa017cde03155f7832071d30959049052cb3ece"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:9e25b7088f9ccbfc0dfcaa52bf969300ca229e10ecf758974ebcbb080a4b37bb"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:613fc4ee9eaef26dc5840666214dd6fbcebcf32f46e76f4abc473059f4e13dda"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_31_riscv64.whl", hash = "sha256:85264a90ff4c05c1568dd65f5921c837614b67c60358fb4c17df3b7f2e90690a"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:fe71bca7d547acb17027c7fd1624ff8aae623499c498d3e7011182c4de5c25e0"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:a05fa4f41f37ec97c9c260441a940450a192f78d774d2b097eee1379f1e1246a"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:df1d2a1996755b24b9ecee92cb4d36c28f86f464a6a173349c26bab41e94b8c2"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:8895840ac4809e5f60c88fd07617cd71326e73d6e5a8aa783c5c0f7c24985de2"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-win32.whl", hash = "sha256:3684a59b158a7683aaeb8e25352e9a9dd2122cec78f2d8530266e4f91b4c7b3f"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-win_amd64.whl", hash = "sha256:7bd530e6a530bb3ea892f194fafa455f3516ac25ecf7143fd33c09be62b0470a"},
|
||||
{file = "rpds_py-2026.5.1-cp311-cp311-win_arm64.whl", hash = "sha256:0a5ae4dbe43c1076983b72616496919872ae7bbe7a1e21cc48336bc3154d130b"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-macosx_10_12_x86_64.whl", hash = "sha256:3abe24a66e57adcfa645d718063a5fa5103ecc71ddbf26d78af8f9368018ff1d"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:58b1d94308ddf0b1982f61f2eb54bf92997c9ece8a8093ef014250f4a517906c"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:0fa92420128dadce7f54bd73ba1825a273e9268fe9e35dbf7e6362890efa4e08"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:ca653c6546386227cd9800d1bef6a348099acf8db4250341da6d90f663d6dfcb"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:66c93681c4729e4e3ecba31b8179fae083ff3118841672835140338b4b9867c1"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:40ff257542e04796880e011e15cd4dc21c2599975df2aaa8f2c8495ca574e1a5"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b6825cc329b290e93c5f6a9be2393118a763f6ccf6abd83704e0c102ca583644"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_31_riscv64.whl", hash = "sha256:de42116e69cb53b911cc34aee5ab98f36c597b822545045d49e938818b99e5e4"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c0f920015df2a504bebaba6d4c31ccf3fcf942f92655c086da30b671aad19aa6"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0408a24e44feb919423dc6d9da677cb5cddb894d2ca9e763967d156d9c60fab4"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:cea68bcd53467561ae2f96a6bdad1544299ba97b5b0ddcd5ac3d376e5c781c24"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:4be8b1d2a705cc37d08256004e1d07de143fa0075c8e85a3df020b776f62b732"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-win32.whl", hash = "sha256:6736718bd4fc49cbcb538ba30516fdbef161522acefb739657d48b97bd864fed"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-win_amd64.whl", hash = "sha256:0a7d1eec967df0e9b22614a5e177622e0c89611d03727fa0cb48e45028907870"},
|
||||
{file = "rpds_py-2026.5.1-cp312-cp312-win_arm64.whl", hash = "sha256:1841d067089e117142d79b98aa0df2f08b52f2ecc1819dd2700636c0db74a473"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-macosx_10_12_x86_64.whl", hash = "sha256:efef4ac29c6ff495531eb17ee705b62841ecaa291b7c7077e848ea03e237164d"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:c39f5b67a8a2e67179ada2a954227d670fe65fa9098457f698f56ddf248709b3"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:b5c30f3f04eef4fbd362226a6f31d7c8895ca4fbb6e0b790f6890a98d8da8559"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:277f6c82f0580848796c7ecc8a7173aa3bfb928e4ff831261c2f60a81dc270db"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:63c2c4c213f1a4e3f3de28ecab029dbdee976324e729c0d7a55211be72576b02"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:3350ec808fb538fe71a1f94dfaa0e29c598dfad805ce49f0caec5ae3183c652b"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:b1b964e3ab599e718dc46c018d104b1ebc007cbc6567d827c94a687fca56d77e"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_31_riscv64.whl", hash = "sha256:19cb09fab7b7fc96b2a6e28f2e34b72a3705ff27b37edb77455316e5d3f3dc9b"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:abe76bcdba31e576cb83eeb8797aa0d882b738fef6dc65d0601fc753806a5b46"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8bff7073db3899158fff55ebf57b113a67030af26f80a18978f9f0aa60250ddf"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:8ba264fa49be666cd9cc56bf34ec7002fb3d27a4aee5bcb4d43d0d18feb1bb6f"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:4860b603ddda0475a8885499b3729e90229d480105b42651962a5397d995fa89"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-win32.whl", hash = "sha256:7944270ae71383f6e2657dd7d5ce4eeb4ac2d0059a6738f0510583d462ab4842"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-win_amd64.whl", hash = "sha256:88647f43a73c4e01be19b04ceef0c8d3a1958153604d13c773becd8016f2a0cf"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313-win_arm64.whl", hash = "sha256:453895624ecf7db7063b1004e44037522bbaef9ff6a945e59bc71662d7a03abd"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:b4e4bc98639ec915f512fde3aa7a95e0041d95d9c3cc86eea841fa63cb1e8600"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:cacedb7a6e167680acba45ad5716e89067d225dc80da0d7040cae8c81d4572fa"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:68700371c5d7ae1412862ddfa719090925c93ecf351c566d66f09d04b136ea00"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:296c799becfa849c779c8725494fe9ed94959ed886787df4364b058465bad7f0"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:d3858b908218ee108d0bbfb2095ccc237648053c9bf98affad7cb079acaf1d97"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:4fb8d2e7cb2f850b169806d61d1b991738acec96500a75c30f49caf064ce7cef"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:27b74c10ed6a8f190f4287f53bcfea348b92a84a9c9f70d30183d1e6172d580d"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_31_riscv64.whl", hash = "sha256:b9a6528956191c48c52294a592dbd4a8386d7048bdb25c0efcb6b966466c6d83"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:af03e34e860047bc7a352b842856fcf78798fbb81132cc98bd2f907ab4eb9cd2"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:fea6e836d10abbe191d557d33bd58bd5987725fe63aa1eefe557d230209855bd"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_i686.whl", hash = "sha256:fc0c0f878ea770a0a8a462456c5ad36fc9fe6358e6b76fdadc7f17575e0b8bf1"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:e0b360f316d966b048b085857630b3cc51f3db2f07b06f440eac8f695374d1e3"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-win32.whl", hash = "sha256:a2999883eedf72fdfb7520b92c7d4ec2572a71ff40239377aa604cc529eecafc"},
|
||||
{file = "rpds_py-2026.5.1-cp313-cp313t-win_amd64.whl", hash = "sha256:e07be2a9d7122bd6e82dea89814ef8dc893feb1aae97fec1630f3263bbb30e55"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-macosx_10_12_x86_64.whl", hash = "sha256:1f2c391c3059798093b65df23aca2cac150460ae9c630d99dec83d703d9485b9"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:413b424f7c4ee65ab5e5be91f5731be0f8b41a1ee2b12dfe810d716312e95a78"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:2c595a1d9255dce0599e13130d1440ab2506654f2b50294226ee06402f8fef63"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:1c27c5f6102eac8c03e7595a00827a53b271ba40a53b59ff8709170e0855ea4a"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:6c7fcf61d44cacecaf3aea542b0e053db77972a4573e7ceda16fb2b399161195"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:2c817a189d4ee14290420e5ff051e4dd6baa13f3edf84685071dee07a6d538ee"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:21846aac0ed2e0589f38c12dc44e77bb64e494b771eadbcf169cba00566ba7ba"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_31_riscv64.whl", hash = "sha256:b317c87a13f769a4e787819bd508aaa5d69aa09b0880de9af6d3a8a54571cdec"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:ce87129d9f2c14fa6c4a8601fb80eb4488c80d38a20cd13758ef11123e14995d"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:9cdddb6c1207d284d94fd1530adf57fbd797fe7c4b8704ba85f49414f2557e7d"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:4e237e139f94d3c036fd28eb9f564c99055476ff4ff05cd42be55ce349b5aa02"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:ed0954b524873214369184a9c82b0eaa45a3fbb9a798cd95b17e0d98499e7ea0"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-win32.whl", hash = "sha256:2d88621d6a7d4dfa633d21abe90f280bb205274e16b1d1e61c6ad4640b2453b7"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-win_amd64.whl", hash = "sha256:cef8ac28d26f4dda3533060c20fbf80a325458fa9fd23ea72a73cdfa8e978838"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314-win_arm64.whl", hash = "sha256:eaaea962c68cdc68d4a533ba985ab8e9484277910bbfaa2ab3ef7732667bfed8"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-macosx_10_12_x86_64.whl", hash = "sha256:21942f52dbbd5f8758bf021213d28bd45c39e873e65e2407faf5f1846f5761ad"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:f414556f6e3958300ff941e40c9f97e3dc9774ddd1b3434c475d73dd354bbed3"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:ef1013a8625c74043210190b246f5b1551e09757c1f356c6e4160ef96c5bc081"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:cc68e231a77a5f0d774ae278a1f8e55c0456501820847c1e4efb3829f3441df6"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:9baffb505aff33acc69b422a19f77806680f3c8632227d79f48de8a810d1c2c5"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b8d2f912928d426e8cfa396f7f3f8d29a59e6689c86dcca3c420730c1096322b"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:90f628283be835db980c941767d41c9a27b5239e54ba0a9c1335247e82406964"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_31_riscv64.whl", hash = "sha256:1ebb2f0ab7e16132995a72de805170e0203df0c3dd22e1ef1cd1fdd90bd7a131"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:f3df3d16ded76f1f8c9cdebd0e1ea55fdf4c23b812de189814da7cf229c22a81"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:9af8905b8f854990e40d5206aa5ac58d9b0fe0b7f351ff2bb086c20f6c8c6a47"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_i686.whl", hash = "sha256:036a36a87fb1cd3b214d11c4b3c4f7d2ddad933625dca1c900b56a057c07740a"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:62ae3853454fe9ef283a03c96c2d835d39e84b14643a9d62c82ef0fb87d702ca"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-win32.whl", hash = "sha256:6c3d771a46ec18b12af06ce36243a9a80b07a5d0515236332d90863ca8bb326a"},
|
||||
{file = "rpds_py-2026.5.1-cp314-cp314t-win_amd64.whl", hash = "sha256:c93c629be4636cf54337bd5f06c104d55e42ced54d681f6fe21ae510a65116f6"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-macosx_10_12_x86_64.whl", hash = "sha256:3574b55c604b8f75dacb007136508bbc0db406e626301778096a133327e7f2fb"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-macosx_11_0_arm64.whl", hash = "sha256:94068eb3ae6d43f5a786b7db96a406a34e6d5c24489feef32fd6e8946ea7b291"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:f3a5b10e8ce894825f380a8f1b6444cf73c294dfea62afbb2d13e3a9e630cec1"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:fc09f82e63d4bcd58149572f857a431bae851dc747e313c3b5bdf7abb907fda8"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:e10464d17df3b582745c25cec695cb9558bca2cb6ddb631aee1787fc72c767b2"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:ba05adbf15d994c38ec0b7ab32e858e5110c21e9009a00a86545fd220f84e038"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:77c004fdc7b891967106f78ddfd7b076bfe6813c6139c6fff6aed3bcaa960b26"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_31_riscv64.whl", hash = "sha256:83bcf894486c9d78dd290d3c0124ff6dd8875d3025e2090a8ec49fcc37c55fdd"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:c3df104083952a0e0c6f10de33e440eabe98fb6317d23e1a58c68f6df08d01b9"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_aarch64.whl", hash = "sha256:980450826cf22e133c57e0835070bdd0dd3f73b9b708c3ce223def2cb9469e14"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_i686.whl", hash = "sha256:205dde846f24332ab0c1188699a043b8d165b79bb84529ce272c45048ff6be01"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-musllinux_1_2_x86_64.whl", hash = "sha256:3966b82dd563176396df030f3dd52a6e54cb69b718e95e78bd555ed3d1e0185d"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-win32.whl", hash = "sha256:7818f8d0a415be74d2be3590b0a1c1f463a642f4d0217e7d10602dceef5b79aa"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-win_amd64.whl", hash = "sha256:b3cc20c0d800af78fd0fac68086e28c1856cec51ea528bb81ea851aa40d39325"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315-win_arm64.whl", hash = "sha256:3609e9939a8a76cd904cf98a3f1f13b5dc7e150adeaee89e0ea09652ea213e16"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-macosx_10_12_x86_64.whl", hash = "sha256:5d333a7127d4b307601ac37792bee01bb95c867cbfacf21b6375b804d6bbd723"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-macosx_11_0_arm64.whl", hash = "sha256:b5f077b44a4f7808520f66dae234988d867deb9aed9be5da057ce9ba831b2a41"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:55d8f9b7b78c9538fc9e04e82ec0e888ff0c3cffcfad152c77e57cd09351a98a"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:e3a8ae58895ac107ed934a6bf51e5846f95c53b9b940c2c6d310838fd5846358"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0957cf3c2b8632ec7aaebffebea8005b353cc2a237b6e2ae3c2cac0820704cfb"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:c396c1304de421050b3681ea70f371874b54d41b0151e96109758144c231e30b"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:aad1bff7f666b9598e573815affd666aac6a13a585dde336f843e33350c7fadc"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_31_riscv64.whl", hash = "sha256:656a042550878f12d45752452d47094b7cfe5ad1e9d7b87b5a22ad3ae5ff8015"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:73c4bd4f70294737b5206a3e8e30ccadbf8a60301831c8ea23eec5dbeea1ecfa"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_aarch64.whl", hash = "sha256:43bca78665423cabae77146f2fe7ce55272b6c8d55d82cca83effd42c7e13972"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_i686.whl", hash = "sha256:42d0f20e85e549c870749d0e247f0c10d318a45b7e9676d575d2dcb04a1b2e66"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-musllinux_1_2_x86_64.whl", hash = "sha256:b1be5c35683684d5331b93600c210e8367c254683d8a6df6bd21bd2da3a334fb"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-win32.whl", hash = "sha256:75808f6c38ce7749bb68cc2770161aae5045e6c6f6781a9782e74b93304399df"},
|
||||
{file = "rpds_py-2026.5.1-cp315-cp315t-win_amd64.whl", hash = "sha256:90bd6630002a1c7f09e7843dd79f0d24f3d2897cc25a753480917865d14f15b3"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-macosx_10_12_x86_64.whl", hash = "sha256:edf2765d84e42447f112ad877af8fe1db0089aaec5b28e88d6eab45e7fe99cea"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-macosx_11_0_arm64.whl", hash = "sha256:ad3773236e95f7f33991eb125224b7da66f206504d032a253a02da7e134519fb"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_17_aarch64.manylinux2014_aarch64.whl", hash = "sha256:a04df86b3f0fade39ec8fd0e0aab089b1da9fbd2b48df778a57ef96f5e7d38df"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_17_armv7l.manylinux2014_armv7l.whl", hash = "sha256:6142dbd80c4df62a5d899f0d616d417f84e0bc8d32526c8e5589019d75d028a7"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_17_ppc64le.manylinux2014_ppc64le.whl", hash = "sha256:0b35217adefe87f2fe4db7e9766cabe84744bfe9616d9667be18988928c7f2dc"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_17_s390x.manylinux2014_s390x.whl", hash = "sha256:b95d5e11fc712b752081183a55a244c03cd00570489edd7014d8899f8ceb8162"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_17_x86_64.manylinux2014_x86_64.whl", hash = "sha256:141c9498daf2ace9eda35d2b0e376f9ea8b058d84f2aef4f96fccfd449a2f251"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_31_riscv64.whl", hash = "sha256:6f249f8b860a200ad35193af961183ebe9132710484e6f6ce0cf89fd83c63a9a"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-manylinux_2_5_i686.manylinux1_i686.whl", hash = "sha256:e4abbf391a70be864920858bf360f4fb380577c9a0f732438a1996726e2c195b"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-musllinux_1_2_aarch64.whl", hash = "sha256:c74005a7bb87752acf351c93897ec63ad77a07a0da7ecad9c050e32e7286ba34"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-musllinux_1_2_i686.whl", hash = "sha256:8213afbe8a3a906fb9acb2014423fe3359ee783d0bf90995f70623a3217bfa6c"},
|
||||
{file = "rpds_py-2026.5.1-pp311-pypy311_pp73-musllinux_1_2_x86_64.whl", hash = "sha256:8c43a8a973270fd173bf48cdf80bbe66312421cba68d40845034f174f2389049"},
|
||||
{file = "rpds_py-2026.5.1.tar.gz", hash = "sha256:07b24fea40541e28570e5b795a4a38fbdcd12550c06bd0748005ecc8116ca256"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "s3transfer"
|
||||
version = "0.16.0"
|
||||
@ -4994,6 +5364,29 @@ files = [
|
||||
dev = ["build"]
|
||||
doc = ["sphinx"]
|
||||
|
||||
[[package]]
|
||||
name = "sse-starlette"
|
||||
version = "3.4.5"
|
||||
description = "SSE plugin for Starlette"
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "sse_starlette-3.4.5-py3-none-any.whl", hash = "sha256:e71bad53323f65573c3864a6c3bd0c1eb6e5f092b2e48082b0c35927d19ca296"},
|
||||
{file = "sse_starlette-3.4.5.tar.gz", hash = "sha256:83072538bc211a2f68b7b0422226c4af3e9b62e106e07034664b832ca019842a"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
anyio = ">=4.7.0"
|
||||
starlette = ">=0.49.1"
|
||||
|
||||
[package.extras]
|
||||
daphne = ["daphne (>=4.2.0)"]
|
||||
examples = ["fastapi (>=0.115.12)", "pydantic (>=2)", "uvicorn (>=0.34.0)"]
|
||||
examples-db = ["aiosqlite (>=0.21.0)", "sqlalchemy[asyncio] (>=2.0.41)"]
|
||||
granian = ["granian (>=2.3.1)"]
|
||||
uvicorn = ["uvicorn (>=0.34.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "stack-data"
|
||||
version = "0.6.3"
|
||||
@ -5014,6 +5407,25 @@ pure-eval = "*"
|
||||
[package.extras]
|
||||
tests = ["cython", "littleutils", "pygments", "pytest", "typeguard"]
|
||||
|
||||
[[package]]
|
||||
name = "starlette"
|
||||
version = "1.3.1"
|
||||
description = "The little ASGI library that shines."
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "starlette-1.3.1-py3-none-any.whl", hash = "sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6"},
|
||||
{file = "starlette-1.3.1.tar.gz", hash = "sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
anyio = ">=3.6.2,<5"
|
||||
typing-extensions = {version = ">=4.10.0", markers = "python_version < \"3.13\""}
|
||||
|
||||
[package.extras]
|
||||
full = ["httpx (>=0.27.0,<0.29.0)", "httpx2 (>=2.0.0)", "itsdangerous", "jinja2", "python-multipart (>=0.0.18)", "pyyaml"]
|
||||
|
||||
[[package]]
|
||||
name = "stevedore"
|
||||
version = "5.7.0"
|
||||
@ -5416,6 +5828,18 @@ tzdata = {version = "*", markers = "platform_system == \"Windows\""}
|
||||
[package.extras]
|
||||
devenv = ["check-manifest", "pytest (>=4.3)", "pytest-cov", "pytest-mock (>=3.3)", "zest.releaser"]
|
||||
|
||||
[[package]]
|
||||
name = "uritemplate"
|
||||
version = "4.2.0"
|
||||
description = "Implementation of RFC 6570 URI Templates"
|
||||
optional = false
|
||||
python-versions = ">=3.9"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "uritemplate-4.2.0-py3-none-any.whl", hash = "sha256:962201ba1c4edcab02e60f9a0d3821e82dfc5d2d6662a21abd533879bdb8a686"},
|
||||
{file = "uritemplate-4.2.0.tar.gz", hash = "sha256:480c2ed180878955863323eea31b0ede668795de182617fef9c6ca09e6ec9d0e"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "url-normalize"
|
||||
version = "2.2.1"
|
||||
@ -5451,6 +5875,26 @@ brotli = ["brotli (==1.0.9) ; os_name != \"nt\" and python_version < \"3\" and p
|
||||
secure = ["certifi", "cryptography (>=1.3.4)", "idna (>=2.0.0)", "ipaddress ; python_version == \"2.7\"", "pyOpenSSL (>=0.14)", "urllib3-secure-extra"]
|
||||
socks = ["PySocks (>=1.5.6,!=1.5.7,<2.0)"]
|
||||
|
||||
[[package]]
|
||||
name = "uvicorn"
|
||||
version = "0.49.0"
|
||||
description = "The lightning-fast ASGI server."
|
||||
optional = false
|
||||
python-versions = ">=3.10"
|
||||
groups = ["main"]
|
||||
markers = "sys_platform != \"emscripten\""
|
||||
files = [
|
||||
{file = "uvicorn-0.49.0-py3-none-any.whl", hash = "sha256:ba3d14c3ee7e41c6c654c46c9eb489d33213cdd30aa1696eab1374337c13f68f"},
|
||||
{file = "uvicorn-0.49.0.tar.gz", hash = "sha256:ebf4271aa580d9de97f93192d4595176df6e91f9aae919ca73e4fc07df1e66a3"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
click = ">=7.0"
|
||||
h11 = ">=0.8"
|
||||
|
||||
[package.extras]
|
||||
standard = ["colorama (>=0.4) ; sys_platform == \"win32\"", "httptools (>=0.8.0)", "python-dotenv (>=0.13)", "pyyaml (>=5.1)", "uvloop (>=0.15.1) ; sys_platform != \"win32\" and sys_platform != \"cygwin\" and platform_python_implementation != \"PyPy\"", "watchfiles (>=0.20)", "websockets (>=10.4)"]
|
||||
|
||||
[[package]]
|
||||
name = "vadersentiment"
|
||||
version = "3.3.2"
|
||||
@ -6032,4 +6476,4 @@ cffi = ["cffi (>=1.17,<2.0) ; platform_python_implementation != \"PyPy\" and pyt
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.11,<3.15"
|
||||
content-hash = "cc5b3b44071d6b0ab4f05189580232cc129b4ed694ab3f0673c3d838c3af0f8a"
|
||||
content-hash = "02f8b551a54c53fa2a51622af17ef844e40aa8505c395ed02a7b87eb05647cc5"
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "vrobbler"
|
||||
version = "49.0"
|
||||
version = "58.0"
|
||||
description = ""
|
||||
authors = ["Colin Powell <colin@unbl.ink>"]
|
||||
|
||||
@ -9,7 +9,7 @@ python = ">=3.11,<3.15"
|
||||
Django = "^4.0.3"
|
||||
django-extensions = "^3.1.5"
|
||||
python-dateutil = "^2.8.2"
|
||||
python-dotenv = "^0.20.0"
|
||||
python-dotenv = ">=0.20.0,<2"
|
||||
python-json-logger = "^2.0.2"
|
||||
colorlog = "^6.6.0"
|
||||
httpx = "<=0.27.2"
|
||||
@ -43,6 +43,7 @@ ipython = "^8.14.0"
|
||||
pendulum = "^3"
|
||||
trafilatura = "^1.6.3"
|
||||
django-imagekit = "^5.0.0"
|
||||
django-mcp-server = "^0.5.7"
|
||||
thefuzz = "^0.22.1"
|
||||
dataclass-wizard = "^0.35.0"
|
||||
webdavclient3 = "^3.14.6"
|
||||
@ -64,6 +65,7 @@ fitparse = "^1.2.0"
|
||||
lxml = ">=5.5.0"
|
||||
vaderSentiment = "^3.3.2"
|
||||
sqids = "^0.5.2"
|
||||
python-amazon-paapi = "^6.3.0"
|
||||
|
||||
[tool.poetry.group.test]
|
||||
optional = true
|
||||
|
||||
0
tests/discgolf_tests/__init__.py
Normal file
0
tests/discgolf_tests/__init__.py
Normal file
63
tests/discgolf_tests/conftest.py
Normal file
63
tests/discgolf_tests/conftest.py
Normal file
@ -0,0 +1,63 @@
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
from django.contrib.auth import get_user_model
|
||||
|
||||
User = get_user_model()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def user(db):
|
||||
return User.objects.create(email="golfer@example.com")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_singles_csv_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
Alice,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
Bob,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,4,5,12
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_singles_csv_file(udisc_singles_csv_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_singles_csv_content)
|
||||
return f.name
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_teams_csv_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
Alice+Bob,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
Charlie+Diana,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,4,5,12
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_teams_csv_file(udisc_teams_csv_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_teams_csv_content)
|
||||
return f.name
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_csv_no_par_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Alice,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_csv_no_par_file(udisc_csv_no_par_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_csv_no_par_content)
|
||||
return f.name
|
||||
102
tests/discgolf_tests/test_models.py
Normal file
102
tests/discgolf_tests/test_models.py
Normal file
@ -0,0 +1,102 @@
|
||||
from discgolf.models import DiscGolfCourse, DiscGolfLogData
|
||||
from scrobbles.dataclasses import BaseLogData
|
||||
|
||||
|
||||
class TestDiscGolfCourseModel:
|
||||
def test_create_course(self, db):
|
||||
course = DiscGolfCourse.objects.create(
|
||||
title="Maple Hill",
|
||||
layout_name="Mountains",
|
||||
number_of_holes=18,
|
||||
par_total=54,
|
||||
par_per_hole={"hole_1": 3, "hole_2": 3},
|
||||
)
|
||||
assert course.uuid is not None
|
||||
assert str(course) == "Maple Hill (Mountains)"
|
||||
assert course.subtitle == "Mountains"
|
||||
|
||||
def test_subtitle_fallback(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.subtitle == ""
|
||||
|
||||
def test_logdata_cls(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.logdata_cls is DiscGolfLogData
|
||||
assert issubclass(course.logdata_cls, BaseLogData)
|
||||
|
||||
def test_strings(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.strings.verb == "Playing"
|
||||
assert course.strings.tags == "golf"
|
||||
|
||||
def test_primary_image_url(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.primary_image_url == ""
|
||||
|
||||
def test_get_absolute_url(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
url = course.get_absolute_url()
|
||||
assert str(course.uuid) in url
|
||||
assert url.startswith("/disc-golf/")
|
||||
|
||||
def test_find_or_create_new(self, db):
|
||||
course = DiscGolfCourse.find_or_create(
|
||||
"New Course", layout_name="Default"
|
||||
)
|
||||
assert course.title == "New Course"
|
||||
assert course.layout_name == "Default"
|
||||
|
||||
def test_find_or_create_existing(self, db):
|
||||
created = DiscGolfCourse.objects.create(
|
||||
title="Existing", layout_name="Alpha"
|
||||
)
|
||||
found = DiscGolfCourse.find_or_create("Existing", layout_name="Beta")
|
||||
assert found.id == created.id
|
||||
assert found.layout_name == "Alpha"
|
||||
|
||||
def test_scrobbles_method(self, db, user):
|
||||
from datetime import datetime
|
||||
|
||||
import pytz
|
||||
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
dt1 = datetime(2026, 6, 15, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
dt2 = datetime(2026, 6, 14, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
s1 = Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt1,
|
||||
)
|
||||
s2 = Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt2,
|
||||
)
|
||||
qs = course.scrobbles(user.id)
|
||||
assert list(qs) == [s1, s2]
|
||||
|
||||
|
||||
class TestDiscGolfLogData:
|
||||
def test_basic_logdata(self):
|
||||
data = DiscGolfLogData()
|
||||
assert data.scores is None
|
||||
assert data.weather is None
|
||||
assert data.fun_factor is None
|
||||
assert data.course_name is None
|
||||
|
||||
def test_logdata_with_scores(self):
|
||||
data = DiscGolfLogData(
|
||||
scores={"Alice": {"person_id": 1, "total": 9}},
|
||||
weather="Sunny",
|
||||
fun_factor="High",
|
||||
course_name="Maple Hill",
|
||||
par=9,
|
||||
round_type="Singles",
|
||||
)
|
||||
assert data.scores["Alice"]["total"] == 9
|
||||
assert data.weather == "Sunny"
|
||||
assert data.round_type == "Singles"
|
||||
150
tests/discgolf_tests/test_utils.py
Normal file
150
tests/discgolf_tests/test_utils.py
Normal file
@ -0,0 +1,150 @@
|
||||
from unittest.mock import patch
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from discgolf.utils import _parse_udisc_datetime, import_udisc_csv
|
||||
from people.models import Person
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
|
||||
class TestParserHelpers:
|
||||
def test_parse_udisc_datetime(self):
|
||||
dt = _parse_udisc_datetime("Jun 15, 2026 10:00 AM")
|
||||
assert dt is not None
|
||||
assert dt.year == 2026
|
||||
assert dt.month == 6
|
||||
assert dt.day == 15
|
||||
assert dt.hour == 10
|
||||
assert dt.minute == 0
|
||||
|
||||
def test_parse_udisc_datetime_date_only(self):
|
||||
dt = _parse_udisc_datetime("Jun 15, 2026")
|
||||
assert dt is not None
|
||||
assert dt.year == 2026
|
||||
|
||||
|
||||
class TestImportUdiscCSV:
|
||||
def test_import_singles_creates_course(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.filter(title="Maple Hill").first()
|
||||
assert course is not None
|
||||
assert course.layout_name == "Mountains"
|
||||
assert course.number_of_holes == 3
|
||||
assert course.par_total == 9
|
||||
assert course.par_per_hole == {"hole_1": 3, "hole_2": 3, "hole_3": 3}
|
||||
|
||||
def test_import_singles_creates_scrobble(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 1
|
||||
|
||||
def test_import_singles_logdata(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
log = scrobble.log
|
||||
assert log["course_name"] == "Maple Hill"
|
||||
assert log["par"] == 9
|
||||
assert log["round_type"] == "Singles"
|
||||
assert "Alice" in log["scores"]
|
||||
assert "Bob" in log["scores"]
|
||||
assert log["scores"]["Alice"]["total"] == 9
|
||||
assert log["scores"]["Bob"]["total"] == 12
|
||||
|
||||
def test_import_singles_creates_people(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert Person.objects.filter(name="Alice").exists()
|
||||
assert Person.objects.filter(name="Bob").exists()
|
||||
|
||||
def test_import_teams_creates_scrobble(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 1
|
||||
|
||||
def test_import_teams_logdata(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
assert scrobble.log["round_type"] == "Teams"
|
||||
alice_bob = scrobble.log["scores"]["Alice+Bob"]
|
||||
assert "person_ids" in alice_bob
|
||||
assert len(alice_bob["person_ids"]) == 2
|
||||
|
||||
def test_import_creates_team_people(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
assert Person.objects.filter(name="Alice").exists()
|
||||
assert Person.objects.filter(name="Bob").exists()
|
||||
assert Person.objects.filter(name="Charlie").exists()
|
||||
assert Person.objects.filter(name="Diana").exists()
|
||||
|
||||
def test_import_teams_par_per_hole(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.get(title="Maple Hill")
|
||||
assert course.par_per_hole == {"hole_1": 3, "hole_2": 3, "hole_3": 3}
|
||||
|
||||
def test_import_no_par_returns_empty(self, user, udisc_csv_no_par_file):
|
||||
result = import_udisc_csv(udisc_csv_no_par_file, user.id)
|
||||
assert result == []
|
||||
|
||||
def test_import_empty_csv(self, user, db):
|
||||
import tempfile
|
||||
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write("PlayerName,CourseName,LayoutName,StartDate,Hole1,Total\n")
|
||||
path = f.name
|
||||
|
||||
result = import_udisc_csv(path, user.id)
|
||||
assert result == []
|
||||
|
||||
def test_import_idempotent(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert DiscGolfCourse.objects.filter(title="Maple Hill").count() == 1
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 2
|
||||
|
||||
def test_import_course_defaults_only_on_create(
|
||||
self, user, udisc_singles_csv_file
|
||||
):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.get(title="Maple Hill")
|
||||
assert course.layout_name == "Mountains"
|
||||
|
||||
course.layout_name = "Updated"
|
||||
course.save()
|
||||
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course.refresh_from_db()
|
||||
assert course.layout_name == "Updated"
|
||||
|
||||
@patch("discgolf.utils.ScrobbleNtfyNotification")
|
||||
def test_import_sends_notification(self, mock_notification_class, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
mock_notification_class.assert_called_once()
|
||||
mock_notification_class.return_value.send.assert_called_once()
|
||||
|
||||
def test_import_hole_scores_per_player(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
alice = scrobble.log["scores"]["Alice"]
|
||||
assert alice["hole_1"] == 4
|
||||
assert alice["hole_2"] == 2
|
||||
assert alice["hole_3"] == 3
|
||||
bob = scrobble.log["scores"]["Bob"]
|
||||
assert bob["hole_1"] == 3
|
||||
assert bob["hole_2"] == 4
|
||||
assert bob["hole_3"] == 5
|
||||
|
||||
def test_import_record_error_on_bad_data(self, user, db):
|
||||
import tempfile
|
||||
|
||||
content = """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(content)
|
||||
path = f.name
|
||||
|
||||
errors = []
|
||||
result = import_udisc_csv(path, user.id, record_error=errors.append)
|
||||
assert len(result) == 1
|
||||
course = DiscGolfCourse.objects.first()
|
||||
assert course.title == ""
|
||||
58
tests/discgolf_tests/test_views.py
Normal file
58
tests/discgolf_tests/test_views.py
Normal file
@ -0,0 +1,58 @@
|
||||
from datetime import datetime
|
||||
|
||||
import pytz
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.test import Client
|
||||
from django.urls import reverse
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
User = get_user_model()
|
||||
|
||||
|
||||
class TestDiscGolfCourseViews:
|
||||
def _make_scrobble(self, user, course):
|
||||
dt = datetime(2026, 6, 15, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
return Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt,
|
||||
)
|
||||
|
||||
def test_course_list_anonymous(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(reverse("discgolf:course_list"))
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_course_list_shows_course(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(reverse("discgolf:course_list"))
|
||||
assert response.status_code == 200
|
||||
assert "Maple Hill" in response.content.decode()
|
||||
|
||||
def test_course_detail_anonymous(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(
|
||||
reverse("discgolf:course_detail", kwargs={"slug": course.uuid})
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_course_detail_shows_course(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(
|
||||
title="Maple Hill", layout_name="Mountains"
|
||||
)
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(
|
||||
reverse("discgolf:course_detail", kwargs={"slug": course.uuid})
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert "Maple Hill" in response.content.decode()
|
||||
@ -519,7 +519,7 @@ def test_scrobble_detail_view_with_notes_as_flat_list(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
assert "First note" in response.content.decode()
|
||||
@ -545,7 +545,7 @@ def test_scrobble_detail_view_with_notes_as_dict_timestamps(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
@ -574,7 +574,7 @@ def test_scrobble_detail_view_with_notes_and_labels(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
@ -597,7 +597,7 @@ def test_scrobble_detail_view_post_updates_log(client):
|
||||
"description": "Original description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
|
||||
client.force_login(user)
|
||||
response = client.post(
|
||||
@ -896,7 +896,7 @@ def test_change_visibility_owner_can_change(client):
|
||||
)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 302
|
||||
|
||||
@ -919,7 +919,7 @@ def test_change_visibility_non_owner_gets_404(client):
|
||||
)
|
||||
|
||||
client.force_login(other)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 404
|
||||
|
||||
@ -937,7 +937,7 @@ def test_change_visibility_anonymous_redirects_to_login(client):
|
||||
task=task, media_type="Task", user=user, visibility="private",
|
||||
timestamp=timezone.now(),
|
||||
)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 302
|
||||
assert "/login/" in response.url
|
||||
@ -956,7 +956,7 @@ def test_regenerate_share_token_invalidates_old_sqid(client):
|
||||
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:regenerate-share-token", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:regenerate-share-token", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url)
|
||||
assert response.status_code == 302
|
||||
|
||||
@ -985,7 +985,7 @@ def test_share_analytics_owner_can_view(client):
|
||||
)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
|
||||
@ -1005,7 +1005,7 @@ def test_share_analytics_non_owner_gets_404(client):
|
||||
)
|
||||
|
||||
client.force_login(other)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 404
|
||||
|
||||
@ -1027,7 +1027,7 @@ def test_share_analytics_shows_view_logs(client):
|
||||
client.get(share_url)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
|
||||
@ -4,9 +4,15 @@ from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from vrobbler import context_processors
|
||||
from vrobbler.context_processors import version_info
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_git_cache():
|
||||
context_processors._GIT_COMMIT = None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_request():
|
||||
return MagicMock()
|
||||
|
||||
@ -27,6 +27,7 @@ class BeerAdmin(admin.ModelAdmin):
|
||||
"uuid",
|
||||
"title",
|
||||
)
|
||||
raw_id_fields = ("styles", "producer")
|
||||
ordering = ("-created",)
|
||||
search_fields = ("title",)
|
||||
inlines = [
|
||||
|
||||
@ -27,4 +27,5 @@ class BirdingLocationAdmin(admin.ModelAdmin):
|
||||
class BirdingCSVImportAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
|
||||
raw_id_fields = ("user",)
|
||||
ordering = ("-created",)
|
||||
|
||||
@ -38,6 +38,7 @@ class BoardGameLocationAdmin(admin.ModelAdmin):
|
||||
"uuid",
|
||||
"geo_location",
|
||||
)
|
||||
raw_id_fields = ("geo_location",)
|
||||
ordering = ("-created",)
|
||||
|
||||
|
||||
@ -49,6 +50,7 @@ class BoardGameAdmin(admin.ModelAdmin):
|
||||
"title",
|
||||
"published_year",
|
||||
)
|
||||
raw_id_fields = ("publisher", "publishers", "designers", "expansion_for_boardgame")
|
||||
search_fields = ("title",)
|
||||
ordering = ("-created",)
|
||||
inlines = [
|
||||
|
||||
@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
|
||||
"source": "Lichess",
|
||||
"timezone": user.profile.timezone,
|
||||
"log": log_data,
|
||||
"visibility": "private",
|
||||
}
|
||||
if commit:
|
||||
Scrobble.objects.create(**scrobble_dict)
|
||||
|
||||
@ -1,8 +1,19 @@
|
||||
from books.models import Author, Book, Paper
|
||||
from books.models import Author, Book, Journal, Paper
|
||||
from django.contrib import admin
|
||||
from scrobbles.admin import ScrobbleInline
|
||||
|
||||
|
||||
@admin.register(Journal)
|
||||
class JournalAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = (
|
||||
"title",
|
||||
"website_url",
|
||||
)
|
||||
search_fields = ("title",)
|
||||
ordering = ("-created",)
|
||||
|
||||
|
||||
@admin.register(Author)
|
||||
class AuthorAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
@ -27,6 +38,7 @@ class BookAdmin(admin.ModelAdmin):
|
||||
"first_publish_year",
|
||||
"pages",
|
||||
)
|
||||
raw_id_fields = ("authors",)
|
||||
search_fields = ("name",)
|
||||
ordering = ("-created",)
|
||||
inlines = [
|
||||
@ -34,11 +46,11 @@ class BookAdmin(admin.ModelAdmin):
|
||||
]
|
||||
|
||||
def issue_or_volume(self, obj):
|
||||
return obj.issue_number or obj.volume_number
|
||||
return obj.subtitle
|
||||
|
||||
|
||||
@admin.register(Paper)
|
||||
class BookAdmin(admin.ModelAdmin):
|
||||
class PaperAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = (
|
||||
"title",
|
||||
@ -47,6 +59,7 @@ class BookAdmin(admin.ModelAdmin):
|
||||
"first_publish_year",
|
||||
"pages",
|
||||
)
|
||||
raw_id_fields = ("authors",)
|
||||
search_fields = ("name",)
|
||||
ordering = ("-created",)
|
||||
inlines = [
|
||||
|
||||
@ -1,128 +0,0 @@
|
||||
from enum import Enum
|
||||
from typing import Optional
|
||||
from bs4 import BeautifulSoup
|
||||
import requests
|
||||
import logging
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
USER_AGENT = "Mozilla/5.0 (Android 4.4; Mobile; rv:41.0) Gecko/41.0 Firefox/41.0"
|
||||
AMAZON_SEARCH_URL = "https://www.amazon.com/s?k={amazon_id}"
|
||||
|
||||
|
||||
class AmazonAttribute(Enum):
|
||||
SERIES = 0
|
||||
PAGES = 1
|
||||
LANGUAGE = 2
|
||||
PUBLISHER = 3
|
||||
PUB_DATE = 4
|
||||
DIMENSIONS = 5
|
||||
ISBN_10 = 6
|
||||
ISBN_13 = 7
|
||||
|
||||
|
||||
def strip_and_clean(text):
|
||||
return text.strip("\n").rstrip().lstrip()
|
||||
|
||||
|
||||
def get_rating_from_soup(soup) -> Optional[int]:
|
||||
rating = None
|
||||
try:
|
||||
potential_rating = soup.find("div", class_="allmusic-rating")
|
||||
if potential_rating:
|
||||
rating = int(strip_and_clean(potential_rating.get_text()))
|
||||
except ValueError:
|
||||
pass
|
||||
return rating
|
||||
|
||||
|
||||
def get_review_from_soup(soup) -> str:
|
||||
review = ""
|
||||
try:
|
||||
potential_text = soup.find("div", class_="text")
|
||||
if potential_text:
|
||||
review = strip_and_clean(potential_text.get_text())
|
||||
except ValueError:
|
||||
pass
|
||||
return review
|
||||
|
||||
|
||||
def scrape_data_from_amazon(url) -> dict:
|
||||
data_dict = {}
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
r = requests.get(url, headers=headers)
|
||||
if r.status_code == 200:
|
||||
soup = BeautifulSoup(r.text, "html.parser")
|
||||
# TODO Fix this scraper
|
||||
data_dict["rating"] = get_rating_from_soup(soup)
|
||||
data_dict["review"] = get_review_from_soup(soup)
|
||||
return data_dict
|
||||
|
||||
|
||||
def get_amazon_product_dict(amazon_id: str) -> dict:
|
||||
data_dict = {}
|
||||
url = ""
|
||||
|
||||
search_url = AMAZON_SEARCH_URL.format(amazon_id=amazon_id)
|
||||
headers = {
|
||||
"user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36",
|
||||
"accept-language": "en-GB,en;q=0.9",
|
||||
}
|
||||
|
||||
response = requests.get(search_url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.info(f"Bad http response from Amazon {response}")
|
||||
return data_dict
|
||||
|
||||
soup = BeautifulSoup(response.text, "html.parser")
|
||||
results = soup.find("a", class_="a-link-normal")
|
||||
|
||||
if not results:
|
||||
logger.info(f"No search results for {amazon_id}")
|
||||
return data_dict
|
||||
|
||||
product_url = "https://www.amazon.com" + str(results.get("href", ""))
|
||||
|
||||
data_dict = {}
|
||||
response = requests.get(product_url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.info(f"Bad http response from Amazon {response}")
|
||||
return data_dict
|
||||
|
||||
soup = BeautifulSoup(response.text, "html.parser")
|
||||
try:
|
||||
data_dict["title"] = soup.findAll("span", class_="celwidget")[1].text.strip()
|
||||
data_dict["cover_url"] = soup.find("img", class_="frontImage").get("src")
|
||||
data_dict["summary"] = soup.findAll("div", class_="a-expander-content")[1].text
|
||||
meta = soup.findAll("div", class_="rpi-attribute-value")
|
||||
data_dict["isbn"] = meta[AmazonAttribute.ISBN_10.value].text.strip()
|
||||
pages = meta[AmazonAttribute.PAGES.value].text
|
||||
if "pages" in pages:
|
||||
data_dict["pages"] = (
|
||||
meta[AmazonAttribute.PAGES.value].text.split("pages")[0].strip()
|
||||
)
|
||||
except IndexError as e:
|
||||
logger.error(f"Amazon lookup is failing for this product {amazon_id}: {e}")
|
||||
except AttributeError as e:
|
||||
logger.error(f"Amazon lookup is failing for this product {amazon_id}: {e}")
|
||||
|
||||
return data_dict
|
||||
|
||||
|
||||
def lookup_book_from_amazon(amazon_id: str) -> dict:
|
||||
top = {}
|
||||
|
||||
return {
|
||||
"title": top.get("title"),
|
||||
"isbn": isbn,
|
||||
"openlibrary_id": ol_id,
|
||||
"goodreads_id": get_first("id_goodreads", top),
|
||||
"first_publish_year": top.get("first_publish_year"),
|
||||
"first_sentence": first_sentence,
|
||||
"pages": top.get("number_of_pages_median", None),
|
||||
"cover_url": COVER_URL.format(id=ol_id),
|
||||
"ol_author_id": ol_author_id,
|
||||
"subject_key_list": top.get("subject_key", []),
|
||||
}
|
||||
@ -17,6 +17,7 @@ class MediaSourceTag(str, Enum):
|
||||
LOCG = "source_locg"
|
||||
KOREADER = "source_koreader"
|
||||
SEMANTIC_SCHOLAR = "source_semantic_scholar"
|
||||
AMAZON = "source_amazon"
|
||||
|
||||
@classmethod
|
||||
def choices(cls):
|
||||
|
||||
@ -1,9 +1,8 @@
|
||||
import logging
|
||||
import re
|
||||
import sqlite3
|
||||
from datetime import datetime, timedelta
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from enum import Enum
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import requests
|
||||
from books.constants import BOOKS_TITLES_TO_IGNORE
|
||||
@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
|
||||
return book_id_map
|
||||
|
||||
|
||||
def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
|
||||
def build_page_data(page_rows: list, book_map: dict) -> dict:
|
||||
"""Given rows of page data from KoReader, parse each row and build
|
||||
scrobbles for our user, loading the page data into the page_data
|
||||
field on the scrobble instance.
|
||||
@ -187,18 +186,20 @@ def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
|
||||
book_ids_not_found.append(koreader_book_id)
|
||||
continue
|
||||
|
||||
if "pages" not in book_map[koreader_book_id].keys():
|
||||
book_map[koreader_book_id]["pages"] = {}
|
||||
book_map[koreader_book_id].setdefault("pages", [])
|
||||
|
||||
page_number = page_row[KoReaderPageStatColumn.PAGE.value]
|
||||
duration = page_row[KoReaderPageStatColumn.DURATION.value]
|
||||
start_ts = page_row[KoReaderPageStatColumn.START_TIME.value]
|
||||
|
||||
book_map[koreader_book_id]["pages"][page_number] = {
|
||||
"duration": duration,
|
||||
"start_ts": start_ts,
|
||||
"end_ts": start_ts + duration,
|
||||
}
|
||||
book_map[koreader_book_id]["pages"].append(
|
||||
{
|
||||
"page_number": page_number,
|
||||
"duration": duration,
|
||||
"start_ts": start_ts,
|
||||
"end_ts": start_ts + duration,
|
||||
}
|
||||
)
|
||||
if book_ids_not_found:
|
||||
logger.info(f"Found pages for books not in file: {set(book_ids_not_found)}")
|
||||
return book_map
|
||||
@ -216,7 +217,6 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
pages_not_found.append(book_id)
|
||||
continue
|
||||
|
||||
should_create_scrobble = False
|
||||
scrobble_page_data = {}
|
||||
playback_position_seconds = 0
|
||||
prev_page_stats = {}
|
||||
@ -225,11 +225,12 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
pages_processed = 0
|
||||
total_pages_read = len(book_map[koreader_book_id]["pages"])
|
||||
ordered_pages = sorted(
|
||||
book_map[koreader_book_id]["pages"].items(),
|
||||
key=lambda x: x[1]["start_ts"],
|
||||
book_map[koreader_book_id]["pages"],
|
||||
key=lambda x: x["start_ts"],
|
||||
)
|
||||
|
||||
for cur_page_number, stats in ordered_pages:
|
||||
for stats in ordered_pages:
|
||||
page_number = stats["page_number"]
|
||||
pages_processed += 1
|
||||
|
||||
seconds_from_last_page = 0
|
||||
@ -243,46 +244,52 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
)
|
||||
|
||||
end_of_reading = pages_processed == total_pages_read
|
||||
big_jump_to_this_page = (cur_page_number - last_page_number) > 10
|
||||
big_jump_to_this_page = (page_number - last_page_number) > 10
|
||||
is_session_gap = seconds_from_last_page > SESSION_GAP_SECONDS
|
||||
if (is_session_gap and not big_jump_to_this_page) or end_of_reading:
|
||||
should_create_scrobble = True
|
||||
should_create_scrobble = (
|
||||
is_session_gap and not big_jump_to_this_page
|
||||
) or end_of_reading
|
||||
|
||||
# Always accumulate the current page first
|
||||
scrobble_page_data[page_number] = stats
|
||||
|
||||
if should_create_scrobble:
|
||||
# For end-of-reading, the current page is already accumulated
|
||||
# and belongs in this scrobble. For a session gap, remove the
|
||||
# current page from this scrobble — it starts a new session.
|
||||
if is_session_gap and not end_of_reading:
|
||||
del scrobble_page_data[page_number]
|
||||
|
||||
scrobble_page_data = dict(
|
||||
sorted(
|
||||
scrobble_page_data.items(),
|
||||
key=lambda x: x[1]["start_ts"],
|
||||
)
|
||||
)
|
||||
try:
|
||||
first_page = scrobble_page_data.get(
|
||||
list(scrobble_page_data.keys())[0]
|
||||
)
|
||||
last_page = scrobble_page_data.get(
|
||||
list(scrobble_page_data.keys())[-1]
|
||||
)
|
||||
except IndexError:
|
||||
|
||||
if not scrobble_page_data:
|
||||
logger.error(
|
||||
"Could not process book, no page data found",
|
||||
extra={"scrobble_page_data": scrobble_page_data},
|
||||
)
|
||||
continue
|
||||
|
||||
first_page = next(iter(scrobble_page_data.values()))
|
||||
last_page = scrobble_page_data[
|
||||
list(scrobble_page_data.keys())[-1]
|
||||
]
|
||||
|
||||
timestamp = user.profile.get_timestamp_with_tz(
|
||||
datetime.fromtimestamp(int(first_page.get("start_ts")))
|
||||
datetime.fromtimestamp(
|
||||
int(first_page.get("start_ts"))
|
||||
)
|
||||
)
|
||||
stop_timestamp = user.profile.get_timestamp_with_tz(
|
||||
datetime.fromtimestamp(int(last_page.get("end_ts")))
|
||||
datetime.fromtimestamp(
|
||||
int(last_page.get("end_ts"))
|
||||
)
|
||||
)
|
||||
|
||||
# Adjust for Daylight Saving Time
|
||||
# if timestamp.dst() == timedelta(
|
||||
# 0
|
||||
# ) or stop_timestamp.dst() == timedelta(0):
|
||||
# timestamp = timestamp - timedelta(hours=1)
|
||||
# stop_timestamp = stop_timestamp - timedelta(hours=1)
|
||||
|
||||
scrobble = Scrobble.objects.filter(
|
||||
timestamp=timestamp,
|
||||
book_id=book_id,
|
||||
@ -291,7 +298,7 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
|
||||
if not scrobble:
|
||||
logger.info(
|
||||
f"Queueing scrobble for {book_id}, page {cur_page_number}"
|
||||
f"Queueing scrobble for {book_id}, page {page_number}"
|
||||
)
|
||||
log_data = {
|
||||
"koreader_hash": book_dict.get("hash"),
|
||||
@ -318,16 +325,18 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
timezone=tz,
|
||||
)
|
||||
)
|
||||
# Then start over
|
||||
should_create_scrobble = False
|
||||
# Then start over for the next session
|
||||
playback_position_seconds = 0
|
||||
scrobble_page_data = {}
|
||||
|
||||
# We accumulate pages for the scrobble until we should create a new one
|
||||
scrobble_page_data[cur_page_number] = stats
|
||||
# For session gaps, re-add the current page as the
|
||||
# beginning of the next session's accumulation
|
||||
if is_session_gap and not end_of_reading:
|
||||
scrobble_page_data[page_number] = stats
|
||||
|
||||
last_page_number = cur_page_number
|
||||
last_page_number = page_number
|
||||
prev_page_stats = stats
|
||||
|
||||
if pages_not_found:
|
||||
logger.info(f"Pages not found for books: {set(pages_not_found)}")
|
||||
return scrobbles_to_create
|
||||
@ -335,13 +344,16 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
|
||||
|
||||
def fix_long_play_stats_for_scrobbles(scrobbles: list) -> None:
|
||||
"""Given a list of scrobbles, update pages read, long play seconds and check
|
||||
for media completion"""
|
||||
for media completion.
|
||||
|
||||
Uses the long_play_last_scrobble FK chain to accumulate time.
|
||||
Consider using the recompute_long_play_seconds management command instead.
|
||||
"""
|
||||
|
||||
for scrobble in scrobbles:
|
||||
# But if there's a next scrobble, set pages read to their starting page
|
||||
if scrobble.previous and not scrobble.previous.long_play_complete:
|
||||
if scrobble.long_play_last_scrobble and not scrobble.long_play_last_scrobble.long_play_complete:
|
||||
scrobble.long_play_seconds = scrobble.playback_position_seconds + (
|
||||
scrobble.previous.long_play_seconds or 0
|
||||
scrobble.long_play_last_scrobble.long_play_seconds or 0
|
||||
)
|
||||
else:
|
||||
scrobble.long_play_seconds = scrobble.playback_position_seconds
|
||||
@ -357,9 +369,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
|
||||
|
||||
new_scrobbles = []
|
||||
user = User.objects.filter(id=user_id).first()
|
||||
tz = ZoneInfo("UTC")
|
||||
if user:
|
||||
tz = user.profile.tzinfo
|
||||
|
||||
is_os_file = "https://" not in file_path
|
||||
if is_os_file:
|
||||
@ -375,7 +384,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
|
||||
book_map = build_page_data(
|
||||
cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
|
||||
book_map,
|
||||
tz,
|
||||
)
|
||||
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
|
||||
else:
|
||||
@ -392,7 +400,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
|
||||
_sqlite_bytes(file_path), max_buffer_size=1_048_576
|
||||
):
|
||||
if table_name == "page_stat_data":
|
||||
book_map = build_page_data(rows, book_map, tz)
|
||||
book_map = build_page_data(rows, book_map)
|
||||
new_scrobbles = build_scrobbles_from_book_map(book_map, user)
|
||||
|
||||
logger.info(f"Creating {len(new_scrobbles)} new scrobbles")
|
||||
|
||||
301
vrobbler/apps/books/management/commands/cleanup_book_metadata.py
Normal file
301
vrobbler/apps/books/management/commands/cleanup_book_metadata.py
Normal file
@ -0,0 +1,301 @@
|
||||
import logging
|
||||
import time
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.db import transaction
|
||||
|
||||
from books.constants import READCOMICSONLINE_URL
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MISSING_ALL = [
|
||||
"cover",
|
||||
"summary",
|
||||
"isbn",
|
||||
"pages",
|
||||
"language",
|
||||
"publisher",
|
||||
"publish_year",
|
||||
]
|
||||
|
||||
MISSING_GROUPS = {
|
||||
"cover": lambda b: not bool(b.cover),
|
||||
"summary": lambda b: not b.summary,
|
||||
"isbn": lambda b: not b.isbn_13 and not b.isbn_10,
|
||||
"pages": lambda b: b.pages is None,
|
||||
"language": lambda b: not b.language,
|
||||
"publisher": lambda b: not b.publisher,
|
||||
"publish_year": lambda b: b.first_publish_year is None,
|
||||
}
|
||||
|
||||
|
||||
def _book_matches(book, flags):
|
||||
if not flags:
|
||||
return False
|
||||
for flag in flags:
|
||||
fn = MISSING_GROUPS.get(flag)
|
||||
if fn and fn(book):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Backfill missing metadata on books from Google Books, OpenLibrary, and ComicVine"
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Commit changes to the database",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Number of books to process per batch (default: 100)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sleep",
|
||||
type=float,
|
||||
default=0.5,
|
||||
help="Seconds to sleep between API calls (default: 0.5)",
|
||||
)
|
||||
for flag in MISSING_ALL:
|
||||
parser.add_argument(
|
||||
f"--missing-{flag}",
|
||||
dest="missing_flags",
|
||||
action="append_const",
|
||||
const=flag,
|
||||
help=f"Process books missing {flag}",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--comics-only",
|
||||
action="store_true",
|
||||
help="Only process books with a readcomicsonline.ru URL",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--all",
|
||||
action="store_true",
|
||||
dest="all_missing",
|
||||
help="Process books missing any metadata field",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
from books.models import Book
|
||||
|
||||
commit = options["commit"]
|
||||
batch_size = options["batch_size"]
|
||||
sleep_secs = options["sleep"]
|
||||
flags = options.get("missing_flags") or []
|
||||
comics_only = options["comics_only"]
|
||||
all_missing = options["all_missing"]
|
||||
|
||||
if all_missing:
|
||||
flags = MISSING_ALL
|
||||
|
||||
if not flags and not comics_only:
|
||||
self.stdout.write(
|
||||
"No filters specified. Use --all, --missing-*, or --comics-only."
|
||||
)
|
||||
return
|
||||
|
||||
qs = Book.objects.all()
|
||||
if comics_only:
|
||||
qs = qs.filter(readcomics_url__isnull=False)
|
||||
|
||||
if flags:
|
||||
qs = [b for b in qs.iterator() if _book_matches(b, flags)]
|
||||
else:
|
||||
qs = list(qs)
|
||||
|
||||
total = len(qs)
|
||||
self.stdout.write(f"Found {total} books to process")
|
||||
|
||||
if not commit:
|
||||
self.stdout.write(
|
||||
"Dry run — no API calls will be made. Use --commit to run lookups."
|
||||
)
|
||||
return
|
||||
|
||||
enriched = 0
|
||||
skipped = 0
|
||||
stats = {
|
||||
"cover_fixed": 0,
|
||||
"summary_fixed": 0,
|
||||
"isbn_fixed": 0,
|
||||
"pages_fixed": 0,
|
||||
"language_fixed": 0,
|
||||
"publisher_fixed": 0,
|
||||
"publish_year_fixed": 0,
|
||||
}
|
||||
|
||||
for batch_num, offset in enumerate(range(0, len(qs), batch_size)):
|
||||
batch = qs[offset : offset + batch_size]
|
||||
for book in batch:
|
||||
result = self._enrich_book(book, sleep_secs)
|
||||
if result:
|
||||
enriched += 1
|
||||
for key in stats:
|
||||
if result.get(key):
|
||||
stats[key] += 1
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
self.stdout.write(
|
||||
f" Batch {batch_num + 1}: {offset + len(batch)}/{total} — "
|
||||
f"enriched: {enriched}, skipped: {skipped}"
|
||||
)
|
||||
|
||||
self.stdout.write(
|
||||
f"\nResults (commit={commit}):\n"
|
||||
f" Books enriched: {enriched}\n"
|
||||
f" Books skipped: {skipped}\n"
|
||||
f" Covers fixed: {stats['cover_fixed']}\n"
|
||||
f" Summaries fixed:{stats['summary_fixed']}\n"
|
||||
f" ISBNs fixed: {stats['isbn_fixed']}\n"
|
||||
f" Pages fixed: {stats['pages_fixed']}\n"
|
||||
f" Languages fixed:{stats['language_fixed']}\n"
|
||||
f" Publishers fixed:{stats['publisher_fixed']}\n"
|
||||
f" Publish yrs fixed: {stats['publish_year_fixed']}"
|
||||
)
|
||||
|
||||
def _enrich_book(self, book, sleep_secs):
|
||||
from books.sources.comicvine import (
|
||||
lookup_comic_from_comicvine,
|
||||
lookup_issue_by_comicvine_id,
|
||||
)
|
||||
from books.sources.google import lookup_book_from_google
|
||||
from books.sources.openlibrary import lookup_book_from_openlibrary as lookup_book_from_ol
|
||||
|
||||
title = book.original_title or book.title
|
||||
author_name = book.author.name if book.author else None
|
||||
book_dict = {}
|
||||
|
||||
cv_data = None
|
||||
if book.comicvine_id:
|
||||
cv_data = lookup_issue_by_comicvine_id(str(book.comicvine_id))
|
||||
if not cv_data:
|
||||
cv_data = lookup_comic_from_comicvine(title)
|
||||
if cv_data:
|
||||
book_dict.update(cv_data)
|
||||
|
||||
ol_data = lookup_book_from_ol(title, author=author_name)
|
||||
time.sleep(sleep_secs)
|
||||
google_data = lookup_book_from_google(title)
|
||||
|
||||
if ol_data:
|
||||
for k, v in ol_data.items():
|
||||
book_dict.setdefault(k, v)
|
||||
|
||||
if google_data:
|
||||
for k, v in google_data.items():
|
||||
if v:
|
||||
book_dict.setdefault(k, v)
|
||||
|
||||
if not book_dict:
|
||||
return None
|
||||
|
||||
changed = self._apply(book, book_dict, title)
|
||||
return changed
|
||||
|
||||
def _apply(self, book, data, title):
|
||||
changed = {
|
||||
"cover_fixed": False,
|
||||
"summary_fixed": False,
|
||||
"isbn_fixed": False,
|
||||
"pages_fixed": False,
|
||||
"language_fixed": False,
|
||||
"publisher_fixed": False,
|
||||
"publish_year_fixed": False,
|
||||
}
|
||||
|
||||
update_fields = []
|
||||
|
||||
cover_url = data.pop("cover_url", "")
|
||||
|
||||
if data.get("summary") and not book.summary:
|
||||
book.summary = data["summary"]
|
||||
update_fields.append("summary")
|
||||
changed["summary_fixed"] = True
|
||||
|
||||
if data.get("isbn_13") and not book.isbn_13:
|
||||
book.isbn_13 = data["isbn_13"]
|
||||
update_fields.append("isbn_13")
|
||||
changed["isbn_fixed"] = True
|
||||
|
||||
if data.get("isbn_10") and not book.isbn_10:
|
||||
book.isbn_10 = data["isbn_10"]
|
||||
update_fields.append("isbn_10")
|
||||
changed["isbn_fixed"] = True
|
||||
|
||||
if data.get("pages") and book.pages is None:
|
||||
book.pages = data["pages"]
|
||||
update_fields.append("pages")
|
||||
changed["pages_fixed"] = True
|
||||
|
||||
if data.get("language") and not book.language:
|
||||
book.language = data["language"]
|
||||
update_fields.append("language")
|
||||
changed["language_fixed"] = True
|
||||
|
||||
if data.get("publisher") and not book.publisher:
|
||||
book.publisher = data["publisher"]
|
||||
update_fields.append("publisher")
|
||||
changed["publisher_fixed"] = True
|
||||
|
||||
if data.get("first_publish_year") and book.first_publish_year is None:
|
||||
book.first_publish_year = data["first_publish_year"]
|
||||
update_fields.append("first_publish_year")
|
||||
changed["publish_year_fixed"] = True
|
||||
|
||||
if data.get("openlibrary_id") and not book.openlibrary_id:
|
||||
book.openlibrary_id = data["openlibrary_id"]
|
||||
update_fields.append("openlibrary_id")
|
||||
|
||||
if data.get("comicvine_id") and not book.comicvine_id:
|
||||
book.comicvine_id = data["comicvine_id"]
|
||||
update_fields.append("comicvine_id")
|
||||
|
||||
if data.get("issue_number") and book.issue_number is None:
|
||||
book.issue_number = data["issue_number"]
|
||||
update_fields.append("issue_number")
|
||||
|
||||
if data.get("volume_number") and book.volume_number is None:
|
||||
book.volume_number = data["volume_number"]
|
||||
update_fields.append("volume_number")
|
||||
|
||||
if data.get("volume") and not book.volume:
|
||||
book.volume = data["volume"]
|
||||
update_fields.append("volume")
|
||||
|
||||
if data.get("volume_comicvine_id") and not book.volume_comicvine_id:
|
||||
book.volume_comicvine_id = data["volume_comicvine_id"]
|
||||
update_fields.append("volume_comicvine_id")
|
||||
|
||||
if update_fields:
|
||||
book.save(update_fields=update_fields)
|
||||
self.stdout.write(f" [ENRICHED] {book} — {', '.join(update_fields)}")
|
||||
|
||||
if cover_url and not book.cover:
|
||||
book.save_image_from_url(cover_url)
|
||||
if book.cover:
|
||||
changed["cover_fixed"] = True
|
||||
self.stdout.write(f" [COVER] {book} — cover saved from source")
|
||||
|
||||
genres = data.pop("genres", data.pop("generes", []))
|
||||
if genres:
|
||||
existing = set(book.genre.names())
|
||||
new_genres = [g for g in genres if g not in existing]
|
||||
if new_genres:
|
||||
book.genre.add(*new_genres)
|
||||
self.stdout.write(f" [GENRES] {book} — added {len(new_genres)} genres")
|
||||
|
||||
tags = data.pop("tags", [])
|
||||
if tags:
|
||||
existing_tags = set(book.tags.names())
|
||||
new_tags = [t for t in tags if t not in existing_tags]
|
||||
if new_tags:
|
||||
book.tags.add(*new_tags)
|
||||
self.stdout.write(f" [TAGS] {book} — added {', '.join(new_tags)}")
|
||||
|
||||
return changed if any(changed.values()) else None
|
||||
@ -0,0 +1,130 @@
|
||||
import logging
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.db import transaction
|
||||
|
||||
from books.koreader import SESSION_GAP_SECONDS, fix_long_play_stats_for_scrobbles
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
SESSION_GAP = timedelta(seconds=SESSION_GAP_SECONDS)
|
||||
|
||||
|
||||
def _page_data_keys(pages):
|
||||
return sorted(int(k) for k in (pages or {}))
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Merge orphaned 1-page KOReader scrobbles into the preceding scrobble"
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Commit changes to the database",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
commit = options["commit"]
|
||||
|
||||
qs = Scrobble.objects.filter(
|
||||
media_type="Book", source="KOReader"
|
||||
).order_by("book_id", "timestamp")
|
||||
|
||||
if not qs.exists():
|
||||
self.stdout.write("No KOReader book scrobbles found.")
|
||||
return
|
||||
|
||||
merged = 0
|
||||
affected_books = set()
|
||||
|
||||
# Group by book_id manually since we're iterating in order
|
||||
book_scrobbles = {}
|
||||
for s in qs:
|
||||
book_scrobbles.setdefault(s.book_id, []).append(s)
|
||||
|
||||
if not commit:
|
||||
self.stdout.write("Dry run — no changes will be saved. Use --commit to apply.")
|
||||
|
||||
for book_id, scrobbles in book_scrobbles.items():
|
||||
batch_merged = 0
|
||||
i = 0
|
||||
while i < len(scrobbles) - 1:
|
||||
current = scrobbles[i]
|
||||
orphan = scrobbles[i + 1]
|
||||
|
||||
orphan_pages = orphan.logdata.page_data if orphan.logdata else {}
|
||||
orphan_keys = _page_data_keys(orphan_pages)
|
||||
if len(orphan_keys) != 1:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
current_pages = current.logdata.page_data if current.logdata else {}
|
||||
current_keys = _page_data_keys(current_pages)
|
||||
if not current_keys:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
orphan_page_num = orphan_keys[0]
|
||||
current_last_page = current_keys[-1]
|
||||
|
||||
if orphan_page_num != current_last_page + 1:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# Check that the orphan is close enough in time
|
||||
gap = orphan.timestamp - current.stop_timestamp
|
||||
if gap > SESSION_GAP:
|
||||
i += 1
|
||||
continue
|
||||
|
||||
# Merge orphan into current
|
||||
current_pages[str(orphan_page_num)] = orphan_pages[str(orphan_page_num)]
|
||||
current.log["page_data"] = current_pages
|
||||
current.log["pages_read"] = len(current_pages)
|
||||
current.stop_timestamp = orphan.stop_timestamp
|
||||
current.playback_position_seconds += orphan.playback_position_seconds
|
||||
|
||||
affected_books.add(book_id)
|
||||
|
||||
if commit:
|
||||
with transaction.atomic():
|
||||
current.save(
|
||||
update_fields=[
|
||||
"log",
|
||||
"stop_timestamp",
|
||||
"playback_position_seconds",
|
||||
]
|
||||
)
|
||||
orphan.delete()
|
||||
|
||||
merged += 1
|
||||
batch_merged += 1
|
||||
scrobbles.pop(i + 1)
|
||||
|
||||
if batch_merged:
|
||||
self.stdout.write(
|
||||
f" Book {book_id}: merged {batch_merged} orphan scrobble(s)"
|
||||
)
|
||||
|
||||
self.stdout.write(f"\nTotal orphans merged: {merged}")
|
||||
|
||||
if commit and affected_books:
|
||||
self.stdout.write("Recalculating long_play_stats for affected books...")
|
||||
for book_id in affected_books:
|
||||
scrobbles_to_fix = (
|
||||
Scrobble.objects.filter(book_id=book_id, source="KOReader")
|
||||
.order_by("timestamp")
|
||||
)
|
||||
fix_long_play_stats_for_scrobbles(list(scrobbles_to_fix))
|
||||
|
||||
self.stdout.write(f"Fixed stats for {len(affected_books)} books.")
|
||||
|
||||
if not commit:
|
||||
self.stdout.write(
|
||||
f"\nWould merge {merged} orphan scrobble(s) across "
|
||||
f"{len(affected_books)} book(s)."
|
||||
)
|
||||
@ -0,0 +1,23 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-15 16:09
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("books", "0036_alter_book_genre_alter_paper_genre"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="book",
|
||||
name="volume",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="book",
|
||||
name="volume_comicvine_id",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
]
|
||||
18
vrobbler/apps/books/migrations/0038_paper_pdf_file.py
Normal file
18
vrobbler/apps/books/migrations/0038_paper_pdf_file.py
Normal file
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-23 14:45
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("books", "0037_book_volume_book_volume_comicvine_id"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="paper",
|
||||
name="pdf_file",
|
||||
field=models.FileField(blank=True, null=True, upload_to="papers/pdf/"),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,92 @@
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
import django_extensions.db.fields
|
||||
import uuid
|
||||
|
||||
|
||||
def migrate_journal_data(apps, schema_editor):
|
||||
Paper = apps.get_model("books", "Paper")
|
||||
Journal = apps.get_model("books", "Journal")
|
||||
for paper in Paper.objects.all():
|
||||
old_journal = getattr(paper, "journal", None)
|
||||
if old_journal:
|
||||
journal, _ = Journal.objects.get_or_create(title=str(old_journal))
|
||||
paper._journal_tmp = journal
|
||||
paper.save(update_fields=["_journal_tmp"])
|
||||
|
||||
|
||||
def reverse_migrate_journal_data(apps, schema_editor):
|
||||
Paper = apps.get_model("books", "Paper")
|
||||
for paper in Paper.objects.all():
|
||||
if paper._journal_tmp:
|
||||
paper.journal = paper._journal_tmp.title
|
||||
paper.save(update_fields=["journal"])
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("books", "0038_paper_pdf_file"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="Journal",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
(
|
||||
"uuid",
|
||||
models.UUIDField(
|
||||
blank=True, default=uuid.uuid4, editable=False, null=True
|
||||
),
|
||||
),
|
||||
("title", models.CharField(max_length=255)),
|
||||
("description", models.TextField(blank=True, null=True)),
|
||||
("website_url", models.URLField(blank=True, max_length=500, null=True)),
|
||||
],
|
||||
options={
|
||||
"get_latest_by": "modified",
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="paper",
|
||||
name="_journal_tmp",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="books.journal",
|
||||
),
|
||||
),
|
||||
migrations.RunPython(migrate_journal_data, reverse_migrate_journal_data),
|
||||
migrations.RemoveField(
|
||||
model_name="paper",
|
||||
name="journal",
|
||||
),
|
||||
migrations.RenameField(
|
||||
model_name="paper",
|
||||
old_name="_journal_tmp",
|
||||
new_name="journal",
|
||||
),
|
||||
]
|
||||
@ -4,6 +4,7 @@ from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
from uuid import uuid4
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import requests
|
||||
from books.constants import MediaSourceTag, READCOMICSONLINE_URL
|
||||
@ -19,12 +20,18 @@ from books.openlibrary import (
|
||||
from books.sources.comicvine import (
|
||||
ComicVineClient,
|
||||
lookup_comic_from_comicvine,
|
||||
lookup_issue_by_comicvine_id,
|
||||
)
|
||||
from books.sources.google import lookup_book_from_google
|
||||
from books.sources.amazon import lookup_book_from_amazon
|
||||
from books.sources.openlibrary import (
|
||||
lookup_book_from_openlibrary as lookup_book_from_ol,
|
||||
)
|
||||
from books.sources.semantic import lookup_paper_from_semantic
|
||||
from books.sources.semantic import (
|
||||
lookup_paper_from_semantic,
|
||||
lookup_paper_from_semantic_by_doi,
|
||||
)
|
||||
from books.sources.scihub import SciHubService
|
||||
from books.utils import get_comic_issue_url
|
||||
from django.conf import settings
|
||||
from django.contrib.auth import get_user_model
|
||||
@ -71,7 +78,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
|
||||
_excluded_fields = {"koreader_hash", "page_data"}
|
||||
|
||||
def avg_seconds_per_page(self):
|
||||
if self.page_data:
|
||||
if self.page_data and isinstance(self.page_data, dict):
|
||||
total_duration = 0
|
||||
for page_num, stats in self.page_data.items():
|
||||
total_duration += stats.get("duration", 0)
|
||||
@ -79,6 +86,16 @@ class BookLogData(BaseLogData, LongPlayLogData):
|
||||
return int(total_duration / len(self.page_data))
|
||||
|
||||
|
||||
class Journal(TimeStampedModel):
|
||||
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
|
||||
title = models.CharField(max_length=255)
|
||||
description = models.TextField(**BNULL)
|
||||
website_url = models.URLField(max_length=500, **BNULL)
|
||||
|
||||
def __str__(self):
|
||||
return self.title
|
||||
|
||||
|
||||
class Author(TimeStampedModel):
|
||||
name = models.CharField(max_length=255)
|
||||
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
|
||||
@ -149,6 +166,8 @@ class Book(LongPlayScrobblableMixin):
|
||||
first_sentence = models.TextField(**BNULL)
|
||||
# ComicVine
|
||||
comicvine_id = models.CharField(max_length=255, **BNULL)
|
||||
volume = models.CharField(max_length=255, **BNULL)
|
||||
volume_comicvine_id = models.CharField(max_length=255, **BNULL)
|
||||
readcomics_url = models.CharField(max_length=255, **BNULL)
|
||||
next_readcomics_url = models.CharField(max_length=255, **BNULL)
|
||||
issue_number = models.IntegerField(**BNULL)
|
||||
@ -173,6 +192,9 @@ class Book(LongPlayScrobblableMixin):
|
||||
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
|
||||
|
||||
def __str__(self) -> str:
|
||||
if not self.subtitle:
|
||||
return self.title
|
||||
|
||||
return f"{self.title} - {self.subtitle}"
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
@ -184,12 +206,18 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
subtitle = self.author
|
||||
subtitle_parts = []
|
||||
if self.author:
|
||||
subtitle_parts.append(self.author.name)
|
||||
if self.issue_number and "Issue" not in str(self.title):
|
||||
subtitle += " - Issue {self.issue_number}"
|
||||
subtitle_parts.append(f"Issue {self.issue_number}")
|
||||
if self.volume_number and "Volume" not in str(self.title):
|
||||
subtitle += " - Volume {self.volume_number}"
|
||||
return subtitle
|
||||
subtitle_parts.append(f"Volume {self.volume_number}")
|
||||
if len(subtitle_parts) > 1:
|
||||
return " / ".join(subtitle_parts)
|
||||
if len(subtitle_parts) == 1:
|
||||
return subtitle_parts[0]
|
||||
return ""
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
@ -211,7 +239,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
@property
|
||||
def resume_start_url(self):
|
||||
return reverse("scrobbles:start", kwargs={"uuid": self.uuid}) + "?resume=1"
|
||||
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid}) + "?resume=1"
|
||||
|
||||
@classmethod
|
||||
def get_from_comicvine(
|
||||
@ -219,36 +247,24 @@ class Book(LongPlayScrobblableMixin):
|
||||
) -> "Book":
|
||||
book, created = cls.objects.get_or_create(title=title)
|
||||
|
||||
if not created:
|
||||
if not created and not overwrite:
|
||||
return book
|
||||
|
||||
book_dict = lookup_comic_from_comicvine(title)
|
||||
if not book_dict:
|
||||
return book
|
||||
|
||||
if created or overwrite:
|
||||
author_list = []
|
||||
author_dicts = book_dict.pop("author_dicts")
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
if author_dict.get("authorId"):
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
semantic_id=author_dict.get("authorId")
|
||||
)
|
||||
author_list.append(author)
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
# TODO enrich author?
|
||||
...
|
||||
tags = book_dict.pop("tags", [])
|
||||
genres = book_dict.pop("genres", book_dict.pop("generes", []))
|
||||
|
||||
for k, v in book_dict.items():
|
||||
setattr(book, k, v)
|
||||
book.save()
|
||||
for k, v in book_dict.items():
|
||||
setattr(book, k, v)
|
||||
book.save()
|
||||
|
||||
if author_list:
|
||||
book.authors.add(*author_list)
|
||||
genres = book_dict.pop("genres", [])
|
||||
if genres:
|
||||
book.genre.add(*genres)
|
||||
if genres:
|
||||
book.genre.add(*genres)
|
||||
if tags:
|
||||
book.tags.add(*tags)
|
||||
return book
|
||||
|
||||
@classmethod
|
||||
@ -259,6 +275,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
url: str = "",
|
||||
enrich: bool = True,
|
||||
commit: bool = True,
|
||||
amazon_id: str | None = None,
|
||||
):
|
||||
"""Given a title, get a Book instance.
|
||||
|
||||
@ -282,24 +299,50 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
book_dict = None
|
||||
source_tag = None
|
||||
tried_comicvine = False
|
||||
if READCOMICSONLINE_URL in url:
|
||||
book_dict = lookup_comic_from_comicvine(title)
|
||||
tried_comicvine = True
|
||||
if book_dict:
|
||||
source_tag = MediaSourceTag.COMICVINE
|
||||
book_dict["readcomics_url"] = get_comic_issue_url(url)
|
||||
book_dict["next_readcomics_url"] = next_url_if_exists(
|
||||
book_dict["readcomics_url"]
|
||||
)
|
||||
book_dict["readcomics_url"] = get_comic_issue_url(url)
|
||||
book_dict["next_readcomics_url"] = next_url_if_exists(
|
||||
book_dict["readcomics_url"]
|
||||
)
|
||||
|
||||
if not book_dict:
|
||||
book_dict = lookup_book_from_ol(title, author=author)
|
||||
if book_dict:
|
||||
book_dict = {}
|
||||
ol_data = lookup_book_from_ol(title, author=author)
|
||||
google_data = lookup_book_from_google(title)
|
||||
|
||||
if ol_data:
|
||||
book_dict.update(ol_data)
|
||||
source_tag = MediaSourceTag.OPENLIBRARY
|
||||
|
||||
if not book_dict:
|
||||
book_dict = lookup_book_from_google(title)
|
||||
if book_dict:
|
||||
if google_data:
|
||||
for k, v in google_data.items():
|
||||
if v:
|
||||
book_dict.setdefault(k, v)
|
||||
source_tag = MediaSourceTag.GOOGLE_BOOKS
|
||||
if ol_data and ol_data.get("cover_url"):
|
||||
book_dict["cover_url"] = ol_data["cover_url"]
|
||||
|
||||
# Always try ComicVine as a fallback — it may recognize books that
|
||||
# OL/Google don't flag as comics
|
||||
if not tried_comicvine:
|
||||
cv_data = lookup_comic_from_comicvine(title)
|
||||
if cv_data:
|
||||
for k, v in cv_data.items():
|
||||
if v:
|
||||
book_dict.setdefault(k, v)
|
||||
source_tag = MediaSourceTag.COMICVINE
|
||||
|
||||
# Try Amazon PAAPI as a fallback when given an ASIN
|
||||
if amazon_id and not book_dict:
|
||||
amazon_data = lookup_book_from_amazon(amazon_id)
|
||||
if amazon_data:
|
||||
book_dict.update(amazon_data)
|
||||
source_tag = MediaSourceTag.AMAZON
|
||||
|
||||
if not book_dict:
|
||||
logger.warning(
|
||||
@ -311,6 +354,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
authors = book_dict.pop("authors", [])
|
||||
cover_url = book_dict.pop("cover_url", "")
|
||||
genres = book_dict.pop("genres", book_dict.pop("generes", []))
|
||||
tags = book_dict.pop("tags", [])
|
||||
|
||||
if authors:
|
||||
for author_str in authors:
|
||||
@ -330,6 +374,8 @@ class Book(LongPlayScrobblableMixin):
|
||||
book.save_image_from_url(cover_url)
|
||||
if genres:
|
||||
book.genre.add(*genres)
|
||||
if tags:
|
||||
book.tags.add(*tags)
|
||||
book.authors.add(*author_list)
|
||||
if source_tag:
|
||||
book.tags.add(source_tag.value)
|
||||
@ -367,9 +413,14 @@ class Book(LongPlayScrobblableMixin):
|
||||
data = lookup_comic_from_locg(str(self.title))
|
||||
|
||||
if not data and COMICVINE_API_KEY:
|
||||
logger.warn(f"Checking ComicVine for {self.title}")
|
||||
cv_client = ComicVineClient(api_key=COMICVINE_API_KEY)
|
||||
data = lookup_comic_from_comicvine(str(self.title))
|
||||
if self.comicvine_id:
|
||||
logger.warn(
|
||||
f"Checking ComicVine by ID for {self.title}"
|
||||
)
|
||||
data = lookup_issue_by_comicvine_id(str(self.comicvine_id))
|
||||
if not data:
|
||||
logger.warn(f"Checking ComicVine for {self.title}")
|
||||
data = lookup_comic_from_comicvine(str(self.title))
|
||||
|
||||
if not data:
|
||||
logger.warn(f"Book not found in any sources: {self.title}")
|
||||
@ -407,10 +458,10 @@ class Book(LongPlayScrobblableMixin):
|
||||
)
|
||||
data.pop("pages")
|
||||
|
||||
# Pop this, so we can look it up later
|
||||
# Pop these so they don't get passed to update()
|
||||
cover_url = data.pop("cover_url", "")
|
||||
|
||||
subject_key_list = data.pop("subject_key_list", [])
|
||||
tags = data.pop("tags", [])
|
||||
|
||||
# Fun trick for updating all fields at once
|
||||
Book.objects.filter(pk=self.id).update(**data)
|
||||
@ -418,6 +469,8 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
if subject_key_list:
|
||||
self.genre.add(*subject_key_list)
|
||||
if tags:
|
||||
self.tags.add(*tags)
|
||||
|
||||
if cover_url:
|
||||
r = requests.get(cover_url)
|
||||
@ -463,8 +516,11 @@ class Book(LongPlayScrobblableMixin):
|
||||
if scrobble.logdata.page_data:
|
||||
for page, data in scrobble.logdata.page_data.items():
|
||||
if convert_timestamps:
|
||||
data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
|
||||
data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
|
||||
tz = None
|
||||
if scrobble.timezone:
|
||||
tz = ZoneInfo(scrobble.timezone)
|
||||
data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
|
||||
data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
|
||||
pages[page] = data
|
||||
sorted_pages = OrderedDict(
|
||||
sorted(pages.items(), key=lambda x: x[1]["start_ts"])
|
||||
@ -498,6 +554,21 @@ class Book(LongPlayScrobblableMixin):
|
||||
return progress
|
||||
|
||||
|
||||
@dataclass
|
||||
class PaperLogData(BaseLogData):
|
||||
@classmethod
|
||||
def override_fields(cls) -> dict:
|
||||
from scrobbles.forms import NotesDictField
|
||||
|
||||
fields = {}
|
||||
for base in cls.mro()[1:]:
|
||||
if hasattr(base, "override_fields"):
|
||||
base_fields = base.override_fields()
|
||||
fields.update(base_fields)
|
||||
fields["notes"] = NotesDictField(required=False)
|
||||
return fields
|
||||
|
||||
|
||||
class Paper(LongPlayScrobblableMixin):
|
||||
"""Keeps track of Academic Papers"""
|
||||
|
||||
@ -517,14 +588,29 @@ class Paper(LongPlayScrobblableMixin):
|
||||
language = models.CharField(max_length=4, **BNULL)
|
||||
first_publish_year = models.IntegerField(**BNULL)
|
||||
publish_date = models.DateField(**BNULL)
|
||||
journal = models.CharField(max_length=255, **BNULL)
|
||||
journal = models.ForeignKey(Journal, on_delete=models.DO_NOTHING, **BNULL)
|
||||
journal_volume = models.CharField(max_length=50, **BNULL)
|
||||
abstract = models.TextField(**BNULL)
|
||||
tldr = models.CharField(max_length=255, **BNULL)
|
||||
openaccess_pdf_url = models.CharField(max_length=255, **BNULL)
|
||||
pdf_file = models.FileField(upload_to="papers/pdf/", **BNULL)
|
||||
|
||||
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
|
||||
|
||||
@property
|
||||
def logdata_cls(self):
|
||||
return PaperLogData
|
||||
|
||||
@property
|
||||
def scihub_url(self):
|
||||
if not self.doi_id:
|
||||
return None
|
||||
domain = getattr(settings, "SCIHUB_DOMAIN", "sci-hub.st")
|
||||
return f"https://{domain}/{self.doi_id}"
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("books:paper_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@classmethod
|
||||
def get_from_semantic(cls, title: str, overwrite: bool = False) -> "Paper":
|
||||
paper, created = cls.objects.get_or_create(title=title)
|
||||
@ -535,7 +621,7 @@ class Paper(LongPlayScrobblableMixin):
|
||||
|
||||
if created or overwrite:
|
||||
author_list = []
|
||||
author_dicts = paper_dict.pop("author_dicts")
|
||||
author_dicts = paper_dict.pop("author_dicts", None)
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
if author_dict.get("authorId"):
|
||||
@ -546,8 +632,11 @@ class Paper(LongPlayScrobblableMixin):
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
# TODO enrich author?
|
||||
...
|
||||
|
||||
journal_name = paper_dict.pop("journal_name", None)
|
||||
if journal_name:
|
||||
journal, _ = Journal.objects.get_or_create(title=journal_name)
|
||||
paper.journal = journal
|
||||
|
||||
for k, v in paper_dict.items():
|
||||
setattr(paper, k, v)
|
||||
@ -559,3 +648,78 @@ class Paper(LongPlayScrobblableMixin):
|
||||
if genres:
|
||||
paper.genre.add(*genres)
|
||||
return paper
|
||||
|
||||
@classmethod
|
||||
def find_or_create_by_doi(cls, doi_url: str) -> "Paper":
|
||||
doi = doi_url.replace("https://doi.org/", "").split("?")[0].rstrip("/")
|
||||
paper = cls.objects.filter(doi_id=doi).first()
|
||||
if paper:
|
||||
return paper
|
||||
|
||||
paper = cls(doi_id=doi, title=f"Paper {doi}")
|
||||
paper.save()
|
||||
|
||||
from books.sources.crossref import lookup_paper_from_crossref
|
||||
|
||||
paper_dict = lookup_paper_from_semantic_by_doi(doi)
|
||||
if not paper_dict or not paper_dict.get("abstract"):
|
||||
paper_dict = lookup_paper_from_crossref(doi)
|
||||
|
||||
if paper_dict:
|
||||
author_list = []
|
||||
author_dicts = paper_dict.pop("author_dicts", None)
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
author_id = author_dict.get("authorId")
|
||||
if author_id:
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
semantic_id=author_id
|
||||
)
|
||||
author_list.append(author)
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
else:
|
||||
author_name = author_dict.get("name")
|
||||
if author_name:
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
name=author_name
|
||||
)
|
||||
author_list.append(author)
|
||||
|
||||
journal_name = paper_dict.pop("journal_name", None)
|
||||
if journal_name:
|
||||
journal, _ = Journal.objects.get_or_create(title=journal_name)
|
||||
paper.journal = journal
|
||||
|
||||
for k, v in paper_dict.items():
|
||||
if v is not None:
|
||||
setattr(paper, k, v)
|
||||
paper.save()
|
||||
|
||||
if author_list:
|
||||
paper.authors.add(*author_list)
|
||||
genres = paper_dict.pop("genres", [])
|
||||
if genres:
|
||||
paper.genre.add(*genres)
|
||||
|
||||
if not paper.pdf_file:
|
||||
service = SciHubService()
|
||||
if paper.openaccess_pdf_url:
|
||||
pdf_content = service.fetch_from_url(paper.openaccess_pdf_url)
|
||||
if pdf_content:
|
||||
filename = f"{doi.replace('/', '_')}.pdf"
|
||||
paper.pdf_file.save(filename, ContentFile(pdf_content))
|
||||
if not paper.pdf_file:
|
||||
try:
|
||||
pdf_content = service.fetch_pdf(doi)
|
||||
if pdf_content:
|
||||
filename = f"{doi.replace('/', '_')}.pdf"
|
||||
paper.pdf_file.save(filename, ContentFile(pdf_content))
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
"[paper] sci-hub PDF download failed",
|
||||
extra={"doi": doi, "error": str(e)},
|
||||
)
|
||||
|
||||
return paper
|
||||
|
||||
123
vrobbler/apps/books/sources/amazon.py
Normal file
123
vrobbler/apps/books/sources/amazon.py
Normal file
@ -0,0 +1,123 @@
|
||||
import logging
|
||||
import warnings
|
||||
|
||||
from django.conf import settings
|
||||
|
||||
with warnings.catch_warnings():
|
||||
warnings.simplefilter("ignore", DeprecationWarning)
|
||||
from amazon_paapi import AmazonApi
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_amazon_client = None
|
||||
|
||||
|
||||
def _get_client() -> AmazonApi | None:
|
||||
global _amazon_client
|
||||
if _amazon_client is not None:
|
||||
return _amazon_client
|
||||
|
||||
key = settings.AMAZON_PAAPI_ACCESS_KEY
|
||||
secret = settings.AMAZON_PAAPI_SECRET_KEY
|
||||
tag = settings.AMAZON_PAAPI_ASSOCIATE_TAG
|
||||
country = settings.AMAZON_PAAPI_COUNTRY
|
||||
|
||||
if not all([key, secret, tag]):
|
||||
logger.warning("Amazon PAAPI credentials not configured")
|
||||
return None
|
||||
|
||||
_amazon_client = AmazonApi(key, secret, tag, country)
|
||||
return _amazon_client
|
||||
|
||||
|
||||
def lookup_book_from_amazon(asin: str) -> dict:
|
||||
book_dict: dict = {}
|
||||
|
||||
client = _get_client()
|
||||
if not client:
|
||||
return book_dict
|
||||
|
||||
try:
|
||||
items = client.get_items(
|
||||
items=[asin],
|
||||
Condition="New",
|
||||
LanguagesOfPreference=["en_US"],
|
||||
)
|
||||
except Exception as e:
|
||||
logger.warning(f"Amazon PAAPI lookup failed for {asin}: {e}")
|
||||
return book_dict
|
||||
|
||||
if not items:
|
||||
logger.info(f"No Amazon item found for {asin}")
|
||||
return book_dict
|
||||
|
||||
item = items[0]
|
||||
raw = item.to_dict()
|
||||
item_info = raw.get("item_info", {}) or {}
|
||||
|
||||
book_dict["title"] = _get_nested(item_info, "title", "display_value")
|
||||
if not book_dict.get("title"):
|
||||
book_dict["title"] = _get_nested(item_info, "title", "value")
|
||||
|
||||
contributors = _get_nested(item_info, "by_line_info", "contributors") or []
|
||||
authors = [
|
||||
c["name"]
|
||||
for c in contributors
|
||||
if c.get("role", "").lower() in ("author", "artist", "writer")
|
||||
]
|
||||
if authors:
|
||||
book_dict["authors"] = authors
|
||||
|
||||
publisher = _get_nested(item_info, "by_line_info", "manufacturer")
|
||||
if publisher:
|
||||
book_dict["publisher"] = publisher
|
||||
|
||||
isb_ns = _get_nested(item_info, "external_ids", "isb_ns")
|
||||
if isb_ns and isinstance(isb_ns, list):
|
||||
for isb in isb_ns:
|
||||
if isinstance(isb, dict):
|
||||
if isb.get("type") == "ISBN_13":
|
||||
book_dict["isbn_13"] = isb.get("value")
|
||||
elif isb.get("type") == "ISBN_10":
|
||||
book_dict["isbn_10"] = isb.get("value")
|
||||
|
||||
pages_count = _get_nested(item_info, "content_info", "pages_count")
|
||||
if pages_count and isinstance(pages_count, dict):
|
||||
book_dict["pages"] = pages_count.get("value") or pages_count.get("display_value")
|
||||
|
||||
languages = _get_nested(item_info, "content_info", "languages") or []
|
||||
if languages and isinstance(languages, list):
|
||||
lang = languages[0]
|
||||
if isinstance(lang, dict):
|
||||
book_dict["language"] = lang.get("display_value") or lang.get("value")
|
||||
|
||||
pub_date = _get_nested(item_info, "content_info", "publication_date")
|
||||
if not pub_date:
|
||||
pub_date = _get_nested(item_info, "product_info", "release_date")
|
||||
if pub_date and isinstance(pub_date, dict):
|
||||
book_dict["publish_date"] = pub_date.get("display_value") or pub_date.get("value")
|
||||
|
||||
features = item_info.get("features") or []
|
||||
if features and isinstance(features, list):
|
||||
book_dict["summary"] = " ".join(features[:5])
|
||||
|
||||
images = raw.get("images", {}) or {}
|
||||
primary = images.get("primary", {}) or {}
|
||||
for size in ("large", "hi_res", "medium"):
|
||||
candidate = primary.get(size, {}) or {}
|
||||
url = candidate.get("url")
|
||||
if url:
|
||||
book_dict["cover_url"] = url
|
||||
break
|
||||
|
||||
book_dict["detail_page_url"] = raw.get("detail_page_url")
|
||||
|
||||
return book_dict
|
||||
|
||||
|
||||
def _get_nested(d: dict, *keys):
|
||||
for key in keys:
|
||||
if not isinstance(d, dict):
|
||||
return None
|
||||
d = d.get(key)
|
||||
return d
|
||||
@ -17,8 +17,10 @@ class ComicVineClient(object):
|
||||
account on https://comicvine.gamespot.com/ in order to obtain an API key.
|
||||
"""
|
||||
|
||||
# All API requests made by this client will be made to this URL.
|
||||
API_URL = "https://www.comicvine.com/api/search/"
|
||||
# All API requests made by this client will be made to these URLs.
|
||||
API_URL = "https://comicvine.gamespot.com/api/search/"
|
||||
ISSUE_API_URL = "https://comicvine.gamespot.com/api/issue/4000-{issue_id}/"
|
||||
VOLUME_API_URL = "https://comicvine.gamespot.com/api/volume/4050-{volume_id}/"
|
||||
|
||||
# A valid User-Agent header must be set in order for our API requests to
|
||||
# be accepted, otherwise our request will be rejected with a
|
||||
@ -41,15 +43,12 @@ class ComicVineClient(object):
|
||||
"volume",
|
||||
}
|
||||
|
||||
def __init__(self, api_key, expire_after=300):
|
||||
def __init__(self, api_key):
|
||||
"""
|
||||
Store the API key in a class variable, and install the requests cache,
|
||||
configuring it using the ``expire_after`` parameter.
|
||||
Store the API key in a class variable.
|
||||
|
||||
:param api_key: Your personal ComicVine API key.
|
||||
:type api_key: str
|
||||
:param expire_after: The number of seconds to retain an entry in cache.
|
||||
:type expire_after: int or None
|
||||
"""
|
||||
|
||||
self.api_key = api_key
|
||||
@ -109,14 +108,17 @@ class ComicVineClient(object):
|
||||
:rtype: dict
|
||||
"""
|
||||
|
||||
return {
|
||||
params = {
|
||||
"api_key": self.api_key,
|
||||
"format": "json",
|
||||
"limit": min(10, limit), # hard limit of 10
|
||||
"offset": max(0, offset), # cannot provide negative offset
|
||||
"query": query,
|
||||
"resources": self._validate_resources(resources),
|
||||
}
|
||||
validated = self._validate_resources(resources)
|
||||
if validated:
|
||||
params["resources"] = validated
|
||||
return params
|
||||
|
||||
def _validate_resources(self, resources):
|
||||
"""
|
||||
@ -141,33 +143,35 @@ class ComicVineClient(object):
|
||||
def _query_api(self, params):
|
||||
"""
|
||||
Query the ComicVine API's ``search`` resource, providing the required
|
||||
headers and parameters with the request. Optionally allow the caller
|
||||
of the function to disable the request cache.
|
||||
headers and parameters with the request.
|
||||
|
||||
If an error occurs during the request, handle it accordingly. Upon
|
||||
success, return the JSON from the response.
|
||||
|
||||
:param params: Parameters to include with the request.
|
||||
:type params: dict
|
||||
:param use_cache: Toggle the use of requests_cache.
|
||||
:type use_cache: bool
|
||||
|
||||
:return: The JSON contained in the response.
|
||||
:rtype: dict
|
||||
"""
|
||||
|
||||
# Since we're performing the identical action regardless of whether
|
||||
# or not the request cache is to be used, store the procedure in a
|
||||
# local function to avoid repetition.
|
||||
def __httpget():
|
||||
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
|
||||
response = requests.get(self.API_URL, headers=self.HEADERS, params=params)
|
||||
|
||||
if not response.ok:
|
||||
self._handle_http_error(response)
|
||||
if not response.ok:
|
||||
self._handle_http_error(response)
|
||||
|
||||
return response.json()
|
||||
json_data = response.json()
|
||||
|
||||
return __httpget()
|
||||
if json_data.get("status_code") != 1:
|
||||
error_msg = json_data.get("error", "Unknown ComicVine API error")
|
||||
logger.error(
|
||||
"ComicVine API returned status_code %s: %s",
|
||||
json_data.get("status_code"),
|
||||
error_msg,
|
||||
)
|
||||
return {}
|
||||
|
||||
return json_data
|
||||
|
||||
def _handle_http_error(self, response):
|
||||
"""
|
||||
@ -195,15 +199,81 @@ class ComicVineClient(object):
|
||||
|
||||
raise exception(message)
|
||||
|
||||
def get_issue(self, issue_id: str) -> dict:
|
||||
"""
|
||||
Fetch a single issue by its ComicVine ID directly from the issue detail
|
||||
endpoint, which returns richer data than the search endpoint.
|
||||
|
||||
:param issue_id: The ComicVine numeric ID for the issue (e.g. "538480")
|
||||
:type issue_id: str
|
||||
|
||||
:return: The full JSON response for the issue, or empty dict on failure.
|
||||
:rtype: dict
|
||||
"""
|
||||
params = {
|
||||
"api_key": self.api_key,
|
||||
"format": "json",
|
||||
}
|
||||
url = self.ISSUE_API_URL.format(issue_id=issue_id)
|
||||
response = requests.get(url, headers=self.HEADERS, params=params)
|
||||
|
||||
if not response.ok:
|
||||
self._handle_http_error(response)
|
||||
|
||||
json_data = response.json()
|
||||
|
||||
if json_data.get("status_code") != 1:
|
||||
error_msg = json_data.get("error", "Unknown ComicVine API error")
|
||||
logger.error(
|
||||
"ComicVine API returned status_code %s: %s",
|
||||
json_data.get("status_code"),
|
||||
error_msg,
|
||||
)
|
||||
return {}
|
||||
|
||||
return json_data.get("results", {})
|
||||
|
||||
def get_volume(self, volume_id: str) -> dict:
|
||||
"""
|
||||
Fetch a single volume by its ComicVine ID from the volume detail
|
||||
endpoint. Used to get publisher info and other volume-level metadata.
|
||||
|
||||
:param volume_id: The ComicVine numeric ID for the volume (e.g. "91273")
|
||||
:type volume_id: str
|
||||
|
||||
:return: The full JSON response for the volume, or empty dict on failure.
|
||||
:rtype: dict
|
||||
"""
|
||||
params = {
|
||||
"api_key": self.api_key,
|
||||
"format": "json",
|
||||
}
|
||||
url = self.VOLUME_API_URL.format(volume_id=volume_id)
|
||||
response = requests.get(url, headers=self.HEADERS, params=params)
|
||||
|
||||
if not response.ok:
|
||||
self._handle_http_error(response)
|
||||
|
||||
json_data = response.json()
|
||||
|
||||
if json_data.get("status_code") != 1:
|
||||
error_msg = json_data.get("error", "Unknown ComicVine API error")
|
||||
logger.error(
|
||||
"ComicVine API returned status_code %s: %s",
|
||||
json_data.get("status_code"),
|
||||
error_msg,
|
||||
)
|
||||
return {}
|
||||
|
||||
return json_data.get("results", {})
|
||||
|
||||
|
||||
def lookup_comic_from_comicvine(title: str) -> dict:
|
||||
original_title = title
|
||||
|
||||
issue_number = None
|
||||
volume_nubmer = None
|
||||
resource_type = "issue"
|
||||
if "Issue " in title:
|
||||
resource_type = "issue"
|
||||
issue_number = title.split("Issue ")[1]
|
||||
volume_number = None
|
||||
if "Volume " in title:
|
||||
@ -215,48 +285,136 @@ def lookup_comic_from_comicvine(title: str) -> dict:
|
||||
logger.warning("No ComicVine API key configured, not looking anything up")
|
||||
return {}
|
||||
|
||||
client = ComicVineClient(api_key=getattr(settings, "COMICVINE_API_KEY", None))
|
||||
client = ComicVineClient(api_key=api_key)
|
||||
|
||||
raw_results = client.search(title).get("results")
|
||||
results = [r for r in raw_results if r.get("resource_type") == resource_type]
|
||||
raw_results = client.search(title)
|
||||
if not raw_results:
|
||||
return {}
|
||||
results = raw_results.get("results", [])
|
||||
results = [r for r in results if r.get("resource_type") == resource_type]
|
||||
if not results:
|
||||
logger.warning("No comic found on ComicVine")
|
||||
return {}
|
||||
|
||||
found_result = None
|
||||
for result in results:
|
||||
if result.get("issue_number") == str(issue_number):
|
||||
if issue_number is not None and result.get("issue_number") == str(issue_number):
|
||||
found_result = result
|
||||
break
|
||||
if result.get("volume_number") == str(volume_number):
|
||||
if volume_number is not None and result.get("volume_number") == str(volume_number):
|
||||
found_result = result
|
||||
break
|
||||
|
||||
if not found_result:
|
||||
found_result = results[0]
|
||||
|
||||
logger.info("ComicVine results", extra={"results": results})
|
||||
data_dict = _build_data_dict_from_issue(found_result, original_title)
|
||||
_enrich_with_volume_data(client, data_dict)
|
||||
return data_dict
|
||||
|
||||
if not found_result:
|
||||
logger.warning("No matches found on ComicVine")
|
||||
|
||||
def lookup_issue_by_comicvine_id(comicvine_id: str) -> dict:
|
||||
"""
|
||||
Look up an issue directly by its ComicVine ID using the issue detail
|
||||
endpoint. Returns richer data than the search-based lookup.
|
||||
|
||||
:param comicvine_id: The ComicVine numeric ID for the issue (e.g. "538480")
|
||||
:type comicvine_id: str
|
||||
|
||||
:return: A dict of extracted book metadata, or empty dict on failure.
|
||||
:rtype: dict
|
||||
"""
|
||||
if not comicvine_id:
|
||||
return {}
|
||||
|
||||
title = found_result.get("name")
|
||||
api_key = getattr(settings, "COMICVINE_API_KEY", "")
|
||||
if not api_key:
|
||||
logger.warning("No ComicVine API key configured, not looking anything up")
|
||||
return {}
|
||||
|
||||
if found_result.get("volume"):
|
||||
title = found_result.get("volume").get("name")
|
||||
client = ComicVineClient(api_key=api_key)
|
||||
issue_data = client.get_issue(comicvine_id)
|
||||
if not issue_data:
|
||||
logger.warning("No issue found on ComicVine for ID %s", comicvine_id)
|
||||
return {}
|
||||
|
||||
data_dict = _build_data_dict_from_issue(issue_data, issue_data.get("name", ""))
|
||||
_enrich_with_volume_data(client, data_dict)
|
||||
return data_dict
|
||||
|
||||
|
||||
def _build_data_dict_from_issue(issue_data: dict, original_title: str = "") -> dict:
|
||||
"""
|
||||
Build a book metadata dict from a ComicVine issue resource (either from
|
||||
search results or issue detail endpoint). Both return the same shape of
|
||||
issue data.
|
||||
|
||||
:param issue_data: The issue resource dict from ComicVine.
|
||||
:param original_title: The original search term, if any.
|
||||
|
||||
:return: A dict of extracted book metadata.
|
||||
:rtype: dict
|
||||
"""
|
||||
title = issue_data.get("name")
|
||||
if issue_data.get("volume"):
|
||||
title = issue_data.get("volume").get("name")
|
||||
|
||||
cover_url = None
|
||||
if issue_data.get("image"):
|
||||
cover_url = issue_data["image"].get("original_url")
|
||||
|
||||
volume_name = None
|
||||
volume_cv_id = None
|
||||
publisher_name = None
|
||||
volume_data = issue_data.get("volume")
|
||||
if volume_data:
|
||||
volume_name = volume_data.get("name")
|
||||
volume_cv_id = volume_data.get("id")
|
||||
publisher_data = volume_data.get("publisher")
|
||||
if publisher_data:
|
||||
publisher_name = publisher_data.get("name")
|
||||
|
||||
data_dict = {
|
||||
"title": title,
|
||||
"original_title": original_title,
|
||||
"issue_number": found_result.get("issue_number"),
|
||||
"volume_number": found_result.get("volume_number"),
|
||||
"cover_url": found_result.get("image").get("original_url"),
|
||||
"comicvine_id": found_result.get("id"),
|
||||
"comicvine_data": found_result,
|
||||
"summary": found_result.get("description"),
|
||||
"publish_date": found_result.get("cover_date"),
|
||||
"first_publish_year": found_result.get("cover_date", "")[:4],
|
||||
"issue_number": issue_data.get("issue_number"),
|
||||
"volume_number": issue_data.get("volume_number"),
|
||||
"volume": volume_name,
|
||||
"volume_comicvine_id": volume_cv_id,
|
||||
"publisher": publisher_name,
|
||||
"cover_url": cover_url,
|
||||
"comicvine_id": issue_data.get("id"),
|
||||
"summary": issue_data.get("description"),
|
||||
"publish_date": issue_data.get("cover_date"),
|
||||
"first_publish_year": (issue_data.get("cover_date") or "")[:4],
|
||||
"tags": ["comicbook"],
|
||||
}
|
||||
|
||||
return data_dict
|
||||
|
||||
|
||||
def _enrich_with_volume_data(client: ComicVineClient, data_dict: dict) -> None:
|
||||
"""
|
||||
Follow-up a successful issue lookup by fetching the volume detail and
|
||||
filling in publisher and other volume-level metadata that the issue
|
||||
endpoint doesn't provide.
|
||||
|
||||
:param client: An initialised ComicVineClient instance.
|
||||
:param data_dict: The data dict from an issue lookup (mutated in place).
|
||||
"""
|
||||
volume_cv_id = data_dict.get("volume_comicvine_id")
|
||||
if not volume_cv_id:
|
||||
return
|
||||
|
||||
volume_data = client.get_volume(str(volume_cv_id))
|
||||
if not volume_data:
|
||||
return
|
||||
|
||||
publisher_data = volume_data.get("publisher")
|
||||
if publisher_data:
|
||||
publisher_name = publisher_data.get("name")
|
||||
if publisher_name and not data_dict.get("publisher"):
|
||||
data_dict["publisher"] = publisher_name
|
||||
|
||||
if not data_dict.get("volume"):
|
||||
data_dict["volume"] = volume_data.get("name")
|
||||
|
||||
94
vrobbler/apps/books/sources/crossref.py
Normal file
94
vrobbler/apps/books/sources/crossref.py
Normal file
@ -0,0 +1,94 @@
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
import requests
|
||||
|
||||
CROSSREF_WORK_URL = "https://api.crossref.org/works/{}"
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _strip_jats(text: str) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
text = re.sub(r"</?jats:[^>]*>", "", text)
|
||||
text = re.sub(r"^\s*Abstract\s*", "", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def lookup_paper_from_crossref(doi: str) -> dict:
|
||||
url = CROSSREF_WORK_URL.format(doi)
|
||||
headers = {"User-Agent": "Vrobbler/1.0 (mailto:hello@example.com)"}
|
||||
response = requests.get(url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.warning(
|
||||
"Bad response from Crossref",
|
||||
extra={"doi": doi, "status": response.status_code},
|
||||
)
|
||||
return {"doi_id": doi}
|
||||
|
||||
try:
|
||||
data = response.json()
|
||||
except json.JSONDecodeError:
|
||||
return {"doi_id": doi}
|
||||
|
||||
msg = data.get("message", {})
|
||||
if not msg:
|
||||
return {"doi_id": doi}
|
||||
|
||||
paper_dict = {"doi_id": doi}
|
||||
|
||||
titles = msg.get("title", [])
|
||||
if titles:
|
||||
paper_dict["title"] = titles[0]
|
||||
|
||||
abstract = msg.get("abstract", "")
|
||||
if abstract:
|
||||
paper_dict["abstract"] = _strip_jats(abstract)
|
||||
|
||||
author_dicts = []
|
||||
for author in msg.get("author", []):
|
||||
given = author.get("given", "")
|
||||
family = author.get("family", "")
|
||||
name = f"{given} {family}".strip()
|
||||
if not name:
|
||||
continue
|
||||
entry = {"name": name}
|
||||
orcid = author.get("ORCID", "")
|
||||
if orcid:
|
||||
orcid_id = orcid.replace("https://orcid.org/", "")
|
||||
entry["authorId"] = orcid_id
|
||||
author_dicts.append(entry)
|
||||
if author_dicts:
|
||||
paper_dict["author_dicts"] = author_dicts
|
||||
|
||||
container = msg.get("container-title", [])
|
||||
if container:
|
||||
paper_dict["journal_name"] = container[0]
|
||||
|
||||
volume = msg.get("volume")
|
||||
if volume:
|
||||
paper_dict["journal_volume"] = volume
|
||||
|
||||
page = msg.get("page")
|
||||
if page:
|
||||
try:
|
||||
parts = page.split("-")
|
||||
if len(parts) == 2:
|
||||
paper_dict["pages"] = int(parts[1]) - int(parts[0])
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
for date_field in ("published-print", "published-online", "created"):
|
||||
date_data = msg.get(date_field)
|
||||
if date_data and date_data.get("date-parts"):
|
||||
parts = date_data["date-parts"][0]
|
||||
if len(parts) >= 1:
|
||||
paper_dict["first_publish_year"] = int(parts[0])
|
||||
if len(parts) >= 3:
|
||||
paper_dict["publish_date"] = f"{parts[0]:04d}-{parts[1]:02d}-{parts[2]:02d}"
|
||||
break
|
||||
|
||||
return paper_dict
|
||||
@ -26,8 +26,6 @@ def lookup_book_from_google(title: str) -> dict:
|
||||
if not google_result:
|
||||
return {}
|
||||
|
||||
publish_date = pendulum.parse(google_result.get("publishedDate"))
|
||||
|
||||
isbn_13 = ""
|
||||
isbn_10 = ""
|
||||
for ident in google_result.get("industryIdentifiers", []):
|
||||
@ -35,25 +33,25 @@ def lookup_book_from_google(title: str) -> dict:
|
||||
isbn_13 = ident.get("identifier")
|
||||
if ident.get("type") == "ISBN_10":
|
||||
isbn_10 = ident.get("identifier")
|
||||
# TODO this may lead to issues with the first get if Google changes our title
|
||||
# book_metadata.title = google_result.get("title")
|
||||
# if google_result.get("subtitle"):
|
||||
# book_metadata["title"] = ": ".join(
|
||||
# [google_result.get("title"), google_result.get("subtitle")]
|
||||
# )
|
||||
# book_dict["subtitle"] = google_result.get("subtitle")
|
||||
book_dict["authors"] = google_result.get("authors")
|
||||
book_dict["publisher"] = google_result.get("publisher")
|
||||
book_dict["first_publish_year"] = publish_date.year
|
||||
book_dict["pages"] = google_result.get("pageCount")
|
||||
book_dict["isbn_13"] = isbn_13
|
||||
book_dict["isbn_10"] = isbn_10
|
||||
book_dict["publish_date"] = google_result.get("publishedDate")
|
||||
if len(book_dict["publish_date"]) == 4:
|
||||
book_dict["publish_date"] = f"{book_dict['publish_date']}-1-1"
|
||||
book_dict["language"] = google_result.get("language")
|
||||
book_dict["summary"] = google_result.get("description")
|
||||
book_dict["genres"] = google_result.get("categories")
|
||||
|
||||
raw_date = google_result.get("publishedDate")
|
||||
if raw_date:
|
||||
try:
|
||||
publish_date = pendulum.parse(raw_date)
|
||||
book_dict["first_publish_year"] = publish_date.year
|
||||
except Exception:
|
||||
pass
|
||||
book_dict["publish_date"] = raw_date
|
||||
if len(raw_date) == 4:
|
||||
book_dict["publish_date"] = f"{raw_date}-1-1"
|
||||
book_dict["cover_url"] = (
|
||||
google_result.get("imageLinks", {})
|
||||
.get("thumbnail", "")
|
||||
|
||||
142
vrobbler/apps/books/sources/scihub.py
Normal file
142
vrobbler/apps/books/sources/scihub.py
Normal file
@ -0,0 +1,142 @@
|
||||
import logging
|
||||
from typing import Optional
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from django.conf import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCIHUB_DOMAINS = [
|
||||
"sci-hub.ru",
|
||||
"sci-hub.ee",
|
||||
"sci-hub.st",
|
||||
"sci-hub.do",
|
||||
]
|
||||
|
||||
|
||||
class SciHubService:
|
||||
def __init__(self):
|
||||
self.session = requests.Session()
|
||||
self.session.headers.update(
|
||||
{
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
}
|
||||
)
|
||||
|
||||
def fetch_from_url(self, url: str) -> Optional[bytes]:
|
||||
try:
|
||||
resp = self.session.get(url, timeout=60)
|
||||
if resp.status_code != 200:
|
||||
logger.warning(
|
||||
"[pdf] URL download failed",
|
||||
extra={"status": resp.status_code, "url": url},
|
||||
)
|
||||
return None
|
||||
if not self._looks_like_pdf(resp):
|
||||
return None
|
||||
return resp.content
|
||||
except requests.RequestException as e:
|
||||
logger.error(
|
||||
"[pdf] URL download request failed",
|
||||
extra={"url": url, "error": str(e)},
|
||||
)
|
||||
return None
|
||||
|
||||
def fetch_pdf(self, doi: str) -> Optional[bytes]:
|
||||
configured_domain = getattr(settings, "SCIHUB_DOMAIN", None)
|
||||
domains_to_try = (
|
||||
[configured_domain] + SCIHUB_DOMAINS
|
||||
if configured_domain and configured_domain not in SCIHUB_DOMAINS
|
||||
else SCIHUB_DOMAINS
|
||||
)
|
||||
|
||||
for domain in domains_to_try:
|
||||
url = f"https://{domain}/{doi}"
|
||||
logger.info(
|
||||
"[scihub] trying domain",
|
||||
extra={"domain": domain, "doi": doi},
|
||||
)
|
||||
try:
|
||||
response = self.session.get(url, timeout=30)
|
||||
if response.status_code != 200:
|
||||
continue
|
||||
|
||||
pdf_url = self._extract_pdf_url(response.text, url)
|
||||
if not pdf_url:
|
||||
continue
|
||||
|
||||
pdf_response = self.session.get(pdf_url, timeout=60)
|
||||
if pdf_response.status_code != 200:
|
||||
continue
|
||||
|
||||
if not self._looks_like_pdf(pdf_response):
|
||||
continue
|
||||
|
||||
logger.info(
|
||||
"[scihub] PDF downloaded successfully",
|
||||
extra={
|
||||
"domain": domain,
|
||||
"doi": doi,
|
||||
"size": len(pdf_response.content),
|
||||
},
|
||||
)
|
||||
return pdf_response.content
|
||||
except requests.RequestException as e:
|
||||
logger.debug(
|
||||
"[scihub] domain failed",
|
||||
extra={"domain": domain, "doi": doi, "error": str(e)},
|
||||
)
|
||||
continue
|
||||
|
||||
logger.warning(
|
||||
"[scihub] all domains failed",
|
||||
extra={"doi": doi, "tried": domains_to_try},
|
||||
)
|
||||
return None
|
||||
|
||||
def _looks_like_pdf(self, response: requests.Response) -> bool:
|
||||
content_type = response.headers.get("Content-Type", "")
|
||||
if "application/pdf" in content_type:
|
||||
return True
|
||||
if content_type.startswith("application/octet"):
|
||||
return True
|
||||
if response.url.endswith(".pdf"):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _extract_pdf_url(self, html: str, page_url: str) -> Optional[str]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
|
||||
iframe = soup.find("iframe", {"id": "pdf"})
|
||||
if iframe and iframe.get("src"):
|
||||
src = iframe["src"]
|
||||
if src.startswith("http"):
|
||||
return src
|
||||
return urljoin(page_url, src)
|
||||
|
||||
embed = soup.find("embed", {"type": "application/pdf"})
|
||||
if embed and embed.get("src"):
|
||||
src = embed["src"]
|
||||
if src.startswith("http"):
|
||||
return src
|
||||
return urljoin(page_url, src)
|
||||
|
||||
download_div = soup.find("div", {"id": "download"})
|
||||
if download_div:
|
||||
link = download_div.find("a")
|
||||
if link and link.get("href"):
|
||||
href = link["href"]
|
||||
if href.startswith("http"):
|
||||
return href
|
||||
return urljoin(page_url, href)
|
||||
|
||||
for link in soup.find_all("a", href=True):
|
||||
href = link["href"]
|
||||
if ".pdf" in href:
|
||||
if href.startswith("http"):
|
||||
return href
|
||||
return urljoin(page_url, href)
|
||||
|
||||
return None
|
||||
@ -9,6 +9,7 @@ PAPER_SEARCH_URL = (
|
||||
"https://api.semanticscholar.org/graph/v1/paper/search/match?query={}"
|
||||
)
|
||||
PAPER_DETAIL_URL = "https://api.semanticscholar.org/graph/v1/paper/{}?fields=title,authors,url,year,abstract,externalIds,citationCount,referenceCount,journal,fieldsOfStudy,publicationDate,openAccessPdf"
|
||||
PAPER_DOI_URL = "https://api.semanticscholar.org/graph/v1/paper/DOI:{}?fields=title,authors,url,year,abstract,externalIds,citationCount,referenceCount,journal,fieldsOfStudy,publicationDate,openAccessPdf"
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -39,6 +40,18 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
if not result:
|
||||
return paper_dict
|
||||
|
||||
paper_dict.update(_parse_semantic_result(result))
|
||||
paper_dict.setdefault("title", title)
|
||||
if paper_dict.get("publish_date"):
|
||||
paper_dict["publish_date"] = datetime.strptime(
|
||||
paper_dict["publish_date"], "%Y-%m-%d"
|
||||
)
|
||||
|
||||
return paper_dict
|
||||
|
||||
|
||||
def _parse_semantic_result(result: dict) -> dict:
|
||||
paper_dict = {}
|
||||
page_str = result.get("journal", {}).get("pages")
|
||||
if page_str:
|
||||
try:
|
||||
@ -55,12 +68,10 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
paper_dict["corpus_id"] = result.get("externalIds", {}).get("CorpusId")
|
||||
paper_dict["semantic_title"] = result.get("title")
|
||||
paper_dict["first_publish_year"] = result.get("year")
|
||||
paper_dict["publish_date"] = datetime.strptime(
|
||||
result.get("publicationDate", "1950-01-01"), "%Y-%m-%d"
|
||||
)
|
||||
paper_dict["publish_date"] = result.get("publicationDate")
|
||||
paper_dict["abstract"] = result.get("abstract")
|
||||
paper_dict["tldr"] = result.get("bib", {}).get("abstract")
|
||||
paper_dict["journal"] = result.get("journal", {}).get("name")
|
||||
paper_dict["journal_name"] = result.get("journal", {}).get("name")
|
||||
paper_dict["journal_volume"] = result.get("journal", {}).get("volume")
|
||||
paper_dict["openaccess_pdf_url"] = result.get("openAccessPdf", {}).get("url")
|
||||
paper_dict["base_run_time_seconds"] = paper_dict.get("pages", 10) * getattr(
|
||||
@ -68,5 +79,19 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
)
|
||||
paper_dict["author_dicts"] = result.get("authors")
|
||||
paper_dict["genres"] = result.get("fieldsOfStudy")
|
||||
|
||||
return paper_dict
|
||||
|
||||
|
||||
def lookup_paper_from_semantic_by_doi(doi: str) -> dict:
|
||||
response = get_api_result(PAPER_DOI_URL.format(doi))
|
||||
if not response:
|
||||
return {"doi_id": doi}
|
||||
|
||||
result = json.loads(response.content)
|
||||
if not result:
|
||||
return {"doi_id": doi}
|
||||
|
||||
paper_dict = _parse_semantic_result(result)
|
||||
if not paper_dict.get("title"):
|
||||
paper_dict["title"] = result.get("title", f"Paper {doi}")
|
||||
return paper_dict
|
||||
|
||||
@ -32,8 +32,6 @@ class KoReaderBookRows:
|
||||
DEFAULT_STR = "N/A"
|
||||
DEFAULT_INT = 0
|
||||
DEFAULT_TIME = 1703800469
|
||||
BOOK_ROWS = []
|
||||
PAGE_STATS_ROWS = []
|
||||
|
||||
def _gen_random_row(self, i):
|
||||
wiggle = random.randrange(15)
|
||||
@ -110,6 +108,8 @@ class KoReaderBookRows:
|
||||
end_session = True
|
||||
|
||||
def __init__(self, book_count=0, **kwargs):
|
||||
self.BOOK_ROWS = []
|
||||
self.PAGE_STATS_ROWS = []
|
||||
self._generate_random_book_rows(book_count)
|
||||
self._generate_custom_book_row(**kwargs)
|
||||
self._generate_random_page_stats_rows()
|
||||
|
||||
@ -44,7 +44,10 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
|
||||
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
|
||||
|
||||
scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
|
||||
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
|
||||
# The test data generator adds the session-gap 3600s AFTER the trigger page
|
||||
# (not before), so the first session includes 21 pages (1-21), and each
|
||||
# subsequent session has 20 until the last. The last page is now included
|
||||
# in the final scrobble instead of being orphaned.
|
||||
expected_scrobbles = 6 * len(book_map.keys())
|
||||
assert len(scrobbles) == expected_scrobbles
|
||||
assert len(scrobbles[0].logdata.page_data.keys()) == 21
|
||||
@ -52,7 +55,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
|
||||
assert len(scrobbles[2].logdata.page_data.keys()) == 20
|
||||
assert len(scrobbles[3].logdata.page_data.keys()) == 20
|
||||
assert len(scrobbles[4].logdata.page_data.keys()) == 20
|
||||
assert len(scrobbles[5].logdata.page_data.keys()) == 18
|
||||
assert len(scrobbles[5].logdata.page_data.keys()) == 19
|
||||
|
||||
|
||||
def test_get_author_str_from_row():
|
||||
|
||||
@ -16,4 +16,15 @@ urlpatterns = [
|
||||
views.AuthorDetailView.as_view(),
|
||||
name="author_detail",
|
||||
),
|
||||
path("papers/", views.PaperListView.as_view(), name="paper_list"),
|
||||
path(
|
||||
"papers/<slug:slug>/",
|
||||
views.PaperDetailView.as_view(),
|
||||
name="paper_detail",
|
||||
),
|
||||
path(
|
||||
"papers/<slug:slug>/upload_pdf/",
|
||||
views.PaperUploadPdfView.as_view(),
|
||||
name="paper_upload_pdf",
|
||||
),
|
||||
]
|
||||
|
||||
@ -1,5 +1,8 @@
|
||||
from django.http import HttpResponseRedirect
|
||||
from django.urls import reverse
|
||||
from django.views import View
|
||||
from django.views import generic
|
||||
from books.models import Book, Author
|
||||
from books.models import Book, Author, Paper
|
||||
|
||||
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
|
||||
|
||||
@ -15,3 +18,24 @@ class BookDetailView(ScrobbleableDetailView):
|
||||
class AuthorDetailView(generic.DetailView):
|
||||
model = Author
|
||||
slug_field = "uuid"
|
||||
|
||||
|
||||
class PaperListView(ScrobbleableListView):
|
||||
model = Paper
|
||||
|
||||
|
||||
class PaperDetailView(ScrobbleableDetailView):
|
||||
model = Paper
|
||||
|
||||
|
||||
class PaperUploadPdfView(View):
|
||||
def post(self, request, slug):
|
||||
paper = Paper.objects.filter(uuid=slug).first()
|
||||
if not paper or not request.user.is_authenticated:
|
||||
return HttpResponseRedirect(reverse("books:paper_detail", args=[slug]))
|
||||
|
||||
pdf_file = request.FILES.get("pdf_file")
|
||||
if pdf_file:
|
||||
paper.pdf_file.save(pdf_file.name, pdf_file)
|
||||
|
||||
return HttpResponseRedirect(reverse("books:paper_detail", args=[slug]))
|
||||
|
||||
@ -0,0 +1,146 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-12 16:09
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("charts", "0001_initial"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "album", "rank"],
|
||||
name="charts_char_user_id_1adcde_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "track", "rank"],
|
||||
name="charts_char_user_id_d18aab_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "video", "rank"],
|
||||
name="charts_char_user_id_de9f0a_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "board_game", "rank"],
|
||||
name="charts_char_user_id_d5d58f_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "book", "rank"],
|
||||
name="charts_char_user_id_e877cf_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "food", "rank"],
|
||||
name="charts_char_user_id_a0ad71_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "podcast", "rank"],
|
||||
name="charts_char_user_id_846b80_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "trail", "rank"],
|
||||
name="charts_char_user_id_54feba_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "album", "rank"],
|
||||
name="charts_char_user_id_a3dc49_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "track", "rank"],
|
||||
name="charts_char_user_id_4b01ab_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "video", "rank"],
|
||||
name="charts_char_user_id_2ac9d2_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "board_game", "rank"],
|
||||
name="charts_char_user_id_ba968a_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "book", "rank"],
|
||||
name="charts_char_user_id_e66751_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "food", "rank"],
|
||||
name="charts_char_user_id_d23f06_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "podcast", "rank"],
|
||||
name="charts_char_user_id_be8122_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "week", "trail", "rank"],
|
||||
name="charts_char_user_id_b94ea9_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "day", "artist", "rank"],
|
||||
name="charts_char_user_id_406e0e_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "day", "album", "rank"],
|
||||
name="charts_char_user_id_322b0d_idx",
|
||||
),
|
||||
),
|
||||
migrations.AddIndex(
|
||||
model_name="chartrecord",
|
||||
index=models.Index(
|
||||
fields=["user", "year", "month", "day", "tv_series", "rank"],
|
||||
name="charts_char_user_id_aa44b7_idx",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,109 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-19 16:25
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("charts", "0002_chartrecord_charts_char_user_id_1adcde_idx_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("artist__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "artist"),
|
||||
name="unique_chart_artist_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("album__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "album"),
|
||||
name="unique_chart_album_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("track__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "track"),
|
||||
name="unique_chart_track_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("tv_series__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "tv_series"),
|
||||
name="unique_chart_tv_series_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("video__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "video"),
|
||||
name="unique_chart_video_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("podcast__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "podcast"),
|
||||
name="unique_chart_podcast_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("podcast_episode__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "podcast_episode"),
|
||||
name="unique_chart_podcast_episode_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("board_game__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "board_game"),
|
||||
name="unique_chart_board_game_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("trail__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "trail"),
|
||||
name="unique_chart_trail_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("geo_location__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "geo_location"),
|
||||
name="unique_chart_geo_location_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("food__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "food"),
|
||||
name="unique_chart_food_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("book__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "book"),
|
||||
name="unique_chart_book_period",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -2,6 +2,7 @@ import calendar
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.db import models
|
||||
from django.db.models import Q
|
||||
from django.urls import reverse
|
||||
from django_extensions.db.models import TimeStampedModel
|
||||
|
||||
@ -60,10 +61,91 @@ class ChartRecord(TimeStampedModel):
|
||||
models.Index(fields=["user", "year", "geo_location", "rank"]),
|
||||
models.Index(fields=["user", "year", "food", "rank"]),
|
||||
models.Index(fields=["user", "year", "book", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "artist", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "artist", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "album", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "track", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "tv_series", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "video", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "board_game", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "book", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "food", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "podcast", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "trail", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "artist", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "album", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "track", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "tv_series", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "video", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "board_game", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "book", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "food", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "podcast", "rank"]),
|
||||
models.Index(fields=["user", "year", "week", "trail", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "day", "artist", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "day", "album", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "day", "tv_series", "rank"]),
|
||||
]
|
||||
constraints = [
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "artist"],
|
||||
condition=Q(artist__isnull=False),
|
||||
name="unique_chart_artist_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "album"],
|
||||
condition=Q(album__isnull=False),
|
||||
name="unique_chart_album_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "track"],
|
||||
condition=Q(track__isnull=False),
|
||||
name="unique_chart_track_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "tv_series"],
|
||||
condition=Q(tv_series__isnull=False),
|
||||
name="unique_chart_tv_series_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "video"],
|
||||
condition=Q(video__isnull=False),
|
||||
name="unique_chart_video_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "podcast"],
|
||||
condition=Q(podcast__isnull=False),
|
||||
name="unique_chart_podcast_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "podcast_episode"],
|
||||
condition=Q(podcast_episode__isnull=False),
|
||||
name="unique_chart_podcast_episode_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "board_game"],
|
||||
condition=Q(board_game__isnull=False),
|
||||
name="unique_chart_board_game_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "trail"],
|
||||
condition=Q(trail__isnull=False),
|
||||
name="unique_chart_trail_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "geo_location"],
|
||||
condition=Q(geo_location__isnull=False),
|
||||
name="unique_chart_geo_location_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "food"],
|
||||
condition=Q(food__isnull=False),
|
||||
name="unique_chart_food_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "book"],
|
||||
condition=Q(book__isnull=False),
|
||||
name="unique_chart_book_period",
|
||||
),
|
||||
]
|
||||
|
||||
@property
|
||||
|
||||
@ -7,7 +7,10 @@ register = template.Library()
|
||||
def get_item(dictionary, key):
|
||||
if isinstance(dictionary, dict):
|
||||
return dictionary.get(key)
|
||||
return None
|
||||
try:
|
||||
return dictionary[int(key)]
|
||||
except (IndexError, KeyError, TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
@register.filter
|
||||
|
||||
@ -6,6 +6,7 @@ from typing import Optional
|
||||
import pytz
|
||||
from django.apps import apps
|
||||
from django.conf import settings
|
||||
from django.db import transaction
|
||||
from django.db.models import Count, Q
|
||||
from django.utils import timezone
|
||||
|
||||
@ -186,60 +187,64 @@ def build_charts(
|
||||
ranks = {count: rank for rank, count in enumerate(unique_counts, start=1)}
|
||||
|
||||
media_field = f"{media_type}_id"
|
||||
records_to_create = []
|
||||
records_to_update = []
|
||||
|
||||
existing = ChartRecord.objects.filter(
|
||||
period_filter, user=user, **{media_field + "__isnull": False}
|
||||
)
|
||||
existing_by_media_id = {getattr(r, media_field): r for r in existing}
|
||||
found_media_ids = set()
|
||||
with transaction.atomic():
|
||||
records_to_create = []
|
||||
records_to_update = []
|
||||
|
||||
for result in results:
|
||||
media_id = result[config["values"]]
|
||||
if media_id is None:
|
||||
continue
|
||||
|
||||
found_media_ids.add(media_id)
|
||||
|
||||
chart_record_data = {
|
||||
"user_id": user.id,
|
||||
"year": year,
|
||||
"month": month,
|
||||
"week": week,
|
||||
"day": day,
|
||||
"rank": ranks[result["scrobble_count"]],
|
||||
"count": result["scrobble_count"],
|
||||
}
|
||||
chart_record_data[media_field] = media_id
|
||||
|
||||
if media_id in existing_by_media_id:
|
||||
existing_record = existing_by_media_id[media_id]
|
||||
existing_record.rank = chart_record_data["rank"]
|
||||
existing_record.count = chart_record_data["count"]
|
||||
records_to_update.append(existing_record)
|
||||
else:
|
||||
records_to_create.append(ChartRecord(**chart_record_data))
|
||||
|
||||
ids_to_delete = [
|
||||
r.id for r in existing if getattr(r, media_field) not in found_media_ids
|
||||
]
|
||||
if ids_to_delete:
|
||||
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
|
||||
|
||||
if records_to_update:
|
||||
ChartRecord.objects.bulk_update(
|
||||
records_to_update, ["rank", "count"], batch_size=500
|
||||
existing = ChartRecord.objects.select_for_update().filter(
|
||||
period_filter, user=user, **{media_field + "__isnull": False}
|
||||
)
|
||||
existing_by_media_id = {getattr(r, media_field): r for r in existing}
|
||||
found_media_ids = set()
|
||||
|
||||
if records_to_create:
|
||||
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
|
||||
for result in results:
|
||||
media_id = result[config["values"]]
|
||||
if media_id is None:
|
||||
continue
|
||||
|
||||
logger.info(
|
||||
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
|
||||
f"chart records for {media_type}, period "
|
||||
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
|
||||
)
|
||||
found_media_ids.add(media_id)
|
||||
|
||||
chart_record_data = {
|
||||
"user_id": user.id,
|
||||
"year": year,
|
||||
"month": month,
|
||||
"week": week,
|
||||
"day": day,
|
||||
"rank": ranks[result["scrobble_count"]],
|
||||
"count": result["scrobble_count"],
|
||||
}
|
||||
chart_record_data[media_field] = media_id
|
||||
|
||||
if media_id in existing_by_media_id:
|
||||
existing_record = existing_by_media_id[media_id]
|
||||
existing_record.rank = chart_record_data["rank"]
|
||||
existing_record.count = chart_record_data["count"]
|
||||
records_to_update.append(existing_record)
|
||||
else:
|
||||
records_to_create.append(ChartRecord(**chart_record_data))
|
||||
|
||||
ids_to_delete = [
|
||||
r.id
|
||||
for r in existing
|
||||
if getattr(r, media_field) not in found_media_ids
|
||||
]
|
||||
if ids_to_delete:
|
||||
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
|
||||
|
||||
if records_to_update:
|
||||
ChartRecord.objects.bulk_update(
|
||||
records_to_update, ["rank", "count"], batch_size=500
|
||||
)
|
||||
|
||||
if records_to_create:
|
||||
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
|
||||
|
||||
logger.info(
|
||||
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
|
||||
f"chart records for {media_type}, period "
|
||||
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
|
||||
)
|
||||
|
||||
|
||||
def build_yesterdays_charts(user, media_types: Optional[list] = None) -> None:
|
||||
|
||||
@ -114,108 +114,106 @@ class ChartRecordView(TemplateView):
|
||||
context["current_week"] = current_week
|
||||
context["current_day"] = current_day
|
||||
|
||||
if chart_type == "maloja":
|
||||
context["chart_keys"] = {
|
||||
"today": "Today",
|
||||
"week": "This Week",
|
||||
"month": "This Month",
|
||||
"year": "This Year",
|
||||
"all": "All Time",
|
||||
}
|
||||
context["chart_keys"] = {
|
||||
"today": "Today",
|
||||
"week": "This Week",
|
||||
"month": "This Month",
|
||||
"year": "This Year",
|
||||
"all": "All Time",
|
||||
}
|
||||
|
||||
context["maloja_charts"] = {
|
||||
"artist": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
week=current_week,
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "artist", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "artist")),
|
||||
},
|
||||
"track": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"track",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user, "track", year=current_year, week=current_week
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"track",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "track", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "track")),
|
||||
},
|
||||
"tv_series": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
week=current_week,
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "tv_series", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "tv_series")),
|
||||
},
|
||||
}
|
||||
return context
|
||||
context["maloja_charts"] = {
|
||||
"artist": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
week=current_week,
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"artist",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "artist", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "artist")),
|
||||
},
|
||||
"album": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"album",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user, "album", year=current_year, week=current_week
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"album",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "album", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "album")),
|
||||
},
|
||||
"tv_series": {
|
||||
"today": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
day=current_day,
|
||||
)
|
||||
),
|
||||
"week": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
week=current_week,
|
||||
)
|
||||
),
|
||||
"month": list(
|
||||
self.get_charts_for_period(
|
||||
user,
|
||||
"tv_series",
|
||||
year=current_year,
|
||||
month=current_month,
|
||||
)
|
||||
),
|
||||
"year": list(
|
||||
self.get_charts_for_period(user, "tv_series", year=current_year)
|
||||
),
|
||||
"all": list(self.get_charts_for_period(user, "tv_series")),
|
||||
},
|
||||
}
|
||||
|
||||
if not date_param:
|
||||
context["period"] = "current"
|
||||
@ -440,12 +438,14 @@ class ChartRecordView(TemplateView):
|
||||
return context
|
||||
|
||||
def get_available_years(self, user):
|
||||
return list(
|
||||
ChartRecord.objects.filter(user=user)
|
||||
.values_list("year", flat=True)
|
||||
.distinct()
|
||||
.order_by("-year")
|
||||
)
|
||||
if not hasattr(self, "_available_years"):
|
||||
self._available_years = list(
|
||||
ChartRecord.objects.filter(user=user)
|
||||
.values_list("year", flat=True)
|
||||
.distinct()
|
||||
.order_by("-year")
|
||||
)
|
||||
return self._available_years
|
||||
|
||||
def get_period_type(self):
|
||||
date_param = self.request.GET.get("date")
|
||||
|
||||
0
vrobbler/apps/discgolf/__init__.py
Normal file
0
vrobbler/apps/discgolf/__init__.py
Normal file
14
vrobbler/apps/discgolf/admin.py
Normal file
14
vrobbler/apps/discgolf/admin.py
Normal file
@ -0,0 +1,14 @@
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from django.contrib import admin
|
||||
from scrobbles.admin import ScrobbleInline
|
||||
|
||||
|
||||
@admin.register(DiscGolfCourse)
|
||||
class DiscGolfCourseAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = ("title", "layout_name", "number_of_holes", "par_total", "pdga_slug", "udisc_id")
|
||||
raw_id_fields = ("trail",)
|
||||
search_fields = ("title", "layout_name")
|
||||
inlines = [
|
||||
ScrobbleInline,
|
||||
]
|
||||
0
vrobbler/apps/discgolf/api/__init__.py
Normal file
0
vrobbler/apps/discgolf/api/__init__.py
Normal file
14
vrobbler/apps/discgolf/api/serializers.py
Normal file
14
vrobbler/apps/discgolf/api/serializers.py
Normal file
@ -0,0 +1,14 @@
|
||||
from discgolf import models
|
||||
from rest_framework import serializers
|
||||
|
||||
|
||||
class DiscGolfCourseSerializer(serializers.HyperlinkedModelSerializer):
|
||||
pdga_link = serializers.ReadOnlyField()
|
||||
udisc_link = serializers.ReadOnlyField()
|
||||
|
||||
class Meta:
|
||||
model = models.DiscGolfCourse
|
||||
fields = "__all__"
|
||||
|
||||
|
||||
|
||||
13
vrobbler/apps/discgolf/api/views.py
Normal file
13
vrobbler/apps/discgolf/api/views.py
Normal file
@ -0,0 +1,13 @@
|
||||
from rest_framework import permissions, viewsets
|
||||
|
||||
from discgolf.api import serializers
|
||||
from discgolf import models
|
||||
|
||||
|
||||
class DiscGolfCourseViewSet(viewsets.ModelViewSet):
|
||||
queryset = models.DiscGolfCourse.objects.all().order_by("-created")
|
||||
serializer_class = serializers.DiscGolfCourseSerializer
|
||||
permission_classes = [permissions.IsAuthenticated]
|
||||
|
||||
|
||||
|
||||
6
vrobbler/apps/discgolf/apps.py
Normal file
6
vrobbler/apps/discgolf/apps.py
Normal file
@ -0,0 +1,6 @@
|
||||
from django.apps import AppConfig
|
||||
|
||||
|
||||
class DiscgolfConfig(AppConfig):
|
||||
default_auto_field = "django.db.models.BigAutoField"
|
||||
name = "discgolf"
|
||||
0
vrobbler/apps/discgolf/management/__init__.py
Normal file
0
vrobbler/apps/discgolf/management/__init__.py
Normal file
83
vrobbler/apps/discgolf/migrations/0001_initial.py
Normal file
83
vrobbler/apps/discgolf/migrations/0001_initial.py
Normal file
@ -0,0 +1,83 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:19
|
||||
|
||||
from django.db import migrations, models
|
||||
import django_extensions.db.fields
|
||||
import taggit.managers
|
||||
import uuid
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
initial = True
|
||||
|
||||
dependencies = [
|
||||
("taggit", "0004_alter_taggeditem_content_type_alter_taggeditem_tag"),
|
||||
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="DiscGolfCourse",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
(
|
||||
"uuid",
|
||||
models.UUIDField(
|
||||
blank=True, default=uuid.uuid4, editable=False, null=True
|
||||
),
|
||||
),
|
||||
("title", models.CharField(blank=True, max_length=255, null=True)),
|
||||
("base_run_time_seconds", models.IntegerField(blank=True, null=True)),
|
||||
("description", models.TextField(blank=True, null=True)),
|
||||
(
|
||||
"layout_name",
|
||||
models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
("number_of_holes", models.IntegerField(blank=True, null=True)),
|
||||
("par_total", models.IntegerField(blank=True, null=True)),
|
||||
(
|
||||
"genre",
|
||||
taggit.managers.TaggableManager(
|
||||
blank=True,
|
||||
help_text="A comma-separated list of tags.",
|
||||
through="scrobbles.ObjectWithGenres",
|
||||
to="scrobbles.Genre",
|
||||
verbose_name="Genre",
|
||||
),
|
||||
),
|
||||
(
|
||||
"tags",
|
||||
taggit.managers.TaggableManager(
|
||||
blank=True,
|
||||
help_text="A comma-separated list of tags.",
|
||||
through="taggit.TaggedItem",
|
||||
to="taggit.Tag",
|
||||
verbose_name="Tags",
|
||||
),
|
||||
),
|
||||
],
|
||||
options={
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,25 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:29
|
||||
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("trails", "0009_trail_route_waypoint"),
|
||||
("discgolf", "0001_initial"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="trail",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="trails.trail",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:35
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0002_discgolfcourse_trail"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="par_per_hole",
|
||||
field=models.JSONField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,23 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-21 04:16
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0003_discgolfcourse_par_per_hole"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="pdga_slug",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="udisc_id",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
]
|
||||
0
vrobbler/apps/discgolf/migrations/__init__.py
Normal file
0
vrobbler/apps/discgolf/migrations/__init__.py
Normal file
90
vrobbler/apps/discgolf/models.py
Normal file
90
vrobbler/apps/discgolf/models.py
Normal file
@ -0,0 +1,90 @@
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, TypedDict
|
||||
|
||||
from django.apps import apps
|
||||
from django.db import models
|
||||
from django.urls import reverse
|
||||
from scrobbles.dataclasses import BaseLogData, WithPeopleLogData
|
||||
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
|
||||
|
||||
BNULL = {"blank": True, "null": True}
|
||||
|
||||
|
||||
class DiscGolfSingleScores(TypedDict, total=False):
|
||||
person_id: int
|
||||
total: int
|
||||
|
||||
|
||||
class DiscGolfTeamScores(TypedDict, total=False):
|
||||
person_ids: list[int]
|
||||
total: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiscGolfLogData(BaseLogData, WithPeopleLogData):
|
||||
scores: Optional[dict[str, DiscGolfSingleScores | DiscGolfTeamScores]] = None
|
||||
weather: Optional[str] = None
|
||||
fun_factor: Optional[str] = None
|
||||
course_name: Optional[str] = None
|
||||
par: Optional[int] = None
|
||||
round_type: Optional[str] = None
|
||||
|
||||
|
||||
class DiscGolfCourse(ScrobblableMixin):
|
||||
description = models.TextField(**BNULL)
|
||||
layout_name = models.CharField(max_length=255, **BNULL)
|
||||
number_of_holes = models.IntegerField(**BNULL)
|
||||
par_total = models.IntegerField(**BNULL)
|
||||
par_per_hole = models.JSONField(**BNULL)
|
||||
trail = models.ForeignKey(
|
||||
"trails.Trail", on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
pdga_slug = models.CharField(max_length=255, **BNULL)
|
||||
udisc_id = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
@property
|
||||
def pdga_link(self) -> str:
|
||||
if self.pdga_slug:
|
||||
return f"https://www.pdga.com/course-directory/course/{self.pdga_slug}/"
|
||||
return ""
|
||||
|
||||
@property
|
||||
def udisc_link(self) -> str:
|
||||
if self.udisc_id:
|
||||
return f"https://udisc.com/courses/{self.udisc_id}/"
|
||||
return ""
|
||||
|
||||
def get_absolute_url(self) -> str:
|
||||
return reverse("discgolf:course_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.title} ({self.layout_name or 'Default'})"
|
||||
|
||||
@property
|
||||
def logdata_cls(self):
|
||||
return DiscGolfLogData
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
return self.layout_name or ""
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
return ScrobblableConstants(verb="Playing", tags="golf")
|
||||
|
||||
@property
|
||||
def primary_image_url(self) -> str:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def find_or_create(cls, name: str, **defaults) -> "DiscGolfCourse":
|
||||
course = cls.objects.filter(title=name).first()
|
||||
if not course:
|
||||
course = cls.objects.create(title=name, **defaults)
|
||||
return course
|
||||
|
||||
def scrobbles(self, user_id):
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
return Scrobble.objects.filter(user_id=user_id, disc_golf_course=self).order_by(
|
||||
"-timestamp"
|
||||
)
|
||||
14
vrobbler/apps/discgolf/urls.py
Normal file
14
vrobbler/apps/discgolf/urls.py
Normal file
@ -0,0 +1,14 @@
|
||||
from django.urls import path
|
||||
from discgolf import views
|
||||
|
||||
app_name = "discgolf"
|
||||
|
||||
|
||||
urlpatterns = [
|
||||
path("disc-golf/", views.DiscGolfCourseListView.as_view(), name="course_list"),
|
||||
path(
|
||||
"disc-golf/<slug:slug>/",
|
||||
views.DiscGolfCourseDetailView.as_view(),
|
||||
name="course_detail",
|
||||
),
|
||||
]
|
||||
129
vrobbler/apps/discgolf/utils.py
Normal file
129
vrobbler/apps/discgolf/utils.py
Normal file
@ -0,0 +1,129 @@
|
||||
import csv
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
from dateutil.parser import parse as parse_datetime
|
||||
from django.utils import timezone
|
||||
from people.models import Person
|
||||
from scrobbles.models import Scrobble
|
||||
from scrobbles.notifications import ScrobbleNtfyNotification
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _parse_udisc_datetime(raw: str) -> datetime:
|
||||
return parse_datetime(raw)
|
||||
|
||||
|
||||
def _resolve_player(name: str, user_id: int) -> Person:
|
||||
name = name.strip()
|
||||
existing = Person.objects.filter(name=name, created_by_id=user_id).first()
|
||||
if existing:
|
||||
return existing
|
||||
return Person.objects.create(name=name, created_by_id=user_id)
|
||||
|
||||
|
||||
def import_udisc_csv(
|
||||
file_path: str, user_id: int, record_error=None
|
||||
) -> list[Scrobble]:
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
new_scrobbles = []
|
||||
|
||||
with open(file_path, newline="", encoding="utf-8-sig") as f:
|
||||
reader = csv.DictReader(f)
|
||||
rows = list(reader)
|
||||
|
||||
if not rows:
|
||||
return []
|
||||
|
||||
par_row = None
|
||||
player_rows = []
|
||||
for row in rows:
|
||||
name = row.get("PlayerName", "").strip()
|
||||
if name.lower() == "par":
|
||||
par_row = row
|
||||
else:
|
||||
player_rows.append(row)
|
||||
|
||||
if not par_row:
|
||||
return []
|
||||
|
||||
course_name = par_row.get("CourseName", "").strip()
|
||||
layout_name = par_row.get("LayoutName", "").strip()
|
||||
start_date_raw = par_row.get("StartDate", "").strip()
|
||||
start_dt = _parse_udisc_datetime(start_date_raw) if start_date_raw else timezone.now()
|
||||
|
||||
number_of_holes = sum(1 for k in par_row if k.startswith("Hole") and k[4:].isdigit())
|
||||
par_total_str = par_row.get("Total", "").strip()
|
||||
par_total = int(par_total_str) if par_total_str.isdigit() else None
|
||||
|
||||
par_per_hole = {}
|
||||
for k, v in par_row.items():
|
||||
if k.startswith("Hole") and k[4:].isdigit() and v:
|
||||
hole_num = int(k[4:])
|
||||
try:
|
||||
par_per_hole[f"hole_{hole_num}"] = int(v)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
course, _ = DiscGolfCourse.objects.get_or_create(
|
||||
title=course_name,
|
||||
defaults={
|
||||
"layout_name": layout_name,
|
||||
"number_of_holes": number_of_holes,
|
||||
"par_total": par_total,
|
||||
"par_per_hole": par_per_hole or None,
|
||||
},
|
||||
)
|
||||
|
||||
is_teams = "+" in player_rows[0].get("PlayerName", "") if player_rows else False
|
||||
round_type = "Teams" if is_teams else "Singles"
|
||||
|
||||
scores = {}
|
||||
for row in player_rows:
|
||||
player_name = row.get("PlayerName", "").strip()
|
||||
hole_scores = {}
|
||||
for k, v in row.items():
|
||||
if k.startswith("Hole") and k[4:].isdigit() and v:
|
||||
hole_num = int(k[4:])
|
||||
try:
|
||||
hole_scores[f"hole_{hole_num}"] = int(v)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
total_str = row.get("Total", "").strip()
|
||||
total = int(total_str) if total_str.isdigit() else None
|
||||
if total is not None:
|
||||
hole_scores["total"] = total
|
||||
|
||||
if is_teams:
|
||||
people = player_name.split("+")
|
||||
person_ids = [_resolve_player(p.strip(), user_id).id for p in people]
|
||||
hole_scores["person_ids"] = person_ids
|
||||
else:
|
||||
person = _resolve_player(player_name, user_id)
|
||||
hole_scores["person_id"] = person.id
|
||||
|
||||
scores[player_name] = hole_scores
|
||||
|
||||
log = {
|
||||
"scores": scores,
|
||||
"course_name": course_name,
|
||||
"par": par_total,
|
||||
"round_type": round_type,
|
||||
}
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": start_dt,
|
||||
"source": "uDisc",
|
||||
"playback_position_seconds": 0,
|
||||
"log": log,
|
||||
}
|
||||
|
||||
scrobble = Scrobble.create_or_update(course, user_id, scrobble_dict)
|
||||
if scrobble:
|
||||
new_scrobbles.append(scrobble)
|
||||
ScrobbleNtfyNotification(scrobble).send()
|
||||
|
||||
return new_scrobbles
|
||||
32
vrobbler/apps/discgolf/views.py
Normal file
32
vrobbler/apps/discgolf/views.py
Normal file
@ -0,0 +1,32 @@
|
||||
from django.apps import apps
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
from scrobbles.views import (
|
||||
ScrobbleableListView,
|
||||
ScrobbleableDetailView,
|
||||
ChartContextMixin,
|
||||
)
|
||||
|
||||
|
||||
class DiscGolfCourseListView(ScrobbleableListView):
|
||||
model = DiscGolfCourse
|
||||
|
||||
|
||||
class DiscGolfCourseDetailView(ScrobbleableDetailView, ChartContextMixin):
|
||||
model = DiscGolfCourse
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
context = super().get_context_data(**kwargs)
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
context["trail_gpx_url"] = None
|
||||
latest = (
|
||||
Scrobble.objects.filter(
|
||||
trail=self.object.trail, gpx_file__isnull=False
|
||||
)
|
||||
.order_by("-timestamp")
|
||||
.first()
|
||||
)
|
||||
if latest and latest.gpx_file:
|
||||
context["trail_gpx_url"] = latest.gpx_file.url
|
||||
return context
|
||||
@ -21,6 +21,7 @@ class FoodAdmin(admin.ModelAdmin):
|
||||
"uuid",
|
||||
"title",
|
||||
)
|
||||
raw_id_fields = ("category",)
|
||||
ordering = ("-created",)
|
||||
search_fields = ("title",)
|
||||
inlines = [
|
||||
|
||||
@ -1,4 +1,7 @@
|
||||
import time
|
||||
|
||||
from django.contrib import admin
|
||||
from django.http import HttpRequest
|
||||
|
||||
from locations.models import GeoLocation
|
||||
|
||||
@ -14,9 +17,29 @@ class GeoLocationAdmin(admin.ModelAdmin):
|
||||
"lon",
|
||||
"title",
|
||||
"altitude",
|
||||
"city",
|
||||
"state_province",
|
||||
"country",
|
||||
)
|
||||
ordering = ("-created",)
|
||||
search_fields = ("title",)
|
||||
actions = ["reverse_geocode_selected"]
|
||||
inlines = [
|
||||
ScrobbleInline,
|
||||
]
|
||||
|
||||
@admin.action(description="Reverse geocode selected locations")
|
||||
def reverse_geocode_selected(self, request: HttpRequest, queryset):
|
||||
updated = 0
|
||||
errors = 0
|
||||
for i, location in enumerate(queryset.iterator()):
|
||||
if location.reverse_geocode():
|
||||
updated += 1
|
||||
else:
|
||||
errors += 1
|
||||
if i < queryset.count() - 1:
|
||||
time.sleep(1.1)
|
||||
msg = f"Reverse geocoded {updated} locations"
|
||||
if errors:
|
||||
msg += f", {errors} failed"
|
||||
self.message_user(request, msg)
|
||||
|
||||
@ -0,0 +1,38 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-21 04:26
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("locations", "0010_clean_start"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="city",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="country",
|
||||
field=models.CharField(blank=True, max_length=100, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="postal_code",
|
||||
field=models.CharField(blank=True, max_length=20, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="state_province",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="street",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -28,6 +28,14 @@ class GeoLocationLogData(BaseLogData, WithPeopleLogData):
|
||||
activity: str = ""
|
||||
detected_at: str = ""
|
||||
|
||||
@classmethod
|
||||
def from_log_dict(cls, log_dict: dict) -> dict:
|
||||
instance_data = log_dict.get("movement_detection", {}).copy()
|
||||
for field_name in ["description", "notes", "with_people_ids"]:
|
||||
if field_name in log_dict:
|
||||
instance_data[field_name] = log_dict[field_name]
|
||||
return instance_data
|
||||
|
||||
|
||||
class GeoLocation(ScrobblableMixin):
|
||||
COMPLETION_PERCENT = getattr(settings, "LOCATION_COMPLETION_PERCENT", 100)
|
||||
@ -37,6 +45,11 @@ class GeoLocation(ScrobblableMixin):
|
||||
truncated_lat = models.FloatField(**BNULL)
|
||||
truncated_lon = models.FloatField(**BNULL)
|
||||
altitude = models.FloatField(**BNULL)
|
||||
street = models.TextField(**BNULL)
|
||||
city = models.CharField(max_length=255, **BNULL)
|
||||
state_province = models.CharField(max_length=255, **BNULL)
|
||||
postal_code = models.CharField(max_length=20, **BNULL)
|
||||
country = models.CharField(max_length=100, **BNULL)
|
||||
|
||||
class Meta:
|
||||
unique_together = [["lat", "lon", "altitude"]]
|
||||
@ -47,6 +60,11 @@ class GeoLocation(ScrobblableMixin):
|
||||
|
||||
return f"{self.lat} x {self.lon}"
|
||||
|
||||
@property
|
||||
def display_address(self) -> str:
|
||||
parts = [self.street, self.city, self.state_province, self.postal_code, self.country]
|
||||
return ", ".join(p for p in parts if p)
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("locations:geolocation_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@ -113,6 +131,17 @@ class GeoLocation(ScrobblableMixin):
|
||||
|
||||
return fetch_current_weather(self.lat, self.lon)
|
||||
|
||||
def reverse_geocode(self) -> bool:
|
||||
from locations.utils import reverse_geocode
|
||||
|
||||
result = reverse_geocode(self.lat, self.lon)
|
||||
if result is None:
|
||||
return False
|
||||
for field, value in result.items():
|
||||
setattr(self, field, value)
|
||||
self.save(update_fields=list(result.keys()))
|
||||
return True
|
||||
|
||||
def loc_diff(self, old_lat_lon: tuple) -> tuple:
|
||||
return (
|
||||
abs(Decimal(old_lat_lon[0]) - Decimal(self.lat)),
|
||||
|
||||
@ -201,6 +201,50 @@ def detect_movement(
|
||||
return result
|
||||
|
||||
|
||||
NOMINATIM_URL = "https://nominatim.openstreetmap.org/reverse"
|
||||
|
||||
USER_AGENT = "Vrobbler/1.0 (https://github.com/secstate/vrobbler)"
|
||||
|
||||
|
||||
def reverse_geocode(lat: float, lon: float) -> Optional[dict]:
|
||||
"""Reverse geocode lat/lon to an address using Nominatim.
|
||||
|
||||
Returns a dict with address fields, or None on failure.
|
||||
Nominatim usage policy: max 1 request per second.
|
||||
"""
|
||||
params = {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"format": "json",
|
||||
}
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
try:
|
||||
resp = requests.get(NOMINATIM_URL, params=params, headers=headers, timeout=10)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException as e:
|
||||
logger.warning("Failed to reverse geocode %s,%s: %s", lat, lon, e)
|
||||
return None
|
||||
|
||||
data = resp.json()
|
||||
if "error" in data:
|
||||
logger.warning("Nominatim error for %s,%s: %s", lat, lon, data["error"])
|
||||
return None
|
||||
|
||||
address = data.get("address", {})
|
||||
return {
|
||||
"street": address.get("road")
|
||||
or address.get("pedestrian")
|
||||
or address.get("footway"),
|
||||
"city": address.get("city")
|
||||
or address.get("town")
|
||||
or address.get("village")
|
||||
or address.get("hamlet"),
|
||||
"state_province": address.get("state"),
|
||||
"postal_code": address.get("postcode"),
|
||||
"country": address.get("country"),
|
||||
}
|
||||
|
||||
|
||||
NWS_URL = "https://forecast.weather.gov/MapClick.php"
|
||||
|
||||
|
||||
|
||||
@ -1,8 +1,22 @@
|
||||
from django.core.cache import cache
|
||||
|
||||
from music.models import Artist, Album
|
||||
|
||||
CACHE_TTL = 300
|
||||
|
||||
|
||||
def music_lists(request):
|
||||
artist_list = cache.get("music_lists_artist_list")
|
||||
if artist_list is None:
|
||||
artist_list = list(Artist.objects.all().only("id", "name"))
|
||||
cache.set("music_lists_artist_list", artist_list, CACHE_TTL)
|
||||
|
||||
album_list = cache.get("music_lists_album_list")
|
||||
if album_list is None:
|
||||
album_list = list(Album.objects.all().only("id", "name"))
|
||||
cache.set("music_lists_album_list", album_list, CACHE_TTL)
|
||||
|
||||
return {
|
||||
"artist_list": Artist.objects.all(),
|
||||
"album_list": Album.objects.all(),
|
||||
"artist_list": artist_list,
|
||||
"album_list": album_list,
|
||||
}
|
||||
|
||||
@ -236,19 +236,6 @@ class Artist(TimeStampedModel):
|
||||
)
|
||||
artist.fix_metadata()
|
||||
|
||||
# TODO: See if this alt_names stuff actually works or causes hard to debug problems
|
||||
# If we did find our artist, but the found name is slightly differnt, record that
|
||||
# if artist and alt_name:
|
||||
# if not artist.alt_names:
|
||||
# artist.alt_names = alt_name
|
||||
# else:
|
||||
# artist.alt_names += f"\\{alt_name}"
|
||||
# logger.info(
|
||||
# f"Add alt_name {alt_name} to artist {artist}",
|
||||
# extra={"alt_name": alt_name, "artist_id": artist.id},
|
||||
# )
|
||||
# artist.save(update_fields=["alt_names"])
|
||||
|
||||
return artist
|
||||
|
||||
|
||||
@ -600,8 +587,9 @@ class Track(ScrobblableMixin):
|
||||
def __str__(self):
|
||||
return f"{self.title} by {self.artist}"
|
||||
|
||||
@property
|
||||
def logdata_cls(self):
|
||||
return TrackLogData()
|
||||
return TrackLogData
|
||||
|
||||
@property
|
||||
def primary_album(self):
|
||||
|
||||
@ -6,6 +6,7 @@ from people.models import Person, PersonScrobble
|
||||
class PersonAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = ("name", "bgg_username", "bgstats_id")
|
||||
raw_id_fields = ("user", "created_by")
|
||||
ordering = ("-created",)
|
||||
search_fields = ("name",)
|
||||
|
||||
|
||||
@ -18,6 +18,7 @@ class PodcastAdmin(admin.ModelAdmin):
|
||||
"producer",
|
||||
"active",
|
||||
)
|
||||
raw_id_fields = ("producer",)
|
||||
ordering = ("name",)
|
||||
|
||||
|
||||
@ -28,6 +29,7 @@ class PodcastEpisodeAdmin(admin.ModelAdmin):
|
||||
"title",
|
||||
"podcast",
|
||||
)
|
||||
raw_id_fields = ("podcast",)
|
||||
list_filter = ("podcast",)
|
||||
ordering = ("-created",)
|
||||
inlines = [
|
||||
|
||||
@ -6,6 +6,7 @@ from profiles.models import UserProfile
|
||||
@admin.register(UserProfile)
|
||||
class UserProfileAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
raw_id_fields = ("user",)
|
||||
ordering = ("-created",)
|
||||
readonly_fields = ("timezone_change_log",)
|
||||
exclude = (
|
||||
|
||||
@ -1,6 +1,8 @@
|
||||
from django import forms
|
||||
|
||||
from profiles.models import UserProfile
|
||||
from scrobbles.constants import Visibility
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
|
||||
class UserProfileForm(forms.ModelForm):
|
||||
@ -38,6 +40,7 @@ class UserProfileForm(forms.ModelForm):
|
||||
"enable_public_widgets",
|
||||
"widget_custom_css",
|
||||
"home_scrobble_limit",
|
||||
"live_now_playing",
|
||||
"weigh_in_units",
|
||||
]
|
||||
widgets = {
|
||||
@ -45,3 +48,51 @@ class UserProfileForm(forms.ModelForm):
|
||||
"archivebox_password": forms.PasswordInput(render_value=True),
|
||||
"webdav_pass": forms.PasswordInput(render_value=True),
|
||||
}
|
||||
|
||||
|
||||
MEDIA_TYPE_LABELS = {
|
||||
mt.value: mt.label for mt in Scrobble.MediaType
|
||||
}
|
||||
|
||||
INHERIT = ""
|
||||
|
||||
|
||||
class BulkVisibilityForm(forms.Form):
|
||||
bulk_action = forms.ChoiceField(
|
||||
choices=[
|
||||
(Visibility.PUBLIC, "Public"),
|
||||
(Visibility.PRIVATE, "Private"),
|
||||
],
|
||||
widget=forms.RadioSelect,
|
||||
required=False,
|
||||
label="Set all non-shared scrobbles to",
|
||||
)
|
||||
|
||||
def __init__(self, *args, **kwargs):
|
||||
self.profile = kwargs.pop("profile")
|
||||
super().__init__(*args, **kwargs)
|
||||
media_types = Scrobble.MediaType.values
|
||||
choices = [
|
||||
(Visibility.PUBLIC, "Public"),
|
||||
(Visibility.SHARED, "Shared"),
|
||||
(Visibility.PRIVATE, "Private"),
|
||||
]
|
||||
existing_overrides = self.profile.media_type_visibility or {}
|
||||
for mt in sorted(media_types):
|
||||
label = MEDIA_TYPE_LABELS.get(mt, mt)
|
||||
self.fields[f"media_type_{mt}"] = forms.ChoiceField(
|
||||
choices=choices,
|
||||
required=False,
|
||||
label=label,
|
||||
initial=existing_overrides.get(mt, Visibility.PRIVATE),
|
||||
)
|
||||
|
||||
def clean(self):
|
||||
cleaned = super().clean()
|
||||
overrides = {}
|
||||
for mt in Scrobble.MediaType.values:
|
||||
val = cleaned.get(f"media_type_{mt}")
|
||||
if val:
|
||||
overrides[mt] = val
|
||||
cleaned["media_type_visibility"] = overrides
|
||||
return cleaned
|
||||
|
||||
@ -0,0 +1,20 @@
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("profiles", "0037_alter_userprofile_default_scrobble_visibility"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="userprofile",
|
||||
name="media_type_visibility",
|
||||
field=models.JSONField(
|
||||
blank=True,
|
||||
default=dict,
|
||||
help_text='Per-media-type visibility overrides, e.g. {"Video": "public", "Track": "private"}',
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-22 02:44
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("profiles", "0038_userprofile_media_type_visibility"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="userprofile",
|
||||
name="live_now_playing",
|
||||
field=models.BooleanField(default=False),
|
||||
),
|
||||
]
|
||||
@ -90,8 +90,16 @@ class UserProfile(TimeStampedModel):
|
||||
default="private",
|
||||
)
|
||||
|
||||
media_type_visibility = models.JSONField(
|
||||
default=dict,
|
||||
blank=True,
|
||||
help_text="Per-media-type visibility overrides, e.g. {\"Video\": \"public\", \"Track\": \"private\"}",
|
||||
)
|
||||
|
||||
home_scrobble_limit = models.IntegerField(default=20)
|
||||
|
||||
live_now_playing = models.BooleanField(default=False)
|
||||
|
||||
weigh_in_units = models.CharField(
|
||||
max_length=16,
|
||||
choices=WeighUnit.choices,
|
||||
@ -134,6 +142,11 @@ class UserProfile(TimeStampedModel):
|
||||
return history
|
||||
|
||||
def get_timestamp_with_tz(self, timestamp):
|
||||
from django.conf import settings
|
||||
|
||||
server_tz = ZoneInfo(settings.TIME_ZONE)
|
||||
ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
|
||||
|
||||
timezone = self.tzinfo
|
||||
if self.timezone_change_log:
|
||||
change_list = self.historic_timezone_changes
|
||||
@ -144,13 +157,13 @@ class UserProfile(TimeStampedModel):
|
||||
end = None
|
||||
|
||||
if end:
|
||||
if start <= timestamp.replace(tzinfo=end.timezone) <= end:
|
||||
if start <= ref_dt <= end:
|
||||
timezone = start.timezone
|
||||
else:
|
||||
if start <= timestamp.replace(tzinfo=start.timezone):
|
||||
if start <= ref_dt:
|
||||
timezone = start.timezone
|
||||
|
||||
return timestamp.replace(tzinfo=timezone)
|
||||
return ref_dt.astimezone(timezone)
|
||||
|
||||
def adjust_timezone_of_scrobbles(self, commit=False):
|
||||
current_dt = None
|
||||
|
||||
@ -6,4 +6,9 @@ app_name = "profiles"
|
||||
|
||||
urlpatterns = [
|
||||
path("settings/", views.ProfileFormView.as_view(), name="profile_settings"),
|
||||
path(
|
||||
"settings/visibility/",
|
||||
views.BulkVisibilityView.as_view(),
|
||||
name="bulk_visibility",
|
||||
),
|
||||
]
|
||||
|
||||
@ -1,8 +1,12 @@
|
||||
from django.contrib import messages
|
||||
from django.contrib.auth.mixins import LoginRequiredMixin
|
||||
from django.db.models import Count, Q
|
||||
from django.http.response import HttpResponseBadRequest
|
||||
from django.urls import reverse_lazy
|
||||
from django.views.generic import FormView
|
||||
from profiles.forms import UserProfileForm
|
||||
from profiles.forms import BulkVisibilityForm, UserProfileForm
|
||||
from scrobbles.constants import Visibility
|
||||
from scrobbles.models import Scrobble
|
||||
from tasks.todoist import generate_todoist_oauth_url
|
||||
|
||||
|
||||
@ -30,3 +34,46 @@ class ProfileFormView(LoginRequiredMixin, FormView):
|
||||
context["profile"] = self.request.user.profile
|
||||
context["todoist_oauth_url"] = generate_todoist_oauth_url(self.request.user.id)
|
||||
return context
|
||||
|
||||
|
||||
class BulkVisibilityView(LoginRequiredMixin, FormView):
|
||||
template_name = "profiles/visibility_settings.html"
|
||||
form_class = BulkVisibilityForm
|
||||
success_url = reverse_lazy("profiles:bulk_visibility")
|
||||
|
||||
def get_form_kwargs(self):
|
||||
kwargs = super().get_form_kwargs()
|
||||
kwargs["profile"] = self.request.user.profile
|
||||
return kwargs
|
||||
|
||||
def form_valid(self, form):
|
||||
request = self.request
|
||||
profile = request.user.profile
|
||||
|
||||
bulk_action = form.cleaned_data.get("bulk_action")
|
||||
if bulk_action:
|
||||
qs = Scrobble.objects.filter(
|
||||
user=request.user,
|
||||
).exclude(visibility=Visibility.SHARED)
|
||||
total = qs.count()
|
||||
qs.update(visibility=bulk_action)
|
||||
messages.success(
|
||||
request,
|
||||
f"Updated {total} scrobble(s) to {bulk_action}.",
|
||||
)
|
||||
|
||||
profile.media_type_visibility = form.cleaned_data["media_type_visibility"]
|
||||
profile.save(update_fields=["media_type_visibility"])
|
||||
messages.success(request, "Per-media-type visibility overrides saved.")
|
||||
return super().form_valid(form)
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
ctx = super().get_context_data(**kwargs)
|
||||
profile = self.request.user.profile
|
||||
qs = Scrobble.objects.filter(user=self.request.user)
|
||||
ctx["scrobble_count"] = qs.count()
|
||||
ctx["visibility_counts"] = qs.values("visibility").annotate(
|
||||
count=Count("id")
|
||||
)
|
||||
ctx["profile"] = profile
|
||||
return ctx
|
||||
|
||||
@ -21,6 +21,7 @@ class PuzzleAdmin(admin.ModelAdmin):
|
||||
"uuid",
|
||||
"title",
|
||||
)
|
||||
raw_id_fields = ("manufacturer",)
|
||||
ordering = ("-created",)
|
||||
search_fields = ("title",)
|
||||
inlines = [
|
||||
|
||||
@ -12,6 +12,7 @@ from scrobbles.models import (
|
||||
Scrobble,
|
||||
ShareViewLog,
|
||||
TrailGPXImport,
|
||||
UDiscCSVImport,
|
||||
)
|
||||
from scrobbles.mixins import Genre
|
||||
|
||||
@ -19,35 +20,11 @@ from scrobbles.mixins import Genre
|
||||
class ScrobbleInline(admin.TabularInline):
|
||||
model = Scrobble
|
||||
extra = 0
|
||||
raw_id_fields = (
|
||||
"video",
|
||||
"channel",
|
||||
"podcast_episode",
|
||||
"track",
|
||||
"video_game",
|
||||
"book",
|
||||
"paper",
|
||||
"sport_event",
|
||||
"food",
|
||||
"board_game",
|
||||
"geo_location",
|
||||
"task",
|
||||
"puzzle",
|
||||
"mood",
|
||||
"brick_set",
|
||||
"trail",
|
||||
"beer",
|
||||
"web_page",
|
||||
"life_event",
|
||||
"birding_location",
|
||||
"user",
|
||||
)
|
||||
exclude = (
|
||||
"scrobble_log",
|
||||
"timezone",
|
||||
"videogame_save_data",
|
||||
"screenshot",
|
||||
)
|
||||
per_page = 15
|
||||
ordering = ("-timestamp",)
|
||||
show_change_link = True
|
||||
fields = ("timestamp", "media_type", "source", "in_progress")
|
||||
readonly_fields = fields
|
||||
|
||||
|
||||
class ImportBaseAdmin(admin.ModelAdmin):
|
||||
@ -97,6 +74,10 @@ class ScaleCSVImportAdmin(ImportBaseAdmin): ...
|
||||
class TrailGPXImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(UDiscCSVImport)
|
||||
class UDiscCSVImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(Genre)
|
||||
class GenreAdmin(admin.ModelAdmin):
|
||||
list_display = (
|
||||
@ -121,7 +102,9 @@ class ScrobbleAdmin(admin.ModelAdmin):
|
||||
"user",
|
||||
)
|
||||
raw_id_fields = (
|
||||
"user",
|
||||
"video",
|
||||
"channel",
|
||||
"podcast_episode",
|
||||
"track",
|
||||
"sport_event",
|
||||
@ -140,6 +123,8 @@ class ScrobbleAdmin(admin.ModelAdmin):
|
||||
"web_page",
|
||||
"life_event",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
"long_play_last_scrobble",
|
||||
)
|
||||
list_filter = (
|
||||
"is_paused",
|
||||
@ -152,6 +137,7 @@ class ScrobbleAdmin(admin.ModelAdmin):
|
||||
"user",
|
||||
)
|
||||
ordering = ("-timestamp",)
|
||||
readonly_fields = ("share_token_version", "share_view_count")
|
||||
|
||||
def media_name(self, obj):
|
||||
return obj.media_obj
|
||||
@ -178,14 +164,19 @@ class FavoriteMediaAdmin(admin.ModelAdmin):
|
||||
list_filter = ("media_type", "sent_to_mopidy", "user")
|
||||
date_hierarchy = "created"
|
||||
raw_id_fields = (
|
||||
"user",
|
||||
"video",
|
||||
"channel",
|
||||
"track",
|
||||
"podcast_episode",
|
||||
"sport_event",
|
||||
"book",
|
||||
"paper",
|
||||
"video_game",
|
||||
"board_game",
|
||||
"geo_location",
|
||||
"puzzle",
|
||||
"food",
|
||||
"task",
|
||||
"mood",
|
||||
"brick_set",
|
||||
@ -194,4 +185,5 @@ class FavoriteMediaAdmin(admin.ModelAdmin):
|
||||
"web_page",
|
||||
"life_event",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
)
|
||||
|
||||
@ -14,6 +14,7 @@ LONG_PLAY_MEDIA = {
|
||||
"books": "Book",
|
||||
"bricksets": "BrickSet",
|
||||
"tasks": "Task",
|
||||
"papers": "Paper",
|
||||
}
|
||||
|
||||
# Media types that should just be finished if they go over time
|
||||
@ -25,17 +26,26 @@ AUTO_FINISH_MEDIA = {
|
||||
}
|
||||
|
||||
PLAY_AGAIN_MEDIA = {
|
||||
"videogames": "VideoGame",
|
||||
"videos": "Video",
|
||||
"music": "Track",
|
||||
"podcasts": "PodcastEpisode",
|
||||
"sports": "SportEvent",
|
||||
"books": "Book",
|
||||
"videogames": "VideoGame",
|
||||
"boardgames": "BoardGame",
|
||||
"moods": "Mood",
|
||||
"bricksets": "BrickSet",
|
||||
"locations": "GeoLocation",
|
||||
"trails": "Trail",
|
||||
"beers": "Beer",
|
||||
"puzzles": "Puzzle",
|
||||
"foods": "Food",
|
||||
"locations": "GeoLocation",
|
||||
"videos": "Video",
|
||||
"tasks": "Task",
|
||||
"webpages": "WebPage",
|
||||
"lifeevents": "LifeEvent",
|
||||
"moods": "Mood",
|
||||
"bricksets": "BrickSet",
|
||||
"channels": "Channel",
|
||||
"birds": "BirdingLocation",
|
||||
"discgolf": "DiscGolfCourse",
|
||||
}
|
||||
|
||||
MEDIA_END_PADDING_SECONDS = {
|
||||
@ -52,6 +62,7 @@ SCROBBLE_CONTENT_URLS = {
|
||||
"-b": ["https://www.amazon.com/"],
|
||||
"-t": ["https://app.todoist.com/app/task/{id}"],
|
||||
"-p": ["https://www.ipdb.plus/IPDb/puzzle.php?id="],
|
||||
"-pp": ["https://doi.org/"],
|
||||
"-l": ["https://brickset.com/sets/"],
|
||||
"-c": ["https://readcomicsonline.ru"],
|
||||
"-h": ["https://www.twitch.tv/"],
|
||||
@ -73,6 +84,8 @@ MANUAL_SCROBBLE_FNS = {
|
||||
"-c": "manual_scrobble_book",
|
||||
"-f": "manual_scrobble_food",
|
||||
"-h": "manual_scrobble_twitch_channel",
|
||||
"-dg": "manual_scrobble_discgolf",
|
||||
"-pp": "manual_scrobble_paper",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -27,15 +27,19 @@ def month_color(request):
|
||||
|
||||
def now_playing(request):
|
||||
user = request.user
|
||||
now = timezone.now()
|
||||
if not user.is_authenticated:
|
||||
return {}
|
||||
|
||||
return {
|
||||
"now_playing_list": Scrobble.objects.filter(
|
||||
in_progress=True,
|
||||
is_paused=False,
|
||||
user=user,
|
||||
).exclude(
|
||||
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
|
||||
)
|
||||
"now_playing_list": list(
|
||||
Scrobble.objects.filter(
|
||||
in_progress=True,
|
||||
is_paused=False,
|
||||
user=user,
|
||||
)
|
||||
.exclude(
|
||||
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
|
||||
)
|
||||
.select_related("track", "video", "podcast_episode")
|
||||
),
|
||||
}
|
||||
|
||||
@ -50,6 +50,18 @@ class BaseLogData(JSONDataclass):
|
||||
def override_fields(cls) -> dict:
|
||||
return {}
|
||||
|
||||
@classmethod
|
||||
def from_log_dict(cls, log_dict: dict) -> dict:
|
||||
"""Extract LogData keyword arguments from a stored log dict.
|
||||
|
||||
Override in subclasses to handle custom nesting/structure.
|
||||
"""
|
||||
return {
|
||||
k: v
|
||||
for k, v in log_dict.items()
|
||||
if k in cls.__dataclass_fields__
|
||||
}
|
||||
|
||||
def notes_as_str(self, separator: str = " | ") -> str:
|
||||
import html
|
||||
import re
|
||||
@ -207,7 +219,7 @@ class BaseLogData(JSONDataclass):
|
||||
|
||||
@dataclass
|
||||
class LongPlayLogData(JSONDataclass):
|
||||
long_play_complete: bool = False
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
|
||||
@ -76,6 +76,7 @@ class LastFM:
|
||||
in_progress=False,
|
||||
media_type=Scrobble.MediaType.TRACK,
|
||||
timezone=tz_timestamp.tzinfo.name,
|
||||
visibility="private",
|
||||
)
|
||||
# Vrobbler scrobbles on finish, LastFM scrobbles on start
|
||||
seconds_eariler = timestamp - timedelta(seconds=20)
|
||||
@ -92,7 +93,6 @@ class LastFM:
|
||||
new_scrobbles.append(new_scrobble)
|
||||
|
||||
created = Scrobble.objects.bulk_create(new_scrobbles)
|
||||
# TODO Add a notification for users that their import is complete
|
||||
logger.info(
|
||||
f"Last.fm import fnished",
|
||||
extra={
|
||||
|
||||
@ -111,6 +111,7 @@ def import_scale_csv(file_path, user_id):
|
||||
played_to_completion=True,
|
||||
in_progress=False,
|
||||
media_type=Scrobble.MediaType.TASK,
|
||||
visibility="private",
|
||||
)
|
||||
new_scrobbles.append(new_scrobble)
|
||||
|
||||
|
||||
@ -305,6 +305,7 @@ def import_trail_gpx(file_path, user_id, original_filename=None):
|
||||
played_to_completion=True,
|
||||
in_progress=False,
|
||||
media_type=Scrobble.MediaType.TRAIL,
|
||||
visibility="private",
|
||||
)
|
||||
|
||||
_, ext = os.path.splitext(file_path)
|
||||
|
||||
@ -80,6 +80,7 @@ def import_audioscrobbler_tsv_file(file_path, user_id):
|
||||
in_progress=False,
|
||||
media_type=Scrobble.MediaType.TRACK,
|
||||
timezone=timestamp.tzinfo.name,
|
||||
visibility="private",
|
||||
)
|
||||
existing = Scrobble.objects.filter(
|
||||
timestamp=timestamp, track=track, user=user
|
||||
|
||||
@ -19,6 +19,7 @@ DEFAULT_RETROARCH_PATH = "var/retroarch/"
|
||||
DEFAULT_BGSTATS_PATH = "var/bgstats/"
|
||||
DEFAULT_EBIRD_PATH = "var/ebird/"
|
||||
DEFAULT_SCALE_PATH = "var/scale/"
|
||||
DEFAULT_UDISC_PATH = "var/udisc/"
|
||||
|
||||
|
||||
def import_from_webdav_for_all_users(
|
||||
@ -48,6 +49,7 @@ def import_from_webdav_for_all_users(
|
||||
bgstats_count = 0
|
||||
ebird_count = 0
|
||||
scale_count = 0
|
||||
udisc_count = 0
|
||||
|
||||
for user_id in webdav_enabled_user_ids:
|
||||
client = get_webdav_client(user_id)
|
||||
@ -78,15 +80,20 @@ def import_from_webdav_for_all_users(
|
||||
)
|
||||
logger.info("Scanning WebDAV scale for user %s", user_id)
|
||||
scale_count += scan_webdav_for_scale(client, user_id)
|
||||
logger.info("Scanning WebDAV udisc for user %s", user_id)
|
||||
udisc_count += scan_webdav_for_udisc(
|
||||
client, user_id, include_processed=include_processed
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale WebDAV imports",
|
||||
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale, %d uDisc WebDAV imports",
|
||||
ko_count,
|
||||
gpx_count,
|
||||
retro_count,
|
||||
bgstats_count,
|
||||
ebird_count,
|
||||
scale_count,
|
||||
udisc_count,
|
||||
extra={
|
||||
"koreader": ko_count,
|
||||
"trail_gpx": gpx_count,
|
||||
@ -94,9 +101,10 @@ def import_from_webdav_for_all_users(
|
||||
"bgstats": bgstats_count,
|
||||
"ebird": ebird_count,
|
||||
"scale": scale_count,
|
||||
"udisc": udisc_count,
|
||||
},
|
||||
)
|
||||
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count
|
||||
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count, udisc_count
|
||||
|
||||
|
||||
def scan_webdav_for_koreader(
|
||||
@ -153,7 +161,7 @@ def scan_webdav_for_koreader(
|
||||
)
|
||||
|
||||
if update_etag_only:
|
||||
if last_import and remote_etag:
|
||||
if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
|
||||
last_import.webdav_etag = remote_etag
|
||||
last_import.save(update_fields=["webdav_etag"])
|
||||
logger.info(
|
||||
@ -163,7 +171,7 @@ def scan_webdav_for_koreader(
|
||||
)
|
||||
return 0
|
||||
|
||||
if last_import and last_import.webdav_etag and remote_etag:
|
||||
if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
|
||||
if last_import.webdav_etag == remote_etag:
|
||||
logger.info(
|
||||
"koreader stats file unchanged (ETag match)",
|
||||
@ -811,3 +819,115 @@ def scan_webdav_for_scale(webdav_client, user_id):
|
||||
os.unlink(tmp.name)
|
||||
|
||||
return new_imports
|
||||
|
||||
|
||||
def scan_webdav_for_udisc(webdav_client, user_id, include_processed=False):
|
||||
"""Download .csv files from WebDAV var/udisc/ and queue imports for new files.
|
||||
|
||||
After importing, files are moved to var/udisc/processed/ so they are
|
||||
not re-imported on subsequent scans unless *include_processed* is True.
|
||||
"""
|
||||
from scrobbles.models import UDiscCSVImport
|
||||
from scrobbles.tasks import process_udisc_csv_import
|
||||
|
||||
udisc_path = DEFAULT_UDISC_PATH
|
||||
try:
|
||||
webdav_client.info(udisc_path)
|
||||
except:
|
||||
logger.info("No var/udisc/ directory on webdav", extra={"user_id": user_id})
|
||||
return 0
|
||||
|
||||
try:
|
||||
files = webdav_client.list(udisc_path)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Could not list var/udisc/",
|
||||
extra={"user_id": user_id, "error": str(e)},
|
||||
)
|
||||
return 0
|
||||
|
||||
processed_dir = f"{udisc_path}processed/"
|
||||
try:
|
||||
webdav_client.mkdir(processed_dir, recursive=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
new_imports = 0
|
||||
already_imported = set(
|
||||
UDiscCSVImport.objects.filter(user_id=user_id).values_list(
|
||||
"original_filename", flat=True
|
||||
)
|
||||
)
|
||||
|
||||
for fname in files:
|
||||
fname = os.path.basename(fname)
|
||||
if not fname.lower().endswith(".csv"):
|
||||
continue
|
||||
if fname == "processed":
|
||||
continue
|
||||
if fname in already_imported:
|
||||
logger.debug(f"Skipping already-imported {fname}")
|
||||
continue
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
|
||||
try:
|
||||
webdav_client.download_sync(
|
||||
remote_path=f"{udisc_path}{fname}", local_path=tmp.name
|
||||
)
|
||||
imp = UDiscCSVImport.objects.create(
|
||||
user_id=user_id,
|
||||
original_filename=fname,
|
||||
)
|
||||
with open(tmp.name, "rb") as f:
|
||||
imp.csv_file.save(fname, f, save=True)
|
||||
|
||||
stem, ext = os.path.splitext(fname)
|
||||
ts = datetime.now().strftime("%Y%m%dT%H%M%S")
|
||||
webdav_client.move(
|
||||
f"{udisc_path}{fname}",
|
||||
f"{processed_dir}{stem}_{ts}{ext}",
|
||||
)
|
||||
|
||||
process_udisc_csv_import.delay(imp.id)
|
||||
new_imports += 1
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to import uDisc CSV file {fname}: {e}")
|
||||
finally:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
if include_processed:
|
||||
try:
|
||||
processed_files = webdav_client.list(processed_dir)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Could not list var/udisc/processed/",
|
||||
extra={"user_id": user_id, "error": str(e)},
|
||||
)
|
||||
return new_imports
|
||||
|
||||
for fname in processed_files:
|
||||
fname = os.path.basename(fname)
|
||||
if not fname.lower().endswith(".csv"):
|
||||
continue
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
|
||||
try:
|
||||
webdav_client.download_sync(
|
||||
remote_path=f"{processed_dir}{fname}", local_path=tmp.name
|
||||
)
|
||||
imp = UDiscCSVImport.objects.create(
|
||||
user_id=user_id,
|
||||
original_filename=fname,
|
||||
)
|
||||
with open(tmp.name, "rb") as f:
|
||||
imp.csv_file.save(fname, f, save=True)
|
||||
process_udisc_csv_import.delay(imp.id)
|
||||
new_imports += 1
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import processed uDisc CSV file {fname}: {e}"
|
||||
)
|
||||
finally:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
return new_imports
|
||||
|
||||
@ -0,0 +1,174 @@
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.db import connection
|
||||
|
||||
from scrobbles.constants import LONG_PLAY_MEDIA
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
BATCH_SIZE = 1000
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
"Backfill long_play_last_scrobble FK chains, then recompute "
|
||||
"long_play_seconds by walking forward through scrobbles in "
|
||||
"timestamp order with a running accumulator."
|
||||
)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--dry-run",
|
||||
action="store_true",
|
||||
help="Show what would be changed without making changes",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--media-type",
|
||||
type=str,
|
||||
help="Only process a specific media type (e.g., Book, VideoGame)",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
dry_run = options["dry_run"]
|
||||
media_type = options.get("media_type")
|
||||
|
||||
media_types = list(LONG_PLAY_MEDIA.values())
|
||||
if media_type:
|
||||
if media_type not in media_types:
|
||||
self.stdout.write(
|
||||
self.style.ERROR(
|
||||
f"Invalid media type '{media_type}'. "
|
||||
f"Valid: {', '.join(media_types)}"
|
||||
)
|
||||
)
|
||||
return
|
||||
media_types = [media_type]
|
||||
|
||||
# Step 1: backfill long_play_last_scrobble
|
||||
self.stdout.write("Step 1: Backfilling long_play_last_scrobble chains...")
|
||||
total_backfilled = 0
|
||||
for mt in media_types:
|
||||
n = self._backfill_chain(mt, dry_run)
|
||||
total_backfilled += n
|
||||
self.stdout.write(f" {mt}: {n} scrobbles linked")
|
||||
|
||||
if dry_run:
|
||||
self.stdout.write(
|
||||
self.style.WARNING(
|
||||
f"Would backfill {total_backfilled} scrobbles total. "
|
||||
"Run without --dry-run to apply."
|
||||
)
|
||||
)
|
||||
else:
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(f"Backfilled {total_backfilled} scrobbles")
|
||||
)
|
||||
|
||||
# Step 2: recompute long_play_seconds
|
||||
self.stdout.write(
|
||||
"\nStep 2: Recomputing long_play_seconds in timestamp order..."
|
||||
)
|
||||
|
||||
total_updated = 0
|
||||
for mt in media_types:
|
||||
n = self._recompute_for_media_type(mt, dry_run)
|
||||
total_updated += n
|
||||
self.stdout.write(f" {mt}: {n} scrobbles updated")
|
||||
|
||||
if dry_run:
|
||||
self.stdout.write(
|
||||
self.style.WARNING(
|
||||
f"Dry run: would update {total_updated} scrobbles total. "
|
||||
"Use without --dry-run to apply."
|
||||
)
|
||||
)
|
||||
else:
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(f"Updated {total_updated} scrobbles")
|
||||
)
|
||||
|
||||
def _recompute_for_media_type(self, media_type: str, dry_run: bool) -> int:
|
||||
"""Process scrobbles for a single media type in timestamp order with a
|
||||
running accumulator, avoiding O(n2) FK chain walks."""
|
||||
fk = _media_type_to_fk(media_type)
|
||||
fk_id = f"{fk}_id"
|
||||
|
||||
scrobbles = Scrobble.objects.filter(
|
||||
media_type=media_type,
|
||||
**{f"{fk}__isnull": False},
|
||||
playback_position_seconds__isnull=False,
|
||||
).order_by(fk_id, "user_id", "timestamp")
|
||||
|
||||
total = scrobbles.count()
|
||||
if not total:
|
||||
return 0
|
||||
|
||||
updated = 0
|
||||
batch = []
|
||||
last_key = None
|
||||
running_total = 0
|
||||
|
||||
for scrobble in scrobbles.iterator():
|
||||
key = (getattr(scrobble, fk_id), scrobble.user_id)
|
||||
|
||||
if key != last_key:
|
||||
running_total = 0
|
||||
last_key = key
|
||||
|
||||
running_total += scrobble.playback_position_seconds or 0
|
||||
|
||||
if scrobble.long_play_seconds != running_total:
|
||||
updated += 1
|
||||
if not dry_run:
|
||||
scrobble.long_play_seconds = running_total
|
||||
batch.append(scrobble)
|
||||
if len(batch) >= BATCH_SIZE:
|
||||
Scrobble.objects.bulk_update(batch, ["long_play_seconds"])
|
||||
batch = []
|
||||
|
||||
if scrobble.long_play_complete:
|
||||
running_total = 0
|
||||
|
||||
if batch:
|
||||
Scrobble.objects.bulk_update(batch, ["long_play_seconds"])
|
||||
|
||||
return updated
|
||||
|
||||
def _backfill_chain(self, media_type: str, dry_run: bool) -> int:
|
||||
"""Set long_play_last_scrobble on each scrobble to the previous
|
||||
scrobble for the same media+user using a single UPDATE with a
|
||||
correlated subquery."""
|
||||
fk = _media_type_to_fk(media_type)
|
||||
table = Scrobble._meta.db_table
|
||||
|
||||
if dry_run:
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
f"SELECT COUNT(*) FROM {table} "
|
||||
f"WHERE long_play_last_scrobble_id IS NULL "
|
||||
f"AND {fk}_id IS NOT NULL"
|
||||
)
|
||||
return cursor.fetchone()[0]
|
||||
|
||||
with connection.cursor() as cursor:
|
||||
cursor.execute(
|
||||
f"UPDATE {table} "
|
||||
f"SET long_play_last_scrobble_id = ("
|
||||
f" SELECT id FROM {table} AS prev "
|
||||
f" WHERE prev.{fk}_id = {table}.{fk}_id "
|
||||
f" AND prev.user_id = {table}.user_id "
|
||||
f" AND prev.timestamp < {table}.timestamp "
|
||||
f" ORDER BY prev.timestamp DESC LIMIT 1"
|
||||
f") "
|
||||
f"WHERE long_play_last_scrobble_id IS NULL "
|
||||
f"AND {fk}_id IS NOT NULL"
|
||||
)
|
||||
return cursor.rowcount
|
||||
|
||||
|
||||
def _media_type_to_fk(media_type):
|
||||
mapping = {
|
||||
"VideoGame": "video_game",
|
||||
"Book": "book",
|
||||
"BrickSet": "brick_set",
|
||||
"Task": "task",
|
||||
}
|
||||
return mapping.get(media_type)
|
||||
518
vrobbler/apps/scrobbles/mcp.py
Normal file
518
vrobbler/apps/scrobbles/mcp.py
Normal file
@ -0,0 +1,518 @@
|
||||
from mcp_server import MCPToolset
|
||||
from scrobbles.models import Scrobble
|
||||
from scrobbles.constants import LONG_PLAY_MEDIA
|
||||
|
||||
|
||||
class ScrobbleToolset(MCPToolset):
|
||||
def list_recent_scrobbles(
|
||||
self,
|
||||
days: int = 7,
|
||||
media_type: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""List scrobbles from the last N days, optionally filtered by media type.
|
||||
Valid media_type values: Video, Track, PodcastEpisode, SportEvent, Book,
|
||||
Paper, VideoGame, BoardGame, GeoLocation, Trail, Beer, Puzzle, Food, Task,
|
||||
WebPage, LifeEvent, Mood, BrickSet, Channel, BirdingLocation, DiscGolfCourse
|
||||
"""
|
||||
qs = (
|
||||
Scrobble.objects.filter(user=self.request.user)
|
||||
.select_related(
|
||||
"video", "track", "book", "video_game", "board_game",
|
||||
"beer", "puzzle", "food", "trail", "task", "web_page",
|
||||
"life_event", "mood", "brick_set", "podcast_episode",
|
||||
"sport_event", "geo_location", "birding_location",
|
||||
"disc_golf_course", "channel",
|
||||
)
|
||||
.order_by("-timestamp")
|
||||
)
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
qs = qs.filter(timestamp__gte=timezone.now() - datetime.timedelta(days=days))
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
qs = qs[:limit]
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_scrobble(self, uuid: str) -> dict | None:
|
||||
"""Get a single scrobble by its UUID."""
|
||||
try:
|
||||
s = Scrobble.objects.filter(user=self.request.user).get(uuid=uuid)
|
||||
except Scrobble.DoesNotExist:
|
||||
return None
|
||||
return _scrobble_to_dict(s)
|
||||
|
||||
def search_scrobbles(
|
||||
self, query: str, media_type: str | None = None, limit: int = 20
|
||||
) -> list[dict]:
|
||||
"""Search scrobbles by text in their log data or related media titles."""
|
||||
from django.db.models import Q
|
||||
qs = Scrobble.objects.filter(user=self.request.user).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
qs = qs.filter(
|
||||
Q(log__icontains=query)
|
||||
| Q(video__title__icontains=query)
|
||||
| Q(track__title__icontains=query)
|
||||
| Q(book__title__icontains=query)
|
||||
| Q(video_game__title__icontains=query)
|
||||
| Q(board_game__title__icontains=query)
|
||||
| Q(beer__title__icontains=query)
|
||||
| Q(food__title__icontains=query)
|
||||
| Q(trail__title__icontains=query)
|
||||
| Q(task__title__icontains=query)
|
||||
| Q(web_page__title__icontains=query)
|
||||
| Q(life_event__title__icontains=query)
|
||||
| Q(puzzle__title__icontains=query)
|
||||
| Q(brick_set__title__icontains=query)
|
||||
| Q(podcast_episode__title__icontains=query)
|
||||
)[:limit]
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_scrobbles_by_date(
|
||||
self, date: str, media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get scrobbles for a specific date (YYYY-MM-DD format)."""
|
||||
import datetime
|
||||
try:
|
||||
dt = datetime.datetime.strptime(date, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
return []
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
timestamp__date=dt,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_in_progress_scrobbles(
|
||||
self, media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get scrobbles currently in progress (started but not finished).
|
||||
These are long-play items like books, video games, brick sets, or tasks."""
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
in_progress=True,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_long_play_scrobbles(
|
||||
self, status: str = "in_progress", media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get long-play scrobbles (books, video games, brick sets, tasks).
|
||||
Status can be 'in_progress' or 'completed'."""
|
||||
types = list(LONG_PLAY_MEDIA.values())
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
media_type__in=types,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
if status == "in_progress":
|
||||
qs = qs.filter(in_progress=True)
|
||||
elif status == "completed":
|
||||
qs = qs.filter(in_progress=False)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
|
||||
class MediaToolset(MCPToolset):
|
||||
def get_book(self, uuid: str) -> dict | None:
|
||||
"""Get a book by UUID."""
|
||||
from books.models import Book
|
||||
try:
|
||||
b = Book.objects.get(uuid=uuid)
|
||||
except Book.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(b, fields=["title", "pages", "language",
|
||||
"first_publish_year", "isbn_13",
|
||||
"publisher", "summary"])
|
||||
|
||||
def list_books(self, author: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List books, optionally filtered by author name."""
|
||||
from books.models import Book
|
||||
qs = Book.objects.all().order_by("title")
|
||||
if author:
|
||||
qs = qs.filter(authors__name__icontains=author)
|
||||
return [_media_to_dict(b, fields=["title", "pages", "language",
|
||||
"first_publish_year", "isbn_13",
|
||||
"publisher"]) for b in qs[:limit]]
|
||||
|
||||
def get_track(self, uuid: str) -> dict | None:
|
||||
"""Get a music track by UUID."""
|
||||
from music.models import Track
|
||||
try:
|
||||
t = Track.objects.select_related("artist_fk").get(uuid=uuid)
|
||||
except Track.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "base_run_time_seconds",
|
||||
"artist_fk__name", "genre"])
|
||||
|
||||
def list_tracks(self, artist: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List music tracks, optionally filtered by artist name."""
|
||||
from music.models import Track
|
||||
qs = Track.objects.select_related("artist_fk").all().order_by("title")
|
||||
if artist:
|
||||
qs = qs.filter(artist_fk__name__icontains=artist)
|
||||
return [_media_to_dict(t, fields=["title", "base_run_time_seconds",
|
||||
"artist_fk__name", "genre"])
|
||||
for t in qs[:limit]]
|
||||
|
||||
def get_video(self, uuid: str) -> dict | None:
|
||||
"""Get a video by UUID."""
|
||||
from videos.models import Video
|
||||
try:
|
||||
v = Video.objects.select_related("tv_series", "channel").get(uuid=uuid)
|
||||
except Video.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(v, fields=["title", "year", "overview",
|
||||
"imdb_id", "imdb_rating",
|
||||
"tv_series__name", "channel__title",
|
||||
"season_number", "episode_number"])
|
||||
|
||||
def list_videos(self, series: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List videos, optionally filtered by series name."""
|
||||
from videos.models import Video
|
||||
qs = Video.objects.select_related("tv_series", "channel").all().order_by("title")
|
||||
if series:
|
||||
qs = qs.filter(tv_series__name__icontains=series)
|
||||
return [_media_to_dict(v, fields=["title", "year", "overview",
|
||||
"tv_series__name", "channel__title",
|
||||
"season_number", "episode_number"])
|
||||
for v in qs[:limit]]
|
||||
|
||||
def get_board_game(self, uuid: str) -> dict | None:
|
||||
"""Get a board game by UUID."""
|
||||
from boardgames.models import BoardGame
|
||||
try:
|
||||
bg = BoardGame.objects.get(uuid=uuid)
|
||||
except BoardGame.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bg, fields=["title", "genre"])
|
||||
|
||||
def list_board_games(self, limit: int = 20) -> list[dict]:
|
||||
"""List board games."""
|
||||
from boardgames.models import BoardGame
|
||||
qs = BoardGame.objects.all().order_by("title")[:limit]
|
||||
return [_media_to_dict(bg, fields=["title", "genre"]) for bg in qs]
|
||||
|
||||
def get_podcast_episode(self, uuid: str) -> dict | None:
|
||||
"""Get a podcast episode by UUID."""
|
||||
from podcasts.models import PodcastEpisode
|
||||
try:
|
||||
pe = PodcastEpisode.objects.select_related("podcast", "producer").get(
|
||||
uuid=uuid
|
||||
)
|
||||
except PodcastEpisode.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(pe, fields=["title", "podcast__title",
|
||||
"producer__name", "base_run_time_seconds"])
|
||||
|
||||
def get_beer(self, uuid: str) -> dict | None:
|
||||
"""Get a beer by UUID."""
|
||||
from beers.models import Beer
|
||||
try:
|
||||
b = Beer.objects.select_related("style", "producer").get(uuid=uuid)
|
||||
except Beer.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(b, fields=["title", "style__name",
|
||||
"producer__name", "abv"])
|
||||
|
||||
def get_brick_set(self, uuid: str) -> dict | None:
|
||||
"""Get a brick set (LEGO) by UUID."""
|
||||
from bricksets.models import BrickSet
|
||||
try:
|
||||
bs = BrickSet.objects.get(uuid=uuid)
|
||||
except BrickSet.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bs, fields=["title", "piece_count", "set_number"])
|
||||
|
||||
def get_video_game(self, uuid: str) -> dict | None:
|
||||
"""Get a video game by UUID."""
|
||||
from videogames.models import VideoGame
|
||||
try:
|
||||
vg = VideoGame.objects.get(uuid=uuid)
|
||||
except VideoGame.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(vg, fields=["title", "genre",
|
||||
"base_run_time_seconds"])
|
||||
|
||||
def get_puzzle(self, uuid: str) -> dict | None:
|
||||
"""Get a puzzle by UUID."""
|
||||
from puzzles.models import Puzzle
|
||||
try:
|
||||
p = Puzzle.objects.select_related("manufacturer").get(uuid=uuid)
|
||||
except Puzzle.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(p, fields=["title", "piece_count",
|
||||
"manufacturer__name"])
|
||||
|
||||
def get_web_page(self, uuid: str) -> dict | None:
|
||||
"""Get a web page by UUID."""
|
||||
from webpages.models import WebPage
|
||||
try:
|
||||
wp = WebPage.objects.select_related("domain").get(uuid=uuid)
|
||||
except WebPage.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(wp, fields=["title", "url", "domain__name"])
|
||||
|
||||
def get_task(self, uuid: str) -> dict | None:
|
||||
"""Get a task by UUID."""
|
||||
from tasks.models import Task
|
||||
try:
|
||||
t = Task.objects.get(uuid=uuid)
|
||||
except Task.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "completed"])
|
||||
|
||||
def get_trail(self, uuid: str) -> dict | None:
|
||||
"""Get a trail by UUID."""
|
||||
from trails.models import Trail
|
||||
try:
|
||||
t = Trail.objects.get(uuid=uuid)
|
||||
except Trail.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "genre", "base_run_time_seconds"])
|
||||
|
||||
def get_geo_location(self, uuid: str) -> dict | None:
|
||||
"""Get a geo location by UUID."""
|
||||
from locations.models import GeoLocation
|
||||
try:
|
||||
gl = GeoLocation.objects.get(uuid=uuid)
|
||||
except GeoLocation.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(gl, fields=["title", "latitude", "longitude"])
|
||||
|
||||
def get_life_event(self, uuid: str) -> dict | None:
|
||||
"""Get a life event by UUID."""
|
||||
from lifeevents.models import LifeEvent
|
||||
try:
|
||||
le = LifeEvent.objects.get(uuid=uuid)
|
||||
except LifeEvent.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(le, fields=["title", "event_date", "genre"])
|
||||
|
||||
def get_mood(self, uuid: str) -> dict | None:
|
||||
"""Get a mood entry by UUID."""
|
||||
from moods.models import Mood
|
||||
try:
|
||||
m = Mood.objects.get(uuid=uuid)
|
||||
except Mood.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(m, fields=["title", "mood_type", "mood_quality"])
|
||||
|
||||
def get_food(self, uuid: str) -> dict | None:
|
||||
"""Get a food entry by UUID."""
|
||||
from foods.models import Food
|
||||
try:
|
||||
f = Food.objects.select_related("category").get(uuid=uuid)
|
||||
except Food.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(f, fields=["title", "category__name"])
|
||||
|
||||
def get_bird_sighting(self, uuid: str) -> dict | None:
|
||||
"""Get a bird sighting by UUID."""
|
||||
from birds.models import BirdSighting
|
||||
try:
|
||||
bs = BirdSighting.objects.select_related("bird").get(uuid=uuid)
|
||||
except BirdSighting.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bs, fields=["title", "bird__common_name",
|
||||
"bird__scientific_name", "location"])
|
||||
|
||||
def get_disc_golf_course(self, uuid: str) -> dict | None:
|
||||
"""Get a disc golf course by UUID."""
|
||||
from discgolf.models import DiscGolfCourse
|
||||
try:
|
||||
dg = DiscGolfCourse.objects.get(uuid=uuid)
|
||||
except DiscGolfCourse.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(dg, fields=["title", "holes", "location"])
|
||||
|
||||
|
||||
class StatsToolset(MCPToolset):
|
||||
def get_scrobble_counts(self, days: int = 30) -> list[dict]:
|
||||
"""Get scrobble counts grouped by media type for the last N days."""
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
from django.db.models import Count
|
||||
|
||||
cutoff = timezone.now() - datetime.timedelta(days=days)
|
||||
qs = (
|
||||
Scrobble.objects.filter(user=self.request.user, timestamp__gte=cutoff)
|
||||
.values("media_type")
|
||||
.annotate(count=Count("id"))
|
||||
.order_by("-count")
|
||||
)
|
||||
return list(qs)
|
||||
|
||||
def get_top_media(
|
||||
self, media_type: str, days: int = 30, limit: int = 10
|
||||
) -> list[dict]:
|
||||
"""Get the most-scrobbled items of a given media type in the last N days.
|
||||
Valid media_type values: Video, Track, Book, BoardGame, Beer, etc."""
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
from django.db.models import Count
|
||||
|
||||
cutoff = timezone.now() - datetime.timedelta(days=days)
|
||||
|
||||
rel_field = _media_type_to_rel_field(media_type)
|
||||
if not rel_field:
|
||||
return []
|
||||
|
||||
qs = (
|
||||
Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
media_type=media_type,
|
||||
timestamp__gte=cutoff,
|
||||
)
|
||||
.values(rel_field)
|
||||
.annotate(count=Count("id"))
|
||||
.order_by("-count")
|
||||
)[:limit]
|
||||
|
||||
results = []
|
||||
for row in qs:
|
||||
obj_id = row[rel_field]
|
||||
if obj_id is None:
|
||||
continue
|
||||
results.append({"id": obj_id, "count": row["count"]})
|
||||
return results
|
||||
|
||||
|
||||
def _media_type_to_rel_field(media_type: str) -> str | None:
|
||||
mapping = {
|
||||
"Video": "video",
|
||||
"Track": "track",
|
||||
"PodcastEpisode": "podcast_episode",
|
||||
"SportEvent": "sport_event",
|
||||
"Book": "book",
|
||||
"Paper": "paper",
|
||||
"VideoGame": "video_game",
|
||||
"BoardGame": "board_game",
|
||||
"GeoLocation": "geo_location",
|
||||
"Trail": "trail",
|
||||
"Beer": "beer",
|
||||
"Puzzle": "puzzle",
|
||||
"Food": "food",
|
||||
"Task": "task",
|
||||
"WebPage": "web_page",
|
||||
"LifeEvent": "life_event",
|
||||
"Mood": "mood",
|
||||
"BrickSet": "brick_set",
|
||||
"Channel": "channel",
|
||||
"BirdingLocation": "birding_location",
|
||||
"DiscGolfCourse": "disc_golf_course",
|
||||
}
|
||||
return mapping.get(media_type)
|
||||
|
||||
|
||||
def _scrobble_to_dict(s: Scrobble) -> dict:
|
||||
result = {
|
||||
"uuid": str(s.uuid),
|
||||
"media_type": s.media_type,
|
||||
"timestamp": s.timestamp.isoformat() if s.timestamp else None,
|
||||
"stop_timestamp": s.stop_timestamp.isoformat() if s.stop_timestamp else None,
|
||||
"in_progress": s.in_progress,
|
||||
"played_to_completion": s.played_to_completion,
|
||||
"source": s.source,
|
||||
"visibility": s.visibility,
|
||||
"timezone": s.timezone,
|
||||
}
|
||||
if s.log:
|
||||
result["log"] = s.log
|
||||
|
||||
rel = _scrobble_related_to_dict(s)
|
||||
if rel:
|
||||
result["media"] = rel
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _scrobble_related_to_dict(s: Scrobble) -> dict | None:
|
||||
if s.video:
|
||||
return _media_to_dict(s.video, fields=["title", "year", "imdb_id",
|
||||
"imdb_rating"])
|
||||
if s.track:
|
||||
return _media_to_dict(s.track, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.book:
|
||||
return _media_to_dict(s.book, fields=["title", "pages"])
|
||||
if s.video_game:
|
||||
return _media_to_dict(s.video_game, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.board_game:
|
||||
return _media_to_dict(s.board_game, fields=["title"])
|
||||
if s.beer:
|
||||
return _media_to_dict(s.beer, fields=["title"])
|
||||
if s.puzzle:
|
||||
return _media_to_dict(s.puzzle, fields=["title", "piece_count"])
|
||||
if s.food:
|
||||
return _media_to_dict(s.food, fields=["title"])
|
||||
if s.trail:
|
||||
return _media_to_dict(s.trail, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.task:
|
||||
return _media_to_dict(s.task, fields=["title"])
|
||||
if s.web_page:
|
||||
return _media_to_dict(s.web_page, fields=["title", "url"])
|
||||
if s.life_event:
|
||||
return _media_to_dict(s.life_event, fields=["title", "event_date"])
|
||||
if s.mood:
|
||||
return _media_to_dict(s.mood, fields=["title"])
|
||||
if s.brick_set:
|
||||
return _media_to_dict(s.brick_set, fields=["title", "set_number"])
|
||||
if s.podcast_episode:
|
||||
return _media_to_dict(s.podcast_episode, fields=["title"])
|
||||
if s.sport_event:
|
||||
return {"title": str(s.sport_event)}
|
||||
if s.geo_location:
|
||||
return _media_to_dict(s.geo_location, fields=["title", "latitude",
|
||||
"longitude"])
|
||||
if s.birding_location:
|
||||
return _media_to_dict(s.birding_location, fields=["title"])
|
||||
if s.disc_golf_course:
|
||||
return _media_to_dict(s.disc_golf_course, fields=["title", "holes"])
|
||||
if s.channel:
|
||||
return _media_to_dict(s.channel, fields=["title"])
|
||||
return None
|
||||
|
||||
|
||||
def _media_to_dict(obj, fields: list[str] | None = None) -> dict:
|
||||
if obj is None:
|
||||
return {}
|
||||
result = {}
|
||||
if hasattr(obj, "uuid"):
|
||||
result["uuid"] = str(obj.uuid)
|
||||
if hasattr(obj, "title"):
|
||||
result["title"] = obj.title
|
||||
|
||||
if fields is None:
|
||||
return result
|
||||
|
||||
resolved = _resolve_fields(obj, fields)
|
||||
for k, v in resolved.items():
|
||||
if k not in result:
|
||||
result[k] = v
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _resolve_fields(obj, fields: list[str]) -> dict:
|
||||
result = {}
|
||||
for field in fields:
|
||||
parts = field.split("__")
|
||||
val = obj
|
||||
try:
|
||||
for part in parts:
|
||||
val = getattr(val, part)
|
||||
except AttributeError:
|
||||
continue
|
||||
if val is not None:
|
||||
if hasattr(val, "all"):
|
||||
val = [str(v) for v in val.all()]
|
||||
result[field] = val
|
||||
return result
|
||||
@ -0,0 +1,20 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
def backfill_null_visibility(apps, schema_editor):
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
Scrobble.objects.filter(visibility__isnull=True).update(visibility="private")
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0094_scrobble_share_view_count_alter_scrobble_visibility_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RunPython(
|
||||
backfill_null_visibility,
|
||||
reverse_code=migrations.RunPython.noop,
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,35 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
def convert_page_data_to_dict(apps, schema_editor):
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
|
||||
log__page_data=None
|
||||
):
|
||||
page_data = scrobble.log.get("page_data")
|
||||
if isinstance(page_data, list):
|
||||
new_page_data = {}
|
||||
for entry in page_data:
|
||||
page_num = entry.get("page_number")
|
||||
if page_num is not None:
|
||||
try:
|
||||
page_num = int(page_num)
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
new_page_data[page_num] = entry
|
||||
scrobble.log["page_data"] = new_page_data
|
||||
scrobble.save(update_fields=["log"])
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0095_backfill_null_visibility"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RunPython(
|
||||
convert_page_data_to_dict,
|
||||
reverse_code=migrations.RunPython.noop,
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,19 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-12 16:09
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0096_convert_book_page_data_to_dict"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddIndex(
|
||||
model_name="scrobble",
|
||||
index=models.Index(
|
||||
fields=["user", "-timestamp"], name="scrobbles_s_user_id_d367a7_idx"
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,25 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-15 17:48
|
||||
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0097_scrobble_scrobbles_s_user_id_d367a7_idx"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="scrobble",
|
||||
name="long_play_last_scrobble",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.SET_NULL,
|
||||
related_name="next_long_play_scrobbles",
|
||||
to="scrobbles.scrobble",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,156 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:19
|
||||
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
import django_extensions.db.fields
|
||||
import scrobbles.models
|
||||
import uuid
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0001_initial"),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="favoritemedia",
|
||||
name="disc_golf",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.CASCADE,
|
||||
to="discgolf.discgolfcourse",
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="scrobble",
|
||||
name="disc_golf",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="discgolf.discgolfcourse",
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="favoritemedia",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("DiscGolf", "Disc golf"),
|
||||
],
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="scrobble",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("DiscGolf", "Disc golf"),
|
||||
],
|
||||
default="Video",
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name="UDiscCSVImport",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
("uuid", models.UUIDField(default=uuid.uuid4, editable=False)),
|
||||
("processing_started", models.DateTimeField(blank=True, null=True)),
|
||||
("processed_finished", models.DateTimeField(blank=True, null=True)),
|
||||
("process_log", models.TextField(blank=True, null=True)),
|
||||
("process_count", models.IntegerField(blank=True, null=True)),
|
||||
("error_log", models.TextField(blank=True, null=True)),
|
||||
(
|
||||
"csv_file",
|
||||
models.FileField(
|
||||
blank=True,
|
||||
null=True,
|
||||
upload_to=scrobbles.models.UDiscCSVImport.get_path,
|
||||
),
|
||||
),
|
||||
(
|
||||
"original_filename",
|
||||
models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
(
|
||||
"user",
|
||||
models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
related_name="scrobbles_udisccsvimport_set",
|
||||
to=settings.AUTH_USER_MODEL,
|
||||
),
|
||||
),
|
||||
],
|
||||
options={
|
||||
"verbose_name": "uDisc CSV Import",
|
||||
},
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,84 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:53
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0099_favoritemedia_disc_golf_scrobble_disc_golf_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RenameField(
|
||||
model_name="favoritemedia",
|
||||
old_name="disc_golf",
|
||||
new_name="disc_golf_course",
|
||||
),
|
||||
migrations.RenameField(
|
||||
model_name="scrobble",
|
||||
old_name="disc_golf",
|
||||
new_name="disc_golf_course",
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="favoritemedia",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("BirdingLocation", "Birding location"),
|
||||
("DiscGolfCourse", "Disc golf"),
|
||||
],
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="scrobble",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("BirdingLocation", "Birding location"),
|
||||
("DiscGolfCourse", "Disc golf"),
|
||||
],
|
||||
default="Video",
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -114,7 +114,7 @@ class ScrobblableMixin(TimeStampedModel):
|
||||
|
||||
@property
|
||||
def start_url(self):
|
||||
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid})
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
@ -162,26 +162,21 @@ class LongPlayScrobblableMixin(ScrobblableMixin):
|
||||
return False
|
||||
|
||||
def get_longplay_finish_url(self):
|
||||
return reverse("scrobbles:longplay-finish", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:longplay-finish", kwargs={"media_uuid": self.uuid})
|
||||
|
||||
def first_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
|
||||
return (
|
||||
get_scrobbles_for_media(self, user)
|
||||
.filter(
|
||||
log__long_play_complete=False,
|
||||
log__serial_scrobble_id__isnull=True,
|
||||
)
|
||||
.order_by("timestamp")
|
||||
.first()
|
||||
)
|
||||
last = self.last_long_play_scrobble_for_user(user)
|
||||
if not last:
|
||||
return None
|
||||
current = last
|
||||
while current.long_play_last_scrobble and not current.long_play_last_scrobble.long_play_complete:
|
||||
current = current.long_play_last_scrobble
|
||||
return current
|
||||
|
||||
def last_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
|
||||
return (
|
||||
get_scrobbles_for_media(self, user)
|
||||
.filter(
|
||||
log__long_play_complete=False,
|
||||
log__serial_scrobble_id__isnull=False,
|
||||
)
|
||||
.order_by("timestamp")
|
||||
.last()
|
||||
.filter(long_play_complete=False)
|
||||
.order_by("-timestamp")
|
||||
.first()
|
||||
)
|
||||
|
||||
@ -11,6 +11,7 @@ import pendulum
|
||||
import pytz
|
||||
from beers.models import Beer
|
||||
from birds.models import BirdingLocation
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from boardgames.models import BoardGame
|
||||
from books.koreader import process_koreader_sqlite_file
|
||||
from books.models import Book, BookLogData, BookPageLogData, Paper
|
||||
@ -51,7 +52,10 @@ from scrobbles.constants import (
|
||||
MEDIA_END_PADDING_SECONDS,
|
||||
)
|
||||
from scrobbles.importers.lastfm import LastFM
|
||||
from scrobbles.notifications import ScrobbleNtfyNotification
|
||||
from scrobbles.notifications import (
|
||||
LastFmImportNtfyNotification,
|
||||
ScrobbleNtfyNotification,
|
||||
)
|
||||
from scrobbles.utils import get_file_md5_hash, media_class_to_foreign_key
|
||||
from sports.models import SportEvent
|
||||
from taggit.managers import TaggableManager
|
||||
@ -225,7 +229,9 @@ class KoReaderImport(BaseFileImportMixin):
|
||||
|
||||
self.mark_started()
|
||||
try:
|
||||
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
|
||||
scrobbles = process_koreader_sqlite_file(
|
||||
self.upload_file_path, self.user.id
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
@ -272,7 +278,9 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
|
||||
|
||||
self.mark_started()
|
||||
try:
|
||||
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
|
||||
scrobbles = import_audioscrobbler_tsv_file(
|
||||
self.upload_file_path, self.user.id
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
@ -424,6 +432,8 @@ class LastFmImport(BaseFileImportMixin):
|
||||
try:
|
||||
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
|
||||
self.record_log(scrobbles)
|
||||
if scrobbles:
|
||||
LastFmImportNtfyNotification(self, len(scrobbles)).send()
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
@ -608,6 +618,85 @@ class EBirdCSVImport(BaseFileImportMixin):
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class UDiscCSVImport(BaseFileImportMixin):
|
||||
class Meta:
|
||||
verbose_name = "uDisc CSV Import"
|
||||
|
||||
user = models.ForeignKey(
|
||||
User,
|
||||
on_delete=models.DO_NOTHING,
|
||||
**BNULL,
|
||||
related_name="scrobbles_udisccsvimport_set",
|
||||
)
|
||||
|
||||
@property
|
||||
def import_type(self) -> str:
|
||||
return "uDisc"
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("scrobbles:udisc-csv-import-detail", kwargs={"slug": self.uuid})
|
||||
|
||||
def get_path(instance, filename):
|
||||
extension = filename.split(".")[-1]
|
||||
uuid = instance.uuid
|
||||
return f"udisc-csv-uploads/{uuid}.{extension}"
|
||||
|
||||
@property
|
||||
def upload_file_path(self):
|
||||
if getattr(settings, "USE_S3_STORAGE"):
|
||||
path = self.csv_file.url
|
||||
else:
|
||||
path = self.csv_file.path
|
||||
return path
|
||||
|
||||
csv_file = models.FileField(upload_to=get_path, **BNULL)
|
||||
original_filename = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
def process(self, force=False):
|
||||
from discgolf.utils import import_udisc_csv
|
||||
|
||||
if self.processed_finished and not force:
|
||||
logger.info(f"{self} already processed on {self.processed_finished}")
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
try:
|
||||
scrobbles = import_udisc_csv(
|
||||
self.upload_file_path, self.user_id, record_error=self.record_error
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
TYPE_FK_PREFETCHES: dict[str, tuple[str, ...]] = {
|
||||
"Video": ("video",),
|
||||
"Track": ("track", "track__artist_fk"),
|
||||
"PodcastEpisode": ("podcast_episode", "podcast_episode__podcast"),
|
||||
"SportEvent": ("sport_event",),
|
||||
"Book": ("book",),
|
||||
"Paper": ("paper",),
|
||||
"VideoGame": ("video_game",),
|
||||
"BoardGame": ("board_game",),
|
||||
"GeoLocation": ("geo_location",),
|
||||
"Trail": ("trail",),
|
||||
"Beer": ("beer",),
|
||||
"Puzzle": ("puzzle",),
|
||||
"Food": ("food",),
|
||||
"Task": ("task",),
|
||||
"WebPage": ("web_page",),
|
||||
"LifeEvent": ("life_event",),
|
||||
"Mood": ("mood",),
|
||||
"BrickSet": ("brick_set",),
|
||||
"Channel": ("channel",),
|
||||
"BirdingLocation": ("birding_location",),
|
||||
"DiscGolfCourse": ("disc_golf_course",),
|
||||
}
|
||||
|
||||
|
||||
class ScrobbleQuerySet(models.QuerySet):
|
||||
def with_related(self):
|
||||
return self.select_related("user").prefetch_related(
|
||||
@ -632,8 +721,16 @@ class ScrobbleQuerySet(models.QuerySet):
|
||||
"mood",
|
||||
"brick_set",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
)
|
||||
|
||||
def with_related_for_types(self, media_types: list[str]):
|
||||
prefetches = []
|
||||
for t in media_types:
|
||||
if t in TYPE_FK_PREFETCHES:
|
||||
prefetches.extend(TYPE_FK_PREFETCHES[t])
|
||||
return self.select_related("user").prefetch_related(*prefetches)
|
||||
|
||||
|
||||
class ShareViewLog(TimeStampedModel):
|
||||
scrobble = models.ForeignKey(
|
||||
@ -675,6 +772,7 @@ class Scrobble(TimeStampedModel):
|
||||
BRICKSET = "BrickSet", "Brick set"
|
||||
CHANNEL = "Channel", "Channel"
|
||||
BIRDING_LOCATION = "BirdingLocation", "Birding location"
|
||||
DISC_GOLF = "DiscGolfCourse", "Disc golf"
|
||||
|
||||
@classmethod
|
||||
def list(cls):
|
||||
@ -705,6 +803,9 @@ class Scrobble(TimeStampedModel):
|
||||
birding_location = models.ForeignKey(
|
||||
BirdingLocation, on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
disc_golf_course = models.ForeignKey(
|
||||
DiscGolfCourse, on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
media_type = models.CharField(
|
||||
max_length=20, choices=MediaType.choices, default=MediaType.VIDEO
|
||||
)
|
||||
@ -758,8 +859,20 @@ class Scrobble(TimeStampedModel):
|
||||
format="JPEG",
|
||||
options={"quality": 75},
|
||||
)
|
||||
screenshot_large = ImageSpecField(
|
||||
source="screenshot",
|
||||
processors=[ResizeToFit(800, 800)],
|
||||
format="JPEG",
|
||||
options={"quality": 85},
|
||||
)
|
||||
long_play_seconds = models.BigIntegerField(**BNULL)
|
||||
long_play_complete = models.BooleanField(**BNULL)
|
||||
long_play_last_scrobble = models.ForeignKey(
|
||||
"self",
|
||||
**BNULL,
|
||||
on_delete=models.SET_NULL,
|
||||
related_name="next_long_play_scrobbles",
|
||||
)
|
||||
|
||||
class Meta:
|
||||
indexes = [
|
||||
@ -774,6 +887,7 @@ class Scrobble(TimeStampedModel):
|
||||
"is_paused",
|
||||
]
|
||||
),
|
||||
models.Index(fields=["user", "-timestamp"]),
|
||||
]
|
||||
|
||||
@classmethod
|
||||
@ -806,11 +920,14 @@ class Scrobble(TimeStampedModel):
|
||||
@classmethod
|
||||
def as_dict_by_type(cls, scrobble_qs: models.QuerySet) -> dict:
|
||||
scrobbles_by_type = defaultdict(list)
|
||||
scrobbles = (
|
||||
scrobble_qs.with_related()
|
||||
if hasattr(scrobble_qs, "with_related")
|
||||
else scrobble_qs
|
||||
)
|
||||
|
||||
if hasattr(scrobble_qs, "with_related"):
|
||||
media_types_present = list(
|
||||
scrobble_qs.values_list("media_type", flat=True).distinct()
|
||||
)
|
||||
scrobbles = scrobble_qs.with_related_for_types(media_types_present)
|
||||
else:
|
||||
scrobbles = scrobble_qs
|
||||
|
||||
for scrobble in scrobbles:
|
||||
scrobbles_by_type[scrobble.media_type].append(scrobble)
|
||||
@ -825,7 +942,7 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
# Remove any locations without titles
|
||||
if "GeoLocation" in scrobbles_by_type.keys():
|
||||
for loc_scrobble in scrobbles_by_type["GeoLocation"]:
|
||||
for loc_scrobble in list(scrobbles_by_type["GeoLocation"]):
|
||||
if not loc_scrobble.media_obj.title:
|
||||
scrobbles_by_type["GeoLocation"].remove(loc_scrobble)
|
||||
scrobbles_by_type["GeoLocation_count"] -= 1
|
||||
@ -850,7 +967,7 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
@property
|
||||
def finish_url(self) -> str:
|
||||
return reverse("scrobbles:finish", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:finish", kwargs={"pk": self.pk})
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
class_name = self.media_obj.__class__.__name__
|
||||
@ -892,10 +1009,7 @@ class Scrobble(TimeStampedModel):
|
||||
return super(Scrobble, self).save(*args, **kwargs)
|
||||
|
||||
def get_absolute_url(self):
|
||||
if not self.uuid:
|
||||
self.uuid = uuid4()
|
||||
self.save(update_fields=["uuid"])
|
||||
return reverse("scrobbles:detail", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:detail", kwargs={"pk": self.pk})
|
||||
|
||||
def get_share_url(self):
|
||||
if self.visibility == Visibility.PRIVATE:
|
||||
@ -907,27 +1021,6 @@ class Scrobble(TimeStampedModel):
|
||||
self.share_token_version += 1
|
||||
self.save(update_fields=["share_token_version"])
|
||||
|
||||
def push_to_archivebox(self):
|
||||
pushable_media = hasattr(self.media_obj, "push_to_archivebox") and callable(
|
||||
self.media_obj.push_to_archivebox
|
||||
)
|
||||
|
||||
if pushable_media and self.user.profile.archivebox_url:
|
||||
try:
|
||||
self.media_obj.push_to_archivebox(
|
||||
url=self.user.profile.archivebox_url,
|
||||
username=self.user.profile.archivebox_username,
|
||||
password=self.user.profile.archivebox_password,
|
||||
)
|
||||
except Exception:
|
||||
logger.info(
|
||||
"Failed to push URL to archivebox",
|
||||
extra={
|
||||
"archivebox_url": self.user.profile.archivebox_url,
|
||||
"archivebox_username": self.user.profile.archivebox_username,
|
||||
},
|
||||
)
|
||||
|
||||
@property
|
||||
def logdata(self) -> Optional[logdata.BaseLogData]:
|
||||
if self.media_obj:
|
||||
@ -936,6 +1029,8 @@ class Scrobble(TimeStampedModel):
|
||||
logdata_cls = logdata.BaseLogData
|
||||
|
||||
log_dict = self.log
|
||||
if isinstance(log_dict, logdata.BaseLogData):
|
||||
log_dict = log_dict.asdict
|
||||
if isinstance(self.log, str):
|
||||
# There's nothing stopping django from saving a string in a JSONField :(
|
||||
logger.warning(
|
||||
@ -947,24 +1042,8 @@ class Scrobble(TimeStampedModel):
|
||||
if not log_dict:
|
||||
log_dict = {}
|
||||
|
||||
# Special handling for GeoLocationLogData - data is nested under 'movement_detection'
|
||||
# TODO there's a better way to fix this this at the LogData level
|
||||
if logdata_cls.__name__ == "GeoLocationLogData":
|
||||
instance_data = log_dict.get("movement_detection", {}).copy()
|
||||
# Add top-level fields that GeoLocationLogData expects from BaseLogData/WithPeopleLogData
|
||||
for field_name in ["description", "notes", "with_people_ids"]:
|
||||
if field_name in log_dict:
|
||||
instance_data[field_name] = log_dict[field_name]
|
||||
try:
|
||||
return logdata_cls(**instance_data)
|
||||
except Exception as e:
|
||||
logger.warning("Log data could not be loaded", e)
|
||||
return logdata_cls()
|
||||
|
||||
# Strip log-only keys (stored in JSONField but not part of LogData dataclass)
|
||||
logdata_kwargs = {
|
||||
k: v for k, v in log_dict.items() if k in logdata_cls().__dataclass_fields__
|
||||
}
|
||||
# Use LogData's from_log_dict to handle any custom nesting/structure
|
||||
logdata_kwargs = logdata_cls.from_log_dict(log_dict)
|
||||
|
||||
try:
|
||||
return logdata_cls(**logdata_kwargs)
|
||||
@ -1141,8 +1220,8 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
if self.is_long_play:
|
||||
long_play_secs = 0
|
||||
if self.previous and not self.previous.long_play_complete:
|
||||
long_play_secs = self.previous.long_play_seconds or 0
|
||||
if self.long_play_last_scrobble and not self.long_play_last_scrobble.long_play_complete:
|
||||
long_play_secs = self.long_play_last_scrobble.long_play_seconds or 0
|
||||
percent = int(((playback_seconds + long_play_secs) / run_time_secs) * 100)
|
||||
|
||||
return percent
|
||||
@ -1298,6 +1377,10 @@ class Scrobble(TimeStampedModel):
|
||||
media_obj = self.channel
|
||||
if self.birding_location:
|
||||
media_obj = self.birding_location
|
||||
if self.paper:
|
||||
media_obj = self.paper
|
||||
if self.disc_golf_course:
|
||||
media_obj = self.disc_golf_course
|
||||
return media_obj
|
||||
|
||||
def __str__(self):
|
||||
@ -1412,20 +1495,18 @@ class Scrobble(TimeStampedModel):
|
||||
or scrobble_data["playback_status"] == "stopped"
|
||||
):
|
||||
if read_log_page:
|
||||
page_list = scrobble.log.get("page_data", [])
|
||||
if page_list:
|
||||
for page in page_list:
|
||||
page_data = scrobble.log.get("page_data", {})
|
||||
if page_data:
|
||||
for page_num, page in page_data.items():
|
||||
if not page.get("end_ts", None):
|
||||
page["end_ts"] = int(timezone.now().timestamp())
|
||||
page["duration"] = page["end_ts"] - page.get("start_ts")
|
||||
|
||||
page_list.append(
|
||||
BookPageLogData(
|
||||
page_number=read_log_page,
|
||||
start_ts=int(timezone.now().timestamp()),
|
||||
)
|
||||
page_data[read_log_page] = BookPageLogData(
|
||||
page_number=read_log_page,
|
||||
start_ts=int(timezone.now().timestamp()),
|
||||
)
|
||||
scrobble.log["page_data"] = page_list
|
||||
scrobble.log["page_data"] = page_data
|
||||
scrobble.save(update_fields=["log"])
|
||||
elif "log" in scrobble_data.keys() and scrobble.log:
|
||||
scrobble_data["log"] = scrobble.log | scrobble_data["log"]
|
||||
@ -1436,13 +1517,13 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
if read_log_page:
|
||||
scrobble_data["log"] = BookLogData(
|
||||
page_data=[
|
||||
BookPageLogData(
|
||||
page_data={
|
||||
read_log_page: BookPageLogData(
|
||||
page_number=read_log_page,
|
||||
start_ts=int(timezone.now().timestamp()),
|
||||
)
|
||||
]
|
||||
)
|
||||
}
|
||||
).asdict
|
||||
|
||||
logger.info(
|
||||
f"[scrobbling] creating new scrobble",
|
||||
@ -1457,9 +1538,27 @@ class Scrobble(TimeStampedModel):
|
||||
"calories", None
|
||||
):
|
||||
if media.calories:
|
||||
scrobble_data["log"] = FoodLogData(calories=media.calories)
|
||||
scrobble_data["log"] = FoodLogData(calories=media.calories).asdict
|
||||
|
||||
if mtype not in LONG_PLAY_MEDIA.values():
|
||||
scrobble_data.pop("long_play_complete", None)
|
||||
|
||||
scrobble = cls.create(scrobble_data)
|
||||
|
||||
if mtype in LONG_PLAY_MEDIA.values():
|
||||
last_finished = (
|
||||
cls.objects.filter(
|
||||
models.Q(**{key: media}),
|
||||
user_id=user_id,
|
||||
timestamp__lt=scrobble.timestamp,
|
||||
)
|
||||
.order_by("-timestamp")
|
||||
.first()
|
||||
)
|
||||
if last_finished:
|
||||
scrobble.long_play_last_scrobble = last_finished
|
||||
scrobble.save(update_fields=["long_play_last_scrobble"])
|
||||
|
||||
return scrobble
|
||||
|
||||
@classmethod
|
||||
@ -1612,6 +1711,18 @@ class Scrobble(TimeStampedModel):
|
||||
cls,
|
||||
scrobble_data: dict,
|
||||
) -> "Scrobble":
|
||||
if "visibility" not in scrobble_data:
|
||||
user = scrobble_data.get("user")
|
||||
media_type = scrobble_data.get("media_type")
|
||||
override = None
|
||||
if user and media_type:
|
||||
try:
|
||||
profile = user.profile
|
||||
overrides = profile.media_type_visibility or {}
|
||||
override = overrides.get(media_type)
|
||||
except user.__class__.profile.RelatedObjectDoesNotExist:
|
||||
pass
|
||||
scrobble_data["visibility"] = override or Visibility.PRIVATE
|
||||
scrobble = cls.objects.create(**scrobble_data)
|
||||
if not (
|
||||
scrobble.media_type == cls.MediaType.GEO_LOCATION
|
||||
@ -1706,8 +1817,8 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
# Set our playback seconds, and calc long play seconds
|
||||
self.playback_position_seconds = seconds_elapsed
|
||||
if self.previous:
|
||||
past_seconds = self.previous.long_play_seconds or 0
|
||||
if self.long_play_last_scrobble:
|
||||
past_seconds = self.long_play_last_scrobble.long_play_seconds or 0
|
||||
|
||||
self.long_play_seconds = past_seconds + seconds_elapsed
|
||||
|
||||
@ -1768,33 +1879,27 @@ class Scrobble(TimeStampedModel):
|
||||
return False
|
||||
|
||||
def calculate_reading_stats(self, commit=True):
|
||||
page_data = self.log.get("page_data")
|
||||
page_data = self.logdata.page_data
|
||||
|
||||
if page_data:
|
||||
# --- Sort safely by numeric page_number ---
|
||||
def safe_page_number(entry):
|
||||
try:
|
||||
return int(getattr("page_number", entry), 0)
|
||||
except (ValueError, TypeError):
|
||||
return float("inf")
|
||||
|
||||
if isinstance(page_data, dict):
|
||||
logger.warning("Page data is dict, migrate koreader data")
|
||||
return
|
||||
valid_pages = sorted(int(k) for k in page_data.keys())
|
||||
if valid_pages:
|
||||
self.log["page_start"] = min(valid_pages)
|
||||
self.log["page_end"] = max(valid_pages)
|
||||
self.log["pages_read"] = len(set(valid_pages))
|
||||
elif isinstance(page_data, list):
|
||||
valid_pages = []
|
||||
for page in page_data:
|
||||
try:
|
||||
valid_pages.append(int(page["page_number"]))
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
page_data.sort(key=safe_page_number)
|
||||
|
||||
valid_pages = []
|
||||
for page in page_data:
|
||||
try:
|
||||
valid_pages.append(int(page["page_number"]))
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
if valid_pages:
|
||||
self.log["page_start"] = min(valid_pages)
|
||||
self.log["page_end"] = max(valid_pages)
|
||||
self.log["pages_read"] = len(set(valid_pages))
|
||||
if valid_pages:
|
||||
self.log["page_start"] = min(valid_pages)
|
||||
self.log["page_end"] = max(valid_pages)
|
||||
self.log["pages_read"] = len(set(valid_pages))
|
||||
else:
|
||||
page_start = self.log.get("page_start")
|
||||
page_end = self.log.get("page_end")
|
||||
@ -1834,9 +1939,10 @@ class FavoriteMedia(TimeStampedModel):
|
||||
birding_location = models.ForeignKey(
|
||||
BirdingLocation, on_delete=models.CASCADE, **BNULL
|
||||
)
|
||||
media_type = models.CharField(
|
||||
max_length=20, choices=Scrobble.MediaType.choices
|
||||
disc_golf_course = models.ForeignKey(
|
||||
DiscGolfCourse, on_delete=models.CASCADE, **BNULL
|
||||
)
|
||||
media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
|
||||
sent_to_mopidy = models.BooleanField(default=False)
|
||||
|
||||
class Meta:
|
||||
@ -1886,6 +1992,8 @@ class FavoriteMedia(TimeStampedModel):
|
||||
media_obj = self.channel
|
||||
if self.birding_location:
|
||||
media_obj = self.birding_location
|
||||
if self.disc_golf_course:
|
||||
media_obj = self.disc_golf_course
|
||||
return media_obj
|
||||
|
||||
@classmethod
|
||||
@ -1915,6 +2023,7 @@ class FavoriteMedia(TimeStampedModel):
|
||||
"Mood": "mood",
|
||||
"BrickSet": "brick_set",
|
||||
"BirdingLocation": "birding_location",
|
||||
"DiscGolfCourse": "disc_golf_course",
|
||||
}
|
||||
|
||||
fk = fk_map.get(media_type)
|
||||
|
||||
@ -79,6 +79,27 @@ class ScrobbleNtfyNotification(ScrobbleNotification):
|
||||
)
|
||||
|
||||
|
||||
class LastFmImportNtfyNotification(BasicNtfyNotification):
|
||||
def __init__(self, lfm_import, scrobble_count):
|
||||
super().__init__(lfm_import.user.profile)
|
||||
self.ntfy_str = f"Imported {scrobble_count} scrobble(s) from Last.fm"
|
||||
self.click_url = lfm_import.get_absolute_url()
|
||||
self.title = "Last.fm Import Complete"
|
||||
|
||||
def send(self):
|
||||
if self.profile and self.profile.ntfy_enabled and self.profile.ntfy_url:
|
||||
requests.post(
|
||||
self.profile.ntfy_url,
|
||||
data=self.ntfy_str.encode(encoding="utf-8"),
|
||||
headers={
|
||||
"Title": self.title,
|
||||
"Priority": "default",
|
||||
"Tags": "musical_note",
|
||||
"Click": self.click_url,
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
class MoodNtfyNotification(BasicNtfyNotification):
|
||||
def __init__(self, profile, **kwargs):
|
||||
super().__init__(profile)
|
||||
|
||||
@ -9,10 +9,11 @@ import requests
|
||||
from beers.models import Beer
|
||||
from boardgames.models import BoardGame, BoardGameDesigner, BoardGameLocation
|
||||
from books.constants import READCOMICSONLINE_URL
|
||||
from books.models import Book, BookLogData, BookPageLogData
|
||||
from books.models import Book, BookLogData, BookPageLogData, Paper
|
||||
from books.utils import parse_readcomicsonline_uri
|
||||
from bricksets.models import BrickSet
|
||||
from dateutil.parser import parse
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from django.utils import timezone
|
||||
from foods.models import Food
|
||||
from foods.sources.rscraper import RecipeScraperService
|
||||
@ -32,6 +33,7 @@ from scrobbles.constants import (
|
||||
)
|
||||
from scrobbles.models import Scrobble
|
||||
from scrobbles.notifications import ScrobbleNtfyNotification
|
||||
from scrobbles.tasks import push_scrobble_to_archivebox
|
||||
from scrobbles.utils import (
|
||||
convert_to_seconds,
|
||||
extract_domain,
|
||||
@ -329,8 +331,6 @@ def manual_scrobble_book(
|
||||
|
||||
source = READCOMICSONLINE_URL.replace("https://", "")
|
||||
|
||||
# TODO: Check for scrobble of this book already and if so, update the page count
|
||||
|
||||
book = Book.find_or_create(title, url=url, enrich=True)
|
||||
|
||||
scrobble_dict = {
|
||||
@ -357,10 +357,7 @@ def manual_scrobble_book(
|
||||
|
||||
if action == "stop":
|
||||
if url:
|
||||
if isinstance(scrobble.log, "BookLogData"):
|
||||
scrobble.log.resume_url = next_url_if_exists(url)
|
||||
else:
|
||||
scrobble.log["resume_url"] = next_url_if_exists(url)
|
||||
scrobble.log["resume_url"] = next_url_if_exists(url)
|
||||
scrobble.save(update_fields=["log"])
|
||||
scrobble.stop(force_finish=True)
|
||||
|
||||
@ -540,6 +537,7 @@ def email_scrobble_board_game(
|
||||
"playback_position_seconds": duration_seconds,
|
||||
"source": "BG Stats",
|
||||
"log": log_data,
|
||||
"visibility": "private",
|
||||
}
|
||||
|
||||
scrobble = None
|
||||
@ -643,6 +641,8 @@ def manual_scrobble_from_url(
|
||||
item_id = "tt" + str(item_id)
|
||||
elif content_key == "-h" and "twitch.tv" in url:
|
||||
item_id = url
|
||||
elif content_key == "-pp" and "doi.org" in url:
|
||||
item_id = url
|
||||
|
||||
scrobble_fn = MANUAL_SCROBBLE_FNS[content_key]
|
||||
return eval(scrobble_fn)(item_id, user_id, source=source, action=action)
|
||||
@ -997,6 +997,38 @@ def manual_scrobble_task(
|
||||
return scrobble
|
||||
|
||||
|
||||
def manual_scrobble_paper(
|
||||
doi_url: str,
|
||||
user_id: int,
|
||||
source: str = "Bookmarklet",
|
||||
action: Optional[str] = None,
|
||||
):
|
||||
paper = Paper.find_or_create_by_doi(doi_url)
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": timezone.now(),
|
||||
"playback_position_seconds": 0,
|
||||
"source": source,
|
||||
}
|
||||
logger.info(
|
||||
"[vrobbler-scrobble] paper scrobble request received",
|
||||
extra={
|
||||
"paper_id": paper.id,
|
||||
"user_id": user_id,
|
||||
"scrobble_dict": scrobble_dict,
|
||||
"media_type": Scrobble.MediaType.PAPER,
|
||||
},
|
||||
)
|
||||
|
||||
scrobble = Scrobble.create_or_update(paper, user_id, scrobble_dict)
|
||||
|
||||
if action == "stop":
|
||||
scrobble.stop(force_finish=True)
|
||||
|
||||
return scrobble
|
||||
|
||||
|
||||
def manual_scrobble_webpage(
|
||||
url: str,
|
||||
user_id: int,
|
||||
@ -1030,8 +1062,7 @@ def manual_scrobble_webpage(
|
||||
if action == "stop":
|
||||
scrobble.stop(force_finish=True)
|
||||
else:
|
||||
# possibly async this?
|
||||
scrobble.push_to_archivebox()
|
||||
push_scrobble_to_archivebox.delay(scrobble.id)
|
||||
|
||||
return scrobble
|
||||
|
||||
@ -1152,7 +1183,7 @@ def web_scrobbler_scrobble_video_or_song(
|
||||
artist_name = data_dict.get("artist")
|
||||
track_name = data_dict.get("track")
|
||||
tracks = Track.objects.filter(
|
||||
artist__name=data_dict.get("artist"), title=data_dict.get("track")
|
||||
artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
|
||||
)
|
||||
if tracks.count() > 1:
|
||||
logger.warning(
|
||||
@ -1331,3 +1362,32 @@ def manual_scrobble_food(
|
||||
)
|
||||
|
||||
return Scrobble.create_or_update(food, user_id, scrobble_dict)
|
||||
|
||||
|
||||
def manual_scrobble_discgolf(
|
||||
item_id: str,
|
||||
user_id: int,
|
||||
source: str = "Vrobbler",
|
||||
action: Optional[str] = None,
|
||||
):
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
course, _ = DiscGolfCourse.objects.get_or_create(title=item_id.strip())
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": timezone.now(),
|
||||
"playback_position_seconds": 0,
|
||||
"source": source,
|
||||
}
|
||||
|
||||
logger.info(
|
||||
"[scrobblers] manual disc golf scrobble request received",
|
||||
extra={
|
||||
"course_id": course.id,
|
||||
"user_id": user_id,
|
||||
"scrobble_dict": scrobble_dict,
|
||||
},
|
||||
)
|
||||
|
||||
return Scrobble.create_or_update(course, user_id, scrobble_dict)
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
import logging
|
||||
|
||||
from django.core.cache import cache
|
||||
from django.db.models.signals import post_delete, post_save
|
||||
from django.dispatch import receiver
|
||||
from django.utils import timezone
|
||||
@ -9,6 +10,7 @@ from scrobbles.tasks import (
|
||||
add_favorite_to_mopidy_playlist,
|
||||
CHARTABLE_MEDIA_TYPES,
|
||||
remove_favorite_from_mopidy_playlist,
|
||||
reverse_geocode_geolocation,
|
||||
SCROBBLES_WITHOUT_CHARTS,
|
||||
update_charts_for_timestamp,
|
||||
)
|
||||
@ -52,6 +54,11 @@ def _update_charts_for_timestamp(user, ts):
|
||||
if ts is None:
|
||||
return
|
||||
|
||||
lock_key = f"chart_update_{user.id}"
|
||||
if not cache.add(lock_key, "locked", timeout=30):
|
||||
logger.info(f"Chart update already queued for user {user.id}, skipping")
|
||||
return
|
||||
|
||||
if timezone.is_naive(ts):
|
||||
ts = timezone.make_aware(ts)
|
||||
|
||||
@ -80,6 +87,31 @@ def add_tags_from_task_title(sender, instance, **kwargs):
|
||||
instance.tags.add(tag)
|
||||
|
||||
|
||||
@receiver(post_save, sender=Scrobble)
|
||||
def reverse_geocode_on_scrobble_creation(sender, instance, created, **kwargs):
|
||||
if not created:
|
||||
return
|
||||
if not instance.geo_location_id:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: scrobble %s has no geo_location",
|
||||
instance.id,
|
||||
)
|
||||
return
|
||||
if instance.geo_location.postal_code:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: geo_location %s already has postal_code %s",
|
||||
instance.geo_location_id,
|
||||
instance.geo_location.postal_code,
|
||||
)
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Enqueuing reverse geocode for geo_location %s",
|
||||
instance.geo_location_id,
|
||||
)
|
||||
reverse_geocode_geolocation.delay(instance.geo_location_id)
|
||||
|
||||
|
||||
@receiver(post_save, sender=FavoriteMedia)
|
||||
def add_to_mopidy_playlist_on_favorite(sender, instance, created, **kwargs):
|
||||
if not created:
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user