Compare commits
26 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 59e8339e94 | |||
| 9277db97e5 | |||
| e755dc6641 | |||
| 782f5c15d6 | |||
| 2f4fae7d02 | |||
| 4b7c5aa58d | |||
| d4f82f2d6f | |||
| 106d25c20f | |||
| d77caa2783 | |||
| b5bfad73ef | |||
| 274b2704ed | |||
| 80fcb6c002 | |||
| c6f3c90006 | |||
| 387dee7d37 | |||
| 188e899357 | |||
| 30b005fa46 | |||
| 72f739ee5a | |||
| 56ee14512d | |||
| 8c947d35dd | |||
| 61bab1f734 | |||
| 42ce6df9bd | |||
| cbd46df4bc | |||
| e7203cdb9b | |||
| 7246adfeb6 | |||
| a5606951c5 | |||
| 0b4537b7ed |
240
PROJECT.org
240
PROJECT.org
@ -1,6 +1,20 @@
|
||||
#+title: Vrobbler Project
|
||||
|
||||
* Overview
|
||||
Vrobbler began humbly enough as a way to use Jellyfin's webhook to keep track of
|
||||
the shows and movies I was watching. More specifically, I broke my ankle a few
|
||||
days after Christmas in 2022 and spent the next four months very slowly
|
||||
recovering after surgical repair. So once I had the webhook working, and
|
||||
scrobbling videos, it was only a matter of time till I expaned it to mopidy to
|
||||
replicate LastFM. Then I added board games, books via KoReader, sports events,
|
||||
podcasts ... it just keeps going. Vrobbler is now a sort of Frankenstein's
|
||||
monster of scrobbling an entire life.
|
||||
|
||||
I am still unconvinced I can keep this going, but being able to scrobble org
|
||||
tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
|
||||
to be sometimes cathartic and sometimes functional as I try to remember when I
|
||||
did a thing.
|
||||
|
||||
* Features
|
||||
** Beer
|
||||
*** Triggers
|
||||
@ -70,7 +84,6 @@ fetching and simple saving.
|
||||
**** Bookmarklet
|
||||
*** Metadata sources
|
||||
**** Scraper
|
||||
* Release history
|
||||
* Chores
|
||||
** DONE Document various vrobbler features :chore:personal:project:vrobbler:documentation:
|
||||
:PROPERTIES:
|
||||
@ -79,71 +92,8 @@ fetching and simple saving.
|
||||
:LOGBOOK:
|
||||
CLOCK: [2025-07-09 Wed 09:55]--[2025-07-09 Wed 10:15] => 0:20
|
||||
:END:
|
||||
* Backlog [3/23]
|
||||
** TODO [#A] Add classmethod for metadata fetching to tracks :vrobbler:feature:music:personal:project:
|
||||
:PROPERTIES:
|
||||
:ID: bc4b45e5-4c65-13c5-ab7b-1937d3fbf5c2
|
||||
:END:
|
||||
:LOGBOOK:
|
||||
CLOCK: [2025-07-09 Wed 10:15]
|
||||
:END:
|
||||
|
||||
** TODO Add importer class for IMAP imports :vrobbler:feature:imap:importers:project:personal:
|
||||
** TODO Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
|
||||
** TODO [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
|
||||
** TODO [#A] Fix small bug in views for TV series where next episode is now None :vrobbler:bug:personal:videos:
|
||||
|
||||
#+begin_src python
|
||||
ERROR django.request:241 log_response Internal Server Error: /series/c24100d1-da45-4abe-86bf-27cfce9b1f89/
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
|
||||
response = get_response(request)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
|
||||
response = wrapped_callback(request, *callback_args, **callback_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
|
||||
return self.dispatch(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
|
||||
return super().dispatch(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
|
||||
return handler(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/detail.py", line 109, in get
|
||||
context = self.get_context_data(object=self.object)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/videos/views.py", line 33, in get_context_data
|
||||
context_data["next_episode_id"] = "tt" + next_episode_id
|
||||
~~~~~^~~~~~~~~~~~~~~~~
|
||||
TypeError: can only concatenate str (not "NoneType") to str
|
||||
#+end_src
|
||||
|
||||
** TODO [#A] Send periodic check notifications for board games and beer that ask if you're still scrobbling :vrobbler:personal:project:beers:boardgames:notifications:feature:
|
||||
** TODO [#A] Fix raw text webpage title not truncating to 254 chars :vrobbler:personal:bug:webpages:
|
||||
** TODO [#A] Fix koreader scrobble imports to use DST properly :vrobbler:personal:bug:books:imports:
|
||||
The page data has the canonical date something was read in it, but it seems to be an hour off. I traced this back to being off during DST, so we just need the importer to be aware of whether a user is using DST or not and roll back an hour for part of the year.
|
||||
|
||||
Also, we'd need to adjust any old scrobbles that took place with DST off to roll them back by an hour.
|
||||
** TODO [#A] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
|
||||
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
|
||||
Pretty clear, I would love to make trails more useful. Historically I wasn't
|
||||
hiking a lot, which made the source for this a bit silly. But it's clear that
|
||||
AllTrails is the best source, though having TrailForks is nice to.
|
||||
** TODO [#B] Add `garmin_activity_id` to the TrailMetadataLog class :vrobbler:trails:feature:personal:project:
|
||||
Would be nice to have some loose connection to the actual event in my Garmin profile.
|
||||
** TODO [#B] Explore a way to add metadata editing to scrobbles after saving :vrobbler:spike:scrobbling:personal:project:
|
||||
Could be as simple as a JSON form on the scrobble detail page (do I have have one of those yet?).
|
||||
** TODO [#B] Explore a good way to show notes and descriptions from scrobbles to users :personal:project:scrobbling:vrobbler:spike:
|
||||
** TODO [#B] Add webdav syncing to retroarch imports :vrobbler:videogames:webdav:feature:project:personal:
|
||||
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
|
||||
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
|
||||
** TODO [#C] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
|
||||
Rather than pick up an existing Podcast using the podcast title in the mopidy
|
||||
file name, Vrobbler creates a new podcast with no enriched data. Not a big deal
|
||||
for my use as the volume of podcasts I listen to makes manual fixes easy. But
|
||||
it's annoying.
|
||||
* Backlog [3/27]
|
||||
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
|
||||
** TODO [#C] Move to using more robust mopidy-webhooks pacakge form pypi :utility:improvement:
|
||||
:PROPERTIES:
|
||||
:ID: ab31fdc3-359c-1b1d-6b9d-546b476021ba
|
||||
@ -442,6 +392,164 @@ it's annoying.
|
||||
** TODO [#C] Allow users to see tasks on calendar view :vrobbler:personal:project:templates:feature:
|
||||
https://codepen.io/oliviale/pen/QYqybo
|
||||
** TODO [#C] Come up with a possible flow using WebDAV and super-productivity for tasks :personal:feature:project:vrobbler:tasks:
|
||||
** TODO [#B] Add importer class for IMAP imports :vrobbler:feature:imap:importers:project:personal:
|
||||
** TODO [#B] Clean up follow up notifications for board games and beer that ask if you're still scrobbling :vrobbler:personal:project:beers:boardgames:notifications:feature:
|
||||
|
||||
- Note taken on [2025-09-30 Tue 09:32]
|
||||
|
||||
I added this feature in a very rough way, but now we should add "Action"
|
||||
headers so that we can either Finish or Cancel the associated scrobble:
|
||||
|
||||
https://docs.ntfy.sh/publish/#send-http-request
|
||||
|
||||
** TODO [#B] Add AllTrails as a source for Trail data :vrobbler:trails:feature:personal:project:
|
||||
Pretty clear, I would love to make trails more useful. Historically I wasn't
|
||||
hiking a lot, which made the source for this a bit silly. But it's clear that
|
||||
AllTrails is the best source, though having TrailForks is nice to.
|
||||
** TODO [#B] Add `garmin_activity_id` to the TrailMetadataLog class :vrobbler:trails:feature:personal:project:
|
||||
Would be nice to have some loose connection to the actual event in my Garmin profile.
|
||||
** TODO [#B] Explore a way to add metadata editing to scrobbles after saving :vrobbler:spike:scrobbling:personal:project:
|
||||
Could be as simple as a JSON form on the scrobble detail page (do I have have one of those yet?).
|
||||
** TODO [#B] Explore a good way to show notes and descriptions from scrobbles to users :personal:project:scrobbling:vrobbler:spike:
|
||||
** TODO [#B] Add webdav syncing to retroarch imports :vrobbler:videogames:webdav:feature:project:personal:
|
||||
** TODO [#B] Add CSV endpoint for book scrobbles that LibraryThing can ingest :personal:project:books:feature:export:
|
||||
https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-library-thing-can-ingest-6X7QPMRp265xMXqg#comment-6X7QrXq6gJjMP4hg
|
||||
** TODO [#B] Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
|
||||
** TODO [#B] Fix PuzzleLogData has no attribute form :vrobbler:puzzles:personal:project:logdata:
|
||||
** TODO [#B] Scrape ComicBookRoundUp ratings for comic book metadata :vrobbler:books:feature:comicbook:personal:project:
|
||||
|
||||
- Note taken on [2025-09-25 Thu 10:51]
|
||||
|
||||
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
|
||||
** TODO [#B] Add PuzzleLogData class with with_people and completed :vrobbler:feature:puzzles:logdata:personal:project:
|
||||
** TODO [#A] Add classmethod for metadata fetching to tracks :vrobbler:feature:music:personal:project:
|
||||
:PROPERTIES:
|
||||
:ID: bc4b45e5-4c65-13c5-ab7b-1937d3fbf5c2
|
||||
:END:
|
||||
|
||||
** TODO [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
|
||||
** TODO [#A] Fix views for TV series where next episode is now None :vrobbler:bug:personal:videos:
|
||||
|
||||
#+begin_src python
|
||||
ERROR django.request:241 log_response Internal Server Error: /series/c24100d1-da45-4abe-86bf-27cfce9b1f89/
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 55, in inner
|
||||
response = get_response(request)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 197, in _get_response
|
||||
response = wrapped_callback(request, *callback_args, **callback_kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 104, in view
|
||||
return self.dispatch(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/contrib/auth/mixins.py", line 73, in dispatch
|
||||
return super().dispatch(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/base.py", line 143, in dispatch
|
||||
return handler(request, *args, **kwargs)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/django/views/generic/detail.py", line 109, in get
|
||||
context = self.get_context_data(object=self.object)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/videos/views.py", line 33, in get_context_data
|
||||
context_data["next_episode_id"] = "tt" + next_episode_id
|
||||
~~~~~^~~~~~~~~~~~~~~~~
|
||||
TypeError: can only concatenate str (not "NoneType") to str
|
||||
#+end_src
|
||||
|
||||
** TODO [#A] Fix koreader scrobble imports to use DST properly :vrobbler:personal:bug:books:imports:
|
||||
|
||||
- Note taken on [2025-09-25 Thu 10:37] \\
|
||||
|
||||
This may already be fixed ... need to check.
|
||||
|
||||
- Note taken on [2025-02-25 12:34] \\
|
||||
|
||||
The page data has the canonical date something was read in it, but it seems
|
||||
to be an hour off. I traced this back to being off during DST, so we just need
|
||||
the importer to be aware of whether a user is using DST or not and roll back
|
||||
an hour for part of the year. Also, we'd need to adjust any old scrobbles that
|
||||
took place with DST off to roll them back by an hour.
|
||||
|
||||
** TODO [#A] Allow scrobbling from the Food list page's start links :vrobbler:bug:food:scrobbling:personal:project:
|
||||
https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
|
||||
** TODO [#A] Puzzles (and all longplays) should have a "Completed?" column on their detail page :vrobbler:bug:puzzles:personal:project:
|
||||
** TODO [#A] Fix raw text webpage title not truncating to 254 chars :vrobbler:personal:bug:webpages:
|
||||
|
||||
- Note taken on [2025-09-30 Tue 09:33]
|
||||
|
||||
This may have already been resolved ... need to just confirm it.
|
||||
* Version 31.0 [3/3]
|
||||
** DONE [#A] Stop comic book webpage scrobbles from overwriting old scrobbles :vrobbler:personal:bug:books:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 4b2ec068-a281-a88b-c31d-6248d6eb0aa0
|
||||
:END:
|
||||
** DONE [#A] Add page calculation to manually scrobbled books :vrobbler:personal:feature:books:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: b2e313b3-5c35-57e7-8933-627535baf34b
|
||||
:END:
|
||||
** DONE [#A] Fix bug in scrobbling comics where google fails :vrobbler:personal:bug:books:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 9a870c05-6d20-0803-d35d-c03fbe1d0ee1
|
||||
:END:
|
||||
* Version 30.0 [3/3]
|
||||
** DONE [#A] Fix readcomicsonline browsing to update pages :vrobbler:books:feature:comicbook:personal:project:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 981b215a-6473-5fc7-d4cc-51b3eddec4c3
|
||||
:END:
|
||||
** DONE [#B] Redirect webpages back to the original page when starting or stopping :vrobbler:project:webpages:bug:
|
||||
:PROPERTIES:
|
||||
:ID: 6183d03a-452b-51d5-cceb-5bfeada947aa
|
||||
:END:
|
||||
|
||||
** DONE [#B] Fix ComicVine as source for comic book metadata :vrobbler:books:feature:comicbook:personal:project:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: d22cec3f-117f-f203-33a5-efbefa8a5cee
|
||||
:END:
|
||||
* Version 29.0 [1/1]
|
||||
** DONE HOTFIX podcast lookups, final
|
||||
* Version 28.0 [1/1]
|
||||
** DONE HOTFIX podcast lookups
|
||||
* Version 27.0 [3/3]
|
||||
** DONE [#A] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
|
||||
:PROPERTIES:
|
||||
:ID: 7377ef6c-5fa7-9e4e-9080-f9810a76118c
|
||||
:END:
|
||||
|
||||
Rather than pick up an existing Podcast using the podcast title in the mopidy
|
||||
file name, Vrobbler creates a new podcast with no enriched data. Not a big deal
|
||||
for my use as the volume of podcasts I listen to makes manual fixes easy. But
|
||||
it's annoying.
|
||||
|
||||
** DONE [#A] Allow reading comic books from readcomicsoline.ru :vrobbler:books:feature:comicbook:personal:project:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 7c7e9ecc-b675-68c3-764f-ef771ce5d88f
|
||||
:END:
|
||||
|
||||
- Note taken on [2025-09-25 Thu 10:52]
|
||||
|
||||
Things to consider are whether we scrobble the issue on one page, send it to
|
||||
archivebox? (yes), and how best to enrich the data
|
||||
|
||||
** DONE [#A] Add RSS feed lookups to podcasts :vrobbler:personal:feature:podcasts:
|
||||
:PROPERTIES:
|
||||
:ID: d60645b0-7578-97c1-0278-05bd9de4269c
|
||||
:END:
|
||||
|
||||
- Note taken on [2025-10-14 Tue 10:08]
|
||||
|
||||
Turns out the Podcast plugin for mopidy does a pretty good job of showing the
|
||||
latest file without having to scroll the bottom using only Muse to not parse
|
||||
the podcast title name. BUT, now we're getting urls like this:
|
||||
|
||||
https://nsf.libsyn.com/rss#77e01251-cb20-4609-b577-d48e985d2e7b
|
||||
|
||||
This is great, because there's more context there, but it has to read out of
|
||||
the RSS feed. We should add a check in the podcast util to sniff out the file
|
||||
referenced in the # in that url and populate the info from there. This should
|
||||
actually be much more reliable than the current state of the podcast lookup
|
||||
which depends on the file to be name properly.
|
||||
|
||||
* Version 26.0 [3/3]
|
||||
** DONE Clean up templates for scrobble details :vrobbler:personal:bug:templates:
|
||||
:PROPERTIES:
|
||||
|
||||
42
poetry.lock
generated
42
poetry.lock
generated
@ -1602,6 +1602,21 @@ files = [
|
||||
[package.extras]
|
||||
devel = ["colorama", "json-spec", "jsonschema", "pylint", "pytest", "pytest-benchmark", "pytest-cache", "validictory"]
|
||||
|
||||
[[package]]
|
||||
name = "feedparser"
|
||||
version = "6.0.12"
|
||||
description = "Universal feed parser, handles RSS 0.9x, RSS 1.0, RSS 2.0, CDF, Atom 0.3, and Atom 1.0 feeds"
|
||||
optional = false
|
||||
python-versions = ">=3.6"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "feedparser-6.0.12-py3-none-any.whl", hash = "sha256:6bbff10f5a52662c00a2e3f86a38928c37c48f77b3c511aedcd51de933549324"},
|
||||
{file = "feedparser-6.0.12.tar.gz", hash = "sha256:64f76ce90ae3e8ef5d1ede0f8d3b50ce26bcce71dd8ae5e82b1cd2d4a5f94228"},
|
||||
]
|
||||
|
||||
[package.dependencies]
|
||||
sgmllib3k = "*"
|
||||
|
||||
[[package]]
|
||||
name = "filelock"
|
||||
version = "3.18.0"
|
||||
@ -4403,6 +4418,17 @@ enabler = ["pytest-enabler (>=2.2)"]
|
||||
test = ["build[virtualenv] (>=1.0.3)", "filelock (>=3.4.0)", "ini2toml[lite] (>=0.14)", "jaraco.develop (>=7.21) ; python_version >= \"3.9\" and sys_platform != \"cygwin\"", "jaraco.envs (>=2.2)", "jaraco.path (>=3.7.2)", "jaraco.test (>=5.5)", "packaging (>=24.2)", "pip (>=19.1)", "pyproject-hooks (!=1.1)", "pytest (>=6,!=8.1.*)", "pytest-home (>=0.5)", "pytest-perf ; sys_platform != \"cygwin\"", "pytest-subprocess", "pytest-timeout", "pytest-xdist (>=3)", "tomli-w (>=1.0.0)", "virtualenv (>=13.0.0)", "wheel (>=0.44.0)"]
|
||||
type = ["importlib_metadata (>=7.0.2) ; python_version < \"3.10\"", "jaraco.develop (>=7.21) ; sys_platform != \"cygwin\"", "mypy (==1.14.*)", "pytest-mypy"]
|
||||
|
||||
[[package]]
|
||||
name = "sgmllib3k"
|
||||
version = "1.0.0"
|
||||
description = "Py3k port of sgmllib."
|
||||
optional = false
|
||||
python-versions = "*"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "sgmllib3k-1.0.0.tar.gz", hash = "sha256:7868fb1c8bfa764c1ac563d3cf369c381d1325d36124933a726f29fcdaa812e9"},
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "shellingham"
|
||||
version = "1.5.4"
|
||||
@ -4730,6 +4756,20 @@ webencodings = ">=0.4"
|
||||
doc = ["sphinx", "sphinx_rtd_theme"]
|
||||
test = ["pytest", "ruff"]
|
||||
|
||||
[[package]]
|
||||
name = "titlecase"
|
||||
version = "2.4.1"
|
||||
description = "Python Port of John Gruber's titlecase.pl"
|
||||
optional = false
|
||||
python-versions = ">=3.7"
|
||||
groups = ["main"]
|
||||
files = [
|
||||
{file = "titlecase-2.4.1.tar.gz", hash = "sha256:7d83a277ccbbda11a2944e78a63e5ccaf3d32f828c594312e4862f9a07f635f5"},
|
||||
]
|
||||
|
||||
[package.extras]
|
||||
regex = ["regex (>=2020.4.4)"]
|
||||
|
||||
[[package]]
|
||||
name = "tld"
|
||||
version = "0.13"
|
||||
@ -5499,4 +5539,4 @@ cffi = ["cffi (>=1.11)"]
|
||||
[metadata]
|
||||
lock-version = "2.1"
|
||||
python-versions = ">=3.9,<3.12"
|
||||
content-hash = "3a483aefea0a3afebf187b17b7df72a158788024ca8121b512b39567fb5ec8ca"
|
||||
content-hash = "2e297ef6f8c524840a381ad793946c87b601d81afd569e882fe58120a5f84626"
|
||||
|
||||
@ -56,6 +56,8 @@ poetry-bumpversion = "^0.3.3"
|
||||
orgparse = "^0.4.20250520"
|
||||
tmdbv3api = "^1.9.0"
|
||||
themoviedb = "^1.0.2"
|
||||
feedparser = "^6.0.12"
|
||||
titlecase = "^2.4.1"
|
||||
|
||||
[tool.poetry.group.test]
|
||||
optional = true
|
||||
|
||||
@ -115,6 +115,12 @@ def mopidy_podcast_request_data():
|
||||
mopidy_uri=mopidy_uri, artist="NPR", album="Up First"
|
||||
).request_json
|
||||
|
||||
@pytest.fixture
|
||||
def mopidy_podcast_https_request_data():
|
||||
mopidy_uri = "podcast+https://feeds.npr.org/510318/podcast.xml#85b9c4c4-ae09-43d9-8853-31ccf43f68e6"
|
||||
return MopidyRequest(
|
||||
mopidy_uri=mopidy_uri, artist="NPR", album="Up First"
|
||||
).request_json
|
||||
|
||||
class JellyfinTrackRequest:
|
||||
name = "Emotion"
|
||||
|
||||
@ -5,3 +5,5 @@ BOOKS_TITLES_TO_IGNORE = [
|
||||
"zb2rhkSwygt9vjkAEBj7tP5KVgFqejJqsJ2W3bYsrgiiKK8XL",
|
||||
"zb2rhchGpo7P27mofV9hYjT63d9ZaQnbQ6LSfzmkvsYzvARif",
|
||||
]
|
||||
|
||||
READCOMICSONLINE_URL = "https://readcomicsonline.ru"
|
||||
|
||||
@ -0,0 +1,33 @@
|
||||
# Generated by Django 4.2.19 on 2025-10-20 18:35
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
('books', '0028_delete_page'),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name='book',
|
||||
name='comicvine_id',
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='book',
|
||||
name='issue_number',
|
||||
field=models.IntegerField(blank=True, max_length=5, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='book',
|
||||
name='original_title',
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name='book',
|
||||
name='volume_number',
|
||||
field=models.IntegerField(blank=True, max_length=5, null=True),
|
||||
),
|
||||
]
|
||||
@ -26,7 +26,7 @@ from scrobbles.mixins import (
|
||||
from scrobbles.utils import get_scrobbles_for_media
|
||||
from taggit.managers import TaggableManager
|
||||
from thefuzz import fuzz
|
||||
from vrobbler.apps.books.comicvine import (
|
||||
from vrobbler.apps.books.sources.comicvine import (
|
||||
ComicVineClient,
|
||||
lookup_comic_from_comicvine,
|
||||
)
|
||||
@ -135,6 +135,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
)
|
||||
|
||||
title = models.CharField(max_length=255)
|
||||
original_title = models.CharField(max_length=255, **BNULL)
|
||||
authors = models.ManyToManyField(Author, blank=True)
|
||||
koreader_data_by_hash = models.JSONField(**BNULL)
|
||||
isbn_13 = models.CharField(max_length=255, **BNULL)
|
||||
@ -145,6 +146,11 @@ class Book(LongPlayScrobblableMixin):
|
||||
publish_date = models.DateField(**BNULL)
|
||||
publisher = models.CharField(max_length=255, **BNULL)
|
||||
first_sentence = models.TextField(**BNULL)
|
||||
# ComicVine
|
||||
comicvine_id = models.CharField(max_length=255, **BNULL)
|
||||
issue_number = models.IntegerField(max_length=5, **BNULL)
|
||||
volume_number = models.IntegerField(max_length=5, **BNULL)
|
||||
# OpenLibrary
|
||||
openlibrary_id = models.CharField(max_length=255, **BNULL)
|
||||
cover = models.ImageField(upload_to="books/covers/", **BNULL)
|
||||
cover_small = ImageSpecField(
|
||||
@ -188,6 +194,43 @@ class Book(LongPlayScrobblableMixin):
|
||||
def get_absolute_url(self):
|
||||
return reverse("books:book_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@classmethod
|
||||
def get_from_comicvine(cls, title: str, overwrite: bool = False, force_new: bool =False) -> "Book":
|
||||
book, created = cls.objects.get_or_create(title=title)
|
||||
if not created and not overwrite and not force_new:
|
||||
book, created = cls.objects.get_or_create(original_title=title)
|
||||
logger.info("Found comic by original title, use force_new=True to override")
|
||||
return book
|
||||
|
||||
book_dict = lookup_comic_from_comicvine(title)
|
||||
|
||||
if created or overwrite:
|
||||
author_list = []
|
||||
author_dicts = book_dict.pop("author_dicts")
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
if author_dict.get("authorId"):
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
semantic_id=author_dict.get("authorId")
|
||||
)
|
||||
author_list.append(author)
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
# TODO enrich author?
|
||||
...
|
||||
|
||||
for k, v in book_dict.items():
|
||||
setattr(book, k, v)
|
||||
book.save()
|
||||
|
||||
if author_list:
|
||||
book.authors.add(*author_list)
|
||||
genres = book_dict.pop("genres", [])
|
||||
if genres:
|
||||
book.genre.add(*genres)
|
||||
return book
|
||||
|
||||
@classmethod
|
||||
def find_or_create(
|
||||
cls, title: str, enrich: bool = False, commit: bool = True
|
||||
@ -215,14 +258,13 @@ class Book(LongPlayScrobblableMixin):
|
||||
return book
|
||||
|
||||
book_dict = lookup_book_from_google(title)
|
||||
if not book_dict or book_dict.get("isbn_10"):
|
||||
book_dict = lookup_comic_from_comicvine(title)
|
||||
|
||||
author_list = []
|
||||
authors = book_dict.pop("authors")
|
||||
cover_url = book_dict.pop("cover_url")
|
||||
try:
|
||||
genres = book_dict.pop("generes")
|
||||
except:
|
||||
genres = []
|
||||
authors = book_dict.pop("authors", [])
|
||||
cover_url = book_dict.pop("cover_url", "")
|
||||
genres = book_dict.pop("generes", [])
|
||||
|
||||
if authors:
|
||||
for author_str in authors:
|
||||
@ -248,7 +290,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
return book
|
||||
|
||||
def save_image_from_url(self, url: str, force_update: bool = False):
|
||||
if not self.cover or (force_update and url):
|
||||
if url and (not self.cover or force_update):
|
||||
r = requests.get(url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{self.title}_{self.uuid}.jpg"
|
||||
|
||||
@ -3,7 +3,6 @@ ComicVine API Information & Documentation:
|
||||
https://comicvine.gamespot.com/api/
|
||||
https://comicvine.gamespot.com/api/documentation
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
from django.conf import settings
|
||||
|
||||
@ -200,34 +199,71 @@ class ComicVineClient(object):
|
||||
|
||||
|
||||
def lookup_comic_from_comicvine(title: str) -> dict:
|
||||
original_title = title
|
||||
|
||||
issue_number = None
|
||||
volume_nubmer = None
|
||||
resource_type = "issue"
|
||||
if "Issue " in title:
|
||||
resource_type = "issue"
|
||||
issue_number = title.split("Issue ")[1]
|
||||
volume_number = None
|
||||
if "Volume " in title:
|
||||
resource_type = "volume"
|
||||
volume_number = title.split("Volume ")[1]
|
||||
|
||||
api_key = getattr(settings, "COMICVINE_API_KEY", "")
|
||||
if not api_key:
|
||||
logger.warn("No ComicVine API key configured, not looking anything up")
|
||||
logger.warning("No ComicVine API key configured, not looking anything up")
|
||||
return {}
|
||||
|
||||
client = ComicVineClient(
|
||||
api_key=getattr(settings, "COMICVINE_API_KEY", None)
|
||||
)
|
||||
result = [
|
||||
r
|
||||
for r in client.search(title).get("results")
|
||||
if r.get("resource_type") == "volume"
|
||||
][0]
|
||||
|
||||
if "volume" not in result.keys():
|
||||
logger.warn("No result found on ComicVine", extra={"title": title})
|
||||
raw_results = client.search(title).get("results")
|
||||
results = [
|
||||
r
|
||||
for r in raw_results
|
||||
if r.get("resource_type") == resource_type
|
||||
]
|
||||
print(results)
|
||||
if not results:
|
||||
logger.warning("No comic found on ComicVine")
|
||||
return {}
|
||||
|
||||
title = " ".join([result.get("volume").get("name"), result.get("name)")])
|
||||
found_result = None
|
||||
for result in results:
|
||||
print("checking ", result.get("issue_number"), " to ", str(issue_number))
|
||||
if result.get("issue_number") == str(issue_number):
|
||||
found_result = result
|
||||
break
|
||||
if result.get("volume_number") == str(volume_number):
|
||||
found_result = result
|
||||
break
|
||||
|
||||
if not found_result:
|
||||
found_result = results[0]
|
||||
|
||||
logger.info("ComicVine results", extra={"results": results})
|
||||
|
||||
if not found_result:
|
||||
logger.warning("No matches found on ComicVine")
|
||||
return {}
|
||||
|
||||
title = found_result.get("name")
|
||||
|
||||
if found_result.get("volume"):
|
||||
title = found_result.get("volume").get("name")
|
||||
|
||||
data_dict = {
|
||||
"title": title,
|
||||
"cover_url": result.get("image").get("original_url"),
|
||||
"comicvine_data": {
|
||||
"id": result.get("id"),
|
||||
"site_detail_url": result.get("site_detail_url"),
|
||||
"description": result.get("description"),
|
||||
"image": result.get("image").get("original_url"),
|
||||
},
|
||||
"original_title": original_title,
|
||||
"issue_number": found_result.get("issue_number"),
|
||||
"volume_number": found_result.get("volume_number"),
|
||||
"cover_url": found_result.get("image").get("original_url"),
|
||||
"comicvine_id": found_result.get("id"),
|
||||
"comicvine_data": found_result,
|
||||
}
|
||||
|
||||
return data_dict
|
||||
@ -29,6 +29,9 @@ def lookup_book_from_google(title: str) -> dict:
|
||||
google_result = (
|
||||
json.loads(response.content).get("items", [{}])[0].get("volumeInfo")
|
||||
)
|
||||
if not google_result:
|
||||
return {}
|
||||
|
||||
publish_date = pendulum.parse(google_result.get("publishedDate"))
|
||||
|
||||
isbn_13 = ""
|
||||
@ -59,13 +62,15 @@ def lookup_book_from_google(title: str) -> dict:
|
||||
book_dict["genres"] = google_result.get("categories")
|
||||
book_dict["cover_url"] = (
|
||||
google_result.get("imageLinks", {})
|
||||
.get("thumbnail")
|
||||
.get("thumbnail", "")
|
||||
.replace("zoom=1", "zoom=15")
|
||||
.replace("&edge=curl", "")
|
||||
)
|
||||
|
||||
book_dict["run_time_seconds"] = book_dict.get("pages", 10) * getattr(
|
||||
settings, "AVERAGE_PAGE_READING_SECONDS", 60
|
||||
)
|
||||
book_dict["run_time_seconds"] = 3600
|
||||
if book_dict.get("pages"):
|
||||
book_dict["run_time_seconds"] = book_dict.get("pages", 10) * getattr(
|
||||
settings, "AVERAGE_PAGE_READING_SECONDS", 60
|
||||
)
|
||||
|
||||
return book_dict
|
||||
|
||||
21
vrobbler/apps/books/utils.py
Normal file
21
vrobbler/apps/books/utils.py
Normal file
@ -0,0 +1,21 @@
|
||||
from titlecase import titlecase
|
||||
|
||||
def parse_readcomicsonline_uri(uri: str) -> tuple:
|
||||
try:
|
||||
path = uri.split("comic/")[1]
|
||||
except IndexError:
|
||||
return "", "", ""
|
||||
|
||||
parts = path.split('/')
|
||||
title = ""
|
||||
volume = 1
|
||||
page = 1
|
||||
if len(parts) == 2:
|
||||
title = titlecase(parts[0].replace("-", " "))
|
||||
volume = parts[1]
|
||||
if len(parts) == 3:
|
||||
title = titlecase(parts[0].replace("-", " "))
|
||||
volume = parts[1]
|
||||
page = parts[2]
|
||||
|
||||
return title, volume, page
|
||||
@ -143,43 +143,46 @@ class PodcastEpisode(ScrobblableMixin):
|
||||
def find_or_create(
|
||||
cls,
|
||||
title: str,
|
||||
podcast_name: str,
|
||||
pub_date: str,
|
||||
number: int = 0,
|
||||
mopidy_uri: str = "",
|
||||
producer_name: str = "",
|
||||
episode_num: int = 0,
|
||||
run_time_seconds: int = 1800,
|
||||
mopidy_uri: str = "",
|
||||
podcast_name: str = "",
|
||||
podcast_producer: str = "",
|
||||
podcast_description: str = "",
|
||||
enrich: bool = True,
|
||||
) -> "PodcastEpisode":
|
||||
"""Given a data dict from Mopidy, finds or creates a podcast and
|
||||
producer before saving the epsiode so it can be scrobbled.
|
||||
|
||||
"""
|
||||
log_context={"mopidy_uri": mopidy_uri, "media_type": "Podcast"}
|
||||
producer = None
|
||||
if producer_name:
|
||||
producer = Producer.find_or_create(producer_name)
|
||||
if podcast_producer:
|
||||
producer = Producer.find_or_create(podcast_producer)
|
||||
|
||||
podcast = Podcast.objects.filter(
|
||||
name__iexact=podcast_name,
|
||||
).first()
|
||||
if not podcast:
|
||||
podcast = Podcast.objects.create(
|
||||
name=podcast_name, producer=producer
|
||||
)
|
||||
if enrich:
|
||||
podcast.fix_metadata()
|
||||
podcast, created = Podcast.objects.get_or_create(name=podcast_name, defaults={"description": podcast_description})
|
||||
log_context["podcast_id"] = podcast.id
|
||||
log_context["podcast_name"] = podcast.name
|
||||
if created:
|
||||
logger.info("Created new podcast", extra=log_context)
|
||||
if enrich and created:
|
||||
logger.info("Enriching new podcast", extra=log_context)
|
||||
podcast.fix_metadata()
|
||||
|
||||
episode = cls.objects.filter(
|
||||
title__iexact=title, podcast=podcast
|
||||
).first()
|
||||
if not episode:
|
||||
episode = cls.objects.create(
|
||||
title=title,
|
||||
podcast=podcast,
|
||||
run_time_seconds=run_time_seconds,
|
||||
number=number,
|
||||
pub_date=pub_date,
|
||||
mopidy_uri=mopidy_uri,
|
||||
)
|
||||
episode, created = cls.objects.get_or_create(
|
||||
title=title,
|
||||
podcast=podcast,
|
||||
defaults={
|
||||
"run_time_seconds": run_time_seconds,
|
||||
"number": episode_num,
|
||||
"pub_date": pub_date,
|
||||
"mopidy_uri": mopidy_uri,
|
||||
}
|
||||
)
|
||||
if created:
|
||||
log_context["episode_id"] = episode.id
|
||||
log_context["episode_title"] = episode.title
|
||||
logger.info("Created new podcast episode", extra=log_context)
|
||||
|
||||
return episode
|
||||
|
||||
@ -1,7 +1,10 @@
|
||||
import logging
|
||||
import os
|
||||
from typing import Any
|
||||
from urllib.parse import unquote
|
||||
|
||||
import feedparser
|
||||
import requests
|
||||
from dateutil.parser import ParserError, parse
|
||||
from podcasts.models import PodcastEpisode
|
||||
|
||||
@ -10,26 +13,95 @@ logger = logging.getLogger(__name__)
|
||||
# TODO This should be configurable in settings or per deploy
|
||||
PODCAST_DATE_FORMAT = "YYYY-MM-DD"
|
||||
|
||||
def parse_duration(d):
|
||||
if not d:
|
||||
return None
|
||||
if d.isdigit():
|
||||
return int(d)
|
||||
parts = [int(p) for p in d.split(":")]
|
||||
while len(parts) < 3:
|
||||
parts.insert(0, 0)
|
||||
h, m, s = parts
|
||||
return h * 3600 + m * 60 + s
|
||||
|
||||
def fetch_metadata_from_rss(uri: str) -> dict[str, Any]:
|
||||
log_context = {"mopidy_uri": uri, "media_type": "Podcast"}
|
||||
podcast_data: dict[str, Any] = {}
|
||||
|
||||
rss_url = uri.split("#")[0].split("podcast+")[1]
|
||||
target_guid = uri.split("#")[1]
|
||||
|
||||
log_context["rss_url"] = rss_url
|
||||
log_context["target_guid"] = target_guid
|
||||
|
||||
try:
|
||||
resp = requests.get(rss_url, timeout=10)
|
||||
feed = feedparser.parse(resp.text)
|
||||
except IndexError:
|
||||
logger.warning("Tried to parse uri as RSS feed, but no target found", extra=log_context)
|
||||
return podcast_data
|
||||
|
||||
podcast_publisher = getattr(feed.feed, "itunes_publisher", "")
|
||||
try:
|
||||
podcast_owner = feed.feed.itunes_owner.get("name") if isinstance(feed.feed.itunes_owner, dict) else feed.feed.itunes_owner
|
||||
podcast_other = feed.feed.get("managingeditor") or feed.feed.get("copyright")
|
||||
except AttributeError:
|
||||
podcast_owner = None
|
||||
podcast_other = None
|
||||
|
||||
podcast_data = {
|
||||
"podcast_name": getattr(feed.feed, "title", ""),
|
||||
# "podcast_description": getattr(feed.feed, "description", ""),
|
||||
# "podcast_link": getattr(feed.feed, "link", ""),
|
||||
"podcast_producer": podcast_publisher or podcast_owner or podcast_other
|
||||
}
|
||||
|
||||
for entry in feed.entries:
|
||||
if entry.get("guid") == target_guid:
|
||||
logger.info("🎧 Episode found in RSS feed", extra=log_context)
|
||||
podcast_data["title"] = entry.title
|
||||
podcast_data["episode_num"] = int(entry.get("itunes_episode", 0))
|
||||
podcast_data["pub_date"] = parse(entry.get("published", None))
|
||||
podcast_data["run_time_seconds"] = parse_duration(entry.get("itunes_duration", None))
|
||||
# podcast_data["description"] = entry.get("description", None)
|
||||
# podcast_data["episode_url"] = entry.enclosures[0].href if entry.get("enclosures") else None
|
||||
return podcast_data
|
||||
else:
|
||||
logger.info("Episode not found in RSS feed.")
|
||||
|
||||
|
||||
def parse_mopidy_uri(uri: str) -> dict[str, Any]:
|
||||
podcast_data: dict[str, Any] = {}
|
||||
|
||||
def parse_mopidy_uri(uri: str) -> dict:
|
||||
logger.debug(f"Parsing URI: {uri}")
|
||||
if "podcast+https" in uri:
|
||||
return fetch_metadata_from_rss(uri)
|
||||
|
||||
|
||||
parsed_uri = os.path.splitext(unquote(uri))[0].split("/")
|
||||
|
||||
podcast_data = {
|
||||
"title": parsed_uri[-1],
|
||||
"episode_num": None,
|
||||
"podcast_name": parsed_uri[-2].strip(),
|
||||
"pub_date": None,
|
||||
}
|
||||
|
||||
|
||||
episode_str = parsed_uri[-1]
|
||||
podcast_name = parsed_uri[-2].strip()
|
||||
episode_num = None
|
||||
episode_num_pad = 0
|
||||
|
||||
try:
|
||||
# Without episode numbers the date will lead
|
||||
pub_date = parse(episode_str[0:10])
|
||||
podcast_data["pub_date"] = parse(episode_str[0:10])
|
||||
except ParserError:
|
||||
episode_num = int(episode_str.split("-")[0])
|
||||
episode_num_pad = len(str(episode_num)) + 1
|
||||
podcast_data["episode_num"] = int(episode_str.split("-")[0])
|
||||
episode_num_pad = len(str(podcast_data["episode_num"])) + 1
|
||||
|
||||
try:
|
||||
# Beacuse we have epsiode numbers on
|
||||
pub_date = parse(
|
||||
podcast_data["pub_date"] = parse(
|
||||
episode_str[
|
||||
episode_num_pad : len(PODCAST_DATE_FORMAT)
|
||||
+ episode_num_pad
|
||||
@ -39,41 +111,19 @@ def parse_mopidy_uri(uri: str) -> dict:
|
||||
pub_date = ""
|
||||
|
||||
gap_to_strip = 0
|
||||
if pub_date:
|
||||
if podcast_data["pub_date"]:
|
||||
gap_to_strip += len(PODCAST_DATE_FORMAT)
|
||||
if episode_num:
|
||||
if podcast_data["episode_num"]:
|
||||
gap_to_strip += episode_num_pad
|
||||
|
||||
episode_name = episode_str[gap_to_strip:].replace("-", " ").strip()
|
||||
podcast_data["title"] = episode_str[gap_to_strip:].replace("-", " ").strip()
|
||||
|
||||
return {
|
||||
"episode_filename": episode_name,
|
||||
"episode_num": episode_num,
|
||||
"podcast_name": podcast_name,
|
||||
"pub_date": pub_date,
|
||||
}
|
||||
return podcast_data
|
||||
|
||||
|
||||
def get_or_create_podcast(post_data: dict) -> PodcastEpisode:
|
||||
logger.info("Looking up podcast", extra={"post_data": post_data, "media_type": "Podcast"})
|
||||
mopidy_uri = post_data.get("mopidy_uri", "")
|
||||
parsed_data = parse_mopidy_uri(mopidy_uri)
|
||||
|
||||
producer_dict = {"name": post_data.get("artist")}
|
||||
|
||||
podcast_name = post_data.get("album")
|
||||
if not podcast_name:
|
||||
podcast_name = parsed_data.get("podcast_name")
|
||||
podcast_dict = {"name": podcast_name}
|
||||
|
||||
episode_name = parsed_data.get("episode_filename")
|
||||
episode_dict = {
|
||||
"title": episode_name,
|
||||
"run_time_seconds": post_data.get("run_time"),
|
||||
"number": parsed_data.get("episode_num"),
|
||||
"pub_date": parsed_data.get("pub_date"),
|
||||
"mopidy_uri": mopidy_uri,
|
||||
}
|
||||
|
||||
return PodcastEpisode.find_or_create(
|
||||
podcast_dict, producer_dict, episode_dict
|
||||
)
|
||||
return PodcastEpisode.find_or_create(**parsed_data)
|
||||
|
||||
@ -18,6 +18,8 @@ PLAY_AGAIN_MEDIA = {
|
||||
"bricksets": "BrickSet",
|
||||
"trails": "Trail",
|
||||
"beers": "Beer",
|
||||
"foods": "Food",
|
||||
"locations": "GeoLocation",
|
||||
}
|
||||
|
||||
MEDIA_END_PADDING_SECONDS = {
|
||||
@ -35,6 +37,7 @@ SCROBBLE_CONTENT_URLS = {
|
||||
"-t": ["https://app.todoist.com/app/task/{id}"],
|
||||
"-p": ["https://www.ipdb.plus/IPDb/puzzle.php?id="],
|
||||
"-l": ["https://brickset.com/sets/"],
|
||||
"-c": ["https://readcomicsonline.ru"],
|
||||
}
|
||||
|
||||
EXCLUDE_FROM_NOW_PLAYING = ("GeoLocation",)
|
||||
@ -50,6 +53,7 @@ MANUAL_SCROBBLE_FNS = {
|
||||
"-t": "manual_scrobble_task",
|
||||
"-p": "manual_scrobble_puzzle",
|
||||
"-l": "manual_scrobble_brickset",
|
||||
"-c": "manual_scrobble_book",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -12,7 +12,7 @@ import pytz
|
||||
from beers.models import Beer
|
||||
from boardgames.models import BoardGame
|
||||
from books.koreader import process_koreader_sqlite_file
|
||||
from books.models import Book, Paper
|
||||
from books.models import Book, Paper, BookPageLogData, BookLogData
|
||||
from bricksets.models import BrickSet
|
||||
from dataclass_wizard.errors import ParseError
|
||||
from django.conf import settings
|
||||
@ -775,7 +775,7 @@ class Scrobble(TimeStampedModel):
|
||||
and user.profile.redirect_to_webpage
|
||||
):
|
||||
logger.info(f"Redirecting to {self.media_obj.url}")
|
||||
redirect_url = self.media_obj.get_read_url()
|
||||
redirect_url = self.media_obj.url
|
||||
|
||||
if (
|
||||
self.media_type == self.MediaType.VIDEO
|
||||
@ -991,7 +991,7 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
@property
|
||||
def can_be_updated(self) -> bool:
|
||||
if self.media_obj.__class__.__name__ in LONG_PLAY_MEDIA.values():
|
||||
if self.media_obj.__class__.__name__ in LONG_PLAY_MEDIA.values() and self.source != "readcomicsonline.ru":
|
||||
logger.info(
|
||||
"[scrobbling] cannot be updated, long play media",
|
||||
extra={
|
||||
@ -1131,6 +1131,8 @@ class Scrobble(TimeStampedModel):
|
||||
media_query = models.Q(**{key: media})
|
||||
scrobble_data[key + "_id"] = media.id
|
||||
skip_in_progress_check = kwargs.get("skip_in_progress_check", False)
|
||||
read_log_page = kwargs.get("read_log_page", None)
|
||||
|
||||
|
||||
# Find our last scrobble of this media item (track, video, etc)
|
||||
scrobble = (
|
||||
@ -1154,7 +1156,7 @@ class Scrobble(TimeStampedModel):
|
||||
)
|
||||
return scrobble
|
||||
|
||||
if not skip_in_progress_check:
|
||||
if not skip_in_progress_check or read_log_page:
|
||||
logger.info(
|
||||
f"[create_or_update] check for existing scrobble to update ",
|
||||
extra={
|
||||
@ -1170,15 +1172,35 @@ class Scrobble(TimeStampedModel):
|
||||
# If it's marked as stopped, send it through our update mechanism, which will complete it
|
||||
if scrobble and (
|
||||
scrobble.can_be_updated
|
||||
or (read_log_page and scrobble.can_be_updated)
|
||||
or scrobble_data["playback_status"] == "stopped"
|
||||
):
|
||||
if "log" in scrobble_data.keys() and scrobble.log:
|
||||
if read_log_page:
|
||||
page_list = scrobble.log.get("page_data", [])
|
||||
if page_list:
|
||||
for page in page_list:
|
||||
if not page.get("end_ts", None):
|
||||
page["end_ts"] = int(timezone.now().timestamp())
|
||||
page["duration"] = page["end_ts"] - page.get("start_ts")
|
||||
|
||||
page_list.append(
|
||||
BookPageLogData(
|
||||
page_number=read_log_page,
|
||||
start_ts=int(timezone.now().timestamp())
|
||||
)
|
||||
)
|
||||
scrobble.log["page_data"] = page_list
|
||||
scrobble.save(update_fields=["log"])
|
||||
elif "log" in scrobble_data.keys() and scrobble.log:
|
||||
scrobble_data["log"] = scrobble.log | scrobble_data["log"]
|
||||
return scrobble.update(scrobble_data)
|
||||
|
||||
# Discard status before creating
|
||||
scrobble_data.pop("playback_status")
|
||||
|
||||
if read_log_page:
|
||||
scrobble_data["log"] = BookLogData(page_data=[BookPageLogData(page_number=read_log_page, start_ts=int(timezone.now().timestamp()))])
|
||||
|
||||
logger.info(
|
||||
f"[scrobbling] creating new scrobble",
|
||||
extra={
|
||||
@ -1371,6 +1393,9 @@ class Scrobble(TimeStampedModel):
|
||||
if class_name in LONG_PLAY_MEDIA.values():
|
||||
self.finish_long_play()
|
||||
|
||||
if class_name == "Book":
|
||||
self.calculate_reading_stats()
|
||||
|
||||
logger.info(
|
||||
f"[scrobbling] stopped",
|
||||
extra={
|
||||
@ -1466,3 +1491,40 @@ class Scrobble(TimeStampedModel):
|
||||
beyond_completion = False
|
||||
|
||||
return beyond_completion
|
||||
|
||||
def calculate_reading_stats(self, commit=True):
|
||||
# --- Sort safely by numeric page_number ---
|
||||
def safe_page_number(entry):
|
||||
try:
|
||||
return int(getattr("page_number", entry), 0)
|
||||
except (ValueError, TypeError):
|
||||
return float("inf") # push invalid entries to the end
|
||||
|
||||
page_data = self.log.get("page_data")
|
||||
|
||||
if not page_data:
|
||||
logger.warning("No page data found to calculate")
|
||||
return
|
||||
|
||||
if isinstance(page_data, dict):
|
||||
logger.warning("Page data is dict, migrate koreader data")
|
||||
return
|
||||
|
||||
page_data.sort(key=safe_page_number)
|
||||
|
||||
# --- Extract valid numeric page numbers ---
|
||||
valid_pages = []
|
||||
for page in page_data:
|
||||
try:
|
||||
valid_pages.append(int(page["page_number"]))
|
||||
except (ValueError, TypeError):
|
||||
continue
|
||||
|
||||
# --- Compute stats ---
|
||||
if valid_pages:
|
||||
self.log["page_start"] = min(valid_pages)
|
||||
self.log["page_end"] = max(valid_pages)
|
||||
self.log["pages_read"] = len(set(valid_pages))
|
||||
|
||||
if commit:
|
||||
self.save(update_fields=["log"])
|
||||
|
||||
@ -7,7 +7,9 @@ import pendulum
|
||||
import pytz
|
||||
from beers.models import Beer
|
||||
from boardgames.models import BoardGame, BoardGameDesigner, BoardGameLocation
|
||||
from books.models import Book
|
||||
from books.constants import READCOMICSONLINE_URL
|
||||
from books.models import Book, BookLogData, BookPageLogData
|
||||
from books.utils import parse_readcomicsonline_uri
|
||||
from bricksets.models import BrickSet
|
||||
from dateutil.parser import parse
|
||||
from django.utils import timezone
|
||||
@ -56,18 +58,11 @@ def mopidy_scrobble_media(post_data: dict, user_id: int) -> Scrobble:
|
||||
|
||||
if media_type == Scrobble.MediaType.PODCAST_EPISODE:
|
||||
parsed_data = parse_mopidy_uri(post_data.get("mopidy_uri", ""))
|
||||
podcast_name = post_data.get(
|
||||
"album", parsed_data.get("podcast_name", "")
|
||||
)
|
||||
if not parsed_data:
|
||||
logger.warning("Tried to scrobble podcast but no uri found", extra={"post_data": post_data})
|
||||
return Scrobble()
|
||||
|
||||
media_obj = PodcastEpisode.find_or_create(
|
||||
title=parsed_data.get("episode_filename", ""),
|
||||
podcast_name=podcast_name,
|
||||
producer_name=post_data.get("artist", ""),
|
||||
number=parsed_data.get("episode_num", ""),
|
||||
pub_date=parsed_data.get("pub_date", ""),
|
||||
mopidy_uri=post_data.get("mopidy_uri", ""),
|
||||
)
|
||||
media_obj = PodcastEpisode.find_or_create(**parsed_data)
|
||||
else:
|
||||
media_obj = Track.find_or_create(
|
||||
title=post_data.get("name", ""),
|
||||
@ -262,13 +257,41 @@ def manual_scrobble_video_game(
|
||||
def manual_scrobble_book(
|
||||
title: str, user_id: int, action: Optional[str] = None
|
||||
):
|
||||
book = Book.find_or_create(title)
|
||||
log = {}
|
||||
source = "Vrobbler"
|
||||
page = None
|
||||
|
||||
if READCOMICSONLINE_URL in title:
|
||||
title, volume, page = parse_readcomicsonline_uri(title)
|
||||
if not title:
|
||||
logger.info(
|
||||
"[scrobblers] manual book scrobble request failed",
|
||||
extra={
|
||||
"title": title,
|
||||
"user_id": user_id,
|
||||
"media_type": Scrobble.MediaType.BOOK,
|
||||
},
|
||||
)
|
||||
return
|
||||
|
||||
title = f"{title} - Issue {volume}"
|
||||
|
||||
if not page:
|
||||
page = 1
|
||||
|
||||
logger.info("[scrobblers] Book page included in scrobble, should update!")
|
||||
|
||||
source = READCOMICSONLINE_URL.replace("https://", "")
|
||||
|
||||
# TODO: Check for scrobble of this book already and if so, update the page count
|
||||
|
||||
book = Book.find_or_create(title, enrich=True)
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": timezone.now(),
|
||||
"playback_position_seconds": 0,
|
||||
"source": "Vrobbler",
|
||||
"source": source,
|
||||
"long_play_complete": False,
|
||||
}
|
||||
|
||||
@ -282,7 +305,13 @@ def manual_scrobble_book(
|
||||
},
|
||||
)
|
||||
|
||||
return Scrobble.create_or_update(book, user_id, scrobble_dict)
|
||||
|
||||
scrobble = Scrobble.create_or_update(book, user_id, scrobble_dict, read_log_page=page)
|
||||
|
||||
if action == "stop":
|
||||
scrobble.stop(force_finish=True)
|
||||
|
||||
return scrobble
|
||||
|
||||
|
||||
def manual_scrobble_board_game(
|
||||
@ -539,6 +568,8 @@ def manual_scrobble_from_url(
|
||||
|
||||
if content_key == "-i" and "v=" in url:
|
||||
item_id = url.split("v=")[1].split("&")[0]
|
||||
elif content_key == "-c" and "comics" in url:
|
||||
item_id = url
|
||||
elif content_key == "-i" and "title/tt" in url:
|
||||
item_id = "tt" + str(item_id)
|
||||
|
||||
@ -875,8 +906,13 @@ def manual_scrobble_webpage(
|
||||
)
|
||||
|
||||
scrobble = Scrobble.create_or_update(webpage, user_id, scrobble_dict)
|
||||
# possibly async this?
|
||||
scrobble.push_to_archivebox()
|
||||
|
||||
if action == "stop":
|
||||
scrobble.stop(force_finish=True)
|
||||
else:
|
||||
# possibly async this?
|
||||
scrobble.push_to_archivebox()
|
||||
|
||||
return scrobble
|
||||
|
||||
|
||||
|
||||
@ -1,15 +1,18 @@
|
||||
import hashlib
|
||||
import logging
|
||||
import re
|
||||
from datetime import datetime, timedelta
|
||||
from datetime import date, datetime, timedelta
|
||||
from typing import TYPE_CHECKING, Optional
|
||||
from urllib.parse import urlparse
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pendulum
|
||||
import pytz
|
||||
from django.apps import apps
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.db import models
|
||||
from django.db.models.fields.json import KeyTextTransform
|
||||
from django.db.models.functions import Cast, TruncDate
|
||||
from django.utils import timezone
|
||||
from profiles.models import UserProfile
|
||||
from profiles.utils import now_user_timezone
|
||||
@ -371,3 +374,37 @@ def fix_playback_position_seconds(*scrobbles: "Scrobble", commit=True) -> list["
|
||||
scrobble.save(update_fields=["playback_position_seconds"])
|
||||
|
||||
return updated_scrobbles
|
||||
|
||||
|
||||
def base_scrobble_qs(user_id: int) -> models.QuerySet:
|
||||
"""Base queryset with calories extracted as integer and day annotated."""
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
return (
|
||||
Scrobble.objects
|
||||
.annotate(day=TruncDate("timestamp"))
|
||||
.annotate(calories_int=Cast(KeyTextTransform("calories", "log"), models.IntegerField()))
|
||||
.filter(calories_int__isnull=False, user_id=user_id)
|
||||
)
|
||||
|
||||
def get_daily_calories_for_user_by_day(user_id: int, date: date| str) -> int:
|
||||
"""Return total calories for a user on a specific day."""
|
||||
|
||||
if isinstance(date, str):
|
||||
date = pendulum.parse(date)
|
||||
|
||||
qs = base_scrobble_qs(user_id).filter(day=date)
|
||||
agg = qs.aggregate(total_calories=models.Sum("calories_int"))
|
||||
|
||||
return agg["total_calories"] or 0
|
||||
|
||||
def get_daily_calorie_dict_for_user(user_id: int) -> dict[date, int]:
|
||||
"""Return {day: total_calories} for all days with scrobbles, in one query."""
|
||||
qs = (
|
||||
base_scrobble_qs(user_id)
|
||||
.values("day")
|
||||
.annotate(total_calories=models.Sum("calories_int"))
|
||||
.order_by("day")
|
||||
)
|
||||
|
||||
return {entry["day"]: entry["total_calories"] for entry in qs}
|
||||
|
||||
@ -2,26 +2,27 @@ import calendar
|
||||
import json
|
||||
import logging
|
||||
from datetime import datetime, timedelta
|
||||
from dateutil.relativedelta import relativedelta
|
||||
|
||||
from django.shortcuts import redirect
|
||||
import pendulum
|
||||
import pytz
|
||||
from dateutil.relativedelta import relativedelta
|
||||
from django.apps import apps
|
||||
from django.contrib import messages
|
||||
from django.contrib.auth.mixins import LoginRequiredMixin
|
||||
from django.db.models import Count, Q, Max
|
||||
from django.core.paginator import EmptyPage, PageNotAnInteger, Paginator
|
||||
from django.db.models import Count, Max, Q
|
||||
from django.db.models.query import QuerySet
|
||||
from django.http import FileResponse, HttpResponseRedirect, JsonResponse
|
||||
from django.shortcuts import redirect
|
||||
from django.urls import reverse_lazy
|
||||
from django.utils import timezone
|
||||
from django.views.decorators.csrf import csrf_exempt
|
||||
from django.views.generic import DetailView, FormView, TemplateView
|
||||
from django.views.generic.edit import CreateView
|
||||
from django.views.generic.list import ListView
|
||||
from moods.models import Mood
|
||||
from music.aggregators import live_charts, scrobble_counts, week_of_scrobbles
|
||||
|
||||
from django.core.paginator import Paginator, EmptyPage, PageNotAnInteger
|
||||
from pendulum.parsing.exceptions import ParserError
|
||||
from rest_framework import status
|
||||
from rest_framework.decorators import (
|
||||
api_view,
|
||||
@ -54,10 +55,10 @@ from scrobbles.tasks import (
|
||||
process_tsv_import,
|
||||
)
|
||||
from scrobbles.utils import (
|
||||
get_daily_calories_for_user_by_day,
|
||||
get_long_plays_completed,
|
||||
get_long_plays_in_progress,
|
||||
)
|
||||
from moods.models import Mood
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -146,7 +147,7 @@ class RecentScrobbleList(ListView):
|
||||
if date_str:
|
||||
try:
|
||||
date = pendulum.parse(date_str)
|
||||
except:
|
||||
except ParserError:
|
||||
pass
|
||||
if date_str:
|
||||
if date_str == "this_week" or "-W" in date_str:
|
||||
@ -230,6 +231,7 @@ class RecentScrobbleList(ListView):
|
||||
user=self.request.user,
|
||||
)
|
||||
data["counts"] = [] # scrobble_counts(user)
|
||||
data["daily_calories"] = get_daily_calories_for_user_by_day(self.request.user.id, date)
|
||||
else:
|
||||
data["weekly_data"] = week_of_scrobbles()
|
||||
data["counts"] = scrobble_counts()
|
||||
@ -623,9 +625,9 @@ def scrobble_start(request, uuid):
|
||||
|
||||
if (
|
||||
user.profile.redirect_to_webpage
|
||||
and media_obj.__class__.__name__ == Scrobble.MediaType.WEBPAGE
|
||||
and (media_obj.__class__.__name__ == Scrobble.MediaType.WEBPAGE or media_obj.__class__.__name__ == Scrobble.MediaType.BOOK)
|
||||
):
|
||||
logger.info(f"Redirecting to {media_obj} detail apge")
|
||||
logger.info(f"Redirecting to {media_obj} detail page")
|
||||
return HttpResponseRedirect(media_obj.url)
|
||||
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
@ -12,6 +12,7 @@
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for obj in object_list %}
|
||||
{% if obj.title %}
|
||||
<tr>
|
||||
<td><a href="{{obj.scrobble_set.last.get_absolute_url}}">{{obj.scrobble_set.last.local_timestamp}}
|
||||
<td><a href="{{obj.get_absolute_url}}">{{obj}}</a></td>
|
||||
@ -20,6 +21,7 @@
|
||||
<td><a type="button" class="btn btn-sm btn-primary" href="{{obj.start_url}}">Scrobble</a></td>
|
||||
{% endif %}
|
||||
</tr>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
|
||||
@ -105,6 +105,7 @@
|
||||
{% with scrobbles=Beer count=Beer_count time=Beer_time %}
|
||||
{% include "scrobbles/_scrobble_table.html" %}
|
||||
{% endwith %}
|
||||
{% else %}
|
||||
<p>No beer today</p>
|
||||
{% endif %}
|
||||
|
||||
|
||||
@ -4,7 +4,7 @@
|
||||
<div class="tab-pane fade show" id="latest-beers" role="tabpanel"
|
||||
aria-labelledby="latest-beers-tab">
|
||||
<div class="table-responsive">
|
||||
{{count}} scrobble {% if time %}| {{time|natural_duration}}{% endif %}
|
||||
{{count}} scrobble {% if time %}| {{time|natural_duration}}{% endif %}{% if daily_calories %}| {{daily_calories}} calories{% endif %}
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
|
||||
Reference in New Issue
Block a user