Compare commits
54 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5e0dffdc7a | |||
| 2283a6c640 | |||
| 327ba94c63 | |||
| ee59cde882 | |||
| c7b4656679 | |||
| 04f9e00c9c | |||
| c2dabd1dac | |||
| 7a0cb8b9d0 | |||
| 1c2c570c4b | |||
| 0671ab432f | |||
| 893867419a | |||
| d9dfec81aa | |||
| 948fbc19bf | |||
| 7d708ad8a6 | |||
| e0505cb82c | |||
| ab6459e4b0 | |||
| c001248d1b | |||
| f1c777d4ef | |||
| 931488c288 | |||
| ab897fd848 | |||
| 4f189b4d66 | |||
| 1487504318 | |||
| 0655363a0d | |||
| dccc80c615 | |||
| 4f91d5b40b | |||
| cb01781615 | |||
| 1f5fada8b1 | |||
| 31888a85cb | |||
| 22d8b0787e | |||
| 8cc559752b | |||
| db3f9696fa | |||
| 407d570c82 | |||
| 033239260f | |||
| 9f854dc735 | |||
| f29272a853 | |||
| 4e56d9420a | |||
| 852a257159 | |||
| 68ff230f13 | |||
| 57a952a6d1 | |||
| 718fcf7392 | |||
| 52adcf83c7 | |||
| 0061623f7e | |||
| ec73e5151e | |||
| 2c90dd38b5 | |||
| c6b1e42d7a | |||
| fcf86d5b3f | |||
| 6fde9ec8d2 | |||
| 0f1882b21f | |||
| e819a2db0d | |||
| e03cf6c9b1 | |||
| 471e70ff7f | |||
| 255e335d7a | |||
| c8cf80b513 | |||
| b4180afbed |
262
PROJECT.org
262
PROJECT.org
@ -88,7 +88,7 @@ fetching and simple saving.
|
||||
*** Metadata sources
|
||||
**** Scraper
|
||||
|
||||
* Backlog [0/20] :vrobbler:project:personal:
|
||||
* Backlog [0/22] :vrobbler:project:personal:
|
||||
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :bug:music:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
|
||||
@ -579,6 +579,18 @@ named constants for maintainability.
|
||||
- ~vrobbler/apps/scrobbles/importers/tsv.py~ (line 55) -- ="S"= completion status
|
||||
|
||||
|
||||
** TODO [#A] Deduplicate BGG plays before posting :boardgames:bgg:duplication:
|
||||
:PROPERTIES:
|
||||
:ID: e9b842bf-0049-42e7-a060-f3ebd0067d2f
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
No check for existing BGG plays before posting, which can create duplicates.
|
||||
Should look up past plays by =bggeek_id= first.
|
||||
|
||||
File: ~vrobbler/apps/boardgames/bgg.py~ (line 117)
|
||||
|
||||
** TODO [#C] Clean up naming of =bgsplay= parsing :importers:refactoring:
|
||||
:PROPERTIES:
|
||||
:ID: c751dbbc-464a-4e63-9fe3-e034303f7b54
|
||||
@ -590,6 +602,254 @@ We should rename `email_scrobble_board_game` to reflect the fact that it's just
|
||||
a helper method to create board game scrobbles given a json blob. It's
|
||||
independent of the email flow it was originally creatdd for
|
||||
|
||||
** TODO [#B] Is there way to create unique slugs for media instances :media_types:
|
||||
|
||||
* Version 58.2 [2/2]
|
||||
** DONE [#B] Add more robust webpage scraping :webpages:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: 84d9bfa5-75c0-0718-764e-379f7456602a
|
||||
:END:
|
||||
** DONE [#B] Time of Day Categories trend :trends:
|
||||
:PROPERTIES:
|
||||
:ID: 6598074f-2290-46db-967b-29f45d30be29
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Added a "Time of Day Categories" trend that groups scrobbles for Books, Trails,
|
||||
Birding Locations, and Board Games into Early Bird (5-10:59am), Day Jay (11am-6:59pm),
|
||||
and Night Owl (7pm-4:59am) buckets. Shows both overall and per-media-type breakdowns.
|
||||
|
||||
|
||||
* Version 58.1 [1/1]
|
||||
** DONE [#B] Add auto genre tagging for papers :books:papers:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: e6b5c3a5-7fc6-b530-96c2-b5962a716db6
|
||||
:END:
|
||||
|
||||
* Version 58.0 [1/1]
|
||||
** DONE [#B] Add scrobbling of Papers via webpages with doi.org links in them :feature:papers:
|
||||
:PROPERTIES:
|
||||
:ID: d30bb8aa-eefd-002c-38d5-3f2fcef345f2
|
||||
:END:
|
||||
|
||||
* Version 57.1 [1/1]
|
||||
** DONE [#A] Write poetry lock file :bug:deps:
|
||||
:PROPERTIES:
|
||||
:ID: 0f5a6f4b-a486-ba7e-bbce-f7581274398c
|
||||
:END:
|
||||
|
||||
* Version 57.0 [5/5]
|
||||
** DONE [#A] Scrobble button on some media list pages dont work :bug:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: a3a5c707-2e3d-a6b1-0f7f-4c6f7433aa1f
|
||||
:END:
|
||||
** DONE [#B] Use HTMx to update the Now Playing widget :feature:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 5f5631fc-9ee1-d5a5-d0f8-94fea6fbbfa4
|
||||
:END:
|
||||
** DONE [#B] Add a live page that updates the scrobble list via JS polling :feature:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 58790d76-dc6e-8aa5-2dc0-e64fe786fbf1
|
||||
:END:
|
||||
** DONE [#A] Turns out we cant cache the now playing widget :bug:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 9ce669ea-c000-cdfe-a634-ad5cdaeae81c
|
||||
:END:
|
||||
** DONE [#C] What would it look like to add an MCP server to expose scrobbles and media items? :mcpserver:feature:
|
||||
:PROPERTIES:
|
||||
:ID: c5fca159-c7e0-5795-7c05-bbc48f539650
|
||||
:END:
|
||||
|
||||
* Version 56.4 [3/3]
|
||||
** DONE [#B] Add ability to do reverse address lookup on lat-long pairs :geolocations:feature:
|
||||
:PROPERTIES:
|
||||
:ID: 86c071ff-7638-41ba-6b65-1382df1cb5aa
|
||||
:END:
|
||||
** DONE [#B] Add address fields to GeoLocation :addresses:geolocation:
|
||||
:PROPERTIES:
|
||||
:ID: a55ae508-07ab-ccdd-e453-846bd3fca6fb
|
||||
:END:
|
||||
** DONE [#B] Add better detail template for Disc Golf Courses :discgolf:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 12cee67c-f723-0fa3-848a-cbc6e4d65fc3
|
||||
:END:
|
||||
|
||||
* Version 56.3 [1/1]
|
||||
** DONE [#A] Fix bug in importer script from discgolf being added :bug:
|
||||
:PROPERTIES:
|
||||
:ID: c3733f96-18f1-eef8-f5d9-edaf97e35623
|
||||
:END:
|
||||
|
||||
* Version 56.2 [1/1]
|
||||
** DONE [#A] Fix bug in creating people when importing course plays :discgolf:bug:
|
||||
:PROPERTIES:
|
||||
:ID: 255e9886-098b-39ae-1077-25e43223660e
|
||||
:END:
|
||||
|
||||
* Version 56.1 [1/1]
|
||||
** DONE [#A] Add tests to discgolf app :discgolf:tests:
|
||||
:PROPERTIES:
|
||||
:ID: 28e8344e-c3cf-19af-ce1c-cb821d4fcb5f
|
||||
:END:
|
||||
|
||||
* Version 56.0 [1/1]
|
||||
** DONE [#B] Add DiscGolf as a scrobbleable media :discgolf:
|
||||
:PROPERTIES:
|
||||
:ID: 8cdde5d3-0ae5-7d5a-99d2-200c86afae03
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
I have a csv file fro the uDisc disc golf scoring app that looks like:
|
||||
|
||||
Singles round. Note second row is the par for the course
|
||||
#+begin_src csv
|
||||
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
|
||||
Par,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,27,,,3,3,3,3,3,3,3,3,3
|
||||
Colin Powell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,30,3,,2,3,4,4,3,4,3,3,4
|
||||
Asa Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,5,4,4,8,5,5,4,4,5
|
||||
Emma Sweet,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,4,5,6,3,4,3,5,6
|
||||
Jane Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,44,17,,4,5,5,5,5,5,4,6,5
|
||||
Nabby Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,59,32,,6,6,7,7,6,7,6,6,8
|
||||
Silas Sewell,DR Front 9,Custom Layout,2026-06-19 1535-0400,2026-06-19 1725-0400,41,14,,5,5,4,5,3,5,4,4,6`
|
||||
#+end_src
|
||||
|
||||
Teams of two or more persons. Note second row is the par for the course
|
||||
#+begin_src csv
|
||||
PlayerName,CourseName,LayoutName,StartDate,EndDate,Total,+/-,RoundRating,Hole1,Hole2,Hole3,Hole4,Hole5,Hole6,Hole7,Hole8,Hole9
|
||||
Par,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,27,,,3,3,3,3,3,3,3,3,3
|
||||
Colin Powell + Asa Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,29,2,,3,4,2,3,2,3,5,3,4
|
||||
Emma Sweet + Jane Sewell,Peninsula Links,Main,2026-06-19 2322-0400,2026-06-19 2323-0400,28,1,,4,3,4,2,3,4,3,3,2
|
||||
#+end_src
|
||||
|
||||
We should add a new app called discgolf that has the following data models:
|
||||
|
||||
- DiscGolfRound - scrobblable media + course_id, round_type (Singles, Teams)
|
||||
- DiscGolfCourse - name, layoutname, number_of_holes
|
||||
|
||||
And the logdata for a DiscGolfOuting scrobble should have:
|
||||
+ {person: {hole_number: score}, total: int}
|
||||
+ {team: {name: "", people: [person, person], hole_number: score}, total: int}
|
||||
+ weather
|
||||
+ fun_factor (miserable, not great, so-so, good, excellent, party time)
|
||||
|
||||
|
||||
* Version 55.6 [1/1]
|
||||
** DONE [#A] Figure out why historical Lastfm imports don't work :importers:lastfm:music:
|
||||
:PROPERTIES:
|
||||
:ID: 71b18c1b-de96-6d93-20fa-de2ec0df1288
|
||||
:END:
|
||||
|
||||
* Version 55.5 [1/1]
|
||||
** DONE [#B] Fix bug in lastfm import for new users :importers:lastfm:music:
|
||||
:PROPERTIES:
|
||||
:ID: d034966d-0c7f-e512-4cf8-7329c9026b6f
|
||||
:END:
|
||||
|
||||
* Version 55.4 [1/1]
|
||||
** DONE [#A] Tighten up the speed of startup and first request :perf:
|
||||
:PROPERTIES:
|
||||
:ID: 9ee8834c-6be2-d04b-df6d-56375504083f
|
||||
:END:
|
||||
|
||||
|
||||
* Version 55.3 [3/3]
|
||||
** DONE [#C] =alt_names= feature for artists (commented out / dead code) :music:dead-code:
|
||||
:PROPERTIES:
|
||||
:ID: e22060a2-5f7a-4f33-9056-309ecd27159c
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
File: ~vrobbler/apps/music/models.py~ (line 236)
|
||||
|
||||
An entire block of code for tracking alternate artist names is commented
|
||||
out. The TODO questions whether it even works. Review: either implement
|
||||
properly or remove the dead code.
|
||||
|
||||
** DONE [#A] Put chart rebuilds in a lower priority task queue :charts:tasks:
|
||||
:PROPERTIES:
|
||||
:ID: 43c90de0-fc1c-1139-dac7-9b7c82006b2e
|
||||
:END:
|
||||
** DONE [#A] Check for existing book scrobble and update page count :books:scrobbling:
|
||||
:PROPERTIES:
|
||||
:ID: 1a0609bc-6b16-4da4-96c1-59588229e4b4
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
File: ~vrobbler/apps/scrobbles/scrobblers.py~ (line 330)
|
||||
|
||||
When scrobbling a book (comic), the code doesn't check for prior scrobbles to
|
||||
update reading progress. Needed for proper page-count tracking.
|
||||
|
||||
|
||||
* Version 55.2 [2/2]
|
||||
** DONE [#A] Fix bug in scrobble id in calendar view :templates:
|
||||
:PROPERTIES:
|
||||
:ID: 8cb34852-b18f-e794-cd9b-fb1ecad70a0d
|
||||
:END:
|
||||
** DONE [#A] Video game cleanup script should clear out broken images :metadata:videogames:
|
||||
:PROPERTIES:
|
||||
:ID: ca1f1ea9-0f79-082c-5ff7-867671faff4b
|
||||
:END:
|
||||
|
||||
* Version 55.1 [1/1]
|
||||
** DONE [#A] Clean up metadata scrapping for video games :metadata:videogames:
|
||||
:PROPERTIES:
|
||||
:ID: fbc421b5-21a3-4aed-9062-c59192ead065
|
||||
:END:
|
||||
|
||||
* Version 55.0 [3/3]
|
||||
** DONE [#B] Use pk ID for scrobble detail view, not uuid :scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 9cc3b285-e478-041e-394b-3d550aefbe1d
|
||||
:END:
|
||||
** DONE [#B] Display videogame screenshots on scrobble detail if they exist :videogames:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 0406d082-20f6-0d12-76e2-f281c4801468
|
||||
:END:
|
||||
** DONE [#B] Add autotagging to webpages based on domain, title :webpages:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: f658435b-f7a0-42e6-b9f6-226678a77a55
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
For easier filtering, like we do with tasks, we should auto tag WebPage instances
|
||||
based on the domain name split part by periods (so news.ycombinator.com tags: news, ycombinator, com)
|
||||
|
||||
And also based on the nouns in the title.
|
||||
|
||||
|
||||
* Version 54.5 [1/1]
|
||||
** DONE Fix bug in generating mood trends :trends:
|
||||
:PROPERTIES:
|
||||
:ID: 8e75abfa-8e70-d85b-00a4-a4813bbce879
|
||||
:END:
|
||||
|
||||
* Version 54.4 [2/2]
|
||||
** DONE [#A] Remove all-time trends :trends:
|
||||
:PROPERTIES:
|
||||
:ID: 53b231d1-7677-8cd3-1d88-dae110aba1e6
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
All time trends take forever to calculate and don't provide too much data
|
||||
|
||||
** DONE [#B] Add a trend around moods :moods:trends:
|
||||
:PROPERTIES:
|
||||
:ID: fba3f4ae-8f97-ee0b-e762-31630884518a
|
||||
:END:
|
||||
|
||||
* Version 54.3 [1/1]
|
||||
** DONE [#B] Fix bug in series metadata cleanup script :videos:metadta:
|
||||
:PROPERTIES:
|
||||
:ID: 85448702-907c-5d63-f5af-7795661d7c46
|
||||
:END:
|
||||
|
||||
* Version 54.2 [4/4]
|
||||
** DONE [#B] Add script to clean up TV series metadata :videos:metadata:
|
||||
:PROPERTIES:
|
||||
|
||||
2
Procfile
2
Procfile
@ -1,2 +1,2 @@
|
||||
web: python manage.py runserver 0.0.0.0:8014
|
||||
worker: celery -A vrobbler worker -l DEBUG
|
||||
worker: celery -A vrobbler worker -Q default,charts -l DEBUG
|
||||
|
||||
835
poetry.lock
generated
835
poetry.lock
generated
File diff suppressed because it is too large
Load Diff
@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "vrobbler"
|
||||
version = "54.2"
|
||||
version = "58.2"
|
||||
description = ""
|
||||
authors = ["Colin Powell <colin@unbl.ink>"]
|
||||
|
||||
@ -9,8 +9,10 @@ python = ">=3.11,<3.15"
|
||||
Django = "^4.0.3"
|
||||
django-extensions = "^3.1.5"
|
||||
python-dateutil = "^2.8.2"
|
||||
python-dotenv = "^0.20.0"
|
||||
python-dotenv = ">=0.20.0,<2"
|
||||
python-json-logger = "^2.0.2"
|
||||
cloudscraper = "^1.2.71"
|
||||
curl-cffi = "^0.15.0"
|
||||
colorlog = "^6.6.0"
|
||||
httpx = "<=0.27.2"
|
||||
djangorestframework = "^3.13.1"
|
||||
@ -43,6 +45,7 @@ ipython = "^8.14.0"
|
||||
pendulum = "^3"
|
||||
trafilatura = "^1.6.3"
|
||||
django-imagekit = "^5.0.0"
|
||||
django-mcp-server = "^0.5.7"
|
||||
thefuzz = "^0.22.1"
|
||||
dataclass-wizard = "^0.35.0"
|
||||
webdavclient3 = "^3.14.6"
|
||||
@ -65,6 +68,7 @@ lxml = ">=5.5.0"
|
||||
vaderSentiment = "^3.3.2"
|
||||
sqids = "^0.5.2"
|
||||
python-amazon-paapi = "^6.3.0"
|
||||
yake = "^0.7.3"
|
||||
|
||||
[tool.poetry.group.test]
|
||||
optional = true
|
||||
|
||||
0
tests/discgolf_tests/__init__.py
Normal file
0
tests/discgolf_tests/__init__.py
Normal file
63
tests/discgolf_tests/conftest.py
Normal file
63
tests/discgolf_tests/conftest.py
Normal file
@ -0,0 +1,63 @@
|
||||
import tempfile
|
||||
|
||||
import pytest
|
||||
from django.contrib.auth import get_user_model
|
||||
|
||||
User = get_user_model()
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def user(db):
|
||||
return User.objects.create(email="golfer@example.com")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_singles_csv_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
Alice,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
Bob,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,4,5,12
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_singles_csv_file(udisc_singles_csv_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_singles_csv_content)
|
||||
return f.name
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_teams_csv_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
Alice+Bob,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
Charlie+Diana,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",3,4,5,12
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_teams_csv_file(udisc_teams_csv_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_teams_csv_content)
|
||||
return f.name
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_csv_no_par_content():
|
||||
return """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Alice,Maple Hill,Mountains,"Jun 15, 2026 10:00 AM",4,2,3,9
|
||||
"""
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def udisc_csv_no_par_file(udisc_csv_no_par_content):
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(udisc_csv_no_par_content)
|
||||
return f.name
|
||||
102
tests/discgolf_tests/test_models.py
Normal file
102
tests/discgolf_tests/test_models.py
Normal file
@ -0,0 +1,102 @@
|
||||
from discgolf.models import DiscGolfCourse, DiscGolfLogData
|
||||
from scrobbles.dataclasses import BaseLogData
|
||||
|
||||
|
||||
class TestDiscGolfCourseModel:
|
||||
def test_create_course(self, db):
|
||||
course = DiscGolfCourse.objects.create(
|
||||
title="Maple Hill",
|
||||
layout_name="Mountains",
|
||||
number_of_holes=18,
|
||||
par_total=54,
|
||||
par_per_hole={"hole_1": 3, "hole_2": 3},
|
||||
)
|
||||
assert course.uuid is not None
|
||||
assert str(course) == "Maple Hill (Mountains)"
|
||||
assert course.subtitle == "Mountains"
|
||||
|
||||
def test_subtitle_fallback(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.subtitle == ""
|
||||
|
||||
def test_logdata_cls(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.logdata_cls is DiscGolfLogData
|
||||
assert issubclass(course.logdata_cls, BaseLogData)
|
||||
|
||||
def test_strings(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.strings.verb == "Playing"
|
||||
assert course.strings.tags == "golf"
|
||||
|
||||
def test_primary_image_url(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
assert course.primary_image_url == ""
|
||||
|
||||
def test_get_absolute_url(self, db):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
url = course.get_absolute_url()
|
||||
assert str(course.uuid) in url
|
||||
assert url.startswith("/disc-golf/")
|
||||
|
||||
def test_find_or_create_new(self, db):
|
||||
course = DiscGolfCourse.find_or_create(
|
||||
"New Course", layout_name="Default"
|
||||
)
|
||||
assert course.title == "New Course"
|
||||
assert course.layout_name == "Default"
|
||||
|
||||
def test_find_or_create_existing(self, db):
|
||||
created = DiscGolfCourse.objects.create(
|
||||
title="Existing", layout_name="Alpha"
|
||||
)
|
||||
found = DiscGolfCourse.find_or_create("Existing", layout_name="Beta")
|
||||
assert found.id == created.id
|
||||
assert found.layout_name == "Alpha"
|
||||
|
||||
def test_scrobbles_method(self, db, user):
|
||||
from datetime import datetime
|
||||
|
||||
import pytz
|
||||
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
dt1 = datetime(2026, 6, 15, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
dt2 = datetime(2026, 6, 14, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
s1 = Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt1,
|
||||
)
|
||||
s2 = Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt2,
|
||||
)
|
||||
qs = course.scrobbles(user.id)
|
||||
assert list(qs) == [s1, s2]
|
||||
|
||||
|
||||
class TestDiscGolfLogData:
|
||||
def test_basic_logdata(self):
|
||||
data = DiscGolfLogData()
|
||||
assert data.scores is None
|
||||
assert data.weather is None
|
||||
assert data.fun_factor is None
|
||||
assert data.course_name is None
|
||||
|
||||
def test_logdata_with_scores(self):
|
||||
data = DiscGolfLogData(
|
||||
scores={"Alice": {"person_id": 1, "total": 9}},
|
||||
weather="Sunny",
|
||||
fun_factor="High",
|
||||
course_name="Maple Hill",
|
||||
par=9,
|
||||
round_type="Singles",
|
||||
)
|
||||
assert data.scores["Alice"]["total"] == 9
|
||||
assert data.weather == "Sunny"
|
||||
assert data.round_type == "Singles"
|
||||
150
tests/discgolf_tests/test_utils.py
Normal file
150
tests/discgolf_tests/test_utils.py
Normal file
@ -0,0 +1,150 @@
|
||||
from unittest.mock import patch
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from discgolf.utils import _parse_udisc_datetime, import_udisc_csv
|
||||
from people.models import Person
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
|
||||
class TestParserHelpers:
|
||||
def test_parse_udisc_datetime(self):
|
||||
dt = _parse_udisc_datetime("Jun 15, 2026 10:00 AM")
|
||||
assert dt is not None
|
||||
assert dt.year == 2026
|
||||
assert dt.month == 6
|
||||
assert dt.day == 15
|
||||
assert dt.hour == 10
|
||||
assert dt.minute == 0
|
||||
|
||||
def test_parse_udisc_datetime_date_only(self):
|
||||
dt = _parse_udisc_datetime("Jun 15, 2026")
|
||||
assert dt is not None
|
||||
assert dt.year == 2026
|
||||
|
||||
|
||||
class TestImportUdiscCSV:
|
||||
def test_import_singles_creates_course(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.filter(title="Maple Hill").first()
|
||||
assert course is not None
|
||||
assert course.layout_name == "Mountains"
|
||||
assert course.number_of_holes == 3
|
||||
assert course.par_total == 9
|
||||
assert course.par_per_hole == {"hole_1": 3, "hole_2": 3, "hole_3": 3}
|
||||
|
||||
def test_import_singles_creates_scrobble(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 1
|
||||
|
||||
def test_import_singles_logdata(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
log = scrobble.log
|
||||
assert log["course_name"] == "Maple Hill"
|
||||
assert log["par"] == 9
|
||||
assert log["round_type"] == "Singles"
|
||||
assert "Alice" in log["scores"]
|
||||
assert "Bob" in log["scores"]
|
||||
assert log["scores"]["Alice"]["total"] == 9
|
||||
assert log["scores"]["Bob"]["total"] == 12
|
||||
|
||||
def test_import_singles_creates_people(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert Person.objects.filter(name="Alice").exists()
|
||||
assert Person.objects.filter(name="Bob").exists()
|
||||
|
||||
def test_import_teams_creates_scrobble(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 1
|
||||
|
||||
def test_import_teams_logdata(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
assert scrobble.log["round_type"] == "Teams"
|
||||
alice_bob = scrobble.log["scores"]["Alice+Bob"]
|
||||
assert "person_ids" in alice_bob
|
||||
assert len(alice_bob["person_ids"]) == 2
|
||||
|
||||
def test_import_creates_team_people(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
assert Person.objects.filter(name="Alice").exists()
|
||||
assert Person.objects.filter(name="Bob").exists()
|
||||
assert Person.objects.filter(name="Charlie").exists()
|
||||
assert Person.objects.filter(name="Diana").exists()
|
||||
|
||||
def test_import_teams_par_per_hole(self, user, udisc_teams_csv_file):
|
||||
import_udisc_csv(udisc_teams_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.get(title="Maple Hill")
|
||||
assert course.par_per_hole == {"hole_1": 3, "hole_2": 3, "hole_3": 3}
|
||||
|
||||
def test_import_no_par_returns_empty(self, user, udisc_csv_no_par_file):
|
||||
result = import_udisc_csv(udisc_csv_no_par_file, user.id)
|
||||
assert result == []
|
||||
|
||||
def test_import_empty_csv(self, user, db):
|
||||
import tempfile
|
||||
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write("PlayerName,CourseName,LayoutName,StartDate,Hole1,Total\n")
|
||||
path = f.name
|
||||
|
||||
result = import_udisc_csv(path, user.id)
|
||||
assert result == []
|
||||
|
||||
def test_import_idempotent(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
assert DiscGolfCourse.objects.filter(title="Maple Hill").count() == 1
|
||||
assert Scrobble.objects.filter(source="uDisc").count() == 2
|
||||
|
||||
def test_import_course_defaults_only_on_create(
|
||||
self, user, udisc_singles_csv_file
|
||||
):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course = DiscGolfCourse.objects.get(title="Maple Hill")
|
||||
assert course.layout_name == "Mountains"
|
||||
|
||||
course.layout_name = "Updated"
|
||||
course.save()
|
||||
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
course.refresh_from_db()
|
||||
assert course.layout_name == "Updated"
|
||||
|
||||
@patch("discgolf.utils.ScrobbleNtfyNotification")
|
||||
def test_import_sends_notification(self, mock_notification_class, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
mock_notification_class.assert_called_once()
|
||||
mock_notification_class.return_value.send.assert_called_once()
|
||||
|
||||
def test_import_hole_scores_per_player(self, user, udisc_singles_csv_file):
|
||||
import_udisc_csv(udisc_singles_csv_file, user.id)
|
||||
scrobble = Scrobble.objects.filter(source="uDisc").first()
|
||||
alice = scrobble.log["scores"]["Alice"]
|
||||
assert alice["hole_1"] == 4
|
||||
assert alice["hole_2"] == 2
|
||||
assert alice["hole_3"] == 3
|
||||
bob = scrobble.log["scores"]["Bob"]
|
||||
assert bob["hole_1"] == 3
|
||||
assert bob["hole_2"] == 4
|
||||
assert bob["hole_3"] == 5
|
||||
|
||||
def test_import_record_error_on_bad_data(self, user, db):
|
||||
import tempfile
|
||||
|
||||
content = """PlayerName,CourseName,LayoutName,StartDate,Hole1,Hole2,Hole3,Total
|
||||
Par,,Mountains,"Jun 15, 2026 10:00 AM",3,3,3,9
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8-sig"
|
||||
) as f:
|
||||
f.write(content)
|
||||
path = f.name
|
||||
|
||||
errors = []
|
||||
result = import_udisc_csv(path, user.id, record_error=errors.append)
|
||||
assert len(result) == 1
|
||||
course = DiscGolfCourse.objects.first()
|
||||
assert course.title == ""
|
||||
58
tests/discgolf_tests/test_views.py
Normal file
58
tests/discgolf_tests/test_views.py
Normal file
@ -0,0 +1,58 @@
|
||||
from datetime import datetime
|
||||
|
||||
import pytz
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.test import Client
|
||||
from django.urls import reverse
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
User = get_user_model()
|
||||
|
||||
|
||||
class TestDiscGolfCourseViews:
|
||||
def _make_scrobble(self, user, course):
|
||||
dt = datetime(2026, 6, 15, 14, 0, 0, tzinfo=pytz.UTC)
|
||||
return Scrobble.objects.create(
|
||||
user=user,
|
||||
disc_golf_course=course,
|
||||
media_type=Scrobble.MediaType.DISC_GOLF,
|
||||
timestamp=dt,
|
||||
)
|
||||
|
||||
def test_course_list_anonymous(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(reverse("discgolf:course_list"))
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_course_list_shows_course(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(reverse("discgolf:course_list"))
|
||||
assert response.status_code == 200
|
||||
assert "Maple Hill" in response.content.decode()
|
||||
|
||||
def test_course_detail_anonymous(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(title="Maple Hill")
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(
|
||||
reverse("discgolf:course_detail", kwargs={"slug": course.uuid})
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
def test_course_detail_shows_course(self, db, user):
|
||||
course = DiscGolfCourse.objects.create(
|
||||
title="Maple Hill", layout_name="Mountains"
|
||||
)
|
||||
self._make_scrobble(user, course)
|
||||
client = Client()
|
||||
response = client.get(
|
||||
reverse("discgolf:course_detail", kwargs={"slug": course.uuid})
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert "Maple Hill" in response.content.decode()
|
||||
@ -519,7 +519,7 @@ def test_scrobble_detail_view_with_notes_as_flat_list(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
assert "First note" in response.content.decode()
|
||||
@ -545,7 +545,7 @@ def test_scrobble_detail_view_with_notes_as_dict_timestamps(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
@ -574,7 +574,7 @@ def test_scrobble_detail_view_with_notes_and_labels(client):
|
||||
"description": "Test description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
@ -597,7 +597,7 @@ def test_scrobble_detail_view_post_updates_log(client):
|
||||
"description": "Original description",
|
||||
},
|
||||
)
|
||||
url = reverse("scrobbles:detail", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:detail", kwargs={"pk": scrobble.id})
|
||||
|
||||
client.force_login(user)
|
||||
response = client.post(
|
||||
@ -896,7 +896,7 @@ def test_change_visibility_owner_can_change(client):
|
||||
)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 302
|
||||
|
||||
@ -919,7 +919,7 @@ def test_change_visibility_non_owner_gets_404(client):
|
||||
)
|
||||
|
||||
client.force_login(other)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 404
|
||||
|
||||
@ -937,7 +937,7 @@ def test_change_visibility_anonymous_redirects_to_login(client):
|
||||
task=task, media_type="Task", user=user, visibility="private",
|
||||
timestamp=timezone.now(),
|
||||
)
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:change-visibility", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url, {"visibility": "shared"})
|
||||
assert response.status_code == 302
|
||||
assert "/login/" in response.url
|
||||
@ -956,7 +956,7 @@ def test_regenerate_share_token_invalidates_old_sqid(client):
|
||||
old_sqid = encode_scrobble_share(scrobble.id, scrobble.share_token_version)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:regenerate-share-token", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:regenerate-share-token", kwargs={"pk": scrobble.id})
|
||||
response = client.post(url)
|
||||
assert response.status_code == 302
|
||||
|
||||
@ -985,7 +985,7 @@ def test_share_analytics_owner_can_view(client):
|
||||
)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
|
||||
@ -1005,7 +1005,7 @@ def test_share_analytics_non_owner_gets_404(client):
|
||||
)
|
||||
|
||||
client.force_login(other)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 404
|
||||
|
||||
@ -1027,7 +1027,7 @@ def test_share_analytics_shows_view_logs(client):
|
||||
client.get(share_url)
|
||||
|
||||
client.force_login(user)
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"uuid": scrobble.uuid})
|
||||
url = reverse("scrobbles:share-analytics", kwargs={"pk": scrobble.id})
|
||||
response = client.get(url)
|
||||
assert response.status_code == 200
|
||||
content = response.content.decode()
|
||||
|
||||
@ -4,9 +4,15 @@ from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
from vrobbler import context_processors
|
||||
from vrobbler.context_processors import version_info
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def reset_git_cache():
|
||||
context_processors._GIT_COMMIT = None
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mock_request():
|
||||
return MagicMock()
|
||||
|
||||
@ -1,8 +1,19 @@
|
||||
from books.models import Author, Book, Paper
|
||||
from books.models import Author, Book, Journal, Paper
|
||||
from django.contrib import admin
|
||||
from scrobbles.admin import ScrobbleInline
|
||||
|
||||
|
||||
@admin.register(Journal)
|
||||
class JournalAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = (
|
||||
"title",
|
||||
"website_url",
|
||||
)
|
||||
search_fields = ("title",)
|
||||
ordering = ("-created",)
|
||||
|
||||
|
||||
@admin.register(Author)
|
||||
class AuthorAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
|
||||
18
vrobbler/apps/books/migrations/0038_paper_pdf_file.py
Normal file
18
vrobbler/apps/books/migrations/0038_paper_pdf_file.py
Normal file
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-23 14:45
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("books", "0037_book_volume_book_volume_comicvine_id"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="paper",
|
||||
name="pdf_file",
|
||||
field=models.FileField(blank=True, null=True, upload_to="papers/pdf/"),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,92 @@
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
import django_extensions.db.fields
|
||||
import uuid
|
||||
|
||||
|
||||
def migrate_journal_data(apps, schema_editor):
|
||||
Paper = apps.get_model("books", "Paper")
|
||||
Journal = apps.get_model("books", "Journal")
|
||||
for paper in Paper.objects.all():
|
||||
old_journal = getattr(paper, "journal", None)
|
||||
if old_journal:
|
||||
journal, _ = Journal.objects.get_or_create(title=str(old_journal))
|
||||
paper._journal_tmp = journal
|
||||
paper.save(update_fields=["_journal_tmp"])
|
||||
|
||||
|
||||
def reverse_migrate_journal_data(apps, schema_editor):
|
||||
Paper = apps.get_model("books", "Paper")
|
||||
for paper in Paper.objects.all():
|
||||
if paper._journal_tmp:
|
||||
paper.journal = paper._journal_tmp.title
|
||||
paper.save(update_fields=["journal"])
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("books", "0038_paper_pdf_file"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="Journal",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
(
|
||||
"uuid",
|
||||
models.UUIDField(
|
||||
blank=True, default=uuid.uuid4, editable=False, null=True
|
||||
),
|
||||
),
|
||||
("title", models.CharField(max_length=255)),
|
||||
("description", models.TextField(blank=True, null=True)),
|
||||
("website_url", models.URLField(blank=True, max_length=500, null=True)),
|
||||
],
|
||||
options={
|
||||
"get_latest_by": "modified",
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="paper",
|
||||
name="_journal_tmp",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="books.journal",
|
||||
),
|
||||
),
|
||||
migrations.RunPython(migrate_journal_data, reverse_migrate_journal_data),
|
||||
migrations.RemoveField(
|
||||
model_name="paper",
|
||||
name="journal",
|
||||
),
|
||||
migrations.RenameField(
|
||||
model_name="paper",
|
||||
old_name="_journal_tmp",
|
||||
new_name="journal",
|
||||
),
|
||||
]
|
||||
@ -27,7 +27,11 @@ from books.sources.amazon import lookup_book_from_amazon
|
||||
from books.sources.openlibrary import (
|
||||
lookup_book_from_openlibrary as lookup_book_from_ol,
|
||||
)
|
||||
from books.sources.semantic import lookup_paper_from_semantic
|
||||
from books.sources.semantic import (
|
||||
lookup_paper_from_semantic,
|
||||
lookup_paper_from_semantic_by_doi,
|
||||
)
|
||||
from books.sources.scihub import SciHubService
|
||||
from books.utils import get_comic_issue_url
|
||||
from django.conf import settings
|
||||
from django.contrib.auth import get_user_model
|
||||
@ -82,6 +86,16 @@ class BookLogData(BaseLogData, LongPlayLogData):
|
||||
return int(total_duration / len(self.page_data))
|
||||
|
||||
|
||||
class Journal(TimeStampedModel):
|
||||
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
|
||||
title = models.CharField(max_length=255)
|
||||
description = models.TextField(**BNULL)
|
||||
website_url = models.URLField(max_length=500, **BNULL)
|
||||
|
||||
def __str__(self):
|
||||
return self.title
|
||||
|
||||
|
||||
class Author(TimeStampedModel):
|
||||
name = models.CharField(max_length=255)
|
||||
uuid = models.UUIDField(default=uuid4, editable=False, **BNULL)
|
||||
@ -225,7 +239,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
@property
|
||||
def resume_start_url(self):
|
||||
return reverse("scrobbles:start", kwargs={"uuid": self.uuid}) + "?resume=1"
|
||||
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid}) + "?resume=1"
|
||||
|
||||
@classmethod
|
||||
def get_from_comicvine(
|
||||
@ -540,6 +554,21 @@ class Book(LongPlayScrobblableMixin):
|
||||
return progress
|
||||
|
||||
|
||||
@dataclass
|
||||
class PaperLogData(BaseLogData):
|
||||
@classmethod
|
||||
def override_fields(cls) -> dict:
|
||||
from scrobbles.forms import NotesDictField
|
||||
|
||||
fields = {}
|
||||
for base in cls.mro()[1:]:
|
||||
if hasattr(base, "override_fields"):
|
||||
base_fields = base.override_fields()
|
||||
fields.update(base_fields)
|
||||
fields["notes"] = NotesDictField(required=False)
|
||||
return fields
|
||||
|
||||
|
||||
class Paper(LongPlayScrobblableMixin):
|
||||
"""Keeps track of Academic Papers"""
|
||||
|
||||
@ -559,14 +588,29 @@ class Paper(LongPlayScrobblableMixin):
|
||||
language = models.CharField(max_length=4, **BNULL)
|
||||
first_publish_year = models.IntegerField(**BNULL)
|
||||
publish_date = models.DateField(**BNULL)
|
||||
journal = models.CharField(max_length=255, **BNULL)
|
||||
journal = models.ForeignKey(Journal, on_delete=models.DO_NOTHING, **BNULL)
|
||||
journal_volume = models.CharField(max_length=50, **BNULL)
|
||||
abstract = models.TextField(**BNULL)
|
||||
tldr = models.CharField(max_length=255, **BNULL)
|
||||
openaccess_pdf_url = models.CharField(max_length=255, **BNULL)
|
||||
pdf_file = models.FileField(upload_to="papers/pdf/", **BNULL)
|
||||
|
||||
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
|
||||
|
||||
@property
|
||||
def logdata_cls(self):
|
||||
return PaperLogData
|
||||
|
||||
@property
|
||||
def scihub_url(self):
|
||||
if not self.doi_id:
|
||||
return None
|
||||
domain = getattr(settings, "SCIHUB_DOMAIN", "sci-hub.st")
|
||||
return f"https://{domain}/{self.doi_id}"
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("books:paper_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@classmethod
|
||||
def get_from_semantic(cls, title: str, overwrite: bool = False) -> "Paper":
|
||||
paper, created = cls.objects.get_or_create(title=title)
|
||||
@ -577,7 +621,7 @@ class Paper(LongPlayScrobblableMixin):
|
||||
|
||||
if created or overwrite:
|
||||
author_list = []
|
||||
author_dicts = paper_dict.pop("author_dicts")
|
||||
author_dicts = paper_dict.pop("author_dicts", None)
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
if author_dict.get("authorId"):
|
||||
@ -588,8 +632,11 @@ class Paper(LongPlayScrobblableMixin):
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
# TODO enrich author?
|
||||
...
|
||||
|
||||
journal_name = paper_dict.pop("journal_name", None)
|
||||
if journal_name:
|
||||
journal, _ = Journal.objects.get_or_create(title=journal_name)
|
||||
paper.journal = journal
|
||||
|
||||
for k, v in paper_dict.items():
|
||||
setattr(paper, k, v)
|
||||
@ -601,3 +648,78 @@ class Paper(LongPlayScrobblableMixin):
|
||||
if genres:
|
||||
paper.genre.add(*genres)
|
||||
return paper
|
||||
|
||||
@classmethod
|
||||
def find_or_create_by_doi(cls, doi_url: str) -> "Paper":
|
||||
doi = doi_url.replace("https://doi.org/", "").split("?")[0].rstrip("/")
|
||||
paper = cls.objects.filter(doi_id=doi).first()
|
||||
if paper:
|
||||
return paper
|
||||
|
||||
paper = cls(doi_id=doi, title=f"Paper {doi}")
|
||||
paper.save()
|
||||
|
||||
from books.sources.crossref import lookup_paper_from_crossref
|
||||
|
||||
paper_dict = lookup_paper_from_semantic_by_doi(doi)
|
||||
if not paper_dict or not paper_dict.get("abstract"):
|
||||
paper_dict = lookup_paper_from_crossref(doi)
|
||||
|
||||
if paper_dict:
|
||||
author_list = []
|
||||
author_dicts = paper_dict.pop("author_dicts", None)
|
||||
if author_dicts:
|
||||
for author_dict in author_dicts:
|
||||
author_id = author_dict.get("authorId")
|
||||
if author_id:
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
semantic_id=author_id
|
||||
)
|
||||
author_list.append(author)
|
||||
if a_created:
|
||||
author.name = author_dict.get("name")
|
||||
author.save()
|
||||
else:
|
||||
author_name = author_dict.get("name")
|
||||
if author_name:
|
||||
author, a_created = Author.objects.get_or_create(
|
||||
name=author_name
|
||||
)
|
||||
author_list.append(author)
|
||||
|
||||
journal_name = paper_dict.pop("journal_name", None)
|
||||
if journal_name:
|
||||
journal, _ = Journal.objects.get_or_create(title=journal_name)
|
||||
paper.journal = journal
|
||||
|
||||
for k, v in paper_dict.items():
|
||||
if v is not None:
|
||||
setattr(paper, k, v)
|
||||
paper.save()
|
||||
|
||||
if author_list:
|
||||
paper.authors.add(*author_list)
|
||||
genres = paper_dict.pop("genres", [])
|
||||
if genres:
|
||||
paper.genre.add(*genres)
|
||||
|
||||
if not paper.pdf_file:
|
||||
service = SciHubService()
|
||||
if paper.openaccess_pdf_url:
|
||||
pdf_content = service.fetch_from_url(paper.openaccess_pdf_url)
|
||||
if pdf_content:
|
||||
filename = f"{doi.replace('/', '_')}.pdf"
|
||||
paper.pdf_file.save(filename, ContentFile(pdf_content))
|
||||
if not paper.pdf_file:
|
||||
try:
|
||||
pdf_content = service.fetch_pdf(doi)
|
||||
if pdf_content:
|
||||
filename = f"{doi.replace('/', '_')}.pdf"
|
||||
paper.pdf_file.save(filename, ContentFile(pdf_content))
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
"[paper] sci-hub PDF download failed",
|
||||
extra={"doi": doi, "error": str(e)},
|
||||
)
|
||||
|
||||
return paper
|
||||
|
||||
149
vrobbler/apps/books/sources/crossref.py
Normal file
149
vrobbler/apps/books/sources/crossref.py
Normal file
@ -0,0 +1,149 @@
|
||||
import json
|
||||
import logging
|
||||
import re
|
||||
|
||||
import requests
|
||||
import yake
|
||||
|
||||
CROSSREF_WORK_URL = "https://api.crossref.org/works/{}"
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
_STOPWORDS = {
|
||||
"this", "that", "these", "those", "the", "a", "an", "in", "on", "at",
|
||||
"to", "for", "of", "and", "or", "is", "are", "was", "were", "be",
|
||||
"been", "being", "have", "has", "had", "do", "does", "did", "will",
|
||||
"would", "can", "could", "may", "might", "shall", "should", "not",
|
||||
"no", "nor", "with", "from", "by", "as", "at", "but", "if", "because",
|
||||
"while", "although", "however", "we", "our", "their", "its", "it",
|
||||
"they", "them", "also", "more", "most", "new", "such", "into",
|
||||
"across", "between", "through", "about", "after", "before", "during",
|
||||
"within", "without", "other", "many", "some", "each", "every", "both",
|
||||
"few", "own", "via",
|
||||
}
|
||||
|
||||
_DROP_PHRASES = {
|
||||
"paper", "study", "studies", "research", "introduction", "conclusion",
|
||||
"conclusions", "background", "methods", "results", "findings",
|
||||
"analysis", "approach", "approaches", "framework", "theory",
|
||||
"theories", "concept", "concepts", "model", "models", "process",
|
||||
"processes", "role", "roles", "factor", "factors", "effect",
|
||||
"effects", "impact", "implication", "implications", "actor", "actors",
|
||||
"article", "chapter", "section", "discussion", "review", "overview",
|
||||
"summary", "methodology", "special issue", "implications",
|
||||
"limitations", "findings", "purpose", "objective", "objectives",
|
||||
"design", "setting", "participants", "sample", "data",
|
||||
"contemporary", "little", "empirical", "theoretical",
|
||||
"organizations", "dissent",
|
||||
}
|
||||
|
||||
|
||||
def _strip_jats(text: str) -> str:
|
||||
if not text:
|
||||
return ""
|
||||
text = re.sub(r"</?jats:[^>]*>", "", text)
|
||||
text = re.sub(r"^\s*Abstract\s*", "", text)
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _extract_genres_from_abstract(abstract: str, max_keywords: int = 8) -> list[str]:
|
||||
if not abstract or len(abstract) < 50:
|
||||
return []
|
||||
kw_extractor = yake.KeywordExtractor(lan="en", n=2, top=max_keywords)
|
||||
keywords = kw_extractor.extract_keywords(abstract)
|
||||
genres = []
|
||||
seen = set()
|
||||
for kw, score in keywords:
|
||||
kw_lower = kw.lower().strip()
|
||||
if kw_lower in seen or kw_lower in _DROP_PHRASES:
|
||||
continue
|
||||
words = [w for w in kw_lower.split() if w not in _STOPWORDS]
|
||||
cleaned = " ".join(words)
|
||||
if not cleaned or len(cleaned) < 3 or cleaned in seen:
|
||||
continue
|
||||
if cleaned in _DROP_PHRASES:
|
||||
continue
|
||||
seen.add(cleaned)
|
||||
genres.append(cleaned)
|
||||
return genres
|
||||
|
||||
|
||||
def lookup_paper_from_crossref(doi: str) -> dict:
|
||||
url = CROSSREF_WORK_URL.format(doi)
|
||||
headers = {"User-Agent": "Vrobbler/1.0 (mailto:hello@example.com)"}
|
||||
response = requests.get(url, headers=headers)
|
||||
|
||||
if response.status_code != 200:
|
||||
logger.warning(
|
||||
"Bad response from Crossref",
|
||||
extra={"doi": doi, "status": response.status_code},
|
||||
)
|
||||
return {"doi_id": doi}
|
||||
|
||||
try:
|
||||
data = response.json()
|
||||
except json.JSONDecodeError:
|
||||
return {"doi_id": doi}
|
||||
|
||||
msg = data.get("message", {})
|
||||
if not msg:
|
||||
return {"doi_id": doi}
|
||||
|
||||
paper_dict = {"doi_id": doi}
|
||||
|
||||
titles = msg.get("title", [])
|
||||
if titles:
|
||||
paper_dict["title"] = titles[0]
|
||||
|
||||
abstract = msg.get("abstract", "")
|
||||
if abstract:
|
||||
stripped = _strip_jats(abstract)
|
||||
paper_dict["abstract"] = stripped
|
||||
genres = _extract_genres_from_abstract(stripped)
|
||||
if genres:
|
||||
paper_dict["genres"] = genres
|
||||
|
||||
author_dicts = []
|
||||
for author in msg.get("author", []):
|
||||
given = author.get("given", "")
|
||||
family = author.get("family", "")
|
||||
name = f"{given} {family}".strip()
|
||||
if not name:
|
||||
continue
|
||||
entry = {"name": name}
|
||||
orcid = author.get("ORCID", "")
|
||||
if orcid:
|
||||
orcid_id = orcid.replace("https://orcid.org/", "")
|
||||
entry["authorId"] = orcid_id
|
||||
author_dicts.append(entry)
|
||||
if author_dicts:
|
||||
paper_dict["author_dicts"] = author_dicts
|
||||
|
||||
container = msg.get("container-title", [])
|
||||
if container:
|
||||
paper_dict["journal_name"] = container[0]
|
||||
|
||||
volume = msg.get("volume")
|
||||
if volume:
|
||||
paper_dict["journal_volume"] = volume
|
||||
|
||||
page = msg.get("page")
|
||||
if page:
|
||||
try:
|
||||
parts = page.split("-")
|
||||
if len(parts) == 2:
|
||||
paper_dict["pages"] = int(parts[1]) - int(parts[0])
|
||||
except (ValueError, IndexError):
|
||||
pass
|
||||
|
||||
for date_field in ("published-print", "published-online", "created"):
|
||||
date_data = msg.get(date_field)
|
||||
if date_data and date_data.get("date-parts"):
|
||||
parts = date_data["date-parts"][0]
|
||||
if len(parts) >= 1:
|
||||
paper_dict["first_publish_year"] = int(parts[0])
|
||||
if len(parts) >= 3:
|
||||
paper_dict["publish_date"] = f"{parts[0]:04d}-{parts[1]:02d}-{parts[2]:02d}"
|
||||
break
|
||||
|
||||
return paper_dict
|
||||
142
vrobbler/apps/books/sources/scihub.py
Normal file
142
vrobbler/apps/books/sources/scihub.py
Normal file
@ -0,0 +1,142 @@
|
||||
import logging
|
||||
from typing import Optional
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import requests
|
||||
from bs4 import BeautifulSoup
|
||||
from django.conf import settings
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
SCIHUB_DOMAINS = [
|
||||
"sci-hub.ru",
|
||||
"sci-hub.ee",
|
||||
"sci-hub.st",
|
||||
"sci-hub.do",
|
||||
]
|
||||
|
||||
|
||||
class SciHubService:
|
||||
def __init__(self):
|
||||
self.session = requests.Session()
|
||||
self.session.headers.update(
|
||||
{
|
||||
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
|
||||
}
|
||||
)
|
||||
|
||||
def fetch_from_url(self, url: str) -> Optional[bytes]:
|
||||
try:
|
||||
resp = self.session.get(url, timeout=60)
|
||||
if resp.status_code != 200:
|
||||
logger.warning(
|
||||
"[pdf] URL download failed",
|
||||
extra={"status": resp.status_code, "url": url},
|
||||
)
|
||||
return None
|
||||
if not self._looks_like_pdf(resp):
|
||||
return None
|
||||
return resp.content
|
||||
except requests.RequestException as e:
|
||||
logger.error(
|
||||
"[pdf] URL download request failed",
|
||||
extra={"url": url, "error": str(e)},
|
||||
)
|
||||
return None
|
||||
|
||||
def fetch_pdf(self, doi: str) -> Optional[bytes]:
|
||||
configured_domain = getattr(settings, "SCIHUB_DOMAIN", None)
|
||||
domains_to_try = (
|
||||
[configured_domain] + SCIHUB_DOMAINS
|
||||
if configured_domain and configured_domain not in SCIHUB_DOMAINS
|
||||
else SCIHUB_DOMAINS
|
||||
)
|
||||
|
||||
for domain in domains_to_try:
|
||||
url = f"https://{domain}/{doi}"
|
||||
logger.info(
|
||||
"[scihub] trying domain",
|
||||
extra={"domain": domain, "doi": doi},
|
||||
)
|
||||
try:
|
||||
response = self.session.get(url, timeout=30)
|
||||
if response.status_code != 200:
|
||||
continue
|
||||
|
||||
pdf_url = self._extract_pdf_url(response.text, url)
|
||||
if not pdf_url:
|
||||
continue
|
||||
|
||||
pdf_response = self.session.get(pdf_url, timeout=60)
|
||||
if pdf_response.status_code != 200:
|
||||
continue
|
||||
|
||||
if not self._looks_like_pdf(pdf_response):
|
||||
continue
|
||||
|
||||
logger.info(
|
||||
"[scihub] PDF downloaded successfully",
|
||||
extra={
|
||||
"domain": domain,
|
||||
"doi": doi,
|
||||
"size": len(pdf_response.content),
|
||||
},
|
||||
)
|
||||
return pdf_response.content
|
||||
except requests.RequestException as e:
|
||||
logger.debug(
|
||||
"[scihub] domain failed",
|
||||
extra={"domain": domain, "doi": doi, "error": str(e)},
|
||||
)
|
||||
continue
|
||||
|
||||
logger.warning(
|
||||
"[scihub] all domains failed",
|
||||
extra={"doi": doi, "tried": domains_to_try},
|
||||
)
|
||||
return None
|
||||
|
||||
def _looks_like_pdf(self, response: requests.Response) -> bool:
|
||||
content_type = response.headers.get("Content-Type", "")
|
||||
if "application/pdf" in content_type:
|
||||
return True
|
||||
if content_type.startswith("application/octet"):
|
||||
return True
|
||||
if response.url.endswith(".pdf"):
|
||||
return True
|
||||
return False
|
||||
|
||||
def _extract_pdf_url(self, html: str, page_url: str) -> Optional[str]:
|
||||
soup = BeautifulSoup(html, "html.parser")
|
||||
|
||||
iframe = soup.find("iframe", {"id": "pdf"})
|
||||
if iframe and iframe.get("src"):
|
||||
src = iframe["src"]
|
||||
if src.startswith("http"):
|
||||
return src
|
||||
return urljoin(page_url, src)
|
||||
|
||||
embed = soup.find("embed", {"type": "application/pdf"})
|
||||
if embed and embed.get("src"):
|
||||
src = embed["src"]
|
||||
if src.startswith("http"):
|
||||
return src
|
||||
return urljoin(page_url, src)
|
||||
|
||||
download_div = soup.find("div", {"id": "download"})
|
||||
if download_div:
|
||||
link = download_div.find("a")
|
||||
if link and link.get("href"):
|
||||
href = link["href"]
|
||||
if href.startswith("http"):
|
||||
return href
|
||||
return urljoin(page_url, href)
|
||||
|
||||
for link in soup.find_all("a", href=True):
|
||||
href = link["href"]
|
||||
if ".pdf" in href:
|
||||
if href.startswith("http"):
|
||||
return href
|
||||
return urljoin(page_url, href)
|
||||
|
||||
return None
|
||||
@ -9,6 +9,7 @@ PAPER_SEARCH_URL = (
|
||||
"https://api.semanticscholar.org/graph/v1/paper/search/match?query={}"
|
||||
)
|
||||
PAPER_DETAIL_URL = "https://api.semanticscholar.org/graph/v1/paper/{}?fields=title,authors,url,year,abstract,externalIds,citationCount,referenceCount,journal,fieldsOfStudy,publicationDate,openAccessPdf"
|
||||
PAPER_DOI_URL = "https://api.semanticscholar.org/graph/v1/paper/DOI:{}?fields=title,authors,url,year,abstract,externalIds,citationCount,referenceCount,journal,fieldsOfStudy,publicationDate,openAccessPdf"
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -39,6 +40,18 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
if not result:
|
||||
return paper_dict
|
||||
|
||||
paper_dict.update(_parse_semantic_result(result))
|
||||
paper_dict.setdefault("title", title)
|
||||
if paper_dict.get("publish_date"):
|
||||
paper_dict["publish_date"] = datetime.strptime(
|
||||
paper_dict["publish_date"], "%Y-%m-%d"
|
||||
)
|
||||
|
||||
return paper_dict
|
||||
|
||||
|
||||
def _parse_semantic_result(result: dict) -> dict:
|
||||
paper_dict = {}
|
||||
page_str = result.get("journal", {}).get("pages")
|
||||
if page_str:
|
||||
try:
|
||||
@ -55,12 +68,10 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
paper_dict["corpus_id"] = result.get("externalIds", {}).get("CorpusId")
|
||||
paper_dict["semantic_title"] = result.get("title")
|
||||
paper_dict["first_publish_year"] = result.get("year")
|
||||
paper_dict["publish_date"] = datetime.strptime(
|
||||
result.get("publicationDate", "1950-01-01"), "%Y-%m-%d"
|
||||
)
|
||||
paper_dict["publish_date"] = result.get("publicationDate")
|
||||
paper_dict["abstract"] = result.get("abstract")
|
||||
paper_dict["tldr"] = result.get("bib", {}).get("abstract")
|
||||
paper_dict["journal"] = result.get("journal", {}).get("name")
|
||||
paper_dict["journal_name"] = result.get("journal", {}).get("name")
|
||||
paper_dict["journal_volume"] = result.get("journal", {}).get("volume")
|
||||
paper_dict["openaccess_pdf_url"] = result.get("openAccessPdf", {}).get("url")
|
||||
paper_dict["base_run_time_seconds"] = paper_dict.get("pages", 10) * getattr(
|
||||
@ -68,5 +79,19 @@ def lookup_paper_from_semantic(title: str) -> dict:
|
||||
)
|
||||
paper_dict["author_dicts"] = result.get("authors")
|
||||
paper_dict["genres"] = result.get("fieldsOfStudy")
|
||||
|
||||
return paper_dict
|
||||
|
||||
|
||||
def lookup_paper_from_semantic_by_doi(doi: str) -> dict:
|
||||
response = get_api_result(PAPER_DOI_URL.format(doi))
|
||||
if not response:
|
||||
return {"doi_id": doi}
|
||||
|
||||
result = json.loads(response.content)
|
||||
if not result:
|
||||
return {"doi_id": doi}
|
||||
|
||||
paper_dict = _parse_semantic_result(result)
|
||||
if not paper_dict.get("title"):
|
||||
paper_dict["title"] = result.get("title", f"Paper {doi}")
|
||||
return paper_dict
|
||||
|
||||
@ -16,4 +16,15 @@ urlpatterns = [
|
||||
views.AuthorDetailView.as_view(),
|
||||
name="author_detail",
|
||||
),
|
||||
path("papers/", views.PaperListView.as_view(), name="paper_list"),
|
||||
path(
|
||||
"papers/<slug:slug>/",
|
||||
views.PaperDetailView.as_view(),
|
||||
name="paper_detail",
|
||||
),
|
||||
path(
|
||||
"papers/<slug:slug>/upload_pdf/",
|
||||
views.PaperUploadPdfView.as_view(),
|
||||
name="paper_upload_pdf",
|
||||
),
|
||||
]
|
||||
|
||||
@ -1,5 +1,8 @@
|
||||
from django.http import HttpResponseRedirect
|
||||
from django.urls import reverse
|
||||
from django.views import View
|
||||
from django.views import generic
|
||||
from books.models import Book, Author
|
||||
from books.models import Book, Author, Paper
|
||||
|
||||
from scrobbles.views import ScrobbleableListView, ScrobbleableDetailView
|
||||
|
||||
@ -15,3 +18,24 @@ class BookDetailView(ScrobbleableDetailView):
|
||||
class AuthorDetailView(generic.DetailView):
|
||||
model = Author
|
||||
slug_field = "uuid"
|
||||
|
||||
|
||||
class PaperListView(ScrobbleableListView):
|
||||
model = Paper
|
||||
|
||||
|
||||
class PaperDetailView(ScrobbleableDetailView):
|
||||
model = Paper
|
||||
|
||||
|
||||
class PaperUploadPdfView(View):
|
||||
def post(self, request, slug):
|
||||
paper = Paper.objects.filter(uuid=slug).first()
|
||||
if not paper or not request.user.is_authenticated:
|
||||
return HttpResponseRedirect(reverse("books:paper_detail", args=[slug]))
|
||||
|
||||
pdf_file = request.FILES.get("pdf_file")
|
||||
if pdf_file:
|
||||
paper.pdf_file.save(pdf_file.name, pdf_file)
|
||||
|
||||
return HttpResponseRedirect(reverse("books:paper_detail", args=[slug]))
|
||||
|
||||
@ -0,0 +1,109 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-19 16:25
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("charts", "0002_chartrecord_charts_char_user_id_1adcde_idx_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("artist__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "artist"),
|
||||
name="unique_chart_artist_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("album__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "album"),
|
||||
name="unique_chart_album_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("track__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "track"),
|
||||
name="unique_chart_track_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("tv_series__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "tv_series"),
|
||||
name="unique_chart_tv_series_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("video__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "video"),
|
||||
name="unique_chart_video_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("podcast__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "podcast"),
|
||||
name="unique_chart_podcast_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("podcast_episode__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "podcast_episode"),
|
||||
name="unique_chart_podcast_episode_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("board_game__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "board_game"),
|
||||
name="unique_chart_board_game_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("trail__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "trail"),
|
||||
name="unique_chart_trail_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("geo_location__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "geo_location"),
|
||||
name="unique_chart_geo_location_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("food__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "food"),
|
||||
name="unique_chart_food_period",
|
||||
),
|
||||
),
|
||||
migrations.AddConstraint(
|
||||
model_name="chartrecord",
|
||||
constraint=models.UniqueConstraint(
|
||||
condition=models.Q(("book__isnull", False)),
|
||||
fields=("user", "year", "month", "week", "day", "book"),
|
||||
name="unique_chart_book_period",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -2,6 +2,7 @@ import calendar
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.db import models
|
||||
from django.db.models import Q
|
||||
from django.urls import reverse
|
||||
from django_extensions.db.models import TimeStampedModel
|
||||
|
||||
@ -84,6 +85,68 @@ class ChartRecord(TimeStampedModel):
|
||||
models.Index(fields=["user", "year", "month", "day", "album", "rank"]),
|
||||
models.Index(fields=["user", "year", "month", "day", "tv_series", "rank"]),
|
||||
]
|
||||
constraints = [
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "artist"],
|
||||
condition=Q(artist__isnull=False),
|
||||
name="unique_chart_artist_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "album"],
|
||||
condition=Q(album__isnull=False),
|
||||
name="unique_chart_album_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "track"],
|
||||
condition=Q(track__isnull=False),
|
||||
name="unique_chart_track_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "tv_series"],
|
||||
condition=Q(tv_series__isnull=False),
|
||||
name="unique_chart_tv_series_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "video"],
|
||||
condition=Q(video__isnull=False),
|
||||
name="unique_chart_video_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "podcast"],
|
||||
condition=Q(podcast__isnull=False),
|
||||
name="unique_chart_podcast_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "podcast_episode"],
|
||||
condition=Q(podcast_episode__isnull=False),
|
||||
name="unique_chart_podcast_episode_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "board_game"],
|
||||
condition=Q(board_game__isnull=False),
|
||||
name="unique_chart_board_game_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "trail"],
|
||||
condition=Q(trail__isnull=False),
|
||||
name="unique_chart_trail_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "geo_location"],
|
||||
condition=Q(geo_location__isnull=False),
|
||||
name="unique_chart_geo_location_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "food"],
|
||||
condition=Q(food__isnull=False),
|
||||
name="unique_chart_food_period",
|
||||
),
|
||||
models.UniqueConstraint(
|
||||
fields=["user", "year", "month", "week", "day", "book"],
|
||||
condition=Q(book__isnull=False),
|
||||
name="unique_chart_book_period",
|
||||
),
|
||||
]
|
||||
|
||||
@property
|
||||
def media_obj(self):
|
||||
|
||||
@ -6,6 +6,7 @@ from typing import Optional
|
||||
import pytz
|
||||
from django.apps import apps
|
||||
from django.conf import settings
|
||||
from django.db import transaction
|
||||
from django.db.models import Count, Q
|
||||
from django.utils import timezone
|
||||
|
||||
@ -186,60 +187,64 @@ def build_charts(
|
||||
ranks = {count: rank for rank, count in enumerate(unique_counts, start=1)}
|
||||
|
||||
media_field = f"{media_type}_id"
|
||||
records_to_create = []
|
||||
records_to_update = []
|
||||
|
||||
existing = ChartRecord.objects.filter(
|
||||
period_filter, user=user, **{media_field + "__isnull": False}
|
||||
)
|
||||
existing_by_media_id = {getattr(r, media_field): r for r in existing}
|
||||
found_media_ids = set()
|
||||
with transaction.atomic():
|
||||
records_to_create = []
|
||||
records_to_update = []
|
||||
|
||||
for result in results:
|
||||
media_id = result[config["values"]]
|
||||
if media_id is None:
|
||||
continue
|
||||
|
||||
found_media_ids.add(media_id)
|
||||
|
||||
chart_record_data = {
|
||||
"user_id": user.id,
|
||||
"year": year,
|
||||
"month": month,
|
||||
"week": week,
|
||||
"day": day,
|
||||
"rank": ranks[result["scrobble_count"]],
|
||||
"count": result["scrobble_count"],
|
||||
}
|
||||
chart_record_data[media_field] = media_id
|
||||
|
||||
if media_id in existing_by_media_id:
|
||||
existing_record = existing_by_media_id[media_id]
|
||||
existing_record.rank = chart_record_data["rank"]
|
||||
existing_record.count = chart_record_data["count"]
|
||||
records_to_update.append(existing_record)
|
||||
else:
|
||||
records_to_create.append(ChartRecord(**chart_record_data))
|
||||
|
||||
ids_to_delete = [
|
||||
r.id for r in existing if getattr(r, media_field) not in found_media_ids
|
||||
]
|
||||
if ids_to_delete:
|
||||
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
|
||||
|
||||
if records_to_update:
|
||||
ChartRecord.objects.bulk_update(
|
||||
records_to_update, ["rank", "count"], batch_size=500
|
||||
existing = ChartRecord.objects.select_for_update().filter(
|
||||
period_filter, user=user, **{media_field + "__isnull": False}
|
||||
)
|
||||
existing_by_media_id = {getattr(r, media_field): r for r in existing}
|
||||
found_media_ids = set()
|
||||
|
||||
if records_to_create:
|
||||
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
|
||||
for result in results:
|
||||
media_id = result[config["values"]]
|
||||
if media_id is None:
|
||||
continue
|
||||
|
||||
logger.info(
|
||||
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
|
||||
f"chart records for {media_type}, period "
|
||||
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
|
||||
)
|
||||
found_media_ids.add(media_id)
|
||||
|
||||
chart_record_data = {
|
||||
"user_id": user.id,
|
||||
"year": year,
|
||||
"month": month,
|
||||
"week": week,
|
||||
"day": day,
|
||||
"rank": ranks[result["scrobble_count"]],
|
||||
"count": result["scrobble_count"],
|
||||
}
|
||||
chart_record_data[media_field] = media_id
|
||||
|
||||
if media_id in existing_by_media_id:
|
||||
existing_record = existing_by_media_id[media_id]
|
||||
existing_record.rank = chart_record_data["rank"]
|
||||
existing_record.count = chart_record_data["count"]
|
||||
records_to_update.append(existing_record)
|
||||
else:
|
||||
records_to_create.append(ChartRecord(**chart_record_data))
|
||||
|
||||
ids_to_delete = [
|
||||
r.id
|
||||
for r in existing
|
||||
if getattr(r, media_field) not in found_media_ids
|
||||
]
|
||||
if ids_to_delete:
|
||||
ChartRecord.objects.filter(id__in=ids_to_delete).delete()
|
||||
|
||||
if records_to_update:
|
||||
ChartRecord.objects.bulk_update(
|
||||
records_to_update, ["rank", "count"], batch_size=500
|
||||
)
|
||||
|
||||
if records_to_create:
|
||||
ChartRecord.objects.bulk_create(records_to_create, batch_size=500)
|
||||
|
||||
logger.info(
|
||||
f"Built {len(records_to_create)} new, {len(records_to_update)} updated "
|
||||
f"chart records for {media_type}, period "
|
||||
f"{year}-{month or ''}-{week or ''}-{day or ''} for {user}"
|
||||
)
|
||||
|
||||
|
||||
def build_yesterdays_charts(user, media_types: Optional[list] = None) -> None:
|
||||
|
||||
0
vrobbler/apps/discgolf/__init__.py
Normal file
0
vrobbler/apps/discgolf/__init__.py
Normal file
14
vrobbler/apps/discgolf/admin.py
Normal file
14
vrobbler/apps/discgolf/admin.py
Normal file
@ -0,0 +1,14 @@
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from django.contrib import admin
|
||||
from scrobbles.admin import ScrobbleInline
|
||||
|
||||
|
||||
@admin.register(DiscGolfCourse)
|
||||
class DiscGolfCourseAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = ("title", "layout_name", "number_of_holes", "par_total", "pdga_slug", "udisc_id")
|
||||
raw_id_fields = ("trail",)
|
||||
search_fields = ("title", "layout_name")
|
||||
inlines = [
|
||||
ScrobbleInline,
|
||||
]
|
||||
0
vrobbler/apps/discgolf/api/__init__.py
Normal file
0
vrobbler/apps/discgolf/api/__init__.py
Normal file
14
vrobbler/apps/discgolf/api/serializers.py
Normal file
14
vrobbler/apps/discgolf/api/serializers.py
Normal file
@ -0,0 +1,14 @@
|
||||
from discgolf import models
|
||||
from rest_framework import serializers
|
||||
|
||||
|
||||
class DiscGolfCourseSerializer(serializers.HyperlinkedModelSerializer):
|
||||
pdga_link = serializers.ReadOnlyField()
|
||||
udisc_link = serializers.ReadOnlyField()
|
||||
|
||||
class Meta:
|
||||
model = models.DiscGolfCourse
|
||||
fields = "__all__"
|
||||
|
||||
|
||||
|
||||
13
vrobbler/apps/discgolf/api/views.py
Normal file
13
vrobbler/apps/discgolf/api/views.py
Normal file
@ -0,0 +1,13 @@
|
||||
from rest_framework import permissions, viewsets
|
||||
|
||||
from discgolf.api import serializers
|
||||
from discgolf import models
|
||||
|
||||
|
||||
class DiscGolfCourseViewSet(viewsets.ModelViewSet):
|
||||
queryset = models.DiscGolfCourse.objects.all().order_by("-created")
|
||||
serializer_class = serializers.DiscGolfCourseSerializer
|
||||
permission_classes = [permissions.IsAuthenticated]
|
||||
|
||||
|
||||
|
||||
6
vrobbler/apps/discgolf/apps.py
Normal file
6
vrobbler/apps/discgolf/apps.py
Normal file
@ -0,0 +1,6 @@
|
||||
from django.apps import AppConfig
|
||||
|
||||
|
||||
class DiscgolfConfig(AppConfig):
|
||||
default_auto_field = "django.db.models.BigAutoField"
|
||||
name = "discgolf"
|
||||
0
vrobbler/apps/discgolf/management/__init__.py
Normal file
0
vrobbler/apps/discgolf/management/__init__.py
Normal file
83
vrobbler/apps/discgolf/migrations/0001_initial.py
Normal file
83
vrobbler/apps/discgolf/migrations/0001_initial.py
Normal file
@ -0,0 +1,83 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:19
|
||||
|
||||
from django.db import migrations, models
|
||||
import django_extensions.db.fields
|
||||
import taggit.managers
|
||||
import uuid
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
initial = True
|
||||
|
||||
dependencies = [
|
||||
("taggit", "0004_alter_taggeditem_content_type_alter_taggeditem_tag"),
|
||||
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.CreateModel(
|
||||
name="DiscGolfCourse",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
(
|
||||
"uuid",
|
||||
models.UUIDField(
|
||||
blank=True, default=uuid.uuid4, editable=False, null=True
|
||||
),
|
||||
),
|
||||
("title", models.CharField(blank=True, max_length=255, null=True)),
|
||||
("base_run_time_seconds", models.IntegerField(blank=True, null=True)),
|
||||
("description", models.TextField(blank=True, null=True)),
|
||||
(
|
||||
"layout_name",
|
||||
models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
("number_of_holes", models.IntegerField(blank=True, null=True)),
|
||||
("par_total", models.IntegerField(blank=True, null=True)),
|
||||
(
|
||||
"genre",
|
||||
taggit.managers.TaggableManager(
|
||||
blank=True,
|
||||
help_text="A comma-separated list of tags.",
|
||||
through="scrobbles.ObjectWithGenres",
|
||||
to="scrobbles.Genre",
|
||||
verbose_name="Genre",
|
||||
),
|
||||
),
|
||||
(
|
||||
"tags",
|
||||
taggit.managers.TaggableManager(
|
||||
blank=True,
|
||||
help_text="A comma-separated list of tags.",
|
||||
through="taggit.TaggedItem",
|
||||
to="taggit.Tag",
|
||||
verbose_name="Tags",
|
||||
),
|
||||
),
|
||||
],
|
||||
options={
|
||||
"abstract": False,
|
||||
},
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,25 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:29
|
||||
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("trails", "0009_trail_route_waypoint"),
|
||||
("discgolf", "0001_initial"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="trail",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="trails.trail",
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:35
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0002_discgolfcourse_trail"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="par_per_hole",
|
||||
field=models.JSONField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,23 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-21 04:16
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0003_discgolfcourse_par_per_hole"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="pdga_slug",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="discgolfcourse",
|
||||
name="udisc_id",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
]
|
||||
0
vrobbler/apps/discgolf/migrations/__init__.py
Normal file
0
vrobbler/apps/discgolf/migrations/__init__.py
Normal file
90
vrobbler/apps/discgolf/models.py
Normal file
90
vrobbler/apps/discgolf/models.py
Normal file
@ -0,0 +1,90 @@
|
||||
from dataclasses import dataclass
|
||||
from typing import Optional, TypedDict
|
||||
|
||||
from django.apps import apps
|
||||
from django.db import models
|
||||
from django.urls import reverse
|
||||
from scrobbles.dataclasses import BaseLogData, WithPeopleLogData
|
||||
from scrobbles.mixins import ScrobblableConstants, ScrobblableMixin
|
||||
|
||||
BNULL = {"blank": True, "null": True}
|
||||
|
||||
|
||||
class DiscGolfSingleScores(TypedDict, total=False):
|
||||
person_id: int
|
||||
total: int
|
||||
|
||||
|
||||
class DiscGolfTeamScores(TypedDict, total=False):
|
||||
person_ids: list[int]
|
||||
total: int
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiscGolfLogData(BaseLogData, WithPeopleLogData):
|
||||
scores: Optional[dict[str, DiscGolfSingleScores | DiscGolfTeamScores]] = None
|
||||
weather: Optional[str] = None
|
||||
fun_factor: Optional[str] = None
|
||||
course_name: Optional[str] = None
|
||||
par: Optional[int] = None
|
||||
round_type: Optional[str] = None
|
||||
|
||||
|
||||
class DiscGolfCourse(ScrobblableMixin):
|
||||
description = models.TextField(**BNULL)
|
||||
layout_name = models.CharField(max_length=255, **BNULL)
|
||||
number_of_holes = models.IntegerField(**BNULL)
|
||||
par_total = models.IntegerField(**BNULL)
|
||||
par_per_hole = models.JSONField(**BNULL)
|
||||
trail = models.ForeignKey(
|
||||
"trails.Trail", on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
pdga_slug = models.CharField(max_length=255, **BNULL)
|
||||
udisc_id = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
@property
|
||||
def pdga_link(self) -> str:
|
||||
if self.pdga_slug:
|
||||
return f"https://www.pdga.com/course-directory/course/{self.pdga_slug}/"
|
||||
return ""
|
||||
|
||||
@property
|
||||
def udisc_link(self) -> str:
|
||||
if self.udisc_id:
|
||||
return f"https://udisc.com/courses/{self.udisc_id}/"
|
||||
return ""
|
||||
|
||||
def get_absolute_url(self) -> str:
|
||||
return reverse("discgolf:course_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
def __str__(self):
|
||||
return f"{self.title} ({self.layout_name or 'Default'})"
|
||||
|
||||
@property
|
||||
def logdata_cls(self):
|
||||
return DiscGolfLogData
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
return self.layout_name or ""
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
return ScrobblableConstants(verb="Playing", tags="golf")
|
||||
|
||||
@property
|
||||
def primary_image_url(self) -> str:
|
||||
return ""
|
||||
|
||||
@classmethod
|
||||
def find_or_create(cls, name: str, **defaults) -> "DiscGolfCourse":
|
||||
course = cls.objects.filter(title=name).first()
|
||||
if not course:
|
||||
course = cls.objects.create(title=name, **defaults)
|
||||
return course
|
||||
|
||||
def scrobbles(self, user_id):
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
return Scrobble.objects.filter(user_id=user_id, disc_golf_course=self).order_by(
|
||||
"-timestamp"
|
||||
)
|
||||
14
vrobbler/apps/discgolf/urls.py
Normal file
14
vrobbler/apps/discgolf/urls.py
Normal file
@ -0,0 +1,14 @@
|
||||
from django.urls import path
|
||||
from discgolf import views
|
||||
|
||||
app_name = "discgolf"
|
||||
|
||||
|
||||
urlpatterns = [
|
||||
path("disc-golf/", views.DiscGolfCourseListView.as_view(), name="course_list"),
|
||||
path(
|
||||
"disc-golf/<slug:slug>/",
|
||||
views.DiscGolfCourseDetailView.as_view(),
|
||||
name="course_detail",
|
||||
),
|
||||
]
|
||||
129
vrobbler/apps/discgolf/utils.py
Normal file
129
vrobbler/apps/discgolf/utils.py
Normal file
@ -0,0 +1,129 @@
|
||||
import csv
|
||||
import logging
|
||||
from datetime import datetime
|
||||
|
||||
from dateutil.parser import parse as parse_datetime
|
||||
from django.utils import timezone
|
||||
from people.models import Person
|
||||
from scrobbles.models import Scrobble
|
||||
from scrobbles.notifications import ScrobbleNtfyNotification
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _parse_udisc_datetime(raw: str) -> datetime:
|
||||
return parse_datetime(raw)
|
||||
|
||||
|
||||
def _resolve_player(name: str, user_id: int) -> Person:
|
||||
name = name.strip()
|
||||
existing = Person.objects.filter(name=name, created_by_id=user_id).first()
|
||||
if existing:
|
||||
return existing
|
||||
return Person.objects.create(name=name, created_by_id=user_id)
|
||||
|
||||
|
||||
def import_udisc_csv(
|
||||
file_path: str, user_id: int, record_error=None
|
||||
) -> list[Scrobble]:
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
new_scrobbles = []
|
||||
|
||||
with open(file_path, newline="", encoding="utf-8-sig") as f:
|
||||
reader = csv.DictReader(f)
|
||||
rows = list(reader)
|
||||
|
||||
if not rows:
|
||||
return []
|
||||
|
||||
par_row = None
|
||||
player_rows = []
|
||||
for row in rows:
|
||||
name = row.get("PlayerName", "").strip()
|
||||
if name.lower() == "par":
|
||||
par_row = row
|
||||
else:
|
||||
player_rows.append(row)
|
||||
|
||||
if not par_row:
|
||||
return []
|
||||
|
||||
course_name = par_row.get("CourseName", "").strip()
|
||||
layout_name = par_row.get("LayoutName", "").strip()
|
||||
start_date_raw = par_row.get("StartDate", "").strip()
|
||||
start_dt = _parse_udisc_datetime(start_date_raw) if start_date_raw else timezone.now()
|
||||
|
||||
number_of_holes = sum(1 for k in par_row if k.startswith("Hole") and k[4:].isdigit())
|
||||
par_total_str = par_row.get("Total", "").strip()
|
||||
par_total = int(par_total_str) if par_total_str.isdigit() else None
|
||||
|
||||
par_per_hole = {}
|
||||
for k, v in par_row.items():
|
||||
if k.startswith("Hole") and k[4:].isdigit() and v:
|
||||
hole_num = int(k[4:])
|
||||
try:
|
||||
par_per_hole[f"hole_{hole_num}"] = int(v)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
|
||||
course, _ = DiscGolfCourse.objects.get_or_create(
|
||||
title=course_name,
|
||||
defaults={
|
||||
"layout_name": layout_name,
|
||||
"number_of_holes": number_of_holes,
|
||||
"par_total": par_total,
|
||||
"par_per_hole": par_per_hole or None,
|
||||
},
|
||||
)
|
||||
|
||||
is_teams = "+" in player_rows[0].get("PlayerName", "") if player_rows else False
|
||||
round_type = "Teams" if is_teams else "Singles"
|
||||
|
||||
scores = {}
|
||||
for row in player_rows:
|
||||
player_name = row.get("PlayerName", "").strip()
|
||||
hole_scores = {}
|
||||
for k, v in row.items():
|
||||
if k.startswith("Hole") and k[4:].isdigit() and v:
|
||||
hole_num = int(k[4:])
|
||||
try:
|
||||
hole_scores[f"hole_{hole_num}"] = int(v)
|
||||
except (ValueError, TypeError):
|
||||
pass
|
||||
total_str = row.get("Total", "").strip()
|
||||
total = int(total_str) if total_str.isdigit() else None
|
||||
if total is not None:
|
||||
hole_scores["total"] = total
|
||||
|
||||
if is_teams:
|
||||
people = player_name.split("+")
|
||||
person_ids = [_resolve_player(p.strip(), user_id).id for p in people]
|
||||
hole_scores["person_ids"] = person_ids
|
||||
else:
|
||||
person = _resolve_player(player_name, user_id)
|
||||
hole_scores["person_id"] = person.id
|
||||
|
||||
scores[player_name] = hole_scores
|
||||
|
||||
log = {
|
||||
"scores": scores,
|
||||
"course_name": course_name,
|
||||
"par": par_total,
|
||||
"round_type": round_type,
|
||||
}
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": start_dt,
|
||||
"source": "uDisc",
|
||||
"playback_position_seconds": 0,
|
||||
"log": log,
|
||||
}
|
||||
|
||||
scrobble = Scrobble.create_or_update(course, user_id, scrobble_dict)
|
||||
if scrobble:
|
||||
new_scrobbles.append(scrobble)
|
||||
ScrobbleNtfyNotification(scrobble).send()
|
||||
|
||||
return new_scrobbles
|
||||
32
vrobbler/apps/discgolf/views.py
Normal file
32
vrobbler/apps/discgolf/views.py
Normal file
@ -0,0 +1,32 @@
|
||||
from django.apps import apps
|
||||
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
from scrobbles.views import (
|
||||
ScrobbleableListView,
|
||||
ScrobbleableDetailView,
|
||||
ChartContextMixin,
|
||||
)
|
||||
|
||||
|
||||
class DiscGolfCourseListView(ScrobbleableListView):
|
||||
model = DiscGolfCourse
|
||||
|
||||
|
||||
class DiscGolfCourseDetailView(ScrobbleableDetailView, ChartContextMixin):
|
||||
model = DiscGolfCourse
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
context = super().get_context_data(**kwargs)
|
||||
Scrobble = apps.get_model("scrobbles", "Scrobble")
|
||||
context["trail_gpx_url"] = None
|
||||
latest = (
|
||||
Scrobble.objects.filter(
|
||||
trail=self.object.trail, gpx_file__isnull=False
|
||||
)
|
||||
.order_by("-timestamp")
|
||||
.first()
|
||||
)
|
||||
if latest and latest.gpx_file:
|
||||
context["trail_gpx_url"] = latest.gpx_file.url
|
||||
return context
|
||||
@ -1,4 +1,7 @@
|
||||
import time
|
||||
|
||||
from django.contrib import admin
|
||||
from django.http import HttpRequest
|
||||
|
||||
from locations.models import GeoLocation
|
||||
|
||||
@ -14,9 +17,29 @@ class GeoLocationAdmin(admin.ModelAdmin):
|
||||
"lon",
|
||||
"title",
|
||||
"altitude",
|
||||
"city",
|
||||
"state_province",
|
||||
"country",
|
||||
)
|
||||
ordering = ("-created",)
|
||||
search_fields = ("title",)
|
||||
actions = ["reverse_geocode_selected"]
|
||||
inlines = [
|
||||
ScrobbleInline,
|
||||
]
|
||||
|
||||
@admin.action(description="Reverse geocode selected locations")
|
||||
def reverse_geocode_selected(self, request: HttpRequest, queryset):
|
||||
updated = 0
|
||||
errors = 0
|
||||
for i, location in enumerate(queryset.iterator()):
|
||||
if location.reverse_geocode():
|
||||
updated += 1
|
||||
else:
|
||||
errors += 1
|
||||
if i < queryset.count() - 1:
|
||||
time.sleep(1.1)
|
||||
msg = f"Reverse geocoded {updated} locations"
|
||||
if errors:
|
||||
msg += f", {errors} failed"
|
||||
self.message_user(request, msg)
|
||||
|
||||
@ -0,0 +1,38 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-21 04:26
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("locations", "0010_clean_start"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="city",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="country",
|
||||
field=models.CharField(blank=True, max_length=100, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="postal_code",
|
||||
field=models.CharField(blank=True, max_length=20, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="state_province",
|
||||
field=models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="geolocation",
|
||||
name="street",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -45,6 +45,11 @@ class GeoLocation(ScrobblableMixin):
|
||||
truncated_lat = models.FloatField(**BNULL)
|
||||
truncated_lon = models.FloatField(**BNULL)
|
||||
altitude = models.FloatField(**BNULL)
|
||||
street = models.TextField(**BNULL)
|
||||
city = models.CharField(max_length=255, **BNULL)
|
||||
state_province = models.CharField(max_length=255, **BNULL)
|
||||
postal_code = models.CharField(max_length=20, **BNULL)
|
||||
country = models.CharField(max_length=100, **BNULL)
|
||||
|
||||
class Meta:
|
||||
unique_together = [["lat", "lon", "altitude"]]
|
||||
@ -55,6 +60,11 @@ class GeoLocation(ScrobblableMixin):
|
||||
|
||||
return f"{self.lat} x {self.lon}"
|
||||
|
||||
@property
|
||||
def display_address(self) -> str:
|
||||
parts = [self.street, self.city, self.state_province, self.postal_code, self.country]
|
||||
return ", ".join(p for p in parts if p)
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("locations:geolocation_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@ -121,6 +131,17 @@ class GeoLocation(ScrobblableMixin):
|
||||
|
||||
return fetch_current_weather(self.lat, self.lon)
|
||||
|
||||
def reverse_geocode(self) -> bool:
|
||||
from locations.utils import reverse_geocode
|
||||
|
||||
result = reverse_geocode(self.lat, self.lon)
|
||||
if result is None:
|
||||
return False
|
||||
for field, value in result.items():
|
||||
setattr(self, field, value)
|
||||
self.save(update_fields=list(result.keys()))
|
||||
return True
|
||||
|
||||
def loc_diff(self, old_lat_lon: tuple) -> tuple:
|
||||
return (
|
||||
abs(Decimal(old_lat_lon[0]) - Decimal(self.lat)),
|
||||
|
||||
@ -201,6 +201,50 @@ def detect_movement(
|
||||
return result
|
||||
|
||||
|
||||
NOMINATIM_URL = "https://nominatim.openstreetmap.org/reverse"
|
||||
|
||||
USER_AGENT = "Vrobbler/1.0 (https://github.com/secstate/vrobbler)"
|
||||
|
||||
|
||||
def reverse_geocode(lat: float, lon: float) -> Optional[dict]:
|
||||
"""Reverse geocode lat/lon to an address using Nominatim.
|
||||
|
||||
Returns a dict with address fields, or None on failure.
|
||||
Nominatim usage policy: max 1 request per second.
|
||||
"""
|
||||
params = {
|
||||
"lat": lat,
|
||||
"lon": lon,
|
||||
"format": "json",
|
||||
}
|
||||
headers = {"User-Agent": USER_AGENT}
|
||||
try:
|
||||
resp = requests.get(NOMINATIM_URL, params=params, headers=headers, timeout=10)
|
||||
resp.raise_for_status()
|
||||
except requests.RequestException as e:
|
||||
logger.warning("Failed to reverse geocode %s,%s: %s", lat, lon, e)
|
||||
return None
|
||||
|
||||
data = resp.json()
|
||||
if "error" in data:
|
||||
logger.warning("Nominatim error for %s,%s: %s", lat, lon, data["error"])
|
||||
return None
|
||||
|
||||
address = data.get("address", {})
|
||||
return {
|
||||
"street": address.get("road")
|
||||
or address.get("pedestrian")
|
||||
or address.get("footway"),
|
||||
"city": address.get("city")
|
||||
or address.get("town")
|
||||
or address.get("village")
|
||||
or address.get("hamlet"),
|
||||
"state_province": address.get("state"),
|
||||
"postal_code": address.get("postcode"),
|
||||
"country": address.get("country"),
|
||||
}
|
||||
|
||||
|
||||
NWS_URL = "https://forecast.weather.gov/MapClick.php"
|
||||
|
||||
|
||||
|
||||
@ -1,8 +1,22 @@
|
||||
from django.core.cache import cache
|
||||
|
||||
from music.models import Artist, Album
|
||||
|
||||
CACHE_TTL = 300
|
||||
|
||||
|
||||
def music_lists(request):
|
||||
artist_list = cache.get("music_lists_artist_list")
|
||||
if artist_list is None:
|
||||
artist_list = list(Artist.objects.all().only("id", "name"))
|
||||
cache.set("music_lists_artist_list", artist_list, CACHE_TTL)
|
||||
|
||||
album_list = cache.get("music_lists_album_list")
|
||||
if album_list is None:
|
||||
album_list = list(Album.objects.all().only("id", "name"))
|
||||
cache.set("music_lists_album_list", album_list, CACHE_TTL)
|
||||
|
||||
return {
|
||||
"artist_list": Artist.objects.all(),
|
||||
"album_list": Album.objects.all(),
|
||||
"artist_list": artist_list,
|
||||
"album_list": album_list,
|
||||
}
|
||||
|
||||
@ -236,19 +236,6 @@ class Artist(TimeStampedModel):
|
||||
)
|
||||
artist.fix_metadata()
|
||||
|
||||
# TODO: See if this alt_names stuff actually works or causes hard to debug problems
|
||||
# If we did find our artist, but the found name is slightly differnt, record that
|
||||
# if artist and alt_name:
|
||||
# if not artist.alt_names:
|
||||
# artist.alt_names = alt_name
|
||||
# else:
|
||||
# artist.alt_names += f"\\{alt_name}"
|
||||
# logger.info(
|
||||
# f"Add alt_name {alt_name} to artist {artist}",
|
||||
# extra={"alt_name": alt_name, "artist_id": artist.id},
|
||||
# )
|
||||
# artist.save(update_fields=["alt_names"])
|
||||
|
||||
return artist
|
||||
|
||||
|
||||
|
||||
@ -40,6 +40,7 @@ class UserProfileForm(forms.ModelForm):
|
||||
"enable_public_widgets",
|
||||
"widget_custom_css",
|
||||
"home_scrobble_limit",
|
||||
"live_now_playing",
|
||||
"weigh_in_units",
|
||||
]
|
||||
widgets = {
|
||||
|
||||
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-22 02:44
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("profiles", "0038_userprofile_media_type_visibility"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="userprofile",
|
||||
name="live_now_playing",
|
||||
field=models.BooleanField(default=False),
|
||||
),
|
||||
]
|
||||
@ -98,6 +98,8 @@ class UserProfile(TimeStampedModel):
|
||||
|
||||
home_scrobble_limit = models.IntegerField(default=20)
|
||||
|
||||
live_now_playing = models.BooleanField(default=False)
|
||||
|
||||
weigh_in_units = models.CharField(
|
||||
max_length=16,
|
||||
choices=WeighUnit.choices,
|
||||
|
||||
@ -12,6 +12,7 @@ from scrobbles.models import (
|
||||
Scrobble,
|
||||
ShareViewLog,
|
||||
TrailGPXImport,
|
||||
UDiscCSVImport,
|
||||
)
|
||||
from scrobbles.mixins import Genre
|
||||
|
||||
@ -73,6 +74,10 @@ class ScaleCSVImportAdmin(ImportBaseAdmin): ...
|
||||
class TrailGPXImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(UDiscCSVImport)
|
||||
class UDiscCSVImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(Genre)
|
||||
class GenreAdmin(admin.ModelAdmin):
|
||||
list_display = (
|
||||
@ -118,6 +123,7 @@ class ScrobbleAdmin(admin.ModelAdmin):
|
||||
"web_page",
|
||||
"life_event",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
"long_play_last_scrobble",
|
||||
)
|
||||
list_filter = (
|
||||
@ -179,4 +185,5 @@ class FavoriteMediaAdmin(admin.ModelAdmin):
|
||||
"web_page",
|
||||
"life_event",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
)
|
||||
|
||||
@ -14,6 +14,7 @@ LONG_PLAY_MEDIA = {
|
||||
"books": "Book",
|
||||
"bricksets": "BrickSet",
|
||||
"tasks": "Task",
|
||||
"papers": "Paper",
|
||||
}
|
||||
|
||||
# Media types that should just be finished if they go over time
|
||||
@ -25,17 +26,26 @@ AUTO_FINISH_MEDIA = {
|
||||
}
|
||||
|
||||
PLAY_AGAIN_MEDIA = {
|
||||
"videogames": "VideoGame",
|
||||
"videos": "Video",
|
||||
"music": "Track",
|
||||
"podcasts": "PodcastEpisode",
|
||||
"sports": "SportEvent",
|
||||
"books": "Book",
|
||||
"videogames": "VideoGame",
|
||||
"boardgames": "BoardGame",
|
||||
"moods": "Mood",
|
||||
"bricksets": "BrickSet",
|
||||
"locations": "GeoLocation",
|
||||
"trails": "Trail",
|
||||
"beers": "Beer",
|
||||
"puzzles": "Puzzle",
|
||||
"foods": "Food",
|
||||
"locations": "GeoLocation",
|
||||
"videos": "Video",
|
||||
"tasks": "Task",
|
||||
"webpages": "WebPage",
|
||||
"lifeevents": "LifeEvent",
|
||||
"moods": "Mood",
|
||||
"bricksets": "BrickSet",
|
||||
"channels": "Channel",
|
||||
"birds": "BirdingLocation",
|
||||
"discgolf": "DiscGolfCourse",
|
||||
}
|
||||
|
||||
MEDIA_END_PADDING_SECONDS = {
|
||||
@ -52,6 +62,7 @@ SCROBBLE_CONTENT_URLS = {
|
||||
"-b": ["https://www.amazon.com/"],
|
||||
"-t": ["https://app.todoist.com/app/task/{id}"],
|
||||
"-p": ["https://www.ipdb.plus/IPDb/puzzle.php?id="],
|
||||
"-pp": ["https://doi.org/"],
|
||||
"-l": ["https://brickset.com/sets/"],
|
||||
"-c": ["https://readcomicsonline.ru"],
|
||||
"-h": ["https://www.twitch.tv/"],
|
||||
@ -73,6 +84,8 @@ MANUAL_SCROBBLE_FNS = {
|
||||
"-c": "manual_scrobble_book",
|
||||
"-f": "manual_scrobble_food",
|
||||
"-h": "manual_scrobble_twitch_channel",
|
||||
"-dg": "manual_scrobble_discgolf",
|
||||
"-pp": "manual_scrobble_paper",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@ -27,15 +27,19 @@ def month_color(request):
|
||||
|
||||
def now_playing(request):
|
||||
user = request.user
|
||||
now = timezone.now()
|
||||
if not user.is_authenticated:
|
||||
return {}
|
||||
|
||||
return {
|
||||
"now_playing_list": Scrobble.objects.filter(
|
||||
in_progress=True,
|
||||
is_paused=False,
|
||||
user=user,
|
||||
).exclude(
|
||||
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
|
||||
)
|
||||
"now_playing_list": list(
|
||||
Scrobble.objects.filter(
|
||||
in_progress=True,
|
||||
is_paused=False,
|
||||
user=user,
|
||||
)
|
||||
.exclude(
|
||||
media_type__in=EXCLUDE_FROM_NOW_PLAYING,
|
||||
)
|
||||
.select_related("track", "video", "podcast_episode")
|
||||
),
|
||||
}
|
||||
|
||||
@ -19,6 +19,7 @@ DEFAULT_RETROARCH_PATH = "var/retroarch/"
|
||||
DEFAULT_BGSTATS_PATH = "var/bgstats/"
|
||||
DEFAULT_EBIRD_PATH = "var/ebird/"
|
||||
DEFAULT_SCALE_PATH = "var/scale/"
|
||||
DEFAULT_UDISC_PATH = "var/udisc/"
|
||||
|
||||
|
||||
def import_from_webdav_for_all_users(
|
||||
@ -48,6 +49,7 @@ def import_from_webdav_for_all_users(
|
||||
bgstats_count = 0
|
||||
ebird_count = 0
|
||||
scale_count = 0
|
||||
udisc_count = 0
|
||||
|
||||
for user_id in webdav_enabled_user_ids:
|
||||
client = get_webdav_client(user_id)
|
||||
@ -78,15 +80,20 @@ def import_from_webdav_for_all_users(
|
||||
)
|
||||
logger.info("Scanning WebDAV scale for user %s", user_id)
|
||||
scale_count += scan_webdav_for_scale(client, user_id)
|
||||
logger.info("Scanning WebDAV udisc for user %s", user_id)
|
||||
udisc_count += scan_webdav_for_udisc(
|
||||
client, user_id, include_processed=include_processed
|
||||
)
|
||||
|
||||
logger.info(
|
||||
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale WebDAV imports",
|
||||
"Started %d KOReader, %d Trail GPX, %d Retroarch, %d BGStats, %d eBird, %d Scale, %d uDisc WebDAV imports",
|
||||
ko_count,
|
||||
gpx_count,
|
||||
retro_count,
|
||||
bgstats_count,
|
||||
ebird_count,
|
||||
scale_count,
|
||||
udisc_count,
|
||||
extra={
|
||||
"koreader": ko_count,
|
||||
"trail_gpx": gpx_count,
|
||||
@ -94,9 +101,10 @@ def import_from_webdav_for_all_users(
|
||||
"bgstats": bgstats_count,
|
||||
"ebird": ebird_count,
|
||||
"scale": scale_count,
|
||||
"udisc": udisc_count,
|
||||
},
|
||||
)
|
||||
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count
|
||||
return ko_count, gpx_count, retro_count, bgstats_count, ebird_count, scale_count, udisc_count
|
||||
|
||||
|
||||
def scan_webdav_for_koreader(
|
||||
@ -811,3 +819,115 @@ def scan_webdav_for_scale(webdav_client, user_id):
|
||||
os.unlink(tmp.name)
|
||||
|
||||
return new_imports
|
||||
|
||||
|
||||
def scan_webdav_for_udisc(webdav_client, user_id, include_processed=False):
|
||||
"""Download .csv files from WebDAV var/udisc/ and queue imports for new files.
|
||||
|
||||
After importing, files are moved to var/udisc/processed/ so they are
|
||||
not re-imported on subsequent scans unless *include_processed* is True.
|
||||
"""
|
||||
from scrobbles.models import UDiscCSVImport
|
||||
from scrobbles.tasks import process_udisc_csv_import
|
||||
|
||||
udisc_path = DEFAULT_UDISC_PATH
|
||||
try:
|
||||
webdav_client.info(udisc_path)
|
||||
except:
|
||||
logger.info("No var/udisc/ directory on webdav", extra={"user_id": user_id})
|
||||
return 0
|
||||
|
||||
try:
|
||||
files = webdav_client.list(udisc_path)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Could not list var/udisc/",
|
||||
extra={"user_id": user_id, "error": str(e)},
|
||||
)
|
||||
return 0
|
||||
|
||||
processed_dir = f"{udisc_path}processed/"
|
||||
try:
|
||||
webdav_client.mkdir(processed_dir, recursive=True)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
new_imports = 0
|
||||
already_imported = set(
|
||||
UDiscCSVImport.objects.filter(user_id=user_id).values_list(
|
||||
"original_filename", flat=True
|
||||
)
|
||||
)
|
||||
|
||||
for fname in files:
|
||||
fname = os.path.basename(fname)
|
||||
if not fname.lower().endswith(".csv"):
|
||||
continue
|
||||
if fname == "processed":
|
||||
continue
|
||||
if fname in already_imported:
|
||||
logger.debug(f"Skipping already-imported {fname}")
|
||||
continue
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
|
||||
try:
|
||||
webdav_client.download_sync(
|
||||
remote_path=f"{udisc_path}{fname}", local_path=tmp.name
|
||||
)
|
||||
imp = UDiscCSVImport.objects.create(
|
||||
user_id=user_id,
|
||||
original_filename=fname,
|
||||
)
|
||||
with open(tmp.name, "rb") as f:
|
||||
imp.csv_file.save(fname, f, save=True)
|
||||
|
||||
stem, ext = os.path.splitext(fname)
|
||||
ts = datetime.now().strftime("%Y%m%dT%H%M%S")
|
||||
webdav_client.move(
|
||||
f"{udisc_path}{fname}",
|
||||
f"{processed_dir}{stem}_{ts}{ext}",
|
||||
)
|
||||
|
||||
process_udisc_csv_import.delay(imp.id)
|
||||
new_imports += 1
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to import uDisc CSV file {fname}: {e}")
|
||||
finally:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
if include_processed:
|
||||
try:
|
||||
processed_files = webdav_client.list(processed_dir)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
"Could not list var/udisc/processed/",
|
||||
extra={"user_id": user_id, "error": str(e)},
|
||||
)
|
||||
return new_imports
|
||||
|
||||
for fname in processed_files:
|
||||
fname = os.path.basename(fname)
|
||||
if not fname.lower().endswith(".csv"):
|
||||
continue
|
||||
|
||||
tmp = tempfile.NamedTemporaryFile(delete=False, suffix=fname)
|
||||
try:
|
||||
webdav_client.download_sync(
|
||||
remote_path=f"{processed_dir}{fname}", local_path=tmp.name
|
||||
)
|
||||
imp = UDiscCSVImport.objects.create(
|
||||
user_id=user_id,
|
||||
original_filename=fname,
|
||||
)
|
||||
with open(tmp.name, "rb") as f:
|
||||
imp.csv_file.save(fname, f, save=True)
|
||||
process_udisc_csv_import.delay(imp.id)
|
||||
new_imports += 1
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to import processed uDisc CSV file {fname}: {e}"
|
||||
)
|
||||
finally:
|
||||
os.unlink(tmp.name)
|
||||
|
||||
return new_imports
|
||||
|
||||
518
vrobbler/apps/scrobbles/mcp.py
Normal file
518
vrobbler/apps/scrobbles/mcp.py
Normal file
@ -0,0 +1,518 @@
|
||||
from mcp_server import MCPToolset
|
||||
from scrobbles.models import Scrobble
|
||||
from scrobbles.constants import LONG_PLAY_MEDIA
|
||||
|
||||
|
||||
class ScrobbleToolset(MCPToolset):
|
||||
def list_recent_scrobbles(
|
||||
self,
|
||||
days: int = 7,
|
||||
media_type: str | None = None,
|
||||
limit: int = 50,
|
||||
) -> list[dict]:
|
||||
"""List scrobbles from the last N days, optionally filtered by media type.
|
||||
Valid media_type values: Video, Track, PodcastEpisode, SportEvent, Book,
|
||||
Paper, VideoGame, BoardGame, GeoLocation, Trail, Beer, Puzzle, Food, Task,
|
||||
WebPage, LifeEvent, Mood, BrickSet, Channel, BirdingLocation, DiscGolfCourse
|
||||
"""
|
||||
qs = (
|
||||
Scrobble.objects.filter(user=self.request.user)
|
||||
.select_related(
|
||||
"video", "track", "book", "video_game", "board_game",
|
||||
"beer", "puzzle", "food", "trail", "task", "web_page",
|
||||
"life_event", "mood", "brick_set", "podcast_episode",
|
||||
"sport_event", "geo_location", "birding_location",
|
||||
"disc_golf_course", "channel",
|
||||
)
|
||||
.order_by("-timestamp")
|
||||
)
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
qs = qs.filter(timestamp__gte=timezone.now() - datetime.timedelta(days=days))
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
qs = qs[:limit]
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_scrobble(self, uuid: str) -> dict | None:
|
||||
"""Get a single scrobble by its UUID."""
|
||||
try:
|
||||
s = Scrobble.objects.filter(user=self.request.user).get(uuid=uuid)
|
||||
except Scrobble.DoesNotExist:
|
||||
return None
|
||||
return _scrobble_to_dict(s)
|
||||
|
||||
def search_scrobbles(
|
||||
self, query: str, media_type: str | None = None, limit: int = 20
|
||||
) -> list[dict]:
|
||||
"""Search scrobbles by text in their log data or related media titles."""
|
||||
from django.db.models import Q
|
||||
qs = Scrobble.objects.filter(user=self.request.user).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
qs = qs.filter(
|
||||
Q(log__icontains=query)
|
||||
| Q(video__title__icontains=query)
|
||||
| Q(track__title__icontains=query)
|
||||
| Q(book__title__icontains=query)
|
||||
| Q(video_game__title__icontains=query)
|
||||
| Q(board_game__title__icontains=query)
|
||||
| Q(beer__title__icontains=query)
|
||||
| Q(food__title__icontains=query)
|
||||
| Q(trail__title__icontains=query)
|
||||
| Q(task__title__icontains=query)
|
||||
| Q(web_page__title__icontains=query)
|
||||
| Q(life_event__title__icontains=query)
|
||||
| Q(puzzle__title__icontains=query)
|
||||
| Q(brick_set__title__icontains=query)
|
||||
| Q(podcast_episode__title__icontains=query)
|
||||
)[:limit]
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_scrobbles_by_date(
|
||||
self, date: str, media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get scrobbles for a specific date (YYYY-MM-DD format)."""
|
||||
import datetime
|
||||
try:
|
||||
dt = datetime.datetime.strptime(date, "%Y-%m-%d").date()
|
||||
except ValueError:
|
||||
return []
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
timestamp__date=dt,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_in_progress_scrobbles(
|
||||
self, media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get scrobbles currently in progress (started but not finished).
|
||||
These are long-play items like books, video games, brick sets, or tasks."""
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
in_progress=True,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
def get_long_play_scrobbles(
|
||||
self, status: str = "in_progress", media_type: str | None = None
|
||||
) -> list[dict]:
|
||||
"""Get long-play scrobbles (books, video games, brick sets, tasks).
|
||||
Status can be 'in_progress' or 'completed'."""
|
||||
types = list(LONG_PLAY_MEDIA.values())
|
||||
qs = Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
media_type__in=types,
|
||||
).order_by("-timestamp")
|
||||
if media_type:
|
||||
qs = qs.filter(media_type=media_type)
|
||||
if status == "in_progress":
|
||||
qs = qs.filter(in_progress=True)
|
||||
elif status == "completed":
|
||||
qs = qs.filter(in_progress=False)
|
||||
return [_scrobble_to_dict(s) for s in qs]
|
||||
|
||||
|
||||
class MediaToolset(MCPToolset):
|
||||
def get_book(self, uuid: str) -> dict | None:
|
||||
"""Get a book by UUID."""
|
||||
from books.models import Book
|
||||
try:
|
||||
b = Book.objects.get(uuid=uuid)
|
||||
except Book.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(b, fields=["title", "pages", "language",
|
||||
"first_publish_year", "isbn_13",
|
||||
"publisher", "summary"])
|
||||
|
||||
def list_books(self, author: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List books, optionally filtered by author name."""
|
||||
from books.models import Book
|
||||
qs = Book.objects.all().order_by("title")
|
||||
if author:
|
||||
qs = qs.filter(authors__name__icontains=author)
|
||||
return [_media_to_dict(b, fields=["title", "pages", "language",
|
||||
"first_publish_year", "isbn_13",
|
||||
"publisher"]) for b in qs[:limit]]
|
||||
|
||||
def get_track(self, uuid: str) -> dict | None:
|
||||
"""Get a music track by UUID."""
|
||||
from music.models import Track
|
||||
try:
|
||||
t = Track.objects.select_related("artist_fk").get(uuid=uuid)
|
||||
except Track.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "base_run_time_seconds",
|
||||
"artist_fk__name", "genre"])
|
||||
|
||||
def list_tracks(self, artist: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List music tracks, optionally filtered by artist name."""
|
||||
from music.models import Track
|
||||
qs = Track.objects.select_related("artist_fk").all().order_by("title")
|
||||
if artist:
|
||||
qs = qs.filter(artist_fk__name__icontains=artist)
|
||||
return [_media_to_dict(t, fields=["title", "base_run_time_seconds",
|
||||
"artist_fk__name", "genre"])
|
||||
for t in qs[:limit]]
|
||||
|
||||
def get_video(self, uuid: str) -> dict | None:
|
||||
"""Get a video by UUID."""
|
||||
from videos.models import Video
|
||||
try:
|
||||
v = Video.objects.select_related("tv_series", "channel").get(uuid=uuid)
|
||||
except Video.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(v, fields=["title", "year", "overview",
|
||||
"imdb_id", "imdb_rating",
|
||||
"tv_series__name", "channel__title",
|
||||
"season_number", "episode_number"])
|
||||
|
||||
def list_videos(self, series: str | None = None, limit: int = 20) -> list[dict]:
|
||||
"""List videos, optionally filtered by series name."""
|
||||
from videos.models import Video
|
||||
qs = Video.objects.select_related("tv_series", "channel").all().order_by("title")
|
||||
if series:
|
||||
qs = qs.filter(tv_series__name__icontains=series)
|
||||
return [_media_to_dict(v, fields=["title", "year", "overview",
|
||||
"tv_series__name", "channel__title",
|
||||
"season_number", "episode_number"])
|
||||
for v in qs[:limit]]
|
||||
|
||||
def get_board_game(self, uuid: str) -> dict | None:
|
||||
"""Get a board game by UUID."""
|
||||
from boardgames.models import BoardGame
|
||||
try:
|
||||
bg = BoardGame.objects.get(uuid=uuid)
|
||||
except BoardGame.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bg, fields=["title", "genre"])
|
||||
|
||||
def list_board_games(self, limit: int = 20) -> list[dict]:
|
||||
"""List board games."""
|
||||
from boardgames.models import BoardGame
|
||||
qs = BoardGame.objects.all().order_by("title")[:limit]
|
||||
return [_media_to_dict(bg, fields=["title", "genre"]) for bg in qs]
|
||||
|
||||
def get_podcast_episode(self, uuid: str) -> dict | None:
|
||||
"""Get a podcast episode by UUID."""
|
||||
from podcasts.models import PodcastEpisode
|
||||
try:
|
||||
pe = PodcastEpisode.objects.select_related("podcast", "producer").get(
|
||||
uuid=uuid
|
||||
)
|
||||
except PodcastEpisode.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(pe, fields=["title", "podcast__title",
|
||||
"producer__name", "base_run_time_seconds"])
|
||||
|
||||
def get_beer(self, uuid: str) -> dict | None:
|
||||
"""Get a beer by UUID."""
|
||||
from beers.models import Beer
|
||||
try:
|
||||
b = Beer.objects.select_related("style", "producer").get(uuid=uuid)
|
||||
except Beer.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(b, fields=["title", "style__name",
|
||||
"producer__name", "abv"])
|
||||
|
||||
def get_brick_set(self, uuid: str) -> dict | None:
|
||||
"""Get a brick set (LEGO) by UUID."""
|
||||
from bricksets.models import BrickSet
|
||||
try:
|
||||
bs = BrickSet.objects.get(uuid=uuid)
|
||||
except BrickSet.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bs, fields=["title", "piece_count", "set_number"])
|
||||
|
||||
def get_video_game(self, uuid: str) -> dict | None:
|
||||
"""Get a video game by UUID."""
|
||||
from videogames.models import VideoGame
|
||||
try:
|
||||
vg = VideoGame.objects.get(uuid=uuid)
|
||||
except VideoGame.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(vg, fields=["title", "genre",
|
||||
"base_run_time_seconds"])
|
||||
|
||||
def get_puzzle(self, uuid: str) -> dict | None:
|
||||
"""Get a puzzle by UUID."""
|
||||
from puzzles.models import Puzzle
|
||||
try:
|
||||
p = Puzzle.objects.select_related("manufacturer").get(uuid=uuid)
|
||||
except Puzzle.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(p, fields=["title", "piece_count",
|
||||
"manufacturer__name"])
|
||||
|
||||
def get_web_page(self, uuid: str) -> dict | None:
|
||||
"""Get a web page by UUID."""
|
||||
from webpages.models import WebPage
|
||||
try:
|
||||
wp = WebPage.objects.select_related("domain").get(uuid=uuid)
|
||||
except WebPage.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(wp, fields=["title", "url", "domain__name"])
|
||||
|
||||
def get_task(self, uuid: str) -> dict | None:
|
||||
"""Get a task by UUID."""
|
||||
from tasks.models import Task
|
||||
try:
|
||||
t = Task.objects.get(uuid=uuid)
|
||||
except Task.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "completed"])
|
||||
|
||||
def get_trail(self, uuid: str) -> dict | None:
|
||||
"""Get a trail by UUID."""
|
||||
from trails.models import Trail
|
||||
try:
|
||||
t = Trail.objects.get(uuid=uuid)
|
||||
except Trail.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(t, fields=["title", "genre", "base_run_time_seconds"])
|
||||
|
||||
def get_geo_location(self, uuid: str) -> dict | None:
|
||||
"""Get a geo location by UUID."""
|
||||
from locations.models import GeoLocation
|
||||
try:
|
||||
gl = GeoLocation.objects.get(uuid=uuid)
|
||||
except GeoLocation.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(gl, fields=["title", "latitude", "longitude"])
|
||||
|
||||
def get_life_event(self, uuid: str) -> dict | None:
|
||||
"""Get a life event by UUID."""
|
||||
from lifeevents.models import LifeEvent
|
||||
try:
|
||||
le = LifeEvent.objects.get(uuid=uuid)
|
||||
except LifeEvent.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(le, fields=["title", "event_date", "genre"])
|
||||
|
||||
def get_mood(self, uuid: str) -> dict | None:
|
||||
"""Get a mood entry by UUID."""
|
||||
from moods.models import Mood
|
||||
try:
|
||||
m = Mood.objects.get(uuid=uuid)
|
||||
except Mood.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(m, fields=["title", "mood_type", "mood_quality"])
|
||||
|
||||
def get_food(self, uuid: str) -> dict | None:
|
||||
"""Get a food entry by UUID."""
|
||||
from foods.models import Food
|
||||
try:
|
||||
f = Food.objects.select_related("category").get(uuid=uuid)
|
||||
except Food.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(f, fields=["title", "category__name"])
|
||||
|
||||
def get_bird_sighting(self, uuid: str) -> dict | None:
|
||||
"""Get a bird sighting by UUID."""
|
||||
from birds.models import BirdSighting
|
||||
try:
|
||||
bs = BirdSighting.objects.select_related("bird").get(uuid=uuid)
|
||||
except BirdSighting.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(bs, fields=["title", "bird__common_name",
|
||||
"bird__scientific_name", "location"])
|
||||
|
||||
def get_disc_golf_course(self, uuid: str) -> dict | None:
|
||||
"""Get a disc golf course by UUID."""
|
||||
from discgolf.models import DiscGolfCourse
|
||||
try:
|
||||
dg = DiscGolfCourse.objects.get(uuid=uuid)
|
||||
except DiscGolfCourse.DoesNotExist:
|
||||
return None
|
||||
return _media_to_dict(dg, fields=["title", "holes", "location"])
|
||||
|
||||
|
||||
class StatsToolset(MCPToolset):
|
||||
def get_scrobble_counts(self, days: int = 30) -> list[dict]:
|
||||
"""Get scrobble counts grouped by media type for the last N days."""
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
from django.db.models import Count
|
||||
|
||||
cutoff = timezone.now() - datetime.timedelta(days=days)
|
||||
qs = (
|
||||
Scrobble.objects.filter(user=self.request.user, timestamp__gte=cutoff)
|
||||
.values("media_type")
|
||||
.annotate(count=Count("id"))
|
||||
.order_by("-count")
|
||||
)
|
||||
return list(qs)
|
||||
|
||||
def get_top_media(
|
||||
self, media_type: str, days: int = 30, limit: int = 10
|
||||
) -> list[dict]:
|
||||
"""Get the most-scrobbled items of a given media type in the last N days.
|
||||
Valid media_type values: Video, Track, Book, BoardGame, Beer, etc."""
|
||||
from django.utils import timezone
|
||||
import datetime
|
||||
from django.db.models import Count
|
||||
|
||||
cutoff = timezone.now() - datetime.timedelta(days=days)
|
||||
|
||||
rel_field = _media_type_to_rel_field(media_type)
|
||||
if not rel_field:
|
||||
return []
|
||||
|
||||
qs = (
|
||||
Scrobble.objects.filter(
|
||||
user=self.request.user,
|
||||
media_type=media_type,
|
||||
timestamp__gte=cutoff,
|
||||
)
|
||||
.values(rel_field)
|
||||
.annotate(count=Count("id"))
|
||||
.order_by("-count")
|
||||
)[:limit]
|
||||
|
||||
results = []
|
||||
for row in qs:
|
||||
obj_id = row[rel_field]
|
||||
if obj_id is None:
|
||||
continue
|
||||
results.append({"id": obj_id, "count": row["count"]})
|
||||
return results
|
||||
|
||||
|
||||
def _media_type_to_rel_field(media_type: str) -> str | None:
|
||||
mapping = {
|
||||
"Video": "video",
|
||||
"Track": "track",
|
||||
"PodcastEpisode": "podcast_episode",
|
||||
"SportEvent": "sport_event",
|
||||
"Book": "book",
|
||||
"Paper": "paper",
|
||||
"VideoGame": "video_game",
|
||||
"BoardGame": "board_game",
|
||||
"GeoLocation": "geo_location",
|
||||
"Trail": "trail",
|
||||
"Beer": "beer",
|
||||
"Puzzle": "puzzle",
|
||||
"Food": "food",
|
||||
"Task": "task",
|
||||
"WebPage": "web_page",
|
||||
"LifeEvent": "life_event",
|
||||
"Mood": "mood",
|
||||
"BrickSet": "brick_set",
|
||||
"Channel": "channel",
|
||||
"BirdingLocation": "birding_location",
|
||||
"DiscGolfCourse": "disc_golf_course",
|
||||
}
|
||||
return mapping.get(media_type)
|
||||
|
||||
|
||||
def _scrobble_to_dict(s: Scrobble) -> dict:
|
||||
result = {
|
||||
"uuid": str(s.uuid),
|
||||
"media_type": s.media_type,
|
||||
"timestamp": s.timestamp.isoformat() if s.timestamp else None,
|
||||
"stop_timestamp": s.stop_timestamp.isoformat() if s.stop_timestamp else None,
|
||||
"in_progress": s.in_progress,
|
||||
"played_to_completion": s.played_to_completion,
|
||||
"source": s.source,
|
||||
"visibility": s.visibility,
|
||||
"timezone": s.timezone,
|
||||
}
|
||||
if s.log:
|
||||
result["log"] = s.log
|
||||
|
||||
rel = _scrobble_related_to_dict(s)
|
||||
if rel:
|
||||
result["media"] = rel
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _scrobble_related_to_dict(s: Scrobble) -> dict | None:
|
||||
if s.video:
|
||||
return _media_to_dict(s.video, fields=["title", "year", "imdb_id",
|
||||
"imdb_rating"])
|
||||
if s.track:
|
||||
return _media_to_dict(s.track, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.book:
|
||||
return _media_to_dict(s.book, fields=["title", "pages"])
|
||||
if s.video_game:
|
||||
return _media_to_dict(s.video_game, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.board_game:
|
||||
return _media_to_dict(s.board_game, fields=["title"])
|
||||
if s.beer:
|
||||
return _media_to_dict(s.beer, fields=["title"])
|
||||
if s.puzzle:
|
||||
return _media_to_dict(s.puzzle, fields=["title", "piece_count"])
|
||||
if s.food:
|
||||
return _media_to_dict(s.food, fields=["title"])
|
||||
if s.trail:
|
||||
return _media_to_dict(s.trail, fields=["title",
|
||||
"base_run_time_seconds"])
|
||||
if s.task:
|
||||
return _media_to_dict(s.task, fields=["title"])
|
||||
if s.web_page:
|
||||
return _media_to_dict(s.web_page, fields=["title", "url"])
|
||||
if s.life_event:
|
||||
return _media_to_dict(s.life_event, fields=["title", "event_date"])
|
||||
if s.mood:
|
||||
return _media_to_dict(s.mood, fields=["title"])
|
||||
if s.brick_set:
|
||||
return _media_to_dict(s.brick_set, fields=["title", "set_number"])
|
||||
if s.podcast_episode:
|
||||
return _media_to_dict(s.podcast_episode, fields=["title"])
|
||||
if s.sport_event:
|
||||
return {"title": str(s.sport_event)}
|
||||
if s.geo_location:
|
||||
return _media_to_dict(s.geo_location, fields=["title", "latitude",
|
||||
"longitude"])
|
||||
if s.birding_location:
|
||||
return _media_to_dict(s.birding_location, fields=["title"])
|
||||
if s.disc_golf_course:
|
||||
return _media_to_dict(s.disc_golf_course, fields=["title", "holes"])
|
||||
if s.channel:
|
||||
return _media_to_dict(s.channel, fields=["title"])
|
||||
return None
|
||||
|
||||
|
||||
def _media_to_dict(obj, fields: list[str] | None = None) -> dict:
|
||||
if obj is None:
|
||||
return {}
|
||||
result = {}
|
||||
if hasattr(obj, "uuid"):
|
||||
result["uuid"] = str(obj.uuid)
|
||||
if hasattr(obj, "title"):
|
||||
result["title"] = obj.title
|
||||
|
||||
if fields is None:
|
||||
return result
|
||||
|
||||
resolved = _resolve_fields(obj, fields)
|
||||
for k, v in resolved.items():
|
||||
if k not in result:
|
||||
result[k] = v
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def _resolve_fields(obj, fields: list[str]) -> dict:
|
||||
result = {}
|
||||
for field in fields:
|
||||
parts = field.split("__")
|
||||
val = obj
|
||||
try:
|
||||
for part in parts:
|
||||
val = getattr(val, part)
|
||||
except AttributeError:
|
||||
continue
|
||||
if val is not None:
|
||||
if hasattr(val, "all"):
|
||||
val = [str(v) for v in val.all()]
|
||||
result[field] = val
|
||||
return result
|
||||
@ -0,0 +1,156 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:19
|
||||
|
||||
from django.conf import settings
|
||||
from django.db import migrations, models
|
||||
import django.db.models.deletion
|
||||
import django_extensions.db.fields
|
||||
import scrobbles.models
|
||||
import uuid
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("discgolf", "0001_initial"),
|
||||
migrations.swappable_dependency(settings.AUTH_USER_MODEL),
|
||||
("scrobbles", "0098_scrobble_long_play_last_scrobble"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="favoritemedia",
|
||||
name="disc_golf",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.CASCADE,
|
||||
to="discgolf.discgolfcourse",
|
||||
),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="scrobble",
|
||||
name="disc_golf",
|
||||
field=models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
to="discgolf.discgolfcourse",
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="favoritemedia",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("DiscGolf", "Disc golf"),
|
||||
],
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="scrobble",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("DiscGolf", "Disc golf"),
|
||||
],
|
||||
default="Video",
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.CreateModel(
|
||||
name="UDiscCSVImport",
|
||||
fields=[
|
||||
(
|
||||
"id",
|
||||
models.BigAutoField(
|
||||
auto_created=True,
|
||||
primary_key=True,
|
||||
serialize=False,
|
||||
verbose_name="ID",
|
||||
),
|
||||
),
|
||||
(
|
||||
"created",
|
||||
django_extensions.db.fields.CreationDateTimeField(
|
||||
auto_now_add=True, verbose_name="created"
|
||||
),
|
||||
),
|
||||
(
|
||||
"modified",
|
||||
django_extensions.db.fields.ModificationDateTimeField(
|
||||
auto_now=True, verbose_name="modified"
|
||||
),
|
||||
),
|
||||
("uuid", models.UUIDField(default=uuid.uuid4, editable=False)),
|
||||
("processing_started", models.DateTimeField(blank=True, null=True)),
|
||||
("processed_finished", models.DateTimeField(blank=True, null=True)),
|
||||
("process_log", models.TextField(blank=True, null=True)),
|
||||
("process_count", models.IntegerField(blank=True, null=True)),
|
||||
("error_log", models.TextField(blank=True, null=True)),
|
||||
(
|
||||
"csv_file",
|
||||
models.FileField(
|
||||
blank=True,
|
||||
null=True,
|
||||
upload_to=scrobbles.models.UDiscCSVImport.get_path,
|
||||
),
|
||||
),
|
||||
(
|
||||
"original_filename",
|
||||
models.CharField(blank=True, max_length=255, null=True),
|
||||
),
|
||||
(
|
||||
"user",
|
||||
models.ForeignKey(
|
||||
blank=True,
|
||||
null=True,
|
||||
on_delete=django.db.models.deletion.DO_NOTHING,
|
||||
related_name="scrobbles_udisccsvimport_set",
|
||||
to=settings.AUTH_USER_MODEL,
|
||||
),
|
||||
),
|
||||
],
|
||||
options={
|
||||
"verbose_name": "uDisc CSV Import",
|
||||
},
|
||||
),
|
||||
]
|
||||
@ -0,0 +1,84 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-20 04:53
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0099_favoritemedia_disc_golf_scrobble_disc_golf_and_more"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RenameField(
|
||||
model_name="favoritemedia",
|
||||
old_name="disc_golf",
|
||||
new_name="disc_golf_course",
|
||||
),
|
||||
migrations.RenameField(
|
||||
model_name="scrobble",
|
||||
old_name="disc_golf",
|
||||
new_name="disc_golf_course",
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="favoritemedia",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("BirdingLocation", "Birding location"),
|
||||
("DiscGolfCourse", "Disc golf"),
|
||||
],
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
migrations.AlterField(
|
||||
model_name="scrobble",
|
||||
name="media_type",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("Video", "Video"),
|
||||
("Track", "Track"),
|
||||
("PodcastEpisode", "Podcast episode"),
|
||||
("SportEvent", "Sport event"),
|
||||
("Book", "Book"),
|
||||
("Paper", "Paper"),
|
||||
("VideoGame", "Video game"),
|
||||
("BoardGame", "Board game"),
|
||||
("GeoLocation", "GeoLocation"),
|
||||
("Trail", "Trail"),
|
||||
("Beer", "Beer"),
|
||||
("Puzzle", "Puzzle"),
|
||||
("Food", "Food"),
|
||||
("Task", "Task"),
|
||||
("WebPage", "Web Page"),
|
||||
("LifeEvent", "Life event"),
|
||||
("Mood", "Mood"),
|
||||
("BrickSet", "Brick set"),
|
||||
("Channel", "Channel"),
|
||||
("BirdingLocation", "Birding location"),
|
||||
("DiscGolfCourse", "Disc golf"),
|
||||
],
|
||||
default="Video",
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -114,7 +114,7 @@ class ScrobblableMixin(TimeStampedModel):
|
||||
|
||||
@property
|
||||
def start_url(self):
|
||||
return reverse("scrobbles:start", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:start", kwargs={"media_uuid": self.uuid})
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
@ -162,7 +162,7 @@ class LongPlayScrobblableMixin(ScrobblableMixin):
|
||||
return False
|
||||
|
||||
def get_longplay_finish_url(self):
|
||||
return reverse("scrobbles:longplay-finish", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:longplay-finish", kwargs={"media_uuid": self.uuid})
|
||||
|
||||
def first_long_play_scrobble_for_user(self, user) -> Optional["Scrobble"]:
|
||||
last = self.last_long_play_scrobble_for_user(user)
|
||||
|
||||
@ -11,6 +11,7 @@ import pendulum
|
||||
import pytz
|
||||
from beers.models import Beer
|
||||
from birds.models import BirdingLocation
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from boardgames.models import BoardGame
|
||||
from books.koreader import process_koreader_sqlite_file
|
||||
from books.models import Book, BookLogData, BookPageLogData, Paper
|
||||
@ -617,6 +618,60 @@ class EBirdCSVImport(BaseFileImportMixin):
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class UDiscCSVImport(BaseFileImportMixin):
|
||||
class Meta:
|
||||
verbose_name = "uDisc CSV Import"
|
||||
|
||||
user = models.ForeignKey(
|
||||
User,
|
||||
on_delete=models.DO_NOTHING,
|
||||
**BNULL,
|
||||
related_name="scrobbles_udisccsvimport_set",
|
||||
)
|
||||
|
||||
@property
|
||||
def import_type(self) -> str:
|
||||
return "uDisc"
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("scrobbles:udisc-csv-import-detail", kwargs={"slug": self.uuid})
|
||||
|
||||
def get_path(instance, filename):
|
||||
extension = filename.split(".")[-1]
|
||||
uuid = instance.uuid
|
||||
return f"udisc-csv-uploads/{uuid}.{extension}"
|
||||
|
||||
@property
|
||||
def upload_file_path(self):
|
||||
if getattr(settings, "USE_S3_STORAGE"):
|
||||
path = self.csv_file.url
|
||||
else:
|
||||
path = self.csv_file.path
|
||||
return path
|
||||
|
||||
csv_file = models.FileField(upload_to=get_path, **BNULL)
|
||||
original_filename = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
def process(self, force=False):
|
||||
from discgolf.utils import import_udisc_csv
|
||||
|
||||
if self.processed_finished and not force:
|
||||
logger.info(f"{self} already processed on {self.processed_finished}")
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
try:
|
||||
scrobbles = import_udisc_csv(
|
||||
self.upload_file_path, self.user_id, record_error=self.record_error
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
TYPE_FK_PREFETCHES: dict[str, tuple[str, ...]] = {
|
||||
"Video": ("video",),
|
||||
"Track": ("track", "track__artist_fk"),
|
||||
@ -638,6 +693,7 @@ TYPE_FK_PREFETCHES: dict[str, tuple[str, ...]] = {
|
||||
"BrickSet": ("brick_set",),
|
||||
"Channel": ("channel",),
|
||||
"BirdingLocation": ("birding_location",),
|
||||
"DiscGolfCourse": ("disc_golf_course",),
|
||||
}
|
||||
|
||||
|
||||
@ -665,6 +721,7 @@ class ScrobbleQuerySet(models.QuerySet):
|
||||
"mood",
|
||||
"brick_set",
|
||||
"birding_location",
|
||||
"disc_golf_course",
|
||||
)
|
||||
|
||||
def with_related_for_types(self, media_types: list[str]):
|
||||
@ -715,6 +772,7 @@ class Scrobble(TimeStampedModel):
|
||||
BRICKSET = "BrickSet", "Brick set"
|
||||
CHANNEL = "Channel", "Channel"
|
||||
BIRDING_LOCATION = "BirdingLocation", "Birding location"
|
||||
DISC_GOLF = "DiscGolfCourse", "Disc golf"
|
||||
|
||||
@classmethod
|
||||
def list(cls):
|
||||
@ -745,6 +803,9 @@ class Scrobble(TimeStampedModel):
|
||||
birding_location = models.ForeignKey(
|
||||
BirdingLocation, on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
disc_golf_course = models.ForeignKey(
|
||||
DiscGolfCourse, on_delete=models.DO_NOTHING, **BNULL
|
||||
)
|
||||
media_type = models.CharField(
|
||||
max_length=20, choices=MediaType.choices, default=MediaType.VIDEO
|
||||
)
|
||||
@ -798,6 +859,12 @@ class Scrobble(TimeStampedModel):
|
||||
format="JPEG",
|
||||
options={"quality": 75},
|
||||
)
|
||||
screenshot_large = ImageSpecField(
|
||||
source="screenshot",
|
||||
processors=[ResizeToFit(800, 800)],
|
||||
format="JPEG",
|
||||
options={"quality": 85},
|
||||
)
|
||||
long_play_seconds = models.BigIntegerField(**BNULL)
|
||||
long_play_complete = models.BooleanField(**BNULL)
|
||||
long_play_last_scrobble = models.ForeignKey(
|
||||
@ -900,7 +967,7 @@ class Scrobble(TimeStampedModel):
|
||||
|
||||
@property
|
||||
def finish_url(self) -> str:
|
||||
return reverse("scrobbles:finish", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:finish", kwargs={"pk": self.pk})
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
class_name = self.media_obj.__class__.__name__
|
||||
@ -942,10 +1009,7 @@ class Scrobble(TimeStampedModel):
|
||||
return super(Scrobble, self).save(*args, **kwargs)
|
||||
|
||||
def get_absolute_url(self):
|
||||
if not self.uuid:
|
||||
self.uuid = uuid4()
|
||||
self.save(update_fields=["uuid"])
|
||||
return reverse("scrobbles:detail", kwargs={"uuid": self.uuid})
|
||||
return reverse("scrobbles:detail", kwargs={"pk": self.pk})
|
||||
|
||||
def get_share_url(self):
|
||||
if self.visibility == Visibility.PRIVATE:
|
||||
@ -1313,6 +1377,10 @@ class Scrobble(TimeStampedModel):
|
||||
media_obj = self.channel
|
||||
if self.birding_location:
|
||||
media_obj = self.birding_location
|
||||
if self.paper:
|
||||
media_obj = self.paper
|
||||
if self.disc_golf_course:
|
||||
media_obj = self.disc_golf_course
|
||||
return media_obj
|
||||
|
||||
def __str__(self):
|
||||
@ -1871,6 +1939,9 @@ class FavoriteMedia(TimeStampedModel):
|
||||
birding_location = models.ForeignKey(
|
||||
BirdingLocation, on_delete=models.CASCADE, **BNULL
|
||||
)
|
||||
disc_golf_course = models.ForeignKey(
|
||||
DiscGolfCourse, on_delete=models.CASCADE, **BNULL
|
||||
)
|
||||
media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
|
||||
sent_to_mopidy = models.BooleanField(default=False)
|
||||
|
||||
@ -1921,6 +1992,8 @@ class FavoriteMedia(TimeStampedModel):
|
||||
media_obj = self.channel
|
||||
if self.birding_location:
|
||||
media_obj = self.birding_location
|
||||
if self.disc_golf_course:
|
||||
media_obj = self.disc_golf_course
|
||||
return media_obj
|
||||
|
||||
@classmethod
|
||||
@ -1950,6 +2023,7 @@ class FavoriteMedia(TimeStampedModel):
|
||||
"Mood": "mood",
|
||||
"BrickSet": "brick_set",
|
||||
"BirdingLocation": "birding_location",
|
||||
"DiscGolfCourse": "disc_golf_course",
|
||||
}
|
||||
|
||||
fk = fk_map.get(media_type)
|
||||
|
||||
@ -9,10 +9,11 @@ import requests
|
||||
from beers.models import Beer
|
||||
from boardgames.models import BoardGame, BoardGameDesigner, BoardGameLocation
|
||||
from books.constants import READCOMICSONLINE_URL
|
||||
from books.models import Book, BookLogData, BookPageLogData
|
||||
from books.models import Book, BookLogData, BookPageLogData, Paper
|
||||
from books.utils import parse_readcomicsonline_uri
|
||||
from bricksets.models import BrickSet
|
||||
from dateutil.parser import parse
|
||||
from discgolf.models import DiscGolfCourse
|
||||
from django.utils import timezone
|
||||
from foods.models import Food
|
||||
from foods.sources.rscraper import RecipeScraperService
|
||||
@ -330,8 +331,6 @@ def manual_scrobble_book(
|
||||
|
||||
source = READCOMICSONLINE_URL.replace("https://", "")
|
||||
|
||||
# TODO: Check for scrobble of this book already and if so, update the page count
|
||||
|
||||
book = Book.find_or_create(title, url=url, enrich=True)
|
||||
|
||||
scrobble_dict = {
|
||||
@ -642,6 +641,8 @@ def manual_scrobble_from_url(
|
||||
item_id = "tt" + str(item_id)
|
||||
elif content_key == "-h" and "twitch.tv" in url:
|
||||
item_id = url
|
||||
elif content_key == "-pp" and "doi.org" in url:
|
||||
item_id = url
|
||||
|
||||
scrobble_fn = MANUAL_SCROBBLE_FNS[content_key]
|
||||
return eval(scrobble_fn)(item_id, user_id, source=source, action=action)
|
||||
@ -996,6 +997,38 @@ def manual_scrobble_task(
|
||||
return scrobble
|
||||
|
||||
|
||||
def manual_scrobble_paper(
|
||||
doi_url: str,
|
||||
user_id: int,
|
||||
source: str = "Bookmarklet",
|
||||
action: Optional[str] = None,
|
||||
):
|
||||
paper = Paper.find_or_create_by_doi(doi_url)
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": timezone.now(),
|
||||
"playback_position_seconds": 0,
|
||||
"source": source,
|
||||
}
|
||||
logger.info(
|
||||
"[vrobbler-scrobble] paper scrobble request received",
|
||||
extra={
|
||||
"paper_id": paper.id,
|
||||
"user_id": user_id,
|
||||
"scrobble_dict": scrobble_dict,
|
||||
"media_type": Scrobble.MediaType.PAPER,
|
||||
},
|
||||
)
|
||||
|
||||
scrobble = Scrobble.create_or_update(paper, user_id, scrobble_dict)
|
||||
|
||||
if action == "stop":
|
||||
scrobble.stop(force_finish=True)
|
||||
|
||||
return scrobble
|
||||
|
||||
|
||||
def manual_scrobble_webpage(
|
||||
url: str,
|
||||
user_id: int,
|
||||
@ -1329,3 +1362,32 @@ def manual_scrobble_food(
|
||||
)
|
||||
|
||||
return Scrobble.create_or_update(food, user_id, scrobble_dict)
|
||||
|
||||
|
||||
def manual_scrobble_discgolf(
|
||||
item_id: str,
|
||||
user_id: int,
|
||||
source: str = "Vrobbler",
|
||||
action: Optional[str] = None,
|
||||
):
|
||||
from discgolf.models import DiscGolfCourse
|
||||
|
||||
course, _ = DiscGolfCourse.objects.get_or_create(title=item_id.strip())
|
||||
|
||||
scrobble_dict = {
|
||||
"user_id": user_id,
|
||||
"timestamp": timezone.now(),
|
||||
"playback_position_seconds": 0,
|
||||
"source": source,
|
||||
}
|
||||
|
||||
logger.info(
|
||||
"[scrobblers] manual disc golf scrobble request received",
|
||||
extra={
|
||||
"course_id": course.id,
|
||||
"user_id": user_id,
|
||||
"scrobble_dict": scrobble_dict,
|
||||
},
|
||||
)
|
||||
|
||||
return Scrobble.create_or_update(course, user_id, scrobble_dict)
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
import logging
|
||||
|
||||
from django.core.cache import cache
|
||||
from django.db.models.signals import post_delete, post_save
|
||||
from django.dispatch import receiver
|
||||
from django.utils import timezone
|
||||
@ -9,6 +10,7 @@ from scrobbles.tasks import (
|
||||
add_favorite_to_mopidy_playlist,
|
||||
CHARTABLE_MEDIA_TYPES,
|
||||
remove_favorite_from_mopidy_playlist,
|
||||
reverse_geocode_geolocation,
|
||||
SCROBBLES_WITHOUT_CHARTS,
|
||||
update_charts_for_timestamp,
|
||||
)
|
||||
@ -52,6 +54,11 @@ def _update_charts_for_timestamp(user, ts):
|
||||
if ts is None:
|
||||
return
|
||||
|
||||
lock_key = f"chart_update_{user.id}"
|
||||
if not cache.add(lock_key, "locked", timeout=30):
|
||||
logger.info(f"Chart update already queued for user {user.id}, skipping")
|
||||
return
|
||||
|
||||
if timezone.is_naive(ts):
|
||||
ts = timezone.make_aware(ts)
|
||||
|
||||
@ -80,6 +87,31 @@ def add_tags_from_task_title(sender, instance, **kwargs):
|
||||
instance.tags.add(tag)
|
||||
|
||||
|
||||
@receiver(post_save, sender=Scrobble)
|
||||
def reverse_geocode_on_scrobble_creation(sender, instance, created, **kwargs):
|
||||
if not created:
|
||||
return
|
||||
if not instance.geo_location_id:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: scrobble %s has no geo_location",
|
||||
instance.id,
|
||||
)
|
||||
return
|
||||
if instance.geo_location.postal_code:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: geo_location %s already has postal_code %s",
|
||||
instance.geo_location_id,
|
||||
instance.geo_location.postal_code,
|
||||
)
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Enqueuing reverse geocode for geo_location %s",
|
||||
instance.geo_location_id,
|
||||
)
|
||||
reverse_geocode_geolocation.delay(instance.geo_location_id)
|
||||
|
||||
|
||||
@receiver(post_save, sender=FavoriteMedia)
|
||||
def add_to_mopidy_playlist_on_favorite(sender, instance, created, **kwargs):
|
||||
if not created:
|
||||
|
||||
@ -12,6 +12,7 @@ from charts.utils import (
|
||||
from django.apps import apps
|
||||
from django.conf import settings
|
||||
from django.contrib.auth import get_user_model
|
||||
from django.core.cache import cache
|
||||
from django.db import models
|
||||
from django.utils import timezone
|
||||
|
||||
@ -170,6 +171,16 @@ def process_ebird_csv_import(import_id):
|
||||
birding_import.process()
|
||||
|
||||
|
||||
@shared_task
|
||||
def process_udisc_csv_import(import_id):
|
||||
UDiscCSVImport = apps.get_model("scrobbles", "UDiscCSVImport")
|
||||
udisc_import = UDiscCSVImport.objects.filter(id=import_id).first()
|
||||
if not udisc_import:
|
||||
logger.warn(f"UDiscCSVImport not found with id {import_id}")
|
||||
return
|
||||
udisc_import.process()
|
||||
|
||||
|
||||
@shared_task
|
||||
def process_scale_csv_import(import_id):
|
||||
ScaleCSVImport = apps.get_model("scrobbles", "ScaleCSVImport")
|
||||
@ -241,6 +252,11 @@ def update_charts_for_timestamp(user_id, year, month, day, week):
|
||||
logger.error(f"User with id {user_id} not found")
|
||||
return
|
||||
|
||||
lock_key = f"chart_update_running_{user_id}"
|
||||
if not cache.add(lock_key, "locked", timeout=300):
|
||||
logger.info(f"Chart update already running for user {user_id}, skipping")
|
||||
return
|
||||
|
||||
try:
|
||||
build_daily_charts(user, year, month, day, CHARTABLE_MEDIA_TYPES)
|
||||
build_weekly_charts(user, year, week, CHARTABLE_MEDIA_TYPES)
|
||||
@ -250,6 +266,8 @@ def update_charts_for_timestamp(user_id, year, month, day, week):
|
||||
logger.info(f"[charts] Updated charts for {user} on {date_str}")
|
||||
except Exception as e:
|
||||
logger.error(f"[charts] Failed to update charts: {e}")
|
||||
finally:
|
||||
cache.delete(lock_key)
|
||||
|
||||
|
||||
@shared_task
|
||||
@ -708,3 +726,41 @@ def add_scrobble_to_mopidy_monthly_playlist(scrobble_id):
|
||||
break
|
||||
|
||||
add_track_to_mopidy_monthly_playlist(scrobble)
|
||||
|
||||
|
||||
@shared_task
|
||||
def reverse_geocode_geolocation(geo_location_id):
|
||||
from locations.models import GeoLocation
|
||||
|
||||
location = GeoLocation.objects.filter(id=geo_location_id).first()
|
||||
if not location:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: geo_location %s not found",
|
||||
geo_location_id,
|
||||
)
|
||||
return
|
||||
if location.postal_code:
|
||||
logger.info(
|
||||
"Skipping reverse geocode: geo_location %s already has postal_code %s",
|
||||
geo_location_id,
|
||||
location.postal_code,
|
||||
)
|
||||
return
|
||||
|
||||
logger.info(
|
||||
"Reverse geocoding geo_location %s (%s, %s)",
|
||||
geo_location_id,
|
||||
location.lat,
|
||||
location.lon,
|
||||
)
|
||||
if location.reverse_geocode():
|
||||
logger.info(
|
||||
"Reverse geocode succeeded for geo_location %s: %s",
|
||||
geo_location_id,
|
||||
location.display_address,
|
||||
)
|
||||
else:
|
||||
logger.warning(
|
||||
"Reverse geocode failed for geo_location %s",
|
||||
geo_location_id,
|
||||
)
|
||||
|
||||
@ -5,6 +5,7 @@ from tasks.webhooks import EmacsWebhookView, TodoistWebhookView
|
||||
app_name = "scrobbles"
|
||||
|
||||
urlpatterns = [
|
||||
path("now-playing/", views.NowPlayingPartialView.as_view(), name="now-playing-partial"),
|
||||
path("calendar/", views.ScrobbleCalendarView.as_view(), name="calendar"),
|
||||
path("search/", views.ScrobbleSearchView.as_view(), name="search"),
|
||||
path("status/", views.ScrobbleStatusView.as_view(), name="status"),
|
||||
@ -44,7 +45,7 @@ urlpatterns = [
|
||||
name="lookup-manual-scrobble",
|
||||
),
|
||||
path(
|
||||
"long-play-finish/<slug:uuid>/",
|
||||
"long-play-finish/<slug:media_uuid>/",
|
||||
views.scrobble_longplay_finish,
|
||||
name="longplay-finish",
|
||||
),
|
||||
@ -147,6 +148,11 @@ urlpatterns = [
|
||||
views.ScrobbleBirdingCSVImportDetailView.as_view(),
|
||||
name="ebird-csv-import-detail",
|
||||
),
|
||||
path(
|
||||
"imports/udisc-csv/<slug:slug>/",
|
||||
views.ScrobbleUDiscCSVImportDetailView.as_view(),
|
||||
name="udisc-csv-import-detail",
|
||||
),
|
||||
path(
|
||||
"long-plays/",
|
||||
views.ScrobbleLongPlaysView.as_view(),
|
||||
@ -160,38 +166,38 @@ urlpatterns = [
|
||||
name="shared-detail",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/",
|
||||
"scrobbles/<int:pk>/",
|
||||
views.ScrobbleDetailView.as_view(),
|
||||
name="detail",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/regenerate-share-token/",
|
||||
"scrobbles/<int:pk>/regenerate-share-token/",
|
||||
views.RegenerateShareTokenView.as_view(),
|
||||
name="regenerate-share-token",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/change-visibility/",
|
||||
"scrobbles/<int:pk>/change-visibility/",
|
||||
views.ChangeVisibilityView.as_view(),
|
||||
name="change-visibility",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/share-analytics/",
|
||||
"scrobbles/<int:pk>/share-analytics/",
|
||||
views.ScrobbleShareAnalyticsView.as_view(),
|
||||
name="share-analytics",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/add-to-mopidy-queue/",
|
||||
"scrobbles/<int:pk>/add-to-mopidy-queue/",
|
||||
views.add_to_mopidy_queue,
|
||||
name="add-to-mopidy-queue",
|
||||
),
|
||||
path(
|
||||
"scrobbles/<slug:uuid>/add-to-mopidy-monthly-playlist/",
|
||||
"scrobbles/<int:pk>/add-to-mopidy-monthly-playlist/",
|
||||
views.add_to_mopidy_monthly_playlist,
|
||||
name="add-to-mopidy-monthly-playlist",
|
||||
),
|
||||
path("scrobbles/<slug:uuid>/start/", views.scrobble_start, name="start"),
|
||||
path("scrobbles/<slug:uuid>/finish/", views.scrobble_finish, name="finish"),
|
||||
path("scrobbles/<slug:uuid>/cancel/", views.scrobble_cancel, name="cancel"),
|
||||
path("scrobbles/<slug:media_uuid>/start/", views.scrobble_start, name="start"),
|
||||
path("scrobbles/<int:pk>/finish/", views.scrobble_finish, name="finish"),
|
||||
path("scrobbles/<int:pk>/cancel/", views.scrobble_cancel, name="cancel"),
|
||||
path(
|
||||
"favorite/<str:media_type>/<int:object_id>/toggle/",
|
||||
views.toggle_favorite,
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
import hashlib
|
||||
import html
|
||||
import logging
|
||||
import requests
|
||||
import re
|
||||
@ -153,10 +154,11 @@ def import_lastfm_for_all_users(restart=False):
|
||||
last_processed = lfm_import.processed_finished
|
||||
else:
|
||||
logger.info(
|
||||
f"Not resuming failed LastFM import {lfm_import.id} for user {user_id}, use restart=True to restart"
|
||||
"No existing LastFM import, we should start a monthly parsing of lastFm for this user going back to 2002"
|
||||
"No existing LastFM import for user %s, "
|
||||
"starting a full parse",
|
||||
user_id,
|
||||
)
|
||||
continue
|
||||
last_processed = None
|
||||
|
||||
lfm_client = LastFM(user=get_user_model().objects.filter(id=user_id).first())
|
||||
|
||||
@ -795,6 +797,7 @@ def tokenize_title_to_tags(title: str) -> list[str]:
|
||||
if not title:
|
||||
return []
|
||||
|
||||
title = html.unescape(title)
|
||||
cleaned = re.sub(r"[\(\)\[\]\{\}]", "", title)
|
||||
cleaned = re.sub(r"[^\w\s]", "", cleaned)
|
||||
|
||||
|
||||
@ -87,6 +87,7 @@ from scrobbles.models import (
|
||||
ScrobbleQuerySet,
|
||||
ShareViewLog,
|
||||
TrailGPXImport,
|
||||
UDiscCSVImport,
|
||||
)
|
||||
from scrobbles.scrobblers import *
|
||||
from scrobbles.tasks import (
|
||||
@ -360,11 +361,35 @@ class RecentScrobbleList(ListView):
|
||||
return Scrobble.objects.all().order_by("-timestamp")
|
||||
|
||||
|
||||
class NowPlayingPartialView(LoginRequiredMixin, TemplateView):
|
||||
template_name = "scrobbles/_now_playing.html"
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
ctx = super().get_context_data(**kwargs)
|
||||
from scrobbles.constants import EXCLUDE_FROM_NOW_PLAYING
|
||||
|
||||
ctx["now_playing_list"] = list(
|
||||
Scrobble.objects.filter(
|
||||
in_progress=True,
|
||||
is_paused=False,
|
||||
user=self.request.user,
|
||||
)
|
||||
.exclude(media_type__in=EXCLUDE_FROM_NOW_PLAYING)
|
||||
.select_related("track", "video", "podcast_episode")
|
||||
)
|
||||
return ctx
|
||||
|
||||
|
||||
class ScrobbleListView(LoginRequiredMixin, ListView):
|
||||
model = Scrobble
|
||||
paginate_by = 100
|
||||
template_name = "scrobbles/scrobble_all_list.html"
|
||||
|
||||
def get_template_names(self):
|
||||
if self.request.headers.get("HX-Request"):
|
||||
return ["scrobbles/_scrobble_all_content.html"]
|
||||
return ["scrobbles/scrobble_all_list.html"]
|
||||
|
||||
def get_queryset(self):
|
||||
qs = Scrobble.objects.filter(user=self.request.user).order_by("-timestamp")
|
||||
tags_param = self.request.GET.get("tags", "")
|
||||
@ -535,6 +560,8 @@ class BaseScrobbleImportDetailView(DetailView):
|
||||
title = "Scale CSV Import"
|
||||
if self.model == TrailGPXImport:
|
||||
title = "Trail GPX Import"
|
||||
if self.model == UDiscCSVImport:
|
||||
title = "uDisc CSV Import"
|
||||
context_data["title"] = title
|
||||
return context_data
|
||||
|
||||
@ -571,6 +598,10 @@ class ScrobbleBirdingCSVImportDetailView(BaseScrobbleImportDetailView):
|
||||
model = EBirdCSVImport
|
||||
|
||||
|
||||
class ScrobbleUDiscCSVImportDetailView(BaseScrobbleImportDetailView):
|
||||
model = UDiscCSVImport
|
||||
|
||||
|
||||
class ManualScrobbleView(FormView):
|
||||
form_class = ScrobbleForm
|
||||
template_name = "scrobbles/manual_form.html"
|
||||
@ -579,7 +610,12 @@ class ManualScrobbleView(FormView):
|
||||
item_str = form.cleaned_data.get("item_id")
|
||||
logger.debug(f"Looking for scrobblable media with input {item_str}")
|
||||
|
||||
key, item_id = item_str[:2], item_str[3:]
|
||||
if len(item_str) > 2 and item_str[:3] in MANUAL_SCROBBLE_FNS:
|
||||
key = item_str[:3]
|
||||
item_id = item_str[4:]
|
||||
else:
|
||||
key = item_str[:2]
|
||||
item_id = item_str[3:]
|
||||
scrobble_fn = MANUAL_SCROBBLE_FNS[key]
|
||||
scrobble = eval(scrobble_fn)(item_id, self.request.user.id)
|
||||
|
||||
@ -839,10 +875,10 @@ def import_audioscrobbler_file(request):
|
||||
|
||||
@api_view(["GET"])
|
||||
@permission_classes([IsAuthenticated])
|
||||
def scrobble_start(request, uuid):
|
||||
def scrobble_start(request, media_uuid):
|
||||
logger.info(
|
||||
"[scrobble_start] called",
|
||||
extra={"request": request, "uuid": uuid},
|
||||
extra={"request": request, "media_uuid": media_uuid},
|
||||
)
|
||||
user = request.user
|
||||
success_url = request.META.get("HTTP_REFERER")
|
||||
@ -853,14 +889,14 @@ def scrobble_start(request, uuid):
|
||||
media_obj = None
|
||||
for app, model in PLAY_AGAIN_MEDIA.items():
|
||||
media_model = apps.get_model(app_label=app, model_name=model)
|
||||
media_obj = media_model.objects.filter(uuid=uuid).first()
|
||||
media_obj = media_model.objects.filter(uuid=media_uuid).first()
|
||||
if media_obj:
|
||||
break
|
||||
|
||||
if not media_obj:
|
||||
logger.info(
|
||||
"[scrobble_start] media object not found",
|
||||
extra={"uuid": uuid, "user_id": user.id},
|
||||
extra={"media_uuid": media_uuid, "user_id": user.id},
|
||||
)
|
||||
raise Exception("No media object provided to scrobble")
|
||||
|
||||
@ -887,7 +923,7 @@ def scrobble_start(request, uuid):
|
||||
if last_scrobble and last_scrobble.logdata:
|
||||
next_page = last_scrobble.logdata.page_end + 1
|
||||
log_data = {"page_start": next_page}
|
||||
media_obj.scrobble_for_user(user_id, log=log_data)
|
||||
scrobble = media_obj.scrobble_for_user(user_id, log=log_data)
|
||||
|
||||
if scrobble:
|
||||
messages.add_message(
|
||||
@ -897,7 +933,7 @@ def scrobble_start(request, uuid):
|
||||
)
|
||||
else:
|
||||
messages.add_message(
|
||||
request, messages.ERROR, f"Media with uuid {uuid} not found."
|
||||
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
|
||||
)
|
||||
|
||||
if (
|
||||
@ -915,7 +951,7 @@ def scrobble_start(request, uuid):
|
||||
|
||||
|
||||
@api_view(["GET"])
|
||||
def scrobble_longplay_finish(request, uuid):
|
||||
def scrobble_longplay_finish(request, media_uuid):
|
||||
user = request.user
|
||||
success_url = request.META.get("HTTP_REFERER")
|
||||
|
||||
@ -923,7 +959,7 @@ def scrobble_longplay_finish(request, uuid):
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
# Try scrobble UUID first
|
||||
scrobble = Scrobble.objects.filter(uuid=uuid, user=user).first()
|
||||
scrobble = Scrobble.objects.filter(uuid=media_uuid, user=user).first()
|
||||
if scrobble:
|
||||
if scrobble.long_play_complete == True:
|
||||
scrobble.long_play_complete = None
|
||||
@ -947,13 +983,13 @@ def scrobble_longplay_finish(request, uuid):
|
||||
media_obj = None
|
||||
for app, model in LONG_PLAY_MEDIA.items():
|
||||
media_model = apps.get_model(app_label=app, model_name=model)
|
||||
media_obj = media_model.objects.filter(uuid=uuid).first()
|
||||
media_obj = media_model.objects.filter(uuid=media_uuid).first()
|
||||
if media_obj:
|
||||
break
|
||||
|
||||
if not media_obj:
|
||||
messages.add_message(
|
||||
request, messages.ERROR, f"Media with uuid {uuid} not found."
|
||||
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
|
||||
)
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
@ -976,14 +1012,14 @@ def scrobble_longplay_finish(request, uuid):
|
||||
)
|
||||
else:
|
||||
messages.add_message(
|
||||
request, messages.ERROR, f"Media with uuid {uuid} not found."
|
||||
request, messages.ERROR, f"Media with uuid {media_uuid} not found."
|
||||
)
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
|
||||
@api_view(["GET"])
|
||||
@permission_classes([IsAuthenticated])
|
||||
def scrobble_finish(request, uuid):
|
||||
def scrobble_finish(request, pk):
|
||||
user = request.user
|
||||
success_url = request.META.get("HTTP_REFERER")
|
||||
if not success_url:
|
||||
@ -992,7 +1028,7 @@ def scrobble_finish(request, uuid):
|
||||
if not user.is_authenticated:
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
scrobble = Scrobble.objects.filter(user=user, uuid=uuid).first()
|
||||
scrobble = Scrobble.objects.filter(user=user, pk=pk).first()
|
||||
if scrobble:
|
||||
scrobble.stop(force_finish=True)
|
||||
messages.add_message(
|
||||
@ -1007,14 +1043,14 @@ def scrobble_finish(request, uuid):
|
||||
|
||||
@api_view(["GET"])
|
||||
@permission_classes([IsAuthenticated])
|
||||
def scrobble_cancel(request, uuid):
|
||||
def scrobble_cancel(request, pk):
|
||||
user = request.user
|
||||
success_url = reverse_lazy("vrobbler-home")
|
||||
|
||||
if not user.is_authenticated:
|
||||
return HttpResponseRedirect(success_url)
|
||||
|
||||
scrobble = Scrobble.objects.filter(user=user, uuid=uuid).first()
|
||||
scrobble = Scrobble.objects.filter(user=user, pk=pk).first()
|
||||
if scrobble:
|
||||
scrobble.cancel()
|
||||
messages.add_message(
|
||||
@ -1028,11 +1064,11 @@ def scrobble_cancel(request, uuid):
|
||||
|
||||
|
||||
@require_POST
|
||||
def add_to_mopidy_queue(request, uuid):
|
||||
def add_to_mopidy_queue(request, pk):
|
||||
if not request.user.is_authenticated:
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
|
||||
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
|
||||
mopidy_url = request.user.profile.mopidy_api_url
|
||||
|
||||
if not mopidy_url:
|
||||
@ -1041,22 +1077,22 @@ def add_to_mopidy_queue(request, uuid):
|
||||
messages.ERROR,
|
||||
"Mopidy API URL not configured in your profile settings.",
|
||||
)
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
from scrobbles.tasks import add_scrobble_to_mopidy_queue as task
|
||||
|
||||
task.delay(scrobble.id)
|
||||
msg = f'Adding "{scrobble.media_obj}" to Mopidy queue.'
|
||||
messages.add_message(request, messages.SUCCESS, msg)
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
|
||||
@require_POST
|
||||
def add_to_mopidy_monthly_playlist(request, uuid):
|
||||
def add_to_mopidy_monthly_playlist(request, pk):
|
||||
if not request.user.is_authenticated:
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
|
||||
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
|
||||
profile = request.user.profile
|
||||
pattern = profile.monthly_mopidy_playlist_pattern
|
||||
|
||||
@ -1066,7 +1102,7 @@ def add_to_mopidy_monthly_playlist(request, uuid):
|
||||
messages.ERROR,
|
||||
"Monthly playlist pattern or Mopidy API URL not configured in your profile.",
|
||||
)
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
now = now_user_timezone(profile)
|
||||
playlist_name = DateFormat(now).format(pattern)
|
||||
@ -1079,7 +1115,7 @@ def add_to_mopidy_monthly_playlist(request, uuid):
|
||||
messages.SUCCESS,
|
||||
f'Adding "{scrobble.media_obj}" to monthly playlist "{playlist_name}".',
|
||||
)
|
||||
return redirect("scrobbles:detail", uuid=uuid)
|
||||
return redirect("scrobbles:detail", pk=pk)
|
||||
|
||||
|
||||
@require_POST
|
||||
@ -1108,6 +1144,7 @@ def toggle_favorite(request, media_type, object_id):
|
||||
"Mood": ("moods", "Mood"),
|
||||
"BrickSet": ("bricksets", "BrickSet"),
|
||||
"BirdingLocation": ("birds", "BirdingLocation"),
|
||||
"DiscGolfCourse": ("discgolf", "DiscGolfCourse"),
|
||||
}
|
||||
|
||||
app_label, model_name = app_model_map.get(media_type, (None, None))
|
||||
@ -1184,8 +1221,6 @@ class ScrobbleStatusView(LoginRequiredMixin, TemplateView):
|
||||
|
||||
class ScrobbleDetailView(DetailView):
|
||||
model = Scrobble
|
||||
slug_field = "uuid"
|
||||
slug_url_kwarg = "uuid"
|
||||
paginate_by = 100
|
||||
|
||||
def get_object(self, queryset=None):
|
||||
@ -1385,15 +1420,15 @@ class ScrobbleExploreView(ListView):
|
||||
|
||||
|
||||
class RegenerateShareTokenView(LoginRequiredMixin, View):
|
||||
def post(self, request, uuid):
|
||||
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
|
||||
def post(self, request, pk):
|
||||
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
|
||||
scrobble.regenerate_share_token()
|
||||
return redirect(scrobble.get_absolute_url())
|
||||
|
||||
|
||||
class ChangeVisibilityView(LoginRequiredMixin, View):
|
||||
def post(self, request, uuid):
|
||||
scrobble = get_object_or_404(Scrobble, uuid=uuid, user=request.user)
|
||||
def post(self, request, pk):
|
||||
scrobble = get_object_or_404(Scrobble, pk=pk, user=request.user)
|
||||
visibility = request.POST.get("visibility")
|
||||
if visibility not in (Visibility.PUBLIC, Visibility.SHARED, Visibility.PRIVATE):
|
||||
return redirect(scrobble.get_absolute_url())
|
||||
@ -1404,8 +1439,6 @@ class ChangeVisibilityView(LoginRequiredMixin, View):
|
||||
|
||||
class ScrobbleShareAnalyticsView(LoginRequiredMixin, DetailView):
|
||||
model = Scrobble
|
||||
slug_field = "uuid"
|
||||
slug_url_kwarg = "uuid"
|
||||
template_name = "scrobbles/scrobble_share_analytics.html"
|
||||
|
||||
def get_queryset(self):
|
||||
@ -1721,6 +1754,7 @@ class ScrobbleCalendarView(LoginRequiredMixin, TemplateView):
|
||||
for scrobble in day_map[day_num]:
|
||||
day_scrobbles.append(
|
||||
{
|
||||
"id": scrobble.pk,
|
||||
"uuid": scrobble.uuid,
|
||||
"emoji": self.MEDIA_EMOJI.get(scrobble.media_type, "📌"),
|
||||
"title": (
|
||||
|
||||
@ -0,0 +1,29 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-22 02:44
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
(
|
||||
"trends",
|
||||
"0002_alter_trendresult_unique_together_trendresult_period_and_more",
|
||||
),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AlterField(
|
||||
model_name="trendresult",
|
||||
name="period",
|
||||
field=models.CharField(
|
||||
choices=[
|
||||
("last_30", "Last 30 days"),
|
||||
("last_90", "Last 90 days"),
|
||||
("last_year", "Last year"),
|
||||
],
|
||||
default="last_30",
|
||||
max_length=20,
|
||||
),
|
||||
),
|
||||
]
|
||||
@ -8,7 +8,6 @@ PERIOD_CHOICES = [
|
||||
("last_30", "Last 30 days"),
|
||||
("last_90", "Last 90 days"),
|
||||
("last_year", "Last year"),
|
||||
("all_time", "All time"),
|
||||
]
|
||||
|
||||
|
||||
@ -18,7 +17,7 @@ class TrendResult(TimeStampedModel):
|
||||
period = models.CharField(
|
||||
max_length=20,
|
||||
choices=PERIOD_CHOICES,
|
||||
default="all_time",
|
||||
default="last_30",
|
||||
)
|
||||
computed_at = models.DateTimeField(auto_now_add=True)
|
||||
data = models.JSONField(default=dict)
|
||||
|
||||
78
vrobbler/apps/trends/templates/trends/_mood_by_time.html
Normal file
78
vrobbler/apps/trends/templates/trends/_mood_by_time.html
Normal file
@ -0,0 +1,78 @@
|
||||
<div class="row">
|
||||
<div class="col-md-6">
|
||||
<h5 class="mb-3">By Hour of Day</h5>
|
||||
{% if data.hours %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Hour</th>
|
||||
<th class="text-end">Avg Quality</th>
|
||||
<th class="text-end">Check-ins</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for entry in data.hours %}
|
||||
{% if entry.count > 0 %}
|
||||
<tr>
|
||||
<td>
|
||||
{% if entry.hour == 0 %}
|
||||
12 AM
|
||||
{% elif entry.hour < 12 %}
|
||||
{{ entry.hour }} AM
|
||||
{% elif entry.hour == 12 %}
|
||||
12 PM
|
||||
{% else %}
|
||||
{{ entry.hour|add:"-12" }} PM
|
||||
{% endif %}
|
||||
</td>
|
||||
<td class="text-end">
|
||||
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
|
||||
{{ entry.avg_quality }}
|
||||
</span>
|
||||
</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
</tr>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No hourly data.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
<div class="col-md-6">
|
||||
<h5 class="mb-3">By Day of Week</h5>
|
||||
{% if data.days %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Day</th>
|
||||
<th class="text-end">Avg Quality</th>
|
||||
<th class="text-end">Check-ins</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for entry in data.days %}
|
||||
{% if entry.count > 0 %}
|
||||
<tr>
|
||||
<td>{{ entry.day_name }}</td>
|
||||
<td class="text-end">
|
||||
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
|
||||
{{ entry.avg_quality }}
|
||||
</span>
|
||||
</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
</tr>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No daily data.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
@ -0,0 +1,45 @@
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
{% if data.moods %}
|
||||
<p class="text-muted mb-3">
|
||||
Total mood check-ins{% if current_period_label %} ({{ current_period_label }}){% endif %}: <strong>{{ data.total }}</strong>
|
||||
· Positive: <strong>{{ data.positive_count }}</strong>
|
||||
· Negative: <strong>{{ data.negative_count }}</strong>
|
||||
</p>
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Mood</th>
|
||||
<th class="text-end">Count</th>
|
||||
<th>Distribution</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% with max=data.moods.0.count %}
|
||||
{% for entry in data.moods %}
|
||||
<tr>
|
||||
<td>{{ entry.mood }}</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
<td style="width: 40%;">
|
||||
{% if max > 0 %}
|
||||
<div class="progress" style="height: 12px;">
|
||||
<div class="progress-bar" role="progressbar"
|
||||
style="width: {% widthratio entry.count max 100 %}%;"
|
||||
aria-valuenow="{{ entry.count }}"
|
||||
aria-valuemin="0" aria-valuemax="{{ max }}">
|
||||
</div>
|
||||
</div>
|
||||
{% endif %}
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
{% endwith %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No mood distribution data found.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
47
vrobbler/apps/trends/templates/trends/_mood_streaks.html
Normal file
47
vrobbler/apps/trends/templates/trends/_mood_streaks.html
Normal file
@ -0,0 +1,47 @@
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
{% if data.current_streak %}
|
||||
<div class="alert alert-info">
|
||||
<strong>Current streak:</strong>
|
||||
{{ data.current_streak.length }} consecutive
|
||||
<span class="{% if data.current_streak.mood_type == 'positive' %}text-success{% else %}text-danger{% endif %}">
|
||||
{{ data.current_streak.mood_type }}
|
||||
</span>
|
||||
check-ins since {{ data.current_streak.start_date }}.
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if data.streaks %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>#</th>
|
||||
<th>Mood Type</th>
|
||||
<th class="text-end">Length</th>
|
||||
<th>Start</th>
|
||||
<th>End</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for streak in data.streaks %}
|
||||
<tr>
|
||||
<td>{{ forloop.counter }}</td>
|
||||
<td>
|
||||
<span class="{% if streak.mood_type == 'positive' %}text-success{% elif streak.mood_type == 'negative' %}text-danger{% endif %}">
|
||||
{{ streak.mood_type|title }}
|
||||
</span>
|
||||
</td>
|
||||
<td class="text-end">{{ streak.length }}</td>
|
||||
<td>{{ streak.start_date }}</td>
|
||||
<td>{{ streak.end_date }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No streak data found.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
39
vrobbler/apps/trends/templates/trends/_mood_trajectory.html
Normal file
39
vrobbler/apps/trends/templates/trends/_mood_trajectory.html
Normal file
@ -0,0 +1,39 @@
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
{% if data.trajectory %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Date</th>
|
||||
<th class="text-end">Avg Quality</th>
|
||||
<th class="text-end">Check-ins</th>
|
||||
<th>Mood Bar</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for entry in data.trajectory %}
|
||||
<tr>
|
||||
<td>{{ entry.date }}</td>
|
||||
<td class="text-end">{{ entry.avg_quality }}</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
<td style="width: 40%;">
|
||||
<div class="progress" style="height: 16px;">
|
||||
<div class="progress-bar {% if entry.avg_quality >= 5 %}bg-success{% elif entry.avg_quality >= 4 %}bg-info{% elif entry.avg_quality >= 3 %}bg-warning{% else %}bg-danger{% endif %}"
|
||||
role="progressbar"
|
||||
style="width: {% widthratio entry.avg_quality 7 100 %}%;"
|
||||
aria-valuenow="{{ entry.avg_quality }}"
|
||||
aria-valuemin="1" aria-valuemax="7">
|
||||
</div>
|
||||
</div>
|
||||
</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No mood check-in data found.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
64
vrobbler/apps/trends/templates/trends/_mood_weather.html
Normal file
64
vrobbler/apps/trends/templates/trends/_mood_weather.html
Normal file
@ -0,0 +1,64 @@
|
||||
<div class="row">
|
||||
<div class="col-md-6">
|
||||
<h5 class="mb-3">By Weather Condition</h5>
|
||||
{% if data.conditions %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Condition</th>
|
||||
<th class="text-end">Avg Quality</th>
|
||||
<th class="text-end">Check-ins</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for entry in data.conditions %}
|
||||
<tr>
|
||||
<td>{{ entry.condition }}</td>
|
||||
<td class="text-end">
|
||||
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
|
||||
{{ entry.avg_quality }}
|
||||
</span>
|
||||
</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No weather-linked mood data found.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
<div class="col-md-6">
|
||||
<h5 class="mb-3">By Temperature Range</h5>
|
||||
{% if data.temp_ranges %}
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Temp Range</th>
|
||||
<th class="text-end">Avg Quality</th>
|
||||
<th class="text-end">Check-ins</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for entry in data.temp_ranges %}
|
||||
<tr>
|
||||
<td>{{ entry.range }}</td>
|
||||
<td class="text-end">
|
||||
<span class="{% if entry.avg_quality >= 5 %}text-success{% elif entry.avg_quality >= 4 %}text-info{% elif entry.avg_quality >= 3 %}text-warning{% else %}text-danger{% endif %}">
|
||||
{{ entry.avg_quality }}
|
||||
</span>
|
||||
</td>
|
||||
<td class="text-end">{{ entry.count }}</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% else %}
|
||||
<p class="text-muted">No temperature-linked mood data found.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
@ -0,0 +1,65 @@
|
||||
<div class="row">
|
||||
<div class="col-12">
|
||||
{% if data.total and data.total > 0 %}
|
||||
<h5>Overall</h5>
|
||||
<div class="table-responsive mb-4">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Category</th>
|
||||
<th class="text-end">Scrobbles</th>
|
||||
<th class="text-end">%</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for slug, info in data.categories.items %}
|
||||
<tr>
|
||||
<td>{{ info.label }}</td>
|
||||
<td class="text-end">{{ info.count }}</td>
|
||||
<td class="text-end">{{ info.pct }}%</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
<tr class="table-secondary">
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="text-end"><strong>{{ data.total }}</strong></td>
|
||||
<td class="text-end"></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
|
||||
<h5>By Media Type</h5>
|
||||
{% for mt, mt_data in data.by_media_type.items %}
|
||||
<h6 class="mt-3">{{ mt }}</h6>
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
<thead>
|
||||
<tr>
|
||||
<th>Category</th>
|
||||
<th class="text-end">Scrobbles</th>
|
||||
<th class="text-end">%</th>
|
||||
</tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
{% for slug, info in mt_data.categories.items %}
|
||||
<tr>
|
||||
<td>{{ info.label }}</td>
|
||||
<td class="text-end">{{ info.count }}</td>
|
||||
<td class="text-end">{{ info.pct }}%</td>
|
||||
</tr>
|
||||
{% endfor %}
|
||||
<tr class="table-secondary">
|
||||
<td><strong>Total</strong></td>
|
||||
<td class="text-end"><strong>{{ mt_data.total }}</strong></td>
|
||||
<td class="text-end"></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</div>
|
||||
{% endfor %}
|
||||
|
||||
{% else %}
|
||||
<p class="text-muted">No data found for Books, Trails, Birding Locations, or Board Games in this period.</p>
|
||||
{% endif %}
|
||||
</div>
|
||||
</div>
|
||||
@ -55,6 +55,9 @@
|
||||
{% elif trend.slug == "reading-pace-vs-activity" %}
|
||||
{% include "trends/_reading_pace.html" %}
|
||||
|
||||
{% elif trend.slug == "time-of-day-categories" %}
|
||||
{% include "trends/_time_of_day_categories.html" %}
|
||||
|
||||
{% elif trend.slug == "trending-up" %}
|
||||
{% include "trends/_trending_up.html" %}
|
||||
|
||||
@ -67,5 +70,20 @@
|
||||
{% elif trend.slug == "activity-distribution" %}
|
||||
{% include "trends/_activity_distribution.html" %}
|
||||
|
||||
{% elif trend.slug == "mood-trajectory" %}
|
||||
{% include "trends/_mood_trajectory.html" %}
|
||||
|
||||
{% elif trend.slug == "mood-by-time" %}
|
||||
{% include "trends/_mood_by_time.html" %}
|
||||
|
||||
{% elif trend.slug == "mood-distribution" %}
|
||||
{% include "trends/_mood_distribution.html" %}
|
||||
|
||||
{% elif trend.slug == "mood-streaks" %}
|
||||
{% include "trends/_mood_streaks.html" %}
|
||||
|
||||
{% elif trend.slug == "mood-weather" %}
|
||||
{% include "trends/_mood_weather.html" %}
|
||||
|
||||
{% endif %}
|
||||
{% endblock %}
|
||||
|
||||
@ -7,7 +7,15 @@ from trends.trends.concurrent import (
|
||||
compute_concurrent_listening,
|
||||
compute_concurrent_reading,
|
||||
)
|
||||
from trends.trends.mood import (
|
||||
compute_mood_by_time,
|
||||
compute_mood_distribution,
|
||||
compute_mood_streaks,
|
||||
compute_mood_trajectory,
|
||||
compute_mood_weather,
|
||||
)
|
||||
from trends.trends.reading import compute_reading_pace_vs_activity
|
||||
from trends.trends.time_of_day import compute_time_of_day_categories
|
||||
from trends.trends.trending import compute_trending_up
|
||||
|
||||
TREND_REGISTRY = {}
|
||||
@ -28,9 +36,17 @@ compute_activity_distribution = register("activity-distribution")(
|
||||
# compute_concurrent_listening
|
||||
# )
|
||||
compute_concurrent_reading = register("concurrent-reading")(compute_concurrent_reading)
|
||||
compute_mood_by_time = register("mood-by-time")(compute_mood_by_time)
|
||||
compute_mood_distribution = register("mood-distribution")(compute_mood_distribution)
|
||||
compute_mood_streaks = register("mood-streaks")(compute_mood_streaks)
|
||||
compute_mood_trajectory = register("mood-trajectory")(compute_mood_trajectory)
|
||||
compute_mood_weather = register("mood-weather")(compute_mood_weather)
|
||||
compute_peak_hours = register("peak-hours")(compute_peak_hours)
|
||||
compute_reading_pace_vs_activity = register("reading-pace-vs-activity")(
|
||||
compute_reading_pace_vs_activity
|
||||
)
|
||||
compute_time_of_day_categories = register("time-of-day-categories")(
|
||||
compute_time_of_day_categories
|
||||
)
|
||||
compute_trending_up = register("trending-up")(compute_trending_up)
|
||||
compute_weekly_rhythm = register("weekly-rhythm")(compute_weekly_rhythm)
|
||||
|
||||
208
vrobbler/apps/trends/trends/mood.py
Normal file
208
vrobbler/apps/trends/trends/mood.py
Normal file
@ -0,0 +1,208 @@
|
||||
from collections import Counter, defaultdict
|
||||
|
||||
from django.db.models import Q
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
|
||||
def _mood_scrobbles(user, period="last_30"):
|
||||
from trends.utils import get_date_range
|
||||
|
||||
start, end = get_date_range(period)
|
||||
filters = Q(user=user, media_type=Scrobble.MediaType.MOOD)
|
||||
if start:
|
||||
filters &= Q(timestamp__gte=start)
|
||||
if end:
|
||||
filters &= Q(timestamp__lte=end)
|
||||
return Scrobble.objects.filter(filters).select_related("mood")
|
||||
|
||||
|
||||
def _parse_quality(raw):
|
||||
try:
|
||||
return int(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def _avg_quality(values):
|
||||
nums = [v for v in values if v is not None]
|
||||
if not nums:
|
||||
return 0.0
|
||||
return round(sum(nums) / len(nums), 2)
|
||||
|
||||
|
||||
def compute_mood_trajectory(user, period="last_30"):
|
||||
scrobbles = _mood_scrobbles(user, period).order_by("timestamp")
|
||||
by_date = defaultdict(list)
|
||||
for s in scrobbles:
|
||||
quality = _parse_quality(s.log.get("mood_quality"))
|
||||
if quality is not None:
|
||||
day_key = s.timestamp.strftime("%Y-%m-%d")
|
||||
by_date[day_key].append(quality)
|
||||
|
||||
trajectory = []
|
||||
for date_key in sorted(by_date):
|
||||
values = by_date[date_key]
|
||||
trajectory.append(
|
||||
{
|
||||
"date": date_key,
|
||||
"avg_quality": _avg_quality(values),
|
||||
"count": len(values),
|
||||
}
|
||||
)
|
||||
|
||||
return {"trajectory": trajectory}
|
||||
|
||||
|
||||
def compute_mood_by_time(user, period="last_30"):
|
||||
scrobbles = _mood_scrobbles(user, period)
|
||||
by_hour = defaultdict(list)
|
||||
by_day = defaultdict(list)
|
||||
|
||||
for s in scrobbles:
|
||||
quality = _parse_quality(s.log.get("mood_quality"))
|
||||
if quality is not None and s.timestamp:
|
||||
by_hour[s.timestamp.hour].append(quality)
|
||||
by_day[s.timestamp.isoweekday()].append(quality)
|
||||
|
||||
hours = []
|
||||
for h in range(24):
|
||||
vals = by_hour.get(h, [])
|
||||
hours.append(
|
||||
{
|
||||
"hour": h,
|
||||
"avg_quality": _avg_quality(vals),
|
||||
"count": len(vals),
|
||||
}
|
||||
)
|
||||
|
||||
DAY_NAMES = {
|
||||
1: "Monday",
|
||||
2: "Tuesday",
|
||||
3: "Wednesday",
|
||||
4: "Thursday",
|
||||
5: "Friday",
|
||||
6: "Saturday",
|
||||
7: "Sunday",
|
||||
}
|
||||
days = []
|
||||
for d in range(1, 8):
|
||||
vals = by_day.get(d, [])
|
||||
days.append(
|
||||
{
|
||||
"day_index": d,
|
||||
"day_name": DAY_NAMES[d],
|
||||
"avg_quality": _avg_quality(vals),
|
||||
"count": len(vals),
|
||||
}
|
||||
)
|
||||
|
||||
return {"hours": hours, "days": days}
|
||||
|
||||
|
||||
def compute_mood_distribution(user, period="last_30"):
|
||||
scrobbles = _mood_scrobbles(user, period)
|
||||
mood_counts = Counter()
|
||||
type_counts = Counter()
|
||||
|
||||
for s in scrobbles:
|
||||
if s.mood and s.mood.title:
|
||||
mood_counts[s.mood.title] += 1
|
||||
mood_type = s.log.get("mood_type")
|
||||
if mood_type:
|
||||
type_counts[mood_type] += 1
|
||||
|
||||
moods = [
|
||||
{"mood": mood, "count": count}
|
||||
for mood, count in mood_counts.most_common()
|
||||
]
|
||||
total = sum(mood_counts.values())
|
||||
|
||||
return {
|
||||
"moods": moods,
|
||||
"total": total,
|
||||
"positive_count": type_counts.get("positive", 0),
|
||||
"negative_count": type_counts.get("negative", 0),
|
||||
}
|
||||
|
||||
|
||||
def compute_mood_streaks(user, period="last_30"):
|
||||
scrobbles = list(
|
||||
_mood_scrobbles(user, period).order_by("timestamp")
|
||||
)
|
||||
if not scrobbles:
|
||||
return {"streaks": [], "current_streak": None}
|
||||
|
||||
streaks = []
|
||||
current_start = scrobbles[0].timestamp.date()
|
||||
current_type = scrobbles[0].log.get("mood_type") or "unknown"
|
||||
current_length = 1
|
||||
|
||||
for s in scrobbles[1:]:
|
||||
mood_type = s.log.get("mood_type") or "unknown"
|
||||
if mood_type == current_type:
|
||||
current_length += 1
|
||||
else:
|
||||
streaks.append(
|
||||
{
|
||||
"start_date": current_start.isoformat(),
|
||||
"end_date": scrobbles[scrobbles.index(s) - 1].timestamp.date().isoformat(),
|
||||
"mood_type": current_type,
|
||||
"length": current_length,
|
||||
}
|
||||
)
|
||||
current_start = s.timestamp.date()
|
||||
current_type = mood_type
|
||||
current_length = 1
|
||||
|
||||
streaks.append(
|
||||
{
|
||||
"start_date": current_start.isoformat(),
|
||||
"end_date": scrobbles[-1].timestamp.date().isoformat(),
|
||||
"mood_type": current_type,
|
||||
"length": current_length,
|
||||
}
|
||||
)
|
||||
|
||||
streaks.sort(key=lambda x: x["length"], reverse=True)
|
||||
|
||||
current_streak = {
|
||||
"mood_type": current_type,
|
||||
"length": current_length,
|
||||
"start_date": current_start.isoformat(),
|
||||
}
|
||||
|
||||
return {"streaks": streaks[:10], "current_streak": current_streak}
|
||||
|
||||
|
||||
def compute_mood_weather(user, period="last_30"):
|
||||
scrobbles = _mood_scrobbles(user, period)
|
||||
by_condition = defaultdict(list)
|
||||
by_temp_range = defaultdict(list)
|
||||
|
||||
for s in scrobbles:
|
||||
quality = _parse_quality(s.log.get("mood_quality"))
|
||||
if quality is None:
|
||||
continue
|
||||
desc = s.log.get("weather_description")
|
||||
temp = s.log.get("weather_temp")
|
||||
if desc:
|
||||
by_condition[desc].append(quality)
|
||||
if temp is not None:
|
||||
try:
|
||||
temp_f = float(temp)
|
||||
except (TypeError, ValueError):
|
||||
continue
|
||||
bucket = f"{(int(temp_f) // 10) * 10}-{(int(temp_f) // 10) * 10 + 9}F"
|
||||
by_temp_range[bucket].append(quality)
|
||||
|
||||
conditions = [
|
||||
{"condition": cond, "avg_quality": _avg_quality(vals), "count": len(vals)}
|
||||
for cond, vals in sorted(by_condition.items(), key=lambda x: len(x[1]), reverse=True)
|
||||
]
|
||||
|
||||
temp_ranges = [
|
||||
{"range": rng, "avg_quality": _avg_quality(vals), "count": len(vals)}
|
||||
for rng, vals in sorted(by_temp_range.items())
|
||||
]
|
||||
|
||||
return {"conditions": conditions, "temp_ranges": temp_ranges}
|
||||
89
vrobbler/apps/trends/trends/time_of_day.py
Normal file
89
vrobbler/apps/trends/trends/time_of_day.py
Normal file
@ -0,0 +1,89 @@
|
||||
from collections import OrderedDict
|
||||
|
||||
from django.db.models import Count, Q
|
||||
from django.db.models.functions import Extract
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
|
||||
TARGET_MEDIA_TYPES = ["Book", "Trail", "BirdingLocation", "BoardGame"]
|
||||
|
||||
CATEGORIES = OrderedDict(
|
||||
[
|
||||
("early_bird", {"label": "Early Bird", "hours": {5, 6, 7, 8, 9, 10}}),
|
||||
("day_jay", {"label": "Day Jay", "hours": {11, 12, 13, 14, 15, 16, 17, 18}}),
|
||||
("night_owl", {"label": "Night Owl", "hours": {19, 20, 21, 22, 23, 0, 1, 2, 3, 4}}),
|
||||
]
|
||||
)
|
||||
|
||||
|
||||
def _categorize_hour(hour):
|
||||
for slug, cat in CATEGORIES.items():
|
||||
if hour in cat["hours"]:
|
||||
return slug
|
||||
return None
|
||||
|
||||
|
||||
def compute_time_of_day_categories(user, period="last_30"):
|
||||
from trends.utils import get_date_range
|
||||
|
||||
start, end = get_date_range(period)
|
||||
filters = Q(user=user, media_type__in=TARGET_MEDIA_TYPES, timestamp__isnull=False)
|
||||
if start:
|
||||
filters &= Q(timestamp__gte=start)
|
||||
if end:
|
||||
filters &= Q(timestamp__lte=end)
|
||||
|
||||
qs = (
|
||||
Scrobble.objects.filter(filters)
|
||||
.annotate(hour=Extract("timestamp", "hour"))
|
||||
.values("media_type", "hour")
|
||||
.annotate(count=Count("id"))
|
||||
.order_by("media_type", "hour")
|
||||
)
|
||||
|
||||
raw = {}
|
||||
for row in qs:
|
||||
mt = row["media_type"]
|
||||
raw.setdefault(mt, {})[row["hour"]] = row["count"]
|
||||
|
||||
by_media_type = {}
|
||||
grand_totals = {"early_bird": 0, "day_jay": 0, "night_owl": 0}
|
||||
grand_total = 0
|
||||
|
||||
for mt in TARGET_MEDIA_TYPES:
|
||||
mt_data = raw.get(mt, {})
|
||||
cat_counts = {"early_bird": 0, "day_jay": 0, "night_owl": 0}
|
||||
mt_total = 0
|
||||
for hour, count in mt_data.items():
|
||||
slug = _categorize_hour(hour)
|
||||
if slug:
|
||||
cat_counts[slug] += count
|
||||
mt_total += count
|
||||
by_media_type[mt] = {
|
||||
"total": mt_total,
|
||||
"categories": {},
|
||||
}
|
||||
for slug in CATEGORIES:
|
||||
c = cat_counts[slug]
|
||||
by_media_type[mt]["categories"][slug] = {
|
||||
"count": c,
|
||||
"pct": round((c / mt_total * 100), 1) if mt_total else 0,
|
||||
"label": CATEGORIES[slug]["label"],
|
||||
}
|
||||
grand_totals[slug] += c
|
||||
grand_total += mt_total
|
||||
|
||||
categories = {}
|
||||
for slug in CATEGORIES:
|
||||
c = grand_totals[slug]
|
||||
categories[slug] = {
|
||||
"count": c,
|
||||
"pct": round((c / grand_total * 100), 1) if grand_total else 0,
|
||||
"label": CATEGORIES[slug]["label"],
|
||||
}
|
||||
|
||||
return {
|
||||
"categories": categories,
|
||||
"total": grand_total,
|
||||
"by_media_type": by_media_type,
|
||||
}
|
||||
@ -10,7 +10,6 @@ PERIOD_DAYS = {
|
||||
"last_30": 30,
|
||||
"last_90": 90,
|
||||
"last_year": 365,
|
||||
"all_time": None,
|
||||
}
|
||||
|
||||
PERIOD_LABELS = dict(PERIOD_CHOICES)
|
||||
@ -19,8 +18,16 @@ TIME_BOUND_TRENDS = {
|
||||
"activity-distribution",
|
||||
"concurrent-reading",
|
||||
"concurrent-listening",
|
||||
"mood-by-time",
|
||||
"mood-distribution",
|
||||
"mood-streaks",
|
||||
"mood-trajectory",
|
||||
"mood-weather",
|
||||
"peak-hours",
|
||||
"reading-pace-vs-activity",
|
||||
"time-of-day-categories",
|
||||
"trending-up",
|
||||
"weekly-rhythm",
|
||||
}
|
||||
|
||||
TREND_PERIOD_OVERRIDES = {
|
||||
@ -32,9 +39,7 @@ def get_supported_periods(trend_slug):
|
||||
if trend_slug in TREND_PERIOD_OVERRIDES:
|
||||
slugs = TREND_PERIOD_OVERRIDES[trend_slug]
|
||||
return {s: PERIOD_LABELS[s] for s in slugs}
|
||||
if trend_slug in TIME_BOUND_TRENDS:
|
||||
return dict(PERIOD_LABELS)
|
||||
return {"all_time": PERIOD_LABELS["all_time"]}
|
||||
return dict(PERIOD_LABELS)
|
||||
|
||||
|
||||
def get_period_days(period):
|
||||
@ -61,7 +66,7 @@ def get_period_nav(current_period, trend_slug):
|
||||
return prev_period, next_period
|
||||
|
||||
|
||||
def compute_and_save_trend(user, slug, period="all_time"):
|
||||
def compute_and_save_trend(user, slug, period="last_30"):
|
||||
"""Compute a single trend for a given period and persist the result.
|
||||
|
||||
Returns elapsed seconds on success, raises on failure.
|
||||
|
||||
@ -20,6 +20,31 @@ TREND_METADATA = {
|
||||
"description": "What music did you listen to while reading books?",
|
||||
"icon": "📖",
|
||||
},
|
||||
"mood-trajectory": {
|
||||
"title": "Mood Trajectory",
|
||||
"description": "How your mood quality has changed over time.",
|
||||
"icon": "📈",
|
||||
},
|
||||
"mood-by-time": {
|
||||
"title": "Mood by Time",
|
||||
"description": "How your mood varies by hour of day and day of week.",
|
||||
"icon": "🕐",
|
||||
},
|
||||
"mood-distribution": {
|
||||
"title": "Mood Distribution",
|
||||
"description": "Which moods you feel most often.",
|
||||
"icon": "🎭",
|
||||
},
|
||||
"mood-streaks": {
|
||||
"title": "Mood Streaks",
|
||||
"description": "Your longest runs of positive and negative moods.",
|
||||
"icon": "🔥",
|
||||
},
|
||||
"mood-weather": {
|
||||
"title": "Mood & Weather",
|
||||
"description": "How weather conditions correlate with your mood.",
|
||||
"icon": "🌤",
|
||||
},
|
||||
"peak-hours": {
|
||||
"title": "Peak Activity Hours",
|
||||
"description": "What time of day are you most active?",
|
||||
@ -30,6 +55,11 @@ TREND_METADATA = {
|
||||
"description": "Compare how long you read per session with and without concurrent music.",
|
||||
"icon": "📊",
|
||||
},
|
||||
"time-of-day-categories": {
|
||||
"title": "Time of Day Categories",
|
||||
"description": "Are you an early bird, day jay, or night owl? Categorized by Books, Trails, Birding Locations, and Board Games.",
|
||||
"icon": "🦉",
|
||||
},
|
||||
"trending-up": {
|
||||
"title": "Trending Media Types",
|
||||
"description": "Which media types have you been consuming more or less of recently?",
|
||||
@ -86,7 +116,7 @@ class TrendDetailView(LoginRequiredMixin, TemplateView):
|
||||
ctx["trend_not_found"] = True
|
||||
return ctx
|
||||
|
||||
period = self.request.GET.get("period", "all_time")
|
||||
period = self.request.GET.get("period", "last_30")
|
||||
|
||||
meta = TREND_METADATA.get(slug, {})
|
||||
ctx["trend"] = {
|
||||
|
||||
@ -10,26 +10,27 @@ def hrs_to_secs(hrs: float) -> int:
|
||||
return int(hrs * 60 * 60)
|
||||
|
||||
|
||||
def lookup_game_from_hltb(name_or_id: str) -> Optional[dict]:
|
||||
def lookup_game_from_hltb(name_or_id: str, search_by_title: bool = False) -> Optional[dict]:
|
||||
"""Lookup game on HowLongToBeat.com via HLtB ID or a name string and return
|
||||
the data in a dictonary mapped to our internal game fields
|
||||
|
||||
"""
|
||||
hltb_game = {}
|
||||
|
||||
try:
|
||||
hltb_id = int(name_or_id)
|
||||
except ValueError:
|
||||
hltb_id = None
|
||||
if not search_by_title:
|
||||
try:
|
||||
hltb_id = int(name_or_id)
|
||||
except ValueError:
|
||||
hltb_id = None
|
||||
|
||||
if hltb_id:
|
||||
hltb_game = HowLongToBeat().search_from_id(hltb_id)
|
||||
logger.info(f"Found game on HLtB for ID {hltb_id}")
|
||||
if hltb_id:
|
||||
hltb_game = HowLongToBeat().search_from_id(hltb_id)
|
||||
logger.info(f"Found game on HLtB for ID {hltb_id}")
|
||||
|
||||
if not hltb_game:
|
||||
results = HowLongToBeat().search(name_or_id)
|
||||
if not results:
|
||||
logger.warn(f"Lookup of game on HLtB failed for ID {name_or_id}")
|
||||
logger.warn(f"Lookup of game on HLtB failed via search {name_or_id!r}")
|
||||
return
|
||||
|
||||
hltb_game = results[0]
|
||||
|
||||
@ -19,6 +19,7 @@ GAMES_URL = "https://api.igdb.com/v4/games"
|
||||
ALT_NAMES_URL = "https://api.igdb.com/v4/alternative_names"
|
||||
SCREENSHOT_URL = "https://api.igdb.com/v4/screenshots"
|
||||
COVER_URL = "https://api.igdb.com/v4/covers"
|
||||
PLATFORMS_URL = "https://api.igdb.com/v4/platforms"
|
||||
|
||||
IGDB_CLIENT_ID = getattr(settings, "IGDB_CLIENT_ID")
|
||||
IGDB_CLIENT_SECRET = getattr(settings, "IGDB_CLIENT_SECRET")
|
||||
@ -35,6 +36,20 @@ def get_igdb_token() -> str:
|
||||
return results.get("access_token")
|
||||
|
||||
|
||||
def lookup_platform_names(platform_ids: list, headers: dict) -> list:
|
||||
"""Resolve IGDB platform IDs to platform names"""
|
||||
if not platform_ids:
|
||||
return []
|
||||
ids_str = ",".join(str(pid) for pid in platform_ids)
|
||||
body = f"fields name; where id = ({ids_str});"
|
||||
resp = requests.post(PLATFORMS_URL, data=body, headers=headers)
|
||||
if resp.status_code != 200:
|
||||
logger.warn(f"Failed to resolve platform IDs {platform_ids}")
|
||||
return []
|
||||
results = json.loads(resp.content)
|
||||
return [p["name"] for p in results if "name" in p]
|
||||
|
||||
|
||||
def lookup_game_id_from_gdb(name: str) -> str:
|
||||
|
||||
headers = {
|
||||
@ -62,9 +77,10 @@ def lookup_game_id_from_gdb(name: str) -> str:
|
||||
"details": results.get("details"),
|
||||
},
|
||||
)
|
||||
# Sort our result by IDs so we always get the lowest ID, which is likely to be the least esoteric game
|
||||
results = sorted(results, key=lambda k: k.get("game", 250000))
|
||||
return results[0].get("game", "")
|
||||
# Sort results by release date (oldest first) to prefer the original game
|
||||
results = [r for r in results if r.get("game")]
|
||||
results = sorted(results, key=lambda k: k.get("published_at") or 9999999999)
|
||||
return results[0].get("game", "") if results else ""
|
||||
|
||||
|
||||
def lookup_game_from_igdb(name_or_igdb_id: str) -> Dict:
|
||||
@ -118,6 +134,16 @@ def lookup_game_from_igdb(name_or_igdb_id: str) -> Dict:
|
||||
for genre in game.get("genres"):
|
||||
genres.append(genre["name"])
|
||||
|
||||
platforms = []
|
||||
if "release_dates" in game.keys():
|
||||
platform_ids = set()
|
||||
for rd in game["release_dates"]:
|
||||
pid = rd.get("platform")
|
||||
if pid is not None:
|
||||
platform_ids.add(pid)
|
||||
if platform_ids:
|
||||
platforms = lookup_platform_names(list(platform_ids), headers)
|
||||
|
||||
game_dict = {
|
||||
"igdb_id": game.get("id"),
|
||||
"title": game.get("name"),
|
||||
@ -129,6 +155,7 @@ def lookup_game_from_igdb(name_or_igdb_id: str) -> Dict:
|
||||
"release_date": release_date,
|
||||
"summary": game.get("summary"),
|
||||
"genres": genres,
|
||||
"platforms": platforms,
|
||||
}
|
||||
|
||||
return game_dict
|
||||
|
||||
0
vrobbler/apps/videogames/management/__init__.py
Normal file
0
vrobbler/apps/videogames/management/__init__.py
Normal file
@ -0,0 +1,529 @@
|
||||
import logging
|
||||
import time
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
MISSING_ALL = [
|
||||
"cover",
|
||||
"screenshot",
|
||||
"summary",
|
||||
"rating",
|
||||
"release_date",
|
||||
"release_year",
|
||||
"igdb_id",
|
||||
"hltb_id",
|
||||
]
|
||||
|
||||
MISSING_GROUPS = {
|
||||
"cover": lambda g: not bool(g.cover),
|
||||
"screenshot": lambda g: not bool(g.screenshot),
|
||||
"summary": lambda g: not g.summary,
|
||||
"rating": lambda g: g.rating is None,
|
||||
"release_date": lambda g: g.release_date is None,
|
||||
"release_year": lambda g: g.release_year is None,
|
||||
"igdb_id": lambda g: g.igdb_id is None,
|
||||
"hltb_id": lambda g: g.hltb_id is None,
|
||||
}
|
||||
|
||||
|
||||
def _game_matches(game, flags):
|
||||
if not flags:
|
||||
return False
|
||||
for flag in flags:
|
||||
fn = MISSING_GROUPS.get(flag)
|
||||
if fn and fn(game):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Backfill missing metadata on video games from IGDB and HowLongToBeat"
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Commit changes to the database",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--batch-size",
|
||||
type=int,
|
||||
default=100,
|
||||
help="Number of games to process per batch (default: 100)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--sleep",
|
||||
type=float,
|
||||
default=0.5,
|
||||
help="Seconds to sleep between API calls (default: 0.5)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--force",
|
||||
action="store_true",
|
||||
help="Re-fetch metadata even if data already exists",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--game-id",
|
||||
type=int,
|
||||
help="Only process a specific game by ID",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--fix-broken-images",
|
||||
action="store_true",
|
||||
help="Check and refetch broken/deleted game images (cover, screenshot, hltb_cover)",
|
||||
)
|
||||
for flag in MISSING_ALL:
|
||||
parser.add_argument(
|
||||
f"--missing-{flag}",
|
||||
dest="missing_flags",
|
||||
action="append_const",
|
||||
const=flag,
|
||||
help=f"Process games missing {flag}",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--all",
|
||||
action="store_true",
|
||||
dest="all_missing",
|
||||
help="Process games missing any metadata field",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
from videogames.models import VideoGame
|
||||
|
||||
commit = options["commit"]
|
||||
batch_size = options["batch_size"]
|
||||
sleep_secs = options["sleep"]
|
||||
force = options["force"]
|
||||
game_id = options["game_id"]
|
||||
fix_broken_images = options.get("fix_broken_images", False)
|
||||
flags = options.get("missing_flags") or []
|
||||
all_missing = options["all_missing"]
|
||||
|
||||
if all_missing:
|
||||
flags = MISSING_ALL
|
||||
|
||||
if not flags and not game_id and not force and not fix_broken_images:
|
||||
self.stdout.write(
|
||||
"No filters specified. Use --all, --missing-*, --game-id, --force, or --fix-broken-images."
|
||||
)
|
||||
return
|
||||
|
||||
if game_id:
|
||||
qs = VideoGame.objects.filter(id=game_id)
|
||||
else:
|
||||
qs = VideoGame.objects.all()
|
||||
|
||||
if flags:
|
||||
qs = [g for g in qs.iterator() if _game_matches(g, flags)]
|
||||
else:
|
||||
qs = list(qs)
|
||||
|
||||
total = len(qs)
|
||||
self.stdout.write(f"Found {total} games to process")
|
||||
|
||||
if not commit:
|
||||
self.stdout.write(
|
||||
"Dry run — no API calls will be made. Use --commit to run lookups."
|
||||
)
|
||||
return
|
||||
|
||||
title_mismatches = []
|
||||
enriched = 0
|
||||
skipped = 0
|
||||
stats = {
|
||||
"cover_fixed": 0,
|
||||
"screenshot_fixed": 0,
|
||||
"summary_fixed": 0,
|
||||
"rating_fixed": 0,
|
||||
"release_date_fixed": 0,
|
||||
"release_year_fixed": 0,
|
||||
"igdb_id_found": 0,
|
||||
"hltb_id_found": 0,
|
||||
"images_fixed": 0,
|
||||
}
|
||||
|
||||
enriched_any = bool(flags or game_id or force)
|
||||
|
||||
if enriched_any:
|
||||
for batch_num, offset in enumerate(range(0, len(qs), batch_size)):
|
||||
batch = qs[offset : offset + batch_size]
|
||||
for game in batch:
|
||||
result = self._enrich_game(game, sleep_secs, force)
|
||||
self._check_retroarch_name(game, title_mismatches)
|
||||
if result:
|
||||
enriched += 1
|
||||
for key in stats:
|
||||
if result.get(key):
|
||||
stats[key] += 1
|
||||
else:
|
||||
skipped += 1
|
||||
|
||||
self.stdout.write(
|
||||
f" Batch {batch_num + 1}: {offset + len(batch)}/{total} — "
|
||||
f"enriched: {enriched}, skipped: {skipped}"
|
||||
)
|
||||
|
||||
if fix_broken_images:
|
||||
broken_stats = self._fix_broken_images(qs, sleep_secs)
|
||||
stats["images_fixed"] = broken_stats["images_fixed"]
|
||||
|
||||
self.stdout.write(
|
||||
f"\nResults (commit={commit}):\n"
|
||||
f" Games enriched: {enriched}\n"
|
||||
f" Games skipped: {skipped}\n"
|
||||
f" Covers fixed: {stats['cover_fixed']}\n"
|
||||
f" Screenshots fixed: {stats['screenshot_fixed']}\n"
|
||||
f" Summaries fixed: {stats['summary_fixed']}\n"
|
||||
f" Ratings fixed: {stats['rating_fixed']}\n"
|
||||
f" Release dates fixed: {stats['release_date_fixed']}\n"
|
||||
f" Release years fixed: {stats['release_year_fixed']}\n"
|
||||
f" IGDB IDs found: {stats['igdb_id_found']}\n"
|
||||
f" HLtB IDs found: {stats['hltb_id_found']}"
|
||||
)
|
||||
if fix_broken_images:
|
||||
self.stdout.write(f" Broken images fixed: {stats['images_fixed']}")
|
||||
|
||||
if title_mismatches:
|
||||
self.stdout.write("\nTitle vs retroarch_name mismatches (not auto-fixed):")
|
||||
for retroarch_name, title, game_id in title_mismatches:
|
||||
self.stdout.write(
|
||||
f" Game #{game_id}: retroarch_name={retroarch_name!r} vs title={title!r}"
|
||||
)
|
||||
|
||||
def _clean_retroarch_name(self, name):
|
||||
if not name:
|
||||
return ""
|
||||
name = name.strip()
|
||||
if "(" in name:
|
||||
name = name.split("(")[0].strip()
|
||||
return name
|
||||
|
||||
def _check_retroarch_name(self, game, mismatches):
|
||||
if not game.retroarch_name:
|
||||
return
|
||||
cleaned = self._clean_retroarch_name(game.retroarch_name)
|
||||
if cleaned.lower() != game.title.lower():
|
||||
mismatches.append((game.retroarch_name, game.title, game.id))
|
||||
if "retroarch-mismatch" not in game.tags.names():
|
||||
game.tags.add("retroarch-mismatch")
|
||||
self.stdout.write(
|
||||
f" [TAG] {game} — tagged as retroarch-mismatch"
|
||||
)
|
||||
|
||||
def _enrich_game(self, game, sleep_secs, force):
|
||||
from videogames.igdb import lookup_game_id_from_gdb, lookup_game_from_igdb
|
||||
from videogames.howlongtobeat import lookup_game_from_hltb
|
||||
|
||||
search_name = self._clean_retroarch_name(game.retroarch_name) or game.title
|
||||
|
||||
changed = {}
|
||||
|
||||
if not game.hltb_id:
|
||||
hltb_data = None
|
||||
if search_name:
|
||||
hltb_data = lookup_game_from_hltb(search_name, search_by_title=True)
|
||||
time.sleep(sleep_secs)
|
||||
if not hltb_data and game.title and game.title != search_name:
|
||||
hltb_data = lookup_game_from_hltb(game.title, search_by_title=True)
|
||||
time.sleep(sleep_secs)
|
||||
if hltb_data:
|
||||
result = self._apply_hltb_data(game, hltb_data, force)
|
||||
if result:
|
||||
changed.update(result)
|
||||
|
||||
igdb_data = None
|
||||
if not game.igdb_id and search_name:
|
||||
igdb_id = lookup_game_id_from_gdb(search_name)
|
||||
time.sleep(sleep_secs)
|
||||
if igdb_id:
|
||||
igdb_data = lookup_game_from_igdb(str(igdb_id))
|
||||
time.sleep(sleep_secs)
|
||||
elif game.igdb_id:
|
||||
igdb_data = lookup_game_from_igdb(str(game.igdb_id))
|
||||
time.sleep(sleep_secs)
|
||||
|
||||
if igdb_data:
|
||||
igdb_title = igdb_data.get("title", "")
|
||||
igdb_title_clean = self._clean_retroarch_name(igdb_title)
|
||||
if igdb_title_clean.lower() == search_name.lower():
|
||||
if game.igdb_id is None and igdb_data.get("igdb_id"):
|
||||
game.igdb_id = int(igdb_data["igdb_id"])
|
||||
game.save(update_fields=["igdb_id"])
|
||||
changed["igdb_id_found"] = True
|
||||
self.stdout.write(f" [IGDB_ID] {game} — found IGDB ID {game.igdb_id}")
|
||||
result = self._apply_igdb_data(game, igdb_data, force)
|
||||
if result:
|
||||
changed.update(result)
|
||||
else:
|
||||
self.stdout.write(
|
||||
f" [IGDB] {game} — title mismatch (IGDB: {igdb_title!r} vs expected: {search_name!r}), re-searching…"
|
||||
)
|
||||
resolved = False
|
||||
for candidate in (search_name, game.title if game.title != search_name else None):
|
||||
if not candidate:
|
||||
continue
|
||||
new_id = lookup_game_id_from_gdb(candidate)
|
||||
time.sleep(sleep_secs)
|
||||
if not new_id:
|
||||
continue
|
||||
new_data = lookup_game_from_igdb(str(new_id))
|
||||
time.sleep(sleep_secs)
|
||||
if not new_data:
|
||||
continue
|
||||
new_title = new_data.get("title", "")
|
||||
new_title_clean = self._clean_retroarch_name(new_title)
|
||||
if new_title_clean.lower() == candidate.lower():
|
||||
game.igdb_id = int(new_id)
|
||||
if new_title and new_title != game.title:
|
||||
game.title = new_title
|
||||
changed["title_updated"] = True
|
||||
self.stdout.write(f" [TITLE] {game} — updated title to {new_title!r} from IGDB")
|
||||
game.save(update_fields=["igdb_id"] + (["title"] if changed.get("title_updated") else []))
|
||||
changed["igdb_id_found"] = True
|
||||
self.stdout.write(f" [IGDB_ID] {game} — re-found IGDB ID {game.igdb_id}")
|
||||
if "igdb-mismatch" in game.tags.names():
|
||||
game.tags.remove("igdb-mismatch")
|
||||
self.stdout.write(f" [TAG] {game} — removed igdb-mismatch tag")
|
||||
result = self._apply_igdb_data(game, new_data, force)
|
||||
if result:
|
||||
changed.update(result)
|
||||
resolved = True
|
||||
break
|
||||
|
||||
if not resolved and "igdb-mismatch" not in game.tags.names():
|
||||
game.tags.add("igdb-mismatch")
|
||||
self.stdout.write(f" [TAG] {game} — tagged igdb-mismatch")
|
||||
|
||||
# If retroarch-mismatch tag exists but no longer applies, remove it
|
||||
if "retroarch-mismatch" in game.tags.names():
|
||||
cleaned = self._clean_retroarch_name(game.retroarch_name or "")
|
||||
if cleaned.lower() == game.title.lower():
|
||||
game.tags.remove("retroarch-mismatch")
|
||||
self.stdout.write(f" [TAG] {game} — removed retroarch-mismatch tag (title now matches)")
|
||||
|
||||
return changed if changed else None
|
||||
|
||||
def _apply_igdb_data(self, game, data, force):
|
||||
from django.core.files.base import ContentFile
|
||||
|
||||
import requests
|
||||
|
||||
changed = {
|
||||
"cover_fixed": False,
|
||||
"screenshot_fixed": False,
|
||||
"summary_fixed": False,
|
||||
"rating_fixed": False,
|
||||
"release_date_fixed": False,
|
||||
}
|
||||
|
||||
update_fields = []
|
||||
|
||||
if data.get("alternative_name") and not game.alternative_name:
|
||||
game.alternative_name = data["alternative_name"]
|
||||
update_fields.append("alternative_name")
|
||||
|
||||
if data.get("summary") and (not game.summary or force):
|
||||
game.summary = data["summary"]
|
||||
update_fields.append("summary")
|
||||
changed["summary_fixed"] = True
|
||||
|
||||
if data.get("rating") is not None and (game.rating is None or force):
|
||||
game.rating = data["rating"]
|
||||
update_fields.append("rating")
|
||||
changed["rating_fixed"] = True
|
||||
|
||||
if data.get("rating_count") is not None and (game.rating_count is None or force):
|
||||
game.rating_count = data["rating_count"]
|
||||
update_fields.append("rating_count")
|
||||
|
||||
if data.get("release_date") and (game.release_date is None or force):
|
||||
game.release_date = data["release_date"]
|
||||
update_fields.append("release_date")
|
||||
changed["release_date_fixed"] = True
|
||||
|
||||
if update_fields:
|
||||
game.save(update_fields=update_fields)
|
||||
self.stdout.write(f" [IGDB] {game} — {', '.join(update_fields)}")
|
||||
|
||||
cover_url = data.get("cover_url")
|
||||
if cover_url:
|
||||
r = requests.get(cover_url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
game.cover.save(fname, ContentFile(r.content), save=True)
|
||||
changed["cover_fixed"] = True
|
||||
self.stdout.write(f" [COVER] {game} — cover saved from IGDB")
|
||||
|
||||
screenshot_url = data.get("screenshot_url")
|
||||
if screenshot_url:
|
||||
r = requests.get(screenshot_url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
game.screenshot.save(fname, ContentFile(r.content), save=True)
|
||||
changed["screenshot_fixed"] = True
|
||||
self.stdout.write(f" [SCREENSHOT] {game} — screenshot saved from IGDB")
|
||||
|
||||
genres = data.get("genres", [])
|
||||
if genres:
|
||||
existing = set(game.genre.names())
|
||||
new_genres = [g for g in genres if g not in existing]
|
||||
if new_genres:
|
||||
game.genre.add(*new_genres)
|
||||
self.stdout.write(f" [GENRES] {game} — added {len(new_genres)} genres")
|
||||
|
||||
platforms = data.get("platforms", [])
|
||||
if platforms:
|
||||
existing = set(game.platforms.values_list("name", flat=True))
|
||||
new_platforms = [p for p in platforms if p not in existing]
|
||||
if new_platforms:
|
||||
from videogames.models import VideoGamePlatform
|
||||
|
||||
for name in new_platforms:
|
||||
p, _ = VideoGamePlatform.objects.get_or_create(name=name)
|
||||
game.platforms.add(p)
|
||||
self.stdout.write(f" [PLATFORMS] {game} — added {len(new_platforms)} platforms")
|
||||
|
||||
if "igdb-enriched" not in game.tags.names():
|
||||
game.tags.add("igdb-enriched")
|
||||
self.stdout.write(f" [TAG] {game} — tagged igdb-enriched")
|
||||
|
||||
return changed if any(changed.values()) else None
|
||||
|
||||
def _apply_hltb_data(self, game, data, force):
|
||||
from django.core.files.base import ContentFile
|
||||
|
||||
import requests
|
||||
|
||||
changed = {
|
||||
"hltb_id_found": False,
|
||||
"release_year_fixed": False,
|
||||
}
|
||||
|
||||
update_fields = []
|
||||
|
||||
hltb_title = data.get("title", "")
|
||||
if hltb_title and hltb_title != game.title:
|
||||
game.title = hltb_title
|
||||
update_fields.append("title")
|
||||
self.stdout.write(f" [TITLE] {game} — updated title to {hltb_title!r}")
|
||||
|
||||
if data.get("hltb_id") and (game.hltb_id is None or force):
|
||||
game.hltb_id = data["hltb_id"]
|
||||
update_fields.append("hltb_id")
|
||||
changed["hltb_id_found"] = True
|
||||
self.stdout.write(f" [HLTB_ID] {game} — found HLtB ID {data['hltb_id']}")
|
||||
|
||||
if data.get("release_year") and (game.release_year is None or force):
|
||||
game.release_year = data["release_year"]
|
||||
update_fields.append("release_year")
|
||||
changed["release_year_fixed"] = True
|
||||
|
||||
if data.get("main_story_time") and (game.main_story_time is None or force):
|
||||
game.main_story_time = data["main_story_time"]
|
||||
update_fields.append("main_story_time")
|
||||
|
||||
if data.get("main_extra_time") and (game.main_extra_time is None or force):
|
||||
game.main_extra_time = data["main_extra_time"]
|
||||
update_fields.append("main_extra_time")
|
||||
|
||||
if data.get("completionist_time") and (game.completionist_time is None or force):
|
||||
game.completionist_time = data["completionist_time"]
|
||||
update_fields.append("completionist_time")
|
||||
|
||||
if data.get("hltb_score") is not None and (game.hltb_score is None or force):
|
||||
game.hltb_score = data["hltb_score"]
|
||||
update_fields.append("hltb_score")
|
||||
|
||||
if update_fields:
|
||||
game.save(update_fields=update_fields)
|
||||
self.stdout.write(f" [HLTB] {game} — {', '.join(update_fields)}")
|
||||
|
||||
cover_url = data.get("cover_url")
|
||||
if cover_url:
|
||||
headers = {"User-Agent": "Vrobbler 0.11.12"}
|
||||
r = requests.get(cover_url, headers=headers)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_cover_{game.uuid}.jpg"
|
||||
game.hltb_cover.save(fname, ContentFile(r.content), save=True)
|
||||
self.stdout.write(f" [HLTB_COVER] {game} — cover saved from HLtB")
|
||||
|
||||
platforms = data.get("platforms", [])
|
||||
if platforms:
|
||||
existing = set(game.platforms.values_list("name", flat=True))
|
||||
new_platforms = [p for p in platforms if p not in existing]
|
||||
if new_platforms:
|
||||
from videogames.models import VideoGamePlatform
|
||||
|
||||
for name in new_platforms:
|
||||
p, _ = VideoGamePlatform.objects.get_or_create(name=name)
|
||||
game.platforms.add(p)
|
||||
self.stdout.write(f" [PLATFORMS] {game} — added {len(new_platforms)} platforms")
|
||||
|
||||
if "hltb-enriched" not in game.tags.names():
|
||||
game.tags.add("hltb-enriched")
|
||||
self.stdout.write(f" [TAG] {game} — tagged hltb-enriched")
|
||||
|
||||
return changed if any(changed.values()) else None
|
||||
|
||||
def _fix_broken_images(self, games, sleep_secs):
|
||||
from django.core.files.base import ContentFile
|
||||
|
||||
import requests
|
||||
|
||||
from videogames.igdb import lookup_game_from_igdb
|
||||
from videogames.howlongtobeat import lookup_game_from_hltb
|
||||
|
||||
stats = {"cover_fixed": 0, "screenshot_fixed": 0, "images_fixed": 0}
|
||||
|
||||
for game in games:
|
||||
for field_name, source in [
|
||||
("cover", "igdb"),
|
||||
("screenshot", "igdb"),
|
||||
("hltb_cover", "hltb"),
|
||||
]:
|
||||
field = getattr(game, field_name)
|
||||
if not field.name:
|
||||
continue
|
||||
if field.storage.exists(field.name):
|
||||
continue
|
||||
|
||||
self.stdout.write(
|
||||
f" [IMAGE] {game} — {field_name} is broken (file missing), refetching…"
|
||||
)
|
||||
|
||||
if source == "igdb" and game.igdb_id:
|
||||
data = lookup_game_from_igdb(str(game.igdb_id))
|
||||
time.sleep(sleep_secs)
|
||||
if not data:
|
||||
continue
|
||||
url = data.get("cover_url" if field_name == "cover" else "screenshot_url")
|
||||
if not url:
|
||||
continue
|
||||
r = requests.get(url)
|
||||
if r.status_code != 200:
|
||||
continue
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
getattr(game, field_name).save(fname, ContentFile(r.content), save=True)
|
||||
stats["images_fixed"] += 1
|
||||
self.stdout.write(f" [IMAGE] {game} — {field_name} refetched from IGDB")
|
||||
|
||||
elif source == "hltb" and game.hltb_id:
|
||||
data = lookup_game_from_hltb(str(game.hltb_id))
|
||||
time.sleep(sleep_secs)
|
||||
if not data:
|
||||
continue
|
||||
url = data.get("cover_url")
|
||||
if not url:
|
||||
continue
|
||||
headers = {"User-Agent": "Vrobbler 0.11.12"}
|
||||
r = requests.get(url, headers=headers)
|
||||
if r.status_code != 200:
|
||||
continue
|
||||
fname = f"{game.title}_cover_{game.uuid}.jpg"
|
||||
game.hltb_cover.save(fname, ContentFile(r.content), save=True)
|
||||
stats["images_fixed"] += 1
|
||||
self.stdout.write(f" [IMAGE] {game} — hltb_cover refetched from HLtB")
|
||||
|
||||
return stats
|
||||
@ -215,12 +215,16 @@ class VideoGame(LongPlayScrobblableMixin):
|
||||
def fix_metadata(self, force_update: bool = False):
|
||||
from videogames.utils import (
|
||||
get_or_create_videogame,
|
||||
load_game_data_from_hltb,
|
||||
load_game_data_from_igdb,
|
||||
)
|
||||
|
||||
if self.hltb_id and force_update:
|
||||
get_or_create_videogame(str(self.hltb_id), force_update)
|
||||
|
||||
if not self.hltb_id:
|
||||
load_game_data_from_hltb(self.id)
|
||||
|
||||
if not self.igdb_id:
|
||||
# This almost never works without intervention
|
||||
self.igdb_id = lookup_game_id_from_gdb(self.title)
|
||||
|
||||
@ -153,6 +153,13 @@ def import_retroarch_lrtl_files(playlog_path: str, user_id: int) -> List[dict]:
|
||||
continue
|
||||
|
||||
logger.info(f"Queued scrobble for game {found_game.id}")
|
||||
|
||||
log_data = {"emulated": True}
|
||||
if last_scrobble and last_scrobble.log:
|
||||
prev = last_scrobble.log
|
||||
if prev.get("emulator"):
|
||||
log_data["emulator"] = prev["emulator"]
|
||||
|
||||
new_scrobbles.append(
|
||||
Scrobble(
|
||||
video_game_id=found_game.id,
|
||||
@ -168,6 +175,7 @@ def import_retroarch_lrtl_files(playlog_path: str, user_id: int) -> List[dict]:
|
||||
user_id=user_id,
|
||||
source="Retroarch",
|
||||
media_type=Scrobble.MediaType.VIDEO_GAME,
|
||||
log=log_data,
|
||||
)
|
||||
)
|
||||
created_scrobbles = Scrobble.objects.bulk_create(new_scrobbles)
|
||||
|
||||
@ -15,13 +15,11 @@ def natural_duration(value):
|
||||
seconds = remainder % 60
|
||||
parts = []
|
||||
if days:
|
||||
parts.append(f"{days} day{'s' if days != 1 else ''}")
|
||||
parts.append(f"{days} d")
|
||||
if hours:
|
||||
parts.append(f"{hours} hour{'s' if hours != 1 else ''}")
|
||||
parts.append(f"{hours} hr")
|
||||
if minutes:
|
||||
parts.append(f"{minutes} minute{'s' if minutes != 1 else ''}")
|
||||
parts.append(f"{minutes} min")
|
||||
if seconds or not parts:
|
||||
parts.append(f"{seconds} second{'s' if seconds != 1 else ''}")
|
||||
if len(parts) == 1:
|
||||
return parts[0]
|
||||
return ", ".join(parts[:-1]) + " and " + parts[-1]
|
||||
parts.append(f"{seconds} sec")
|
||||
return ", ".join(parts)
|
||||
|
||||
@ -7,8 +7,6 @@ from videogames.howlongtobeat import lookup_game_from_hltb
|
||||
from videogames.igdb import lookup_game_from_igdb
|
||||
from videogames.models import VideoGame, VideoGamePlatform
|
||||
|
||||
from vrobbler.apps.videogames.exceptions import GameNotFound
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@ -16,22 +14,33 @@ def get_or_create_videogame(
|
||||
name_or_id: str,
|
||||
force_update: bool = False,
|
||||
) -> Optional[VideoGame]:
|
||||
"""Look up game by name or ID from HowLongToBeat"""
|
||||
"""Look up game by name or ID from HowLongToBeat, then enrich with IGDB"""
|
||||
|
||||
game_dict = lookup_game_from_hltb(name_or_id)
|
||||
hltb_data = lookup_game_from_hltb(name_or_id)
|
||||
|
||||
if not game_dict:
|
||||
game_dict = lookup_game_from_igdb(name_or_id)
|
||||
if hltb_data:
|
||||
game = _create_update_from_dict(hltb_data, force_update)
|
||||
else:
|
||||
igdb_data = lookup_game_from_igdb(name_or_id)
|
||||
if igdb_data:
|
||||
game = _create_update_from_dict(igdb_data, force_update)
|
||||
else:
|
||||
return None
|
||||
|
||||
if not game_dict:
|
||||
return
|
||||
if game:
|
||||
game.fix_metadata()
|
||||
return game
|
||||
|
||||
|
||||
def _create_update_from_dict(
|
||||
game_dict: dict, force_update: bool = False
|
||||
) -> Optional[VideoGame]:
|
||||
|
||||
# Create missing platforms and prep for loading after create
|
||||
platform_ids = []
|
||||
if "platforms" in game_dict.keys():
|
||||
platforms = game_dict.get("platforms", [])
|
||||
if platforms:
|
||||
for platform in game_dict.get("platforms", []):
|
||||
for platform in platforms:
|
||||
p, _created = VideoGamePlatform.objects.get_or_create(name=platform)
|
||||
platform_ids.append(p.id)
|
||||
game_dict.pop("platforms")
|
||||
@ -48,7 +57,7 @@ def get_or_create_videogame(
|
||||
|
||||
title = game_dict.get("title")
|
||||
if not title:
|
||||
raise GameNotFound(name_or_id)
|
||||
return None
|
||||
|
||||
hltb_id = game_dict.get("hltb_id")
|
||||
igdb_id = game_dict.get("igdb_id")
|
||||
@ -69,21 +78,19 @@ def get_or_create_videogame(
|
||||
VideoGame.objects.filter(pk=game.id).update(**game_dict)
|
||||
game.refresh_from_db()
|
||||
|
||||
# Associate plaforms
|
||||
if platform_ids:
|
||||
game.platforms.add(*platform_ids)
|
||||
|
||||
if genres:
|
||||
game.genre.add(*genres)
|
||||
|
||||
if not game.screenshot and screenshot_url:
|
||||
if screenshot_url:
|
||||
r = requests.get(screenshot_url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
game.screenshot.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
# Go get cover image if the URL is present
|
||||
if cover_url and not game.hltb_cover:
|
||||
if cover_url:
|
||||
headers = {"User-Agent": "Vrobbler 0.11.12"}
|
||||
r = requests.get(cover_url, headers=headers)
|
||||
logger.debug(r.status_code)
|
||||
@ -91,12 +98,89 @@ def get_or_create_videogame(
|
||||
fname = f"{game.title}_cover_{game.uuid}.jpg"
|
||||
game.hltb_cover.save(fname, ContentFile(r.content), save=True)
|
||||
logger.debug("Loaded cover image from HLtB")
|
||||
game.fix_metadata()
|
||||
|
||||
tag = "hltb-enriched" if hltb_id else "igdb-enriched"
|
||||
if tag not in game.tags.names():
|
||||
game.tags.add(tag)
|
||||
logger.info(f"Game {game} tagged {tag}")
|
||||
|
||||
return game
|
||||
|
||||
|
||||
def load_game_data_from_igdb(game_id: int, igdb_id: str = "") -> Optional[VideoGame]:
|
||||
def load_game_data_from_hltb(
|
||||
game_id: int, expected_title: str = ""
|
||||
) -> Optional[VideoGame]:
|
||||
"""Look up HLtB data for an existing game and apply it"""
|
||||
game = VideoGame.objects.filter(id=game_id).first()
|
||||
if not game:
|
||||
logger.warn(f"Video game with ID {game_id} not found")
|
||||
return
|
||||
|
||||
logger.info(f"Looking up HLtB data for {game}")
|
||||
hltb_data = lookup_game_from_hltb(game.title, search_by_title=True)
|
||||
if not hltb_data:
|
||||
logger.warn(f"No HLtB data found for {game}")
|
||||
return
|
||||
|
||||
update_fields = []
|
||||
|
||||
hltb_title = hltb_data.get("title", "")
|
||||
if hltb_title and hltb_title != game.title:
|
||||
game.title = hltb_title
|
||||
update_fields.append("title")
|
||||
logger.info(f"Game {game.id} title updated to {hltb_title!r}")
|
||||
|
||||
if hltb_data.get("hltb_id") and (game.hltb_id is None):
|
||||
game.hltb_id = hltb_data["hltb_id"]
|
||||
update_fields.append("hltb_id")
|
||||
|
||||
if hltb_data.get("release_year") and (game.release_year is None):
|
||||
game.release_year = hltb_data["release_year"]
|
||||
update_fields.append("release_year")
|
||||
|
||||
if hltb_data.get("main_story_time") and (game.main_story_time is None):
|
||||
game.main_story_time = hltb_data["main_story_time"]
|
||||
update_fields.append("main_story_time")
|
||||
|
||||
if hltb_data.get("main_extra_time") and (game.main_extra_time is None):
|
||||
game.main_extra_time = hltb_data["main_extra_time"]
|
||||
update_fields.append("main_extra_time")
|
||||
|
||||
if hltb_data.get("completionist_time") and (game.completionist_time is None):
|
||||
game.completionist_time = hltb_data["completionist_time"]
|
||||
update_fields.append("completionist_time")
|
||||
|
||||
if hltb_data.get("hltb_score") is not None and (game.hltb_score is None):
|
||||
game.hltb_score = hltb_data["hltb_score"]
|
||||
update_fields.append("hltb_score")
|
||||
|
||||
if update_fields:
|
||||
game.save(update_fields=update_fields)
|
||||
|
||||
platforms = hltb_data.get("platforms", [])
|
||||
if platforms:
|
||||
for name in platforms:
|
||||
p, _ = VideoGamePlatform.objects.get_or_create(name=name)
|
||||
game.platforms.add(p)
|
||||
|
||||
cover_url = hltb_data.get("cover_url")
|
||||
if cover_url:
|
||||
headers = {"User-Agent": "Vrobbler 0.11.12"}
|
||||
r = requests.get(cover_url, headers=headers)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_cover_{game.uuid}.jpg"
|
||||
game.hltb_cover.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
if "hltb-enriched" not in game.tags.names():
|
||||
game.tags.add("hltb-enriched")
|
||||
logger.info(f"Game {game} tagged hltb-enriched")
|
||||
|
||||
return game
|
||||
|
||||
|
||||
def load_game_data_from_igdb(
|
||||
game_id: int, igdb_id: str = "", expected_title: str = ""
|
||||
) -> Optional[VideoGame]:
|
||||
"""Look up game, if it doesn't exist, lookup data from igdb"""
|
||||
game = VideoGame.objects.filter(id=game_id).first()
|
||||
if not game:
|
||||
@ -116,25 +200,68 @@ def load_game_data_from_igdb(game_id: int, igdb_id: str = "") -> Optional[VideoG
|
||||
logger.warn(f"No game data found on IGDB for ID {igdb_id}")
|
||||
return
|
||||
|
||||
igdb_title = game_dict.get("title", "")
|
||||
igdb_title_clean = igdb_title.split(" (")[0].strip() if " (" in igdb_title else igdb_title
|
||||
expected = expected_title or game.title
|
||||
if igdb_title_clean.lower() != expected.lower():
|
||||
logger.info(
|
||||
f"IGDB title {igdb_title!r} doesn't match expected {expected!r} for {game} — re-searching…"
|
||||
)
|
||||
from videogames.igdb import lookup_game_id_from_gdb
|
||||
|
||||
new_id = lookup_game_id_from_gdb(expected)
|
||||
if new_id:
|
||||
new_data = lookup_game_from_igdb(str(new_id))
|
||||
if new_data:
|
||||
new_title = new_data.get("title", "")
|
||||
new_title_clean = new_title.split(" (")[0].strip() if " (" in new_title else new_title
|
||||
if new_title_clean.lower() == expected.lower():
|
||||
game_dict = new_data
|
||||
igdb_id = int(new_id)
|
||||
if game.igdb_id != igdb_id:
|
||||
game.igdb_id = igdb_id
|
||||
game.save(update_fields=["igdb_id"])
|
||||
logger.info(f"Game {game} IGDB ID updated to {igdb_id}")
|
||||
|
||||
igdb_title = game_dict.get("title", "")
|
||||
igdb_title_clean = igdb_title.split(" (")[0].strip() if " (" in igdb_title else igdb_title
|
||||
if igdb_title_clean.lower() != expected.lower():
|
||||
if "igdb-mismatch" not in game.tags.names():
|
||||
game.tags.add("igdb-mismatch")
|
||||
logger.info(
|
||||
f"Game {game} tagged igdb-mismatch (IGDB: {igdb_title!r} vs expected: {expected!r})"
|
||||
)
|
||||
return
|
||||
|
||||
screenshot_url = game_dict.pop("screenshot_url")
|
||||
cover_url = game_dict.pop("cover_url")
|
||||
genres = game_dict.pop("genres")
|
||||
platforms = game_dict.pop("platforms", [])
|
||||
|
||||
VideoGame.objects.filter(pk=game.id).update(**game_dict)
|
||||
game.refresh_from_db()
|
||||
|
||||
game.genre.add(*genres)
|
||||
|
||||
if not game.screenshot and screenshot_url:
|
||||
if platforms:
|
||||
for name in platforms:
|
||||
p, _ = VideoGamePlatform.objects.get_or_create(name=name)
|
||||
game.platforms.add(p)
|
||||
|
||||
if screenshot_url:
|
||||
r = requests.get(screenshot_url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
game.screenshot.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
if not game.cover and cover_url:
|
||||
if cover_url:
|
||||
r = requests.get(cover_url)
|
||||
if r.status_code == 200:
|
||||
fname = f"{game.title}_{game.uuid}.jpg"
|
||||
game.cover.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
if "igdb-enriched" not in game.tags.names():
|
||||
game.tags.add("igdb-enriched")
|
||||
logger.info(f"Game {game} tagged igdb-enriched")
|
||||
|
||||
return game
|
||||
|
||||
@ -1,8 +1,24 @@
|
||||
from django.core.cache import cache
|
||||
|
||||
from videos.models import Video, Series
|
||||
|
||||
CACHE_TTL = 300
|
||||
|
||||
|
||||
def video_lists(request):
|
||||
movie_list = cache.get("video_lists_movie_list")
|
||||
if movie_list is None:
|
||||
movie_list = list(
|
||||
Video.objects.filter(video_type=Video.VideoType.MOVIE).only("id", "title")
|
||||
)
|
||||
cache.set("video_lists_movie_list", movie_list, CACHE_TTL)
|
||||
|
||||
series_list = cache.get("video_lists_series_list")
|
||||
if series_list is None:
|
||||
series_list = list(Series.objects.all().only("id", "name"))
|
||||
cache.set("video_lists_series_list", series_list, CACHE_TTL)
|
||||
|
||||
return {
|
||||
"movie_list": Video.objects.filter(video_type=Video.VideoType.MOVIE),
|
||||
"series_list": Series.objects.all(),
|
||||
"movie_list": movie_list,
|
||||
"series_list": series_list,
|
||||
}
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
import logging
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
from django.db import transaction
|
||||
from django.db import models, transaction
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
@ -25,6 +25,11 @@ class Command(BaseCommand):
|
||||
type=str,
|
||||
help="Only process series with this imdb_id",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--needs-metadata",
|
||||
action="store_true",
|
||||
help="Only process series missing imdb_id or cover image",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
from videos.models import Series
|
||||
@ -32,18 +37,30 @@ class Command(BaseCommand):
|
||||
force = options["force"]
|
||||
dry_run = options["dry_run"]
|
||||
imdb_id = options["imdb_id"]
|
||||
needs_metadata = options["needs_metadata"]
|
||||
|
||||
qs = Series.objects.all()
|
||||
if imdb_id:
|
||||
qs = qs.filter(imdb_id=imdb_id)
|
||||
if needs_metadata:
|
||||
qs = qs.filter(
|
||||
models.Q(imdb_id__isnull=True)
|
||||
| models.Q(imdb_id="")
|
||||
| models.Q(cover_image__isnull=True)
|
||||
| models.Q(cover_image="")
|
||||
)
|
||||
|
||||
total = qs.count()
|
||||
self.stdout.write(f"Processing {total} series")
|
||||
|
||||
if dry_run:
|
||||
for series in qs.iterator():
|
||||
has_imdb = bool(series.imdb_id)
|
||||
has_image = bool(series.cover_image)
|
||||
self.stdout.write(
|
||||
f" [DRY RUN] Would fix {series.name} (imdb_id={series.imdb_id})"
|
||||
f" [DRY RUN] Would fix {series.name}"
|
||||
f" (imdb_id={'✓' if has_imdb else '✗'}"
|
||||
f", image={'✓' if has_image else '✗'})"
|
||||
)
|
||||
return
|
||||
|
||||
|
||||
@ -321,13 +321,27 @@ class Series(TimeStampedModel):
|
||||
return not last_scrobble.played_to_completion
|
||||
|
||||
def fix_metadata(self, force_update=False):
|
||||
name_or_id = self.name
|
||||
if self.imdb_id:
|
||||
name_or_id = self.imdb_id
|
||||
video_metadata: VideoMetadata = lookup_video_from_tmdb(name_or_id)
|
||||
from tmdbv3api import TV
|
||||
|
||||
if not video_metadata.title:
|
||||
logger.warning(f"No imdb data for {self}")
|
||||
if not self.imdb_id:
|
||||
tv = TV()
|
||||
results = tv.search(self.name)
|
||||
if results:
|
||||
show_id = results[0].id
|
||||
external_ids = tv.external_ids(show_id)
|
||||
if external_ids and external_ids.imdb_id:
|
||||
self.imdb_id = external_ids.imdb_id
|
||||
self.save(update_fields=["imdb_id"])
|
||||
else:
|
||||
logger.warning(f"No IMDB ID found on TMDB for {self}")
|
||||
return
|
||||
else:
|
||||
logger.warning(f"No results on TMDB for {self.name}")
|
||||
return
|
||||
|
||||
video_metadata = lookup_video_from_tmdb(self.imdb_id)
|
||||
if not video_metadata or not video_metadata.title:
|
||||
logger.warning(f"No metadata for {self}")
|
||||
return
|
||||
|
||||
if video_metadata.cover_url and (not self.cover_image or force_update):
|
||||
@ -336,8 +350,8 @@ class Series(TimeStampedModel):
|
||||
fname = f"{self.name}_{self.uuid}.jpg"
|
||||
self.cover_image.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
self.plot = video_metadata.plot
|
||||
self.imdb_rating = video_metadata.imdb_rating
|
||||
self.plot = video_metadata.plot or ""
|
||||
self.imdb_rating = getattr(video_metadata, "imdb_rating", None)
|
||||
self.save()
|
||||
|
||||
if video_metadata.genres:
|
||||
|
||||
@ -1,4 +1,5 @@
|
||||
import logging
|
||||
import os
|
||||
|
||||
import pendulum
|
||||
from django.conf import settings
|
||||
@ -8,6 +9,8 @@ from videos.metadata import VideoMetadata, VideoType
|
||||
|
||||
TMDB_KEY = getattr(settings, "TMDB_API_KEY", "")
|
||||
|
||||
os.environ.setdefault("TMDB_API_KEY", TMDB_KEY)
|
||||
|
||||
tmdb = TMDb(key=TMDB_KEY, language="en-US", region="US")
|
||||
|
||||
TMDB_IMAGE_URL = "https://image.tmdb.org/t/p/original"
|
||||
@ -43,6 +46,28 @@ def lookup_video_from_tmdb(name_or_id: str, kind: str = "movie") -> VideoMetadat
|
||||
) # TODO: enrich this with TMDB url
|
||||
video_metadata.year = pendulum.parse(media.release_date).year
|
||||
video_metadata.genres = [g.get("name", "") for g in media.genres]
|
||||
video_metadata.tmdb_id = media.id
|
||||
video_metadata.base_run_time_seconds = media.runtime * 60
|
||||
video_metadata.plot = media.overview
|
||||
video_metadata.overview = media.overview
|
||||
video_metadata.tmdb_rating = media.vote_average
|
||||
|
||||
if len(tmdb_result.tv_results) > 0:
|
||||
media = TV().details(tmdb_result.tv_results[0].id)
|
||||
video_metadata.video_type = VideoType.TV_EPISODE.value
|
||||
video_metadata.title = media.name
|
||||
video_metadata.cover_url = (
|
||||
TMDB_IMAGE_URL + media.poster_path
|
||||
)
|
||||
video_metadata.year = pendulum.parse(media.first_air_date).year if media.first_air_date else None
|
||||
video_metadata.genres = [g.get("name", "") for g in media.genres]
|
||||
video_metadata.tmdb_id = media.id
|
||||
video_metadata.base_run_time_seconds = (
|
||||
media.episode_run_time[0] * 60 if media.episode_run_time else 1800
|
||||
)
|
||||
video_metadata.plot = media.overview
|
||||
video_metadata.overview = media.overview
|
||||
video_metadata.tmdb_rating = media.vote_average
|
||||
|
||||
if len(tmdb_result.tv_episode_results) > 0:
|
||||
video_metadata.video_type = VideoType.TV_EPISODE.value
|
||||
@ -63,15 +88,12 @@ def lookup_video_from_tmdb(name_or_id: str, kind: str = "movie") -> VideoMetadat
|
||||
series.save()
|
||||
video_metadata.tv_series_id = series.id
|
||||
|
||||
video_metadata.tmdb_id = media.id
|
||||
video_metadata.plot = media.overview
|
||||
video_metadata.overview = media.overview
|
||||
|
||||
if not media:
|
||||
logger.warning("Video not found on TMDB", extra={"imdb_id": imdb_id})
|
||||
return video_metadata
|
||||
|
||||
video_metadata.tmdb_id = media.id
|
||||
video_metadata.base_run_time_seconds = media.runtime * 60
|
||||
video_metadata.plot = media.overview
|
||||
video_metadata.overview = media.overview
|
||||
video_metadata.tmdb_rating = media.vote_average
|
||||
# video_metadata.next_imdb_id = imdb_result.get("next episode", None)
|
||||
|
||||
return video_metadata
|
||||
|
||||
@ -3,3 +3,6 @@ from django.apps import AppConfig
|
||||
|
||||
class WebpagesConfig(AppConfig):
|
||||
name = "webpages"
|
||||
|
||||
def ready(self):
|
||||
import webpages.signals # noqa
|
||||
|
||||
0
vrobbler/apps/webpages/management/__init__.py
Normal file
0
vrobbler/apps/webpages/management/__init__.py
Normal file
@ -0,0 +1,68 @@
|
||||
import re
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
|
||||
from scrobbles.utils import tokenize_title_to_tags
|
||||
from webpages.models import WebPage
|
||||
|
||||
|
||||
def _clean(s: str) -> str:
|
||||
return re.sub(r"[^\x20-\x7e]", "", s)
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = "Backfill auto tags on webpages from domain and title"
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--commit",
|
||||
action="store_true",
|
||||
help="Actually add tags",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
commit = options["commit"]
|
||||
|
||||
webpages = WebPage.objects.all()
|
||||
total = webpages.count()
|
||||
updated_count = 0
|
||||
skipped_count = 0
|
||||
|
||||
for i, webpage in enumerate(webpages.iterator(), start=1):
|
||||
new_tags = set()
|
||||
|
||||
if webpage.domain:
|
||||
parts = webpage.domain.root.split(".")
|
||||
for part in parts:
|
||||
part = part.strip().lower()
|
||||
if part and part != "www":
|
||||
new_tags.add(part)
|
||||
|
||||
if webpage.title:
|
||||
title_tags = tokenize_title_to_tags(webpage.title)
|
||||
new_tags.update(title_tags)
|
||||
|
||||
existing_tags = {
|
||||
t.name for t in webpage.tags.all()
|
||||
}
|
||||
tags_to_add = new_tags - existing_tags
|
||||
|
||||
if tags_to_add:
|
||||
updated_count += 1
|
||||
if commit:
|
||||
for tag in tags_to_add:
|
||||
webpage.tags.add(tag)
|
||||
self.stdout.write(
|
||||
f"[{i}/{total}] Added tags to {_clean(str(webpage))}: "
|
||||
f"{sorted(tags_to_add)}"
|
||||
)
|
||||
else:
|
||||
self.stdout.write(
|
||||
f"[{i}/{total}] [DRY RUN] Would add tags to "
|
||||
f"{_clean(str(webpage))}: {sorted(tags_to_add)}"
|
||||
)
|
||||
else:
|
||||
skipped_count += 1
|
||||
|
||||
self.stdout.write(f"\nDone. {updated_count} webpages to update, "
|
||||
f"{skipped_count} already up to date.")
|
||||
@ -2,6 +2,8 @@ import logging
|
||||
from typing import Dict, Optional
|
||||
from uuid import uuid4
|
||||
|
||||
import cloudscraper
|
||||
from curl_cffi import requests as curl_requests
|
||||
import pendulum
|
||||
import requests
|
||||
import trafilatura
|
||||
@ -45,6 +47,37 @@ class Domain(TimeStampedModel):
|
||||
)
|
||||
|
||||
|
||||
def _fetch_url_raw(url: str) -> Optional[str]:
|
||||
"""Fetch raw HTML for a URL.
|
||||
|
||||
Tries three strategies in order:
|
||||
1. trafilatura (standard, works for most sites)
|
||||
2. cloudscraper (handles basic Cloudflare JS challenges)
|
||||
3. curl_cffi (impersonates browser TLS fingerprint for aggressive
|
||||
Cloudflare / bot detection)
|
||||
"""
|
||||
raw = trafilatura.fetch_url(url)
|
||||
if raw:
|
||||
return raw
|
||||
logger.debug("trafilatura returned nothing for %s, trying cloudscraper", url)
|
||||
try:
|
||||
scraper = cloudscraper.create_scraper()
|
||||
resp = scraper.get(url, timeout=30)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
except Exception as exc:
|
||||
logger.debug("cloudscraper failed for %s: %s, trying curl_cffi", url, exc)
|
||||
try:
|
||||
resp = curl_requests.get(
|
||||
url, impersonate="chrome124", timeout=30
|
||||
)
|
||||
resp.raise_for_status()
|
||||
return resp.text
|
||||
except Exception as e:
|
||||
logger.warning("curl_cffi also failed for %s: %s", url, e)
|
||||
return None
|
||||
|
||||
|
||||
class WebPage(ScrobblableMixin):
|
||||
COMPLETION_PERCENT = getattr(settings, "WEBSITE_COMPLETION_PERCENT", 100)
|
||||
|
||||
@ -80,7 +113,7 @@ class WebPage(ScrobblableMixin):
|
||||
|
||||
def _update_extract_from_web(self, raw_text: str = "", force=True):
|
||||
if not raw_text:
|
||||
raw_text = requests.get(self.url, headers=headers).text
|
||||
raw_text = _fetch_url_raw(self.url)
|
||||
if not self.extract or force:
|
||||
self.extract = trafilatura.extract(raw_text)
|
||||
self.save(update_fields=["extract"])
|
||||
@ -259,7 +292,7 @@ class WebPage(ScrobblableMixin):
|
||||
return False
|
||||
|
||||
def fetch_data_from_web(self, save=True, force=True):
|
||||
raw_text = trafilatura.fetch_url(self.url)
|
||||
raw_text = _fetch_url_raw(self.url)
|
||||
if not self.extract or force:
|
||||
self.extract = trafilatura.extract(
|
||||
raw_text,
|
||||
|
||||
29
vrobbler/apps/webpages/signals.py
Normal file
29
vrobbler/apps/webpages/signals.py
Normal file
@ -0,0 +1,29 @@
|
||||
import logging
|
||||
|
||||
from django.db.models.signals import post_save
|
||||
from django.dispatch import receiver
|
||||
|
||||
from scrobbles.utils import tokenize_title_to_tags
|
||||
from webpages.models import WebPage
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
@receiver(post_save, sender=WebPage)
|
||||
def add_auto_tags_to_webpage(sender, instance, **kwargs):
|
||||
existing_tags = {t.name for t in instance.tags.all()}
|
||||
|
||||
if instance.domain:
|
||||
domain_parts = instance.domain.root.split(".")
|
||||
for part in domain_parts:
|
||||
part = part.strip().lower()
|
||||
if part and part != "www" and part not in existing_tags:
|
||||
instance.tags.add(part)
|
||||
existing_tags.add(part)
|
||||
|
||||
if instance.title:
|
||||
title_tags = tokenize_title_to_tags(instance.title)
|
||||
for tag in title_tags:
|
||||
if tag not in existing_tags:
|
||||
instance.tags.add(tag)
|
||||
existing_tags.add(tag)
|
||||
@ -16,8 +16,23 @@ from webpages.models import WebPage
|
||||
class WebPageListView(ScrobbleableListView):
|
||||
model = WebPage
|
||||
|
||||
def get_queryset(self):
|
||||
queryset = super().get_queryset()
|
||||
tag_name = self.request.GET.get("tag")
|
||||
if tag_name:
|
||||
from django.contrib.contenttypes.models import ContentType
|
||||
from taggit.models import TaggedItem
|
||||
ct = ContentType.objects.get_for_model(WebPage)
|
||||
webpages = TaggedItem.objects.filter(
|
||||
content_type=ct,
|
||||
tag__name__iexact=tag_name,
|
||||
).values_list("object_id", flat=True)
|
||||
queryset = queryset.filter(pk__in=webpages)
|
||||
return queryset
|
||||
|
||||
def get_context_data(self, **kwargs):
|
||||
context_data = super().get_context_data(**kwargs)
|
||||
context_data["active_tag"] = self.request.GET.get("tag", "")
|
||||
user = self.request.user
|
||||
now = timezone.now()
|
||||
start_day_of_week = now - datetime.timedelta(days=now.weekday())
|
||||
|
||||
@ -4,46 +4,55 @@ from importlib.metadata import version as get_version
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
_GIT_COMMIT = None
|
||||
|
||||
|
||||
def version_info(request):
|
||||
global _GIT_COMMIT
|
||||
|
||||
try:
|
||||
app_version = get_version("vrobbler")
|
||||
except Exception:
|
||||
app_version = "unknown"
|
||||
|
||||
commit = os.environ.get("VROBBLER_COMMIT")
|
||||
if not commit:
|
||||
# Try to import from _commit.py module first
|
||||
if commit:
|
||||
return {"app_version": app_version, "git_commit": commit}
|
||||
|
||||
if _GIT_COMMIT is not None:
|
||||
return {"app_version": app_version, "git_commit": _GIT_COMMIT}
|
||||
|
||||
try:
|
||||
from vrobbler._commit import commit as _commit
|
||||
except ImportError:
|
||||
pass
|
||||
else:
|
||||
if _commit and _commit != "unknown":
|
||||
_GIT_COMMIT = _commit
|
||||
return {"app_version": app_version, "git_commit": _GIT_COMMIT}
|
||||
|
||||
commit_file = Path("/var/lib/vrobbler/commit.txt")
|
||||
if commit_file.exists():
|
||||
try:
|
||||
from vrobbler._commit import commit as _commit
|
||||
except ImportError:
|
||||
commit = commit_file.read_text().strip()
|
||||
if commit:
|
||||
_GIT_COMMIT = commit
|
||||
return {"app_version": app_version, "git_commit": _GIT_COMMIT}
|
||||
except OSError:
|
||||
pass
|
||||
else:
|
||||
if _commit and _commit != "unknown":
|
||||
return {"app_version": app_version, "git_commit": _commit}
|
||||
|
||||
# Try to read from commit file (written during deploy)
|
||||
commit_file = Path("/var/lib/vrobbler/commit.txt")
|
||||
if commit_file.exists():
|
||||
try:
|
||||
commit = commit_file.read_text().strip()
|
||||
if commit:
|
||||
return {"app_version": app_version, "git_commit": commit}
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
# Fall back to git command
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
try:
|
||||
commit = (
|
||||
subprocess.check_output(
|
||||
["git", "rev-parse", "--short", "HEAD"],
|
||||
cwd=PROJECT_ROOT,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
.decode("utf-8")
|
||||
.strip()
|
||||
PROJECT_ROOT = Path(__file__).resolve().parent.parent
|
||||
try:
|
||||
_GIT_COMMIT = (
|
||||
subprocess.check_output(
|
||||
["git", "rev-parse", "--short", "HEAD"],
|
||||
cwd=PROJECT_ROOT,
|
||||
stderr=subprocess.DEVNULL,
|
||||
)
|
||||
except (subprocess.SubprocessError, FileNotFoundError):
|
||||
commit = "unknown"
|
||||
.decode("utf-8")
|
||||
.strip()
|
||||
)
|
||||
except (subprocess.SubprocessError, FileNotFoundError):
|
||||
_GIT_COMMIT = "unknown"
|
||||
|
||||
return {"app_version": app_version, "git_commit": commit}
|
||||
return {"app_version": app_version, "git_commit": _GIT_COMMIT}
|
||||
|
||||
@ -122,6 +122,15 @@ CELERY_ACCEPT_CONTENT = ["json"]
|
||||
CELERY_RESULT_EXTENDED = True
|
||||
CELERY_BROKER_CONNECTION_RETRY_ON_STARTUP = True
|
||||
|
||||
CELERY_TASK_CREATE_MISSING_QUEUES = True
|
||||
CELERY_TASK_ROUTES = {
|
||||
"scrobbles.tasks.update_charts_for_timestamp": {"queue": "charts"},
|
||||
"scrobbles.tasks.create_yesterdays_charts": {"queue": "charts"},
|
||||
"scrobbles.tasks.rebuild_weekly_charts": {"queue": "charts"},
|
||||
"scrobbles.tasks.rebuild_monthly_charts": {"queue": "charts"},
|
||||
"scrobbles.tasks.rebuild_yearly_charts": {"queue": "charts"},
|
||||
}
|
||||
|
||||
CELERY_BEAT_SCHEDULE = {
|
||||
"build-yesterdays-charts": {
|
||||
"task": "scrobbles.tasks.create_yesterdays_charts",
|
||||
@ -219,6 +228,7 @@ INSTALLED_APPS = [
|
||||
"foods",
|
||||
"lifeevents",
|
||||
"moods",
|
||||
"discgolf",
|
||||
"birds",
|
||||
"mathfilters",
|
||||
"rest_framework",
|
||||
@ -226,6 +236,7 @@ INSTALLED_APPS = [
|
||||
"allauth.account",
|
||||
"allauth.socialaccount",
|
||||
"django_celery_results",
|
||||
"mcp_server",
|
||||
]
|
||||
|
||||
SITE_ID = 1
|
||||
@ -319,6 +330,13 @@ REST_FRAMEWORK = {
|
||||
"PAGE_SIZE": 200,
|
||||
}
|
||||
|
||||
DJANGO_MCP_AUTHENTICATION_CLASSES = [
|
||||
"oauth2_provider.contrib.rest_framework.OAuth2Authentication",
|
||||
"rest_framework.authentication.TokenAuthentication",
|
||||
"rest_framework.authentication.SessionAuthentication",
|
||||
]
|
||||
DJANGO_MCP_GET_SERVER_INSTRUCTIONS_TOOL = True
|
||||
|
||||
LOGIN_REDIRECT_URL = "/"
|
||||
|
||||
AUTH_PASSWORD_VALIDATORS = [
|
||||
@ -352,11 +370,11 @@ USE_TZ = True
|
||||
# Static files (CSS, JavaScript, Images)
|
||||
# https://docs.djangoproject.com/en/3.1/howto/static-files/
|
||||
#
|
||||
from storages.backends import s3boto3
|
||||
|
||||
USE_S3_STORAGE = os.getenv("VROBBLER_USE_S3", "False").lower() in TRUTHY
|
||||
|
||||
if USE_S3_STORAGE:
|
||||
from storages.backends import s3boto3
|
||||
|
||||
AWS_S3_ENDPOINT_URL = os.getenv("AWS_S3_ENDPOINT_URL", "")
|
||||
AWS_STORAGE_BUCKET_NAME = os.getenv("AWS_STORAGE_BUCKET_NAME", "")
|
||||
AWS_S3_ACCESS_KEY_ID = os.getenv("AWS_S3_ACCESS_KEY_ID")
|
||||
@ -379,6 +397,8 @@ else:
|
||||
MEDIA_URL = os.getenv("VROBBLER_MEDIA_URL", "/media/")
|
||||
|
||||
|
||||
SCIHUB_DOMAIN = os.getenv("VROBBLER_SCIHUB_DOMAIN", "sci-hub.st")
|
||||
|
||||
JSON_LOGGING = os.getenv("VROBBLER_JSON_LOGGING", "false").lower() in TRUTHY
|
||||
LOG_TYPE = "json" if JSON_LOGGING else "log"
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user