Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 568772a0e6 | |||
| 91c3376256 | |||
| 58639c6fc1 | |||
| 228441ddc5 | |||
| 6341075f07 | |||
| a135b9f5f2 | |||
| 9088412d1e | |||
| c7339fbe31 | |||
| 4ce3dc03c5 | |||
| 5a4ef678a8 | |||
| 5ca22efeaa | |||
| 912ea8bfac | |||
| b541e1084d | |||
| c9b9da4abc |
@ -14,3 +14,11 @@ ro class method should call the utility function.
|
||||
|
||||
Be sure to check pyproject.toml for project defaults. Specifically for black and
|
||||
isort expectations.
|
||||
|
||||
All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
|
||||
|
||||
In local development, environment variables for various sensitive values live in a .envrc file
|
||||
|
||||
The .envrc file can be loaded into a shell environment to allow access to most third party services
|
||||
|
||||
Care should be taken when using .envrc that we do not spam services we use in production with requests
|
||||
|
||||
150
PROJECT.org
150
PROJECT.org
@ -17,6 +17,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
|
||||
to be sometimes cathartic and sometimes functional as I try to remember when I
|
||||
did a thing.
|
||||
|
||||
|
||||
* Features
|
||||
** Beer
|
||||
*** Triggers
|
||||
@ -86,7 +87,8 @@ fetching and simple saving.
|
||||
**** Bookmarklet
|
||||
*** Metadata sources
|
||||
**** Scraper
|
||||
* Backlog [0/12] :vrobbler:project:personal:
|
||||
|
||||
* Backlog [0/14] :vrobbler:project:personal:
|
||||
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
|
||||
:PROPERTIES:
|
||||
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
|
||||
@ -498,7 +500,118 @@ needed import celery task. This is how the WebDAV celery task currently works.
|
||||
This would also be an opporunity to clean up the code around WebDAV imports
|
||||
and make them more re-usable for other import services.
|
||||
|
||||
** TODO [#B] Add OMDB source as backup when TMDB returns nothing :videos:metadata:imdb:
|
||||
|
||||
** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: fd86a11a-73ec-470d-b5e3-2d90ba9137c8
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
|
||||
tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
|
||||
this put into it and then we stop trying to pull the artist apart when we run into it.
|
||||
|
||||
|
||||
** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
|
||||
|
||||
In most cases, we should be able to trust this data to created music tracks or videos rather
|
||||
than going to third-party services to enrich. Thus, for tracks and videos we should search in
|
||||
the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
|
||||
enrich further.
|
||||
|
||||
If not found, tracks and videos from mopidy and jellyfin should be created as completely as
|
||||
possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
|
||||
and start the scrobble. A separate celery task should be kicked off to enrich the track or video
|
||||
async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
|
||||
to go try to enrich the media instance. Should this enrichment fail, tag the scrobble as "enrichment-failed"
|
||||
log a warning and move on.
|
||||
|
||||
|
||||
* Version 48.1 [2/2]
|
||||
** DONE [#A] Generate a report of tracks with mistmatched metadata :music:tracks:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: 684b8cd2-a3c1-4995-ba9e-7abdb02c37f2
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
We should have a management command that outputs a CSV file of track IDs where
|
||||
the log["raw_data"]["Artist"] (for Jellyfin) or log["raw_data"]["artist"]
|
||||
(mopidy) value does not match the Track.artists names.
|
||||
|
||||
And we should see the same thing for albums (log["raw_data"]["Album"] or
|
||||
log["raw_data]["album"]).
|
||||
|
||||
It should output the fields "track_id", "track_artist_name", "track_album_name",
|
||||
"raw_artist", "raw_album", "source"
|
||||
|
||||
Where source is either Jellyfin or Mopidy based on the keys.
|
||||
|
||||
Put the file /tmp/metadata-report.csv by default and overwrite exsiting reports.
|
||||
|
||||
The command should also accept a file-path to overide this default.
|
||||
|
||||
|
||||
** DONE [#A] Date parsing failing in eBird imports :birds:ebird:importers:
|
||||
:PROPERTIES:
|
||||
:ID: 72e0254f-39d8-4843-9857-623e0362d77e
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
On line 45 in the apps/birds/importer.py file, the import is thorowing this
|
||||
error:
|
||||
|
||||
#+begin_src python
|
||||
ValueError: time data 'Jun 7, 2026, 5:15 PM' does not match format '%B %d, %Y %I:%M %p'
|
||||
#+end_src
|
||||
|
||||
Historically other files starting on May 24 worked, so I suspect this is a
|
||||
problem of the date formatter expecting Long month names and zero-padded days
|
||||
and the only sample we had was a three-letter month (May) and days with two
|
||||
digites(24 through the 31)
|
||||
|
||||
We should also add a "error_log" to the importers so that errors tha occur are
|
||||
surfaced, even when 0 successful files were processed. And we should make sure
|
||||
all importers do this as well.
|
||||
|
||||
|
||||
|
||||
* Version 48.0 [2/2]
|
||||
** DONE [#B] Show team or player images on sport detail and scrobble detail :sports:templates:
|
||||
:PROPERTIES:
|
||||
:ID: 68c17383-ee6e-4b5f-b3f5-1b637a0a3ea8
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
On the sport event detail page, we should show the images of the teams or
|
||||
players invovled.
|
||||
|
||||
Also, those images for the sport event should be shown on the scrobble detail
|
||||
page for sport event scrobble details.
|
||||
|
||||
** DONE [#B] Add fix_metadta method to Video instances :videos:metadata:
|
||||
:PROPERTIES:
|
||||
:ID: 9df5404d-1b60-4eee-b7cf-1f7e6dfade65
|
||||
:END:
|
||||
|
||||
*** Description
|
||||
|
||||
Turns out we don't have a fix_metadata method for videos. We should add that using
|
||||
the basic logic from find_or_create on the Video model.
|
||||
|
||||
|
||||
* Version 47.2 [1/1]
|
||||
** DONE [#B] Add OMDB source as backup when TMDB returns nothing :videos:metadata:imdb:
|
||||
:PROPERTIES:
|
||||
:ID: 20195445-7fdd-49be-9767-103b12da0bfb
|
||||
:END:
|
||||
@ -512,6 +625,14 @@ epiodes up in three parts, while they were always broadcast three-in-one, and
|
||||
that's how IMDB lists them. Thus, the IMDB ID means nothing, and the videos end
|
||||
up unenriched.
|
||||
|
||||
|
||||
* Version 47.1 [1/1]
|
||||
** DONE [#A] Untangle the sports migrations errors :sports:bug:migrations:
|
||||
:PROPERTIES:
|
||||
:ID: 4d50ca2e-f45b-dde8-e3c9-cd84f353b349
|
||||
:END:
|
||||
|
||||
|
||||
* Version 47.0 [1/1]
|
||||
** DONE [#B] Change sports scrobbling a bit :feature:sports:scrobbles:
|
||||
:PROPERTIES:
|
||||
@ -927,6 +1048,7 @@ Format should be: "%Y-%m-%dT%H:%M:%S.%fZ"
|
||||
:ID: 68a011b2-bb6f-3ba8-2312-5947c41db9ac
|
||||
:END:
|
||||
|
||||
|
||||
* Version 39.0 [3/3]
|
||||
** DONE [#B] Clean up org-mode tasks metadata :bug:tasks:metadata:
|
||||
:PROPERTIES:
|
||||
@ -981,6 +1103,7 @@ If the task is completed, don't touch it.
|
||||
:ID: 63dc633c-4382-e6a5-e663-b01871ce86ce
|
||||
:END:
|
||||
|
||||
|
||||
* Version 38.0 [38/38]
|
||||
** DONE [#A] Fix release flow to be easier to trigger :pyproject:release:tooling:
|
||||
:PROPERTIES:
|
||||
@ -1372,6 +1495,7 @@ urllib.error.HTTPError: HTTP Error 500: Internal Server Error
|
||||
|
||||
This may have already been resolved ... need to just confirm it.
|
||||
|
||||
|
||||
* Version 37.0 [4/4]
|
||||
** DONE [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
|
||||
:PROPERTIES:
|
||||
@ -1390,11 +1514,13 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
|
||||
:PROPERTIES:
|
||||
:ID: e3e49a9a-67d2-8ad8-1114-6f05effee9b7
|
||||
:END:
|
||||
|
||||
* Version 36.0 [1/1]
|
||||
** DONE [#A] Refactor how videos are scrobbled :vrobbler:vidoes:feature:personal:project:
|
||||
:PROPERTIES:
|
||||
:ID: 6034a11d-5376-994d-9a4b-e1640e258cfa
|
||||
:END:
|
||||
|
||||
* Version 35.0 [3/3]
|
||||
** DONE [#B] Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
|
||||
:PROPERTIES:
|
||||
@ -1408,6 +1534,7 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
|
||||
:PROPERTIES:
|
||||
:ID: d1ba1ca1-509b-13a9-1307-b2dc94a2eafe
|
||||
:END:
|
||||
|
||||
* Version 34.0 [4/4]
|
||||
** DONE [#A] Use bgg-api for BoardGameGeek lookups :vrobbler:feature:boardgames:personal:project:
|
||||
:PROPERTIES:
|
||||
@ -1467,6 +1594,7 @@ TypeError: can only concatenate str (not "NoneType") to str
|
||||
|
||||
A good lesson in using constants for things.
|
||||
|
||||
|
||||
* Version 33.0 [3/3]
|
||||
** DONE [#A] Fix bug where scrobble is_stale only uses seconds not total_seconds :vrobbler:bug:scrobbles:personal:project:
|
||||
:PROPERTIES:
|
||||
@ -1481,6 +1609,7 @@ TypeError: can only concatenate str (not "NoneType") to str
|
||||
:ID: 4955cc34-0882-50db-92f7-f36a95bf57a4
|
||||
:END:
|
||||
<2025-10-28 Tue>
|
||||
|
||||
* Version 32.0 [2/2]
|
||||
** DONE [#B] Save path to reading source on book scrobbles and show it on the detail page :vrobbler:feature:books:personal:project:
|
||||
:PROPERTIES:
|
||||
@ -1490,6 +1619,7 @@ TypeError: can only concatenate str (not "NoneType") to str
|
||||
:PROPERTIES:
|
||||
:ID: 9fe09567-11a3-7083-53c7-07458a9591d0
|
||||
:END:
|
||||
|
||||
* Version 31.0 [3/3]
|
||||
** DONE [#A] Stop comic book webpage scrobbles from overwriting old scrobbles :vrobbler:personal:bug:books:scrobbling:
|
||||
:PROPERTIES:
|
||||
@ -1503,6 +1633,7 @@ TypeError: can only concatenate str (not "NoneType") to str
|
||||
:PROPERTIES:
|
||||
:ID: 9a870c05-6d20-0803-d35d-c03fbe1d0ee1
|
||||
:END:
|
||||
|
||||
* Version 30.0 [3/3]
|
||||
** DONE [#A] Fix readcomicsonline browsing to update pages :vrobbler:books:feature:comicbook:personal:project:scrobbling:
|
||||
:PROPERTIES:
|
||||
@ -1517,10 +1648,13 @@ TypeError: can only concatenate str (not "NoneType") to str
|
||||
:PROPERTIES:
|
||||
:ID: d22cec3f-117f-f203-33a5-efbefa8a5cee
|
||||
:END:
|
||||
|
||||
* Version 29.0 [1/1]
|
||||
** DONE HOTFIX podcast lookups, final
|
||||
|
||||
* Version 28.0 [1/1]
|
||||
** DONE HOTFIX podcast lookups
|
||||
|
||||
* Version 27.0 [3/3]
|
||||
** DONE [#A] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
|
||||
:PROPERTIES:
|
||||
@ -1561,6 +1695,7 @@ it's annoying.
|
||||
actually be much more reliable than the current state of the podcast lookup
|
||||
which depends on the file to be name properly.
|
||||
|
||||
|
||||
* Version 26.0 [3/3]
|
||||
** DONE Clean up templates for scrobble details :vrobbler:personal:bug:templates:
|
||||
:PROPERTIES:
|
||||
@ -1574,6 +1709,7 @@ it's annoying.
|
||||
:PROPERTIES:
|
||||
:ID: c03a38ce-b337-f4fa-adba-aee08d4329f5
|
||||
:END:
|
||||
|
||||
* Version 25.0 [3/3]
|
||||
** DONE Add basic food templates and fix urls :food:vrobbler:personal:project:bug:urls:
|
||||
:PROPERTIES:
|
||||
@ -1587,6 +1723,7 @@ it's annoying.
|
||||
:PROPERTIES:
|
||||
:ID: 7debfbaf-cdd8-f49b-57ff-804bfe7c9236
|
||||
:END:
|
||||
|
||||
* Version 24.0 [2/2]
|
||||
** DONE Clean up logdata for various media :personal:feature:project:vrobbler:logdata:
|
||||
:PROPERTIES:
|
||||
@ -1597,6 +1734,7 @@ it's annoying.
|
||||
:ID: 1a1c0aa6-0313-c8be-1676-5d6adddef0a4
|
||||
:END:
|
||||
|
||||
|
||||
* Version 23.0 [3/3]
|
||||
** DONE Add dynamic forms for LogData classes :personal:feature:vrobbler:project:forms:logdata:
|
||||
:PROPERTIES:
|
||||
@ -1607,16 +1745,19 @@ it's annoying.
|
||||
:PROPERTIES:
|
||||
:ID: 99f6bd77-dc8f-6ed1-0321-32a52c944264
|
||||
:END:
|
||||
|
||||
* Version 19.0 [1/1]
|
||||
** DONE Add periodic check for mood :vrobbler:feature:moods:personal:project:
|
||||
:PROPERTIES:
|
||||
:ID: 55404488-c69f-0dd5-838e-1d1e15c873eb
|
||||
:END:
|
||||
|
||||
* Version 18.7 [1/1]
|
||||
** DONE Use the timezone history log to fix old Scrobbles that fall into those timezone blocks :vrobbler:chore:scrobbles:project:personal:
|
||||
:PROPERTIES:
|
||||
:ID: 9d055ac1-584b-20c8-7ad9-9ce36b329dc7
|
||||
:END:
|
||||
|
||||
* Version 18.4 [2/2]
|
||||
** DONE Track timezone changes for profiles :vrobbler:feature:profiles:personal:project:
|
||||
:PROPERTIES:
|
||||
@ -1630,11 +1771,13 @@ it's annoying.
|
||||
- Note taken on [2025-07-20 Sun 16:21]
|
||||
|
||||
This thing is kicking my butt. As it stands it works, but the scrobbles are not assigned to the tracks properly.
|
||||
|
||||
* Version 18.3 [1/1]
|
||||
** DONE Add timezone awarness to IMAP importer :personal:project:vrobbler:feature:importer:imap:timezones:
|
||||
:PROPERTIES:
|
||||
:ID: 05837b7c-96aa-6190-3678-e2ae7c7cac75
|
||||
:END:
|
||||
|
||||
* Version 18 [4/4]
|
||||
** DONE Condense tracks of the same title by the same artist with multiple albums :vrobbler:feature:music:project:personal:
|
||||
:PROPERTIES:
|
||||
@ -1756,6 +1899,7 @@ it's annoying.
|
||||
:PROPERTIES:
|
||||
:ID: 23f485e3-988c-6198-c79d-91fdf92f001c
|
||||
:END:
|
||||
|
||||
* Version 17.0 [6/6]
|
||||
** DONE [#A] Fix bug in new task label lookup for Emacs/Org-mode :vrobbler:bug:tasks:
|
||||
:PROPERTIES:
|
||||
@ -1819,6 +1963,7 @@ Not sure if the problem is in my Emacs hook sending or Vrobbler itself.
|
||||
:PROPERTIES:
|
||||
:ID: df58f8d0-fa4a-2037-c7d7-e5388c239042
|
||||
:END:
|
||||
|
||||
* Version 0.16.0 [19/19]
|
||||
** DONE [#A] Jellyfin, bandcamp tracks from Mopidy create duplicate music tracks :bug:scrobbling:music:
|
||||
:PROPERTIES:
|
||||
@ -1924,6 +2069,7 @@ out using that.
|
||||
** DONE Fix bug in Jellyfin scrobbles that spam more scrobbles after completion :scrobbling:videos:bug:
|
||||
This was fixed a while ago, but there's a new manifested bug. Going to create a
|
||||
separate bug tracking ticket for that.
|
||||
|
||||
* Version 0.11.4 [9/9]
|
||||
** DONE Add rudimentary video game scrobbling :improvement:content:videogames:
|
||||
CLOSED: [2023-03-07 Tue 11:11]
|
||||
|
||||
@ -1,6 +1,6 @@
|
||||
[tool.poetry]
|
||||
name = "vrobbler"
|
||||
version = "47.0"
|
||||
version = "48.1"
|
||||
description = ""
|
||||
authors = ["Colin Powell <colin@unbl.ink>"]
|
||||
|
||||
|
||||
@ -1,3 +1,4 @@
|
||||
import tempfile
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
@ -128,6 +129,23 @@ class TestBirdingCSVImportModel:
|
||||
assert imp.import_type == "Birding CSV"
|
||||
assert "Birding" in str(imp)
|
||||
|
||||
def test_record_error(self, db, user):
|
||||
imp = BirdingCSVImport.objects.create(user=user)
|
||||
assert imp.error_log is None
|
||||
imp.record_error("test error")
|
||||
imp.refresh_from_db()
|
||||
assert imp.error_log is not None
|
||||
assert "test error" in imp.error_log
|
||||
|
||||
def test_record_error_appends(self, db, user):
|
||||
imp = BirdingCSVImport.objects.create(user=user)
|
||||
imp.record_error("first error")
|
||||
imp.record_error("second error")
|
||||
imp.refresh_from_db()
|
||||
assert imp.error_log.count("\n") == 1
|
||||
assert "first error" in imp.error_log
|
||||
assert "second error" in imp.error_log
|
||||
|
||||
@pytest.mark.django_db(transaction=True)
|
||||
def test_process_via_model(self, user, birding_csv_file):
|
||||
imp = BirdingCSVImport.objects.create(user=user)
|
||||
@ -137,3 +155,35 @@ class TestBirdingCSVImportModel:
|
||||
imp.refresh_from_db()
|
||||
assert imp.process_count == 1
|
||||
assert imp.processed_finished is not None
|
||||
|
||||
def test_record_error_on_bad_csv(self, user, db):
|
||||
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
|
||||
Canada Goose,6,Test Park,Stationary,"Bad Date",4:15 PM,9 minute(s),,,4,true,4 species,
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8"
|
||||
) as f:
|
||||
f.write(content)
|
||||
file_path = f.name
|
||||
|
||||
errors = []
|
||||
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
|
||||
assert len(scrobbles) == 0
|
||||
assert len(errors) == 1
|
||||
assert "Could not parse date/time" in errors[0]
|
||||
|
||||
def test_record_error_on_bad_location(self, user, db):
|
||||
content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
|
||||
Canada Goose,6,,Stationary,"May 10, 2026",4:15 PM,9 minute(s),,,4,true,4 species,
|
||||
"""
|
||||
with tempfile.NamedTemporaryFile(
|
||||
mode="w", suffix=".csv", delete=False, encoding="utf-8"
|
||||
) as f:
|
||||
f.write(content)
|
||||
file_path = f.name
|
||||
|
||||
errors = []
|
||||
scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
|
||||
assert len(scrobbles) == 0
|
||||
assert len(errors) == 1
|
||||
assert "Skipping rows with no location" in errors[0]
|
||||
|
||||
@ -26,5 +26,5 @@ class BirdingLocationAdmin(admin.ModelAdmin):
|
||||
@admin.register(BirdingCSVImport)
|
||||
class BirdingCSVImportAdmin(admin.ModelAdmin):
|
||||
date_hierarchy = "created"
|
||||
list_display = ("uuid", "process_count", "processed_finished", "processing_started")
|
||||
list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
|
||||
ordering = ("-created",)
|
||||
|
||||
@ -2,7 +2,9 @@ import csv
|
||||
import logging
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from datetime import datetime, timedelta
|
||||
from datetime import timedelta
|
||||
|
||||
from dateutil import parser
|
||||
|
||||
from django.contrib.auth import get_user_model
|
||||
|
||||
@ -35,11 +37,12 @@ def parse_coords(location_str):
|
||||
|
||||
def parse_timestamp(date_str, time_str):
|
||||
try:
|
||||
dt = datetime.strptime(f"{date_str} {time_str}", "%B %d, %Y %I:%M %p")
|
||||
dt_str = f"{date_str} {time_str}".strip()
|
||||
dt = parser.parse(dt_str)
|
||||
return dt
|
||||
except (ValueError, TypeError):
|
||||
try:
|
||||
dt = datetime.strptime(date_str, "%B %d, %Y")
|
||||
dt = parser.parse(date_str)
|
||||
return dt
|
||||
except (ValueError, TypeError):
|
||||
logger.warning(f"Could not parse date/time: {date_str} {time_str}")
|
||||
@ -61,7 +64,7 @@ def parse_int(value):
|
||||
return None
|
||||
|
||||
|
||||
def import_birding_csv(file_path, user_id):
|
||||
def import_birding_csv(file_path, user_id, record_error=None):
|
||||
user = User.objects.get(id=user_id)
|
||||
new_scrobbles = []
|
||||
|
||||
@ -80,11 +83,17 @@ def import_birding_csv(file_path, user_id):
|
||||
|
||||
for (location_str, date_str, time_str), sighting_rows in groups.items():
|
||||
if not location_str:
|
||||
logger.warning("Skipping rows with no location")
|
||||
msg = "Skipping rows with no location"
|
||||
logger.warning(msg)
|
||||
if record_error:
|
||||
record_error(msg)
|
||||
continue
|
||||
|
||||
timestamp = parse_timestamp(date_str, time_str)
|
||||
if not timestamp:
|
||||
msg = f"Could not parse date/time: {date_str} {time_str}"
|
||||
if record_error:
|
||||
record_error(msg)
|
||||
continue
|
||||
|
||||
timestamp = user.profile.get_timestamp_with_tz(timestamp)
|
||||
|
||||
@ -0,0 +1,18 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-08 14:57
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("birds", "0002_birdingcsvimport"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="birdingcsvimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -61,9 +61,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
|
||||
@cached_property
|
||||
def bird_list(self) -> str:
|
||||
if self.birds:
|
||||
return ", ".join(
|
||||
[BirdSightingEntry(**b).__str__() for b in self.birds]
|
||||
)
|
||||
return ", ".join([BirdSightingEntry(**b).__str__() for b in self.birds])
|
||||
return ""
|
||||
|
||||
def as_html(self) -> str:
|
||||
@ -80,9 +78,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
|
||||
)
|
||||
|
||||
if self.area:
|
||||
html_parts.append(
|
||||
f'<div class="birding-area">Area: {self.area}</div>'
|
||||
)
|
||||
html_parts.append(f'<div class="birding-area">Area: {self.area}</div>')
|
||||
|
||||
if self.party_size:
|
||||
html_parts.append(
|
||||
@ -105,9 +101,7 @@ class BirdSightingLogData(BaseLogData, WithPeopleLogData):
|
||||
)
|
||||
|
||||
if self.guide:
|
||||
html_parts.append(
|
||||
f'<div class="birding-guide">Guide: {self.guide}</div>'
|
||||
)
|
||||
html_parts.append(f'<div class="birding-guide">Guide: {self.guide}</div>')
|
||||
|
||||
if self.duration_minutes:
|
||||
html_parts.append(
|
||||
@ -183,9 +177,7 @@ class Bird(TimeStampedModel):
|
||||
|
||||
class BirdingLocation(ScrobblableMixin):
|
||||
description = models.TextField(**BNULL)
|
||||
geo_location = models.ForeignKey(
|
||||
GeoLocation, **BNULL, on_delete=models.DO_NOTHING
|
||||
)
|
||||
geo_location = models.ForeignKey(GeoLocation, **BNULL, on_delete=models.DO_NOTHING)
|
||||
ebird_hotspot_id = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
def get_absolute_url(self):
|
||||
@ -193,7 +185,7 @@ class BirdingLocation(ScrobblableMixin):
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
return ""
|
||||
return self.geo_location
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
@ -224,6 +216,7 @@ class BirdingCSVImport(TimeStampedModel):
|
||||
processed_finished = models.DateTimeField(**BNULL)
|
||||
process_log = models.TextField(**BNULL)
|
||||
process_count = models.IntegerField(**BNULL)
|
||||
error_log = models.TextField(**BNULL)
|
||||
csv_file = models.FileField(upload_to="birding-csv-uploads/", **BNULL)
|
||||
|
||||
class Meta:
|
||||
@ -269,9 +262,7 @@ class BirdingCSVImport(TimeStampedModel):
|
||||
self.save(update_fields=["process_log", "process_count"])
|
||||
return
|
||||
for count, scrobble in enumerate(scrobbles):
|
||||
scrobble_str = (
|
||||
f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
|
||||
)
|
||||
scrobble_str = f"{scrobble.id}\t{scrobble.timestamp}\t{scrobble.media_obj}"
|
||||
log_line = f"{scrobble_str}"
|
||||
if count > 0:
|
||||
log_line = "\n" + log_line
|
||||
@ -279,6 +270,14 @@ class BirdingCSVImport(TimeStampedModel):
|
||||
self.process_count = len(scrobbles)
|
||||
self.save(update_fields=["process_log", "process_count"])
|
||||
|
||||
def record_error(self, error_message):
|
||||
log_line = f"{timezone.now().isoformat()}: {error_message}"
|
||||
if self.error_log:
|
||||
self.error_log += "\n" + log_line
|
||||
else:
|
||||
self.error_log = log_line
|
||||
self.save(update_fields=["error_log"])
|
||||
|
||||
def scrobbles(self):
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
@ -297,6 +296,13 @@ class BirdingCSVImport(TimeStampedModel):
|
||||
from birds.importer import import_birding_csv
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = import_birding_csv(
|
||||
self.upload_file_path, self.user_id, record_error=self.record_error
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
@ -266,8 +266,9 @@ class BoardGame(ScrobblableMixin):
|
||||
"self", **BNULL, on_delete=models.DO_NOTHING
|
||||
)
|
||||
|
||||
def __str__(self):
|
||||
return self.title
|
||||
@property
|
||||
def subtitle(self) -> str:
|
||||
return self.publisher
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("boardgames:boardgame_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@ -173,11 +173,7 @@ class Book(LongPlayScrobblableMixin):
|
||||
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
|
||||
|
||||
def __str__(self) -> str:
|
||||
if self.issue_number and "Issue" not in str(self.title):
|
||||
return f"{self.title} - Issue {self.issue_number}"
|
||||
if self.volume_number and "Volume" not in str(self.title):
|
||||
return f"{self.title} - Volume {self.volume_number}"
|
||||
return f"{self.title}"
|
||||
return f"{self.title} - {self.subtitle}"
|
||||
|
||||
def save(self, *args, **kwargs):
|
||||
if self.pages:
|
||||
@ -188,7 +184,12 @@ class Book(LongPlayScrobblableMixin):
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
return f" by {self.author}"
|
||||
subtitle = self.author
|
||||
if self.issue_number and "Issue" not in str(self.title):
|
||||
subtitle += " - Issue {self.issue_number}"
|
||||
if self.volume_number and "Volume" not in str(self.title):
|
||||
subtitle += " - Volume {self.volume_number}"
|
||||
return subtitle
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
|
||||
@ -0,0 +1,127 @@
|
||||
import csv
|
||||
import logging
|
||||
|
||||
from django.core.management.base import BaseCommand
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
def _get_source(raw_data):
|
||||
if "Artist" in raw_data:
|
||||
return "Jellyfin"
|
||||
if "artist" in raw_data:
|
||||
return "Mopidy"
|
||||
return None
|
||||
|
||||
|
||||
def _get_raw_values(raw_data, source):
|
||||
if source == "Jellyfin":
|
||||
return raw_data.get("Artist", ""), raw_data.get("Album", "")
|
||||
return raw_data.get("artist", ""), raw_data.get("album", "")
|
||||
|
||||
|
||||
def _normalize(name):
|
||||
return name.strip().casefold()
|
||||
|
||||
|
||||
def _artist_mismatch(raw_artist, track_artist_names):
|
||||
if not raw_artist or not track_artist_names:
|
||||
return False
|
||||
track_names = [_normalize(n) for n in track_artist_names.split(" / ")]
|
||||
raw = _normalize(raw_artist)
|
||||
if raw in track_names:
|
||||
return False
|
||||
if raw == _normalize(track_artist_names):
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _album_mismatch(raw_album, track_album_name):
|
||||
if not raw_album or not track_album_name:
|
||||
return False
|
||||
return _normalize(raw_album) != _normalize(track_album_name)
|
||||
|
||||
|
||||
class Command(BaseCommand):
|
||||
help = (
|
||||
"Outputs a CSV of track IDs where raw metadata from scrobble logs "
|
||||
"does not match the track's stored artists or album"
|
||||
)
|
||||
|
||||
def add_arguments(self, parser):
|
||||
parser.add_argument(
|
||||
"--file-path",
|
||||
type=str,
|
||||
default="/tmp/metadata-report.csv",
|
||||
help="Output CSV file path (default: /tmp/metadata-report.csv)",
|
||||
)
|
||||
|
||||
def handle(self, *args, **options):
|
||||
from scrobbles.models import Scrobble
|
||||
|
||||
file_path = options["file_path"]
|
||||
|
||||
qs = (
|
||||
Scrobble.objects.filter(media_type=Scrobble.MediaType.TRACK)
|
||||
.exclude(log__isnull=True)
|
||||
.exclude(log={})
|
||||
.select_related("track__album")
|
||||
.prefetch_related("track__artists")
|
||||
.iterator()
|
||||
)
|
||||
|
||||
rows = []
|
||||
for scrobble in qs:
|
||||
track = scrobble.track
|
||||
if not track:
|
||||
continue
|
||||
|
||||
raw_data = scrobble.log.get("raw_data")
|
||||
if not raw_data:
|
||||
continue
|
||||
|
||||
source = _get_source(raw_data)
|
||||
if not source:
|
||||
continue
|
||||
|
||||
raw_artist, raw_album = _get_raw_values(raw_data, source)
|
||||
if not raw_artist and not raw_album:
|
||||
continue
|
||||
|
||||
track_artist_names = " / ".join(
|
||||
track.artists.all().values_list("name", flat=True)
|
||||
)
|
||||
track_album_name = track.album.name if track.album else ""
|
||||
|
||||
if _artist_mismatch(raw_artist, track_artist_names) or _album_mismatch(
|
||||
raw_album, track_album_name
|
||||
):
|
||||
rows.append(
|
||||
{
|
||||
"track_id": track.id,
|
||||
"track_artist_name": track_artist_names,
|
||||
"track_album_name": track_album_name,
|
||||
"raw_artist": raw_artist,
|
||||
"raw_album": raw_album,
|
||||
"source": source,
|
||||
}
|
||||
)
|
||||
|
||||
fieldnames = [
|
||||
"track_id",
|
||||
"track_artist_name",
|
||||
"track_album_name",
|
||||
"raw_artist",
|
||||
"raw_album",
|
||||
"source",
|
||||
]
|
||||
with open(file_path, "w", newline="") as f:
|
||||
writer = csv.DictWriter(f, fieldnames=fieldnames)
|
||||
writer.writeheader()
|
||||
writer.writerows(rows)
|
||||
|
||||
self.stdout.write(
|
||||
self.style.SUCCESS(
|
||||
f"Wrote {len(rows)} mismatched track(s) to {file_path}"
|
||||
)
|
||||
)
|
||||
@ -20,6 +20,7 @@ class ScrobbleInline(admin.TabularInline):
|
||||
extra = 0
|
||||
raw_id_fields = (
|
||||
"video",
|
||||
"channel",
|
||||
"podcast_episode",
|
||||
"track",
|
||||
"video_game",
|
||||
@ -30,6 +31,7 @@ class ScrobbleInline(admin.TabularInline):
|
||||
"board_game",
|
||||
"geo_location",
|
||||
"task",
|
||||
"puzzle",
|
||||
"mood",
|
||||
"brick_set",
|
||||
"trail",
|
||||
@ -54,52 +56,44 @@ class ImportBaseAdmin(admin.ModelAdmin):
|
||||
"process_count",
|
||||
"processed_finished",
|
||||
"processing_started",
|
||||
"error_log",
|
||||
)
|
||||
ordering = ("-created",)
|
||||
|
||||
|
||||
@admin.register(AudioScrobblerTSVImport)
|
||||
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class AudioScrobblerTSVImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(LastFmImport)
|
||||
class LastFmImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class LastFmImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(KoReaderImport)
|
||||
class KoReaderImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class KoReaderImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(RetroarchImport)
|
||||
class RetroarchImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class RetroarchImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
class RetroarchImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class RetroarchImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(BGStatsImport)
|
||||
class BGStatsImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class BGStatsImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(EBirdCSVImport)
|
||||
class EBirdCSVImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class EBirdCSVImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(ScaleCSVImport)
|
||||
class ScaleCSVImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class ScaleCSVImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(TrailGPXImport)
|
||||
class TrailGPXImportAdmin(ImportBaseAdmin):
|
||||
...
|
||||
class TrailGPXImportAdmin(ImportBaseAdmin): ...
|
||||
|
||||
|
||||
@admin.register(Genre)
|
||||
|
||||
@ -0,0 +1,53 @@
|
||||
# Generated by Django 4.2.29 on 2026-06-08 14:57
|
||||
|
||||
from django.db import migrations, models
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("scrobbles", "0089_favoritemedia"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.AddField(
|
||||
model_name="audioscrobblertsvimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="bgstatsimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="ebirdcsvimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="koreaderimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="lastfmimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="retroarchimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="scalecsvimport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
migrations.AddField(
|
||||
model_name="trailgpximport",
|
||||
name="error_log",
|
||||
field=models.TextField(blank=True, null=True),
|
||||
),
|
||||
]
|
||||
@ -65,6 +65,9 @@ class ScrobblableMixin(TimeStampedModel):
|
||||
class Meta:
|
||||
abstract = True
|
||||
|
||||
def __str__(self) -> str:
|
||||
return f"{self.title} - {self.subtitle}"
|
||||
|
||||
@property
|
||||
def run_time_seconds(self) -> int:
|
||||
run_time = 900
|
||||
|
||||
@ -74,6 +74,7 @@ class BaseFileImportMixin(TimeStampedModel):
|
||||
processed_finished = models.DateTimeField(**BNULL)
|
||||
process_log = models.TextField(**BNULL)
|
||||
process_count = models.IntegerField(**BNULL)
|
||||
error_log = models.TextField(**BNULL)
|
||||
|
||||
class Meta:
|
||||
abstract = True
|
||||
@ -158,6 +159,14 @@ class BaseFileImportMixin(TimeStampedModel):
|
||||
self.process_count = len(scrobbles)
|
||||
self.save(update_fields=["process_log", "process_count"])
|
||||
|
||||
def record_error(self, error_message):
|
||||
log_line = f"{timezone.now().isoformat()}: {error_message}"
|
||||
if self.error_log:
|
||||
self.error_log += "\n" + log_line
|
||||
else:
|
||||
self.error_log = log_line
|
||||
self.save(update_fields=["error_log"])
|
||||
|
||||
@property
|
||||
def upload_file_path(self):
|
||||
raise NotImplementedError
|
||||
@ -213,9 +222,14 @@ class KoReaderImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class AudioScrobblerTSVImport(BaseFileImportMixin):
|
||||
@ -255,10 +269,14 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
|
||||
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class ScaleCSVImport(BaseFileImportMixin):
|
||||
@ -297,9 +315,14 @@ class ScaleCSVImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class TrailGPXImport(BaseFileImportMixin):
|
||||
@ -337,11 +360,16 @@ class TrailGPXImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = import_trail_gpx(
|
||||
self.upload_file_path, self.user.id, self.original_filename
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = import_trail_gpx(
|
||||
self.upload_file_path, self.user.id, self.original_filename
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class LastFmImport(BaseFileImportMixin):
|
||||
@ -391,11 +419,14 @@ class LastFmImport(BaseFileImportMixin):
|
||||
last_processed = last_import.processed_finished
|
||||
|
||||
self.mark_started()
|
||||
|
||||
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
|
||||
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class RetroarchImport(BaseFileImportMixin):
|
||||
@ -426,43 +457,48 @@ class RetroarchImport(BaseFileImportMixin):
|
||||
logger.info(f"You told me to force import from Retroarch")
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = None
|
||||
try:
|
||||
if self.lrtl_file:
|
||||
import os
|
||||
import tempfile
|
||||
import zipfile
|
||||
|
||||
if self.lrtl_file:
|
||||
import os
|
||||
import tempfile
|
||||
import zipfile
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
try:
|
||||
zip_path = os.path.join(tmpdir, "archive.zip")
|
||||
with open(zip_path, "wb") as f:
|
||||
f.write(self.lrtl_file.read())
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
zf.extractall(tmpdir)
|
||||
os.unlink(zip_path)
|
||||
scrobbles = retroarch.import_retroarch_lrtl_files(
|
||||
tmpdir + "/",
|
||||
self.user.id,
|
||||
)
|
||||
finally:
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmpdir, ignore_errors=True)
|
||||
else:
|
||||
if not self.user.profile.retroarch_path:
|
||||
logger.info(
|
||||
"Trying to import Retroarch logs, but user has no retroarch_path configured"
|
||||
)
|
||||
self.mark_finished()
|
||||
return
|
||||
|
||||
tmpdir = tempfile.mkdtemp()
|
||||
try:
|
||||
zip_path = os.path.join(tmpdir, "archive.zip")
|
||||
with open(zip_path, "wb") as f:
|
||||
f.write(self.lrtl_file.read())
|
||||
with zipfile.ZipFile(zip_path, "r") as zf:
|
||||
zf.extractall(tmpdir)
|
||||
os.unlink(zip_path)
|
||||
scrobbles = retroarch.import_retroarch_lrtl_files(
|
||||
tmpdir + "/",
|
||||
self.user.profile.retroarch_path,
|
||||
self.user.id,
|
||||
)
|
||||
finally:
|
||||
import shutil
|
||||
|
||||
shutil.rmtree(tmpdir, ignore_errors=True)
|
||||
else:
|
||||
if not self.user.profile.retroarch_path:
|
||||
logger.info(
|
||||
"Tying to import Retroarch logs, but user has no retroarch_path configured"
|
||||
)
|
||||
self.mark_finished()
|
||||
return
|
||||
|
||||
scrobbles = retroarch.import_retroarch_lrtl_files(
|
||||
self.user.profile.retroarch_path,
|
||||
self.user.id,
|
||||
)
|
||||
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class BGStatsImport(BaseFileImportMixin):
|
||||
@ -499,17 +535,21 @@ class BGStatsImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
try:
|
||||
import json
|
||||
|
||||
import json
|
||||
from scrobbles.scrobblers import email_scrobble_board_game
|
||||
|
||||
from scrobbles.scrobblers import email_scrobble_board_game
|
||||
with open(self.upload_file_path, "r", encoding="utf-8") as f:
|
||||
parsed_json = json.load(f)
|
||||
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
|
||||
|
||||
with open(self.upload_file_path, "r", encoding="utf-8") as f:
|
||||
parsed_json = json.load(f)
|
||||
scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
|
||||
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class EBirdCSVImport(BaseFileImportMixin):
|
||||
@ -554,9 +594,16 @@ class EBirdCSVImport(BaseFileImportMixin):
|
||||
return
|
||||
|
||||
self.mark_started()
|
||||
scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
|
||||
self.record_log(scrobbles)
|
||||
self.mark_finished()
|
||||
try:
|
||||
scrobbles = import_birding_csv(
|
||||
self.upload_file_path, self.user_id, record_error=self.record_error
|
||||
)
|
||||
self.record_log(scrobbles)
|
||||
except Exception as e:
|
||||
self.record_error(f"Import failed: {e}")
|
||||
logger.exception(f"Import failed for {self}")
|
||||
finally:
|
||||
self.mark_finished()
|
||||
|
||||
|
||||
class ScrobbleQuerySet(models.QuerySet):
|
||||
|
||||
@ -184,7 +184,7 @@ class ScrobbleableDetailView(ChartContextMixin, DetailView):
|
||||
def get_context_data(self, **kwargs):
|
||||
context_data = super().get_context_data(**kwargs)
|
||||
scrobbles = []
|
||||
if not self.request.user.is_anonymous:
|
||||
if not self.request.user.is_anonymous and hasattr(self.object, "scrobble_set"):
|
||||
scrobbles = self.object.scrobble_set.filter(
|
||||
user=self.request.user
|
||||
).order_by("-timestamp")
|
||||
@ -439,8 +439,7 @@ class ScrobbleListView(LoginRequiredMixin, ListView):
|
||||
full_qs = getattr(self, "_full_queryset", None)
|
||||
if full_qs is not None and getattr(self, "tag_list", []):
|
||||
total = (
|
||||
full_qs.aggregate(total=Sum("playback_position_seconds"))["total"]
|
||||
or 0
|
||||
full_qs.aggregate(total=Sum("playback_position_seconds"))["total"] or 0
|
||||
)
|
||||
ctx["total_time_seconds"] = total
|
||||
return ctx
|
||||
@ -1245,8 +1244,7 @@ class ScrobbleDetailView(DetailView):
|
||||
track=media_obj, user=self.object.user
|
||||
).order_by("-timestamp")[:20]
|
||||
context["has_mopidy_uri"] = any(
|
||||
(s.log or {}).get("raw_data", {}).get("mopidy_uri")
|
||||
for s in scrobbles
|
||||
(s.log or {}).get("raw_data", {}).get("mopidy_uri") for s in scrobbles
|
||||
)
|
||||
else:
|
||||
context["has_mopidy_uri"] = False
|
||||
|
||||
@ -108,20 +108,4 @@ class Migration(migrations.Migration):
|
||||
migrations.RunPython(
|
||||
migrate_sport_event_data, reverse_code=migrations.RunPython.noop
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="home_team",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="away_team",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="player_one",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="player_two",
|
||||
),
|
||||
]
|
||||
|
||||
@ -0,0 +1,27 @@
|
||||
from django.db import migrations
|
||||
|
||||
|
||||
class Migration(migrations.Migration):
|
||||
|
||||
dependencies = [
|
||||
("sports", "0021_team_logo"),
|
||||
]
|
||||
|
||||
operations = [
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="home_team",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="away_team",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="player_one",
|
||||
),
|
||||
migrations.RemoveField(
|
||||
model_name="sportevent",
|
||||
name="player_two",
|
||||
),
|
||||
]
|
||||
@ -123,17 +123,17 @@ class SportEvent(ScrobblableMixin):
|
||||
super(SportEvent, self).save(*args, **kwargs)
|
||||
|
||||
def __str__(self) -> str:
|
||||
league = self.league
|
||||
if self.league and self.league.abbreviation_str:
|
||||
league = self.league.abbreviation_str
|
||||
return f"{self.title} - {league}"
|
||||
return f"{self.title} - {self.subtitle}"
|
||||
|
||||
def get_absolute_url(self):
|
||||
return reverse("sports:event_detail", kwargs={"slug": self.uuid})
|
||||
|
||||
@property
|
||||
def subtitle(self) -> str:
|
||||
return self.comp_str
|
||||
league = self.league
|
||||
if self.league and self.league.abbreviation_str:
|
||||
league = self.league.abbreviation_str
|
||||
return f"{league} {self.get_event_type_display()}"
|
||||
|
||||
@property
|
||||
def comp_str(self) -> str:
|
||||
|
||||
@ -163,12 +163,9 @@ class VideoGame(LongPlayScrobblableMixin):
|
||||
platforms = models.ManyToManyField(VideoGamePlatform)
|
||||
retroarch_name = models.CharField(max_length=255, **BNULL)
|
||||
|
||||
def __str__(self):
|
||||
return self.title
|
||||
|
||||
@property
|
||||
def subtitle(self):
|
||||
return f" On {self.platforms.first()}"
|
||||
return f"{self.platforms.first()}"
|
||||
|
||||
@property
|
||||
def strings(self) -> ScrobblableConstants:
|
||||
|
||||
@ -71,6 +71,7 @@ class VideoMetadata:
|
||||
twitch_id: Optional[str] = "",
|
||||
base_run_time_seconds: int = 900,
|
||||
):
|
||||
self.title = ""
|
||||
self.imdb_id = imdb_id
|
||||
self.youtube_id = youtube_id
|
||||
self.twitch_id = twitch_id
|
||||
|
||||
@ -20,6 +20,7 @@ from scrobbles.mixins import (
|
||||
)
|
||||
from taggit.managers import TaggableManager
|
||||
from videos.metadata import VideoMetadata
|
||||
from videos.sources.omdb import lookup_video_from_omdb
|
||||
from videos.sources.tmdb import lookup_video_from_tmdb
|
||||
from videos.sources.youtube import lookup_video_from_youtube
|
||||
|
||||
@ -224,6 +225,8 @@ class Series(TimeStampedModel):
|
||||
|
||||
def is_episode_playing(self, user_id: int) -> bool:
|
||||
last_scrobble = self.scrobbles_for_user(user_id, include_playing=True).first()
|
||||
if not last_scrobble:
|
||||
return False
|
||||
return not last_scrobble.played_to_completion
|
||||
|
||||
def fix_metadata(self, force_update=False):
|
||||
@ -255,9 +258,20 @@ class Series(TimeStampedModel):
|
||||
logger.info("Series not created and overwrite=False, returning")
|
||||
return series
|
||||
|
||||
vdict, _, cover, genres = lookup_video_from_tmdb(
|
||||
imdb_id
|
||||
).as_dict_with_cover_and_genres()
|
||||
try:
|
||||
metadata = lookup_video_from_tmdb(imdb_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"TMDB lookup failed for series {imdb_id}: {e}")
|
||||
metadata = None
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
metadata = lookup_video_from_omdb(imdb_id)
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
logger.warning(f"No metadata found for series {imdb_id} from TMDB or OMDB")
|
||||
return series
|
||||
|
||||
vdict, _, cover, genres = metadata.as_dict_with_cover_and_genres()
|
||||
vdict.pop("video_type")
|
||||
|
||||
vdict["name"] = vdict.pop("title")
|
||||
@ -406,6 +420,84 @@ class Video(ScrobblableMixin):
|
||||
fname = f"{self.title}_{self.uuid}.jpg"
|
||||
self.cover_image.save(fname, ContentFile(r.content), save=True)
|
||||
|
||||
def fix_metadata(self, force_update: bool = False) -> None:
|
||||
if self.video_type == self.VideoType.YOUTUBE and self.youtube_id:
|
||||
vdict, _, cover, genres = lookup_video_from_youtube(
|
||||
self.youtube_id
|
||||
).as_dict_with_cover_and_genres()
|
||||
|
||||
for k, v in vdict.items():
|
||||
setattr(self, k, v)
|
||||
self.save()
|
||||
|
||||
if cover:
|
||||
self.save_image_from_url(cover)
|
||||
if genres:
|
||||
self.genre.add(*genres)
|
||||
return
|
||||
|
||||
if self.video_type == self.VideoType.TWITCH and self.twitch_id:
|
||||
from videos.sources.twitch import lookup_video_from_twitch
|
||||
|
||||
metadata = lookup_video_from_twitch(self.twitch_id)
|
||||
self.title = metadata.title or f"Twitch VOD {self.twitch_id}"
|
||||
self.overview = metadata.overview
|
||||
self.base_run_time_seconds = metadata.base_run_time_seconds
|
||||
if metadata.upload_date:
|
||||
self.upload_date = metadata.upload_date
|
||||
if metadata.year:
|
||||
self.year = metadata.year
|
||||
self.video_type = Video.VideoType.TWITCH
|
||||
|
||||
if metadata.channel_id:
|
||||
from videos.models import Channel
|
||||
|
||||
self.channel = Channel.objects.filter(
|
||||
id=metadata.channel_id
|
||||
).first()
|
||||
|
||||
self.save()
|
||||
|
||||
if metadata.cover_url:
|
||||
self.save_image_from_url(metadata.cover_url)
|
||||
return
|
||||
|
||||
if self.imdb_id:
|
||||
try:
|
||||
metadata = lookup_video_from_tmdb(self.imdb_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"TMDB lookup failed for {self.imdb_id}: {e}")
|
||||
metadata = None
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
metadata = lookup_video_from_omdb(self.imdb_id)
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
logger.warning(f"No metadata found for {self} from TMDB or OMDB")
|
||||
return
|
||||
|
||||
vdict, series_id, cover, genres = (
|
||||
metadata.as_dict_with_cover_and_genres()
|
||||
)
|
||||
|
||||
for k, v in vdict.items():
|
||||
setattr(self, k, v)
|
||||
|
||||
if series_id:
|
||||
self.tv_series = Series.find_or_create(imdb_id=series_id)
|
||||
|
||||
self.save()
|
||||
|
||||
if cover:
|
||||
self.save_image_from_url(cover)
|
||||
if genres:
|
||||
self.genre.add(*genres)
|
||||
return
|
||||
|
||||
logger.warning(
|
||||
f"No metadata source available for {self} (type={self.video_type})"
|
||||
)
|
||||
|
||||
@classmethod
|
||||
def get_from_youtube_id(cls, youtube_id: str, overwrite: bool = False) -> "Video":
|
||||
video, created = cls.objects.get_or_create(youtube_id=youtube_id)
|
||||
@ -432,9 +524,20 @@ class Video(ScrobblableMixin):
|
||||
if not created and not overwrite:
|
||||
return video
|
||||
|
||||
vdict, series_id, cover, genres = lookup_video_from_tmdb(
|
||||
imdb_id
|
||||
).as_dict_with_cover_and_genres()
|
||||
try:
|
||||
metadata = lookup_video_from_tmdb(imdb_id)
|
||||
except Exception as e:
|
||||
logger.warning(f"TMDB lookup failed for {imdb_id}: {e}")
|
||||
metadata = None
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
metadata = lookup_video_from_omdb(imdb_id)
|
||||
|
||||
if not metadata or not metadata.title:
|
||||
logger.warning(f"No metadata found for {imdb_id} from TMDB or OMDB")
|
||||
return video
|
||||
|
||||
vdict, series_id, cover, genres = metadata.as_dict_with_cover_and_genres()
|
||||
|
||||
if created or overwrite:
|
||||
for k, v in vdict.items():
|
||||
|
||||
83
vrobbler/apps/videos/sources/omdb.py
Normal file
83
vrobbler/apps/videos/sources/omdb.py
Normal file
@ -0,0 +1,83 @@
|
||||
import logging
|
||||
import re
|
||||
from typing import Optional
|
||||
|
||||
import requests
|
||||
from django.conf import settings
|
||||
from videos.metadata import VideoMetadata, VideoType
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
OMDB_API_KEY = getattr(settings, "OMDB_API_KEY", "")
|
||||
|
||||
OMDB_URL = "https://www.omdbapi.com/"
|
||||
RUNTIME_RE = re.compile(r"(\d+)\s*min")
|
||||
|
||||
|
||||
def lookup_video_from_omdb(imdb_id: str) -> Optional[VideoMetadata]:
|
||||
if not imdb_id.startswith("tt"):
|
||||
imdb_id = f"tt{imdb_id}"
|
||||
|
||||
if not OMDB_API_KEY:
|
||||
logger.warning("No OMDB API key configured")
|
||||
return None
|
||||
|
||||
params = {"apikey": OMDB_API_KEY, "i": imdb_id, "plot": "full"}
|
||||
|
||||
try:
|
||||
response = requests.get(OMDB_URL, params=params)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
except requests.exceptions.RequestException as e:
|
||||
logger.error(f"OMDB API error for {imdb_id}: {e}")
|
||||
return None
|
||||
|
||||
if data.get("Response") == "False":
|
||||
logger.info(f"OMDB no result for {imdb_id}: {data.get('Error')}")
|
||||
return None
|
||||
|
||||
metadata = VideoMetadata(imdb_id=imdb_id)
|
||||
metadata.title = data.get("Title")
|
||||
metadata.plot = data.get("Plot")
|
||||
metadata.overview = data.get("Plot")
|
||||
|
||||
raw_year = data.get("Year")
|
||||
if raw_year and raw_year.isdigit():
|
||||
metadata.year = int(raw_year)
|
||||
|
||||
raw_rating = data.get("imdbRating")
|
||||
if raw_rating and raw_rating != "N/A":
|
||||
metadata.imdb_rating = raw_rating
|
||||
|
||||
raw_cover = data.get("Poster")
|
||||
if raw_cover and raw_cover != "N/A":
|
||||
metadata.cover_url = raw_cover
|
||||
|
||||
raw_runtime = data.get("Runtime")
|
||||
if raw_runtime:
|
||||
match = RUNTIME_RE.match(raw_runtime)
|
||||
if match:
|
||||
metadata.base_run_time_seconds = int(match.group(1)) * 60
|
||||
|
||||
media_type = data.get("Type")
|
||||
if media_type == "movie":
|
||||
metadata.video_type = VideoType.MOVIE.value
|
||||
elif media_type in ("series", "episode"):
|
||||
metadata.video_type = VideoType.TV_EPISODE.value
|
||||
|
||||
if media_type == "episode":
|
||||
raw_season = data.get("Season")
|
||||
if raw_season and raw_season != "N/A":
|
||||
metadata.season_number = int(raw_season)
|
||||
raw_episode = data.get("Episode")
|
||||
if raw_episode and raw_episode != "N/A":
|
||||
metadata.episode_number = int(raw_episode)
|
||||
series_imdb_id = data.get("seriesID")
|
||||
if series_imdb_id and series_imdb_id != "N/A":
|
||||
metadata.tv_series_imdb_id = series_imdb_id
|
||||
|
||||
raw_genres = data.get("Genre")
|
||||
if raw_genres:
|
||||
metadata.genres = [g.strip() for g in raw_genres.split(",") if g.strip()]
|
||||
|
||||
return metadata
|
||||
@ -3,14 +3,11 @@ import logging
|
||||
import pendulum
|
||||
from django.conf import settings
|
||||
from themoviedb import TMDb
|
||||
from tmdbv3api import TV, Movie, TMDb as TMDb_direct
|
||||
from tmdbv3api import TV, Movie
|
||||
from videos.metadata import VideoMetadata, VideoType
|
||||
|
||||
TMDB_KEY = getattr(settings, "TMDB_API_KEY", "")
|
||||
|
||||
tmdb_direct = TMDb_direct()
|
||||
tmdb_direct.api_key = TMDB_KEY
|
||||
|
||||
tmdb = TMDb(key=TMDB_KEY, language="en-US", region="US")
|
||||
|
||||
TMDB_IMAGE_URL = "https://image.tmdb.org/t/p/original"
|
||||
@ -36,7 +33,7 @@ def lookup_video_from_tmdb(name_or_id: str, kind: str = "movie") -> VideoMetadat
|
||||
video_metadata = VideoMetadata(imdb_id=imdb_id)
|
||||
|
||||
media = None
|
||||
show = None
|
||||
show_data = None
|
||||
if len(tmdb_result.movie_results) > 0:
|
||||
media = Movie().details(tmdb_result.movie_results[0].id)
|
||||
video_metadata.video_type = VideoType.MOVIE.value
|
||||
|
||||
@ -32,7 +32,11 @@ class SeriesDetailView(LoginRequiredMixin, ChartContextMixin, generic.DetailView
|
||||
context_data = super().get_context_data(**kwargs)
|
||||
|
||||
context_data["scrobbles"] = self.object.scrobbles_for_user(user_id)
|
||||
next_episode_id = self.object.last_scrobbled_episode(user_id).next_imdb_id or ""
|
||||
next_episode_id = ""
|
||||
if self.object.last_scrobbled_episode(user_id):
|
||||
next_episode_id = (
|
||||
self.object.last_scrobbled_episode(user_id).next_imdb_id or ""
|
||||
)
|
||||
if self.object.is_episode_playing(user_id):
|
||||
next_episode_id = ""
|
||||
if next_episode_id:
|
||||
|
||||
@ -60,6 +60,7 @@ THEAUDIODB_API_KEY = os.getenv("VROBBLER_THEAUDIODB_API_KEY", "2")
|
||||
PODCASTINDEX_API_KEY = os.getenv("VROBBLER_PODCASTINDEX_API_KEY", "")
|
||||
PODCASTINDEX_API_SECRET = os.getenv("VROBBLER_PODCASTINDEX_API_SECRET", "")
|
||||
TMDB_API_KEY = os.getenv("VROBBLER_TMDB_API_KEY", "")
|
||||
OMDB_API_KEY = os.getenv("VROBBLER_OMDB_API_KEY", "")
|
||||
LASTFM_API_KEY = os.getenv("VROBBLER_LASTFM_API_KEY")
|
||||
LASTFM_SECRET_KEY = os.getenv("VROBBLER_LASTFM_SECRET_KEY")
|
||||
IGDB_CLIENT_ID = os.getenv("VROBBLER_IGDB_CLIENT_ID")
|
||||
|
||||
@ -50,7 +50,11 @@
|
||||
|
||||
<h1 class="d-flex align-items-center gap-2">
|
||||
{% if object.media_type == "Video" %}🎬{% elif object.media_type == "Track" %}🎵{% elif object.media_type == "PodcastEpisode" %}🎙️{% elif object.media_type == "SportEvent" %}⚽{% elif object.media_type == "Book" %}📚{% elif object.media_type == "Paper" %}📄{% elif object.media_type == "VideoGame" %}🎮{% elif object.media_type == "BoardGame" %}🎲{% elif object.media_type == "GeoLocation" %}📍{% elif object.media_type == "Trail" %}🥾{% elif object.media_type == "Beer" %}🍺{% elif object.media_type == "Puzzle" %}🧩{% elif object.media_type == "Food" %}🍔{% elif object.media_type == "Task" %}✅{% elif object.media_type == "WebPage" %}🌐{% elif object.media_type == "LifeEvent" %}🎉{% elif object.media_type == "Mood" %}😊{% elif object.media_type == "BrickSet" %}🧱{% elif object.media_type == "Channel" %}📺{% endif %}
|
||||
{% if object.media_obj.get_absolute_url %}<a href="{{ object.media_obj.get_absolute_url }}">{% endif %}{{ object.media_obj }}{% if object.media_obj.get_absolute_url %}</a>{% endif %}
|
||||
{% if object.media_obj.get_absolute_url %}
|
||||
<a href="{{ object.media_obj.get_absolute_url }}">{% endif %}
|
||||
{{ object.media_obj.title }}
|
||||
{% if object.media_obj.get_absolute_url %}</a>
|
||||
{% endif %}
|
||||
{% if user.is_authenticated and object.media_obj %}
|
||||
<button id="favorite-btn"
|
||||
data-url="{% url 'scrobbles:toggle-favorite' object.media_type object.media_obj.id %}"
|
||||
@ -62,7 +66,15 @@
|
||||
</button>
|
||||
{% endif %}
|
||||
</h1>
|
||||
<h2>{{ object.media_obj.subtitle }}</h2>
|
||||
<p>
|
||||
{% if object.media_type == "SportEvent" %}
|
||||
{% for team in object.media_obj.teams.all %}
|
||||
<img src="{{team.logo.url}}" width=150 />
|
||||
{% endfor %}
|
||||
{% endif %}
|
||||
{% if object.media_type == "Task" and object.logdata.title %}
|
||||
</p>
|
||||
<h2>{{ object.logdata.title }}</h2>
|
||||
{% endif %}
|
||||
<h3 class="text-muted">{{ object.local_timestamp }}</h3>
|
||||
|
||||
@ -1,13 +1,16 @@
|
||||
{% extends "base_detail.html" %}
|
||||
|
||||
{% block title %}{{object}}{% endblock %}
|
||||
{% block title %}{{object.title}}{% endblock %}
|
||||
|
||||
{% block details %}
|
||||
<h2>{{object.subtitle}}</h2>
|
||||
<h2>{{object.league}} {{object.get_event_type_display}}</h2>
|
||||
|
||||
<div class="row">
|
||||
<h2>{{object.tv_series}}</h2>
|
||||
<div class="col-md">
|
||||
{% for team in object.teams.all %}
|
||||
<img src="{{team.logo.url}}" width=150 />
|
||||
{% endfor %}
|
||||
<hr />
|
||||
<h3>Last scrobbles</h3>
|
||||
<div class="table-responsive">
|
||||
<table class="table table-striped table-sm">
|
||||
|
||||
Reference in New Issue
Block a user