[release] Bump to version 48.1

- Generate a report of tracks with mistmatched metadata
- Date parsing failing in eBird imports

AGENTS.md

@@ -14,3 +14,11 @@ ro class method should call the utility function.
 
 Be sure to check pyproject.toml for project defaults. Specifically for black and
 isort expectations.
+
+All tasks live in the PROJECT.org file and include an org ID that is a uuid to make them unique.
+
+In local development, environment variables for various sensitive values live in a .envrc file
+
+The .envrc file can be loaded into a shell environment to allow access to most third party services
+
+Care should be taken when using .envrc that we do not spam services we use in production with requests

PROJECT.org

@@ -17,6 +17,7 @@ tasks, Todoist tasks, web pages I've read and trails I've hiked has turned out
 to be sometimes cathartic and sometimes functional as I try to remember when I
 did a thing.
 
+
 * Features
 ** Beer
 *** Triggers
@@ -86,7 +87,8 @@ fetching and simple saving.
 **** Bookmarklet
 *** Metadata sources
 **** Scraper
-* Backlog [0/12] :vrobbler:project:personal:
+
+* Backlog [0/14] :vrobbler:project:personal:
 ** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
 :PROPERTIES:
 :ID:       702462cf-d54b-48c6-8a7c-78b8de751deb
@@ -498,6 +500,91 @@ needed import celery task. This is how the WebDAV celery task currently works.
 This would also be an opporunity to clean up the code around WebDAV imports
 and make them more re-usable for other import services.
 
+
+** TODO [#A] Add an exception list of artists as a constant that are exempted from splitting :music:artists:metadata:
+:PROPERTIES:
+:ID:       fd86a11a-73ec-470d-b5e3-2d90ba9137c8
+:END:
+
+*** Description
+
+Certain artists like "Simon & Garfunkel" are actually one artist. While we don't want to mess with splitting up
+tracks into featured artists, we should have a "LITERAL_ARTIST_TITLES" constant that can have exceptions like
+this put into it and then we stop trying to pull the artist apart when we run into it.
+
+
+** TODO [#A] Before enriching anything, trust the POST data :feature:scrobbles:metadata:
+:PROPERTIES:
+:ID:       db6b05f8-09f4-49f5-9838-fbacc9fe9cd0
+:END:
+
+*** Description
+
+Both Jellyfin and Mopidy provide a decent amount of metadata when they POST to our webhooks.
+
+In most cases, we should be able to trust this data to created music tracks or videos rather
+than going to third-party services to enrich. Thus, for tracks and videos we should search in
+the local database for imdb_id or musicbrainz_id for the specific content and, if found, not
+enrich further.
+
+If not found, tracks and videos from mopidy and jellyfin should be created as completely as
+possible using only the POST data from the webhooks, tagged the scrobble with "webhook-metadata-only"
+and start the scrobble. A separate celery task should be kicked off to enrich the track or video
+async with the POST data stored in the log["raw_data"] and used by the celery enrichment task
+to go try to enrich the media instance.  Should this enrichment fail, tag the scrobble as "enrichment-failed"
+log a warning and move on.
+
+         
+* Version 48.1 [2/2]
+** DONE [#A] Generate a report of tracks with mistmatched metadata :music:tracks:metadata:
+:PROPERTIES:
+:ID:       684b8cd2-a3c1-4995-ba9e-7abdb02c37f2
+:END:
+
+*** Description
+
+We should have a management command that outputs a CSV file of track IDs where
+the log["raw_data"]["Artist"] (for Jellyfin) or log["raw_data"]["artist"]
+(mopidy) value does not match the Track.artists names.
+
+And we should see the same thing for albums (log["raw_data"]["Album"] or
+log["raw_data]["album"]).
+
+It should output the fields "track_id", "track_artist_name", "track_album_name",
+"raw_artist", "raw_album", "source"
+
+Where source is either Jellyfin or Mopidy based on the keys.
+
+Put the file /tmp/metadata-report.csv by default and overwrite exsiting reports.
+
+The command should also accept a file-path to overide this default.
+
+
+** DONE [#A] Date parsing failing in eBird imports :birds:ebird:importers:
+:PROPERTIES:
+:ID:       72e0254f-39d8-4843-9857-623e0362d77e
+:END:
+
+*** Description
+
+On line 45 in the apps/birds/importer.py file, the import is thorowing this
+error:
+
+#+begin_src python
+ValueError: time data 'Jun 7, 2026, 5:15 PM' does not match format '%B %d, %Y %I:%M %p'
+#+end_src
+
+Historically other files starting on May 24 worked, so I suspect this is a
+problem of the date formatter expecting Long month names and zero-padded days
+and the only sample we had was a three-letter month (May) and days with two
+digites(24 through the 31)
+
+We should also add a "error_log" to the importers so that errors tha occur are
+surfaced, even when 0 successful files were processed. And we should make sure
+all importers do this as well.
+
+
+
 * Version 48.0 [2/2]
 ** DONE [#B] Show team or player images on sport detail and scrobble detail :sports:templates:
 :PROPERTIES:
@@ -545,6 +632,7 @@ up unenriched.
 :ID:       4d50ca2e-f45b-dde8-e3c9-cd84f353b349
 :END:
 
+
 * Version 47.0 [1/1]
 ** DONE [#B] Change sports scrobbling a bit :feature:sports:scrobbles:
 :PROPERTIES:
@@ -960,6 +1048,7 @@ Format should be: "%Y-%m-%dT%H:%M:%S.%fZ"
 :ID:       68a011b2-bb6f-3ba8-2312-5947c41db9ac
 :END:
 
+
 * Version 39.0 [3/3]
 ** DONE [#B] Clean up org-mode tasks metadata :bug:tasks:metadata:
 :PROPERTIES:
@@ -1014,6 +1103,7 @@ If the task is completed, don't touch it.
 :ID:       63dc633c-4382-e6a5-e663-b01871ce86ce
 :END:
 
+
 * Version 38.0 [38/38]
 ** DONE [#A] Fix release flow to be easier to trigger :pyproject:release:tooling:
 :PROPERTIES:
@@ -1405,6 +1495,7 @@ urllib.error.HTTPError: HTTP Error 500: Internal Server Error
 
   This may have already been resolved ... need to just confirm it.
 
+  
 * Version 37.0 [4/4]
 ** DONE [#A] Tasks from org-mode should properly update notes and leave them out of the body :vrobbler:bug:tasks:
 :PROPERTIES:
@@ -1423,11 +1514,13 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
 :PROPERTIES:
 :ID:       e3e49a9a-67d2-8ad8-1114-6f05effee9b7
 :END:
+
 * Version 36.0 [1/1]
 ** DONE [#A] Refactor how videos are scrobbled :vrobbler:vidoes:feature:personal:project:
 :PROPERTIES:
 :ID:       6034a11d-5376-994d-9a4b-e1640e258cfa
 :END:
+
 * Version 35.0 [3/3]
 ** DONE [#B] Add youtube link in place of IMDB on video detail page :vrobbler:feature:videos:personal:project:
 :PROPERTIES:
@@ -1441,6 +1534,7 @@ https://life.lab.unbl.ink/scrobble/e39779c8-62a5-46a6-bdef-fb7662810dc6/start/
 :PROPERTIES:
 :ID:       d1ba1ca1-509b-13a9-1307-b2dc94a2eafe
 :END:
+
 * Version 34.0 [4/4]
 ** DONE [#A] Use bgg-api for BoardGameGeek lookups :vrobbler:feature:boardgames:personal:project:
 :PROPERTIES:
@@ -1500,6 +1594,7 @@ TypeError: can only concatenate str (not "NoneType") to str
 
   A good lesson in using constants for things.
 
+  
 * Version 33.0 [3/3]
 ** DONE [#A] Fix bug where scrobble is_stale only uses seconds not total_seconds :vrobbler:bug:scrobbles:personal:project:
 :PROPERTIES:
@@ -1514,6 +1609,7 @@ TypeError: can only concatenate str (not "NoneType") to str
 :ID:       4955cc34-0882-50db-92f7-f36a95bf57a4
 :END:
 <2025-10-28 Tue>
+
 * Version 32.0 [2/2]
 ** DONE [#B] Save path to reading source on book scrobbles and show it on the detail page :vrobbler:feature:books:personal:project:
 :PROPERTIES:
@@ -1523,6 +1619,7 @@ TypeError: can only concatenate str (not "NoneType") to str
 :PROPERTIES:
 :ID:       9fe09567-11a3-7083-53c7-07458a9591d0
 :END:
+
 * Version 31.0 [3/3]
 ** DONE [#A] Stop comic book webpage scrobbles from overwriting old scrobbles :vrobbler:personal:bug:books:scrobbling:
 :PROPERTIES:
@@ -1536,6 +1633,7 @@ TypeError: can only concatenate str (not "NoneType") to str
 :PROPERTIES:
 :ID:       9a870c05-6d20-0803-d35d-c03fbe1d0ee1
 :END:
+
 * Version 30.0 [3/3]
 ** DONE [#A] Fix readcomicsonline browsing to update pages :vrobbler:books:feature:comicbook:personal:project:scrobbling:
 :PROPERTIES:
@@ -1550,10 +1648,13 @@ TypeError: can only concatenate str (not "NoneType") to str
 :PROPERTIES:
 :ID:       d22cec3f-117f-f203-33a5-efbefa8a5cee
 :END:
+
 * Version 29.0 [1/1]
 ** DONE HOTFIX podcast lookups, final
+
 * Version 28.0 [1/1]
 ** DONE HOTFIX podcast lookups
+
 * Version 27.0 [3/3]
 ** DONE [#A] Fix bug where podcast scrobbling creates duplicate Podcast :project:vrobbler:scrobbling:podcasts:bug:personal:
 :PROPERTIES:
@@ -1594,6 +1695,7 @@ it's annoying.
   actually be much more reliable than the current state of the podcast lookup
   which depends on the file to be name properly.
 
+  
 * Version 26.0 [3/3]
 ** DONE Clean up templates for scrobble details :vrobbler:personal:bug:templates:
 :PROPERTIES:
@@ -1607,6 +1709,7 @@ it's annoying.
 :PROPERTIES:
 :ID:       c03a38ce-b337-f4fa-adba-aee08d4329f5
 :END:
+
 * Version 25.0 [3/3]
 ** DONE Add basic food templates and fix urls :food:vrobbler:personal:project:bug:urls:
 :PROPERTIES:
@@ -1620,6 +1723,7 @@ it's annoying.
 :PROPERTIES:
 :ID:       7debfbaf-cdd8-f49b-57ff-804bfe7c9236
 :END:
+
 * Version 24.0 [2/2]
 ** DONE Clean up logdata for various media :personal:feature:project:vrobbler:logdata:
 :PROPERTIES:
@@ -1630,6 +1734,7 @@ it's annoying.
 :ID:       1a1c0aa6-0313-c8be-1676-5d6adddef0a4
 :END:
 
+
 * Version 23.0 [3/3]
 ** DONE Add dynamic forms for LogData classes :personal:feature:vrobbler:project:forms:logdata:
 :PROPERTIES:
@@ -1640,16 +1745,19 @@ it's annoying.
 :PROPERTIES:
 :ID:       99f6bd77-dc8f-6ed1-0321-32a52c944264
 :END:
+
 * Version 19.0 [1/1]
 ** DONE Add periodic check for mood :vrobbler:feature:moods:personal:project:
 :PROPERTIES:
 :ID:       55404488-c69f-0dd5-838e-1d1e15c873eb
 :END:
+
 * Version 18.7 [1/1]
 ** DONE Use the timezone history log to fix old Scrobbles that fall into those timezone blocks :vrobbler:chore:scrobbles:project:personal:
 :PROPERTIES:
 :ID:       9d055ac1-584b-20c8-7ad9-9ce36b329dc7
 :END:
+
 * Version 18.4 [2/2]
 ** DONE Track timezone changes for profiles :vrobbler:feature:profiles:personal:project:
 :PROPERTIES:
@@ -1663,11 +1771,13 @@ it's annoying.
 - Note taken on [2025-07-20 Sun 16:21]
 
   This thing is kicking my butt. As it stands it works, but the scrobbles are not assigned to the tracks properly.
+  
 * Version 18.3 [1/1]
 ** DONE Add timezone awarness to IMAP importer :personal:project:vrobbler:feature:importer:imap:timezones:
 :PROPERTIES:
 :ID:       05837b7c-96aa-6190-3678-e2ae7c7cac75
 :END:
+
 * Version 18 [4/4]
 ** DONE Condense tracks of the same title by the same artist with multiple albums :vrobbler:feature:music:project:personal:
 :PROPERTIES:
@@ -1789,6 +1899,7 @@ it's annoying.
 :PROPERTIES:
 :ID:       23f485e3-988c-6198-c79d-91fdf92f001c
 :END:
+
 * Version 17.0 [6/6]
 ** DONE [#A] Fix bug in new task label lookup for Emacs/Org-mode :vrobbler:bug:tasks:
 :PROPERTIES:
@@ -1852,6 +1963,7 @@ Not sure if the problem is in my Emacs hook sending or Vrobbler itself.
 :PROPERTIES:
 :ID:       df58f8d0-fa4a-2037-c7d7-e5388c239042
 :END:
+
 * Version 0.16.0 [19/19]
 ** DONE [#A] Jellyfin, bandcamp tracks from Mopidy create duplicate music tracks :bug:scrobbling:music:
 :PROPERTIES:
@@ -1957,6 +2069,7 @@ out using that.
 ** DONE Fix bug in Jellyfin scrobbles that spam more scrobbles after completion :scrobbling:videos:bug:
 This was fixed a while ago, but there's a new manifested bug. Going to create a
 separate bug tracking ticket for that.
+
 * Version 0.11.4 [9/9]
 ** DONE Add rudimentary video game scrobbling :improvement:content:videogames:
 CLOSED: [2023-03-07 Tue 11:11]

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "vrobbler"
-version = "48.0"
+version = "48.1"
 description = ""
 authors = ["Colin Powell <colin@unbl.ink>"]
 

tests/birds_tests/test_importer.py

@@ -1,3 +1,4 @@
+import tempfile
 from datetime import timedelta
 
 import pytest
@@ -128,6 +129,23 @@ class TestBirdingCSVImportModel:
         assert imp.import_type == "Birding CSV"
         assert "Birding" in str(imp)
 
+    def test_record_error(self, db, user):
+        imp = BirdingCSVImport.objects.create(user=user)
+        assert imp.error_log is None
+        imp.record_error("test error")
+        imp.refresh_from_db()
+        assert imp.error_log is not None
+        assert "test error" in imp.error_log
+
+    def test_record_error_appends(self, db, user):
+        imp = BirdingCSVImport.objects.create(user=user)
+        imp.record_error("first error")
+        imp.record_error("second error")
+        imp.refresh_from_db()
+        assert imp.error_log.count("\n") == 1
+        assert "first error" in imp.error_log
+        assert "second error" in imp.error_log
+
     @pytest.mark.django_db(transaction=True)
     def test_process_via_model(self, user, birding_csv_file):
         imp = BirdingCSVImport.objects.create(user=user)
@@ -137,3 +155,35 @@ class TestBirdingCSVImportModel:
         imp.refresh_from_db()
         assert imp.process_count == 1
         assert imp.processed_finished is not None
+
+    def test_record_error_on_bad_csv(self, user, db):
+        content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
+Canada Goose,6,Test Park,Stationary,"Bad Date",4:15 PM,9 minute(s),,,4,true,4 species,
+"""
+        with tempfile.NamedTemporaryFile(
+            mode="w", suffix=".csv", delete=False, encoding="utf-8"
+        ) as f:
+            f.write(content)
+            file_path = f.name
+
+        errors = []
+        scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
+        assert len(scrobbles) == 0
+        assert len(errors) == 1
+        assert "Could not parse date/time" in errors[0]
+
+    def test_record_error_on_bad_location(self, user, db):
+        content = """Species,Count,Location,Observation Type,Observation Date,Start Time,Duration,Distance,Area,Party Size,Complete Checklist,# of species,Details
+Canada Goose,6,,Stationary,"May 10, 2026",4:15 PM,9 minute(s),,,4,true,4 species,
+"""
+        with tempfile.NamedTemporaryFile(
+            mode="w", suffix=".csv", delete=False, encoding="utf-8"
+        ) as f:
+            f.write(content)
+            file_path = f.name
+
+        errors = []
+        scrobbles = import_birding_csv(file_path, user.id, record_error=errors.append)
+        assert len(scrobbles) == 0
+        assert len(errors) == 1
+        assert "Skipping rows with no location" in errors[0]

vrobbler/apps/birds/admin.py

@@ -26,5 +26,5 @@ class BirdingLocationAdmin(admin.ModelAdmin):
 @admin.register(BirdingCSVImport)
 class BirdingCSVImportAdmin(admin.ModelAdmin):
     date_hierarchy = "created"
-    list_display = ("uuid", "process_count", "processed_finished", "processing_started")
+    list_display = ("uuid", "process_count", "processed_finished", "processing_started", "error_log")
     ordering = ("-created",)

vrobbler/apps/birds/importer.py

@@ -2,7 +2,9 @@ import csv
 import logging
 import re
 from collections import defaultdict
-from datetime import datetime, timedelta
+from datetime import timedelta
+
+from dateutil import parser
 
 from django.contrib.auth import get_user_model
 
@@ -35,11 +37,12 @@ def parse_coords(location_str):
 
 def parse_timestamp(date_str, time_str):
     try:
-        dt = datetime.strptime(f"{date_str} {time_str}", "%B %d, %Y %I:%M %p")
+        dt_str = f"{date_str} {time_str}".strip()
+        dt = parser.parse(dt_str)
         return dt
     except (ValueError, TypeError):
         try:
-            dt = datetime.strptime(date_str, "%B %d, %Y")
+            dt = parser.parse(date_str)
             return dt
         except (ValueError, TypeError):
             logger.warning(f"Could not parse date/time: {date_str} {time_str}")
@@ -61,7 +64,7 @@ def parse_int(value):
         return None
 
 
-def import_birding_csv(file_path, user_id):
+def import_birding_csv(file_path, user_id, record_error=None):
     user = User.objects.get(id=user_id)
     new_scrobbles = []
 
@@ -80,11 +83,17 @@ def import_birding_csv(file_path, user_id):
 
     for (location_str, date_str, time_str), sighting_rows in groups.items():
         if not location_str:
-            logger.warning("Skipping rows with no location")
+            msg = "Skipping rows with no location"
+            logger.warning(msg)
+            if record_error:
+                record_error(msg)
             continue
 
         timestamp = parse_timestamp(date_str, time_str)
         if not timestamp:
+            msg = f"Could not parse date/time: {date_str} {time_str}"
+            if record_error:
+                record_error(msg)
             continue
 
         timestamp = user.profile.get_timestamp_with_tz(timestamp)

vrobbler/apps/birds/migrations/0003_birdingcsvimport_error_log.py

@@ -0,0 +1,18 @@
+# Generated by Django 4.2.29 on 2026-06-08 14:57
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("birds", "0002_birdingcsvimport"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="birdingcsvimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+    ]

vrobbler/apps/birds/models.py

@@ -216,6 +216,7 @@ class BirdingCSVImport(TimeStampedModel):
     processed_finished = models.DateTimeField(**BNULL)
     process_log = models.TextField(**BNULL)
     process_count = models.IntegerField(**BNULL)
+    error_log = models.TextField(**BNULL)
     csv_file = models.FileField(upload_to="birding-csv-uploads/", **BNULL)
 
     class Meta:
@@ -269,6 +270,14 @@ class BirdingCSVImport(TimeStampedModel):
         self.process_count = len(scrobbles)
         self.save(update_fields=["process_log", "process_count"])
 
+    def record_error(self, error_message):
+        log_line = f"{timezone.now().isoformat()}: {error_message}"
+        if self.error_log:
+            self.error_log += "\n" + log_line
+        else:
+            self.error_log = log_line
+        self.save(update_fields=["error_log"])
+
     def scrobbles(self):
         from scrobbles.models import Scrobble
 
@@ -287,6 +296,13 @@ class BirdingCSVImport(TimeStampedModel):
         from birds.importer import import_birding_csv
 
         self.mark_started()
-        scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = import_birding_csv(
+                self.upload_file_path, self.user_id, record_error=self.record_error
+            )
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()

vrobbler/apps/music/management/commands/report_mismatched_metadata.py

@@ -0,0 +1,127 @@
+import csv
+import logging
+
+from django.core.management.base import BaseCommand
+
+logger = logging.getLogger(__name__)
+
+
+def _get_source(raw_data):
+    if "Artist" in raw_data:
+        return "Jellyfin"
+    if "artist" in raw_data:
+        return "Mopidy"
+    return None
+
+
+def _get_raw_values(raw_data, source):
+    if source == "Jellyfin":
+        return raw_data.get("Artist", ""), raw_data.get("Album", "")
+    return raw_data.get("artist", ""), raw_data.get("album", "")
+
+
+def _normalize(name):
+    return name.strip().casefold()
+
+
+def _artist_mismatch(raw_artist, track_artist_names):
+    if not raw_artist or not track_artist_names:
+        return False
+    track_names = [_normalize(n) for n in track_artist_names.split(" / ")]
+    raw = _normalize(raw_artist)
+    if raw in track_names:
+        return False
+    if raw == _normalize(track_artist_names):
+        return False
+    return True
+
+
+def _album_mismatch(raw_album, track_album_name):
+    if not raw_album or not track_album_name:
+        return False
+    return _normalize(raw_album) != _normalize(track_album_name)
+
+
+class Command(BaseCommand):
+    help = (
+        "Outputs a CSV of track IDs where raw metadata from scrobble logs "
+        "does not match the track's stored artists or album"
+    )
+
+    def add_arguments(self, parser):
+        parser.add_argument(
+            "--file-path",
+            type=str,
+            default="/tmp/metadata-report.csv",
+            help="Output CSV file path (default: /tmp/metadata-report.csv)",
+        )
+
+    def handle(self, *args, **options):
+        from scrobbles.models import Scrobble
+
+        file_path = options["file_path"]
+
+        qs = (
+            Scrobble.objects.filter(media_type=Scrobble.MediaType.TRACK)
+            .exclude(log__isnull=True)
+            .exclude(log={})
+            .select_related("track__album")
+            .prefetch_related("track__artists")
+            .iterator()
+        )
+
+        rows = []
+        for scrobble in qs:
+            track = scrobble.track
+            if not track:
+                continue
+
+            raw_data = scrobble.log.get("raw_data")
+            if not raw_data:
+                continue
+
+            source = _get_source(raw_data)
+            if not source:
+                continue
+
+            raw_artist, raw_album = _get_raw_values(raw_data, source)
+            if not raw_artist and not raw_album:
+                continue
+
+            track_artist_names = " / ".join(
+                track.artists.all().values_list("name", flat=True)
+            )
+            track_album_name = track.album.name if track.album else ""
+
+            if _artist_mismatch(raw_artist, track_artist_names) or _album_mismatch(
+                raw_album, track_album_name
+            ):
+                rows.append(
+                    {
+                        "track_id": track.id,
+                        "track_artist_name": track_artist_names,
+                        "track_album_name": track_album_name,
+                        "raw_artist": raw_artist,
+                        "raw_album": raw_album,
+                        "source": source,
+                    }
+                )
+
+        fieldnames = [
+            "track_id",
+            "track_artist_name",
+            "track_album_name",
+            "raw_artist",
+            "raw_album",
+            "source",
+        ]
+        with open(file_path, "w", newline="") as f:
+            writer = csv.DictWriter(f, fieldnames=fieldnames)
+            writer.writeheader()
+            writer.writerows(rows)
+
+        self.stdout.write(
+            self.style.SUCCESS(
+                f"Wrote {len(rows)} mismatched track(s) to {file_path}"
+            )
+        )

vrobbler/apps/scrobbles/admin.py

@@ -56,6 +56,7 @@ class ImportBaseAdmin(admin.ModelAdmin):
         "process_count",
         "processed_finished",
         "processing_started",
+        "error_log",
     )
     ordering = ("-created",)
 

vrobbler/apps/scrobbles/migrations/0090_audioscrobblertsvimport_error_log_and_more.py

@@ -0,0 +1,53 @@
+# Generated by Django 4.2.29 on 2026-06-08 14:57
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("scrobbles", "0089_favoritemedia"),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name="audioscrobblertsvimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="bgstatsimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="ebirdcsvimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="koreaderimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="lastfmimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="retroarchimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="scalecsvimport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+        migrations.AddField(
+            model_name="trailgpximport",
+            name="error_log",
+            field=models.TextField(blank=True, null=True),
+        ),
+    ]

vrobbler/apps/scrobbles/models.py

@@ -74,6 +74,7 @@ class BaseFileImportMixin(TimeStampedModel):
     processed_finished = models.DateTimeField(**BNULL)
     process_log = models.TextField(**BNULL)
     process_count = models.IntegerField(**BNULL)
+    error_log = models.TextField(**BNULL)
 
     class Meta:
         abstract = True
@@ -158,6 +159,14 @@ class BaseFileImportMixin(TimeStampedModel):
         self.process_count = len(scrobbles)
         self.save(update_fields=["process_log", "process_count"])
 
+    def record_error(self, error_message):
+        log_line = f"{timezone.now().isoformat()}: {error_message}"
+        if self.error_log:
+            self.error_log += "\n" + log_line
+        else:
+            self.error_log = log_line
+        self.save(update_fields=["error_log"])
+
     @property
     def upload_file_path(self):
         raise NotImplementedError
@@ -213,9 +222,14 @@ class KoReaderImport(BaseFileImportMixin):
             return
 
         self.mark_started()
-        scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class AudioScrobblerTSVImport(BaseFileImportMixin):
@@ -255,10 +269,14 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
             return
 
         self.mark_started()
-
-        scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class ScaleCSVImport(BaseFileImportMixin):
@@ -297,9 +315,14 @@ class ScaleCSVImport(BaseFileImportMixin):
             return
 
         self.mark_started()
-        scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = import_scale_csv(self.upload_file_path, self.user.id)
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class TrailGPXImport(BaseFileImportMixin):
@@ -337,11 +360,16 @@ class TrailGPXImport(BaseFileImportMixin):
             return
 
         self.mark_started()
-        scrobbles = import_trail_gpx(
-            self.upload_file_path, self.user.id, self.original_filename
-        )
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = import_trail_gpx(
+                self.upload_file_path, self.user.id, self.original_filename
+            )
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class LastFmImport(BaseFileImportMixin):
@@ -391,11 +419,14 @@ class LastFmImport(BaseFileImportMixin):
                 last_processed = last_import.processed_finished
 
         self.mark_started()
-
-        scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
-
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = lastfm.import_from_lastfm(last_processed, time_to=time_to)
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class RetroarchImport(BaseFileImportMixin):
@@ -426,43 +457,48 @@ class RetroarchImport(BaseFileImportMixin):
             logger.info(f"You told me to force import from Retroarch")
 
         self.mark_started()
+        scrobbles = None
+        try:
+            if self.lrtl_file:
+                import os
+                import tempfile
+                import zipfile
 
-        if self.lrtl_file:
-            import os
-            import tempfile
-            import zipfile
+                tmpdir = tempfile.mkdtemp()
+                try:
+                    zip_path = os.path.join(tmpdir, "archive.zip")
+                    with open(zip_path, "wb") as f:
+                        f.write(self.lrtl_file.read())
+                    with zipfile.ZipFile(zip_path, "r") as zf:
+                        zf.extractall(tmpdir)
+                    os.unlink(zip_path)
+                    scrobbles = retroarch.import_retroarch_lrtl_files(
+                        tmpdir + "/",
+                        self.user.id,
+                    )
+                finally:
+                    import shutil
+
+                    shutil.rmtree(tmpdir, ignore_errors=True)
+            else:
+                if not self.user.profile.retroarch_path:
+                    logger.info(
+                        "Trying to import Retroarch logs, but user has no retroarch_path configured"
+                    )
+                    self.mark_finished()
+                    return
 
-            tmpdir = tempfile.mkdtemp()
-            try:
-                zip_path = os.path.join(tmpdir, "archive.zip")
-                with open(zip_path, "wb") as f:
-                    f.write(self.lrtl_file.read())
-                with zipfile.ZipFile(zip_path, "r") as zf:
-                    zf.extractall(tmpdir)
-                os.unlink(zip_path)
                 scrobbles = retroarch.import_retroarch_lrtl_files(
-                    tmpdir + "/",
+                    self.user.profile.retroarch_path,
                     self.user.id,
                 )
-            finally:
-                import shutil
 
-                shutil.rmtree(tmpdir, ignore_errors=True)
-        else:
-            if not self.user.profile.retroarch_path:
-                logger.info(
-                    "Tying to import Retroarch logs, but user has no retroarch_path configured"
-                )
-                self.mark_finished()
-                return
-
-            scrobbles = retroarch.import_retroarch_lrtl_files(
-                self.user.profile.retroarch_path,
-                self.user.id,
-            )
-
-        self.record_log(scrobbles)
-        self.mark_finished()
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class BGStatsImport(BaseFileImportMixin):
@@ -499,17 +535,21 @@ class BGStatsImport(BaseFileImportMixin):
             return
 
         self.mark_started()
+        try:
+            import json
 
-        import json
+            from scrobbles.scrobblers import email_scrobble_board_game
 
-        from scrobbles.scrobblers import email_scrobble_board_game
+            with open(self.upload_file_path, "r", encoding="utf-8") as f:
+                parsed_json = json.load(f)
+            scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
 
-        with open(self.upload_file_path, "r", encoding="utf-8") as f:
-            parsed_json = json.load(f)
-        scrobbles = email_scrobble_board_game(parsed_json, self.user_id)
-
-        self.record_log(scrobbles)
-        self.mark_finished()
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class EBirdCSVImport(BaseFileImportMixin):
@@ -554,9 +594,16 @@ class EBirdCSVImport(BaseFileImportMixin):
             return
 
         self.mark_started()
-        scrobbles = import_birding_csv(self.upload_file_path, self.user_id)
-        self.record_log(scrobbles)
-        self.mark_finished()
+        try:
+            scrobbles = import_birding_csv(
+                self.upload_file_path, self.user_id, record_error=self.record_error
+            )
+            self.record_log(scrobbles)
+        except Exception as e:
+            self.record_error(f"Import failed: {e}")
+            logger.exception(f"Import failed for {self}")
+        finally:
+            self.mark_finished()
 
 
 class ScrobbleQuerySet(models.QuerySet):