[release] Bump to version 51.1

- Fix scrobbling comic books

PROJECT.org

@@ -88,7 +88,7 @@ fetching and simple saving.
 *** Metadata sources
 **** Scraper
 
-* Backlog [0/15] :vrobbler:project:personal:
+* Backlog [0/14] :vrobbler:project:personal:
 ** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
 :PROPERTIES:
 :ID:       702462cf-d54b-48c6-8a7c-78b8de751deb
@@ -451,32 +451,6 @@ https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-li
 :PROPERTIES:
 :ID:       79f867c3-1288-4143-b6bf-2a452983ee9f
 :END:
-** TODO [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
-:PROPERTIES:
-:ID:       79758cba-a440-48b6-a637-efb88827acf2
-:END:
-
-- Note taken on [2025-09-25 Thu 10:37] \\
-
-  This may already be fixed ... need to check.
-
-- Note taken on [2025-02-25 12:34] \\
-
-  The page data has the canonical date something was read in it, but it seems
-  to be an hour off. I traced this back to being off during DST, so we just need
-  the importer to be aware of whether a user is using DST or not and roll back
-  an hour for part of the year. Also, we'd need to adjust any old scrobbles that
-  took place with DST off to roll them back by an hour.
-      
-  
-*** Description
-
-This is a long-standing problem when daylight saving time takes effect. Time is
-manually set on a KoReader device (or at least, always saved in local time). So
-whatever time KoReader reports, we need to know, given the date and the user
-profile's historic timezone, how many hours to adjust the KoReader time to get
-to GMT to save it in the database.
-
 ** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
 :PROPERTIES:
 :ID:       b1426d92-2feb-4d15-9738-d5b7b0594f96
@@ -544,6 +518,134 @@ log a warning and move on.
 We should have a global view `/favorites/` that shows the logged in users's
 favorited media objects.
 
+* Version 51.1 [1/1]
+** DONE [#A] Fix scrobbling comic books :books:scrobbles:bug:
+:PROPERTIES:
+:ID:       8dfbff19-3fa4-f3b8-21c7-7a416498000c
+:END:
+
+*** Description
+
+At some point logdata and log got confused, and now when you try
+to scrobble a comic book, it just throws errors. We should look
+into where the confusion happened and fix it.
+
+
+* Version 51.0 [3/3]
+** DONE [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
+:PROPERTIES:
+:ID:       79758cba-a440-48b6-a637-efb88827acf2
+:END:
+
+- Note taken on [2025-09-25 Thu 10:37] \\
+
+  This may already be fixed ... need to check.
+
+- Note taken on [2025-02-25 12:34] \\
+
+  The page data has the canonical date something was read in it, but it seems
+  to be an hour off. I traced this back to being off during DST, so we just need
+  the importer to be aware of whether a user is using DST or not and roll back
+  an hour for part of the year. Also, we'd need to adjust any old scrobbles that
+  took place with DST off to roll them back by an hour.
+      
+  
+*** Description
+
+This is a long-standing problem when daylight saving time takes effect. Time is
+manually set on a KoReader device (or at least, always saved in local time). So
+whatever time KoReader reports, we need to know, given the date and the user
+profile's historic timezone, how many hours to adjust the KoReader time to get
+to GMT to save it in the database.
+
+** DONE [#A] Fix book scrobbles where page_data is a list :bug:books:scrobbles:
+:PROPERTIES:
+:ID:       35b323fa-ccc0-4009-b227-8a0f12bbd469
+:END:
+
+*** Description
+
+Comic scrobbling is currently kind of janky. Most of the problems boil down to them
+storing saved page data in a list of dicts rather than a dict keyed off of the page number that
+was read.
+
+We need to adjust comic scrobbling to use a dict of pages keyed off the page number, and also
+write a migration script that runs as a data migration to update any book scrobbles that may
+have page_data as a list.
+
+*** Example data
+#+begin_src python
+{"notes": null, "page_end": 27, "page_data": [{"notes": null, "end_ts": 1771815895, "duration": 14, "start_ts": 1771815881, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815908, "duration": 13, "start_ts": 1771815895, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815913, "duration": 5, "start_ts": 1771815908, "description": null, "page_number": "2"}, {"notes": null, "end_ts": 1771815933, "duration": 20, "start_ts": 1771815913, "description": null, "page_number": "3"}, {"notes": null, "end_ts": 1771815945, "duration": 12, "start_ts": 1771815933, "description": null, "page_number": "4"}, {"notes": null, "end_ts": 1771815983, "duration": 38, "start_ts": 1771815945, "description": null, "page_number": "5"}, {"notes": null, "end_ts": 1771816007, "duration": 24, "start_ts": 1771815983, "description": null, "page_number": "6"}, {"notes": null, "end_ts": 1771816011, "duration": 4, "start_ts": 1771816007, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816013, "duration": 2, "start_ts": 1771816011, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816052, "duration": 39, "start_ts": 1771816013, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816127, "duration": 75, "start_ts": 1771816052, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816134, "duration": 7, "start_ts": 1771816127, "description": null, "page_number": "9"}, {"notes": null, "end_ts": 1771816196, "duration": 62, "start_ts": 1771816134, "description": null, "page_number": "10"}, {"notes": null, "end_ts": 1771816262, "duration": 66, "start_ts": 1771816196, "description": null, "page_number": "11"}, {"notes": null, "end_ts": 1771816293, "duration": 31, "start_ts": 1771816262, "description": null, "page_number": "12"}, {"notes": null, "end_ts": 1771816322, "duration": 29, "start_ts": 1771816293, "description": null, "page_number": "13"}, {"notes": null, "end_ts": 1771816330, "duration": 8, "start_ts": 1771816322, "description": null, "page_number": "14"}, {"notes": null, "end_ts": 1771816368, "duration": 38, "start_ts": 1771816330, "description": null, "page_number": "15"}, {"notes": null, "end_ts": 1771816388, "duration": 20, "start_ts": 1771816368, "description": null, "page_number": "16"}, {"notes": null, "end_ts": 1771816482, "duration": 94, "start_ts": 1771816388, "description": null, "page_number": "17"}, {"notes": null, "end_ts": 1771816550, "duration": 68, "start_ts": 1771816482, "description": null, "page_number": "18"}, {"notes": null, "end_ts": 1771816567, "duration": 17, "start_ts": 1771816550, "description": null, "page_number": "19"}, {"notes": null, "end_ts": 1771816586, "duration": 19, "start_ts": 1771816567, "description": null, "page_number": "20"}, {"notes": null, "end_ts": 1771816597, "duration": 11, "start_ts": 1771816586, "description": null, "page_number": "21"}, {"notes": null, "end_ts": 1771816616, "duration": 19, "start_ts": 1771816597, "description": null, "page_number": "22"}, {"notes": null, "end_ts": 1771816640, "duration": 24, "start_ts": 1771816616, "description": null, "page_number": "23"}, {"notes": null, "end_ts": 1771816690, "duration": 50, "start_ts": 1771816640, "description": null, "page_number": "24"}, {"notes": null, "end_ts": 1771816702, "duration": 12, "start_ts": 1771816690, "description": null, "page_number": "25"}, {"notes": null, "end_ts": 1771816823, "duration": 121, "start_ts": 1771816702, "description": null, "page_number": "26"}, {"notes": null, "end_ts": null, "duration": null, "start_ts": 1771816823, "description": null, "page_number": "27"}], "page_start": 1, "pages_read": 27, "resume_url": null, "description": null, "koreader_hash": null, "long_play_complete": false}
+
+#+end_src
+** DONE [#A] Lichess imports do not set default visbility :boardgames:bug:importers:lichess:
+:PROPERTIES:
+:ID:       a78f7c72-a20a-8db2-cde0-d92a731d4fba
+:END:
+
+* Version 50.2 [2/2]
+** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers:
+:PROPERTIES:
+:ID:       b50141fd-cda6-4a3a-afd3-cd8499e7523e
+:END:
+
+*** Description
+
+When you read a single page in a book in Koreader and try to import it, the scrobble is only
+created the day after, not on the day of the reading.
+
+** DONE [#A] Fix bugs in celery tasks causing imports to fail :bug:celery:tasks:
+:PROPERTIES:
+:ID:       d1171cb0-6413-44b8-a68a-019a4d2fb285
+:END:
+
+*** Description
+
+Seems like all celery tasks are failing for different reasons except the chart
+updates.
+
+*** Errors
+****  scrobbles.tasks.send_notification_for_in_progress
+#+begin_src bash
+KeyError: 'track'
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
+return self.cursor.execute(sql, params)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+psycopg2.errors.UndefinedColumn: column music_track.artist_id does not exist
+LINE 1: ..."."title", "music_track"."base_run_time_seconds", "music_tra...
+^
+HINT: Perhaps you meant to reference the column "music_track.artist_fk_id".
+#+end_src
+
+**** scrobbles.tasks.import_from_webdav_all_users
+#+begin_src bash
+File "/usr/local/lib/python3.11/site-packages/vrobbler/apps/scrobbles/importers/webdav.py", line 166, in scan_webdav_for_koreader
+if last_import and last_import.webdav_etag and remote_etag:
+^^^^^^^^^^^^^^^^^^^^^^^
+AttributeError: 'KoReaderImport' object has no attribute 'webdav_etag'
+#+end_src
+
+**** scrobbles.tasks.process_bgstats_import
+#+begin_src bash
+
+Traceback (most recent call last):
+File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
+return self.cursor.execute(sql, params)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+psycopg2.errors.NotNullViolation: null value in column "visibility" of relation "scrobbles_scrobble" violates not-null constraint
+DETAIL: Failing row contains (374463, 2026-06-11 13:27:06.528319+00, 2026-06-11 13:27:06.52834+00, 2026-06-11 13:17:34+00, 180, f, f, BG Stats, 1, null, t, {"players": [{"new": false, "win": false, "rank": 0, "role": "",..., null, null, null, 8e73ceec-b731-4623-9637-712bbf9f76ce, null, null, null, null, , null, BoardGame, 324, null, null, America/New_York, null, null, , null, null, , null, null, null, null, null, null, null, null, null, null, null).
+#+end_src
+
+* Version 50.1 [1/1]
+** DONE [#B] Fix bug in charts where only #1 is displayed :charts:templates:
+:PROPERTIES:
+:ID:       7136dffb-e6b7-184b-48ac-bb09bae0b0f0
+:END:
+
 * Version 50.0 [2/2]
 ** DONE [#A] Allow updating all a user's scrobble visibility at once :scrobbles:sharing:feature:
 :PROPERTIES:

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "vrobbler"
-version = "50.0"
+version = "51.1"
 description = ""
 authors = ["Colin Powell <colin@unbl.ink>"]
 

vrobbler/apps/boardgames/sources/lichess.py

@@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
             "source": "Lichess",
             "timezone": user.profile.timezone,
             "log": log_data,
+            "visibility": "private",
         }
         if commit:
             Scrobble.objects.create(**scrobble_dict)

vrobbler/apps/books/koreader.py

@@ -1,9 +1,8 @@
 import logging
 import re
 import sqlite3
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 from enum import Enum
-from zoneinfo import ZoneInfo
 
 import requests
 from books.constants import BOOKS_TITLES_TO_IGNORE
@@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
     return book_id_map
 
 
-def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
+def build_page_data(page_rows: list, book_map: dict) -> dict:
     """Given rows of page data from KoReader, parse each row and build
     scrobbles for our user, loading the page data into the page_data
     field on the scrobble instance.
@@ -187,18 +186,20 @@ def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict:
             book_ids_not_found.append(koreader_book_id)
             continue
 
-        if "pages" not in book_map[koreader_book_id].keys():
-            book_map[koreader_book_id]["pages"] = {}
+        book_map[koreader_book_id].setdefault("pages", [])
 
         page_number = page_row[KoReaderPageStatColumn.PAGE.value]
         duration = page_row[KoReaderPageStatColumn.DURATION.value]
         start_ts = page_row[KoReaderPageStatColumn.START_TIME.value]
 
-        book_map[koreader_book_id]["pages"][page_number] = {
-            "duration": duration,
-            "start_ts": start_ts,
-            "end_ts": start_ts + duration,
-        }
+        book_map[koreader_book_id]["pages"].append(
+            {
+                "page_number": page_number,
+                "duration": duration,
+                "start_ts": start_ts,
+                "end_ts": start_ts + duration,
+            }
+        )
     if book_ids_not_found:
         logger.info(f"Found pages for books not in file: {set(book_ids_not_found)}")
     return book_map
@@ -225,11 +226,12 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
         pages_processed = 0
         total_pages_read = len(book_map[koreader_book_id]["pages"])
         ordered_pages = sorted(
-            book_map[koreader_book_id]["pages"].items(),
-            key=lambda x: x[1]["start_ts"],
+            book_map[koreader_book_id]["pages"],
+            key=lambda x: x["start_ts"],
         )
 
-        for cur_page_number, stats in ordered_pages:
+        for stats in ordered_pages:
+            page_number = stats["page_number"]
             pages_processed += 1
 
             seconds_from_last_page = 0
@@ -243,12 +245,14 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
             )
 
             end_of_reading = pages_processed == total_pages_read
-            big_jump_to_this_page = (cur_page_number - last_page_number) > 10
+            big_jump_to_this_page = (page_number - last_page_number) > 10
             is_session_gap = seconds_from_last_page > SESSION_GAP_SECONDS
             if (is_session_gap and not big_jump_to_this_page) or end_of_reading:
                 should_create_scrobble = True
 
             if should_create_scrobble:
+                if not scrobble_page_data:
+                    scrobble_page_data[page_number] = stats
                 scrobble_page_data = dict(
                     sorted(
                         scrobble_page_data.items(),
@@ -270,19 +274,16 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
                     continue
 
                 timestamp = user.profile.get_timestamp_with_tz(
-                    datetime.fromtimestamp(int(first_page.get("start_ts")))
+                    datetime.fromtimestamp(
+                        int(first_page.get("start_ts"))
+                    )
                 )
                 stop_timestamp = user.profile.get_timestamp_with_tz(
-                    datetime.fromtimestamp(int(last_page.get("end_ts")))
+                    datetime.fromtimestamp(
+                        int(last_page.get("end_ts"))
+                    )
                 )
 
-                # Adjust for Daylight Saving Time
-                # if timestamp.dst() == timedelta(
-                #    0
-                # ) or stop_timestamp.dst() == timedelta(0):
-                #    timestamp = timestamp - timedelta(hours=1)
-                #    stop_timestamp = stop_timestamp - timedelta(hours=1)
-
                 scrobble = Scrobble.objects.filter(
                     timestamp=timestamp,
                     book_id=book_id,
@@ -291,7 +292,7 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
 
                 if not scrobble:
                     logger.info(
-                        f"Queueing scrobble for {book_id}, page {cur_page_number}"
+                        f"Queueing scrobble for {book_id}, page {page_number}"
                     )
                     log_data = {
                         "koreader_hash": book_dict.get("hash"),
@@ -324,10 +325,84 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
                 scrobble_page_data = {}
 
             # We accumulate pages for the scrobble until we should create a new one
-            scrobble_page_data[cur_page_number] = stats
+            scrobble_page_data[page_number] = stats
 
-            last_page_number = cur_page_number
+            last_page_number = page_number
             prev_page_stats = stats
+
+        # Handle leftover pages that never triggered a session gap
+        if scrobble_page_data:
+            scrobble_page_data = dict(
+                sorted(
+                    scrobble_page_data.items(),
+                    key=lambda x: x[1]["start_ts"],
+                )
+            )
+            try:
+                first_page = scrobble_page_data.get(
+                    list(scrobble_page_data.keys())[0]
+                )
+                last_page = scrobble_page_data.get(
+                    list(scrobble_page_data.keys())[-1]
+                )
+            except IndexError:
+                logger.error(
+                    "Could not process book, no page data found",
+                    extra={"scrobble_page_data": scrobble_page_data},
+                )
+                continue
+
+            playback_position_seconds = sum(
+                p["duration"] for p in scrobble_page_data.values()
+            )
+
+            timestamp = user.profile.get_timestamp_with_tz(
+                datetime.fromtimestamp(
+                    int(first_page.get("start_ts"))
+                )
+            )
+            stop_timestamp = user.profile.get_timestamp_with_tz(
+                datetime.fromtimestamp(
+                    int(last_page.get("end_ts"))
+                )
+            )
+
+            scrobble = Scrobble.objects.filter(
+                timestamp=timestamp,
+                book_id=book_id,
+                user_id=user.id,
+            ).first()
+
+            if not scrobble:
+                logger.info(
+                    f"Queueing scrobble for {book_id}, page {list(scrobble_page_data.keys())[0]}"
+                )
+                log_data = {
+                    "koreader_hash": book_dict.get("hash"),
+                    "page_data": scrobble_page_data,
+                    "pages_read": len(scrobble_page_data.keys()),
+                }
+                if hasattr(timestamp.tzinfo, "tzname"):
+                    tz = timestamp.tzinfo.tzname
+                if hasattr(timestamp.tzinfo, "name"):
+                    tz = timestamp.tzinfo.name
+                scrobbles_to_create.append(
+                    Scrobble(
+                        book_id=book_id,
+                        user_id=user.id,
+                        source="KOReader",
+                        media_type=Scrobble.MediaType.BOOK,
+                        timestamp=timestamp,
+                        log=log_data,
+                        stop_timestamp=stop_timestamp,
+                        playback_position_seconds=playback_position_seconds,
+                        in_progress=False,
+                        played_to_completion=True,
+                        long_play_complete=False,
+                        timezone=tz,
+                    )
+                )
+
     if pages_not_found:
         logger.info(f"Pages not found for books: {set(pages_not_found)}")
     return scrobbles_to_create
@@ -357,9 +432,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
 
     new_scrobbles = []
     user = User.objects.filter(id=user_id).first()
-    tz = ZoneInfo("UTC")
-    if user:
-        tz = user.profile.tzinfo
 
     is_os_file = "https://" not in file_path
     if is_os_file:
@@ -375,7 +447,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
         book_map = build_page_data(
             cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
             book_map,
-            tz,
         )
         new_scrobbles = build_scrobbles_from_book_map(book_map, user)
     else:
@@ -392,7 +463,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
             _sqlite_bytes(file_path), max_buffer_size=1_048_576
         ):
             if table_name == "page_stat_data":
-                book_map = build_page_data(rows, book_map, tz)
+                book_map = build_page_data(rows, book_map)
                 new_scrobbles = build_scrobbles_from_book_map(book_map, user)
 
     logger.info(f"Creating {len(new_scrobbles)} new scrobbles")

vrobbler/apps/books/models.py

@@ -4,6 +4,7 @@ from dataclasses import dataclass
 from datetime import datetime
 from typing import Optional
 from uuid import uuid4
+from zoneinfo import ZoneInfo
 
 import requests
 from books.constants import MediaSourceTag, READCOMICSONLINE_URL
@@ -71,7 +72,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
     _excluded_fields = {"koreader_hash", "page_data"}
 
     def avg_seconds_per_page(self):
-        if self.page_data:
+        if self.page_data and isinstance(self.page_data, dict):
             total_duration = 0
             for page_num, stats in self.page_data.items():
                 total_duration += stats.get("duration", 0)
@@ -173,6 +174,9 @@ class Book(LongPlayScrobblableMixin):
     genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
 
     def __str__(self) -> str:
+        if not self.subtitle:
+            return self.title
+
         return f"{self.title} - {self.subtitle}"
 
     def save(self, *args, **kwargs):
@@ -184,12 +188,18 @@ class Book(LongPlayScrobblableMixin):
 
     @property
     def subtitle(self):
-        subtitle = self.author
+        subtitle_parts = []
+        if self.author:
+            subtitle_parts.append(self.author.name)
         if self.issue_number and "Issue" not in str(self.title):
-            subtitle += " - Issue {self.issue_number}"
+            subtitle_parts.append(f"Issue {self.issue_number}")
         if self.volume_number and "Volume" not in str(self.title):
-            subtitle += " - Volume {self.volume_number}"
-        return subtitle
+            subtitle_parts.append(f"Volume {self.volume_number}")
+        if len(subtitle_parts) > 1:
+            return " / ".join(subtitle_parts)
+        if len(subtitle_parts) == 1:
+            return subtitle_parts[0]
+        return ""
 
     @property
     def strings(self) -> ScrobblableConstants:
@@ -463,8 +473,11 @@ class Book(LongPlayScrobblableMixin):
             if scrobble.logdata.page_data:
                 for page, data in scrobble.logdata.page_data.items():
                     if convert_timestamps:
-                        data["start_ts"] = datetime.fromtimestamp(data["start_ts"])
-                        data["end_ts"] = datetime.fromtimestamp(data["end_ts"])
+                        tz = None
+                        if scrobble.timezone:
+                            tz = ZoneInfo(scrobble.timezone)
+                        data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
+                        data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
                     pages[page] = data
         sorted_pages = OrderedDict(
             sorted(pages.items(), key=lambda x: x[1]["start_ts"])

vrobbler/apps/books/tests/conftest.py

@@ -32,8 +32,6 @@ class KoReaderBookRows:
     DEFAULT_STR = "N/A"
     DEFAULT_INT = 0
     DEFAULT_TIME = 1703800469
-    BOOK_ROWS = []
-    PAGE_STATS_ROWS = []
 
     def _gen_random_row(self, i):
         wiggle = random.randrange(15)
@@ -110,6 +108,8 @@ class KoReaderBookRows:
                     end_session = True
 
     def __init__(self, book_count=0, **kwargs):
+        self.BOOK_ROWS = []
+        self.PAGE_STATS_ROWS = []
         self._generate_random_book_rows(book_count)
         self._generate_custom_book_row(**kwargs)
         self._generate_random_page_stats_rows()

vrobbler/apps/books/tests/test_koreader.py

@@ -44,8 +44,12 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
     book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
 
     scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
-    # Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read )
-    expected_scrobbles = 6 * len(book_map.keys())
+    # The test data generator adds the session-gap 3600s AFTER the trigger page
+    # (not before), so the first session includes 21 pages (1-21), and each
+    # subsequent session has 20 until the last. The last trigger page (120) was
+    # previously orphaned by the loop structure; the post-loop fix now creates a
+    # scrobble for it.
+    expected_scrobbles = 7 * len(book_map.keys())
     assert len(scrobbles) == expected_scrobbles
     assert len(scrobbles[0].logdata.page_data.keys()) == 21
     assert len(scrobbles[1].logdata.page_data.keys()) == 20
@@ -53,6 +57,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
     assert len(scrobbles[3].logdata.page_data.keys()) == 20
     assert len(scrobbles[4].logdata.page_data.keys()) == 20
     assert len(scrobbles[5].logdata.page_data.keys()) == 18
+    assert len(scrobbles[6].logdata.page_data.keys()) == 1
 
 
 def test_get_author_str_from_row():

vrobbler/apps/charts/templatetags/chart_tags.py

@@ -7,7 +7,10 @@ register = template.Library()
 def get_item(dictionary, key):
     if isinstance(dictionary, dict):
         return dictionary.get(key)
-    return None
+    try:
+        return dictionary[int(key)]
+    except (IndexError, KeyError, TypeError, ValueError):
+        return None
 
 
 @register.filter

vrobbler/apps/profiles/models.py

@@ -140,6 +140,11 @@ class UserProfile(TimeStampedModel):
         return history
 
     def get_timestamp_with_tz(self, timestamp):
+        from django.conf import settings
+
+        server_tz = ZoneInfo(settings.TIME_ZONE)
+        ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
+
         timezone = self.tzinfo
         if self.timezone_change_log:
             change_list = self.historic_timezone_changes
@@ -150,13 +155,13 @@ class UserProfile(TimeStampedModel):
                     end = None
 
                 if end:
-                    if start <= timestamp.replace(tzinfo=end.timezone) <= end:
+                    if start <= ref_dt <= end:
                         timezone = start.timezone
                 else:
-                    if start <= timestamp.replace(tzinfo=start.timezone):
+                    if start <= ref_dt:
                         timezone = start.timezone
 
-        return timestamp.replace(tzinfo=timezone)
+        return ref_dt.astimezone(timezone)
 
     def adjust_timezone_of_scrobbles(self, commit=False):
         current_dt = None

vrobbler/apps/scrobbles/importers/webdav.py

@@ -153,7 +153,7 @@ def scan_webdav_for_koreader(
     )
 
     if update_etag_only:
-        if last_import and remote_etag:
+        if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
             last_import.webdav_etag = remote_etag
             last_import.save(update_fields=["webdav_etag"])
             logger.info(
@@ -163,7 +163,7 @@ def scan_webdav_for_koreader(
             )
         return 0
 
-    if last_import and last_import.webdav_etag and remote_etag:
+    if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
         if last_import.webdav_etag == remote_etag:
             logger.info(
                 "koreader stats file unchanged (ETag match)",

vrobbler/apps/scrobbles/migrations/0096_convert_book_page_data_to_dict.py

@@ -0,0 +1,35 @@
+from django.db import migrations
+
+
+def convert_page_data_to_dict(apps, schema_editor):
+    Scrobble = apps.get_model("scrobbles", "Scrobble")
+    for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
+        log__page_data=None
+    ):
+        page_data = scrobble.log.get("page_data")
+        if isinstance(page_data, list):
+            new_page_data = {}
+            for entry in page_data:
+                page_num = entry.get("page_number")
+                if page_num is not None:
+                    try:
+                        page_num = int(page_num)
+                    except (ValueError, TypeError):
+                        continue
+                    new_page_data[page_num] = entry
+            scrobble.log["page_data"] = new_page_data
+            scrobble.save(update_fields=["log"])
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ("scrobbles", "0095_backfill_null_visibility"),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            convert_page_data_to_dict,
+            reverse_code=migrations.RunPython.noop,
+        ),
+    ]

vrobbler/apps/scrobbles/models.py

@@ -225,7 +225,9 @@ class KoReaderImport(BaseFileImportMixin):
 
         self.mark_started()
         try:
-            scrobbles = process_koreader_sqlite_file(self.upload_file_path, self.user.id)
+            scrobbles = process_koreader_sqlite_file(
+                self.upload_file_path, self.user.id
+            )
             self.record_log(scrobbles)
         except Exception as e:
             self.record_error(f"Import failed: {e}")
@@ -272,7 +274,9 @@ class AudioScrobblerTSVImport(BaseFileImportMixin):
 
         self.mark_started()
         try:
-            scrobbles = import_audioscrobbler_tsv_file(self.upload_file_path, self.user.id)
+            scrobbles = import_audioscrobbler_tsv_file(
+                self.upload_file_path, self.user.id
+            )
             self.record_log(scrobbles)
         except Exception as e:
             self.record_error(f"Import failed: {e}")
@@ -936,6 +940,8 @@ class Scrobble(TimeStampedModel):
             logdata_cls = logdata.BaseLogData
 
         log_dict = self.log
+        if isinstance(log_dict, logdata.BaseLogData):
+            log_dict = log_dict.asdict
         if isinstance(self.log, str):
             # There's nothing stopping django from saving a string in a JSONField :(
             logger.warning(
@@ -1412,20 +1418,18 @@ class Scrobble(TimeStampedModel):
                 or scrobble_data["playback_status"] == "stopped"
             ):
                 if read_log_page:
-                    page_list = scrobble.log.get("page_data", [])
-                    if page_list:
-                        for page in page_list:
+                    page_data = scrobble.log.get("page_data", {})
+                    if page_data:
+                        for page_num, page in page_data.items():
                             if not page.get("end_ts", None):
                                 page["end_ts"] = int(timezone.now().timestamp())
                                 page["duration"] = page["end_ts"] - page.get("start_ts")
 
-                    page_list.append(
-                        BookPageLogData(
-                            page_number=read_log_page,
-                            start_ts=int(timezone.now().timestamp()),
-                        )
+                    page_data[read_log_page] = BookPageLogData(
+                        page_number=read_log_page,
+                        start_ts=int(timezone.now().timestamp()),
                     )
-                    scrobble.log["page_data"] = page_list
+                    scrobble.log["page_data"] = page_data
                     scrobble.save(update_fields=["log"])
                 elif "log" in scrobble_data.keys() and scrobble.log:
                     scrobble_data["log"] = scrobble.log | scrobble_data["log"]
@@ -1436,13 +1440,13 @@ class Scrobble(TimeStampedModel):
 
         if read_log_page:
             scrobble_data["log"] = BookLogData(
-                page_data=[
-                    BookPageLogData(
+                page_data={
+                    read_log_page: BookPageLogData(
                         page_number=read_log_page,
                         start_ts=int(timezone.now().timestamp()),
                     )
-                ]
-            )
+                }
+            ).asdict
 
         logger.info(
             f"[scrobbling] creating new scrobble",
@@ -1457,7 +1461,7 @@ class Scrobble(TimeStampedModel):
             "calories", None
         ):
             if media.calories:
-                scrobble_data["log"] = FoodLogData(calories=media.calories)
+                scrobble_data["log"] = FoodLogData(calories=media.calories).asdict
 
         scrobble = cls.create(scrobble_data)
         return scrobble
@@ -1780,33 +1784,27 @@ class Scrobble(TimeStampedModel):
         return False
 
     def calculate_reading_stats(self, commit=True):
-        page_data = self.log.get("page_data")
+        page_data = self.logdata.page_data
 
         if page_data:
-            # --- Sort safely by numeric page_number ---
-            def safe_page_number(entry):
-                try:
-                    return int(getattr("page_number", entry), 0)
-                except (ValueError, TypeError):
-                    return float("inf")
-
             if isinstance(page_data, dict):
-                logger.warning("Page data is dict, migrate koreader data")
-                return
+                valid_pages = sorted(int(k) for k in page_data.keys())
+                if valid_pages:
+                    self.log["page_start"] = min(valid_pages)
+                    self.log["page_end"] = max(valid_pages)
+                    self.log["pages_read"] = len(set(valid_pages))
+            elif isinstance(page_data, list):
+                valid_pages = []
+                for page in page_data:
+                    try:
+                        valid_pages.append(int(page["page_number"]))
+                    except (ValueError, TypeError):
+                        continue
 
-            page_data.sort(key=safe_page_number)
-
-            valid_pages = []
-            for page in page_data:
-                try:
-                    valid_pages.append(int(page["page_number"]))
-                except (ValueError, TypeError):
-                    continue
-
-            if valid_pages:
-                self.log["page_start"] = min(valid_pages)
-                self.log["page_end"] = max(valid_pages)
-                self.log["pages_read"] = len(set(valid_pages))
+                if valid_pages:
+                    self.log["page_start"] = min(valid_pages)
+                    self.log["page_end"] = max(valid_pages)
+                    self.log["pages_read"] = len(set(valid_pages))
         else:
             page_start = self.log.get("page_start")
             page_end = self.log.get("page_end")
@@ -1846,9 +1844,7 @@ class FavoriteMedia(TimeStampedModel):
     birding_location = models.ForeignKey(
         BirdingLocation, on_delete=models.CASCADE, **BNULL
     )
-    media_type = models.CharField(
-        max_length=20, choices=Scrobble.MediaType.choices
-    )
+    media_type = models.CharField(max_length=20, choices=Scrobble.MediaType.choices)
     sent_to_mopidy = models.BooleanField(default=False)
 
     class Meta:

vrobbler/apps/scrobbles/scrobblers.py

@@ -540,6 +540,7 @@ def email_scrobble_board_game(
             "playback_position_seconds": duration_seconds,
             "source": "BG Stats",
             "log": log_data,
+            "visibility": "private",
         }
 
         scrobble = None
@@ -1152,7 +1153,7 @@ def web_scrobbler_scrobble_video_or_song(
     artist_name = data_dict.get("artist")
     track_name = data_dict.get("track")
     tracks = Track.objects.filter(
-        artist__name=data_dict.get("artist"), title=data_dict.get("track")
+        artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
     )
     if tracks.count() > 1:
         logger.warning(

vrobbler/settings.py

@@ -145,7 +145,7 @@ CELERY_BEAT_SCHEDULE = {
     },
     "import-from-webdav": {
         "task": "scrobbles.tasks.import_from_webdav_all_users",
-        "schedule": crontab(minute="*/3"),
+        "schedule": crontab(minute="*/2"),
     },
     # Deprecated: BG Stats files now picked up from WebDAV var/bgstats/
     # "import-from-imap": {