Compare commits

...

7 Commits
50.2 ... 51.0

Author SHA1 Message Date
a731427f6e [release] Bump to version 51.0
All checks were successful
build / test (push) Successful in 2m52s
deploy / test (push) Successful in 2m25s
deploy / build-and-deploy (push) Successful in 49s
- Fix koreader scrobble imports to use DST properly
- Fix book scrobbles where page_data is a list
- Lichess imports do not set default visbility
2026-06-11 18:39:31 -04:00
410da163fe [books] Fix koreader imports, maybe forever 2026-06-11 18:38:51 -04:00
a171192a6f [books] Fix list page_data for old book scrobbles
All checks were successful
build / test (push) Successful in 2m2s
2026-06-11 11:00:04 -04:00
c16b61db40 [importers] Fix strange bug where celery version mismatches
All checks were successful
build / test (push) Successful in 2m9s
2026-06-11 10:48:38 -04:00
29cb6a4991 [books] Really fix the subtitle bug 2026-06-11 10:48:21 -04:00
25c28e8335 [books] Fix bug in subtitle gen
All checks were successful
build / test (push) Successful in 2m5s
2026-06-11 10:34:20 -04:00
25626be3b6 [boardgames] Add visibility to lichess imports 2026-06-11 10:02:37 -04:00
11 changed files with 238 additions and 87 deletions

View File

@ -88,7 +88,7 @@ fetching and simple saving.
*** Metadata sources *** Metadata sources
**** Scraper **** Scraper
* Backlog [0/15] :vrobbler:project:personal: * Backlog [0/14] :vrobbler:project:personal:
** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles: ** TODO [#C] Create small utility to clean up tracks scrobbled with wonky playback times :vrobbler:personal:bug:music:scrobbles:
:PROPERTIES: :PROPERTIES:
:ID: 702462cf-d54b-48c6-8a7c-78b8de751deb :ID: 702462cf-d54b-48c6-8a7c-78b8de751deb
@ -451,32 +451,6 @@ https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-li
:PROPERTIES: :PROPERTIES:
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f :ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
:END: :END:
** TODO [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
- Note taken on [2025-09-25 Thu 10:37] \\
This may already be fixed ... need to check.
- Note taken on [2025-02-25 12:34] \\
The page data has the canonical date something was read in it, but it seems
to be an hour off. I traced this back to being off during DST, so we just need
the importer to be aware of whether a user is using DST or not and roll back
an hour for part of the year. Also, we'd need to adjust any old scrobbles that
took place with DST off to roll them back by an hour.
*** Description
This is a long-standing problem when daylight saving time takes effect. Time is
manually set on a KoReader device (or at least, always saved in local time). So
whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers: ** TODO [#B] Make IMAP and WebDAV configurable :webdav:feature:imap:importers:
:PROPERTIES: :PROPERTIES:
:ID: b1426d92-2feb-4d15-9738-d5b7b0594f96 :ID: b1426d92-2feb-4d15-9738-d5b7b0594f96
@ -544,6 +518,58 @@ log a warning and move on.
We should have a global view `/favorites/` that shows the logged in users's We should have a global view `/favorites/` that shows the logged in users's
favorited media objects. favorited media objects.
* Version 51.0 [3/3]
** DONE [#B] Fix koreader scrobble imports to use DST properly :bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
- Note taken on [2025-09-25 Thu 10:37] \\
This may already be fixed ... need to check.
- Note taken on [2025-02-25 12:34] \\
The page data has the canonical date something was read in it, but it seems
to be an hour off. I traced this back to being off during DST, so we just need
the importer to be aware of whether a user is using DST or not and roll back
an hour for part of the year. Also, we'd need to adjust any old scrobbles that
took place with DST off to roll them back by an hour.
*** Description
This is a long-standing problem when daylight saving time takes effect. Time is
manually set on a KoReader device (or at least, always saved in local time). So
whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** DONE [#A] Fix book scrobbles where page_data is a list :bug:books:scrobbles:
:PROPERTIES:
:ID: 35b323fa-ccc0-4009-b227-8a0f12bbd469
:END:
*** Description
Comic scrobbling is currently kind of janky. Most of the problems boil down to them
storing saved page data in a list of dicts rather than a dict keyed off of the page number that
was read.
We need to adjust comic scrobbling to use a dict of pages keyed off the page number, and also
write a migration script that runs as a data migration to update any book scrobbles that may
have page_data as a list.
*** Example data
#+begin_src python
{"notes": null, "page_end": 27, "page_data": [{"notes": null, "end_ts": 1771815895, "duration": 14, "start_ts": 1771815881, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815908, "duration": 13, "start_ts": 1771815895, "description": null, "page_number": "1"}, {"notes": null, "end_ts": 1771815913, "duration": 5, "start_ts": 1771815908, "description": null, "page_number": "2"}, {"notes": null, "end_ts": 1771815933, "duration": 20, "start_ts": 1771815913, "description": null, "page_number": "3"}, {"notes": null, "end_ts": 1771815945, "duration": 12, "start_ts": 1771815933, "description": null, "page_number": "4"}, {"notes": null, "end_ts": 1771815983, "duration": 38, "start_ts": 1771815945, "description": null, "page_number": "5"}, {"notes": null, "end_ts": 1771816007, "duration": 24, "start_ts": 1771815983, "description": null, "page_number": "6"}, {"notes": null, "end_ts": 1771816011, "duration": 4, "start_ts": 1771816007, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816013, "duration": 2, "start_ts": 1771816011, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816052, "duration": 39, "start_ts": 1771816013, "description": null, "page_number": "7"}, {"notes": null, "end_ts": 1771816127, "duration": 75, "start_ts": 1771816052, "description": null, "page_number": "8"}, {"notes": null, "end_ts": 1771816134, "duration": 7, "start_ts": 1771816127, "description": null, "page_number": "9"}, {"notes": null, "end_ts": 1771816196, "duration": 62, "start_ts": 1771816134, "description": null, "page_number": "10"}, {"notes": null, "end_ts": 1771816262, "duration": 66, "start_ts": 1771816196, "description": null, "page_number": "11"}, {"notes": null, "end_ts": 1771816293, "duration": 31, "start_ts": 1771816262, "description": null, "page_number": "12"}, {"notes": null, "end_ts": 1771816322, "duration": 29, "start_ts": 1771816293, "description": null, "page_number": "13"}, {"notes": null, "end_ts": 1771816330, "duration": 8, "start_ts": 1771816322, "description": null, "page_number": "14"}, {"notes": null, "end_ts": 1771816368, "duration": 38, "start_ts": 1771816330, "description": null, "page_number": "15"}, {"notes": null, "end_ts": 1771816388, "duration": 20, "start_ts": 1771816368, "description": null, "page_number": "16"}, {"notes": null, "end_ts": 1771816482, "duration": 94, "start_ts": 1771816388, "description": null, "page_number": "17"}, {"notes": null, "end_ts": 1771816550, "duration": 68, "start_ts": 1771816482, "description": null, "page_number": "18"}, {"notes": null, "end_ts": 1771816567, "duration": 17, "start_ts": 1771816550, "description": null, "page_number": "19"}, {"notes": null, "end_ts": 1771816586, "duration": 19, "start_ts": 1771816567, "description": null, "page_number": "20"}, {"notes": null, "end_ts": 1771816597, "duration": 11, "start_ts": 1771816586, "description": null, "page_number": "21"}, {"notes": null, "end_ts": 1771816616, "duration": 19, "start_ts": 1771816597, "description": null, "page_number": "22"}, {"notes": null, "end_ts": 1771816640, "duration": 24, "start_ts": 1771816616, "description": null, "page_number": "23"}, {"notes": null, "end_ts": 1771816690, "duration": 50, "start_ts": 1771816640, "description": null, "page_number": "24"}, {"notes": null, "end_ts": 1771816702, "duration": 12, "start_ts": 1771816690, "description": null, "page_number": "25"}, {"notes": null, "end_ts": 1771816823, "duration": 121, "start_ts": 1771816702, "description": null, "page_number": "26"}, {"notes": null, "end_ts": null, "duration": null, "start_ts": 1771816823, "description": null, "page_number": "27"}], "page_start": 1, "pages_read": 27, "resume_url": null, "description": null, "koreader_hash": null, "long_play_complete": false}
#+end_src
** DONE [#A] Lichess imports do not set default visbility :boardgames:bug:importers:lichess:
:PROPERTIES:
:ID: a78f7c72-a20a-8db2-cde0-d92a731d4fba
:END:
* Version 50.2 [2/2] * Version 50.2 [2/2]
** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers: ** DONE [#B] Koreader imports only import single-page scrobbles the next day :bug:books:importers:
:PROPERTIES: :PROPERTIES:

View File

@ -1,6 +1,6 @@
[tool.poetry] [tool.poetry]
name = "vrobbler" name = "vrobbler"
version = "50.2" version = "51.0"
description = "" description = ""
authors = ["Colin Powell <colin@unbl.ink>"] authors = ["Colin Powell <colin@unbl.ink>"]

View File

@ -103,6 +103,7 @@ def import_chess_games_for_user_id(user_id: int, commit: bool = False) -> dict:
"source": "Lichess", "source": "Lichess",
"timezone": user.profile.timezone, "timezone": user.profile.timezone,
"log": log_data, "log": log_data,
"visibility": "private",
} }
if commit: if commit:
Scrobble.objects.create(**scrobble_dict) Scrobble.objects.create(**scrobble_dict)

View File

@ -1,9 +1,8 @@
import logging import logging
import re import re
import sqlite3 import sqlite3
from datetime import datetime, timedelta from datetime import datetime, timedelta, timezone
from enum import Enum from enum import Enum
from zoneinfo import ZoneInfo
import requests import requests
from books.constants import BOOKS_TITLES_TO_IGNORE from books.constants import BOOKS_TITLES_TO_IGNORE
@ -174,7 +173,7 @@ def build_book_map(rows) -> dict:
return book_id_map return book_id_map
def build_page_data(page_rows: list, book_map: dict, user_tz=None) -> dict: def build_page_data(page_rows: list, book_map: dict) -> dict:
"""Given rows of page data from KoReader, parse each row and build """Given rows of page data from KoReader, parse each row and build
scrobbles for our user, loading the page data into the page_data scrobbles for our user, loading the page data into the page_data
field on the scrobble instance. field on the scrobble instance.
@ -275,10 +274,14 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
continue continue
timestamp = user.profile.get_timestamp_with_tz( timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(first_page.get("start_ts"))) datetime.fromtimestamp(
int(first_page.get("start_ts"))
)
) )
stop_timestamp = user.profile.get_timestamp_with_tz( stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(int(last_page.get("end_ts"))) datetime.fromtimestamp(
int(last_page.get("end_ts"))
)
) )
scrobble = Scrobble.objects.filter( scrobble = Scrobble.objects.filter(
@ -326,6 +329,80 @@ def build_scrobbles_from_book_map(book_map: dict, user: "User") -> list["Scrobbl
last_page_number = page_number last_page_number = page_number
prev_page_stats = stats prev_page_stats = stats
# Handle leftover pages that never triggered a session gap
if scrobble_page_data:
scrobble_page_data = dict(
sorted(
scrobble_page_data.items(),
key=lambda x: x[1]["start_ts"],
)
)
try:
first_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[0]
)
last_page = scrobble_page_data.get(
list(scrobble_page_data.keys())[-1]
)
except IndexError:
logger.error(
"Could not process book, no page data found",
extra={"scrobble_page_data": scrobble_page_data},
)
continue
playback_position_seconds = sum(
p["duration"] for p in scrobble_page_data.values()
)
timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(
int(first_page.get("start_ts"))
)
)
stop_timestamp = user.profile.get_timestamp_with_tz(
datetime.fromtimestamp(
int(last_page.get("end_ts"))
)
)
scrobble = Scrobble.objects.filter(
timestamp=timestamp,
book_id=book_id,
user_id=user.id,
).first()
if not scrobble:
logger.info(
f"Queueing scrobble for {book_id}, page {list(scrobble_page_data.keys())[0]}"
)
log_data = {
"koreader_hash": book_dict.get("hash"),
"page_data": scrobble_page_data,
"pages_read": len(scrobble_page_data.keys()),
}
if hasattr(timestamp.tzinfo, "tzname"):
tz = timestamp.tzinfo.tzname
if hasattr(timestamp.tzinfo, "name"):
tz = timestamp.tzinfo.name
scrobbles_to_create.append(
Scrobble(
book_id=book_id,
user_id=user.id,
source="KOReader",
media_type=Scrobble.MediaType.BOOK,
timestamp=timestamp,
log=log_data,
stop_timestamp=stop_timestamp,
playback_position_seconds=playback_position_seconds,
in_progress=False,
played_to_completion=True,
long_play_complete=False,
timezone=tz,
)
)
if pages_not_found: if pages_not_found:
logger.info(f"Pages not found for books: {set(pages_not_found)}") logger.info(f"Pages not found for books: {set(pages_not_found)}")
return scrobbles_to_create return scrobbles_to_create
@ -355,9 +432,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
new_scrobbles = [] new_scrobbles = []
user = User.objects.filter(id=user_id).first() user = User.objects.filter(id=user_id).first()
tz = ZoneInfo("UTC")
if user:
tz = user.profile.tzinfo
is_os_file = "https://" not in file_path is_os_file = "https://" not in file_path
if is_os_file: if is_os_file:
@ -373,7 +447,6 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
book_map = build_page_data( book_map = build_page_data(
cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"), cur.execute("SELECT * from page_stat_data ORDER BY id_book, start_time"),
book_map, book_map,
tz,
) )
new_scrobbles = build_scrobbles_from_book_map(book_map, user) new_scrobbles = build_scrobbles_from_book_map(book_map, user)
else: else:
@ -390,7 +463,7 @@ def process_koreader_sqlite_file(file_path, user_id) -> list:
_sqlite_bytes(file_path), max_buffer_size=1_048_576 _sqlite_bytes(file_path), max_buffer_size=1_048_576
): ):
if table_name == "page_stat_data": if table_name == "page_stat_data":
book_map = build_page_data(rows, book_map, tz) book_map = build_page_data(rows, book_map)
new_scrobbles = build_scrobbles_from_book_map(book_map, user) new_scrobbles = build_scrobbles_from_book_map(book_map, user)
logger.info(f"Creating {len(new_scrobbles)} new scrobbles") logger.info(f"Creating {len(new_scrobbles)} new scrobbles")

View File

@ -4,6 +4,7 @@ from dataclasses import dataclass
from datetime import datetime from datetime import datetime
from typing import Optional from typing import Optional
from uuid import uuid4 from uuid import uuid4
from zoneinfo import ZoneInfo
import requests import requests
from books.constants import MediaSourceTag, READCOMICSONLINE_URL from books.constants import MediaSourceTag, READCOMICSONLINE_URL
@ -71,7 +72,7 @@ class BookLogData(BaseLogData, LongPlayLogData):
_excluded_fields = {"koreader_hash", "page_data"} _excluded_fields = {"koreader_hash", "page_data"}
def avg_seconds_per_page(self): def avg_seconds_per_page(self):
if self.page_data: if self.page_data and isinstance(self.page_data, dict):
total_duration = 0 total_duration = 0
for page_num, stats in self.page_data.items(): for page_num, stats in self.page_data.items():
total_duration += stats.get("duration", 0) total_duration += stats.get("duration", 0)
@ -173,6 +174,9 @@ class Book(LongPlayScrobblableMixin):
genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre") genre = TaggableManager(through=ObjectWithGenres, blank=True, verbose_name="Genre")
def __str__(self) -> str: def __str__(self) -> str:
if not self.subtitle:
return self.title
return f"{self.title} - {self.subtitle}" return f"{self.title} - {self.subtitle}"
def save(self, *args, **kwargs): def save(self, *args, **kwargs):
@ -184,12 +188,18 @@ class Book(LongPlayScrobblableMixin):
@property @property
def subtitle(self): def subtitle(self):
subtitle = self.author subtitle_parts = []
if self.author:
subtitle_parts.append(self.author.name)
if self.issue_number and "Issue" not in str(self.title): if self.issue_number and "Issue" not in str(self.title):
subtitle += " - Issue {self.issue_number}" subtitle_parts.append(f"Issue {self.issue_number}")
if self.volume_number and "Volume" not in str(self.title): if self.volume_number and "Volume" not in str(self.title):
subtitle += " - Volume {self.volume_number}" subtitle_parts.append(f"Volume {self.volume_number}")
return subtitle if len(subtitle_parts) > 1:
return " / ".join(subtitle_parts)
if len(subtitle_parts) == 1:
return subtitle_parts[0]
return ""
@property @property
def strings(self) -> ScrobblableConstants: def strings(self) -> ScrobblableConstants:
@ -463,8 +473,11 @@ class Book(LongPlayScrobblableMixin):
if scrobble.logdata.page_data: if scrobble.logdata.page_data:
for page, data in scrobble.logdata.page_data.items(): for page, data in scrobble.logdata.page_data.items():
if convert_timestamps: if convert_timestamps:
data["start_ts"] = datetime.fromtimestamp(data["start_ts"]) tz = None
data["end_ts"] = datetime.fromtimestamp(data["end_ts"]) if scrobble.timezone:
tz = ZoneInfo(scrobble.timezone)
data["start_ts"] = datetime.fromtimestamp(data["start_ts"], tz=tz)
data["end_ts"] = datetime.fromtimestamp(data["end_ts"], tz=tz)
pages[page] = data pages[page] = data
sorted_pages = OrderedDict( sorted_pages = OrderedDict(
sorted(pages.items(), key=lambda x: x[1]["start_ts"]) sorted(pages.items(), key=lambda x: x[1]["start_ts"])

View File

@ -44,8 +44,12 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map) book_map = build_page_data(koreader_rows.PAGE_STATS_ROWS, book_map)
scrobbles = build_scrobbles_from_book_map(book_map, demo_user) scrobbles = build_scrobbles_from_book_map(book_map, demo_user)
# Corresponds to number of sessions per book ( 20 pages per session, 120 +/- 15 pages read ) # The test data generator adds the session-gap 3600s AFTER the trigger page
expected_scrobbles = 6 * len(book_map.keys()) # (not before), so the first session includes 21 pages (1-21), and each
# subsequent session has 20 until the last. The last trigger page (120) was
# previously orphaned by the loop structure; the post-loop fix now creates a
# scrobble for it.
expected_scrobbles = 7 * len(book_map.keys())
assert len(scrobbles) == expected_scrobbles assert len(scrobbles) == expected_scrobbles
assert len(scrobbles[0].logdata.page_data.keys()) == 21 assert len(scrobbles[0].logdata.page_data.keys()) == 21
assert len(scrobbles[1].logdata.page_data.keys()) == 20 assert len(scrobbles[1].logdata.page_data.keys()) == 20
@ -53,6 +57,7 @@ def test_build_scrobbles_from_pages(get_mock, koreader_rows, demo_user, valid_re
assert len(scrobbles[3].logdata.page_data.keys()) == 20 assert len(scrobbles[3].logdata.page_data.keys()) == 20
assert len(scrobbles[4].logdata.page_data.keys()) == 20 assert len(scrobbles[4].logdata.page_data.keys()) == 20
assert len(scrobbles[5].logdata.page_data.keys()) == 18 assert len(scrobbles[5].logdata.page_data.keys()) == 18
assert len(scrobbles[6].logdata.page_data.keys()) == 1
def test_get_author_str_from_row(): def test_get_author_str_from_row():

View File

@ -140,6 +140,11 @@ class UserProfile(TimeStampedModel):
return history return history
def get_timestamp_with_tz(self, timestamp): def get_timestamp_with_tz(self, timestamp):
from django.conf import settings
server_tz = ZoneInfo(settings.TIME_ZONE)
ref_dt = timestamp if timestamp.tzinfo is not None else timestamp.replace(tzinfo=server_tz)
timezone = self.tzinfo timezone = self.tzinfo
if self.timezone_change_log: if self.timezone_change_log:
change_list = self.historic_timezone_changes change_list = self.historic_timezone_changes
@ -150,13 +155,13 @@ class UserProfile(TimeStampedModel):
end = None end = None
if end: if end:
if start <= timestamp.replace(tzinfo=end.timezone) <= end: if start <= ref_dt <= end:
timezone = start.timezone timezone = start.timezone
else: else:
if start <= timestamp.replace(tzinfo=start.timezone): if start <= ref_dt:
timezone = start.timezone timezone = start.timezone
return timestamp.replace(tzinfo=timezone) return ref_dt.astimezone(timezone)
def adjust_timezone_of_scrobbles(self, commit=False): def adjust_timezone_of_scrobbles(self, commit=False):
current_dt = None current_dt = None

View File

@ -153,7 +153,7 @@ def scan_webdav_for_koreader(
) )
if update_etag_only: if update_etag_only:
if last_import and remote_etag: if last_import and remote_etag and hasattr(last_import, "webdav_etag"):
last_import.webdav_etag = remote_etag last_import.webdav_etag = remote_etag
last_import.save(update_fields=["webdav_etag"]) last_import.save(update_fields=["webdav_etag"])
logger.info( logger.info(
@ -163,7 +163,7 @@ def scan_webdav_for_koreader(
) )
return 0 return 0
if last_import and last_import.webdav_etag and remote_etag: if last_import and getattr(last_import, "webdav_etag", None) and remote_etag:
if last_import.webdav_etag == remote_etag: if last_import.webdav_etag == remote_etag:
logger.info( logger.info(
"koreader stats file unchanged (ETag match)", "koreader stats file unchanged (ETag match)",

View File

@ -0,0 +1,35 @@
from django.db import migrations
def convert_page_data_to_dict(apps, schema_editor):
Scrobble = apps.get_model("scrobbles", "Scrobble")
for scrobble in Scrobble.objects.filter(media_type="Book").exclude(
log__page_data=None
):
page_data = scrobble.log.get("page_data")
if isinstance(page_data, list):
new_page_data = {}
for entry in page_data:
page_num = entry.get("page_number")
if page_num is not None:
try:
page_num = int(page_num)
except (ValueError, TypeError):
continue
new_page_data[page_num] = entry
scrobble.log["page_data"] = new_page_data
scrobble.save(update_fields=["log"])
class Migration(migrations.Migration):
dependencies = [
("scrobbles", "0095_backfill_null_visibility"),
]
operations = [
migrations.RunPython(
convert_page_data_to_dict,
reverse_code=migrations.RunPython.noop,
),
]

View File

@ -1412,20 +1412,18 @@ class Scrobble(TimeStampedModel):
or scrobble_data["playback_status"] == "stopped" or scrobble_data["playback_status"] == "stopped"
): ):
if read_log_page: if read_log_page:
page_list = scrobble.log.get("page_data", []) page_data = scrobble.log.get("page_data", {})
if page_list: if page_data:
for page in page_list: for page_num, page in page_data.items():
if not page.get("end_ts", None): if not page.get("end_ts", None):
page["end_ts"] = int(timezone.now().timestamp()) page["end_ts"] = int(timezone.now().timestamp())
page["duration"] = page["end_ts"] - page.get("start_ts") page["duration"] = page["end_ts"] - page.get("start_ts")
page_list.append( page_data[read_log_page] = BookPageLogData(
BookPageLogData( page_number=read_log_page,
page_number=read_log_page, start_ts=int(timezone.now().timestamp()),
start_ts=int(timezone.now().timestamp()),
)
) )
scrobble.log["page_data"] = page_list scrobble.log["page_data"] = page_data
scrobble.save(update_fields=["log"]) scrobble.save(update_fields=["log"])
elif "log" in scrobble_data.keys() and scrobble.log: elif "log" in scrobble_data.keys() and scrobble.log:
scrobble_data["log"] = scrobble.log | scrobble_data["log"] scrobble_data["log"] = scrobble.log | scrobble_data["log"]
@ -1436,12 +1434,12 @@ class Scrobble(TimeStampedModel):
if read_log_page: if read_log_page:
scrobble_data["log"] = BookLogData( scrobble_data["log"] = BookLogData(
page_data=[ page_data={
BookPageLogData( read_log_page: BookPageLogData(
page_number=read_log_page, page_number=read_log_page,
start_ts=int(timezone.now().timestamp()), start_ts=int(timezone.now().timestamp()),
) )
] }
) )
logger.info( logger.info(
@ -1783,30 +1781,24 @@ class Scrobble(TimeStampedModel):
page_data = self.log.get("page_data") page_data = self.log.get("page_data")
if page_data: if page_data:
# --- Sort safely by numeric page_number ---
def safe_page_number(entry):
try:
return int(getattr("page_number", entry), 0)
except (ValueError, TypeError):
return float("inf")
if isinstance(page_data, dict): if isinstance(page_data, dict):
logger.warning("Page data is dict, migrate koreader data") valid_pages = sorted(int(k) for k in page_data.keys())
return if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
elif isinstance(page_data, list):
valid_pages = []
for page in page_data:
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
page_data.sort(key=safe_page_number) if valid_pages:
self.log["page_start"] = min(valid_pages)
valid_pages = [] self.log["page_end"] = max(valid_pages)
for page in page_data: self.log["pages_read"] = len(set(valid_pages))
try:
valid_pages.append(int(page["page_number"]))
except (ValueError, TypeError):
continue
if valid_pages:
self.log["page_start"] = min(valid_pages)
self.log["page_end"] = max(valid_pages)
self.log["pages_read"] = len(set(valid_pages))
else: else:
page_start = self.log.get("page_start") page_start = self.log.get("page_start")
page_end = self.log.get("page_end") page_end = self.log.get("page_end")

View File

@ -540,6 +540,7 @@ def email_scrobble_board_game(
"playback_position_seconds": duration_seconds, "playback_position_seconds": duration_seconds,
"source": "BG Stats", "source": "BG Stats",
"log": log_data, "log": log_data,
"visibility": "private",
} }
scrobble = None scrobble = None
@ -1152,7 +1153,7 @@ def web_scrobbler_scrobble_video_or_song(
artist_name = data_dict.get("artist") artist_name = data_dict.get("artist")
track_name = data_dict.get("track") track_name = data_dict.get("track")
tracks = Track.objects.filter( tracks = Track.objects.filter(
artist__name=data_dict.get("artist"), title=data_dict.get("track") artist_fk__name=data_dict.get("artist"), title=data_dict.get("track")
) )
if tracks.count() > 1: if tracks.count() > 1:
logger.warning( logger.warning(