Compare commits

...

4 Commits
38.0 ... 39.0

Author SHA1 Message Date
96d1d7ac6b [release] Bump to version 39.0
Some checks failed
build & deploy / test (push) Failing after 1m43s
build & deploy / build-and-deploy (push) Has been skipped
- Clean up org-mode tasks metadata
- Actually push branches up and add a just command to do it
- Try to fix deploy failing with bad release plan
2026-05-31 23:17:52 -04:00
009b2ba243 [scrobbles] Big update for notes in tasks and boardgames
Some checks failed
build & deploy / build-and-deploy (push) Has been cancelled
build & deploy / test (push) Has been cancelled
2026-05-31 23:17:16 -04:00
4f051ae250 [tooling] Add push command to just
All checks were successful
build & deploy / test (push) Successful in 2m7s
build & deploy / build-and-deploy (push) Has been skipped
2026-05-31 13:56:57 -04:00
7e9fbb1bf6 [tooling] Try to fix CI script
All checks were successful
build & deploy / test (push) Successful in 1m58s
build & deploy / build-and-deploy (push) Has been skipped
2026-05-31 13:10:31 -04:00
16 changed files with 362 additions and 74 deletions

View File

@ -72,7 +72,7 @@ jobs:
build-and-deploy:
# Only deploy on tags (equivalent to Drone when: ref: refs/tags/*)
# if: startsWith(gitea.ref, 'refs/tags/')
if: startsWith(gitea.ref, 'refs/tags/')
needs: [test]
runs-on: ubuntu-latest
@ -120,10 +120,13 @@ jobs:
key: ${{ secrets.JAIL_KEY }}
command_timeout: 2m
script: |
set -e
mkdir -p /var/lib/vrobbler
echo "${{ gitea.sha }}" | cut -c1-8 > /var/lib/vrobbler/commit.txt
pip uninstall -y vrobbler
pip install /var/lib/vrobbler/dist/*.whl
python -c "import vrobbler; print(f'vrobbler {vrobbler.__version__} installed OK')"
rm -f /var/lib/vrobbler/dist/*.whl
vrobbler migrate
vrobbler collectstatic --noinput
immortalctl restart vrobbler-celery && immortalctl restart vrobbler-celerybeat && immortalctl restart vrobbler

View File

@ -93,7 +93,7 @@ fetching and simple saving.
:LOGBOOK:
CLOCK: [2025-07-09 Wed 09:55]--[2025-07-09 Wed 10:15] => 0:20
:END:
* Backlog [0/16] :vrobbler:project:personal:
* Backlog [0/15] :vrobbler:project:personal:
** TODO [#C] Add sentiment parsing for Scrobbles with notes :vrobbler:project:scrobbles:sentiment:
:PROPERTIES:
:ID: 37781d6a-f3b0-48b2-bf98-33c2c791cf85
@ -450,11 +450,11 @@ https://app.todoist.com/app/task/add-a-csv-endpoint-for-users-book-reads-that-li
- Note taken on [2025-09-25 Thu 10:51]
As an example https://comicbookroundup.com/comic-books/reviews/humanoids-publishing/the-history-of-science-fiction
** TODO [#A] Find page numbers for comic books from ComicVine :vrobbler:feature:books:personal:project:
** TODO [#B] Find page numbers for comic books from ComicVine :vrobbler:feature:books:personal:project:
:PROPERTIES:
:ID: 79f867c3-1288-4143-b6bf-2a452983ee9f
:END:
** TODO [#A] Fix koreader scrobble imports to use DST properly :vrobbler:personal:bug:books:imports:
** TODO [#B] Fix koreader scrobble imports to use DST properly :vrobbler:personal:bug:books:imports:
:PROPERTIES:
:ID: 79758cba-a440-48b6-a637-efb88827acf2
:END:
@ -480,14 +480,59 @@ whatever time KoReader reports, we need to know, given the date and the user
profile's historic timezone, how many hours to adjust the KoReader time to get
to GMT to save it in the database.
** TODO [#A] When creating org-mode tasks, don't copy comments :vrobbler:bug:scrobbles:tasks:
* Version 39.0 [3/3]
** DONE [#B] Clean up org-mode tasks metadata :bug:tasks:metadata:
:PROPERTIES:
:ID: 0c762d09-fc69-4e75-be40-7eaaf04f178e
:END:
*** Description
It would be nice to not duplicate comments that exist on a task when it's first scrobbled.
Org-mode tasks have a `Description` subheader, which should populate the
"Description" of a task.
The title should come from the actual TODO content, with tags coming from the
tags in colons after the task. This behaviour should already work.
Next, any comments should go under a `Comments` subheader. Under these should be a list tag like:
"Note taken on [2026-05-31 Sun 20:03]"
Which declares it's a note with a timestamp like "YYYY-MM-DD d HH:MM".
When scrobbling a task from org-mode, the description should always be copied to
the scrobble's log["description"].
Any comments that exist on the task when the scrobble is first created, should
be ignored.
Any comments on a task that is updated (a re-POST'd while the task is in
progress, should add a comment in the log["notes"], a dictionary, keyed off the
timestamp the note string. If a note for that datetime already exists, the
content should be replaced with the new comment.
Additionally, when a task is re-POST'd while the task is in progress, the
description should also be compared, and if different, the newest description
should be used.
Note, the biggest change for this flow is making sure comments that already
exist are not added to any new tasks and that both comments and descriptions are
added or updated if the new values are different than what is already in the
currently in-progress task.
If the task is completed, don't touch it.
*** Comments
- Note taken on [2026-05-31 Sun 20:03]
** DONE [#A] Actually push branches up and add a just command to do it :release:justfile:tooling:
:PROPERTIES:
:ID: 50aa5daa-a802-6aa9-38a3-218b7a9d4b34
:END:
** DONE [#A] Try to fix deploy failing with bad release plan :release:tooling:pyproject:
:PROPERTIES:
:ID: 63dc633c-4382-e6a5-e663-b01871ce86ce
:END:
* Version 38.0 [38/38]
** DONE [#A] Fix release flow to be easier to trigger :pyproject:release:tooling:

View File

@ -17,3 +17,7 @@ celery-beat:
release kind="minor":
poetry run python scripts/release.py {{kind}}
push:
git push && git push gitea
git push --tags && git push --tags gitea

View File

@ -1,6 +1,6 @@
[tool.poetry]
name = "vrobbler"
version = "38.0"
version = "39.0"
description = ""
authors = ["Colin Powell <colin@unbl.ink>"]

View File

@ -90,6 +90,26 @@ class BoardGameLogData(BaseLogData, LongPlayLogData):
"variant",
}
@classmethod
def override_fields(cls) -> dict:
from scrobbles.forms import NotesDictField
fields = {}
for base in cls.mro()[1:]:
if hasattr(base, "override_fields"):
base_fields = base.override_fields()
fields.update(base_fields)
custom_fields = {
"notes": NotesDictField(required=False),
"location_id": forms.ModelChoiceField(
queryset=BoardGameLocation.objects.all(),
required=False,
widget=forms.Select(),
),
}
fields.update(custom_fields)
return fields
@cached_property
def location(self):
if not self.location_id:
@ -133,23 +153,6 @@ class BoardGameLogData(BaseLogData, LongPlayLogData):
return "".join(html_parts)
@classmethod
def override_fields(cls) -> dict:
fields = {}
for base in cls.mro()[1:]:
if hasattr(base, "override_fields"):
base_fields = base.override_fields()
fields.update(base_fields)
custom_fields = {
"location_id": forms.ModelChoiceField(
queryset=BoardGameLocation.objects.all(),
required=False,
widget=forms.Select(),
)
}
fields.update(custom_fields)
return fields
class BoardGamePublisher(TimeStampedModel):
name = models.CharField(max_length=255)

View File

@ -57,14 +57,28 @@ class BaseLogData(JSONDataclass):
if not self.notes:
return ""
if isinstance(self.notes, dict):
lines = []
for ts, text in self.notes.items():
note_text = " ".join(text.strip().split())
lines.append(f"{ts}: {note_text}")
return separator.join(lines)
if isinstance(self.notes, str):
return html.escape(re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", self.notes))
if isinstance(self.notes, list):
cleaned_notes = []
for note in self.notes:
cleaned = html.escape(re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", note))
cleaned_notes.append(cleaned)
if isinstance(note, dict):
timestamp, note_text = next(iter(note.items()))
note_text = " ".join(note_text.strip().split())
cleaned_notes.append(f"{timestamp}: {note_text}")
elif isinstance(note, str):
cleaned = html.escape(
re.sub(r"\s*\[start:\d+\|end:\d+\]$", "", note)
)
cleaned_notes.append(cleaned)
return separator.join(cleaned_notes)
return ""
@ -76,18 +90,35 @@ class BaseLogData(JSONDataclass):
if not self.notes:
return ""
notes_list = []
if isinstance(self.notes, str):
notes_list = [self.notes]
elif isinstance(self.notes, list):
notes_list = self.notes
html_notes = []
for note in notes_list:
for line in note.split("\n"):
if isinstance(self.notes, dict):
for ts, text in self.notes.items():
note_text = " ".join(text.strip().split())
html_notes.append(
f'<div class="sticky-note">{ts}: {note_text}</div>'
)
elif isinstance(self.notes, str):
for line in self.notes.split("\n"):
if line.strip():
escaped_line = html.escape(line)
html_notes.append(f'<div class="sticky-note">{escaped_line}</div>')
elif isinstance(self.notes, list):
for note in self.notes:
if isinstance(note, dict):
timestamp, note_text = next(iter(note.items()))
note_text = " ".join(note_text.strip().split())
html_notes.append(
f'<div class="sticky-note">{timestamp}: {note_text}</div>'
)
elif isinstance(note, str):
for line in note.split("\n"):
if line.strip():
escaped_line = html.escape(line)
html_notes.append(
f'<div class="sticky-note">{escaped_line}</div>'
)
return mark_safe("".join(html_notes))

View File

@ -1,3 +1,4 @@
import json
from dataclasses import fields
from typing import Union, get_args, get_origin
@ -89,7 +90,7 @@ def form_from_dataclass(dataclass):
form_cls = type(f"{dataclass.__name__}Form", (forms.Form,), form_fields)
if "notes" in form_cls.base_fields:
if "notes" in form_cls.base_fields and "notes" not in override_fields:
form_cls.base_fields["notes"] = forms.CharField(
required=False,
widget=forms.Textarea(attrs={"rows": 4}),
@ -101,3 +102,47 @@ def form_from_dataclass(dataclass):
form_cls.clean_notes = clean_notes
return form_cls
class NotesDictWidget(forms.Widget):
template_name = "tasks/task_notes_widget.html"
class Media:
js = ("tasks/task_notes.js",)
def value_from_datadict(self, data, files, name):
timestamps = data.getlist(f"{name}_timestamps")
contents = data.getlist(f"{name}_contents")
return {
"timestamps": timestamps,
"contents": contents,
}
def get_context(self, name, value, attrs):
context = super().get_context(name, value, attrs)
notes = {}
if value:
if isinstance(value, str):
try:
notes = json.loads(value)
except (json.JSONDecodeError, TypeError):
notes = {}
elif isinstance(value, dict):
notes = value
context["widget"]["notes"] = notes
return context
class NotesDictField(forms.Field):
widget = NotesDictWidget
def clean(self, value):
if not value:
return {}
result = {}
timestamps = value.get("timestamps", [])
contents = value.get("contents", [])
for i, ts in enumerate(timestamps):
if i < len(contents) and ts and contents[i].strip():
result[ts] = contents[i].strip()
return result if result else {}

View File

@ -1,9 +1,7 @@
from django.core.management.base import BaseCommand
from vrobbler.apps.tasks.utils import (
convert_notes_to_dict,
convert_old_boardgame_log_to_new,
convert_old_orgmode_log_to_new,
convert_old_todoist_log_to_new,
convert_tasks_notes_list_to_dict,
)
@ -21,7 +19,5 @@ class Command(BaseCommand):
commit = True
else:
print("No changes will be saved, use --commit to save")
convert_old_orgmode_log_to_new(commit)
convert_old_todoist_log_to_new(commit)
convert_notes_to_dict(commit)
convert_tasks_notes_list_to_dict(commit)
convert_old_boardgame_log_to_new(commit)

View File

@ -450,11 +450,11 @@ def email_scrobble_board_game(
locations[location_dict.get("id")] = location
scrobbles_created = []
second = 0
for play_dict in bgstat_data.get("plays", []):
hour = None
minute = None
second = None
comments = None
if "comments" in play_dict.keys():
for line in play_dict.get("comments", "").split("\n"):
if "Learning to play" in line:
@ -469,7 +469,7 @@ def email_scrobble_board_game(
except IndexError:
second = 0
log_data["notes"] = [play_dict.get("comments")]
comments = play_dict.get("comments")
log_data["expansion_ids"] = []
try:
base_game = base_games[play_dict.get("gameRefId")]
@ -527,6 +527,9 @@ def email_scrobble_board_game(
duration_seconds = base_game.run_time_seconds
stop_timestamp = timestamp + timedelta(seconds=duration_seconds)
if comments:
log_data["notes"] = {str(int(stop_timestamp.timestamp())): comments}
logger.info(f"Creating scrobble for {base_game} at {timestamp}")
log_data["raw_data"] = bgstat_data
scrobble_dict = {
@ -685,9 +688,10 @@ def todoist_scrobble_update_task(
)
return
timestamp = todoist_note.get("posted_at") or str(int(timezone.now().timestamp()))
if not scrobble.log.get("notes"):
scrobble.log["notes"] = []
scrobble.log["notes"].append(todoist_note.get("notes"))
scrobble.log["notes"] = {}
scrobble.log["notes"][timestamp] = todoist_note.get("notes")
scrobble.save(update_fields=["log"])
logger.info(
"[todoist_scrobble_update_task] todoist note added",
@ -783,7 +787,10 @@ def todoist_scrobble_task(
def emacs_scrobble_update_task(
emacs_id: str, emacs_notes: dict, user_id: int
emacs_id: str,
emacs_notes: list,
user_id: int,
description: Optional[str] = None,
) -> Optional[Scrobble]:
scrobble = Scrobble.objects.filter(
in_progress=True,
@ -803,27 +810,30 @@ def emacs_scrobble_update_task(
)
return
notes_updated = False
log_updated = False
if not scrobble.log.get("notes"):
scrobble.log["notes"] = {}
for note in emacs_notes:
try:
existing_note_ts = [
n.get("timestamp") for n in scrobble.log.get("notes", [])
]
except AttributeError:
existing_note_ts = []
if not scrobble.log.get('notes"'):
scrobble.log["notes"] = []
if note.get("timestamp") not in existing_note_ts:
scrobble.log["notes"].append({note.get("timestamp"): note.get("content")})
notes_updated = True
timestamp = note.get("timestamp")
content = note.get("content")
if timestamp:
existing = scrobble.log["notes"].get(timestamp)
if existing != content:
scrobble.log["notes"][timestamp] = content
log_updated = True
if notes_updated:
if description is not None and scrobble.log.get("description") != description:
scrobble.log["description"] = description
log_updated = True
if log_updated:
scrobble.save(update_fields=["log"])
logger.info(
"[emacs_scrobble_update_task] emacs note added",
"[emacs_scrobble_update_task] emacs scrobble updated",
extra={
"emacs_note": emacs_notes,
"emacs_id": emacs_id,
"user_id": user_id,
"media_type": Scrobble.MediaType.TASK,
},
@ -865,7 +875,7 @@ def emacs_scrobble_task(
logger.info(
"[emacs_scrobble_task] cannot start already started task",
extra={
"ormode_id": orgmode_id,
"orgmode_id": orgmode_id,
},
)
return in_progress_scrobble
@ -884,9 +894,7 @@ def emacs_scrobble_task(
if in_progress_scrobble:
return in_progress_scrobble
notes = task_data.pop("notes")
if notes:
task_data["notes"] = [note.get("content") for note in notes]
task_data.pop("notes", None)
task_data["title"] = task_data.pop("description")
task_data["description"] = task_data.pop("body")
task_data["labels"] = task_data.pop("labels")

View File

@ -1024,7 +1024,11 @@ class ScrobbleDetailView(DetailView):
FormClass = self.get_form_class()
log = self.object.log or {}
log["notes"] = self.object.logdata.notes_as_str(separator="\n")
notes = log.get("notes")
if isinstance(notes, dict):
log["notes"] = notes
else:
log["notes"] = self.object.logdata.notes_as_str(separator="\n")
return FormClass(initial=log)

View File

@ -0,0 +1,47 @@
import json
from django import forms
class TaskNotesWidget(forms.Widget):
template_name = "tasks/task_notes_widget.html"
class Media:
js = ("tasks/task_notes.js",)
def value_from_datadict(self, data, files, name):
timestamps = data.getlist(f"{name}_timestamps")
contents = data.getlist(f"{name}_contents")
return {
"timestamps": timestamps,
"contents": contents,
}
def get_context(self, name, value, attrs):
context = super().get_context(name, value, attrs)
notes = {}
if value:
if isinstance(value, str):
try:
notes = json.loads(value)
except (json.JSONDecodeError, TypeError):
notes = {}
elif isinstance(value, dict):
notes = value
context["widget"]["notes"] = notes
return context
class TaskNotesField(forms.Field):
widget = TaskNotesWidget
def clean(self, value):
if not value:
return {}
result = {}
timestamps = value.get("timestamps", [])
contents = value.get("contents", [])
for i, ts in enumerate(timestamps):
if i < len(contents) and ts and contents[i].strip():
result[ts] = contents[i].strip()
return result if result else {}

View File

@ -38,6 +38,18 @@ class TaskLogData(BaseLogData):
"todoist_project_id",
}
@classmethod
def override_fields(cls) -> dict:
from scrobbles.forms import NotesDictField
fields = {}
for base in cls.mro()[1:]:
if hasattr(base, "override_fields"):
base_fields = base.override_fields()
fields.update(base_fields)
fields["notes"] = NotesDictField(required=False)
return fields
def notes_as_str(self, separator: str = " | ") -> str:
"""Return formatted notes with line breaks and no keys"""
labels_str = ""
@ -47,13 +59,14 @@ class TaskLogData(BaseLogData):
lines = []
if self.notes:
notes = self.notes
if isinstance(notes, dict):
notes = [{k: v} for k, v in notes.items()]
if isinstance(notes, str):
notes = [notes]
for note in notes:
if isinstance(note, dict):
timestamp, note_text = next(iter(note.items()))
# Flatten newlines and clean whitespace
note_text = " ".join(note_text.strip().split())
lines.append(f"{timestamp}: {note_text}")
if isinstance(note, list):
@ -67,6 +80,8 @@ class TaskLogData(BaseLogData):
return ""
notes = self.notes
if isinstance(notes, dict):
notes = [{k: v} for k, v in notes.items()]
if isinstance(notes, str):
notes = [notes]

View File

@ -0,0 +1,36 @@
(function() {
function addNoteRow(container, widgetName, timestamp, content) {
var row = document.createElement('div');
row.className = 'task-note-row row mb-2 align-items-start';
row.innerHTML =
'<div class="col-md-3">' +
'<input type="hidden" name="' + widgetName + '_timestamps" value="' + timestamp + '">' +
'</div>' +
'<div class="col-md-7">' +
'<textarea name="' + widgetName + '_contents" class="form-control" rows="2">' + (content || '') + '</textarea>' +
'</div>' +
'<div class="col-md-2">' +
'<button type="button" class="btn btn-sm btn-outline-danger remove-note">&times;</button>' +
'</div>';
container.appendChild(row);
}
document.addEventListener('click', function(e) {
var addBtn = e.target.closest('.add-note');
if (addBtn) {
e.preventDefault();
var container = document.getElementById('task-notes-container');
var now = Math.floor(Date.now() / 1000);
var widgetName = addBtn.getAttribute('data-widget-name');
addNoteRow(container, widgetName, String(now), '');
}
});
document.addEventListener('click', function(e) {
var removeBtn = e.target.closest('.remove-note');
if (removeBtn) {
e.preventDefault();
removeBtn.closest('.task-note-row').remove();
}
});
})();

View File

@ -0,0 +1,17 @@
{% load static %}
<div id="task-notes-container" class="task-notes-widget">
{% for timestamp, content in widget.notes.items %}
<div class="task-note-row row mb-2 align-items-start">
<div class="col-md-3">
<input type="hidden" name="{{widget.name}}_timestamps" value="{{timestamp}}">
</div>
<div class="col-md-7">
<textarea name="{{widget.name}}_contents" class="form-control" rows="2">{{content}}</textarea>
</div>
<div class="col-md-2">
<button type="button" class="btn btn-sm btn-outline-danger remove-note">&times;</button>
</div>
</div>
{% endfor %}
</div>
<button type="button" class="btn btn-sm btn-outline-primary add-note" data-widget-name="{{widget.name}}">+ Add Note</button>

View File

@ -83,18 +83,50 @@ def convert_notes_to_dict(commit=False):
scrobble.log["notes"] = [
value for d in note_list for value in d.values()
]
else:
scrobble.log["notes"] = note_list
count += 1
if commit:
scrobble.save(update_fields=["log"])
print(f"Updated {count} todoist tasks scrobbles")
def convert_tasks_notes_list_to_dict(commit=False):
scrobbles = Scrobble.objects.filter(task__isnull=False, log__notes__isnull=False)
count = 0
for scrobble in scrobbles:
notes = scrobble.log.get("notes")
if isinstance(notes, list):
key = str(int(scrobble.timestamp.timestamp()) + 10)
parts = []
for note in notes:
if isinstance(note, dict):
parts.append(next(iter(note.values()), ""))
elif isinstance(note, list):
parts.extend(str(x) for x in note)
else:
parts.append(str(note))
scrobble.log["notes"] = {key: "\n".join(parts)}
count += 1
if commit:
scrobble.save(update_fields=["log"])
print(f"Updated {count} task scrobbles notes from list to dict")
def convert_old_boardgame_log_to_new(commit=False):
scrobbles = Scrobble.objects.filter(board_game__isnull=False, log__has_key="notes")
count = 0
for scrobble in scrobbles:
if isinstance(scrobble.log.get("notes"), str):
scrobble.log["notes"] = [scrobble.log.pop("notes")]
notes = scrobble.log.get("notes")
if isinstance(notes, str):
scrobble.log["notes"] = [notes]
notes = [notes]
if isinstance(notes, list):
key_ts = scrobble.stop_timestamp or scrobble.timestamp
scrobble.log["notes"] = {str(int(key_ts.timestamp())): "\n".join(
str(n) for n in notes
)}
count += 1
if commit:
scrobble.save(update_fields=["log"])
print(f"Updated {scrobbles.count()} board game scrobbles")
print(f"Updated {count} board game scrobbles")

View File

@ -136,6 +136,7 @@ class TodoistWebhookView(APIView):
"todoist_type": todoist_type,
"todoist_event": todoist_event,
"updated_at": task_data.get("updated_at"),
"posted_at": event_data.get("posted_at"),
"details": task_data.get("description"),
"notes": event_data.get("content"),
"is_deleted": (
@ -231,11 +232,12 @@ class EmacsWebhookView(APIView):
status=status.HTTP_304_NOT_MODIFIED,
)
if task_in_progress and post_data.get("notes"):
if task_in_progress:
emacs_scrobble_update_task(
post_data.get("source_id"),
post_data.get("notes"),
post_data.get("notes", []),
user_id,
description=post_data.get("body"),
)
self.logger.info(