Files

T

xiaoming 069af30dba Initial import: Music_Server, MusicFree, catalog-sync

2026-05-23 16:51:14 +08:00

33 KiB

Raw Blame History

Music_Cloud Snapshot and Private Origin Implementation Plan

For agentic workers: REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (- [ ]) syntax for tracking.

Goal: Add snapshot export on the Music_Cloud side so the public service can consume a read-only catalog mirror, and add the signed private-origin primitives needed for NAS-only fallback streaming.

Architecture: Keep all authoritative writes in Music_Cloud, export a compact read model into catalog_read.db, publish it with a manifest.json, and add signed-token helpers that the live NAS web stack can use for private-origin playback without exposing filesystem paths.

Tech Stack: Python 3, sqlite3, dataclasses, json, hashlib, hmac, base64, unittest

Repository root: D:\source\musicdl-catalog-sync-worktrees\catalog-sync

File Structure

Create: musicdl/catalogsync/export/__init__.py
Create: musicdl/catalogsync/export/models.py
Create: musicdl/catalogsync/export/service.py
Create: musicdl/catalogsync/private_origin/__init__.py
Create: musicdl/catalogsync/private_origin/tokens.py
Create: tests/catalogsync/test_snapshot_models.py
Create: tests/catalogsync/test_snapshot_export.py
Create: tests/catalogsync/test_private_origin_tokens.py
Modify: musicdl/catalogsync/cli.py
Create: scripts/catalogsync/export_snapshot.ps1

Task 1: Define the snapshot manifest contract

Files:

Create: musicdl/catalogsync/export/__init__.py
Create: musicdl/catalogsync/export/models.py
Test: tests/catalogsync/test_snapshot_models.py
Step 1: Write the failing test

import unittest

from musicdl.catalogsync.export.models import SnapshotManifest


class SnapshotManifestTests(unittest.TestCase):
    def test_round_trip_manifest_dict(self):
        manifest = SnapshotManifest(
            snapshot_id="snap-001",
            generated_at="2026-04-19T08:00:00Z",
            schema_version=1,
            playlist_count=12,
            track_count=34,
            file_count=56,
            cover_count=7,
        )

        payload = manifest.to_dict()
        restored = SnapshotManifest.from_dict(payload)

        self.assertEqual("snap-001", restored.snapshot_id)
        self.assertEqual(34, restored.track_count)


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_snapshot_models -v
Expected: ERROR with ModuleNotFoundError: No module named 'musicdl.catalogsync.export.models'

Step 3: Write minimal implementation

musicdl/catalogsync/export/__init__.py

from .models import SnapshotManifest

__all__ = ["SnapshotManifest"]

musicdl/catalogsync/export/models.py

from dataclasses import asdict, dataclass


@dataclass(frozen=True)
class SnapshotManifest:
    snapshot_id: str
    generated_at: str
    schema_version: int
    playlist_count: int
    track_count: int
    file_count: int
    cover_count: int

    def to_dict(self) -> dict:
        return asdict(self)

    @classmethod
    def from_dict(cls, payload: dict) -> "SnapshotManifest":
        return cls(
            snapshot_id=str(payload["snapshot_id"]),
            generated_at=str(payload["generated_at"]),
            schema_version=int(payload["schema_version"]),
            playlist_count=int(payload["playlist_count"]),
            track_count=int(payload["track_count"]),
            file_count=int(payload["file_count"]),
            cover_count=int(payload["cover_count"]),
        )

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_snapshot_models -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_snapshot_models.py musicdl/catalogsync/export/__init__.py musicdl/catalogsync/export/models.py
git commit -m "feat: add snapshot manifest model"

Task 2: Export a read-only catalog snapshot database

Files:

Create: musicdl/catalogsync/export/service.py
Test: tests/catalogsync/test_snapshot_export.py
Step 1: Write the failing test

import json
import sqlite3
import tempfile
import unittest
from pathlib import Path

from musicdl.catalogsync.export.service import export_snapshot


class SnapshotExportTests(unittest.TestCase):
    def test_export_snapshot_writes_read_model_and_manifest(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            tmpdir_path = Path(tmpdir)
            source_db = tmpdir_path / "catalogsync.db"
            target_dir = tmpdir_path / "snapshot"
            target_db = target_dir / "catalog_read.db"
            target_manifest = target_dir / "manifest.json"

            conn = sqlite3.connect(source_db)
            conn.executescript(
                '''
                create table playlists (
                    id integer primary key,
                    platform text not null,
                    remote_playlist_id text not null,
                    name text not null,
                    description text,
                    cover_url text,
                    play_count integer default 0,
                    collected_song_count integer default 0
                );
                create table songs (
                    id integer primary key,
                    platform text not null,
                    remote_song_id text not null,
                    name text not null,
                    singers text,
                    ext text,
                    file_size_bytes integer,
                    metadata_json text
                );
                create table playlist_songs (
                    playlist_id integer not null,
                    song_id integer not null,
                    position integer not null
                );
                insert into playlists values (1, 'netease', '18165', '娴嬭瘯姝屽崟', 'desc', 'https://img/1.jpg', 99, 1);
                insert into songs values (10, 'netease', '65800', '娴峰笨浣?, '椹篃 / Crabbit', 'flac', 123456, '{"duration_ms": 0}');
                insert into playlist_songs values (1, 10, 1);
                '''
            )
            conn.commit()
            conn.close()

            manifest = export_snapshot(source_db=source_db, target_dir=target_dir)

            self.assertTrue(target_db.exists())
            self.assertTrue(target_manifest.exists())
            self.assertEqual(1, manifest.playlist_count)
            self.assertEqual(1, manifest.track_count)

            read_conn = sqlite3.connect(target_db)
            row = read_conn.execute(
                "select playlist_id, name, play_count, song_count from catalog_playlists"
            ).fetchone()
            read_conn.close()

            self.assertEqual((1, "娴嬭瘯姝屽崟", 99, 1), row)

            payload = json.loads(target_manifest.read_text(encoding="utf-8"))
            self.assertEqual("snap-playlists-1-tracks-1", payload["snapshot_id"])


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_snapshot_export -v
Expected: ERROR with ImportError because export_snapshot does not exist yet

Step 3: Write minimal implementation

musicdl/catalogsync/export/service.py

import json
import sqlite3
from datetime import datetime, timezone
from pathlib import Path

from .models import SnapshotManifest


def export_snapshot(source_db: Path, target_dir: Path) -> SnapshotManifest:
    source_db = Path(source_db)
    target_dir = Path(target_dir)
    target_dir.mkdir(parents=True, exist_ok=True)
    target_db = target_dir / "catalog_read.db"
    target_manifest = target_dir / "manifest.json"

    if target_db.exists():
        target_db.unlink()

    source_conn = sqlite3.connect(source_db)
    source_conn.row_factory = sqlite3.Row
    target_conn = sqlite3.connect(target_db)

    target_conn.executescript(
        """
        create table catalog_playlists (
            playlist_id integer primary key,
            platform text not null,
            remote_playlist_id text not null,
            name text not null,
            description text,
            cover_url text,
            play_count integer not null default 0,
            song_count integer not null default 0
        );
        create table catalog_tracks (
            song_id integer primary key,
            platform text not null,
            remote_song_id text not null,
            name text not null,
            singers text,
            ext text,
            file_size_bytes integer,
            metadata_json text
        );
        create table catalog_playlist_tracks (
            playlist_id integer not null,
            song_id integer not null,
            position integer not null
        );
        """
    )

    playlists = source_conn.execute(
        """
        select
            p.id as playlist_id,
            p.platform,
            p.remote_playlist_id,
            p.name,
            p.description,
            p.cover_url,
            coalesce(p.play_count, 0) as play_count,
            coalesce(p.collected_song_count, 0) as song_count
        from playlists p
        """
    ).fetchall()

    tracks = source_conn.execute(
        """
        select
            s.id as song_id,
            s.platform,
            s.remote_song_id,
            s.name,
            s.singers,
            s.ext,
            s.file_size_bytes,
            s.metadata_json
        from songs s
        """
    ).fetchall()

    playlist_tracks = source_conn.execute(
        """
        select playlist_id, song_id, position
        from playlist_songs
        order by playlist_id, position
        """
    ).fetchall()

    target_conn.executemany(
        """
        insert into catalog_playlists (
            playlist_id, platform, remote_playlist_id, name, description, cover_url, play_count, song_count
        ) values (
            :playlist_id, :platform, :remote_playlist_id, :name, :description, :cover_url, :play_count, :song_count
        )
        """,
        playlists,
    )
    target_conn.executemany(
        """
        insert into catalog_tracks (
            song_id, platform, remote_song_id, name, singers, ext, file_size_bytes, metadata_json
        ) values (
            :song_id, :platform, :remote_song_id, :name, :singers, :ext, :file_size_bytes, :metadata_json
        )
        """,
        tracks,
    )
    target_conn.executemany(
        """
        insert into catalog_playlist_tracks (playlist_id, song_id, position)
        values (:playlist_id, :song_id, :position)
        """,
        playlist_tracks,
    )
    target_conn.commit()
    target_conn.close()
    source_conn.close()

    manifest = SnapshotManifest(
        snapshot_id=f"snap-playlists-{len(playlists)}-tracks-{len(tracks)}",
        generated_at=datetime.now(timezone.utc).isoformat(),
        schema_version=1,
        playlist_count=len(playlists),
        track_count=len(tracks),
        file_count=0,
        cover_count=sum(1 for row in playlists if row["cover_url"]),
    )
    target_manifest.write_text(
        json.dumps(manifest.to_dict(), ensure_ascii=False, indent=2),
        encoding="utf-8",
    )
    return manifest

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_snapshot_export -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_snapshot_export.py musicdl/catalogsync/export/service.py
git commit -m "feat: export read-only catalog snapshot"

Task 3: Add a CLI entrypoint and operator script for snapshot export

Files:

Modify: musicdl/catalogsync/cli.py
Create: scripts/catalogsync/export_snapshot.ps1
Test: tests/catalogsync/test_snapshot_cli.py
Step 1: Write the failing test

import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch

from musicdl.catalogsync.cli import main


class SnapshotCliTests(unittest.TestCase):
    def test_export_snapshot_command_dispatches_to_service(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            source_db = Path(tmpdir) / "catalogsync.db"
            source_db.touch()
            target_dir = Path(tmpdir) / "snapshot"

            with patch("musicdl.catalogsync.cli.export_snapshot") as export_mock:
                exit_code = main(
                    [
                        "export-snapshot",
                        "--db",
                        str(source_db),
                        "--target-dir",
                        str(target_dir),
                    ]
                )

            self.assertEqual(0, exit_code)
            export_mock.assert_called_once_with(source_db=source_db, target_dir=target_dir)


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_snapshot_cli -v
Expected: ERROR because the parser does not know export-snapshot

Step 3: Write minimal implementation

musicdl/catalogsync/cli.py

from pathlib import Path
import argparse

from musicdl.catalogsync.export.service import export_snapshot


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(prog="musicdl-catalogsync")
    subparsers = parser.add_subparsers(dest="command", required=True)

    export_parser = subparsers.add_parser("export-snapshot")
    export_parser.add_argument("--db", required=True)
    export_parser.add_argument("--target-dir", required=True)

    return parser


def main(argv=None) -> int:
    parser = build_parser()
    args = parser.parse_args(argv)

    if args.command == "export-snapshot":
        export_snapshot(source_db=Path(args.db), target_dir=Path(args.target_dir))
        return 0

    parser.error(f"unsupported command: {args.command}")

scripts/catalogsync/export_snapshot.ps1

param(
    [Parameter(Mandatory = $true)]
    [string]$DbPath,
    [Parameter(Mandatory = $true)]
    [string]$TargetDir
)

$ErrorActionPreference = "Stop"
python -m musicdl.catalogsync.cli export-snapshot --db $DbPath --target-dir $TargetDir
if ($LASTEXITCODE -ne 0) {
    throw "export-snapshot failed with exit code $LASTEXITCODE"
}
Write-Host "Snapshot exported to $TargetDir"

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_snapshot_cli -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_snapshot_cli.py musicdl/catalogsync/cli.py scripts/catalogsync/export_snapshot.ps1
git commit -m "feat: add snapshot export command"

Task 4: Add signed private-origin token helpers for NAS fallback playback

Files:

Create: musicdl/catalogsync/private_origin/__init__.py
Create: musicdl/catalogsync/private_origin/tokens.py
Test: tests/catalogsync/test_private_origin_tokens.py
Step 1: Write the failing test

import time
import unittest

from musicdl.catalogsync.private_origin.tokens import create_origin_token, verify_origin_token


class PrivateOriginTokenTests(unittest.TestCase):
    def test_create_and_verify_origin_token(self):
        now = int(time.time())
        token = create_origin_token(
            secret="test-secret",
            locator="netease/椹篃/娴峰笨浣?flac",
            expires_at=now + 300,
        )

        payload = verify_origin_token(
            secret="test-secret",
            token=token,
            now=now,
        )

        self.assertEqual("netease/椹篃/娴峰笨浣?flac", payload["locator"])


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_private_origin_tokens -v
Expected: ERROR with ModuleNotFoundError

Step 3: Write minimal implementation

musicdl/catalogsync/private_origin/__init__.py

from .tokens import create_origin_token, verify_origin_token

__all__ = ["create_origin_token", "verify_origin_token"]

musicdl/catalogsync/private_origin/tokens.py

import base64
import hashlib
import hmac
import json


def _sign(secret: str, payload_json: str) -> str:
    digest = hmac.new(
        secret.encode("utf-8"),
        payload_json.encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()
    return digest


def create_origin_token(secret: str, locator: str, expires_at: int) -> str:
    payload = {"locator": locator, "expires_at": int(expires_at)}
    payload_json = json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
    signature = _sign(secret=secret, payload_json=payload_json)
    envelope = {"payload": payload, "sig": signature}
    return base64.urlsafe_b64encode(
        json.dumps(envelope, ensure_ascii=False, separators=(",", ":")).encode("utf-8")
    ).decode("ascii")


def verify_origin_token(secret: str, token: str, now: int) -> dict:
    envelope_json = base64.urlsafe_b64decode(token.encode("ascii")).decode("utf-8")
    envelope = json.loads(envelope_json)
    payload = envelope["payload"]
    payload_json = json.dumps(payload, ensure_ascii=False, separators=(",", ":"))
    expected_sig = _sign(secret=secret, payload_json=payload_json)
    if not hmac.compare_digest(expected_sig, envelope["sig"]):
        raise ValueError("invalid signature")
    if int(payload["expires_at"]) < int(now):
        raise ValueError("token expired")
    return payload

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_private_origin_tokens -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_private_origin_tokens.py musicdl/catalogsync/private_origin/__init__.py musicdl/catalogsync/private_origin/tokens.py
git commit -m "feat: add private origin token helpers"

Task 5: Expand snapshot export to file availability and toplist read models

Files:

Modify: musicdl/catalogsync/export/service.py
Test: tests/catalogsync/test_snapshot_export.py
Step 1: Write the failing test

import sqlite3
import tempfile
import unittest
from pathlib import Path

from musicdl.catalogsync.export.service import export_snapshot


class SnapshotExportExtendedTests(unittest.TestCase):
    def test_export_snapshot_writes_track_files_and_toplists(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            tmpdir_path = Path(tmpdir)
            source_db = tmpdir_path / "catalogsync.db"
            target_dir = tmpdir_path / "snapshot"

            conn = sqlite3.connect(source_db)
            conn.executescript(
                '''
                create table playlists (
                    id integer primary key,
                    platform text not null,
                    remote_playlist_id text not null,
                    name text not null,
                    description text,
                    cover_url text,
                    play_count integer default 0,
                    collected_song_count integer default 0
                );
                create table songs (
                    id integer primary key,
                    platform text not null,
                    remote_song_id text not null,
                    name text not null,
                    singers text,
                    ext text,
                    file_size_bytes integer,
                    metadata_json text
                );
                create table playlist_songs (playlist_id integer not null, song_id integer not null, position integer not null);
                create table playlist_pools (
                    id integer primary key,
                    platform text not null,
                    pool_kind text not null,
                    external_id text not null,
                    name text not null,
                    url text,
                    metadata_json text
                );
                create table pool_playlists (pool_id integer not null, playlist_id integer not null);
                create table storage_backends (
                    id integer primary key,
                    name text not null,
                    backend_type text not null,
                    base_path text,
                    config_json text
                );
                create table file_assets (
                    id integer primary key,
                    song_id integer not null,
                    quality_label text,
                    ext text,
                    file_size_bytes integer
                );
                create table file_locations (
                    id integer primary key,
                    file_asset_id integer not null,
                    backend_id integer not null,
                    locator text not null,
                    status text not null,
                    is_primary integer not null
                );
                insert into playlists values (1, 'kuwo', '16', '酷我飙升榜', 'desc', 'https://img/top.jpg', 1000, 1);
                insert into songs values (10, 'netease', '65800', '海屿你', '马也 / Crabbit', 'flac', 123456, '{"duration_ms": 0}');
                insert into playlist_songs values (1, 10, 1);
                insert into playlist_pools values (5, 'kuwo', 'toplist', 'kuwo_top_16', '酷我榜单', null, '{}');
                insert into pool_playlists values (5, 1);
                insert into storage_backends values (2, 'main-s3', 'object_storage', null, '{"public_base_url": "https://cdn.example"}');
                insert into file_assets values (7, 10, 'super', 'flac', 123456);
                insert into file_locations values (9, 7, 2, 'music/netease/test.flac', 'active', 1);
                '''
            )
            conn.commit()
            conn.close()

            export_snapshot(source_db=source_db, target_dir=target_dir)

            read_conn = sqlite3.connect(target_dir / "catalog_read.db")
            track_file_row = read_conn.execute(
                "select song_id, backend_type, backend_name, locator from catalog_track_files"
            ).fetchone()
            toplist_row = read_conn.execute(
                "select toplist_id, group_name, song_count from catalog_toplists"
            ).fetchone()
            read_conn.close()

            self.assertEqual((10, "object_storage", "main-s3", "music/netease/test.flac"), track_file_row)
            self.assertEqual(("kuwo_top_16", "酷我榜单", 1), toplist_row)


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_snapshot_export.SnapshotExportExtendedTests -v
Expected: FAIL with sqlite3.OperationalError: no such table: catalog_track_files

Step 3: Write minimal implementation

musicdl/catalogsync/export/service.py

import json
import sqlite3
from datetime import datetime, timezone
from pathlib import Path

from .models import SnapshotManifest


def export_snapshot(source_db: Path, target_dir: Path) -> SnapshotManifest:
    source_db = Path(source_db)
    target_dir = Path(target_dir)
    target_dir.mkdir(parents=True, exist_ok=True)
    target_db = target_dir / "catalog_read.db"
    target_manifest = target_dir / "manifest.json"

    if target_db.exists():
        target_db.unlink()

    source_conn = sqlite3.connect(source_db)
    source_conn.row_factory = sqlite3.Row
    target_conn = sqlite3.connect(target_db)

    target_conn.executescript(
        """
        create table catalog_playlists (
            playlist_id integer primary key,
            platform text not null,
            remote_playlist_id text not null,
            name text not null,
            description text,
            cover_url text,
            play_count integer not null default 0,
            song_count integer not null default 0
        );
        create table catalog_tracks (
            song_id integer primary key,
            platform text not null,
            remote_song_id text not null,
            name text not null,
            singers text,
            ext text,
            file_size_bytes integer,
            metadata_json text
        );
        create table catalog_playlist_tracks (
            playlist_id integer not null,
            song_id integer not null,
            position integer not null
        );
        create table catalog_track_files (
            song_id integer not null,
            quality_label text,
            ext text,
            file_size_bytes integer,
            backend_type text,
            backend_name text,
            locator text,
            public_url text,
            status text,
            is_primary integer
        );
        create table catalog_toplists (
            toplist_id text primary key,
            platform text not null,
            name text not null,
            description text,
            cover_url text,
            play_count integer not null,
            song_count integer not null,
            group_name text not null
        );
        """
    )

    playlists = source_conn.execute(
        """
        select
            p.id as playlist_id,
            p.platform,
            p.remote_playlist_id,
            p.name,
            p.description,
            p.cover_url,
            coalesce(p.play_count, 0) as play_count,
            coalesce(p.collected_song_count, 0) as song_count
        from playlists p
        """
    ).fetchall()
    tracks = source_conn.execute(
        """
        select
            s.id as song_id,
            s.platform,
            s.remote_song_id,
            s.name,
            s.singers,
            s.ext,
            s.file_size_bytes,
            s.metadata_json
        from songs s
        """
    ).fetchall()
    playlist_tracks = source_conn.execute(
        """
        select playlist_id, song_id, position
        from playlist_songs
        order by playlist_id, position
        """
    ).fetchall()
    track_files = source_conn.execute(
        """
        select
            fa.song_id,
            fa.quality_label,
            fa.ext,
            fa.file_size_bytes,
            sb.backend_type,
            sb.name as backend_name,
            fl.locator,
            case
                when sb.backend_type = 'object_storage'
                then coalesce(json_extract(sb.config_json, '$.public_base_url'), '') || '/' || fl.locator
                else null
            end as public_url,
            fl.status,
            fl.is_primary
        from file_locations fl
        join file_assets fa on fa.id = fl.file_asset_id
        join storage_backends sb on sb.id = fl.backend_id
        where fl.status = 'active'
        """
    ).fetchall()
    toplists = source_conn.execute(
        """
        select
            pp.external_id as toplist_id,
            p.platform,
            p.name,
            p.description,
            p.cover_url,
            coalesce(p.play_count, 0) as play_count,
            coalesce(p.collected_song_count, 0) as song_count,
            pp.name as group_name
        from playlists p
        join pool_playlists rel on rel.playlist_id = p.id
        join playlist_pools pp on pp.id = rel.pool_id
        where pp.pool_kind = 'toplist'
        """
    ).fetchall()

    target_conn.executemany(
        """
        insert into catalog_playlists (
            playlist_id, platform, remote_playlist_id, name, description, cover_url, play_count, song_count
        ) values (
            :playlist_id, :platform, :remote_playlist_id, :name, :description, :cover_url, :play_count, :song_count
        )
        """,
        playlists,
    )
    target_conn.executemany(
        """
        insert into catalog_tracks (
            song_id, platform, remote_song_id, name, singers, ext, file_size_bytes, metadata_json
        ) values (
            :song_id, :platform, :remote_song_id, :name, :singers, :ext, :file_size_bytes, :metadata_json
        )
        """,
        tracks,
    )
    target_conn.executemany(
        """
        insert into catalog_playlist_tracks (playlist_id, song_id, position)
        values (:playlist_id, :song_id, :position)
        """,
        playlist_tracks,
    )
    target_conn.executemany(
        """
        insert into catalog_track_files (
            song_id, quality_label, ext, file_size_bytes, backend_type, backend_name, locator, public_url, status, is_primary
        ) values (
            :song_id, :quality_label, :ext, :file_size_bytes, :backend_type, :backend_name, :locator, :public_url, :status, :is_primary
        )
        """,
        track_files,
    )
    target_conn.executemany(
        """
        insert into catalog_toplists (
            toplist_id, platform, name, description, cover_url, play_count, song_count, group_name
        ) values (
            :toplist_id, :platform, :name, :description, :cover_url, :play_count, :song_count, :group_name
        )
        """,
        toplists,
    )
    target_conn.commit()
    target_conn.close()
    source_conn.close()

    manifest = SnapshotManifest(
        snapshot_id=f"snap-playlists-{len(playlists)}-tracks-{len(tracks)}",
        generated_at=datetime.now(timezone.utc).isoformat(),
        schema_version=1,
        playlist_count=len(playlists),
        track_count=len(tracks),
        file_count=len(track_files),
        cover_count=sum(1 for row in playlists if row["cover_url"]),
    )
    target_manifest.write_text(json.dumps(manifest.to_dict(), ensure_ascii=False, indent=2), encoding="utf-8")
    return manifest

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_snapshot_export.SnapshotExportExtendedTests -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_snapshot_export.py musicdl/catalogsync/export/service.py
git commit -m "feat: export track files and toplists in snapshot"

Task 6: Add the NAS private-origin streaming route to the ops web app

Files:

Create: musicdl/catalogsync/private_origin/service.py
Modify: musicdl/catalogsync/ops/web.py
Test: tests/catalogsync/test_private_origin_web.py
Step 1: Write the failing test

import tempfile
import unittest
from pathlib import Path

from fastapi.testclient import TestClient

from musicdl.catalogsync.ops.web import create_app
from musicdl.catalogsync.private_origin.tokens import create_origin_token


class PrivateOriginWebTests(unittest.TestCase):
    def test_private_origin_stream_serves_local_file_when_token_is_valid(self):
        with tempfile.TemporaryDirectory() as tmpdir:
            music_file = Path(tmpdir) / "test.flac"
            music_file.write_bytes(b"test-audio")
            token = create_origin_token(
                secret="secret-123",
                locator=str(music_file),
                expires_at=4102444800,
            )

            app = create_app(
                db_path=Path(tmpdir) / "catalogsync.db",
                env_file=Path(tmpdir) / "catalogsync.env",
            )
            app.state.private_origin_secret = "secret-123"
            client = TestClient(app)
            response = client.get(f"/api/private-origin/stream/{token}")

            self.assertEqual(200, response.status_code)
            self.assertEqual(b"test-audio", response.content)


if __name__ == "__main__":
    unittest.main()

Step 2: Run test to verify it fails

Run: python -m unittest tests.catalogsync.test_private_origin_web -v
Expected: FAIL with 404 Not Found for /api/private-origin/stream/{token}

Step 3: Write minimal implementation

musicdl/catalogsync/private_origin/service.py

from pathlib import Path

from fastapi import HTTPException

from .tokens import verify_origin_token


def resolve_private_origin_file(secret: str, token: str, now: int) -> Path:
    payload = verify_origin_token(secret=secret, token=token, now=now)
    file_path = Path(payload["locator"]).resolve()
    if not file_path.exists() or not file_path.is_file():
        raise HTTPException(status_code=404, detail="private origin file not found")
    return file_path

musicdl/catalogsync/ops/web.py

import time
from pathlib import Path

from fastapi import FastAPI, HTTPException
from fastapi.responses import FileResponse

from musicdl.catalogsync.private_origin.service import resolve_private_origin_file


def create_app(db_path: Path, env_file: Path) -> FastAPI:
    app = FastAPI(title="Catalogsync Operations Console")

    @app.get("/api/private-origin/stream/{token}")
    def private_origin_stream(token: str):
        secret = getattr(app.state, "private_origin_secret", "")
        if not secret:
            raise HTTPException(status_code=503, detail="private origin secret not configured")
        file_path = resolve_private_origin_file(secret=secret, token=token, now=int(time.time()))
        return FileResponse(path=file_path, filename=file_path.name)

    return app

Step 4: Run test to verify it passes

Run: python -m unittest tests.catalogsync.test_private_origin_web -v
Expected: OK

Step 5: Commit

git add tests/catalogsync/test_private_origin_web.py musicdl/catalogsync/private_origin/service.py musicdl/catalogsync/ops/web.py
git commit -m "feat: add nas private origin stream route"

33 KiB Raw Blame History

Music_Cloud Snapshot and Private Origin Implementation Plan

File Structure

Task 1: Define the snapshot manifest contract

Task 2: Export a read-only catalog snapshot database

Task 3: Add a CLI entrypoint and operator script for snapshot export

Task 4: Add signed private-origin token helpers for NAS fallback playback

Task 5: Expand snapshot export to file availability and toplist read models

Task 6: Add the NAS private-origin streaming route to the ops web app

33 KiB

Raw Blame History