Initial import: Music_Server, MusicFree, catalog-sync
This commit is contained in:
@@ -0,0 +1,70 @@
|
||||
# Catalog Sync Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add an independent catalog-sync CLI that harvests playlist pools, persists playlist/song/artist relationships, and automates deduplicated downloads.
|
||||
|
||||
**Architecture:** Implement a new `musicdl.catalogsync` package with a SQLite repository layer, per-source collectors, playlist-sync services, and a CLI wrapper that reuses existing `musicdl` source clients for parsing and downloading. Model downloaded media as logical file assets plus storage locations so local paths work now and cloud/object-storage locations fit later.
|
||||
|
||||
**Tech Stack:** Python stdlib (`sqlite3`, `json`, `pathlib`, `hashlib`, `shutil`), `click`, existing `musicdl` source clients, `requests`, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add failing tests for schema and normalization helpers
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/catalogsync/test_db.py`
|
||||
- Create: `tests/catalogsync/test_models.py`
|
||||
- Create: `tests/catalogsync/fixtures/`
|
||||
|
||||
- [ ] Write failing tests for schema creation and song dedupe helpers.
|
||||
- [ ] Run the focused unittest commands and verify they fail for missing modules.
|
||||
- [ ] Implement the minimal schema and helper modules to satisfy the tests.
|
||||
- [ ] Re-run the focused tests and verify they pass.
|
||||
|
||||
### Task 2: Add failing tests for collector parsing helpers
|
||||
|
||||
**Files:**
|
||||
- Create: `tests/catalogsync/test_collectors.py`
|
||||
- Create: `musicdl/catalogsync/collectors/`
|
||||
|
||||
- [ ] Write fixture-driven failing tests for NetEase, QQ, and Kuwo collector parsing helpers.
|
||||
- [ ] Run the focused unittest commands and verify they fail.
|
||||
- [ ] Implement minimal collector modules and parsing helpers.
|
||||
- [ ] Re-run the focused tests and verify they pass.
|
||||
|
||||
### Task 3: Implement repository, sync services, and download planner
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/db.py`
|
||||
- Create: `musicdl/catalogsync/repository.py`
|
||||
- Create: `musicdl/catalogsync/services.py`
|
||||
- Create: `musicdl/catalogsync/downloader.py`
|
||||
|
||||
- [ ] Write failing service tests for playlist sync, derived artist sync, and download dedupe.
|
||||
- [ ] Run the focused unittest commands and verify they fail.
|
||||
- [ ] Implement the repository and service layer.
|
||||
- [ ] Re-run the focused tests and verify they pass.
|
||||
|
||||
### Task 4: Implement CLI and package integration
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `setup.py`
|
||||
- Modify: `musicdl/__init__.py`
|
||||
|
||||
- [ ] Write failing CLI smoke tests around argument parsing and DB initialization.
|
||||
- [ ] Run the focused unittest commands and verify they fail.
|
||||
- [ ] Implement the CLI entrypoint and wire a new console script.
|
||||
- [ ] Re-run the focused tests and verify they pass.
|
||||
|
||||
### Task 5: Verify end-to-end behavior and document usage
|
||||
|
||||
**Files:**
|
||||
- Create: `docs/catalogsync.md`
|
||||
- Modify: `README.md`
|
||||
|
||||
- [ ] Run the full focused unittest suite for `tests/catalogsync`.
|
||||
- [ ] Run a manual CLI smoke flow against a temporary SQLite DB.
|
||||
- [ ] Update user-facing docs with command examples and caveats.
|
||||
- [ ] Re-run final verification commands and capture results.
|
||||
@@ -0,0 +1,485 @@
|
||||
# Download Layout And NAS Deployment Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Change `musicdl.catalogsync` downloads to land under `LIBRARY_DIR/<platform>/<first_artist>/...`, preserve relative locators for later upload reuse, and add portable NAS/Linux deployment scripts plus `.env`-driven runtime layout.
|
||||
|
||||
**Architecture:** Add a small runtime/layout helper module for path building, safe filename components, config defaults, and directory creation. Reuse the existing downloader and CLI, but route download destinations through the new path helper and add deploy/runtime scripts under `scripts/catalogsync` so target machines can be bootstrapped and then run from `catalogsync/bin` with `catalogsync.env`.
|
||||
|
||||
**Tech Stack:** Python stdlib (`pathlib`, `dataclasses`, `tempfile`, `re`), `click`, existing `musicdl.catalogsync` modules, PowerShell, POSIX shell, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add runtime/layout helper tests and implementation
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/runtime.py`
|
||||
- Create: `tests/catalogsync/test_runtime.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing runtime/layout tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class RuntimeLayoutTests(unittest.TestCase):
|
||||
def test_runtime_config_builds_defaults_from_root_dir(self):
|
||||
from musicdl.catalogsync.runtime import CatalogSyncRuntimeConfig
|
||||
|
||||
config = CatalogSyncRuntimeConfig.from_mapping(
|
||||
{
|
||||
"ROOT_DIR": "/volume4/Music_Cloud",
|
||||
"PYTHON_BIN": "python3",
|
||||
}
|
||||
)
|
||||
|
||||
self.assertEqual(Path("/volume4/Music_Cloud/catalogsync"), config.app_home)
|
||||
self.assertEqual(Path("/volume4/Music_Cloud/library"), config.library_dir)
|
||||
self.assertEqual(Path("/volume4/Music_Cloud/catalogsync/data/catalogsync.db"), config.db_path)
|
||||
self.assertEqual("platform_first_artist", config.download_layout)
|
||||
|
||||
def test_runtime_config_ensure_directories_creates_expected_tree(self):
|
||||
from musicdl.catalogsync.runtime import CatalogSyncRuntimeConfig
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root_dir = Path(tmpdir) / "Music_Cloud"
|
||||
config = CatalogSyncRuntimeConfig.from_mapping({"ROOT_DIR": str(root_dir)})
|
||||
|
||||
config.ensure_directories()
|
||||
|
||||
self.assertTrue((root_dir / "library").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "app").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "bin").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "config").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "data").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "inputs").is_dir())
|
||||
self.assertTrue((root_dir / "catalogsync" / "logs").is_dir())
|
||||
|
||||
def test_build_download_relative_dir_uses_platform_and_first_artist(self):
|
||||
from musicdl.catalogsync.runtime import build_download_relative_dir
|
||||
|
||||
relative_dir = build_download_relative_dir(
|
||||
platform="qq",
|
||||
singers="Singer A / Singer B",
|
||||
)
|
||||
|
||||
self.assertEqual(Path("qq") / "Singer A", relative_dir)
|
||||
|
||||
def test_build_download_relative_dir_falls_back_to_unknown_artist(self):
|
||||
from musicdl.catalogsync.runtime import build_download_relative_dir
|
||||
|
||||
relative_dir = build_download_relative_dir(
|
||||
platform="netease",
|
||||
singers="",
|
||||
)
|
||||
|
||||
self.assertEqual(Path("netease") / "Unknown Artist", relative_dir)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime/layout tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_runtime -v`
|
||||
Expected: FAIL with import error for `musicdl.catalogsync.runtime` or missing helper functions
|
||||
|
||||
- [ ] **Step 3: Implement the minimal runtime/layout helper module**
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
INVALID_PATH_CHARS_RE = re.compile(r'[<>:"/\\|?*\x00-\x1f]')
|
||||
|
||||
|
||||
def sanitize_path_component(value: str, fallback: str) -> str:
|
||||
cleaned = INVALID_PATH_CHARS_RE.sub("_", (value or "").strip()).rstrip(". ")
|
||||
return cleaned or fallback
|
||||
|
||||
|
||||
def pick_first_artist_name(singers: str | None) -> str:
|
||||
for candidate in re.split(r"\s*(?:/|,|&|\|)\s*", singers or ""):
|
||||
if candidate.strip():
|
||||
return sanitize_path_component(candidate, "Unknown Artist")
|
||||
return "Unknown Artist"
|
||||
|
||||
|
||||
def build_download_relative_dir(platform: str, singers: str | None) -> Path:
|
||||
return Path(sanitize_path_component(platform, "unknown")) / pick_first_artist_name(singers)
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class CatalogSyncRuntimeConfig:
|
||||
root_dir: Path
|
||||
app_home: Path
|
||||
library_dir: Path
|
||||
db_path: Path
|
||||
input_dir: Path
|
||||
log_dir: Path
|
||||
python_bin: str
|
||||
venv_dir: Path
|
||||
download_layout: str
|
||||
|
||||
@classmethod
|
||||
def from_mapping(cls, mapping: dict[str, str]) -> "CatalogSyncRuntimeConfig":
|
||||
root_dir = Path(mapping["ROOT_DIR"]).resolve()
|
||||
app_home = Path(mapping.get("APP_HOME", root_dir / "catalogsync")).resolve()
|
||||
library_dir = Path(mapping.get("LIBRARY_DIR", root_dir / "library")).resolve()
|
||||
return cls(
|
||||
root_dir=root_dir,
|
||||
app_home=app_home,
|
||||
library_dir=library_dir,
|
||||
db_path=Path(mapping.get("DB_PATH", app_home / "data" / "catalogsync.db")).resolve(),
|
||||
input_dir=Path(mapping.get("INPUT_DIR", app_home / "inputs")).resolve(),
|
||||
log_dir=Path(mapping.get("LOG_DIR", app_home / "logs")).resolve(),
|
||||
python_bin=mapping.get("PYTHON_BIN", "python3"),
|
||||
venv_dir=Path(mapping.get("VENV_DIR", app_home / "app" / ".venv")).resolve(),
|
||||
download_layout=mapping.get("DOWNLOAD_LAYOUT", "platform_first_artist"),
|
||||
)
|
||||
|
||||
def ensure_directories(self) -> None:
|
||||
for path in (
|
||||
self.root_dir,
|
||||
self.library_dir,
|
||||
self.app_home / "app",
|
||||
self.app_home / "bin",
|
||||
self.app_home / "config",
|
||||
self.app_home / "data",
|
||||
self.app_home / "inputs",
|
||||
self.app_home / "logs",
|
||||
):
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime/layout tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_runtime -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/runtime.py tests/catalogsync/test_runtime.py
|
||||
git commit -m "feat: add runtime layout helpers"
|
||||
```
|
||||
|
||||
### Task 2: Route downloader output through `platform/first_artist`
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Add failing downloader layout tests**
|
||||
|
||||
```python
|
||||
def test_catalog_downloader_records_platform_first_artist_locator(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.downloader import CatalogDownloader
|
||||
from musicdl.catalogsync.models import CatalogSong
|
||||
from musicdl.catalogsync.repository import CatalogRepository
|
||||
|
||||
class FakeClient:
|
||||
def download(self, song_infos, num_threadings=1, auto_supplement_song=False):
|
||||
save_path = Path(song_infos[0].work_dir) / "song-c.mp3"
|
||||
save_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
save_path.write_bytes(b"fake-audio")
|
||||
return [SimpleNamespace(save_path=str(save_path))]
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
library_root = Path(tmpdir) / "library"
|
||||
initialize_database(db_path, default_library_root=library_root).close()
|
||||
repo = CatalogRepository(db_path)
|
||||
repo.upsert_song(
|
||||
CatalogSong(
|
||||
platform="qq",
|
||||
remote_song_id="song-c",
|
||||
name="Song C",
|
||||
singers="Singer A / Singer B",
|
||||
ext="mp3",
|
||||
file_size_bytes=80,
|
||||
metadata={"snapshot": {"identifier": "song-c"}},
|
||||
)
|
||||
)
|
||||
downloader = CatalogDownloader(repository=repo)
|
||||
|
||||
with patch("musicdl.catalogsync.downloader.deserialize_song_info", return_value=SimpleNamespace(singers="Singer A / Singer B")):
|
||||
with patch.object(downloader, "get_client", return_value=FakeClient()):
|
||||
downloader.download_pending(library_root=library_root, limit=1)
|
||||
|
||||
location = repo._fetchone(
|
||||
"SELECT locator FROM file_locations ORDER BY id DESC LIMIT 1"
|
||||
)
|
||||
|
||||
self.assertEqual("qq/Singer A/song-c.mp3", location["locator"])
|
||||
|
||||
def test_catalog_downloader_uses_unknown_artist_fallback_directory(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.downloader import CatalogDownloader
|
||||
from musicdl.catalogsync.models import CatalogSong
|
||||
from musicdl.catalogsync.repository import CatalogRepository
|
||||
|
||||
class FakeClient:
|
||||
def download(self, song_infos, num_threadings=1, auto_supplement_song=False):
|
||||
save_path = Path(song_infos[0].work_dir) / "song-a.flac"
|
||||
save_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
save_path.write_bytes(b"fake-audio")
|
||||
return [SimpleNamespace(save_path=str(save_path))]
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
library_root = Path(tmpdir) / "library"
|
||||
initialize_database(db_path, default_library_root=library_root).close()
|
||||
repo = CatalogRepository(db_path)
|
||||
repo.upsert_song(
|
||||
CatalogSong(
|
||||
platform="netease",
|
||||
remote_song_id="song-a",
|
||||
name="Song A",
|
||||
singers="",
|
||||
ext="flac",
|
||||
file_size_bytes=100,
|
||||
metadata={"snapshot": {"identifier": "song-a"}},
|
||||
)
|
||||
)
|
||||
downloader = CatalogDownloader(repository=repo)
|
||||
|
||||
with patch("musicdl.catalogsync.downloader.deserialize_song_info", return_value=SimpleNamespace(singers="")):
|
||||
with patch.object(downloader, "get_client", return_value=FakeClient()):
|
||||
downloader.download_pending(library_root=library_root, limit=1)
|
||||
|
||||
location = repo._fetchone(
|
||||
"SELECT locator FROM file_locations ORDER BY id DESC LIMIT 1"
|
||||
)
|
||||
|
||||
self.assertEqual("netease/Unknown Artist/song-a.flac", location["locator"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused downloader tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services.CatalogServiceTests.test_catalog_downloader_records_platform_first_artist_locator tests.catalogsync.test_services.CatalogServiceTests.test_catalog_downloader_uses_unknown_artist_fallback_directory -v`
|
||||
Expected: FAIL because the downloader still writes `platform/filename`
|
||||
|
||||
- [ ] **Step 3: Implement the downloader layout change**
|
||||
|
||||
```python
|
||||
from .runtime import build_download_relative_dir
|
||||
```
|
||||
|
||||
```python
|
||||
relative_dir = build_download_relative_dir(
|
||||
platform=row["platform"],
|
||||
singers=getattr(song_info, "singers", None) or row.get("singers"),
|
||||
)
|
||||
target_dir = target_root / relative_dir
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
song_info.work_dir = str(target_dir)
|
||||
```
|
||||
|
||||
Keep the locator writeback based on the actual saved file:
|
||||
|
||||
```python
|
||||
saved_path = Path(saved_song.save_path)
|
||||
relative_path = saved_path.relative_to(target_root).as_posix()
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused downloader tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services.CatalogServiceTests.test_catalog_downloader_records_platform_first_artist_locator tests.catalogsync.test_services.CatalogServiceTests.test_catalog_downloader_uses_unknown_artist_fallback_directory -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Run the broader catalogsync tests affected by downloader changes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services tests.catalogsync.test_cli -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/downloader.py tests/catalogsync/test_services.py
|
||||
git commit -m "feat: store downloads under platform and first artist"
|
||||
```
|
||||
|
||||
### Task 3: Add portable deployment and runtime script templates
|
||||
|
||||
**Files:**
|
||||
- Create: `scripts/catalogsync/bootstrap_to_linux.ps1`
|
||||
- Create: `scripts/catalogsync/templates/catalogsync.env.example`
|
||||
- Create: `scripts/catalogsync/templates/download_all.sh`
|
||||
- Create: `scripts/catalogsync/templates/download_from_file.sh`
|
||||
- Modify: `tests/catalogsync/test_runtime.py`
|
||||
|
||||
- [ ] **Step 1: Add failing tests for deployment template content**
|
||||
|
||||
```python
|
||||
def test_catalogsync_env_example_contains_required_keys(self):
|
||||
template = Path("scripts/catalogsync/templates/catalogsync.env.example").read_text(encoding="utf-8")
|
||||
self.assertIn("ROOT_DIR=", template)
|
||||
self.assertIn("APP_HOME=", template)
|
||||
self.assertIn("LIBRARY_DIR=", template)
|
||||
self.assertIn("DB_PATH=", template)
|
||||
self.assertIn("INPUT_DIR=", template)
|
||||
self.assertIn("LOG_DIR=", template)
|
||||
self.assertIn("DOWNLOAD_LAYOUT=platform_first_artist", template)
|
||||
|
||||
def test_runtime_script_template_uses_configured_library_dir(self):
|
||||
script = Path("scripts/catalogsync/templates/download_from_file.sh").read_text(encoding="utf-8")
|
||||
self.assertIn("LIBRARY_DIR", script)
|
||||
self.assertIn("INPUT_DIR", script)
|
||||
self.assertIn("musicdl.catalogsync.cli run", script)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused runtime/template tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_runtime.RuntimeLayoutTests.test_catalogsync_env_example_contains_required_keys tests.catalogsync.test_runtime.RuntimeLayoutTests.test_runtime_script_template_uses_configured_library_dir -v`
|
||||
Expected: FAIL because the template files do not exist yet
|
||||
|
||||
- [ ] **Step 3: Add the deployment and runtime script templates**
|
||||
|
||||
`scripts/catalogsync/templates/catalogsync.env.example`:
|
||||
|
||||
```bash
|
||||
ROOT_DIR=/volume4/Music_Cloud
|
||||
APP_HOME=/volume4/Music_Cloud/catalogsync
|
||||
LIBRARY_DIR=/volume4/Music_Cloud/library
|
||||
|
||||
DB_PATH=/volume4/Music_Cloud/catalogsync/data/catalogsync.db
|
||||
INPUT_DIR=/volume4/Music_Cloud/catalogsync/inputs
|
||||
LOG_DIR=/volume4/Music_Cloud/catalogsync/logs
|
||||
|
||||
PYTHON_BIN=python3
|
||||
VENV_DIR=/volume4/Music_Cloud/catalogsync/app/.venv
|
||||
|
||||
DOWNLOAD_LAYOUT=platform_first_artist
|
||||
```
|
||||
|
||||
`scripts/catalogsync/templates/download_all.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
APP_HOME="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
CONFIG_FILE="${APP_HOME}/config/catalogsync.env"
|
||||
source "${CONFIG_FILE}"
|
||||
|
||||
mkdir -p "${LIBRARY_DIR}" "${APP_HOME}/data" "${INPUT_DIR}" "${LOG_DIR}"
|
||||
|
||||
"${PYTHON_BIN}" -m musicdl.catalogsync.cli run \
|
||||
--db "${DB_PATH}" \
|
||||
--library-root "${LIBRARY_DIR}" \
|
||||
"$@"
|
||||
```
|
||||
|
||||
`scripts/catalogsync/templates/download_from_file.sh`:
|
||||
|
||||
```bash
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
if [[ $# -lt 1 ]]; then
|
||||
echo "usage: $0 <playlist-file> [extra args...]"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
PLAYLIST_FILE="$1"
|
||||
shift
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
APP_HOME="$(cd "${SCRIPT_DIR}/.." && pwd)"
|
||||
CONFIG_FILE="${APP_HOME}/config/catalogsync.env"
|
||||
source "${CONFIG_FILE}"
|
||||
|
||||
mkdir -p "${LIBRARY_DIR}" "${APP_HOME}/data" "${INPUT_DIR}" "${LOG_DIR}"
|
||||
|
||||
"${PYTHON_BIN}" -m musicdl.catalogsync.cli run \
|
||||
--db "${DB_PATH}" \
|
||||
--library-root "${LIBRARY_DIR}" \
|
||||
--playlist-file "${PLAYLIST_FILE}" \
|
||||
"$@"
|
||||
```
|
||||
|
||||
`scripts/catalogsync/bootstrap_to_linux.ps1` should:
|
||||
|
||||
```powershell
|
||||
param(
|
||||
[string]$Host,
|
||||
[int]$Port = 22,
|
||||
[string]$User,
|
||||
[string]$RootDir = "/volume4/Music_Cloud"
|
||||
)
|
||||
|
||||
$AppHome = "$RootDir/catalogsync"
|
||||
$RemoteDirs = @(
|
||||
$RootDir,
|
||||
"$RootDir/library",
|
||||
"$AppHome/app",
|
||||
"$AppHome/bin",
|
||||
"$AppHome/config",
|
||||
"$AppHome/data",
|
||||
"$AppHome/inputs",
|
||||
"$AppHome/logs"
|
||||
)
|
||||
```
|
||||
|
||||
Then use `ssh` and `scp` to:
|
||||
|
||||
- create the remote directories
|
||||
- copy the application files into `$AppHome/app`
|
||||
- copy the shell script templates into `$AppHome/bin`
|
||||
- copy `catalogsync.env.example` into `$AppHome/config/catalogsync.env.example` if missing
|
||||
|
||||
- [ ] **Step 4: Re-run the focused runtime/template tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_runtime -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add scripts/catalogsync tests/catalogsync/test_runtime.py
|
||||
git commit -m "feat: add portable catalogsync deployment scripts"
|
||||
```
|
||||
|
||||
### Task 4: Document the new layout and verify the full flow
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
- Modify: `README.md`
|
||||
|
||||
- [ ] **Step 1: Update user-facing docs with the new deployment layout**
|
||||
|
||||
Add:
|
||||
|
||||
- the `/volume4/Music_Cloud/library` versus `/volume4/Music_Cloud/catalogsync` split
|
||||
- the `platform/first_artist` download layout
|
||||
- the `catalogsync.env` example
|
||||
- the `scripts/catalogsync/bootstrap_to_linux.ps1` usage
|
||||
- the target-side `download_all.sh` and `download_from_file.sh` usage
|
||||
|
||||
- [ ] **Step 2: Run the full catalogsync unittest suite**
|
||||
|
||||
Run: `python -m unittest discover -s tests/catalogsync -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Run a local smoke check for CLI help**
|
||||
|
||||
Run: `python -m musicdl.catalogsync.cli run --help`
|
||||
Expected: output includes `--playlist-file`
|
||||
|
||||
- [ ] **Step 4: Inspect the generated diff**
|
||||
|
||||
Run: `git diff --stat`
|
||||
Expected: only the planned runtime/layout/downloader/docs files changed
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/catalogsync.md README.md
|
||||
git commit -m "docs: describe NAS download layout workflow"
|
||||
```
|
||||
@@ -0,0 +1,476 @@
|
||||
# Playlist File Run Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a `--playlist-file` option to `musicdl.catalogsync` so `run` can process only playlists listed in a text file while preserving the current default workflow when the option is absent.
|
||||
|
||||
**Architecture:** Keep the existing `collect -> sync -> download` path unchanged and add a narrow file-driven branch in `run`. Parse playlist-file lines into normalized playlist import entries, upsert them into the existing catalog tables under a `manual_file` pool, then sync and download only the referenced playlist IDs.
|
||||
|
||||
**Tech Stack:** Python stdlib (`pathlib`, `dataclasses`, `urllib.parse`), `click`, existing `musicdl.catalogsync` modules, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add file playlist parsing and manual-file pool support
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/manual_playlists.py`
|
||||
- Modify: `musicdl/catalogsync/models.py`
|
||||
- Modify: `musicdl/catalogsync/repository.py`
|
||||
- Test: `tests/catalogsync/test_manual_playlists.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for playlist-file parsing**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ManualPlaylistParsingTests(unittest.TestCase):
|
||||
def test_parse_playlist_file_supports_url_and_platform_url_lines(self):
|
||||
from musicdl.catalogsync.manual_playlists import parse_playlist_file
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
playlist_file = Path(tmpdir) / "playlists.txt"
|
||||
playlist_file.write_text(
|
||||
"\n".join(
|
||||
[
|
||||
"# comment",
|
||||
"https://music.163.com/#/playlist?id=17745989905",
|
||||
"qq,https://y.qq.com/n/ryqq/playlist/7707261125",
|
||||
"https://music.163.com/#/playlist?id=17745989905",
|
||||
"",
|
||||
]
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
parsed = parse_playlist_file(playlist_file)
|
||||
|
||||
self.assertEqual(5, parsed.total_lines)
|
||||
self.assertEqual(0, parsed.skipped_lines)
|
||||
self.assertEqual(2, len(parsed.entries))
|
||||
self.assertEqual("netease", parsed.entries[0].platform)
|
||||
self.assertEqual("17745989905", parsed.entries[0].remote_id)
|
||||
self.assertEqual("qq", parsed.entries[1].platform)
|
||||
self.assertEqual("7707261125", parsed.entries[1].remote_id)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused parser test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_manual_playlists.ManualPlaylistParsingTests.test_parse_playlist_file_supports_url_and_platform_url_lines -v`
|
||||
Expected: FAIL with import error for `musicdl.catalogsync.manual_playlists` or missing parser helpers
|
||||
|
||||
- [ ] **Step 3: Implement the parser module and manual-file pool helper**
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from .models import PlaylistCandidate
|
||||
|
||||
|
||||
@dataclass(slots=True)
|
||||
class ParsedPlaylistFile:
|
||||
entries: list[PlaylistCandidate]
|
||||
total_lines: int
|
||||
skipped_lines: int
|
||||
|
||||
|
||||
def parse_playlist_file(path: str | Path) -> ParsedPlaylistFile:
|
||||
raise NotImplementedError
|
||||
```
|
||||
|
||||
```python
|
||||
def get_or_create_manual_file_pool(self, playlist_file: str | Path) -> int:
|
||||
resolved = Path(playlist_file).resolve()
|
||||
return self.upsert_playlist_pool(
|
||||
platform="manual",
|
||||
pool_kind="manual_file",
|
||||
external_id=f"manual_file:{resolved}",
|
||||
name=f"Manual File Import: {resolved.name}",
|
||||
url=str(resolved),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused parser tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_manual_playlists -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/manual_playlists.py musicdl/catalogsync/models.py musicdl/catalogsync/repository.py tests/catalogsync/test_manual_playlists.py
|
||||
git commit -m "feat: add playlist file parsing support"
|
||||
```
|
||||
|
||||
### Task 2: Add manual playlist import and targeted sync
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/services.py`
|
||||
- Modify: `musicdl/catalogsync/repository.py`
|
||||
- Test: `tests/catalogsync/test_manual_playlists.py`
|
||||
- Test: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for manual playlist import and targeted sync**
|
||||
|
||||
```python
|
||||
def test_import_manual_playlists_creates_manual_pool_and_returns_playlist_ids(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.manual_playlists import parse_playlist_file
|
||||
from musicdl.catalogsync.repository import CatalogRepository
|
||||
from musicdl.catalogsync.services import CatalogSyncService
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
playlist_file = Path(tmpdir) / "playlists.txt"
|
||||
playlist_file.write_text(
|
||||
"https://music.163.com/#/playlist?id=17745989905\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
initialize_database(db_path).close()
|
||||
repo = CatalogRepository(db_path)
|
||||
service = CatalogSyncService(repo)
|
||||
|
||||
parsed = parse_playlist_file(playlist_file)
|
||||
playlist_ids = service.import_manual_playlists(playlist_file, parsed.entries)
|
||||
|
||||
self.assertEqual(1, len(playlist_ids))
|
||||
self.assertEqual(1, repo.count_rows("playlists"))
|
||||
self.assertEqual(
|
||||
1,
|
||||
len(repo.list_pool_playlist_links(repo.get_or_create_manual_file_pool(playlist_file))),
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
def test_sync_specific_playlists_only_processes_requested_playlist_ids(self):
|
||||
from unittest.mock import patch
|
||||
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.models import PlaylistCandidate
|
||||
from musicdl.catalogsync.repository import CatalogRepository
|
||||
from musicdl.catalogsync.services import CatalogSyncService
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
initialize_database(db_path).close()
|
||||
repo = CatalogRepository(db_path)
|
||||
service = CatalogSyncService(repo)
|
||||
|
||||
playlist_a = repo.upsert_playlist(
|
||||
PlaylistCandidate(
|
||||
platform="netease",
|
||||
pool_kind="manual_file",
|
||||
remote_id="17745989905",
|
||||
name="Playlist A",
|
||||
url="https://music.163.com/#/playlist?id=17745989905",
|
||||
)
|
||||
)
|
||||
playlist_b = repo.upsert_playlist(
|
||||
PlaylistCandidate(
|
||||
platform="netease",
|
||||
pool_kind="manual_file",
|
||||
remote_id="17729789137",
|
||||
name="Playlist B",
|
||||
url="https://music.163.com/#/playlist?id=17729789137",
|
||||
)
|
||||
)
|
||||
|
||||
with patch.object(service, "resolve_playlist_song_infos", return_value=[] ) as resolver:
|
||||
service.sync_specific_playlists([playlist_b])
|
||||
|
||||
called_row = resolver.call_args[0][0]
|
||||
self.assertEqual(playlist_b, int(called_row["id"]))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused services tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_manual_playlists tests.catalogsync.test_services -v`
|
||||
Expected: FAIL with missing `import_manual_playlists`, `list_playlists_by_ids`, and `sync_specific_playlists`
|
||||
|
||||
- [ ] **Step 3: Implement manual playlist import and targeted sync entry points**
|
||||
|
||||
```python
|
||||
def list_playlists_by_ids(self, playlist_ids: list[int]) -> list[sqlite3.Row]:
|
||||
placeholders = ", ".join("?" for _ in playlist_ids)
|
||||
return self._fetchall(
|
||||
f"SELECT * FROM playlists WHERE id IN ({placeholders}) ORDER BY id ASC",
|
||||
tuple(playlist_ids),
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
def import_manual_playlists(self, playlist_file: str | Path, candidates: list[PlaylistCandidate]) -> list[int]:
|
||||
pool_id = self.repository.get_or_create_manual_file_pool(playlist_file)
|
||||
playlist_ids = []
|
||||
for candidate in candidates:
|
||||
playlist_id = self.repository.upsert_playlist(candidate)
|
||||
self.repository.link_pool_playlist(pool_id, playlist_id)
|
||||
playlist_ids.append(playlist_id)
|
||||
return playlist_ids
|
||||
|
||||
def sync_specific_playlists(self, playlist_ids: list[int]) -> int:
|
||||
processed = 0
|
||||
for playlist_row in self.repository.list_playlists_by_ids(playlist_ids):
|
||||
song_infos = self.resolve_playlist_song_infos(playlist_row)
|
||||
for pool_id in self.repository.get_pool_ids_for_playlist(int(playlist_row["id"])):
|
||||
self.store_playlist_songs(int(playlist_row["id"]), pool_id, song_infos)
|
||||
processed += len(song_infos)
|
||||
return processed
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused services tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_manual_playlists tests.catalogsync.test_services -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/services.py musicdl/catalogsync/repository.py tests/catalogsync/test_manual_playlists.py tests/catalogsync/test_services.py
|
||||
git commit -m "feat: add manual playlist import and targeted sync"
|
||||
```
|
||||
|
||||
### Task 3: Add targeted download planning for selected playlists
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `musicdl/catalogsync/repository.py`
|
||||
- Test: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Write a failing test for limiting downloads to selected playlist IDs**
|
||||
|
||||
```python
|
||||
def test_download_planner_can_limit_queue_to_specific_playlists(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.models import CatalogSong, PlaylistCandidate
|
||||
from musicdl.catalogsync.repository import CatalogRepository
|
||||
from musicdl.catalogsync.downloader import DownloadPlanner
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
initialize_database(db_path).close()
|
||||
repo = CatalogRepository(db_path)
|
||||
|
||||
playlist_a = repo.upsert_playlist(
|
||||
PlaylistCandidate(
|
||||
platform="qq",
|
||||
pool_kind="manual_file",
|
||||
remote_id="7707261125",
|
||||
name="Playlist A",
|
||||
url="https://y.qq.com/n/ryqq/playlist/7707261125",
|
||||
)
|
||||
)
|
||||
playlist_b = repo.upsert_playlist(
|
||||
PlaylistCandidate(
|
||||
platform="qq",
|
||||
pool_kind="manual_file",
|
||||
remote_id="7578943835",
|
||||
name="Playlist B",
|
||||
url="https://y.qq.com/n/ryqq/playlist/7578943835",
|
||||
)
|
||||
)
|
||||
song_a = repo.upsert_song(CatalogSong(platform="qq", remote_song_id="song-a", name="A"))
|
||||
song_b = repo.upsert_song(CatalogSong(platform="qq", remote_song_id="song-b", name="B"))
|
||||
repo.link_playlist_song(playlist_a, song_a, 1)
|
||||
repo.link_playlist_song(playlist_b, song_b, 1)
|
||||
|
||||
planner = DownloadPlanner(repo)
|
||||
queue = planner.build_download_queue(playlist_ids=[playlist_a])
|
||||
|
||||
self.assertEqual([song_a], [item["song_id"] for item in queue])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused download planner test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services.CatalogServiceTests.test_download_planner_can_limit_queue_to_specific_playlists -v`
|
||||
Expected: FAIL because `playlist_ids` is unsupported
|
||||
|
||||
- [ ] **Step 3: Implement repository and downloader filtering by playlist IDs**
|
||||
|
||||
```python
|
||||
def list_pending_download_songs(
|
||||
self,
|
||||
sources: list[str] | None = None,
|
||||
limit: int | None = None,
|
||||
playlist_ids: list[int] | None = None,
|
||||
) -> list[sqlite3.Row]:
|
||||
query = """
|
||||
SELECT DISTINCT s.*
|
||||
FROM songs s
|
||||
JOIN playlist_songs ps ON ps.song_id = s.id
|
||||
WHERE NOT EXISTS (
|
||||
SELECT 1
|
||||
FROM file_locations fl
|
||||
JOIN file_assets fa ON fa.id = fl.file_asset_id
|
||||
JOIN storage_backends sb ON sb.id = fl.backend_id
|
||||
WHERE fa.song_id = s.id
|
||||
AND fl.status = 'active'
|
||||
AND sb.backend_type = 'local_fs'
|
||||
)
|
||||
"""
|
||||
```
|
||||
|
||||
```python
|
||||
def build_download_queue(
|
||||
self,
|
||||
sources: list[str] | None = None,
|
||||
limit: int | None = None,
|
||||
playlist_ids: list[int] | None = None,
|
||||
) -> list[dict]:
|
||||
rows = self.repository.list_pending_download_songs(
|
||||
sources=sources,
|
||||
limit=limit,
|
||||
playlist_ids=playlist_ids,
|
||||
)
|
||||
queue = []
|
||||
for row in rows:
|
||||
if self.repository.song_has_active_local_file(int(row["id"])):
|
||||
continue
|
||||
item = dict(row)
|
||||
item["song_id"] = int(row["id"])
|
||||
queue.append(item)
|
||||
return queue
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused download tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/downloader.py musicdl/catalogsync/repository.py tests/catalogsync/test_services.py
|
||||
git commit -m "feat: limit downloads to selected playlists"
|
||||
```
|
||||
|
||||
### Task 4: Wire `--playlist-file` into the CLI
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
- Test: `tests/catalogsync/test_manual_playlists.py`
|
||||
|
||||
- [ ] **Step 1: Write failing CLI tests for the file-driven `run` path**
|
||||
|
||||
```python
|
||||
def test_run_command_uses_playlist_file_branch_without_collect(self):
|
||||
from musicdl.catalogsync.cli import cli
|
||||
|
||||
runner = CliRunner()
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
playlist_file = Path(tmpdir) / "playlists.txt"
|
||||
playlist_file.write_text(
|
||||
"https://music.163.com/#/playlist?id=17745989905\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
with patch("musicdl.catalogsync.cli.CatalogSyncApplication") as app_cls:
|
||||
app = app_cls.return_value
|
||||
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"run",
|
||||
"--db",
|
||||
str(db_path),
|
||||
"--library-root",
|
||||
str(Path(tmpdir) / "library"),
|
||||
"--playlist-file",
|
||||
str(playlist_file),
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(0, result.exit_code, msg=result.output)
|
||||
app.collect_playlists.assert_not_called()
|
||||
app.run_playlist_file.assert_called_once()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused CLI test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_cli.CatalogCliTests.test_run_command_uses_playlist_file_branch_without_collect -v`
|
||||
Expected: FAIL because `--playlist-file` and `run_playlist_file` do not exist
|
||||
|
||||
- [ ] **Step 3: Implement the CLI branch and application method**
|
||||
|
||||
```python
|
||||
def run_playlist_file(self, playlist_file: str, sources: list[str] | None = None, limit: int | None = None):
|
||||
parsed = parse_playlist_file(playlist_file)
|
||||
playlist_ids = self.service.import_manual_playlists(playlist_file, parsed.entries)
|
||||
if limit is not None:
|
||||
playlist_ids = playlist_ids[:limit]
|
||||
synced = self.service.sync_specific_playlists(playlist_ids)
|
||||
downloaded = self.downloader.download_pending(
|
||||
self.library_root,
|
||||
sources=sources,
|
||||
limit=limit,
|
||||
playlist_ids=playlist_ids,
|
||||
)
|
||||
return parsed, synced, downloaded
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused CLI tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_cli -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/cli.py tests/catalogsync/test_cli.py tests/catalogsync/test_manual_playlists.py
|
||||
git commit -m "feat: add playlist file run option"
|
||||
```
|
||||
|
||||
### Task 5: Update docs and run final verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
- Modify: `README.md`
|
||||
- Test: `tests/catalogsync/test_manual_playlists.py`
|
||||
- Test: `tests/catalogsync/test_cli.py`
|
||||
- Test: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Update docs with `--playlist-file` examples and file format rules**
|
||||
|
||||
```markdown
|
||||
### 从文件读取歌单
|
||||
|
||||
```bash
|
||||
musicdl-catalogsync run --db D:\catalogsync\catalogsync.db --library-root E:\MusicLibrary --playlist-file D:\lists\playlists.txt
|
||||
```
|
||||
|
||||
文件支持：
|
||||
- 一行一个歌单 URL
|
||||
- `平台,歌单URL`
|
||||
- `#` 注释行
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the full catalogsync test suite**
|
||||
|
||||
Run: `python -m unittest discover -s tests/catalogsync -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Run a manual smoke flow using a temporary playlist file**
|
||||
|
||||
Run:
|
||||
```bash
|
||||
python -m musicdl.catalogsync.cli init-db --db .tmp\playlist-file\catalogsync.db --library-root .tmp\playlist-file\library
|
||||
python -m musicdl.catalogsync.cli run --db .tmp\playlist-file\catalogsync.db --library-root .tmp\playlist-file\library --playlist-file .tmp\playlist-file\playlists.txt
|
||||
```
|
||||
|
||||
Expected:
|
||||
- command exits 0
|
||||
- the file-driven branch skips `collect`
|
||||
- playlists from the file are imported and processed
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/catalogsync.md README.md
|
||||
git commit -m "docs: add playlist file run usage"
|
||||
```
|
||||
@@ -0,0 +1,958 @@
|
||||
# Catalogsync Operations Console Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Build a NAS-local operations console for `musicdl.catalogsync` with queue-based `collect/sync/download/upload` jobs, soft pause and resume, crash-safe recovery, song-level retry, worker visibility, and `catalogsync.env` configuration management.
|
||||
|
||||
**Architecture:** Add a new `musicdl.catalogsync.ops` package for execution-state persistence, env revision management, job orchestration, and FastAPI-backed UI/API endpoints. Reuse the existing catalog tables and execution services by wrapping them in stage and item executors rather than rebuilding the domain logic from scratch.
|
||||
|
||||
**Tech Stack:** Python 3, unittest, SQLite, FastAPI, Jinja2, Server-Sent Events, existing `musicdl.catalogsync` services/downloader/uploader, NAS shell templates.
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
### New files
|
||||
|
||||
- `musicdl/catalogsync/ops/__init__.py`
|
||||
- exports the operations-console public entry points
|
||||
- `musicdl/catalogsync/ops/models.py`
|
||||
- enums, dataclasses, and small helpers for job, stage, item, and worker states
|
||||
- `musicdl/catalogsync/ops/repository.py`
|
||||
- CRUD for `job_runs`, `job_stages`, `job_items`, `job_workers`, `job_commands`, `job_events`, `job_logs`, and `config_revisions`
|
||||
- `musicdl/catalogsync/ops/config.py`
|
||||
- load, validate, snapshot, version, and apply `catalogsync.env`
|
||||
- `musicdl/catalogsync/ops/executors.py`
|
||||
- adapters that run `collect`, `sync`, `download`, and `upload` one work item at a time
|
||||
- `musicdl/catalogsync/ops/runner.py`
|
||||
- queue scheduler, command polling, pause/resume, and crash recovery
|
||||
- `musicdl/catalogsync/ops/web.py`
|
||||
- FastAPI app factory, API routes, page routes, and SSE stream
|
||||
- `musicdl/catalogsync/templates/ops/base.html`
|
||||
- shared layout
|
||||
- `musicdl/catalogsync/templates/ops/dashboard.html`
|
||||
- dashboard page
|
||||
- `musicdl/catalogsync/templates/ops/jobs.html`
|
||||
- queue and active-job page
|
||||
- `musicdl/catalogsync/templates/ops/job_detail.html`
|
||||
- per-job detail page
|
||||
- `musicdl/catalogsync/templates/ops/playlists.html`
|
||||
- playlist-pool and playlist status page
|
||||
- `musicdl/catalogsync/templates/ops/songs.html`
|
||||
- worker and song-processing page
|
||||
- `musicdl/catalogsync/templates/ops/logs.html`
|
||||
- log and exception page
|
||||
- `musicdl/catalogsync/templates/ops/config.html`
|
||||
- env editor and revision page
|
||||
- `musicdl/catalogsync/static/ops/app.js`
|
||||
- lightweight browser logic for SSE updates and page actions
|
||||
- `scripts/catalogsync/templates/serve_console.sh`
|
||||
- NAS runtime script for the web console
|
||||
- `tests/catalogsync/test_ops_db.py`
|
||||
- schema and repository coverage for operations tables
|
||||
- `tests/catalogsync/test_ops_config.py`
|
||||
- env loading, snapshot, revision, and apply tests
|
||||
- `tests/catalogsync/test_ops_runner.py`
|
||||
- state-machine, pause/resume, and recovery tests
|
||||
- `tests/catalogsync/test_ops_executors.py`
|
||||
- per-stage executor tests
|
||||
- `tests/catalogsync/test_ops_api.py`
|
||||
- FastAPI route and SSE tests
|
||||
|
||||
### Modified files
|
||||
|
||||
- `musicdl/catalogsync/db.py`
|
||||
- create the new operations tables
|
||||
- `musicdl/catalogsync/repository.py`
|
||||
- add query helpers that the operations layer can reuse from catalog data
|
||||
- `musicdl/catalogsync/services.py`
|
||||
- expose a playlist-row based sync unit for the runner
|
||||
- `musicdl/catalogsync/downloader.py`
|
||||
- expose song-level download execution for one queued item
|
||||
- `musicdl/catalogsync/uploader.py`
|
||||
- expose upload-task-level execution for one queued item
|
||||
- `musicdl/catalogsync/cli.py`
|
||||
- add the `serve` command
|
||||
- `musicdl/catalogsync/runtime.py`
|
||||
- add web host, port, and config-path runtime fields
|
||||
- `scripts/catalogsync/templates/catalogsync.env.example`
|
||||
- add web-console settings
|
||||
- `scripts/catalogsync/templates/install_runtime.sh`
|
||||
- install any new web-console dependencies
|
||||
- `docs/catalogsync.md`
|
||||
- document the console workflow and runtime script
|
||||
- `README.md`
|
||||
- add a concise operations-console entry
|
||||
- `requirements.txt`
|
||||
- add the web-console runtime dependencies
|
||||
- `setup.py`
|
||||
- include templates/static assets in packaging
|
||||
- `MANIFEST.in`
|
||||
- include template/static files for source distributions
|
||||
- `tests/catalogsync/test_cli.py`
|
||||
- cover the new `serve` command
|
||||
- `tests/catalogsync/test_runtime.py`
|
||||
- cover the new runtime config fields
|
||||
|
||||
### Dependency notes
|
||||
|
||||
- Add `fastapi`
|
||||
- Add `uvicorn`
|
||||
- Add `jinja2`
|
||||
- Add `python-multipart`
|
||||
|
||||
These should be added only once and then reused by the `serve` command, tests, and NAS runtime script.
|
||||
|
||||
### Task 1: Add Operations Schema And Repository Primitives
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/ops/__init__.py`
|
||||
- Create: `musicdl/catalogsync/ops/models.py`
|
||||
- Create: `musicdl/catalogsync/ops/repository.py`
|
||||
- Modify: `musicdl/catalogsync/db.py`
|
||||
- Test: `tests/catalogsync/test_ops_db.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing schema and repository tests**
|
||||
|
||||
```python
|
||||
import sqlite3
|
||||
import tempfile
|
||||
import unittest
|
||||
from contextlib import closing
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class OperationsSchemaTests(unittest.TestCase):
|
||||
def test_initialize_database_creates_operations_tables(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
|
||||
expected_tables = {
|
||||
"job_runs",
|
||||
"job_stages",
|
||||
"job_items",
|
||||
"job_workers",
|
||||
"job_commands",
|
||||
"job_events",
|
||||
"job_logs",
|
||||
"config_revisions",
|
||||
}
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
initialize_database(db_path).close()
|
||||
with closing(sqlite3.connect(db_path)) as conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
||||
).fetchall()
|
||||
}
|
||||
|
||||
self.assertTrue(expected_tables.issubset(tables))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted test and verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_db -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the operations tables and repository module do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement the schema, enums, and repository**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/db.py
|
||||
REQUIRED_TABLES |= {
|
||||
"job_runs",
|
||||
"job_stages",
|
||||
"job_items",
|
||||
"job_workers",
|
||||
"job_commands",
|
||||
"job_events",
|
||||
"job_logs",
|
||||
"config_revisions",
|
||||
}
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/models.py
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
|
||||
|
||||
class JobStatus(StrEnum):
|
||||
QUEUED = "queued"
|
||||
RUNNING = "running"
|
||||
PAUSE_REQUESTED = "pause_requested"
|
||||
PAUSED = "paused"
|
||||
COMPLETED = "completed"
|
||||
COMPLETED_WITH_ERRORS = "completed_with_errors"
|
||||
FAILED = "failed"
|
||||
CANCELED = "canceled"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class JobCreateRequest:
|
||||
job_type: str
|
||||
config_snapshot: dict
|
||||
sources: list[str] | None = None
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/repository.py
|
||||
class OperationsRepository:
|
||||
def create_job_run(self, job_type: str, config_snapshot: dict, sources=None, download_sources=None, playlist_scope=None) -> int:
|
||||
with connect_database(self.db_path) as conn:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO job_runs (job_type, config_snapshot_json, sources, download_sources, playlist_scope_json)
|
||||
VALUES (?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
job_type,
|
||||
json.dumps(config_snapshot, ensure_ascii=False),
|
||||
",".join(sources or []),
|
||||
",".join(download_sources or []),
|
||||
json.dumps(playlist_scope or {}, ensure_ascii=False),
|
||||
),
|
||||
)
|
||||
return int(cursor.lastrowid)
|
||||
|
||||
def create_job_stage(self, job_id: int, stage_type: str, seq_no: int) -> int:
|
||||
with connect_database(self.db_path) as conn:
|
||||
cursor = conn.execute(
|
||||
"INSERT INTO job_stages (job_run_id, stage_type, seq_no) VALUES (?, ?, ?)",
|
||||
(job_id, stage_type, seq_no),
|
||||
)
|
||||
return int(cursor.lastrowid)
|
||||
|
||||
def create_job_item(self, job_stage_id: int, item_type: str, item_key: str, **extra) -> int:
|
||||
with connect_database(self.db_path) as conn:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO job_items (job_stage_id, item_type, item_key, playlist_pool_id, playlist_id, song_id, file_location_id, payload_json)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
""",
|
||||
(
|
||||
job_stage_id,
|
||||
item_type,
|
||||
item_key,
|
||||
extra.get("playlist_pool_id"),
|
||||
extra.get("playlist_id"),
|
||||
extra.get("song_id"),
|
||||
extra.get("file_location_id"),
|
||||
json.dumps(extra.get("payload_json") or {}, ensure_ascii=False),
|
||||
),
|
||||
)
|
||||
return int(cursor.lastrowid)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted test and verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_db -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- the test output shows the new operations tables are created
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/db.py musicdl/catalogsync/ops/__init__.py musicdl/catalogsync/ops/models.py musicdl/catalogsync/ops/repository.py tests/catalogsync/test_ops_db.py
|
||||
git commit -m "feat: add operations schema and repository primitives"
|
||||
```
|
||||
|
||||
### Task 2: Add Env Revision Management And Job Config Snapshots
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/ops/config.py`
|
||||
- Modify: `musicdl/catalogsync/ops/repository.py`
|
||||
- Test: `tests/catalogsync/test_ops_config.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing config and revision tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class EnvManagerTests(unittest.TestCase):
|
||||
def test_load_snapshot_and_save_revision(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.config import CatalogsyncEnvManager
|
||||
from musicdl.catalogsync.ops.repository import OperationsRepository
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
env_path = Path(tmpdir) / "catalogsync.env"
|
||||
env_path.write_text(
|
||||
"LIBRARY_DIR=/volume4/Music_Cloud/library\nDOWNLOAD_SOURCES=qq,kuwo,migu\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
initialize_database(db_path).close()
|
||||
repo = OperationsRepository(db_path)
|
||||
manager = CatalogsyncEnvManager(env_path=env_path, repository=repo)
|
||||
|
||||
snapshot = manager.build_job_snapshot()
|
||||
revision_id = manager.save_revision(note="initial import")
|
||||
revisions = manager.list_revisions()
|
||||
|
||||
self.assertEqual(["qq", "kuwo", "migu"], snapshot["download_sources"])
|
||||
self.assertEqual(revision_id, revisions[0]["id"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted test and verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_config -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the env manager and config revision methods do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement the env manager and revision methods**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/config.py
|
||||
class CatalogsyncEnvManager:
|
||||
SNAPSHOT_KEYS = {
|
||||
"DB_PATH",
|
||||
"LIBRARY_DIR",
|
||||
"DOWNLOAD_SOURCES",
|
||||
"OBJECT_BACKEND_NAME",
|
||||
"OBJECT_BUCKET",
|
||||
"OBJECT_ENDPOINT",
|
||||
}
|
||||
|
||||
def load_current(self) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
for line in self.env_path.read_text(encoding="utf-8").splitlines():
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
key, value = line.split("=", 1)
|
||||
values[key.strip()] = value.strip()
|
||||
return values
|
||||
|
||||
def build_job_snapshot(self) -> dict:
|
||||
current = self.load_current()
|
||||
return {
|
||||
"library_root": current.get("LIBRARY_DIR", ""),
|
||||
"download_sources": [item for item in current.get("DOWNLOAD_SOURCES", "").split(",") if item],
|
||||
"env_values": {key: current[key] for key in self.SNAPSHOT_KEYS if key in current},
|
||||
}
|
||||
|
||||
def save_revision(self, note: str | None = None) -> int:
|
||||
content = self.env_path.read_text(encoding="utf-8")
|
||||
digest = hashlib.sha256(content.encode("utf-8")).hexdigest()
|
||||
return self.repository.create_config_revision(
|
||||
file_path=str(self.env_path),
|
||||
content_text=content,
|
||||
content_hash=digest,
|
||||
note=note,
|
||||
)
|
||||
|
||||
def apply_revision(self, revision_id: int) -> None:
|
||||
row = self.repository.get_config_revision(revision_id)
|
||||
self.env_path.write_text(row["content_text"], encoding="utf-8")
|
||||
self.repository.mark_config_revision_applied(revision_id)
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/repository.py
|
||||
def create_config_revision(self, file_path: str, content_text: str, content_hash: str, note: str | None = None) -> int:
|
||||
with connect_database(self.db_path) as conn:
|
||||
cursor = conn.execute(
|
||||
"""
|
||||
INSERT INTO config_revisions (file_path, content_text, content_hash, note)
|
||||
VALUES (?, ?, ?, ?)
|
||||
""",
|
||||
(file_path, content_text, content_hash, note),
|
||||
)
|
||||
return int(cursor.lastrowid)
|
||||
|
||||
def get_config_revision(self, revision_id: int):
|
||||
with connect_database(self.db_path) as conn:
|
||||
return conn.execute("SELECT * FROM config_revisions WHERE id = ?", (revision_id,)).fetchone()
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted test and verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_config -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- revision apply rewrites the env file and marks the revision as applied
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/ops/config.py musicdl/catalogsync/ops/repository.py tests/catalogsync/test_ops_config.py
|
||||
git commit -m "feat: add env revision and job snapshot management"
|
||||
```
|
||||
|
||||
### Task 3: Implement The Runner State Machine And Recovery Logic
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/ops/runner.py`
|
||||
- Modify: `musicdl/catalogsync/ops/models.py`
|
||||
- Modify: `musicdl/catalogsync/ops/repository.py`
|
||||
- Test: `tests/catalogsync/test_ops_runner.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing runner-state tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class RunnerStateTests(unittest.TestCase):
|
||||
def test_recover_orphaned_jobs_converts_running_items_to_interrupted(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.repository import OperationsRepository
|
||||
from musicdl.catalogsync.ops.runner import CatalogsyncRunner
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
initialize_database(db_path).close()
|
||||
repo = OperationsRepository(db_path)
|
||||
runner = CatalogsyncRunner(db_path=db_path, env_path=Path(tmpdir) / "catalogsync.env")
|
||||
|
||||
job_id = repo.create_job_run("download_only", {"library_root": "/tmp/library"})
|
||||
stage_id = repo.create_job_stage(job_id, "download", 1)
|
||||
item_id = repo.create_job_item(stage_id, "song", "song:1", song_id=1)
|
||||
repo.mark_job_running(job_id)
|
||||
repo.mark_stage_running(stage_id)
|
||||
repo.mark_item_running(item_id, worker_id=1)
|
||||
|
||||
runner.recover_incomplete_jobs()
|
||||
|
||||
job = repo.get_job_run(job_id)
|
||||
item = repo.get_job_item(item_id)
|
||||
|
||||
self.assertEqual("paused", job["status"])
|
||||
self.assertEqual("interrupted", item["status"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted test and verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_runner -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the runner, command flow, and recovery helpers do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement the runner, command polling, and recovery**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/runner.py
|
||||
class CatalogsyncRunner:
|
||||
def recover_incomplete_jobs(self) -> None:
|
||||
for job in self.repository.list_recoverable_jobs():
|
||||
self.repository.pause_job_for_recovery(int(job["id"]))
|
||||
for item in self.repository.list_running_items(int(job["id"])):
|
||||
self.repository.mark_item_interrupted(int(item["id"]), last_error="Runner restarted during execution")
|
||||
self.repository.add_job_event(int(job["id"]), "recovery_requeued", "Recovered job after runner restart")
|
||||
|
||||
def apply_pending_commands(self) -> None:
|
||||
for command in self.repository.list_pending_commands():
|
||||
if command["command_type"] == "pause":
|
||||
self.repository.request_job_pause(int(command["job_run_id"]))
|
||||
elif command["command_type"] == "resume":
|
||||
self.repository.resume_job(int(command["job_run_id"]))
|
||||
elif command["command_type"] == "retry_item":
|
||||
self.repository.requeue_item(int(command["target_item_id"]), force=False)
|
||||
elif command["command_type"] == "force_retry_item":
|
||||
self.repository.requeue_item(int(command["target_item_id"]), force=True)
|
||||
self.repository.mark_command_applied(int(command["id"]))
|
||||
|
||||
def reconcile_pause_state(self, job_id: int) -> None:
|
||||
if self.repository.job_has_running_items(job_id):
|
||||
return
|
||||
self.repository.finalize_pause(job_id)
|
||||
|
||||
def loop_once(self) -> None:
|
||||
self.apply_pending_commands()
|
||||
active_job = self.repository.claim_next_runnable_job()
|
||||
if active_job is None:
|
||||
return
|
||||
self.repository.mark_job_running(int(active_job["id"]))
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/repository.py
|
||||
def request_job_pause(self, job_id: int) -> None:
|
||||
with connect_database(self.db_path) as conn:
|
||||
conn.execute("UPDATE job_runs SET status = 'pause_requested' WHERE id = ?", (job_id,))
|
||||
conn.execute(
|
||||
"UPDATE job_stages SET status = 'pause_requested' WHERE job_run_id = ? AND status = 'running'",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
def pause_job_for_recovery(self, job_id: int) -> None:
|
||||
with connect_database(self.db_path) as conn:
|
||||
conn.execute("UPDATE job_runs SET status = 'paused' WHERE id = ?", (job_id,))
|
||||
conn.execute(
|
||||
"UPDATE job_stages SET status = 'paused' WHERE job_run_id = ? AND status IN ('running', 'pause_requested')",
|
||||
(job_id,),
|
||||
)
|
||||
|
||||
def mark_item_interrupted(self, item_id: int, last_error: str | None = None) -> None:
|
||||
with connect_database(self.db_path) as conn:
|
||||
conn.execute(
|
||||
"UPDATE job_items SET status = 'interrupted', worker_id = NULL, ended_at = CURRENT_TIMESTAMP, last_error = ? WHERE id = ?",
|
||||
(last_error, item_id),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted test and verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_runner -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- orphaned running items become `interrupted`
|
||||
- soft pause waits until no item remains running before closing the stage
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/ops/runner.py musicdl/catalogsync/ops/models.py musicdl/catalogsync/ops/repository.py tests/catalogsync/test_ops_runner.py
|
||||
git commit -m "feat: add operations runner state machine and recovery"
|
||||
```
|
||||
|
||||
### Task 4: Add Stage Executors And Single-Item Execution Hooks
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/ops/executors.py`
|
||||
- Modify: `musicdl/catalogsync/services.py`
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `musicdl/catalogsync/uploader.py`
|
||||
- Modify: `musicdl/catalogsync/ops/repository.py`
|
||||
- Test: `tests/catalogsync/test_ops_executors.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing executor integration tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
|
||||
class StageExecutorTests(unittest.TestCase):
|
||||
def test_download_executor_marks_item_succeeded(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.executors import DownloadStageExecutor
|
||||
from musicdl.catalogsync.ops.repository import OperationsRepository
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
library_root = Path(tmpdir) / "library"
|
||||
initialize_database(db_path, default_library_root=library_root).close()
|
||||
repo = OperationsRepository(db_path)
|
||||
stage_id = repo.create_job_stage(repo.create_job_run("download_only", {"library_root": str(library_root)}), "download", 1)
|
||||
item_id = repo.create_job_item(stage_id, "song", "song:1", song_id=1, payload_json={"row": {"id": 1, "platform": "qq"}})
|
||||
|
||||
executor = DownloadStageExecutor(db_path=db_path, library_root=library_root, download_sources=["qq"])
|
||||
with patch("musicdl.catalogsync.downloader.CatalogDownloader.download_song_row", return_value=True):
|
||||
executor.process_item(item_id=item_id, worker_name="download-1")
|
||||
|
||||
item = repo.get_job_item(item_id)
|
||||
|
||||
self.assertEqual("succeeded", item["status"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted test and verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_executors -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the executor layer and single-item helpers do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement executor adapters and expose item-level hooks**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/downloader.py
|
||||
class CatalogDownloader:
|
||||
def download_song_row(
|
||||
self,
|
||||
row: dict,
|
||||
library_root: str | Path,
|
||||
download_sources: list[str] | None = None,
|
||||
worker_callback=None,
|
||||
) -> bool:
|
||||
default_root = Path(library_root).resolve()
|
||||
if worker_callback:
|
||||
worker_callback(
|
||||
current_song_id=int(row["id"]),
|
||||
current_playlist_id=row.get("playlist_id"),
|
||||
current_display_text=f'{row.get("name", row["id"])} / {row.get("singers", "")}'.strip(" /"),
|
||||
)
|
||||
return self._download_one(row=row, default_root=default_root, download_sources=download_sources)
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/uploader.py
|
||||
class CatalogUploader:
|
||||
def process_upload_task_row(self, task_row, backend_name: str) -> str:
|
||||
backend = self.get_backend(backend_name)
|
||||
return self._process_task(task_row, backend, uploader=None)
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/services.py
|
||||
class CatalogSyncService:
|
||||
def sync_playlist_row(self, playlist_row) -> int:
|
||||
song_infos = self.resolve_playlist_song_infos(playlist_row)
|
||||
source_pool_ids = self.repository.get_pool_ids_for_playlist(int(playlist_row["id"]))
|
||||
linked_count = 0
|
||||
for source_pool_id in source_pool_ids:
|
||||
linked_count += self.store_playlist_songs(
|
||||
playlist_id=int(playlist_row["id"]),
|
||||
source_pool_id=source_pool_id,
|
||||
song_infos=song_infos,
|
||||
)
|
||||
return linked_count
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/executors.py
|
||||
class DownloadStageExecutor:
|
||||
def process_item(self, item_id: int, worker_name: str) -> None:
|
||||
item = self.ops_repo.claim_item(item_id=item_id, worker_name=worker_name)
|
||||
row = self.ops_repo.build_download_row(item_id)
|
||||
ok = self.downloader.download_song_row(
|
||||
row=row,
|
||||
library_root=self.library_root,
|
||||
download_sources=self.download_sources,
|
||||
worker_callback=lambda **state: self.ops_repo.update_worker_state(worker_name=worker_name, **state),
|
||||
)
|
||||
if ok:
|
||||
self.ops_repo.mark_item_succeeded(item_id)
|
||||
else:
|
||||
self.ops_repo.mark_item_failed(item_id, "download returned no file")
|
||||
|
||||
|
||||
class SyncStageExecutor:
|
||||
def process_item(self, item_id: int, worker_name: str) -> None:
|
||||
item = self.ops_repo.claim_item(item_id=item_id, worker_name=worker_name)
|
||||
playlist_row = self.ops_repo.get_playlist_row_for_item(item_id)
|
||||
linked_count = self.service.sync_playlist_row(playlist_row)
|
||||
self.ops_repo.mark_item_succeeded(item_id, result_payload={"linked_count": linked_count})
|
||||
|
||||
|
||||
class UploadStageExecutor:
|
||||
def process_item(self, item_id: int, worker_name: str) -> None:
|
||||
item = self.ops_repo.claim_item(item_id=item_id, worker_name=worker_name)
|
||||
upload_row = self.ops_repo.get_upload_row_for_item(item_id)
|
||||
result = self.uploader.process_upload_task_row(upload_row, backend_name=self.backend_name)
|
||||
if result == "succeeded":
|
||||
self.ops_repo.mark_item_succeeded(item_id)
|
||||
else:
|
||||
self.ops_repo.mark_item_failed(item_id, f"upload result: {result}")
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted test and verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_executors -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- download, sync, and upload work items can be processed one at a time
|
||||
- the item records and worker state update correctly
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/downloader.py musicdl/catalogsync/uploader.py musicdl/catalogsync/services.py musicdl/catalogsync/ops/executors.py musicdl/catalogsync/ops/repository.py tests/catalogsync/test_ops_executors.py
|
||||
git commit -m "feat: add stage executors for operations console"
|
||||
```
|
||||
|
||||
### Task 5: Build The FastAPI UI And Management API
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/ops/web.py`
|
||||
- Create: `musicdl/catalogsync/templates/ops/base.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/dashboard.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/jobs.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/job_detail.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/playlists.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/songs.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/logs.html`
|
||||
- Create: `musicdl/catalogsync/templates/ops/config.html`
|
||||
- Create: `musicdl/catalogsync/static/ops/app.js`
|
||||
- Modify: `setup.py`
|
||||
- Modify: `MANIFEST.in`
|
||||
- Test: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing API and page tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
|
||||
|
||||
class OperationsApiTests(unittest.TestCase):
|
||||
def test_dashboard_and_jobs_endpoints_render(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.web import create_app
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
env_path = Path(tmpdir) / "catalogsync.env"
|
||||
env_path.write_text("LIBRARY_DIR=/volume4/Music_Cloud/library\n", encoding="utf-8")
|
||||
initialize_database(db_path).close()
|
||||
client = TestClient(create_app(db_path=db_path, env_path=env_path))
|
||||
|
||||
dashboard = client.get("/dashboard")
|
||||
jobs = client.get("/api/jobs")
|
||||
|
||||
self.assertEqual(200, dashboard.status_code)
|
||||
self.assertEqual(200, jobs.status_code)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted test and verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the FastAPI app, templates, and API endpoints do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement the FastAPI app, pages, APIs, and SSE**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/ops/web.py
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.responses import HTMLResponse, StreamingResponse
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
from fastapi.templating import Jinja2Templates
|
||||
|
||||
|
||||
def create_app(db_path: str | Path, env_path: str | Path) -> FastAPI:
|
||||
app = FastAPI(title="Catalogsync Operations Console")
|
||||
repo = OperationsRepository(db_path)
|
||||
env_manager = CatalogsyncEnvManager(env_path=env_path, repository=repo)
|
||||
templates = Jinja2Templates(directory=str(Path(__file__).resolve().parents[1] / "templates"))
|
||||
app.mount("/static", StaticFiles(directory=str(Path(__file__).resolve().parents[1] / "static")), name="static")
|
||||
@app.get("/dashboard", response_class=HTMLResponse)
|
||||
def dashboard(request: Request):
|
||||
return templates.TemplateResponse(
|
||||
request,
|
||||
"ops/dashboard.html",
|
||||
{"title": "总览", "summary": repo.get_dashboard_summary()},
|
||||
)
|
||||
|
||||
@app.get("/api/jobs")
|
||||
def api_jobs():
|
||||
return {"items": repo.list_jobs()}
|
||||
|
||||
@app.get("/api/events/stream")
|
||||
def api_events():
|
||||
def event_stream():
|
||||
while True:
|
||||
yield f"data: {json.dumps(repo.get_live_snapshot(), ensure_ascii=False)}\n\n"
|
||||
return StreamingResponse(event_stream(), media_type="text/event-stream")
|
||||
```
|
||||
|
||||
```html
|
||||
<!-- musicdl/catalogsync/templates/ops/base.html -->
|
||||
<!DOCTYPE html>
|
||||
<html lang="zh-CN">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<title>{{ title or "Catalogsync Console" }}</title>
|
||||
<script src="/static/ops/app.js" defer></script>
|
||||
</head>
|
||||
<body data-sse-url="/api/events/stream">
|
||||
<nav>
|
||||
<a href="/dashboard">总览</a>
|
||||
<a href="/jobs">任务中心</a>
|
||||
<a href="/playlists">歌单池</a>
|
||||
<a href="/songs">歌曲处理</a>
|
||||
<a href="/logs">日志异常</a>
|
||||
<a href="/config">配置管理</a>
|
||||
</nav>
|
||||
{% block content %}{% endblock %}
|
||||
</body>
|
||||
</html>
|
||||
```
|
||||
|
||||
```text
|
||||
# MANIFEST.in
|
||||
recursive-include musicdl/catalogsync/templates/ops *.html
|
||||
recursive-include musicdl/catalogsync/static/ops *.js
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the targeted test and verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- page routes render
|
||||
- queue-control and config endpoints return the expected status codes
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/ops/web.py musicdl/catalogsync/templates/ops/base.html musicdl/catalogsync/templates/ops/dashboard.html musicdl/catalogsync/templates/ops/jobs.html musicdl/catalogsync/templates/ops/job_detail.html musicdl/catalogsync/templates/ops/playlists.html musicdl/catalogsync/templates/ops/songs.html musicdl/catalogsync/templates/ops/logs.html musicdl/catalogsync/templates/ops/config.html musicdl/catalogsync/static/ops/app.js setup.py MANIFEST.in tests/catalogsync/test_ops_api.py
|
||||
git commit -m "feat: add operations console web app"
|
||||
```
|
||||
|
||||
### Task 6: Wire The CLI, Runtime Scripts, Docs, And Final Verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `musicdl/catalogsync/runtime.py`
|
||||
- Modify: `requirements.txt`
|
||||
- Modify: `setup.py`
|
||||
- Modify: `scripts/catalogsync/templates/catalogsync.env.example`
|
||||
- Modify: `scripts/catalogsync/templates/install_runtime.sh`
|
||||
- Create: `scripts/catalogsync/templates/serve_console.sh`
|
||||
- Modify: `docs/catalogsync.md`
|
||||
- Modify: `README.md`
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
- Modify: `tests/catalogsync/test_runtime.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing CLI and runtime tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from click.testing import CliRunner
|
||||
|
||||
|
||||
class CatalogConsoleCliTests(unittest.TestCase):
|
||||
def test_serve_command_builds_web_app(self):
|
||||
from musicdl.catalogsync.cli import cli
|
||||
|
||||
runner = CliRunner()
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
env_path = Path(tmpdir) / "catalogsync.env"
|
||||
env_path.write_text("LIBRARY_DIR=/volume4/Music_Cloud/library\n", encoding="utf-8")
|
||||
|
||||
with patch("musicdl.catalogsync.cli.uvicorn.run") as uvicorn_run:
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
[
|
||||
"serve",
|
||||
"--db",
|
||||
str(db_path),
|
||||
"--env-file",
|
||||
str(env_path),
|
||||
"--host",
|
||||
"0.0.0.0",
|
||||
"--port",
|
||||
"8421",
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(0, result.exit_code, msg=result.output)
|
||||
uvicorn_run.assert_called_once()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the targeted tests and verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_cli tests.catalogsync.test_runtime -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAIL` because the `serve` command and web runtime fields do not exist yet
|
||||
|
||||
- [ ] **Step 3: Implement the serve command, runtime fields, dependencies, and NAS script**
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/runtime.py
|
||||
@dataclass
|
||||
class CatalogSyncRuntimeConfig:
|
||||
root_dir: Path
|
||||
app_home: Path
|
||||
library_dir: Path
|
||||
db_path: Path
|
||||
input_dir: Path
|
||||
log_dir: Path
|
||||
python_bin: str
|
||||
venv_dir: Path
|
||||
download_layout: str
|
||||
env_file: Path
|
||||
web_host: str
|
||||
web_port: int
|
||||
```
|
||||
|
||||
```python
|
||||
# musicdl/catalogsync/cli.py
|
||||
@cli.command("serve")
|
||||
@click.option("--db", "db_path", required=True, type=click.Path(dir_okay=False))
|
||||
@click.option("--env-file", required=True, type=click.Path(dir_okay=False, exists=True))
|
||||
@click.option("--host", default="0.0.0.0", show_default=True)
|
||||
@click.option("--port", default=8421, type=int, show_default=True)
|
||||
def serve_command(db_path: str, env_file: str, host: str, port: int):
|
||||
import uvicorn
|
||||
from .ops.web import create_app
|
||||
|
||||
app = create_app(db_path=db_path, env_path=env_file)
|
||||
uvicorn.run(app, host=host, port=port)
|
||||
```
|
||||
|
||||
```bash
|
||||
# scripts/catalogsync/templates/serve_console.sh
|
||||
"${VENV_DIR}/bin/python" -m musicdl.catalogsync.cli serve \
|
||||
--db "${DB_PATH}" \
|
||||
--env-file "${ENV_FILE:-${CONFIG_FILE}}" \
|
||||
--host "${WEB_HOST:-0.0.0.0}" \
|
||||
--port "${WEB_PORT:-8421}"
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Update docs and runtime templates**
|
||||
|
||||
```text
|
||||
# scripts/catalogsync/templates/catalogsync.env.example
|
||||
ENV_FILE=/volume4/Music_Cloud/catalogsync/config/catalogsync.env
|
||||
WEB_HOST=0.0.0.0
|
||||
WEB_PORT=8421
|
||||
```
|
||||
|
||||
Document in:
|
||||
|
||||
- `docs/catalogsync.md`
|
||||
- `README.md`
|
||||
|
||||
- [ ] **Step 5: Run the full verification suite**
|
||||
|
||||
Run: `python -m unittest discover -s tests/catalogsync -v`
|
||||
|
||||
Expected:
|
||||
|
||||
- `OK`
|
||||
- all previous catalogsync tests still pass
|
||||
- the new operations-console tests pass
|
||||
|
||||
Run: `python -m musicdl.catalogsync.cli serve --help`
|
||||
|
||||
Expected:
|
||||
|
||||
- help output lists `--db`, `--env-file`, `--host`, and `--port`
|
||||
|
||||
- [ ] **Step 6: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/cli.py musicdl/catalogsync/runtime.py requirements.txt setup.py scripts/catalogsync/templates/catalogsync.env.example scripts/catalogsync/templates/install_runtime.sh scripts/catalogsync/templates/serve_console.sh docs/catalogsync.md README.md tests/catalogsync/test_cli.py tests/catalogsync/test_runtime.py
|
||||
git commit -m "feat: ship operations console runtime and docs"
|
||||
```
|
||||
@@ -0,0 +1,427 @@
|
||||
# Object Storage Upload Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add an S3-compatible object storage upload pipeline to `musicdl.catalogsync`, persist remote locations and backend presence, and expose it through dedicated CLI commands while keeping the existing local download workflow intact.
|
||||
|
||||
**Architecture:** Extend the SQLite schema and repository so upload state is queue-driven and derived from the existing `file_assets` plus `file_locations` model. Add a focused `uploader.py` module that plans missing uploads, resolves credentials from environment variables, uploads to an object backend with limited concurrency, and records remote locations plus summary presence rows.
|
||||
|
||||
**Tech Stack:** Python stdlib (`json`, `os`, `pathlib`, `threading`, `concurrent.futures`), `sqlite3`, `click`, `boto3`, existing `musicdl.catalogsync` modules, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Extend schema and repository for object storage uploads
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/db.py`
|
||||
- Modify: `musicdl/catalogsync/repository.py`
|
||||
- Modify: `tests/catalogsync/test_db.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing schema and repository tests**
|
||||
|
||||
```python
|
||||
class DatabaseSchemaTests(unittest.TestCase):
|
||||
def test_initialize_database_creates_upload_tables(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
conn = initialize_database(db_path)
|
||||
conn.close()
|
||||
|
||||
with closing(sqlite3.connect(db_path)) as verify_conn:
|
||||
tables = {
|
||||
row[0]
|
||||
for row in verify_conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
||||
).fetchall()
|
||||
}
|
||||
|
||||
self.assertIn("song_backend_presence", tables)
|
||||
self.assertIn("upload_tasks", tables)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused schema tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_db -v`
|
||||
Expected: FAIL because `song_backend_presence` and `upload_tasks` do not exist yet.
|
||||
|
||||
- [ ] **Step 3: Implement schema, backend upsert, presence refresh, and upload queue helpers**
|
||||
|
||||
```python
|
||||
REQUIRED_TABLES = {
|
||||
"playlist_pools",
|
||||
"playlists",
|
||||
"pool_playlists",
|
||||
"artist_pools",
|
||||
"artists",
|
||||
"pool_artists",
|
||||
"songs",
|
||||
"playlist_songs",
|
||||
"artist_songs",
|
||||
"storage_backends",
|
||||
"file_assets",
|
||||
"file_locations",
|
||||
"download_tasks",
|
||||
"song_backend_presence",
|
||||
"upload_tasks",
|
||||
}
|
||||
|
||||
def upsert_object_storage_backend(
|
||||
self,
|
||||
name: str,
|
||||
container_name: str,
|
||||
endpoint: str,
|
||||
region: str | None,
|
||||
base_prefix: str | None,
|
||||
credential_env_prefix: str,
|
||||
addressing_style: str | None = None,
|
||||
public_base_url: str | None = None,
|
||||
) -> int:
|
||||
config = {
|
||||
"endpoint": endpoint,
|
||||
"region": region,
|
||||
"base_prefix": base_prefix,
|
||||
"addressing_style": addressing_style,
|
||||
"public_base_url": public_base_url,
|
||||
"credential_env_prefix": credential_env_prefix,
|
||||
}
|
||||
return self._upsert_backend_row(name=name, backend_type="object_storage", container_name=container_name, config=config)
|
||||
|
||||
def get_backend_by_name(self, name: str) -> sqlite3.Row | None:
|
||||
return self._fetchone("SELECT * FROM storage_backends WHERE name = ?", (name,))
|
||||
|
||||
def record_remote_file(
|
||||
self,
|
||||
file_asset_id: int,
|
||||
backend_id: int,
|
||||
container_name: str,
|
||||
locator: str,
|
||||
public_url: str | None,
|
||||
download_url: str | None,
|
||||
) -> int:
|
||||
return self._execute(
|
||||
"""
|
||||
INSERT INTO file_locations (
|
||||
file_asset_id, backend_id, container_name, locator, absolute_path,
|
||||
public_url, download_url, status, is_primary
|
||||
) VALUES (?, ?, ?, ?, NULL, ?, ?, 'active', 0)
|
||||
ON CONFLICT(file_asset_id, backend_id, locator) DO UPDATE SET
|
||||
public_url = excluded.public_url,
|
||||
download_url = excluded.download_url,
|
||||
status = excluded.status,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(file_asset_id, backend_id, container_name, locator, public_url, download_url),
|
||||
)
|
||||
|
||||
def refresh_song_backend_presence(self, song_id: int, backend_id: int) -> None:
|
||||
self._execute_presence_refresh(song_id=song_id, backend_id=backend_id)
|
||||
def enqueue_upload_task(
|
||||
self,
|
||||
file_asset_id: int,
|
||||
source_location_id: int,
|
||||
target_backend_id: int,
|
||||
target_container_name: str,
|
||||
target_locator: str,
|
||||
) -> int:
|
||||
return self._execute(
|
||||
"""
|
||||
INSERT INTO upload_tasks (
|
||||
file_asset_id, source_location_id, target_backend_id, target_container_name, target_locator
|
||||
) VALUES (?, ?, ?, ?, ?)
|
||||
ON CONFLICT(file_asset_id, target_backend_id, target_locator) DO NOTHING
|
||||
""",
|
||||
(file_asset_id, source_location_id, target_backend_id, target_container_name, target_locator),
|
||||
)
|
||||
|
||||
def list_pending_upload_tasks(self, target_backend_id: int, limit: int | None = None) -> list[sqlite3.Row]:
|
||||
return self._fetchall(
|
||||
"""
|
||||
SELECT ut.*, fl.absolute_path, fa.song_id
|
||||
FROM upload_tasks ut
|
||||
JOIN file_locations fl ON fl.id = ut.source_location_id
|
||||
JOIN file_assets fa ON fa.id = ut.file_asset_id
|
||||
WHERE ut.target_backend_id = ? AND ut.status IN ('pending', 'failed')
|
||||
ORDER BY ut.id ASC
|
||||
LIMIT COALESCE(?, -1)
|
||||
""",
|
||||
(target_backend_id, limit),
|
||||
)
|
||||
|
||||
def mark_upload_task_status(self, task_id: int, status: str, last_error: str | None = None) -> None:
|
||||
self._execute(
|
||||
"""
|
||||
UPDATE upload_tasks
|
||||
SET status = ?, last_error = ?, updated_at = CURRENT_TIMESTAMP
|
||||
WHERE id = ?
|
||||
""",
|
||||
(status, last_error, task_id),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused schema tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_db -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/db.py musicdl/catalogsync/repository.py tests/catalogsync/test_db.py
|
||||
git commit -m "feat: add upload queue schema and repository helpers"
|
||||
```
|
||||
|
||||
### Task 2: Add the object storage uploader module with limited concurrency
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/uploader.py`
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Write failing uploader tests**
|
||||
|
||||
```python
|
||||
class ObjectStorageUploadTests(unittest.TestCase):
|
||||
def test_upload_runner_records_remote_location_and_presence(self):
|
||||
uploader = CatalogUploader(repository=repo, worker_count=2)
|
||||
backend_id = repo.upsert_object_storage_backend(
|
||||
name="main-s3",
|
||||
container_name="music-bucket",
|
||||
endpoint="https://s3.example.com",
|
||||
region="auto",
|
||||
base_prefix="music",
|
||||
credential_env_prefix="CATALOGSYNC_MAIN_S3",
|
||||
)
|
||||
|
||||
task_count = uploader.enqueue_missing_uploads(backend_name="main-s3")
|
||||
self.assertEqual(1, task_count)
|
||||
|
||||
with patch("musicdl.catalogsync.uploader.build_s3_client", return_value=fake_client):
|
||||
summary = uploader.run(backend_name="main-s3")
|
||||
|
||||
self.assertEqual(1, summary["succeeded"])
|
||||
self.assertTrue(repo.song_has_active_backend_file(song_id, backend_id))
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused uploader tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services.ObjectStorageUploadTests -v`
|
||||
Expected: FAIL because `musicdl.catalogsync.uploader` and upload repository helpers do not exist yet.
|
||||
|
||||
- [ ] **Step 3: Implement the upload planner and S3-compatible runner**
|
||||
|
||||
```python
|
||||
class CatalogUploader:
|
||||
def enqueue_missing_uploads(self, backend_name: str, song_ids: list[int] | None = None) -> int:
|
||||
backend = self.repository.get_backend_by_name(backend_name)
|
||||
candidates = self.repository.list_missing_object_upload_candidates(int(backend["id"]), song_ids=song_ids)
|
||||
return sum(
|
||||
1
|
||||
for row in candidates
|
||||
if self.repository.enqueue_upload_task(
|
||||
file_asset_id=int(row["file_asset_id"]),
|
||||
source_location_id=int(row["source_location_id"]),
|
||||
target_backend_id=int(backend["id"]),
|
||||
target_container_name=backend["container_name"],
|
||||
target_locator=row["target_locator"],
|
||||
)
|
||||
)
|
||||
|
||||
def run(self, backend_name: str, limit: int | None = None) -> dict[str, int]:
|
||||
tasks = self.repository.list_pending_upload_tasks(target_backend_id=backend_id, limit=limit)
|
||||
return self._run_tasks(tasks, backend)
|
||||
|
||||
class S3CompatibleUploader:
|
||||
def upload_file(self, local_path: Path, container_name: str, locator: str) -> dict[str, str | None]:
|
||||
self.client.upload_file(str(local_path), container_name, locator, ExtraArgs=extra_args or None)
|
||||
return {"public_url": public_url, "download_url": None}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused uploader tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services.ObjectStorageUploadTests -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/uploader.py tests/catalogsync/test_services.py
|
||||
git commit -m "feat: add s3 compatible uploader"
|
||||
```
|
||||
|
||||
### Task 3: Expose backend registration and upload execution through the CLI
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
- Modify: `setup.py`
|
||||
- Modify: `requirements.txt`
|
||||
|
||||
- [ ] **Step 1: Write failing CLI tests for backend registration and upload**
|
||||
|
||||
```python
|
||||
def test_register_object_backend_command_wires_application_method(self):
|
||||
from musicdl.catalogsync.cli import cli
|
||||
|
||||
with patch("musicdl.catalogsync.cli.CatalogSyncApplication") as app_cls:
|
||||
result = CliRunner().invoke(
|
||||
cli,
|
||||
[
|
||||
"register-object-backend",
|
||||
"--db", str(db_path),
|
||||
"--name", "main-s3",
|
||||
"--bucket", "music-bucket",
|
||||
"--endpoint", "https://s3.example.com",
|
||||
"--region", "auto",
|
||||
"--base-prefix", "music",
|
||||
"--credential-env-prefix", "CATALOGSYNC_MAIN_S3",
|
||||
],
|
||||
)
|
||||
|
||||
self.assertEqual(0, result.exit_code, msg=result.output)
|
||||
app_cls.return_value.register_object_backend.assert_called_once()
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused CLI tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_cli -v`
|
||||
Expected: FAIL because the new commands and application methods do not exist yet.
|
||||
|
||||
- [ ] **Step 3: Add CLI methods and commands**
|
||||
|
||||
```python
|
||||
class CatalogSyncApplication:
|
||||
def register_object_backend(self, **kwargs):
|
||||
return self.repository.upsert_object_storage_backend(**kwargs)
|
||||
|
||||
def upload_files(self, backend_name: str, workers: int = 4, limit: int | None = None, enqueue_only: bool = False):
|
||||
uploader = CatalogUploader(self.repository, worker_count=workers)
|
||||
queued = uploader.enqueue_missing_uploads(backend_name=backend_name)
|
||||
return {"queued": queued} if enqueue_only else uploader.run(backend_name=backend_name, limit=limit)
|
||||
|
||||
@cli.command("register-object-backend")
|
||||
@click.option("--name", required=True)
|
||||
@click.option("--bucket", "container_name", required=True)
|
||||
@click.option("--endpoint", required=True)
|
||||
@click.option("--workers", type=int, default=4, show_default=True)
|
||||
@click.option("--enqueue-only/--run", default=False, show_default=True)
|
||||
def upload_command(db_path: str, library_root: str | None, backend_name: str, workers: int, limit: int | None, enqueue_only: bool):
|
||||
app = CatalogSyncApplication(db_path=db_path, library_root=library_root)
|
||||
result = app.upload_files(
|
||||
backend_name=backend_name,
|
||||
workers=workers,
|
||||
limit=limit,
|
||||
enqueue_only=enqueue_only,
|
||||
)
|
||||
click.echo(result)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused CLI tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_cli -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/cli.py tests/catalogsync/test_cli.py setup.py requirements.txt
|
||||
git commit -m "feat: add upload cli commands"
|
||||
```
|
||||
|
||||
### Task 4: Add bounded concurrency to download and upload execution
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `musicdl/catalogsync/uploader.py`
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
|
||||
- [ ] **Step 1: Write failing concurrency and disk-switch tests**
|
||||
|
||||
```python
|
||||
def test_catalog_downloader_reuses_new_root_after_space_prompt(self):
|
||||
downloader = CatalogDownloader(repository=repo, worker_count=2)
|
||||
with patch("builtins.input", side_effect=[str(second_root)]):
|
||||
count = downloader.download_pending(library_root=first_root, limit=2)
|
||||
self.assertEqual(2, count)
|
||||
|
||||
def test_catalog_uploader_uses_bounded_workers(self):
|
||||
uploader = CatalogUploader(repository=repo, worker_count=3)
|
||||
summary = uploader.run(backend_name="main-s3")
|
||||
self.assertEqual(3, summary["workers"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused concurrency tests to verify they fail**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services -v`
|
||||
Expected: FAIL because downloader and uploader are still single-threaded or do not report bounded worker behavior yet.
|
||||
|
||||
- [ ] **Step 3: Implement limited worker pools and one-time root switching**
|
||||
|
||||
```python
|
||||
class CatalogDownloader:
|
||||
def __init__(self, repository: CatalogRepository, work_dir: str = "musicdl_outputs/catalogsync", worker_count: int = 3):
|
||||
self.worker_count = max(1, worker_count)
|
||||
self._current_library_root: Path | None = None
|
||||
|
||||
def ensure_space(self, root_path: str | Path, required_bytes: int | None) -> Path:
|
||||
if self._current_library_root is None:
|
||||
self._current_library_root = Path(root_path).resolve()
|
||||
root = self._current_library_root
|
||||
while required_bytes and shutil.disk_usage(root).free < required_bytes:
|
||||
root = Path(input("磁盘空间不足，请输入新的下载目录继续: ").strip()).resolve()
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
self._current_library_root = root
|
||||
return root
|
||||
|
||||
def download_pending(self, library_root: str | Path, sources: list[str] | None = None, limit: int | None = None, playlist_ids: list[int] | None = None) -> int:
|
||||
with ThreadPoolExecutor(max_workers=self.worker_count) as executor:
|
||||
futures = [executor.submit(self._download_one, row, default_root) for row in queue]
|
||||
return sum(1 for future in as_completed(futures) if future.result())
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Re-run the focused concurrency tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_services -v`
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/downloader.py musicdl/catalogsync/uploader.py tests/catalogsync/test_services.py
|
||||
git commit -m "feat: add bounded download and upload concurrency"
|
||||
```
|
||||
|
||||
### Task 5: Document the operator workflow and run final verification
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
- Modify: `README.md`
|
||||
|
||||
- [ ] **Step 1: Update operator docs for object storage upload**
|
||||
|
||||
```markdown
|
||||
## Object Storage Upload
|
||||
|
||||
1. Register one backend with `musicdl-catalogsync register-object-backend`
|
||||
2. Export `${PREFIX}_ACCESS_KEY_ID` and `${PREFIX}_SECRET_ACCESS_KEY`
|
||||
3. Run `musicdl-catalogsync upload --backend main-s3`
|
||||
4. Inspect `file_locations`, `song_backend_presence`, and `upload_tasks`
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Verify help output and test suite**
|
||||
|
||||
Run: `python -m unittest discover -s tests/catalogsync -v`
|
||||
Expected: PASS
|
||||
|
||||
Run: `python -m musicdl.catalogsync.cli run --help`
|
||||
Expected: PASS and include existing run options
|
||||
|
||||
Run: `python -m musicdl.catalogsync.cli upload --help`
|
||||
Expected: PASS and include object storage upload options
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/catalogsync.md README.md
|
||||
git commit -m "docs: add object storage upload workflow"
|
||||
```
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,276 @@
|
||||
# Task Tree Dashboard Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Replace the dashboard Task Center detail tables with a stable task -> playlist -> song tree that updates node state in place.
|
||||
|
||||
**Architecture:** Keep the existing FastAPI endpoints and lazy playlist-song endpoint, but change the repository task query to keep finished tasks visible and change the dashboard frontend from table redraws to keyed tree-node patching. The top dashboard cards remain unchanged in this iteration.
|
||||
|
||||
**Tech Stack:** Python, FastAPI, Jinja2 templates, vanilla JavaScript, unittest
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Keep finished tasks in the Task Center query
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/repository.py`
|
||||
- Modify: `tests/catalogsync/test_ops_repository.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing repository test**
|
||||
|
||||
```python
|
||||
def test_list_task_center_rows_includes_completed_jobs(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.models import JobStatus
|
||||
from musicdl.catalogsync.ops.repository import OpsRepository
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
initialize_database(db_path).close()
|
||||
repo = OpsRepository(db_path)
|
||||
|
||||
completed_job_id = repo.create_job(
|
||||
job_type="download_only",
|
||||
config_snapshot={},
|
||||
status=JobStatus.COMPLETED,
|
||||
playlist_scope={"playlist_ids": [42]},
|
||||
)
|
||||
|
||||
rows = repo.list_task_center_rows(limit=20)
|
||||
|
||||
rows_by_id = {int(row["id"]): row for row in rows}
|
||||
self.assertIn(completed_job_id, rows_by_id)
|
||||
self.assertEqual("completed", rows_by_id[completed_job_id]["status"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_repository.OpsRepositoryTaskCenterTests.test_list_task_center_rows_includes_completed_jobs -v`
|
||||
|
||||
Expected: FAIL because completed jobs are filtered out of `list_task_center_rows()`.
|
||||
|
||||
- [ ] **Step 3: Write the minimal implementation**
|
||||
|
||||
```python
|
||||
rows = self._fetchall(
|
||||
"""
|
||||
SELECT *
|
||||
FROM job_runs
|
||||
WHERE status IN (?, ?, ?, ?, ?, ?, ?, ?)
|
||||
...
|
||||
""",
|
||||
(
|
||||
JobStatus.RUNNING.value,
|
||||
JobStatus.PAUSE_REQUESTED.value,
|
||||
JobStatus.QUEUED.value,
|
||||
JobStatus.PAUSED.value,
|
||||
JobStatus.COMPLETED.value,
|
||||
JobStatus.COMPLETED_WITH_ERRORS.value,
|
||||
JobStatus.FAILED.value,
|
||||
JobStatus.CANCELED.value,
|
||||
...
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_repository.OpsRepositoryTaskCenterTests.test_list_task_center_rows_includes_completed_jobs -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/ops/repository.py tests/catalogsync/test_ops_repository.py
|
||||
git commit -m "test: keep completed tasks in task center"
|
||||
```
|
||||
|
||||
### Task 2: Lock dashboard HTML to the tree shell
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/templates/ops/dashboard.html`
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing dashboard HTML test**
|
||||
|
||||
```python
|
||||
def test_dashboard_page_renders_task_tree_shell_without_detail_tables(self):
|
||||
from musicdl.catalogsync.ops.models import JobStatus
|
||||
from musicdl.catalogsync.ops.repository import OpsRepository
|
||||
|
||||
client, db_path, _ = self._build_client()
|
||||
repo = OpsRepository(db_path)
|
||||
job_id = repo.create_job(
|
||||
job_type="download_only",
|
||||
config_snapshot={},
|
||||
status=JobStatus.RUNNING,
|
||||
)
|
||||
|
||||
response = client.get("/dashboard")
|
||||
html = response.text
|
||||
|
||||
self.assertIn('data-task-tree-root', html)
|
||||
self.assertIn(f'data-task-node="{job_id}"', html)
|
||||
self.assertNotIn("<h3>Summary</h3>", html)
|
||||
self.assertNotIn("<h3>Stages</h3>", html)
|
||||
self.assertNotIn("<h3>Workers</h3>", html)
|
||||
self.assertNotIn("<h3>Running Items</h3>", html)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api.OperationsApiTests.test_dashboard_page_renders_task_tree_shell_without_detail_tables -v`
|
||||
|
||||
Expected: FAIL because the template still renders the table/detail shell.
|
||||
|
||||
- [ ] **Step 3: Write the minimal template implementation**
|
||||
|
||||
```html
|
||||
<div class="card">
|
||||
<h2>Task Center</h2>
|
||||
<div class="task-tree" data-task-tree-root>
|
||||
{% for row in task_rows %}
|
||||
<section class="task-node" data-task-node="{{ row.id }}">
|
||||
<div class="task-node__header">
|
||||
<button type="button" data-task-toggle="{{ row.id }}">+</button>
|
||||
<div class="task-node__meta">
|
||||
<strong>{{ row.display_name }}</strong>
|
||||
<div class="muted">{{ row.job_type }}</div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="task-node__children" data-task-children="{{ row.id }}" hidden></div>
|
||||
</section>
|
||||
{% endfor %}
|
||||
</div>
|
||||
</div>
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api.OperationsApiTests.test_dashboard_page_renders_task_tree_shell_without_detail_tables -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/templates/ops/dashboard.html tests/catalogsync/test_ops_api.py
|
||||
git commit -m "feat: replace task center table with tree shell"
|
||||
```
|
||||
|
||||
### Task 3: Replace Task Center redraw with keyed tree patching
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/static/ops/app.js`
|
||||
- Modify: `musicdl/catalogsync/templates/ops/base.html`
|
||||
|
||||
- [ ] **Step 1: Add the tree patch helpers**
|
||||
|
||||
```javascript
|
||||
function upsertTaskTree(rows) {
|
||||
var root = document.querySelector("[data-task-tree-root]");
|
||||
if (!root) {
|
||||
return;
|
||||
}
|
||||
var seen = {};
|
||||
rows.forEach(function (row) {
|
||||
var id = String(row.id);
|
||||
seen[id] = true;
|
||||
var node = ensureTaskNode(root, row);
|
||||
patchTaskNode(node, row);
|
||||
});
|
||||
pruneMissingTaskNodes(root, seen);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Switch `updateDashboard()` away from `setTaskRows()`**
|
||||
|
||||
```javascript
|
||||
if (Object.prototype.hasOwnProperty.call(payload, "task_rows")) {
|
||||
dashboardState.taskRows = payload.task_rows || [];
|
||||
pruneTaskState(dashboardState.taskRows);
|
||||
upsertTaskTree(dashboardState.taskRows);
|
||||
restoreExpandedTaskRows();
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Rebuild expanded task rendering as playlist nodes only**
|
||||
|
||||
```javascript
|
||||
function applyTaskDetail(jobId, payload) {
|
||||
dashboardState.detailCache[String(jobId)] = payload;
|
||||
patchPlaylistTree(String(jobId), payload.playlist_progress || []);
|
||||
restoreExpandedPlaylistRows(String(jobId));
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Rebuild playlist song rendering as song child nodes**
|
||||
|
||||
```javascript
|
||||
function applyPlaylistSongs(jobId, playlistId, songs) {
|
||||
var key = playlistKey(jobId, playlistId);
|
||||
var body = document.querySelector('[data-playlist-song-list="' + key + '"]');
|
||||
patchSongTree(body, songs || []);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 5: Version the static asset to force the browser to pick up the new script**
|
||||
|
||||
```html
|
||||
<script src="/static/ops/app.js?v=20260417_task_tree_v3" defer></script>
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run targeted API tests**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api tests.catalogsync.test_ops_repository -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/static/ops/app.js musicdl/catalogsync/templates/ops/base.html
|
||||
git commit -m "feat: patch dashboard task tree in place"
|
||||
```
|
||||
|
||||
### Task 4: Final verification and docs sync
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
|
||||
- [ ] **Step 1: Document the new dashboard behavior**
|
||||
|
||||
```markdown
|
||||
## Task Center
|
||||
|
||||
The dashboard Task Center now renders a tree:
|
||||
|
||||
- task
|
||||
- playlist
|
||||
- song
|
||||
|
||||
Task state updates patch the existing node in place. Expanding a task no longer renders Summary, Stages, Workers, or Running Items tables.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run regression verification**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api tests.catalogsync.test_ops_repository -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
- [ ] **Step 3: Manual browser verification**
|
||||
|
||||
Run the dashboard, expand one task and one playlist, wait through multiple refresh cycles, and verify:
|
||||
|
||||
- no large detail tables appear
|
||||
- paused/completed tasks stay visible
|
||||
- expanded nodes remain expanded
|
||||
- the task tree does not visibly flash as a full block
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/catalogsync.md
|
||||
git commit -m "docs: describe task tree dashboard"
|
||||
```
|
||||
@@ -0,0 +1,61 @@
|
||||
# Playlist Export On Download Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 把歌单导出目录的生成时机从 `sync` 挪到“所选歌单下载完成后”，并增加“输出所选歌单”按钮做状态分流导出。
|
||||
|
||||
**Architecture:** 后端以 `CatalogRepository` 的歌单状态为准，把所选歌单分成“直接导出 / download_only / sync_download”三组。`sync` 阶段不再写 `playlists/`，而是在带歌单作用域的下载阶段结束后统一刷新对应歌单的导出目录。
|
||||
|
||||
**Tech Stack:** Python, FastAPI, SQLite, Jinja2, vanilla JavaScript, pytest
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Lock behavior with tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
- Modify: `tests/catalogsync/test_ops_runner.py`
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] 写失败测试，证明 `sync_playlist_row()` 不再直接写 `playlists/`
|
||||
- [ ] 写失败测试，证明作用域下载任务完成后会刷新歌单导出目录
|
||||
- [ ] 写失败测试，证明 `POST /api/playlists/export` 会把歌单按状态分流
|
||||
|
||||
### Task 2: Implement backend export routing
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/repository.py`
|
||||
- Modify: `musicdl/catalogsync/services.py`
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
|
||||
- [ ] 增加“按歌单 id 查询导出状态”的仓库方法
|
||||
- [ ] 删除 `sync_playlist_row()` 中的自动导出调用
|
||||
- [ ] 新增 `POST /api/playlists/export`
|
||||
- [ ] 复用现有建任务逻辑返回 `download_only` / `sync_download` 任务信息
|
||||
|
||||
### Task 3: Export after scoped download completes
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/runner.py`
|
||||
|
||||
- [ ] 在下载阶段结束后，对 `playlist_scope.playlist_ids` 执行歌单目录刷新
|
||||
- [ ] 仅对 scoped download job 生效
|
||||
- [ ] 出错只记录事件，不破坏下载阶段主状态
|
||||
|
||||
### Task 4: Update playlist UI
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/templates/ops/playlists.html`
|
||||
- Modify: `musicdl/catalogsync/static/ops/app.js`
|
||||
|
||||
- [ ] 新增 `Export Selected Playlists` 按钮
|
||||
- [ ] 前端处理返回结果，分别提示直接导出数量和新建任务
|
||||
- [ ] 保持单歌单 `Export Folder` 弹窗按钮可用
|
||||
|
||||
### Task 5: Verify
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
|
||||
- [ ] 运行相关 pytest 用例
|
||||
- [ ] 更新项目文档，写明“sync 不导出、download 导出、export selected 分流”
|
||||
@@ -0,0 +1,168 @@
|
||||
# Task Center Bandwidth Summary Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Show real-time aggregate download speed next to the Task Center heading, while rendering upload speed as a placeholder.
|
||||
|
||||
**Architecture:** Reuse existing per-worker `speed_bytes_per_sec` values already stored in `job_workers`, aggregate them in the dashboard payload as `transfer_stats`, and render/update a single Task Center header node from server-rendered HTML plus dashboard refreshes. Upload remains explicitly unimplemented and is shown as a placeholder string.
|
||||
|
||||
**Tech Stack:** FastAPI, Jinja2, vanilla JavaScript, `unittest`
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Lock the API and page contract with tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing API test**
|
||||
|
||||
Add coverage that seeds a running download worker with `speed_bytes_per_sec=2 * 1024 * 1024`, calls `/api/dashboard`, and asserts:
|
||||
|
||||
```python
|
||||
self.assertEqual("2.0 MB/s", payload["transfer_stats"]["download_speed_text"])
|
||||
self.assertEqual("-", payload["transfer_stats"]["upload_speed_text"])
|
||||
```
|
||||
|
||||
Also render `/dashboard` and assert the Task Center area includes:
|
||||
|
||||
```python
|
||||
self.assertIn("Task Center", html)
|
||||
self.assertIn("Down 2.0 MB/s", html)
|
||||
self.assertIn("Up -", html)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api.OperationsApiTests.test_dashboard_transfer_stats_exposes_download_speed_and_upload_placeholder -v`
|
||||
|
||||
Expected: FAIL because `transfer_stats` and the new header text do not exist yet.
|
||||
|
||||
### Task 2: Lock the browser refresh behavior with a frontend test
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_frontend.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing frontend test**
|
||||
|
||||
Expose `updateDashboard`, create a fake `[data-task-center-transfer]` node, call:
|
||||
|
||||
```javascript
|
||||
api.updateDashboard({
|
||||
transfer_stats: {
|
||||
download_speed_text: "2.0 MB/s",
|
||||
upload_speed_text: "-"
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
and assert the node text becomes:
|
||||
|
||||
```javascript
|
||||
"Down 2.0 MB/s | Up -"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_frontend.OperationsFrontendTests.test_update_dashboard_refreshes_task_center_transfer_summary -v`
|
||||
|
||||
Expected: FAIL because no Task Center transfer node is updated yet.
|
||||
|
||||
### Task 3: Implement backend aggregation and initial render
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Modify: `musicdl/catalogsync/templates/ops/dashboard.html`
|
||||
|
||||
- [ ] **Step 1: Add backend transfer aggregation**
|
||||
|
||||
In `musicdl/catalogsync/ops/web.py`, extend worker serialization to include numeric speed fields and add a helper that returns:
|
||||
|
||||
```python
|
||||
{
|
||||
"download_speed_bytes_per_sec": ...,
|
||||
"download_speed_text": ...,
|
||||
"upload_speed_bytes_per_sec": 0,
|
||||
"upload_speed_text": "-",
|
||||
}
|
||||
```
|
||||
|
||||
using the sum of active download worker `speed_bytes_per_sec`.
|
||||
|
||||
- [ ] **Step 2: Add the server-rendered Task Center summary node**
|
||||
|
||||
In `musicdl/catalogsync/templates/ops/dashboard.html`, change the Task Center heading area to include:
|
||||
|
||||
```html
|
||||
<span class="muted" data-task-center-transfer>Down {{ transfer_stats.download_speed_text }} | Up {{ transfer_stats.upload_speed_text }}</span>
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Run the API test to verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_api.OperationsApiTests.test_dashboard_transfer_stats_exposes_download_speed_and_upload_placeholder -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
### Task 4: Implement live refresh wiring
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/static/ops/app.js`
|
||||
|
||||
- [ ] **Step 1: Update the dashboard refresh path**
|
||||
|
||||
In `updateDashboard(payload)`, look up `[data-task-center-transfer]` and set:
|
||||
|
||||
```javascript
|
||||
"Down " + downloadSpeedText + " | Up " + uploadSpeedText
|
||||
```
|
||||
|
||||
with defaults of `"0 B/s"` for missing download speed and `"-"` for upload.
|
||||
|
||||
- [ ] **Step 2: Run the frontend test to verify it passes**
|
||||
|
||||
Run: `python -m unittest tests.catalogsync.test_ops_frontend.OperationsFrontendTests.test_update_dashboard_refreshes_task_center_transfer_summary -v`
|
||||
|
||||
Expected: PASS
|
||||
|
||||
### Task 5: Regression verification
|
||||
|
||||
**Files:**
|
||||
- Modify: none
|
||||
|
||||
- [ ] **Step 1: Run focused regression**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m unittest tests.catalogsync.test_ops_api tests.catalogsync.test_ops_frontend -v
|
||||
node -e "new Function(require('fs').readFileSync('musicdl/catalogsync/static/ops/app.js','utf8')); console.log('app.js syntax ok')"
|
||||
```
|
||||
|
||||
Expected: all selected tests PASS and `app.js syntax ok` prints.
|
||||
|
||||
- [ ] **Step 2: Sync to NAS and restart**
|
||||
|
||||
Sync these files to `/volume4/Music_Cloud/catalogsync/app/...`:
|
||||
|
||||
```text
|
||||
musicdl/catalogsync/ops/web.py
|
||||
musicdl/catalogsync/templates/ops/dashboard.html
|
||||
musicdl/catalogsync/static/ops/app.js
|
||||
```
|
||||
|
||||
Then restart:
|
||||
|
||||
```bash
|
||||
nohup bash /volume4/Music_Cloud/catalogsync/bin/serve_console.sh >/dev/null 2>&1 &
|
||||
```
|
||||
|
||||
- [ ] **Step 3: Verify deployed page**
|
||||
|
||||
Check:
|
||||
|
||||
```bash
|
||||
http://127.0.0.1:18080/dashboard
|
||||
http://127.0.0.1:18080/api/dashboard?include_task_rows=false
|
||||
```
|
||||
|
||||
Expected: the Task Center heading shows `Down ... | Up -`.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Download Runner And Dashboard Fixes Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Fix catalog-sync so download jobs start promptly on NAS and dashboard download speed only reflects truly active download workers.
|
||||
|
||||
**Architecture:** Tighten dashboard worker selection to current running items only, and remove the pre-worker playlist export refresh that can block a download stage before any download worker starts. Keep playlist export behavior during item completion and stage finalization.
|
||||
|
||||
**Tech Stack:** Python, sqlite3, unittest, FastAPI ops dashboard
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Lock The Regression With Tests
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
- Modify: `tests/catalogsync/test_ops_runner.py`
|
||||
|
||||
- [ ] Add a dashboard API regression test that seeds one real running download worker plus stale historical workers and expects transfer speed to only include the live worker.
|
||||
- [ ] Add a runner regression test that keeps a stage open with pending downloads and expects already-completed playlist exports not to run before pending download workers start.
|
||||
- [ ] Run targeted tests first and confirm they fail for the expected reason.
|
||||
|
||||
### Task 2: Apply The Minimal Fix
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Modify: `musicdl/catalogsync/ops/runner.py`
|
||||
|
||||
- [ ] Restrict dashboard worker rows to workers whose current job item is still running under an active job.
|
||||
- [ ] Remove the pre-worker playlist artifact refresh from download stage startup so worker claiming is not blocked by export work.
|
||||
- [ ] Keep existing per-playlist export on item completion and full export refresh on stage completion.
|
||||
|
||||
### Task 3: Verify The Fix
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
- Modify: `tests/catalogsync/test_ops_runner.py`
|
||||
|
||||
- [ ] Run the focused regression tests and confirm they pass.
|
||||
- [ ] Run a slightly wider ops test slice to catch nearby regressions.
|
||||
@@ -0,0 +1,765 @@
|
||||
# Playlist Export Local ZIP Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** 把 `Export` 统一改成“导出到前端本地 ZIP”，服务端先确保 NAS `playlists/` 目录存在，再打包 ZIP 返回浏览器下载。
|
||||
|
||||
**Architecture:** 复用现有 `playlist_artifacts.py` 目录生成链路，不直接从数据库拼 ZIP。新增一个轻量 `export_bundles.py` 负责 ZIP 文件名、临时 bundle 目录和打包逻辑；`ops/web.py` 只负责状态分流、下载接口与 HTTP 响应；前端 `app.js` 只负责触发下载或显示“已入队”的状态。
|
||||
|
||||
**Tech Stack:** Python, FastAPI, Starlette `FileResponse`, `zipfile`, SQLite, vanilla JavaScript, Node-based frontend tests, pytest
|
||||
|
||||
---
|
||||
|
||||
## File Structure
|
||||
|
||||
- Create: `musicdl/catalogsync/export_bundles.py`
|
||||
- 负责：
|
||||
- 生成单歌单 / 多歌单 ZIP 文件名
|
||||
- 把现有歌单目录打成 ZIP
|
||||
- 在服务端临时 bundle 目录落地 ZIP
|
||||
- 根据 token 找回 bundle 路径
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- 负责：
|
||||
- `GET /api/playlists/{playlist_id}/export.zip`
|
||||
- `POST /api/playlists/export-zip`
|
||||
- `GET /api/exports/bundles/{token}.zip`
|
||||
- 保留并弱化 `GET /api/playlists/{playlist_id}/export-folder`
|
||||
- 可选保留 `POST /api/playlists/export` 兼容旧前端，但内部改成调用新逻辑
|
||||
- Modify: `musicdl/catalogsync/templates/ops/playlists.html`
|
||||
- 负责按钮文案从 `Export Folder` / `Export Selected Playlists` 改为 `Export` / `Export Selected`
|
||||
- Modify: `musicdl/catalogsync/static/ops/app.js`
|
||||
- 负责：
|
||||
- 单歌单导出直接下载 ZIP
|
||||
- 批量导出根据后端返回决定“自动下载 ZIP”还是“显示 queued 提示”
|
||||
- 不再把 `Export` 当成“仅 NAS 上生成目录”
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
- 负责后端 API 行为测试
|
||||
- Create: `tests/catalogsync/test_export_bundles.py`
|
||||
- 负责 ZIP 打包 helper 纯逻辑测试
|
||||
- Modify: `tests/catalogsync/test_ops_frontend.py`
|
||||
- 负责前端按钮文案和下载流程测试
|
||||
- Modify: `docs/catalogsync.md`
|
||||
- 负责记录 `Export` 语义更新
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Add ZIP Bundle Helper
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/export_bundles.py`
|
||||
- Test: `tests/catalogsync/test_export_bundles.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
import zipfile
|
||||
|
||||
|
||||
class ExportBundleTests(unittest.TestCase):
|
||||
def test_build_single_playlist_zip_creates_expected_top_level_folder(self):
|
||||
from musicdl.catalogsync.export_bundles import create_single_playlist_bundle
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
playlist_dir = root / "Playlist A_100"
|
||||
(playlist_dir / "covers").mkdir(parents=True, exist_ok=True)
|
||||
(playlist_dir / "playlist.yaml").write_text("playlist_id: 100\n", encoding="utf-8")
|
||||
(playlist_dir / ".playlist_meta.json").write_text("{}", encoding="utf-8")
|
||||
(playlist_dir / "covers" / "playlist-cover.jpg").write_bytes(b"cover")
|
||||
|
||||
bundle_path = create_single_playlist_bundle(
|
||||
bundle_root=root / "bundles",
|
||||
playlist_dir=playlist_dir,
|
||||
playlist={"id": 100, "platform": "qq", "name": "Playlist A"},
|
||||
)
|
||||
|
||||
self.assertTrue(bundle_path.exists())
|
||||
with zipfile.ZipFile(bundle_path) as zf:
|
||||
names = set(zf.namelist())
|
||||
|
||||
self.assertIn("Playlist A_100/playlist.yaml", names)
|
||||
self.assertIn("Playlist A_100/.playlist_meta.json", names)
|
||||
self.assertIn("Playlist A_100/covers/playlist-cover.jpg", names)
|
||||
|
||||
def test_build_multi_playlist_zip_wraps_directories_under_playlists_root(self):
|
||||
from musicdl.catalogsync.export_bundles import create_multi_playlist_bundle
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
playlist_a = root / "Playlist A_100"
|
||||
playlist_b = root / "Playlist B_200"
|
||||
playlist_a.mkdir(parents=True, exist_ok=True)
|
||||
playlist_b.mkdir(parents=True, exist_ok=True)
|
||||
(playlist_a / "playlist.yaml").write_text("playlist_id: 100\n", encoding="utf-8")
|
||||
(playlist_b / "playlist.yaml").write_text("playlist_id: 200\n", encoding="utf-8")
|
||||
|
||||
bundle_path = create_multi_playlist_bundle(
|
||||
bundle_root=root / "bundles",
|
||||
playlist_dirs=[playlist_a, playlist_b],
|
||||
)
|
||||
|
||||
with zipfile.ZipFile(bundle_path) as zf:
|
||||
names = set(zf.namelist())
|
||||
|
||||
self.assertIn("playlists/Playlist A_100/playlist.yaml", names)
|
||||
self.assertIn("playlists/Playlist B_200/playlist.yaml", names)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_export_bundles.py -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL with `ModuleNotFoundError` or missing function errors for `musicdl.catalogsync.export_bundles`
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from .runtime import sanitize_path_component
|
||||
|
||||
|
||||
def _token() -> str:
|
||||
return str(int(time.time() * 1000))
|
||||
|
||||
|
||||
def build_single_bundle_name(playlist: dict[str, object]) -> str:
|
||||
platform = sanitize_path_component(str(playlist.get("platform") or ""), "playlist")
|
||||
playlist_id = sanitize_path_component(str(playlist.get("id") or ""), "0")
|
||||
name = sanitize_path_component(str(playlist.get("name") or ""), "playlist")
|
||||
return f"playlist-{platform}-{playlist_id}-{name}.zip"
|
||||
|
||||
|
||||
def build_multi_bundle_name() -> str:
|
||||
return "playlists-export-" + time.strftime("%Y%m%d-%H%M%S") + ".zip"
|
||||
|
||||
|
||||
def _zip_tree(zf: zipfile.ZipFile, source_dir: Path, archive_root: str) -> None:
|
||||
for path in sorted(source_dir.rglob("*")):
|
||||
if not path.is_file():
|
||||
continue
|
||||
relative = path.relative_to(source_dir).as_posix()
|
||||
zf.write(path, f"{archive_root}/{relative}")
|
||||
|
||||
|
||||
def create_single_playlist_bundle(*, bundle_root: Path, playlist_dir: Path, playlist: dict[str, object]) -> Path:
|
||||
bundle_root.mkdir(parents=True, exist_ok=True)
|
||||
destination = bundle_root / (_token() + "-" + build_single_bundle_name(playlist))
|
||||
with zipfile.ZipFile(destination, "w", compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
_zip_tree(zf, playlist_dir, playlist_dir.name)
|
||||
return destination
|
||||
|
||||
|
||||
def create_multi_playlist_bundle(*, bundle_root: Path, playlist_dirs: list[Path]) -> Path:
|
||||
bundle_root.mkdir(parents=True, exist_ok=True)
|
||||
destination = bundle_root / (_token() + "-" + build_multi_bundle_name())
|
||||
with zipfile.ZipFile(destination, "w", compression=zipfile.ZIP_DEFLATED) as zf:
|
||||
for playlist_dir in playlist_dirs:
|
||||
_zip_tree(zf, playlist_dir, f"playlists/{playlist_dir.name}")
|
||||
return destination
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_export_bundles.py -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_export_bundles.py musicdl/catalogsync/export_bundles.py
|
||||
git commit -m "feat: add playlist export zip bundle helpers"
|
||||
```
|
||||
|
||||
### Task 2: Add Single-Playlist ZIP Export API
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Test: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
def test_api_playlist_export_zip_downloads_single_playlist_bundle(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.web import create_app
|
||||
import zipfile
|
||||
import io
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
db_path = root / "catalogsync.db"
|
||||
env_path = root / "catalogsync.env"
|
||||
env_path.write_text(f"ROOT_DIR={root.as_posix()}\n", encoding="utf-8")
|
||||
initialize_database(db_path, default_library_root=root / "library").close()
|
||||
|
||||
playlist_id = self._seed_playlist(
|
||||
db_path,
|
||||
platform="qq",
|
||||
pool_kind="manual_file",
|
||||
remote_id="playlist-export-zip",
|
||||
name="Playlist Export Zip",
|
||||
)
|
||||
song_id = self._seed_song(
|
||||
db_path,
|
||||
platform="qq",
|
||||
remote_id="song-export-zip",
|
||||
name="Song Export Zip",
|
||||
)
|
||||
self._link_playlist_song(db_path, playlist_id=playlist_id, song_id=song_id, position=1)
|
||||
self._mark_local_downloaded(
|
||||
db_path,
|
||||
song_id=song_id,
|
||||
relative_path="qq/Singer A/song-export-zip.mp3",
|
||||
)
|
||||
|
||||
app = create_app(db_path=db_path, env_path=env_path)
|
||||
client = TestClient(app)
|
||||
self.addCleanup(client.close)
|
||||
|
||||
response = client.get(f"/api/playlists/{playlist_id}/export.zip")
|
||||
|
||||
self.assertEqual(200, response.status_code)
|
||||
self.assertEqual("application/zip", response.headers["content-type"])
|
||||
archive = zipfile.ZipFile(io.BytesIO(response.content))
|
||||
self.assertIn(f"Playlist Export Zip_{playlist_id}/playlist.yaml", archive.namelist())
|
||||
|
||||
|
||||
def test_api_playlist_export_zip_returns_409_for_unsynced_playlist(self):
|
||||
client, db_path, _ = self._build_client()
|
||||
playlist_id = self._seed_playlist(
|
||||
db_path,
|
||||
platform="qq",
|
||||
pool_kind="manual_file",
|
||||
remote_id="playlist-export-unsynced",
|
||||
name="Playlist Export Unsynced",
|
||||
)
|
||||
|
||||
response = client.get(f"/api/playlists/{playlist_id}/export.zip")
|
||||
|
||||
self.assertEqual(409, response.status_code)
|
||||
payload = response.json()
|
||||
self.assertEqual("unsynced", payload["state_code"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "playlist_export_zip_downloads_single_playlist_bundle or playlist_export_zip_returns_409_for_unsynced_playlist" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL with `404` because `/api/playlists/{playlist_id}/export.zip` does not exist yet
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
from fastapi.responses import FileResponse
|
||||
|
||||
from musicdl.catalogsync.export_bundles import create_single_playlist_bundle
|
||||
|
||||
|
||||
def _playlist_state_or_404(catalog_repo: CatalogRepository, playlist_id: int) -> dict[str, Any]:
|
||||
rows = catalog_repo.list_playlist_export_state_rows([playlist_id])
|
||||
if not rows:
|
||||
raise HTTPException(status_code=404, detail="playlist not found")
|
||||
return dict(rows[0])
|
||||
|
||||
|
||||
@app.get("/api/playlists/{playlist_id}/export.zip")
|
||||
def api_playlist_export_zip(playlist_id: int):
|
||||
state_row = _playlist_state_or_404(catalog_repo, playlist_id)
|
||||
if str(state_row.get("state_code") or "") != "downloaded":
|
||||
raise HTTPException(
|
||||
status_code=409,
|
||||
detail={
|
||||
"state_code": str(state_row.get("state_code") or ""),
|
||||
"playlist_id": playlist_id,
|
||||
"message": "playlist is not ready for immediate export",
|
||||
},
|
||||
)
|
||||
|
||||
env_values = env_manager.load_current()
|
||||
playlists_root = _resolve_playlists_root(env_values, catalog_repo)
|
||||
if playlists_root is None:
|
||||
raise HTTPException(status_code=500, detail="playlists root is not configured")
|
||||
|
||||
service = CatalogSyncService(repository=catalog_repo, playlists_root=playlists_root)
|
||||
playlist_dir = service.ensure_playlist_artifacts_for_playlist(playlist_id)
|
||||
if playlist_dir is None or not playlist_dir.exists():
|
||||
raise HTTPException(status_code=404, detail="playlist export folder not found")
|
||||
|
||||
bundle_root = Path(env_values.get("ROOT_DIR") or playlists_root.parent) / "export-bundles"
|
||||
bundle_path = create_single_playlist_bundle(
|
||||
bundle_root=bundle_root,
|
||||
playlist_dir=playlist_dir,
|
||||
playlist=state_row,
|
||||
)
|
||||
return FileResponse(
|
||||
bundle_path,
|
||||
media_type="application/zip",
|
||||
filename=bundle_path.name.split("-", 1)[1],
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "playlist_export_zip_downloads_single_playlist_bundle or playlist_export_zip_returns_409_for_unsynced_playlist" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_api.py musicdl/catalogsync/ops/web.py
|
||||
git commit -m "feat: add single playlist zip export api"
|
||||
```
|
||||
|
||||
### Task 3: Add Bulk Export ZIP Prepare + Download API
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Test: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests**
|
||||
|
||||
```python
|
||||
def test_api_playlists_export_zip_returns_download_url_when_all_selected_playlists_are_ready(self):
|
||||
from musicdl.catalogsync.ops.web import create_app
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
db_path = root / "catalogsync.db"
|
||||
env_path = root / "catalogsync.env"
|
||||
env_path.write_text(f"ROOT_DIR={root.as_posix()}\n", encoding="utf-8")
|
||||
initialize_database(db_path, default_library_root=root / "library").close()
|
||||
|
||||
playlist_a = self._seed_playlist(db_path, platform="qq", pool_kind="manual_file", remote_id="bulk-a", name="Bulk A")
|
||||
playlist_b = self._seed_playlist(db_path, platform="qq", pool_kind="manual_file", remote_id="bulk-b", name="Bulk B")
|
||||
song_a = self._seed_song(db_path, platform="qq", remote_id="bulk-song-a", name="Bulk Song A")
|
||||
song_b = self._seed_song(db_path, platform="qq", remote_id="bulk-song-b", name="Bulk Song B")
|
||||
self._link_playlist_song(db_path, playlist_id=playlist_a, song_id=song_a, position=1)
|
||||
self._link_playlist_song(db_path, playlist_id=playlist_b, song_id=song_b, position=1)
|
||||
self._mark_local_downloaded(db_path, song_id=song_a, relative_path="qq/Singer A/bulk-song-a.mp3")
|
||||
self._mark_local_downloaded(db_path, song_id=song_b, relative_path="qq/Singer A/bulk-song-b.mp3")
|
||||
|
||||
app = create_app(db_path=db_path, env_path=env_path)
|
||||
client = TestClient(app)
|
||||
self.addCleanup(client.close)
|
||||
|
||||
response = client.post("/api/playlists/export-zip", json={"playlist_ids": [playlist_a, playlist_b]})
|
||||
payload = response.json()
|
||||
download_response = client.get(payload["download_url"])
|
||||
|
||||
self.assertEqual(200, response.status_code)
|
||||
self.assertEqual("ready", payload["status"])
|
||||
self.assertEqual(200, download_response.status_code)
|
||||
self.assertEqual("application/zip", download_response.headers["content-type"])
|
||||
|
||||
|
||||
def test_api_playlists_export_zip_queues_jobs_when_any_selected_playlist_is_not_ready(self):
|
||||
client, db_path, _ = self._build_client()
|
||||
downloaded_playlist = self._seed_playlist(
|
||||
db_path,
|
||||
platform="qq",
|
||||
pool_kind="manual_file",
|
||||
remote_id="bulk-ready",
|
||||
name="Bulk Ready",
|
||||
)
|
||||
song_id = self._seed_song(db_path, platform="qq", remote_id="bulk-ready-song", name="Bulk Ready Song")
|
||||
self._link_playlist_song(db_path, playlist_id=downloaded_playlist, song_id=song_id, position=1)
|
||||
self._mark_local_downloaded(db_path, song_id=song_id, relative_path="qq/Singer A/bulk-ready-song.mp3")
|
||||
|
||||
unsynced_playlist = self._seed_playlist(
|
||||
db_path,
|
||||
platform="netease",
|
||||
pool_kind="playlist_square",
|
||||
remote_id="bulk-unsynced",
|
||||
name="Bulk Unsynced",
|
||||
)
|
||||
|
||||
response = client.post("/api/playlists/export-zip", json={"playlist_ids": [downloaded_playlist, unsynced_playlist]})
|
||||
|
||||
self.assertEqual(200, response.status_code)
|
||||
payload = response.json()
|
||||
self.assertEqual("queued", payload["status"])
|
||||
self.assertEqual([downloaded_playlist], payload["ready_playlist_ids"])
|
||||
self.assertEqual([unsynced_playlist], payload["blocked_playlist_ids"])
|
||||
self.assertIsNotNone(payload["sync_download_job"])
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "playlists_export_zip_returns_download_url_when_all_selected_playlists_are_ready or playlists_export_zip_queues_jobs_when_any_selected_playlist_is_not_ready" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL with `404` because `/api/playlists/export-zip` and `/api/exports/bundles/{token}.zip` do not exist yet
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```python
|
||||
from musicdl.catalogsync.export_bundles import create_multi_playlist_bundle
|
||||
|
||||
|
||||
def _bundle_root(env_values: dict[str, str], playlists_root: Path) -> Path:
|
||||
root_dir = str(env_values.get("ROOT_DIR") or "").strip()
|
||||
if root_dir:
|
||||
path = Path(root_dir).resolve() / "export-bundles"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
path = playlists_root.parent / "export-bundles"
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
return path
|
||||
|
||||
|
||||
@app.post("/api/playlists/export-zip")
|
||||
def api_export_selected_playlists_zip(payload: PlaylistBulkRequest):
|
||||
playlist_ids = _normalize_playlist_ids(payload.playlist_ids)
|
||||
if not playlist_ids:
|
||||
raise HTTPException(status_code=422, detail="playlist_ids is required")
|
||||
|
||||
state_rows = catalog_repo.list_playlist_export_state_rows(playlist_ids)
|
||||
rows_by_id = {int(row["id"]): dict(row) for row in state_rows}
|
||||
|
||||
ready_ids: list[int] = []
|
||||
blocked_ids: list[int] = []
|
||||
sync_download_ids: list[int] = []
|
||||
download_ids: list[int] = []
|
||||
for playlist_id in playlist_ids:
|
||||
row = rows_by_id.get(playlist_id)
|
||||
if row is None:
|
||||
continue
|
||||
state_code = str(row.get("state_code") or "")
|
||||
if state_code == "downloaded":
|
||||
ready_ids.append(playlist_id)
|
||||
else:
|
||||
blocked_ids.append(playlist_id)
|
||||
if state_code == "unsynced":
|
||||
sync_download_ids.append(playlist_id)
|
||||
elif state_code != "downloading":
|
||||
download_ids.append(playlist_id)
|
||||
|
||||
if blocked_ids:
|
||||
return {
|
||||
"status": "queued",
|
||||
"message": f"{len(blocked_ids)} playlists queued for sync/download before export.",
|
||||
"ready_playlist_ids": ready_ids,
|
||||
"blocked_playlist_ids": blocked_ids,
|
||||
"download_job": _create_scoped_playlist_job(repo, env_manager, job_type="download_only", playlist_ids=download_ids, requested_by=payload.requested_by),
|
||||
"sync_download_job": _create_scoped_playlist_job(repo, env_manager, job_type="sync_download", playlist_ids=sync_download_ids, requested_by=payload.requested_by),
|
||||
}
|
||||
|
||||
env_values = env_manager.load_current()
|
||||
playlists_root = _resolve_playlists_root(env_values, catalog_repo)
|
||||
service = CatalogSyncService(repository=catalog_repo, playlists_root=playlists_root)
|
||||
playlist_dirs = [service.ensure_playlist_artifacts_for_playlist(playlist_id) for playlist_id in playlist_ids]
|
||||
valid_dirs = [path for path in playlist_dirs if path is not None and path.exists()]
|
||||
bundle_path = create_multi_playlist_bundle(bundle_root=_bundle_root(env_values, playlists_root), playlist_dirs=valid_dirs)
|
||||
return {
|
||||
"status": "ready",
|
||||
"playlist_ids": playlist_ids,
|
||||
"download_url": f"/api/exports/bundles/{bundle_path.name}",
|
||||
}
|
||||
|
||||
|
||||
@app.get("/api/exports/bundles/{bundle_name}")
|
||||
def api_export_bundle_download(bundle_name: str):
|
||||
env_values = env_manager.load_current()
|
||||
playlists_root = _resolve_playlists_root(env_values, catalog_repo)
|
||||
if playlists_root is None:
|
||||
raise HTTPException(status_code=500, detail="playlists root is not configured")
|
||||
bundle_path = _bundle_root(env_values, playlists_root) / bundle_name
|
||||
if not bundle_path.exists():
|
||||
raise HTTPException(status_code=404, detail="bundle not found")
|
||||
filename = bundle_name.split("-", 1)[1] if "-" in bundle_name else bundle_name
|
||||
return FileResponse(bundle_path, media_type="application/zip", filename=filename)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "playlists_export_zip_returns_download_url_when_all_selected_playlists_are_ready or playlists_export_zip_queues_jobs_when_any_selected_playlist_is_not_ready" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_api.py musicdl/catalogsync/ops/web.py
|
||||
git commit -m "feat: add bulk playlist export zip workflow"
|
||||
```
|
||||
|
||||
### Task 4: Rename Export Buttons and Switch Frontend to Browser Download
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/templates/ops/playlists.html`
|
||||
- Modify: `musicdl/catalogsync/static/ops/app.js`
|
||||
- Test: `tests/catalogsync/test_ops_frontend.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing frontend tests**
|
||||
|
||||
```python
|
||||
def test_playlist_modal_export_button_text_is_export(self):
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
html = (repo_root / "musicdl/catalogsync/templates/ops/playlists.html").read_text(encoding="utf-8")
|
||||
self.assertIn(">Export</button>", html)
|
||||
self.assertNotIn(">Export Folder</button>", html)
|
||||
|
||||
|
||||
def test_export_selected_action_uses_download_url_when_backend_returns_ready(self):
|
||||
repo_root = Path(__file__).resolve().parents[2]
|
||||
script = textwrap.dedent(
|
||||
r'''
|
||||
const fs = require("fs");
|
||||
const path = require("path");
|
||||
const sourcePath = path.join(process.cwd(), "musicdl/catalogsync/static/ops/app.js");
|
||||
const source = fs.readFileSync(sourcePath, "utf8");
|
||||
let navigatedTo = "";
|
||||
|
||||
const exportButton = {
|
||||
handlers: {},
|
||||
disabled: false,
|
||||
getAttribute(name) {
|
||||
if (name === "data-playlist-action") return "export-selected";
|
||||
return null;
|
||||
},
|
||||
addEventListener(type, handler) {
|
||||
this.handlers[type] = handler;
|
||||
},
|
||||
};
|
||||
const checkbox = { checked: true, value: "101", addEventListener() {} };
|
||||
const selectionCount = { textContent: "" };
|
||||
const root = {
|
||||
querySelectorAll(selector) {
|
||||
if (selector === "[data-playlist-checkbox]") return [checkbox];
|
||||
if (selector === "[data-playlist-action]") return [exportButton];
|
||||
return [];
|
||||
},
|
||||
querySelector(selector) {
|
||||
if (selector === "[data-playlist-selection-count]") return selectionCount;
|
||||
return null;
|
||||
},
|
||||
addEventListener() {},
|
||||
};
|
||||
const body = { getAttribute() { return ""; }, appendChild() {}, removeChild() {} };
|
||||
const document = {
|
||||
body,
|
||||
querySelector(selector) {
|
||||
if (selector === "[data-playlists-page]") return root;
|
||||
return null;
|
||||
},
|
||||
querySelectorAll() { return []; },
|
||||
createElement() { return { click() {}, remove() {} }; },
|
||||
};
|
||||
const windowObj = {
|
||||
Number,
|
||||
setTimeout(fn) { fn(); return 1; },
|
||||
clearTimeout() {},
|
||||
alert() {},
|
||||
URL: { createObjectURL() { return "blob:test"; }, revokeObjectURL() {} },
|
||||
Blob: function Blob() {},
|
||||
location: { set href(value) { navigatedTo = value; }, get href() { return navigatedTo; } },
|
||||
fetch(url, options) {
|
||||
if (url !== "/api/playlists/export-zip") throw new Error("unexpected url: " + url);
|
||||
return Promise.resolve({
|
||||
ok: true,
|
||||
json() {
|
||||
return Promise.resolve({ status: "ready", download_url: "/api/exports/bundles/token.zip" });
|
||||
}
|
||||
});
|
||||
},
|
||||
};
|
||||
|
||||
global.window = windowObj;
|
||||
global.document = document;
|
||||
global.setTimeout = windowObj.setTimeout;
|
||||
global.clearTimeout = windowObj.clearTimeout;
|
||||
eval(source);
|
||||
|
||||
exportButton.handlers.click({});
|
||||
Promise.resolve().then(() => {
|
||||
if (navigatedTo !== "/api/exports/bundles/token.zip") {
|
||||
throw new Error("unexpected download target: " + navigatedTo);
|
||||
}
|
||||
process.exit(0);
|
||||
}).catch((error) => {
|
||||
console.error(error && error.stack ? error.stack : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
'''
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test to verify it fails**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_frontend.py -k "playlist_modal_export_button_text_is_export or export_selected_action_uses_download_url_when_backend_returns_ready" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- FAIL because current template still says `Export Folder`
|
||||
- FAIL because current JS still posts to `/api/playlists/export` and only shows status text
|
||||
|
||||
- [ ] **Step 3: Write minimal implementation**
|
||||
|
||||
```javascript
|
||||
var endpointMap = {
|
||||
sync: "/api/playlists/sync",
|
||||
download: "/api/playlists/download",
|
||||
"sync-download": "/api/playlists/sync-download",
|
||||
"export-selected": "/api/playlists/export-zip",
|
||||
"mark-wanted": "/api/playlists/mark-wanted",
|
||||
"unmark-wanted": "/api/playlists/unmark-wanted",
|
||||
};
|
||||
|
||||
function startBrowserDownload(url) {
|
||||
window.location.href = String(url || "");
|
||||
}
|
||||
|
||||
function handleExportSelectedResponse(data) {
|
||||
if (data && data.status === "ready" && data.download_url) {
|
||||
startBrowserDownload(data.download_url);
|
||||
return;
|
||||
}
|
||||
var messages = [];
|
||||
if (data && data.message) {
|
||||
messages.push(String(data.message));
|
||||
}
|
||||
if (data && data.download_job && data.download_job.id) {
|
||||
messages.push("download job #" + String(data.download_job.id));
|
||||
}
|
||||
if (data && data.sync_download_job && data.sync_download_job.id) {
|
||||
messages.push("sync+download job #" + String(data.sync_download_job.id));
|
||||
}
|
||||
showMessage(messages.join("; ") || "export queued", false);
|
||||
window.setTimeout(function () {
|
||||
window.location.reload();
|
||||
}, 1200);
|
||||
}
|
||||
|
||||
function handleSinglePlaylistExportDownload(playlistId) {
|
||||
window.location.href = "/api/playlists/" + encodeURIComponent(playlistId) + "/export.zip";
|
||||
}
|
||||
```
|
||||
|
||||
Template snippet:
|
||||
|
||||
```html
|
||||
<button type="button" class="secondary" data-playlist-export disabled>Export</button>
|
||||
<button type="button" class="secondary" data-playlist-action="export-selected">Export Selected</button>
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_frontend.py -k "playlist_modal_export_button_text_is_export or export_selected_action_uses_download_url_when_backend_returns_ready" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_frontend.py musicdl/catalogsync/templates/ops/playlists.html musicdl/catalogsync/static/ops/app.js
|
||||
git commit -m "feat: switch export ui to browser zip download"
|
||||
```
|
||||
|
||||
### Task 5: Document the New Export Semantics and Run Full Regression
|
||||
|
||||
**Files:**
|
||||
- Modify: `docs/catalogsync.md`
|
||||
|
||||
- [ ] **Step 1: Update docs with the new semantics**
|
||||
|
||||
```markdown
|
||||
## 2026-04-19 Playlist Export To Local ZIP
|
||||
|
||||
- `Export` now means browser download to the user's local machine.
|
||||
- NAS `playlists/` remains the internal artifact cache and bundle source.
|
||||
- `GET /api/playlists/{playlist_id}/export.zip` downloads one playlist ZIP.
|
||||
- `POST /api/playlists/export-zip` prepares bulk export:
|
||||
- returns `status=ready` with `download_url` when every selected playlist is export-ready
|
||||
- returns `status=queued` with background job info when any selected playlist still needs sync/download
|
||||
- `GET /api/exports/bundles/{token}.zip` downloads a prepared bulk ZIP bundle.
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run focused regression suites**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_export_bundles.py -q
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -q
|
||||
python -m pytest tests/catalogsync/test_ops_frontend.py -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- all tests PASS
|
||||
|
||||
- [ ] **Step 3: Run the broader regression that already covers current export + runner behavior**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_services.py -q
|
||||
python -m pytest tests/catalogsync/test_ops_runner.py -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- all tests PASS
|
||||
- no regression to the existing “download stage refreshes playlist artifacts” behavior
|
||||
|
||||
- [ ] **Step 4: Commit**
|
||||
|
||||
```bash
|
||||
git add docs/catalogsync.md
|
||||
git commit -m "docs: describe local zip export workflow"
|
||||
```
|
||||
@@ -0,0 +1,756 @@
|
||||
# Download Dual-Pool Pipeline Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Split the download stage into resolver workers and downloader workers so configured download concurrency is spent on real transfers instead of long source-resolution work.
|
||||
|
||||
**Architecture:** Keep deferred snapshots and current database schema unchanged, refactor `CatalogDownloader` into explicit resolve-only and download-only phases, and add a runner-level in-memory `ready_queue` that connects a small resolver pool to a larger downloader pool during `download` stages.
|
||||
|
||||
**Tech Stack:** Python, sqlite3, unittest, ThreadPoolExecutor, queue.Queue, FastAPI ops dashboard
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
### Existing Files To Modify
|
||||
|
||||
- `musicdl/catalogsync/downloader.py`
|
||||
- Split current mixed `resolve + download` flow into reusable resolve-only and download-only methods.
|
||||
- `musicdl/catalogsync/ops/executors.py`
|
||||
- Add download-stage helpers for resolve-only and download-only item handling without breaking existing item status semantics.
|
||||
- `musicdl/catalogsync/ops/runner.py`
|
||||
- Add the runner-level dual-pool execution path for `download` stages.
|
||||
- `tests/catalogsync/test_services.py`
|
||||
- Lock the downloader API refactor with unit tests.
|
||||
- `tests/catalogsync/test_ops_executors.py`
|
||||
- Lock download-stage executor behavior for resolved tasks and failure handling.
|
||||
- `tests/catalogsync/test_ops_runner.py`
|
||||
- Lock the dual-pool queueing model, worker split, and stage lifecycle.
|
||||
- `tests/catalogsync/test_ops_api.py`
|
||||
- Lock dashboard worker visibility for resolver and downloader worker families if any payload expectations need adjustment.
|
||||
|
||||
### New Files To Create
|
||||
|
||||
- None required for the first implementation. Keep the change focused and schema-free.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Lock `CatalogDownloader` Into Resolve And Download Phases
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing tests for resolve-only and download-only behavior**
|
||||
|
||||
Add tests near the existing downloader tests in `tests/catalogsync/test_services.py` for:
|
||||
|
||||
```python
|
||||
def test_catalog_downloader_resolve_song_row_returns_resolved_payload():
|
||||
payload = downloader.resolve_song_row(
|
||||
row={
|
||||
"id": song_id,
|
||||
"playlist_id": 123,
|
||||
"platform": "netease",
|
||||
"name": "Song Resolve",
|
||||
"singers": "Singer Resolve",
|
||||
"ext": "mp3",
|
||||
"file_size_bytes": 24,
|
||||
"metadata_json": '{"snapshot":{"identifier":"song-resolve"}}',
|
||||
},
|
||||
library_root=library_root,
|
||||
download_sources=["qq", "kuwo"],
|
||||
worker_callback=lambda **state: worker_updates.append(dict(state)),
|
||||
)
|
||||
|
||||
assert payload is not None
|
||||
assert payload["row"]["id"] == song_id
|
||||
assert payload["display_text"] == "Song Resolve / Singer Resolve"
|
||||
assert payload["resolved_song_info"].source == "QQMusicClient"
|
||||
```
|
||||
|
||||
```python
|
||||
def test_catalog_downloader_download_resolved_song_reports_progress_and_records_file():
|
||||
ok = downloader.download_resolved_song(
|
||||
resolved_payload=resolved_payload,
|
||||
worker_callback=lambda **state: worker_updates.append(dict(state)),
|
||||
)
|
||||
|
||||
assert ok is True
|
||||
assert any(int(state.get("downloaded_bytes") or 0) > 0 for state in worker_updates)
|
||||
assert repo.song_has_active_local_file(song_id) is True
|
||||
```
|
||||
|
||||
```python
|
||||
def test_catalog_downloader_download_song_row_remains_a_compatibility_wrapper():
|
||||
ok = downloader.download_song_row(
|
||||
row=row,
|
||||
library_root=library_root,
|
||||
download_sources=["qq"],
|
||||
worker_callback=lambda **state: worker_updates.append(dict(state)),
|
||||
)
|
||||
|
||||
assert ok is True
|
||||
assert any("resolving source" in str(state.get("last_progress_text") or "") for state in worker_updates)
|
||||
assert any("starting download via" in str(state.get("last_progress_text") or "") for state in worker_updates)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused tests and verify they fail for missing APIs**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_services.py -k "resolve_song_row or download_resolved_song or compatibility_wrapper" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAILED`
|
||||
- failure mentions missing `resolve_song_row` or `download_resolved_song`, or existing wrapper behavior not matching the new tests
|
||||
|
||||
- [ ] **Step 3: Implement minimal resolve-only and download-only APIs in `CatalogDownloader`**
|
||||
|
||||
Refactor `musicdl/catalogsync/downloader.py` toward this shape:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ResolvedDownloadPayload:
|
||||
row: dict[str, object]
|
||||
display_text: str
|
||||
default_root: Path
|
||||
target_root: Path
|
||||
backend_id: int
|
||||
expected_bytes: int | None
|
||||
resolved_song_info: object
|
||||
|
||||
|
||||
def resolve_song_row(
|
||||
self,
|
||||
row,
|
||||
library_root: str | Path,
|
||||
download_sources: list[str] | None = None,
|
||||
worker_callback=None,
|
||||
) -> ResolvedDownloadPayload | None:
|
||||
row_dict = dict(row)
|
||||
default_root = Path(library_root).resolve()
|
||||
self._current_library_root = default_root
|
||||
self.repository.ensure_local_backend(default_root, name="default-local", is_default=True)
|
||||
display_name = str(row_dict.get("name") or row_dict.get("id") or "")
|
||||
singers = str(row_dict.get("singers") or "").strip()
|
||||
display_text = f"{display_name} / {singers}".strip(" /")
|
||||
self._emit_worker_progress(row_dict, worker_callback, display_text=display_text)
|
||||
|
||||
metadata = json.loads(row_dict["metadata_json"]) if row_dict.get("metadata_json") else {}
|
||||
song_info = deserialize_song_info(metadata.get("snapshot"))
|
||||
if song_info is None:
|
||||
return None
|
||||
|
||||
resolve_progress_callback = None
|
||||
if worker_callback is not None:
|
||||
resolve_progress_callback = lambda message: self._emit_worker_progress(
|
||||
row_dict,
|
||||
worker_callback,
|
||||
display_text=display_text,
|
||||
last_progress_text=message,
|
||||
)
|
||||
|
||||
song_info = self.resolve_song_info_for_download(
|
||||
row=row_dict,
|
||||
song_info=song_info,
|
||||
download_sources=download_sources,
|
||||
progress_callback=resolve_progress_callback,
|
||||
)
|
||||
download_platform = self._detect_download_platform(song_info, row_dict["platform"])
|
||||
target_root = self.ensure_space(
|
||||
default_root,
|
||||
getattr(song_info, "file_size_bytes", None) or row_dict.get("file_size_bytes"),
|
||||
)
|
||||
is_default_root = target_root.resolve() == default_root
|
||||
backend_id = self.repository.ensure_local_backend(
|
||||
target_root,
|
||||
name="default-local" if is_default_root else None,
|
||||
is_default=is_default_root,
|
||||
)
|
||||
expected_bytes = int(getattr(song_info, "file_size_bytes", None) or row_dict.get("file_size_bytes") or 0) or None
|
||||
return ResolvedDownloadPayload(
|
||||
row=row_dict,
|
||||
display_text=display_text,
|
||||
default_root=default_root,
|
||||
target_root=target_root,
|
||||
backend_id=backend_id,
|
||||
expected_bytes=expected_bytes,
|
||||
resolved_song_info=song_info,
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
def download_resolved_song(
|
||||
self,
|
||||
resolved_payload: ResolvedDownloadPayload,
|
||||
worker_callback=None,
|
||||
) -> bool:
|
||||
row = resolved_payload.row
|
||||
song_info = resolved_payload.resolved_song_info
|
||||
download_platform = self._detect_download_platform(song_info, row["platform"])
|
||||
client = self.get_client(download_platform)
|
||||
singers = self._normalize_singers(getattr(song_info, "singers", None)) or self._normalize_singers(row.get("singers"))
|
||||
relative_dir = build_download_relative_dir(platform=download_platform, singers=singers)
|
||||
target_dir = resolved_payload.target_root / relative_dir
|
||||
target_dir.mkdir(parents=True, exist_ok=True)
|
||||
song_info.work_dir = str(target_dir)
|
||||
if hasattr(song_info, "_save_path"):
|
||||
song_info._save_path = None
|
||||
self._emit_worker_progress(
|
||||
row,
|
||||
worker_callback,
|
||||
display_text=resolved_payload.display_text,
|
||||
last_progress_text=f"starting download via {download_platform}",
|
||||
)
|
||||
# keep existing monitor, client.download, and record_local_file logic here
|
||||
```
|
||||
|
||||
```python
|
||||
def download_song_row(...):
|
||||
resolved_payload = self.resolve_song_row(
|
||||
row=row,
|
||||
library_root=library_root,
|
||||
download_sources=download_sources,
|
||||
worker_callback=worker_callback,
|
||||
)
|
||||
if resolved_payload is None:
|
||||
return False
|
||||
return self.download_resolved_song(
|
||||
resolved_payload=resolved_payload,
|
||||
worker_callback=worker_callback,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the focused tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_services.py -k "resolve_song_row or download_resolved_song or compatibility_wrapper" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
- no regressions in existing progress tests around `resolving source ...` and `starting download via ...`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_services.py musicdl/catalogsync/downloader.py
|
||||
git commit -m "refactor: split downloader resolve and transfer phases"
|
||||
```
|
||||
|
||||
### Task 2: Lock Download Executor Helpers For Resolved Tasks
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_executors.py`
|
||||
- Modify: `musicdl/catalogsync/ops/executors.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing executor tests for resolve-only and download-only item handling**
|
||||
|
||||
Add focused tests in `tests/catalogsync/test_ops_executors.py`:
|
||||
|
||||
```python
|
||||
def test_download_executor_resolve_item_marks_failed_when_resolution_returns_none():
|
||||
with patch.object(CatalogDownloader, "resolve_song_row", return_value=None):
|
||||
executor.process_resolve_item(
|
||||
item_id=item_id,
|
||||
worker_name="resolve-1",
|
||||
ready_queue=ready_queue,
|
||||
)
|
||||
|
||||
item = ops_repo.get_item(item_id)
|
||||
assert item.status == ItemStatus.FAILED
|
||||
assert ready_queue.empty()
|
||||
```
|
||||
|
||||
```python
|
||||
def test_download_executor_resolve_item_enqueues_resolved_payload():
|
||||
resolved_payload = SimpleNamespace(row=row, display_text="Song A / Singer A")
|
||||
with patch.object(CatalogDownloader, "resolve_song_row", return_value=resolved_payload):
|
||||
executor.process_resolve_item(
|
||||
item_id=item_id,
|
||||
worker_name="resolve-1",
|
||||
ready_queue=ready_queue,
|
||||
)
|
||||
|
||||
queued = ready_queue.get_nowait()
|
||||
assert queued.item_id == item_id
|
||||
assert queued.resolved_payload is resolved_payload
|
||||
```
|
||||
|
||||
```python
|
||||
def test_download_executor_download_resolved_item_marks_item_succeeded():
|
||||
with patch.object(CatalogDownloader, "download_resolved_song", return_value=True):
|
||||
executor.process_download_task(
|
||||
task=resolved_task,
|
||||
worker_name="download-1",
|
||||
)
|
||||
|
||||
item = ops_repo.get_item(item_id)
|
||||
assert item.status == ItemStatus.SUCCEEDED
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused executor tests and verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_executors.py -k "process_resolve_item or process_download_task" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAILED`
|
||||
- failure mentions missing `process_resolve_item` or `process_download_task`
|
||||
|
||||
- [ ] **Step 3: Add minimal executor helpers without removing current compatibility path**
|
||||
|
||||
Extend `musicdl/catalogsync/ops/executors.py` around `DownloadStageExecutor` with helpers shaped like:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class ResolvedStageDownloadTask:
|
||||
item_id: int
|
||||
playlist_id: int | None
|
||||
row: dict[str, object]
|
||||
resolved_payload: object
|
||||
|
||||
|
||||
def process_resolve_item(self, item_id: int, worker_name: str, *, ready_queue) -> None:
|
||||
row = self.ops_repo.build_download_row(item_id=item_id)
|
||||
song_id = int(row.get("id") or row.get("song_id") or 0)
|
||||
if song_id > 0 and self.catalog_repo.song_has_active_local_file(song_id):
|
||||
self.ops_repo.update_worker_state(
|
||||
worker_name=worker_name,
|
||||
current_job_item_id=item_id,
|
||||
status="running",
|
||||
current_song_id=song_id,
|
||||
current_playlist_id=row.get("playlist_id"),
|
||||
current_display_text=str(row.get("name") or row.get("id") or song_id),
|
||||
last_progress_text="already downloaded",
|
||||
)
|
||||
_ensure_transition_applied(
|
||||
self.ops_repo.mark_item_succeeded(
|
||||
item_id=item_id,
|
||||
result_payload={"already_downloaded": True},
|
||||
),
|
||||
item_id=item_id,
|
||||
action="mark_item_succeeded",
|
||||
)
|
||||
return
|
||||
resolved_payload = self.downloader.resolve_song_row(
|
||||
row=row,
|
||||
library_root=self.library_root,
|
||||
download_sources=self.download_sources,
|
||||
worker_callback=lambda **state: self.ops_repo.update_worker_state(
|
||||
worker_name=worker_name,
|
||||
current_job_item_id=item_id,
|
||||
status="running",
|
||||
**state,
|
||||
),
|
||||
)
|
||||
if resolved_payload is None:
|
||||
_ensure_transition_applied(
|
||||
self.ops_repo.mark_item_failed(item_id=item_id, error_message="resolve returned no downloadable song"),
|
||||
item_id=item_id,
|
||||
action="mark_item_failed",
|
||||
)
|
||||
return
|
||||
ready_queue.put(
|
||||
ResolvedStageDownloadTask(
|
||||
item_id=item_id,
|
||||
playlist_id=row.get("playlist_id"),
|
||||
row=row,
|
||||
resolved_payload=resolved_payload,
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
```python
|
||||
def process_download_task(self, task: ResolvedStageDownloadTask, worker_name: str) -> None:
|
||||
succeeded = self.downloader.download_resolved_song(
|
||||
resolved_payload=task.resolved_payload,
|
||||
worker_callback=lambda **state: self.ops_repo.update_worker_state(
|
||||
worker_name=worker_name,
|
||||
current_job_item_id=task.item_id,
|
||||
status="running",
|
||||
**state,
|
||||
),
|
||||
)
|
||||
if succeeded:
|
||||
_ensure_transition_applied(
|
||||
self.ops_repo.mark_item_succeeded(item_id=task.item_id),
|
||||
item_id=task.item_id,
|
||||
action="mark_item_succeeded",
|
||||
)
|
||||
return
|
||||
_ensure_transition_applied(
|
||||
self.ops_repo.mark_item_failed(item_id=task.item_id, error_message="download returned no file"),
|
||||
item_id=task.item_id,
|
||||
action="mark_item_failed",
|
||||
)
|
||||
```
|
||||
|
||||
Keep `process_item(...)` as the compatibility path by delegating to `download_song_row(...)`.
|
||||
|
||||
- [ ] **Step 4: Run the focused executor tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_executors.py -k "process_resolve_item or process_download_task" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
- existing `DownloadStageExecutor.process_item(...)` tests remain green
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_executors.py musicdl/catalogsync/ops/executors.py
|
||||
git commit -m "refactor: add staged download executor helpers"
|
||||
```
|
||||
|
||||
### Task 3: Lock The Runner-Level Dual-Pool Pipeline
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_runner.py`
|
||||
- Modify: `musicdl/catalogsync/ops/runner.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing runner tests for worker splitting and queue-driven execution**
|
||||
|
||||
Add tests in `tests/catalogsync/test_ops_runner.py` for:
|
||||
|
||||
```python
|
||||
def test_download_stage_splits_workers_into_resolve_and_download_pools():
|
||||
resolver_workers, download_workers = runner._download_stage_worker_split(total_workers=10)
|
||||
assert resolver_workers == 3
|
||||
assert download_workers == 7
|
||||
```
|
||||
|
||||
```python
|
||||
def test_download_stage_pipeline_processes_items_through_ready_queue():
|
||||
processed = []
|
||||
|
||||
class FakeDownloadExecutor:
|
||||
def process_resolve_item(self, item_id, worker_name, *, ready_queue):
|
||||
ready_queue.put(SimpleNamespace(item_id=item_id, playlist_id=None, resolved_payload=f"payload-{item_id}"))
|
||||
|
||||
def process_download_task(self, task, worker_name):
|
||||
processed.append((task.item_id, worker_name, task.resolved_payload))
|
||||
repo.mark_item_succeeded(task.item_id)
|
||||
|
||||
with patch.object(runner, "_build_executor", return_value=FakeDownloadExecutor()):
|
||||
runner._run_stage(job, stage)
|
||||
|
||||
assert processed
|
||||
assert all(worker_name.startswith("download-") for _, worker_name, _ in processed)
|
||||
```
|
||||
|
||||
```python
|
||||
def test_download_stage_pipeline_uses_single_thread_compatibility_when_worker_count_is_one():
|
||||
calls = []
|
||||
|
||||
class FakeDownloadExecutor:
|
||||
def process_item(self, item_id, worker_name, *, already_claimed=False):
|
||||
calls.append((item_id, worker_name, already_claimed))
|
||||
repo.mark_item_succeeded(item_id)
|
||||
|
||||
with patch.object(runner, "_build_executor", return_value=FakeDownloadExecutor()):
|
||||
runner._run_stage(job, stage)
|
||||
|
||||
assert calls
|
||||
assert calls[0][1] == "download-1"
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused runner tests and verify they fail**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_runner.py -k "worker_split or ready_queue or single_thread_compatibility" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `FAILED`
|
||||
- failure mentions missing `_download_stage_worker_split` or the existing `_run_stage(...)` shape not matching the new behavior
|
||||
|
||||
- [ ] **Step 3: Implement the runner-level dual-pool path for download stages**
|
||||
|
||||
Refactor `musicdl/catalogsync/ops/runner.py` so `download` stages use a specialized path:
|
||||
|
||||
```python
|
||||
def _download_stage_worker_split(self, total_workers: int) -> tuple[int, int]:
|
||||
normalized_total = max(int(total_workers or 0), 1)
|
||||
if normalized_total == 1:
|
||||
return 1, 0
|
||||
if normalized_total == 2:
|
||||
return 1, 1
|
||||
resolver_workers = max(1, min(3, normalized_total // 3))
|
||||
download_workers = max(1, normalized_total - resolver_workers)
|
||||
return resolver_workers, download_workers
|
||||
```
|
||||
|
||||
```python
|
||||
def _run_download_stage_pipeline(self, job, stage, executor, worker_count: int) -> None:
|
||||
resolver_workers, download_workers = self._download_stage_worker_split(worker_count)
|
||||
if download_workers == 0:
|
||||
self._run_stage_with_single_pool(job, stage, executor, worker_count)
|
||||
return
|
||||
|
||||
ready_queue: Queue = Queue(maxsize=max(1, download_workers * 2))
|
||||
stop_event = threading.Event()
|
||||
sentinel = object()
|
||||
|
||||
def resolver_loop(worker_index: int) -> None:
|
||||
worker_name = f"resolve-{worker_index + 1}"
|
||||
while not stop_event.is_set():
|
||||
active_job = self.repository.get_job(job.id)
|
||||
if active_job is None or active_job.status in {JobStatus.PAUSE_REQUESTED, JobStatus.CANCELED}:
|
||||
stop_event.set()
|
||||
return
|
||||
item = self.repository.claim_next_stage_item(stage.id, worker_name)
|
||||
if item is None:
|
||||
return
|
||||
executor.process_resolve_item(item.id, worker_name, ready_queue=ready_queue)
|
||||
|
||||
def download_loop(worker_index: int) -> None:
|
||||
worker_name = f"download-{worker_index + 1}"
|
||||
while True:
|
||||
task = ready_queue.get()
|
||||
if task is sentinel:
|
||||
return
|
||||
executor.process_download_task(task, worker_name)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=resolver_workers + download_workers) as pool:
|
||||
resolver_futures = [pool.submit(resolver_loop, index) for index in range(resolver_workers)]
|
||||
download_futures = [pool.submit(download_loop, index) for index in range(download_workers)]
|
||||
for future in resolver_futures:
|
||||
future.result()
|
||||
for _ in range(download_workers):
|
||||
ready_queue.put(sentinel)
|
||||
for future in download_futures:
|
||||
future.result()
|
||||
```
|
||||
|
||||
Then update `_run_stage(...)` to branch:
|
||||
|
||||
```python
|
||||
if refreshed_stage.stage_type == StageType.DOWNLOAD.value:
|
||||
self._run_download_stage_pipeline(job, refreshed_stage, executor, worker_count)
|
||||
else:
|
||||
self._run_stage_with_single_pool(job, refreshed_stage, executor, worker_count)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the focused runner tests and verify they pass**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_runner.py -k "worker_split or ready_queue or single_thread_compatibility" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
- current worker-count tests still pass
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_runner.py musicdl/catalogsync/ops/runner.py
|
||||
git commit -m "feat: run download stage through dual-pool pipeline"
|
||||
```
|
||||
|
||||
### Task 4: Lock Dashboard And Worker-State Expectations
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Modify: `musicdl/catalogsync/ops/repository.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing dashboard regression for resolver and downloader worker families**
|
||||
|
||||
Add a test in `tests/catalogsync/test_ops_api.py` shaped like:
|
||||
|
||||
```python
|
||||
def test_dashboard_exposes_resolver_and_downloader_workers_during_download_stage():
|
||||
repo.update_worker_state(
|
||||
worker_name="resolve-1",
|
||||
current_job_item_id=item_id_a,
|
||||
status="running",
|
||||
current_song_id=song_a_id,
|
||||
current_display_text="Song A / Singer A",
|
||||
last_progress_text="resolving source qq (1/6)",
|
||||
)
|
||||
repo.update_worker_state(
|
||||
worker_name="download-1",
|
||||
current_job_item_id=item_id_b,
|
||||
status="running",
|
||||
current_song_id=song_b_id,
|
||||
current_display_text="Song B / Singer B",
|
||||
last_progress_text="12.00MB/48.00MB",
|
||||
downloaded_bytes=12 * 1024 * 1024,
|
||||
total_bytes=48 * 1024 * 1024,
|
||||
speed_bytes_per_sec=3 * 1024 * 1024,
|
||||
progress_percent=25,
|
||||
)
|
||||
|
||||
payload = client.get("/api/dashboard?include_task_rows=false").json()
|
||||
|
||||
worker_names = [worker["worker_name"] for worker in payload["workers"]]
|
||||
assert "resolve-1" in worker_names
|
||||
assert "download-1" in worker_names
|
||||
assert payload["transfer_stats"]["download_speed_bytes_per_sec"] == 3 * 1024 * 1024
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused dashboard test and verify it fails only if payload logic needs adjustment**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "resolver_and_downloader_workers" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- either `FAIL` because payload assumptions need adjustment
|
||||
- or `PASS`, which means no production code change is needed here
|
||||
|
||||
- [ ] **Step 3: Apply the minimal payload or repository changes only if the test requires them**
|
||||
|
||||
If worker lookup or naming assumptions need tightening, keep the code minimal, for example:
|
||||
|
||||
```python
|
||||
# no-op if existing worker queries already behave correctly
|
||||
# only adjust helper logic if it filters by a fixed "download-" prefix anywhere
|
||||
```
|
||||
|
||||
The target is not to redesign dashboard data, only to ensure resolver workers remain visible and transfer stats still reflect downloader workers only.
|
||||
|
||||
- [ ] **Step 4: Re-run the focused dashboard test and verify it passes**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_api.py -k "resolver_and_downloader_workers" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_ops_api.py musicdl/catalogsync/ops/web.py musicdl/catalogsync/ops/repository.py
|
||||
git commit -m "test: cover resolver and downloader worker visibility"
|
||||
```
|
||||
|
||||
### Task 5: Final Verification And NAS Reality Check
|
||||
|
||||
**Files:**
|
||||
- Modify: `tests/catalogsync/test_services.py`
|
||||
- Modify: `tests/catalogsync/test_ops_executors.py`
|
||||
- Modify: `tests/catalogsync/test_ops_runner.py`
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `musicdl/catalogsync/ops/executors.py`
|
||||
- Modify: `musicdl/catalogsync/ops/runner.py`
|
||||
|
||||
- [ ] **Step 1: Run the full targeted regression slice**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_services.py tests/catalogsync/test_ops_executors.py tests/catalogsync/test_ops_runner.py tests/catalogsync/test_ops_api.py -k "download or transfer_stats or resolver" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
- existing download progress tests still green
|
||||
- new dual-pool runner tests green
|
||||
|
||||
- [ ] **Step 2: Run a slightly wider ops regression slice**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
python -m pytest tests/catalogsync/test_ops_repository.py tests/catalogsync/test_ops_frontend.py -k "worker or transfer or dashboard" -q
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- `PASS`
|
||||
- no unexpected worker-state regressions
|
||||
|
||||
- [ ] **Step 3: Deploy to NAS**
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
powershell -ExecutionPolicy Bypass -File .\deploy-catalogsync.ps1
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- deploy completes successfully
|
||||
- health check passes for `http://127.0.0.1:18080/dashboard`
|
||||
|
||||
- [ ] **Step 4: Verify production behavior on NAS**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
$env:NAS_192168543_PASSWORD='Nie@159357'
|
||||
powershell -ExecutionPolicy Bypass -File 'C:\Users\Administrator\.codex\skills\nas-ssh-192168543\scripts\run.ps1' "curl -fsS http://127.0.0.1:18080/api/dashboard?include_task_rows=false | python3 -m json.tool | head -n 160"
|
||||
```
|
||||
|
||||
Expected:
|
||||
|
||||
- worker list includes both `resolve-*` and `download-*`
|
||||
- at least some `download-*` workers show non-zero transfer stats simultaneously under active load
|
||||
- `resolve-*` workers show `resolving source ...` text instead of pretending to be downloading
|
||||
|
||||
- [ ] **Step 5: Commit final integration changes**
|
||||
|
||||
```bash
|
||||
git add tests/catalogsync/test_services.py tests/catalogsync/test_ops_executors.py tests/catalogsync/test_ops_runner.py tests/catalogsync/test_ops_api.py musicdl/catalogsync/downloader.py musicdl/catalogsync/ops/executors.py musicdl/catalogsync/ops/runner.py musicdl/catalogsync/ops/web.py musicdl/catalogsync/ops/repository.py
|
||||
git commit -m "feat: split download stage into resolver and transfer pools"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Self-Review
|
||||
|
||||
### Spec Coverage
|
||||
|
||||
- dual-pool architecture: covered by Task 3
|
||||
- downloader API split: covered by Task 1
|
||||
- executor support for resolved tasks: covered by Task 2
|
||||
- worker-family visibility: covered by Task 4
|
||||
- NAS verification of real concurrency: covered by Task 5
|
||||
|
||||
No spec gaps remain for this iteration.
|
||||
|
||||
### Placeholder Scan
|
||||
|
||||
- no `TODO`, `TBD`, or “implement later” placeholders remain
|
||||
- each code-changing task includes concrete method names, commands, and code shapes
|
||||
|
||||
### Type Consistency
|
||||
|
||||
- `ResolvedDownloadPayload` is used consistently between Task 1 and Task 2
|
||||
- `ResolvedStageDownloadTask` is used consistently between Task 2 and Task 3
|
||||
- runner dual-pool entry point is consistently named `_run_download_stage_pipeline(...)`
|
||||
|
||||
@@ -0,0 +1,631 @@
|
||||
# Resolver Source Ranking Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Add a persistent resolver side database that learns fallback success rates by original source and reorders fallback sources after warmup without touching the main catalog business tables.
|
||||
|
||||
**Architecture:** Create a dedicated `resolver_stats.db` side database and repository, then wire `MultiSourceSongResolver` to ask that repository for ranked fallback order. Keep preferred-source resolution first, record only fallback attempts and successes, and continue trying later sources if the learned top two fail.
|
||||
|
||||
**Tech Stack:** Python, sqlite3, unittest, Click CLI, FastAPI ops web
|
||||
|
||||
---
|
||||
|
||||
## File Map
|
||||
|
||||
- Create: `musicdl/catalogsync/resolver_stats.py`
|
||||
Dedicated resolver side-database bootstrap, default path helper, and ranking repository.
|
||||
- Modify: `musicdl/catalogsync/resolver.py`
|
||||
Preferred-source-first resolver flow plus ranked fallback traversal and resilient stats recording.
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
Construct `MultiSourceSongResolver` with a `ResolverStatsRepository` derived from the main database path.
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
Initialize the resolver side database during CLI app startup and `init-db`.
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
Initialize the resolver side database during web app startup.
|
||||
- Create: `tests/catalogsync/test_resolver_stats.py`
|
||||
Unit tests for side-database schema, warmup, ranking, and grouping.
|
||||
- Modify: `tests/catalogsync/test_resolver.py`
|
||||
Integration-style resolver tests for warmup behavior, ranked top-two traversal, continuation, and graceful fallback.
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
CLI startup tests for side-database creation.
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
Web startup test for side-database creation.
|
||||
|
||||
### Task 1: Add The Resolver Stats Side Database
|
||||
|
||||
**Files:**
|
||||
- Create: `musicdl/catalogsync/resolver_stats.py`
|
||||
- Create: `tests/catalogsync/test_resolver_stats.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing side-database tests**
|
||||
|
||||
```python
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
class ResolverStatsRepositoryTests(unittest.TestCase):
|
||||
def test_initialize_resolver_stats_database_creates_stats_table(self):
|
||||
from musicdl.catalogsync.resolver_stats import initialize_resolver_stats_database
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "resolver_stats.db"
|
||||
conn = initialize_resolver_stats_database(db_path)
|
||||
try:
|
||||
table_names = {
|
||||
row["name"]
|
||||
for row in conn.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type = 'table'"
|
||||
).fetchall()
|
||||
}
|
||||
finally:
|
||||
conn.close()
|
||||
|
||||
self.assertIn("resolver_source_stats", table_names)
|
||||
|
||||
def test_rank_fallback_sources_keeps_config_order_before_warmup(self):
|
||||
from musicdl.catalogsync.resolver_stats import ResolverStatsRepository
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
repo = ResolverStatsRepository(Path(tmpdir) / "resolver_stats.db")
|
||||
repo.record_fallback_result("qq", "kuwo", succeeded=True)
|
||||
ranked = repo.rank_fallback_sources(
|
||||
"qq",
|
||||
["kuwo", "migu", "qianqian"],
|
||||
warmup_attempts=1000,
|
||||
)
|
||||
|
||||
self.assertEqual(["kuwo", "migu", "qianqian"], ranked)
|
||||
|
||||
def test_rank_fallback_sources_reorders_after_warmup_per_origin_source(self):
|
||||
from musicdl.catalogsync.resolver_stats import ResolverStatsRepository
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
repo = ResolverStatsRepository(Path(tmpdir) / "resolver_stats.db")
|
||||
for _ in range(800):
|
||||
repo.record_fallback_result("qq", "migu", succeeded=True)
|
||||
for _ in range(200):
|
||||
repo.record_fallback_result("qq", "kuwo", succeeded=False)
|
||||
ranked = repo.rank_fallback_sources(
|
||||
"qq",
|
||||
["kuwo", "migu", "qianqian"],
|
||||
warmup_attempts=1000,
|
||||
)
|
||||
|
||||
self.assertEqual(["migu", "kuwo", "qianqian"], ranked)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused side-database tests and verify they fail**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_resolver_stats.py -q`
|
||||
|
||||
Expected: `ModuleNotFoundError` or missing symbol failures for `musicdl.catalogsync.resolver_stats`.
|
||||
|
||||
- [ ] **Step 3: Write the minimal side-database implementation**
|
||||
|
||||
```python
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from contextlib import suppress
|
||||
from pathlib import Path
|
||||
|
||||
SQLITE_BUSY_TIMEOUT_MS = 30000
|
||||
RESOLVER_FALLBACK_WARMUP_ATTEMPTS = 1000
|
||||
|
||||
SCHEMA_STATEMENTS = [
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS resolver_source_stats (
|
||||
origin_source TEXT NOT NULL,
|
||||
candidate_source TEXT NOT NULL,
|
||||
attempt_count INTEGER NOT NULL DEFAULT 0,
|
||||
resolve_success_count INTEGER NOT NULL DEFAULT 0,
|
||||
last_attempt_at TEXT,
|
||||
last_success_at TEXT,
|
||||
created_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
updated_at TEXT DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY(origin_source, candidate_source)
|
||||
)
|
||||
""",
|
||||
"""
|
||||
CREATE INDEX IF NOT EXISTS idx_resolver_source_stats_origin
|
||||
ON resolver_source_stats (origin_source)
|
||||
""",
|
||||
]
|
||||
|
||||
|
||||
def default_resolver_stats_db_path(db_path: str | Path) -> Path:
|
||||
return Path(db_path).resolve().with_name("resolver_stats.db")
|
||||
|
||||
|
||||
def connect_resolver_stats_database(db_path: str | Path) -> sqlite3.Connection:
|
||||
path = Path(db_path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
conn = sqlite3.connect(path, timeout=SQLITE_BUSY_TIMEOUT_MS / 1000)
|
||||
conn.row_factory = sqlite3.Row
|
||||
conn.execute(f"PRAGMA busy_timeout = {SQLITE_BUSY_TIMEOUT_MS}")
|
||||
with suppress(sqlite3.OperationalError):
|
||||
conn.execute("PRAGMA journal_mode = WAL")
|
||||
with suppress(sqlite3.OperationalError):
|
||||
conn.execute("PRAGMA synchronous = NORMAL")
|
||||
return conn
|
||||
|
||||
|
||||
def initialize_resolver_stats_database(db_path: str | Path) -> sqlite3.Connection:
|
||||
conn = connect_resolver_stats_database(db_path)
|
||||
for statement in SCHEMA_STATEMENTS:
|
||||
conn.execute(statement)
|
||||
conn.commit()
|
||||
return conn
|
||||
|
||||
|
||||
class ResolverStatsRepository:
|
||||
def __init__(self, db_path: str | Path):
|
||||
self.db_path = Path(db_path)
|
||||
initialize_resolver_stats_database(self.db_path).close()
|
||||
|
||||
def record_fallback_result(self, origin_source: str, candidate_source: str, *, succeeded: bool) -> None:
|
||||
with connect_resolver_stats_database(self.db_path) as conn:
|
||||
conn.execute(
|
||||
"""
|
||||
INSERT INTO resolver_source_stats (
|
||||
origin_source, candidate_source, attempt_count, resolve_success_count,
|
||||
last_attempt_at, last_success_at
|
||||
)
|
||||
VALUES (?, ?, 1, ?, CURRENT_TIMESTAMP, CASE WHEN ? THEN CURRENT_TIMESTAMP ELSE NULL END)
|
||||
ON CONFLICT(origin_source, candidate_source) DO UPDATE SET
|
||||
attempt_count = attempt_count + 1,
|
||||
resolve_success_count = resolve_success_count + excluded.resolve_success_count,
|
||||
last_attempt_at = CURRENT_TIMESTAMP,
|
||||
last_success_at = CASE
|
||||
WHEN excluded.resolve_success_count > 0 THEN CURRENT_TIMESTAMP
|
||||
ELSE resolver_source_stats.last_success_at
|
||||
END,
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
""",
|
||||
(origin_source, candidate_source, 1 if succeeded else 0, 1 if succeeded else 0),
|
||||
)
|
||||
|
||||
def rank_fallback_sources(
|
||||
self,
|
||||
origin_source: str,
|
||||
fallback_sources: list[str],
|
||||
*,
|
||||
warmup_attempts: int = RESOLVER_FALLBACK_WARMUP_ATTEMPTS,
|
||||
) -> list[str]:
|
||||
ordered = list(fallback_sources)
|
||||
if not ordered:
|
||||
return []
|
||||
with connect_resolver_stats_database(self.db_path) as conn:
|
||||
rows = conn.execute(
|
||||
"""
|
||||
SELECT candidate_source, attempt_count, resolve_success_count
|
||||
FROM resolver_source_stats
|
||||
WHERE origin_source = ?
|
||||
""",
|
||||
(origin_source,),
|
||||
).fetchall()
|
||||
total_attempts = sum(int(row["attempt_count"]) for row in rows)
|
||||
if total_attempts < warmup_attempts:
|
||||
return ordered
|
||||
stats = {
|
||||
str(row["candidate_source"]): (
|
||||
(int(row["resolve_success_count"]) + 1) / (int(row["attempt_count"]) + 2)
|
||||
)
|
||||
for row in rows
|
||||
}
|
||||
return sorted(ordered, key=lambda source: (-stats.get(source, 0.5), ordered.index(source)))
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the side-database tests and verify they pass**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_resolver_stats.py -q`
|
||||
|
||||
Expected: `3 passed`
|
||||
|
||||
- [ ] **Step 5: Commit the side-database foundation**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/resolver_stats.py tests/catalogsync/test_resolver_stats.py
|
||||
git commit -m "feat: add resolver stats side database"
|
||||
```
|
||||
|
||||
### Task 2: Teach The Resolver To Use Ranked Fallback Sources
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/resolver.py`
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `tests/catalogsync/test_resolver.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing resolver behavior tests**
|
||||
|
||||
```python
|
||||
def test_resolver_uses_ranked_top_two_fallback_sources_after_warmup(self):
|
||||
from musicdl.catalogsync.resolver import MultiSourceSongResolver
|
||||
from musicdl.modules.utils.data import SongInfo
|
||||
|
||||
class FakeStatsRepo:
|
||||
def rank_fallback_sources(self, origin_source, fallback_sources, warmup_attempts=1000):
|
||||
self.rank_call = (origin_source, list(fallback_sources), warmup_attempts)
|
||||
return ["migu", "kuwo", "qianqian"]
|
||||
|
||||
def record_fallback_result(self, origin_source, candidate_source, *, succeeded):
|
||||
self.records.append((origin_source, candidate_source, succeeded))
|
||||
|
||||
def __init__(self):
|
||||
self.records = []
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, source, result=None, calls=None):
|
||||
self.source = source
|
||||
self.result = list(result or [])
|
||||
self.calls = calls
|
||||
|
||||
def search(self, keyword, num_threadings=1, request_overrides=None, rule=None, main_process_context=None):
|
||||
self.calls.append(self.source)
|
||||
return list(self.result)
|
||||
|
||||
snapshot = SongInfo(
|
||||
source="QQMusicClient",
|
||||
identifier="song-1",
|
||||
song_name="Song 1",
|
||||
singers="Singer 1",
|
||||
raw_data={"search": {"id": "song-1"}},
|
||||
download_url=None,
|
||||
download_url_status={},
|
||||
)
|
||||
migu_hit = SongInfo(
|
||||
source="MiguMusicClient",
|
||||
identifier="migu-song-1",
|
||||
song_name="Song 1",
|
||||
singers="Singer 1",
|
||||
ext="mp3",
|
||||
download_url="https://example.com/song-1.mp3",
|
||||
download_url_status={"ok": True},
|
||||
)
|
||||
search_calls = []
|
||||
stats_repo = FakeStatsRepo()
|
||||
resolver = MultiSourceSongResolver(
|
||||
client_factory=lambda platform: {
|
||||
"qq": FakeClient("qq", [], search_calls),
|
||||
"kuwo": FakeClient("kuwo", [], search_calls),
|
||||
"migu": FakeClient("migu", [migu_hit], search_calls),
|
||||
"qianqian": FakeClient("qianqian", [], search_calls),
|
||||
}[platform],
|
||||
resolver_stats_repo=stats_repo,
|
||||
)
|
||||
|
||||
resolved = resolver.resolve_song_info(
|
||||
row={"platform": "qq", "name": "Song 1", "singers": "Singer 1", "remote_song_id": "song-1"},
|
||||
snapshot_song_info=snapshot,
|
||||
download_sources=["qq", "kuwo", "migu", "qianqian"],
|
||||
)
|
||||
|
||||
self.assertEqual(["qq", "migu"], search_calls)
|
||||
self.assertEqual(
|
||||
[("qq", "migu", True)],
|
||||
stats_repo.records,
|
||||
)
|
||||
self.assertEqual("MiguMusicClient", resolved.source)
|
||||
|
||||
def test_resolver_continues_after_ranked_top_two_fail(self):
|
||||
from musicdl.catalogsync.resolver import MultiSourceSongResolver
|
||||
from musicdl.modules.utils.data import SongInfo
|
||||
|
||||
class FakeStatsRepo:
|
||||
def rank_fallback_sources(self, origin_source, fallback_sources, warmup_attempts=1000):
|
||||
return ["migu", "kuwo", "qianqian"]
|
||||
|
||||
def record_fallback_result(self, origin_source, candidate_source, *, succeeded):
|
||||
self.records.append((candidate_source, succeeded))
|
||||
|
||||
def __init__(self):
|
||||
self.records = []
|
||||
|
||||
class FakeClient:
|
||||
def __init__(self, source, result, calls):
|
||||
self.source = source
|
||||
self.result = list(result)
|
||||
self.calls = calls
|
||||
|
||||
def search(self, keyword, num_threadings=1, request_overrides=None, rule=None, main_process_context=None):
|
||||
self.calls.append(self.source)
|
||||
return list(self.result)
|
||||
|
||||
snapshot = SongInfo(
|
||||
source="QQMusicClient",
|
||||
identifier="song-2",
|
||||
song_name="Song 2",
|
||||
singers="Singer 2",
|
||||
raw_data={"search": {"id": "song-2"}},
|
||||
download_url=None,
|
||||
download_url_status={},
|
||||
)
|
||||
qianqian_hit = SongInfo(
|
||||
source="QianqianMusicClient",
|
||||
identifier="qianqian-song-2",
|
||||
song_name="Song 2",
|
||||
singers="Singer 2",
|
||||
ext="mp3",
|
||||
download_url="https://example.com/song-2.mp3",
|
||||
download_url_status={"ok": True},
|
||||
)
|
||||
calls = []
|
||||
stats_repo = FakeStatsRepo()
|
||||
resolver = MultiSourceSongResolver(
|
||||
client_factory=lambda platform: {
|
||||
"qq": FakeClient("qq", [], calls),
|
||||
"migu": FakeClient("migu", [], calls),
|
||||
"kuwo": FakeClient("kuwo", [], calls),
|
||||
"qianqian": FakeClient("qianqian", [qianqian_hit], calls),
|
||||
}[platform],
|
||||
resolver_stats_repo=stats_repo,
|
||||
)
|
||||
|
||||
resolved = resolver.resolve_song_info(
|
||||
row={"platform": "qq", "name": "Song 2", "singers": "Singer 2", "remote_song_id": "song-2"},
|
||||
snapshot_song_info=snapshot,
|
||||
download_sources=["qq", "kuwo", "migu", "qianqian"],
|
||||
)
|
||||
|
||||
self.assertEqual(["qq", "migu", "kuwo", "qianqian"], calls)
|
||||
self.assertEqual("QianqianMusicClient", resolved.source)
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the focused resolver tests and verify they fail**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_resolver.py -q`
|
||||
|
||||
Expected: failures for unexpected source order and missing `resolver_stats_repo` support.
|
||||
|
||||
- [ ] **Step 3: Write the minimal resolver and downloader integration**
|
||||
|
||||
```python
|
||||
class MultiSourceSongResolver:
|
||||
def __init__(
|
||||
self,
|
||||
client_factory,
|
||||
request_overrides_factory=None,
|
||||
resolver_stats_repo=None,
|
||||
warmup_attempts: int = RESOLVER_FALLBACK_WARMUP_ATTEMPTS,
|
||||
):
|
||||
self.client_factory = client_factory
|
||||
self.request_overrides_factory = request_overrides_factory or (lambda timeout: {"timeout": timeout})
|
||||
self.resolver_stats_repo = resolver_stats_repo
|
||||
self.warmup_attempts = int(warmup_attempts)
|
||||
|
||||
def _rank_fallback_sources(self, origin_source: str, fallback_sources: list[str]) -> list[str]:
|
||||
if self.resolver_stats_repo is None:
|
||||
return list(fallback_sources)
|
||||
try:
|
||||
return self.resolver_stats_repo.rank_fallback_sources(
|
||||
origin_source,
|
||||
list(fallback_sources),
|
||||
warmup_attempts=self.warmup_attempts,
|
||||
)
|
||||
except Exception:
|
||||
return list(fallback_sources)
|
||||
|
||||
def _record_fallback_result(self, origin_source: str, candidate_source: str, *, succeeded: bool) -> None:
|
||||
if self.resolver_stats_repo is None:
|
||||
return
|
||||
try:
|
||||
self.resolver_stats_repo.record_fallback_result(
|
||||
origin_source,
|
||||
candidate_source,
|
||||
succeeded=succeeded,
|
||||
)
|
||||
except Exception:
|
||||
return
|
||||
|
||||
def resolve_song_info(self, row, snapshot_song_info, download_sources=None, progress_callback=None):
|
||||
target_song_info = self._build_target_song_info(row=row, snapshot_song_info=snapshot_song_info)
|
||||
preferred_source = normalize_source_name(getattr(target_song_info, "source", None) or row.get("platform"))
|
||||
ordered_sources = dedupe_preserve_order(list(download_sources or DEFAULT_DOWNLOAD_SOURCES))
|
||||
fallback_sources = [source for source in ordered_sources if source != preferred_source]
|
||||
ranked_fallback_sources = self._rank_fallback_sources(preferred_source, fallback_sources)
|
||||
|
||||
candidate_rows = []
|
||||
if preferred_source in ordered_sources:
|
||||
self._emit_progress(progress_callback, f"resolving source {preferred_source} (1/{len(ordered_sources)})")
|
||||
client = self.client_factory(preferred_source)
|
||||
refreshed_song = self._refresh_song_info(client, target_song_info)
|
||||
if self._has_valid_download_url(refreshed_song):
|
||||
merged_refreshed = merge_resolved_song_info(target_song_info, refreshed_song)
|
||||
refreshed_match_priority = song_info_match_priority(merged_refreshed, target_song_info)
|
||||
candidate_rows.append((merged_refreshed, refreshed_match_priority, 0))
|
||||
if is_high_confidence_match(refreshed_match_priority):
|
||||
return merged_refreshed
|
||||
search_candidates = self._search_source_candidates(preferred_source, build_resolve_keyword(target_song_info, row))
|
||||
best_candidate = self._pick_best_candidate(search_candidates, target_song_info, 0)
|
||||
if best_candidate is not None:
|
||||
merged_candidate = merge_resolved_song_info(target_song_info, best_candidate)
|
||||
match_priority = song_info_match_priority(merged_candidate, target_song_info)
|
||||
candidate_rows.append((merged_candidate, match_priority, 0))
|
||||
if is_high_confidence_match(match_priority):
|
||||
return merged_candidate
|
||||
|
||||
for offset, source in enumerate(ranked_fallback_sources, start=2):
|
||||
self._emit_progress(progress_callback, f"resolving source {source} ({offset}/{len(ordered_sources)})")
|
||||
search_candidates = self._search_source_candidates(source, build_resolve_keyword(target_song_info, row))
|
||||
best_candidate = self._pick_best_candidate(search_candidates, target_song_info, offset)
|
||||
succeeded = best_candidate is not None
|
||||
self._record_fallback_result(preferred_source, source, succeeded=succeeded)
|
||||
if not succeeded:
|
||||
continue
|
||||
return merge_resolved_song_info(target_song_info, best_candidate)
|
||||
|
||||
if candidate_rows:
|
||||
candidate_rows.sort(key=lambda item: (match_priority_group(item[1]), search_result_quality_group(item[0]), -candidate_file_size_bytes(item[0]), item[2], item[1]))
|
||||
return candidate_rows[0][0]
|
||||
return target_song_info
|
||||
|
||||
|
||||
class CatalogDownloader:
|
||||
def __init__(self, repository, work_dir="musicdl_outputs/catalogsync", worker_count=DEFAULT_DOWNLOAD_WORKERS):
|
||||
self.repository = repository
|
||||
resolver_stats_repo = ResolverStatsRepository(default_resolver_stats_db_path(self.repository.db_path))
|
||||
self._resolver = MultiSourceSongResolver(
|
||||
client_factory=lambda platform: self.get_client(platform),
|
||||
request_overrides_factory=lambda timeout: self._request_overrides(timeout),
|
||||
resolver_stats_repo=resolver_stats_repo,
|
||||
)
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the resolver-focused tests and verify they pass**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_resolver.py tests/catalogsync/test_resolver_stats.py -q`
|
||||
|
||||
Expected: all resolver and resolver-stats tests pass.
|
||||
|
||||
- [ ] **Step 5: Commit the ranked resolver behavior**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/resolver.py musicdl/catalogsync/downloader.py tests/catalogsync/test_resolver.py tests/catalogsync/test_resolver_stats.py
|
||||
git commit -m "feat: rank resolver fallback sources by origin"
|
||||
```
|
||||
|
||||
### Task 3: Initialize The Side Database At Startup Boundaries
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Write the failing startup initialization tests**
|
||||
|
||||
```python
|
||||
def test_init_db_command_creates_resolver_stats_side_db(self):
|
||||
from musicdl.catalogsync.cli import cli
|
||||
|
||||
runner = CliRunner()
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
db_path = Path(tmpdir) / "catalogsync.db"
|
||||
side_db_path = Path(tmpdir) / "resolver_stats.db"
|
||||
library_root = Path(tmpdir) / "library"
|
||||
|
||||
result = runner.invoke(
|
||||
cli,
|
||||
["init-db", "--db", str(db_path), "--library-root", str(library_root)],
|
||||
)
|
||||
|
||||
self.assertEqual(0, result.exit_code, msg=result.output)
|
||||
self.assertTrue(side_db_path.exists())
|
||||
|
||||
def test_create_app_initializes_resolver_stats_side_db(self):
|
||||
from musicdl.catalogsync.db import initialize_database
|
||||
from musicdl.catalogsync.ops.web import create_app
|
||||
|
||||
with tempfile.TemporaryDirectory(ignore_cleanup_errors=True) as tmpdir:
|
||||
root = Path(tmpdir)
|
||||
db_path = root / "catalogsync.db"
|
||||
env_path = root / "catalogsync.env"
|
||||
side_db_path = root / "resolver_stats.db"
|
||||
env_path.write_text("ROOT_DIR=/music\nDOWNLOAD_SOURCES=qq\n", encoding="utf-8")
|
||||
initialize_database(db_path).close()
|
||||
|
||||
app = create_app(db_path=db_path, env_path=env_path)
|
||||
|
||||
self.assertIsNotNone(app)
|
||||
self.assertTrue(side_db_path.exists())
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run the startup tests and verify they fail**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_cli.py -k "resolver_stats_side_db" tests/catalogsync/test_ops_api.py -k "resolver_stats_side_db" -q`
|
||||
|
||||
Expected: assertions fail because `resolver_stats.db` is not created yet.
|
||||
|
||||
- [ ] **Step 3: Wire startup initialization to the side database**
|
||||
|
||||
```python
|
||||
from .resolver_stats import default_resolver_stats_db_path, initialize_resolver_stats_database
|
||||
|
||||
|
||||
class CatalogSyncApplication:
|
||||
def __init__(self, db_path: str, library_root: str | None = None):
|
||||
self.db_path = db_path
|
||||
self.library_root = library_root
|
||||
initialize_database(db_path, default_library_root=library_root).close()
|
||||
initialize_resolver_stats_database(default_resolver_stats_db_path(db_path)).close()
|
||||
self.repository = CatalogRepository(db_path)
|
||||
self.service = CatalogSyncService(self.repository)
|
||||
self.downloader = CatalogDownloader(self.repository)
|
||||
|
||||
def init_db(self):
|
||||
initialize_database(self.db_path, default_library_root=self.library_root).close()
|
||||
initialize_resolver_stats_database(default_resolver_stats_db_path(self.db_path)).close()
|
||||
|
||||
|
||||
def create_app(db_path: str | Path, env_path: str | Path, *, start_runner: bool = False, runner_sleep_seconds: float = 1.0) -> FastAPI:
|
||||
db_file = Path(db_path)
|
||||
initialize_database(db_file).close()
|
||||
initialize_resolver_stats_database(default_resolver_stats_db_path(db_file)).close()
|
||||
...
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run the startup tests and verify they pass**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_cli.py -k "resolver_stats_side_db" tests/catalogsync/test_ops_api.py -k "resolver_stats_side_db" -q`
|
||||
|
||||
Expected: `2 passed`
|
||||
|
||||
- [ ] **Step 5: Commit the startup wiring**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/cli.py musicdl/catalogsync/ops/web.py tests/catalogsync/test_cli.py tests/catalogsync/test_ops_api.py
|
||||
git commit -m "feat: initialize resolver stats database on startup"
|
||||
```
|
||||
|
||||
### Task 4: Run Full Verification And NAS Validation
|
||||
|
||||
**Files:**
|
||||
- Modify: `musicdl/catalogsync/resolver.py`
|
||||
- Modify: `musicdl/catalogsync/downloader.py`
|
||||
- Modify: `musicdl/catalogsync/cli.py`
|
||||
- Modify: `musicdl/catalogsync/ops/web.py`
|
||||
- Modify: `musicdl/catalogsync/resolver_stats.py`
|
||||
- Modify: `tests/catalogsync/test_resolver.py`
|
||||
- Modify: `tests/catalogsync/test_resolver_stats.py`
|
||||
- Modify: `tests/catalogsync/test_cli.py`
|
||||
- Modify: `tests/catalogsync/test_ops_api.py`
|
||||
|
||||
- [ ] **Step 1: Run the full local regression slice**
|
||||
|
||||
Run: `python -m pytest tests/catalogsync/test_resolver_stats.py tests/catalogsync/test_resolver.py tests/catalogsync/test_cli.py tests/catalogsync/test_services.py tests/catalogsync/test_ops_executors.py tests/catalogsync/test_ops_runner.py tests/catalogsync/test_ops_api.py tests/catalogsync/test_runtime.py -q`
|
||||
|
||||
Expected: all tests pass, with only the existing known warning if it still appears.
|
||||
|
||||
- [ ] **Step 2: Deploy to NAS**
|
||||
|
||||
Run: `powershell -ExecutionPolicy Bypass -File .\deploy-catalogsync.ps1`
|
||||
|
||||
Expected: deploy completes successfully, health check passes, and single-instance check passes.
|
||||
|
||||
- [ ] **Step 3: Sample NAS dashboard and confirm dual-download bursts still appear**
|
||||
|
||||
Run:
|
||||
|
||||
```powershell
|
||||
powershell -ExecutionPolicy Bypass -File 'C:\Users\Administrator\.codex\skills\nas-ssh-192168543\scripts\run.ps1' "python3 - <<'PY'
|
||||
import json, urllib.request
|
||||
with urllib.request.urlopen('http://127.0.0.1:18080/api/dashboard?include_task_rows=false', timeout=10) as resp:
|
||||
data = json.load(resp)
|
||||
print(json.dumps({
|
||||
'downloaded_songs': data['download_stats']['downloaded_songs'],
|
||||
'speed_bps': data['transfer_stats']['download_speed_bytes_per_sec'],
|
||||
'workers': [w['worker_name'] for w in data['workers'] if w.get('status') == 'running'],
|
||||
}, ensure_ascii=False))
|
||||
PY"
|
||||
```
|
||||
|
||||
Expected: running workers still include `download-1` and `download-2` during active bursts, and `downloaded_songs` continues increasing.
|
||||
|
||||
- [ ] **Step 4: Commit the verified end-to-end implementation**
|
||||
|
||||
```bash
|
||||
git add musicdl/catalogsync/resolver_stats.py musicdl/catalogsync/resolver.py musicdl/catalogsync/downloader.py musicdl/catalogsync/cli.py musicdl/catalogsync/ops/web.py tests/catalogsync/test_resolver_stats.py tests/catalogsync/test_resolver.py tests/catalogsync/test_cli.py tests/catalogsync/test_ops_api.py
|
||||
git commit -m "feat: persist resolver fallback source rankings"
|
||||
```
|
||||
Reference in New Issue
Block a user