530 lines
14 KiB
Markdown
530 lines
14 KiB
Markdown
# Catalogsync Task Center And Download Lanes Design
|
|
|
|
## Goal
|
|
|
|
Rework the current operations console so the NAS web UI behaves like a real task center:
|
|
|
|
- `Dashboard` becomes the primary task control page
|
|
- task list and task detail are merged into one page with row expansion instead of forced page jumps
|
|
- all jobs that contain a `download` stage are serialized into one download lane
|
|
- collect and sync jobs can still run without being blocked by the download lane
|
|
- operators can run `sync_only` against selected playlists to fix `song_count = 0` or incomplete playlists
|
|
- download jobs surface real-time throughput, including per-task aggregate speed and per-worker song speed
|
|
|
|
This design extends the existing operations-console design rather than replacing the underlying SQLite-backed execution model.
|
|
|
|
## Confirmed Decisions
|
|
|
|
The following points were confirmed during design review:
|
|
|
|
- `Dashboard` should become the main task center instead of relying on `/jobs/{id}` as the normal interaction path
|
|
- the preferred layout is a single task table with inline expansion for details
|
|
- all job types that include a `download` stage are treated as download-class jobs:
|
|
- `catalog_sync`
|
|
- `sync_download`
|
|
- `download_only`
|
|
- `download_upload`
|
|
- download-class jobs may be created freely, but only one may run at a time
|
|
- non-download jobs such as `collect_only` and `sync_only` are not restricted by the single-download-job rule
|
|
- playlists with `song_count = 0` do not get a new dedicated status
|
|
- the playlists page must add `sync selected playlists`, implemented as `sync_only + playlist_scope`
|
|
- the task center should use compact icon-style controls:
|
|
- one toggle control for pause and resume
|
|
- one `X`-style control for cancel
|
|
- the task center should show download speed if a real value can be captured from the download pipeline
|
|
|
|
## Scope
|
|
|
|
### In Scope
|
|
|
|
- redesign the `Dashboard` page into a task center
|
|
- reduce the importance of `/jobs` and `/jobs/{id}` while keeping them available as fallback routes
|
|
- add lane-aware scheduling rules so only one download-class job runs at a time
|
|
- keep collect and sync jobs runnable without the global single-job bottleneck
|
|
- add playlist-bulk sync from the playlists page
|
|
- add structured task summary data for dashboard rows
|
|
- add real-time download throughput display for download workers and download tasks
|
|
- preserve the existing pause, resume, cancel, retry, and recovery model where possible
|
|
|
|
### Out Of Scope
|
|
|
|
- changing the core collect, sync, download, and upload business logic for provider behavior
|
|
- redefining playlist status taxonomy beyond the current states
|
|
- removing `/jobs/{id}` completely
|
|
- redesigning the UI as a separate SPA
|
|
- cross-machine or distributed workers
|
|
- changing upload scheduling policy in this iteration beyond fitting into the task-center UI
|
|
|
|
## User Experience Design
|
|
|
|
## Dashboard As Task Center
|
|
|
|
`/dashboard` becomes the single primary operator page.
|
|
|
|
The page should be reorganized into three layers:
|
|
|
|
1. Summary cards
|
|
2. Quick actions
|
|
3. Main task table
|
|
|
|
### Summary Cards
|
|
|
|
The top row should remain compact and operator-focused:
|
|
|
|
- total jobs
|
|
- running jobs
|
|
- queued download jobs
|
|
- paused jobs
|
|
- failed or completed-with-errors jobs
|
|
- running download songs
|
|
|
|
These cards are status indicators, not the main interaction surface.
|
|
|
|
### Quick Actions
|
|
|
|
Keep quick-launch actions, but make them secondary to the task table:
|
|
|
|
- full pipeline
|
|
- collect only
|
|
- sync only
|
|
- download only
|
|
- upload only
|
|
|
|
The existing manual job-creation form may remain, but it should be visually reduced so the page reads as a task center first.
|
|
|
|
### Main Task Table
|
|
|
|
Replace the current split between `Active Job`, `Recent Jobs`, and the hard jump into `/jobs/{id}` with one central task table.
|
|
|
|
Recommended columns:
|
|
|
|
- `ID`
|
|
- `Task`
|
|
- `Status`
|
|
- `Scope`
|
|
- `Primary Progress`
|
|
- `Active Workers`
|
|
- `Lane`
|
|
- `Actions`
|
|
|
|
Each row should support inline expansion.
|
|
|
|
### Inline Expanded Task Details
|
|
|
|
Expanding a row reveals the most useful parts of the current task detail page:
|
|
|
|
- stage summary
|
|
- playlist progress
|
|
- running items
|
|
- recent commands
|
|
- recent errors
|
|
|
|
The goal is that normal pause, resume, cancel, and progress inspection no longer require navigation away from `Dashboard`.
|
|
|
|
## Route Roles
|
|
|
|
### `/dashboard`
|
|
|
|
Primary operator view and daily control surface.
|
|
|
|
### `/jobs`
|
|
|
|
Fallback archive-like list for all jobs. It can be simplified because the main operator path is now `Dashboard`.
|
|
|
|
### `/jobs/{id}`
|
|
|
|
Keep as a deep-link route for troubleshooting, bookmarks, and future direct links from logs or notifications. It should no longer be the primary interaction path.
|
|
|
|
## Playlist Page Behavior
|
|
|
|
The playlists page should keep its current filtering and progress features and add one new bulk action:
|
|
|
|
- `sync selected playlists`
|
|
|
|
This action creates a `sync_only` job with `playlist_scope.playlist_ids`.
|
|
|
|
It must:
|
|
|
|
- sync the selected playlists
|
|
- update playlist-song links
|
|
- update `song_count`
|
|
|
|
It must not:
|
|
|
|
- download song files
|
|
- implicitly turn into `sync_download`
|
|
|
|
### `song_count = 0` Handling
|
|
|
|
Playlists with zero songs remain in the existing status model.
|
|
|
|
No extra dedicated state is introduced for this iteration.
|
|
|
|
To help operators understand what to do, the row may show a light hint such as:
|
|
|
|
- `0 songs, sync recommended`
|
|
|
|
But the filter model stays unchanged.
|
|
|
|
## Task Summary Model
|
|
|
|
The dashboard task table needs a job-summary projection that is richer than the current recent-jobs payload.
|
|
|
|
Each task row should expose:
|
|
|
|
- `id`
|
|
- `job_type`
|
|
- `display_name`
|
|
- `status`
|
|
- `scope_summary`
|
|
- `lane_type`
|
|
- `queue_position`
|
|
- `primary_progress_text`
|
|
- `primary_progress_percent`
|
|
- `active_worker_count`
|
|
- `can_pause`
|
|
- `can_resume`
|
|
- `can_cancel`
|
|
- `expanded_detail_payload`
|
|
|
|
### Display Name
|
|
|
|
Map internal job types to friendlier operator labels. Examples:
|
|
|
|
- `catalog_sync` -> `Full Pipeline`
|
|
- `collect_only` -> `Collect`
|
|
- `sync_only` with playlist scope -> `Sync Selected Playlists`
|
|
- `download_only` with playlist scope -> `Download Selected Playlists`
|
|
- `sync_download` with playlist scope -> `Sync Then Download`
|
|
|
|
### Scope Summary
|
|
|
|
Examples:
|
|
|
|
- `All sources`
|
|
- `12 playlists`
|
|
- `3 sources`
|
|
|
|
### Primary Progress
|
|
|
|
The progress shown in the main task row depends on task type:
|
|
|
|
- collect jobs:
|
|
- collected sources or collected pools summary
|
|
- sync jobs:
|
|
- synced playlists / target playlists
|
|
- download jobs:
|
|
- downloaded songs / target download songs
|
|
- upload jobs:
|
|
- uploaded files / target uploads
|
|
|
|
## Scheduling Design
|
|
|
|
## Current Limitation
|
|
|
|
The current runner effectively behaves like a global single-active-job scheduler.
|
|
|
|
That is insufficient for the new requirement because it would still block pure collect or sync jobs behind a long-running download-class job.
|
|
|
|
## Lane Model
|
|
|
|
Introduce two scheduler lanes:
|
|
|
|
- `download`
|
|
- `general`
|
|
|
|
### Download Lane
|
|
|
|
Contains any job whose stage sequence includes `download`.
|
|
|
|
This includes:
|
|
|
|
- `catalog_sync`
|
|
- `sync_download`
|
|
- `download_only`
|
|
- `download_upload`
|
|
|
|
Policy:
|
|
|
|
- only one download-lane job may be running at a time
|
|
- additional download-lane jobs remain queued in lane order
|
|
|
|
### General Lane
|
|
|
|
Contains jobs without a `download` stage.
|
|
|
|
This includes:
|
|
|
|
- `collect_only`
|
|
- `sync_only`
|
|
|
|
Policy:
|
|
|
|
- these jobs are not blocked by the single-download-job rule
|
|
- multiple general-lane jobs may run concurrently
|
|
|
|
### Recommended Default Concurrency
|
|
|
|
Assume:
|
|
|
|
- `DOWNLOAD_LANE_CONCURRENCY = 1`
|
|
- `GENERAL_LANE_CONCURRENCY = 3`
|
|
|
|
`GENERAL_LANE_CONCURRENCY` should be configurable later through env or runner settings, but default `3` is acceptable for the first implementation.
|
|
|
|
## Lane Assignment Rule
|
|
|
|
Lane assignment should be derived from the job stage sequence, not from separate operator flags.
|
|
|
|
This avoids drift between UI intent and scheduler behavior.
|
|
|
|
If a job contains `download` in `JOB_STAGE_SEQUENCES`, it belongs to the `download` lane.
|
|
|
|
## Queue Position
|
|
|
|
For download-lane jobs, the dashboard should expose:
|
|
|
|
- `running`
|
|
- `queued #1`
|
|
- `queued #2`
|
|
|
|
For general-lane jobs, queue display can be simpler:
|
|
|
|
- `general`
|
|
- `running`
|
|
- `queued`
|
|
|
|
The exact wording can stay simple as long as the operator can tell which jobs are blocked by the single-download rule.
|
|
|
|
## Runner Refactor Strategy
|
|
|
|
Do not rewrite stage executors.
|
|
|
|
Instead, refactor the scheduler layer so:
|
|
|
|
- lane eligibility is computed when choosing runnable jobs
|
|
- the runner can hold one active download-lane job
|
|
- the runner can hold multiple active general-lane jobs
|
|
- each running job continues to use the existing stage and worker machinery
|
|
|
|
This keeps risk concentrated in the orchestration layer rather than in provider-specific logic.
|
|
|
|
## Download Speed Design
|
|
|
|
## Requirement
|
|
|
|
The task center should display real download throughput rather than parsing console text heuristically.
|
|
|
|
### Why Text Parsing Is Rejected
|
|
|
|
Many providers already emit `MB/s` in rich terminal progress, but that output is not a stable API:
|
|
|
|
- formats differ by provider
|
|
- text may change without notice
|
|
- not all clients expose identical progress lines
|
|
|
|
Therefore the design must use structured progress reporting inside the download pipeline.
|
|
|
|
## Structured Throughput Model
|
|
|
|
During download-stage execution, each download worker should publish structured progress fields such as:
|
|
|
|
- `downloaded_bytes`
|
|
- `total_bytes`
|
|
- `speed_bytes_per_sec`
|
|
- `progress_percent`
|
|
|
|
These values should update the worker state and be aggregatable per task.
|
|
|
|
### Task-Level Speed
|
|
|
|
The dashboard row for a download-class task should show total live throughput across active download workers.
|
|
|
|
Example:
|
|
|
|
- `62 / 300 songs | 18.4 MB/s`
|
|
|
|
### Worker-Level Speed
|
|
|
|
The expanded worker section for a running download task should show, per worker:
|
|
|
|
- current song
|
|
- current speed
|
|
- downloaded bytes / total bytes when known
|
|
|
|
Example:
|
|
|
|
- `download-2 | Moonlight | 6.2 MB/s | 21.4 / 41.0 MB`
|
|
|
|
### Fallback Behavior
|
|
|
|
If structured speed is not available for a particular worker or provider:
|
|
|
|
- show `-`
|
|
- do not synthesize or guess a value from text logs
|
|
|
|
## API Changes
|
|
|
|
## Dashboard Payload
|
|
|
|
`GET /api/dashboard` must evolve from a light summary into a task-center payload.
|
|
|
|
It should return:
|
|
|
|
- summary cards
|
|
- quick-launch defaults
|
|
- task-center rows
|
|
- row detail summaries for tasks expanded by the UI
|
|
|
|
Each task row should include the task summary fields described above.
|
|
|
|
## New Playlist Sync Endpoint
|
|
|
|
Add:
|
|
|
|
- `POST /api/playlists/sync`
|
|
|
|
Behavior:
|
|
|
|
- validate `playlist_ids`
|
|
- create `sync_only`
|
|
- write `playlist_scope.playlist_ids`
|
|
- return created job summary
|
|
|
|
Existing playlist bulk endpoints remain:
|
|
|
|
- `mark-wanted`
|
|
- `unmark-wanted`
|
|
- `download`
|
|
- `sync-download`
|
|
|
|
## Job Detail Endpoint
|
|
|
|
`GET /api/jobs/{id}` remains the source of full detail.
|
|
|
|
The dashboard inline expansion may either:
|
|
|
|
- reuse the existing detail payload directly
|
|
- or consume a trimmed detail projection
|
|
|
|
The implementation may start by reusing the current payload for safety.
|
|
|
|
## Data Model Extensions
|
|
|
|
Prefer extending existing operations tables rather than introducing a second job schema.
|
|
|
|
### Job Summary Computation
|
|
|
|
The repository layer should compute dashboard-friendly projections rather than forcing templates to derive them ad hoc.
|
|
|
|
### Worker Progress Extension
|
|
|
|
`job_workers` state must be able to carry structured download progress.
|
|
|
|
This may be done by:
|
|
|
|
- adding typed columns
|
|
- or adding a compact JSON progress payload
|
|
|
|
Recommended preference:
|
|
|
|
- keep existing visible scalar columns
|
|
- add a small JSON payload if multiple dynamic throughput fields are needed
|
|
|
|
The implementation plan can choose the exact storage form.
|
|
|
|
## UI Controls
|
|
|
|
The dashboard task table should use compact icon-first controls:
|
|
|
|
- pause icon when the job is pausable
|
|
- resume icon when the job is resumable
|
|
- cancel `X` icon when the job is cancelable
|
|
- expand toggle for inline details
|
|
|
|
Low-frequency actions such as `retry item` remain inside expanded detail sections or the fallback detail page.
|
|
|
|
## Testing Strategy
|
|
|
|
## Scheduler Tests
|
|
|
|
Add tests that prove:
|
|
|
|
- only one download-lane job runs at a time
|
|
- a second download-lane job remains queued
|
|
- a general-lane `sync_only` job can run while a download-lane job is active
|
|
- `catalog_sync` is correctly classified into the download lane
|
|
|
|
## API Tests
|
|
|
|
Add tests for:
|
|
|
|
- `POST /api/playlists/sync`
|
|
- dashboard payload includes lane and task-summary fields
|
|
- dashboard renders compact action controls
|
|
- inline task detail data is available
|
|
|
|
## UI Rendering Tests
|
|
|
|
At minimum verify:
|
|
|
|
- dashboard contains the main task table
|
|
- dashboard no longer depends on a jump to detail as the primary control path
|
|
- playlists page contains `sync selected playlists`
|
|
- download tasks render speed fields and real values when available
|
|
|
|
## Regression Tests
|
|
|
|
Protect:
|
|
|
|
- existing pause, resume, cancel command flow
|
|
- `wanted_only=` empty-query compatibility
|
|
- playlist progress rendering
|
|
- task playlist progress rendering
|
|
|
|
## Rollout Plan
|
|
|
|
Recommended rollout order:
|
|
|
|
1. add dashboard-oriented task summary repository helpers
|
|
2. add lane-aware scheduling rules
|
|
3. add playlist bulk sync endpoint and button
|
|
4. redesign dashboard into the primary task center
|
|
5. add structured download throughput reporting
|
|
6. redeploy to NAS and verify live behavior
|
|
|
|
This order keeps correctness and scheduling changes ahead of cosmetic UI work.
|
|
|
|
## Risks
|
|
|
|
Primary technical risk:
|
|
|
|
- refactoring the runner from a single-global-job loop into a lane-aware multi-job scheduler
|
|
|
|
Risk reduction:
|
|
|
|
- keep stage executors intact
|
|
- concentrate changes in job selection and orchestration
|
|
- verify lane rules with focused tests before refining UI
|
|
|
|
Secondary risk:
|
|
|
|
- structured speed reporting may require touching downloader integration points across multiple providers
|
|
|
|
Risk reduction:
|
|
|
|
- start with download-stage worker instrumentation in `catalogsync`
|
|
- expose speed only when a real structured value is available
|
|
- degrade gracefully to `-` rather than inventing numbers
|
|
|
|
## Success Criteria
|
|
|
|
The design is successful when:
|
|
|
|
- operators can manage tasks primarily from `Dashboard`
|
|
- normal control flow does not require bouncing into `/jobs/{id}`
|
|
- multiple download-class jobs can be queued while only one runs
|
|
- collect and sync jobs are no longer unnecessarily blocked behind downloads
|
|
- selected playlists can be synced directly from the playlists page
|
|
- running download tasks show meaningful live throughput in the task center
|