musicdl-catalog-sync-suite/catalog-sync/docs/superpowers/specs/2026-04-17-task-center-download-lanes-design.md

Catalogsync Task Center And Download Lanes Design

Goal

Rework the current operations console so the NAS web UI behaves like a real task center:

• Dashboard becomes the primary task control page
• task list and task detail are merged into one page with row expansion instead of forced page jumps
• all jobs that contain a download stage are serialized into one download lane
• collect and sync jobs can still run without being blocked by the download lane
• operators can run sync_only against selected playlists to fix song_count = 0 or incomplete playlists
• download jobs surface real-time throughput, including per-task aggregate speed and per-worker song speed

This design extends the existing operations-console design rather than replacing the underlying SQLite-backed execution model.

Confirmed Decisions

The following points were confirmed during design review:

• Dashboard should become the main task center instead of relying on /jobs/{id} as the normal interaction path
• the preferred layout is a single task table with inline expansion for details
• all job types that include a download stage are treated as download-class jobs:
  • catalog_sync
  • sync_download
  • download_only
  • download_upload
• download-class jobs may be created freely, but only one may run at a time
• non-download jobs such as collect_only and sync_only are not restricted by the single-download-job rule
• playlists with song_count = 0 do not get a new dedicated status
• the playlists page must add sync selected playlists, implemented as sync_only + playlist_scope
• the task center should use compact icon-style controls:
  • one toggle control for pause and resume
  • one X-style control for cancel
• the task center should show download speed if a real value can be captured from the download pipeline

Scope

In Scope

• redesign the Dashboard page into a task center
• reduce the importance of /jobs and /jobs/{id} while keeping them available as fallback routes
• add lane-aware scheduling rules so only one download-class job runs at a time
• keep collect and sync jobs runnable without the global single-job bottleneck
• add playlist-bulk sync from the playlists page
• add structured task summary data for dashboard rows
• add real-time download throughput display for download workers and download tasks
• preserve the existing pause, resume, cancel, retry, and recovery model where possible

Out Of Scope

• changing the core collect, sync, download, and upload business logic for provider behavior
• redefining playlist status taxonomy beyond the current states
• removing /jobs/{id} completely
• redesigning the UI as a separate SPA
• cross-machine or distributed workers
• changing upload scheduling policy in this iteration beyond fitting into the task-center UI

User Experience Design

Dashboard As Task Center

/dashboard becomes the single primary operator page.

The page should be reorganized into three layers:

1. Summary cards
2. Quick actions
3. Main task table

Summary Cards

The top row should remain compact and operator-focused:

• total jobs
• running jobs
• queued download jobs
• paused jobs
• failed or completed-with-errors jobs
• running download songs

These cards are status indicators, not the main interaction surface.

Quick Actions

Keep quick-launch actions, but make them secondary to the task table:

• full pipeline
• collect only
• sync only
• download only
• upload only

The existing manual job-creation form may remain, but it should be visually reduced so the page reads as a task center first.

Main Task Table

Replace the current split between Active Job, Recent Jobs, and the hard jump into /jobs/{id} with one central task table.

Recommended columns:

• ID
• Task
• Status
• Scope
• Primary Progress
• Active Workers
• Lane
• Actions

Each row should support inline expansion.

Inline Expanded Task Details

Expanding a row reveals the most useful parts of the current task detail page:

• stage summary
• playlist progress
• running items
• recent commands
• recent errors

The goal is that normal pause, resume, cancel, and progress inspection no longer require navigation away from Dashboard.

Route Roles

/dashboard

Primary operator view and daily control surface.

/jobs

Fallback archive-like list for all jobs. It can be simplified because the main operator path is now Dashboard.

/jobs/{id}

Keep as a deep-link route for troubleshooting, bookmarks, and future direct links from logs or notifications. It should no longer be the primary interaction path.

Playlist Page Behavior

The playlists page should keep its current filtering and progress features and add one new bulk action:

• sync selected playlists

This action creates a sync_only job with playlist_scope.playlist_ids.

It must:

• sync the selected playlists
• update playlist-song links
• update song_count

It must not:

• download song files
• implicitly turn into sync_download

song_count = 0 Handling

Playlists with zero songs remain in the existing status model.

No extra dedicated state is introduced for this iteration.

To help operators understand what to do, the row may show a light hint such as:

• 0 songs, sync recommended

But the filter model stays unchanged.

Task Summary Model

The dashboard task table needs a job-summary projection that is richer than the current recent-jobs payload.

Each task row should expose:

• id
• job_type
• display_name
• status
• scope_summary
• lane_type
• queue_position
• primary_progress_text
• primary_progress_percent
• active_worker_count
• can_pause
• can_resume
• can_cancel
• expanded_detail_payload

Display Name

Map internal job types to friendlier operator labels. Examples:

• catalog_sync -> Full Pipeline
• collect_only -> Collect
• sync_only with playlist scope -> Sync Selected Playlists
• download_only with playlist scope -> Download Selected Playlists
• sync_download with playlist scope -> Sync Then Download

Scope Summary

Examples:

• All sources
• 12 playlists
• 3 sources

Primary Progress

The progress shown in the main task row depends on task type:

• collect jobs:
  • collected sources or collected pools summary
• sync jobs:
  • synced playlists / target playlists
• download jobs:
  • downloaded songs / target download songs
• upload jobs:
  • uploaded files / target uploads

Scheduling Design

Current Limitation

The current runner effectively behaves like a global single-active-job scheduler.

That is insufficient for the new requirement because it would still block pure collect or sync jobs behind a long-running download-class job.

Lane Model

Introduce two scheduler lanes:

• download
• general

Download Lane

Contains any job whose stage sequence includes download.

This includes:

• catalog_sync
• sync_download
• download_only
• download_upload

Policy:

• only one download-lane job may be running at a time
• additional download-lane jobs remain queued in lane order

General Lane

Contains jobs without a download stage.

This includes:

• collect_only
• sync_only

Policy:

• these jobs are not blocked by the single-download-job rule
• multiple general-lane jobs may run concurrently

Recommended Default Concurrency

Assume:

• DOWNLOAD_LANE_CONCURRENCY = 1
• GENERAL_LANE_CONCURRENCY = 3

GENERAL_LANE_CONCURRENCY should be configurable later through env or runner settings, but default 3 is acceptable for the first implementation.

Lane Assignment Rule

Lane assignment should be derived from the job stage sequence, not from separate operator flags.

This avoids drift between UI intent and scheduler behavior.

If a job contains download in JOB_STAGE_SEQUENCES, it belongs to the download lane.

Queue Position

For download-lane jobs, the dashboard should expose:

• running
• queued #1
• queued #2

For general-lane jobs, queue display can be simpler:

• general
• running
• queued

The exact wording can stay simple as long as the operator can tell which jobs are blocked by the single-download rule.

Runner Refactor Strategy

Do not rewrite stage executors.

Instead, refactor the scheduler layer so:

• lane eligibility is computed when choosing runnable jobs
• the runner can hold one active download-lane job
• the runner can hold multiple active general-lane jobs
• each running job continues to use the existing stage and worker machinery

This keeps risk concentrated in the orchestration layer rather than in provider-specific logic.

Download Speed Design

Requirement

The task center should display real download throughput rather than parsing console text heuristically.

Why Text Parsing Is Rejected

Many providers already emit MB/s in rich terminal progress, but that output is not a stable API:

• formats differ by provider
• text may change without notice
• not all clients expose identical progress lines

Therefore the design must use structured progress reporting inside the download pipeline.

Structured Throughput Model

During download-stage execution, each download worker should publish structured progress fields such as:

• downloaded_bytes
• total_bytes
• speed_bytes_per_sec
• progress_percent

These values should update the worker state and be aggregatable per task.

Task-Level Speed

The dashboard row for a download-class task should show total live throughput across active download workers.

Example:

• 62 / 300 songs | 18.4 MB/s

Worker-Level Speed

The expanded worker section for a running download task should show, per worker:

• current song
• current speed
• downloaded bytes / total bytes when known

Example:

• download-2 | Moonlight | 6.2 MB/s | 21.4 / 41.0 MB

Fallback Behavior

If structured speed is not available for a particular worker or provider:

• show -
• do not synthesize or guess a value from text logs

API Changes

Dashboard Payload

GET /api/dashboard must evolve from a light summary into a task-center payload.

It should return:

• summary cards
• quick-launch defaults
• task-center rows
• row detail summaries for tasks expanded by the UI

Each task row should include the task summary fields described above.

New Playlist Sync Endpoint

Add:

• POST /api/playlists/sync

Behavior:

• validate playlist_ids
• create sync_only
• write playlist_scope.playlist_ids
• return created job summary

Existing playlist bulk endpoints remain:

• mark-wanted
• unmark-wanted
• download
• sync-download

Job Detail Endpoint

GET /api/jobs/{id} remains the source of full detail.

The dashboard inline expansion may either:

• reuse the existing detail payload directly
• or consume a trimmed detail projection

The implementation may start by reusing the current payload for safety.

Data Model Extensions

Prefer extending existing operations tables rather than introducing a second job schema.

Job Summary Computation

The repository layer should compute dashboard-friendly projections rather than forcing templates to derive them ad hoc.

Worker Progress Extension

job_workers state must be able to carry structured download progress.

This may be done by:

• adding typed columns
• or adding a compact JSON progress payload

Recommended preference:

• keep existing visible scalar columns
• add a small JSON payload if multiple dynamic throughput fields are needed

The implementation plan can choose the exact storage form.

UI Controls

The dashboard task table should use compact icon-first controls:

• pause icon when the job is pausable
• resume icon when the job is resumable
• cancel X icon when the job is cancelable
• expand toggle for inline details

Low-frequency actions such as retry item remain inside expanded detail sections or the fallback detail page.

Testing Strategy

Scheduler Tests

Add tests that prove:

• only one download-lane job runs at a time
• a second download-lane job remains queued
• a general-lane sync_only job can run while a download-lane job is active
• catalog_sync is correctly classified into the download lane

API Tests

Add tests for:

• POST /api/playlists/sync
• dashboard payload includes lane and task-summary fields
• dashboard renders compact action controls
• inline task detail data is available

UI Rendering Tests

At minimum verify:

• dashboard contains the main task table
• dashboard no longer depends on a jump to detail as the primary control path
• playlists page contains sync selected playlists
• download tasks render speed fields and real values when available

Regression Tests

Protect:

• existing pause, resume, cancel command flow
• wanted_only= empty-query compatibility
• playlist progress rendering
• task playlist progress rendering

Rollout Plan

Recommended rollout order:

1. add dashboard-oriented task summary repository helpers
2. add lane-aware scheduling rules
3. add playlist bulk sync endpoint and button
4. redesign dashboard into the primary task center
5. add structured download throughput reporting
6. redeploy to NAS and verify live behavior

This order keeps correctness and scheduling changes ahead of cosmetic UI work.

Risks

Primary technical risk:

• refactoring the runner from a single-global-job loop into a lane-aware multi-job scheduler

Risk reduction:

• keep stage executors intact
• concentrate changes in job selection and orchestration
• verify lane rules with focused tests before refining UI

Secondary risk:

• structured speed reporting may require touching downloader integration points across multiple providers

Risk reduction:

• start with download-stage worker instrumentation in catalogsync
• expose speed only when a real structured value is available
• degrade gracefully to - rather than inventing numbers

Success Criteria

The design is successful when:

• operators can manage tasks primarily from Dashboard
• normal control flow does not require bouncing into /jobs/{id}
• multiple download-class jobs can be queued while only one runs
• collect and sync jobs are no longer unnecessarily blocked behind downloads
• selected playlists can be synced directly from the playlists page
• running download tasks show meaningful live throughput in the task center