New Models tab: HuggingFace discovery, download queue, and installed model management #18

New issue

Closed

opened 2026-04-08 22:18:19 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-08 22:18:19 -07:00

Owner

Context: Model lifecycle (discover → download → manage) is a separate concern from running benchmarks. Adding it to BenchmarkView would make it too dense. A dedicated Models tab gives each stage room to breathe and makes the workflow explicit.

Tab layout (Models tab — new):

HF repo-ID input: accepts https://huggingface.co/org/model or bare org/model → calls HF Hub API to fetch task type, size, description, license
Approval queue: models pending your go-ahead (card per model, Approve / Dismiss)
Download queue: approved models being fetched via huggingface_hub.snapshot_download() with SSE progress
Installed models: table of on-disk models (name, task type, size, adapter) with Delete button

Benchmark tab changes (minimal):

Model selector: checkboxes to include/exclude installed models in the next run
No other structural changes

Backend — new router app/models.py:

GET /api/models/lookup?repo_id= — fetch HF Hub metadata, return task type + adapter recommendation
GET /api/models/installed — scan models/ dir, return list
GET /api/models/queue — return approval + download queue state (JSONL-persisted)
POST /api/models/queue — add model to approval queue
POST /api/models/queue/{id}/approve — move to download queue, start download
DELETE /api/models/queue/{id} — dismiss from queue
GET /api/models/download/stream — SSE progress for active download
DELETE /api/models/installed/{name} — remove model from disk

HF pipeline_tag → adapter mapping:

zero-shot-classification / text-classification → ZeroShotAdapter
sentence-similarity / text-ranking → RerankerAdapter
text-generation → GenerationAdapter (future)
other → warn, still allow download

Out of scope: Multi-model parallel downloads; training newly downloaded models immediately.

Acceptance criteria:

Paste HF link → see model card preview before approving
Approve → download starts, progress visible in UI
Installed model appears in Benchmark tab model selector
Queue state survives API restart
Delete removes model files from disk

Related: app/api.py SSE pattern (reuse for download stream); scripts/benchmark_classifier.py MODEL_REGISTRY (extend with dynamic models)

**Context:** Model lifecycle (discover → download → manage) is a separate concern from running benchmarks. Adding it to BenchmarkView would make it too dense. A dedicated Models tab gives each stage room to breathe and makes the workflow explicit. **Tab layout (Models tab — new):** - HF repo-ID input: accepts `https://huggingface.co/org/model` or bare `org/model` → calls HF Hub API to fetch task type, size, description, license - Approval queue: models pending your go-ahead (card per model, Approve / Dismiss) - Download queue: approved models being fetched via `huggingface_hub.snapshot_download()` with SSE progress - Installed models: table of on-disk models (name, task type, size, adapter) with Delete button **Benchmark tab changes (minimal):** - Model selector: checkboxes to include/exclude installed models in the next run - No other structural changes **Backend — new router `app/models.py`:** - `GET /api/models/lookup?repo_id=` — fetch HF Hub metadata, return task type + adapter recommendation - `GET /api/models/installed` — scan models/ dir, return list - `GET /api/models/queue` — return approval + download queue state (JSONL-persisted) - `POST /api/models/queue` — add model to approval queue - `POST /api/models/queue/{id}/approve` — move to download queue, start download - `DELETE /api/models/queue/{id}` — dismiss from queue - `GET /api/models/download/stream` — SSE progress for active download - `DELETE /api/models/installed/{name}` — remove model from disk **HF pipeline_tag → adapter mapping:** - `zero-shot-classification` / `text-classification` → ZeroShotAdapter - `sentence-similarity` / `text-ranking` → RerankerAdapter - `text-generation` → GenerationAdapter (future) - other → warn, still allow download **Out of scope:** Multi-model parallel downloads; training newly downloaded models immediately. **Acceptance criteria:** - Paste HF link → see model card preview before approving - Approve → download starts, progress visible in UI - Installed model appears in Benchmark tab model selector - Queue state survives API restart - Delete removes model files from disk **Related:** `app/api.py` SSE pattern (reuse for download stream); `scripts/benchmark_classifier.py` MODEL_REGISTRY (extend with dynamic models)

pyr0ball added this to the Beta — Benchmark Harness milestone 2026-04-08 22:18:19 -07:00

pyr0ball added the

enhancement

label 2026-04-08 22:18:19 -07:00

pyr0ball changed title from ~~Benchmark tab: HuggingFace model discovery, download queue, and multi-category benchmarking~~ to New Models tab: HuggingFace discovery, download queue, and installed model management

2026-04-08 22:21:16 -07:00

pyr0ball closed this issue

2026-04-09 12:07:47 -07:00