feat: extract train module + SQLite finetune job queue #43

New issue

Closed

opened 2026-05-01 12:12:00 -07:00 by pyr0ball · 1 comment

pyr0ball commented

2026-05-01 12:12:00 -07:00

Owner

Context

Finetune is currently a fire-and-forget SSE endpoint buried in api.py. Replace with a proper job queue so multiple job types can be queued, monitored, and reviewed.

Work

Create app/train/train.py with SQLite-backed job queue
Jobs table: id, type, model_key, status, config_json, created_at, started_at, completed_at, error
Statuses: queued → running → completed | failed
Extract existing finetune routes from api.py into train.py
SSE log streaming from running jobs
Routes: POST /api/train/jobs, GET /api/train/jobs, GET /api/train/jobs/{id}/stream

Acceptance

Existing classifier finetune still works via new queue
Job status persists across server restarts
Live log streaming works via SSE

## Context Finetune is currently a fire-and-forget SSE endpoint buried in `api.py`. Replace with a proper job queue so multiple job types can be queued, monitored, and reviewed. ## Work - Create `app/train/train.py` with SQLite-backed job queue - Jobs table: `id, type, model_key, status, config_json, created_at, started_at, completed_at, error` - Statuses: `queued → running → completed | failed` - Extract existing finetune routes from `api.py` into `train.py` - SSE log streaming from running jobs - Routes: `POST /api/train/jobs`, `GET /api/train/jobs`, `GET /api/train/jobs/{id}/stream` ## Acceptance - Existing classifier finetune still works via new queue - Job status persists across server restarts - Live log streaming works via SSE

pyr0ball added this to the v2 — Pipeline Architecture milestone 2026-05-01 12:12:00 -07:00

pyr0ball added the

backend

reorg

labels 2026-05-01 12:12:00 -07:00

pyr0ball referenced this issue

2026-05-01 12:12:32 -07:00

feat: LLM SFT finetuning backend (TRL + PEFT/LoRA) #46

pyr0ball referenced this issue

2026-05-01 12:13:12 -07:00

feat: Train › Jobs view — finetune job queue UI #49

pyr0ball referenced this issue

2026-05-01 12:13:45 -07:00

feat: writing style finetuning backend (Phase 2) #52

pyr0ball referenced this issue

2026-05-01 12:13:46 -07:00

feat: voice/TTS finetuning backend (Phase 2) #53

pyr0ball referenced this issue

2026-05-01 12:13:46 -07:00

feat: cf-orch dispatch for training jobs (Phase 3) #54

pyr0ball commented

2026-05-06 23:00:43 -07:00

Author

Owner

Shipped in the Apr 19–May 4 sprint. SQLite-backed job queue, Popen subprocess, SSE log streaming, and path traversal guard on python_bin all landed in app/train/train.py.

pyr0ball closed this issue

2026-05-06 23:00:44 -07:00