|
|
dffb1d0d7a
|
feat: cf-orch LLM benchmark integration (Phase 1)
Backend (app/cforch.py — new APIRouter at /api/cforch):
- GET /tasks — reads bench_tasks.yaml, returns tasks + deduplicated types
- GET /models — reads bench_models.yaml, returns model list with service/tags
- GET /run — SSE endpoint; spawns cf-orch benchmark.py subprocess with
--filter-tasks, --filter-tags, --coordinator, --ollama-url; strips ANSI
codes; emits progress/result/complete/error events; 409 guard on concurrency
- GET /results — returns latest bench_results/*/summary.json; 404 if none
- POST /cancel — terminates running benchmark subprocess
- All paths configurable via label_tool.yaml cforch: section
- 13 tests; follows sft.py/models.py testability seam pattern
Frontend:
- BenchmarkView: mode toggle (Classifier / LLM Eval); LLM Eval panel with
task picker (by type, select-all + indeterminate), model picker (by service),
SSE run log, results table with best-per-column highlighting
- StatsView: LLM Benchmark section showing quality_by_task_type table across
models; hidden when no results; fetches /api/cforch/results on mount
SFT candidate pipeline: cf-orch runs that produce sft_candidates.jsonl are
auto-discovered by the existing bench_results_dir config in sft.py — no
additional wiring needed.
|
2026-04-09 10:46:06 -07:00 |
|
