feat: env var LLM config, cf-orch coordinator auth, SFT default bench path (#10, #14) #22

Merged

pyr0ball merged 3 commits from feat/env-config-sft-import into main

2026-04-09 12:30:56 -07:00

pyr0ball commented

2026-04-09 12:29:02 -07:00

Owner

Summary

#10 — env var fallbacks for CF_ORCH_URL, CF_LICENSE_KEY, OLLAMA_HOST, OLLAMA_MODEL in cf-orch integration; label_tool.yaml takes priority when both present; license key forwarded to benchmark subprocess env; GET /api/cforch/config exposes resolved connection state (key redacted); Settings UI shows connection status pill
#14 — _DEFAULT_BENCH_RESULTS_DIR set to circuitforge-orch bench results path so Corrections tab finds runs out of the box; testability seam to override in tests
Also includes feat/cforch-benchmark (LLM benchmark integration, PR #21 base)

New files

.env.example — documents all supported env vars
config/label_tool.yaml.example — full cforch: section with all keys documented

Test plan

136 tests passing (conda run -n cf python -m pytest tests/ --ignore=tests/test_finetune.py ...)
Settings tab shows connection status pill (● Ollama / ● cf-orch coordinator / ○ Not configured)
Corrections tab: scan for runs finds runs from circuitforge-orch bench_results without any config
Set CF_ORCH_URL=http://localhost:7700 in env, restart server — Settings pill shows ● cf-orch coordinator via env vars

🤖 Generated with Claude Code

## Summary - **#10** — env var fallbacks for `CF_ORCH_URL`, `CF_LICENSE_KEY`, `OLLAMA_HOST`, `OLLAMA_MODEL` in cf-orch integration; `label_tool.yaml` takes priority when both present; license key forwarded to benchmark subprocess env; `GET /api/cforch/config` exposes resolved connection state (key redacted); Settings UI shows connection status pill - **#14** — `_DEFAULT_BENCH_RESULTS_DIR` set to circuitforge-orch bench results path so Corrections tab finds runs out of the box; testability seam to override in tests - Also includes `feat/cforch-benchmark` (LLM benchmark integration, PR #21 base) ## New files - `.env.example` — documents all supported env vars - `config/label_tool.yaml.example` — full `cforch:` section with all keys documented ## Test plan - [ ] 136 tests passing (`conda run -n cf python -m pytest tests/ --ignore=tests/test_finetune.py ...`) - [ ] Settings tab shows connection status pill (● Ollama / ● cf-orch coordinator / ○ Not configured) - [ ] Corrections tab: scan for runs finds runs from circuitforge-orch bench_results without any config - [ ] Set `CF_ORCH_URL=http://localhost:7700` in env, restart server — Settings pill shows ● cf-orch coordinator via env vars 🤖 Generated with [Claude Code](https://claude.ai/claude-code)

pyr0ball added 3 commits 2026-04-09 12:29:02 -07:00

feat: cf-orch LLM benchmark integration (Phase 1) dffb1d0d7a

Backend (app/cforch.py — new APIRouter at /api/cforch):
- GET /tasks — reads bench_tasks.yaml, returns tasks + deduplicated types
- GET /models — reads bench_models.yaml, returns model list with service/tags
- GET /run — SSE endpoint; spawns cf-orch benchmark.py subprocess with
  --filter-tasks, --filter-tags, --coordinator, --ollama-url; strips ANSI
  codes; emits progress/result/complete/error events; 409 guard on concurrency
- GET /results — returns latest bench_results/*/summary.json; 404 if none
- POST /cancel — terminates running benchmark subprocess
- All paths configurable via label_tool.yaml cforch: section
- 13 tests; follows sft.py/models.py testability seam pattern

Frontend:
- BenchmarkView: mode toggle (Classifier / LLM Eval); LLM Eval panel with
  task picker (by type, select-all + indeterminate), model picker (by service),
  SSE run log, results table with best-per-column highlighting
- StatsView: LLM Benchmark section showing quality_by_task_type table across
  models; hidden when no results; fetches /api/cforch/results on mount

SFT candidate pipeline: cf-orch runs that produce sft_candidates.jsonl are
auto-discovered by the existing bench_results_dir config in sft.py — no
additional wiring needed.

feat(#10 ): env var LLM config + cf-orch coordinator auth a271278dc9

- _load_cforch_config() falls back to CF_ORCH_URL / CF_LICENSE_KEY /
  OLLAMA_HOST / OLLAMA_MODEL env vars when label_tool.yaml cforch: key
  is absent or empty (yaml wins when both present)
- CF_LICENSE_KEY forwarded to benchmark subprocess env so cf-orch agent
  can authenticate without it appearing in command args
- GET /api/cforch/config endpoint — returns resolved connection state;
  redacts license key (returns license_key_set bool only)
- SettingsView: connection status pill (cf-orch / Ollama / unconfigured)
  loaded from /api/cforch/config on mount; shows env vs yaml source
- .env.example documenting all relevant vars
- config/label_tool.yaml.example: full cforch: section with all keys
- environment.yml: add circuitforge-core>=0.9.0 dependency
- .gitignore: add .env
- 4 new tests (17 total in test_cforch.py); 136 passing overall

Closes #10

feat(#14 ): default bench_results_dir + testability seam 891142570b

- sft.py: _DEFAULT_BENCH_RESULTS_DIR set to circuitforge-orch bench
  results path; set_default_bench_results_dir() seam for test isolation
- test fixture resets default to tmp_path to avoid real-fs interference
- 136 tests passing

Closes #14