pyr0ball opened issue Circuit-Forge/avocet#8

pyr0ball
Sun, 15 Mar 2026 16:20:46 -0700

8#[Feature] Fine-tune email classifier — WeightedTrainer, FineTunedAdapter, SSE training UI#

Summary

Full fine-tune pipeline implemented on branch feat/vue-label-tab (2026-03-15).

What was built

New files

scripts/finetune_classifier.py — CLI + library: load_and_prepare_data(), WeightedTrainer, run_finetune(), compute_class_weights(), compute_metrics_for_trainer()

tests/test_finetune.py — 15 unit tests + 1 integration test

Modified files

scripts/classifier_adapters.py — FineTunedAdapter (loads saved checkpoint via HF pipeline; GPU-routed)

scripts/benchmark_classifier.py — discover_finetuned_models(), auto-merge fine-tuned adapters into benchmark at startup

app/api.py — GET /api/finetune/status, GET /api/finetune/run (SSE subprocess stream, path-traversal-safe)

web/src/views/BenchmarkView.vue — trained models badge row + collapsible fine-tune section (auto-triggers benchmark on completion)

environment.yml — added scikit-learn>=1.4

Key design notes

Multi-file JSONL: load_and_prepare_data() accepts Path | list[Path]; deduplicates by MD5(subject + body[:100]), last-write-wins. Supports accumulating multiple labeling sessions over time.

Supported models: deberta-small (100M, ~15 min, no VRAM conflict) and bge-m3 (600M, requires stopping Peregrine vLLM first)

WeightedTrainer: inverse-frequency class weights; compute_loss(**kwargs) absorbs num_items_in_batch (Transformers 4.38+); weights moved .to(device) for GPU training

Auto-discovery: discover_finetuned_models() scans models/*/training_info.json at startup — trained models appear in benchmark automatically

SSE streaming: training stdout piped via StreamingResponse; Vue useApiSSE composable handles progress/complete/error

Path traversal protection: score file query params resolved and prefix-checked against _DATA_DIR

CLI usage

# Single score file (default)
conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small --epochs 5

# Multiple score files (merged, last-write-wins dedup)
conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small \
  --score data/run1.jsonl --score data/run2.jsonl

Test results

job-seeker env: 79/79 pass

job-seeker-classifiers env: 44/44 unit tests pass (integration test skipped — needs GPU headroom)

Vue build: ✓

Known limitation

Integration test (test_integration_finetune_on_example_data) hits CUDA OOM when Peregrine vLLM is running. Run with ./manage.sh stop first.