[Feature] Fine-tune email classifier — WeightedTrainer, FineTunedAdapter, SSE training UI #8

New issue

Closed

opened 2026-03-15 16:20:46 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-03-15 16:20:46 -07:00

Owner

Summary

Full fine-tune pipeline implemented on branch feat/vue-label-tab (2026-03-15).

What was built

New files

scripts/finetune_classifier.py — CLI + library: load_and_prepare_data(), WeightedTrainer, run_finetune(), compute_class_weights(), compute_metrics_for_trainer()
tests/test_finetune.py — 15 unit tests + 1 integration test

Modified files

scripts/classifier_adapters.py — FineTunedAdapter (loads saved checkpoint via HF pipeline; GPU-routed)
scripts/benchmark_classifier.py — discover_finetuned_models(), auto-merge fine-tuned adapters into benchmark at startup
app/api.py — GET /api/finetune/status, GET /api/finetune/run (SSE subprocess stream, path-traversal-safe)
web/src/views/BenchmarkView.vue — trained models badge row + collapsible fine-tune section (auto-triggers benchmark on completion)
environment.yml — added scikit-learn>=1.4

Key design notes

Multi-file JSONL: load_and_prepare_data() accepts Path | list[Path]; deduplicates by MD5(subject + body[:100]), last-write-wins. Supports accumulating multiple labeling sessions over time.
Supported models: deberta-small (100M, ~15 min, no VRAM conflict) and bge-m3 (600M, requires stopping Peregrine vLLM first)
WeightedTrainer: inverse-frequency class weights; compute_loss(**kwargs) absorbs num_items_in_batch (Transformers 4.38+); weights moved .to(device) for GPU training
Auto-discovery: discover_finetuned_models() scans models/*/training_info.json at startup — trained models appear in benchmark automatically
SSE streaming: training stdout piped via StreamingResponse; Vue useApiSSE composable handles progress/complete/error
Path traversal protection: score file query params resolved and prefix-checked against _DATA_DIR

CLI usage

# Single score file (default)
conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small --epochs 5

# Multiple score files (merged, last-write-wins dedup)
conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small \
  --score data/run1.jsonl --score data/run2.jsonl

Test results

job-seeker env: 79/79 pass
job-seeker-classifiers env: 44/44 unit tests pass (integration test skipped — needs GPU headroom)
Vue build: ✓

Known limitation

Integration test (test_integration_finetune_on_example_data) hits CUDA OOM when Peregrine vLLM is running. Run with ./manage.sh stop first.

## Summary Full fine-tune pipeline implemented on branch `feat/vue-label-tab` (2026-03-15). ## What was built ### New files - `scripts/finetune_classifier.py` — CLI + library: `load_and_prepare_data()`, `WeightedTrainer`, `run_finetune()`, `compute_class_weights()`, `compute_metrics_for_trainer()` - `tests/test_finetune.py` — 15 unit tests + 1 integration test ### Modified files - `scripts/classifier_adapters.py` — `FineTunedAdapter` (loads saved checkpoint via HF pipeline; GPU-routed) - `scripts/benchmark_classifier.py` — `discover_finetuned_models()`, auto-merge fine-tuned adapters into benchmark at startup - `app/api.py` — `GET /api/finetune/status`, `GET /api/finetune/run` (SSE subprocess stream, path-traversal-safe) - `web/src/views/BenchmarkView.vue` — trained models badge row + collapsible fine-tune section (auto-triggers benchmark on completion) - `environment.yml` — added `scikit-learn>=1.4` ## Key design notes - **Multi-file JSONL**: `load_and_prepare_data()` accepts `Path | list[Path]`; deduplicates by MD5(subject + body[:100]), last-write-wins. Supports accumulating multiple labeling sessions over time. - **Supported models**: `deberta-small` (100M, ~15 min, no VRAM conflict) and `bge-m3` (600M, requires stopping Peregrine vLLM first) - **WeightedTrainer**: inverse-frequency class weights; `compute_loss(**kwargs)` absorbs `num_items_in_batch` (Transformers 4.38+); weights moved `.to(device)` for GPU training - **Auto-discovery**: `discover_finetuned_models()` scans `models/*/training_info.json` at startup — trained models appear in benchmark automatically - **SSE streaming**: training stdout piped via `StreamingResponse`; Vue `useApiSSE` composable handles progress/complete/error - **Path traversal protection**: score file query params resolved and prefix-checked against `_DATA_DIR` ## CLI usage ```bash # Single score file (default) conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small --epochs 5 # Multiple score files (merged, last-write-wins dedup) conda run -n job-seeker-classifiers python scripts/finetune_classifier.py --model deberta-small \ --score data/run1.jsonl --score data/run2.jsonl ``` ## Test results - `job-seeker` env: 79/79 pass - `job-seeker-classifiers` env: 44/44 unit tests pass (integration test skipped — needs GPU headroom) - Vue build: ✓ ## Known limitation Integration test (`test_integration_finetune_on_example_data`) hits CUDA OOM when Peregrine vLLM is running. Run with `./manage.sh stop` first.