[New Feature] Email classifier benchmark harness — evaluate adapter models against labeled gold set #6

New issue

Open

opened 2026-03-04 09:31:51 -08:00 by pyr0ball · 0 comments

pyr0ball commented

2026-03-04 09:31:51 -08:00

Owner

Overview

Design doc: docs/plans/2026-02-26-email-classifier-benchmark-design.md
Plan doc: docs/plans/2026-02-26-email-classifier-benchmark-plan.md

Implement a benchmark harness that:

Loads a labeled gold set (JSONL with ground-truth labels)
Runs each configured classifier adapter (rule-based, zero-shot, fine-tuned) against the gold set
Reports per-label precision/recall/F1 and macro averages
Saves results to data/benchmark_results/ with timestamp
Supports ./manage.sh benchmark and ./manage.sh compare CLI commands

Key Files

scripts/benchmark.py — main harness entry point
scripts/classifier_adapters.py — adapter interface (already started)
tests/test_benchmark.py — TDD tests

Acceptance Criteria

./manage.sh benchmark runs all adapters and prints a results table
./manage.sh compare diffs two benchmark result files
./manage.sh score prints current model scores
Results saved as JSON with metadata (date, model version, dataset size)
All tests pass

## Overview Design doc: `docs/plans/2026-02-26-email-classifier-benchmark-design.md` Plan doc: `docs/plans/2026-02-26-email-classifier-benchmark-plan.md` Implement a benchmark harness that: 1. Loads a labeled gold set (JSONL with ground-truth labels) 2. Runs each configured classifier adapter (rule-based, zero-shot, fine-tuned) against the gold set 3. Reports per-label precision/recall/F1 and macro averages 4. Saves results to `data/benchmark_results/` with timestamp 5. Supports `./manage.sh benchmark` and `./manage.sh compare` CLI commands ## Key Files - `scripts/benchmark.py` — main harness entry point - `scripts/classifier_adapters.py` — adapter interface (already started) - `tests/test_benchmark.py` — TDD tests ## Acceptance Criteria - `./manage.sh benchmark` runs all adapters and prints a results table - `./manage.sh compare` diffs two benchmark result files - `./manage.sh score` prints current model scores - Results saved as JSON with metadata (date, model version, dataset size) - All tests pass