pyr0ball opened issue Circuit-Forge/avocet#6

pyr0ball
Wed, 04 Mar 2026 09:31:51 -0800

6#[New Feature] Email classifier benchmark harness — evaluate adapter models against labeled gold set#

Overview

Design doc: docs/plans/2026-02-26-email-classifier-benchmark-design.md
Plan doc: docs/plans/2026-02-26-email-classifier-benchmark-plan.md

Implement a benchmark harness that:

1. Loads a labeled gold set (JSONL with ground-truth labels)

2. Runs each configured classifier adapter (rule-based, zero-shot, fine-tuned) against the gold set

3. Reports per-label precision/recall/F1 and macro averages

4. Saves results to data/benchmark_results/ with timestamp

5. Supports ./manage.sh benchmark and ./manage.sh compare CLI commands

Key Files

scripts/benchmark.py — main harness entry point

scripts/classifier_adapters.py — adapter interface (already started)

tests/test_benchmark.py — TDD tests

Acceptance Criteria

./manage.sh benchmark runs all adapters and prints a results table

./manage.sh compare diffs two benchmark result files

./manage.sh score prints current model scores

Results saved as JSON with metadata (date, model version, dataset size)

All tests pass