# Email Classifier Benchmark — Design

**Date:** 2026-02-26
**Status:** Approved

## Problem

The current `classify_stage_signal()` in `scripts/imap_sync.py` uses `llama3.1:8b` via
Ollama for 6-label email classification. This is slow, requires a running Ollama instance,
and accuracy is unverified against alternatives. This design establishes a benchmark harness
to evaluate HuggingFace-native classifiers as potential replacements.

## Labels

```
interview_scheduled  offer_received  rejected
positive_response    survey_received  neutral
```

## Approach: Standalone Benchmark Script (Approach B)

Two new files; nothing in `imap_sync.py` changes until a winner is chosen.

```
scripts/
  benchmark_classifier.py     — CLI entry point
  classifier_adapters.py      — adapter classes (reusable by imap_sync later)

data/
  email_eval.jsonl            — labeled ground truth (gitignored — contains email content)
  email_eval.jsonl.example    — committed example with fake emails

scripts/classifier_service/
  environment.yml             — new conda env: job-seeker-classifiers
```

## Adapter Pattern

```
ClassifierAdapter (ABC)
  .classify(subject, body) → str   # one of the 6 labels
  .name → str
  .model_id → str
  .load() / .unload()              # explicit lifecycle

ZeroShotAdapter(ClassifierAdapter)
  # uses transformers pipeline("zero-shot-classification")
  # candidate_labels = list of 6 labels
  # works for: DeBERTa, BART-MNLI, BGE-M3-ZeroShot, XLM-RoBERTa

GLiClassAdapter(ClassifierAdapter)
  # uses gliclass library (pip install gliclass)
  # GLiClassModel + ZeroShotClassificationPipeline
  # works for: gliclass-instruct-large-v1.0

RerankerAdapter(ClassifierAdapter)
  # uses FlagEmbedding reranker.compute_score()
  # scores (email_text, label_description) pairs; highest = predicted label
  # works for: bge-reranker-v2-m3
```

## Model Registry

| Short name | Model | Params | Adapter | Default |
|------------|-------|--------|---------|---------|
| `deberta-zeroshot` | MoritzLaurer/DeBERTa-v3-large-zeroshot-v2.0 | 400M | ZeroShot | ✅ |
| `deberta-small` | cross-encoder/nli-deberta-v3-small | 100M | ZeroShot | ✅ |
| `gliclass-large` | knowledgator/gliclass-instruct-large-v1.0 | 400M | GLiClass | ✅ |
| `bart-mnli` | facebook/bart-large-mnli | 400M | ZeroShot | ✅ |
| `bge-m3-zeroshot` | MoritzLaurer/bge-m3-zeroshot-v2.0 | 600M | ZeroShot | ✅ |
| `bge-reranker` | BAAI/bge-reranker-v2-m3 | 600M | Reranker | ❌ (`--include-slow`) |
| `deberta-xlarge` | microsoft/deberta-xlarge-mnli | 750M | ZeroShot | ❌ (`--include-slow`) |
| `mdeberta-mnli` | MoritzLaurer/mDeBERTa-v3-base-mnli-xnli | 300M | ZeroShot | ❌ (`--include-slow`) |
| `xlm-roberta-anli` | vicgalle/xlm-roberta-large-xnli-anli | 600M | ZeroShot | ❌ (`--include-slow`) |

## CLI Modes

### `--compare` (live IMAP, visual table)
Extends the pattern of `test_email_classify.py`. Pulls emails via IMAP, shows a table:
```
Subject                                              | Phrase | llama3 | deberta-zs | deberta-sm | gliclass | bart | bge-m3
```
- Phrase-filter column shows BLOCK/pass (same gate as production)
- `llama3` column = current production baseline
- HF model columns follow

### `--eval` (ground-truth evaluation)
Reads `data/email_eval.jsonl`, runs all models, reports per-label and aggregate metrics:
- Per-label: precision, recall, F1
- Aggregate: macro-F1, accuracy
- Latency: ms/email per model

JSONL format:
```jsonl
{"subject": "Interview invitation", "body": "We'd like to schedule...", "label": "interview_scheduled"}
{"subject": "Your application", "body": "We regret to inform you...", "label": "rejected"}
```

### `--list-models`
Prints the registry with sizes, adapter types, and default/slow flags.

## Conda Environment

New env `job-seeker-classifiers` — isolated from `job-seeker` (no torch there).

Key deps:
- `torch` (CUDA-enabled)
- `transformers`
- `gliclass`
- `FlagEmbedding` (for bge-reranker only)
- `sentence-transformers` (optional, for future embedding-based approaches)

## GPU

Auto-select (`device="cuda"` when available, CPU fallback). No GPU pinning — models
load one at a time so VRAM pressure is sequential, not cumulative.

## Error Handling

- Model load failures: skip that column, print warning, continue
- Classification errors: show `ERR` in cell, continue
- IMAP failures: propagate (same as existing harness)
- Missing eval file: clear error message pointing to `data/email_eval.jsonl.example`

## What Does Not Change (Yet)

- `scripts/imap_sync.py` — production classifier unchanged
- `scripts/llm_router.py` — unchanged
- `staging.db` schema — unchanged

After benchmark results are reviewed, a separate PR will wire the winning model
into `classify_stage_signal()` as an opt-in backend in `llm_router.py`.