Merlin pipeline adapter — gesture/landmark labeled training data #40

New issue

Open

opened 2026-04-26 21:45:10 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-26 21:45:10 -07:00

Owner

Avocet currently handles email classification training data. Extend to support Merlin gesture training data.

Data type: normalized landmark vectors (63-dim float32 for hands, face landmark subsets for blink/gaze/head). Stored as .npy or .jsonl in ~/.merlin/training/<profile>/.

Avocet changes needed:

New data adapter: MerlinLandmarkAdapter — reads landmark vectors from ~/.merlin/training/, presents uncertain (low-confidence) samples to the review queue alongside their gesture class label and signal quality score.
Review UI extension: card-stack labeler gets a new mode for gesture samples. Show gesture name, signal quality score, animated replay of the landmark sequence (stick figure or hand outline), Accept/Reject/Re-record buttons.
Trainer integration: after labeling session, one-click trigger to run Merlin's local MLP/gradient boost trainer. Progress shows in Avocet. Writes model to ~/.merlin/models/<profile>.joblib.
Benchmark hook: track per-gesture class accuracy across training sessions (same pattern as email benchmark harness).

This makes Avocet the universal training harness for the menagerie — not just email. Merlin is the first second consumer; EEG epoch labeling (Phase C) will be the third.

See Circuit-Forge/merlin#1 for the Merlin-side recorder and test mode.

Avocet currently handles email classification training data. Extend to support Merlin gesture training data. **Data type:** normalized landmark vectors (63-dim float32 for hands, face landmark subsets for blink/gaze/head). Stored as `.npy` or `.jsonl` in `~/.merlin/training/<profile>/`. **Avocet changes needed:** 1. **New data adapter:** `MerlinLandmarkAdapter` — reads landmark vectors from `~/.merlin/training/`, presents uncertain (low-confidence) samples to the review queue alongside their gesture class label and signal quality score. 2. **Review UI extension:** card-stack labeler gets a new mode for gesture samples. Show gesture name, signal quality score, animated replay of the landmark sequence (stick figure or hand outline), Accept/Reject/Re-record buttons. 3. **Trainer integration:** after labeling session, one-click trigger to run Merlin's local MLP/gradient boost trainer. Progress shows in Avocet. Writes model to `~/.merlin/models/<profile>.joblib`. 4. **Benchmark hook:** track per-gesture class accuracy across training sessions (same pattern as email benchmark harness). This makes Avocet the universal training harness for the menagerie — not just email. Merlin is the first second consumer; EEG epoch labeling (Phase C) will be the third. See Circuit-Forge/merlin#1 for the Merlin-side recorder and test mode.