Recipe scanner: photo of cookbook/recipe card/handwritten note → structured recipe + pantry cross-reference #9

New issue

Open

opened 2026-04-01 22:01:32 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-01 22:01:32 -07:00

Owner

Feature

Allow users to photograph a paper recipe (cookbook page, index card, handwritten note) and have it parsed into a structured recipe automatically.

Pipeline

📸 photo → Dolphin-v2 (layout) → LLM (semantics) → structured recipe → pantry cross-reference

Stage 1 — Dolphin-v2 (structure):

Identifies headings (recipe name), lists (ingredients, steps), tables (nutrition), paragraphs (notes)
Handles both printed and handwritten sources
Preprocessing pass (contrast enhancement) recommended for faded/old cards

Stage 2 — LLM (semantics):

Extracts {item, qty, unit} tuples from ingredient list
Parses steps into ordered list
Normalises units (e.g. "2 cups" → {qty: 2, unit: "cup"})

Stage 3 — Pantry cross-reference:

Matches extracted ingredients against pantry inventory
Shows "you have 7 of 9 ingredients" before user saves recipe
Flags missing items for optional shopping list addition

Use cases

Cookbook pages
Printed recipe cards
Handwritten family recipes / index cards
Napkin notes / informal recipes

Notes

Same camera UX as existing barcode/receipt scanning (Phase 1)
Dolphin-v2 needs ~8GB VRAM — falls back to LLM-only parsing on lower-tier hardware
Handwritten degradation: add image preprocessing step for old/faded cards
Model ref: ByteDance/Dolphin-v2 (HuggingFace)

## Feature Allow users to photograph a paper recipe (cookbook page, index card, handwritten note) and have it parsed into a structured recipe automatically. ## Pipeline ``` 📸 photo → Dolphin-v2 (layout) → LLM (semantics) → structured recipe → pantry cross-reference ``` **Stage 1 — Dolphin-v2 (structure):** - Identifies headings (recipe name), lists (ingredients, steps), tables (nutrition), paragraphs (notes) - Handles both printed and handwritten sources - Preprocessing pass (contrast enhancement) recommended for faded/old cards **Stage 2 — LLM (semantics):** - Extracts `{item, qty, unit}` tuples from ingredient list - Parses steps into ordered list - Normalises units (e.g. "2 cups" → `{qty: 2, unit: "cup"}`) **Stage 3 — Pantry cross-reference:** - Matches extracted ingredients against pantry inventory - Shows "you have 7 of 9 ingredients" before user saves recipe - Flags missing items for optional shopping list addition ## Use cases - Cookbook pages - Printed recipe cards - Handwritten family recipes / index cards - Napkin notes / informal recipes ## Notes - Same camera UX as existing barcode/receipt scanning (Phase 1) - Dolphin-v2 needs ~8GB VRAM — falls back to LLM-only parsing on lower-tier hardware - Handwritten degradation: add image preprocessing step for old/faded cards - Model ref: `ByteDance/Dolphin-v2` (HuggingFace)