Add weekly_harvest.sh wrapper that:
- Runs discover_current_menu.py to fetch this week's 23 active menu slugs
- Runs scrape_live.py with --resume to scrape only new slugs
- Appends timestamped output to /Library/Assets/kiwi/pipeline/logs/
Cron entry added to system crontab:
0 23 * * 0 (every Sunday 23:00)
Logs: /Library/Assets/kiwi/pipeline/logs/purple_carrot_harvest.log
Add three new scripts for Purple Carrot recipe pipeline:
- discover_current_menu.py: fetches this week's active menu slugs from
/plant-based-recipes using requests (server-rendered HTML, no JS needed).
Accumulates slugs across weekly runs for building a recipe corpus over time.
- discover_slugs_categories.py: crawls recipe-category listing pages with
?page=N pagination to discover historical slug inventory. Note: category
archive slugs (past menu items) 404 when scraped live; only use for
identifying currently-featured recipes per category.
- scrape_live.py: updated with --slugs-from flag (load slug inventory from
any parquet, not just the default Wayback one) and fresh-context-per-slug
pattern to bypass Cloudflare session-level bot detection (which fires on
the 2nd+ request in a shared browser context).
Discovery: the live site only renders full ingredient/instruction content for
recipes currently on the active weekly menu. 23/23 current menu recipes
scraped successfully (100% hit rate vs ~1% for archived slugs).
scrape_recipes.py:
- Switch CDX to HTTPS (avoids HTTP 503 rate-limit bucket)
- Restrict product API CDX to 2019–2021 window (pre-HelloFresh instruction stripping)
- Replace inline CDX requests with _cdx_get() helper: retries on 429/503 with
exponential backoff (15s, 30s, 60s, 120s)
- Increase HTML fallback CDX limit from 5 to 10 timestamps
- Bump CDX_DELAY 0.5s → 3.0s and REPLAY_DELAY 1.2s → 2.0s (polite scraping)
- Fix KeyError: 0 on hero_images dict (normalise dict to list before indexing)
discover_wayback.py:
- Switch CDX to HTTPS
scripts/pipeline/log_utils.py (new):
- attach_pipeline_log(script_name): adds a JSON FileHandler to the root logger
writing to /Library/Assets/logs/pipeline/<script>_<ts>.jsonl for Avocet
Turnstone training data ingestion (kiwi#141 / avocet#67)
discover_wayback.py — enumerates recipe slugs from archived menu API
(/api/v2/menus/<id>) and product API (/api/v1/products/*) plus
recipe-category HTML pages. Writes incremental JSONL manifest to
/Library/Assets/kiwi/pipeline/pc_slugs.jsonl.
scrape_recipes.py — fetches full recipe data per slug using three-tier
fallback: product API JSON (oldest captures first), HTML inline state
(__NEXT_DATA__ / __INITIAL_STATE__), and JSON-LD structured data.
Outputs recipes_purplecarrot.parquet in food.com columnar format so
build_recipe_index.py imports it unchanged. Includes SourceURL column
for recipe attribution UI (kiwi#139). Checkpoints every 50 recipes.
Initial discovery: 158 slugs from menu 1536 + product_api pass.
Re-run discover_wayback.py after archive.org stabilizes to pick up
older slugs from recipe-category pages.
Backlog: live Playwright scraper for post-Wayback recipes (kiwi#137).
backfill_meal_tags.py merges meal: tags from title-only matching
into existing inferred_tags without re-deriving all other signals.
~10x faster than infer_recipe_tags.py --force for meal-tag-only
updates: 3.19M recipes in ~5-10min vs ~2.5h for full re-derivation.
Multiple concurrent users browsing the 3.2M recipe corpus would cause FTS5 page
cache contention and slow per-request queries. Solution: pre-compute counts for
all category/subcategory keyword sets into a small SQLite cache.
- browse_counts_cache.py: refresh(), load_into_memory(), is_stale() helpers
- config.py: BROWSE_COUNTS_PATH setting (default DATA_DIR/browse_counts.db)
- main.py: warms in-memory cache on startup; runs nightly refresh task every 24h
- infer_recipe_tags.py: auto-refreshes cache after a successful tag run so the
app picks up updated FTS counts without a restart
- build_recipe_index.py: add _parse_r_vector() for food.com R format, add
_parse_allrecipes_text() for corbt/all-recipes text format, _row_to_fields()
dispatcher handles both columnar (food.com) and single-text (all-recipes)
- build_flavorgraph_index.py: switch from graph.json to nodes/edges CSVs
matching actual FlavorGraph repo structure
- download_datasets.py: switch recipe source to corbt/all-recipes (2.1M
recipes, 807MB) replacing near-empty AkashPS11/recipes_data_food.com
- 007_recipe_corpus.sql: add UNIQUE constraint on external_id to prevent
duplicate inserts on pipeline reruns
