Switches to OrchestratedScheduler in cloud mode so concurrent recipe_llm
jobs fan out across all registered cf-orch GPU nodes instead of serializing
on one. Under load this eliminates poll timeouts from queue backup.
USE_ORCH_SCHEDULER env var gives explicit control independent of CLOUD_MODE:
unset follow CLOUD_MODE (cloud=orch, local=local)
true OrchestratedScheduler always (e.g. multi-GPU local rig)
false LocalScheduler always (e.g. cloud single-GPU dev instance)
ImportError fallback: if circuitforge_orch is not installed and orch is
requested, logs a warning and falls back to LocalScheduler gracefully.
Adds the recipe_jobs table and background task pipeline for level 3/4
recipe generation. POST ?async=true returns 202 with job_id; clients
poll GET /recipes/jobs/{job_id} until status=done.
Key fix: _enqueue_recipe_job now calls scheduler.enqueue() after
insert_task() to wake the in-memory work queue immediately. Without
this, tasks sat in 'queued' until the scheduler's 30s idle cycle or
an API restart triggered _load_queued_tasks().
- Migration 034: recipe_jobs table (job_id, user_id, status, request,
result, error) with indexes on job_id and user_id/created_at
- Store: create/get/update_running/complete/fail recipe job methods
- runner.py: recipe_llm task type + _run_recipe_llm handler; MUST
call fail_recipe_job() before re-raising so status stays consistent
- CLOUD_MODE guard: falls back to sync generation (scheduler only
polls shared settings DB, not per-user DBs)
- L4 wildcard is covered by the same req.level in (3, 4) dispatch
Uses circuitforge_core.tasks.scheduler. VRAM detection via cf-orch when
available, falling back to unlimited. Adds expiry_llm_fallback task type
to background-predict expiry dates for items the LUT doesn't cover.
