Integrate task-model assignment routing via cf-orch /api/inference/task #135

New issue

Open

opened 2026-05-13 07:38:13 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-05-13 07:38:13 -07:00

Owner

Background

cf-orch #60 shipped a three-layer task-model assignment system:

Layer 1 (assignments.yaml): product.task → model_id
Layer 2 (model_registry.yaml): model_id → {service_type, vram_mb, alias, ...}
Layer 3: per-node catalogs (existing, unchanged)

This exposes a new endpoint: POST /api/inference/task which accepts {product, task, prompt, ...} and resolves the right model+node automatically — no hardcoded model IDs or service type checks in product code.

Spec: circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md

What needs to change in Kiwi

1. Register tasks in `assignments.yaml`

kiwi:
  meal_plan:
    model_id: ibm-granite--granite-4.1-8b
    description: Meal plan generation from pantry
  recipe_generation:
    model_id: ibm-granite--granite-4.1-8b
    description: Recipe suggestions from available ingredients
  ocr:
    model_id: <vision-model-slug>
    description: Receipt and barcode OCR (vision model)
  barcode_scan:
    model_id: <vision-model-slug>
    description: Barcode / product lookup via vision

2. Migration path for call sites

services/coordinator_proxy.py — main cf-orch proxy layer
services/meal_plan/llm_router.py — meal plan LLM routing
services/meal_plan/llm_planner.py — planner calls
services/ocr/vl_model.py — vision/OCR model selection

Replace explicit model selection with POST /api/inference/task calls using {"product": "kiwi", "task": "<task_name>", ...}.

Note for vision tasks (ocr, barcode_scan): The model registry service_type should be set to cf-vision for these entries. The /api/inference/task endpoint allocates appropriately based on service_type from Layer 2.

3. Backwards compatibility

This is opt-in — existing POST /api/inference calls continue to work. Migrate call sites incrementally.

Acceptance Criteria

assignments.yaml has entries for kiwi tasks (at minimum meal_plan and recipe_generation)
services/coordinator_proxy.py or llm_router.py migrated to /api/inference/task
Vision task entries use service_type: cf-vision in model registry
Tests updated / added

cf-orch #60 (task-model assignment layer — already shipped)
circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md

## Background cf-orch #60 shipped a three-layer task-model assignment system: - **Layer 1** (`assignments.yaml`): `product.task → model_id` - **Layer 2** (`model_registry.yaml`): `model_id → {service_type, vram_mb, alias, ...}` - **Layer 3**: per-node catalogs (existing, unchanged) This exposes a new endpoint: `POST /api/inference/task` which accepts `{product, task, prompt, ...}` and resolves the right model+node automatically — no hardcoded model IDs or service type checks in product code. Spec: `circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md` ## What needs to change in Kiwi ### 1. Register tasks in `assignments.yaml` ```yaml kiwi: meal_plan: model_id: ibm-granite--granite-4.1-8b description: Meal plan generation from pantry recipe_generation: model_id: ibm-granite--granite-4.1-8b description: Recipe suggestions from available ingredients ocr: model_id: <vision-model-slug> description: Receipt and barcode OCR (vision model) barcode_scan: model_id: <vision-model-slug> description: Barcode / product lookup via vision ``` ### 2. Migration path for call sites - `services/coordinator_proxy.py` — main cf-orch proxy layer - `services/meal_plan/llm_router.py` — meal plan LLM routing - `services/meal_plan/llm_planner.py` — planner calls - `services/ocr/vl_model.py` — vision/OCR model selection Replace explicit model selection with `POST /api/inference/task` calls using `{"product": "kiwi", "task": "<task_name>", ...}`. **Note for vision tasks (ocr, barcode_scan):** The model registry `service_type` should be set to `cf-vision` for these entries. The `/api/inference/task` endpoint allocates appropriately based on service_type from Layer 2. ### 3. Backwards compatibility This is **opt-in** — existing `POST /api/inference` calls continue to work. Migrate call sites incrementally. ## Acceptance Criteria - [ ] `assignments.yaml` has entries for kiwi tasks (at minimum `meal_plan` and `recipe_generation`) - [ ] `services/coordinator_proxy.py` or `llm_router.py` migrated to `/api/inference/task` - [ ] Vision task entries use `service_type: cf-vision` in model registry - [ ] Tests updated / added ## Related - cf-orch #60 (task-model assignment layer — already shipped) - `circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md`