Integrate task-model assignment routing via cf-orch /api/inference/task #115

New issue

Closed

opened 2026-05-13 07:38:03 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-05-13 07:38:03 -07:00

Owner

Background

cf-orch #60 shipped a three-layer task-model assignment system:

Layer 1 (assignments.yaml): product.task → model_id
Layer 2 (model_registry.yaml): model_id → {service_type, vram_mb, alias, ...}
Layer 3: per-node catalogs (existing, unchanged)

This exposes a new endpoint: POST /api/inference/task which accepts {product, task, prompt, ...} and resolves the right model+node automatically — no hardcoded model IDs in product code.

Spec: circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md

What needs to change in Peregrine

1. Register tasks in `assignments.yaml`

peregrine:
  cover_letter:
    model_id: ibm-granite--granite-4.1-8b
    description: Cover letter generation
  ats_rewrite:
    model_id: ibm-granite--granite-4.1-8b
    description: ATS keyword rewriting
  job_research:
    model_id: ibm-granite--granite-4.1-8b
    description: Job post research / summarization
  interview_prep:
    model_id: ibm-granite--granite-4.1-8b
    description: Interview question generation

(Entries managed via Avocet Assignments UI or direct YAML edit + cf-orch reload.)

2. Migration path for call sites

The following files currently call cf-orch directly with explicit model selection. Update them to call POST /api/inference/task with {"product": "peregrine", "task": "<task_name>", ...} instead:

app/wizard/step_inference.py — inference step
dev_api.py / dev-api.py — development API endpoints
resume_matcher/apps/backend/app/llm.py — LLM routing

3. Backwards compatibility

This is opt-in — existing POST /api/inference calls continue to work. Migrate call sites incrementally.

Acceptance Criteria

assignments.yaml has entries for all four peregrine tasks
At least one call site migrated to /api/inference/task
All migrated call sites remove hardcoded model IDs
Tests updated / added for the new routing path

cf-orch #60 (task-model assignment layer — already shipped)
circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md

## Background cf-orch #60 shipped a three-layer task-model assignment system: - **Layer 1** (`assignments.yaml`): `product.task → model_id` - **Layer 2** (`model_registry.yaml`): `model_id → {service_type, vram_mb, alias, ...}` - **Layer 3**: per-node catalogs (existing, unchanged) This exposes a new endpoint: `POST /api/inference/task` which accepts `{product, task, prompt, ...}` and resolves the right model+node automatically — no hardcoded model IDs in product code. Spec: `circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md` ## What needs to change in Peregrine ### 1. Register tasks in `assignments.yaml` ```yaml peregrine: cover_letter: model_id: ibm-granite--granite-4.1-8b description: Cover letter generation ats_rewrite: model_id: ibm-granite--granite-4.1-8b description: ATS keyword rewriting job_research: model_id: ibm-granite--granite-4.1-8b description: Job post research / summarization interview_prep: model_id: ibm-granite--granite-4.1-8b description: Interview question generation ``` (Entries managed via Avocet Assignments UI or direct YAML edit + cf-orch reload.) ### 2. Migration path for call sites The following files currently call cf-orch directly with explicit model selection. Update them to call `POST /api/inference/task` with `{"product": "peregrine", "task": "<task_name>", ...}` instead: - `app/wizard/step_inference.py` — inference step - `dev_api.py` / `dev-api.py` — development API endpoints - `resume_matcher/apps/backend/app/llm.py` — LLM routing ### 3. Backwards compatibility This is **opt-in** — existing `POST /api/inference` calls continue to work. Migrate call sites incrementally. ## Acceptance Criteria - [ ] `assignments.yaml` has entries for all four peregrine tasks - [ ] At least one call site migrated to `/api/inference/task` - [ ] All migrated call sites remove hardcoded model IDs - [ ] Tests updated / added for the new routing path ## Related - cf-orch #60 (task-model assignment layer — already shipped) - `circuitforge-plans/circuitforge-orch/superpowers/specs/2026-05-13-task-model-assignments-design.md`