Per-user LLM config: expose in Settings UI + cloud admin tooling #49

New issue

Closed

opened 2026-04-02 09:06:34 -07:00 by pyr0ball · 1 comment

pyr0ball commented

2026-04-02 09:06:34 -07:00

Owner

Background

Per-user llm.yaml overrides are now wired into the cover letter generation path (task_runner.py → generate() picks up <user_data>/peregrine/config/llm.yaml when it exists, falling back to the global cloud config).

Meghan's file was written manually at /devl/menagerie-data/5b99ca9f-b0ef-46fb-aa21-ee8d5a0ebc21/peregrine/config/llm.yaml with meghan-cover-writer:latest as her ollama cover letter model.

Problem

This is a manual operation right now. There is no:

Settings UI for users to select their custom fine-tuned model
Admin tooling to set/inspect per-user LLM config from outside the container
Coverage in company_research.py or other LLM callers (only generate_cover_letter is wired)

Tasks

Extend per-user config wiring to company_research.py (also calls LLMRouter)
Settings UI: "Custom cover letter model" field in Settings → System (Paid tier+); writes to per-user llm.yaml
Admin script or endpoint to set/inspect per-user LLM config without docker exec
Consider: should the model list come from a live ollama /api/tags call so the UI shows available models?
Document the per-user llm.yaml override pattern in CLAUDE.md

meghan-cover-writer:latest is the fine-tuned Llama 3.2 3B model for Meghan's cover letters
Global cloud default is llama3.1:8b (no fine-tunes for anonymous users)

## Background Per-user `llm.yaml` overrides are now wired into the cover letter generation path (`task_runner.py` → `generate()` picks up `<user_data>/peregrine/config/llm.yaml` when it exists, falling back to the global cloud config). Meghan's file was written manually at `/devl/menagerie-data/5b99ca9f-b0ef-46fb-aa21-ee8d5a0ebc21/peregrine/config/llm.yaml` with `meghan-cover-writer:latest` as her ollama cover letter model. ## Problem This is a manual operation right now. There is no: - Settings UI for users to select their custom fine-tuned model - Admin tooling to set/inspect per-user LLM config from outside the container - Coverage in `company_research.py` or other LLM callers (only `generate_cover_letter` is wired) ## Tasks - [ ] Extend per-user config wiring to `company_research.py` (also calls LLMRouter) - [ ] Settings UI: "Custom cover letter model" field in Settings → System (Paid tier+); writes to per-user `llm.yaml` - [ ] Admin script or endpoint to set/inspect per-user LLM config without docker exec - [ ] Consider: should the model list come from a live ollama `/api/tags` call so the UI shows available models? - [ ] Document the per-user `llm.yaml` override pattern in CLAUDE.md ## Related - `meghan-cover-writer:latest` is the fine-tuned Llama 3.2 3B model for Meghan's cover letters - Global cloud default is `llama3.1:8b` (no fine-tunes for anonymous users)

pyr0ball added this to the Paid Tier GA milestone 2026-04-04 16:33:18 -07:00

pyr0ball commented

2026-04-06 10:13:17 -07:00

Author

Owner

Implemented. Three parts:

company_research.py now accepts config_path and uses the per-user llm.yaml when present (same pattern as cover_letter). Key gotcha: must call scripts/llm_router.py shim (LLMRouter() with no args when no user config) — importing CONFIG_PATH from circuitforge_core directly bypasses the tri-level repo/user/env config resolution and breaks cloud.
Three new API endpoints: GET/PUT /api/settings/llm/cover-letter-model, GET /api/settings/llm/ollama-models. PUT writes to per-user config/llm.yaml, injecting a cover_letter backend at the front of the fallback order.
Settings → System now shows a Custom Cover Letter Model section for cloud + paid users, populated from live Ollama /api/tags call.

Implemented. Three parts: 1. company_research.py now accepts config_path and uses the per-user llm.yaml when present (same pattern as cover_letter). Key gotcha: must call scripts/llm_router.py shim (LLMRouter() with no args when no user config) — importing CONFIG_PATH from circuitforge_core directly bypasses the tri-level repo/user/env config resolution and breaks cloud. 2. Three new API endpoints: GET/PUT /api/settings/llm/cover-letter-model, GET /api/settings/llm/ollama-models. PUT writes to per-user config/llm.yaml, injecting a cover_letter backend at the front of the fallback order. 3. Settings → System now shows a Custom Cover Letter Model section for cloud + paid users, populated from live Ollama /api/tags call.

pyr0ball closed this issue

2026-04-06 10:13:17 -07:00