Design ref: llama-conductor patterns for log retrieval cascade + provenance tracking #59

New issue

Open

opened 2026-05-31 09:44:00 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-05-31 09:44:00 -07:00

Owner

Reference project

GitHub: https://github.com/BobbyLLM/llama-conductor
License: AGPL-3.0 — do NOT incorporate code; patterns only

Relevant patterns for Turnstone

Retrieval cascade ordering

llama-conductor enforces a strict source priority before falling back to model inference:

Cheatsheets → Codex (wiki) → Scratchpad → Web → Model weights (last resort)

For Turnstone the equivalent would be:

Log index (exact match) → Known pattern DB → Past diagnoses → LLM inference (last resort)

The key principle: model inference only fires after deterministic sources are exhausted. This prevents the LLM from hallucinating a diagnosis when a real log pattern already exists.

Provenance footer on every response

Every llama-conductor answer includes: Confidence: <tier> | Source: <path>

Turnstone should surface the same — when a diagnosis is returned, the UI should show:

Which log lines triggered it
Whether the match was exact (pattern DB), fuzzy (embedding), or inferred (LLM)
Confidence tier (matched / probable / inferred)

This is directly aligned with the Human Approval step in the CF pipeline and reduces blind trust in LLM output.

Explicit refusal when evidence is absent

llama-conductor refuses to answer rather than speculate when no grounding evidence exists. Turnstone should do the same: if the log pattern is not in the index and the LLM confidence is below threshold, return "insufficient evidence" rather than a low-confidence guess.

Next steps

Map Turnstone's current retrieval path against the cascade model above
Add source and confidence_tier fields to diagnosis response schema
Define confidence tiers: exact_match, pattern_match, llm_inferred, insufficient_evidence
Add refusal/low-confidence path to diagnosis endpoint

## Reference project **GitHub:** https://github.com/BobbyLLM/llama-conductor **License:** AGPL-3.0 — do NOT incorporate code; patterns only ## Relevant patterns for Turnstone ### Retrieval cascade ordering llama-conductor enforces a strict source priority before falling back to model inference: ``` Cheatsheets → Codex (wiki) → Scratchpad → Web → Model weights (last resort) ``` For Turnstone the equivalent would be: ``` Log index (exact match) → Known pattern DB → Past diagnoses → LLM inference (last resort) ``` The key principle: model inference only fires after deterministic sources are exhausted. This prevents the LLM from hallucinating a diagnosis when a real log pattern already exists. ### Provenance footer on every response Every llama-conductor answer includes: `Confidence: <tier> | Source: <path>` Turnstone should surface the same — when a diagnosis is returned, the UI should show: - Which log lines triggered it - Whether the match was exact (pattern DB), fuzzy (embedding), or inferred (LLM) - Confidence tier (matched / probable / inferred) This is directly aligned with the Human Approval step in the CF pipeline and reduces blind trust in LLM output. ### Explicit refusal when evidence is absent llama-conductor refuses to answer rather than speculate when no grounding evidence exists. Turnstone should do the same: if the log pattern is not in the index and the LLM confidence is below threshold, return "insufficient evidence" rather than a low-confidence guess. ## Next steps - [ ] Map Turnstone's current retrieval path against the cascade model above - [ ] Add `source` and `confidence_tier` fields to diagnosis response schema - [ ] Define confidence tiers: `exact_match`, `pattern_match`, `llm_inferred`, `insufficient_evidence` - [ ] Add refusal/low-confidence path to diagnosis endpoint