feat: technical-level post-processor for diagnosis output

pyr0ball commented

2026-05-27 22:23:18 -07:00

Owner

The LLM synthesis output is a structured sysadmin diagnosis (VERDICT / TIMELINE / ROOT CAUSES / RECOMMENDED ACTIONS / INVESTIGATE FURTHER). This is appropriate for a technical operator, but can be hard to parse for a homelab user, a support engineer, or someone who didn't configure the system.

Add a user-configurable technical level setting that reframes the diagnosis output for the operator's background.

Proposed levels

Level	Audience	Framing
`homelab`	Self-hosted enthusiast, comfortable with Linux basics	Plain English, service names spelled out, actions concrete and copy-pasteable
`sysadmin`	Professional sysadmin / DevOps (default)	Technical, current structured format, no hand-holding
`executive`	Non-technical stakeholder	Jargon-free impact summary: what broke, what it affects, what was done

Implementation options

Option A — adjusted system prompt (one LLM call)
Pass technical_level into SummarySynthesizer._SYSTEM_PROMPT and instruct the model to write at the appropriate level. Simpler, but the technical analysis is shaped from the start.

Option B — post-processing pass (two LLM calls)
Synthesizer always generates the full technical diagnosis. A second "translator" call rewrites it for the configured level. Slower, but the technical version is preserved (useful for sysadmin-level review even if the operator prefers homelab framing).

Settings integration

Add technical_level: str = "sysadmin" to the preferences JSON (data/prefs.json)
Expose it in the Settings panel UI (radio group or select)
Pass from rest.py → diagnose_stream → run_pipeline → SummarySynthesizer.synthesize
Label in Settings: "Diagnosis language" or "Explanation level"

Acceptance criteria

Homelab level: output contains no unexplained acronyms, uses plain English service names (e.g. "your download client" not "qBittorrent"), and each action item is copy-pasteable
Sysadmin level: current structured format unchanged
Executive level: ≤4 sentences, no hostnames, no command syntax
Level persists across sessions (saved in prefs)
Changing level in settings takes effect on next diagnose run (no reload needed)

The LLM synthesis output is a structured sysadmin diagnosis (VERDICT / TIMELINE / ROOT CAUSES / RECOMMENDED ACTIONS / INVESTIGATE FURTHER). This is appropriate for a technical operator, but can be hard to parse for a homelab user, a support engineer, or someone who didn't configure the system. Add a user-configurable technical level setting that reframes the diagnosis output for the operator's background. ## Proposed levels | Level | Audience | Framing | |-------|----------|---------| | `homelab` | Self-hosted enthusiast, comfortable with Linux basics | Plain English, service names spelled out, actions concrete and copy-pasteable | | `sysadmin` | Professional sysadmin / DevOps (default) | Technical, current structured format, no hand-holding | | `executive` | Non-technical stakeholder | Jargon-free impact summary: what broke, what it affects, what was done | ## Implementation options **Option A — adjusted system prompt (one LLM call)** Pass `technical_level` into `SummarySynthesizer._SYSTEM_PROMPT` and instruct the model to write at the appropriate level. Simpler, but the technical analysis is shaped from the start. **Option B — post-processing pass (two LLM calls)** Synthesizer always generates the full technical diagnosis. A second "translator" call rewrites it for the configured level. Slower, but the technical version is preserved (useful for sysadmin-level review even if the operator prefers homelab framing). ## Settings integration - Add `technical_level: str = "sysadmin"` to the preferences JSON (`data/prefs.json`) - Expose it in the Settings panel UI (radio group or select) - Pass from `rest.py` → `diagnose_stream` → `run_pipeline` → `SummarySynthesizer.synthesize` - Label in Settings: "Diagnosis language" or "Explanation level" ## Acceptance criteria - Homelab level: output contains no unexplained acronyms, uses plain English service names (e.g. "your download client" not "qBittorrent"), and each action item is copy-pasteable - Sysadmin level: current structured format unchanged - Executive level: ≤4 sentences, no hostnames, no command syntax - Level persists across sessions (saved in prefs) - Changing level in settings takes effect on next diagnose run (no reload needed)