Add scan_log_directories() to discover.py that recursively walks
/var/log and /opt, filters to readable log files, and scores each
candidate by recency (mtime, 0.7 weight), file size (0.3), and
keyword match against an optional problem-context query (shifts
weights to 0.4/0.2/0.4 when a query is provided).
- GET /api/setup/scan?query=...&max_results=N — new API endpoint
- SourcesView: "Scan" button opens a panel with ranked candidates,
checkboxes, and "Add selected" to write to sources.yaml
- 13 new unit tests, 466 passing total
Closes: #23
Security Alerts:
- Client-side duplicate collapsing via anomaly_label + text fingerprint
- ×N count badge chip on collapsed rows; toggle to expand
- Skeleton shimmer rows replace "Loading..." text
Dashboard:
- Clickable Recent Criticals — inline LLM explanation via SSE stream
- ±5 min time window scoped to source_id for useful context
- Explanation cache keyed by entry_id (no re-fetch on re-expand)
- Default diagnose query injected on Diagnose button navigation to
prevent local models hallucinating from bare log data
- Stat card and source-health skeleton shimmer loading states
Backend:
- anomaly.py: 4-attempt retry on "database is locked" with 10s backoff
- search.py: migrate build_fts_index to get_conn() (WAL race fix);
add timeline_events to stats_summary for clickable criticals feature
- theme.css: @keyframes shimmer + .loading-shimmer utility;
prefers-reduced-motion degrades gracefully to static muted block
- New ChatDiagnose.vue: multi-turn chat UI in the Diagnose tab
- Textarea input (auto-grows) for long free-form problem descriptions
- Source suggestion pre-flight: debounced POST /api/sources/suggest
identifies relevant log sources from the query text and shows them
as interactive chips (deselect to exclude before searching)
- Conversation history preserved across turns with LLM reasoning,
collapsible log entries, and "Save as incident" per turn
- Reuses existing /api/diagnose/stream — no new pipeline
- DiagnoseView.vue: Chat is now default tab; viewport-height layout
- POST /api/sources/suggest: token-overlap source ranking, no LLM
- Fix: add missing 'import re' causing 500 on suggest route
Adds IncidentTimeline.vue — a pure SVG time-axis component rendered inside the
incident detail drawer when entries are present:
- Horizontal strip scaled to incident window (preserveAspectRatio=none)
- Event ticks colored by severity, height proportional to severity level
- 50-bin density shading shows burst periods as blue bands
- Gap markers (dashed lines) for silence > 10% of window or > 60s
- Hover tooltip showing nearest entry's severity, time, and truncated text
- Click-to-scroll: clicking a tick highlights and scrolls to its entry in the list below
- Legend showing only severity levels present in the incident
Also fixes a pre-existing bug: get_incident_endpoint and both build_bundle callers
were passing INCIDENTS_DB_PATH to get_incident_entries/build_bundle, causing all
incident entry lookups to silently search the empty incidents DB instead of the
main log DB. This made all incident detail views show "No log entries found".
Closes: #57
- Fix loadScorerStatus: was spreading data.state + data.config (both
undefined); API returns flat object; now uses data directly
- Fix v-for to use filteredDetections (was using raw detections array,
breaking the Unacknowledged tab filter)
- Fix double-prefix URL bug: BASE already contains /turnstone, so
fetches to ${BASE}/turnstone/api/... doubled the prefix → returned
SPA HTML → silent JSON parse failure. Fixed all fetch URLs to use
${BASE}/api/... in SecurityAlertsView and DashboardView
- Add CybersecStatus interface to replace Record<string, unknown>
- Add scorer field to Detection interface; show 'cybersec' badge in
label cell when scorer !== 'anomaly'
- Add cybersecStatus.running to cybersec badge (pulse animation)
- Add ANOMALY / CYBERSEC stats rows side-by-side
- Add 'Run cybersec' button with cybersecTriggerLoading state and
runCybersec() function posting to /api/cybersec/run
- Rename 'Run scorer' → 'Run anomaly' for clarity
Closes: #11
Second-pass cybersec classifier using DeBERTa-v3-base-mnli (already
cached — no download required). Runs after each anomaly scoring pass on
entries flagged by the anomaly scorer or with pattern matches.
Architecture:
- app/services/cybersec.py: zero-shot-classification pipeline with 5
cybersec candidate labels (auth failure, privilege escalation, network
intrusion, malware, data exfiltration). Writes ml_score/ml_label/
ml_scored_at to log_entries; inserts high-confidence hits into
detections with scorer='cybersec'.
- app/tasks/cybersec_scorer.py: async background task (same shape as
anomaly_scorer.py).
- REST: GET/POST /turnstone/api/cybersec/status|run|detections.
GET /turnstone/api/anomaly/detections now accepts scorer= filter.
Schema: ml_score, ml_label, ml_scored_at added to log_entries; scorer
column added to detections (idempotent migrations + DDL for both SQLite
and Postgres).
UI: Security Alerts view gains Source dropdown (All / Anomaly / Cybersec)
and cybersec scorer status badge. Label dropdown split into optgroups.
Deployment: TURNSTONE_CYBERSEC_MODEL/DEVICE/THRESHOLD vars added to
.env.example, docker-compose.yml, docker-standalone.sh.
Tests: 10 new tests — no model, no eligible entries, scoring, detection
creation, normal label suppression, threshold filtering, pattern-tag
filtering, idempotency, list filtering, scorer column filter.
416/416 passing.
Closes: #9
New SecurityAlertsView (/alerts route) surfaces the detections table built
in #10. Features:
- All / Unacknowledged tab filter with live counts
- Label dropdown (SECURITY_ANOMALY, SYSTEM_FAILURE, NETWORK_ANOMALY, etc.)
- Score confidence bar per detection (colour-coded by threshold)
- Acknowledge drawer: full log text, optional notes, in-place row dim on save
- Scorer status badge + manual "Run scorer" button
- Config warning when TURNSTONE_ANOMALY_MODEL is unset
Dashboard: new "Unreviewed Alerts" stat card (red border when > 0) links
to /alerts so alerts surface on the landing page without navigating away.
Closes: #11
HTML buttons get a ~#efefef background and 2px outset border from the
browser UA stylesheet. In light mode these blend in; in dark mode they
render as stark white boxes. Adding a global button reset in theme.css
clears the UA defaults — explicit bg-* utility classes still win.
Affects: theme toggle, hamburger nav button, dashboard diagnose buttons,
and all other icon/text buttons that had no explicit bg class.
Bumps version to 0.6.2.
- Replace bg-accent/10 with bg-accent-muted on selected radio cards
(opacity modifiers on CSS variable colors are silently dropped by
UnoCSS, causing full-opacity solid blue backgrounds)
- Add explicit left-0.5 to toggle switch thumb and set off-state to
translate-x-0 — without an explicit left the browser auto-placed the
thumb 18px inside the track, causing 14px overflow when translated on
source_ids with 3+ colon segments (e.g. muninn-journal:Muninn:ssh.service)
are now aggregated by their prefix:host key at the SQL level in both
list_sources() and stats_summary(). This collapses ~19K transient systemd
unit rows (crash-loop scope entries from Muninn) into ~24 grouped rows.
- list_sources: SQL CASE/INSTR group-by stem + unit_count field
- stats_summary: same stem grouping for dashboard source health table
- delete endpoint: LIKE-based cascade delete covers grouped stems
- SourcesView: unit_count badge (e.g. "2686 units") on grouped rows;
delete confirmation names the unit count when deleting a group
- Bump version to v0.6.1
Truncation fix: call_llm() in _llm_client.py now accepts max_tokens (default
2048) and passes it in both the cf-orch task payload and the OpenAI-compat
fallback body. Hypothesizer uses max_tokens=1024 (JSON array output);
synthesizer and legacy summarize use 2048 (structured 5-section narrative).
Without this, backends use their own default (often 512 tokens), causing
mid-sentence truncation of the diagnosis output.
UI fix: reasoning card changed from bg-accent/5 border-accent/30 (opacity
modifiers on CSS variables don't compose reliably across themes) to the
callout pattern: bg-surface-raised with a solid border-l-4 border-accent.
Header label changed from text-text-dim to text-accent for visual anchoring.
Text remains text-text-primary for guaranteed contrast on both light and dark
themes.
Tracks: #56 (technical-level post-processor, filed as follow-on feature)
Closes turnstone#22.
## Transport layer (app/glean/ssh.py)
- SSHTransport context manager: key-only auth, paramiko backend
- SSHConnectionError / SSHCommandError exception hierarchy
- exec_stream() generator: yields stdout lines, raises SSHCommandError on
non-zero exit (isinstance(int) guard for test-mock safety)
- Command builders: _build_journald_command, _build_syslog_command,
_build_plaintext_command, _build_docker_command
- 18 unit tests in tests/test_glean_ssh.py
## Pipeline integration (app/glean/pipeline.py)
- _stream_and_write(): per-item error isolation — SSHCommandError skips
one glean item without aborting the rest of the host connection
- _glean_ssh_source(): one SSHTransport per host, dispatches all glean
items (journald/syslog/plaintext/docker); SSHConnectionError aborts host
- glean_sources(): splits local vs SSH sources; local → _glean_files();
SSH → _glean_ssh_source(); shared compiled patterns and DB connection
- glean_ssh_source(): public wrapper for REST use — manages DB connection,
pattern compilation, FTS rebuild lifecycle
- 15 integration tests in tests/test_glean_pipeline_ssh.py
- All 285 tests passing
## REST layer (app/rest.py)
- GET /api/sources/configured: reads sources.yaml and enriches with DB
stats; SSH sources appear before first glean (entry_count=0); sub-source
IDs (rack01/journald, rack01/docker/myapp) aggregated per host entry
- POST /api/sources/{id}/glean: detects transport:ssh and dispatches to
glean_ssh_source() wrapper; local sources unchanged
- Import: glean_ssh_source as _glean_ssh_source
## Frontend (web/src/views/SourcesView.vue)
- Fetches /api/sources/configured (primary) + /api/sources (DB-only) in
parallel; merges into unified SourceRow list
- SSH sources show: ssh badge (with user@host tooltip), glean-type pills
(journald/syslog/docker/etc.), host subtitle
- SSH sub-source IDs (rack01/journald) suppressed from the DB-only list
since they are covered by the parent SSH row
- DB-only sources (uploads) appear below configured sources with 'uploaded'
badge; reglean button disabled (not in sources.yaml)
- Delete zeroes out configured-source stats in-place rather than removing
the row (so the source remains visible for re-gleaning)
overflow-hidden and overflow-x-auto on the same element conflict in Tailwind's
CSS generation order. The shorthand overflow:hidden can override overflow-x:auto,
clipping the rightmost column (diagnose buttons). Fix: outer div keeps
overflow-hidden for rounded corners, inner div handles overflow-x-auto scrolling.
- App: hamburger menu on mobile, nav links hidden below md breakpoint
- LogSearch: collapsible sidebar on mobile, stacks above results vertically
- Incidents/Sources: overflow-x-auto on table containers, min-w to preserve
column layout on desktop; drawer action buttons flex-wrap on small screens
- Bundles: flex-wrap on header row, hide source_host + timestamp below sm
- General: p-4 sm:p-6 padding on all standard views
- wizard.py: wrap syslog_port int() in try/except to default 514 on non-numeric input
- ContextView: add try/catch to doDelete, doDeleteFact, addFact for network errors
- ContextView: arrow-key navigation for tablist (ArrowLeft/ArrowRight)
- DiagnoseView: arrow-key navigation for tablist (ArrowLeft/ArrowRight)
- WizardOverlay: reset current_step to last schema step when clicking 'Go back and edit'
- WizardOverlay: focus trap on Tab/Shift+Tab within dialog element
Adds /context route with tabbed UI for managing uploaded documents and
manually-entered environment facts. Includes inline confirm-before-delete,
add-fact form with category/key/value fields, wizard CTA panel, and
stub components for DocUploadZone and WizardOverlay (Task 14).
- Diagnose: add source_filter param threaded through entries_in_window,
search, _diagnose, and DiagnoseRequest — clicking diagnose on a
dashboard source now scopes both keyword and window hits to that source
- QuickCapture: read route.query.source; show scope badge with clear ✕;
auto-run when source param is present without a query
- DashboardView: pass source= (not q=) when navigating to diagnose
- collect_cluster_logs.sh: auto-discover Docker containers on all nodes
(Heimdall non-watched, Navi, Strahl via SSH); collect Cass Plex logs
via SSH; write to per-node dirs for directory-mode ingest
- turnstone-cluster.service: add --reload for hot-reload during dev
Turnstone now calls /v1/chat/completions instead of Ollama's /api/generate.
This format works with both local Ollama (>=0.1.24) and a remote cf-orch
coordinator, enabling GPU-less nodes like Contributor2's to route diagnoses through
the cluster without any local model.
- llm.py: OpenAI-compat messages format, optional Bearer auth header
- diagnose.py: thread llm_api_key through the call chain
- rest.py: llm_api_key pref (default empty), SettingsBody field, passed to diagnose
- SettingsView.vue: API Key field, label updated from "Ollama URL" to "LLM Endpoint URL"
- tests: updated mocks for new response shape; added bearer token assertion test
Adds background watcher that tails active log sources and ingests entries
in near-real-time, keeping the DB fresh without manual ingest runs.
- app/watch/watcher.py: Watcher + WatchSource using subprocess + select
loop; flushes every 10s or 100 lines; syncs FTS index every 3 flushes
- patterns/watch.yaml: declarative source config (journald/docker/podman)
- app/rest.py: lifespan context manager starts/stops watcher on app
startup/shutdown; GET /api/watch/status + POST /api/watch/reload
- web/src/views/DashboardView.vue: live/manual indicator chip + stale
banner copy adapts to whether live watching is active
- tests/test_watch_watcher.py: 16 tests covering config load, command
building, docker timestamp stripping, orchestrator lifecycle
Closes#4
Turnstone incidents now carry an issue_type tag (free-text with datalist
suggestions) used to categorize patterns for signature building.
Backend:
- Incident model gains issue_type; additive ALTER TABLE migration keeps
existing DBs working without a full schema rebuild
- New received_bundles table stores incoming JSON bundles with indexes on
bundled_at and issue_type
- build_bundle() assembles incident + related log entries into a versioned
bundle dict; store_bundle()/list_bundles()/get_bundle() for the receiver
- POST /api/incidents/{id}/send — pushes bundle to TURNSTONE_BUNDLE_ENDPOINT
- GET /api/incidents/{id}/bundle — export without sending
- POST /api/bundles — receive and store an incoming bundle
- GET /api/bundles — list all received bundles
- TURNSTONE_SOURCE_HOST and TURNSTONE_BUNDLE_ENDPOINT env vars; auto-set
source host from hostname in podman-standalone.sh
Frontend:
- Incidents form: issue_type field with datalist suggestions; Type column
in the table; Send Bundle button + status feedback in the detail drawer
- New BundlesView: collapsible bundle rows, inline JSON parse (no extra
round-trip), Export JSON download button
- Router and nav updated with /bundles route
- Add GET /api/stats endpoint with 24h windowed aggregation (criticals,
errors, per-source health, recent criticals list)
- Fix timestamp format bug: strftime('%Y-%m-%dT%H:%M:%S', ...) to match
stored ISO-8601 T-separated timestamps (datetime('now') uses space)
- Add composite index idx_ts_repeat(timestamp_iso, repeat_count) — drops
stats query from 3.5 s to <1 ms by resolving both WHERE conditions
from the index without table row fetches
- New DashboardView: 3 stat cards, source health table with health dots,
diagnose-per-source button, recent criticals panel, zero-state card
- Router default / → /dashboard; Dashboard first in nav
- DiagnoseView: reads ?q= query param on mount and auto-runs; shows
formatted LLM summary block
- LogEntryRow: expand/collapse for long entries (>200 chars or multiline)
- New IncidentsView.vue: create/list/delete incidents, inline entry drawer
- Time-bucket quick-pick: Ongoing, Just now, Last hour, Last day
- Optional custom datetime-local range picker for precise control
- Incident label doubles as the LLM/FTS search term for entry retrieval
- Entry drawer shows up to 100 associated log entries with severity colour
- Add /incidents route and nav link (between Diagnose and Sources)
