Commit graph

98 commits

Author SHA1 Message Date
82977f365b feat: periodic ingest scheduler + Orchard submission pipeline
Adds asyncio-native background scheduler (TURNSTONE_INGEST_INTERVAL,
default 900s) that runs batch ingest then pushes pattern-matched entries
to a remote CF harvest endpoint (TURNSTONE_SUBMIT_ENDPOINT).

- app/tasks/ingest_scheduler.py: IngestState, scheduler_loop, run_once,
  submit_matched, _query_matched_since — asyncio.Lock prevents concurrent runs
- app/rest.py: POST /api/ingest/batch (pre-parsed entry receiver),
  GET /api/tasks/ingest/status, POST /api/tasks/ingest (manual trigger),
  TURNSTONE_INGEST_INTERVAL + TURNSTONE_SUBMIT_ENDPOINT env wiring in lifespan
- docker-compose.submissions.yml: segregated daniel (8536) + xander (8537)
  receiving instances on Heimdall, isolated DBs under
  /devl/docker/turnstone-submissions/<node>/
- podman-standalone.sh: pass-through for TURNSTONE_SUBMIT_ENDPOINT +
  TURNSTONE_SOURCE_HOST
- app/ingest/mqtt_subscriber.py: MQTT log source adapter
- app/ingest/wazuh.py: Wazuh alert JSON adapter
- tests/test_ingest_wazuh.py: Wazuh adapter test suite
2026-05-20 08:57:25 -07:00
f09de1c28a fix: make sqlite-vec download non-fatal in Dockerfile 2026-05-19 13:02:15 -07:00
95d7c141ca fix: use curl instead of wget for sqlite-vec download in Dockerfile 2026-05-19 13:01:45 -07:00
16fe5f70a5 feat: Alpha milestone — corpus management, upload ingest, harvester agent
Closes #1 (incident tagging — already implemented), #2, #3, #5.

- feat(api): DELETE /api/sources/{id} — purge entries + FTS rows for a source
- feat(api): POST /api/sources/{id}/ingest — re-ingest from sources.yaml
- feat(api): POST /api/ingest/upload — multipart log file upload with auto-detect
- feat(ui): SourcesView reingest + delete buttons and upload file input (#2)
- feat(harvester): harvester.py push + incident subcommands (#5)
- feat(harvester): Dockerfile, docker-compose.yml, harvester.sh (containerless)
- feat(config): GPU_SERVER_URL → CF_ORCH_URL resolution + write-back (#20)
- docs: .env.example, README Configuration table, version bump to 0.5.0
2026-05-19 07:45:58 -07:00
c40e79bb9e docs: bump version badge to match latest Forgejo release 2026-05-17 11:19:13 -07:00
560eaf706d fix(ui): nested overflow wrapper to prevent overflow-hidden clipping table columns
overflow-hidden and overflow-x-auto on the same element conflict in Tailwind's
CSS generation order. The shorthand overflow:hidden can override overflow-x:auto,
clipping the rightmost column (diagnose buttons). Fix: outer div keeps
overflow-hidden for rounded corners, inner div handles overflow-x-auto scrolling.
2026-05-16 09:11:42 -07:00
f5132d5a5c feat(ui): mobile fixes for Dashboard and Diagnose views
- DashboardView: p-4 sm:p-6 padding, overflow-x-auto on source health table
- DiagnoseView: p-4 sm:p-6 padding
- QuickCapture: px-4 sm:px-6 + shrink-0 on Search button to avoid input squeeze
2026-05-16 09:04:37 -07:00
1538a3cf69 feat(ui): mobile responsive layout
- App: hamburger menu on mobile, nav links hidden below md breakpoint
- LogSearch: collapsible sidebar on mobile, stacks above results vertically
- Incidents/Sources: overflow-x-auto on table containers, min-w to preserve
  column layout on desktop; drawer action buttons flex-wrap on small screens
- Bundles: flex-wrap on header row, hide source_host + timestamp below sm
- General: p-4 sm:p-6 padding on all standard views
2026-05-16 02:11:58 -07:00
9052939ae1 fix(blocklist): render llm_score, fix load() error handling, fix severity override mutations
- BlocklistView: display llm_score/llm_reason when non-null (spec gap)
- BlocklistView: set scanError on non-ok load() response (was silent)
- SettingsView: replace in-place splice/property mutation with immutable
  spread pattern in toggleOverride/deleteOverride
2026-05-16 01:57:18 -07:00
171fa578eb feat(blocklist): BlocklistView + Pi-hole settings UI 2026-05-15 21:23:03 -07:00
7f63f155e2 fix(blocklist): get_candidate for O(1) push/unblock, 400 on malformed device_names JSON 2026-05-15 21:19:02 -07:00
e44c6fd680 feat(blocklist): 6 REST endpoints + Pi-hole settings fields
Add blocklist candidate listing, scan trigger, status update,
push/unblock to Pi-hole, and connection test endpoints.
Add pihole_url/version/api_key and router_source_ids/device_names
fields to SettingsBody and prefs handling in patch_settings.
Add PiholeClient.__post_init__ validation so 503 fires naturally
when url/api_key are unconfigured (mock-safe: bypassed in tests).
2026-05-15 21:15:09 -07:00
c813832cbe feat(blocklist): extraction scan + candidate CRUD + full test suite 2026-05-15 21:05:49 -07:00
0e887837d1 fix(blocklist): validate _v6_auth session JSON, add auth-failure test 2026-05-15 21:03:03 -07:00
a683297d8b feat(blocklist): Pi-hole v5/v6 API client + tests
PiholeClient dataclass supporting both Pi-hole v5 (PHP /admin/api.php)
and v6 (REST /api/) with public block/unblock/test_connection methods.
9 tests covering both API versions, auth flow, and error handling.
2026-05-15 21:00:01 -07:00
1a3c753093 fix(blocklist): remove premature imports from blocklist.py (Task 2 scope) 2026-05-15 20:58:04 -07:00
8832061de2 feat(blocklist): telemetry YAML list + loader + domain matcher
Adds patterns/telemetry.yaml with 6 rule groups (samsung, belkin, roku, lg, amazon, advertising).
Adds app/services/blocklist.py with TelemetryRule and BlocklistCandidate dataclasses, load_telemetry_rules(), and matches_telemetry() with exact and subdomain matching.
6 new TestTelemetry tests pass; 199 total passing.
2026-05-15 20:54:40 -07:00
2967036503 feat(blocklist): blocklist_candidates schema + tests
Add blocklist_candidates table and indexes to _SCHEMA in pipeline.py.
Add TestSchema tests verifying table existence, column set, and status/hit_count defaults.
All 193 tests pass.
2026-05-15 20:51:00 -07:00
1fb726bfc1 docs: update status badge to beta 2026-05-15 20:13:47 -07:00
9e5c5da7e9 chore: remove stale load_patterns import from rest.py 2026-05-13 21:52:03 -07:00
950a854b58 fix: tautulli — hmac token compare, public pattern loader, startup cache, endpoint tests 2026-05-13 19:08:49 -07:00
72800332c9 fix: tautulli — entry_id collision on missing ts, token settings, test coverage 2026-05-13 19:04:07 -07:00
b61a85dc62 feat: Tautulli webhook ingest endpoint — plex events -> log_entries
POST /turnstone/api/ingest/tautulli accepts Tautulli notification agent
payloads and stores them as log_entries under source 'tautulli'. Severity
maps error->CRITICAL, buffer->WARN, all others->None. Optional bearer token
auth via X-Tautulli-Token header + tautulli_token pref. FTS index rebuilt
as a background task after each write. 28 new tests, all passing.
2026-05-13 18:41:03 -07:00
63af5aa14b fix: time window regex misses fuzzy quantifiers like 'last few hours'
The relative-time regex only matched digits between 'last/past' and
the unit, so 'last few hours' fell through to dateparser which then
found the bare word 'hours' and resolved it as midnight local time.

Extended the regex to capture 'few', 'couple of', 'several', 'a few'
as approximate quantifiers, mapped to 3 units each. Numeric expressions
and bare 'last hour' still work as before.
2026-05-13 18:32:54 -07:00
32f44700f9 fix: ingestors treat naive log timestamps as local time, not UTC
All five parsers (plex, syslog, servarr, qbittorrent, plaintext) were
using .replace(tzinfo=timezone.utc) on naive datetimes parsed from log
files, which slaps a UTC label on what is actually local-time data.
On a UTC-7 system a 2pm entry was stored as 14:00Z instead of 21:00Z,
causing time-window searches to return zero results.

Fix: use .astimezone(timezone.utc) instead, which treats the naive
datetime as local time and converts correctly.

Tests updated to round-trip back to local time for assertion so they
pass on any timezone, not just UTC.
2026-05-13 18:16:33 -07:00
e6075f80b3 fix: final review fixes — port guard, network error handling, wizard back nav, tablist arrow keys, dialog focus trap
- wizard.py: wrap syslog_port int() in try/except to default 514 on non-numeric input
- ContextView: add try/catch to doDelete, doDeleteFact, addFact for network errors
- ContextView: arrow-key navigation for tablist (ArrowLeft/ArrowRight)
- DiagnoseView: arrow-key navigation for tablist (ArrowLeft/ArrowRight)
- WizardOverlay: reset current_step to last schema step when clicking 'Go back and edit'
- WizardOverlay: focus trap on Tab/Shift+Tab within dialog element
2026-05-13 17:40:40 -07:00
7895366d67 fix: drag flicker guard, error body parsing, wizard session restore answer 2026-05-13 17:07:56 -07:00
e0bb4f0d8d feat: WizardOverlay and DocUploadZone — accessible multi-step wizard and upload UI 2026-05-13 17:04:15 -07:00
5f25d9a350 fix: add error handling to context doc/fact load functions 2026-05-13 17:00:29 -07:00
e8a1e2d77d feat: Context view — document and fact management with accessible tables
Adds /context route with tabbed UI for managing uploaded documents and
manually-entered environment facts. Includes inline confirm-before-delete,
add-fact form with category/key/value fields, wizard CTA panel, and
stub components for DocUploadZone and WizardOverlay (Task 14).
2026-05-13 16:57:38 -07:00
2fbf623f02 fix: a11y — tab panels v-show, radio roving-tabindex, table header label 2026-05-13 16:53:41 -07:00
b41ca4910a fix: a11y — tablist, health dots, table headers, switch roles, nav landmark 2026-05-13 16:48:38 -07:00
5d6ecfea62 fix: a11y — QuickCapture label/role/aria-live/spinner, LogEntryRow expand button 2026-05-13 16:42:46 -07:00
82d3d37790 feat: optional sqlite-vec embedding pipeline for Paid-tier RAG 2026-05-13 16:32:57 -07:00
074240c061 feat: context REST API — docs, facts, wizard, and debug endpoints
Wires the context/RAG layer into FastAPI via a dedicated _ctx router
(/turnstone/api/context/*): document upload (POST/GET/DELETE /docs),
fact CRUD (POST/GET/DELETE /facts), wizard state machine
(/wizard/schema, /wizard/step, /wizard/apply), and a debug search
endpoint (/debug/search). All blocking DB calls are dispatched via
asyncio.to_thread to keep the event loop free.
2026-05-13 16:31:07 -07:00
f19f896300 feat: inject environment context into diagnose pipeline and LLM prompt
- Add context_block param to summarize() and thread it into _PROMPT_TEMPLATE
- Wire retrieve_context/format_context_block into diagnose_stream() before
  log search; emit context SSE event (facts + chunks) to the client
- 3 new tests covering prompt injection and SSE event emission (155 total, all pass)
2026-05-13 16:29:26 -07:00
abb61a6e90 feat: wizard state machine — structured Q&A writes context facts and source config 2026-05-13 16:25:52 -07:00
de662725ee feat: context retriever — keyword fact lookup and chunk search 2026-05-13 16:23:54 -07:00
d7b892bfcf feat: doc upload adapter — writes facts, document, and chunks to context store 2026-05-13 16:21:55 -07:00
c17bbf6e26 feat: context chunker — type detection, YAML extraction, text chunking
- Implement document type detection for yaml/json/markdown/text
- Extract service facts from docker-compose YAML (names, images, ports)
- Split text into overlapping word chunks (300-word default with 50-word overlap)
- Enforce 5 MB file size limit
- Comprehensive TDD test suite: 15 tests passing
2026-05-13 15:54:51 -07:00
36c9e607b7 feat: context store — fact and document CRUD 2026-05-13 15:53:03 -07:00
2a2f2e311a feat: add context_facts, context_documents, context_chunks tables to schema 2026-05-13 15:51:19 -07:00
067d565894 fix: a11y foundation — text-dim contrast, focus-visible, prefers-reduced-motion 2026-05-13 15:48:12 -07:00
734e81c8ca feat: SSE streaming diagnose, severity filter pills, per-source-cap search
- diagnose_stream() async generator: status/summary/entries/reasoning/done events
- POST /api/diagnose/stream SSE endpoint wired in rest.py
- entries_in_window() gains per_source_cap to prevent high-volume sources crowding results
- QuickCapture: severity filter pills, filtered entries view, pipeline status spinner
- llm.py: remove overly broad HTTPStatusError re-raise
2026-05-13 15:45:35 -07:00
909bb3f78b feat: try cf-orch task endpoint first; fall back to direct model call
POST /api/inference/task with product=turnstone task=log_analysis routes to
the security reasoning model assigned in cf-orch. Falls back to the OpenAI-
compat /v1/chat/completions path on 404 (no assignment) or if the task
endpoint is absent (local instances, xanderland).
2026-05-13 08:20:29 -07:00
b88c6d7ebf feat: source-scoped diagnose; multi-node Docker log collection
- Diagnose: add source_filter param threaded through entries_in_window,
  search, _diagnose, and DiagnoseRequest — clicking diagnose on a
  dashboard source now scopes both keyword and window hits to that source
- QuickCapture: read route.query.source; show scope badge with clear ✕;
  auto-run when source param is present without a query
- DashboardView: pass source= (not q=) when navigating to diagnose
- collect_cluster_logs.sh: auto-discover Docker containers on all nodes
  (Heimdall non-watched, Navi, Strahl via SSH); collect Cass Plex logs
  via SSH; write to per-node dirs for directory-mode ingest
- turnstone-cluster.service: add --reload for hot-reload during dev
2026-05-13 08:10:42 -07:00
03b796eb6e fix: correct cf-orch port to 7700; fix relative time parsing in diagnose; fix syslog PRI prefix 2026-05-13 05:33:41 -07:00
0923f40071 fix: write ingest log to data dir (alan lacks /var/log write access) 2026-05-13 05:20:56 -07:00
2f07bd2c0e fix: run collect service as alan user; call ingest directly without Docker 2026-05-13 05:17:43 -07:00
d80d4875db feat: add UDP syslog receiver for network device log collection
scripts/syslog_receiver.py: asyncio UDP server listening on port 5140,
appends raw syslog lines to network-syslog.txt for the Turnstone live
watcher to tail. Requires no root — port 5140 is non-privileged.

scripts/turnstone-syslog-receiver.service: systemd unit for auto-start.

app/ingest/syslog.py: strip optional RFC 3164 <PRI> prefix before
parsing so network-forwarded syslog (OpenWRT logd, Arista EOS, etc.)
is handled correctly without the PRI value breaking the regex.
2026-05-13 04:58:51 -07:00