Commit graph

7 commits

Author SHA1 Message Date
286778d6a9 feat: journald export + system failure patterns
- Add scripts/export_journal.sh — dumps recent journal (priority 0-5,
  20min window) to /opt/turnstone/data/journal-export.jsonl; idempotent
  via entry_id deduplication so overlap is safe
- Add system-journal source to sources.yaml pointing at the export file
- Add 9 system-level patterns to default.yaml:
  systemd_fail, oom_kill, disk_hw_error, fs_error, kernel_error,
  ssh_brute, container_crash, smart_error, nfs_error
2026-05-11 06:54:42 -07:00
f46aba8165 feat: multi-source ingest via sources.yaml + servarr parser
- Add servarr.py parser for all *arr services (sonarr/radarr/lidarr/
  prowlarr/readarr/whisparr/bazarr) — pipe-delimited format with
  component prefix prepended for searchability
- Add ingest_sources() to pipeline.py; reads sources.yaml, skips
  missing paths with a warning so cron keeps running if a service
  is down
- Add --sources mode to ingest_corpus.py CLI; legacy positional args
  unchanged for backward compat
- Add patterns/sources.yaml with all of Contributor2's discovered service
  log paths (qbit, 7 servarr services, nzbget, tautulli, jellyseerr)
- Replace per-service volume mounts in podman-standalone.sh with
  /opt:/opt:ro + /var/log:/var/log:ro; adding a new source now
  requires only editing sources.yaml — no container restart
2026-05-11 06:26:32 -07:00
f6b261280c feat: plain-text and Plex log ingestors
- app/ingest/plex.py: Plex Media Server log parser
  Regex-based line parser for 'Mon DD, YYYY HH:MM:SS.mmm [pid] LEVEL - msg'
  format. Handles multi-line entries (stack traces). Detects plex_eae_failure
  and all other patterns via shared pattern library.
- app/ingest/plaintext.py: generic fallback parser for unrecognized formats
  Extracts timestamps (ISO 8601, syslog, common log) and severity via regex.
- pipeline.py: detect plex format via is_plex_log(); fall back to plaintext
  instead of skipping; process *.log files alongside *.jsonl; add ingest_file()
  for single-file ingestion.
- scripts/ingest_corpus.py: accept single file or directory as target
- manage.sh: ingest-plex command SSHes to Cass (or HOST arg), pulls
  Plex Media Server.log, and ingests it directly
2026-05-08 17:50:01 -07:00
1f5854e90b chore: standardize manage.sh, remove start_dev.sh
- manage.sh: start/stop/restart/status/logs/open/dev/ingest/build-fts/test
  following avocet pattern (PID files, colored output, native processes)
- start mode: builds Vue SPA, uvicorn on :8534, python http.server on :8535
- dev mode: uvicorn --reload + Vite HMR, trap cleanup on exit
- scripts/start_dev.sh: removed (superseded by manage.sh dev)
2026-05-08 16:58:12 -07:00
eef84d55be feat: Vue 3 frontend and FastAPI REST layer
- app/rest.py: FastAPI app wrapping search/diagnose/sources with CORS
- web/: Vue 3 + Vite + UnoCSS + Pinia frontend at port 8535
  - LogSearchView: sidebar filters (source, severity, limit) + FTS search
  - DiagnoseView: layered symptom investigation matching MCP diagnose tool
  - SourcesView: corpus table with entry count, error count, time range
  - LogEntryRow: severity badge, pattern chips, repeat count, timestamp
  - StatusDot: live API health indicator in nav
- scripts/start_dev.sh: launch FastAPI (:8534) + Vite dev server (:8535)
- .gitignore: add web/node_modules/ and web/dist/
- Caddy: /turnstone* route added to menagerie.circuitforge.tech block
  (API → :8534 with /turnstone strip, SPA fallback → :8535)
2026-05-08 16:27:59 -07:00
3431be5bfa feat: plex EAE watchdog and plex_eae_failure pattern
Add plex_eae_failure pattern to default.yaml targeting the EasyAudioEncoder
crash signature (EAE timeout + I/O error pair, 5s cadence). Pattern fires
when EAE's WAV handoff files stop appearing in the pms temp directory.

Add watch_plex.py: tail-based watchdog that counts EAE timeout events and
auto-restarts plexmediaserver after N consecutive hits (default 3, ~15s of
failure). Includes cooldown, dry-run mode, and a systemd unit template.
2026-05-08 13:41:34 -07:00
bbe4b1e360 feat: initial Turnstone POC — ingest, FTS search, MCP server
Ingest pipeline (journald / Caddy / Docker-wrapped formats) with
per-source state tracking (repeat dedup, out-of-order detection),
named pattern tagging at ingest time, and idempotent SHA1-keyed writes.

FTS5 search layer with porter stemmer, severity/source/pattern/time
filters, and BM25 ranking. MCP server (FastMCP stdio) with three tools:
search_logs, diagnose, list_log_sources — compatible with both
Claude Code and Copilot CLI.

WAL mode enabled on all connections. FTS index auto-built after ingest.
MCP configs included for Claude Code (.mcp.json) and Copilot CLI
(.github/copilot/mcp.json).
2026-05-08 12:12:34 -07:00