turnstone

Circuit-Forge/turnstone

Fork 0

Commit graph

Author	SHA1	Message	Date
pyr0ball	9ec60ea7ff	feat: syslog and dmesg parsers with graceful journald fallback - Add syslog.py — RFC 3164 parser for /var/log/syslog, /var/log/messages, auth.log, kern.log; ident prepended to message text for searchability - Add dmesg_log.py — handles both relative [secs.usecs] and human-readable [Dow Mon DD HH:MM:SS YYYY] formats; relative timestamps preserved as raw - Wire both into pipeline.py auto-detection (before plaintext fallback) - Update export_journal.sh: checks for journalctl availability, falls back gracefully on non-systemd systems; adds dmesg -T export (falls back to plain dmesg on older kernels) - Add syslog entries (commented) + dmesg source to sources.yaml - 30 tests covering both parsers (detection + parse correctness)	2026-05-11 06:57:38 -07:00
pyr0ball	1b6482701c	feat: journald export + system failure patterns - Add scripts/export_journal.sh — dumps recent journal (priority 0-5, 20min window) to /opt/turnstone/data/journal-export.jsonl; idempotent via entry_id deduplication so overlap is safe - Add system-journal source to sources.yaml pointing at the export file - Add 9 system-level patterns to default.yaml: systemd_fail, oom_kill, disk_hw_error, fs_error, kernel_error, ssh_brute, container_crash, smart_error, nfs_error	2026-05-11 06:54:42 -07:00
pyr0ball	f9691277d8	feat: multi-source ingest via sources.yaml + servarr parser - Add servarr.py parser for all *arr services (sonarr/radarr/lidarr/ prowlarr/readarr/whisparr/bazarr) — pipe-delimited format with component prefix prepended for searchability - Add ingest_sources() to pipeline.py; reads sources.yaml, skips missing paths with a warning so cron keeps running if a service is down - Add --sources mode to ingest_corpus.py CLI; legacy positional args unchanged for backward compat - Add patterns/sources.yaml with all of Xander's discovered service log paths (qbit, 7 servarr services, nzbget, tautulli, jellyseerr) - Replace per-service volume mounts in podman-standalone.sh with /opt:/opt:ro + /var/log:/var/log:ro; adding a new source now requires only editing sources.yaml — no container restart	2026-05-11 06:26:32 -07:00

Author

SHA1

Message

Date

pyr0ball

9ec60ea7ff

feat: syslog and dmesg parsers with graceful journald fallback

- Add syslog.py — RFC 3164 parser for /var/log/syslog, /var/log/messages,
  auth.log, kern.log; ident prepended to message text for searchability
- Add dmesg_log.py — handles both relative [secs.usecs] and human-readable
  [Dow Mon DD HH:MM:SS YYYY] formats; relative timestamps preserved as raw
- Wire both into pipeline.py auto-detection (before plaintext fallback)
- Update export_journal.sh: checks for journalctl availability, falls back
  gracefully on non-systemd systems; adds dmesg -T export (falls back to
  plain dmesg on older kernels)
- Add syslog entries (commented) + dmesg source to sources.yaml
- 30 tests covering both parsers (detection + parse correctness)

2026-05-11 06:57:38 -07:00

pyr0ball

1b6482701c

feat: journald export + system failure patterns

- Add scripts/export_journal.sh — dumps recent journal (priority 0-5,
  20min window) to /opt/turnstone/data/journal-export.jsonl; idempotent
  via entry_id deduplication so overlap is safe
- Add system-journal source to sources.yaml pointing at the export file
- Add 9 system-level patterns to default.yaml:
  systemd_fail, oom_kill, disk_hw_error, fs_error, kernel_error,
  ssh_brute, container_crash, smart_error, nfs_error

2026-05-11 06:54:42 -07:00

pyr0ball

f9691277d8

feat: multi-source ingest via sources.yaml + servarr parser

- Add servarr.py parser for all *arr services (sonarr/radarr/lidarr/
  prowlarr/readarr/whisparr/bazarr) — pipe-delimited format with
  component prefix prepended for searchability
- Add ingest_sources() to pipeline.py; reads sources.yaml, skips
  missing paths with a warning so cron keeps running if a service
  is down
- Add --sources mode to ingest_corpus.py CLI; legacy positional args
  unchanged for backward compat
- Add patterns/sources.yaml with all of Xander's discovered service
  log paths (qbit, 7 servarr services, nzbget, tautulli, jellyseerr)
- Replace per-service volume mounts in podman-standalone.sh with
  /opt:/opt:ro + /var/log:/var/log:ro; adding a new source now
  requires only editing sources.yaml — no container restart

2026-05-11 06:26:32 -07:00

3 commits