pyr0ball
346ea6e0c6
- Add syslog.py — RFC 3164 parser for /var/log/syslog, /var/log/messages, auth.log, kern.log; ident prepended to message text for searchability - Add dmesg_log.py — handles both relative [secs.usecs] and human-readable [Dow Mon DD HH:MM:SS YYYY] formats; relative timestamps preserved as raw - Wire both into pipeline.py auto-detection (before plaintext fallback) - Update export_journal.sh: checks for journalctl availability, falls back gracefully on non-systemd systems; adds dmesg -T export (falls back to plain dmesg on older kernels) - Add syslog entries (commented) + dmesg source to sources.yaml - 30 tests covering both parsers (detection + parse correctness)
72 lines
2.8 KiB
YAML
72 lines
2.8 KiB
YAML
# Turnstone log sources — edit this file to add or remove services.
|
|
# NOTE: the system-journal entry requires export_journal.sh to run on the HOST
|
|
# before the container ingest step. See crontab setup instructions in the README.
|
|
# Run ingest manually:
|
|
# sudo podman exec turnstone python scripts/ingest_corpus.py \
|
|
# --sources /patterns/sources.yaml --db /data/turnstone.db
|
|
#
|
|
# Paths here are container-side paths under the /opt bind mount.
|
|
# Missing paths are skipped with a warning — safe to leave entries for
|
|
# services that are temporarily down.
|
|
|
|
sources:
|
|
# ── System (exported by export_journal.sh on the host) ───────────────────
|
|
# journal-export.jsonl and dmesg-export.txt are written to /opt/turnstone/data/
|
|
# by the export script before each ingest run.
|
|
- id: system-journal
|
|
path: /data/journal-export.jsonl
|
|
|
|
- id: dmesg
|
|
path: /data/dmesg-export.txt
|
|
|
|
# ── Syslog / rsyslog (direct file reads via /var/log bind mount) ──────────
|
|
# Uncomment the file(s) present on your system.
|
|
# Debian/Ubuntu:
|
|
# - id: syslog
|
|
# path: /var/log/syslog
|
|
# - id: auth-log
|
|
# path: /var/log/auth.log
|
|
# - id: kern-log
|
|
# path: /var/log/kern.log
|
|
# RHEL/Fedora/Rocky:
|
|
# - id: messages
|
|
# path: /var/log/messages
|
|
# - id: secure
|
|
# path: /var/log/secure
|
|
|
|
# ── Download ─────────────────────────────────────────────────────────────
|
|
- id: qbittorrent
|
|
path: /opt/qbittorrent/config/data/logs/qbittorrent.log
|
|
|
|
# ── Servarr stack ─────────────────────────────────────────────────────────
|
|
- id: sonarr
|
|
path: /opt/sonarr/config/logs/sonarr.0.txt
|
|
|
|
- id: radarr
|
|
path: /opt/radarr/config/logs/radarr.0.txt
|
|
|
|
- id: lidarr
|
|
path: /opt/lidarr/config/logs/Lidarr.0.txt
|
|
|
|
- id: readarr
|
|
path: /opt/readarr/config/logs/readarr.0.txt
|
|
|
|
- id: whisparr
|
|
path: /opt/whisparr/config/logs/whisparr.0.txt
|
|
|
|
- id: prowlarr
|
|
path: /opt/prowlarr/config/logs/prowlarr.0.txt
|
|
|
|
- id: bazarr
|
|
path: /opt/bazarr/config/log/bazarr.log
|
|
|
|
# ── Usenet ────────────────────────────────────────────────────────────────
|
|
- id: nzbget
|
|
path: /opt/nzbget/config/nzbget.log
|
|
|
|
# ── Media / Requests ─────────────────────────────────────────────────────
|
|
- id: tautulli
|
|
path: /opt/tautulli/config/logs/tautulli.log
|
|
|
|
- id: jellyseerr
|
|
path: /opt/jellyseerr/config/logs/jellyseerr.log
|