pyr0ball
63c742a708
Adds asyncio-native background scheduler (TURNSTONE_INGEST_INTERVAL, default 900s) that runs batch ingest then pushes pattern-matched entries to a remote CF harvest endpoint (TURNSTONE_SUBMIT_ENDPOINT). - app/tasks/ingest_scheduler.py: IngestState, scheduler_loop, run_once, submit_matched, _query_matched_since — asyncio.Lock prevents concurrent runs - app/rest.py: POST /api/ingest/batch (pre-parsed entry receiver), GET /api/tasks/ingest/status, POST /api/tasks/ingest (manual trigger), TURNSTONE_INGEST_INTERVAL + TURNSTONE_SUBMIT_ENDPOINT env wiring in lifespan - docker-compose.submissions.yml: segregated contrib1 (8536) + contrib2 (8537) receiving instances on Heimdall, isolated DBs under /devl/docker/turnstone-submissions/<node>/ - podman-standalone.sh: pass-through for TURNSTONE_SUBMIT_ENDPOINT + TURNSTONE_SOURCE_HOST - app/ingest/mqtt_subscriber.py: MQTT log source adapter - app/ingest/wazuh.py: Wazuh alert JSON adapter - tests/test_ingest_wazuh.py: Wazuh adapter test suite
51 lines
1.5 KiB
YAML
51 lines
1.5 KiB
YAML
# Turnstone Harvester — sources.example.yaml
|
|
# Copy to sources.yaml and adjust paths for your system.
|
|
# The harvester reads this file and POSTs each log file to Turnstone.
|
|
#
|
|
# Each source needs:
|
|
# id: Short identifier (used as source_id in Turnstone)
|
|
# path: Absolute path to the log file on the host
|
|
|
|
sources:
|
|
# System journal (export with: journalctl -o json-pretty > /var/log/journal-export.jsonl)
|
|
# - id: system-journal
|
|
# path: /var/log/journal-export.jsonl
|
|
|
|
# Syslog
|
|
- id: syslog
|
|
path: /var/log/syslog
|
|
|
|
# Docker daemon log
|
|
# - id: docker
|
|
# path: /var/log/docker.log
|
|
|
|
# Podman events (rootful)
|
|
# - id: podman
|
|
# path: /var/log/podman-events.log
|
|
|
|
# Caddy access log
|
|
# - id: caddy
|
|
# path: /var/log/caddy/access.log
|
|
|
|
# Arr stack — adjust container paths to match your setup
|
|
# - id: sonarr
|
|
# path: /opt/sonarr/config/logs/sonarr.0.txt
|
|
# - id: radarr
|
|
# path: /opt/radarr/config/logs/radarr.0.txt
|
|
# - id: prowlarr
|
|
# path: /opt/prowlarr/config/logs/prowlarr.0.txt
|
|
|
|
# qBittorrent
|
|
# - id: qbittorrent
|
|
# path: /opt/qbittorrent/config/data/logs/qbittorrent.log
|
|
|
|
# Jellyfin
|
|
# - id: jellyfin
|
|
# path: /opt/jellyfin/log/jellyfin.log
|
|
|
|
# Wazuh SIEM — alerts.json on the Wazuh manager
|
|
# Turnstone auto-detects this format; source_id is qualified per agent automatically.
|
|
# For push-based ingestion from Wazuh custom integrations, use:
|
|
# POST /api/ingest/wazuh/alert (single alert JSON body)
|
|
# - id: wazuh
|
|
# path: /var/ossec/logs/alerts/alerts.json
|