chore: enumerate log sources to harvest from huginn

pyr0ball commented

2026-05-27 08:50:49 -07:00

Owner

Hey @nimdah — Turnstone (the log aggregation + LLM diagnosis tool running on Heimdall) can pull logs from Huginn via SSH and ingest them into the cluster database. Since Huginn is the WireGuard entry point and public edge proxy for opensourcesolarpunk.com, it's particularly worth watching for tunnel failures, Caddy errors, and ddclient DDNS events.

Here's a starter list based on Huginn's known role. Please review, fill in paths, and add anything we're missing.

Suggested sources for Huginn

# huginn (Daniel) log sources for Turnstone SSH collection
# Review and fill in actual paths / remove anything not applicable.

sources:
  # ── System journal (catch-all: systemd failures, WireGuard events, OOM) ──────
  - id: huginn-journal
    path: /devl/turnstone-cluster/data/huginn-journal.jsonl
    # Collected via: ssh huginn journalctl --output=json --priority=0..5 --since '20 minutes ago'
    # Key things to watch: wg-quick@ failures, systemd unit restarts, OOM events

  # ── Caddy (public edge proxy for opensourcesolarpunk.com) ────────────────────
  - id: huginn-caddy-access
    path: /devl/turnstone-cluster/data/huginn/caddy-access.log
    # Caddy structured access log — confirm path (typically /var/log/caddy/access.log
    # or output to stdout via journald: journalctl -u caddy)
    # We already have a Caddy parser for structured JSON output.

  - id: huginn-caddy-error
    path: /devl/turnstone-cluster/data/huginn/caddy-error.log
    # Caddy error log if written to file, otherwise captured in huginn-journal via journald

  # ── ddclient (Dynamic DNS for the public IP) ──────────────────────────────────
  - id: huginn-ddclient
    path: /devl/turnstone-cluster/data/huginn/ddclient.log
    # ddclient typically logs to /var/log/ddclient.log or journald (ddclient.service)
    # Watching for: failed updates, IP change events, auth failures

  # ── WireGuard tunnel ─────────────────────────────────────────────────────────
  # WireGuard events are in the system journal (wg-quick@wg0.service).
  # Captured automatically via huginn-journal above — no separate source needed
  # unless you want a dedicated WireGuard log view.

  # ── Docker containers (if any running on Huginn) ─────────────────────────────
  # List any Docker containers you're running and their relevant log paths:
  #
  # - id: huginn-<service-name>
  #   path: /devl/turnstone-cluster/data/huginn/<service-name>.jsonl
  #   # docker logs huginn/<container-name> --since 20m

What to fill in

Caddy — does it log to a file or only to journald? If file, what's the path?
ddclient — does it log to /var/log/ddclient.log or journald?
Docker containers — what's running on Huginn? List any services and we'll add them.
Anything else on the machine worth watching (nginx, fail2ban, unbound, etc.)?

Notes

The WireGuard tunnel events are already captured in the system journal source — we have vpn_tunnel_fail and vpn_handshake patterns that will fire on them.
Huginn is reachable via ssh huginn when the WireGuard VPN is up. If the VPN is down, Turnstone will skip it with a warning (same as other unreachable nodes).
Alan will add Huginn to the SSH collect timer once you've confirmed the list.

Hey @nimdah — Turnstone (the log aggregation + LLM diagnosis tool running on Heimdall) can pull logs from Huginn via SSH and ingest them into the cluster database. Since Huginn is the WireGuard entry point and public edge proxy for `opensourcesolarpunk.com`, it's particularly worth watching for tunnel failures, Caddy errors, and ddclient DDNS events. Here's a starter list based on Huginn's known role. Please review, fill in paths, and add anything we're missing. --- ## Suggested sources for Huginn ```yaml # huginn (Daniel) log sources for Turnstone SSH collection # Review and fill in actual paths / remove anything not applicable. sources: # ── System journal (catch-all: systemd failures, WireGuard events, OOM) ────── - id: huginn-journal path: /devl/turnstone-cluster/data/huginn-journal.jsonl # Collected via: ssh huginn journalctl --output=json --priority=0..5 --since '20 minutes ago' # Key things to watch: wg-quick@ failures, systemd unit restarts, OOM events # ── Caddy (public edge proxy for opensourcesolarpunk.com) ──────────────────── - id: huginn-caddy-access path: /devl/turnstone-cluster/data/huginn/caddy-access.log # Caddy structured access log — confirm path (typically /var/log/caddy/access.log # or output to stdout via journald: journalctl -u caddy) # We already have a Caddy parser for structured JSON output. - id: huginn-caddy-error path: /devl/turnstone-cluster/data/huginn/caddy-error.log # Caddy error log if written to file, otherwise captured in huginn-journal via journald # ── ddclient (Dynamic DNS for the public IP) ────────────────────────────────── - id: huginn-ddclient path: /devl/turnstone-cluster/data/huginn/ddclient.log # ddclient typically logs to /var/log/ddclient.log or journald (ddclient.service) # Watching for: failed updates, IP change events, auth failures # ── WireGuard tunnel ───────────────────────────────────────────────────────── # WireGuard events are in the system journal (wg-quick@wg0.service). # Captured automatically via huginn-journal above — no separate source needed # unless you want a dedicated WireGuard log view. # ── Docker containers (if any running on Huginn) ───────────────────────────── # List any Docker containers you're running and their relevant log paths: # # - id: huginn-<service-name> # path: /devl/turnstone-cluster/data/huginn/<service-name>.jsonl # # docker logs huginn/<container-name> --since 20m ``` --- ## What to fill in 1. **Caddy** — does it log to a file or only to journald? If file, what's the path? 2. **ddclient** — does it log to `/var/log/ddclient.log` or journald? 3. **Docker containers** — what's running on Huginn? List any services and we'll add them. 4. **Anything else** on the machine worth watching (nginx, fail2ban, unbound, etc.)? ## Notes - The **WireGuard tunnel** events are already captured in the system journal source — we have `vpn_tunnel_fail` and `vpn_handshake` patterns that will fire on them. - Huginn is reachable via `ssh huginn` when the WireGuard VPN is up. If the VPN is down, Turnstone will skip it with a warning (same as other unreachable nodes). - Alan will add Huginn to the SSH collect timer once you've confirmed the list.

pyr0ball added this to the beta milestone 2026-05-27 08:50:49 -07:00

pyr0ball added the

deployment

docs

labels 2026-05-27 08:50:49 -07:00

nimdah was assigned by pyr0ball

2026-05-27 08:50:49 -07:00

pyr0ball changed title from ~~chore: enumerate log sources to harvest from huginn (Daniel)~~ to chore: enumerate log sources to harvest from huginn

2026-06-14 13:43:25 -07:00

pyr0ball referenced this issue

2026-06-14 15:17:15 -07:00

chore: enumerate log sources to harvest from example-node #54

pyr0ball commented

2026-06-14 15:17:15 -07:00

Author

Owner

Resolved by the live Orchard setup. Huginn is now submitting to harvest.circuitforge.tech/huginn (#55). The example-node (node2) has its sources enumerated in patterns/sources-example.yaml and the compose setup in docker-compose.submissions.yml (#54). The graft API (#27) handles future node enrollment automatically.

Resolved by the live Orchard setup. Huginn is now submitting to harvest.circuitforge.tech/huginn (#55). The example-node (node2) has its sources enumerated in `patterns/sources-example.yaml` and the compose setup in `docker-compose.submissions.yml` (#54). The graft API (#27) handles future node enrollment automatically.

pyr0ball closed this issue

2026-06-14 15:17:15 -07:00

chore: enumerate log sources to harvest from huginn #55

Suggested sources for Huginn

What to fill in

Notes