Turnstone

Diagnostic log intelligence for self-hosted infrastructure.

Turnstone ingests logs from your services, indexes them for full-text and pattern search, and lets you tag incidents, build diagnostic bundles, and query across your infrastructure — from a web UI or an MCP-compatible agent client.

---

What it does

Service logs (journald, Docker, syslog, Caddy, Plex, arr stack, qBittorrent, dmesg)
  → Ingest pipeline (auto-detect format, parse, deduplicate, pattern-tag)
  → SQLite + FTS index
  → REST API → Vue web UI  /  MCP server → agent clients (Orchard)

Human workflow: Search logs by symptom or time window, create incidents, attach relevant log entries, bundle everything into a diagnostic package for hand-off or archival.

Agent workflow: MCP tools expose search, incident management, and diagnose over a standard protocol — Orchard agents can query Turnstone as part of automated triage and resolution pipelines.

---

Features

• Multi-source glean — journald, Docker, syslog, Caddy, dmesg, Plex, Servarr (arr stack), qBittorrent, plaintext; paths configured in patterns/sources.yaml
• Pattern tagging — named regex patterns applied at glean time (service_restart, auth_failure, oom, segfault, disk_full, timeout, …); extend in patterns/default.yaml
• Full-text search — SQLite FTS5 index across all ingested entries; filter by source, severity, time window
• Natural-language time queries — "what happened yesterday morning", "show me errors from the last 3 hours"; powered by dateparser
• Incident management — create, label, and track incidents; attach supporting log entries
• Diagnostic bundles — group log entries + incident metadata into a shareable bundle for escalation or archival
• MCP server — exposes search, incident, and diagnose tools to MCP-compatible agent clients
• Dark/light theme — Vue 3 + UnoCSS, system-aware

---

Quick start (Docker)

git clone https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone.git
cd turnstone

# Edit sources to match your paths
cp patterns/sources.yaml.example patterns/sources.yaml
$EDITOR patterns/sources.yaml

docker build -t turnstone:latest .
docker run -d --name turnstone \
  -p 8534:8534 \
  -v $(pwd)/data:/data \
  -v $(pwd)/patterns:/patterns \
  turnstone:latest

Open http://localhost:8534/turnstone/

---

Quick start (dev)

# Backend
conda run -n cf pip install -r requirements.txt
conda run -n cf bash manage.sh start

# Frontend (separate terminal, hot-reload)
cd web && npm install && npm run dev

API: http://localhost:8534/turnstone/docs UI: http://localhost:5174/

---

Deployment (Podman + systemd)

See podman-standalone.sh for rootful Podman setup with systemd unit generation. Suitable for hosts that run system Podman rather than Docker Compose.

For Caddy reverse-proxy setup (e.g. menagerie.circuitforge.tech/turnstone), see docs/caddy-routing-pattern.md — all routes are pre-mounted at /turnstone so no prefix stripping is needed.

---

Log source configuration

Edit patterns/sources.yaml to tell Turnstone where your logs live (container-side paths):

sources:
  - id: system-journal
    path: /data/journal-export.jsonl   # exported by export_journal.sh on host

  - id: docker-logs
    path: /var/log/docker              # bind-mounted from host

  - id: caddy
    path: /var/log/caddy/access.log

For journald sources, run scripts/export_journal.sh on the host before each glean (e.g. via cron). Missing paths are skipped with a warning — safe to leave entries for services that are temporarily down.

---

Pattern library

Named patterns in patterns/default.yaml are matched against every log entry at glean time. Matched pattern names are stored and used to boost search relevance for diagnostic queries.

patterns:
  - name: oom
    pattern: "(out of memory|OOM|killed process|cannot allocate)"
    severity: CRITICAL
    description: Out-of-memory condition

Add domain-specific patterns for your stack. Multiple patterns can match a single entry.

---

MCP server

Turnstone exposes an MCP (Model Context Protocol) server for agent clients. Start it alongside the REST API:

conda run -n cf python -m app.mcp_server

Tools exposed: search, diagnose, create_incident, list_incidents, build_bundle.

---

Manage script

bash manage.sh start     # start API (and Vite dev server if --dev)
bash manage.sh stop      # stop API
bash manage.sh restart   # restart
bash manage.sh status    # show process state and port bindings
bash manage.sh logs      # tail API log

---

Configuration

Copy .env.example to .env (or pass as -e flags to Docker/Podman). All variables are optional.

Variable	Default	Description
GPU_SERVER_URL	http://localhost:11434	GPU inference server (Ollama, vLLM, or cf-orch). CF_ORCH_URL is accepted as a backward-compat alias. Paid+ users: leave unset — auto-defaults to https://orch.circuitforge.tech when CF_LICENSE_KEY is present.
CF_LICENSE_KEY	—	CircuitForge Paid+ license key. Enables cloud GPU inference and premium features.
TURNSTONE_DB	/data/turnstone.db	Path to the SQLite database.
TURNSTONE_PATTERNS	./patterns	Pattern directory (default.yaml, sources.yaml, watch.yaml).
TURNSTONE_SOURCE_HOST	unknown	Host identifier stamped on ingested entries.
TURNSTONE_BUNDLE_ENDPOINT	—	Remote URL to push diagnostic bundles for escalation.
TURNSTONE_GLEAN_INTERVAL	900	Seconds between automatic batch glean runs. Set to 0 to disable.

---

Ports

Service	Port	Notes
FastAPI + Vue SPA	8534	Production: REST API + built frontend
Vite HMR	5174	Dev only: hot-reload frontend, proxies /api → 8534

---

License

Private — CircuitForge internal tooling. Not licensed for redistribution.