159 lines
5.4 KiB
Markdown
159 lines
5.4 KiB
Markdown
# Turnstone
|
|
|
|
> **Diagnostic log intelligence for self-hosted infrastructure.**
|
|
|
|
[](https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone)
|
|
[](https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone/releases)
|
|
[](LICENSE)
|
|
[](requirements.txt)
|
|
|
|
Turnstone ingests logs from your services, indexes them for full-text and pattern search, and lets you tag incidents, build diagnostic bundles, and query across your infrastructure — from a web UI or an MCP-compatible agent client.
|
|
|
|
---
|
|
|
|
## What it does
|
|
|
|
```
|
|
Service logs (journald, Docker, syslog, Caddy, Plex, arr stack, qBittorrent, dmesg)
|
|
→ Ingest pipeline (auto-detect format, parse, deduplicate, pattern-tag)
|
|
→ SQLite + FTS index
|
|
→ REST API → Vue web UI / MCP server → agent clients (Orchard)
|
|
```
|
|
|
|
**Human workflow:** Search logs by symptom or time window, create incidents, attach relevant log entries, bundle everything into a diagnostic package for hand-off or archival.
|
|
|
|
**Agent workflow:** MCP tools expose search, incident management, and diagnose over a standard protocol — Orchard agents can query Turnstone as part of automated triage and resolution pipelines.
|
|
|
|
---
|
|
|
|
## Features
|
|
|
|
- **Multi-source ingest** — journald, Docker, syslog, Caddy, dmesg, Plex, Servarr (arr stack), qBittorrent, plaintext; paths configured in `patterns/sources.yaml`
|
|
- **Pattern tagging** — named regex patterns applied at ingest time (`service_restart`, `auth_failure`, `oom`, `segfault`, `disk_full`, `timeout`, …); extend in `patterns/default.yaml`
|
|
- **Full-text search** — SQLite FTS5 index across all ingested entries; filter by source, severity, time window
|
|
- **Natural-language time queries** — "what happened yesterday morning", "show me errors from the last 3 hours"; powered by dateparser
|
|
- **Incident management** — create, label, and track incidents; attach supporting log entries
|
|
- **Diagnostic bundles** — group log entries + incident metadata into a shareable bundle for escalation or archival
|
|
- **MCP server** — exposes search, incident, and diagnose tools to MCP-compatible agent clients
|
|
- **Dark/light theme** — Vue 3 + UnoCSS, system-aware
|
|
|
|
---
|
|
|
|
## Quick start (Docker)
|
|
|
|
```bash
|
|
git clone https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone.git
|
|
cd turnstone
|
|
|
|
# Edit sources to match your paths
|
|
cp patterns/sources.yaml.example patterns/sources.yaml
|
|
$EDITOR patterns/sources.yaml
|
|
|
|
docker build -t turnstone:latest .
|
|
docker run -d --name turnstone \
|
|
-p 8534:8534 \
|
|
-v $(pwd)/data:/data \
|
|
-v $(pwd)/patterns:/patterns \
|
|
turnstone:latest
|
|
```
|
|
|
|
Open `http://localhost:8534/turnstone/`
|
|
|
|
---
|
|
|
|
## Quick start (dev)
|
|
|
|
```bash
|
|
# Backend
|
|
conda run -n cf pip install -r requirements.txt
|
|
conda run -n cf bash manage.sh start
|
|
|
|
# Frontend (separate terminal, hot-reload)
|
|
cd web && npm install && npm run dev
|
|
```
|
|
|
|
API: `http://localhost:8534/turnstone/docs`
|
|
UI: `http://localhost:5174/`
|
|
|
|
---
|
|
|
|
## Deployment (Podman + systemd)
|
|
|
|
See [`podman-standalone.sh`](podman-standalone.sh) for rootful Podman setup with systemd unit generation. Suitable for hosts that run system Podman rather than Docker Compose.
|
|
|
|
For Caddy reverse-proxy setup (e.g. `menagerie.circuitforge.tech/turnstone`), see [`docs/caddy-routing-pattern.md`](docs/caddy-routing-pattern.md) — all routes are pre-mounted at `/turnstone` so no prefix stripping is needed.
|
|
|
|
---
|
|
|
|
## Log source configuration
|
|
|
|
Edit `patterns/sources.yaml` to tell Turnstone where your logs live (container-side paths):
|
|
|
|
```yaml
|
|
sources:
|
|
- id: system-journal
|
|
path: /data/journal-export.jsonl # exported by export_journal.sh on host
|
|
|
|
- id: docker-logs
|
|
path: /var/log/docker # bind-mounted from host
|
|
|
|
- id: caddy
|
|
path: /var/log/caddy/access.log
|
|
```
|
|
|
|
For `journald` sources, run `scripts/export_journal.sh` on the host before each ingest (e.g. via cron). Missing paths are skipped with a warning — safe to leave entries for services that are temporarily down.
|
|
|
|
---
|
|
|
|
## Pattern library
|
|
|
|
Named patterns in `patterns/default.yaml` are matched against every log entry at ingest time. Matched pattern names are stored and used to boost search relevance for diagnostic queries.
|
|
|
|
```yaml
|
|
patterns:
|
|
- name: oom
|
|
pattern: "(out of memory|OOM|killed process|cannot allocate)"
|
|
severity: CRITICAL
|
|
description: Out-of-memory condition
|
|
```
|
|
|
|
Add domain-specific patterns for your stack. Multiple patterns can match a single entry.
|
|
|
|
---
|
|
|
|
## MCP server
|
|
|
|
Turnstone exposes an MCP (Model Context Protocol) server for agent clients. Start it alongside the REST API:
|
|
|
|
```bash
|
|
conda run -n cf python -m app.mcp_server
|
|
```
|
|
|
|
Tools exposed: `search`, `diagnose`, `create_incident`, `list_incidents`, `build_bundle`.
|
|
|
|
---
|
|
|
|
## Manage script
|
|
|
|
```bash
|
|
bash manage.sh start # start API (and Vite dev server if --dev)
|
|
bash manage.sh stop # stop API
|
|
bash manage.sh restart # restart
|
|
bash manage.sh status # show process state and port bindings
|
|
bash manage.sh logs # tail API log
|
|
```
|
|
|
|
---
|
|
|
|
## Ports
|
|
|
|
| Service | Port | Notes |
|
|
|---------|------|-------|
|
|
| FastAPI + Vue SPA | `8534` | Production: REST API + built frontend |
|
|
| Vite HMR | `5174` | Dev only: hot-reload frontend, proxies `/api` → 8534 |
|
|
|
|
---
|
|
|
|
## License
|
|
|
|
Private — CircuitForge internal tooling. Not licensed for redistribution.
|