docs: add README — diagnostic log intelligence layer
This commit is contained in:
parent
fc5029c24e
commit
222cc581ce
1 changed files with 158 additions and 0 deletions
158
README.md
Normal file
158
README.md
Normal file
|
|
@ -0,0 +1,158 @@
|
|||
# Turnstone
|
||||
|
||||
> **Diagnostic log intelligence for self-hosted infrastructure.**
|
||||
|
||||
[](https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone)
|
||||
[](LICENSE)
|
||||
[](requirements.txt)
|
||||
|
||||
Turnstone ingests logs from your services, indexes them for full-text and pattern search, and lets you tag incidents, build diagnostic bundles, and query across your infrastructure — from a web UI or an MCP-compatible agent client.
|
||||
|
||||
---
|
||||
|
||||
## What it does
|
||||
|
||||
```
|
||||
Service logs (journald, Docker, syslog, Caddy, Plex, arr stack, qBittorrent, dmesg)
|
||||
→ Ingest pipeline (auto-detect format, parse, deduplicate, pattern-tag)
|
||||
→ SQLite + FTS index
|
||||
→ REST API → Vue web UI / MCP server → agent clients (Orchard)
|
||||
```
|
||||
|
||||
**Human workflow:** Search logs by symptom or time window, create incidents, attach relevant log entries, bundle everything into a diagnostic package for hand-off or archival.
|
||||
|
||||
**Agent workflow:** MCP tools expose search, incident management, and diagnose over a standard protocol — Orchard agents can query Turnstone as part of automated triage and resolution pipelines.
|
||||
|
||||
---
|
||||
|
||||
## Features
|
||||
|
||||
- **Multi-source ingest** — journald, Docker, syslog, Caddy, dmesg, Plex, Servarr (arr stack), qBittorrent, plaintext; paths configured in `patterns/sources.yaml`
|
||||
- **Pattern tagging** — named regex patterns applied at ingest time (`service_restart`, `auth_failure`, `oom`, `segfault`, `disk_full`, `timeout`, …); extend in `patterns/default.yaml`
|
||||
- **Full-text search** — SQLite FTS5 index across all ingested entries; filter by source, severity, time window
|
||||
- **Natural-language time queries** — "what happened yesterday morning", "show me errors from the last 3 hours"; powered by dateparser
|
||||
- **Incident management** — create, label, and track incidents; attach supporting log entries
|
||||
- **Diagnostic bundles** — group log entries + incident metadata into a shareable bundle for escalation or archival
|
||||
- **MCP server** — exposes search, incident, and diagnose tools to MCP-compatible agent clients
|
||||
- **Dark/light theme** — Vue 3 + UnoCSS, system-aware
|
||||
|
||||
---
|
||||
|
||||
## Quick start (Docker)
|
||||
|
||||
```bash
|
||||
git clone https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone.git
|
||||
cd turnstone
|
||||
|
||||
# Edit sources to match your paths
|
||||
cp patterns/sources.yaml.example patterns/sources.yaml
|
||||
$EDITOR patterns/sources.yaml
|
||||
|
||||
docker build -t turnstone:latest .
|
||||
docker run -d --name turnstone \
|
||||
-p 8534:8534 \
|
||||
-v $(pwd)/data:/data \
|
||||
-v $(pwd)/patterns:/patterns \
|
||||
turnstone:latest
|
||||
```
|
||||
|
||||
Open `http://localhost:8534/turnstone/`
|
||||
|
||||
---
|
||||
|
||||
## Quick start (dev)
|
||||
|
||||
```bash
|
||||
# Backend
|
||||
conda run -n cf pip install -r requirements.txt
|
||||
conda run -n cf bash manage.sh start
|
||||
|
||||
# Frontend (separate terminal, hot-reload)
|
||||
cd web && npm install && npm run dev
|
||||
```
|
||||
|
||||
API: `http://localhost:8534/turnstone/docs`
|
||||
UI: `http://localhost:5174/`
|
||||
|
||||
---
|
||||
|
||||
## Deployment (Podman + systemd)
|
||||
|
||||
See [`podman-standalone.sh`](podman-standalone.sh) for rootful Podman setup with systemd unit generation. Suitable for hosts that run system Podman rather than Docker Compose.
|
||||
|
||||
For Caddy reverse-proxy setup (e.g. `menagerie.circuitforge.tech/turnstone`), see [`docs/caddy-routing-pattern.md`](docs/caddy-routing-pattern.md) — all routes are pre-mounted at `/turnstone` so no prefix stripping is needed.
|
||||
|
||||
---
|
||||
|
||||
## Log source configuration
|
||||
|
||||
Edit `patterns/sources.yaml` to tell Turnstone where your logs live (container-side paths):
|
||||
|
||||
```yaml
|
||||
sources:
|
||||
- id: system-journal
|
||||
path: /data/journal-export.jsonl # exported by export_journal.sh on host
|
||||
|
||||
- id: docker-logs
|
||||
path: /var/log/docker # bind-mounted from host
|
||||
|
||||
- id: caddy
|
||||
path: /var/log/caddy/access.log
|
||||
```
|
||||
|
||||
For `journald` sources, run `scripts/export_journal.sh` on the host before each ingest (e.g. via cron). Missing paths are skipped with a warning — safe to leave entries for services that are temporarily down.
|
||||
|
||||
---
|
||||
|
||||
## Pattern library
|
||||
|
||||
Named patterns in `patterns/default.yaml` are matched against every log entry at ingest time. Matched pattern names are stored and used to boost search relevance for diagnostic queries.
|
||||
|
||||
```yaml
|
||||
patterns:
|
||||
- name: oom
|
||||
pattern: "(out of memory|OOM|killed process|cannot allocate)"
|
||||
severity: CRITICAL
|
||||
description: Out-of-memory condition
|
||||
```
|
||||
|
||||
Add domain-specific patterns for your stack. Multiple patterns can match a single entry.
|
||||
|
||||
---
|
||||
|
||||
## MCP server
|
||||
|
||||
Turnstone exposes an MCP (Model Context Protocol) server for agent clients. Start it alongside the REST API:
|
||||
|
||||
```bash
|
||||
conda run -n cf python -m app.mcp_server
|
||||
```
|
||||
|
||||
Tools exposed: `search`, `diagnose`, `create_incident`, `list_incidents`, `build_bundle`.
|
||||
|
||||
---
|
||||
|
||||
## Manage script
|
||||
|
||||
```bash
|
||||
bash manage.sh start # start API (and Vite dev server if --dev)
|
||||
bash manage.sh stop # stop API
|
||||
bash manage.sh restart # restart
|
||||
bash manage.sh status # show process state and port bindings
|
||||
bash manage.sh logs # tail API log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ports
|
||||
|
||||
| Service | Port | Notes |
|
||||
|---------|------|-------|
|
||||
| FastAPI + Vue SPA | `8534` | Production: REST API + built frontend |
|
||||
| Vite HMR | `5174` | Dev only: hot-reload frontend, proxies `/api` → 8534 |
|
||||
|
||||
---
|
||||
|
||||
## License
|
||||
|
||||
Private — CircuitForge internal tooling. Not licensed for redistribution.
|
||||
Loading…
Reference in a new issue