# Turnstone

> **Diagnostic log intelligence for self-hosted infrastructure.**

[![Status](https://img.shields.io/badge/status-beta-blue)](https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone)
[![Version](https://img.shields.io/badge/version-0.4.0-green)](https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone/releases)
[![License](https://img.shields.io/badge/license-private-red)](LICENSE)
[![Python](https://img.shields.io/badge/python-3.11%2B-blue)](requirements.txt)

Turnstone ingests logs from your services, indexes them for full-text and pattern search, and lets you tag incidents, build diagnostic bundles, and query across your infrastructure — from a web UI or an MCP-compatible agent client.

---

## What it does

```
Service logs (journald, Docker, syslog, Caddy, Plex, arr stack, qBittorrent, dmesg)
  → Ingest pipeline (auto-detect format, parse, deduplicate, pattern-tag)
  → SQLite + FTS index
  → REST API → Vue web UI  /  MCP server → agent clients (Orchard)
```

**Human workflow:** Search logs by symptom or time window, create incidents, attach relevant log entries, bundle everything into a diagnostic package for hand-off or archival.

**Agent workflow:** MCP tools expose search, incident management, and diagnose over a standard protocol — Orchard agents can query Turnstone as part of automated triage and resolution pipelines.

---

## Features

- **Multi-source ingest** — journald, Docker, syslog, Caddy, dmesg, Plex, Servarr (arr stack), qBittorrent, plaintext; paths configured in `patterns/sources.yaml`
- **Pattern tagging** — named regex patterns applied at ingest time (`service_restart`, `auth_failure`, `oom`, `segfault`, `disk_full`, `timeout`, …); extend in `patterns/default.yaml`
- **Full-text search** — SQLite FTS5 index across all ingested entries; filter by source, severity, time window
- **Natural-language time queries** — "what happened yesterday morning", "show me errors from the last 3 hours"; powered by dateparser
- **Incident management** — create, label, and track incidents; attach supporting log entries
- **Diagnostic bundles** — group log entries + incident metadata into a shareable bundle for escalation or archival
- **MCP server** — exposes search, incident, and diagnose tools to MCP-compatible agent clients
- **Dark/light theme** — Vue 3 + UnoCSS, system-aware

---

## Quick start (Docker)

```bash
git clone https://git.opensourcesolarpunk.com/Circuit-Forge/turnstone.git
cd turnstone

# Edit sources to match your paths
cp patterns/sources.yaml.example patterns/sources.yaml
$EDITOR patterns/sources.yaml

docker build -t turnstone:latest .
docker run -d --name turnstone \
  -p 8534:8534 \
  -v $(pwd)/data:/data \
  -v $(pwd)/patterns:/patterns \
  turnstone:latest
```

Open `http://localhost:8534/turnstone/`

---

## Quick start (dev)

```bash
# Backend
conda run -n cf pip install -r requirements.txt
conda run -n cf bash manage.sh start

# Frontend (separate terminal, hot-reload)
cd web && npm install && npm run dev
```

API: `http://localhost:8534/turnstone/docs`
UI: `http://localhost:5174/`

---

## Deployment (Podman + systemd)

See [`podman-standalone.sh`](podman-standalone.sh) for rootful Podman setup with systemd unit generation. Suitable for hosts that run system Podman rather than Docker Compose.

For Caddy reverse-proxy setup (e.g. `menagerie.circuitforge.tech/turnstone`), see [`docs/caddy-routing-pattern.md`](docs/caddy-routing-pattern.md) — all routes are pre-mounted at `/turnstone` so no prefix stripping is needed.

---

## Log source configuration

Edit `patterns/sources.yaml` to tell Turnstone where your logs live (container-side paths):

```yaml
sources:
  - id: system-journal
    path: /data/journal-export.jsonl   # exported by export_journal.sh on host

  - id: docker-logs
    path: /var/log/docker              # bind-mounted from host

  - id: caddy
    path: /var/log/caddy/access.log
```

For `journald` sources, run `scripts/export_journal.sh` on the host before each ingest (e.g. via cron). Missing paths are skipped with a warning — safe to leave entries for services that are temporarily down.

---

## Pattern library

Named patterns in `patterns/default.yaml` are matched against every log entry at ingest time. Matched pattern names are stored and used to boost search relevance for diagnostic queries.

```yaml
patterns:
  - name: oom
    pattern: "(out of memory|OOM|killed process|cannot allocate)"
    severity: CRITICAL
    description: Out-of-memory condition
```

Add domain-specific patterns for your stack. Multiple patterns can match a single entry.

---

## MCP server

Turnstone exposes an MCP (Model Context Protocol) server for agent clients. Start it alongside the REST API:

```bash
conda run -n cf python -m app.mcp_server
```

Tools exposed: `search`, `diagnose`, `create_incident`, `list_incidents`, `build_bundle`.

---

## Manage script

```bash
bash manage.sh start     # start API (and Vite dev server if --dev)
bash manage.sh stop      # stop API
bash manage.sh restart   # restart
bash manage.sh status    # show process state and port bindings
bash manage.sh logs      # tail API log
```

---

## Ports

| Service | Port | Notes |
|---------|------|-------|
| FastAPI + Vue SPA | `8534` | Production: REST API + built frontend |
| Vite HMR | `5174` | Dev only: hot-reload frontend, proxies `/api` → 8534 |

---

## License

Private — CircuitForge internal tooling. Not licensed for redistribution.