pyr0ball
7a2ab0bb46
Implements the Orchard branch grafting system for harvest.circuitforge.tech:
- POST /api/orchard/graft: provisions data dir, starts a new
turnstone-submissions-<slug> Docker container on the next free port
(ORCHARD_PORT_BASE=8538+), injects a handle_path block into the
Caddyfile dynamic-branches marker section, restarts caddy-proxy,
returns {submit_endpoint, api_key}
- GET /api/orchard/branches: list active/inactive branches (admin-only)
- DELETE /api/orchard/branches/<slug>: deactivate branch + stop container
- POST /api/orchard/branches/<slug>/anonymize: HMAC-based IP/username
pseudonymization worker over a branch DB
- POST /api/glean/batch: optional TURNSTONE_BRANCH_KEY auth guard
- anonymized column added to log_entries schema (migration-safe)
- Updated Caddyfile with /huginn/* route (port 8536), /node2/* (8537),
and dynamic-branch marker section
- All endpoints admin-gated via TURNSTONE_ORCHARD_ADMIN_KEY
Closes: #27
104 lines
5.4 KiB
Text
104 lines
5.4 KiB
Text
# Turnstone environment variables
|
|
# Copy to .env and adjust for your setup. All variables are optional unless noted.
|
|
|
|
# --- Database & paths ---
|
|
# TURNSTONE_DB=/data/turnstone.db
|
|
# TURNSTONE_PATTERNS=/patterns
|
|
# TURNSTONE_SOURCE_HOST=my-server
|
|
|
|
# --- GPU / LLM inference ---
|
|
# GPU_SERVER_URL — URL of your GPU inference server (Ollama, vLLM, or cf-orch coordinator).
|
|
# Paid+ users: leave unset to auto-default to https://orch.circuitforge.tech via CF_LICENSE_KEY.
|
|
# Local Ollama (default if unset): http://localhost:11434
|
|
# Local cf-orch coordinator: http://<YOUR_HOST_IP>:7700
|
|
# CF_ORCH_URL is also accepted as a backward-compatible alias.
|
|
# GPU_SERVER_URL=http://localhost:11434
|
|
|
|
# --- CircuitForge license (Paid+) ---
|
|
# Enables cloud GPU inference and premium features.
|
|
# When set, GPU_SERVER_URL defaults to https://orch.circuitforge.tech automatically.
|
|
# CF_LICENSE_KEY=CFG-TRSN-XXXX-XXXX-XXXX
|
|
|
|
# --- Bundle endpoint (optional) ---
|
|
# Remote endpoint to push diagnostic bundles for escalation.
|
|
# TURNSTONE_BUNDLE_ENDPOINT=https://example.com/api/bundles
|
|
|
|
# --- Log corpus export to Avocet (optional) ---
|
|
# Push ERROR/CRITICAL entries and labeled incidents to the Avocet corpus endpoint
|
|
# for logreading fine-tune training. Requires a consent token issued by CF.
|
|
# Contact alan@circuitforge.tech to register your node and receive a token.
|
|
# Watermarks are stored at data/corpus_watermark.txt and data/incident_watermark.txt.
|
|
# AVOCET_CORPUS_ENDPOINT=https://avocet.circuitforge.tech/api/corpus/log-batch
|
|
# AVOCET_CONSENT_TOKEN=your-uuid-token-here
|
|
# TURNSTONE_SOURCE_HOST=my-server-name # defaults to system hostname if unset
|
|
|
|
# --- Periodic batch glean ---
|
|
# Seconds between automatic glean runs from sources.yaml. Set to 0 to disable.
|
|
# TURNSTONE_GLEAN_INTERVAL=900
|
|
|
|
# --- Multi-agent diagnose pipeline (experimental) ---
|
|
# Enable the 5-stage ML pipeline instead of the single-LLM summarize() call.
|
|
# TURNSTONE_MULTI_AGENT_DIAGNOSE=true
|
|
|
|
# Stage 2 — ML severity classifier (optional; falls back to pattern_tags then regex).
|
|
# Recommended: byviz/bylastic_classification_logs (~300MB, downloaded from HuggingFace)
|
|
# TURNSTONE_CLASSIFIER_MODEL=byviz/bylastic_classification_logs
|
|
|
|
# Stage 4 — Embedding backend for false-positive suppression.
|
|
# sentence_transformers: in-process local model (downloads on first use)
|
|
# ollama: uses a running Ollama instance (no download needed if model is already pulled)
|
|
# TURNSTONE_EMBED_BACKEND=sentence_transformers
|
|
# TURNSTONE_EMBED_MODEL=BAAI/bge-small-en-v1.5
|
|
# TURNSTONE_EMBED_DEVICE=cpu
|
|
|
|
# --- Cybersec scoring pipeline (zero-shot, second-pass on flagged entries) ---
|
|
# Runs a zero-shot classifier on entries already flagged by the anomaly scorer
|
|
# or that have pattern matches — a focused second opinion using cybersec vocabulary.
|
|
# The DeBERTa-v3-base-mnli model (required by the diagnose pipeline) is the recommended
|
|
# zero-shot classifier — it produces human-readable cybersec labels with no fine-tuning.
|
|
# TURNSTONE_CYBERSEC_MODEL=MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli
|
|
# TURNSTONE_CYBERSEC_DEVICE=cpu
|
|
# TURNSTONE_CYBERSEC_THRESHOLD=0.60 # lower than anomaly threshold (zero-shot is calibrated differently)
|
|
|
|
# --- Anomaly scoring pipeline (IDS / watchdog) ---
|
|
# Batch-scores every ingested log entry after each glean cycle.
|
|
# Any HuggingFace text-classification model works; the byviz classifier (already
|
|
# required by the diagnose pipeline) is the recommended starting point.
|
|
# Detections above the threshold are inserted into the detections table and
|
|
# surfaced in the Security Alerts tab.
|
|
#
|
|
# Set TURNSTONE_ANOMALY_MODEL to enable; leave unset to disable (safe default).
|
|
# TURNSTONE_ANOMALY_MODEL=byviz/bylastic_classification_logs
|
|
# TURNSTONE_ANOMALY_DEVICE=cpu # or "cuda" / "mps" for GPU inference
|
|
# TURNSTONE_ANOMALY_THRESHOLD=0.80 # confidence floor for detection insertion
|
|
# TURNSTONE_ANOMALY_INTERVAL=0 # standalone loop (0 = glean-triggered only)
|
|
#
|
|
# HuggingFace model cache — share with the host to avoid re-downloading models.
|
|
# HF_HOME=/hf_cache # inside container (set in docker-compose)
|
|
# HF_CACHE_PATH=/Library/Assets/LLM # host bind-mount source (docker-compose only)
|
|
|
|
# --- Air-gapped / offline deployment ---
|
|
# Set to 1 to block all HuggingFace hub network access at runtime.
|
|
# Pre-download models to ~/.cache/huggingface/ before deploying — see docs/air-gapped-deployment.md.
|
|
# TURNSTONE_OFFLINE_MODE=1
|
|
|
|
# --- API authentication ---
|
|
# When set, all /api/ requests require: Authorization: Bearer <token>
|
|
# Generate a token: python -c "import secrets; print(secrets.token_urlsafe(32))"
|
|
# TURNSTONE_API_KEY=your-secret-token-here
|
|
|
|
# --- The Orchard (harvest receiver only) ---
|
|
# Set on the central harvest.circuitforge.tech instance to enable branch management.
|
|
# TURNSTONE_ORCHARD_ADMIN_KEY=your-admin-secret-here
|
|
# TURNSTONE_ORCHARD_DATA_ROOT=/devl/docker/turnstone-submissions
|
|
# TURNSTONE_ORCHARD_CADDYFILE=/devl/caddy-proxy/Caddyfile
|
|
# TURNSTONE_ORCHARD_CADDY_CONTAINER=caddy-proxy
|
|
# TURNSTONE_ORCHARD_HARVEST_HOST=https://harvest.circuitforge.tech
|
|
# TURNSTONE_ORCHARD_PORT_BASE=8538
|
|
# TURNSTONE_ORCHARD_IMAGE=localhost/turnstone:latest
|
|
|
|
# --- Orchard branch (submitting node) ---
|
|
# Set TURNSTONE_SUBMIT_ENDPOINT to push pattern-matched log entries to the harvest receiver.
|
|
# Generate your branch slug and API key via: POST /api/orchard/graft on the harvest instance.
|
|
# TURNSTONE_SUBMIT_ENDPOINT=https://harvest.circuitforge.tech/your-slug
|
|
# TURNSTONE_BRANCH_KEY=api-key-from-graft-response
|