fix: split incidents tables to dedicated turnstone-incidents.db (#60) #61

Closed

pyr0ball wants to merge 0 commits from feat/60-incidents-db into main

pyr0ball commented

2026-06-01 16:07:42 -07:00

Owner

Summary

Moves incidents, received_bundles, sent_bundles out of turnstone.db into a dedicated turnstone-incidents.db file
Eliminates FTS5 write lock contention that was starving the incident API during log bursts
Mirrors the existing turnstone-context.db split pattern

Changes

glean/pipeline.py — add _INCIDENTS_SCHEMA, ensure_incidents_schema(), migrate_incidents_to_dedicated_db(); stub out moved tables in _SCHEMA for backward compat
rest.py — add INCIDENTS_DB_PATH / TURNSTONE_INCIDENTS_DB env var; run migration on startup; pass INCIDENTS_DB_PATH to all incident/bundle endpoints
diagnose/__init__.py + diagnose/pipeline.py + diagnose/suppressor.py — thread incidents_db_path through the suppressor call chain
tests/test_blocklist_endpoints.py — patch CONTEXT_DB_PATH and INCIDENTS_DB_PATH in fixtures (worktree has no data/ dir)

Test plan

372 tests passing locally
Restart Turnstone on Heimdall and confirm migration log line appears
Create an incident via API while a log burst is in progress — confirm no 500/lock error
Verify data/turnstone-incidents.db created on first start

## Summary - Moves `incidents`, `received_bundles`, `sent_bundles` out of `turnstone.db` into a dedicated `turnstone-incidents.db` file - Eliminates FTS5 write lock contention that was starving the incident API during log bursts - Mirrors the existing `turnstone-context.db` split pattern ## Changes - `glean/pipeline.py` — add `_INCIDENTS_SCHEMA`, `ensure_incidents_schema()`, `migrate_incidents_to_dedicated_db()`; stub out moved tables in `_SCHEMA` for backward compat - `rest.py` — add `INCIDENTS_DB_PATH` / `TURNSTONE_INCIDENTS_DB` env var; run migration on startup; pass `INCIDENTS_DB_PATH` to all incident/bundle endpoints - `diagnose/__init__.py` + `diagnose/pipeline.py` + `diagnose/suppressor.py` — thread `incidents_db_path` through the suppressor call chain - `tests/test_blocklist_endpoints.py` — patch `CONTEXT_DB_PATH` and `INCIDENTS_DB_PATH` in fixtures (worktree has no `data/` dir) ## Test plan - [ ] 372 tests passing locally - [ ] Restart Turnstone on Heimdall and confirm migration log line appears - [ ] Create an incident via API while a log burst is in progress — confirm no 500/lock error - [ ] Verify `data/turnstone-incidents.db` created on first start

pyr0ball added 1 commit 2026-06-01 16:07:43 -07:00

fix: split incidents tables to dedicated turnstone-incidents.db (#60 ) 5b151c2509

FTS5 bulk-insert write locks starved the incident API and bundle endpoints
during log bursts (sonarr/radarr, high-volume docker sources). Fix mirrors
the context_facts split (context -> turnstone-context.db):

- Add INCIDENTS_DB_PATH / TURNSTONE_INCIDENTS_DB env var in rest.py
- Add _INCIDENTS_SCHEMA, ensure_incidents_schema(), and
  migrate_incidents_to_dedicated_db() in glean/pipeline.py
- Stub out incidents/received_bundles/sent_bundles in _SCHEMA (no-op
  CREATE IF NOT EXISTS) so legacy single-file deployments still open
- Thread incidents_db_path through diagnose_stream -> run_pipeline ->
  FalsePositiveSuppressor.suppress -> _fetch_resolved_incidents
- One-shot migration on startup: copy existing rows from main DB to
  incidents DB via INSERT OR IGNORE (idempotent, safe to re-run)
- Fix test_blocklist_endpoints fixtures to patch CONTEXT_DB_PATH and
  INCIDENTS_DB_PATH alongside DB_PATH (worktree has no data/ dir)

372 tests passing.

Closes: #60