chore: compliance audit checklist -- data isolation, audit logging, PII egress #48

New issue

Closed

opened 2026-05-26 23:05:26 -07:00 by pyr0ball · 1 comment

pyr0ball commented

2026-05-26 23:05:26 -07:00

Owner

Before Turnstone is deployed to a compliance-sensitive environment, verify and document the following:

Data isolation:

Log entries from different sources are queryable in isolation (source filter enforced in all API endpoints)
No cross-source data leakage in FTS5 queries
SQLite DB file permissions (readable only by the Turnstone process user)

Audit logging:

All API queries logged with timestamp, endpoint, query string (no log body -- just metadata)
Glean operations logged with source ID, entry count, timestamp
Errors logged with enough context for post-hoc audit without PII in message field

PII egress:

LLM diagnose calls: review what fields are sent to the LLM provider (no raw log messages unless explicitly acknowledged by operator)
Avocet harvester: confirm only pattern-tagged entries are exported, and export can be disabled
No external telemetry, analytics, or crash reporting calls

Output: Completed checklist committed to docs/compliance/checklist.md.

Before Turnstone is deployed to a compliance-sensitive environment, verify and document the following: **Data isolation:** - [ ] Log entries from different sources are queryable in isolation (source filter enforced in all API endpoints) - [ ] No cross-source data leakage in FTS5 queries - [ ] SQLite DB file permissions (readable only by the Turnstone process user) **Audit logging:** - [ ] All API queries logged with timestamp, endpoint, query string (no log body -- just metadata) - [ ] Glean operations logged with source ID, entry count, timestamp - [ ] Errors logged with enough context for post-hoc audit without PII in message field **PII egress:** - [ ] LLM diagnose calls: review what fields are sent to the LLM provider (no raw log messages unless explicitly acknowledged by operator) - [ ] Avocet harvester: confirm only pattern-tagged entries are exported, and export can be disabled - [ ] No external telemetry, analytics, or crash reporting calls **Output:** Completed checklist committed to `docs/compliance/checklist.md`.

pyr0ball added this to the Enterprise POC Deliverable milestone 2026-05-26 23:05:26 -07:00

pyr0ball added the

security

docs

compliance

labels 2026-05-26 23:05:26 -07:00

pyr0ball commented

2026-05-28 10:39:38 -07:00

Author

Owner

Implemented:

Audit logging middleware (_audit_middleware in app/rest.py):

Logs every /turnstone/api/* request: method, path, query string (no body), status code, duration ms
Named logger turnstone.audit hooked to uvicorn's error handler at startup
Static files and health endpoint are excluded

Compliance checklist at docs/compliance/checklist.md:

All criteria from the issue verified against current code
Data isolation: ✅ source_filter enforced in all query paths
Audit logging: ✅ middleware + glean scheduler
PII egress: ✅ multi-agent path (not multi-agent legacy path — documented as known limitation)
No external telemetry: ✅ confirmed
GDPR anonymization disclaimer added: anonymized exports cannot be selectively deleted per right-to-erasure

Note: ! sudo kill -9 1551899 1554109 1697462 needed on Heimdall to clear stale uvicorn instances before clean restart.

Implemented: **Audit logging middleware** (`_audit_middleware` in `app/rest.py`): - Logs every `/turnstone/api/*` request: method, path, query string (no body), status code, duration ms - Named logger `turnstone.audit` hooked to uvicorn's error handler at startup - Static files and health endpoint are excluded **Compliance checklist** at `docs/compliance/checklist.md`: - All criteria from the issue verified against current code - Data isolation: ✅ source_filter enforced in all query paths - Audit logging: ✅ middleware + glean scheduler - PII egress: ✅ multi-agent path (not multi-agent legacy path — documented as known limitation) - No external telemetry: ✅ confirmed - GDPR anonymization disclaimer added: anonymized exports cannot be selectively deleted per right-to-erasure Note: `! sudo kill -9 1551899 1554109 1697462` needed on Heimdall to clear stale uvicorn instances before clean restart.

pyr0ball closed this issue

2026-05-28 10:39:39 -07:00