turnstone/app/glean
pyr0ball aa3f9311db feat: SSH remote glean — transport layer, pipeline integration, REST + UI (#22)
Closes turnstone#22.

## Transport layer (app/glean/ssh.py)
- SSHTransport context manager: key-only auth, paramiko backend
- SSHConnectionError / SSHCommandError exception hierarchy
- exec_stream() generator: yields stdout lines, raises SSHCommandError on
  non-zero exit (isinstance(int) guard for test-mock safety)
- Command builders: _build_journald_command, _build_syslog_command,
  _build_plaintext_command, _build_docker_command
- 18 unit tests in tests/test_glean_ssh.py

## Pipeline integration (app/glean/pipeline.py)
- _stream_and_write(): per-item error isolation — SSHCommandError skips
  one glean item without aborting the rest of the host connection
- _glean_ssh_source(): one SSHTransport per host, dispatches all glean
  items (journald/syslog/plaintext/docker); SSHConnectionError aborts host
- glean_sources(): splits local vs SSH sources; local → _glean_files();
  SSH → _glean_ssh_source(); shared compiled patterns and DB connection
- glean_ssh_source(): public wrapper for REST use — manages DB connection,
  pattern compilation, FTS rebuild lifecycle
- 15 integration tests in tests/test_glean_pipeline_ssh.py
- All 285 tests passing

## REST layer (app/rest.py)
- GET /api/sources/configured: reads sources.yaml and enriches with DB
  stats; SSH sources appear before first glean (entry_count=0); sub-source
  IDs (rack01/journald, rack01/docker/myapp) aggregated per host entry
- POST /api/sources/{id}/glean: detects transport:ssh and dispatches to
  glean_ssh_source() wrapper; local sources unchanged
- Import: glean_ssh_source as _glean_ssh_source

## Frontend (web/src/views/SourcesView.vue)
- Fetches /api/sources/configured (primary) + /api/sources (DB-only) in
  parallel; merges into unified SourceRow list
- SSH sources show: ssh badge (with user@host tooltip), glean-type pills
  (journald/syslog/docker/etc.), host subtitle
- SSH sub-source IDs (rack01/journald) suppressed from the DB-only list
  since they are covered by the parent SSH row
- DB-only sources (uploads) appear below configured sources with 'uploaded'
  badge; reglean button disabled (not in sources.yaml)
- Delete zeroes out configured-source stats in-place rather than removing
  the row (so the source remains visible for re-gleaning)
2026-05-21 12:37:30 -07:00
..
__init__.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
base.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
caddy.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
dmesg_log.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
doc_upload.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
docker_log.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
journald.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
mqtt_subscriber.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
pipeline.py feat: SSH remote glean — transport layer, pipeline integration, REST + UI (#22) 2026-05-21 12:37:30 -07:00
plaintext.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
plex.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
qbittorrent.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
servarr.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
ssh.py feat: SSH remote host glean — transport layer and pipeline integration (closes #22, backend) 2026-05-20 23:03:13 -07:00
syslog.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
tautulli.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00
wazuh.py refactor: rename ingest → glean throughout codebase 2026-05-20 23:02:55 -07:00