feat: hybrid BM25 + vector re-ranking for diagnose search (#15) #63

Closed

pyr0ball wants to merge 0 commits from feat/15-hybrid-rag into main

pyr0ball commented

2026-06-01 18:16:48 -07:00

Owner

Summary

Adds _hybrid_search() to search.py: late-fusion BM25 + cosine re-ranking with alpha=0.6 / beta=0.4 weights
search(semantic=True) dispatches to hybrid path; pure BM25 remains the default
diagnose_stream() enables semantic=True so symptom queries surface semantically equivalent entries (ECONNREFUSED, backend gone away, max retries exceeded)
/api/search?semantic=true exposes the hybrid path on the REST endpoint
Graceful fallback to BM25 when embedder unavailable

Test plan

383 tests passing (372 + 11 new)
Run diagnose "database connection failed" — confirm ECONNREFUSED entries surface
Confirm Turnstone starts without sentence-transformers (BM25 fallback path)
Confirm ?semantic=true on /api/search works with BAAI/bge-small-en-v1.5 loaded

## Summary - Adds `_hybrid_search()` to `search.py`: late-fusion BM25 + cosine re-ranking with alpha=0.6 / beta=0.4 weights - `search(semantic=True)` dispatches to hybrid path; pure BM25 remains the default - `diagnose_stream()` enables semantic=True so symptom queries surface semantically equivalent entries (ECONNREFUSED, backend gone away, max retries exceeded) - `/api/search?semantic=true` exposes the hybrid path on the REST endpoint - Graceful fallback to BM25 when embedder unavailable ## Test plan - [ ] 383 tests passing (372 + 11 new) - [ ] Run diagnose "database connection failed" — confirm ECONNREFUSED entries surface - [ ] Confirm Turnstone starts without sentence-transformers (BM25 fallback path) - [ ] Confirm `?semantic=true` on /api/search works with BAAI/bge-small-en-v1.5 loaded

pyr0ball added 1 commit 2026-06-01 18:16:48 -07:00

feat: hybrid BM25 + vector re-ranking for diagnose search (#15 ) 3155bde4ce

Adds late-fusion hybrid search to Turnstone's log retrieval layer:

  hybrid_score = 0.6 * bm25_normalized + 0.4 * cosine_similarity

Implementation:
- _bm25_search() extracts the existing FTS5 BM25 path as a named helper
- _hybrid_search() fetches an oversized BM25 candidate pool (5x limit,
  min 100), embeds the query and each candidate text in-process via the
  existing embeddings service, normalizes BM25 rank to [0,1], combines
  with cosine similarity, and re-ranks
- search() gets semantic=False param that dispatches to _hybrid_search()
  when True; pure BM25 remains the default for all existing call sites
- diagnose_stream() enables semantic=True so symptom-based queries
  ("database connection failed") surface semantically equivalent entries
  ("ECONNREFUSED", "backend gone away", "max retries exceeded")
- /api/search REST endpoint exposes ?semantic=true query param

Graceful degradation: falls back silently to pure BM25 when the embedding
backend is unavailable (EMBEDDING_AVAILABLE=False) or when embed_batch
raises an exception. No new infra — in-process numpy cosine, no vector DB.

11 new tests: BM25 helper, hybrid re-ranking, fallback paths, dispatcher.
372 + 11 = 383 tests passing.

Closes: #15