Bi-encoder embeddings collapse a whole query into one vector, losing
multi-part reasoning structure — queries like "the procedure for setting
an IP on an AVC-X" or "what is the action economy for a fighter casting
a spell while prone" lose nuance. Agent-ModernColBERT is a late-interaction
retriever: per-token embeddings, scored via MaxSim at query time, built
specifically for agentic/multi-hop queries.
Implements Option A from the issue (in-process, via `pylate`) rather than
Option B (managed cf-orch service) — cf-orch already has `agent-moderncolbert`
registered in model_registry.yaml with a `pagepiper/retrieve` assignment
in assignments.yaml pointing at it and referencing this issue directly,
someone had already pre-wired that side.
- app/services/colbert_index.py: new ColBERTIndex class, mirrors
BM25Index's dirty-flag/rebuild-from-SQLite pattern exactly — no
separate per-shelve indexing step needed, just mark_dirty() on the
same callback that already marks BM25 dirty.
- app/services/retriever.py: hybrid_search's semantic half now merges
BM25 with ColBERT MaxSim scores (min-max normalized per-batch, since
MaxSim is unbounded unlike the old sqlite-vec L2-distance path) instead
of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/
adjacent-chunk-window logic is unchanged.
- app/main.py / app/deps.py: per-user ColBERTIndex registry, same
pattern as the existing per-user BM25Index registry.
- Existing BYOK tier gate preserved exactly (llm is None check) — this
is a retrieval-technology swap, not a tier/licensing change. The
ColBERT model runs locally via pylate with no Ollama dependency, but
gating still follows product tiering.
- 12 new tests. pylate is intentionally NOT installed in the dev/test
env — see the cf-sysadmin skill's "Known Gotchas" for why (installing
it directly into the shared `cf` conda env broke several other
services' torch/transformers pins on 2026-07-10). Tests inject fake
pylate modules via sys.modules instead.
Known follow-up (not addressed here): shelve scripts still compute and
store Ollama embeddings into `page_vecs` at shelve time — that table is
no longer read by search/chat now that retrieval uses the ColBERT index.
Removing the now-redundant embedding step is separate cleanup.
Closes: #8
synthesizer: repeat the no-outside-knowledge rule inside the user message turn —
small models (7B) follow user-turn instructions more reliably than system-prompt
alone when parametric memory competes with the retrieved context
retriever: cap each document to max(2, top_k//3) slots in the ranked list so
one book cannot flood all result slots on character-name BM25 matches — forces
coverage across more documents when the answer may be in any of them
- Strengthen synthesizer system prompt: hard 'respond with exactly' constraint
instead of soft 'say so'; removes any wiggle room for the model to supplement
from training data
- Add early return in synthesize() when chunks is empty (belt-and-suspenders
alongside the existing guard in chat.py)
- Add MIN_SIGNAL threshold (0.01) in retriever: if the top combined score is
below the threshold, return empty so the caller's no-results path fires instead
of sending noise chunks to the LLM
Retrieval:
- Add _fetch_adjacent() to retriever: fetches page ± 1 chunks from DB
after ranking so mid-sentence EPUB chunk boundaries don't lose context
- Fix vec DB doc-filter: oversample to top_k*20 before Python filter
instead of post-filtering an already-small global pool (fixes wrong-book
results when searching within a single document)
- top_k default 5 → 10; context per chunk 500 → 1500 chars; citation
snippet 200 → 400 chars
Artifact cleaning:
- Add scripts/text_clean.py: strips ABC Amber LIT Converter watermarks,
processtext.com URLs, bare page numbers, piracy stamps from extracted text
- Wire clean_paragraph() into ingest_pdf.py and new ingest_epub.py
Startup validation:
- _check_vec_schema() at boot: detects embedding dimension mismatch,
deletes stale vec DB, and queues sequential re-embed in background thread
- Sequential _reembed_docs() prevents SQLite lock races on startup re-embed
cf-orch integration:
- Wire CF_ORCH_URL / CF_LICENSE_KEY into LLMRouter backend config so
allocate() fires and keeps the Ollama model warm between requests
Ingestion progress UI:
- GET /api/library/{doc_id}/status now returns vec_count from page_vecs_meta
- DocumentCard.vue polls status every 3 s while processing and shows
two-phase progress: indeterminate animation during extraction,
determinate "Embedding N/M pages" bar once vectors start landing
Other:
- Chat feedback endpoint + thumbs up/down UI (FeedbackButton.vue)
- EPUB ingest script (ingest_epub.py) with heading-based chunking
- migration 002: chat_feedback table
- README.md with setup and feature overview
- app/services/retriever.py: hybrid BM25 + semantic Retriever with BM25-only fallback when llm=None
- app/services/synthesizer.py: LLM answer synthesis with citation assembly over retrieved chunks
- app/api/chat.py: POST /api/chat endpoint with 402 gate when PAGEPIPER_OLLAMA_URL is unset
- tests/test_synthesizer.py: 3 TDD unit tests (mocked LLM, context building, system prompt)
- tests/test_chat_api.py: 2 integration tests (402 without Ollama, 200 with mocked retriever+LLM)
