feat: replace nomic-embed-text retriever with Agent-ModernColBERT #15

Merged

pyr0ball merged 2 commits from feat/colbert-retriever into main

2026-07-10 19:37:11 -07:00

pyr0ball commented

2026-07-10 19:02:52 -07:00

Owner

Summary

Closes #8.

app/services/colbert_index.py: new ColBERTIndex, mirrors BM25Index's dirty-flag/rebuild-from-SQLite pattern.
app/services/retriever.py: hybrid_search's semantic half now merges BM25 with Agent-ModernColBERT MaxSim scores (min-max normalized per-batch) instead of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/adjacent-chunk-window logic unchanged.
Per-user ColBERTIndex registry in app/main.py/app/deps.py, same pattern as the existing BM25 registry.
Existing BYOK tier gate preserved exactly — retrieval-technology swap, not a licensing change. ColBERT runs locally via pylate, no Ollama dependency, but still gated behind the same check.
Implements Option A from the issue. cf-orch already had agent-moderncolbert registered with a pagepiper/retrieve assignment referencing this issue — someone pre-wired that side already.

A note on how this was tested

pylate is intentionally not installed in the dev/test env. Installing it directly into the shared cf conda env on Heimdall broke several other services' torch/transformers pins (vllm, xformers, chatterbox-tts, pyannote-audio) — see the cf-sysadmin skill's "Known Gotchas" for the full incident writeup and the isolated-env remediation (cf-vllm, cf-voice, and a general sandbox env on Sif for this kind of research going forward). Instead:

Real end-to-end validation happened in the new Sif sandbox env (isolated, throwaway) — confirmed the full encode → index → retrieve workflow against the real Agent-ModernColBERT model, including a query matching this issue's own motivating example ("how do I set the IP on the AVC-X") ranking the correct document first.
Unit tests inject fake pylate/pylate.models/pylate.indexes/pylate.retrieve modules via sys.modules so ColBERTIndex's lazy imports resolve to mocks without pylate needing to be installed anywhere near this repo's own dev env.

Known follow-up (not addressed here)

Shelve scripts still compute and store Ollama embeddings into page_vecs at shelve time — that table is no longer read by search/chat now that retrieval uses the ColBERT index. Removing the now-redundant embedding step is separate cleanup.

Test plan

pytest tests/ -q — 69 passed (12 new tests)
Manual: real pylate + Agent-ModernColBERT smoke test in an isolated Sif sandbox env — encode/index/retrieve round-trip correct
Manual: not yet verified against the live /chat endpoint end-to-end (would need pylate actually installed in pagepiper's own Docker image/environment.yml, not yet done)

## Summary Closes #8. - `app/services/colbert_index.py`: new `ColBERTIndex`, mirrors `BM25Index`'s dirty-flag/rebuild-from-SQLite pattern. - `app/services/retriever.py`: `hybrid_search`'s semantic half now merges BM25 with Agent-ModernColBERT MaxSim scores (min-max normalized per-batch) instead of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/adjacent-chunk-window logic unchanged. - Per-user `ColBERTIndex` registry in `app/main.py`/`app/deps.py`, same pattern as the existing BM25 registry. - Existing BYOK tier gate preserved exactly — retrieval-technology swap, not a licensing change. ColBERT runs locally via `pylate`, no Ollama dependency, but still gated behind the same check. - Implements Option A from the issue. cf-orch already had `agent-moderncolbert` registered with a `pagepiper/retrieve` assignment referencing this issue — someone pre-wired that side already. ## A note on how this was tested pylate is **intentionally not installed** in the dev/test env. Installing it directly into the shared `cf` conda env on Heimdall broke several other services' torch/transformers pins (vllm, xformers, chatterbox-tts, pyannote-audio) — see the cf-sysadmin skill's "Known Gotchas" for the full incident writeup and the isolated-env remediation (`cf-vllm`, `cf-voice`, and a general `sandbox` env on Sif for this kind of research going forward). Instead: - Real end-to-end validation happened in the new Sif `sandbox` env (isolated, throwaway) — confirmed the full encode → index → retrieve workflow against the real Agent-ModernColBERT model, including a query matching this issue's own motivating example ("how do I set the IP on the AVC-X") ranking the correct document first. - Unit tests inject fake `pylate`/`pylate.models`/`pylate.indexes`/`pylate.retrieve` modules via `sys.modules` so `ColBERTIndex`'s lazy imports resolve to mocks without pylate needing to be installed anywhere near this repo's own dev env. ## Known follow-up (not addressed here) Shelve scripts still compute and store Ollama embeddings into `page_vecs` at shelve time — that table is no longer read by search/chat now that retrieval uses the ColBERT index. Removing the now-redundant embedding step is separate cleanup. ## Test plan - [x] `pytest tests/ -q` — 69 passed (12 new tests) - [x] Manual: real pylate + Agent-ModernColBERT smoke test in an isolated Sif sandbox env — encode/index/retrieve round-trip correct - [ ] Manual: not yet verified against the live `/chat` endpoint end-to-end (would need `pylate` actually installed in pagepiper's own Docker image/environment.yml, not yet done)

pyr0ball added 2 commits 2026-07-10 19:02:53 -07:00

feat: route pagepiper.rag_query through cf-orch task assignment layer (closes #7 ) b16620385a

When CF_ORCH_URL is set, chat now calls CFOrchClient.task_allocate("pagepiper",
"rag_query") instead of routing through LLMRouter with an explicit model. The
coordinator resolves the assignment (granite-4.1-8b via assignments.yaml) and
returns an allocated URL; pagepiper wraps it in a minimal LLMRouter config for
the Synthesizer. Falls back to LLMRouter on TaskNotFound or allocation failure,
so standalone Ollama installs are unaffected.

Extracts _run_chat() and _build_llm_for_alloc() helpers to keep the endpoint
body readable regardless of which path fires.

feat: replace nomic-embed-text retriever with Agent-ModernColBERT 89a58ec9b0

Bi-encoder embeddings collapse a whole query into one vector, losing
multi-part reasoning structure — queries like "the procedure for setting
an IP on an AVC-X" or "what is the action economy for a fighter casting
a spell while prone" lose nuance. Agent-ModernColBERT is a late-interaction
retriever: per-token embeddings, scored via MaxSim at query time, built
specifically for agentic/multi-hop queries.

Implements Option A from the issue (in-process, via `pylate`) rather than
Option B (managed cf-orch service) — cf-orch already has `agent-moderncolbert`
registered in model_registry.yaml with a `pagepiper/retrieve` assignment
in assignments.yaml pointing at it and referencing this issue directly,
someone had already pre-wired that side.

- app/services/colbert_index.py: new ColBERTIndex class, mirrors
  BM25Index's dirty-flag/rebuild-from-SQLite pattern exactly — no
  separate per-shelve indexing step needed, just mark_dirty() on the
  same callback that already marks BM25 dirty.
- app/services/retriever.py: hybrid_search's semantic half now merges
  BM25 with ColBERT MaxSim scores (min-max normalized per-batch, since
  MaxSim is unbounded unlike the old sqlite-vec L2-distance path) instead
  of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/
  adjacent-chunk-window logic is unchanged.
- app/main.py / app/deps.py: per-user ColBERTIndex registry, same
  pattern as the existing per-user BM25Index registry.
- Existing BYOK tier gate preserved exactly (llm is None check) — this
  is a retrieval-technology swap, not a tier/licensing change. The
  ColBERT model runs locally via pylate with no Ollama dependency, but
  gating still follows product tiering.
- 12 new tests. pylate is intentionally NOT installed in the dev/test
  env — see the cf-sysadmin skill's "Known Gotchas" for why (installing
  it directly into the shared `cf` conda env broke several other
  services' torch/transformers pins on 2026-07-10). Tests inject fake
  pylate modules via sys.modules instead.

Known follow-up (not addressed here): shelve scripts still compute and
store Ollama embeddings into `page_vecs` at shelve time — that table is
no longer read by search/chat now that retrieval uses the ColBERT index.
Removing the now-redundant embedding step is separate cleanup.

Closes: #8