Self-hosted document library manager with BM25 keyword search and RAG chat with page-level citations
pyr0ball
89a58ec9b0
Bi-encoder embeddings collapse a whole query into one vector, losing multi-part reasoning structure — queries like "the procedure for setting an IP on an AVC-X" or "what is the action economy for a fighter casting a spell while prone" lose nuance. Agent-ModernColBERT is a late-interaction retriever: per-token embeddings, scored via MaxSim at query time, built specifically for agentic/multi-hop queries. Implements Option A from the issue (in-process, via `pylate`) rather than Option B (managed cf-orch service) — cf-orch already has `agent-moderncolbert` registered in model_registry.yaml with a `pagepiper/retrieve` assignment in assignments.yaml pointing at it and referencing this issue directly, someone had already pre-wired that side. - app/services/colbert_index.py: new ColBERTIndex class, mirrors BM25Index's dirty-flag/rebuild-from-SQLite pattern exactly — no separate per-shelve indexing step needed, just mark_dirty() on the same callback that already marks BM25 dirty. - app/services/retriever.py: hybrid_search's semantic half now merges BM25 with ColBERT MaxSim scores (min-max normalized per-batch, since MaxSim is unbounded unlike the old sqlite-vec L2-distance path) instead of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/ adjacent-chunk-window logic is unchanged. - app/main.py / app/deps.py: per-user ColBERTIndex registry, same pattern as the existing per-user BM25Index registry. - Existing BYOK tier gate preserved exactly (llm is None check) — this is a retrieval-technology swap, not a tier/licensing change. The ColBERT model runs locally via pylate with no Ollama dependency, but gating still follows product tiering. - 12 new tests. pylate is intentionally NOT installed in the dev/test env — see the cf-sysadmin skill's "Known Gotchas" for why (installing it directly into the shared `cf` conda env broke several other services' torch/transformers pins on 2026-07-10). Tests inject fake pylate modules via sys.modules instead. Known follow-up (not addressed here): shelve scripts still compute and store Ollama embeddings into `page_vecs` at shelve time — that table is no longer read by search/chat now that retrieval uses the ColBERT index. Removing the now-redundant embedding step is separate cleanup. Closes: #8 |
||
|---|---|---|
| app | ||
| config | ||
| docker/web | ||
| docs | ||
| migrations | ||
| scripts | ||
| tests | ||
| web | ||
| .env.cloud.example | ||
| .env.example | ||
| .gitignore | ||
| compose.cloud.yml | ||
| compose.override.yml.example | ||
| compose.yml | ||
| Dockerfile | ||
| environment.yml | ||
| manage.sh | ||
| mkdocs.yml | ||
| pyproject.toml | ||
| README.md | ||
Pagepiper
Search your document library. Get answers with exact page citations.
Self-hosted PDF and EPUB search with BM25 (Best Match 25) full-text indexing and LLM (large language model) synthesis. Drop your documents in, ask a question, get an answer that tells you exactly which page to turn to.
Built for TTRPG (tabletop roleplaying game) players who are tired of ctrl-F'ing through six-hundred-page rulebooks. Works equally well for legal research, technical manuals, academic papers, or any personal document library you want to query in plain language.
No cloud required. Your files stay on your machine.
Screenshots
Library
Chat with citations
Why Pagepiper?
- Your library, not ours. Documents are indexed and stored locally. Nothing is sent to a third-party service unless you explicitly configure a cloud LLM.
- Works without an LLM. BM25 full-text search runs entirely inside the Docker container. No Ollama, no API key, no GPU required for keyword search.
- Answers cite their sources. Every LLM response includes the document name and page number it drew from. You can verify or dispute every answer.
- Hybrid search when you want it. Connect a local Ollama instance to unlock hybrid search — BM25 merged with Agent-ModernColBERT, a late-interaction retriever that scores passages by token-level relevance instead of collapsing your whole question into one vector, so multi-part questions find the right passage even when it doesn't use your exact words.
- Open ingest pipeline. The indexing and search layer is MIT-licensed. Add support for new formats, improve the PDF parser, contribute — the community benefits directly.
Quick Start
Prerequisites: Docker and Docker Compose. Optionally Ollama for LLM-synthesized answers.
git clone https://git.opensourcesolarpunk.com/Circuit-Forge/pagepiper
cd pagepiper
cp .env.example .env
./manage.sh start
Open http://localhost:8521.
Configure
Open .env and set your paths:
# Where Pagepiper stores its SQLite index and uploaded files
PAGEPIPER_DATA_DIR=./data
# Directory to scan for existing PDFs/EPUBs (used by the Scan button)
PAGEPIPER_BOOKS_DIR=/path/to/your/documents
To unlock LLM synthesis and semantic search, add your Ollama endpoint:
PAGEPIPER_OLLAMA_URL=http://localhost:11434
PAGEPIPER_CHAT_MODEL=mistral:7b
PAGEPIPER_EMBED_MODEL=nomic-embed-text
Add documents
Upload via browser — click Upload in the Library view. Files save to data/uploads/ and index automatically.
Scan a directory — set PAGEPIPER_BOOKS_DIR in .env, then click Scan. Pagepiper finds all files recursively and queues them.
Supported Formats
| Format | Ingest | Page-level citations |
|---|---|---|
| Yes | Yes | |
| EPUB | Yes | Yes (chapter/location) |
Stack
| Layer | Technology |
|---|---|
| Backend API | FastAPI + SQLite |
| Full-text search | BM25 (custom index, no external service) |
| Semantic search | Agent-ModernColBERT late-interaction retrieval, via pylate (optional, BYOK-gated) |
| LLM synthesis | Ollama (local, any model) |
| Frontend | Vue 3 SPA served by nginx |
| Deployment | Docker Compose |
Default ports: Web UI 8521, API 8540.
Management
./manage.sh start # Build and start
./manage.sh stop # Stop
./manage.sh restart # Restart
./manage.sh status # Show container status
./manage.sh logs [svc] # Tail logs (pass 'api' or 'web' to filter)
./manage.sh open # Open UI in browser
./manage.sh build # Rebuild images
./manage.sh test # Run test suite
Tiers
| Feature | Free | Paid (BYOK) |
|---|---|---|
| PDF and EPUB upload | Yes | Yes |
| Directory scan | Yes | Yes |
| BM25 full-text search | Yes | Yes |
| Unlimited local ingestion | Yes | Yes |
| Hybrid BM25 + vector search | — | Yes (local Ollama) |
| LLM synthesis with page citations | — | Yes (local Ollama) |
BYOK means you supply your own Ollama instance. No cloud API keys, no usage metering.
Forgejo-primary
Pagepiper is developed and hosted at git.opensourcesolarpunk.com/Circuit-Forge/pagepiper. GitHub mirrors exist for discoverability only. File issues and submit pull requests on Forgejo.
License
Pagepiper uses a split license:
- MIT: Document ingest pipeline, BM25 full-text index, library management, EPUB support — the core discovery and retrieval layer.
- BSL 1.1 (Business Source License): Hybrid vector search, LLM synthesis, RAG (retrieval-augmented generation) chat interface — free for personal non-commercial self-hosting; commercial use or SaaS re-hosting requires a license. Converts to MIT after four years.
A Circuit Forge LLC product. Privacy · Safety · Accessibility — co-equal, non-negotiable.