Multi-format shelve support (DOCX/ODT/Pages/XLSX/ODS/Numbers), the
ingest→shelve terminology rename, GPU_SERVER_URL config rename, and the
Agent-ModernColBERT retriever swap — see release notes.
Found during freeze-branch integration testing (PR #12 + #14 + #15
combined): app/config.py's write-back (os.environ["CF_ORCH_URL"] =
GPU_SERVER_URL) is a module-level side effect monkeypatch never tracks,
so it wasn't being reverted between tests. Once any test_config.py test
set a truthy GPU_SERVER_URL/CF_LICENSE_KEY, the resulting CF_ORCH_URL
wrote leaked into every later test in the same pytest session.
This silently broke all five "skips_embeddings_without_ollama_url"
tests across the shelve_*.py suite (pdf, docx, odt, ods, xlsx) — their
get_llm_config() check no longer saw a clean "nothing configured" state,
so they attempted embedding instead of skipping it. Neither PR #12 nor
#14 alone exercised this combination in the same test session, so it
only surfaced once merged together.
Fix: explicitly pop CF_ORCH_URL/GPU_SERVER_URL/CF_LICENSE_KEY from
os.environ in the autouse teardown fixture before reloading app.config,
rather than relying on monkeypatch to catch a write it never made itself.
Bi-encoder embeddings collapse a whole query into one vector, losing
multi-part reasoning structure — queries like "the procedure for setting
an IP on an AVC-X" or "what is the action economy for a fighter casting
a spell while prone" lose nuance. Agent-ModernColBERT is a late-interaction
retriever: per-token embeddings, scored via MaxSim at query time, built
specifically for agentic/multi-hop queries.
Implements Option A from the issue (in-process, via `pylate`) rather than
Option B (managed cf-orch service) — cf-orch already has `agent-moderncolbert`
registered in model_registry.yaml with a `pagepiper/retrieve` assignment
in assignments.yaml pointing at it and referencing this issue directly,
someone had already pre-wired that side.
- app/services/colbert_index.py: new ColBERTIndex class, mirrors
BM25Index's dirty-flag/rebuild-from-SQLite pattern exactly — no
separate per-shelve indexing step needed, just mark_dirty() on the
same callback that already marks BM25 dirty.
- app/services/retriever.py: hybrid_search's semantic half now merges
BM25 with ColBERT MaxSim scores (min-max normalized per-batch, since
MaxSim is unbounded unlike the old sqlite-vec L2-distance path) instead
of Ollama-embed + sqlite-vec cosine. BM25 merge/rank/per-doc-cap/
adjacent-chunk-window logic is unchanged.
- app/main.py / app/deps.py: per-user ColBERTIndex registry, same
pattern as the existing per-user BM25Index registry.
- Existing BYOK tier gate preserved exactly (llm is None check) — this
is a retrieval-technology swap, not a tier/licensing change. The
ColBERT model runs locally via pylate with no Ollama dependency, but
gating still follows product tiering.
- 12 new tests. pylate is intentionally NOT installed in the dev/test
env — see the cf-sysadmin skill's "Known Gotchas" for why (installing
it directly into the shared `cf` conda env broke several other
services' torch/transformers pins on 2026-07-10). Tests inject fake
pylate modules via sys.modules instead.
Known follow-up (not addressed here): shelve scripts still compute and
store Ollama embeddings into `page_vecs` at shelve time — that table is
no longer read by search/chat now that retrieval uses the ColBERT index.
Removing the now-redundant embedding step is separate cleanup.
Closes: #8
CF_ORCH_URL makes sense internally but isn't self-explanatory for a
local-first user setting up their own GPU rig. Follows the GPU_SERVER_URL
convention Kiwi already established (app/core/config.py there).
- app/config.py: resolves GPU_SERVER_URL -> CF_ORCH_URL (back-compat
alias) -> https://orch.circuitforge.tech default when CF_LICENSE_KEY
is present (Paid+ tiers). Written back to os.environ["CF_ORCH_URL"]
so existing callers (get_llm_config, app/api/chat.py) needed zero
changes.
- .env.example / .env.cloud.example: document GPU_SERVER_URL as the
primary name, CF_ORCH_URL noted as a still-honoured legacy alias.
- compose.cloud.yml: the hardcoded coordinator URL now sets
GPU_SERVER_URL instead of CF_ORCH_URL directly (relies on the same
config.py normalization).
- docs/reference/environment-variables.md updated.
- 6 new tests covering the resolution priority chain and the
write-back behavior legacy callers depend on.
Closes: #10
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap
called out in the PR's original "known gaps" list — Windchill/DocPortal
corpora commonly include parts lists and spec sheets as spreadsheets, not
just prose documents.
- scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window
splitting for large sheets (header row repeated in every window so
each chunk stays self-describing for retrieval).
- scripts/shelve_ods.py — same chunking strategy via odfpy (already a
dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/
TableCell.
- scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX
and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for
.pages. Adds libreoffice-calc to the Docker image alongside the
existing libreoffice-writer.
- Upload button text changed from an ever-growing format list to
"Upload Document or Spreadsheet" — the Supported Formats table in
README/docs is now the source of truth for the full list.
- 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing.
Manually verified via Playwright against an isolated test instance:
XLSX and ODS both upload, shelve to "ready", and store correctly
row-serialized, header-repeated chunks (confirmed via sample-chunks).
BM25 search against a 2-chunk toy corpus returned no hits for terms
split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula
giving an exact 0 for terms in exactly half a tiny corpus
(log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a
defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass
(5 chunks total) diluted this enough to return real results.
Discovered while manually verifying the DOCX/ODT/Pages upload flow —
.playwright-mcp/ (fixtures, screenshots) wasn't ignored and risked
accidental commits of test artifacts.
Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.
- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
tests, docs, frontend). "Glean" (Turnstone's term) was considered and
rejected — that's a harvest metaphor for log/knowledge extraction,
not a fit for documents entering a library. Documented as a general
CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
logic (heading-based chunking, table serialization) already existed
but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
No maintained Python library parses the IWA format directly; libreoffice
bundles libetonyek, the only real open-source Pages parser. Adds
libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
full suite (72 tests) passing.
Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.
When CF_ORCH_URL is set, chat now calls CFOrchClient.task_allocate("pagepiper",
"rag_query") instead of routing through LLMRouter with an explicit model. The
coordinator resolves the assignment (granite-4.1-8b via assignments.yaml) and
returns an allocated URL; pagepiper wraps it in a minimal LLMRouter config for
the Synthesizer. Falls back to LLMRouter on TaskNotFound or allocation failure,
so standalone Ollama installs are unaffected.
Extracts _run_chat() and _build_llm_for_alloc() helpers to keep the endpoint
body readable regardless of which path fires.
Returns up to N randomly sampled page chunks (default 50, max 200) with
chunk_id, doc_id, page_number, and text fields. No tier gate — internal
tool endpoint for same-host corpus benchmarking. Returns [] on empty library.
Implements Option B (fscrypt) from the issue design: OS-level filesystem
encryption for per-user data directories on the cloud host.
- app/startup.py: warn_if_unencrypted() checks for fscrypt at startup in
cloud mode and logs a SECURITY warning if the users/ directory is not
encrypted — catches misconfigured deployments before any data is stored
- app/main.py: call warn_if_unencrypted() during lifespan in cloud mode
- scripts/setup_cloud_fscrypt.sh: operator script to encrypt a user's
data directory with fscrypt (run as root on host before container start);
supports --list and --status subcommands
Key management note: current implementation uses pam_passphrase protector.
For unattended server boot, integrate a raw_key protector from a secrets
manager (Vault, AWS Secrets Manager, etc.) — see script comments.
SQLCipher (Option A) deferred: sqlite-vec virtual table compatibility with
SQLCipher's encrypted VFS needs investigation before committing to that path.
Implements Option A from the issue design: each cloud user gets their own
data directory (DATA_DIR/users/{user_id}/) with separate pagepiper.db,
pagepiper_vecs.db, uploads/, and books/. Local mode is unchanged.
Key changes:
- app/startup.py: extract apply_migrations, reembed_docs,
check_and_rebuild_vec_schema out of main.py (no circular imports)
- app/config.py: add LOCAL_USER_ID constant and user_data_dir() helper
- app/cloud_session.py: extract resolve_authenticated_user(); require_paid_tier
now returns user_id (str) instead of None
- app/deps.py: add UserCtx dataclass (db_path, vec_db_path, data_dir,
watch_dir, bm25) + get_user_ctx dependency; per-user startup guard runs
migrations + vec schema check once per process per user
- app/main.py: _bm25 singleton -> _bm25_map dict keyed by user_id;
add _get_bm25_for(); lifespan only runs startup checks in local mode
- app/api/library.py, search.py, chat.py: thread UserCtx through all
endpoints; remove module-level _mark_bm25_dirty injection pattern
- tests/conftest.py: override get_user_ctx in addition to get_db so all
endpoints get a consistent test UserCtx
Closes#1: rename cloud:* subcommands to cloud-* (hyphen is the
conventional CLI separator; colon syntax is non-standard).
Closes#2: add update (git pull + rebuild) for both local and cloud
stacks, covering the common deploy-from-git workflow.
three-layer approach to stop 7B model from supplementing retrieved context
with training-data knowledge:
1. system prompt redesigned: 'no memory of books/stories/authors' eliminates
the model's self-permission to draw on parametric knowledge
2. quote-first prompt structure: model must commit to a specific quoted passage
before generating an answer — explicit NOT FOUND required when excerpts lack
the answer, preventing the 'excerpt doesn't say X... however in the series...'
escape pattern
3. _strip_escape() post-processor: catches any residual leakage by scanning for
known escape phrases ('in the series', 'by terry goodkind', 'it can be assumed',
etc.) and replacing the response with the canned no-answer message
synthesizer: repeat the no-outside-knowledge rule inside the user message turn —
small models (7B) follow user-turn instructions more reliably than system-prompt
alone when parametric memory competes with the retrieved context
retriever: cap each document to max(2, top_k//3) slots in the ranked list so
one book cannot flood all result slots on character-name BM25 matches — forces
coverage across more documents when the answer may be in any of them
- Strengthen synthesizer system prompt: hard 'respond with exactly' constraint
instead of soft 'say so'; removes any wiggle room for the model to supplement
from training data
- Add early return in synthesize() when chunks is empty (belt-and-suspenders
alongside the existing guard in chat.py)
- Add MIN_SIGNAL threshold (0.01) in retriever: if the top combined score is
below the threshold, return empty so the caller's no-results path fires instead
of sending noise chunks to the LLM
Retrieval:
- Add _fetch_adjacent() to retriever: fetches page ± 1 chunks from DB
after ranking so mid-sentence EPUB chunk boundaries don't lose context
- Fix vec DB doc-filter: oversample to top_k*20 before Python filter
instead of post-filtering an already-small global pool (fixes wrong-book
results when searching within a single document)
- top_k default 5 → 10; context per chunk 500 → 1500 chars; citation
snippet 200 → 400 chars
Artifact cleaning:
- Add scripts/text_clean.py: strips ABC Amber LIT Converter watermarks,
processtext.com URLs, bare page numbers, piracy stamps from extracted text
- Wire clean_paragraph() into ingest_pdf.py and new ingest_epub.py
Startup validation:
- _check_vec_schema() at boot: detects embedding dimension mismatch,
deletes stale vec DB, and queues sequential re-embed in background thread
- Sequential _reembed_docs() prevents SQLite lock races on startup re-embed
cf-orch integration:
- Wire CF_ORCH_URL / CF_LICENSE_KEY into LLMRouter backend config so
allocate() fires and keeps the Ollama model warm between requests
Ingestion progress UI:
- GET /api/library/{doc_id}/status now returns vec_count from page_vecs_meta
- DocumentCard.vue polls status every 3 s while processing and shows
two-phase progress: indeterminate animation during extraction,
determinate "Embedding N/M pages" bar once vectors start landing
Other:
- Chat feedback endpoint + thumbs up/down UI (FeedbackButton.vue)
- EPUB ingest script (ingest_epub.py) with heading-based chunking
- migration 002: chat_feedback table
- README.md with setup and feature overview
- CF_ORCH_URL, CF_APP_NAME, COORDINATOR_URL env vars in api service
- COORDINATOR_PAGEPIPER_KEY wired from .env
- extra_hosts: host.docker.internal:host-gateway for container → host Ollama
- .env.cloud.example updated with COORDINATOR_PAGEPIPER_KEY placeholder
- compose.cloud.yml: pagepiper-cloud project on port 8533 (avoids
conflict with Linnet dev on 8521/Magpie on 8531)
- docker/web/nginx.cloud.conf: handles both /pagepiper/* path (primary
domain, no Caddy strip) and / path (menagerie, Caddy strips prefix)
- docker/web/Dockerfile: NGINX_CONF build arg to select dev vs cloud conf
- .env.cloud.example: cloud env template with BYOK gate vars
- manage.sh: cloud:start|stop|restart|status|logs|build commands
Caddy config updated separately (not in this repo).
DNS record needed: pagepiper.circuitforge.tech → Heimdall edge IP.
Citation dataclass gains bm25_score field populated from the retrieved
chunk. chat.py serializes it. api.ts interface updated to include it.
ChatView passes :bm25-score to CitationPanel so the Nat20 threshold
check in onMounted actually has data to evaluate.
- app/services/retriever.py: hybrid BM25 + semantic Retriever with BM25-only fallback when llm=None
- app/services/synthesizer.py: LLM answer synthesis with citation assembly over retrieved chunks
- app/api/chat.py: POST /api/chat endpoint with 402 gate when PAGEPIPER_OLLAMA_URL is unset
- tests/test_synthesizer.py: 3 TDD unit tests (mocked LLM, context building, system prompt)
- tests/test_chat_api.py: 2 integration tests (402 without Ollama, 200 with mocked retriever+LLM)
Implements GET/DELETE /api/library, POST /api/library/{id}/reingest,
POST /api/library/scan, and GET /api/library/{id}/status. Adds FastAPI
app factory with lifespan migrations, BM25 singleton wiring, get_db
dependency, ingest task registry with cf-orch/BackgroundTasks fallback,
and placeholder search/chat routers. All 5 new tests pass (14 total).
Adds pyproject.toml, environment.yml, Dockerfile, docker/web (Vue+nginx),
compose.yml, compose.override.yml.example, manage.sh, .env.example,
.gitignore, and config stubs for the pagepiper self-hosted PDF library tool.
Port 8521. No secrets committed.
