Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.
- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
tests, docs, frontend). "Glean" (Turnstone's term) was considered and
rejected — that's a harvest metaphor for log/knowledge extraction,
not a fit for documents entering a library. Documented as a general
CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
logic (heading-based chunking, table serialization) already existed
but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
No maintained Python library parses the IWA format directly; libreoffice
bundles libetonyek, the only real open-source Pages parser. Adds
libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
full suite (72 tests) passing.
Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.
Returns up to N randomly sampled page chunks (default 50, max 200) with
chunk_id, doc_id, page_number, and text fields. No tier gate — internal
tool endpoint for same-host corpus benchmarking. Returns [] on empty library.
Implements GET/DELETE /api/library, POST /api/library/{id}/reingest,
POST /api/library/scan, and GET /api/library/{id}/status. Adds FastAPI
app factory with lifespan migrations, BM25 singleton wiring, get_db
dependency, ingest task registry with cf-orch/BackgroundTasks fallback,
and placeholder search/chat routers. All 5 new tests pass (14 total).
