pyr0ball f941ebdeeb feat: add ODT and Apple Pages document support, wire DOCX into UI

Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.

- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
  tests, docs, frontend). "Glean" (Turnstone's term) was considered and
  rejected — that's a harvest metaphor for log/knowledge extraction,
  not a fit for documents entering a library. Documented as a general
  CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
  logic (heading-based chunking, table serialization) already existed
  but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
  No maintained Python library parses the IWA format directly; libreoffice
  bundles libetonyek, the only real open-source Pages parser. Adds
  libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
  full suite (72 tests) passing.

Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.

2026-07-10 13:58:43 -07:00

1.4 KiB

Raw Blame History

Ollama Setup

Hybrid vector search and RAG chat are gated behind a local Ollama instance. This is the BYOK (bring your own key) unlock for the Free tier — no paid subscription required.

Install Ollama

curl -fsSL https://ollama.ai/install.sh | sh

Pull the required models

# Embedding model — converts pages into vectors
ollama pull nomic-embed-text

# Chat model — answers questions using retrieved page excerpts
ollama pull mistral:7b

nomic-embed-text produces 1024-dimensional vectors and runs comfortably on 8 GB of VRAM. mistral:7b requires roughly 5 GB of VRAM. Substitute any compatible model.

Configure Pagepiper

In your .env:

PAGEPIPER_OLLAMA_URL=http://localhost:11434
PAGEPIPER_EMBED_MODEL=nomic-embed-text
PAGEPIPER_CHAT_MODEL=mistral:7b

Restart Pagepiper:

./manage.sh restart

Verify

Upload or re-index a document. The document card should show Embedding N / M pages while shelving. Once complete, the Chat tab becomes active.

Changing embedding models

If you switch PAGEPIPER_EMBED_MODEL, Pagepiper detects the dimension mismatch at startup, deletes the old vector database, and automatically re-embeds all indexed documents in the background. BM25 search remains available throughout.

!!! note Re-embedding a large library can take 30-60 minutes depending on hardware.

1.4 KiB Raw Blame History