pyr0ball f941ebdeeb feat: add ODT and Apple Pages document support, wire DOCX into UI

Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.

- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
  tests, docs, frontend). "Glean" (Turnstone's term) was considered and
  rejected — that's a harvest metaphor for log/knowledge extraction,
  not a fit for documents entering a library. Documented as a general
  CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
  logic (heading-based chunking, table serialization) already existed
  but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
  No maintained Python library parses the IWA format directly; libreoffice
  bundles libetonyek, the only real open-source Pages parser. Adds
  libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
  full suite (72 tests) passing.

Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.

2026-07-10 13:58:43 -07:00

1.3 KiB

Raw Blame History

Quick Start

This guide gets you from zero to searching your first document in under five minutes.

1. Start Pagepiper

./manage.sh start

Open http://localhost:8521 in your browser.

2. Add a document

You have two options:

Upload directly — click Upload PDF / EPUB / DOCX / ODT / Pages in the library header and pick a file from your computer.

Scan a directory — set PAGEPIPER_WATCH_DIR in your .env to a folder of PDFs or EPUBs, then click Scan for PDFs. Pagepiper indexes every file it finds.

3. Wait for indexing

The document card shows progress while text is being extracted and embedded:

Extracting text... (animated bar) — PDF/EPUB/DOCX/ODT/Pages is being parsed into page chunks
Embedding N / M pages (X%) (filling bar) — vectors are being written to the vector store (only when Ollama is configured)

Once the badge shows READY, the document is searchable.

4. Search

Click Search in the navigation. Type any phrase and see ranked page excerpts with scores. Results are instant using BM25 full-text search — no Ollama required.

5. Chat (optional, requires Ollama)

See the Ollama Setup guide to enable hybrid vector search and LLM-powered chat. Once configured, the Chat tab lets you ask natural-language questions and get answers with page citations.

1.3 KiB Raw Blame History