pyr0ball f941ebdeeb feat: add ODT and Apple Pages document support, wire DOCX into UI

Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.

- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
  tests, docs, frontend). "Glean" (Turnstone's term) was considered and
  rejected — that's a harvest metaphor for log/knowledge extraction,
  not a fit for documents entering a library. Documented as a general
  CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
  logic (heading-based chunking, table serialization) already existed
  but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
  No maintained Python library parses the IWA format directly; libreoffice
  bundles libetonyek, the only real open-source Pages parser. Adds
  libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
  full suite (72 tests) passing.

Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.

2026-07-10 13:58:43 -07:00

1.8 KiB

Raw Blame History

Library

The library is the home screen. It shows all indexed documents and lets you add new ones.

Adding documents

Upload — click Upload PDF / EPUB / DOCX / ODT / Pages and select a file. Files up to 200 MB are accepted. The document is saved to data/uploads/ and queued for indexing immediately.

Scan — set PAGEPIPER_WATCH_DIR to a directory in your .env, then click Scan for PDFs. Any PDF, EPUB, DOCX, ODT, or Pages file not already in the library is queued. Re-scanning is safe; already-indexed documents are skipped.

Document states

Badge	Meaning
PROCESSING	Text extraction or embedding in progress
READY	Fully indexed and searchable
ERROR	Indexing failed — see the error message on the card

Shelving progress

While a document is processing, its card shows a live progress bar:

Animated sliding bar while text is being extracted (before page count is known)
"Embedding N / M pages (X%)" once vectors are being written

The card refreshes automatically and emits a library reload when indexing completes.

Re-indexing

Click Re-index on any document card to re-run the full shelve pipeline. This is useful after:

Changing the PAGEPIPER_EMBED_MODEL (dimension mismatch auto-detected at startup, but you can also trigger manually)
A failed shelve you want to retry
Updating to a new version of Pagepiper with an improved extractor

Removing a document

Click Remove to delete the document's metadata, page chunks, and vectors. The source file on disk is not deleted.

Storage

All data lives in the directory set by PAGEPIPER_DATA_DIR (default: data/):

File	Contents
`pagepiper.db`	Document metadata, page chunks, chat feedback
`pagepiper_vecs.db`	sqlite-vec vector store
`uploads/`	Files added via browser upload

1.8 KiB Raw Blame History