pyr0ball f941ebdeeb feat: add ODT and Apple Pages document support, wire DOCX into UI

Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.

- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
  tests, docs, frontend). "Glean" (Turnstone's term) was considered and
  rejected — that's a harvest metaphor for log/knowledge extraction,
  not a fit for documents entering a library. Documented as a general
  CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
  logic (heading-based chunking, table serialization) already existed
  but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
  No maintained Python library parses the IWA format directly; libreoffice
  bundles libetonyek, the only real open-source Pages parser. Adds
  libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
  full suite (72 tests) passing.

Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.

2026-07-10 13:58:43 -07:00

1.9 KiB

Raw Blame History

Architecture

Overview

Browser (Vue 3 SPA)
        |
  nginx (static + /api proxy)
        |
  FastAPI backend
    ├── BM25Index (in-process, rank-bm25)
    ├── Retriever (BM25 + optional vector)
    ├── Synthesizer (LLMRouter → Ollama)
    └── SQLite (page_chunks + metadata)
              +
         sqlite-vec (vectors)

Shelve pipeline

PDF / EPUB / DOCX / ODT / Pages file
    │
    ├─ PDFExtractor (pdfminer + OCR fallback)  ← circuitforge_core
    │   or
    └─ EPUBExtractor (BeautifulSoup + heading chunking)
            │
     text_clean.py (strip artifacts)
            │
     INSERT INTO page_chunks
            │
     Ollama embed (batches of 64)   ← BYOK gate
            │
     sqlite-vec upsert

Retrieval

Hybrid search merges BM25 and semantic results with a 50/50 score blend:

BM25 queries the in-process index (no round-trip to DB)
Semantic query embeds the user query via Ollama, fetches top_k * 20 nearest vectors, filters by doc_id in Python
Hits are merged: BM25 scores and vector scores combined; BM25 hits take priority
Top k results are ranked, then adjacent pages (page ± 1) are fetched to restore context for mid-sentence chunk boundaries

Storage

File	Format	Contents
`pagepiper.db`	SQLite	`documents`, `page_chunks`, `chat_feedback`
`pagepiper_vecs.db`	sqlite-vec	`page_vecs` virtual table + `page_vecs_meta`

The vector database stores one row per page chunk. If the embedding model changes, Pagepiper detects the dimension mismatch at startup (reads CREATE VIRTUAL TABLE DDL from sqlite_master), deletes the vec DB, and queues a background re-embed.

Licensing boundary

Component	License
BM25 search, shelve pipeline, library API	MIT
Hybrid vector search, RAG chat, embedding	BSL 1.1 (BYOK unlocked on Free tier)

1.9 KiB Raw Blame History