pyr0ball
f941ebdeeb
Extends Pagepiper's document shelving pipeline (renamed from "ingest" — see below) to cover the formats most likely to appear in a real-world engineering document corpus, prompted by scoping a STERIS licensing pitch that needs DOCX/ODT coverage. - Rename the ingest pipeline to "shelve" throughout (scripts/, app/api, tests, docs, frontend). "Glean" (Turnstone's term) was considered and rejected — that's a harvest metaphor for log/knowledge extraction, not a fit for documents entering a library. Documented as a general CF naming principle in the org-level CLAUDE.md. - Wire DOCX into the upload/scan UI, README, and docs — the extraction logic (heading-based chunking, table serialization) already existed but wasn't exposed to users or covered by tests. - Add ODT support via odfpy, mirroring DOCX's chunking strategy. - Add Apple Pages support via headless LibreOffice conversion to ODT. No maintained Python library parses the IWA format directly; libreoffice bundles libetonyek, the only real open-source Pages parser. Adds libreoffice-writer to the Docker image (~300-400MB) for this. - 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages; full suite (72 tests) passing. Known gaps not addressed here: no Windchill/DocPortal connector exists yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported, and circuitforge_core.tasks.dispatch_task does not currently exist in circuitforge-core — cf-orch dispatch is dead code, always falling through to local BackgroundTasks. See circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md for the full writeup.
48 lines
1.8 KiB
Markdown
48 lines
1.8 KiB
Markdown
# Library
|
|
|
|
The library is the home screen. It shows all indexed documents and lets you add new ones.
|
|
|
|
## Adding documents
|
|
|
|
**Upload** — click **Upload PDF / EPUB / DOCX / ODT / Pages** and select a file. Files up to 200 MB are accepted. The document is saved to `data/uploads/` and queued for indexing immediately.
|
|
|
|
**Scan** — set `PAGEPIPER_WATCH_DIR` to a directory in your `.env`, then click **Scan for PDFs**. Any PDF, EPUB, DOCX, ODT, or Pages file not already in the library is queued. Re-scanning is safe; already-indexed documents are skipped.
|
|
|
|
## Document states
|
|
|
|
| Badge | Meaning |
|
|
|-------|---------|
|
|
| PROCESSING | Text extraction or embedding in progress |
|
|
| READY | Fully indexed and searchable |
|
|
| ERROR | Indexing failed — see the error message on the card |
|
|
|
|
## Shelving progress
|
|
|
|
While a document is processing, its card shows a live progress bar:
|
|
|
|
- Animated sliding bar while text is being extracted (before page count is known)
|
|
- "Embedding N / M pages (X%)" once vectors are being written
|
|
|
|
The card refreshes automatically and emits a library reload when indexing completes.
|
|
|
|
## Re-indexing
|
|
|
|
Click **Re-index** on any document card to re-run the full shelve pipeline. This is useful after:
|
|
|
|
- Changing the `PAGEPIPER_EMBED_MODEL` (dimension mismatch auto-detected at startup, but you can also trigger manually)
|
|
- A failed shelve you want to retry
|
|
- Updating to a new version of Pagepiper with an improved extractor
|
|
|
|
## Removing a document
|
|
|
|
Click **Remove** to delete the document's metadata, page chunks, and vectors. The source file on disk is not deleted.
|
|
|
|
## Storage
|
|
|
|
All data lives in the directory set by `PAGEPIPER_DATA_DIR` (default: `data/`):
|
|
|
|
| File | Contents |
|
|
|------|---------|
|
|
| `pagepiper.db` | Document metadata, page chunks, chat feedback |
|
|
| `pagepiper_vecs.db` | sqlite-vec vector store |
|
|
| `uploads/` | Files added via browser upload |
|