pyr0ball d39cfbd87a feat: add XLSX, ODS, and Apple Numbers spreadsheet support

Extends the shelve pipeline to cover spreadsheets, closing the Excel gap
called out in the PR's original "known gaps" list — Windchill/DocPortal
corpora commonly include parts lists and spec sheets as spreadsheets, not
just prose documents.

- scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window
  splitting for large sheets (header row repeated in every window so
  each chunk stays self-describing for retrieval).
- scripts/shelve_ods.py — same chunking strategy via odfpy (already a
  dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/
  TableCell.
- scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX
  and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for
  .pages. Adds libreoffice-calc to the Docker image alongside the
  existing libreoffice-writer.
- Upload button text changed from an ever-growing format list to
  "Upload Document or Spreadsheet" — the Supported Formats table in
  README/docs is now the source of truth for the full list.
- 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing.

Manually verified via Playwright against an isolated test instance:
XLSX and ODS both upload, shelve to "ready", and store correctly
row-serialized, header-repeated chunks (confirmed via sample-chunks).
BM25 search against a 2-chunk toy corpus returned no hits for terms
split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula
giving an exact 0 for terms in exactly half a tiny corpus
(log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a
defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass
(5 chunks total) diluted this enough to return real results.

2026-07-10 15:06:16 -07:00

1.8 KiB

Raw Blame History

Library

The library is the home screen. It shows all indexed documents and lets you add new ones.

Adding documents

Upload — click Upload Document or Spreadsheet and select a file. Files up to 200 MB are accepted. The document is saved to data/uploads/ and queued for indexing immediately.

Scan — set PAGEPIPER_WATCH_DIR to a directory in your .env, then click Scan for PDFs. Any supported document or spreadsheet file not already in the library is queued. Re-scanning is safe; already-indexed documents are skipped.

Document states

Badge	Meaning
PROCESSING	Text extraction or embedding in progress
READY	Fully indexed and searchable
ERROR	Indexing failed — see the error message on the card

Shelving progress

While a document is processing, its card shows a live progress bar:

Animated sliding bar while text is being extracted (before page count is known)
"Embedding N / M pages (X%)" once vectors are being written

The card refreshes automatically and emits a library reload when indexing completes.

Re-indexing

Click Re-index on any document card to re-run the full shelve pipeline. This is useful after:

Changing the PAGEPIPER_EMBED_MODEL (dimension mismatch auto-detected at startup, but you can also trigger manually)
A failed shelve you want to retry
Updating to a new version of Pagepiper with an improved extractor

Removing a document

Click Remove to delete the document's metadata, page chunks, and vectors. The source file on disk is not deleted.

Storage

All data lives in the directory set by PAGEPIPER_DATA_DIR (default: data/):

File	Contents
`pagepiper.db`	Document metadata, page chunks, chat feedback
`pagepiper_vecs.db`	sqlite-vec vector store
`uploads/`	Files added via browser upload

1.8 KiB Raw Blame History