pyr0ball d39cfbd87a feat: add XLSX, ODS, and Apple Numbers spreadsheet support

Extends the shelve pipeline to cover spreadsheets, closing the Excel gap
called out in the PR's original "known gaps" list — Windchill/DocPortal
corpora commonly include parts lists and spec sheets as spreadsheets, not
just prose documents.

- scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window
  splitting for large sheets (header row repeated in every window so
  each chunk stays self-describing for retrieval).
- scripts/shelve_ods.py — same chunking strategy via odfpy (already a
  dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/
  TableCell.
- scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX
  and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for
  .pages. Adds libreoffice-calc to the Docker image alongside the
  existing libreoffice-writer.
- Upload button text changed from an ever-growing format list to
  "Upload Document or Spreadsheet" — the Supported Formats table in
  README/docs is now the source of truth for the full list.
- 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing.

Manually verified via Playwright against an isolated test instance:
XLSX and ODS both upload, shelve to "ready", and store correctly
row-serialized, header-repeated chunks (confirmed via sample-chunks).
BM25 search against a 2-chunk toy corpus returned no hits for terms
split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula
giving an exact 0 for terms in exactly half a tiny corpus
(log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a
defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass
(5 chunks total) diluted this enough to return real results.

2026-07-10 15:06:16 -07:00

1.3 KiB

Raw Blame History

Quick Start

This guide gets you from zero to searching your first document in under five minutes.

1. Start Pagepiper

./manage.sh start

Open http://localhost:8521 in your browser.

2. Add a document

You have two options:

Upload directly — click Upload Document or Spreadsheet in the library header and pick a file from your computer.

Scan a directory — set PAGEPIPER_WATCH_DIR in your .env to a folder of PDFs or EPUBs, then click Scan for PDFs. Pagepiper indexes every file it finds.

3. Wait for indexing

The document card shows progress while text is being extracted and embedded:

Extracting text... (animated bar) — the file is being parsed into page chunks
Embedding N / M pages (X%) (filling bar) — vectors are being written to the vector store (only when Ollama is configured)

Once the badge shows READY, the document is searchable.

4. Search

Click Search in the navigation. Type any phrase and see ranked page excerpts with scores. Results are instant using BM25 full-text search — no Ollama required.

5. Chat (optional, requires Ollama)

See the Ollama Setup guide to enable hybrid vector search and LLM-powered chat. Once configured, the Chat tab lets you ask natural-language questions and get answers with page citations.

1.3 KiB Raw Blame History