pyr0ball d39cfbd87a feat: add XLSX, ODS, and Apple Numbers spreadsheet support

Extends the shelve pipeline to cover spreadsheets, closing the Excel gap
called out in the PR's original "known gaps" list — Windchill/DocPortal
corpora commonly include parts lists and spec sheets as spreadsheets, not
just prose documents.

- scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window
  splitting for large sheets (header row repeated in every window so
  each chunk stays self-describing for retrieval).
- scripts/shelve_ods.py — same chunking strategy via odfpy (already a
  dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/
  TableCell.
- scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX
  and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for
  .pages. Adds libreoffice-calc to the Docker image alongside the
  existing libreoffice-writer.
- Upload button text changed from an ever-growing format list to
  "Upload Document or Spreadsheet" — the Supported Formats table in
  README/docs is now the source of truth for the full list.
- 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing.

Manually verified via Playwright against an isolated test instance:
XLSX and ODS both upload, shelve to "ready", and store correctly
row-serialized, header-repeated chunks (confirmed via sample-chunks).
BM25 search against a 2-chunk toy corpus returned no hits for terms
split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula
giving an exact 0 for terms in exactly half a tiny corpus
(log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a
defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass
(5 chunks total) diluted this enough to return real results.

2026-07-10 15:06:16 -07:00

1.9 KiB

Raw Permalink Blame History

Architecture

Overview

Browser (Vue 3 SPA)
        |
  nginx (static + /api proxy)
        |
  FastAPI backend
    ├── BM25Index (in-process, rank-bm25)
    ├── Retriever (BM25 + optional vector)
    ├── Synthesizer (LLMRouter → Ollama)
    └── SQLite (page_chunks + metadata)
              +
         sqlite-vec (vectors)

Shelve pipeline

any supported document or spreadsheet file
    │
    ├─ PDFExtractor (pdfminer + OCR fallback)  ← circuitforge_core
    │   or
    └─ EPUBExtractor (BeautifulSoup + heading chunking)
            │
     text_clean.py (strip artifacts)
            │
     INSERT INTO page_chunks
            │
     Ollama embed (batches of 64)   ← BYOK gate
            │
     sqlite-vec upsert

Retrieval

Hybrid search merges BM25 and semantic results with a 50/50 score blend:

BM25 queries the in-process index (no round-trip to DB)
Semantic query embeds the user query via Ollama, fetches top_k * 20 nearest vectors, filters by doc_id in Python
Hits are merged: BM25 scores and vector scores combined; BM25 hits take priority
Top k results are ranked, then adjacent pages (page ± 1) are fetched to restore context for mid-sentence chunk boundaries

Storage

File	Format	Contents
`pagepiper.db`	SQLite	`documents`, `page_chunks`, `chat_feedback`
`pagepiper_vecs.db`	sqlite-vec	`page_vecs` virtual table + `page_vecs_meta`

The vector database stores one row per page chunk. If the embedding model changes, Pagepiper detects the dimension mismatch at startup (reads CREATE VIRTUAL TABLE DDL from sqlite_master), deletes the vec DB, and queues a background re-embed.

Licensing boundary

Component	License
BM25 search, shelve pipeline, library API	MIT
Hybrid vector search, RAG chat, embedding	BSL 1.1 (BYOK unlocked on Free tier)

1.9 KiB Raw Permalink Blame History