pyr0ball
d39cfbd87a
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap called out in the PR's original "known gaps" list — Windchill/DocPortal corpora commonly include parts lists and spec sheets as spreadsheets, not just prose documents. - scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window splitting for large sheets (header row repeated in every window so each chunk stays self-describing for retrieval). - scripts/shelve_ods.py — same chunking strategy via odfpy (already a dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/ TableCell. - scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for .pages. Adds libreoffice-calc to the Docker image alongside the existing libreoffice-writer. - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth for the full list. - 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing. Manually verified via Playwright against an isolated test instance: XLSX and ODS both upload, shelve to "ready", and store correctly row-serialized, header-repeated chunks (confirmed via sample-chunks). BM25 search against a 2-chunk toy corpus returned no hits for terms split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula giving an exact 0 for terms in exactly half a tiny corpus (log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass (5 chunks total) diluted this enough to return real results.
48 lines
1.8 KiB
Markdown
48 lines
1.8 KiB
Markdown
# Library
|
|
|
|
The library is the home screen. It shows all indexed documents and lets you add new ones.
|
|
|
|
## Adding documents
|
|
|
|
**Upload** — click **Upload Document or Spreadsheet** and select a file. Files up to 200 MB are accepted. The document is saved to `data/uploads/` and queued for indexing immediately.
|
|
|
|
**Scan** — set `PAGEPIPER_WATCH_DIR` to a directory in your `.env`, then click **Scan for PDFs**. Any supported document or spreadsheet file not already in the library is queued. Re-scanning is safe; already-indexed documents are skipped.
|
|
|
|
## Document states
|
|
|
|
| Badge | Meaning |
|
|
|-------|---------|
|
|
| PROCESSING | Text extraction or embedding in progress |
|
|
| READY | Fully indexed and searchable |
|
|
| ERROR | Indexing failed — see the error message on the card |
|
|
|
|
## Shelving progress
|
|
|
|
While a document is processing, its card shows a live progress bar:
|
|
|
|
- Animated sliding bar while text is being extracted (before page count is known)
|
|
- "Embedding N / M pages (X%)" once vectors are being written
|
|
|
|
The card refreshes automatically and emits a library reload when indexing completes.
|
|
|
|
## Re-indexing
|
|
|
|
Click **Re-index** on any document card to re-run the full shelve pipeline. This is useful after:
|
|
|
|
- Changing the `PAGEPIPER_EMBED_MODEL` (dimension mismatch auto-detected at startup, but you can also trigger manually)
|
|
- A failed shelve you want to retry
|
|
- Updating to a new version of Pagepiper with an improved extractor
|
|
|
|
## Removing a document
|
|
|
|
Click **Remove** to delete the document's metadata, page chunks, and vectors. The source file on disk is not deleted.
|
|
|
|
## Storage
|
|
|
|
All data lives in the directory set by `PAGEPIPER_DATA_DIR` (default: `data/`):
|
|
|
|
| File | Contents |
|
|
|------|---------|
|
|
| `pagepiper.db` | Document metadata, page chunks, chat feedback |
|
|
| `pagepiper_vecs.db` | sqlite-vec vector store |
|
|
| `uploads/` | Files added via browser upload |
|