pyr0ball
d39cfbd87a
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap called out in the PR's original "known gaps" list — Windchill/DocPortal corpora commonly include parts lists and spec sheets as spreadsheets, not just prose documents. - scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window splitting for large sheets (header row repeated in every window so each chunk stays self-describing for retrieval). - scripts/shelve_ods.py — same chunking strategy via odfpy (already a dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/ TableCell. - scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for .pages. Adds libreoffice-calc to the Docker image alongside the existing libreoffice-writer. - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth for the full list. - 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing. Manually verified via Playwright against an isolated test instance: XLSX and ODS both upload, shelve to "ready", and store correctly row-serialized, header-repeated chunks (confirmed via sample-chunks). BM25 search against a 2-chunk toy corpus returned no hits for terms split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula giving an exact 0 for terms in exactly half a tiny corpus (log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass (5 chunks total) diluted this enough to return real results.
36 lines
1.3 KiB
Markdown
36 lines
1.3 KiB
Markdown
# Quick Start
|
|
|
|
This guide gets you from zero to searching your first document in under five minutes.
|
|
|
|
## 1. Start Pagepiper
|
|
|
|
```bash
|
|
./manage.sh start
|
|
```
|
|
|
|
Open `http://localhost:8521` in your browser.
|
|
|
|
## 2. Add a document
|
|
|
|
You have two options:
|
|
|
|
**Upload directly** — click **Upload Document or Spreadsheet** in the library header and pick a file from your computer.
|
|
|
|
**Scan a directory** — set `PAGEPIPER_WATCH_DIR` in your `.env` to a folder of PDFs or EPUBs, then click **Scan for PDFs**. Pagepiper indexes every file it finds.
|
|
|
|
## 3. Wait for indexing
|
|
|
|
The document card shows progress while text is being extracted and embedded:
|
|
|
|
- **Extracting text...** (animated bar) — the file is being parsed into page chunks
|
|
- **Embedding N / M pages (X%)** (filling bar) — vectors are being written to the vector store (only when Ollama is configured)
|
|
|
|
Once the badge shows **READY**, the document is searchable.
|
|
|
|
## 4. Search
|
|
|
|
Click **Search** in the navigation. Type any phrase and see ranked page excerpts with scores. Results are instant using BM25 full-text search — no Ollama required.
|
|
|
|
## 5. Chat (optional, requires Ollama)
|
|
|
|
See the [Ollama Setup](ollama-setup.md) guide to enable hybrid vector search and LLM-powered chat. Once configured, the **Chat** tab lets you ask natural-language questions and get answers with page citations.
|