pyr0ball
d39cfbd87a
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap called out in the PR's original "known gaps" list — Windchill/DocPortal corpora commonly include parts lists and spec sheets as spreadsheets, not just prose documents. - scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window splitting for large sheets (header row repeated in every window so each chunk stays self-describing for retrieval). - scripts/shelve_ods.py — same chunking strategy via odfpy (already a dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/ TableCell. - scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for .pages. Adds libreoffice-calc to the Docker image alongside the existing libreoffice-writer. - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth for the full list. - 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing. Manually verified via Playwright against an isolated test instance: XLSX and ODS both upload, shelve to "ready", and store correctly row-serialized, header-repeated chunks (confirmed via sample-chunks). BM25 search against a 2-chunk toy corpus returned no hits for terms split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula giving an exact 0 for terms in exactly half a tiny corpus (log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass (5 chunks total) diluted this enough to return real results.
60 lines
1.9 KiB
Markdown
60 lines
1.9 KiB
Markdown
# Architecture
|
|
|
|
## Overview
|
|
|
|
```
|
|
Browser (Vue 3 SPA)
|
|
|
|
|
nginx (static + /api proxy)
|
|
|
|
|
FastAPI backend
|
|
├── BM25Index (in-process, rank-bm25)
|
|
├── Retriever (BM25 + optional vector)
|
|
├── Synthesizer (LLMRouter → Ollama)
|
|
└── SQLite (page_chunks + metadata)
|
|
+
|
|
sqlite-vec (vectors)
|
|
```
|
|
|
|
## Shelve pipeline
|
|
|
|
```
|
|
any supported document or spreadsheet file
|
|
│
|
|
├─ PDFExtractor (pdfminer + OCR fallback) ← circuitforge_core
|
|
│ or
|
|
└─ EPUBExtractor (BeautifulSoup + heading chunking)
|
|
│
|
|
text_clean.py (strip artifacts)
|
|
│
|
|
INSERT INTO page_chunks
|
|
│
|
|
Ollama embed (batches of 64) ← BYOK gate
|
|
│
|
|
sqlite-vec upsert
|
|
```
|
|
|
|
## Retrieval
|
|
|
|
Hybrid search merges BM25 and semantic results with a 50/50 score blend:
|
|
|
|
1. BM25 queries the in-process index (no round-trip to DB)
|
|
2. Semantic query embeds the user query via Ollama, fetches `top_k * 20` nearest vectors, filters by `doc_id` in Python
|
|
3. Hits are merged: BM25 scores and vector scores combined; BM25 hits take priority
|
|
4. Top `k` results are ranked, then adjacent pages (page ± 1) are fetched to restore context for mid-sentence chunk boundaries
|
|
|
|
## Storage
|
|
|
|
| File | Format | Contents |
|
|
|------|--------|---------|
|
|
| `pagepiper.db` | SQLite | `documents`, `page_chunks`, `chat_feedback` |
|
|
| `pagepiper_vecs.db` | sqlite-vec | `page_vecs` virtual table + `page_vecs_meta` |
|
|
|
|
The vector database stores one row per page chunk. If the embedding model changes, Pagepiper detects the dimension mismatch at startup (reads `CREATE VIRTUAL TABLE` DDL from `sqlite_master`), deletes the vec DB, and queues a background re-embed.
|
|
|
|
## Licensing boundary
|
|
|
|
| Component | License |
|
|
|-----------|---------|
|
|
| BM25 search, shelve pipeline, library API | MIT |
|
|
| Hybrid vector search, RAG chat, embedding | BSL 1.1 (BYOK unlocked on Free tier) |
|