pyr0ball
d39cfbd87a
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap called out in the PR's original "known gaps" list — Windchill/DocPortal corpora commonly include parts lists and spec sheets as spreadsheets, not just prose documents. - scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window splitting for large sheets (header row repeated in every window so each chunk stays self-describing for retrieval). - scripts/shelve_ods.py — same chunking strategy via odfpy (already a dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/ TableCell. - scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for .pages. Adds libreoffice-calc to the Docker image alongside the existing libreoffice-writer. - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth for the full list. - 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing. Manually verified via Playwright against an isolated test instance: XLSX and ODS both upload, shelve to "ready", and store correctly row-serialized, header-repeated chunks (confirmed via sample-chunks). BM25 search against a 2-chunk toy corpus returned no hits for terms split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula giving an exact 0 for terms in exactly half a tiny corpus (log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass (5 chunks total) diluted this enough to return real results.
67 lines
1.8 KiB
YAML
67 lines
1.8 KiB
YAML
site_name: Pagepiper
|
|
site_description: Self-hosted document and spreadsheet library (PDF, EPUB, DOCX, ODT, Apple Pages, XLSX, ODS, Apple Numbers) with BM25 full-text search, hybrid vector retrieval, and LLM-powered RAG chat.
|
|
site_author: Circuit Forge LLC
|
|
site_url: https://docs.circuitforge.tech/pagepiper
|
|
repo_url: https://git.opensourcesolarpunk.com/Circuit-Forge/pagepiper
|
|
repo_name: Circuit-Forge/pagepiper
|
|
|
|
theme:
|
|
name: material
|
|
palette:
|
|
- scheme: default
|
|
primary: brown
|
|
accent: orange
|
|
toggle:
|
|
icon: material/brightness-7
|
|
name: Switch to dark mode
|
|
- scheme: slate
|
|
primary: brown
|
|
accent: amber
|
|
toggle:
|
|
icon: material/brightness-4
|
|
name: Switch to light mode
|
|
features:
|
|
- navigation.tabs
|
|
- navigation.sections
|
|
- navigation.expand
|
|
- navigation.top
|
|
- search.suggest
|
|
- search.highlight
|
|
- content.code.copy
|
|
|
|
markdown_extensions:
|
|
- admonition
|
|
- pymdownx.details
|
|
- pymdownx.superfences:
|
|
custom_fences:
|
|
- name: mermaid
|
|
class: mermaid
|
|
format: !!python/name:pymdownx.superfences.fence_code_format
|
|
- pymdownx.highlight:
|
|
anchor_linenums: true
|
|
- pymdownx.tabbed:
|
|
alternate_style: true
|
|
- tables
|
|
- toc:
|
|
permalink: true
|
|
|
|
nav:
|
|
- Home: index.md
|
|
- Getting Started:
|
|
- Installation: getting-started/installation.md
|
|
- Quick Start: getting-started/quick-start.md
|
|
- Ollama Setup: getting-started/ollama-setup.md
|
|
- User Guide:
|
|
- Library: user-guide/library.md
|
|
- Search: user-guide/search.md
|
|
- Chat: user-guide/chat.md
|
|
- Reference:
|
|
- Architecture: reference/architecture.md
|
|
- Tier System: reference/tier-system.md
|
|
- Environment Variables: reference/environment-variables.md
|
|
|
|
extra_css:
|
|
- stylesheets/theme.css
|
|
|
|
extra_javascript:
|
|
- plausible.js
|