pyr0ball
d39cfbd87a
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap called out in the PR's original "known gaps" list — Windchill/DocPortal corpora commonly include parts lists and spec sheets as spreadsheets, not just prose documents. - scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window splitting for large sheets (header row repeated in every window so each chunk stays self-describing for retrieval). - scripts/shelve_ods.py — same chunking strategy via odfpy (already a dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/ TableCell. - scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for .pages. Adds libreoffice-calc to the Docker image alongside the existing libreoffice-writer. - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth for the full list. - 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing. Manually verified via Playwright against an isolated test instance: XLSX and ODS both upload, shelve to "ready", and store correctly row-serialized, header-repeated chunks (confirmed via sample-chunks). BM25 search against a 2-chunk toy corpus returned no hits for terms split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula giving an exact 0 for terms in exactly half a tiny corpus (log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass (5 chunks total) diluted this enough to return real results.
37 lines
1.3 KiB
Docker
37 lines
1.3 KiB
Docker
FROM continuumio/miniconda3:latest
|
|
|
|
WORKDIR /app
|
|
|
|
# System deps for pytesseract (OCR), pdfplumber, and Apple Pages/Numbers
|
|
# conversion (libreoffice bundles libetonyek, the only maintained open-source
|
|
# parser for Apple's iWork formats — shelve_pages.py / shelve_numbers.py
|
|
# shell out to headless soffice for .pages / .numbers respectively)
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
tesseract-ocr \
|
|
libgl1 \
|
|
libreoffice-writer \
|
|
libreoffice-calc \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install circuitforge-core from sibling directory (compose sets context: ..)
|
|
COPY circuitforge-core/ ./circuitforge-core/
|
|
RUN conda run -n base pip install --no-cache-dir -e "./circuitforge-core[pdf,vector]"
|
|
|
|
# Create pagepiper conda env
|
|
COPY pagepiper/environment.yml .
|
|
RUN conda env create -f environment.yml
|
|
|
|
COPY pagepiper/ ./pagepiper/
|
|
|
|
# Remove gitignored secrets — defence-in-depth
|
|
RUN rm -f /app/pagepiper/.env /app/pagepiper/config/llm.yaml
|
|
|
|
# Install cf-core into pagepiper env + the app itself
|
|
RUN conda run -n pagepiper pip install --no-cache-dir -e "/app/circuitforge-core[pdf,vector]"
|
|
WORKDIR /app/pagepiper
|
|
RUN conda run -n pagepiper pip install --no-cache-dir -e .
|
|
|
|
ENV API_PORT=8522
|
|
EXPOSE $API_PORT
|
|
CMD conda run --no-capture-output -n pagepiper \
|
|
uvicorn app.main:app --host 0.0.0.0 --port ${API_PORT}
|