pyr0ball
f941ebdeeb
Extends Pagepiper's document shelving pipeline (renamed from "ingest" — see below) to cover the formats most likely to appear in a real-world engineering document corpus, prompted by scoping a STERIS licensing pitch that needs DOCX/ODT coverage. - Rename the ingest pipeline to "shelve" throughout (scripts/, app/api, tests, docs, frontend). "Glean" (Turnstone's term) was considered and rejected — that's a harvest metaphor for log/knowledge extraction, not a fit for documents entering a library. Documented as a general CF naming principle in the org-level CLAUDE.md. - Wire DOCX into the upload/scan UI, README, and docs — the extraction logic (heading-based chunking, table serialization) already existed but wasn't exposed to users or covered by tests. - Add ODT support via odfpy, mirroring DOCX's chunking strategy. - Add Apple Pages support via headless LibreOffice conversion to ODT. No maintained Python library parses the IWA format directly; libreoffice bundles libetonyek, the only real open-source Pages parser. Adds libreoffice-writer to the Docker image (~300-400MB) for this. - 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages; full suite (72 tests) passing. Known gaps not addressed here: no Windchill/DocPortal connector exists yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported, and circuitforge_core.tasks.dispatch_task does not currently exist in circuitforge-core — cf-orch dispatch is dead code, always falling through to local BackgroundTasks. See circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md for the full writeup.
35 lines
1.2 KiB
Docker
35 lines
1.2 KiB
Docker
FROM continuumio/miniconda3:latest
|
|
|
|
WORKDIR /app
|
|
|
|
# System deps for pytesseract (OCR), pdfplumber, and Apple Pages conversion
|
|
# (libreoffice-writer bundles libetonyek, the only maintained open-source
|
|
# .pages parser — shelve_pages.py shells out to headless soffice)
|
|
RUN apt-get update && apt-get install -y --no-install-recommends \
|
|
tesseract-ocr \
|
|
libgl1 \
|
|
libreoffice-writer \
|
|
&& rm -rf /var/lib/apt/lists/*
|
|
|
|
# Install circuitforge-core from sibling directory (compose sets context: ..)
|
|
COPY circuitforge-core/ ./circuitforge-core/
|
|
RUN conda run -n base pip install --no-cache-dir -e "./circuitforge-core[pdf,vector]"
|
|
|
|
# Create pagepiper conda env
|
|
COPY pagepiper/environment.yml .
|
|
RUN conda env create -f environment.yml
|
|
|
|
COPY pagepiper/ ./pagepiper/
|
|
|
|
# Remove gitignored secrets — defence-in-depth
|
|
RUN rm -f /app/pagepiper/.env /app/pagepiper/config/llm.yaml
|
|
|
|
# Install cf-core into pagepiper env + the app itself
|
|
RUN conda run -n pagepiper pip install --no-cache-dir -e "/app/circuitforge-core[pdf,vector]"
|
|
WORKDIR /app/pagepiper
|
|
RUN conda run -n pagepiper pip install --no-cache-dir -e .
|
|
|
|
ENV API_PORT=8522
|
|
EXPOSE $API_PORT
|
|
CMD conda run --no-capture-output -n pagepiper \
|
|
uvicorn app.main:app --host 0.0.0.0 --port ${API_PORT}
|