Circuit-Forge/pagepiper
6
0
Fork 0
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions

feat: shelve rename + DOCX/ODT/Pages/XLSX/ODS/Numbers support #12

Merged
pyr0ball merged 4 commits from feat/shelve-multi-format-support into main 2026-07-10 19:34:00 -07:00
Conversation 0 Commits 4 Files changed 41 +2208 -205
pyr0ball commented 2026-07-10 14:00:56 -07:00
Owner
Copy link

Summary

  • Rename the ingest pipeline to shelve throughout (scripts, API, tests, docs, frontend). "Glean" was considered and rejected — that's Turnstone's harvest metaphor, not a fit for documents entering a library. Documented as a general CF naming principle in the org-level CLAUDE.md.
  • Wire DOCX into the upload/scan UI, README, and docs — extraction logic already existed but wasn't exposed or tested.
  • Add ODT support via odfpy, mirroring DOCX's heading-based chunking.
  • Add Apple Pages support via headless LibreOffice conversion to ODT (no maintained Python IWA parser exists; adds libreoffice-writer to the Docker image).
  • Add XLSX support via openpyxl — chunked by sheet, row-windowed for large sheets with the header row repeated in every window.
  • Add ODS support via odfpy, same chunking strategy as XLSX.
  • Add Apple Numbers support via headless LibreOffice conversion to XLSX (mirrors the Pages pattern; adds libreoffice-calc).
  • Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth.
  • 37 new/updated tests; full suite (85 tests) passing.

Prompted by scoping a STERIS licensing pitch that needs broad document and spreadsheet format coverage (parts lists, spec sheets) — see circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md.

Known gaps (not addressed here)

  • No Windchill/DocPortal connector exists yet (metadata-only PowerShell recon only)
  • circuitforge_core.tasks.dispatch_task does not currently exist in circuitforge-core — cf-orch dispatch is dead code, always falling through to local BackgroundTasks

Test plan

  • pytest tests/ -q — 85 passed
  • Manual: built the Docker image on this branch and confirmed libreoffice-writer/libreoffice-calc + soffice install correctly (LibreOffice 25.2.3.2)
  • Manual: stood up isolated test instances (docker compose -p pagepiper-test*, separate ports/env, no shared state with the live dev/cloud instances) and drove them with Playwright — uploaded synthetic DOCX, ODT, PDF, XLSX, and ODS fixtures through the browser; all reached ready with correct page/section/sheet counts; re-shelve round-tripped; BM25 search confirmed working on the larger (5-chunk) DOCX/ODT/PDF corpus (a 2-chunk XLSX/ODS toy corpus hit an expected BM25 IDF-formula edge case, not a defect — see commit message for the math)
## Summary - Rename the ingest pipeline to **shelve** throughout (scripts, API, tests, docs, frontend). "Glean" was considered and rejected — that's Turnstone's harvest metaphor, not a fit for documents entering a library. Documented as a general CF naming principle in the org-level CLAUDE.md. - Wire **DOCX** into the upload/scan UI, README, and docs — extraction logic already existed but wasn't exposed or tested. - Add **ODT** support via odfpy, mirroring DOCX's heading-based chunking. - Add **Apple Pages** support via headless LibreOffice conversion to ODT (no maintained Python IWA parser exists; adds `libreoffice-writer` to the Docker image). - Add **XLSX** support via openpyxl — chunked by sheet, row-windowed for large sheets with the header row repeated in every window. - Add **ODS** support via odfpy, same chunking strategy as XLSX. - Add **Apple Numbers** support via headless LibreOffice conversion to XLSX (mirrors the Pages pattern; adds `libreoffice-calc`). - Upload button text changed from an ever-growing format list to "Upload Document or Spreadsheet" — the Supported Formats table in README/docs is now the source of truth. - 37 new/updated tests; full suite (85 tests) passing. Prompted by scoping a STERIS licensing pitch that needs broad document *and spreadsheet* format coverage (parts lists, spec sheets) — see `circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md`. ## Known gaps (not addressed here) - No Windchill/DocPortal connector exists yet (metadata-only PowerShell recon only) - `circuitforge_core.tasks.dispatch_task` does not currently exist in circuitforge-core — cf-orch dispatch is dead code, always falling through to local `BackgroundTasks` ## Test plan - [x] `pytest tests/ -q` — 85 passed - [x] Manual: built the Docker image on this branch and confirmed `libreoffice-writer`/`libreoffice-calc` + `soffice` install correctly (`LibreOffice 25.2.3.2`) - [x] Manual: stood up isolated test instances (`docker compose -p pagepiper-test*`, separate ports/env, no shared state with the live dev/cloud instances) and drove them with Playwright — uploaded synthetic DOCX, ODT, PDF, XLSX, and ODS fixtures through the browser; all reached `ready` with correct page/section/sheet counts; re-shelve round-tripped; BM25 search confirmed working on the larger (5-chunk) DOCX/ODT/PDF corpus (a 2-chunk XLSX/ODS toy corpus hit an expected BM25 IDF-formula edge case, not a defect — see commit message for the math)
pyr0ball added 2 commits 2026-07-10 14:00:56 -07:00 
When CF_ORCH_URL is set, chat now calls CFOrchClient.task_allocate("pagepiper",
"rag_query") instead of routing through LLMRouter with an explicit model. The
coordinator resolves the assignment (granite-4.1-8b via assignments.yaml) and
returns an allocated URL; pagepiper wraps it in a minimal LLMRouter config for
the Synthesizer. Falls back to LLMRouter on TaskNotFound or allocation failure,
so standalone Ollama installs are unaffected.

Extracts _run_chat() and _build_llm_for_alloc() helpers to keep the endpoint
body readable regardless of which path fires.
Extends Pagepiper's document shelving pipeline (renamed from "ingest" —
see below) to cover the formats most likely to appear in a real-world
engineering document corpus, prompted by scoping a STERIS licensing pitch
that needs DOCX/ODT coverage.

- Rename the ingest pipeline to "shelve" throughout (scripts/, app/api,
  tests, docs, frontend). "Glean" (Turnstone's term) was considered and
  rejected — that's a harvest metaphor for log/knowledge extraction,
  not a fit for documents entering a library. Documented as a general
  CF naming principle in the org-level CLAUDE.md.
- Wire DOCX into the upload/scan UI, README, and docs — the extraction
  logic (heading-based chunking, table serialization) already existed
  but wasn't exposed to users or covered by tests.
- Add ODT support via odfpy, mirroring DOCX's chunking strategy.
- Add Apple Pages support via headless LibreOffice conversion to ODT.
  No maintained Python library parses the IWA format directly; libreoffice
  bundles libetonyek, the only real open-source Pages parser. Adds
  libreoffice-writer to the Docker image (~300-400MB) for this.
- 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages;
  full suite (72 tests) passing.

Known gaps not addressed here: no Windchill/DocPortal connector exists
yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported,
and circuitforge_core.tasks.dispatch_task does not currently exist in
circuitforge-core — cf-orch dispatch is dead code, always falling
through to local BackgroundTasks. See
circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md
for the full writeup.
pyr0ball added 1 commit 2026-07-10 14:18:54 -07:00 
Discovered while manually verifying the DOCX/ODT/Pages upload flow —
.playwright-mcp/ (fixtures, screenshots) wasn't ignored and risked
accidental commits of test artifacts.
pyr0ball added 1 commit 2026-07-10 15:06:20 -07:00 
Extends the shelve pipeline to cover spreadsheets, closing the Excel gap
called out in the PR's original "known gaps" list — Windchill/DocPortal
corpora commonly include parts lists and spec sheets as spreadsheets, not
just prose documents.

- scripts/shelve_xlsx.py — openpyxl, chunked by sheet with row-window
  splitting for large sheets (header row repeated in every window so
  each chunk stays self-describing for retrieval).
- scripts/shelve_ods.py — same chunking strategy via odfpy (already a
  dependency from ODT support), OpenDocumentSpreadsheet's Table/TableRow/
  TableCell.
- scripts/shelve_numbers.py — converts via headless LibreOffice to XLSX
  and delegates to shelve_xlsx, mirroring shelve_pages.py's pattern for
  .pages. Adds libreoffice-calc to the Docker image alongside the
  existing libreoffice-writer.
- Upload button text changed from an ever-growing format list to
  "Upload Document or Spreadsheet" — the Supported Formats table in
  README/docs is now the source of truth for the full list.
- 13 new tests (XLSX, ODS, Numbers); full suite (85 tests) passing.

Manually verified via Playwright against an isolated test instance:
XLSX and ODS both upload, shelve to "ready", and store correctly
row-serialized, header-repeated chunks (confirmed via sample-chunks).
BM25 search against a 2-chunk toy corpus returned no hits for terms
split 1-vs-1 across the two chunks — traced to Okapi BM25's IDF formula
giving an exact 0 for terms in exactly half a tiny corpus
(log((N-n+0.5)/(n+0.5)) = log(1.0) = 0, filtered by `score <= 0`), not a
defect in the new shelvers. The earlier DOCX/ODT/PDF Playwright pass
(5 chunks total) diluted this enough to return real results.
pyr0ball changed title from feat: add ODT and Apple Pages document support, wire DOCX into UI to feat: shelve rename + DOCX/ODT/Pages/XLSX/ODS/Numbers support 2026-07-10 15:06:41 -07:00
pyr0ball merged commit 156f52d6ac into main 2026-07-10 19:34:00 -07:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
backlog

Someday/maybe idea, not a committed roadmap item

No labels
backlog
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Circuit-Forge/pagepiper#12
No description provided.
Delete branch "feat/shelve-multi-format-support"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?