Circuit-Forge/pagepiper
6
0
Fork 0
Code Issues 3 Pull requests Projects Releases 2 Packages Wiki Activity Actions

2 releases 2 tags

RSS feed
  • v0.2.0 afdc211813

    v0.2.0 — Multi-format shelving + Agent-ModernColBERT retriever Stable

    pyr0ball released this 2026-07-10 19:39:09 -07:00 | 1 commits to main since this release

    Highlights

    Document and spreadsheet format support

    Shelving (the extract→chunk→store→embed pipeline, renamed from "ingest" — see below) now covers: PDF, EPUB, DOCX, ODT, Apple Pages, XLSX, ODS, and Apple Numbers. Pages and Numbers convert via headless LibreOffice, since no maintained Python library parses Apple's iWork formats directly.

    Semantic search: Agent-ModernColBERT replaces nomic-embed-text

    Hybrid search's semantic half now uses a late-interaction retriever (lightonai/Agent-ModernColBERT, via pylate) instead of a bi-encoder + cosine similarity. Late-interaction scoring keeps per-token embeddings and computes relevance at query time, which holds up much better on multi-part questions than collapsing a whole query into one vector. Existing BYOK tier gating is unchanged.

    Naming: ingest → shelve

    The document pipeline is now called "shelve" throughout (API routes, scripts, UI) instead of "ingest" — a library/bookshelf metaphor that fits Pagepiper's actual job, versus "ingest"'s vaguer connotation. /api/library/{id}/reingest is now /api/library/{id}/reshelve; /api/ingest/{task_id} is now /api/shelve/{task_id}.

    Config: CF_ORCH_URL → GPU_SERVER_URL

    The GPU/cf-orch coordinator env var is now GPU_SERVER_URL — clearer for self-hosters running their own GPU rig. CF_ORCH_URL is still honored as a legacy alias.

    Breaking changes

    • /api/library/{id}/reingest → /api/library/{id}/reshelve
    • /api/ingest/{task_id} → /api/shelve/{task_id}
    • If you scripted against either endpoint directly, update the path. The web UI is unaffected.

    Known follow-ups (tracked, not in this release)

    • No live Windchill/DocPortal document connector yet (recon only)
    • Shelve scripts still compute and store now-unused Ollama embeddings into page_vecs at shelve time — cleanup pending
    • circuitforge_core.tasks.dispatch_task/get_task_status don't exist yet in circuitforge-core (tracked as circuitforge-core#67) — cf-orch dispatch silently falls back to local BackgroundTasks

    Stats

    103 tests passing. Freeze-tested (PR #12 + #14 + #15 merged together and validated as a unit, including a real Playwright pass and a Docker build) before merging to main — caught and fixed one cross-PR test-isolation bug in the process.

    Downloads