pyr0ball
f941ebdeeb
Extends Pagepiper's document shelving pipeline (renamed from "ingest" — see below) to cover the formats most likely to appear in a real-world engineering document corpus, prompted by scoping a STERIS licensing pitch that needs DOCX/ODT coverage. - Rename the ingest pipeline to "shelve" throughout (scripts/, app/api, tests, docs, frontend). "Glean" (Turnstone's term) was considered and rejected — that's a harvest metaphor for log/knowledge extraction, not a fit for documents entering a library. Documented as a general CF naming principle in the org-level CLAUDE.md. - Wire DOCX into the upload/scan UI, README, and docs — the extraction logic (heading-based chunking, table serialization) already existed but wasn't exposed to users or covered by tests. - Add ODT support via odfpy, mirroring DOCX's chunking strategy. - Add Apple Pages support via headless LibreOffice conversion to ODT. No maintained Python library parses the IWA format directly; libreoffice bundles libetonyek, the only real open-source Pages parser. Adds libreoffice-writer to the Docker image (~300-400MB) for this. - 24 new/updated tests across shelve_docx, shelve_odt, and shelve_pages; full suite (72 tests) passing. Known gaps not addressed here: no Windchill/DocPortal connector exists yet (metadata-only PowerShell recon only), Excel/.xlsx is unsupported, and circuitforge_core.tasks.dispatch_task does not currently exist in circuitforge-core — cf-orch dispatch is dead code, always falling through to local BackgroundTasks. See circuitforge-plans/pagepiper/superpowers/plans/2026-07-10-steris-licensing-pitch.md for the full writeup.
24 lines
807 B
Python
24 lines
807 B
Python
# app/api/shelve.py
|
|
"""Shelve job status polling (proxies cf-orch or checks in-memory registry)."""
|
|
from __future__ import annotations
|
|
|
|
from fastapi import APIRouter, HTTPException
|
|
|
|
router = APIRouter(prefix="/api/shelve", tags=["shelve"])
|
|
|
|
# Populated by _run_shelve_background when cf-orch is unavailable
|
|
_task_registry: dict[str, dict] = {}
|
|
|
|
|
|
@router.get("/{task_id}")
|
|
def get_task_status(task_id: str) -> dict:
|
|
# Check in-memory registry first (BackgroundTasks fallback)
|
|
if task_id in _task_registry:
|
|
return _task_registry[task_id]
|
|
|
|
# Try cf-orch
|
|
try:
|
|
from circuitforge_core.tasks import get_task_status as orch_status # type: ignore[import]
|
|
return orch_status(task_id)
|
|
except Exception:
|
|
raise HTTPException(status_code=404, detail="Task not found")
|