Found during freeze-branch integration testing (PR #12 + #14 + #15
combined): app/config.py's write-back (os.environ["CF_ORCH_URL"] =
GPU_SERVER_URL) is a module-level side effect monkeypatch never tracks,
so it wasn't being reverted between tests. Once any test_config.py test
set a truthy GPU_SERVER_URL/CF_LICENSE_KEY, the resulting CF_ORCH_URL
wrote leaked into every later test in the same pytest session.
This silently broke all five "skips_embeddings_without_ollama_url"
tests across the shelve_*.py suite (pdf, docx, odt, ods, xlsx) — their
get_llm_config() check no longer saw a clean "nothing configured" state,
so they attempted embedding instead of skipping it. Neither PR #12 nor
#14 alone exercised this combination in the same test session, so it
only surfaced once merged together.
Fix: explicitly pop CF_ORCH_URL/GPU_SERVER_URL/CF_LICENSE_KEY from
os.environ in the autouse teardown fixture before reloading app.config,
rather than relying on monkeypatch to catch a write it never made itself.
CF_ORCH_URL makes sense internally but isn't self-explanatory for a
local-first user setting up their own GPU rig. Follows the GPU_SERVER_URL
convention Kiwi already established (app/core/config.py there).
- app/config.py: resolves GPU_SERVER_URL -> CF_ORCH_URL (back-compat
alias) -> https://orch.circuitforge.tech default when CF_LICENSE_KEY
is present (Paid+ tiers). Written back to os.environ["CF_ORCH_URL"]
so existing callers (get_llm_config, app/api/chat.py) needed zero
changes.
- .env.example / .env.cloud.example: document GPU_SERVER_URL as the
primary name, CF_ORCH_URL noted as a still-honoured legacy alias.
- compose.cloud.yml: the hardcoded coordinator URL now sets
GPU_SERVER_URL instead of CF_ORCH_URL directly (relies on the same
config.py normalization).
- docs/reference/environment-variables.md updated.
- 6 new tests covering the resolution priority chain and the
write-back behavior legacy callers depend on.
Closes: #10
Returns up to N randomly sampled page chunks (default 50, max 200) with
chunk_id, doc_id, page_number, and text fields. No tier gate — internal
tool endpoint for same-host corpus benchmarking. Returns [] on empty library.
Implements Option A from the issue design: each cloud user gets their own
data directory (DATA_DIR/users/{user_id}/) with separate pagepiper.db,
pagepiper_vecs.db, uploads/, and books/. Local mode is unchanged.
Key changes:
- app/startup.py: extract apply_migrations, reembed_docs,
check_and_rebuild_vec_schema out of main.py (no circular imports)
- app/config.py: add LOCAL_USER_ID constant and user_data_dir() helper
- app/cloud_session.py: extract resolve_authenticated_user(); require_paid_tier
now returns user_id (str) instead of None
- app/deps.py: add UserCtx dataclass (db_path, vec_db_path, data_dir,
watch_dir, bm25) + get_user_ctx dependency; per-user startup guard runs
migrations + vec schema check once per process per user
- app/main.py: _bm25 singleton -> _bm25_map dict keyed by user_id;
add _get_bm25_for(); lifespan only runs startup checks in local mode
- app/api/library.py, search.py, chat.py: thread UserCtx through all
endpoints; remove module-level _mark_bm25_dirty injection pattern
- tests/conftest.py: override get_user_ctx in addition to get_db so all
endpoints get a consistent test UserCtx
Retrieval:
- Add _fetch_adjacent() to retriever: fetches page ± 1 chunks from DB
after ranking so mid-sentence EPUB chunk boundaries don't lose context
- Fix vec DB doc-filter: oversample to top_k*20 before Python filter
instead of post-filtering an already-small global pool (fixes wrong-book
results when searching within a single document)
- top_k default 5 → 10; context per chunk 500 → 1500 chars; citation
snippet 200 → 400 chars
Artifact cleaning:
- Add scripts/text_clean.py: strips ABC Amber LIT Converter watermarks,
processtext.com URLs, bare page numbers, piracy stamps from extracted text
- Wire clean_paragraph() into ingest_pdf.py and new ingest_epub.py
Startup validation:
- _check_vec_schema() at boot: detects embedding dimension mismatch,
deletes stale vec DB, and queues sequential re-embed in background thread
- Sequential _reembed_docs() prevents SQLite lock races on startup re-embed
cf-orch integration:
- Wire CF_ORCH_URL / CF_LICENSE_KEY into LLMRouter backend config so
allocate() fires and keeps the Ollama model warm between requests
Ingestion progress UI:
- GET /api/library/{doc_id}/status now returns vec_count from page_vecs_meta
- DocumentCard.vue polls status every 3 s while processing and shows
two-phase progress: indeterminate animation during extraction,
determinate "Embedding N/M pages" bar once vectors start landing
Other:
- Chat feedback endpoint + thumbs up/down UI (FeedbackButton.vue)
- EPUB ingest script (ingest_epub.py) with heading-based chunking
- migration 002: chat_feedback table
- README.md with setup and feature overview
- app/services/retriever.py: hybrid BM25 + semantic Retriever with BM25-only fallback when llm=None
- app/services/synthesizer.py: LLM answer synthesis with citation assembly over retrieved chunks
- app/api/chat.py: POST /api/chat endpoint with 402 gate when PAGEPIPER_OLLAMA_URL is unset
- tests/test_synthesizer.py: 3 TDD unit tests (mocked LLM, context building, system prompt)
- tests/test_chat_api.py: 2 integration tests (402 without Ollama, 200 with mocked retriever+LLM)
Implements GET/DELETE /api/library, POST /api/library/{id}/reingest,
POST /api/library/scan, and GET /api/library/{id}/status. Adds FastAPI
app factory with lifespan migrations, BM25 singleton wiring, get_db
dependency, ingest task registry with cf-orch/BackgroundTasks fallback,
and placeholder search/chat routers. All 5 new tests pass (14 total).
