pyr0ball
e52bdb5128
Retrieval:
- Add _fetch_adjacent() to retriever: fetches page ± 1 chunks from DB
after ranking so mid-sentence EPUB chunk boundaries don't lose context
- Fix vec DB doc-filter: oversample to top_k*20 before Python filter
instead of post-filtering an already-small global pool (fixes wrong-book
results when searching within a single document)
- top_k default 5 → 10; context per chunk 500 → 1500 chars; citation
snippet 200 → 400 chars
Artifact cleaning:
- Add scripts/text_clean.py: strips ABC Amber LIT Converter watermarks,
processtext.com URLs, bare page numbers, piracy stamps from extracted text
- Wire clean_paragraph() into ingest_pdf.py and new ingest_epub.py
Startup validation:
- _check_vec_schema() at boot: detects embedding dimension mismatch,
deletes stale vec DB, and queues sequential re-embed in background thread
- Sequential _reembed_docs() prevents SQLite lock races on startup re-embed
cf-orch integration:
- Wire CF_ORCH_URL / CF_LICENSE_KEY into LLMRouter backend config so
allocate() fires and keeps the Ollama model warm between requests
Ingestion progress UI:
- GET /api/library/{doc_id}/status now returns vec_count from page_vecs_meta
- DocumentCard.vue polls status every 3 s while processing and shows
two-phase progress: indeterminate animation during extraction,
determinate "Embedding N/M pages" bar once vectors start landing
Other:
- Chat feedback endpoint + thumbs up/down UI (FeedbackButton.vue)
- EPUB ingest script (ingest_epub.py) with heading-based chunking
- migration 002: chat_feedback table
- README.md with setup and feature overview
38 lines
1.6 KiB
Text
38 lines
1.6 KiB
Text
# pagepiper cloud environment — copy to .env and fill in secrets
|
|
# Used by: docker compose -f compose.cloud.yml -p pagepiper-cloud ...
|
|
|
|
# Data directories (host paths, bind-mounted into the api container)
|
|
PAGEPIPER_DATA_DIR=/devl/pagepiper-cloud-data
|
|
PAGEPIPER_BOOKS_DIR=/devl/pagepiper-cloud-data/books
|
|
|
|
# BYOK gate — set to enable hybrid search and RAG chat (BSL feature)
|
|
# Leave blank to run BM25-only mode (MIT, no Ollama required)
|
|
PAGEPIPER_OLLAMA_URL=
|
|
|
|
# Embedding and chat model selection (only used when PAGEPIPER_OLLAMA_URL is set)
|
|
# mxbai-embed-large (1024-dim) is recommended; nomic-embed-text uses 768-dim
|
|
PAGEPIPER_EMBED_MODEL=mxbai-embed-large
|
|
PAGEPIPER_EMBED_DIMS=1024
|
|
PAGEPIPER_CHAT_MODEL=mistral:7b
|
|
|
|
# Heimdall license server (optional — for per-user tier validation)
|
|
HEIMDALL_URL=https://license.circuitforge.tech
|
|
HEIMDALL_ADMIN_TOKEN=
|
|
|
|
# cf-orch streaming proxy — coordinator product key
|
|
# Must match COORDINATOR_PRODUCT_KEYS["pagepiper"] in cf-orch.env on the coordinator
|
|
COORDINATOR_PAGEPIPER_KEY=
|
|
|
|
# cf-orch coordinator URL — routes chat/embed calls through managed GPU allocation
|
|
# CF_LICENSE_KEY is the auth token sent to the coordinator (same value as COORDINATOR_PAGEPIPER_KEY)
|
|
# Leave CF_ORCH_URL blank to skip allocation and hit PAGEPIPER_OLLAMA_URL directly
|
|
CF_ORCH_URL=
|
|
CF_LICENSE_KEY=
|
|
CF_APP_NAME=pagepiper
|
|
|
|
# Forgejo API token — enables in-app feedback button (files issues to Circuit-Forge/pagepiper)
|
|
FORGEJO_API_TOKEN=
|
|
|
|
# Enable thumbs up/down on chat answers (stores retrieval quality signals locally)
|
|
# Off by default — opt in when you want to collect correction data
|
|
# PAGEPIPER_CHAT_FEEDBACK=true
|