pyr0ball
e52bdb5128
Retrieval:
- Add _fetch_adjacent() to retriever: fetches page ± 1 chunks from DB
after ranking so mid-sentence EPUB chunk boundaries don't lose context
- Fix vec DB doc-filter: oversample to top_k*20 before Python filter
instead of post-filtering an already-small global pool (fixes wrong-book
results when searching within a single document)
- top_k default 5 → 10; context per chunk 500 → 1500 chars; citation
snippet 200 → 400 chars
Artifact cleaning:
- Add scripts/text_clean.py: strips ABC Amber LIT Converter watermarks,
processtext.com URLs, bare page numbers, piracy stamps from extracted text
- Wire clean_paragraph() into ingest_pdf.py and new ingest_epub.py
Startup validation:
- _check_vec_schema() at boot: detects embedding dimension mismatch,
deletes stale vec DB, and queues sequential re-embed in background thread
- Sequential _reembed_docs() prevents SQLite lock races on startup re-embed
cf-orch integration:
- Wire CF_ORCH_URL / CF_LICENSE_KEY into LLMRouter backend config so
allocate() fires and keeps the Ollama model warm between requests
Ingestion progress UI:
- GET /api/library/{doc_id}/status now returns vec_count from page_vecs_meta
- DocumentCard.vue polls status every 3 s while processing and shows
two-phase progress: indeterminate animation during extraction,
determinate "Embedding N/M pages" bar once vectors start landing
Other:
- Chat feedback endpoint + thumbs up/down UI (FeedbackButton.vue)
- EPUB ingest script (ingest_epub.py) with heading-based chunking
- migration 002: chat_feedback table
- README.md with setup and feature overview
50 lines
1.5 KiB
Python
50 lines
1.5 KiB
Python
# tests/conftest.py
|
|
"""Shared fixtures for pagepiper test suite."""
|
|
from __future__ import annotations
|
|
|
|
import sqlite3
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from fastapi.testclient import TestClient
|
|
|
|
|
|
@pytest.fixture
|
|
def test_db(tmp_path) -> str:
|
|
db_path = str(tmp_path / "test.db")
|
|
schema = Path("migrations/001_initial_schema.sql").read_text()
|
|
conn = sqlite3.connect(db_path)
|
|
conn.executescript(schema)
|
|
conn.commit()
|
|
conn.close()
|
|
return db_path
|
|
|
|
|
|
@pytest.fixture
|
|
def client(test_db, tmp_path, monkeypatch):
|
|
monkeypatch.setenv("PAGEPIPER_DATA_DIR", str(tmp_path))
|
|
monkeypatch.setenv("PAGEPIPER_WATCH_DIR", str(tmp_path / "books"))
|
|
(tmp_path / "books").mkdir(exist_ok=True)
|
|
|
|
import app.main as _main_module
|
|
from app.main import app, _bm25
|
|
from app.deps import get_db
|
|
|
|
# Suppress startup side effects — test_db fixture already applies the schema,
|
|
# and vec schema validation is tested separately in test_startup.py
|
|
monkeypatch.setattr(_main_module, "_apply_migrations", lambda: None)
|
|
monkeypatch.setattr(_main_module, "_check_vec_schema", lambda *a, **kw: None)
|
|
|
|
def override_db():
|
|
conn = sqlite3.connect(test_db)
|
|
conn.execute("PRAGMA foreign_keys = ON")
|
|
conn.row_factory = sqlite3.Row
|
|
try:
|
|
yield conn
|
|
finally:
|
|
conn.close()
|
|
|
|
app.dependency_overrides[get_db] = override_db
|
|
_bm25.mark_dirty() # clear any state from previous tests
|
|
yield TestClient(app)
|
|
app.dependency_overrides.clear()
|