Add GET /api/search/async that returns HTTP 202 immediately and streams
scrape results via SSE to avoid nginx 120s timeouts on slow eBay searches.
Backend:
- New GET /api/search/async endpoint submits scraping to ThreadPoolExecutor
and returns {session_id, status: "queued"} before scrape begins
- Background worker runs same pipeline as synchronous search, pushing
typed SSE events: "listings" (initial batch), "update" (enrichment),
"market_price", and None sentinel
- Existing GET /api/updates/{session_id} passes new event types through
as-is (already a generic pass-through); deadline extended to 150s
- Module-level _search_executor (max_workers=4) caps concurrent scrape sessions
Frontend (search.ts):
- search() now calls /api/search/async instead of /api/search
- loading stays true until first "listings" SSE event arrives
- _openUpdates() handles new typed events: "listings", "market_price",
"update"; legacy untyped enrichment events still handled
- cancelSearch() now also closes any open SSE stream
Tests: tests/test_async_search.py (6 tests) covering 202 response,
session_id registration in _update_queues, empty query path, UUID format,
and no-Chromium guarantee. All 159 pre-existing tests still pass.
Closes#49. Also closes Forgejo issue #1 (SSE enrichment streaming, already
implemented; async search completes the picture).
Under high concurrency (100+ users), shared_db write contention causes
database is locked errors in the unguarded post-scrape block. These were
surfacing as 500s because there was no exception handler after line 663.
Now catches OperationalError and returns raw listings with empty trust
scores/sellers (degraded mode) instead of crashing. The SSE queue entry
is cleaned up on this path so no orphaned queue accumulates.
Root cause: shared_db (sellers, market_comps) is SQLite; at 100 concurrent
writers the WAL write queue exceeds the 30s busy timeout. Long-term fix
is migrating shared state to Postgres (see snipe#NN).
Refs: infra#12 load test Phase 2 spike findings
Phase 2 (snipe#4): after bulk-reporting sellers to eBay T&S, Snipe now
persists which sellers were reported so cards show a muted "Reported to
eBay" badge and users aren't prompted to re-report the same seller.
- migration 012 adds reported_sellers table (user DB, UNIQUE on seller)
- Store.mark_reported / list_reported methods
- POST /api/reported + GET /api/reported endpoints
- reported store (frontend) with optimistic update + server persistence
- reportSelected wires into store after opening eBay tabs
Phase 3 prep (snipe#4): community blocklist share toggle
- Settings > Community section: "Share blocklist with community" toggle
(visible only to signed-in cloud users, default OFF)
- Persisted as community.blocklist_share user preference
- Backend community signal publish now gated on opt-in preference;
privacy-by-architecture: sharing is explicit, never implicit
Add _auth_label() classifier: local/anon/guest/authed — no PII, just
enough to distinguish traffic types in docker logs for log-based analytics.
Instrument /api/session: logs new_guest (with UUID) or auth=.../tier=...
on every session bootstrap. Instrument /api/search: expands existing
multi-search log line with auth=, tier=, adapter=, pages=, queries=,
listings= fields for grep/awk analysis of search behaviour by tier.
Add logging.basicConfig so app-level log.info() calls appear in docker
logs alongside the Uvicorn access log (previously suppressed by missing
root handler).
Corrections (#31):
- Add 010_corrections.sql migration (from cf-core CORRECTIONS_MIGRATION_SQL)
- Wire make_corrections_router() at /api/corrections (shared_db, product='snipe')
- get_shared_db() dependency aggregates corrections across all cloud users
Community module (#32#33):
- Init SnipeCommunityStore at startup when COMMUNITY_DB_URL is set
- Graceful skip if COMMUNITY_DB_URL is unset (local mode, community disabled)
- add_to_blocklist() publishes confirmed_scam=True seller_trust signal to
community postgres on every manual blocklist addition (fire-and-forget)
- BlocklistAdd gains flags[] field so active red-flag keys travel with signal
cf-orch community postgres (cf-orch#36) + cf-core module (cf-core#47) both merged.
After a search, the API now returns a session_id. If any trust scores are
partial (pending seller age or category data), the frontend opens a
Server-Sent Events stream to /api/updates/{session_id}. As the background
BTF (account age) and category enrichment threads complete, they re-score
affected listings and push updated TrustScore payloads over SSE. The
frontend patches the trustScores and sellers maps reactively so signal
dots light up without requiring a manual re-search.
Backend:
- _update_queues registry maps session_id -> SimpleQueue (thread-safe bridge)
- _trigger_scraper_enrichment accepts session_id/user_db/query, builds a
seller->listings map, calls _push_updates() after each enrichment pass
which re-scores, saves trust scores, and puts events on the queue
- New GET /api/updates/{session_id} SSE endpoint: polls queue every 500ms,
emits heartbeats every 15s, closes on sentinel None or 90s timeout
- search endpoint generates session_id and returns it in response
Frontend:
- search store adds enriching state and _openUpdates() / closeUpdates()
- On search completion, if partial scores exist, opens EventSource stream
- onmessage: patches trustScores and sellers maps (new Map() to trigger
Vue reactivity), updates marketPrice if included
- on 'done' event or error: closes stream, enriching = false
- SearchView: pulsing 'Updating scores...' badge in toolbar while enriching
- _affiliate_url() helper appends EPN params when EBAY_AFFILIATE_CAMPAIGN_ID set
- Clean /itm/ URLs by default (no affiliate tracking without explicit opt-in)
- affiliate_active flag in search response drives frontend disclosure
- SearchView shows 'Links may include an affiliate code' when active
- .env.example documents EBAY_AFFILIATE_CAMPAIGN_ID with EPN registration link
- Closes#19
- api/main.py: GET /api/feedback/status + POST /api/feedback — creates
Forgejo issues; disabled (503) when FORGEJO_API_TOKEN unset, 403 in
demo mode; includes view, version, platform context in issue body
- FeedbackButton.vue: 2-step modal (type → review → submit); probes
/api/feedback/status on mount, stays hidden until confirmed enabled
- App.vue: mount FeedbackButton with current route name as view context;
import useRoute for reactive route name tracking
- .env.example: document FORGEJO_API_TOKEN / FORGEJO_REPO / FORGEJO_API_URL
- Rename 002_background_tasks.sql → 007_background_tasks.sql to avoid
collision with existing 002_add_listing_format.sql migration
- Add CREATE UNIQUE INDEX on trust_scores(listing_id) in same migration
so save_trust_scores() can use ON CONFLICT upsert semantics
- Add Store.save_trust_scores() — upserts scores keyed by listing_id;
preserves photo_analysis_json so runner writes are never clobbered
- runner.py: replace raw sqlite3.connect() with get_connection() throughout
(timeout=30 + WAL mode); fix connection leak in insert_task via try/finally
- _run_trust_photo_analysis: read 'user_db' from params to write results to
the correct per-user DB in cloud mode (was silently writing to wrong DB)
- main.py lifespan: use _shared_db_path() in cloud mode so background_tasks
queue lives in shared DB, not _LOCAL_SNIPE_DB
- Add _enqueue_vision_tasks() and call it after score_batch() — this is the
missing enqueue call site; gated by features.photo_analysis (Paid tier)
- Test fixture: add missing 'stage' column to background_tasks schema
- Two sidebar fields: 'Must include' and 'Must exclude' (comma-separated)
- Must-exclude terms forwarded to eBay _nkw as -term prefixes (native eBay
support) so exclusions reduce the eBay result set at the source — improves
market comp quality as a side effect
- Must-include applied client-side only (substring, case-insensitive)
- Both applied client-side via passesFilter() for instant response without
re-fetching (cache-friendly)
- Exclude input has subtle red border tint (color-mix) to signal intent
- Hint text: 're-search to apply to eBay' reminds user negatives need a
new search to take effect at the eBay level
- Parallel execution: search() and get_completed_sales() now run
concurrently via ThreadPoolExecutor — each gets its own Store/SQLite
connection for thread safety. First cold search time ~halved.
- Pagination: SearchFilters.pages (default 1) controls how many eBay
result pages are fetched. Both search and sold-comps support up to 3
parallel Playwright sessions per call (capped to avoid Xvfb overload).
UI: segmented 1/2/3/5 pages selector in filter sidebar with cost hint.
- True median: get_completed_sales() now averages the two middle values
for even-length price lists instead of always taking the lower bound.
- Fix suspicious_price false positive: aggregator now checks
signal_scores.get("price_vs_market") == 0 (pre-None-substitution)
so listings without market data are never flagged as suspicious.
- Fix title pollution: scraper strips eBay's hidden screen-reader span
("Opens in a new window or tab") from listing titles via regex.
Lazy-imports playwright/playwright_stealth inside _get() so pure
parsing functions are importable without the full browser stack.
- Tests: 48 pass on host (scraper tests now runnable without Docker),
new regression guards for all three bug fixes.
