SSE/WebSocket live score push for background enrichment #1

New issue

Open

opened 2026-03-26 15:05:38 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-03-26 15:05:38 -07:00

Owner

Problem

Background seller enrichment (BTF /itm/ scrape + _ssn category scrape + Shopping API age lookup) runs after each search in a daemon thread. Enriched data only appears on re-search — users see signal dots greyed out with "↻ Updating: age, categories" and have no way to get updated scores without manually repeating the search.

Proposed Solution

Add a Server-Sent Events (SSE) endpoint: GET /api/search/updates?query=... that streams trust score updates as background enrichment completes.

Backend

_trigger_scraper_enrichment emits events via an asyncio Queue as each seller enriches
FastAPI SSE endpoint reads from queue, streams data: {listing_id, trust_score} JSON events

Frontend

useSearchStore opens SSE connection after search completes
Merges incoming trust_score updates into trustScores map reactively
Signal dots on ListingCard update live without re-search
SSE connection closes when all pending signals are resolved or after 60s timeout

Tier Gate

Free tier — this is a UX improvement for the existing data pipeline, not a new paid feature.

Alternatives Considered

Polling (GET /api/search/status?query=... every 5s): simpler but wastes requests
WebSocket: bidirectional not needed; SSE is simpler for server-push only

Notes

Currently mitigated by: signal dots on trust badge show which signals are pending, hover tooltip says "search again to update", score badge pulses while partial.

## Problem Background seller enrichment (BTF `/itm/` scrape + `_ssn` category scrape + Shopping API age lookup) runs after each search in a daemon thread. Enriched data only appears on re-search — users see signal dots greyed out with "↻ Updating: age, categories" and have no way to get updated scores without manually repeating the search. ## Proposed Solution Add a Server-Sent Events (SSE) endpoint: `GET /api/search/updates?query=...` that streams trust score updates as background enrichment completes. ### Backend - `_trigger_scraper_enrichment` emits events via an asyncio `Queue` as each seller enriches - FastAPI SSE endpoint reads from queue, streams `data: {listing_id, trust_score}` JSON events ### Frontend - `useSearchStore` opens SSE connection after search completes - Merges incoming `trust_score` updates into `trustScores` map reactively - Signal dots on `ListingCard` update live without re-search - SSE connection closes when all pending signals are resolved or after 60s timeout ## Tier Gate Free tier — this is a UX improvement for the existing data pipeline, not a new paid feature. ## Alternatives Considered - **Polling** (`GET /api/search/status?query=...` every 5s): simpler but wastes requests - **WebSocket**: bidirectional not needed; SSE is simpler for server-push only ## Notes Currently mitigated by: signal dots on trust badge show which signals are pending, hover tooltip says "search again to update", score badge pulses while partial.