Circuit-Forge/snipe

Fork 0

SSE/WebSocket live score push for background enrichment #1

New issue

Closed

opened 2026-03-26 15:05:38 -07:00 by pyr0ball · 2 comments

pyr0ball commented

2026-03-26 15:05:38 -07:00

Owner

Problem

Background seller enrichment (BTF /itm/ scrape + _ssn category scrape + Shopping API age lookup) runs after each search in a daemon thread. Enriched data only appears on re-search — users see signal dots greyed out with "↻ Updating: age, categories" and have no way to get updated scores without manually repeating the search.

Proposed Solution

Add a Server-Sent Events (SSE) endpoint: GET /api/search/updates?query=... that streams trust score updates as background enrichment completes.

Backend

_trigger_scraper_enrichment emits events via an asyncio Queue as each seller enriches
FastAPI SSE endpoint reads from queue, streams data: {listing_id, trust_score} JSON events

Frontend

useSearchStore opens SSE connection after search completes
Merges incoming trust_score updates into trustScores map reactively
Signal dots on ListingCard update live without re-search
SSE connection closes when all pending signals are resolved or after 60s timeout

Tier Gate

Free tier — this is a UX improvement for the existing data pipeline, not a new paid feature.

Alternatives Considered

Polling (GET /api/search/status?query=... every 5s): simpler but wastes requests
WebSocket: bidirectional not needed; SSE is simpler for server-push only

Notes

Currently mitigated by: signal dots on trust badge show which signals are pending, hover tooltip says "search again to update", score badge pulses while partial.

## Problem Background seller enrichment (BTF `/itm/` scrape + `_ssn` category scrape + Shopping API age lookup) runs after each search in a daemon thread. Enriched data only appears on re-search — users see signal dots greyed out with "↻ Updating: age, categories" and have no way to get updated scores without manually repeating the search. ## Proposed Solution Add a Server-Sent Events (SSE) endpoint: `GET /api/search/updates?query=...` that streams trust score updates as background enrichment completes. ### Backend - `_trigger_scraper_enrichment` emits events via an asyncio `Queue` as each seller enriches - FastAPI SSE endpoint reads from queue, streams `data: {listing_id, trust_score}` JSON events ### Frontend - `useSearchStore` opens SSE connection after search completes - Merges incoming `trust_score` updates into `trustScores` map reactively - Signal dots on `ListingCard` update live without re-search - SSE connection closes when all pending signals are resolved or after 60s timeout ## Tier Gate Free tier — this is a UX improvement for the existing data pipeline, not a new paid feature. ## Alternatives Considered - **Polling** (`GET /api/search/status?query=...` every 5s): simpler but wastes requests - **WebSocket**: bidirectional not needed; SSE is simpler for server-push only ## Notes Currently mitigated by: signal dots on trust badge show which signals are pending, hover tooltip says "search again to update", score badge pulses while partial.

pyr0ball referenced this issue

2026-03-26 23:32:18 -07:00

Background saved-search monitoring with alert notifications #12

pyr0ball added this to the Alpha — Pipeline milestone 2026-04-04 16:33:19 -07:00

pyr0ball added the

backlog

feature

labels 2026-04-05 21:54:31 -07:00

pyr0ball referenced this issue from a commit

2026-04-05 23:12:29 -07:00

feat: SSE live score push for background enrichment (#1)

pyr0ball closed this issue

2026-04-05 23:16:22 -07:00

pyr0ball referenced this issue from a commit

2026-04-14 06:19:42 -07:00

feat: infra/devops batch — CI/CD, installer, nginx docs, cf-orch agent (v0.3.0)

pyr0ball commented

2026-04-16 13:19:48 -07:00

Author

Owner

SSE live score push is fully implemented and working end-to-end:

/api/updates/{session_id} streams re-scored trust scores from the background enrichment thread via SimpleQueue → async generator
Frontend opens EventSource after search if any scores are partial and a session_id is present
Closes on event: done or after 90s timeout
Verified working in smoke test: 183 partial scores enriched, streamed correctly, done event received

Closing as shipped.

SSE live score push is fully implemented and working end-to-end: - `/api/updates/{session_id}` streams re-scored trust scores from the background enrichment thread via SimpleQueue → async generator - Frontend opens `EventSource` after search if any scores are partial and a session_id is present - Closes on `event: done` or after 90s timeout - Verified working in smoke test: 183 partial scores enriched, streamed correctly, done event received Closing as shipped.

pyr0ball commented

2026-04-20 10:57:11 -07:00

Author

Owner

Implemented in feature/near-term-ux: SSE enrichment streaming via _update_queues + GET /api/updates/{session_id}. Async search (#49) extends this to stream initial listings too.

Implemented in feature/near-term-ux: SSE enrichment streaming via `_update_queues` + `GET /api/updates/{session_id}`. Async search (#49) extends this to stream initial listings too.

pyr0ball referenced this issue from a commit

2026-04-20 11:16:18 -07:00

feat(search): async endpoint + SSE streaming for initial results