Async search endpoint: return job_id immediately, scrape in background #49

New issue

Closed

opened 2026-04-20 08:50:40 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-20 08:50:40 -07:00

Owner

Problem

Search blocks the HTTP connection for 10-120s while Chromium scrapes eBay. Under load this exhausts FastAPI workers and causes 120s nginx timeouts (seen in spike #5 as max=120s ceiling failures).

Proposed Fix

Mirror the Peregrine async task pattern:

POST /api/search — enqueues scrape job, returns {job_id, status: "queued"} immediately (HTTP 202)
Background worker scrapes eBay, writes results to DB
GET /api/search/task/{job_id} — poll for status/results

Frontend already handles async patterns (Peregrine cover letter polling) — adapt that component.

Expected Impact

Zero 120s timeout failures (request returns in <100ms)
Decouples user-perceived latency from scrape duration
Enables retries and progress indication in UI

Priority

Lower than browser pool + caching (those reduce actual latency); this improves resilience but does not fix root cause. Implement after #browser-pool lands.

## Problem Search blocks the HTTP connection for 10-120s while Chromium scrapes eBay. Under load this exhausts FastAPI workers and causes 120s nginx timeouts (seen in spike #5 as max=120s ceiling failures). ## Proposed Fix Mirror the Peregrine async task pattern: 1. `POST /api/search` — enqueues scrape job, returns `{job_id, status: "queued"}` immediately (HTTP 202) 2. Background worker scrapes eBay, writes results to DB 3. `GET /api/search/task/{job_id}` — poll for status/results Frontend already handles async patterns (Peregrine cover letter polling) — adapt that component. ## Expected Impact - Zero 120s timeout failures (request returns in <100ms) - Decouples user-perceived latency from scrape duration - Enables retries and progress indication in UI ## Priority Lower than browser pool + caching (those reduce actual latency); this improves resilience but does not fix root cause. Implement after #browser-pool lands.