Cache search results to avoid redundant scrapes for repeated queries #48

New issue

Closed

opened 2026-04-20 08:50:40 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-20 08:50:40 -07:00

Owner

Problem

Under load (and in normal usage) the same or similar search queries fire multiple times within a short window. Each hits the Kasada scraper unnecessarily. Spike #5: 218 search requests, many likely duplicates.

Proposed Fix

Add a short-TTL cache keyed on (query, filters_hash). On hit, return cached results immediately without touching the scraper.

Backend: SQLite table search_cache(query_hash, results_json, cached_at) — reuse data/snipe.db
TTL: 5 minutes (eBay listings update slowly; configurable via SEARCH_CACHE_TTL_S env)
Cache invalidated on explicit refresh (user-triggered)
Cloud mode: per-user cache vs shared cache TBD (shared is simpler, listings are public)

Expected Impact

Repeat queries return in <100ms
Scraper load reduced proportionally to cache hit rate
Reduces Kasada risk (fewer automated requests = lower detection surface)

Notes

Do NOT cache trust scores — those are per-listing and user-specific
Log cache hit/miss ratio for observability

## Problem Under load (and in normal usage) the same or similar search queries fire multiple times within a short window. Each hits the Kasada scraper unnecessarily. Spike #5: 218 search requests, many likely duplicates. ## Proposed Fix Add a short-TTL cache keyed on `(query, filters_hash)`. On hit, return cached results immediately without touching the scraper. - Backend: SQLite table `search_cache(query_hash, results_json, cached_at)` — reuse `data/snipe.db` - TTL: 5 minutes (eBay listings update slowly; configurable via `SEARCH_CACHE_TTL_S` env) - Cache invalidated on explicit refresh (user-triggered) - Cloud mode: per-user cache vs shared cache TBD (shared is simpler, listings are public) ## Expected Impact - Repeat queries return in <100ms - Scraper load reduced proportionally to cache hit rate - Reduces Kasada risk (fewer automated requests = lower detection surface) ## Notes - Do NOT cache trust scores — those are per-listing and user-specific - Log cache hit/miss ratio for observability