BrowserPool: pool slots unusable cross-thread — pre-warm is effectively a no-op #53

New issue

Closed

opened 2026-05-03 18:37:48 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-05-03 18:37:48 -07:00

Owner

Problem

BrowserPool warms up N Chromium browser slots in background threads at startup. But Playwright's sync API binds to the thread that created it — using a slot from a different thread (e.g. a FastAPI uvicorn thread-pool handler) raises:

cannot switch to a different thread (which happens to have exited)

The _fetch_with_slot() call always fails, _close_slot() terminates the browser, and _fetch_fresh() launches a completely new Xvfb+Chromium instance per request. The pre-warm overhead is wasted.

Root Cause

playwright.sync_api.sync_playwright() creates an internal asyncio event loop bound to the calling thread. The SyncBase._sync() wrapper detects if it's being called from a different thread and raises the error.

Options

Thread-local pool — store a Playwright instance per-thread using threading.local(). Each FastAPI thread creates its own browser on first use and reuses it for subsequent requests on the same thread. No cross-thread sharing.
Playwright async API — rewrite the pool to use playwright.async_api with asyncio. FastAPI supports async route handlers natively. The async API has no thread-binding constraint.
Dedicated browser threads — one thread per slot, with each thread owning its Playwright instance. Requests are dispatched to slot-owner threads via a queue.

Option 1 is the lowest-effort path and compatible with FastAPI's sync route handlers running in a thread pool.

Current Behavior

Pool warms up 2 slots (now that display range is :200+ — see related fix)
Every request triggers the thread-error path → _fetch_fresh() → full Xvfb+Chromium cold start (~10s per fetch)
Pool warm-up is wasted compute

Impact

Each Mercari/eBay scrape pays ~10s cold-start cost instead of reusing a warm context. With traffic this is fine for MVP but becomes a bottleneck under load.

Labels

enhancement, browser-pool, performance

## Problem `BrowserPool` warms up N Chromium browser slots in background threads at startup. But Playwright's sync API binds to the thread that created it — using a slot from a different thread (e.g. a FastAPI uvicorn thread-pool handler) raises: cannot switch to a different thread (which happens to have exited) The `_fetch_with_slot()` call always fails, `_close_slot()` terminates the browser, and `_fetch_fresh()` launches a completely new Xvfb+Chromium instance per request. The pre-warm overhead is wasted. ## Root Cause `playwright.sync_api.sync_playwright()` creates an internal asyncio event loop bound to the calling thread. The `SyncBase._sync()` wrapper detects if it's being called from a different thread and raises the error. ## Options 1. **Thread-local pool** — store a Playwright instance per-thread using `threading.local()`. Each FastAPI thread creates its own browser on first use and reuses it for subsequent requests on the same thread. No cross-thread sharing. 2. **Playwright async API** — rewrite the pool to use `playwright.async_api` with asyncio. FastAPI supports async route handlers natively. The async API has no thread-binding constraint. 3. **Dedicated browser threads** — one thread per slot, with each thread owning its Playwright instance. Requests are dispatched to slot-owner threads via a queue. Option 1 is the lowest-effort path and compatible with FastAPI's sync route handlers running in a thread pool. ## Current Behavior - Pool warms up 2 slots (now that display range is :200+ — see related fix) - Every request triggers the thread-error path → `_fetch_fresh()` → full Xvfb+Chromium cold start (~10s per fetch) - Pool warm-up is wasted compute ## Impact Each Mercari/eBay scrape pays ~10s cold-start cost instead of reusing a warm context. With traffic this is fine for MVP but becomes a bottleneck under load. ## Labels `enhancement`, `browser-pool`, `performance`