96 lines
4.3 KiB
Markdown
96 lines
4.3 KiB
Markdown
# Snipe — Developer Context
|
||
|
||
> eBay listing monitor with seller trust scoring and auction sniping.
|
||
|
||
## Stack
|
||
|
||
| Layer | Tech | Port |
|
||
|-------|------|------|
|
||
| Frontend | Vue 3 + Pinia + UnoCSS + Vite (nginx) | 8509 |
|
||
| API | FastAPI (uvicorn, `network_mode: host`) | 8510 |
|
||
| Scraper | Playwright + playwright-stealth + Xvfb | — |
|
||
| DB | SQLite (`data/snipe.db`) | — |
|
||
| Core | circuitforge-core (editable install) | — |
|
||
|
||
## CLI
|
||
|
||
```bash
|
||
./manage.sh start|stop|restart|status|logs|open|build|test
|
||
```
|
||
|
||
## Docker
|
||
|
||
```bash
|
||
docker compose up -d # start
|
||
docker compose build api web # rebuild after Python/Vue changes
|
||
docker compose logs -f api # tail API logs
|
||
```
|
||
|
||
`compose.override.yml` bind-mounts `./tests` and `./app` for hot reload.
|
||
|
||
nginx proxies `/api/` → `172.17.0.1:8510` (Docker bridge IP — api uses host networking).
|
||
|
||
## Critical Gotchas
|
||
|
||
**Kasada bot protection:** eBay blocks `requests`, `curl_cffi`, headless Playwright, and all `/usr/` and `/fdbk/` seller profile pages. Only headed Chromium via Xvfb passes. `/itm/` listing pages DO load and contain a BTF (below the fold) seller card with "Joined {Mon} {Year}" — use this for account age enrichment.
|
||
|
||
**Xvfb display counter:** Module-level `itertools.cycle(range(200, 300))` issues unique display numbers (`:200`–`:299`) per `_get()` call to prevent lock file collisions when multiple Playwright sessions run in parallel.
|
||
|
||
**HTML cache:** 5-minute in-memory cache keyed by full URL. Prevents duplicate 15s Playwright scrapes within one session. Cleared on restart.
|
||
|
||
**SQLite thread safety:** Each concurrent thread (search + comps run in parallel) must have its own `Store` instance — `sqlite3.connect()` is not thread-safe across threads. See `api/main.py`.
|
||
|
||
**nginx rebuild gotcha:** nginx config is baked into the image at build time. After editing `docker/web/nginx.conf`, always `docker compose build web`.
|
||
|
||
**Playwright imports are lazy:** `sync_playwright` and `Stealth` import inside `_get()` — not at module level — so the pure parsing functions (`scrape_listings`, `scrape_sellers`) can be imported on the host without Docker's browser stack installed.
|
||
|
||
## DB Migrations
|
||
|
||
Auto-applied by `Store.__init__()` via `circuitforge_core.db.run_migrations`.
|
||
Migration files: `app/db/migrations/001_init.sql`, `002_buying_format.sql`, `003_nullable_account_age.sql`
|
||
|
||
## Tests
|
||
|
||
```bash
|
||
# Host (no Docker needed — pure parsing tests)
|
||
conda run -n job-seeker python -m pytest tests/ -v --ignore=tests/test_integration.py
|
||
|
||
# In container
|
||
./manage.sh test
|
||
```
|
||
|
||
48 tests. Scraper tests run on host thanks to lazy Playwright imports.
|
||
|
||
## Trust Scoring Architecture
|
||
|
||
```
|
||
TrustScorer
|
||
├── MetadataScorer → 5 signals × 0–20 = 0–100 composite
|
||
│ account_age, feedback_count, feedback_ratio,
|
||
│ price_vs_market (vs sold comps), category_history
|
||
├── PhotoScorer → phash dedup (free); vision analysis (paid stub)
|
||
└── Aggregator → composite score, red flags, hard filters
|
||
```
|
||
|
||
**Red flag sentinel gotcha:** `signal_scores` uses `None` for missing data; `clean` dict substitutes `None → 0` for arithmetic. Always check `signal_scores.get("key")` (not `clean["key"]`) when gating hard-filter flags — otherwise absent data fires false positives.
|
||
|
||
## Key Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `api/main.py` | FastAPI endpoint — parallel search+comps, serialization |
|
||
| `app/platforms/ebay/scraper.py` | Playwright scraper, HTML cache, page parser |
|
||
| `app/trust/aggregator.py` | Composite score, red flags, hard filters |
|
||
| `app/trust/metadata.py` | 5 metadata signals |
|
||
| `app/db/store.py` | SQLite read/write (batch methods) |
|
||
| `web/src/views/SearchView.vue` | Filter sidebar + results layout |
|
||
| `web/src/stores/search.ts` | Pinia store — API calls, result state |
|
||
| `web/src/components/ListingCard.vue` | Listing card + auction dim style |
|
||
| `web/src/assets/theme.css` | Central theme (CSS custom properties) |
|
||
|
||
## Pending Work
|
||
|
||
- **Keyword filtering** — must-include / must-exclude; negatives forwarded to eBay `_nkw` and applied client-side
|
||
- **Seller enrichment** — background scrape of `/itm/` BTF seller card for account age (`account_age_days`)
|
||
- **Bulk report UI** — multi-select listings → eBay report deep-link + CF community blocklist
|
||
- **Snipe scheduling** — configurable bid-time offset, human approval gate
|