45c758bb53
revert: remove ADD COLUMN IF NOT EXISTS (not a SQLite feature)
...
SQLite does not support ADD COLUMN IF NOT EXISTS regardless of version.
The idempotency fix lives in cf-core's migration runner instead.
2026-04-05 22:24:06 -07:00
59f728cba0
fix: make ALTER TABLE migrations idempotent with IF NOT EXISTS
...
SQLite's executescript() auto-commits each DDL statement, so a partial
migration failure leaves columns in the DB without marking the migration
done. On the next startup the runner retries and hits duplicate column errors.
Use ADD COLUMN IF NOT EXISTS (SQLite 3.35+, shipped in Python 3.11+)
so migrations 004 and 005 are safe to re-run in any partial state.
2026-04-05 22:18:25 -07:00
81e41e39ab
fix: remove duplicate first_seen_at ALTER TABLE in migration 004
...
001_init.sql already defines first_seen_at in the CREATE TABLE statement.
On fresh installs, migration 004 failed with 'duplicate column name: first_seen_at'.
Remove the redundant ALTER TABLE; last_seen_at/times_seen/price_at_first_seen
are still added by 004 as before.
2026-04-05 22:12:47 -07:00
e93e3de207
feat: scammer blocklist, search/listing UI overhaul, tier refactor
...
**Scammer blocklist**
- migration 006: scammer_blocklist table (platform + seller_id unique key,
source: manual|csv_import|community)
- ScammerEntry dataclass + Store.add/remove/list_blocklist methods
- blocklist.ts Pinia store — CRUD, export CSV, import CSV with validation
- BlocklistView.vue — list with search, export/import, bulk-remove; sellers
show on ListingCard with force-score-0 badge
- API: GET/POST/DELETE /api/blocklist + CSV export/import endpoints
- Router: /blocklist route added; AppNav link
**Migration renumber**
- 002_background_tasks.sql → 007_background_tasks.sql (correct sequence
after blocklist; idempotent CREATE IF NOT EXISTS safe for existing DBs)
**Search + listing UI overhaul**
- SearchView.vue: keyword expansion preview, filter chips for condition/
format/price, saved-search quick-run button, paginated results
- ListingCard.vue: trust tier badge, scammer flag overlay, photo count
chip, quick-block button, save-to-search action
- savedSearches store: optimistic update on run, last-run timestamp
**Tier refactor**
- tiers.py: full rewrite with docstring ladder, BYOK LOCAL_VISION_UNLOCKABLE
flag, intentionally-free list with rationale (scammer_db, saved_searches,
market_comps free to maximise adoption)
**Trust aggregator + scraper**
- aggregator.py: blocklist check short-circuits scoring to 0/BAD_ACTOR
- scraper.py: listing format detection, photo count, improved title parsing
**Theme**
- theme.css: trust tier color tokens, badge variants, blocklist badge
2026-04-03 19:08:54 -07:00
d9660093b1
fix(tasks): address code review — cloud DB path, migration number, connection handling, enqueue site
...
- Rename 002_background_tasks.sql → 007_background_tasks.sql to avoid
collision with existing 002_add_listing_format.sql migration
- Add CREATE UNIQUE INDEX on trust_scores(listing_id) in same migration
so save_trust_scores() can use ON CONFLICT upsert semantics
- Add Store.save_trust_scores() — upserts scores keyed by listing_id;
preserves photo_analysis_json so runner writes are never clobbered
- runner.py: replace raw sqlite3.connect() with get_connection() throughout
(timeout=30 + WAL mode); fix connection leak in insert_task via try/finally
- _run_trust_photo_analysis: read 'user_db' from params to write results to
the correct per-user DB in cloud mode (was silently writing to wrong DB)
- main.py lifespan: use _shared_db_path() in cloud mode so background_tasks
queue lives in shared DB, not _LOCAL_SNIPE_DB
- Add _enqueue_vision_tasks() and call it after score_batch() — this is the
missing enqueue call site; gated by features.photo_analysis (Paid tier)
- Test fixture: add missing 'stage' column to background_tasks schema
2026-03-31 17:00:01 -07:00
f7c5e8dc17
feat(tasks): add vision task scheduler for trust_photo_analysis
...
Wires circuitforge_core.tasks.scheduler into Snipe. Adds trust_photo_analysis
background task: downloads primary listing photo, calls LLMRouter with vision
capability, writes result to trust_scores.photo_analysis_json (Paid tier).
photo_analysis_json column already existed in 001_init.sql migration.
2026-03-31 09:27:55 -07:00
9e20759dbe
feat: wire cloud session, Heimdall licensing, and split-store DB isolation
...
- api/cloud_session.py: new module — JWT validation (Directus HS256),
Heimdall provision+tier-resolve, CloudUser+SessionFeatures dataclasses,
compute_features() tier→feature-flag mapping, require_tier() dependency
factory, get_session() FastAPI dependency (local-mode transparent passthrough)
- api/main.py: remove _DB_PATH singleton; all endpoints receive session via
Depends(get_session); shared_store (sellers/comps) and user_store (listings/
saved_searches) created per-request from session.shared_db / session.user_db;
pages capped to features.max_pages; saved_searches limit enforced for free tier;
/api/session endpoint exposes tier+features to frontend; _trigger_scraper_enrichment
receives shared_db Path (background thread creates its own Store)
- app/platforms/ebay/adapter.py, scraper.py: rename store→shared_store parameter
(adapters only touch sellers+comps, never listings — naming reflects this)
- app/trust/__init__.py: rename store→shared_store (TrustScorer reads
sellers+comps from shared DB; listing staging fields come from caller)
- app/db/store.py: refresh_seller_categories gains listing_store param for
split-DB mode (reads listings from user_store, writes categories to self)
- web/src/stores/session.ts: new Pinia store — bootstrap() fetches /api/session,
exposes tier+features reactively; falls back to full-access local defaults
- web/src/App.vue: call session.bootstrap() on mount
- web/src/views/SearchView.vue: import session store; pages buttons disabled+greyed
above features.max_pages with upgrade tooltip
- compose.cloud.yml: add CLOUD_MODE=true + CLOUD_DATA_ROOT env; fix volume mount
- docker/web/nginx.cloud.conf: forward X-CF-Session header from Caddy to API
- .env.example: document cloud env vars (CLOUD_MODE, DIRECTUS_JWT_SECRET, etc.)
2026-03-27 02:07:06 -07:00
98695b00f0
feat(snipe): eBay trust scoring MVP — search, filters, enrichment, comps
...
Core trust scoring:
- Five metadata signals (account age, feedback count/ratio, price vs market,
category history), composited 0–100
- CV-based price signal suppression for heterogeneous search results
(e.g. mixed laptop generations won't false-positive suspicious_price)
- Expanded scratch/dent title detection: evasive redirects, functional problem
phrases, DIY/repair indicators
- Hard filters: new_account, established_bad_actor
- Soft flags: low_feedback, suspicious_price, duplicate_photo, scratch_dent,
long_on_market, significant_price_drop
Search & filtering:
- Browse API adapter (up to 200 items/page) + Playwright scraper fallback
- OR-group query expansion for comprehensive variant coverage
- Must-include (AND/ANY/groups), must-exclude, category, price range filters
- Saved searches with full filter round-trip via URL params
Seller enrichment:
- Background BTF /itm/ scraping for account age (Kasada-safe headed Chromium)
- On-demand enrichment: POST /api/enrich + ListingCard ↻ button
- Category history derived from Browse API categories field (free, no extra calls)
- Shopping API GetUserProfile inline enrichment for API adapter
Market comps:
- eBay Marketplace Insights API with Browse API fallback (catches 403 + 404)
- Comps prioritised in ThreadPoolExecutor (submitted first)
Infrastructure:
- Staging DB fields: times_seen, first_seen_at, price_at_first_seen, category_name
- Migrations 004 (staging tracking) + 005 (listing category)
- eBay webhook handler stub
- Cloud compose stack (compose.cloud.yml)
- Vue frontend: search store, saved searches store, ListingCard, filter sidebar
Docs:
- README fully rewritten to reflect MVP status + full feature documentation
- Roadmap table linked to all 13 Forgejo issues
2026-03-26 23:37:09 -07:00
2ab41219f8
fix: account_age_days=None for scraper tier, stop false new_account flags
...
Scraper can't fetch seller profile age without following each listing's
seller link. Using 0 as sentinel caused every scraped seller to trigger
new_account and account_under_30_days red flags erroneously.
- Seller.account_age_days: int → Optional[int] (None = not yet fetched)
- Migration 003: recreate sellers table without NOT NULL constraint
- MetadataScorer: return None for unknown age → score_is_partial=True
- Aggregator: gate age flags on is not None
- Scraper: account_age_days=None instead of 0
2026-03-25 20:36:43 -07:00
58263d814a
feat(snipe): FastAPI layer, Playwright+Xvfb scraper, caching, tests
...
- FastAPI service (port 8510) wrapping scraper + trust scorer
- Playwright+Xvfb+stealth transport to bypass eBay Kasada bot protection
- li.s-card selector migration (eBay markup change from li.s-item)
- Three-layer caching: HTML (5min), phash (permanent), market comp (6h SQLite)
- Batch DB writes (executemany + single commit) — warm requests <1s
- Unique Xvfb display counter (:200–:299) prevents lock file collisions
- Vue 3 nginx web service (port 8509) proxying /api/ to FastAPI
- Auction card de-emphasis: opacity 0.72 for listings with >1h remaining
- 35 scraper unit tests updated for new li.s-card fixture markup
- tests/ volume-mounted in compose.override.yml for live test editing
2026-03-25 20:09:30 -07:00
6ec0f957b9
feat(snipe): auction support + easter eggs (Konami, The Steal, de-emphasis)
...
Auction metadata:
- Listing model gains buying_format + ends_at fields
- Migration 002 adds columns to existing databases
- scraper.py: parse s-item__time-left → absolute ends_at ISO timestamp
- normaliser.py: extract buyingOptions + itemEndDate from Browse API
- store.py: save/get updated for new fields
Easter eggs (app/ui/components/easter_eggs.py):
- Konami code detector (JS → URL param → Streamlit rerun)
- Web Audio API snipe call synthesis, gated behind sidebar checkbox
(disabled by default for safety/accessibility)
- "The Steal" gold shimmer: trust ≥ 90, price 15–30% below market,
no suspicious_price flag
- Auction de-emphasis: soft caption when > 1h remaining
UI updates:
- listing_row: steal banner + auction notice per row
- Search: inject CSS, check snipe mode, "Ending soon" sort option,
pass market_price from comp cache to row renderer
- app.py: Konami detector + audio enable/disable sidebar toggle
Tests: 22 new tests (72 total, all green)
2026-03-25 14:27:02 -07:00
675146ff1a
feat: add data models, migrations, and store
2026-03-25 13:08:55 -07:00
