Circuit-Forge/snipe
3
0
Fork 0
Code Issues 13 Pull requests 1 Projects Releases 2 Packages Wiki Activity Actions 6
snipe/docs/superpowers/specs/2026-03-25-snipe-circuitforge-core-design.md

14 KiB
Raw Blame History

Snipe MVP + circuitforge-core Extraction — Design Spec

Date: 2026-03-25 Status: Approved Products: snipe (new), circuitforge-core (new), peregrine (updated)

1. Overview

This spec covers two parallel workstreams:

  1. circuitforge-core extraction — hoist the shared scaffold from Peregrine into a private, locally-installable Python package. Peregrine becomes the first downstream consumer. All future CF products depend on it.
  2. Snipe MVP — eBay listing monitor + seller trust scorer, built on top of circuitforge-core. Solves the immediate problem: filtering scam accounts when searching for used GPU listings on eBay.

Design principle: cry once. Pay the extraction cost now while there are only two products; every product after this benefits for free.

2. circuitforge-core

2.1 Repository

  • Repo: git.opensourcesolarpunk.com/Circuit-Forge/circuitforge-core (private)
  • Local path: /Library/Development/CircuitForge/circuitforge-core/
  • Install method: pip install -e ../circuitforge-core (editable local package; graduate to Forgejo Packages private PyPI at product #3)
  • License: BSL 1.1 for AI features, MIT for pipeline/utility layers

2.2 Package Structure

circuitforge-core/
  circuitforge_core/
    pipeline/       # SQLite staging DB, status machine, background task runner
    llm/            # LLM router: fallback chain, BYOK support, vision-aware routing
    vision/         # Vision model wrapper — moondream2 (local) + Claude vision (cloud) [NET-NEW]
    wizard/         # First-run onboarding framework, tier gating, crash recovery
    tiers/          # Tier system (Free/Paid/Premium/Ultra) + Heimdall license client
    db/             # SQLite base class, migration runner
    config/         # Settings loader, env validation, secrets management
  pyproject.toml
  README.md

2.3 Extraction from Peregrine

The following Peregrine modules are extracted (migrated from Peregrine, not net-new):

Peregrine source → Core module Notes
app/wizard/ circuitforge_core/wizard/
scripts/llm_router.py circuitforge_core/llm/router.py Path is scripts/, not app/
app/wizard/tiers.py circuitforge_core/tiers/
SQLite pipeline base circuitforge_core/pipeline/

circuitforge_core/vision/ is net-new — no vision module exists in Peregrine to extract. It is built fresh in core.

Peregrine dependency management: Peregrine uses requirements.txt, not pyproject.toml. The migration adds circuitforge-core to requirements.txt as a local path entry: -e ../circuitforge-core. Snipe is greenfield and uses pyproject.toml from the start. There is no requirement to migrate Peregrine to pyproject.toml as part of this work.

2.4 Docker Build Strategy

Docker build contexts cannot reference paths outside the context directory (COPY ../ is forbidden). Both Peregrine and Snipe resolve this by setting the compose build context to the parent directory:

# compose.yml (snipe or peregrine)
services:
  app:
    build:
      context: ..                          # /Library/Development/CircuitForge/
      dockerfile: snipe/Dockerfile

# snipe/Dockerfile
COPY circuitforge-core/ ./circuitforge-core/
RUN pip install -e ./circuitforge-core
COPY snipe/ ./snipe/
RUN pip install -e ./snipe

In development, compose.override.yml bind-mounts ../circuitforge-core so local edits to core are immediately live without rebuild.

3. Snipe MVP

3.1 Scope

In (v0.1 MVP):

  • eBay listing search (Browse API + Seller API)
  • Metadata trust scoring (free tier)
  • Perceptual hash duplicate photo detection within a search result set (free tier)
  • Faceted filter UI with dynamic, data-driven filter options and sliders
  • On-demand search only
  • SavedSearch DB schema scaffolded but monitoring not wired up

Out (future versions):

  • Background polling / saved search alerts (v0.2)
  • Photo analysis via vision model — real vs marketing shot, EM bag detection (v0.2, paid)
  • Serial number consistency check (v0.2, paid)
  • AI-generated image detection (v0.3, paid)
  • Reverse image search (v0.4, paid)
  • Additional platforms: HiBid, CT Bids, AuctionZip (v0.3+)
  • Bid scheduling / snipe execution (v0.4+)

3.2 Repository

  • Repo: git.opensourcesolarpunk.com/Circuit-Forge/snipe (public discovery layer)
  • Local path: /Library/Development/CircuitForge/snipe/
  • License: MIT (discovery/pipeline), BSL 1.1 (AI features)
  • Product code: CFG-SNPE
  • Port: 8506

3.3 Tech Stack

Follows Peregrine as the reference implementation:

  • UI: Streamlit (Python)
  • DB: SQLite via circuitforge_core.db
  • LLM/Vision: circuitforge_core.llm / circuitforge_core.vision
  • Tiers: circuitforge_core.tiers
  • Containerisation: Docker + compose.yml, managed via manage.sh
  • Python env: conda run -n job-seeker (shared CF env)

3.4 Application Structure

snipe/
  app/
    platforms/
      __init__.py         # PlatformAdapter abstract base class
      ebay/
        adapter.py        # eBay Browse API + Seller API client
        auth.py           # OAuth2 client credentials token manager
        normaliser.py     # Raw API response → Listing / Seller schema
    trust/
      __init__.py         # TrustScorer orchestrator
      metadata.py         # Account age, feedback, price vs market, category history
      photo.py            # Perceptual hash dedup (free); vision analysis (paid, v0.2+)
      aggregator.py       # Weighted composite score + red flag extraction
    ui/
      Search.py           # Main search + results page
      components/
        filters.py        # Dynamic faceted filter sidebar
        listing_row.py    # Listing card with trust badge + red flags + error state
    db/
      models.py           # Listing, Seller, Search, TrustScore, SavedSearch schemas
      migrations/
    wizard/               # First-run onboarding (thin wrapper on core wizard)
    snipe/                # Bid engine placeholder (v0.4)
  manage.sh
  compose.yml
  compose.override.yml
  Dockerfile
  pyproject.toml

3.5 eBay API Credentials

eBay Browse API and Seller API require OAuth 2.0 app-level tokens (client credentials flow — no user auth needed, but a registered eBay developer account and app credentials are required).

Token lifecycle:

  • App token fetched at startup and cached in memory with expiry
  • auth.py handles refresh automatically on expiry (tokens last 2 hours)
  • On token fetch failure: search fails with a user-visible error; no silent fallback

Credentials storage: .env file (gitignored), never hardcoded.

EBAY_CLIENT_ID=...
EBAY_CLIENT_SECRET=...
EBAY_ENV=production   # or sandbox

Rate limits: eBay Browse API — 5,000 calls/day (sandbox), higher on production. Completed sales comps results are cached in SQLite with a 6-hour TTL to avoid redundant calls and stay within limits. Cache miss triggers a fresh fetch; fetch failure degrades gracefully (price vs market signal skipped, score noted as partial).

API split: get_seller() uses the eBay Seller API (different endpoint, same app token). Rate limits are tracked separately. The PlatformAdapter interface does not expose this distinction; it is an internal concern of the eBay adapter.

3.6 Data Model

Listing

id, platform, platform_listing_id, title, price, currency,
condition, seller_id, url, photo_urls (JSON), listing_age_days,
fetched_at, trust_score_id

Seller

id, platform, platform_seller_id, username,
account_age_days, feedback_count, feedback_ratio,
category_history_json, fetched_at

TrustScore

id, listing_id, composite_score,
account_age_score, feedback_count_score, feedback_ratio_score,
price_vs_market_score, category_history_score,
photo_hash_duplicate (bool),
photo_analysis_json (paid, nullable),
red_flags_json, scored_at, score_is_partial (bool)

MarketComp (price comps cache)

id, platform, query_hash, median_price, sample_count, fetched_at, expires_at

SavedSearch (schema scaffolded in v0.1; monitoring not wired until v0.2)

id, name, query, platform, filters_json, created_at, last_run_at

PhotoHash (perceptual hash store for cross-search dedup, v0.2+)

id, listing_id, photo_url, phash, first_seen_at

3.7 Platform Adapter Interface

class PlatformAdapter:
    def search(self, query: str, filters: SearchFilters) -> list[Listing]: ...
    def get_seller(self, seller_id: str) -> Seller: ...
    def get_completed_sales(self, query: str) -> list[Listing]: ...

Adding HiBid or CT Bids later = new adapter, zero changes to trust scorer or UI.

3.8 Trust Scorer

Metadata Signals (Free)

Five signals, each scored 0–20, equal weight. Composite = sum (0–100).

Signal Source Red flag threshold Score 0 condition
Account age eBay Seller API < 30 days < 7 days (also hard-filter)
Feedback count eBay Seller API < 10 < 3
Feedback ratio eBay Seller API < 95% < 80% with count > 20
Price vs market Completed sales comps > 30% below median > 50% below median
Category history Seller past sales No prior electronics sales No prior sales at all

Hard filters (auto-hide regardless of composite score):

  • Account age < 7 days
  • Feedback ratio < 80% with feedback count > 20

Partial scores: If any signal's data source is unavailable (API failure, rate limit), that signal contributes 0 and score_is_partial = True is set on the TrustScore record. The UI surfaces a "⚠ Partial score" indicator on affected listings.

Photo Signals — Anti-Gotcha Layer

Signal Tier Version Method
Perceptual hash dedup within result set Free v0.1 MVP Compare phashes across all listings in the current search response; flag duplicates
Real photo vs marketing shot Paid / Local vision v0.2 Vision model classification
Open box + EM antistatic bag (proof of possession) Paid / Local vision v0.2 Vision model classification
Serial number consistency across photos Paid / Local vision v0.2 Vision model OCR + comparison
AI-generated image detection Paid v0.3 Classifier model
Reverse image search Paid v0.4 Google Lens / TinEye API

v0.1 dedup scope: Perceptual hash comparison is within the current search result set only (not across historical searches). Cross-session dedup uses the PhotoHash table and is a v0.2 feature. Photos are not downloaded to disk in v0.1 — hashes are computed from the image bytes in memory during the search request.

3.9 Tier Gating

Photo analysis features use LOCAL_VISION_UNLOCKABLE (analogous to BYOK_UNLOCKABLE in Peregrine's tiers.py) — they unlock for free-tier users who have a local vision model (moondream2) configured. This is distinct from BYOK (text LLM key), which does not unlock vision features.

Feature Free Paid Local vision unlock
Metadata trust scoring ✓ ✓ —
Perceptual hash dedup (within result set) ✓ ✓ —
Photo analysis (real/marketing/EM bag) — ✓ ✓
Serial number consistency — ✓ ✓
AI generation detection — ✓ —
Reverse image search — ✓ —
Saved searches + background monitoring — ✓ —

Locked features are shown (disabled) in the filter sidebar so free users see what's available. Clicking a locked filter shows a tier upgrade prompt.

3.10 UI — Results Page

Search bar: keywords, max price, condition selector, search button. Sort: trust score (default), price ↑/↓, listing age.

Filter sidebar — all options and counts generated dynamically from the result set. Options with 0 results are hidden (not greyed):

  • Trust score — range slider (min/max from results); colour-band summary (safe/review/skip + counts)
  • Price — min/max text inputs + market avg/median annotation
  • Seller account age — min slider
  • Feedback count — min slider
  • Positive feedback % — min slider
  • Condition — checkboxes (options from data: New, Open Box, Used, For Parts)
  • Photo signals — checkboxes: Real photo, EM bag visible, Open box, No AI-generated (locked, paid)
  • Hide if flagged — checkboxes: New account (<30d), Marketing photo, >30% below market, Duplicate photo
  • Shipping — Free shipping, Local pickup
  • Reset filters button

Listing row (happy path): thumbnail · title · seller summary (username, feedback count, ratio, tenure) · red flag badges · trust score badge (colour-coded: green 80+, amber 50–79, red <50) · score_is_partial indicator if applicable · price · "Open eBay ↗" link. Left border colour matches score band.

Listing row (error states):

  • Seller data unavailable: seller summary shows "Seller data unavailable" in muted text; affected signals show "–" and partial score indicator is set
  • Photo URL 404: thumbnail shows placeholder icon; hash dedup skipped for that photo
  • Trust scoring failed entirely: listing shown with score "?" badge in neutral grey; error logged; "Could not score this listing" tooltip

Hidden results: count shown at bottom ("N results hidden by filters · show anyway"). Clicking reveals them in-place at reduced opacity.

4. Build Order

  1. circuitforge-core — scaffold repo, extract wizard/llm/tiers/pipeline from Peregrine, build vision module net-new, update Peregrine requirements.txt
  2. Snipe scaffold — repo init, Dockerfile, compose.yml (parent context), manage.sh, DB migrations, wizard first-run, .env template
  3. eBay adapter — OAuth2 token manager, Browse API search, Seller API, completed sales comps with cache
  4. Metadata trust scorer — all five signals, aggregator, hard filters, partial score handling
  5. Perceptual hash dedup — in-memory within-result-set comparison
  6. Results UI — search page, listing rows (happy + error states), dynamic filter sidebar
  7. Tier gating — lock photo signals, LOCAL_VISION_UNLOCKABLE gate, upsell prompts in UI

5. Documentation Locations

  • Product spec: snipe/docs/superpowers/specs/2026-03-25-snipe-circuitforge-core-design.md (this file)
  • Internal copy: circuitforge-plans/snipe/2026-03-25-snipe-circuitforge-core-design.md
  • Roadmap: Circuit-Forge/roadmap issues #14 (snipe) and #21 (circuitforge-core)
  • Org-level context: /Library/Development/CircuitForge/CLAUDE.md