feat: self-hosted installer script (install.sh) #25

New issue

Open

opened 2026-04-05 22:24:17 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-04-05 22:24:17 -07:00

Owner

Summary

Add install.sh to the repo root -- a single-command installer for self-hosted Snipe deployments.

What to build

Follow the CircuitForge installer pattern documented at:
circuitforge-plans/shared/2026-04-05-installer-pattern.md

The reference implementation is circuitforge-orch/install.sh (cf-orch#11).

Snipe-specific requirements

No topology selection -- Snipe is single-process, no coordinator/agent split
Browser/Chromium detection -- Snipe uses headed Chromium + Xvfb for Kasada bypass; installer should detect/warn if google-chrome or chromium-browser is absent
Dependencies: Python 3.11+, redis (for background job queue), optional: ollama (for LLM enrichment)
Conda env: detect cf, snipe, or circuitforge env; offer to create snipe env
Env file: ~/.config/circuitforge/snipe.env -- ports (Vue 8509, FastAPI 8510), DB path, optional Heimdall key
License key wiring: same pattern as cf-orch -- optional, CFG-SNPE-* format, populates HEIMDALL_URL
Docker Compose: generate from compose.yml with env_file; no GPU block needed (no cuda)
Systemd unit (optional): both snipe-api and snipe-web or a single snipe target

Key installer decisions (captured in pattern doc)

Approach B for config: generated templates use $SNIPE_* env var placeholders for paths, not hardcoded absolutes
bash install.sh should complete in < 2 min on a fresh Ubuntu box
All prompts must have defaults -- user should be able to press Enter through everything
Check bash -n install.sh passes before committing

Install troubleshooting findings (2026-04-06 -- Strahl test install)

During a real self-hosted install test on strahl (Debian 13, no API keys), several blockers were found and fixed:

Fixed in this session

1. compose.override.yml broke non-dev installs
The override had ../circuitforge-core:/app/circuitforge-core as a bind mount. On any machine without the sibling circuitforge-core checkout, Docker creates an empty directory that shadows the installed package, breaking all circuitforge_core imports. Removed the bind mount; the override now only contains live-reload source mounts for api/, app/, tests/.

2. compose.yml missing network_mode: host on api
nginx.conf (baked into the web image) proxies /api/ to 172.17.0.1:8510 (Docker bridge gateway). Without network_mode: host, the API only listens inside the container network, causing 502s on all /api/ requests. Added network_mode: host to the base compose.yml.

3. Migration 004 partial-failure loop
SQLites executescript() auto-commits each DDL statement individually. A prior crash mid-migration leaves columns in the DB without the migration being recorded as applied. On the next startup, the runner retries and hits duplicate column name errors. Fixed in cf-core migration runner: on duplicate column name error, strip the offending ADD COLUMN statement and retry until the migration completes, then mark it as applied.

4. manage.sh test command used wrong Python env
The test command prefixed conda run -n job-seeker, which does not exist inside Docker. Fixed to use python -m pytest directly.

Still open (filed separately)

snipe#23: publish circuitforge-core to private PyPI (simplifies install, removes sibling-dir requirement)
snipe#24: nginx config for no-Docker self-hosted installs

Validated on Strahl (free tier, no API keys)

Docker build: ~2 min from clean clone
Migrations: all 7 applied correctly on first run
Scraper search: 60 listings returned
Trust scoring: all 60 scored (partial=True, no enrichment data -- expected)
Market price comparison: working
Vision/LLM tasks: gracefully fail-fast when no LLM configured (not blocking)

cf-orch#11 (reference implementation)
snipe#24 (nginx/no-Docker self-hosted)
snipe#23 (Forgejo Packages PyPI)

## Summary Add `install.sh` to the repo root -- a single-command installer for self-hosted Snipe deployments. ## What to build Follow the **CircuitForge installer pattern** documented at: `circuitforge-plans/shared/2026-04-05-installer-pattern.md` The reference implementation is `circuitforge-orch/install.sh` (cf-orch#11). ## Snipe-specific requirements - **No topology selection** -- Snipe is single-process, no coordinator/agent split - **Browser/Chromium detection** -- Snipe uses headed Chromium + Xvfb for Kasada bypass; installer should detect/warn if google-chrome or chromium-browser is absent - **Dependencies**: Python 3.11+, redis (for background job queue), optional: ollama (for LLM enrichment) - **Conda env**: detect `cf`, `snipe`, or `circuitforge` env; offer to create `snipe` env - **Env file**: `~/.config/circuitforge/snipe.env` -- ports (Vue 8509, FastAPI 8510), DB path, optional Heimdall key - **License key wiring**: same pattern as cf-orch -- optional, CFG-SNPE-* format, populates HEIMDALL_URL - **Docker Compose**: generate from `compose.yml` with env_file; no GPU block needed (no cuda) - **Systemd unit** (optional): both snipe-api and snipe-web or a single snipe target ## Key installer decisions (captured in pattern doc) - Approach B for config: generated templates use $SNIPE_* env var placeholders for paths, not hardcoded absolutes - `bash install.sh` should complete in < 2 min on a fresh Ubuntu box - All prompts must have defaults -- user should be able to press Enter through everything - Check `bash -n install.sh` passes before committing ## Install troubleshooting findings (2026-04-06 -- Strahl test install) During a real self-hosted install test on strahl (Debian 13, no API keys), several blockers were found and fixed: ### Fixed in this session **1. compose.override.yml broke non-dev installs** The override had `../circuitforge-core:/app/circuitforge-core` as a bind mount. On any machine without the sibling circuitforge-core checkout, Docker creates an empty directory that shadows the installed package, breaking all circuitforge_core imports. Removed the bind mount; the override now only contains live-reload source mounts for api/, app/, tests/. **2. compose.yml missing network_mode: host on api** nginx.conf (baked into the web image) proxies /api/ to 172.17.0.1:8510 (Docker bridge gateway). Without network_mode: host, the API only listens inside the container network, causing 502s on all /api/ requests. Added network_mode: host to the base compose.yml. **3. Migration 004 partial-failure loop** SQLites executescript() auto-commits each DDL statement individually. A prior crash mid-migration leaves columns in the DB without the migration being recorded as applied. On the next startup, the runner retries and hits duplicate column name errors. Fixed in cf-core migration runner: on duplicate column name error, strip the offending ADD COLUMN statement and retry until the migration completes, then mark it as applied. **4. manage.sh test command used wrong Python env** The test command prefixed conda run -n job-seeker, which does not exist inside Docker. Fixed to use python -m pytest directly. ### Still open (filed separately) - snipe#23: publish circuitforge-core to private PyPI (simplifies install, removes sibling-dir requirement) - snipe#24: nginx config for no-Docker self-hosted installs ### Validated on Strahl (free tier, no API keys) - Docker build: ~2 min from clean clone - Migrations: all 7 applied correctly on first run - Scraper search: 60 listings returned - Trust scoring: all 60 scored (partial=True, no enrichment data -- expected) - Market price comparison: working - Vision/LLM tasks: gracefully fail-fast when no LLM configured (not blocking) ## Related - cf-orch#11 (reference implementation) - snipe#24 (nginx/no-Docker self-hosted) - snipe#23 (Forgejo Packages PyPI)