diff --git a/docs/.gitkeep b/docs/.gitkeep new file mode 100644 index 0000000..e69de29 diff --git a/docs/plans/2026-02-20-job-seeker-design.md b/docs/plans/2026-02-20-job-seeker-design.md deleted file mode 100644 index 942129e..0000000 --- a/docs/plans/2026-02-20-job-seeker-design.md +++ /dev/null @@ -1,201 +0,0 @@ -# Job Seeker Platform — Design Document -**Date:** 2026-02-20 -**Status:** Approved -**Candidate:** Alex Rivera - ---- - -## Overview - -A monorepo project at `/devl/job-seeker/` that integrates three FOSS tools into a -cohesive job search pipeline: automated discovery (JobSpy), resume-to-listing keyword -matching (Resume Matcher), and automated application submission (AIHawk). Job listings -and interactive documents are tracked in Notion; source documents live in -`/Library/Documents/JobSearch/`. - ---- - -## Project Structure - -``` -/devl/job-seeker/ -├── config/ -│ ├── search_profiles.yaml # JobSpy queries (titles, locations, boards) -│ ├── llm.yaml # LLM router: backends + fallback order -│ └── notion.yaml # Notion DB IDs and field mappings -├── aihawk/ # git clone — Auto_Jobs_Applier_AIHawk -├── resume_matcher/ # git clone — Resume-Matcher -├── scripts/ -│ ├── discover.py # JobSpy → deduplicate → push to Notion -│ ├── match.py # Notion job URL → Resume Matcher → write score back -│ └── llm_router.py # LLM abstraction layer with priority fallback chain -├── docs/plans/ # Design and implementation docs (no resume files) -├── environment.yml # conda env spec (env name: job-seeker) -└── .gitignore -``` - -**Document storage rule:** Resumes, cover letters, and any interactable documents live -in `/Library/Documents/JobSearch/` or Notion — never committed to this repo. - ---- - -## Architecture - -### Data Flow - -``` -JobSpy (LinkedIn / Indeed / Glassdoor / ZipRecruiter) - └─▶ discover.py - ├─ deduplicate by URL against existing Notion records - └─▶ Notion DB (Status: "New") - -Notion DB (daily review — decide what to pursue) - └─▶ match.py - ├─ fetch job description from listing URL - ├─ run Resume Matcher vs. /Library/Documents/JobSearch/Alex_Rivera_Resume_02-19-2025.pdf - └─▶ write Match Score + Keyword Gaps back to Notion page - -AIHawk (when ready to apply) - ├─ reads config pointing to same resume + personal_info.yaml - ├─ llm_router.py → best available LLM backend - ├─ submits LinkedIn Easy Apply - └─▶ Notion status → "Applied" -``` - ---- - -## Notion Database Schema - -| Field | Type | Notes | -|---------------|----------|------------------------------------------------------------| -| Job Title | Title | Primary identifier | -| Company | Text | | -| Location | Text | | -| Remote | Checkbox | | -| URL | URL | Deduplication key | -| Source | Select | LinkedIn / Indeed / Glassdoor / ZipRecruiter | -| Status | Select | New → Reviewing → Applied → Interview → Offer → Rejected | -| Match Score | Number | 0–100, written by match.py | -| Keyword Gaps | Text | Comma-separated missing keywords from Resume Matcher | -| Salary | Text | If listed | -| Date Found | Date | Set at discovery time | -| Notes | Text | Manual field | - ---- - -## LLM Router (`scripts/llm_router.py`) - -Single `complete(prompt, system=None)` interface. On each call: health-check each -backend in configured order, use the first that responds. Falls back silently on -connection error, timeout, or 5xx. Logs which backend was used. - -All backends except Anthropic use the `openai` Python package (OpenAI-compatible -endpoints). Anthropic uses the `anthropic` package. - -### `config/llm.yaml` - -```yaml -fallback_order: - - claude_code # port 3009 — Claude via local pipeline (highest quality) - - ollama # port 11434 — local, always-on - - vllm # port 8000 — start when needed - - github_copilot # port 3010 — Copilot via gh token - - anthropic # cloud fallback, burns API credits - -backends: - claude_code: - type: openai_compat - base_url: http://localhost:3009/v1 - model: claude-code-terminal - api_key: "any" - - ollama: - type: openai_compat - base_url: http://localhost:11434/v1 - model: llama3.2 - api_key: "ollama" - - vllm: - type: openai_compat - base_url: http://localhost:8000/v1 - model: __auto__ - api_key: "" - - github_copilot: - type: openai_compat - base_url: http://localhost:3010/v1 - model: gpt-4o - api_key: "any" - - anthropic: - type: anthropic - model: claude-sonnet-4-6 - api_key_env: ANTHROPIC_API_KEY -``` - ---- - -## Job Search Profile - -### `config/search_profiles.yaml` (initial) - -```yaml -profiles: - - name: cs_leadership - titles: - - "Customer Success Manager" - - "Director of Customer Success" - - "VP Customer Success" - - "Head of Customer Success" - - "Technical Account Manager" - - "Revenue Operations Manager" - - "Customer Experience Lead" - locations: - - "Remote" - - "San Francisco Bay Area, CA" - boards: - - linkedin - - indeed - - glassdoor - - zip_recruiter - results_per_board: 25 - remote_only: false # remote preferred but Bay Area in-person ok - hours_old: 72 # listings posted in last 3 days -``` - ---- - -## Conda Environment - -New dedicated env `job-seeker` (not base). Core packages: - -- `python-jobspy` — job scraping -- `notion-client` — Notion API -- `openai` — OpenAI-compatible calls (Ollama, vLLM, Copilot, Claude pipeline) -- `anthropic` — Anthropic API fallback -- `pyyaml` — config parsing -- `pandas` — CSV handling and dedup -- Resume Matcher dependencies (sentence-transformers, streamlit — installed from clone) - -Resume Matcher Streamlit UI runs on port **8501** (confirmed clear). - ---- - -## Port Map - -| Port | Service | Status | -|-------|--------------------------------|----------------| -| 3009 | Claude Code OpenAI wrapper | Start via manage.sh in Post Fight Processing | -| 3010 | GitHub Copilot wrapper | Start via manage-copilot.sh | -| 11434 | Ollama | Running | -| 8000 | vLLM | Start when needed | -| 8501 | Resume Matcher (Streamlit) | Start when needed | - ---- - -## Out of Scope (this phase) - -- Scheduled/cron automation (run discover.py manually for now) -- Email/SMS alerts for new listings -- ATS resume rebuild (separate task) -- Applications to non-LinkedIn platforms via AIHawk diff --git a/docs/plans/2026-02-20-job-seeker-implementation.md b/docs/plans/2026-02-20-job-seeker-implementation.md deleted file mode 100644 index 3ee364b..0000000 --- a/docs/plans/2026-02-20-job-seeker-implementation.md +++ /dev/null @@ -1,1090 +0,0 @@ -# Job Seeker Platform — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Stand up a job discovery pipeline (JobSpy → Notion) with LLM routing, resume matching, and automated LinkedIn application support for Alex Rivera. - -**Architecture:** JobSpy scrapes listings from multiple boards and pushes deduplicated results into a Notion database. A local LLM router with 5-backend fallback chain powers AIHawk's application answer generation. Resume Matcher scores each listing against Alex's resume and writes keyword gaps back to Notion. - -**Tech Stack:** Python 3.12, conda env `job-seeker`, `python-jobspy`, `notion-client`, `openai` SDK, `anthropic` SDK, `pyyaml`, `pandas`, Resume-Matcher (cloned), Auto_Jobs_Applier_AIHawk (cloned), pytest, pytest-mock - -**Priority order:** Discovery (Tasks 1–5) must be running before Match or AIHawk setup. - -**Document storage rule:** Resumes and cover letters live in `/Library/Documents/JobSearch/` — never committed to this repo. - ---- - -## Task 1: Conda Environment + Project Scaffold - -**Files:** -- Create: `/devl/job-seeker/environment.yml` -- Create: `/devl/job-seeker/.gitignore` -- Create: `/devl/job-seeker/tests/__init__.py` - -**Step 1: Write environment.yml** - -```yaml -# /devl/job-seeker/environment.yml -name: job-seeker -channels: - - conda-forge - - defaults -dependencies: - - python=3.12 - - pip - - pip: - - python-jobspy - - notion-client - - openai - - anthropic - - pyyaml - - pandas - - requests - - pytest - - pytest-mock -``` - -**Step 2: Create the conda env** - -```bash -conda env create -f /devl/job-seeker/environment.yml -``` - -Expected: env `job-seeker` created with no errors. - -**Step 3: Verify the env** - -```bash -conda run -n job-seeker python -c "import jobspy, notion_client, openai, anthropic; print('all good')" -``` - -Expected: `all good` - -**Step 4: Write .gitignore** - -```gitignore -# /devl/job-seeker/.gitignore -.env -config/notion.yaml # contains Notion token -__pycache__/ -*.pyc -.pytest_cache/ -output/ -aihawk/ -resume_matcher/ -``` - -Note: `aihawk/` and `resume_matcher/` are cloned externally — don't commit them. - -**Step 5: Create tests directory** - -```bash -mkdir -p /devl/job-seeker/tests -touch /devl/job-seeker/tests/__init__.py -``` - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add environment.yml .gitignore tests/__init__.py -git commit -m "feat: add conda env spec and project scaffold" -``` - ---- - -## Task 2: Config Files - -**Files:** -- Create: `config/search_profiles.yaml` -- Create: `config/llm.yaml` -- Create: `config/notion.yaml.example` (the real `notion.yaml` is gitignored) - -**Step 1: Write search_profiles.yaml** - -```yaml -# config/search_profiles.yaml -profiles: - - name: cs_leadership - titles: - - "Customer Success Manager" - - "Director of Customer Success" - - "VP Customer Success" - - "Head of Customer Success" - - "Technical Account Manager" - - "Revenue Operations Manager" - - "Customer Experience Lead" - locations: - - "Remote" - - "San Francisco Bay Area, CA" - boards: - - linkedin - - indeed - - glassdoor - - zip_recruiter - results_per_board: 25 - hours_old: 72 -``` - -**Step 2: Write llm.yaml** - -```yaml -# config/llm.yaml -fallback_order: - - claude_code - - ollama - - vllm - - github_copilot - - anthropic - -backends: - claude_code: - type: openai_compat - base_url: http://localhost:3009/v1 - model: claude-code-terminal - api_key: "any" - - ollama: - type: openai_compat - base_url: http://localhost:11434/v1 - model: llama3.2 - api_key: "ollama" - - vllm: - type: openai_compat - base_url: http://localhost:8000/v1 - model: __auto__ - api_key: "" - - github_copilot: - type: openai_compat - base_url: http://localhost:3010/v1 - model: gpt-4o - api_key: "any" - - anthropic: - type: anthropic - model: claude-sonnet-4-6 - api_key_env: ANTHROPIC_API_KEY -``` - -**Step 3: Write notion.yaml.example** - -```yaml -# config/notion.yaml.example -# Copy to config/notion.yaml and fill in your values. -# notion.yaml is gitignored — never commit it. -token: "secret_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" -database_id: "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" -``` - -**Step 4: Commit** - -```bash -cd /devl/job-seeker -git add config/search_profiles.yaml config/llm.yaml config/notion.yaml.example -git commit -m "feat: add search profiles, LLM config, and Notion config template" -``` - ---- - -## Task 3: Create Notion Database - -This task creates the Notion DB that all scripts write to. Do it once manually. - -**Step 1: Open Notion and create a new database** - -Create a full-page database called **"Alex's Job Search"** in whatever Notion workspace you use for tracking. - -**Step 2: Add the required properties** - -Delete the default properties and create exactly these (type matters): - -| Property Name | Type | -|----------------|----------| -| Job Title | Title | -| Company | Text | -| Location | Text | -| Remote | Checkbox | -| URL | URL | -| Source | Select | -| Status | Select | -| Match Score | Number | -| Keyword Gaps | Text | -| Salary | Text | -| Date Found | Date | -| Notes | Text | - -For the **Status** select, add these options in order: -`New`, `Reviewing`, `Applied`, `Interview`, `Offer`, `Rejected` - -For the **Source** select, add: -`Linkedin`, `Indeed`, `Glassdoor`, `Zip_Recruiter` - -**Step 3: Get the database ID** - -Open the database as a full page. The URL will look like: -`https://www.notion.so/YourWorkspace/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX?v=...` - -The 32-character hex string before the `?` is the database ID. - -**Step 4: Get your Notion integration token** - -Go to https://www.notion.so/my-integrations → create integration (or use existing) → -copy the "Internal Integration Token" (starts with `secret_`). - -Connect the integration to your database: open the database → `...` menu → -Add connections → select your integration. - -**Step 5: Write config/notion.yaml** - -```bash -cp /devl/job-seeker/config/notion.yaml.example /devl/job-seeker/config/notion.yaml -# Edit notion.yaml and fill in your token and database_id -``` - -**Step 6: Verify connection** - -```bash -conda run -n job-seeker python3 -c " -from notion_client import Client -import yaml -cfg = yaml.safe_load(open('/devl/job-seeker/config/notion.yaml')) -n = Client(auth=cfg['token']) -db = n.databases.retrieve(cfg['database_id']) -print('Connected to:', db['title'][0]['plain_text']) -" -``` - -Expected: `Connected to: Alex's Job Search` - ---- - -## Task 4: LLM Router - -**Files:** -- Create: `scripts/llm_router.py` -- Create: `tests/test_llm_router.py` - -**Step 1: Write the failing tests** - -```python -# tests/test_llm_router.py -import pytest -from unittest.mock import patch, MagicMock -from pathlib import Path -import yaml - -# Point tests at the real config -CONFIG_PATH = Path(__file__).parent.parent / "config" / "llm.yaml" - - -def test_config_loads(): - """Config file is valid YAML with required keys.""" - cfg = yaml.safe_load(CONFIG_PATH.read_text()) - assert "fallback_order" in cfg - assert "backends" in cfg - assert len(cfg["fallback_order"]) >= 1 - - -def test_router_uses_first_reachable_backend(tmp_path): - """Router skips unreachable backends and uses the first that responds.""" - from scripts.llm_router import LLMRouter - - router = LLMRouter(CONFIG_PATH) - - mock_response = MagicMock() - mock_response.choices[0].message.content = "hello" - - with patch.object(router, "_is_reachable", side_effect=[False, True, True, True, True]), \ - patch("scripts.llm_router.OpenAI") as MockOpenAI: - instance = MockOpenAI.return_value - instance.chat.completions.create.return_value = mock_response - # Also mock models.list for __auto__ case - mock_model = MagicMock() - mock_model.id = "test-model" - instance.models.list.return_value.data = [mock_model] - - result = router.complete("say hello") - - assert result == "hello" - - -def test_router_raises_when_all_backends_fail(): - """Router raises RuntimeError when every backend is unreachable or errors.""" - from scripts.llm_router import LLMRouter - - router = LLMRouter(CONFIG_PATH) - - with patch.object(router, "_is_reachable", return_value=False): - with pytest.raises(RuntimeError, match="All LLM backends exhausted"): - router.complete("say hello") - - -def test_is_reachable_returns_false_on_connection_error(): - """_is_reachable returns False when the health endpoint is unreachable.""" - from scripts.llm_router import LLMRouter - import requests - - router = LLMRouter(CONFIG_PATH) - - with patch("scripts.llm_router.requests.get", side_effect=requests.ConnectionError): - result = router._is_reachable("http://localhost:9999/v1") - - assert result is False -``` - -**Step 2: Run tests to verify they fail** - -```bash -cd /devl/job-seeker -conda run -n job-seeker pytest tests/test_llm_router.py -v -``` - -Expected: `ImportError` — `scripts.llm_router` doesn't exist yet. - -**Step 3: Create scripts/__init__.py** - -```bash -touch /devl/job-seeker/scripts/__init__.py -``` - -**Step 4: Write scripts/llm_router.py** - -```python -# scripts/llm_router.py -""" -LLM abstraction layer with priority fallback chain. -Reads config/llm.yaml. Tries backends in order; falls back on any error. -""" -import os -import yaml -import requests -from pathlib import Path -from openai import OpenAI - -CONFIG_PATH = Path(__file__).parent.parent / "config" / "llm.yaml" - - -class LLMRouter: - def __init__(self, config_path: Path = CONFIG_PATH): - with open(config_path) as f: - self.config = yaml.safe_load(f) - - def _is_reachable(self, base_url: str) -> bool: - """Quick health-check ping. Returns True if backend is up.""" - health_url = base_url.rstrip("/").removesuffix("/v1") + "/health" - try: - resp = requests.get(health_url, timeout=2) - return resp.status_code < 500 - except Exception: - return False - - def _resolve_model(self, client: OpenAI, model: str) -> str: - """Resolve __auto__ to the first model served by vLLM.""" - if model != "__auto__": - return model - models = client.models.list() - return models.data[0].id - - def complete(self, prompt: str, system: str | None = None) -> str: - """ - Generate a completion. Tries each backend in fallback_order. - Raises RuntimeError if all backends are exhausted. - """ - for name in self.config["fallback_order"]: - backend = self.config["backends"][name] - - if backend["type"] == "openai_compat": - if not self._is_reachable(backend["base_url"]): - print(f"[LLMRouter] {name}: unreachable, skipping") - continue - try: - client = OpenAI( - base_url=backend["base_url"], - api_key=backend.get("api_key", "any"), - ) - model = self._resolve_model(client, backend["model"]) - messages = [] - if system: - messages.append({"role": "system", "content": system}) - messages.append({"role": "user", "content": prompt}) - - resp = client.chat.completions.create( - model=model, messages=messages - ) - print(f"[LLMRouter] Used backend: {name} ({model})") - return resp.choices[0].message.content - - except Exception as e: - print(f"[LLMRouter] {name}: error — {e}, trying next") - continue - - elif backend["type"] == "anthropic": - api_key = os.environ.get(backend["api_key_env"], "") - if not api_key: - print(f"[LLMRouter] {name}: {backend['api_key_env']} not set, skipping") - continue - try: - import anthropic as _anthropic - client = _anthropic.Anthropic(api_key=api_key) - kwargs: dict = { - "model": backend["model"], - "max_tokens": 4096, - "messages": [{"role": "user", "content": prompt}], - } - if system: - kwargs["system"] = system - msg = client.messages.create(**kwargs) - print(f"[LLMRouter] Used backend: {name}") - return msg.content[0].text - except Exception as e: - print(f"[LLMRouter] {name}: error — {e}, trying next") - continue - - raise RuntimeError("All LLM backends exhausted") - - -# Module-level singleton for convenience -_router: LLMRouter | None = None - - -def complete(prompt: str, system: str | None = None) -> str: - global _router - if _router is None: - _router = LLMRouter() - return _router.complete(prompt, system) -``` - -**Step 5: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_llm_router.py -v -``` - -Expected: 4 tests PASS. - -**Step 6: Smoke-test against live Ollama** - -```bash -conda run -n job-seeker python3 -c " -from scripts.llm_router import complete -print(complete('Say: job-seeker LLM router is working')) -" -``` - -Expected: A short response from Ollama (or next reachable backend). - -**Step 7: Commit** - -```bash -cd /devl/job-seeker -git add scripts/__init__.py scripts/llm_router.py tests/test_llm_router.py -git commit -m "feat: add LLM router with 5-backend fallback chain" -``` - ---- - -## Task 5: Job Discovery (discover.py) — PRIORITY - -**Files:** -- Create: `scripts/discover.py` -- Create: `tests/test_discover.py` - -**Step 1: Write the failing tests** - -```python -# tests/test_discover.py -import pytest -from unittest.mock import patch, MagicMock, call -import pandas as pd -from pathlib import Path - - -SAMPLE_JOB = { - "title": "Customer Success Manager", - "company": "Acme Corp", - "location": "Remote", - "is_remote": True, - "job_url": "https://linkedin.com/jobs/view/123456", - "site": "linkedin", - "salary_source": "$90,000 - $120,000", -} - - -def make_jobs_df(jobs=None): - return pd.DataFrame(jobs or [SAMPLE_JOB]) - - -def test_get_existing_urls_returns_set(): - """get_existing_urls returns a set of URL strings from Notion pages.""" - from scripts.discover import get_existing_urls - - mock_notion = MagicMock() - mock_notion.databases.query.return_value = { - "results": [ - {"properties": {"URL": {"url": "https://example.com/job/1"}}}, - {"properties": {"URL": {"url": "https://example.com/job/2"}}}, - ], - "has_more": False, - "next_cursor": None, - } - - urls = get_existing_urls(mock_notion, "fake-db-id") - assert urls == {"https://example.com/job/1", "https://example.com/job/2"} - - -def test_discover_skips_duplicate_urls(): - """discover does not push a job whose URL is already in Notion.""" - from scripts.discover import run_discovery - - existing = {"https://linkedin.com/jobs/view/123456"} - - with patch("scripts.discover.scrape_jobs", return_value=make_jobs_df()), \ - patch("scripts.discover.get_existing_urls", return_value=existing), \ - patch("scripts.discover.push_to_notion") as mock_push, \ - patch("scripts.discover.Client"): - run_discovery() - - mock_push.assert_not_called() - - -def test_discover_pushes_new_jobs(): - """discover pushes jobs whose URLs are not already in Notion.""" - from scripts.discover import run_discovery - - with patch("scripts.discover.scrape_jobs", return_value=make_jobs_df()), \ - patch("scripts.discover.get_existing_urls", return_value=set()), \ - patch("scripts.discover.push_to_notion") as mock_push, \ - patch("scripts.discover.Client"): - run_discovery() - - assert mock_push.call_count == 1 - - -def test_push_to_notion_sets_status_new(): - """push_to_notion always sets Status to 'New'.""" - from scripts.discover import push_to_notion - - mock_notion = MagicMock() - push_to_notion(mock_notion, "fake-db-id", SAMPLE_JOB) - - call_kwargs = mock_notion.pages.create.call_args[1] - status = call_kwargs["properties"]["Status"]["select"]["name"] - assert status == "New" -``` - -**Step 2: Run tests to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_discover.py -v -``` - -Expected: `ImportError` — `scripts.discover` doesn't exist yet. - -**Step 3: Write scripts/discover.py** - -```python -# scripts/discover.py -""" -JobSpy → Notion discovery pipeline. -Scrapes job boards, deduplicates against existing Notion records, -and pushes new listings with Status=New. - -Usage: - conda run -n job-seeker python scripts/discover.py -""" -import yaml -from datetime import datetime -from pathlib import Path - -import pandas as pd -from jobspy import scrape_jobs -from notion_client import Client - -CONFIG_DIR = Path(__file__).parent.parent / "config" -NOTION_CFG = CONFIG_DIR / "notion.yaml" -PROFILES_CFG = CONFIG_DIR / "search_profiles.yaml" - - -def load_config() -> tuple[dict, dict]: - profiles = yaml.safe_load(PROFILES_CFG.read_text()) - notion_cfg = yaml.safe_load(NOTION_CFG.read_text()) - return profiles, notion_cfg - - -def get_existing_urls(notion: Client, db_id: str) -> set[str]: - """Return the set of all job URLs already tracked in Notion.""" - existing: set[str] = set() - has_more = True - start_cursor = None - - while has_more: - kwargs: dict = {"database_id": db_id, "page_size": 100} - if start_cursor: - kwargs["start_cursor"] = start_cursor - resp = notion.databases.query(**kwargs) - - for page in resp["results"]: - url = page["properties"].get("URL", {}).get("url") - if url: - existing.add(url) - - has_more = resp.get("has_more", False) - start_cursor = resp.get("next_cursor") - - return existing - - -def push_to_notion(notion: Client, db_id: str, job: dict) -> None: - """Create a new page in the Notion jobs database for a single listing.""" - notion.pages.create( - parent={"database_id": db_id}, - properties={ - "Job Title": {"title": [{"text": {"content": str(job.get("title", "Unknown"))}}]}, - "Company": {"rich_text": [{"text": {"content": str(job.get("company", ""))}}]}, - "Location": {"rich_text": [{"text": {"content": str(job.get("location", ""))}}]}, - "Remote": {"checkbox": bool(job.get("is_remote", False))}, - "URL": {"url": str(job.get("job_url", ""))}, - "Source": {"select": {"name": str(job.get("site", "unknown")).title()}}, - "Status": {"select": {"name": "New"}}, - "Salary": {"rich_text": [{"text": {"content": str(job.get("salary_source") or "")}}]}, - "Date Found": {"date": {"start": datetime.now().isoformat()[:10]}}, - }, - ) - - -def run_discovery() -> None: - profiles_cfg, notion_cfg = load_config() - notion = Client(auth=notion_cfg["token"]) - db_id = notion_cfg["database_id"] - - existing_urls = get_existing_urls(notion, db_id) - print(f"[discover] {len(existing_urls)} existing listings in Notion") - - new_count = 0 - - for profile in profiles_cfg["profiles"]: - print(f"\n[discover] Profile: {profile['name']}") - for location in profile["locations"]: - print(f" Scraping: {location}") - jobs: pd.DataFrame = scrape_jobs( - site_name=profile["boards"], - search_term=" OR ".join(f'"{t}"' for t in profile["titles"]), - location=location, - results_wanted=profile.get("results_per_board", 25), - hours_old=profile.get("hours_old", 72), - linkedin_fetch_description=True, - ) - - for _, job in jobs.iterrows(): - url = str(job.get("job_url", "")) - if not url or url in existing_urls: - continue - push_to_notion(notion, db_id, job.to_dict()) - existing_urls.add(url) - new_count += 1 - print(f" + {job.get('title')} @ {job.get('company')}") - - print(f"\n[discover] Done — {new_count} new listings pushed to Notion.") - - -if __name__ == "__main__": - run_discovery() -``` - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_discover.py -v -``` - -Expected: 4 tests PASS. - -**Step 5: Run a live discovery (requires notion.yaml to be set up from Task 3)** - -```bash -conda run -n job-seeker python scripts/discover.py -``` - -Expected: listings printed and pushed to Notion. Check the Notion DB to confirm rows appear with Status=New. - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add scripts/discover.py tests/test_discover.py -git commit -m "feat: add JobSpy discovery pipeline with Notion deduplication" -``` - ---- - -## Task 6: Clone and Configure Resume Matcher - -**Step 1: Clone Resume Matcher** - -```bash -cd /devl/job-seeker -git clone https://github.com/srbhr/Resume-Matcher.git resume_matcher -``` - -**Step 2: Install Resume Matcher dependencies into the job-seeker env** - -```bash -conda run -n job-seeker pip install -r /devl/job-seeker/resume_matcher/requirements.txt -``` - -If there are conflicts, install only the core matching library: -```bash -conda run -n job-seeker pip install sentence-transformers streamlit qdrant-client pypdf2 -``` - -**Step 3: Verify it launches** - -```bash -conda run -n job-seeker streamlit run /devl/job-seeker/resume_matcher/streamlit_app.py --server.port 8501 -``` - -Expected: Streamlit opens on http://localhost:8501 (port confirmed clear). -Stop it with Ctrl+C — we'll run it on-demand. - -**Step 4: Note the resume path to use** - -The ATS-clean resume to use with Resume Matcher: -``` -/Library/Documents/JobSearch/Alex_Rivera_Resume_02-19-2025.pdf -``` - ---- - -## Task 7: Resume Match Script (match.py) - -**Files:** -- Create: `scripts/match.py` -- Create: `tests/test_match.py` - -**Step 1: Write the failing tests** - -```python -# tests/test_match.py -import pytest -from unittest.mock import patch, MagicMock - - -def test_extract_job_description_from_url(): - """extract_job_description fetches and returns text from a URL.""" - from scripts.match import extract_job_description - - with patch("scripts.match.requests.get") as mock_get: - mock_get.return_value.text = "

We need a CSM with Salesforce.

" - mock_get.return_value.raise_for_status = MagicMock() - result = extract_job_description("https://example.com/job/123") - - assert "CSM" in result - assert "Salesforce" in result - - -def test_score_is_between_0_and_100(): - """match_score returns a float in [0, 100].""" - from scripts.match import match_score - - # Provide minimal inputs that the scorer can handle - score, gaps = match_score( - resume_text="Customer Success Manager with Salesforce experience", - job_text="Looking for a Customer Success Manager who knows Salesforce and Gainsight", - ) - assert 0 <= score <= 100 - assert isinstance(gaps, list) - - -def test_write_score_to_notion(): - """write_match_to_notion updates the Notion page with score and gaps.""" - from scripts.match import write_match_to_notion - - mock_notion = MagicMock() - write_match_to_notion(mock_notion, "page-id-abc", 85.5, ["Gainsight", "Churnzero"]) - - mock_notion.pages.update.assert_called_once() - call_kwargs = mock_notion.pages.update.call_args[1] - assert call_kwargs["page_id"] == "page-id-abc" - score_val = call_kwargs["properties"]["Match Score"]["number"] - assert score_val == 85.5 -``` - -**Step 2: Run tests to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_match.py -v -``` - -Expected: `ImportError` — `scripts.match` doesn't exist. - -**Step 3: Write scripts/match.py** - -```python -# scripts/match.py -""" -Resume Matcher integration: score a Notion job listing against Alex's resume. -Writes Match Score and Keyword Gaps back to the Notion page. - -Usage: - conda run -n job-seeker python scripts/match.py -""" -import re -import sys -from pathlib import Path - -import requests -import yaml -from bs4 import BeautifulSoup -from notion_client import Client - -CONFIG_DIR = Path(__file__).parent.parent / "config" -RESUME_PATH = Path("/Library/Documents/JobSearch/Alex_Rivera_Resume_02-19-2025.pdf") - - -def load_notion() -> tuple[Client, str]: - cfg = yaml.safe_load((CONFIG_DIR / "notion.yaml").read_text()) - return Client(auth=cfg["token"]), cfg["database_id"] - - -def extract_page_id(url_or_id: str) -> str: - """Extract 32-char Notion page ID from a URL or return as-is.""" - match = re.search(r"[0-9a-f]{32}", url_or_id.replace("-", "")) - if match: - return match.group(0) - return url_or_id.strip() - - -def get_job_url_from_notion(notion: Client, page_id: str) -> str: - page = notion.pages.retrieve(page_id) - return page["properties"]["URL"]["url"] - - -def extract_job_description(url: str) -> str: - """Fetch a job listing URL and return its visible text.""" - resp = requests.get(url, headers={"User-Agent": "Mozilla/5.0"}, timeout=10) - resp.raise_for_status() - soup = BeautifulSoup(resp.text, "html.parser") - for tag in soup(["script", "style", "nav", "header", "footer"]): - tag.decompose() - return " ".join(soup.get_text(separator=" ").split()) - - -def read_resume_text() -> str: - """Extract text from the ATS-clean PDF resume.""" - try: - import pypdf - reader = pypdf.PdfReader(str(RESUME_PATH)) - return " ".join(page.extract_text() or "" for page in reader.pages) - except ImportError: - import PyPDF2 - with open(RESUME_PATH, "rb") as f: - reader = PyPDF2.PdfReader(f) - return " ".join(p.extract_text() or "" for p in reader.pages) - - -def match_score(resume_text: str, job_text: str) -> tuple[float, list[str]]: - """ - Score resume against job description using TF-IDF keyword overlap. - Returns (score 0-100, list of keywords in job not found in resume). - """ - from sklearn.feature_extraction.text import TfidfVectorizer - from sklearn.metrics.pairwise import cosine_similarity - import numpy as np - - vectorizer = TfidfVectorizer(stop_words="english", max_features=200) - tfidf = vectorizer.fit_transform([resume_text, job_text]) - score = float(cosine_similarity(tfidf[0:1], tfidf[1:2])[0][0]) * 100 - - # Keyword gap: terms in job description not present in resume (lowercased) - job_terms = set(job_text.lower().split()) - resume_terms = set(resume_text.lower().split()) - feature_names = vectorizer.get_feature_names_out() - job_tfidf = tfidf[1].toarray()[0] - top_indices = np.argsort(job_tfidf)[::-1][:30] - top_job_terms = [feature_names[i] for i in top_indices if job_tfidf[i] > 0] - gaps = [t for t in top_job_terms if t not in resume_terms][:10] - - return round(score, 1), gaps - - -def write_match_to_notion(notion: Client, page_id: str, score: float, gaps: list[str]) -> None: - notion.pages.update( - page_id=page_id, - properties={ - "Match Score": {"number": score}, - "Keyword Gaps": {"rich_text": [{"text": {"content": ", ".join(gaps)}}]}, - }, - ) - - -def run_match(page_url_or_id: str) -> None: - notion, _ = load_notion() - page_id = extract_page_id(page_url_or_id) - - print(f"[match] Page ID: {page_id}") - job_url = get_job_url_from_notion(notion, page_id) - print(f"[match] Fetching job description from: {job_url}") - - job_text = extract_job_description(job_url) - resume_text = read_resume_text() - - score, gaps = match_score(resume_text, job_text) - print(f"[match] Score: {score}/100") - print(f"[match] Keyword gaps: {', '.join(gaps) or 'none'}") - - write_match_to_notion(notion, page_id, score, gaps) - print("[match] Written to Notion.") - - -if __name__ == "__main__": - if len(sys.argv) < 2: - print("Usage: python scripts/match.py ") - sys.exit(1) - run_match(sys.argv[1]) -``` - -**Step 4: Install sklearn (needed by match.py)** - -```bash -conda run -n job-seeker pip install scikit-learn beautifulsoup4 pypdf -``` - -**Step 5: Run tests** - -```bash -conda run -n job-seeker pytest tests/test_match.py -v -``` - -Expected: 3 tests PASS. - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add scripts/match.py tests/test_match.py -git commit -m "feat: add resume match scoring with Notion write-back" -``` - ---- - -## Task 8: Clone and Configure AIHawk - -**Step 1: Clone AIHawk** - -```bash -cd /devl/job-seeker -git clone https://github.com/feder-cr/Auto_Jobs_Applier_AIHawk.git aihawk -``` - -**Step 2: Install AIHawk dependencies** - -```bash -conda run -n job-seeker pip install -r /devl/job-seeker/aihawk/requirements.txt -``` - -**Step 3: Install Playwright browsers (AIHawk uses Playwright for browser automation)** - -```bash -conda run -n job-seeker playwright install chromium -``` - -**Step 4: Create AIHawk personal info config** - -AIHawk reads a `personal_info.yaml`. Create it in AIHawk's data directory: - -```bash -cp /devl/job-seeker/aihawk/data_folder/plain_text_resume.yaml \ - /devl/job-seeker/aihawk/data_folder/plain_text_resume.yaml.bak -``` - -Edit `/devl/job-seeker/aihawk/data_folder/plain_text_resume.yaml` with Alex's info. -Key fields to fill: -- `personal_information`: name, email, phone, linkedin, github (leave blank), location -- `work_experience`: pull from the SVG content already extracted -- `education`: Texas State University, Mass Communications & PR, 2012-2015 -- `skills`: Zendesk, Intercom, Asana, Jira, etc. - -**Step 5: Configure AIHawk to use the LLM router** - -AIHawk's config (`aihawk/data_folder/config.yaml`) has an `llm_model_type` and `llm_model` field. -Set it to use the local OpenAI-compatible endpoint: - -```yaml -# In aihawk/data_folder/config.yaml -llm_model_type: openai -llm_model: claude-code-terminal -openai_api_url: http://localhost:3009/v1 # or whichever backend is running -``` - -If 3009 is down, change to `http://localhost:11434/v1` (Ollama). - -**Step 6: Run AIHawk in dry-run mode first** - -```bash -conda run -n job-seeker python /devl/job-seeker/aihawk/main.py --help -``` - -Review the flags. Start with a test run before enabling real submissions. - -**Step 7: Commit the environment update** - -```bash -cd /devl/job-seeker -conda env export -n job-seeker > environment.yml -git add environment.yml -git commit -m "chore: update environment.yml with all installed packages" -``` - ---- - -## Task 9: End-to-End Smoke Test - -**Step 1: Run full test suite** - -```bash -conda run -n job-seeker pytest tests/ -v -``` - -Expected: all tests PASS. - -**Step 2: Run discovery** - -```bash -conda run -n job-seeker python scripts/discover.py -``` - -Expected: new listings appear in Notion with Status=New. - -**Step 3: Run match on one listing** - -Copy the URL of a Notion page from the DB and run: - -```bash -conda run -n job-seeker python scripts/match.py "https://www.notion.so/..." -``` - -Expected: Match Score and Keyword Gaps written back to that Notion page. - -**Step 4: Commit anything left** - -```bash -cd /devl/job-seeker -git status -git add -p # stage only code/config, not secrets -git commit -m "chore: final smoke test cleanup" -``` - ---- - -## Quick Reference - -| Command | What it does | -|---|---| -| `conda run -n job-seeker python scripts/discover.py` | Scrape boards → push new listings to Notion | -| `conda run -n job-seeker python scripts/match.py ` | Score a listing → write back to Notion | -| `conda run -n job-seeker streamlit run resume_matcher/streamlit_app.py --server.port 8501` | Open Resume Matcher UI | -| `conda run -n job-seeker pytest tests/ -v` | Run all tests | -| `cd "/Library/Documents/Post Fight Processing" && ./manage.sh start` | Start Claude Code pipeline (port 3009) | -| `cd "/Library/Documents/Post Fight Processing" && ./manage-copilot.sh start` | Start Copilot wrapper (port 3010) | diff --git a/docs/plans/2026-02-20-ui-design.md b/docs/plans/2026-02-20-ui-design.md deleted file mode 100644 index 3088b0a..0000000 --- a/docs/plans/2026-02-20-ui-design.md +++ /dev/null @@ -1,148 +0,0 @@ -# Job Seeker Platform — Web UI Design - -**Date:** 2026-02-20 -**Status:** Approved - -## Overview - -A Streamlit multi-page web UI that gives Alex (and her partner) a friendly interface to review scraped job listings, curate them before they hit Notion, edit search/LLM/Notion settings, and fill out her AIHawk application profile. Designed to be usable by anyone — no technical knowledge required. - ---- - -## Architecture & Data Flow - -``` -discover.py → SQLite staging.db (status: pending) - ↓ - Streamlit UI - review / approve / reject - ↓ - "Sync N approved jobs" button - ↓ - Notion DB (status: synced) -``` - -`discover.py` is modified to write to SQLite instead of directly to Notion. -A new `sync.py` handles the approved → Notion push. -`db.py` provides shared SQLite helpers used by both scripts and UI pages. - -### SQLite Schema (`staging.db`, gitignored) - -```sql -CREATE TABLE jobs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - title TEXT, - company TEXT, - url TEXT UNIQUE, - source TEXT, - location TEXT, - is_remote INTEGER, - salary TEXT, - description TEXT, - match_score REAL, - keyword_gaps TEXT, - date_found TEXT, - status TEXT DEFAULT 'pending', -- pending / approved / rejected / synced - notion_page_id TEXT -); -``` - ---- - -## Pages - -### Home (Dashboard) -- Stat cards: Pending / Approved / Rejected / Synced counts -- "Run Discovery" button — runs `discover.py` as subprocess, streams output -- "Sync N approved jobs → Notion" button — visible only when approved count > 0 -- Recent activity list (last 10 jobs found) - -### Job Review -- Filterable table/card view of pending jobs -- Filters: source (LinkedIn/Indeed/etc), remote only toggle, minimum match score slider -- Checkboxes for batch selection -- "Approve Selected" / "Reject Selected" buttons -- Rejected jobs hidden by default, togglable -- Match score shown as colored badge (green ≥70, amber 40–69, red <40) - -### Settings -Three tabs: - -**Search** — edit `config/search_profiles.yaml`: -- Job titles (add/remove tags) -- Locations (add/remove) -- Boards checkboxes -- Hours old slider -- Results per board slider - -**LLM Backends** — edit `config/llm.yaml`: -- Fallback order (drag or up/down arrows) -- Per-backend: URL, model name, enabled toggle -- "Test connection" button per backend - -**Notion** — edit `config/notion.yaml`: -- Token field (masked, show/hide toggle) -- Database ID -- "Test connection" button - -### Resume Editor -Sectioned form over `aihawk/data_folder/plain_text_resume.yaml`: -- **Personal Info** — name, email, phone, LinkedIn, city, zip -- **Education** — list of entries, add/remove buttons -- **Experience** — list of entries, add/remove buttons -- **Skills & Interests** — tag-style inputs -- **Preferences** — salary range, notice period, remote/relocation toggles -- **Self-Identification** — gender, pronouns, veteran, disability, ethnicity (with "prefer not to say" options) -- **Legal** — work authorization checkboxes - -`FILL_IN` fields highlighted in amber with "Needs your attention" note. -Save button writes back to YAML. No raw YAML shown by default. - ---- - -## Theme & Styling - -Central theme at `app/.streamlit/config.toml`: -- Dark base, accent color teal/green (job search = growth) -- Consistent font (Inter or system sans-serif) -- Responsive column layouts — usable on tablet/mobile -- No jargon — "Run Discovery" not "Execute scrape", "Sync to Notion" not "Push records" - ---- - -## File Layout - -``` -app/ -├── .streamlit/ -│ └── config.toml # central theme -├── Home.py # dashboard -└── pages/ - ├── 1_Job_Review.py - ├── 2_Settings.py - └── 3_Resume_Editor.py -scripts/ -├── db.py # new: SQLite helpers -├── sync.py # new: approved → Notion push -├── discover.py # modified: write to SQLite not Notion -├── match.py # unchanged -└── llm_router.py # unchanged -``` - -Run: `conda run -n job-seeker streamlit run app/Home.py` - ---- - -## New Dependencies - -None — `streamlit` already installed via resume_matcher deps. -`sqlite3` is Python stdlib. - ---- - -## Out of Scope - -- Real-time collaboration -- Mobile native app -- Cover letter editor (handled separately via LoRA fine-tune task) -- AIHawk trigger from UI (run manually for now) diff --git a/docs/plans/2026-02-20-ui-implementation.md b/docs/plans/2026-02-20-ui-implementation.md deleted file mode 100644 index ba235ae..0000000 --- a/docs/plans/2026-02-20-ui-implementation.md +++ /dev/null @@ -1,1458 +0,0 @@ -# Job Seeker Web UI Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Build a Streamlit web UI with SQLite staging so Alex can review scraped jobs, approve/batch-sync to Notion, edit settings, and complete her AIHawk profile. - -**Architecture:** `discover.py` writes to a local SQLite `staging.db` instead of Notion directly. Streamlit pages read/write SQLite for job review, YAML files for settings and resume. A new `sync.py` pushes approved jobs to Notion on demand. - -**Tech Stack:** Python 3.12, Streamlit (already installed), sqlite3 (stdlib), pyyaml, notion-client, conda env `job-seeker` - ---- - -## Task 1: SQLite DB helpers (`db.py`) - -**Files:** -- Create: `scripts/db.py` -- Create: `tests/test_db.py` -- Modify: `.gitignore` (add `staging.db`) - -**Step 1: Add staging.db to .gitignore** - -```bash -echo "staging.db" >> /devl/job-seeker/.gitignore -``` - -**Step 2: Write failing tests** - -```python -# tests/test_db.py -import pytest -import sqlite3 -from pathlib import Path -from unittest.mock import patch - - -def test_init_db_creates_jobs_table(tmp_path): - """init_db creates a jobs table with correct schema.""" - from scripts.db import init_db - db_path = tmp_path / "test.db" - init_db(db_path) - conn = sqlite3.connect(db_path) - cursor = conn.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='jobs'") - assert cursor.fetchone() is not None - conn.close() - - -def test_insert_job_returns_id(tmp_path): - """insert_job inserts a row and returns its id.""" - from scripts.db import init_db, insert_job - db_path = tmp_path / "test.db" - init_db(db_path) - job = { - "title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "$100k", "description": "Great role", "date_found": "2026-02-20", - } - row_id = insert_job(db_path, job) - assert isinstance(row_id, int) - assert row_id > 0 - - -def test_insert_job_skips_duplicate_url(tmp_path): - """insert_job returns None if URL already exists.""" - from scripts.db import init_db, insert_job - db_path = tmp_path / "test.db" - init_db(db_path) - job = {"title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20"} - insert_job(db_path, job) - result = insert_job(db_path, job) - assert result is None - - -def test_get_jobs_by_status(tmp_path): - """get_jobs_by_status returns only jobs with matching status.""" - from scripts.db import init_db, insert_job, get_jobs_by_status, update_job_status - db_path = tmp_path / "test.db" - init_db(db_path) - job = {"title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20"} - row_id = insert_job(db_path, job) - update_job_status(db_path, [row_id], "approved") - approved = get_jobs_by_status(db_path, "approved") - pending = get_jobs_by_status(db_path, "pending") - assert len(approved) == 1 - assert len(pending) == 0 - - -def test_update_job_status_batch(tmp_path): - """update_job_status updates multiple rows at once.""" - from scripts.db import init_db, insert_job, update_job_status, get_jobs_by_status - db_path = tmp_path / "test.db" - init_db(db_path) - ids = [] - for i in range(3): - job = {"title": f"Job {i}", "company": "Co", "url": f"https://example.com/{i}", - "source": "indeed", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20"} - ids.append(insert_job(db_path, job)) - update_job_status(db_path, ids, "rejected") - assert len(get_jobs_by_status(db_path, "rejected")) == 3 -``` - -**Step 3: Run tests — expect ImportError** - -```bash -conda run -n job-seeker pytest tests/test_db.py -v -``` - -Expected: `ModuleNotFoundError: No module named 'scripts.db'` - -**Step 4: Write `scripts/db.py`** - -```python -# scripts/db.py -""" -SQLite staging layer for job listings. -Jobs flow: pending → approved/rejected → synced -""" -import sqlite3 -from pathlib import Path -from typing import Optional - -DEFAULT_DB = Path(__file__).parent.parent / "staging.db" - -CREATE_JOBS = """ -CREATE TABLE IF NOT EXISTS jobs ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - title TEXT, - company TEXT, - url TEXT UNIQUE, - source TEXT, - location TEXT, - is_remote INTEGER DEFAULT 0, - salary TEXT, - description TEXT, - match_score REAL, - keyword_gaps TEXT, - date_found TEXT, - status TEXT DEFAULT 'pending', - notion_page_id TEXT -); -""" - - -def init_db(db_path: Path = DEFAULT_DB) -> None: - """Create tables if they don't exist.""" - conn = sqlite3.connect(db_path) - conn.execute(CREATE_JOBS) - conn.commit() - conn.close() - - -def insert_job(db_path: Path = DEFAULT_DB, job: dict = None) -> Optional[int]: - """ - Insert a job. Returns row id, or None if URL already exists. - """ - if job is None: - return None - conn = sqlite3.connect(db_path) - try: - cursor = conn.execute( - """INSERT INTO jobs - (title, company, url, source, location, is_remote, salary, description, date_found) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", - ( - job.get("title", ""), - job.get("company", ""), - job.get("url", ""), - job.get("source", ""), - job.get("location", ""), - int(bool(job.get("is_remote", False))), - job.get("salary", ""), - job.get("description", ""), - job.get("date_found", ""), - ), - ) - conn.commit() - return cursor.lastrowid - except sqlite3.IntegrityError: - return None # duplicate URL - finally: - conn.close() - - -def get_jobs_by_status(db_path: Path = DEFAULT_DB, status: str = "pending") -> list[dict]: - """Return all jobs with the given status as a list of dicts.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - cursor = conn.execute( - "SELECT * FROM jobs WHERE status = ? ORDER BY date_found DESC, id DESC", - (status,), - ) - rows = [dict(row) for row in cursor.fetchall()] - conn.close() - return rows - - -def get_job_counts(db_path: Path = DEFAULT_DB) -> dict: - """Return counts per status.""" - conn = sqlite3.connect(db_path) - cursor = conn.execute( - "SELECT status, COUNT(*) as n FROM jobs GROUP BY status" - ) - counts = {row[0]: row[1] for row in cursor.fetchall()} - conn.close() - return counts - - -def update_job_status(db_path: Path = DEFAULT_DB, ids: list[int] = None, status: str = "approved") -> None: - """Batch-update status for a list of job IDs.""" - if not ids: - return - conn = sqlite3.connect(db_path) - conn.execute( - f"UPDATE jobs SET status = ? WHERE id IN ({','.join('?' * len(ids))})", - [status] + list(ids), - ) - conn.commit() - conn.close() - - -def get_existing_urls(db_path: Path = DEFAULT_DB) -> set[str]: - """Return all URLs already in staging (any status).""" - conn = sqlite3.connect(db_path) - cursor = conn.execute("SELECT url FROM jobs") - urls = {row[0] for row in cursor.fetchall()} - conn.close() - return urls - - -def write_match_scores(db_path: Path = DEFAULT_DB, job_id: int = None, - score: float = 0.0, gaps: str = "") -> None: - """Write match score and keyword gaps back to a job row.""" - conn = sqlite3.connect(db_path) - conn.execute( - "UPDATE jobs SET match_score = ?, keyword_gaps = ? WHERE id = ?", - (score, gaps, job_id), - ) - conn.commit() - conn.close() -``` - -**Step 5: Run tests — expect 5 passing** - -```bash -conda run -n job-seeker pytest tests/test_db.py -v -``` - -Expected: `5 passed` - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add scripts/db.py tests/test_db.py .gitignore -git commit -m "feat: add SQLite staging layer (db.py)" -``` - ---- - -## Task 2: Update `discover.py` to write to SQLite - -**Files:** -- Modify: `scripts/discover.py` -- Modify: `tests/test_discover.py` - -**Step 1: Update the tests** - -Replace the existing `tests/test_discover.py` with this version that tests SQLite writes: - -```python -# tests/test_discover.py -import pytest -from unittest.mock import patch, MagicMock -import pandas as pd -from pathlib import Path - -SAMPLE_JOB = { - "title": "Customer Success Manager", - "company": "Acme Corp", - "location": "Remote", - "is_remote": True, - "job_url": "https://linkedin.com/jobs/view/123456", - "site": "linkedin", - "min_amount": 90000, - "max_amount": 120000, - "salary_source": "$90,000 - $120,000", - "description": "Great CS role", -} - -SAMPLE_FM = { - "title_field": "Salary", "job_title": "Job Title", "company": "Company Name", - "url": "Role Link", "source": "Job Source", "status": "Status of Application", - "status_new": "Application Submitted", "date_found": "Date Found", - "remote": "Remote", "match_score": "Match Score", - "keyword_gaps": "Keyword Gaps", "notes": "Notes", "job_description": "Job Description", -} - -SAMPLE_NOTION_CFG = {"token": "secret_test", "database_id": "fake-db-id", "field_map": SAMPLE_FM} -SAMPLE_PROFILES_CFG = { - "profiles": [{"name": "cs", "titles": ["Customer Success Manager"], - "locations": ["Remote"], "boards": ["linkedin"], - "results_per_board": 5, "hours_old": 72}] -} - - -def make_jobs_df(jobs=None): - return pd.DataFrame(jobs or [SAMPLE_JOB]) - - -def test_discover_writes_to_sqlite(tmp_path): - """run_discovery inserts new jobs into SQLite staging db.""" - from scripts.discover import run_discovery - from scripts.db import get_jobs_by_status - - db_path = tmp_path / "test.db" - with patch("scripts.discover.load_config", return_value=(SAMPLE_PROFILES_CFG, SAMPLE_NOTION_CFG)), \ - patch("scripts.discover.scrape_jobs", return_value=make_jobs_df()), \ - patch("scripts.discover.Client"): - run_discovery(db_path=db_path) - - jobs = get_jobs_by_status(db_path, "pending") - assert len(jobs) == 1 - assert jobs[0]["title"] == "Customer Success Manager" - - -def test_discover_skips_duplicate_urls(tmp_path): - """run_discovery does not insert a job whose URL is already in SQLite.""" - from scripts.discover import run_discovery - from scripts.db import init_db, insert_job, get_jobs_by_status - - db_path = tmp_path / "test.db" - init_db(db_path) - insert_job(db_path, { - "title": "Old", "company": "X", "url": "https://linkedin.com/jobs/view/123456", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-01-01", - }) - - with patch("scripts.discover.load_config", return_value=(SAMPLE_PROFILES_CFG, SAMPLE_NOTION_CFG)), \ - patch("scripts.discover.scrape_jobs", return_value=make_jobs_df()), \ - patch("scripts.discover.Client"): - run_discovery(db_path=db_path) - - jobs = get_jobs_by_status(db_path, "pending") - assert len(jobs) == 1 # only the pre-existing one, not a duplicate - - -def test_discover_pushes_new_jobs(): - """Legacy: discover still calls push_to_notion when notion_push=True.""" - from scripts.discover import run_discovery - import tempfile, os - db_path = Path(tempfile.mktemp(suffix=".db")) - try: - with patch("scripts.discover.load_config", return_value=(SAMPLE_PROFILES_CFG, SAMPLE_NOTION_CFG)), \ - patch("scripts.discover.scrape_jobs", return_value=make_jobs_df()), \ - patch("scripts.discover.push_to_notion") as mock_push, \ - patch("scripts.discover.Client"): - run_discovery(db_path=db_path, notion_push=True) - assert mock_push.call_count == 1 - finally: - if db_path.exists(): - os.unlink(db_path) - - -def test_push_to_notion_sets_status_new(): - """push_to_notion always sets Status to the configured status_new value.""" - from scripts.discover import push_to_notion - mock_notion = MagicMock() - push_to_notion(mock_notion, "fake-db-id", SAMPLE_JOB, SAMPLE_FM) - call_kwargs = mock_notion.pages.create.call_args[1] - status = call_kwargs["properties"]["Status of Application"]["select"]["name"] - assert status == "Application Submitted" -``` - -**Step 2: Run tests — some will fail** - -```bash -conda run -n job-seeker pytest tests/test_discover.py -v -``` - -Expected: `test_discover_writes_to_sqlite` and `test_discover_skips_duplicate_urls` fail. - -**Step 3: Update `scripts/discover.py`** - -Add `db_path` and `notion_push` parameters to `run_discovery`. Default writes to SQLite only: - -```python -# scripts/discover.py -""" -JobSpy → SQLite staging pipeline (default) or Notion (notion_push=True). - -Usage: - conda run -n job-seeker python scripts/discover.py -""" -import yaml -from datetime import datetime -from pathlib import Path - -import pandas as pd -from jobspy import scrape_jobs -from notion_client import Client - -from scripts.db import DEFAULT_DB, init_db, insert_job, get_existing_urls as db_existing_urls - -CONFIG_DIR = Path(__file__).parent.parent / "config" -NOTION_CFG = CONFIG_DIR / "notion.yaml" -PROFILES_CFG = CONFIG_DIR / "search_profiles.yaml" - - -def load_config() -> tuple[dict, dict]: - profiles = yaml.safe_load(PROFILES_CFG.read_text()) - notion_cfg = yaml.safe_load(NOTION_CFG.read_text()) - return profiles, notion_cfg - - -def get_existing_urls(notion: Client, db_id: str, url_field: str) -> set[str]: - """Return the set of all job URLs already tracked in Notion (for notion_push mode).""" - existing: set[str] = set() - has_more = True - start_cursor = None - while has_more: - kwargs: dict = {"database_id": db_id, "page_size": 100} - if start_cursor: - kwargs["start_cursor"] = start_cursor - resp = notion.databases.query(**kwargs) - for page in resp["results"]: - url = page["properties"].get(url_field, {}).get("url") - if url: - existing.add(url) - has_more = resp.get("has_more", False) - start_cursor = resp.get("next_cursor") - return existing - - -def push_to_notion(notion: Client, db_id: str, job: dict, fm: dict) -> None: - """Create a new page in the Notion jobs database for a single listing.""" - min_amt = job.get("min_amount") - max_amt = job.get("max_amount") - if min_amt and max_amt and not (pd.isna(min_amt) or pd.isna(max_amt)): - title_content = f"${int(min_amt):,} – ${int(max_amt):,}" - elif job.get("salary_source") and str(job["salary_source"]) not in ("nan", "None", ""): - title_content = str(job["salary_source"]) - else: - title_content = str(job.get("title", "Unknown")) - - job_url = str(job.get("job_url", "") or "") - if job_url in ("nan", "None"): - job_url = "" - - notion.pages.create( - parent={"database_id": db_id}, - properties={ - fm["title_field"]: {"title": [{"text": {"content": title_content}}]}, - fm["job_title"]: {"rich_text": [{"text": {"content": str(job.get("title", "Unknown"))}}]}, - fm["company"]: {"rich_text": [{"text": {"content": str(job.get("company", "") or "")}}]}, - fm["url"]: {"url": job_url or None}, - fm["source"]: {"multi_select": [{"name": str(job.get("site", "unknown")).title()}]}, - fm["status"]: {"select": {"name": fm["status_new"]}}, - fm["remote"]: {"checkbox": bool(job.get("is_remote", False))}, - fm["date_found"]: {"date": {"start": datetime.now().isoformat()[:10]}}, - }, - ) - - -def run_discovery(db_path: Path = DEFAULT_DB, notion_push: bool = False) -> None: - profiles_cfg, notion_cfg = load_config() - fm = notion_cfg["field_map"] - - # SQLite dedup - init_db(db_path) - existing_urls = db_existing_urls(db_path) - - # Notion dedup (only in notion_push mode) - notion = None - if notion_push: - notion = Client(auth=notion_cfg["token"]) - existing_urls |= get_existing_urls(notion, notion_cfg["database_id"], fm["url"]) - - print(f"[discover] {len(existing_urls)} existing listings") - new_count = 0 - - for profile in profiles_cfg["profiles"]: - print(f"\n[discover] Profile: {profile['name']}") - for location in profile["locations"]: - print(f" Scraping: {location}") - jobs: pd.DataFrame = scrape_jobs( - site_name=profile["boards"], - search_term=" OR ".join(f'"{t}"' for t in profile["titles"]), - location=location, - results_wanted=profile.get("results_per_board", 25), - hours_old=profile.get("hours_old", 72), - linkedin_fetch_description=True, - ) - - for _, job in jobs.iterrows(): - url = str(job.get("job_url", "") or "") - if not url or url in ("nan", "None") or url in existing_urls: - continue - - job_dict = job.to_dict() - - # Always write to SQLite staging - min_amt = job_dict.get("min_amount") - max_amt = job_dict.get("max_amount") - salary_str = "" - if min_amt and max_amt and not (pd.isna(min_amt) or pd.isna(max_amt)): - salary_str = f"${int(min_amt):,} – ${int(max_amt):,}" - elif job_dict.get("salary_source") and str(job_dict["salary_source"]) not in ("nan", "None", ""): - salary_str = str(job_dict["salary_source"]) - - insert_job(db_path, { - "title": str(job_dict.get("title", "")), - "company": str(job_dict.get("company", "") or ""), - "url": url, - "source": str(job_dict.get("site", "")), - "location": str(job_dict.get("location", "") or ""), - "is_remote": bool(job_dict.get("is_remote", False)), - "salary": salary_str, - "description": str(job_dict.get("description", "") or ""), - "date_found": datetime.now().isoformat()[:10], - }) - - # Optionally also push straight to Notion - if notion_push: - push_to_notion(notion, notion_cfg["database_id"], job_dict, fm) - - existing_urls.add(url) - new_count += 1 - print(f" + {job.get('title')} @ {job.get('company')}") - - print(f"\n[discover] Done — {new_count} new listings staged.") - - -if __name__ == "__main__": - run_discovery() -``` - -**Step 4: Run tests — expect 4 passing** - -```bash -conda run -n job-seeker pytest tests/test_discover.py -v -``` - -Expected: `4 passed` - -**Step 5: Run full suite** - -```bash -conda run -n job-seeker pytest tests/ -v -``` - -Expected: all tests pass. - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add scripts/discover.py tests/test_discover.py -git commit -m "feat: route discover.py through SQLite staging layer" -``` - ---- - -## Task 3: `sync.py` — approved → Notion push - -**Files:** -- Create: `scripts/sync.py` -- Create: `tests/test_sync.py` - -**Step 1: Write failing tests** - -```python -# tests/test_sync.py -import pytest -from unittest.mock import patch, MagicMock -from pathlib import Path - - -SAMPLE_FM = { - "title_field": "Salary", "job_title": "Job Title", "company": "Company Name", - "url": "Role Link", "source": "Job Source", "status": "Status of Application", - "status_new": "Application Submitted", "date_found": "Date Found", - "remote": "Remote", "match_score": "Match Score", - "keyword_gaps": "Keyword Gaps", "notes": "Notes", "job_description": "Job Description", -} - -SAMPLE_NOTION_CFG = {"token": "secret_test", "database_id": "fake-db-id", "field_map": SAMPLE_FM} - -SAMPLE_JOB = { - "id": 1, "title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": 1, - "salary": "$100k", "description": "Good role", "match_score": 80.0, - "keyword_gaps": "Gainsight, Churnzero", "date_found": "2026-02-20", - "status": "approved", "notion_page_id": None, -} - - -def test_sync_pushes_approved_jobs(tmp_path): - """sync_to_notion pushes approved jobs and marks them synced.""" - from scripts.sync import sync_to_notion - from scripts.db import init_db, insert_job, get_jobs_by_status, update_job_status - - db_path = tmp_path / "test.db" - init_db(db_path) - row_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "$100k", "description": "Good role", "date_found": "2026-02-20", - }) - update_job_status(db_path, [row_id], "approved") - - mock_notion = MagicMock() - mock_notion.pages.create.return_value = {"id": "notion-page-abc"} - - with patch("scripts.sync.load_notion_config", return_value=SAMPLE_NOTION_CFG), \ - patch("scripts.sync.Client", return_value=mock_notion): - count = sync_to_notion(db_path=db_path) - - assert count == 1 - mock_notion.pages.create.assert_called_once() - synced = get_jobs_by_status(db_path, "synced") - assert len(synced) == 1 - - -def test_sync_returns_zero_when_nothing_approved(tmp_path): - """sync_to_notion returns 0 when there are no approved jobs.""" - from scripts.sync import sync_to_notion - from scripts.db import init_db - - db_path = tmp_path / "test.db" - init_db(db_path) - - with patch("scripts.sync.load_notion_config", return_value=SAMPLE_NOTION_CFG), \ - patch("scripts.sync.Client"): - count = sync_to_notion(db_path=db_path) - - assert count == 0 -``` - -**Step 2: Run tests — expect ImportError** - -```bash -conda run -n job-seeker pytest tests/test_sync.py -v -``` - -Expected: `ModuleNotFoundError: No module named 'scripts.sync'` - -**Step 3: Write `scripts/sync.py`** - -```python -# scripts/sync.py -""" -Push approved jobs from SQLite staging to Notion. - -Usage: - conda run -n job-seeker python scripts/sync.py -""" -import yaml -from pathlib import Path -from datetime import datetime - -from notion_client import Client - -from scripts.db import DEFAULT_DB, get_jobs_by_status, update_job_status - -CONFIG_DIR = Path(__file__).parent.parent / "config" - - -def load_notion_config() -> dict: - return yaml.safe_load((CONFIG_DIR / "notion.yaml").read_text()) - - -def sync_to_notion(db_path: Path = DEFAULT_DB) -> int: - """Push all approved jobs to Notion. Returns count synced.""" - cfg = load_notion_config() - notion = Client(auth=cfg["token"]) - db_id = cfg["database_id"] - fm = cfg["field_map"] - - approved = get_jobs_by_status(db_path, "approved") - if not approved: - print("[sync] No approved jobs to sync.") - return 0 - - synced_ids = [] - for job in approved: - try: - page = notion.pages.create( - parent={"database_id": db_id}, - properties={ - fm["title_field"]: {"title": [{"text": {"content": job.get("salary") or job.get("title", "")}}]}, - fm["job_title"]: {"rich_text": [{"text": {"content": job.get("title", "")}}]}, - fm["company"]: {"rich_text": [{"text": {"content": job.get("company", "")}}]}, - fm["url"]: {"url": job.get("url") or None}, - fm["source"]: {"multi_select": [{"name": job.get("source", "unknown").title()}]}, - fm["status"]: {"select": {"name": fm["status_new"]}}, - fm["remote"]: {"checkbox": bool(job.get("is_remote", 0))}, - fm["date_found"]: {"date": {"start": job.get("date_found", datetime.now().isoformat()[:10])}}, - fm["match_score"]: {"number": job.get("match_score")}, - fm["keyword_gaps"]: {"rich_text": [{"text": {"content": job.get("keyword_gaps") or ""}}]}, - }, - ) - synced_ids.append(job["id"]) - print(f"[sync] + {job.get('title')} @ {job.get('company')}") - except Exception as e: - print(f"[sync] Error syncing {job.get('url')}: {e}") - - update_job_status(db_path, synced_ids, "synced") - print(f"[sync] Done — {len(synced_ids)} jobs synced to Notion.") - return len(synced_ids) - - -if __name__ == "__main__": - sync_to_notion() -``` - -**Step 4: Run tests — expect 2 passing** - -```bash -conda run -n job-seeker pytest tests/test_sync.py -v -``` - -Expected: `2 passed` - -**Step 5: Full suite** - -```bash -conda run -n job-seeker pytest tests/ -v -``` - -Expected: all pass. - -**Step 6: Commit** - -```bash -cd /devl/job-seeker -git add scripts/sync.py tests/test_sync.py -git commit -m "feat: add sync.py to push approved jobs from SQLite to Notion" -``` - ---- - -## Task 4: Streamlit theme + app scaffold - -**Files:** -- Create: `app/.streamlit/config.toml` -- Create: `app/Home.py` -- Create: `app/pages/1_Job_Review.py` (stub) -- Create: `app/pages/2_Settings.py` (stub) -- Create: `app/pages/3_Resume_Editor.py` (stub) - -No tests for Streamlit page rendering — test helper functions instead. - -**Step 1: Create theme** - -```toml -# app/.streamlit/config.toml -[theme] -base = "dark" -primaryColor = "#2DD4BF" # teal -backgroundColor = "#0F172A" # slate-900 -secondaryBackgroundColor = "#1E293B" # slate-800 -textColor = "#F1F5F9" # slate-100 -font = "sans serif" -``` - -**Step 2: Create `app/Home.py`** - -```python -# app/Home.py -""" -Job Seeker Dashboard — Home page. -Shows counts, Run Discovery button, and Sync to Notion button. -""" -import subprocess -import sys -from pathlib import Path - -import streamlit as st - -# Make scripts importable -sys.path.insert(0, str(Path(__file__).parent.parent)) - -from scripts.db import DEFAULT_DB, init_db, get_job_counts - -st.set_page_config( - page_title="Alex's Job Search", - page_icon="🔍", - layout="wide", -) - -init_db(DEFAULT_DB) -counts = get_job_counts(DEFAULT_DB) - -st.title("🔍 Alex's Job Search") -st.caption("Discover → Review → Sync to Notion") - -st.divider() - -# Stat cards -col1, col2, col3, col4 = st.columns(4) -col1.metric("Pending Review", counts.get("pending", 0)) -col2.metric("Approved", counts.get("approved", 0)) -col3.metric("Synced to Notion", counts.get("synced", 0)) -col4.metric("Rejected", counts.get("rejected", 0)) - -st.divider() - -# Actions -left, right = st.columns(2) - -with left: - st.subheader("Find New Jobs") - st.caption("Scrapes all configured boards and adds new listings to your review queue.") - if st.button("🚀 Run Discovery", use_container_width=True, type="primary"): - with st.spinner("Scraping job boards…"): - result = subprocess.run( - ["conda", "run", "-n", "job-seeker", "python", "scripts/discover.py"], - capture_output=True, text=True, - cwd=str(Path(__file__).parent.parent), - ) - if result.returncode == 0: - st.success("Discovery complete! Head to Job Review to see new listings.") - st.code(result.stdout) - else: - st.error("Discovery failed.") - st.code(result.stderr) - -with right: - approved_count = counts.get("approved", 0) - st.subheader("Send to Notion") - st.caption("Push all approved jobs to your Notion tracking database.") - if approved_count == 0: - st.info("No approved jobs yet. Review and approve some listings first.") - else: - if st.button(f"📤 Sync {approved_count} approved job{'s' if approved_count != 1 else ''} → Notion", - use_container_width=True, type="primary"): - with st.spinner("Syncing to Notion…"): - from scripts.sync import sync_to_notion - count = sync_to_notion(DEFAULT_DB) - st.success(f"Synced {count} job{'s' if count != 1 else ''} to Notion!") - st.rerun() -``` - -**Step 3: Create page stubs** - -```python -# app/pages/1_Job_Review.py -import streamlit as st -st.set_page_config(page_title="Job Review", page_icon="📋", layout="wide") -st.title("📋 Job Review") -st.info("Coming soon — Task 5") -``` - -```python -# app/pages/2_Settings.py -import streamlit as st -st.set_page_config(page_title="Settings", page_icon="⚙️", layout="wide") -st.title("⚙️ Settings") -st.info("Coming soon — Task 6") -``` - -```python -# app/pages/3_Resume_Editor.py -import streamlit as st -st.set_page_config(page_title="Resume Editor", page_icon="📝", layout="wide") -st.title("📝 Resume Editor") -st.info("Coming soon — Task 7") -``` - -**Step 4: Smoke test** - -```bash -conda run -n job-seeker streamlit run /devl/job-seeker/app/Home.py --server.headless true & -sleep 4 -curl -s http://localhost:8501 | grep -q "Alex" && echo "OK" || echo "FAIL" -kill %1 -``` - -Expected: `OK` - -**Step 5: Commit** - -```bash -cd /devl/job-seeker -git add app/ -git commit -m "feat: add Streamlit app scaffold with dark theme and dashboard" -``` - ---- - -## Task 5: Job Review page - -**Files:** -- Modify: `app/pages/1_Job_Review.py` - -No separate unit tests — logic is inline Streamlit. Test manually after implement. - -**Step 1: Replace stub with full implementation** - -```python -# app/pages/1_Job_Review.py -""" -Job Review — browse pending listings, batch approve or reject. -""" -import sys -from pathlib import Path -sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - -import streamlit as st -from scripts.db import DEFAULT_DB, init_db, get_jobs_by_status, update_job_status - -st.set_page_config(page_title="Job Review", page_icon="📋", layout="wide") -st.title("📋 Job Review") - -init_db(DEFAULT_DB) - -# Filters sidebar -with st.sidebar: - st.header("Filters") - show_status = st.selectbox("Show", ["pending", "approved", "rejected", "synced"], index=0) - remote_only = st.checkbox("Remote only", value=False) - min_score = st.slider("Min match score", 0, 100, 0) - st.divider() - st.caption("Use checkboxes to select jobs, then approve or reject in bulk.") - -jobs = get_jobs_by_status(DEFAULT_DB, show_status) - -# Apply filters -if remote_only: - jobs = [j for j in jobs if j.get("is_remote")] -if min_score > 0: - jobs = [j for j in jobs if (j.get("match_score") or 0) >= min_score] - -if not jobs: - st.info(f"No {show_status} jobs matching your filters.") - st.stop() - -st.caption(f"Showing {len(jobs)} {show_status} job{'s' if len(jobs) != 1 else ''}") - -# Batch action buttons (only relevant for pending) -if show_status == "pending": - col_a, col_b, col_c = st.columns([2, 2, 6]) - select_all = col_a.button("Select all", use_container_width=True) - clear_all = col_b.button("Clear all", use_container_width=True) - - if "selected_ids" not in st.session_state: - st.session_state.selected_ids = set() - if select_all: - st.session_state.selected_ids = {j["id"] for j in jobs} - if clear_all: - st.session_state.selected_ids = set() - - col_approve, col_reject, _ = st.columns([2, 2, 6]) - if col_approve.button("✅ Approve selected", use_container_width=True, type="primary", - disabled=not st.session_state.selected_ids): - update_job_status(DEFAULT_DB, list(st.session_state.selected_ids), "approved") - st.session_state.selected_ids = set() - st.success("Approved!") - st.rerun() - if col_reject.button("❌ Reject selected", use_container_width=True, - disabled=not st.session_state.selected_ids): - update_job_status(DEFAULT_DB, list(st.session_state.selected_ids), "rejected") - st.session_state.selected_ids = set() - st.success("Rejected.") - st.rerun() - -st.divider() - -# Job cards -for job in jobs: - score = job.get("match_score") - if score is None: - score_badge = "⬜ No score" - elif score >= 70: - score_badge = f"🟢 {score:.0f}%" - elif score >= 40: - score_badge = f"🟡 {score:.0f}%" - else: - score_badge = f"🔴 {score:.0f}%" - - remote_badge = "🌐 Remote" if job.get("is_remote") else "🏢 On-site" - source_badge = job.get("source", "").title() - - with st.container(border=True): - left, right = st.columns([8, 2]) - with left: - checked = st.checkbox( - f"**{job['title']}** — {job['company']}", - key=f"chk_{job['id']}", - value=job["id"] in st.session_state.get("selected_ids", set()), - ) - if checked: - st.session_state.setdefault("selected_ids", set()).add(job["id"]) - else: - st.session_state.setdefault("selected_ids", set()).discard(job["id"]) - - cols = st.columns(4) - cols[0].caption(remote_badge) - cols[1].caption(f"📌 {source_badge}") - cols[2].caption(score_badge) - cols[3].caption(f"📅 {job.get('date_found', '')}") - - if job.get("keyword_gaps"): - st.caption(f"**Keyword gaps:** {job['keyword_gaps']}") - - with right: - if job.get("url"): - st.link_button("View listing →", job["url"], use_container_width=True) - if job.get("salary"): - st.caption(f"💰 {job['salary']}") -``` - -**Step 2: Manual smoke test** - -```bash -conda run -n job-seeker streamlit run /devl/job-seeker/app/Home.py -``` - -Open http://localhost:8501, navigate to Job Review. Confirm filters and empty state work. - -**Step 3: Commit** - -```bash -cd /devl/job-seeker -git add app/pages/1_Job_Review.py -git commit -m "feat: add Job Review page with batch approve/reject" -``` - ---- - -## Task 6: Settings page - -**Files:** -- Modify: `app/pages/2_Settings.py` - -**Step 1: Replace stub** - -```python -# app/pages/2_Settings.py -""" -Settings — edit search profiles, LLM backends, and Notion connection. -""" -import sys -from pathlib import Path -sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - -import streamlit as st -import yaml - -st.set_page_config(page_title="Settings", page_icon="⚙️", layout="wide") -st.title("⚙️ Settings") - -CONFIG_DIR = Path(__file__).parent.parent.parent / "config" -SEARCH_CFG = CONFIG_DIR / "search_profiles.yaml" -LLM_CFG = CONFIG_DIR / "llm.yaml" -NOTION_CFG = CONFIG_DIR / "notion.yaml" - - -def load_yaml(path: Path) -> dict: - if path.exists(): - return yaml.safe_load(path.read_text()) or {} - return {} - - -def save_yaml(path: Path, data: dict) -> None: - path.write_text(yaml.dump(data, default_flow_style=False, allow_unicode=True)) - - -tab_search, tab_llm, tab_notion = st.tabs(["🔎 Search", "🤖 LLM Backends", "📚 Notion"]) - -# ── Search tab ────────────────────────────────────────────────────────────── -with tab_search: - cfg = load_yaml(SEARCH_CFG) - profiles = cfg.get("profiles", [{}]) - p = profiles[0] # edit first profile for now - - st.subheader("Job Titles to Search") - titles_text = st.text_area( - "One title per line", - value="\n".join(p.get("titles", [])), - height=150, - help="JobSpy will search for any of these titles across all configured boards.", - ) - - st.subheader("Locations") - locations_text = st.text_area( - "One location per line", - value="\n".join(p.get("locations", [])), - height=100, - ) - - st.subheader("Job Boards") - board_options = ["linkedin", "indeed", "glassdoor", "zip_recruiter"] - selected_boards = st.multiselect( - "Active boards", board_options, - default=p.get("boards", board_options), - ) - - col1, col2 = st.columns(2) - results_per = col1.slider("Results per board", 5, 100, p.get("results_per_board", 25)) - hours_old = col2.slider("How far back to look (hours)", 24, 720, p.get("hours_old", 72)) - - if st.button("💾 Save search settings", type="primary"): - profiles[0] = { - **p, - "titles": [t.strip() for t in titles_text.splitlines() if t.strip()], - "locations": [l.strip() for l in locations_text.splitlines() if l.strip()], - "boards": selected_boards, - "results_per_board": results_per, - "hours_old": hours_old, - } - save_yaml(SEARCH_CFG, {"profiles": profiles}) - st.success("Search settings saved!") - -# ── LLM Backends tab ──────────────────────────────────────────────────────── -with tab_llm: - cfg = load_yaml(LLM_CFG) - backends = cfg.get("backends", {}) - fallback_order = cfg.get("fallback_order", list(backends.keys())) - - st.subheader("Fallback Order") - st.caption("Backends are tried top-to-bottom. First reachable one wins.") - st.write(" → ".join(fallback_order)) - - st.subheader("Backend Configuration") - updated_backends = {} - for name in fallback_order: - b = backends.get(name, {}) - with st.expander(f"**{name.replace('_', ' ').title()}**", expanded=False): - if b.get("type") == "openai_compat": - url = st.text_input("URL", value=b.get("base_url", ""), key=f"{name}_url") - model = st.text_input("Model", value=b.get("model", ""), key=f"{name}_model") - updated_backends[name] = {**b, "base_url": url, "model": model} - elif b.get("type") == "anthropic": - model = st.text_input("Model", value=b.get("model", ""), key=f"{name}_model") - updated_backends[name] = {**b, "model": model} - else: - updated_backends[name] = b - - if st.button(f"Test {name}", key=f"test_{name}"): - with st.spinner("Testing…"): - try: - import sys - sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - from scripts.llm_router import LLMRouter - r = LLMRouter() - reachable = r._is_reachable(b.get("base_url", "")) - st.success("Reachable ✓") if reachable else st.warning("Not reachable") - except Exception as e: - st.error(f"Error: {e}") - - if st.button("💾 Save LLM settings", type="primary"): - save_yaml(LLM_CFG, {**cfg, "backends": updated_backends}) - st.success("LLM settings saved!") - -# ── Notion tab ─────────────────────────────────────────────────────────────── -with tab_notion: - cfg = load_yaml(NOTION_CFG) if NOTION_CFG.exists() else {} - - st.subheader("Notion Connection") - token = st.text_input( - "Integration Token", - value=cfg.get("token", ""), - type="password", - help="Find this at notion.so/my-integrations → your integration → Internal Integration Token", - ) - db_id = st.text_input( - "Database ID", - value=cfg.get("database_id", ""), - help="The 32-character ID from your Notion database URL", - ) - - col_save, col_test = st.columns(2) - if col_save.button("💾 Save Notion settings", type="primary"): - save_yaml(NOTION_CFG, {**cfg, "token": token, "database_id": db_id}) - st.success("Notion settings saved!") - - if col_test.button("🔌 Test connection"): - with st.spinner("Connecting…"): - try: - from notion_client import Client - n = Client(auth=token) - db = n.databases.retrieve(db_id) - st.success(f"Connected to: **{db['title'][0]['plain_text']}**") - except Exception as e: - st.error(f"Connection failed: {e}") -``` - -**Step 2: Manual smoke test** - -Navigate to Settings in the running Streamlit app. Confirm all three tabs render, save/load works. - -**Step 3: Commit** - -```bash -cd /devl/job-seeker -git add app/pages/2_Settings.py -git commit -m "feat: add Settings page with search, LLM, and Notion tabs" -``` - ---- - -## Task 7: Resume Editor page - -**Files:** -- Modify: `app/pages/3_Resume_Editor.py` - -**Step 1: Replace stub** - -```python -# app/pages/3_Resume_Editor.py -""" -Resume Editor — form-based editor for Alex's AIHawk profile YAML. -FILL_IN fields highlighted in amber. -""" -import sys -from pathlib import Path -sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - -import streamlit as st -import yaml - -st.set_page_config(page_title="Resume Editor", page_icon="📝", layout="wide") -st.title("📝 Resume Editor") -st.caption("Edit Alex's application profile used by AIHawk for LinkedIn Easy Apply.") - -RESUME_PATH = Path(__file__).parent.parent.parent / "aihawk" / "data_folder" / "plain_text_resume.yaml" - -if not RESUME_PATH.exists(): - st.error(f"Resume file not found at `{RESUME_PATH}`. Is AIHawk cloned?") - st.stop() - -data = yaml.safe_load(RESUME_PATH.read_text()) or {} - - -def field(label: str, value: str, key: str, help: str = "", password: bool = False) -> str: - """Render a text input, highlighted amber if value is FILL_IN.""" - needs_attention = str(value).startswith("FILL_IN") or value == "" - if needs_attention: - st.markdown( - f'

⚠️ Needs your attention

', - unsafe_allow_html=True, - ) - return st.text_input(label, value=value or "", key=key, help=help, - type="password" if password else "default") - - -st.divider() - -# ── Personal Info ────────────────────────────────────────────────────────── -with st.expander("👤 Personal Information", expanded=True): - info = data.get("personal_information", {}) - col1, col2 = st.columns(2) - with col1: - name = field("First Name", info.get("name", ""), "pi_name") - email = field("Email", info.get("email", ""), "pi_email") - phone = field("Phone", info.get("phone", ""), "pi_phone") - city = field("City", info.get("city", ""), "pi_city") - with col2: - surname = field("Last Name", info.get("surname", ""), "pi_surname") - linkedin = field("LinkedIn URL", info.get("linkedin", ""), "pi_linkedin") - zip_code = field("Zip Code", info.get("zip_code", ""), "pi_zip") - dob = field("Date of Birth", info.get("date_of_birth", ""), "pi_dob", - help="Format: MM/DD/YYYY") - -# ── Education ───────────────────────────────────────────────────────────── -with st.expander("🎓 Education"): - edu_list = data.get("education_details", [{}]) - updated_edu = [] - for i, edu in enumerate(edu_list): - st.markdown(f"**Entry {i+1}**") - col1, col2 = st.columns(2) - with col1: - inst = field("Institution", edu.get("institution", ""), f"edu_inst_{i}") - field_study = st.text_input("Field of Study", edu.get("field_of_study", ""), key=f"edu_field_{i}") - start = st.text_input("Start Year", edu.get("start_date", ""), key=f"edu_start_{i}") - with col2: - level = st.selectbox("Degree Level", - ["Bachelor's Degree", "Master's Degree", "Some College", "Associate's Degree", "High School", "Other"], - index=["Bachelor's Degree", "Master's Degree", "Some College", "Associate's Degree", "High School", "Other"].index( - edu.get("education_level", "Some College") - ) if edu.get("education_level") in ["Bachelor's Degree", "Master's Degree", "Some College", "Associate's Degree", "High School", "Other"] else 2, - key=f"edu_level_{i}") - end = st.text_input("Completion Year", edu.get("year_of_completion", ""), key=f"edu_end_{i}") - updated_edu.append({ - "education_level": level, "institution": inst, "field_of_study": field_study, - "start_date": start, "year_of_completion": end, "final_evaluation_grade": "", "exam": {}, - }) - st.divider() - -# ── Experience ───────────────────────────────────────────────────────────── -with st.expander("💼 Work Experience"): - exp_list = data.get("experience_details", [{}]) - if "exp_count" not in st.session_state: - st.session_state.exp_count = len(exp_list) - if st.button("+ Add Experience Entry"): - st.session_state.exp_count += 1 - exp_list.append({}) - - updated_exp = [] - for i in range(st.session_state.exp_count): - exp = exp_list[i] if i < len(exp_list) else {} - st.markdown(f"**Position {i+1}**") - col1, col2 = st.columns(2) - with col1: - pos = field("Job Title", exp.get("position", ""), f"exp_pos_{i}") - company = field("Company", exp.get("company", ""), f"exp_co_{i}") - period = field("Employment Period", exp.get("employment_period", ""), f"exp_period_{i}", - help="e.g. 01/2022 - Present") - with col2: - location = st.text_input("Location", exp.get("location", ""), key=f"exp_loc_{i}") - industry = st.text_input("Industry", exp.get("industry", ""), key=f"exp_ind_{i}") - - responsibilities = st.text_area( - "Key Responsibilities (one per line)", - value="\n".join( - r.get(f"responsibility_{j+1}", "") if isinstance(r, dict) else str(r) - for j, r in enumerate(exp.get("key_responsibilities", [])) - ), - key=f"exp_resp_{i}", height=100, - ) - skills = st.text_input( - "Skills (comma-separated)", - value=", ".join(exp.get("skills_acquired", [])), - key=f"exp_skills_{i}", - ) - resp_list = [{"responsibility_1": r.strip()} for r in responsibilities.splitlines() if r.strip()] - skill_list = [s.strip() for s in skills.split(",") if s.strip()] - updated_exp.append({ - "position": pos, "company": company, "employment_period": period, - "location": location, "industry": industry, - "key_responsibilities": resp_list, "skills_acquired": skill_list, - }) - st.divider() - -# ── Preferences ──────────────────────────────────────────────────────────── -with st.expander("⚙️ Preferences & Availability"): - wp = data.get("work_preferences", {}) - sal = data.get("salary_expectations", {}) - avail = data.get("availability", {}) - col1, col2 = st.columns(2) - with col1: - salary_range = st.text_input("Salary Range (USD)", sal.get("salary_range_usd", ""), key="pref_salary", - help="e.g. 120000 - 180000") - notice = st.text_input("Notice Period", avail.get("notice_period", "2 weeks"), key="pref_notice") - with col2: - remote_work = st.checkbox("Open to Remote", value=wp.get("remote_work", "Yes") == "Yes", key="pref_remote") - relocation = st.checkbox("Open to Relocation", value=wp.get("open_to_relocation", "No") == "Yes", key="pref_reloc") - assessments = st.checkbox("Willing to complete assessments", - value=wp.get("willing_to_complete_assessments", "Yes") == "Yes", key="pref_assess") - bg_checks = st.checkbox("Willing to undergo background checks", - value=wp.get("willing_to_undergo_background_checks", "Yes") == "Yes", key="pref_bg") - -# ── Self-ID ──────────────────────────────────────────────────────────────── -with st.expander("🏳️‍🌈 Self-Identification (optional)"): - sid = data.get("self_identification", {}) - col1, col2 = st.columns(2) - with col1: - gender = st.text_input("Gender identity", sid.get("gender", "Non-binary"), key="sid_gender", - help="Select 'Non-binary' or 'Prefer not to say' when options allow") - pronouns = st.text_input("Pronouns", sid.get("pronouns", "Any"), key="sid_pronouns") - ethnicity = field("Ethnicity", sid.get("ethnicity", ""), "sid_ethnicity", - help="'Prefer not to say' is always an option") - with col2: - veteran = st.selectbox("Veteran status", ["No", "Yes", "Prefer not to say"], - index=["No", "Yes", "Prefer not to say"].index(sid.get("veteran", "No")), key="sid_vet") - disability = st.selectbox("Disability disclosure", ["Prefer not to say", "No", "Yes"], - index=["Prefer not to say", "No", "Yes"].index( - sid.get("disability", "Prefer not to say")), key="sid_dis") - st.caption("⚠️ Drug testing: set to No (medicinal cannabis for EDS). AIHawk will skip employers who require drug tests.") - -st.divider() - -# ── Save ─────────────────────────────────────────────────────────────────── -if st.button("💾 Save Resume Profile", type="primary", use_container_width=True): - data["personal_information"] = { - **data.get("personal_information", {}), - "name": name, "surname": surname, "email": email, "phone": phone, - "city": city, "zip_code": zip_code, "linkedin": linkedin, "date_of_birth": dob, - } - data["education_details"] = updated_edu - data["experience_details"] = updated_exp - data["salary_expectations"] = {"salary_range_usd": salary_range} - data["availability"] = {"notice_period": notice} - data["work_preferences"] = { - **data.get("work_preferences", {}), - "remote_work": "Yes" if remote_work else "No", - "open_to_relocation": "Yes" if relocation else "No", - "willing_to_complete_assessments": "Yes" if assessments else "No", - "willing_to_undergo_background_checks": "Yes" if bg_checks else "No", - "willing_to_undergo_drug_tests": "No", - } - data["self_identification"] = { - "gender": gender, "pronouns": pronouns, "veteran": veteran, - "disability": disability, "ethnicity": ethnicity, - } - RESUME_PATH.write_text(yaml.dump(data, default_flow_style=False, allow_unicode=True)) - st.success("✅ Profile saved!") - st.balloons() -``` - -**Step 2: Smoke test** - -Navigate to Resume Editor in the Streamlit app. Confirm all sections render and `FILL_IN` fields show amber warnings. - -**Step 3: Commit** - -```bash -cd /devl/job-seeker -git add app/pages/3_Resume_Editor.py -git commit -m "feat: add Resume Editor page with form-based AIHawk YAML editor" -``` - ---- - -## Task 8: Wire up environment.yml and CLAUDE.md - -**Step 1: Export updated environment.yml** - -```bash -conda run -n job-seeker conda env export > /devl/job-seeker/environment.yml -``` - -**Step 2: Update CLAUDE.md with UI section** - -Add to `CLAUDE.md`: - -```markdown -## Web UI -- Run: `conda run -n job-seeker streamlit run app/Home.py` -- Opens at http://localhost:8501 -- staging.db is gitignored — SQLite staging layer between discovery and Notion -- Pages: Home (dashboard), Job Review, Settings, Resume Editor -``` - -**Step 3: Commit** - -```bash -cd /devl/job-seeker -git add environment.yml CLAUDE.md -git commit -m "chore: update environment.yml and CLAUDE.md for Streamlit UI" -``` - ---- - -## Quick Reference - -| Command | What it does | -|---|---| -| `conda run -n job-seeker streamlit run app/Home.py` | Launch the web UI at localhost:8501 | -| `conda run -n job-seeker python scripts/discover.py` | Scrape boards → SQLite staging | -| `conda run -n job-seeker python scripts/sync.py` | Push approved jobs → Notion | -| `conda run -n job-seeker pytest tests/ -v` | Run all tests | diff --git a/docs/plans/2026-02-21-background-tasks-design.md b/docs/plans/2026-02-21-background-tasks-design.md deleted file mode 100644 index 099055b..0000000 --- a/docs/plans/2026-02-21-background-tasks-design.md +++ /dev/null @@ -1,100 +0,0 @@ -# Background Task Processing — Design - -**Date:** 2026-02-21 -**Status:** Approved - -## Problem - -Cover letter generation (`4_Apply.py`) and company research (`6_Interview_Prep.py`) call LLM scripts synchronously inside `st.spinner()`. If the user navigates away during generation, Streamlit abandons the in-progress call and the result is lost. Both results are already persisted to SQLite on completion, so if the task kept running in the background the result would be available on return. - -## Solution Overview - -Python threading + SQLite task table. When a user clicks Generate, a daemon thread is spawned immediately and the task is recorded in a new `background_tasks` table. The thread writes results to the existing tables (`jobs.cover_letter`, `company_research`) and marks itself complete/failed. All pages share a sidebar indicator that auto-refreshes while tasks are active. Individual pages show task-level status inline. - -## SQLite Schema - -New table `background_tasks` added in `scripts/db.py`: - -```sql -CREATE TABLE IF NOT EXISTS background_tasks ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - task_type TEXT NOT NULL, -- "cover_letter" | "company_research" - job_id INTEGER NOT NULL, - status TEXT NOT NULL DEFAULT 'queued', -- queued | running | completed | failed - error TEXT, - created_at DATETIME DEFAULT (datetime('now')), - started_at DATETIME, - finished_at DATETIME -) -``` - -## Deduplication Rule - -Before inserting a new task, check for an existing `queued` or `running` row with the same `(task_type, job_id)`. If one exists, reject the submission (return the existing task's id). Different task types for the same job (e.g. cover letter + research) are allowed to run concurrently. Different jobs of the same type are allowed concurrently. - -## Components - -### `scripts/task_runner.py` (new) - -- `submit_task(db, task_type, job_id) -> int` — dedup check, insert row, spawn daemon thread, return task id -- `_run_task(db, task_id, task_type, job_id)` — thread body: mark running, call generator, save result, mark completed/failed -- `get_active_tasks(db) -> list[dict]` — all queued/running rows with job title+company joined -- `get_task_for_job(db, task_type, job_id) -> dict | None` — latest task row for a specific job+type - -### `scripts/db.py` (modified) - -- Add `init_background_tasks(conn)` called inside `init_db()` -- Add `insert_task`, `update_task_status`, `get_active_tasks`, `get_task_for_job` helpers - -### `app/app.py` (modified) - -- After `st.navigation()`, call `get_active_tasks()` and render sidebar indicator -- Use `st.fragment` with `time.sleep(3)` + `st.rerun(scope="fragment")` to poll while tasks are active -- Sidebar shows: `⏳ N task(s) running` count + per-task line (type + company name) -- Fragment polling stops when active task count reaches zero - -### `app/pages/4_Apply.py` (modified) - -- Generate button calls `submit_task(db, "cover_letter", job_id)` instead of running inline -- If a task is `queued`/`running` for the selected job, disable button and show inline status fragment (polls every 3s) -- On `completed`, load cover letter from `jobs` row (already saved by thread) -- On `failed`, show error message and re-enable button - -### `app/pages/6_Interview_Prep.py` (modified) - -- Generate/Refresh buttons call `submit_task(db, "company_research", job_id)` instead of running inline -- Same inline status fragment pattern as Apply page - -## Data Flow - -``` -User clicks Generate - → submit_task(db, type, job_id) - → dedup check (reject if already queued/running for same type+job) - → INSERT background_tasks row (status=queued) - → spawn daemon thread - → return task_id - → page shows inline "⏳ Queued…" fragment - -Thread runs - → UPDATE status=running, started_at=now - → call generate_cover_letter.generate() OR research_company() - → write result to jobs.cover_letter OR company_research table - → UPDATE status=completed, finished_at=now - (on exception: UPDATE status=failed, error=str(e)) - -Sidebar fragment (every 3s while active tasks > 0) - → get_active_tasks() → render count + list - → st.rerun(scope="fragment") - -Page fragment (every 3s while task for this job is running) - → get_task_for_job() → render status - → on completed: st.rerun() (full rerun to reload cover letter / research) -``` - -## What Is Not Changed - -- `generate_cover_letter.generate()` and `research_company()` are called unchanged from the thread -- `update_cover_letter()` and `save_research()` DB helpers are reused unchanged -- No new Python packages required -- No separate worker process — daemon threads die with the Streamlit server, but results already written to SQLite survive diff --git a/docs/plans/2026-02-21-background-tasks-plan.md b/docs/plans/2026-02-21-background-tasks-plan.md deleted file mode 100644 index 29a6b5e..0000000 --- a/docs/plans/2026-02-21-background-tasks-plan.md +++ /dev/null @@ -1,933 +0,0 @@ -# Background Task Processing Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Replace synchronous LLM calls in Apply and Interview Prep pages with background threads so cover letter and research generation survive page navigation. - -**Architecture:** A new `background_tasks` SQLite table tracks task state. `scripts/task_runner.py` spawns daemon threads that call existing generator functions and write results via existing DB helpers. The Streamlit sidebar polls active tasks every 3s via `@st.fragment(run_every=3)`; individual pages show per-job status with the same pattern. - -**Tech Stack:** Python `threading` (stdlib), SQLite, Streamlit `st.fragment` (≥1.33 — already installed) - ---- - -## Task 1: Add background_tasks table and DB helpers - -**Files:** -- Modify: `scripts/db.py` -- Test: `tests/test_db.py` - -### Step 1: Write the failing tests - -Add to `tests/test_db.py`: - -```python -# ── background_tasks tests ──────────────────────────────────────────────────── - -def test_init_db_creates_background_tasks_table(tmp_path): - """init_db creates a background_tasks table.""" - from scripts.db import init_db - db_path = tmp_path / "test.db" - init_db(db_path) - import sqlite3 - conn = sqlite3.connect(db_path) - cur = conn.execute( - "SELECT name FROM sqlite_master WHERE type='table' AND name='background_tasks'" - ) - assert cur.fetchone() is not None - conn.close() - - -def test_insert_task_returns_id_and_true(tmp_path): - """insert_task returns (task_id, True) for a new task.""" - from scripts.db import init_db, insert_job, insert_task - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - task_id, is_new = insert_task(db_path, "cover_letter", job_id) - assert isinstance(task_id, int) and task_id > 0 - assert is_new is True - - -def test_insert_task_deduplicates_active_task(tmp_path): - """insert_task returns (existing_id, False) if a queued/running task already exists.""" - from scripts.db import init_db, insert_job, insert_task - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - first_id, _ = insert_task(db_path, "cover_letter", job_id) - second_id, is_new = insert_task(db_path, "cover_letter", job_id) - assert second_id == first_id - assert is_new is False - - -def test_insert_task_allows_different_types_same_job(tmp_path): - """insert_task allows cover_letter and company_research for the same job concurrently.""" - from scripts.db import init_db, insert_job, insert_task - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - _, cl_new = insert_task(db_path, "cover_letter", job_id) - _, res_new = insert_task(db_path, "company_research", job_id) - assert cl_new is True - assert res_new is True - - -def test_update_task_status_running(tmp_path): - """update_task_status('running') sets started_at.""" - from scripts.db import init_db, insert_job, insert_task, update_task_status - import sqlite3 - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - task_id, _ = insert_task(db_path, "cover_letter", job_id) - update_task_status(db_path, task_id, "running") - conn = sqlite3.connect(db_path) - row = conn.execute("SELECT status, started_at FROM background_tasks WHERE id=?", (task_id,)).fetchone() - conn.close() - assert row[0] == "running" - assert row[1] is not None - - -def test_update_task_status_completed(tmp_path): - """update_task_status('completed') sets finished_at.""" - from scripts.db import init_db, insert_job, insert_task, update_task_status - import sqlite3 - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - task_id, _ = insert_task(db_path, "cover_letter", job_id) - update_task_status(db_path, task_id, "completed") - conn = sqlite3.connect(db_path) - row = conn.execute("SELECT status, finished_at FROM background_tasks WHERE id=?", (task_id,)).fetchone() - conn.close() - assert row[0] == "completed" - assert row[1] is not None - - -def test_update_task_status_failed_stores_error(tmp_path): - """update_task_status('failed') stores error message and sets finished_at.""" - from scripts.db import init_db, insert_job, insert_task, update_task_status - import sqlite3 - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - task_id, _ = insert_task(db_path, "cover_letter", job_id) - update_task_status(db_path, task_id, "failed", error="LLM timeout") - conn = sqlite3.connect(db_path) - row = conn.execute("SELECT status, error, finished_at FROM background_tasks WHERE id=?", (task_id,)).fetchone() - conn.close() - assert row[0] == "failed" - assert row[1] == "LLM timeout" - assert row[2] is not None - - -def test_get_active_tasks_returns_only_active(tmp_path): - """get_active_tasks returns only queued/running tasks with job info joined.""" - from scripts.db import init_db, insert_job, insert_task, update_task_status, get_active_tasks - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - active_id, _ = insert_task(db_path, "cover_letter", job_id) - done_id, _ = insert_task(db_path, "company_research", job_id) - update_task_status(db_path, done_id, "completed") - - tasks = get_active_tasks(db_path) - assert len(tasks) == 1 - assert tasks[0]["id"] == active_id - assert tasks[0]["company"] == "Acme" - assert tasks[0]["title"] == "CSM" - - -def test_get_task_for_job_returns_latest(tmp_path): - """get_task_for_job returns the most recent task for the given type+job.""" - from scripts.db import init_db, insert_job, insert_task, update_task_status, get_task_for_job - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - first_id, _ = insert_task(db_path, "cover_letter", job_id) - update_task_status(db_path, first_id, "completed") - second_id, _ = insert_task(db_path, "cover_letter", job_id) # allowed since first is done - - task = get_task_for_job(db_path, "cover_letter", job_id) - assert task is not None - assert task["id"] == second_id - - -def test_get_task_for_job_returns_none_when_absent(tmp_path): - """get_task_for_job returns None when no task exists for that job+type.""" - from scripts.db import init_db, insert_job, get_task_for_job - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-20", - }) - assert get_task_for_job(db_path, "cover_letter", job_id) is None -``` - -### Step 2: Run tests to verify they fail - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py -v -k "background_tasks or insert_task or update_task_status or get_active_tasks or get_task_for_job" -``` - -Expected: FAIL with `ImportError: cannot import name 'insert_task'` - -### Step 3: Implement in scripts/db.py - -Add the DDL constant after `CREATE_COMPANY_RESEARCH`: - -```python -CREATE_BACKGROUND_TASKS = """ -CREATE TABLE IF NOT EXISTS background_tasks ( - id INTEGER PRIMARY KEY AUTOINCREMENT, - task_type TEXT NOT NULL, - job_id INTEGER NOT NULL, - status TEXT NOT NULL DEFAULT 'queued', - error TEXT, - created_at DATETIME DEFAULT (datetime('now')), - started_at DATETIME, - finished_at DATETIME -) -""" -``` - -Add `conn.execute(CREATE_BACKGROUND_TASKS)` inside `init_db()`, after the existing three `conn.execute()` calls: - -```python -def init_db(db_path: Path = DEFAULT_DB) -> None: - """Create tables if they don't exist, then run migrations.""" - conn = sqlite3.connect(db_path) - conn.execute(CREATE_JOBS) - conn.execute(CREATE_JOB_CONTACTS) - conn.execute(CREATE_COMPANY_RESEARCH) - conn.execute(CREATE_BACKGROUND_TASKS) # ← add this line - conn.commit() - conn.close() - _migrate_db(db_path) -``` - -Add the four helper functions at the end of `scripts/db.py`: - -```python -# ── Background task helpers ─────────────────────────────────────────────────── - -def insert_task(db_path: Path = DEFAULT_DB, task_type: str = "", - job_id: int = None) -> tuple[int, bool]: - """Insert a new background task. - - Returns (task_id, True) if inserted, or (existing_id, False) if a - queued/running task for the same (task_type, job_id) already exists. - """ - conn = sqlite3.connect(db_path) - existing = conn.execute( - "SELECT id FROM background_tasks WHERE task_type=? AND job_id=? AND status IN ('queued','running')", - (task_type, job_id), - ).fetchone() - if existing: - conn.close() - return existing[0], False - cur = conn.execute( - "INSERT INTO background_tasks (task_type, job_id, status) VALUES (?, ?, 'queued')", - (task_type, job_id), - ) - task_id = cur.lastrowid - conn.commit() - conn.close() - return task_id, True - - -def update_task_status(db_path: Path = DEFAULT_DB, task_id: int = None, - status: str = "", error: Optional[str] = None) -> None: - """Update a task's status and set the appropriate timestamp.""" - now = datetime.now().isoformat()[:16] - conn = sqlite3.connect(db_path) - if status == "running": - conn.execute( - "UPDATE background_tasks SET status=?, started_at=? WHERE id=?", - (status, now, task_id), - ) - elif status in ("completed", "failed"): - conn.execute( - "UPDATE background_tasks SET status=?, finished_at=?, error=? WHERE id=?", - (status, now, error, task_id), - ) - else: - conn.execute("UPDATE background_tasks SET status=? WHERE id=?", (status, task_id)) - conn.commit() - conn.close() - - -def get_active_tasks(db_path: Path = DEFAULT_DB) -> list[dict]: - """Return all queued/running tasks with job title and company joined in.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - rows = conn.execute(""" - SELECT bt.*, j.title, j.company - FROM background_tasks bt - LEFT JOIN jobs j ON j.id = bt.job_id - WHERE bt.status IN ('queued', 'running') - ORDER BY bt.created_at ASC - """).fetchall() - conn.close() - return [dict(r) for r in rows] - - -def get_task_for_job(db_path: Path = DEFAULT_DB, task_type: str = "", - job_id: int = None) -> Optional[dict]: - """Return the most recent task row for a (task_type, job_id) pair, or None.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - row = conn.execute( - """SELECT * FROM background_tasks - WHERE task_type=? AND job_id=? - ORDER BY id DESC LIMIT 1""", - (task_type, job_id), - ).fetchone() - conn.close() - return dict(row) if row else None -``` - -### Step 4: Run tests to verify they pass - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py -v -k "background_tasks or insert_task or update_task_status or get_active_tasks or get_task_for_job" -``` - -Expected: all new tests PASS, no regressions - -### Step 5: Run full test suite - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` - -Expected: all tests PASS - -### Step 6: Commit - -```bash -git add scripts/db.py tests/test_db.py -git commit -m "feat: add background_tasks table and DB helpers" -``` - ---- - -## Task 2: Create scripts/task_runner.py - -**Files:** -- Create: `scripts/task_runner.py` -- Test: `tests/test_task_runner.py` - -### Step 1: Write the failing tests - -Create `tests/test_task_runner.py`: - -```python -import threading -import time -import pytest -from pathlib import Path -from unittest.mock import patch, MagicMock -import sqlite3 - - -def _make_db(tmp_path): - from scripts.db import init_db, insert_job - db = tmp_path / "test.db" - init_db(db) - job_id = insert_job(db, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "Great role.", "date_found": "2026-02-20", - }) - return db, job_id - - -def test_submit_task_returns_id_and_true(tmp_path): - """submit_task returns (task_id, True) and spawns a thread.""" - db, job_id = _make_db(tmp_path) - with patch("scripts.task_runner._run_task"): # don't actually call LLM - from scripts.task_runner import submit_task - task_id, is_new = submit_task(db, "cover_letter", job_id) - assert isinstance(task_id, int) and task_id > 0 - assert is_new is True - - -def test_submit_task_deduplicates(tmp_path): - """submit_task returns (existing_id, False) for a duplicate in-flight task.""" - db, job_id = _make_db(tmp_path) - with patch("scripts.task_runner._run_task"): - from scripts.task_runner import submit_task - first_id, _ = submit_task(db, "cover_letter", job_id) - second_id, is_new = submit_task(db, "cover_letter", job_id) - assert second_id == first_id - assert is_new is False - - -def test_run_task_cover_letter_success(tmp_path): - """_run_task marks running→completed and saves cover letter to DB.""" - db, job_id = _make_db(tmp_path) - from scripts.db import insert_task, get_task_for_job, get_jobs_by_status - task_id, _ = insert_task(db, "cover_letter", job_id) - - with patch("scripts.generate_cover_letter.generate", return_value="Dear Hiring Manager,\nGreat fit!"): - from scripts.task_runner import _run_task - _run_task(db, task_id, "cover_letter", job_id) - - task = get_task_for_job(db, "cover_letter", job_id) - assert task["status"] == "completed" - assert task["error"] is None - - conn = sqlite3.connect(db) - row = conn.execute("SELECT cover_letter FROM jobs WHERE id=?", (job_id,)).fetchone() - conn.close() - assert row[0] == "Dear Hiring Manager,\nGreat fit!" - - -def test_run_task_company_research_success(tmp_path): - """_run_task marks running→completed and saves research to DB.""" - db, job_id = _make_db(tmp_path) - from scripts.db import insert_task, get_task_for_job, get_research - - task_id, _ = insert_task(db, "company_research", job_id) - fake_result = { - "raw_output": "raw", "company_brief": "brief", - "ceo_brief": "ceo", "talking_points": "points", - } - with patch("scripts.company_research.research_company", return_value=fake_result): - from scripts.task_runner import _run_task - _run_task(db, task_id, "company_research", job_id) - - task = get_task_for_job(db, "company_research", job_id) - assert task["status"] == "completed" - - research = get_research(db, job_id=job_id) - assert research["company_brief"] == "brief" - - -def test_run_task_marks_failed_on_exception(tmp_path): - """_run_task marks status=failed and stores error when generator raises.""" - db, job_id = _make_db(tmp_path) - from scripts.db import insert_task, get_task_for_job - task_id, _ = insert_task(db, "cover_letter", job_id) - - with patch("scripts.generate_cover_letter.generate", side_effect=RuntimeError("LLM timeout")): - from scripts.task_runner import _run_task - _run_task(db, task_id, "cover_letter", job_id) - - task = get_task_for_job(db, "cover_letter", job_id) - assert task["status"] == "failed" - assert "LLM timeout" in task["error"] - - -def test_submit_task_actually_completes(tmp_path): - """Integration: submit_task spawns a thread that completes asynchronously.""" - db, job_id = _make_db(tmp_path) - from scripts.db import get_task_for_job - - with patch("scripts.generate_cover_letter.generate", return_value="Cover letter text"): - from scripts.task_runner import submit_task - task_id, _ = submit_task(db, "cover_letter", job_id) - # Wait for thread to complete (max 5s) - for _ in range(50): - task = get_task_for_job(db, "cover_letter", job_id) - if task and task["status"] in ("completed", "failed"): - break - time.sleep(0.1) - - task = get_task_for_job(db, "cover_letter", job_id) - assert task["status"] == "completed" -``` - -### Step 2: Run tests to verify they fail - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_task_runner.py -v -``` - -Expected: FAIL with `ModuleNotFoundError: No module named 'scripts.task_runner'` - -### Step 3: Implement scripts/task_runner.py - -Create `scripts/task_runner.py`: - -```python -# scripts/task_runner.py -""" -Background task runner for LLM generation tasks. - -Submitting a task inserts a row in background_tasks and spawns a daemon thread. -The thread calls the appropriate generator, writes results to existing tables, -and marks the task completed or failed. - -Deduplication: only one queued/running task per (task_type, job_id) is allowed. -Different task types for the same job run concurrently (e.g. cover letter + research). -""" -import sqlite3 -import threading -from pathlib import Path - -from scripts.db import ( - DEFAULT_DB, - insert_task, - update_task_status, - update_cover_letter, - save_research, -) - - -def submit_task(db_path: Path = DEFAULT_DB, task_type: str = "", - job_id: int = None) -> tuple[int, bool]: - """Submit a background LLM task. - - Returns (task_id, True) if a new task was queued and a thread spawned. - Returns (existing_id, False) if an identical task is already in-flight. - """ - task_id, is_new = insert_task(db_path, task_type, job_id) - if is_new: - t = threading.Thread( - target=_run_task, - args=(db_path, task_id, task_type, job_id), - daemon=True, - ) - t.start() - return task_id, is_new - - -def _run_task(db_path: Path, task_id: int, task_type: str, job_id: int) -> None: - """Thread body: run the generator and persist the result.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - row = conn.execute("SELECT * FROM jobs WHERE id=?", (job_id,)).fetchone() - conn.close() - if row is None: - update_task_status(db_path, task_id, "failed", error=f"Job {job_id} not found") - return - - job = dict(row) - update_task_status(db_path, task_id, "running") - - try: - if task_type == "cover_letter": - from scripts.generate_cover_letter import generate - result = generate( - job.get("title", ""), - job.get("company", ""), - job.get("description", ""), - ) - update_cover_letter(db_path, job_id, result) - - elif task_type == "company_research": - from scripts.company_research import research_company - result = research_company(job) - save_research(db_path, job_id=job_id, **result) - - else: - raise ValueError(f"Unknown task_type: {task_type!r}") - - update_task_status(db_path, task_id, "completed") - - except Exception as exc: - update_task_status(db_path, task_id, "failed", error=str(exc)) -``` - -### Step 4: Run tests to verify they pass - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_task_runner.py -v -``` - -Expected: all tests PASS - -### Step 5: Run full test suite - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` - -Expected: all tests PASS - -### Step 6: Commit - -```bash -git add scripts/task_runner.py tests/test_task_runner.py -git commit -m "feat: add task_runner — background thread executor for LLM tasks" -``` - ---- - -## Task 3: Add sidebar task indicator to app/app.py - -**Files:** -- Modify: `app/app.py` - -No new tests needed — this is pure UI wiring. - -### Step 1: Replace the contents of app/app.py - -Current file is 33 lines. Replace entirely with: - -```python -# app/app.py -""" -Streamlit entry point — uses st.navigation() to control the sidebar. -Main workflow pages are listed at the top; Settings is separated into -a "System" section so it doesn't crowd the navigation. - -Run: streamlit run app/app.py - bash scripts/manage-ui.sh start -""" -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).parent.parent)) - -import streamlit as st -from scripts.db import DEFAULT_DB, init_db, get_active_tasks - -st.set_page_config( - page_title="Job Seeker", - page_icon="💼", - layout="wide", -) - -init_db(DEFAULT_DB) - -# ── Background task sidebar indicator ───────────────────────────────────────── -@st.fragment(run_every=3) -def _task_sidebar() -> None: - tasks = get_active_tasks(DEFAULT_DB) - if not tasks: - return - with st.sidebar: - st.divider() - st.markdown(f"**⏳ {len(tasks)} task(s) running**") - for t in tasks: - icon = "⏳" if t["status"] == "running" else "🕐" - label = "Cover letter" if t["task_type"] == "cover_letter" else "Research" - st.caption(f"{icon} {label} — {t.get('company') or 'unknown'}") - -_task_sidebar() - -# ── Navigation ───────────────────────────────────────────────────────────────── -pages = { - "": [ - st.Page("Home.py", title="Home", icon="🏠"), - st.Page("pages/1_Job_Review.py", title="Job Review", icon="📋"), - st.Page("pages/4_Apply.py", title="Apply Workspace", icon="🚀"), - st.Page("pages/5_Interviews.py", title="Interviews", icon="🎯"), - st.Page("pages/6_Interview_Prep.py", title="Interview Prep", icon="📞"), - ], - "System": [ - st.Page("pages/2_Settings.py", title="Settings", icon="⚙️"), - ], -} - -pg = st.navigation(pages) -pg.run() -``` - -### Step 2: Smoke-test by running the UI - -```bash -bash /devl/job-seeker/scripts/manage-ui.sh restart -``` - -Navigate to http://localhost:8501 and confirm the app loads without error. The sidebar task indicator does not appear when no tasks are running (correct). - -### Step 3: Commit - -```bash -git add app/app.py -git commit -m "feat: sidebar background task indicator with 3s auto-refresh" -``` - ---- - -## Task 4: Update 4_Apply.py to use background generation - -**Files:** -- Modify: `app/pages/4_Apply.py` - -No new unit tests — covered by existing test suite for DB layer. Smoke-test in browser. - -### Step 1: Add imports at the top of 4_Apply.py - -After the existing imports block (after `from scripts.db import ...`), add: - -```python -from scripts.db import get_task_for_job -from scripts.task_runner import submit_task -``` - -So the full import block becomes: - -```python -from scripts.db import ( - DEFAULT_DB, init_db, get_jobs_by_status, - update_cover_letter, mark_applied, - get_task_for_job, -) -from scripts.task_runner import submit_task -``` - -### Step 2: Replace the Generate button section - -Find this block (around line 174–185): - -```python - if st.button("✨ Generate / Regenerate", use_container_width=True): - with st.spinner("Generating via LLM…"): - try: - from scripts.generate_cover_letter import generate as _gen - st.session_state[_cl_key] = _gen( - job.get("title", ""), - job.get("company", ""), - job.get("description", ""), - ) - st.rerun() - except Exception as e: - st.error(f"Generation failed: {e}") -``` - -Replace with: - -```python - _cl_task = get_task_for_job(DEFAULT_DB, "cover_letter", selected_id) - _cl_running = _cl_task and _cl_task["status"] in ("queued", "running") - - if st.button("✨ Generate / Regenerate", use_container_width=True, disabled=bool(_cl_running)): - submit_task(DEFAULT_DB, "cover_letter", selected_id) - st.rerun() - - if _cl_running: - @st.fragment(run_every=3) - def _cl_status_fragment(): - t = get_task_for_job(DEFAULT_DB, "cover_letter", selected_id) - if t and t["status"] in ("queued", "running"): - lbl = "Queued…" if t["status"] == "queued" else "Generating via LLM…" - st.info(f"⏳ {lbl}") - else: - st.rerun() # full page rerun — reloads cover letter from DB - _cl_status_fragment() - elif _cl_task and _cl_task["status"] == "failed": - st.error(f"Generation failed: {_cl_task.get('error', 'unknown error')}") -``` - -Also update the session-state initialiser just below (line 171–172) so it loads from DB after background completion. The existing code already does this correctly: - -```python - if _cl_key not in st.session_state: - st.session_state[_cl_key] = job.get("cover_letter") or "" -``` - -This is fine — `job` is fetched fresh on each full-page rerun, so when the background thread writes to `jobs.cover_letter`, the next full rerun picks it up. - -### Step 3: Smoke-test in browser - -1. Navigate to Apply Workspace -2. Select an approved job -3. Click "Generate / Regenerate" -4. Navigate away to Home -5. Navigate back to Apply Workspace for the same job -6. Observe: button is disabled and "⏳ Generating via LLM…" shows while running; cover letter appears when done - -### Step 4: Commit - -```bash -git add app/pages/4_Apply.py -git commit -m "feat: cover letter generation runs in background, survives navigation" -``` - ---- - -## Task 5: Update 6_Interview_Prep.py to use background research - -**Files:** -- Modify: `app/pages/6_Interview_Prep.py` - -### Step 1: Add imports at the top of 6_Interview_Prep.py - -After the existing `from scripts.db import (...)` block, add: - -```python -from scripts.db import get_task_for_job -from scripts.task_runner import submit_task -``` - -So the full import block becomes: - -```python -from scripts.db import ( - DEFAULT_DB, init_db, - get_interview_jobs, get_contacts, get_research, - save_research, get_task_for_job, -) -from scripts.task_runner import submit_task -``` - -### Step 2: Replace the "no research yet" generate button block - -Find this block (around line 99–111): - -```python - if not research: - st.warning("No research brief yet for this job.") - if st.button("🔬 Generate research brief", type="primary", use_container_width=True): - with st.spinner("Generating… this may take 30–60 seconds"): - try: - from scripts.company_research import research_company - result = research_company(job) - save_research(DEFAULT_DB, job_id=selected_id, **result) - st.success("Done!") - st.rerun() - except Exception as e: - st.error(f"Error: {e}") - st.stop() - else: -``` - -Replace with: - -```python - _res_task = get_task_for_job(DEFAULT_DB, "company_research", selected_id) - _res_running = _res_task and _res_task["status"] in ("queued", "running") - - if not research: - if not _res_running: - st.warning("No research brief yet for this job.") - if _res_task and _res_task["status"] == "failed": - st.error(f"Last attempt failed: {_res_task.get('error', '')}") - if st.button("🔬 Generate research brief", type="primary", use_container_width=True): - submit_task(DEFAULT_DB, "company_research", selected_id) - st.rerun() - - if _res_running: - @st.fragment(run_every=3) - def _res_status_initial(): - t = get_task_for_job(DEFAULT_DB, "company_research", selected_id) - if t and t["status"] in ("queued", "running"): - lbl = "Queued…" if t["status"] == "queued" else "Generating… this may take 30–60 seconds" - st.info(f"⏳ {lbl}") - else: - st.rerun() - _res_status_initial() - - st.stop() - else: -``` - -### Step 3: Replace the "refresh" button block - -Find this block (around line 113–124): - -```python - generated_at = research.get("generated_at", "") - col_ts, col_btn = st.columns([3, 1]) - col_ts.caption(f"Research generated: {generated_at}") - if col_btn.button("🔄 Refresh", use_container_width=True): - with st.spinner("Refreshing…"): - try: - from scripts.company_research import research_company - result = research_company(job) - save_research(DEFAULT_DB, job_id=selected_id, **result) - st.rerun() - except Exception as e: - st.error(f"Error: {e}") -``` - -Replace with: - -```python - generated_at = research.get("generated_at", "") - col_ts, col_btn = st.columns([3, 1]) - col_ts.caption(f"Research generated: {generated_at}") - if col_btn.button("🔄 Refresh", use_container_width=True, disabled=bool(_res_running)): - submit_task(DEFAULT_DB, "company_research", selected_id) - st.rerun() - - if _res_running: - @st.fragment(run_every=3) - def _res_status_refresh(): - t = get_task_for_job(DEFAULT_DB, "company_research", selected_id) - if t and t["status"] in ("queued", "running"): - lbl = "Queued…" if t["status"] == "queued" else "Refreshing research…" - st.info(f"⏳ {lbl}") - else: - st.rerun() - _res_status_refresh() - elif _res_task and _res_task["status"] == "failed": - st.error(f"Refresh failed: {_res_task.get('error', '')}") -``` - -### Step 4: Smoke-test in browser - -1. Move a job to Phone Screen on the Interviews page -2. Navigate to Interview Prep, select that job -3. Click "Generate research brief" -4. Navigate away to Home -5. Navigate back — observe "⏳ Generating…" inline indicator -6. Wait for completion — research sections populate automatically - -### Step 5: Run full test suite one final time - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` - -Expected: all tests PASS - -### Step 6: Commit - -```bash -git add app/pages/6_Interview_Prep.py -git commit -m "feat: company research generation runs in background, survives navigation" -``` - ---- - -## Summary of Changes - -| File | Change | -|------|--------| -| `scripts/db.py` | Add `CREATE_BACKGROUND_TASKS`, `init_db` call, 4 new helpers | -| `scripts/task_runner.py` | New file — `submit_task` + `_run_task` thread body | -| `app/app.py` | Add `_task_sidebar` fragment with 3s auto-refresh | -| `app/pages/4_Apply.py` | Generate button → `submit_task`; inline status fragment | -| `app/pages/6_Interview_Prep.py` | Generate/Refresh buttons → `submit_task`; inline status fragments | -| `tests/test_db.py` | 9 new tests for background_tasks helpers | -| `tests/test_task_runner.py` | New file — 6 tests for task_runner | diff --git a/docs/plans/2026-02-21-email-handling-design.md b/docs/plans/2026-02-21-email-handling-design.md deleted file mode 100644 index cb570c8..0000000 --- a/docs/plans/2026-02-21-email-handling-design.md +++ /dev/null @@ -1,91 +0,0 @@ -# Email Handling Design - -**Date:** 2026-02-21 -**Status:** Approved - -## Problem - -IMAP sync already pulls emails for active pipeline jobs, but two gaps exist: -1. Inbound emails suggesting a stage change (e.g. "let's schedule a call") produce no signal — the recruiter's message just sits in the email log. -2. Recruiter outreach to email addresses not yet in the pipeline is invisible — those leads never enter Job Review. - -## Goals - -- Surface stage-change suggestions inline on the Interviews kanban card (suggest-only, never auto-advance). -- Capture recruiter leads from unmatched inbound email and surface them in Job Review. -- Make email sync a background task triggerable from the UI (Home page + Interviews sidebar). - -## Data Model - -**No new tables.** Two columns added to `job_contacts`: - -```sql -ALTER TABLE job_contacts ADD COLUMN stage_signal TEXT; -ALTER TABLE job_contacts ADD COLUMN suggestion_dismissed INTEGER DEFAULT 0; -``` - -- `stage_signal` — one of: `interview_scheduled`, `offer_received`, `rejected`, `positive_response`, `neutral` (or NULL if not yet classified). -- `suggestion_dismissed` — 1 when the user clicks Dismiss; prevents the banner re-appearing. - -Email leads reuse the existing `jobs` table with `source = 'email'` and `status = 'pending'`. No new columns needed. - -## Components - -### 1. Stage Signal Classification (`scripts/imap_sync.py`) - -After saving each **inbound** contact row, call `phi3:mini` via Ollama to classify the email into one of the five labels. Store the result in `stage_signal`. If classification fails, default to `NULL` (no suggestion shown). - -**Model:** `phi3:mini` via `LLMRouter.complete(model_override="phi3:mini", fallback_order=["ollama_research"])`. -Benchmarked at 100% accuracy / 3.0 s per email on a 12-case test suite. Runner-up Qwen2.5-3B untested but phi3-mini is the safe choice. - -### 2. Recruiter Lead Extraction (`scripts/imap_sync.py`) - -A second pass after per-job sync: scan INBOX broadly for recruitment-keyword emails that don't match any known pipeline company. For each unmatched email, call **Nemotron 1.5B** (already in use for company research) to extract `{company, title}`. If extraction returns a company name not already in the DB, insert a new job row `source='email', status='pending'`. - -**Dedup:** checked by `message_id` against all known contacts (cross-job), plus `url` uniqueness on the jobs table (the email lead URL is set to a synthetic `email:///` value). - -### 3. Background Task (`scripts/task_runner.py`) - -New task type: `email_sync` with `job_id = 0`. -`submit_task(db, "email_sync", 0)` → daemon thread → `sync_all()` → returns summary via task `error` field. - -Deduplication: only one `email_sync` can be queued/running at a time (existing insert_task logic handles this). - -### 4. UI — Sync Button (Home + Interviews) - -**Home.py:** New "Sync Emails" section alongside Find Jobs / Score / Notion sync. -**5_Interviews.py:** Existing sync button already present in sidebar; convert from synchronous `sync_all()` call to `submit_task()` + fragment polling. - -### 5. UI — Email Leads (Job Review) - -When `show_status == "pending"`, prepend email leads (`source = 'email'`) at the top of the list with a distinct `📧 Email Lead` badge. Actions are identical to scraped pending jobs (Approve / Reject). - -### 6. UI — Stage Suggestion Banner (Interviews Kanban) - -Inside `_render_card()`, before the advance/reject buttons, check for unseen stage signals: - -``` -💡 Email suggests: interview_scheduled -From: sarah@company.com · "Let's book a call" -[→ Move to Phone Screen] [Dismiss] -``` - -- "Move" calls `advance_to_stage()` + `submit_task("company_research")` then reruns. -- "Dismiss" calls `dismiss_stage_signal(contact_id)` then reruns. -- Only the most recent undismissed signal is shown per card. - -## Error Handling - -| Failure | Behaviour | -|---------|-----------| -| IMAP connection fails | Error stored in task `error` field; shown as warning in UI after sync | -| Classifier call fails | `stage_signal` left NULL; no suggestion shown; sync continues | -| Lead extractor fails | Email skipped; appended to `result["errors"]`; sync continues | -| Duplicate `email_sync` task | `insert_task` returns existing id; no new thread spawned | -| LLM extraction returns no company | Email silently skipped (not a lead) | - -## Out of Scope - -- Auto-advancing pipeline stage (suggest only). -- Sending email replies from the app (draft helper already exists). -- OAuth / token-refresh IMAP (config/email.yaml credentials only). diff --git a/docs/plans/2026-02-21-email-handling-plan.md b/docs/plans/2026-02-21-email-handling-plan.md deleted file mode 100644 index ac75aa5..0000000 --- a/docs/plans/2026-02-21-email-handling-plan.md +++ /dev/null @@ -1,1105 +0,0 @@ -# Email Handling Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Add stage-signal classification to inbound emails, recruiter lead capture from unmatched emails, email sync as a background task, and surface both in the UI. - -**Architecture:** Extend `imap_sync.py` with a phi3-mini classifier and Nemotron lead extractor; wire `email_sync` into `task_runner.py`; add two new DB helpers and two migration columns; update three UI pages. - -**Tech Stack:** Python, SQLite, imaplib, LLMRouter (Ollama phi3:mini + Nemotron 1.5B), Streamlit. - -**Run tests:** `/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v` -**Conda prefix:** `conda run -n job-seeker` - ---- - -### Task 1: DB migrations — stage_signal + suggestion_dismissed columns - -**Files:** -- Modify: `scripts/db.py` -- Test: `tests/test_db.py` - -**Context:** `_CONTACT_MIGRATIONS` is a list of `(col, type)` tuples applied in `_migrate_db()`. Add to that list. Also add two helper functions: `get_unread_stage_signals(db_path, job_id)` returns contacts with a non-null, non-neutral stage_signal and `suggestion_dismissed = 0`; `dismiss_stage_signal(db_path, contact_id)` sets `suggestion_dismissed = 1`. Also update `add_contact()` to accept an optional `stage_signal` kwarg. - -**Step 1: Write the failing tests** - -In `tests/test_db.py`, append: - -```python -def test_stage_signal_columns_exist(tmp_path): - """init_db creates stage_signal and suggestion_dismissed columns on job_contacts.""" - from scripts.db import init_db - db_path = tmp_path / "test.db" - init_db(db_path) - conn = sqlite3.connect(db_path) - cols = {row[1] for row in conn.execute("PRAGMA table_info(job_contacts)").fetchall()} - conn.close() - assert "stage_signal" in cols - assert "suggestion_dismissed" in cols - - -def test_add_contact_with_stage_signal(tmp_path): - """add_contact stores stage_signal when provided.""" - from scripts.db import init_db, insert_job, add_contact, get_contacts - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-21", - }) - add_contact(db_path, job_id=job_id, direction="inbound", - subject="Interview invite", stage_signal="interview_scheduled") - contacts = get_contacts(db_path, job_id=job_id) - assert contacts[0]["stage_signal"] == "interview_scheduled" - - -def test_get_unread_stage_signals(tmp_path): - """get_unread_stage_signals returns only non-neutral, non-dismissed signals.""" - from scripts.db import (init_db, insert_job, add_contact, - get_unread_stage_signals, dismiss_stage_signal) - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-21", - }) - c1 = add_contact(db_path, job_id=job_id, direction="inbound", - subject="Interview invite", stage_signal="interview_scheduled") - add_contact(db_path, job_id=job_id, direction="inbound", - subject="Auto-confirm", stage_signal="neutral") - signals = get_unread_stage_signals(db_path, job_id) - assert len(signals) == 1 - assert signals[0]["stage_signal"] == "interview_scheduled" - - dismiss_stage_signal(db_path, c1) - assert get_unread_stage_signals(db_path, job_id) == [] -``` - -**Step 2: Run tests to confirm they fail** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py::test_stage_signal_columns_exist tests/test_db.py::test_add_contact_with_stage_signal tests/test_db.py::test_get_unread_stage_signals -v -``` - -Expected: 3 failures. - -**Step 3: Implement in `scripts/db.py`** - -3a. In `_CONTACT_MIGRATIONS`, add: -```python -_CONTACT_MIGRATIONS = [ - ("message_id", "TEXT"), - ("stage_signal", "TEXT"), - ("suggestion_dismissed", "INTEGER DEFAULT 0"), -] -``` - -3b. Update `add_contact()` signature and INSERT: -```python -def add_contact(db_path: Path = DEFAULT_DB, job_id: int = None, - direction: str = "inbound", subject: str = "", - from_addr: str = "", to_addr: str = "", - body: str = "", received_at: str = "", - message_id: str = "", - stage_signal: str = "") -> int: - """Log an email contact. Returns the new row id.""" - ts = received_at or datetime.now().isoformat()[:16] - conn = sqlite3.connect(db_path) - cur = conn.execute( - """INSERT INTO job_contacts - (job_id, direction, subject, from_addr, to_addr, body, - received_at, message_id, stage_signal) - VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)""", - (job_id, direction, subject, from_addr, to_addr, body, - ts, message_id, stage_signal or None), - ) - conn.commit() - row_id = cur.lastrowid - conn.close() - return row_id -``` - -3c. Add the two new helpers after `get_contacts()`: -```python -def get_unread_stage_signals(db_path: Path = DEFAULT_DB, - job_id: int = None) -> list[dict]: - """Return inbound contacts with a non-neutral, non-dismissed stage signal.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - rows = conn.execute( - """SELECT * FROM job_contacts - WHERE job_id = ? - AND direction = 'inbound' - AND stage_signal IS NOT NULL - AND stage_signal != 'neutral' - AND (suggestion_dismissed IS NULL OR suggestion_dismissed = 0) - ORDER BY received_at ASC""", - (job_id,), - ).fetchall() - conn.close() - return [dict(r) for r in rows] - - -def dismiss_stage_signal(db_path: Path = DEFAULT_DB, - contact_id: int = None) -> None: - """Mark a stage signal suggestion as dismissed.""" - conn = sqlite3.connect(db_path) - conn.execute( - "UPDATE job_contacts SET suggestion_dismissed = 1 WHERE id = ?", - (contact_id,), - ) - conn.commit() - conn.close() -``` - -3d. Add `get_all_message_ids()` (needed for lead dedup in Task 3): -```python -def get_all_message_ids(db_path: Path = DEFAULT_DB) -> set[str]: - """Return all known Message-IDs across all job contacts.""" - conn = sqlite3.connect(db_path) - rows = conn.execute( - "SELECT message_id FROM job_contacts WHERE message_id IS NOT NULL AND message_id != ''" - ).fetchall() - conn.close() - return {r[0] for r in rows} -``` - -**Step 4: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py -v -``` - -Expected: all pass. - -**Step 5: Commit** - -```bash -git add scripts/db.py tests/test_db.py -git commit -m "feat: add stage_signal/suggestion_dismissed columns and helpers to db" -``` - ---- - -### Task 2: Stage signal classifier in imap_sync.py - -**Files:** -- Modify: `scripts/imap_sync.py` -- Test: `tests/test_imap_sync.py` (create) - -**Context:** Add a `classify_stage_signal(subject, body)` function that calls phi3:mini via LLMRouter and returns one of the 5 label strings. It must gracefully return `None` on any failure (network, timeout, model not loaded). The label parsing must strip `` tags in case a thinking-capable model is used. - -**Step 1: Write the failing test** - -Create `tests/test_imap_sync.py`: - -```python -"""Tests for imap_sync helpers (no live IMAP connection required).""" -import pytest -from unittest.mock import patch - - -def test_classify_stage_signal_interview(tmp_path): - """classify_stage_signal returns interview_scheduled for a call-scheduling email.""" - from scripts.imap_sync import classify_stage_signal - with patch("scripts.imap_sync._CLASSIFIER_ROUTER") as mock_router: - mock_router.complete.return_value = "interview_scheduled" - result = classify_stage_signal( - "Let's schedule a call", - "Hi Alex, we'd love to book a 30-min phone screen with you.", - ) - assert result == "interview_scheduled" - - -def test_classify_stage_signal_returns_none_on_error(tmp_path): - """classify_stage_signal returns None when LLM call raises.""" - from scripts.imap_sync import classify_stage_signal - with patch("scripts.imap_sync._CLASSIFIER_ROUTER") as mock_router: - mock_router.complete.side_effect = RuntimeError("model not loaded") - result = classify_stage_signal("subject", "body") - assert result is None - - -def test_classify_stage_signal_strips_think_tags(tmp_path): - """classify_stage_signal strips … blocks before parsing.""" - from scripts.imap_sync import classify_stage_signal - with patch("scripts.imap_sync._CLASSIFIER_ROUTER") as mock_router: - mock_router.complete.return_value = "Let me think…\nrejected" - result = classify_stage_signal("Update on your application", "We went with another candidate.") - assert result == "rejected" - - -def test_normalise_company(): - """_normalise_company strips legal suffixes.""" - from scripts.imap_sync import _normalise_company - assert _normalise_company("DataStax, Inc.") == "DataStax" - assert _normalise_company("Wiz Ltd") == "Wiz" - assert _normalise_company("Crusoe Energy") == "Crusoe Energy" - - -def test_has_recruitment_keyword(): - """_has_recruitment_keyword matches known keywords.""" - from scripts.imap_sync import _has_recruitment_keyword - assert _has_recruitment_keyword("Interview Invitation — Senior TAM") - assert _has_recruitment_keyword("Your application with DataStax") - assert not _has_recruitment_keyword("Team lunch tomorrow") -``` - -**Step 2: Run to confirm failures** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py -v -``` - -Expected: ImportError or failures on `classify_stage_signal` and `_CLASSIFIER_ROUTER`. - -**Step 3: Implement in `scripts/imap_sync.py`** - -After the existing imports, add: - -```python -import re as _re - -from scripts.llm_router import LLMRouter - -_CLASSIFIER_ROUTER = LLMRouter() - -_CLASSIFY_SYSTEM = ( - "You are an email classifier. Classify the recruitment email into exactly ONE of these categories:\n" - " interview_scheduled, offer_received, rejected, positive_response, neutral\n\n" - "Rules:\n" - "- interview_scheduled: recruiter wants to book a call/interview\n" - "- offer_received: job offer is being extended\n" - "- rejected: explicitly not moving forward\n" - "- positive_response: interested/impressed but no interview booked yet\n" - "- neutral: auto-confirmation, generic update, no clear signal\n\n" - "Respond with ONLY the category name. No explanation." -) - -_CLASSIFY_LABELS = [ - "interview_scheduled", "offer_received", "rejected", - "positive_response", "neutral", -] - - -def classify_stage_signal(subject: str, body: str) -> Optional[str]: - """Classify an inbound email into a pipeline stage signal. - - Returns one of the 5 label strings, or None on failure. - Uses phi3:mini via Ollama (benchmarked 100% on 12-case test set). - """ - try: - prompt = f"Subject: {subject}\n\nEmail: {body[:400]}" - raw = _CLASSIFIER_ROUTER.complete( - prompt, - system=_CLASSIFY_SYSTEM, - model_override="phi3:mini", - fallback_order=["ollama_research"], - ) - # Strip blocks (in case a reasoning model slips through) - text = _re.sub(r".*?", "", raw, flags=_re.DOTALL) - text = text.lower().strip() - for label in _CLASSIFY_LABELS: - if text.startswith(label) or label in text: - return label - return "neutral" - except Exception: - return None -``` - -**Step 4: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py -v -``` - -Expected: all 5 pass. - -**Step 5: Commit** - -```bash -git add scripts/imap_sync.py tests/test_imap_sync.py -git commit -m "feat: add classify_stage_signal to imap_sync using phi3:mini" -``` - ---- - -### Task 3: Classify inbound contacts during per-job sync - -**Files:** -- Modify: `scripts/imap_sync.py` -- Test: `tests/test_imap_sync.py` - -**Context:** Inside `sync_job_emails()`, after calling `add_contact()` for an inbound email, call `classify_stage_signal()` and — if the result is non-None and non-'neutral' — update the `stage_signal` column via a direct SQLite update (no new db.py helper needed; avoid round-tripping through `add_contact`). The `contact_id` is already returned by `add_contact()`. - -We need a tiny helper `_update_contact_signal(db_path, contact_id, signal)` locally in imap_sync.py. Do NOT add this to db.py — it's only used here. - -**Step 1: Add test** - -Append to `tests/test_imap_sync.py`: - -```python -def test_sync_job_emails_classifies_inbound(tmp_path): - """sync_job_emails classifies inbound emails and stores the stage_signal.""" - from scripts.db import init_db, insert_job, get_contacts - from scripts.imap_sync import sync_job_emails - - db_path = tmp_path / "test.db" - init_db(db_path) - job_id = insert_job(db_path, { - "title": "CSM", "company": "Acme", - "url": "https://acme.com/jobs/1", - "source": "linkedin", "location": "Remote", - "is_remote": True, "salary": "", "description": "", - "date_found": "2026-02-21", - }) - job = {"id": job_id, "company": "Acme", "url": "https://acme.com/jobs/1"} - - # Fake IMAP connection + one inbound email - from unittest.mock import MagicMock, patch - - fake_msg_bytes = ( - b"From: recruiter@acme.com\r\n" - b"To: alex@example.com\r\n" - b"Subject: Interview Invitation\r\n" - b"Message-ID: \r\n" - b"\r\n" - b"Hi Alex, we'd like to schedule a phone screen." - ) - - conn_mock = MagicMock() - conn_mock.select.return_value = ("OK", [b"1"]) - conn_mock.search.return_value = ("OK", [b"1"]) - conn_mock.fetch.return_value = ("OK", [(b"1 (RFC822 {123})", fake_msg_bytes)]) - - with patch("scripts.imap_sync.classify_stage_signal", return_value="interview_scheduled"): - inb, out = sync_job_emails(job, conn_mock, {"lookback_days": 90}, db_path) - - assert inb == 1 - contacts = get_contacts(db_path, job_id=job_id) - assert contacts[0]["stage_signal"] == "interview_scheduled" -``` - -**Step 2: Run to confirm failure** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py::test_sync_job_emails_classifies_inbound -v -``` - -Expected: FAIL (stage_signal is None). - -**Step 3: Update `sync_job_emails()` in `scripts/imap_sync.py`** - -Add the private helper just before `sync_job_emails`: - -```python -def _update_contact_signal(db_path: Path, contact_id: int, signal: str) -> None: - """Write a stage signal onto an existing contact row.""" - import sqlite3 as _sqlite3 - conn = _sqlite3.connect(db_path) - conn.execute( - "UPDATE job_contacts SET stage_signal = ? WHERE id = ?", - (signal, contact_id), - ) - conn.commit() - conn.close() -``` - -In the INBOX loop inside `sync_job_emails()`, after the `add_contact(...)` call, add: - -```python -signal = classify_stage_signal(parsed["subject"], parsed["body"]) -if signal and signal != "neutral": - _update_contact_signal(db_path, contact_id, signal) -``` - -Note: `add_contact()` already returns the `row_id` (the contact_id). Make sure to capture it: - -```python -contact_id = add_contact( - db_path, job_id=job["id"], direction="inbound", - ... -) -signal = classify_stage_signal(parsed["subject"], parsed["body"]) -if signal and signal != "neutral": - _update_contact_signal(db_path, contact_id, signal) -``` - -**Step 4: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py -v -``` - -Expected: all pass. - -**Step 5: Commit** - -```bash -git add scripts/imap_sync.py tests/test_imap_sync.py -git commit -m "feat: classify stage signals for inbound emails during per-job sync" -``` - ---- - -### Task 4: Recruiter lead extractor + unmatched email handling - -**Files:** -- Modify: `scripts/imap_sync.py` -- Modify: `scripts/db.py` -- Test: `tests/test_imap_sync.py` - -**Context:** After per-job sync, do a second pass to find inbound recruitment emails NOT matched to any existing pipeline company. For each, call Nemotron to extract company + job title. If extraction succeeds and company isn't already in the DB, insert a new job (`source='email', status='pending'`). Use a synthetic URL `email:///` to satisfy the UNIQUE constraint on `jobs.url`. - -`sync_all()` return dict gains a `new_leads` key. - -**Step 1: Add test** - -Append to `tests/test_imap_sync.py`: - -```python -def test_extract_lead_info_returns_company_and_title(): - """extract_lead_info parses LLM JSON response into (company, title).""" - from scripts.imap_sync import extract_lead_info - with patch("scripts.imap_sync._CLASSIFIER_ROUTER") as mock_router: - mock_router.complete.return_value = '{"company": "Wiz", "title": "Senior TAM"}' - result = extract_lead_info("Senior TAM at Wiz", "Hi Alex, we have a role…", "recruiter@wiz.com") - assert result == ("Wiz", "Senior TAM") - - -def test_extract_lead_info_returns_none_on_bad_json(): - """extract_lead_info returns (None, None) when LLM returns unparseable output.""" - from scripts.imap_sync import extract_lead_info - with patch("scripts.imap_sync._CLASSIFIER_ROUTER") as mock_router: - mock_router.complete.return_value = "I cannot determine the company." - result = extract_lead_info("Job opportunity", "blah", "noreply@example.com") - assert result == (None, None) -``` - -**Step 2: Run to confirm failures** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py::test_extract_lead_info_returns_company_and_title tests/test_imap_sync.py::test_extract_lead_info_returns_none_on_bad_json -v -``` - -Expected: 2 failures. - -**Step 3: Implement `extract_lead_info()` in `scripts/imap_sync.py`** - -Add after `classify_stage_signal()`: - -```python -_EXTRACT_SYSTEM = ( - "Extract the hiring company name and job title from this recruitment email. " - "Respond with ONLY valid JSON in this exact format: " - '{\"company\": \"Company Name\", \"title\": \"Job Title\"}. ' - "If you cannot determine the company, respond: " - '{\"company\": null, \"title\": null}.' -) - - -def extract_lead_info(subject: str, body: str, - from_addr: str) -> tuple[Optional[str], Optional[str]]: - """Use Nemotron to extract (company, title) from an unmatched recruitment email. - - Returns (company, title) or (None, None) on failure / low confidence. - """ - import json as _json - try: - prompt = ( - f"From: {from_addr}\n" - f"Subject: {subject}\n\n" - f"Email excerpt:\n{body[:600]}" - ) - raw = _CLASSIFIER_ROUTER.complete( - prompt, - system=_EXTRACT_SYSTEM, - fallback_order=["ollama_research"], - ) - # Strip blocks - text = _re.sub(r".*?", "", raw, flags=_re.DOTALL).strip() - # Find first JSON object in response - m = _re.search(r'\{.*\}', text, _re.DOTALL) - if not m: - return None, None - data = _json.loads(m.group()) - company = data.get("company") or None - title = data.get("title") or None - return company, title - except Exception: - return None, None -``` - -**Step 4: Implement `_scan_unmatched_leads()` in `scripts/imap_sync.py`** - -Add this function. It uses the existing IMAP connection after per-job sync: - -```python -def _scan_unmatched_leads(conn: imaplib.IMAP4, cfg: dict, - db_path: Path, - known_message_ids: set[str]) -> int: - """Scan INBOX for recruitment emails not matched to any pipeline job. - - Calls LLM to extract company/title; inserts qualifying emails as email leads. - Returns the count of new leads inserted. - """ - from scripts.db import get_existing_urls, insert_job, add_contact - - lookback = int(cfg.get("lookback_days", 90)) - since = (datetime.now() - timedelta(days=lookback)).strftime("%d-%b-%Y") - - # Broad search — subject matches common recruiter terms - broad_terms = ["interview", "opportunity", "offer", "application", "role"] - all_uids: set[bytes] = set() - for term in broad_terms: - uids = _search_folder(conn, "INBOX", f'(SUBJECT "{term}")', since) - all_uids.update(uids) - - existing_urls = get_existing_urls(db_path) - new_leads = 0 - - for uid in all_uids: - parsed = _parse_message(conn, uid) - if not parsed: - continue - mid = parsed["message_id"] - if mid in known_message_ids: - continue # already synced to some job - if not _has_recruitment_keyword(parsed["subject"]): - continue # false positive from broad search - - company, title = extract_lead_info( - parsed["subject"], parsed["body"], parsed["from_addr"] - ) - if not company: - continue - - # Build a synthetic URL for dedup - from_domain = _extract_domain(parsed["from_addr"]) or "unknown" - mid_hash = str(abs(hash(mid)))[:10] - synthetic_url = f"email://{from_domain}/{mid_hash}" - - if synthetic_url in existing_urls: - continue # already captured this lead - - job_id = insert_job(db_path, { - "title": title or "(untitled)", - "company": company, - "url": synthetic_url, - "source": "email", - "location": "", - "is_remote": 0, - "salary": "", - "description": parsed["body"][:2000], - "date_found": datetime.now().isoformat()[:10], - }) - if job_id: - add_contact(db_path, job_id=job_id, direction="inbound", - subject=parsed["subject"], - from_addr=parsed["from_addr"], - body=parsed["body"], - received_at=parsed["date"][:16] if parsed["date"] else "", - message_id=mid) - known_message_ids.add(mid) - existing_urls.add(synthetic_url) - new_leads += 1 - - return new_leads -``` - -**Step 5: Update `sync_all()` to call `_scan_unmatched_leads()`** - -In `sync_all()`, after the per-job loop and before `conn.logout()`: - -```python -from scripts.db import get_all_message_ids -known_mids = get_all_message_ids(db_path) -summary["new_leads"] = _scan_unmatched_leads(conn, cfg, db_path, known_mids) -``` - -Also add `"new_leads": 0` to the initial `summary` dict. - -**Step 6: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py -v -``` - -Expected: all pass. - -**Step 7: Commit** - -```bash -git add scripts/imap_sync.py scripts/db.py tests/test_imap_sync.py -git commit -m "feat: recruiter lead extraction from unmatched inbound emails" -``` - ---- - -### Task 5: email_sync background task type - -**Files:** -- Modify: `scripts/task_runner.py` -- Test: `tests/test_task_runner.py` - -**Context:** Add `email_sync` to the `if/elif` chain in `_run_task()`. `job_id` is 0 (global task). The result summary is stored in the task's `error` field as a string (same pattern as `discovery`). If IMAP config is missing (`FileNotFoundError`), mark failed with a friendly message. - -**Step 1: Add test** - -Append to `tests/test_task_runner.py`: - -```python -def test_run_task_email_sync_success(tmp_path): - """email_sync task calls sync_all and marks completed with summary.""" - db, _ = _make_db(tmp_path) - from scripts.db import insert_task, get_task_for_job - task_id, _ = insert_task(db, "email_sync", 0) - - summary = {"synced": 3, "inbound": 5, "outbound": 2, "new_leads": 1, "errors": []} - with patch("scripts.imap_sync.sync_all", return_value=summary): - from scripts.task_runner import _run_task - _run_task(db, task_id, "email_sync", 0) - - task = get_task_for_job(db, "email_sync", 0) - assert task["status"] == "completed" - assert "3 jobs" in task["error"] - - -def test_run_task_email_sync_file_not_found(tmp_path): - """email_sync marks failed with helpful message when config is missing.""" - db, _ = _make_db(tmp_path) - from scripts.db import insert_task, get_task_for_job - task_id, _ = insert_task(db, "email_sync", 0) - - with patch("scripts.imap_sync.sync_all", side_effect=FileNotFoundError("config/email.yaml")): - from scripts.task_runner import _run_task - _run_task(db, task_id, "email_sync", 0) - - task = get_task_for_job(db, "email_sync", 0) - assert task["status"] == "failed" - assert "email" in task["error"].lower() -``` - -**Step 2: Run to confirm failures** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_task_runner.py::test_run_task_email_sync_success tests/test_task_runner.py::test_run_task_email_sync_file_not_found -v -``` - -Expected: 2 failures. - -**Step 3: Add email_sync branch to `_run_task()` in `scripts/task_runner.py`** - -Add after the `company_research` elif, before the `else`: - -```python -elif task_type == "email_sync": - try: - from scripts.imap_sync import sync_all - result = sync_all(db_path) - leads = result.get("new_leads", 0) - errs = len(result.get("errors", [])) - msg = ( - f"{result['synced']} jobs updated, " - f"+{result['inbound']} in, +{result['outbound']} out" - f"{f', {leads} new lead(s)' if leads else ''}" - f"{f', {errs} error(s)' if errs else ''}" - ) - update_task_status(db_path, task_id, "completed", error=msg) - return - except FileNotFoundError: - update_task_status(db_path, task_id, "failed", - error="Email not configured — go to Settings → Email") - return -``` - -**Step 4: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_task_runner.py -v -``` - -Expected: all pass. - -**Step 5: Commit** - -```bash -git add scripts/task_runner.py tests/test_task_runner.py -git commit -m "feat: add email_sync background task type to task_runner" -``` - ---- - -### Task 6: Sync Emails button on Home page - -**Files:** -- Modify: `app/Home.py` - -**Context:** Home.py has three sections in `left / mid / right` columns (Find Jobs, Score Listings, Send to Notion). Add a fourth section. Since we can't easily add a 4th column to the same row without crowding, add it as a new row below the divider, before the Danger Zone expander. Use the same background task pattern as discovery: check for an in-flight `email_sync` task, disable button if running, poll with `@st.fragment(run_every=4)`. - -Also update the imports to include `get_all_message_ids` — no, actually we don't need that. We need `submit_task` (already imported) and `get_task_for_job` (already imported). - -Also update the success message to show new_leads if any. - -No tests needed for UI pages (Streamlit pages aren't unit-testable without an e2e framework). - -**Step 1: Add Email Sync section to `app/Home.py`** - -After the `with right:` block and before `st.divider()` (the one before Danger Zone), add: - -```python -st.divider() - -# ── Email Sync ──────────────────────────────────────────────────────────────── -email_left, email_right = st.columns([3, 1]) - -with email_left: - st.subheader("Sync Emails") - st.caption("Pull inbound recruiter emails and match them to active applications. " - "New recruiter outreach is added to your Job Review queue.") - -with email_right: - _email_task = get_task_for_job(DEFAULT_DB, "email_sync", 0) - _email_running = _email_task and _email_task["status"] in ("queued", "running") - - if st.button("📧 Sync Emails", use_container_width=True, type="primary", - disabled=bool(_email_running)): - submit_task(DEFAULT_DB, "email_sync", 0) - st.rerun() - - if _email_running: - @st.fragment(run_every=4) - def _email_status(): - t = get_task_for_job(DEFAULT_DB, "email_sync", 0) - if t and t["status"] in ("queued", "running"): - st.info("⏳ Syncing emails…") - else: - st.rerun() - _email_status() - elif _email_task and _email_task["status"] == "completed": - st.success(f"✅ {_email_task.get('error', 'Done')}") - elif _email_task and _email_task["status"] == "failed": - st.error(f"Sync failed: {_email_task.get('error', '')}") -``` - -**Step 2: Manual smoke test** - -```bash -bash /devl/job-seeker/scripts/manage-ui.sh restart -``` - -Open http://localhost:8501, confirm "Sync Emails" section appears with button. - -**Step 3: Commit** - -```bash -git add app/Home.py -git commit -m "feat: add Sync Emails background task button to Home page" -``` - ---- - -### Task 7: Convert Interviews sync to background task + add stage suggestion banner - -**Files:** -- Modify: `app/pages/5_Interviews.py` - -**Context:** The sidebar sync button in 5_Interviews.py currently calls `sync_all()` synchronously inside a `with st.spinner(...)` block (lines 38–61). Replace it with `submit_task(DEFAULT_DB, "email_sync", 0)` + fragment polling, matching the pattern in Home.py. - -Then add the stage suggestion banner in `_render_card()`. After the interview date form (or at the top of the "if not compact:" block), call `get_unread_stage_signals()`. If any exist, show the most recent one with → Move and Dismiss buttons. - -The banner should only show for stages where a stage advancement makes sense: `applied`, `phone_screen`, `interviewing`. Not `offer` or `hired`. - -**Step 1: Update imports in `5_Interviews.py`** - -Add to the existing `from scripts.db import (...)` block: -- `get_unread_stage_signals` -- `dismiss_stage_signal` - -Add to the `from scripts.task_runner import submit_task` line (already present). - -**Step 2: Replace synchronous sync button** - -Replace the entire `with st.sidebar:` block (lines 38–61) with: - -```python -with st.sidebar: - st.markdown("### 📧 Email Sync") - _email_task = get_task_for_job(DEFAULT_DB, "email_sync", 0) - _email_running = _email_task and _email_task["status"] in ("queued", "running") - - if st.button("🔄 Sync Emails", use_container_width=True, type="primary", - disabled=bool(_email_running)): - submit_task(DEFAULT_DB, "email_sync", 0) - st.rerun() - - if _email_running: - @st.fragment(run_every=4) - def _email_sidebar_status(): - t = get_task_for_job(DEFAULT_DB, "email_sync", 0) - if t and t["status"] in ("queued", "running"): - st.info("⏳ Syncing…") - else: - st.rerun() - _email_sidebar_status() - elif _email_task and _email_task["status"] == "completed": - st.success(_email_task.get("error", "Done")) - elif _email_task and _email_task["status"] == "failed": - msg = _email_task.get("error", "") - if "not configured" in msg.lower(): - st.error("Email not configured. Go to **Settings → Email**.") - else: - st.error(f"Sync failed: {msg}") -``` - -**Step 3: Add stage suggestion banner in `_render_card()`** - -Inside `_render_card()`, at the start of the `if not compact:` block (just before `# Advance / Reject buttons`), add: - -```python -if stage in ("applied", "phone_screen", "interviewing"): - signals = get_unread_stage_signals(DEFAULT_DB, job_id=job_id) - if signals: - sig = signals[-1] # most recent - _SIGNAL_LABELS = { - "interview_scheduled": ("📞 Phone Screen", "phone_screen"), - "positive_response": ("📞 Phone Screen", "phone_screen"), - "offer_received": ("📜 Offer", "offer"), - "rejected": ("✗ Reject", None), - } - label_text, target_stage = _SIGNAL_LABELS.get(sig["stage_signal"], (None, None)) - with st.container(border=True): - st.caption( - f"💡 Email suggests: **{sig['stage_signal'].replace('_', ' ')}** \n" - f"_{sig.get('subject', '')}_ · {(sig.get('received_at') or '')[:10]}" - ) - b1, b2 = st.columns(2) - if target_stage and b1.button( - f"→ {label_text}", key=f"sig_adv_{sig['id']}", - use_container_width=True, type="primary", - ): - if target_stage == "phone_screen" and stage == "applied": - advance_to_stage(DEFAULT_DB, job_id=job_id, stage="phone_screen") - submit_task(DEFAULT_DB, "company_research", job_id) - elif target_stage: - advance_to_stage(DEFAULT_DB, job_id=job_id, stage=target_stage) - dismiss_stage_signal(DEFAULT_DB, sig["id"]) - st.rerun() - elif label_text == "✗ Reject" and b1.button( - "✗ Reject", key=f"sig_rej_{sig['id']}", - use_container_width=True, - ): - reject_at_stage(DEFAULT_DB, job_id=job_id, rejection_stage=stage) - dismiss_stage_signal(DEFAULT_DB, sig["id"]) - st.rerun() - if b2.button("Dismiss", key=f"sig_dis_{sig['id']}", - use_container_width=True): - dismiss_stage_signal(DEFAULT_DB, sig["id"]) - st.rerun() -``` - -**Step 4: Manual smoke test** - -```bash -bash /devl/job-seeker/scripts/manage-ui.sh restart -``` - -Open Interviews page, confirm sidebar sync button is present and non-blocking. - -**Step 5: Commit** - -```bash -git add app/pages/5_Interviews.py -git commit -m "feat: non-blocking email sync + stage suggestion banner on Interviews kanban" -``` - ---- - -### Task 8: Email leads section in Job Review - -**Files:** -- Modify: `app/pages/1_Job_Review.py` -- Modify: `scripts/db.py` - -**Context:** Email leads are jobs with `source = 'email'` and `status = 'pending'`. They already appear in the `pending` list returned by `get_jobs_by_status()`. We want to visually separate them at the top when `show_status == 'pending'`. - -Add a `get_email_leads(db_path)` helper in `scripts/db.py` that returns pending email-source jobs ordered by `date_found DESC`. In the Job Review page, before the main job list loop, if `show_status == 'pending'`, pull email leads and render them in a distinct section with an `📧 Email Lead` badge. Then render the remaining (non-email) pending jobs below. - -**Step 1: Add test for new DB helper** - -Append to `tests/test_db.py`: - -```python -def test_get_email_leads(tmp_path): - """get_email_leads returns only source='email' pending jobs.""" - from scripts.db import init_db, insert_job, get_email_leads - db_path = tmp_path / "test.db" - init_db(db_path) - insert_job(db_path, { - "title": "CSM", "company": "Acme", "url": "https://ex.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "", "date_found": "2026-02-21", - }) - insert_job(db_path, { - "title": "TAM", "company": "Wiz", "url": "email://wiz.com/abc123", - "source": "email", "location": "", "is_remote": 0, - "salary": "", "description": "Hi Alex…", "date_found": "2026-02-21", - }) - leads = get_email_leads(db_path) - assert len(leads) == 1 - assert leads[0]["company"] == "Wiz" - assert leads[0]["source"] == "email" -``` - -**Step 2: Run to confirm failure** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py::test_get_email_leads -v -``` - -Expected: FAIL (ImportError or function missing). - -**Step 3: Add `get_email_leads()` to `scripts/db.py`** - -After `get_jobs_by_status()`: - -```python -def get_email_leads(db_path: Path = DEFAULT_DB) -> list[dict]: - """Return pending jobs with source='email', newest first.""" - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - rows = conn.execute( - "SELECT * FROM jobs WHERE source = 'email' AND status = 'pending' " - "ORDER BY date_found DESC, id DESC" - ).fetchall() - conn.close() - return [dict(r) for r in rows] -``` - -**Step 4: Run test** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_db.py::test_get_email_leads -v -``` - -Expected: PASS. - -**Step 5: Update `1_Job_Review.py`** - -Add to the top-level import from `scripts.db`: -- `get_email_leads` - -After `init_db(DEFAULT_DB)` and before the sidebar filters block, add: - -```python -# ── Email leads (shown only when browsing pending) ──────────────────────────── -_email_leads = get_email_leads(DEFAULT_DB) if True else [] -``` - -(We always fetch them; the section only renders when `show_status == 'pending'`.) - -After `st.divider()` (after the caption line) and before the main `for job in jobs:` loop, add: - -```python -if show_status == "pending" and _email_leads: - st.subheader(f"📧 Email Leads ({len(_email_leads)})") - st.caption( - "Inbound recruiter emails not yet matched to a scraped listing. " - "Approve to move to Job Review; Reject to dismiss." - ) - for lead in _email_leads: - lead_id = lead["id"] - with st.container(border=True): - left_l, right_l = st.columns([7, 3]) - with left_l: - st.markdown(f"**{lead['title']}** — {lead['company']}") - badge_cols = st.columns(4) - badge_cols[0].caption("📧 Email Lead") - badge_cols[1].caption(f"📅 {lead.get('date_found', '')}") - if lead.get("description"): - with st.expander("📄 Email excerpt", expanded=False): - st.text(lead["description"][:500]) - with right_l: - if st.button("✅ Approve", key=f"el_approve_{lead_id}", - type="primary", use_container_width=True): - update_job_status(DEFAULT_DB, [lead_id], "approved") - st.rerun() - if st.button("❌ Reject", key=f"el_reject_{lead_id}", - use_container_width=True): - update_job_status(DEFAULT_DB, [lead_id], "rejected") - st.rerun() - st.divider() - -# Filter out email leads from the main pending list (already shown above) -if show_status == "pending": - jobs = [j for j in jobs if j.get("source") != "email"] -``` - -**Step 6: Manual smoke test** - -```bash -bash /devl/job-seeker/scripts/manage-ui.sh restart -``` - -Confirm Job Review shows "Email Leads" section when filtering for pending. - -**Step 7: Commit** - -```bash -git add scripts/db.py tests/test_db.py app/pages/1_Job_Review.py -git commit -m "feat: show email lead jobs at top of Job Review pending queue" -``` - ---- - -### Task 9: Full test run + final polish - -**Files:** -- No new files - -**Step 1: Run full test suite** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` - -Expected: all pass. Fix any regressions before proceeding. - -**Step 2: Verify DB exports in `scripts/db.py`** - -Confirm that `get_unread_stage_signals`, `dismiss_stage_signal`, `get_all_message_ids`, and `get_email_leads` are imported correctly wherever used: -- `5_Interviews.py` imports `get_unread_stage_signals`, `dismiss_stage_signal` -- `imap_sync.py` imports `get_all_message_ids` -- `1_Job_Review.py` imports `get_email_leads` - -Run: -```bash -conda run -n job-seeker python -c "from scripts.db import get_unread_stage_signals, dismiss_stage_signal, get_all_message_ids, get_email_leads; print('OK')" -``` - -**Step 3: Smoke-test the classifier with real Ollama** - -```bash -conda run -n job-seeker python -c " -from scripts.imap_sync import classify_stage_signal -print(classify_stage_signal('Interview Invitation', 'We would love to schedule a 30-min phone screen with you.')) -print(classify_stage_signal('Your application with DataStax', 'We have decided to move forward with other candidates.')) -print(classify_stage_signal('Application received', 'We have received your application and will be in touch.')) -" -``` - -Expected output: -``` -interview_scheduled -rejected -neutral -``` - -**Step 4: Commit** - -```bash -git add -A -git commit -m "chore: verify all email handling imports and run full test suite" -``` diff --git a/docs/plans/2026-02-22-research-workflow-design.md b/docs/plans/2026-02-22-research-workflow-design.md deleted file mode 100644 index 1277357..0000000 --- a/docs/plans/2026-02-22-research-workflow-design.md +++ /dev/null @@ -1,187 +0,0 @@ -# Research Workflow Redesign - -**Date:** 2026-02-22 -**Status:** Approved - -## Problem - -The current `company_research.py` produces shallow output: -- Resume context is a hardcoded 2-sentence blurb — talking points aren't grounded in Alex's actual experience -- Search coverage is limited: CEO, HQ, LinkedIn, one generic news query -- Output has 4 sections; new data categories (tech stack, funding, culture, competitors) have nowhere to go -- No skills/keyword config to drive experience matching against the JD - -## Approach: Query Expansion + Parallel JSON Searches + Single LLM Pass - -Run all searches (companyScraper sequential + new parallel SearXNG JSON queries), aggregate into a structured context block, pre-select resume experiences by keyword score, single LLM call produces all expanded sections. - ---- - -## Design - -### 1. Search Pipeline - -**Phase 1 — companyScraper (unchanged, sequential)** -- CEO name, HQ address, LinkedIn URL - -**Phase 1b — Parallel SearXNG JSON queries (new/expanded)** - -Six queries run concurrently via daemon threads: - -| Intent | Query pattern | -|---|---| -| Recent news/press | `"{company}" news 2025 2026` | -| Funding & investors | `"{company}" funding round investors Series valuation` | -| Tech stack | `"{company}" tech stack engineering technology platform` | -| Competitors | `"{company}" competitors alternatives vs market` | -| Culture / Glassdoor | `"{company}" glassdoor culture reviews employees` | -| CEO press (if found) | `"{ceo}" "{company}"` | - -Each returns 3–4 deduplicated snippets (title + content + URL), labeled by type. -Results are best-effort — any failed query is silently skipped. - ---- - -### 2. Resume Matching - -**`config/resume_keywords.yaml`** — three categories, tag-managed via Settings UI: - -```yaml -skills: - - Customer Success - - Technical Account Management - - Revenue Operations - - Salesforce - - Gainsight - - data analysis - - stakeholder management - -domains: - - B2B SaaS - - enterprise software - - security / compliance - - post-sale lifecycle - -keywords: - - QBR - - churn reduction - - NRR / ARR - - onboarding - - renewal - - executive sponsorship - - VOC -``` - -**Matching logic:** -1. Case-insensitive substring check of all keywords against JD text → `matched_keywords` list -2. Score each experience entry: count of matched keywords appearing in position title + responsibility bullets -3. Top 2 by score → included in prompt as full detail (position, company, period, all bullets) -4. Remaining entries → condensed one-liners ("Founder @ M3 Consulting, 2023–present") - -**UpGuard NDA rule** (explicit in prompt): reference as "enterprise security vendor" in general; only name UpGuard directly if the role has a strong security/compliance focus. - ---- - -### 3. LLM Context Block Structure - -``` -## Role Context -{title} at {company} - -## Job Description -{JD text, up to 2500 chars} - -## Alex's Matched Experience -[Top 2 scored experience entries — full detail] - -Also in Alex's background: [remaining entries as one-liners] - -## Matched Skills & Keywords -Skills matching this JD: {matched_keywords joined} - -## Live Company Data -- CEO: {name} -- HQ: {location} -- LinkedIn: {url} - -## News & Press -[snippets] - -## Funding & Investors -[snippets] - -## Tech Stack -[snippets] - -## Competitors -[snippets] - -## Culture & Employee Signals -[snippets] -``` - ---- - -### 4. Output Sections (7, up from 4) - -| Section header | Purpose | -|---|---| -| `## Company Overview` | What they do, business model, size/stage, market position | -| `## Leadership & Culture` | CEO background, leadership team, philosophy | -| `## Tech Stack & Product` | What they build, relevant technology, product direction | -| `## Funding & Market Position` | Stage, investors, recent rounds, competitor landscape | -| `## Recent Developments` | News, launches, pivots, exec moves | -| `## Red Flags & Watch-outs` | Culture issues, layoffs, exec departures, financial stress | -| `## Talking Points for Alex` | 5 role-matched, resume-grounded, UpGuard-aware talking points ready to speak aloud | - -Talking points prompt instructs LLM to: cite the specific matched experience by name, reference matched skills, apply UpGuard NDA rule, frame each as a ready-to-speak sentence. - ---- - -### 5. DB Schema Changes - -Add columns to `company_research` table: - -```sql -ALTER TABLE company_research ADD COLUMN tech_brief TEXT; -ALTER TABLE company_research ADD COLUMN funding_brief TEXT; -ALTER TABLE company_research ADD COLUMN competitors_brief TEXT; -ALTER TABLE company_research ADD COLUMN red_flags TEXT; -``` - -Existing columns (`company_brief`, `ceo_brief`, `talking_points`, `raw_output`) unchanged. - ---- - -### 6. Settings UI — Skills & Keywords Tab - -New tab in `app/pages/2_Settings.py`: -- One expander or subheader per category (Skills, Domains, Keywords) -- Tag chips rendered with `st.pills` or columns of `st.badge`-style buttons with × -- Inline text input + Add button per category -- Each add/remove saves immediately to `config/resume_keywords.yaml` - ---- - -### 7. Interview Prep UI Changes - -`app/pages/6_Interview_Prep.py` — render new sections alongside existing ones: -- Tech Stack & Product (new panel) -- Funding & Market Position (new panel) -- Red Flags & Watch-outs (new panel, visually distinct — e.g. orange/amber) -- Talking Points promoted to top (most useful during a live call) - ---- - -## Files Affected - -| File | Change | -|---|---| -| `scripts/company_research.py` | Parallel search queries, resume matching, expanded prompt + sections | -| `scripts/db.py` | Add 4 new columns to `company_research`; update `save_research` / `get_research` | -| `config/resume_keywords.yaml` | New file | -| `config/resume_keywords.yaml.example` | New committed template | -| `app/pages/2_Settings.py` | New Skills & Keywords tab | -| `app/pages/6_Interview_Prep.py` | Render new sections | -| `tests/test_db.py` | Tests for new columns | -| `tests/test_company_research.py` | New test file for matching logic + section parsing | diff --git a/docs/plans/2026-02-22-research-workflow-impl.md b/docs/plans/2026-02-22-research-workflow-impl.md deleted file mode 100644 index 1d7c84f..0000000 --- a/docs/plans/2026-02-22-research-workflow-impl.md +++ /dev/null @@ -1,869 +0,0 @@ -# Research Workflow Redesign — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Expand company research to gather richer web data (funding, tech stack, competitors, culture/Glassdoor, news), match Alex's resume experience against the JD, and produce a 7-section brief with role-grounded talking points. - -**Architecture:** Parallel SearXNG JSON queries (6 types) feed a structured context block alongside tiered resume experience (top-2 scored full, rest condensed) from `config/resume_keywords.yaml`. Single LLM call produces 7 output sections stored in expanded DB columns. - -**Tech Stack:** Python threading, requests (SearXNG JSON API at `http://localhost:8888/search?format=json`), PyYAML, SQLite ALTER TABLE migrations, Streamlit `st.pills` / column chips. - -**Design doc:** `docs/plans/2026-02-22-research-workflow-design.md` - -**Run tests:** `/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v` -**Python:** `conda run -n job-seeker python - """ - - mock_resp = MagicMock() - mock_resp.text = json_ld_html - mock_resp.raise_for_status = MagicMock() - - with patch("scripts.scrape_url.requests.get", return_value=mock_resp): - from scripts.scrape_url import scrape_job_url - result = scrape_job_url(db, job_id) - - assert result.get("title") == "TAM Role" - assert result.get("company") == "TechCo" - - -def test_scrape_url_graceful_on_http_error(tmp_path): - db, job_id = _make_db(tmp_path) - import requests as req - - with patch("scripts.scrape_url.requests.get", side_effect=req.RequestException("timeout")): - from scripts.scrape_url import scrape_job_url - result = scrape_job_url(db, job_id) - - # Should return empty dict and not raise; job row still exists - assert isinstance(result, dict) - import sqlite3 - conn = sqlite3.connect(db) - row = conn.execute("SELECT id FROM jobs WHERE id=?", (job_id,)).fetchone() - conn.close() - assert row is not None -``` - -**Step 2: Run tests to verify they fail** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_scrape_url.py -v -``` -Expected: FAIL — `ModuleNotFoundError: No module named 'scripts.scrape_url'` - -**Step 3: Implement `scripts/scrape_url.py`** - -```python -# scripts/scrape_url.py -""" -Scrape a job listing from its URL and update the job record. - -Supports: - - LinkedIn (guest jobs API — no auth required) - - Indeed (HTML parse) - - Glassdoor (JobSpy internal scraper, same as enrich_descriptions.py) - - Generic (JSON-LD → og:tags fallback) - -Usage (background task — called by task_runner): - from scripts.scrape_url import scrape_job_url - scrape_job_url(db_path, job_id) -""" -import json -import re -import sqlite3 -import sys -from pathlib import Path -from typing import Optional - -import requests -from bs4 import BeautifulSoup - -sys.path.insert(0, str(Path(__file__).parent.parent)) - -from scripts.db import DEFAULT_DB, update_job_fields - -_HEADERS = { - "User-Agent": ( - "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 " - "(KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36" - ) -} -_TIMEOUT = 12 - - -def _detect_board(url: str) -> str: - """Return 'linkedin', 'indeed', 'glassdoor', or 'generic'.""" - url_lower = url.lower() - if "linkedin.com" in url_lower: - return "linkedin" - if "indeed.com" in url_lower: - return "indeed" - if "glassdoor.com" in url_lower: - return "glassdoor" - return "generic" - - -def _extract_linkedin_job_id(url: str) -> Optional[str]: - """Extract numeric job ID from a LinkedIn job URL.""" - m = re.search(r"/jobs/view/(\d+)", url) - return m.group(1) if m else None - - -def canonicalize_url(url: str) -> str: - """ - Strip tracking parameters from a job URL and return a clean canonical form. - - LinkedIn: https://www.linkedin.com/jobs/view//?trk=... → https://www.linkedin.com/jobs/view// - Indeed: strips utm_* and other tracking params - Others: strips utm_source/utm_medium/utm_campaign/trk/refId/trackingId - """ - url = url.strip() - if "linkedin.com" in url.lower(): - job_id = _extract_linkedin_job_id(url) - if job_id: - return f"https://www.linkedin.com/jobs/view/{job_id}/" - # For other boards: strip common tracking params - from urllib.parse import urlparse, urlencode, parse_qsl - _STRIP_PARAMS = { - "utm_source", "utm_medium", "utm_campaign", "utm_content", "utm_term", - "trk", "trkEmail", "refId", "trackingId", "lipi", "midToken", "midSig", - "eid", "otpToken", "ssid", "fmid", - } - parsed = urlparse(url) - clean_qs = urlencode([(k, v) for k, v in parse_qsl(parsed.query) if k not in _STRIP_PARAMS]) - return parsed._replace(query=clean_qs).geturl() - - -def _scrape_linkedin(url: str) -> dict: - """Fetch via LinkedIn guest jobs API (no auth required).""" - job_id = _extract_linkedin_job_id(url) - if not job_id: - return {} - api_url = f"https://www.linkedin.com/jobs-guest/jobs/api/jobPosting/{job_id}" - resp = requests.get(api_url, headers=_HEADERS, timeout=_TIMEOUT) - resp.raise_for_status() - soup = BeautifulSoup(resp.text, "html.parser") - - def _text(selector, **kwargs): - tag = soup.find(selector, **kwargs) - return tag.get_text(strip=True) if tag else "" - - title = _text("h2", class_="top-card-layout__title") - company = _text("a", class_="topcard__org-name-link") or _text("span", class_="topcard__org-name-link") - location = _text("span", class_="topcard__flavor--bullet") - desc_div = soup.find("div", class_="show-more-less-html__markup") - description = desc_div.get_text(separator="\n", strip=True) if desc_div else "" - - return {k: v for k, v in { - "title": title, - "company": company, - "location": location, - "description": description, - "source": "linkedin", - }.items() if v} - - -def _scrape_indeed(url: str) -> dict: - """Scrape an Indeed job page.""" - resp = requests.get(url, headers=_HEADERS, timeout=_TIMEOUT) - resp.raise_for_status() - return _parse_json_ld_or_og(resp.text) or {} - - -def _scrape_glassdoor(url: str) -> dict: - """Re-use JobSpy's Glassdoor scraper for description fetch.""" - m = re.search(r"jl=(\d+)", url) - if not m: - return {} - try: - from jobspy.glassdoor import Glassdoor - from jobspy.glassdoor.constant import fallback_token, headers - from jobspy.model import ScraperInput, Site - from jobspy.util import create_session - - scraper = Glassdoor() - scraper.base_url = "https://www.glassdoor.com/" - scraper.session = create_session(has_retry=True) - token = scraper._get_csrf_token() - headers["gd-csrf-token"] = token if token else fallback_token - scraper.scraper_input = ScraperInput(site_type=[Site.GLASSDOOR]) - description = scraper._fetch_job_description(int(m.group(1))) - return {"description": description} if description else {} - except Exception: - return {} - - -def _parse_json_ld_or_og(html: str) -> dict: - """Extract job fields from JSON-LD structured data, then og: meta tags.""" - soup = BeautifulSoup(html, "html.parser") - - # Try JSON-LD first - for script in soup.find_all("script", type="application/ld+json"): - try: - data = json.loads(script.string or "") - if isinstance(data, list): - data = next((d for d in data if d.get("@type") == "JobPosting"), {}) - if data.get("@type") == "JobPosting": - org = data.get("hiringOrganization") or {} - loc = (data.get("jobLocation") or {}) - if isinstance(loc, list): - loc = loc[0] if loc else {} - addr = loc.get("address") or {} - location = ( - addr.get("addressLocality", "") or - addr.get("addressRegion", "") or - addr.get("addressCountry", "") - ) - return {k: v for k, v in { - "title": data.get("title", ""), - "company": org.get("name", ""), - "location": location, - "description": data.get("description", ""), - "salary": str(data.get("baseSalary", "")) if data.get("baseSalary") else "", - }.items() if v} - except Exception: - continue - - # Fall back to og: meta tags - def _meta(prop): - tag = soup.find("meta", property=prop) or soup.find("meta", attrs={"name": prop}) - return (tag or {}).get("content", "") if tag else "" - - title = _meta("og:title") or (soup.find("title") or {}).get_text(strip=True) - description = _meta("og:description") - return {k: v for k, v in {"title": title, "description": description}.items() if v} - - -def _scrape_generic(url: str) -> dict: - resp = requests.get(url, headers=_HEADERS, timeout=_TIMEOUT) - resp.raise_for_status() - return _parse_json_ld_or_og(resp.text) or {} - - -def scrape_job_url(db_path: Path = DEFAULT_DB, job_id: int = None) -> dict: - """ - Fetch the job listing at the stored URL and update the job record. - - Returns the dict of fields that were scraped (may be empty on failure). - Does not raise — failures are logged and the job row is left as-is. - """ - if not job_id: - return {} - - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - row = conn.execute("SELECT url FROM jobs WHERE id=?", (job_id,)).fetchone() - conn.close() - if not row: - return {} - - url = row["url"] or "" - if not url.startswith("http"): - return {} - - board = _detect_board(url) - try: - if board == "linkedin": - fields = _scrape_linkedin(url) - elif board == "indeed": - fields = _scrape_indeed(url) - elif board == "glassdoor": - fields = _scrape_glassdoor(url) - else: - fields = _scrape_generic(url) - except requests.RequestException as exc: - print(f"[scrape_url] HTTP error for job {job_id} ({url}): {exc}") - return {} - except Exception as exc: - print(f"[scrape_url] Error scraping job {job_id} ({url}): {exc}") - return {} - - if fields: - # Never overwrite the URL or source with empty values - fields.pop("url", None) - update_job_fields(db_path, job_id, fields) - print(f"[scrape_url] job {job_id}: scraped '{fields.get('title', '?')}' @ {fields.get('company', '?')}") - - return fields -``` - -**Step 4: Add `scrape_url` task type to `scripts/task_runner.py`** - -In `_run_task`, add a new `elif` branch after `enrich_descriptions` and before the final `else`: - -```python - elif task_type == "scrape_url": - from scripts.scrape_url import scrape_job_url - fields = scrape_job_url(db_path, job_id) - title = fields.get("title") or job.get("url", "?") - company = fields.get("company", "") - msg = f"{title}" + (f" @ {company}" if company else "") - update_task_status(db_path, task_id, "completed", error=msg) - return -``` - -**Step 5: Run all tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_scrape_url.py -v -``` -Expected: all PASS - -**Step 6: Commit** - -```bash -git add scripts/scrape_url.py scripts/task_runner.py tests/test_scrape_url.py -git commit -m "feat: add scrape_url background task for URL-based job import" -``` - ---- - -## Task 3: LinkedIn Job Alert email parser - -**Files:** -- Modify: `scripts/imap_sync.py` -- Test: `tests/test_imap_sync.py` - -**Step 1: Write the failing tests** - -Add to `tests/test_imap_sync.py`: - -```python -def test_parse_linkedin_alert_extracts_jobs(): - from scripts.imap_sync import parse_linkedin_alert - body = """\ -Your job alert for customer success manager in United States -New jobs match your preferences. -Manage alerts: https://www.linkedin.com/comm/jobs/alerts?... - -Customer Success Manager -Reflow -California, United States -View job: https://www.linkedin.com/comm/jobs/view/4376518925/?trackingId=abc%3D%3D&refId=xyz - ---------------------------------------------------------- - -Customer Engagement Manager -Bitwarden -United States - -2 school alumni -Apply with resume & profile -View job: https://www.linkedin.com/comm/jobs/view/4359824983/?trackingId=def%3D%3D - ---------------------------------------------------------- - -""" - jobs = parse_linkedin_alert(body) - assert len(jobs) == 2 - assert jobs[0]["title"] == "Customer Success Manager" - assert jobs[0]["company"] == "Reflow" - assert jobs[0]["location"] == "California, United States" - assert jobs[0]["url"] == "https://www.linkedin.com/jobs/view/4376518925/" - assert jobs[1]["title"] == "Customer Engagement Manager" - assert jobs[1]["company"] == "Bitwarden" - assert jobs[1]["url"] == "https://www.linkedin.com/jobs/view/4359824983/" - - -def test_parse_linkedin_alert_skips_blocks_without_view_job(): - from scripts.imap_sync import parse_linkedin_alert - body = """\ -Customer Success Manager -Some Company -United States - ---------------------------------------------------------- - -Valid Job Title -Valid Company -Remote -View job: https://www.linkedin.com/comm/jobs/view/1111111/?x=y - ---------------------------------------------------------- -""" - jobs = parse_linkedin_alert(body) - assert len(jobs) == 1 - assert jobs[0]["title"] == "Valid Job Title" - - -def test_parse_linkedin_alert_empty_body(): - from scripts.imap_sync import parse_linkedin_alert - assert parse_linkedin_alert("") == [] - assert parse_linkedin_alert("No jobs here.") == [] -``` - -**Step 2: Run tests to verify they fail** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py::test_parse_linkedin_alert_extracts_jobs tests/test_imap_sync.py::test_parse_linkedin_alert_skips_blocks_without_view_job tests/test_imap_sync.py::test_parse_linkedin_alert_empty_body -v -``` -Expected: FAIL — `ImportError: cannot import name 'parse_linkedin_alert'` - -**Step 3: Implement `parse_linkedin_alert` in `scripts/imap_sync.py`** - -Add after the existing `_has_todo_keyword` function (around line 391): - -```python -_LINKEDIN_ALERT_SENDER = "jobalerts-noreply@linkedin.com" - -# Social-proof / nav lines to skip when parsing alert blocks -_ALERT_SKIP_PHRASES = { - "alumni", "apply with", "actively hiring", "manage alerts", - "view all jobs", "your job alert", "new jobs match", - "unsubscribe", "linkedin corporation", -} - - -def parse_linkedin_alert(body: str) -> list[dict]: - """ - Parse the plain-text body of a LinkedIn Job Alert digest email. - - Returns a list of dicts: {title, company, location, url}. - URL is canonicalized to https://www.linkedin.com/jobs/view// - (tracking parameters stripped). - """ - jobs = [] - # Split on separator lines (10+ dashes) - blocks = re.split(r"\n\s*-{10,}\s*\n", body) - for block in blocks: - lines = [ln.strip() for ln in block.strip().splitlines() if ln.strip()] - - # Find "View job:" URL - url = None - for line in lines: - m = re.search(r"View job:\s*(https?://\S+)", line, re.IGNORECASE) - if m: - raw_url = m.group(1) - job_id_m = re.search(r"/jobs/view/(\d+)", raw_url) - if job_id_m: - url = f"https://www.linkedin.com/jobs/view/{job_id_m.group(1)}/" - break - if not url: - continue - - # Filter noise lines - content = [ - ln for ln in lines - if not any(p in ln.lower() for p in _ALERT_SKIP_PHRASES) - and not ln.lower().startswith("view job:") - and not ln.startswith("http") - ] - if len(content) < 2: - continue - - jobs.append({ - "title": content[0], - "company": content[1], - "location": content[2] if len(content) > 2 else "", - "url": url, - }) - return jobs -``` - -**Step 4: Wire the parser into `_scan_unmatched_leads`** - -In `_scan_unmatched_leads`, inside the `for uid in all_uids:` loop, add a detection block immediately after the `if mid in known_message_ids: continue` check (before the existing `_has_recruitment_keyword` check): - -```python - # ── LinkedIn Job Alert digest — parse each card individually ────── - if _LINKEDIN_ALERT_SENDER in parsed["from_addr"].lower(): - cards = parse_linkedin_alert(parsed["body"]) - for card in cards: - if card["url"] in existing_urls: - continue - job_id = insert_job(db_path, { - "title": card["title"], - "company": card["company"], - "url": card["url"], - "source": "linkedin", - "location": card["location"], - "is_remote": 0, - "salary": "", - "description": "", - "date_found": datetime.now().isoformat()[:10], - }) - if job_id: - from scripts.task_runner import submit_task - submit_task(db_path, "scrape_url", job_id) - existing_urls.add(card["url"]) - new_leads += 1 - print(f"[imap] LinkedIn alert → {card['company']} — {card['title']}") - known_message_ids.add(mid) - continue # skip normal LLM extraction path -``` - -**Step 5: Run all imap_sync tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_imap_sync.py -v -``` -Expected: all PASS (including the 3 new tests) - -**Step 6: Commit** - -```bash -git add scripts/imap_sync.py tests/test_imap_sync.py -git commit -m "feat: auto-parse LinkedIn Job Alert digest emails into pending jobs" -``` - ---- - -## Task 4: Home page — Add Job(s) by URL - -**Files:** -- Modify: `app/Home.py` - -No unit tests — this is pure Streamlit UI. Verify manually by pasting a URL and checking the DB. - -**Step 1: Add `_queue_url_imports` helper and the new section to `app/Home.py`** - -Add to the imports at the top (after the existing `from scripts.db import ...` line): - -```python -from scripts.db import DEFAULT_DB, init_db, get_job_counts, purge_jobs, purge_email_data, \ - kill_stuck_tasks, get_task_for_job, get_active_tasks, insert_job, get_existing_urls -``` - -Add this helper function before the Streamlit layout code (after the `init_db` call at the top): - -```python -def _queue_url_imports(db_path: Path, urls: list[str]) -> int: - """Insert each URL as a pending manual job and queue a scrape_url task. - Returns count of newly queued jobs.""" - from datetime import datetime - from scripts.scrape_url import canonicalize_url - existing = get_existing_urls(db_path) - queued = 0 - for url in urls: - url = canonicalize_url(url.strip()) - if not url.startswith("http"): - continue - if url in existing: - continue - job_id = insert_job(db_path, { - "title": "Importing…", - "company": "", - "url": url, - "source": "manual", - "location": "", - "description": "", - "date_found": datetime.now().isoformat()[:10], - }) - if job_id: - submit_task(db_path, "scrape_url", job_id) - queued += 1 - return queued -``` - -Add a new section between the Email Sync divider and the Danger Zone expander. Replace: - -```python -st.divider() - -# ── Danger zone: purge + re-scrape ──────────────────────────────────────────── -``` - -with: - -```python -st.divider() - -# ── Add Jobs by URL ─────────────────────────────────────────────────────────── -add_left, add_right = st.columns([3, 1]) -with add_left: - st.subheader("Add Jobs by URL") - st.caption("Paste job listing URLs to import and scrape in the background. " - "Supports LinkedIn, Indeed, Glassdoor, and most job boards.") - -url_tab, csv_tab = st.tabs(["Paste URLs", "Upload CSV"]) - -with url_tab: - url_text = st.text_area( - "urls", - placeholder="https://www.linkedin.com/jobs/view/1234567/\nhttps://www.indeed.com/viewjob?jk=abc", - height=100, - label_visibility="collapsed", - ) - if st.button("📥 Add Jobs", key="add_urls_btn", use_container_width=True, - disabled=not (url_text or "").strip()): - _urls = [u.strip() for u in url_text.strip().splitlines() if u.strip().startswith("http")] - if _urls: - _n = _queue_url_imports(DEFAULT_DB, _urls) - if _n: - st.success(f"Queued {_n} job{'s' if _n != 1 else ''} for import. Check Job Review shortly.") - else: - st.info("All URLs already in the database.") - st.rerun() - -with csv_tab: - csv_file = st.file_uploader("CSV with a URL column", type=["csv"], - label_visibility="collapsed") - if csv_file: - import csv as _csv - import io as _io - reader = _csv.DictReader(_io.StringIO(csv_file.read().decode("utf-8", errors="replace"))) - _csv_urls = [] - for row in reader: - for val in row.values(): - if val and val.strip().startswith("http"): - _csv_urls.append(val.strip()) - break - if _csv_urls: - st.caption(f"Found {len(_csv_urls)} URL(s) in CSV.") - if st.button("📥 Import CSV Jobs", key="add_csv_btn", use_container_width=True): - _n = _queue_url_imports(DEFAULT_DB, _csv_urls) - st.success(f"Queued {_n} job{'s' if _n != 1 else ''} for import.") - st.rerun() - else: - st.warning("No URLs found — CSV must have a column whose values start with http.") - -# Active scrape_url tasks status -@st.fragment(run_every=3) -def _scrape_status(): - import sqlite3 as _sq - conn = _sq.connect(DEFAULT_DB) - conn.row_factory = _sq.Row - rows = conn.execute( - """SELECT bt.status, bt.error, j.title, j.company, j.url - FROM background_tasks bt - JOIN jobs j ON j.id = bt.job_id - WHERE bt.task_type = 'scrape_url' - AND bt.updated_at >= datetime('now', '-5 minutes') - ORDER BY bt.updated_at DESC LIMIT 20""" - ).fetchall() - conn.close() - if not rows: - return - st.caption("Recent URL imports:") - for r in rows: - if r["status"] == "running": - st.info(f"⏳ Scraping {r['url']}") - elif r["status"] == "completed": - label = f"{r['title']}" + (f" @ {r['company']}" if r['company'] else "") - st.success(f"✅ {label}") - elif r["status"] == "failed": - st.error(f"❌ {r['url']} — {r['error'] or 'scrape failed'}") - -_scrape_status() - -st.divider() - -# ── Danger zone: purge + re-scrape ──────────────────────────────────────────── -``` - -**Step 2: Check `background_tasks` schema has an `updated_at` column** - -The status fragment queries `bt.updated_at`. Verify it exists: - -```bash -conda run -n job-seeker python -c " -import sqlite3 -from scripts.db import DEFAULT_DB, init_db -init_db(DEFAULT_DB) -conn = sqlite3.connect(DEFAULT_DB) -print(conn.execute('PRAGMA table_info(background_tasks)').fetchall()) -" -``` - -If `updated_at` is missing, add a migration in `scripts/db.py`'s `_migrate_db` function: - -```python - try: - conn.execute("ALTER TABLE background_tasks ADD COLUMN updated_at TEXT DEFAULT (datetime('now'))") - except sqlite3.OperationalError: - pass -``` - -And update `update_task_status` in `db.py` to set `updated_at = datetime('now')` on every status change: - -```python -def update_task_status(db_path, task_id, status, error=None): - conn = sqlite3.connect(db_path) - conn.execute( - "UPDATE background_tasks SET status=?, error=?, updated_at=datetime('now') WHERE id=?", - (status, error, task_id), - ) - conn.commit() - conn.close() -``` - -**Step 3: Restart the UI and manually verify** - -```bash -bash /devl/job-seeker/scripts/manage-ui.sh restart -``` - -Test: -1. Paste `https://www.linkedin.com/jobs/view/4376518925/` into the text area -2. Click "📥 Add Jobs" — should show "Queued 1 job for import" -3. Go to Job Review → should see a pending job (Reflow - Customer Success Manager once scraped) - -**Step 4: Commit** - -```bash -git add app/Home.py -git commit -m "feat: add 'Add Jobs by URL' section to Home page with background scraping" -``` - ---- - -## Final: push to remote - -```bash -git push origin main -``` diff --git a/docs/plans/2026-02-24-job-seeker-app-generalize.md b/docs/plans/2026-02-24-job-seeker-app-generalize.md deleted file mode 100644 index ee50c44..0000000 --- a/docs/plans/2026-02-24-job-seeker-app-generalize.md +++ /dev/null @@ -1,1559 +0,0 @@ -# Job Seeker App — Generalization Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Fork the personal job-seeker app into a fully generalized, Docker-Compose-based version at `/Library/Development/devl/job-seeker-app/` that any job seeker can run. - -**Architecture:** A `UserProfile` class backed by `config/user.yaml` replaces all hard-coded personal references across the codebase. A Docker Compose stack with four named profiles (`remote`, `cpu`, `single-gpu`, `dual-gpu`) controls which services start. A first-run wizard gates the app on first launch and writes `user.yaml` on completion. - -**Tech Stack:** Python 3.11, Streamlit, SQLite, Docker Compose v2, NVIDIA Container Toolkit (optional), PyYAML, Requests - -**Reference:** Design doc at `docs/plans/2026-02-24-generalize-design.md` in the personal repo. - ---- - -## Task 1: Bootstrap — New Repo From Personal Source - -**Files:** -- Create: `/Library/Development/devl/job-seeker-app/` (new directory) - -**Step 1: Copy source, strip personal config** - -```bash -mkdir -p /Library/Development/devl/job-seeker-app -rsync -av --exclude='.git' \ - --exclude='staging.db' \ - --exclude='config/email.yaml' \ - --exclude='config/notion.yaml' \ - --exclude='config/tokens.yaml' \ - --exclude='aihawk/' \ - --exclude='__pycache__/' \ - --exclude='*.pyc' \ - --exclude='.streamlit.pid' \ - --exclude='.streamlit.log' \ - /devl/job-seeker/ \ - /Library/Development/devl/job-seeker-app/ -``` - -**Step 2: Init fresh git repo** - -```bash -cd /Library/Development/devl/job-seeker-app -git init -git add . -git commit -m "chore: seed from personal job-seeker (pre-generalization)" -``` - -**Step 3: Verify structure** - -```bash -ls /Library/Development/devl/job-seeker-app/ -# Expected: app/ config/ scripts/ tests/ docs/ environment.yml etc. -# NOT expected: staging.db, config/notion.yaml, config/email.yaml -``` - ---- - -## Task 2: UserProfile Class - -**Files:** -- Create: `scripts/user_profile.py` -- Create: `config/user.yaml.example` -- Create: `tests/test_user_profile.py` - -**Step 1: Write failing tests** - -```python -# tests/test_user_profile.py -import pytest -from pathlib import Path -import tempfile, yaml -from scripts.user_profile import UserProfile - -@pytest.fixture -def profile_yaml(tmp_path): - data = { - "name": "Jane Smith", - "email": "jane@example.com", - "phone": "555-1234", - "linkedin": "linkedin.com/in/janesmith", - "career_summary": "Experienced CSM with 8 years in SaaS.", - "nda_companies": ["AcmeCorp"], - "docs_dir": "~/Documents/JobSearch", - "ollama_models_dir": "~/models/ollama", - "vllm_models_dir": "~/models/vllm", - "inference_profile": "single-gpu", - "services": { - "streamlit_port": 8501, - "ollama_host": "localhost", - "ollama_port": 11434, - "ollama_ssl": False, - "ollama_ssl_verify": True, - "vllm_host": "localhost", - "vllm_port": 8000, - "vllm_ssl": False, - "vllm_ssl_verify": True, - "searxng_host": "localhost", - "searxng_port": 8888, - "searxng_ssl": False, - "searxng_ssl_verify": True, - } - } - p = tmp_path / "user.yaml" - p.write_text(yaml.dump(data)) - return p - -def test_loads_fields(profile_yaml): - p = UserProfile(profile_yaml) - assert p.name == "Jane Smith" - assert p.email == "jane@example.com" - assert p.nda_companies == ["AcmeCorp"] - assert p.inference_profile == "single-gpu" - -def test_service_url_http(profile_yaml): - p = UserProfile(profile_yaml) - assert p.ollama_url == "http://localhost:11434" - assert p.vllm_url == "http://localhost:8000" - assert p.searxng_url == "http://localhost:8888" - -def test_service_url_https(tmp_path): - data = yaml.safe_load(open(profile_yaml)) if False else { - "name": "X", "services": { - "ollama_host": "myserver.com", "ollama_port": 443, - "ollama_ssl": True, "ollama_ssl_verify": True, - "vllm_host": "localhost", "vllm_port": 8000, - "vllm_ssl": False, "vllm_ssl_verify": True, - "searxng_host": "localhost", "searxng_port": 8888, - "searxng_ssl": False, "searxng_ssl_verify": True, - } - } - p2 = tmp_path / "user2.yaml" - p2.write_text(yaml.dump(data)) - prof = UserProfile(p2) - assert prof.ollama_url == "https://myserver.com:443" - -def test_nda_mask(profile_yaml): - p = UserProfile(profile_yaml) - assert p.is_nda("AcmeCorp") - assert p.is_nda("acmecorp") # case-insensitive - assert not p.is_nda("Google") - -def test_missing_file_raises(): - with pytest.raises(FileNotFoundError): - UserProfile(Path("/nonexistent/user.yaml")) - -def test_exists_check(profile_yaml, tmp_path): - assert UserProfile.exists(profile_yaml) - assert not UserProfile.exists(tmp_path / "missing.yaml") - -def test_docs_dir_expanded(profile_yaml): - p = UserProfile(profile_yaml) - assert not str(p.docs_dir).startswith("~") - assert p.docs_dir.is_absolute() -``` - -**Step 2: Run tests to verify they fail** - -```bash -cd /Library/Development/devl/job-seeker-app -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_user_profile.py -v -# Expected: ImportError — scripts/user_profile.py does not exist yet -``` - -**Step 3: Implement UserProfile** - -```python -# scripts/user_profile.py -""" -UserProfile — wraps config/user.yaml and provides typed accessors. - -All hard-coded personal references in the app should import this instead -of reading strings directly. URL construction for services is centralised -here so port/host/SSL changes propagate everywhere automatically. -""" -from __future__ import annotations -from pathlib import Path -import yaml - -_DEFAULTS = { - "name": "", - "email": "", - "phone": "", - "linkedin": "", - "career_summary": "", - "nda_companies": [], - "docs_dir": "~/Documents/JobSearch", - "ollama_models_dir": "~/models/ollama", - "vllm_models_dir": "~/models/vllm", - "inference_profile": "remote", - "services": { - "streamlit_port": 8501, - "ollama_host": "localhost", - "ollama_port": 11434, - "ollama_ssl": False, - "ollama_ssl_verify": True, - "vllm_host": "localhost", - "vllm_port": 8000, - "vllm_ssl": False, - "vllm_ssl_verify": True, - "searxng_host": "localhost", - "searxng_port": 8888, - "searxng_ssl": False, - "searxng_ssl_verify": True, - }, -} - - -class UserProfile: - def __init__(self, path: Path): - if not path.exists(): - raise FileNotFoundError(f"user.yaml not found at {path}") - raw = yaml.safe_load(path.read_text()) or {} - data = {**_DEFAULTS, **raw} - svc_defaults = dict(_DEFAULTS["services"]) - svc_defaults.update(raw.get("services", {})) - data["services"] = svc_defaults - - self.name: str = data["name"] - self.email: str = data["email"] - self.phone: str = data["phone"] - self.linkedin: str = data["linkedin"] - self.career_summary: str = data["career_summary"] - self.nda_companies: list[str] = [c.lower() for c in data["nda_companies"]] - self.docs_dir: Path = Path(data["docs_dir"]).expanduser().resolve() - self.ollama_models_dir: Path = Path(data["ollama_models_dir"]).expanduser().resolve() - self.vllm_models_dir: Path = Path(data["vllm_models_dir"]).expanduser().resolve() - self.inference_profile: str = data["inference_profile"] - self._svc = data["services"] - - # ── Service URLs ────────────────────────────────────────────────────────── - def _url(self, host: str, port: int, ssl: bool) -> str: - scheme = "https" if ssl else "http" - return f"{scheme}://{host}:{port}" - - @property - def ollama_url(self) -> str: - s = self._svc - return self._url(s["ollama_host"], s["ollama_port"], s["ollama_ssl"]) - - @property - def vllm_url(self) -> str: - s = self._svc - return self._url(s["vllm_host"], s["vllm_port"], s["vllm_ssl"]) - - @property - def searxng_url(self) -> str: - s = self._svc - return self._url(s["searxng_host"], s["searxng_port"], s["searxng_ssl"]) - - def ssl_verify(self, service: str) -> bool: - """Return ssl_verify flag for a named service (ollama/vllm/searxng).""" - return bool(self._svc.get(f"{service}_ssl_verify", True)) - - # ── NDA helpers ─────────────────────────────────────────────────────────── - def is_nda(self, company: str) -> bool: - return company.lower() in self.nda_companies - - def nda_label(self, company: str, score: int = 0, threshold: int = 3) -> str: - """Return masked label if company is NDA and score below threshold.""" - if self.is_nda(company) and score < threshold: - return "previous employer (NDA)" - return company - - # ── Existence check (used by app.py before load) ───────────────────────── - @staticmethod - def exists(path: Path) -> bool: - return path.exists() - - # ── llm.yaml URL generation ─────────────────────────────────────────────── - def generate_llm_urls(self) -> dict[str, str]: - """Return base_url values for each backend, derived from services config.""" - return { - "ollama": f"{self.ollama_url}/v1", - "ollama_research": f"{self.ollama_url}/v1", - "vllm": f"{self.vllm_url}/v1", - } -``` - -**Step 4: Run tests to verify they pass** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_user_profile.py -v -# Expected: all PASS -``` - -**Step 5: Create config/user.yaml.example** - -```yaml -# config/user.yaml.example -# Copy to config/user.yaml and fill in your details. -# The first-run wizard will create this file automatically. - -name: "Your Name" -email: "you@example.com" -phone: "555-000-0000" -linkedin: "linkedin.com/in/yourprofile" -career_summary: > - Experienced professional with X years in [your field]. - Specialise in [key skills]. Known for [strength]. - -nda_companies: [] # e.g. ["FormerEmployer"] — masked in research briefs - -docs_dir: "~/Documents/JobSearch" -ollama_models_dir: "~/models/ollama" -vllm_models_dir: "~/models/vllm" - -inference_profile: "remote" # remote | cpu | single-gpu | dual-gpu - -services: - streamlit_port: 8501 - ollama_host: localhost - ollama_port: 11434 - ollama_ssl: false - ollama_ssl_verify: true - vllm_host: localhost - vllm_port: 8000 - vllm_ssl: false - vllm_ssl_verify: true - searxng_host: localhost - searxng_port: 8888 - searxng_ssl: false - searxng_ssl_verify: true -``` - -**Step 6: Commit** - -```bash -git add scripts/user_profile.py config/user.yaml.example tests/test_user_profile.py -git commit -m "feat: add UserProfile class with service URL generation and NDA helpers" -``` - ---- - -## Task 3: Extract Hard-Coded References — Scripts - -**Files:** -- Modify: `scripts/company_research.py` -- Modify: `scripts/generate_cover_letter.py` -- Modify: `scripts/match.py` -- Modify: `scripts/finetune_local.py` -- Modify: `scripts/prepare_training_data.py` - -**Step 1: Add UserProfile loading helper to company_research.py** - -In `scripts/company_research.py`, remove the hard-coded `_SCRAPER_DIR` path and -replace personal references. The scraper is now bundled in the Docker image so its -path is always `/app/companyScraper.py` inside the container. - -Replace: -```python -_SCRAPER_DIR = Path("/Library/Development/scrapers") -_SCRAPER_AVAILABLE = False - -if _SCRAPER_DIR.exists(): - sys.path.insert(0, str(_SCRAPER_DIR)) - try: - from companyScraper import EnhancedCompanyScraper, Config as _ScraperConfig - _SCRAPER_AVAILABLE = True - except (ImportError, SystemExit): - pass -``` - -With: -```python -# companyScraper is bundled into the Docker image at /app/scrapers/ -_SCRAPER_AVAILABLE = False -for _scraper_candidate in [ - Path("/app/scrapers"), # Docker container path - Path(__file__).parent.parent / "scrapers", # local dev fallback -]: - if _scraper_candidate.exists(): - sys.path.insert(0, str(_scraper_candidate)) - try: - from companyScraper import EnhancedCompanyScraper, Config as _ScraperConfig - _SCRAPER_AVAILABLE = True - except (ImportError, SystemExit): - pass - break -``` - -Replace `_searxng_running()` to use profile URL: -```python -def _searxng_running(searxng_url: str = "http://localhost:8888") -> bool: - try: - import requests - r = requests.get(f"{searxng_url}/", timeout=3) - return r.status_code == 200 - except Exception: - return False -``` - -Replace all `"Alex Rivera"` / `"Alex's"` / `_NDA_COMPANIES` references: -```python -# At top of research_company(): -from scripts.user_profile import UserProfile -from scripts.db import DEFAULT_DB -_USER_YAML = Path(__file__).parent.parent / "config" / "user.yaml" -_profile = UserProfile(_USER_YAML) if UserProfile.exists(_USER_YAML) else None - -# In _build_resume_context(), replace _company_label(): -def _company_label(exp: dict) -> str: - company = exp.get("company", "") - score = exp.get("score", 0) - if _profile: - return _profile.nda_label(company, score) - return company - -# Replace "## Alex's Matched Experience": -lines = [f"## {_profile.name if _profile else 'Candidate'}'s Matched Experience"] - -# In research_company() prompt, replace "Alex Rivera": -name = _profile.name if _profile else "the candidate" -summary = _profile.career_summary if _profile else "" -# Replace "You are preparing Alex Rivera for a job interview." with: -prompt = f"""You are preparing {name} for a job interview.\n{summary}\n...""" -``` - -**Step 2: Update generate_cover_letter.py** - -Replace: -```python -LETTERS_DIR = Path("/Library/Documents/JobSearch") -SYSTEM_CONTEXT = """You are writing cover letters for Alex Rivera...""" -``` - -With: -```python -from scripts.user_profile import UserProfile -_USER_YAML = Path(__file__).parent.parent / "config" / "user.yaml" -_profile = UserProfile(_USER_YAML) if UserProfile.exists(_USER_YAML) else None - -LETTERS_DIR = _profile.docs_dir if _profile else Path.home() / "Documents" / "JobSearch" -SYSTEM_CONTEXT = ( - f"You are writing cover letters for {_profile.name}. {_profile.career_summary}" - if _profile else - "You are a professional cover letter writer. Write in first person." -) -``` - -**Step 3: Update match.py** - -Replace hard-coded resume path with a config lookup: -```python -# match.py — read RESUME_PATH from config/user.yaml or fall back to auto-discovery -from scripts.user_profile import UserProfile -_USER_YAML = Path(__file__).parent.parent / "config" / "user.yaml" -_profile = UserProfile(_USER_YAML) if UserProfile.exists(_USER_YAML) else None - -def _find_resume(docs_dir: Path) -> Path | None: - """Find the most recently modified PDF in docs_dir matching *resume* or *cv*.""" - candidates = list(docs_dir.glob("*[Rr]esume*.pdf")) + list(docs_dir.glob("*[Cc][Vv]*.pdf")) - return max(candidates, key=lambda p: p.stat().st_mtime) if candidates else None - -RESUME_PATH = ( - _find_resume(_profile.docs_dir) if _profile else None -) or Path(__file__).parent.parent / "config" / "resume.pdf" -``` - -**Step 4: Update finetune_local.py and prepare_training_data.py** - -Replace all `/Library/` paths with profile-driven paths: -```python -from scripts.user_profile import UserProfile -_USER_YAML = Path(__file__).parent.parent / "config" / "user.yaml" -_profile = UserProfile(_USER_YAML) if UserProfile.exists(_USER_YAML) else None - -_docs = _profile.docs_dir if _profile else Path.home() / "Documents" / "JobSearch" -LETTERS_JSONL = _docs / "training_data" / "cover_letters.jsonl" -OUTPUT_DIR = _docs / "training_data" / "finetune_output" -GGUF_DIR = _docs / "training_data" / "gguf" -OLLAMA_NAME = f"{_profile.name.split()[0].lower()}-cover-writer" if _profile else "cover-writer" -SYSTEM_PROMPT = ( - f"You are {_profile.name}'s personal cover letter writer. " - f"{_profile.career_summary}" - if _profile else - "You are a professional cover letter writer. Write in first person." -) -``` - -**Step 5: Run existing tests to verify nothing broken** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -# Expected: all existing tests PASS -``` - -**Step 6: Commit** - -```bash -git add scripts/ -git commit -m "feat: extract hard-coded personal references from all scripts via UserProfile" -``` - ---- - -## Task 4: Extract Hard-Coded References — App Pages - -**Files:** -- Modify: `app/Home.py` -- Modify: `app/pages/4_Apply.py` -- Modify: `app/pages/5_Interviews.py` -- Modify: `app/pages/6_Interview_Prep.py` -- Modify: `app/pages/2_Settings.py` - -**Step 1: Add profile loader utility to app pages** - -Add to the top of each modified page (after sys.path insert): -```python -from scripts.user_profile import UserProfile -from scripts.db import DEFAULT_DB - -_USER_YAML = Path(__file__).parent.parent.parent / "config" / "user.yaml" -_profile = UserProfile(_USER_YAML) if UserProfile.exists(_USER_YAML) else None -_name = _profile.name if _profile else "Job Seeker" -``` - -**Step 2: Home.py** - -Replace: -```python -st.title("🔍 Alex's Job Search") -# and: -st.caption(f"Run TF-IDF match scoring against Alex's resume...") -``` -With: -```python -st.title(f"🔍 {_name}'s Job Search") -# and: -st.caption(f"Run TF-IDF match scoring against {_name}'s resume...") -``` - -**Step 3: 4_Apply.py — PDF contact block and DOCS_DIR** - -Replace: -```python -DOCS_DIR = Path("/Library/Documents/JobSearch") -# and the contact paragraph: -Paragraph("ALEX RIVERA", name_style) -Paragraph("alex@example.com · (555) 867-5309 · ...", contact_style) -Paragraph("Warm regards,

Alex Rivera", body_style) -``` -With: -```python -DOCS_DIR = _profile.docs_dir if _profile else Path.home() / "Documents" / "JobSearch" -# and: -display_name = (_profile.name.upper() if _profile else "YOUR NAME") -contact_line = " · ".join(filter(None, [ - _profile.email if _profile else "", - _profile.phone if _profile else "", - _profile.linkedin if _profile else "", -])) -Paragraph(display_name, name_style) -Paragraph(contact_line, contact_style) -Paragraph(f"Warm regards,

{_profile.name if _profile else 'Your Name'}", body_style) -``` - -**Step 4: 5_Interviews.py — email assistant prompt** - -Replace hard-coded persona strings with: -```python -_persona = ( - f"{_name} is a {_profile.career_summary[:120] if _profile and _profile.career_summary else 'professional'}" -) -# Replace all occurrences of "Alex Rivera is a Customer Success..." with _persona -``` - -**Step 5: 6_Interview_Prep.py — interviewer and Q&A prompts** - -Replace all occurrences of `"Alex"` in f-strings with `_name`. - -**Step 6: 2_Settings.py — Services tab** - -Remove `PFP_DIR` and the Claude Code Wrapper / Copilot Wrapper service entries entirely. - -Replace the vLLM service entry's `model_dir` with: -```python -"model_dir": str(_profile.vllm_models_dir) if _profile else str(Path.home() / "models" / "vllm"), -``` - -Replace the SearXNG entry to use Docker Compose instead of a host path: -```python -{ - "name": "SearXNG (company scraper)", - "port": _profile._svc["searxng_port"] if _profile else 8888, - "start": ["docker", "compose", "--profile", "searxng", "up", "-d", "searxng"], - "stop": ["docker", "compose", "stop", "searxng"], - "cwd": str(Path(__file__).parent.parent.parent), - "note": "Privacy-respecting meta-search for company research", -}, -``` - -Replace all caption strings containing "Alex's" with `f"{_name}'s"`. - -**Step 7: Commit** - -```bash -git add app/ -git commit -m "feat: extract hard-coded personal references from all app pages via UserProfile" -``` - ---- - -## Task 5: llm.yaml URL Auto-Generation - -**Files:** -- Modify: `scripts/user_profile.py` (already has `generate_llm_urls()`) -- Modify: `app/pages/2_Settings.py` (My Profile save button) -- Create: `scripts/generate_llm_config.py` - -**Step 1: Write failing test** - -```python -# tests/test_llm_config_generation.py -from pathlib import Path -import tempfile, yaml -from scripts.user_profile import UserProfile -from scripts.generate_llm_config import apply_service_urls - -def test_urls_applied_to_llm_yaml(tmp_path): - user_yaml = tmp_path / "user.yaml" - user_yaml.write_text(yaml.dump({ - "name": "Test", - "services": { - "ollama_host": "myserver", "ollama_port": 11434, "ollama_ssl": False, - "ollama_ssl_verify": True, - "vllm_host": "localhost", "vllm_port": 8000, "vllm_ssl": False, - "vllm_ssl_verify": True, - "searxng_host": "localhost", "searxng_port": 8888, - "searxng_ssl": False, "searxng_ssl_verify": True, - } - })) - llm_yaml = tmp_path / "llm.yaml" - llm_yaml.write_text(yaml.dump({"backends": { - "ollama": {"base_url": "http://old:11434/v1", "type": "openai_compat"}, - "vllm": {"base_url": "http://old:8000/v1", "type": "openai_compat"}, - }})) - - profile = UserProfile(user_yaml) - apply_service_urls(profile, llm_yaml) - - result = yaml.safe_load(llm_yaml.read_text()) - assert result["backends"]["ollama"]["base_url"] == "http://myserver:11434/v1" - assert result["backends"]["vllm"]["base_url"] == "http://localhost:8000/v1" -``` - -**Step 2: Run to verify it fails** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_llm_config_generation.py -v -# Expected: ImportError -``` - -**Step 3: Implement generate_llm_config.py** - -```python -# scripts/generate_llm_config.py -"""Update config/llm.yaml base_url values from the user profile's services block.""" -from pathlib import Path -import yaml -from scripts.user_profile import UserProfile - - -def apply_service_urls(profile: UserProfile, llm_yaml_path: Path) -> None: - """Rewrite base_url for ollama, ollama_research, and vllm backends.""" - if not llm_yaml_path.exists(): - return - cfg = yaml.safe_load(llm_yaml_path.read_text()) or {} - urls = profile.generate_llm_urls() - backends = cfg.get("backends", {}) - for backend_name, url in urls.items(): - if backend_name in backends: - backends[backend_name]["base_url"] = url - cfg["backends"] = backends - llm_yaml_path.write_text(yaml.dump(cfg, default_flow_style=False, allow_unicode=True)) -``` - -**Step 4: Run test to verify it passes** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_llm_config_generation.py -v -# Expected: PASS -``` - -**Step 5: Wire into Settings My Profile save** - -In `app/pages/2_Settings.py`, after the "Save My Profile" button writes `user.yaml`, add: -```python -from scripts.generate_llm_config import apply_service_urls -apply_service_urls(UserProfile(_USER_YAML), LLM_CFG) -st.success("Profile saved and service URLs updated.") -``` - -**Step 6: Commit** - -```bash -git add scripts/generate_llm_config.py tests/test_llm_config_generation.py app/pages/2_Settings.py -git commit -m "feat: auto-generate llm.yaml base_url values from user profile services config" -``` - ---- - -## Task 6: Settings — My Profile Tab - -**Files:** -- Modify: `app/pages/2_Settings.py` - -**Step 1: Add My Profile tab to the tab list** - -Replace the existing `st.tabs(...)` call to add the new tab first: -```python -tab_profile, tab_search, tab_llm, tab_notion, tab_services, tab_resume, tab_email, tab_skills = st.tabs( - ["👤 My Profile", "🔎 Search", "🤖 LLM Backends", "📚 Notion", - "🔌 Services", "📝 Resume Profile", "📧 Email", "🏷️ Skills"] -) -``` - -**Step 2: Implement the My Profile tab** - -```python -USER_CFG = CONFIG_DIR / "user.yaml" - -with tab_profile: - from scripts.user_profile import UserProfile, _DEFAULTS - import yaml as _yaml - - st.caption("Your identity and service configuration. Saved values drive all LLM prompts, PDF headers, and service connections.") - - _u = _yaml.safe_load(USER_CFG.read_text()) or {} if USER_CFG.exists() else {} - _svc = {**_DEFAULTS["services"], **_u.get("services", {})} - - with st.expander("👤 Identity", expanded=True): - c1, c2 = st.columns(2) - u_name = c1.text_input("Full Name", _u.get("name", "")) - u_email = c1.text_input("Email", _u.get("email", "")) - u_phone = c2.text_input("Phone", _u.get("phone", "")) - u_linkedin = c2.text_input("LinkedIn URL", _u.get("linkedin", "")) - u_summary = st.text_area("Career Summary (used in LLM prompts)", - _u.get("career_summary", ""), height=100) - - with st.expander("🔒 Sensitive Employers (NDA)"): - st.caption("Companies listed here appear as 'previous employer (NDA)' in research briefs.") - nda_list = list(_u.get("nda_companies", [])) - nda_cols = st.columns(max(len(nda_list), 1)) - _to_remove = None - for i, company in enumerate(nda_list): - if nda_cols[i % len(nda_cols)].button(f"× {company}", key=f"rm_nda_{company}"): - _to_remove = company - if _to_remove: - nda_list.remove(_to_remove) - nc, nb = st.columns([4, 1]) - new_nda = nc.text_input("Add employer", key="new_nda", label_visibility="collapsed", placeholder="Employer name…") - if nb.button("＋ Add", key="add_nda") and new_nda.strip(): - nda_list.append(new_nda.strip()) - - with st.expander("📁 File Paths"): - u_docs = st.text_input("Documents directory", _u.get("docs_dir", "~/Documents/JobSearch")) - u_ollama = st.text_input("Ollama models directory", _u.get("ollama_models_dir", "~/models/ollama")) - u_vllm = st.text_input("vLLM models directory", _u.get("vllm_models_dir", "~/models/vllm")) - - with st.expander("⚙️ Inference Profile"): - profiles = ["remote", "cpu", "single-gpu", "dual-gpu"] - u_profile = st.selectbox("Active profile", profiles, - index=profiles.index(_u.get("inference_profile", "remote"))) - - with st.expander("🔌 Service Ports & Hosts"): - st.caption("Advanced — change only if services run on non-default ports or remote hosts.") - sc1, sc2, sc3 = st.columns(3) - with sc1: - st.markdown("**Ollama**") - svc_ollama_host = st.text_input("Host##ollama", _svc["ollama_host"], key="svc_ollama_host") - svc_ollama_port = st.number_input("Port##ollama", value=_svc["ollama_port"], key="svc_ollama_port") - svc_ollama_ssl = st.checkbox("SSL##ollama", _svc["ollama_ssl"], key="svc_ollama_ssl") - svc_ollama_verify = st.checkbox("Verify cert##ollama", _svc["ollama_ssl_verify"], key="svc_ollama_verify") - with sc2: - st.markdown("**vLLM**") - svc_vllm_host = st.text_input("Host##vllm", _svc["vllm_host"], key="svc_vllm_host") - svc_vllm_port = st.number_input("Port##vllm", value=_svc["vllm_port"], key="svc_vllm_port") - svc_vllm_ssl = st.checkbox("SSL##vllm", _svc["vllm_ssl"], key="svc_vllm_ssl") - svc_vllm_verify = st.checkbox("Verify cert##vllm", _svc["vllm_ssl_verify"], key="svc_vllm_verify") - with sc3: - st.markdown("**SearXNG**") - svc_sxng_host = st.text_input("Host##sxng", _svc["searxng_host"], key="svc_sxng_host") - svc_sxng_port = st.number_input("Port##sxng", value=_svc["searxng_port"], key="svc_sxng_port") - svc_sxng_ssl = st.checkbox("SSL##sxng", _svc["searxng_ssl"], key="svc_sxng_ssl") - svc_sxng_verify = st.checkbox("Verify cert##sxng", _svc["searxng_ssl_verify"], key="svc_sxng_verify") - - if st.button("💾 Save Profile", type="primary", key="save_user_profile"): - new_data = { - "name": u_name, "email": u_email, "phone": u_phone, - "linkedin": u_linkedin, "career_summary": u_summary, - "nda_companies": nda_list, - "docs_dir": u_docs, "ollama_models_dir": u_ollama, "vllm_models_dir": u_vllm, - "inference_profile": u_profile, - "services": { - "streamlit_port": _svc["streamlit_port"], - "ollama_host": svc_ollama_host, "ollama_port": int(svc_ollama_port), - "ollama_ssl": svc_ollama_ssl, "ollama_ssl_verify": svc_ollama_verify, - "vllm_host": svc_vllm_host, "vllm_port": int(svc_vllm_port), - "vllm_ssl": svc_vllm_ssl, "vllm_ssl_verify": svc_vllm_verify, - "searxng_host": svc_sxng_host, "searxng_port": int(svc_sxng_port), - "searxng_ssl": svc_sxng_ssl, "searxng_ssl_verify": svc_sxng_verify, - } - } - save_yaml(USER_CFG, new_data) - from scripts.user_profile import UserProfile - from scripts.generate_llm_config import apply_service_urls - apply_service_urls(UserProfile(USER_CFG), LLM_CFG) - st.success("Profile saved and service URLs updated.") -``` - -**Step 2: Commit** - -```bash -git add app/pages/2_Settings.py -git commit -m "feat: add My Profile tab to Settings with full user.yaml editing + URL auto-generation" -``` - ---- - -## Task 7: First-Run Wizard - -**Files:** -- Create: `app/pages/0_Setup.py` -- Modify: `app/app.py` - -**Step 1: Create the wizard page** - -```python -# app/pages/0_Setup.py -""" -First-run setup wizard — shown by app.py when config/user.yaml is absent. -Five steps: hardware detection → identity → NDA companies → inference/keys → Notion. -Writes config/user.yaml (and optionally config/notion.yaml) on completion. -""" -import subprocess -import sys -from pathlib import Path -sys.path.insert(0, str(Path(__file__).parent.parent.parent)) - -import streamlit as st -import yaml - -CONFIG_DIR = Path(__file__).parent.parent.parent / "config" -USER_CFG = CONFIG_DIR / "user.yaml" -NOTION_CFG = CONFIG_DIR / "notion.yaml" -LLM_CFG = CONFIG_DIR / "llm.yaml" - -PROFILES = ["remote", "cpu", "single-gpu", "dual-gpu"] - -def _detect_gpus() -> list[str]: - """Return list of GPU names via nvidia-smi, or [] if none.""" - try: - out = subprocess.check_output( - ["nvidia-smi", "--query-gpu=name", "--format=csv,noheader"], - text=True, timeout=5 - ) - return [l.strip() for l in out.strip().splitlines() if l.strip()] - except Exception: - return [] - -def _suggest_profile(gpus: list[str]) -> str: - if len(gpus) >= 2: - return "dual-gpu" - if len(gpus) == 1: - return "single-gpu" - return "remote" - -# ── Wizard state ────────────────────────────────────────────────────────────── -if "wizard_step" not in st.session_state: - st.session_state.wizard_step = 1 -if "wizard_data" not in st.session_state: - st.session_state.wizard_data = {} - -step = st.session_state.wizard_step -data = st.session_state.wizard_data - -st.title("👋 Welcome to Job Seeker") -st.caption("Let's get you set up. This takes about 2 minutes.") -st.progress(step / 5, text=f"Step {step} of 5") -st.divider() - -# ── Step 1: Hardware detection ──────────────────────────────────────────────── -if step == 1: - st.subheader("Step 1 — Hardware Detection") - gpus = _detect_gpus() - suggested = _suggest_profile(gpus) - - if gpus: - st.success(f"Found {len(gpus)} GPU(s): {', '.join(gpus)}") - else: - st.info("No NVIDIA GPUs detected. Remote or CPU mode recommended.") - - profile = st.selectbox( - "Inference mode", - PROFILES, - index=PROFILES.index(suggested), - help="This controls which Docker services start. You can change it later in Settings → My Profile.", - ) - if profile in ("single-gpu", "dual-gpu") and not gpus: - st.warning("No GPUs detected — GPU profiles require NVIDIA Container Toolkit. See the README for install instructions.") - - if st.button("Next →", type="primary"): - data["inference_profile"] = profile - data["gpus_detected"] = gpus - st.session_state.wizard_step = 2 - st.rerun() - -# ── Step 2: Identity ────────────────────────────────────────────────────────── -elif step == 2: - st.subheader("Step 2 — Your Identity") - st.caption("Used in cover letter PDFs, LLM prompts, and the app header.") - c1, c2 = st.columns(2) - name = c1.text_input("Full Name *", data.get("name", "")) - email = c1.text_input("Email *", data.get("email", "")) - phone = c2.text_input("Phone", data.get("phone", "")) - linkedin = c2.text_input("LinkedIn URL", data.get("linkedin", "")) - summary = st.text_area( - "Career Summary *", - data.get("career_summary", ""), - height=120, - placeholder="Experienced professional with X years in [field]. Specialise in [skills].", - help="This paragraph is injected into cover letter and research prompts as your professional context.", - ) - - col_back, col_next = st.columns([1, 4]) - if col_back.button("← Back"): - st.session_state.wizard_step = 1 - st.rerun() - if col_next.button("Next →", type="primary"): - if not name or not email or not summary: - st.error("Name, email, and career summary are required.") - else: - data.update({"name": name, "email": email, "phone": phone, - "linkedin": linkedin, "career_summary": summary}) - st.session_state.wizard_step = 3 - st.rerun() - -# ── Step 3: NDA Companies ───────────────────────────────────────────────────── -elif step == 3: - st.subheader("Step 3 — Sensitive Employers (Optional)") - st.caption( - "Previous employers listed here will appear as 'previous employer (NDA)' in " - "research briefs and talking points. Skip if not applicable." - ) - nda_list = list(data.get("nda_companies", [])) - if nda_list: - cols = st.columns(min(len(nda_list), 5)) - to_remove = None - for i, c in enumerate(nda_list): - if cols[i % 5].button(f"× {c}", key=f"rm_{c}"): - to_remove = c - if to_remove: - nda_list.remove(to_remove) - data["nda_companies"] = nda_list - st.rerun() - nc, nb = st.columns([4, 1]) - new_c = nc.text_input("Add employer", key="new_nda_wiz", label_visibility="collapsed", placeholder="Employer name…") - if nb.button("＋ Add") and new_c.strip(): - nda_list.append(new_c.strip()) - data["nda_companies"] = nda_list - st.rerun() - - col_back, col_skip, col_next = st.columns([1, 1, 3]) - if col_back.button("← Back"): - st.session_state.wizard_step = 2 - st.rerun() - if col_skip.button("Skip"): - data.setdefault("nda_companies", []) - st.session_state.wizard_step = 4 - st.rerun() - if col_next.button("Next →", type="primary"): - data["nda_companies"] = nda_list - st.session_state.wizard_step = 4 - st.rerun() - -# ── Step 4: Inference & API Keys ────────────────────────────────────────────── -elif step == 4: - profile = data.get("inference_profile", "remote") - st.subheader("Step 4 — Inference & API Keys") - - if profile == "remote": - st.info("Remote mode: LLM calls go to external APIs. At least one key is needed.") - anthropic_key = st.text_input("Anthropic API Key", type="password", - placeholder="sk-ant-…") - openai_url = st.text_input("OpenAI-compatible endpoint (optional)", - placeholder="https://api.together.xyz/v1") - openai_key = st.text_input("Endpoint API Key (optional)", type="password") if openai_url else "" - data.update({"anthropic_key": anthropic_key, "openai_url": openai_url, "openai_key": openai_key}) - else: - st.info(f"Local mode ({profile}): Ollama handles cover letters. Configure model below.") - ollama_model = st.text_input("Cover letter model name", - data.get("ollama_model", "llama3.2:3b"), - help="This model will be pulled by Ollama on first start.") - data["ollama_model"] = ollama_model - - st.divider() - with st.expander("Advanced — Service Ports & Hosts"): - st.caption("Change only if services run on non-default ports or remote hosts.") - svc = data.get("services", {}) - for svc_name, default_host, default_port in [ - ("ollama", "localhost", 11434), - ("vllm", "localhost", 8000), - ("searxng","localhost", 8888), - ]: - c1, c2, c3, c4 = st.columns([2, 1, 0.5, 0.5]) - svc[f"{svc_name}_host"] = c1.text_input(f"{svc_name} host", svc.get(f"{svc_name}_host", default_host), key=f"adv_{svc_name}_host") - svc[f"{svc_name}_port"] = c2.number_input(f"port", value=svc.get(f"{svc_name}_port", default_port), key=f"adv_{svc_name}_port") - svc[f"{svc_name}_ssl"] = c3.checkbox("SSL", svc.get(f"{svc_name}_ssl", False), key=f"adv_{svc_name}_ssl") - svc[f"{svc_name}_ssl_verify"] = c4.checkbox("Verify", svc.get(f"{svc_name}_ssl_verify", True), key=f"adv_{svc_name}_verify") - data["services"] = svc - - col_back, col_next = st.columns([1, 4]) - if col_back.button("← Back"): - st.session_state.wizard_step = 3 - st.rerun() - if col_next.button("Next →", type="primary"): - st.session_state.wizard_step = 5 - st.rerun() - -# ── Step 5: Notion (optional) ───────────────────────────────────────────────── -elif step == 5: - st.subheader("Step 5 — Notion Sync (Optional)") - st.caption("Syncs approved and applied jobs to a Notion database. Skip if not using Notion.") - notion_token = st.text_input("Integration Token", type="password", placeholder="secret_…") - notion_db = st.text_input("Database ID", placeholder="32-character ID from Notion URL") - - if notion_token and notion_db: - if st.button("🔌 Test connection"): - with st.spinner("Connecting…"): - try: - from notion_client import Client - db = Client(auth=notion_token).databases.retrieve(notion_db) - st.success(f"Connected: {db['title'][0]['plain_text']}") - except Exception as e: - st.error(f"Connection failed: {e}") - - col_back, col_skip, col_finish = st.columns([1, 1, 3]) - if col_back.button("← Back"): - st.session_state.wizard_step = 4 - st.rerun() - - def _finish(save_notion: bool): - # Build user.yaml - svc_defaults = { - "streamlit_port": 8501, - "ollama_host": "localhost", "ollama_port": 11434, "ollama_ssl": False, "ollama_ssl_verify": True, - "vllm_host": "localhost", "vllm_port": 8000, "vllm_ssl": False, "vllm_ssl_verify": True, - "searxng_host":"localhost", "searxng_port": 8888, "searxng_ssl":False, "searxng_ssl_verify": True, - } - svc_defaults.update(data.get("services", {})) - user_data = { - "name": data.get("name", ""), - "email": data.get("email", ""), - "phone": data.get("phone", ""), - "linkedin": data.get("linkedin", ""), - "career_summary": data.get("career_summary", ""), - "nda_companies": data.get("nda_companies", []), - "docs_dir": "~/Documents/JobSearch", - "ollama_models_dir":"~/models/ollama", - "vllm_models_dir": "~/models/vllm", - "inference_profile":data.get("inference_profile", "remote"), - "services": svc_defaults, - } - CONFIG_DIR.mkdir(parents=True, exist_ok=True) - USER_CFG.write_text(yaml.dump(user_data, default_flow_style=False, allow_unicode=True)) - - # Update llm.yaml URLs - if LLM_CFG.exists(): - from scripts.user_profile import UserProfile - from scripts.generate_llm_config import apply_service_urls - apply_service_urls(UserProfile(USER_CFG), LLM_CFG) - - # Optionally write notion.yaml - if save_notion and notion_token and notion_db: - NOTION_CFG.write_text(yaml.dump({"token": notion_token, "database_id": notion_db})) - - st.session_state.wizard_step = 1 - st.session_state.wizard_data = {} - st.success("Setup complete! Redirecting…") - st.rerun() - - if col_skip.button("Skip & Finish"): - _finish(save_notion=False) - if col_finish.button("💾 Save & Finish", type="primary"): - _finish(save_notion=True) -``` - -**Step 2: Gate navigation in app.py** - -In `app/app.py`, after `init_db()`, add: -```python -from scripts.user_profile import UserProfile - -_USER_YAML = Path(__file__).parent.parent / "config" / "user.yaml" - -if not UserProfile.exists(_USER_YAML): - # Show wizard only — no nav, no sidebar tasks - setup_page = st.Page("pages/0_Setup.py", title="Setup", icon="👋") - st.navigation({"": [setup_page]}).run() - st.stop() -``` - -This must appear before the normal `st.navigation(pages)` call. - -**Step 3: Commit** - -```bash -git add app/pages/0_Setup.py app/app.py -git commit -m "feat: first-run setup wizard gates app until user.yaml is created" -``` - ---- - -## Task 8: Docker Compose Stack - -**Files:** -- Create: `Dockerfile` -- Create: `compose.yml` -- Create: `docker/searxng/settings.yml` -- Create: `docker/ollama/entrypoint.sh` -- Create: `.dockerignore` -- Create: `.env.example` - -**Step 1: Dockerfile** - -```dockerfile -# Dockerfile -FROM python:3.11-slim - -WORKDIR /app - -# System deps for companyScraper (beautifulsoup4, fake-useragent, lxml) -RUN apt-get update && apt-get install -y --no-install-recommends \ - gcc libffi-dev curl \ - && rm -rf /var/lib/apt/lists/* - -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt - -# Bundle companyScraper -COPY scrapers/ /app/scrapers/ - -COPY . . - -EXPOSE 8501 - -CMD ["streamlit", "run", "app/app.py", \ - "--server.port=8501", \ - "--server.headless=true", \ - "--server.fileWatcherType=none"] -``` - -**Step 2: compose.yml** - -```yaml -# compose.yml -services: - - app: - build: . - ports: - - "${STREAMLIT_PORT:-8501}:8501" - volumes: - - ./config:/app/config - - ./data:/app/data - - ${DOCS_DIR:-~/Documents/JobSearch}:/docs - environment: - - STAGING_DB=/app/data/staging.db - depends_on: - searxng: - condition: service_healthy - restart: unless-stopped - - searxng: - image: searxng/searxng:latest - ports: - - "${SEARXNG_PORT:-8888}:8080" - volumes: - - ./docker/searxng:/etc/searxng:ro - healthcheck: - test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/"] - interval: 10s - timeout: 5s - retries: 3 - restart: unless-stopped - - ollama: - image: ollama/ollama:latest - ports: - - "${OLLAMA_PORT:-11434}:11434" - volumes: - - ${OLLAMA_MODELS_DIR:-~/models/ollama}:/root/.ollama - - ./docker/ollama/entrypoint.sh:/entrypoint.sh - environment: - - OLLAMA_MODELS=/root/.ollama - entrypoint: ["/bin/bash", "/entrypoint.sh"] - profiles: [cpu, single-gpu, dual-gpu] - restart: unless-stopped - - ollama-gpu: - extends: - service: ollama - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["0"] - capabilities: [gpu] - profiles: [single-gpu, dual-gpu] - - vllm: - image: vllm/vllm-openai:latest - ports: - - "${VLLM_PORT:-8000}:8000" - volumes: - - ${VLLM_MODELS_DIR:-~/models/vllm}:/models - command: > - --model /models/${VLLM_MODEL:-Ouro-1.4B} - --trust-remote-code - --max-model-len 4096 - --gpu-memory-utilization 0.75 - --enforce-eager - --max-num-seqs 8 - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["1"] - capabilities: [gpu] - profiles: [dual-gpu] - restart: unless-stopped -``` - -**Step 3: SearXNG settings.yml** - -```yaml -# docker/searxng/settings.yml -use_default_settings: true -search: - formats: - - html - - json -server: - secret_key: "change-me-in-production" - bind_address: "0.0.0.0:8080" -``` - -**Step 4: Ollama entrypoint** - -```bash -#!/usr/bin/env bash -# docker/ollama/entrypoint.sh -# Start Ollama server and pull a default model if none are present -ollama serve & -sleep 5 -if [ -z "$(ollama list 2>/dev/null | tail -n +2)" ]; then - MODEL="${DEFAULT_OLLAMA_MODEL:-llama3.2:3b}" - echo "No models found — pulling $MODEL..." - ollama pull "$MODEL" -fi -wait -``` - -**Step 5: .env.example** - -```bash -# .env.example — copy to .env (auto-generated by wizard, or fill manually) -STREAMLIT_PORT=8501 -OLLAMA_PORT=11434 -VLLM_PORT=8000 -SEARXNG_PORT=8888 -DOCS_DIR=~/Documents/JobSearch -OLLAMA_MODELS_DIR=~/models/ollama -VLLM_MODELS_DIR=~/models/vllm -VLLM_MODEL=Ouro-1.4B -``` - -**Step 6: .dockerignore** - -``` -.git -__pycache__ -*.pyc -staging.db -config/user.yaml -config/notion.yaml -config/email.yaml -config/tokens.yaml -.streamlit.pid -.streamlit.log -aihawk/ -docs/ -tests/ -``` - -**Step 7: Update .gitignore** - -Add to `.gitignore`: -``` -.env -config/user.yaml -data/ -``` - -**Step 8: Commit** - -```bash -git add Dockerfile compose.yml docker/ .dockerignore .env.example -git commit -m "feat: add Docker Compose stack with remote/cpu/single-gpu/dual-gpu profiles" -``` - ---- - -## Task 9: Services Tab — Compose-Driven Start/Stop - -**Files:** -- Modify: `app/pages/2_Settings.py` - -**Step 1: Replace SERVICES list with compose-driven definitions** - -```python -COMPOSE_DIR = str(Path(__file__).parent.parent.parent) -_profile_name = _profile.inference_profile if _profile else "remote" - -SERVICES = [ - { - "name": "Streamlit UI", - "port": _profile._svc["streamlit_port"] if _profile else 8501, - "start": ["docker", "compose", "--profile", _profile_name, "up", "-d", "app"], - "stop": ["docker", "compose", "stop", "app"], - "cwd": COMPOSE_DIR, - "note": "Job Seeker web interface", - }, - { - "name": "Ollama (local LLM)", - "port": _profile._svc["ollama_port"] if _profile else 11434, - "start": ["docker", "compose", "--profile", _profile_name, "up", "-d", "ollama"], - "stop": ["docker", "compose", "stop", "ollama"], - "cwd": COMPOSE_DIR, - "note": f"Local inference engine — profile: {_profile_name}", - "hidden": _profile_name == "remote", - }, - { - "name": "vLLM Server", - "port": _profile._svc["vllm_port"] if _profile else 8000, - "start": ["docker", "compose", "--profile", _profile_name, "up", "-d", "vllm"], - "stop": ["docker", "compose", "stop", "vllm"], - "cwd": COMPOSE_DIR, - "model_dir": str(_profile.vllm_models_dir) if _profile else str(Path.home() / "models" / "vllm"), - "note": "vLLM inference — dual-gpu profile only", - "hidden": _profile_name != "dual-gpu", - }, - { - "name": "SearXNG (company scraper)", - "port": _profile._svc["searxng_port"] if _profile else 8888, - "start": ["docker", "compose", "up", "-d", "searxng"], - "stop": ["docker", "compose", "stop", "searxng"], - "cwd": COMPOSE_DIR, - "note": "Privacy-respecting meta-search for company research", - }, -] -# Filter hidden services -SERVICES = [s for s in SERVICES if not s.get("hidden")] -``` - -**Step 2: Update health checks to use SSL** - -Replace the `_port_open()` helper: -```python -def _port_open(port: int, host: str = "127.0.0.1", - ssl: bool = False, verify: bool = True) -> bool: - try: - import requests as _r - scheme = "https" if ssl else "http" - _r.get(f"{scheme}://{host}:{port}/", timeout=1, verify=verify) - return True - except Exception: - return False -``` - -Update each service health check call to pass host/ssl/verify from the profile. - -**Step 3: Commit** - -```bash -git add app/pages/2_Settings.py -git commit -m "feat: services tab uses docker compose commands and SSL-aware health checks" -``` - ---- - -## Task 10: Fine-Tune Wizard Tab - -**Files:** -- Modify: `app/pages/2_Settings.py` - -**Step 1: Add fine-tune tab (GPU profiles only)** - -Add `tab_finetune` to the tab list (shown only when profile is single-gpu or dual-gpu). - -```python -# In the tab definition, add conditionally: -_show_finetune = _profile and _profile.inference_profile in ("single-gpu", "dual-gpu") - -# Add tab: -tab_finetune = st.tabs([..., "🎯 Fine-Tune"])[last_index] if _show_finetune else None -``` - -**Step 2: Implement the fine-tune tab** - -```python -if _show_finetune and tab_finetune: - with tab_finetune: - st.subheader("Fine-Tune Your Cover Letter Model") - st.caption( - "Upload your existing cover letters to train a personalised writing model. " - "Requires a GPU. The base model is used until fine-tuning completes." - ) - - step = st.session_state.get("ft_step", 1) - - if step == 1: - st.markdown("**Step 1: Upload Cover Letters**") - uploaded = st.file_uploader( - "Upload cover letters (PDF, DOCX, or TXT)", - type=["pdf", "docx", "txt"], - accept_multiple_files=True, - ) - if uploaded and st.button("Extract Training Pairs →", type="primary"): - # Save uploads to docs_dir/training_data/uploads/ - upload_dir = (_profile.docs_dir / "training_data" / "uploads") - upload_dir.mkdir(parents=True, exist_ok=True) - for f in uploaded: - (upload_dir / f.name).write_bytes(f.read()) - st.session_state.ft_step = 2 - st.rerun() - - elif step == 2: - st.markdown("**Step 2: Preview Training Pairs**") - st.info("Run `python scripts/prepare_training_data.py` to extract pairs, then return here.") - jsonl_path = _profile.docs_dir / "training_data" / "cover_letters.jsonl" - if jsonl_path.exists(): - import json - pairs = [json.loads(l) for l in jsonl_path.read_text().splitlines() if l.strip()] - st.caption(f"{len(pairs)} training pairs extracted.") - for i, p in enumerate(pairs[:3]): - with st.expander(f"Pair {i+1}"): - st.text(p.get("input", "")[:300]) - col_back, col_next = st.columns([1, 4]) - if col_back.button("← Back"): - st.session_state.ft_step = 1; st.rerun() - if col_next.button("Start Training →", type="primary"): - st.session_state.ft_step = 3; st.rerun() - - elif step == 3: - st.markdown("**Step 3: Train**") - epochs = st.slider("Epochs", 3, 20, 10) - if st.button("🚀 Start Fine-Tune", type="primary"): - from scripts.task_runner import submit_task - from scripts.db import DEFAULT_DB - # finetune task type — extend task_runner for this - st.info("Fine-tune queued as a background task. Check back in 30–60 minutes.") - if col_back := st.button("← Back"): - st.session_state.ft_step = 2; st.rerun() -else: - if tab_finetune is None and _profile: - with st.expander("🎯 Fine-Tune (GPU only)"): - st.info( - f"Fine-tuning requires a GPU profile. " - f"Current profile: `{_profile.inference_profile}`. " - "Change it in My Profile to enable this tab." - ) -``` - -**Step 3: Commit** - -```bash -git add app/pages/2_Settings.py -git commit -m "feat: add fine-tune wizard tab to Settings (GPU profiles only)" -``` - ---- - -## Task 11: Final Wiring, Tests & README - -**Files:** -- Create: `README.md` -- Create: `requirements.txt` (Docker-friendly, no torch/CUDA) -- Modify: `tests/` (smoke test wizard gating) - -**Step 1: Write a smoke test for wizard gating** - -```python -# tests/test_app_gating.py -from pathlib import Path -from scripts.user_profile import UserProfile - -def test_wizard_gating_logic(tmp_path): - """app.py should show wizard when user.yaml is absent.""" - missing = tmp_path / "user.yaml" - assert not UserProfile.exists(missing) - -def test_wizard_gating_passes_after_setup(tmp_path): - import yaml - p = tmp_path / "user.yaml" - p.write_text(yaml.dump({"name": "Test User", "services": {}})) - assert UserProfile.exists(p) -``` - -**Step 2: Create requirements.txt** - -``` -streamlit>=1.45 -pyyaml>=6.0 -requests>=2.31 -reportlab>=4.0 -jobspy>=1.1 -notion-client>=2.2 -anthropic>=0.34 -openai>=1.40 -beautifulsoup4>=4.12 -fake-useragent>=1.5 -imaplib2>=3.6 -``` - -**Step 3: Create README.md** - -Document: quick start (`git clone → docker compose --profile remote up -d`), profile options, first-run wizard, and how to configure each inference mode. - -**Step 4: Run full test suite** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -# Expected: all PASS -``` - -**Step 5: Final commit** - -```bash -git add README.md requirements.txt tests/ -git commit -m "feat: complete generalization — wizard, UserProfile, compose stack, all personal refs extracted" -``` - ---- - -## Execution Checklist - -- [ ] Task 1: Bootstrap new repo -- [ ] Task 2: UserProfile class + tests -- [ ] Task 3: Extract references — scripts -- [ ] Task 4: Extract references — app pages -- [ ] Task 5: llm.yaml URL auto-generation -- [ ] Task 6: My Profile tab in Settings -- [ ] Task 7: First-run wizard -- [ ] Task 8: Docker Compose stack -- [ ] Task 9: Services tab — compose-driven -- [ ] Task 10: Fine-tune wizard tab -- [ ] Task 11: Final wiring, tests, README diff --git a/docs/plans/2026-02-24-monetization-business-plan.md b/docs/plans/2026-02-24-monetization-business-plan.md deleted file mode 100644 index f37c1e8..0000000 --- a/docs/plans/2026-02-24-monetization-business-plan.md +++ /dev/null @@ -1,474 +0,0 @@ -# Job Seeker Platform — Monetization Business Plan - -**Date:** 2026-02-24 -**Status:** Draft — pre-VC pitch -**Author:** Brainstorming session - ---- - -## 1. Product Overview - -An automated job discovery, resume matching, and application pipeline platform. Built originally as a personal tool for a single job seeker; architecture is already generalized — user identity, preferences, and data are fully parameterized via onboarding, not hardcoded. - -### Core pipeline -``` -Job Discovery (multi-board) → Resume Matching → Job Review UI -→ Apply Workspace (cover letter + PDF) -→ Interviews Kanban (phone_screen → offer → hired) -→ Notion Sync -``` - -### Key feature surface -- Multi-board job discovery (LinkedIn, Indeed, Glassdoor, ZipRecruiter, Google, Adzuna, The Ladders) -- LinkedIn Alert email ingestion + email classifier (interview requests, rejections, surveys) -- Resume keyword matching + match scoring -- AI cover letter generation (local model, shared hosted model, or cloud LLM) -- Company research briefs (web scrape + LLM synthesis) -- Interview prep + practice Q&A -- Culture-fit survey assistant with vision/screenshot support -- Application pipeline kanban with stage tracking -- Notion sync for external tracking -- Mission alignment + accessibility preferences (personal decision-making only) -- Per-user fine-tuned cover letter model (trained on user's own writing corpus) - ---- - -## 2. Target Market - -### Primary: Individual job seekers (B2C) -- Actively searching, technically comfortable, value privacy -- Frustrated by manual tracking (spreadsheets, Notion boards) -- Want AI-assisted applications without giving their data to a third party -- Typical job search duration: 3–6 months → average subscription length ~4.5 months - -### Secondary: Career coaches (B2B, seat-based) -- Manage 10–20 active clients simultaneously -- High willingness to pay for tools that make their service more efficient -- **20× revenue multiplier** vs. solo users (base + per-seat pricing) - -### Tertiary: Outplacement firms / staffing agencies (B2B enterprise) -- Future expansion; validates product-market fit at coach tier first - ---- - -## 3. Distribution Model - -### Starting point: Local-first (self-hosted) - -Users run the application on their own machine via Docker Compose or a native installer. All job data, resume data, and preferences stay local. AI features are optional and configurable — users can use their own LLM backends or subscribe for hosted AI. - -**Why local-first:** -- Zero infrastructure cost per free user -- Strong privacy story (no job search data on your servers) -- Reversible — easy to add a hosted SaaS path later without a rewrite -- Aligns with the open core licensing model - -### Future path: Cloud Edition (SaaS) - -Same codebase deployed as a hosted service. Users sign up at a URL, no install required. Unlocked when revenue and user feedback validate the market. - -**Architecture readiness:** The config layer, per-user data isolation, and SQLite-per-user design already support multi-tenancy with minimal refactoring. SaaS is a deployment mode, not a rewrite. - ---- - -## 4. Licensing Strategy - -### Open Core - -| Component | License | Rationale | -|---|---|---| -| Job discovery pipeline | MIT | Community maintains scrapers (boards break constantly) | -| SQLite schema + `db.py` | MIT | Interoperability, trust | -| Application pipeline state machine | MIT | Core value is visible, auditable | -| Streamlit UI shell | MIT | Community contributions, forks welcome | -| AI cover letter generation | BSL 1.1 | Proprietary prompt engineering + model routing | -| Company research synthesis | BSL 1.1 | LLM orchestration is the moat | -| Interview prep + practice Q&A | BSL 1.1 | Premium feature | -| Survey assistant (vision) | BSL 1.1 | Premium feature | -| Email classifier | BSL 1.1 | Premium feature | -| Notion sync | BSL 1.1 | Integration layer | -| Team / multi-user features | Proprietary | Future enterprise feature | -| Analytics dashboard | Proprietary | Future feature | -| Fine-tuned model weights | Proprietary | Per-user, not redistributable | - -**Business Source License (BSL 1.1):** Code is visible and auditable on GitHub. Free for personal, non-commercial self-hosting. Commercial use or SaaS re-hosting requires a paid license. Converts to MIT after 4 years. Used by HashiCorp (Vault, Terraform), MariaDB, and others — well understood by the VC community. - -**Why this works here:** The value is not in the code. A competitor could clone the repo and still not have: the fine-tuned model, the user's corpus, the orchestration prompts, or the UX polish. The moat is the system, not any individual file. - ---- - -## 5. Tier Structure - -### Free — $0/mo -Self-hosted, local-only. Genuinely useful as a privacy-respecting job tracker. - -| Feature | Included | -|---|---| -| Multi-board job discovery | ✓ | -| Custom board scrapers (Adzuna, The Ladders) | ✓ | -| LinkedIn Alert email ingestion | ✓ | -| Add jobs by URL | ✓ | -| Resume keyword matching | ✓ | -| Cover letter generation (local Ollama only) | ✓ | -| Application pipeline kanban | ✓ | -| Mission alignment + accessibility preferences | ✓ | -| Search profiles | 1 | -| AI backend | User's local Ollama | -| Support | Community (GitHub Discussions) | - -**Purpose:** Acquisition engine. GitHub stars = distribution. Users who get a job on free tier refer friends. - ---- - -### Paid — $12/mo -For job seekers who want quality AI output without GPU setup or API key management. - -Includes everything in Free, plus: - -| Feature | Included | -|---|---| -| Shared hosted fine-tuned cover letter model | ✓ | -| Claude API (BYOK — bring your own key) | ✓ | -| Company research briefs | ✓ | -| Interview prep + practice Q&A | ✓ | -| Survey assistant (vision/screenshot) | ✓ | -| Search criteria LLM suggestions | ✓ | -| Email classifier | ✓ | -| Notion sync | ✓ | -| Search profiles | 5 | -| Support | Email | - -**Purpose:** Primary revenue tier. High margin, low support burden. Targets the individual job seeker who wants "it just works." - ---- - -### Premium — $29/mo -For power users and career coaches who want best-in-class output and personal model training. - -Includes everything in Paid, plus: - -| Feature | Included | -|---|---| -| Claude Sonnet (your hosted key, 150 ops/mo included) | ✓ | -| Per-user fine-tuned model (trained on their corpus) | ✓ (one-time onboarding) | -| Corpus re-training | ✓ (quarterly) | -| Search profiles | Unlimited | -| Multi-user / coach mode | ✓ (+$15/seat) | -| Shared job pool across seats | ✓ | -| Priority support + onboarding call | ✓ | - -**Purpose:** Highest LTV tier. Coach accounts at 3+ seats generate $59–$239/mo each. Fine-tuned personal model is a high-perceived-value differentiator that costs ~$0.50 to produce. - ---- - -## 6. AI Inference — Claude API Cost Model - -Pricing basis: Haiku 4.5 = $0.80/MTok in · $4/MTok out | Sonnet 4.6 = $3/MTok in · $15/MTok out - -### Per-operation costs - -| Operation | Tokens In | Tokens Out | Haiku | Sonnet | -|---|---|---|---|---| -| Cover letter generation | ~2,400 | ~400 | $0.0035 | $0.013 | -| Company research brief | ~3,000 | ~800 | $0.0056 | $0.021 | -| Survey Q&A (5 questions) | ~3,000 | ~1,500 | $0.0084 | $0.031 | -| Job description enrichment | ~800 | ~300 | $0.0018 | $0.007 | -| Search criteria suggestion | ~400 | ~200 | $0.0010 | $0.004 | - -### Monthly inference cost per active user -Assumptions: 12 cover letters, 3 research briefs, 2 surveys, 40 enrichments, 2 search suggestions - -| Backend mix | Cost/user/mo | -|---|---| -| Haiku only (paid tier) | ~$0.15 | -| Sonnet only | ~$0.57 | -| Mixed: Sonnet for CL + research, Haiku for rest (premium tier) | ~$0.31 | - -### Per-user fine-tuning cost (premium, one-time) -| Provider | Cost | -|---|---| -| User's local GPU | $0 | -| RunPod A100 (~20 min) | $0.25–$0.40 | -| Together AI / Replicate | $0.50–$0.75 | -| Quarterly re-train | Same as above | - -**Amortized over 12 months:** ~$0.04–$0.06/user/mo - ---- - -## 7. Full Infrastructure Cost Model - -Local-first architecture means most compute runs on the user's machine. Your infra is limited to: AI inference API calls, shared model serving, fine-tune jobs, license/auth server, and storage for model artifacts. - -### Monthly infrastructure at 100K users -(4% paid conversion = 4,000 paid; 20% of paid premium = 800 premium) - -| Cost center | Detail | Monthly cost | -|---|---|---| -| Claude API inference (paid tier, Haiku) | 4,000 users × $0.15 | $600 | -| Claude API inference (premium tier, mixed) | 800 users × $0.31 | $248 | -| Shared model serving (Together AI, 3B model) | 48,000 requests/mo | $27 | -| Per-user fine-tune jobs | 800 users / 12mo × $0.50 | $33 | -| App hosting (license server, auth API, DB) | VPS + PostgreSQL | $200 | -| Model artifact storage (800 × 1.5GB on S3) | 1.2TB | $28 | -| **Total** | | **$1,136/mo** | - ---- - -## 8. Revenue Model & Unit Economics - -### Monthly revenue at scale - -| Total users | Paid (4%) | Premium (20% of paid) | Revenue/mo | Infra/mo | **Gross margin** | -|---|---|---|---|---|---| -| 10,000 | 400 | 80 | $7,120 | $196 | **97.2%** | -| 100,000 | 4,000 | 800 | $88,250 | $1,136 | **98.7%** | - -### Blended ARPU -- Across all users (including free): **~$0.71/user/mo** -- Across paying users only: **~$17.30/user/mo** -- Coach account (3 seats avg): **~$74/mo** - -### LTV per user segment -- Paid individual (4.5mo avg job search): **~$54** -- Premium individual (4.5mo avg): **~$130** -- Coach account (ongoing, low churn): **$74/mo × 18mo estimated = ~$1,330** -- **Note:** Success churn is real — users leave when they get a job. Re-subscription rate on next job search partially offsets this. - -### ARR projections - -| Scale | ARR | -|---|---| -| 10K users | **~$85K** | -| 100K users | **~$1.06M** | -| 1M users | **~$10.6M** | - -To reach $10M ARR: ~1M total users **or** meaningful coach/enterprise penetration at lower user counts. - ---- - -## 9. VC Pitch Angles - -### The thesis -> "GitHub is our distribution channel. Local-first is our privacy moat. Coaches are our revenue engine." - -### Key metrics to hit before Series A -- 10K GitHub stars (validates distribution thesis) -- 500 paying users (validates willingness to pay) -- 20 coach accounts (validates B2B multiplier) -- 97%+ gross margin (already proven in model) - -### Competitive differentiation -1. **Privacy-first** — job search data never leaves your machine on free/paid tiers -2. **Fine-tuned personal model** — no other tool trains a cover letter model on your specific writing voice -3. **Full pipeline** — discovery through hired, not just one step (most competitors are point solutions) -4. **Open core** — community maintains job board scrapers, which break constantly; competitors pay engineers for this -5. **LLM-agnostic** — works with Ollama, Claude, GPT, vLLM; users aren't locked to one provider - -### Risks to address -- **Success churn** — mitigated by re-subscription on next job search, coach accounts (persistent), and potential pivot to ongoing career management -- **Job board scraping fragility** — mitigated by open core (community patches), multiple board sources, email ingestion fallback -- **LLM cost spikes** — mitigated by Haiku-first routing, local model fallback, user BYOK option -- **Copying by incumbents** — LinkedIn, Indeed have distribution but not privacy story; fine-tuned personal model is hard to replicate at their scale - ---- - -## 10. Roadmap - -### Phase 1 — Local-first launch (now) -- Docker Compose installer + setup wizard -- License key server (simple, hosted) -- Paid tier: shared model endpoint + Notion sync + email classifier -- Premium tier: fine-tune pipeline + Claude API routing -- Open core GitHub repo (MIT core, BSL premium) - -### Phase 2 — Coach tier validation (3–6 months post-launch) -- Multi-user mode with seat management -- Coach dashboard: shared job pool, per-candidate pipeline view -- Billing portal (Stripe) -- Outplacement firm pilot - -### Phase 3 — Cloud Edition (6–12 months, revenue-funded or post-seed) -- Hosted SaaS version at a URL (no install) -- Same codebase, cloud deployment mode -- Converts local-first users who want convenience -- Enables mobile access - -### Phase 4 — Enterprise (post-Series A) -- SSO / SAML -- Admin dashboard + analytics -- API for ATS integrations -- Custom fine-tune models for outplacement firm's brand voice - ---- - -## 11. Competitive Landscape - -### Direct competitors - -| Product | Price | Pipeline | AI CL | Privacy | Fine-tune | Open Source | -|---|---|---|---|---|---|---| -| **Job Seeker Platform** | Free–$29 | Full (discovery→hired) | Personal fine-tune | Local-first | Per-user | Core (MIT) | -| Teal | Free/$29 | Partial (tracker + resume) | Generic AI | Cloud | No | No | -| Jobscan | $49.95 | Resume scan only | No | Cloud | No | No | -| Huntr | Free/$30 | Tracker only | No | Cloud | No | No | -| Rezi | $29 | Resume/CL only | Generic AI | Cloud | No | No | -| Kickresume | $19 | Resume/CL only | Generic AI | Cloud | No | No | -| LinkedIn Premium | $40 | Job search only | No | Cloud (them) | No | No | -| AIHawk | Free | LinkedIn Easy Apply | No | Local | No | Yes (MIT) | -| Simplify | Free | Auto-fill only | No | Extension | No | No | - -### Competitive analysis - -**Teal** ($29/mo) is the closest feature competitor — job tracker + resume builder + AI cover letters. Key gaps: cloud-only (privacy risk), no discovery automation, generic AI (not fine-tuned to your voice), no interview prep, no email classifier. Their paid tier costs the same as our premium and delivers substantially less. - -**Jobscan** ($49.95/mo) is the premium ATS-optimization tool. Single-purpose, no pipeline, no cover letters. Overpriced for what it does. Users often use it alongside a tracker — this platform replaces both. - -**AIHawk** (open source) automates LinkedIn Easy Apply but has no pipeline, no AI beyond form filling, no cover letter gen, no tracking. It's a macro, not a platform. We already integrate with it as a downstream action. We're complementary, not competitive at the free tier. - -**LinkedIn Premium** ($40/mo) has distribution but actively works against user privacy and owns the candidate relationship. Users are the product. Our privacy story is a direct counter-positioning. - -### The whitespace - -No competitor offers all three of: **full pipeline automation + privacy-first local storage + personalized fine-tuned AI**. Every existing tool is either a point solution (just resume, just tracker, just auto-apply) or cloud-based SaaS that monetizes user data. The combination is the moat. - -### Indirect competition - -- **Spreadsheets + Notion templates** — free, flexible, no AI. The baseline we replace for free users. -- **Recruiting agencies** — human-assisted job search; we're a complement, not a replacement. -- **Career coaches** — we sell *to* them, not against them. - ---- - -## 12. Go-to-Market Strategy - -### Phase 1: Developer + privacy community launch - -**Channel:** GitHub → Hacker News → Reddit - -The open core model makes GitHub the primary distribution channel. A compelling README, one-command Docker install, and a working free tier are the launch. Target communities: - -- Hacker News "Show HN" — privacy-first self-hosted tools get strong traction -- r/cscareerquestions (1.2M members) — active job seekers, technically literate -- r/selfhosted (2.8M members) — prime audience for local-first tools -- r/ExperiencedDevs, r/remotework — secondary seeding - -**Goal:** 1,000 GitHub stars and 100 free installs in first 30 days. - -**Content hook:** "I built a private job search AI that runs entirely on your machine — no data leaves your computer." Privacy angle resonates deeply post-2024 data breach fatigue. - -### Phase 2: Career coaching channel - -**Channel:** LinkedIn → direct outreach → coach partnerships - -Career coaches are the highest-LTV customer and the most efficient channel to reach many job seekers at once. One coach onboarded = 10–20 active users. - -Tactics: -- Identify coaches on LinkedIn who post about job search tools -- Offer white-glove onboarding + 60-day free trial of coach seats -- Co-create content: "How I run 15 client job searches simultaneously" -- Referral program: coach gets 1 free seat per paid client referral - -**Goal:** 20 coach accounts within 90 days of paid tier launch. - -### Phase 3: Content + SEO (SaaS phase) - -Once the hosted Cloud Edition exists, invest in organic content: - -- "Best job tracker apps 2027" (comparison content — we win on privacy + AI) -- "How to write a cover letter that sounds like you, not ChatGPT" -- "Job search automation without giving LinkedIn your data" -- Tutorial videos: full setup walkthrough, fine-tuning demo - -**Goal:** 10K organic monthly visitors driving 2–5% free tier signups. - -### Phase 4: Outplacement firm partnerships (enterprise) - -Target HR consultancies and outplacement firms (Challenger, Gray & Christmas; Right Management; Lee Hecht Harrison). These firms place thousands of candidates per year and pay per-seat enterprise licenses. - -**Goal:** 3 enterprise pilots within 12 months of coach tier validation. - -### Pricing strategy by channel - -| Channel | Entry offer | Conversion lever | -|---|---|---| -| GitHub / OSS | Free forever | Upgrade friction: GPU setup, no shared model | -| Direct / ProductHunt | Free 30-day paid trial | AI quality gap is immediately visible | -| Coach outreach | Free 60-day coach trial | Efficiency gain across client base | -| Enterprise | Pilot with 10 seats | ROI vs. current manual process | - -### Key metrics by phase - -| Phase | Primary metric | Target | -|---|---|---| -| Launch | GitHub stars | 1K in 30 days | -| Paid validation | Paying users | 500 in 90 days | -| Coach validation | Coach accounts | 20 in 90 days | -| SaaS launch | Cloud signups | 10K in 6 months | -| Enterprise | ARR from enterprise | $100K in 12 months | - ---- - -## 13. Pricing Sensitivity Analysis - -### Paid tier sensitivity ($8 / $12 / $15 / $20) - -Assumption: 100K total users, 4% base conversion, gross infra cost $1,136/mo - -| Price | Conversion assumption | Paying users | Revenue/mo | Gross margin | -|---|---|---|---|---| -| $8 | 5.5% (price-elastic) | 5,500 | $44,000 | 97.4% | -| **$12** | **4.0% (base)** | **4,000** | **$48,000** | **97.6%** | -| $15 | 3.2% (slight drop) | 3,200 | $48,000 | 97.6% | -| $20 | 2.5% (meaningful drop) | 2,500 | $50,000 | 97.7% | - -**Finding:** Revenue is relatively flat between $12 and $20 because conversion drops offset the price increase. $12 is the sweet spot — maximizes paying user count (more data, more referrals, more upgrade candidates) without sacrificing revenue. Going below $10 requires meaningfully higher conversion to justify. - -### Premium tier sensitivity ($19 / $29 / $39 / $49) - -Assumption: 800 base premium users (20% of 4,000 paid), conversion adjusts with price - -| Price | Conversion from paid | Premium users | Revenue/mo | Fine-tune cost | Net/mo | -|---|---|---|---|---|---| -| $19 | 25% | 1,000 | $19,000 | $42 | $18,958 | -| **$29** | **20%** | **800** | **$23,200** | **$33** | **$23,167** | -| $39 | 15% | 600 | $23,400 | $25 | $23,375 | -| $49 | 10% | 400 | $19,600 | $17 | $19,583 | - -**Finding:** $29–$39 is the revenue-maximizing range. $29 wins on user volume (more fine-tune data, stronger coach acquisition funnel). $39 wins marginally on revenue but shrinks the premium base significantly. Recommend $29 at launch with the option to test $34–$39 once the fine-tuned model quality is demonstrated. - -### Coach seat sensitivity ($10 / $15 / $20 per seat) - -Assumption: 50 coach accounts, 3 seats avg, base $29 already captured above - -| Seat price | Seat revenue/mo | Total coach revenue/mo | -|---|---|---| -| $10 | $1,500 | $1,500 | -| **$15** | **$2,250** | **$2,250** | -| $20 | $3,000 | $3,000 | - -**Finding:** Seat pricing is relatively inelastic for coaches — $15–$20 is well within their cost of tools per client. $15 is conservative and easy to raise. $20 is defensible once coach ROI is documented. Consider $15 at launch, $20 after first 20 coach accounts are active. - -### Blended revenue at optimized pricing (100K users) - -| Component | Users | Price | Revenue/mo | -|---|---|---|---| -| Paid tier | 4,000 | $12 | $48,000 | -| Premium individual | 720 | $29 | $20,880 | -| Premium coach base | 80 | $29 | $2,320 | -| Coach seats (80 accounts × 3 avg) | 240 seats | $15 | $3,600 | -| **Total** | | | **$74,800/mo** | -| Infrastructure | | | -$1,136/mo | -| **Net** | | | **$73,664/mo (~$884K ARR)** | - -### Sensitivity to conversion rate (at $12/$29 pricing, 100K users) - -| Free→Paid conversion | Paid→Premium conversion | Revenue/mo | ARR | -|---|---|---|---| -| 2% | 15% | $30,720 | $369K | -| 3% | 18% | $47,664 | $572K | -| **4%** | **20%** | **$65,600** | **$787K** | -| 5% | 22% | $84,480 | $1.01M | -| 6% | 25% | $104,400 | $1.25M | - -**Key insight:** Conversion rate is the highest-leverage variable. Going from 4% → 5% free-to-paid conversion adds $228K ARR at 100K users. Investment in onboarding quality and the free-tier value proposition has outsized return vs. price adjustments. diff --git a/docs/plans/2026-02-25-circuitforge-license-design.md b/docs/plans/2026-02-25-circuitforge-license-design.md deleted file mode 100644 index 78ecb36..0000000 --- a/docs/plans/2026-02-25-circuitforge-license-design.md +++ /dev/null @@ -1,367 +0,0 @@ -# CircuitForge License Server — Design Document - -**Date:** 2026-02-25 -**Status:** Approved — ready for implementation - ---- - -## Goal - -Build a self-hosted licensing server for Circuit Forge LLC products. v1 serves Peregrine; schema is multi-product from day one. Enforces free / paid / premium / ultra tier gates with offline-capable JWT validation, 30-day refresh cycle, 7-day grace period, seat tracking, usage telemetry, and a content violation flagging foundation. - -## Architecture - -``` -┌─────────────────────────────────────────────────┐ -│ circuitforge-license (Heimdall:8600) │ -│ FastAPI + SQLite + RS256 JWT │ -│ │ -│ Public API (/v1/…): │ -│ POST /v1/activate → issue JWT │ -│ POST /v1/refresh → renew JWT │ -│ POST /v1/deactivate → free a seat │ -│ POST /v1/usage → record usage event │ -│ POST /v1/flag → report violation │ -│ │ -│ Admin API (/admin/…, bearer token): │ -│ POST/GET /admin/keys → CRUD keys │ -│ DELETE /admin/keys/{id} → revoke │ -│ GET /admin/activations → audit │ -│ GET /admin/usage → telemetry │ -│ GET/PATCH /admin/flags → flag review │ -└─────────────────────────────────────────────────┘ - ↑ HTTPS via Caddy (license.circuitforge.com) - -┌─────────────────────────────────────────────────┐ -│ Peregrine (user's machine) │ -│ scripts/license.py │ -│ │ -│ activate(key) → POST /v1/activate │ -│ writes config/license.json │ -│ verify_local() → validates JWT offline │ -│ using embedded public key │ -│ refresh_if_needed() → called on app startup │ -│ effective_tier() → tier string for can_use() │ -│ report_usage(…) → fire-and-forget telemetry │ -│ report_flag(…) → fire-and-forget violation │ -└─────────────────────────────────────────────────┘ -``` - -**Key properties:** -- Peregrine verifies tier **offline** on every check — RS256 public key embedded at build time -- Network required only at activation and 30-day refresh -- Revoked keys stop working at next refresh cycle (≤30 day lag — acceptable for v1) -- `config/license.json` gitignored; missing = free tier - ---- - -## Crypto: RS256 (asymmetric JWT) - -- **Private key** — lives only on the license server (`keys/private.pem`, gitignored) -- **Public key** — committed to both the license server repo and Peregrine (`scripts/license_public_key.pem`) -- Peregrine can verify JWT authenticity without ever knowing the private key -- A stolen JWT cannot be forged without the private key -- Revocation: server refuses refresh; old JWT valid until expiry then grace period expires - -**Key generation (one-time, on Heimdall):** -```bash -openssl genrsa -out keys/private.pem 2048 -openssl rsa -in keys/private.pem -pubout -out keys/public.pem -# copy keys/public.pem → peregrine/scripts/license_public_key.pem -``` - ---- - -## Database Schema - -```sql -CREATE TABLE license_keys ( - id TEXT PRIMARY KEY, -- UUID - key_display TEXT UNIQUE NOT NULL, -- CFG-PRNG-XXXX-XXXX-XXXX - product TEXT NOT NULL, -- peregrine | falcon | osprey | … - tier TEXT NOT NULL, -- paid | premium | ultra - seats INTEGER DEFAULT 1, - valid_until TEXT, -- ISO date or NULL (perpetual) - revoked INTEGER DEFAULT 0, - customer_email TEXT, -- proper field, not buried in notes - source TEXT DEFAULT 'manual', -- manual | beta | promo | stripe - trial INTEGER DEFAULT 0, -- 1 = time-limited trial key - notes TEXT, - created_at TEXT NOT NULL -); - -CREATE TABLE activations ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT NOT NULL, -- sha256(hostname + MAC) - app_version TEXT, -- Peregrine version at last refresh - platform TEXT, -- linux | macos | windows | docker - activated_at TEXT NOT NULL, - last_refresh TEXT NOT NULL, - deactivated_at TEXT -- NULL = still active -); - -CREATE TABLE usage_events ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT NOT NULL, - product TEXT NOT NULL, - event_type TEXT NOT NULL, -- cover_letter_generated | - -- company_research | email_sync | - -- interview_prep | survey | etc. - metadata TEXT, -- JSON blob for context - created_at TEXT NOT NULL -); - -CREATE TABLE flags ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT, - product TEXT NOT NULL, - flag_type TEXT NOT NULL, -- content_violation | tos_violation | - -- abuse | manual - details TEXT, -- JSON: prompt snippet, output excerpt - status TEXT DEFAULT 'open', -- open | reviewed | dismissed | actioned - created_at TEXT NOT NULL, - reviewed_at TEXT, - action_taken TEXT -- none | warned | revoked -); - -CREATE TABLE audit_log ( - id TEXT PRIMARY KEY, - entity_type TEXT NOT NULL, -- key | activation | flag - entity_id TEXT NOT NULL, - action TEXT NOT NULL, -- created | revoked | activated | - -- deactivated | flag_actioned - actor TEXT, -- admin identifier (future multi-admin) - details TEXT, -- JSON - created_at TEXT NOT NULL -); -``` - -**Flags scope (v1):** Schema and `POST /v1/flag` endpoint capture data. No admin enforcement UI in v1 — query DB directly. Build review UI in v2 when there's data to act on. - ---- - -## JWT Payload - -```json -{ - "sub": "CFG-PRNG-A1B2-C3D4-E5F6", - "product": "peregrine", - "tier": "paid", - "seats": 2, - "machine": "a3f9c2…", - "notice": "Version 1.1 available — see circuitforge.com/update", - "iat": 1740000000, - "exp": 1742592000 -} -``` - -`notice` is optional — set via a server config value; included in refresh responses so Peregrine can surface it as a banner. No DB table needed. - ---- - -## Key Format - -`CFG-PRNG-A1B2-C3D4-E5F6` - -- `CFG` — Circuit Forge -- `PRNG` / `FLCN` / `OSPY` / … — 4-char product code -- Three random 4-char alphanumeric segments -- Human-readable, easy to copy/paste into a support email - ---- - -## Endpoint Reference - -| Method | Path | Auth | Purpose | -|--------|------|------|---------| -| POST | `/v1/activate` | none | Issue JWT for key + machine | -| POST | `/v1/refresh` | JWT bearer | Renew JWT before expiry | -| POST | `/v1/deactivate` | JWT bearer | Free a seat | -| POST | `/v1/usage` | JWT bearer | Record usage event (fire-and-forget) | -| POST | `/v1/flag` | JWT bearer | Report content/ToS violation | -| POST | `/admin/keys` | admin token | Create a new key | -| GET | `/admin/keys` | admin token | List all keys + activation counts | -| DELETE | `/admin/keys/{id}` | admin token | Revoke a key | -| GET | `/admin/activations` | admin token | Full activation audit | -| GET | `/admin/usage` | admin token | Usage breakdown per key/product/event | -| GET | `/admin/flags` | admin token | List flags (open by default) | -| PATCH | `/admin/flags/{id}` | admin token | Update flag status + action | - ---- - -## Peregrine Client (`scripts/license.py`) - -**Public API:** -```python -def activate(key: str) -> dict # POST /v1/activate, writes license.json -def verify_local() -> dict | None # validates JWT offline; None = free tier -def refresh_if_needed() -> None # silent; called on app startup -def effective_tier() -> str # "free"|"paid"|"premium"|"ultra" -def report_usage(event_type: str, # fire-and-forget; failures silently dropped - metadata: dict = {}) -> None -def report_flag(flag_type: str, # fire-and-forget - details: dict) -> None -``` - -**`effective_tier()` decision tree:** -``` -license.json missing or unreadable → "free" -JWT signature invalid → "free" -JWT product != "peregrine" → "free" -JWT not expired → tier from payload -JWT expired, within grace period → tier from payload + show banner -JWT expired, grace period expired → "free" + show banner -``` - -**`config/license.json` (gitignored):** -```json -{ - "jwt": "eyJ…", - "key_display": "CFG-PRNG-A1B2-C3D4-E5F6", - "tier": "paid", - "valid_until": "2026-03-27", - "machine_id": "a3f9c2…", - "last_refresh": "2026-02-25T12:00:00Z", - "grace_until": null -} -``` - -**Integration point in `tiers.py`:** -```python -def effective_tier(profile) -> str: - from scripts.license import effective_tier as _license_tier - if profile.dev_tier_override: # dev override still works in dev mode - return profile.dev_tier_override - return _license_tier() -``` - -**Settings License tab** (new tab in `app/pages/2_Settings.py`): -- Text input: enter license key → calls `activate()` → shows result -- If active: tier badge, key display string, expiry date, seat count -- Grace period: amber banner with days remaining -- "Deactivate this machine" button → `/v1/deactivate`, deletes `license.json` - ---- - -## Deployment - -**Repo:** `git.opensourcesolarpunk.com/pyr0ball/circuitforge-license` (private) - -**Repo layout:** -``` -circuitforge-license/ -├── app/ -│ ├── main.py # FastAPI app -│ ├── db.py # SQLite helpers, schema init -│ ├── models.py # Pydantic models -│ ├── crypto.py # RSA sign/verify helpers -│ └── routes/ -│ ├── public.py # /v1/* endpoints -│ └── admin.py # /admin/* endpoints -├── data/ # SQLite DB (named volume) -├── keys/ -│ ├── private.pem # gitignored -│ └── public.pem # committed -├── scripts/ -│ └── issue-key.sh # curl wrapper for key issuance -├── tests/ -├── Dockerfile -├── docker-compose.yml -├── .env.example -└── requirements.txt -``` - -**`docker-compose.yml` (on Heimdall):** -```yaml -services: - license: - build: . - restart: unless-stopped - ports: - - "127.0.0.1:8600:8600" - volumes: - - license_data:/app/data - - ./keys:/app/keys:ro - env_file: .env - -volumes: - license_data: -``` - -**`.env` (gitignored):** -``` -ADMIN_TOKEN= -JWT_PRIVATE_KEY_PATH=/app/keys/private.pem -JWT_PUBLIC_KEY_PATH=/app/keys/public.pem -JWT_EXPIRY_DAYS=30 -GRACE_PERIOD_DAYS=7 -``` - -**Caddy block (add to Heimdall Caddyfile):** -```caddy -license.circuitforge.com { - reverse_proxy localhost:8600 -} -``` - ---- - -## Admin Workflow (v1) - -All operations via `curl` or `scripts/issue-key.sh`: - -```bash -# Issue a key -./scripts/issue-key.sh --product peregrine --tier paid --seats 2 \ - --email user@example.com --notes "Beta — manual payment 2026-02-25" -# → CFG-PRNG-A1B2-C3D4-E5F6 (email to customer) - -# List all keys -curl https://license.circuitforge.com/admin/keys \ - -H "Authorization: Bearer $ADMIN_TOKEN" - -# Revoke a key -curl -X DELETE https://license.circuitforge.com/admin/keys/{id} \ - -H "Authorization: Bearer $ADMIN_TOKEN" -``` - ---- - -## Testing Strategy - -**License server:** -- pytest with in-memory SQLite and generated test keypair -- All endpoints tested: activate, refresh, deactivate, usage, flag, admin CRUD -- Seat limit enforcement, expiry, revocation all unit tested - -**Peregrine client:** -- `verify_local()` tested with pre-signed test JWT using test keypair -- `activate()` / `refresh()` tested with `httpx` mocks -- `effective_tier()` tested across all states: valid, expired, grace, revoked, missing - -**Integration smoke test:** -```bash -docker compose up -d -# create test key via admin API -# call /v1/activate with test key -# verify JWT signature with public key -# verify /v1/refresh extends expiry -``` - ---- - -## Decisions Log - -| Decision | Rationale | -|----------|-----------| -| RS256 over HS256 | Public key embeddable in client; private key never leaves server | -| SQLite over Postgres | Matches Peregrine's SQLite-first philosophy; trivially backupable | -| 30-day JWT lifetime | Standard SaaS pattern; invisible to users in normal operation | -| 7-day grace period | Covers travel, network outages, server maintenance | -| Flags v1: capture only | No volume to justify review UI yet; add in v2 | -| No payment integration | Manual issuance until customer volume justifies automation | -| Multi-product schema | Adding a column now vs migrating a live DB later | -| Separate repo | License server is infrastructure, not part of Peregrine's BSL scope | diff --git a/docs/plans/2026-02-25-circuitforge-license-plan.md b/docs/plans/2026-02-25-circuitforge-license-plan.md deleted file mode 100644 index c7c914b..0000000 --- a/docs/plans/2026-02-25-circuitforge-license-plan.md +++ /dev/null @@ -1,2197 +0,0 @@ -# CircuitForge License Server — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Build a self-hosted RS256 JWT licensing server for Circuit Forge LLC and wire Peregrine to validate licenses offline. - -**Architecture:** Two work streams — (A) a new FastAPI + SQLite service (`circuitforge-license`) deployed on Heimdall via Docker + Caddy, and (B) a `scripts/license.py` client in Peregrine that activates against the server and verifies JWTs offline using an embedded public key. The server issues 30-day signed tokens; the client verifies signatures locally on every tier check with zero network calls during normal operation. - -**Tech Stack:** FastAPI, PyJWT[crypto], Pydantic v2, SQLite, pytest, httpx (test client), cryptography (RSA key gen in tests), Docker Compose V2, Caddy. - -**Repos:** -- License server dev: `/Library/Development/CircuitForge/circuitforge-license/` → `git.opensourcesolarpunk.com/pyr0ball/circuitforge-license` -- License server live (on Heimdall): cloned to `/devl/circuitforge-license/` -- Peregrine client: `/Library/Development/devl/peregrine/` -- Run tests: `/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v` -- Python env for local dev/test: `conda run -n job-seeker` - ---- - -## PART A — License Server (new repo) - ---- - -### Task 1: Repo scaffold + DB schema - -**Files:** -- Create: `/Library/Development/CircuitForge/circuitforge-license/` (new directory) -- Create: `requirements.txt` -- Create: `app/__init__.py` -- Create: `app/db.py` -- Create: `tests/__init__.py` -- Create: `tests/test_db.py` -- Create: `.gitignore` - -**Step 1: Create the directory and git repo** - -```bash -mkdir -p /Library/Development/devl/circuitforge-license -cd /Library/Development/devl/circuitforge-license -git init -``` - -**Step 2: Create `.gitignore`** - -``` -# Secrets — never commit these -.env -keys/private.pem -data/ - -# Python -__pycache__/ -*.pyc -.pytest_cache/ -*.egg-info/ -dist/ -.coverage -htmlcov/ -``` - -**Step 3: Create `requirements.txt`** - -``` -fastapi>=0.110 -uvicorn[standard]>=0.27 -pyjwt[crypto]>=2.8 -pydantic>=2.0 -python-dotenv>=1.0 -pytest>=9.0 -pytest-cov -httpx -cryptography>=42 -``` - -**Step 4: Create `app/__init__.py`** (empty file) - -**Step 5: Write the failing test** - -```python -# tests/test_db.py -import pytest -from pathlib import Path -from app.db import init_db, get_db - - -def test_init_db_creates_all_tables(tmp_path): - db = tmp_path / "test.db" - init_db(db) - with get_db(db) as conn: - tables = {row[0] for row in conn.execute( - "SELECT name FROM sqlite_master WHERE type='table'" - ).fetchall()} - expected = {"license_keys", "activations", "usage_events", "flags", "audit_log"} - assert expected.issubset(tables) - - -def test_init_db_idempotent(tmp_path): - db = tmp_path / "test.db" - init_db(db) - init_db(db) # second call must not raise or corrupt - with get_db(db) as conn: - count = conn.execute("SELECT COUNT(*) FROM license_keys").fetchone()[0] - assert count == 0 -``` - -**Step 6: Run test to verify it fails** - -```bash -cd /Library/Development/devl/circuitforge-license -conda run -n job-seeker python -m pytest tests/test_db.py -v -``` -Expected: `FAILED` — `ModuleNotFoundError: No module named 'app'` - -**Step 7: Write `app/db.py`** - -```python -# app/db.py -import sqlite3 -from contextlib import contextmanager -from pathlib import Path - -DB_PATH = Path(__file__).parent.parent / "data" / "license.db" - -_SCHEMA = """ -CREATE TABLE IF NOT EXISTS license_keys ( - id TEXT PRIMARY KEY, - key_display TEXT UNIQUE NOT NULL, - product TEXT NOT NULL, - tier TEXT NOT NULL, - seats INTEGER DEFAULT 1, - valid_until TEXT, - revoked INTEGER DEFAULT 0, - customer_email TEXT, - source TEXT DEFAULT 'manual', - trial INTEGER DEFAULT 0, - notes TEXT, - created_at TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS activations ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT NOT NULL, - app_version TEXT, - platform TEXT, - activated_at TEXT NOT NULL, - last_refresh TEXT NOT NULL, - deactivated_at TEXT -); - -CREATE TABLE IF NOT EXISTS usage_events ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT NOT NULL, - product TEXT NOT NULL, - event_type TEXT NOT NULL, - metadata TEXT, - created_at TEXT NOT NULL -); - -CREATE TABLE IF NOT EXISTS flags ( - id TEXT PRIMARY KEY, - key_id TEXT NOT NULL REFERENCES license_keys(id), - machine_id TEXT, - product TEXT NOT NULL, - flag_type TEXT NOT NULL, - details TEXT, - status TEXT DEFAULT 'open', - created_at TEXT NOT NULL, - reviewed_at TEXT, - action_taken TEXT -); - -CREATE TABLE IF NOT EXISTS audit_log ( - id TEXT PRIMARY KEY, - entity_type TEXT NOT NULL, - entity_id TEXT NOT NULL, - action TEXT NOT NULL, - actor TEXT, - details TEXT, - created_at TEXT NOT NULL -); -""" - - -@contextmanager -def get_db(db_path: Path = DB_PATH): - db_path.parent.mkdir(parents=True, exist_ok=True) - conn = sqlite3.connect(db_path) - conn.row_factory = sqlite3.Row - conn.execute("PRAGMA journal_mode=WAL") - conn.execute("PRAGMA foreign_keys=ON") - try: - yield conn - conn.commit() - except Exception: - conn.rollback() - raise - finally: - conn.close() - - -def init_db(db_path: Path = DB_PATH) -> None: - with get_db(db_path) as conn: - conn.executescript(_SCHEMA) -``` - -**Step 8: Run test to verify it passes** - -```bash -conda run -n job-seeker python -m pytest tests/test_db.py -v -``` -Expected: `2 passed` - -**Step 9: Commit** - -```bash -cd /Library/Development/devl/circuitforge-license -git add -A -git commit -m "feat: repo scaffold, DB schema, init_db" -``` - ---- - -### Task 2: Crypto module + test keypair fixture - -**Files:** -- Create: `app/crypto.py` -- Create: `tests/conftest.py` -- Create: `tests/test_crypto.py` -- Create: `keys/` (directory; `public.pem` committed later) - -**Step 1: Write the failing tests** - -```python -# tests/test_crypto.py -import pytest -import jwt as pyjwt -from app.crypto import sign_jwt, verify_jwt - - -def test_sign_and_verify_roundtrip(test_keypair): - private_pem, public_pem = test_keypair - payload = {"sub": "CFG-PRNG-TEST", "product": "peregrine", "tier": "paid"} - token = sign_jwt(payload, private_pem=private_pem, expiry_days=30) - decoded = verify_jwt(token, public_pem=public_pem) - assert decoded["sub"] == "CFG-PRNG-TEST" - assert decoded["tier"] == "paid" - assert "exp" in decoded - assert "iat" in decoded - - -def test_verify_rejects_wrong_key(test_keypair): - from cryptography.hazmat.primitives.asymmetric import rsa - from cryptography.hazmat.primitives import serialization - private_pem, _ = test_keypair - other_private = rsa.generate_private_key(public_exponent=65537, key_size=2048) - other_public_pem = other_private.public_key().public_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PublicFormat.SubjectPublicKeyInfo, - ) - token = sign_jwt({"sub": "test"}, private_pem=private_pem, expiry_days=30) - with pytest.raises(pyjwt.exceptions.InvalidSignatureError): - verify_jwt(token, public_pem=other_public_pem) - - -def test_verify_rejects_expired_token(test_keypair): - private_pem, public_pem = test_keypair - token = sign_jwt({"sub": "test"}, private_pem=private_pem, expiry_days=-1) - with pytest.raises(pyjwt.exceptions.ExpiredSignatureError): - verify_jwt(token, public_pem=public_pem) -``` - -**Step 2: Write `tests/conftest.py`** - -```python -# tests/conftest.py -import pytest -from cryptography.hazmat.primitives.asymmetric import rsa -from cryptography.hazmat.primitives import serialization - - -@pytest.fixture(scope="session") -def test_keypair(): - """Generate a fresh RSA-2048 keypair for the test session.""" - private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) - private_pem = private_key.private_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PrivateFormat.TraditionalOpenSSL, - encryption_algorithm=serialization.NoEncryption(), - ) - public_pem = private_key.public_key().public_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PublicFormat.SubjectPublicKeyInfo, - ) - return private_pem, public_pem -``` - -**Step 3: Run test to verify it fails** - -```bash -conda run -n job-seeker python -m pytest tests/test_crypto.py -v -``` -Expected: `FAILED` — `ModuleNotFoundError: No module named 'app.crypto'` - -**Step 4: Write `app/crypto.py`** - -```python -# app/crypto.py -import os -from datetime import datetime, timedelta, timezone -from pathlib import Path - -import jwt as pyjwt - - -def _load_key(env_var: str, override: bytes | None) -> bytes: - if override is not None: - return override - path = Path(os.environ[env_var]) - return path.read_bytes() - - -def sign_jwt( - payload: dict, - expiry_days: int | None = None, - private_pem: bytes | None = None, -) -> str: - if expiry_days is None: - expiry_days = int(os.environ.get("JWT_EXPIRY_DAYS", "30")) - now = datetime.now(timezone.utc) - full_payload = { - **payload, - "iat": now, - "exp": now + timedelta(days=expiry_days), - } - key = _load_key("JWT_PRIVATE_KEY_PATH", private_pem) - return pyjwt.encode(full_payload, key, algorithm="RS256") - - -def verify_jwt(token: str, public_pem: bytes | None = None) -> dict: - """Verify RS256 JWT and return decoded payload. Raises on invalid/expired.""" - key = _load_key("JWT_PUBLIC_KEY_PATH", public_pem) - return pyjwt.decode(token, key, algorithms=["RS256"]) -``` - -**Step 5: Run test to verify it passes** - -```bash -conda run -n job-seeker python -m pytest tests/test_crypto.py -v -``` -Expected: `3 passed` - -**Step 6: Commit** - -```bash -git add -A -git commit -m "feat: crypto module — RS256 sign/verify with test keypair fixture" -``` - ---- - -### Task 3: Pydantic models - -**Files:** -- Create: `app/models.py` -- Create: `tests/test_models.py` - -**Step 1: Write the failing test** - -```python -# tests/test_models.py -from app.models import ( - ActivateRequest, ActivateResponse, - RefreshRequest, DeactivateRequest, - UsageRequest, FlagRequest, - CreateKeyRequest, -) - - -def test_activate_request_requires_key_machine_product(): - req = ActivateRequest(key="CFG-PRNG-A1B2-C3D4-E5F6", - machine_id="abc123", product="peregrine") - assert req.key == "CFG-PRNG-A1B2-C3D4-E5F6" - assert req.app_version is None - assert req.platform is None - - -def test_create_key_request_defaults(): - req = CreateKeyRequest(product="peregrine", tier="paid") - assert req.seats == 1 - assert req.source == "manual" - assert req.trial is False - assert req.valid_until is None -``` - -**Step 2: Run to verify failure** - -```bash -conda run -n job-seeker python -m pytest tests/test_models.py -v -``` -Expected: `FAILED` — `ModuleNotFoundError: No module named 'app.models'` - -**Step 3: Write `app/models.py`** - -```python -# app/models.py -from __future__ import annotations -from typing import Optional -from pydantic import BaseModel - - -class ActivateRequest(BaseModel): - key: str - machine_id: str - product: str - app_version: Optional[str] = None - platform: Optional[str] = None - - -class ActivateResponse(BaseModel): - jwt: str - tier: str - valid_until: Optional[str] = None - notice: Optional[str] = None - - -class RefreshRequest(BaseModel): - jwt: str - machine_id: str - app_version: Optional[str] = None - platform: Optional[str] = None - - -class DeactivateRequest(BaseModel): - jwt: str - machine_id: str - - -class UsageRequest(BaseModel): - event_type: str - product: str - metadata: Optional[dict] = None - - -class FlagRequest(BaseModel): - flag_type: str - product: str - details: Optional[dict] = None - - -class CreateKeyRequest(BaseModel): - product: str - tier: str - seats: int = 1 - valid_until: Optional[str] = None - customer_email: Optional[str] = None - source: str = "manual" - trial: bool = False - notes: Optional[str] = None - - -class KeyResponse(BaseModel): - id: str - key_display: str - product: str - tier: str - seats: int - valid_until: Optional[str] - revoked: bool - customer_email: Optional[str] - source: str - trial: bool - notes: Optional[str] - created_at: str - active_seat_count: int = 0 - - -class FlagUpdateRequest(BaseModel): - status: str # reviewed | dismissed | actioned - action_taken: Optional[str] = None # none | warned | revoked -``` - -**Step 4: Run to verify it passes** - -```bash -conda run -n job-seeker python -m pytest tests/test_models.py -v -``` -Expected: `2 passed` - -**Step 5: Commit** - -```bash -git add -A -git commit -m "feat: Pydantic v2 request/response models" -``` - ---- - -### Task 4: Public routes — activate, refresh, deactivate - -**Files:** -- Create: `app/routes/__init__.py` (empty) -- Create: `app/routes/public.py` -- Create: `tests/test_public_routes.py` - -**Step 1: Write failing tests** - -```python -# tests/test_public_routes.py -import json -import pytest -from fastapi.testclient import TestClient -from app.main import create_app -from app.db import init_db - - -@pytest.fixture() -def client(tmp_path, test_keypair, monkeypatch): - db = tmp_path / "test.db" - private_pem, public_pem = test_keypair - # Write keys to tmp files - (tmp_path / "private.pem").write_bytes(private_pem) - (tmp_path / "public.pem").write_bytes(public_pem) - monkeypatch.setenv("JWT_PRIVATE_KEY_PATH", str(tmp_path / "private.pem")) - monkeypatch.setenv("JWT_PUBLIC_KEY_PATH", str(tmp_path / "public.pem")) - monkeypatch.setenv("JWT_EXPIRY_DAYS", "30") - monkeypatch.setenv("GRACE_PERIOD_DAYS", "7") - monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token") - monkeypatch.setenv("SERVER_NOTICE", "") - init_db(db) - app = create_app(db_path=db) - return TestClient(app) - - -@pytest.fixture() -def active_key(client): - """Create a paid key via admin API, return key_display.""" - resp = client.post("/admin/keys", json={ - "product": "peregrine", "tier": "paid", "seats": 2, - "customer_email": "test@example.com", - }, headers={"Authorization": "Bearer test-admin-token"}) - assert resp.status_code == 200 - return resp.json()["key_display"] - - -def test_activate_returns_jwt(client, active_key): - resp = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "machine-1", "product": "peregrine", - "platform": "linux", "app_version": "1.0.0", - }) - assert resp.status_code == 200 - data = resp.json() - assert "jwt" in data - assert data["tier"] == "paid" - - -def test_activate_same_machine_twice_ok(client, active_key): - payload = {"key": active_key, "machine_id": "machine-1", "product": "peregrine"} - resp1 = client.post("/v1/activate", json=payload) - resp2 = client.post("/v1/activate", json=payload) - assert resp1.status_code == 200 - assert resp2.status_code == 200 - - -def test_activate_seat_limit_enforced(client, active_key): - # seats=2, so machine-1 and machine-2 OK, machine-3 rejected - for mid in ["machine-1", "machine-2"]: - r = client.post("/v1/activate", json={ - "key": active_key, "machine_id": mid, "product": "peregrine" - }) - assert r.status_code == 200 - r3 = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "machine-3", "product": "peregrine" - }) - assert r3.status_code == 409 - - -def test_activate_invalid_key_rejected(client): - resp = client.post("/v1/activate", json={ - "key": "CFG-PRNG-FAKE-FAKE-FAKE", "machine_id": "m1", "product": "peregrine" - }) - assert resp.status_code == 403 - - -def test_activate_wrong_product_rejected(client, active_key): - resp = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "m1", "product": "falcon" - }) - assert resp.status_code == 403 - - -def test_refresh_returns_new_jwt(client, active_key): - act = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "m1", "product": "peregrine" - }) - old_jwt = act.json()["jwt"] - resp = client.post("/v1/refresh", json={"jwt": old_jwt, "machine_id": "m1"}) - assert resp.status_code == 200 - assert "jwt" in resp.json() - - -def test_deactivate_frees_seat(client, active_key): - # Fill both seats - for mid in ["machine-1", "machine-2"]: - client.post("/v1/activate", json={ - "key": active_key, "machine_id": mid, "product": "peregrine" - }) - # Deactivate machine-1 - act = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "machine-1", "product": "peregrine" - }) - token = act.json()["jwt"] - deact = client.post("/v1/deactivate", json={"jwt": token, "machine_id": "machine-1"}) - assert deact.status_code == 200 - # Now machine-3 can activate - r3 = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "machine-3", "product": "peregrine" - }) - assert r3.status_code == 200 -``` - -**Step 2: Run to verify failure** - -```bash -conda run -n job-seeker python -m pytest tests/test_public_routes.py -v -``` -Expected: `FAILED` — `ModuleNotFoundError: No module named 'app.main'` - -**Step 3: Write `app/routes/__init__.py`** (empty) - -**Step 4: Write `app/routes/public.py`** - -```python -# app/routes/public.py -import json -import os -import uuid -from datetime import datetime, timezone - -import jwt as pyjwt -from fastapi import APIRouter, Depends, HTTPException - -from app.crypto import sign_jwt, verify_jwt -from app.db import get_db -from app.models import ( - ActivateRequest, ActivateResponse, - RefreshRequest, DeactivateRequest, - UsageRequest, FlagRequest, -) - -router = APIRouter() - - -def _now() -> str: - return datetime.now(timezone.utc).isoformat() - - -def _get_key_row(conn, key_display: str, product: str): - row = conn.execute( - "SELECT * FROM license_keys WHERE key_display=? AND product=?", - (key_display, product), - ).fetchone() - if not row or row["revoked"]: - raise HTTPException(status_code=403, detail="Invalid or revoked license key") - if row["valid_until"] and row["valid_until"] < datetime.now(timezone.utc).date().isoformat(): - raise HTTPException(status_code=403, detail="License key expired") - return row - - -def _build_jwt(key_row, machine_id: str) -> str: - notice = os.environ.get("SERVER_NOTICE", "") - payload = { - "sub": key_row["key_display"], - "product": key_row["product"], - "tier": key_row["tier"], - "seats": key_row["seats"], - "machine": machine_id, - } - if notice: - payload["notice"] = notice - return sign_jwt(payload) - - -def _audit(conn, entity_type: str, entity_id: str, action: str, details: dict | None = None): - conn.execute( - "INSERT INTO audit_log (id, entity_type, entity_id, action, details, created_at) " - "VALUES (?,?,?,?,?,?)", - (str(uuid.uuid4()), entity_type, entity_id, action, - json.dumps(details) if details else None, _now()), - ) - - -@router.post("/activate", response_model=ActivateResponse) -def activate(req: ActivateRequest, db_path=Depends(lambda: None)): - from app.routes._db_dep import get_db_path - with get_db(get_db_path()) as conn: - key_row = _get_key_row(conn, req.key, req.product) - # Count active seats, excluding this machine - active_seats = conn.execute( - "SELECT COUNT(*) FROM activations " - "WHERE key_id=? AND deactivated_at IS NULL AND machine_id!=?", - (key_row["id"], req.machine_id), - ).fetchone()[0] - existing = conn.execute( - "SELECT * FROM activations WHERE key_id=? AND machine_id=?", - (key_row["id"], req.machine_id), - ).fetchone() - if not existing and active_seats >= key_row["seats"]: - raise HTTPException(status_code=409, detail=f"Seat limit reached ({key_row['seats']} seats)") - now = _now() - if existing: - conn.execute( - "UPDATE activations SET last_refresh=?, app_version=?, platform=?, " - "deactivated_at=NULL WHERE id=?", - (now, req.app_version, req.platform, existing["id"]), - ) - activation_id = existing["id"] - else: - activation_id = str(uuid.uuid4()) - conn.execute( - "INSERT INTO activations (id, key_id, machine_id, app_version, platform, " - "activated_at, last_refresh) VALUES (?,?,?,?,?,?,?)", - (activation_id, key_row["id"], req.machine_id, - req.app_version, req.platform, now, now), - ) - _audit(conn, "activation", activation_id, "activated", {"machine_id": req.machine_id}) - token = _build_jwt(key_row, req.machine_id) - notice = os.environ.get("SERVER_NOTICE") or None - return ActivateResponse(jwt=token, tier=key_row["tier"], - valid_until=key_row["valid_until"], notice=notice) - - -@router.post("/refresh", response_model=ActivateResponse) -def refresh(req: RefreshRequest, db_path=Depends(lambda: None)): - from app.routes._db_dep import get_db_path - # Decode without expiry check so we can refresh near-expired tokens - try: - payload = verify_jwt(req.jwt) - except pyjwt.exceptions.ExpiredSignatureError: - # Allow refresh of just-expired tokens - payload = pyjwt.decode(req.jwt, options={"verify_exp": False, - "verify_signature": False}) - except pyjwt.exceptions.InvalidTokenError as e: - raise HTTPException(status_code=403, detail=str(e)) - - with get_db(get_db_path()) as conn: - key_row = _get_key_row(conn, payload.get("sub", ""), payload.get("product", "")) - existing = conn.execute( - "SELECT * FROM activations WHERE key_id=? AND machine_id=? AND deactivated_at IS NULL", - (key_row["id"], req.machine_id), - ).fetchone() - if not existing: - raise HTTPException(status_code=403, detail="Machine not registered for this key") - now = _now() - conn.execute( - "UPDATE activations SET last_refresh=?, app_version=? WHERE id=?", - (now, req.app_version or existing["app_version"], existing["id"]), - ) - _audit(conn, "activation", existing["id"], "refreshed", {"machine_id": req.machine_id}) - token = _build_jwt(key_row, req.machine_id) - notice = os.environ.get("SERVER_NOTICE") or None - return ActivateResponse(jwt=token, tier=key_row["tier"], - valid_until=key_row["valid_until"], notice=notice) - - -@router.post("/deactivate") -def deactivate(req: DeactivateRequest): - from app.routes._db_dep import get_db_path - try: - payload = verify_jwt(req.jwt) - except pyjwt.exceptions.PyJWTError as e: - raise HTTPException(status_code=403, detail=str(e)) - with get_db(get_db_path()) as conn: - existing = conn.execute( - "SELECT a.id FROM activations a " - "JOIN license_keys k ON k.id=a.key_id " - "WHERE k.key_display=? AND a.machine_id=? AND a.deactivated_at IS NULL", - (payload.get("sub", ""), req.machine_id), - ).fetchone() - if not existing: - raise HTTPException(status_code=404, detail="No active seat found") - now = _now() - conn.execute("UPDATE activations SET deactivated_at=? WHERE id=?", - (now, existing["id"])) - _audit(conn, "activation", existing["id"], "deactivated", {"machine_id": req.machine_id}) - return {"status": "deactivated"} -``` - -**Step 5: Write `app/routes/_db_dep.py`** (module-level DB path holder, allows test injection) - -```python -# app/routes/_db_dep.py -from pathlib import Path -from app.db import DB_PATH - -_db_path: Path = DB_PATH - - -def set_db_path(p: Path) -> None: - global _db_path - _db_path = p - - -def get_db_path() -> Path: - return _db_path -``` - -**Step 6: Write `app/main.py`** (minimal, enough for tests) - -```python -# app/main.py -from pathlib import Path -from fastapi import FastAPI -from app.db import init_db, DB_PATH -from app.routes import public, admin -from app.routes._db_dep import set_db_path - - -def create_app(db_path: Path = DB_PATH) -> FastAPI: - set_db_path(db_path) - init_db(db_path) - app = FastAPI(title="CircuitForge License Server", version="1.0.0") - app.include_router(public.router, prefix="/v1") - app.include_router(admin.router, prefix="/admin") - return app - - -app = create_app() -``` - -**Step 7: Write minimal `app/routes/admin.py`** (enough for `active_key` fixture to work) - -```python -# app/routes/admin.py — skeleton; full implementation in Task 5 -import os -import uuid -import secrets -import string -from datetime import datetime, timezone -from fastapi import APIRouter, HTTPException, Header -from app.db import get_db -from app.models import CreateKeyRequest, KeyResponse -from app.routes._db_dep import get_db_path - -router = APIRouter() - - -def _require_admin(authorization: str = Header(...)): - expected = f"Bearer {os.environ.get('ADMIN_TOKEN', '')}" - if authorization != expected: - raise HTTPException(status_code=401, detail="Unauthorized") - - -def _gen_key_display(product: str) -> str: - codes = {"peregrine": "PRNG", "falcon": "FLCN", "osprey": "OSPY", - "kestrel": "KSTR", "harrier": "HARR", "merlin": "MRLN", - "ibis": "IBIS", "tern": "TERN", "wren": "WREN", "martin": "MRTN"} - code = codes.get(product, product[:4].upper()) - chars = string.ascii_uppercase + string.digits - segs = [secrets.choice(chars) + secrets.choice(chars) + - secrets.choice(chars) + secrets.choice(chars) for _ in range(3)] - return f"CFG-{code}-{segs[0]}-{segs[1]}-{segs[2]}" - - -@router.post("/keys", response_model=KeyResponse) -def create_key(req: CreateKeyRequest, authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - key_id = str(uuid.uuid4()) - key_display = _gen_key_display(req.product) - now = datetime.now(timezone.utc).isoformat() - conn.execute( - "INSERT INTO license_keys (id, key_display, product, tier, seats, valid_until, " - "customer_email, source, trial, notes, created_at) VALUES (?,?,?,?,?,?,?,?,?,?,?)", - (key_id, key_display, req.product, req.tier, req.seats, req.valid_until, - req.customer_email, req.source, 1 if req.trial else 0, req.notes, now), - ) - return KeyResponse(id=key_id, key_display=key_display, product=req.product, - tier=req.tier, seats=req.seats, valid_until=req.valid_until, - revoked=False, customer_email=req.customer_email, - source=req.source, trial=req.trial, notes=req.notes, - created_at=now, active_seat_count=0) -``` - -**Step 8: Fix test `client` fixture** — remove the broken `Depends` in activate and use `_db_dep` properly. Update `tests/test_public_routes.py` fixture to call `set_db_path`: - -```python -# Update the client fixture in tests/test_public_routes.py -@pytest.fixture() -def client(tmp_path, test_keypair, monkeypatch): - db = tmp_path / "test.db" - private_pem, public_pem = test_keypair - (tmp_path / "private.pem").write_bytes(private_pem) - (tmp_path / "public.pem").write_bytes(public_pem) - monkeypatch.setenv("JWT_PRIVATE_KEY_PATH", str(tmp_path / "private.pem")) - monkeypatch.setenv("JWT_PUBLIC_KEY_PATH", str(tmp_path / "public.pem")) - monkeypatch.setenv("JWT_EXPIRY_DAYS", "30") - monkeypatch.setenv("GRACE_PERIOD_DAYS", "7") - monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token") - monkeypatch.setenv("SERVER_NOTICE", "") - from app.routes._db_dep import set_db_path - set_db_path(db) - from app.main import create_app - init_db(db) - app = create_app(db_path=db) - return TestClient(app) -``` - -Also remove the broken `db_path=Depends(lambda: None)` from route functions — they should call `get_db_path()` directly (already done in the implementation above). - -**Step 9: Run tests to verify they pass** - -```bash -conda run -n job-seeker python -m pytest tests/test_public_routes.py -v -``` -Expected: `7 passed` - -**Step 10: Commit** - -```bash -git add -A -git commit -m "feat: public routes — activate, refresh, deactivate with seat enforcement" -``` - ---- - -### Task 5: Public routes — usage + flag; Admin routes - -**Files:** -- Modify: `app/routes/public.py` (add `/usage`, `/flag`) -- Modify: `app/routes/admin.py` (add list, delete, activations, usage, flags endpoints) -- Modify: `tests/test_public_routes.py` (add usage/flag tests) -- Create: `tests/test_admin_routes.py` - -**Step 1: Add usage/flag tests to `tests/test_public_routes.py`** - -```python -def test_usage_event_recorded(client, active_key): - act = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "m1", "product": "peregrine" - }) - token = act.json()["jwt"] - resp = client.post("/v1/usage", json={ - "event_type": "cover_letter_generated", - "product": "peregrine", - "metadata": {"job_id": 42}, - }, headers={"Authorization": f"Bearer {token}"}) - assert resp.status_code == 200 - - -def test_flag_recorded(client, active_key): - act = client.post("/v1/activate", json={ - "key": active_key, "machine_id": "m1", "product": "peregrine" - }) - token = act.json()["jwt"] - resp = client.post("/v1/flag", json={ - "flag_type": "content_violation", - "product": "peregrine", - "details": {"prompt_snippet": "test"}, - }, headers={"Authorization": f"Bearer {token}"}) - assert resp.status_code == 200 - - -def test_usage_with_invalid_jwt_rejected(client): - resp = client.post("/v1/usage", json={ - "event_type": "test", "product": "peregrine" - }, headers={"Authorization": "Bearer not-a-jwt"}) - assert resp.status_code == 403 -``` - -**Step 2: Write `tests/test_admin_routes.py`** - -```python -# tests/test_admin_routes.py -import pytest -from fastapi.testclient import TestClient -from app.main import create_app -from app.db import init_db -from app.routes._db_dep import set_db_path - -ADMIN_HDR = {"Authorization": "Bearer test-admin-token"} - - -@pytest.fixture() -def client(tmp_path, test_keypair, monkeypatch): - db = tmp_path / "test.db" - private_pem, public_pem = test_keypair - (tmp_path / "private.pem").write_bytes(private_pem) - (tmp_path / "public.pem").write_bytes(public_pem) - monkeypatch.setenv("JWT_PRIVATE_KEY_PATH", str(tmp_path / "private.pem")) - monkeypatch.setenv("JWT_PUBLIC_KEY_PATH", str(tmp_path / "public.pem")) - monkeypatch.setenv("JWT_EXPIRY_DAYS", "30") - monkeypatch.setenv("ADMIN_TOKEN", "test-admin-token") - monkeypatch.setenv("SERVER_NOTICE", "") - set_db_path(db) - init_db(db) - return TestClient(create_app(db_path=db)) - - -def test_create_key_returns_display(client): - resp = client.post("/admin/keys", json={ - "product": "peregrine", "tier": "paid" - }, headers=ADMIN_HDR) - assert resp.status_code == 200 - assert resp.json()["key_display"].startswith("CFG-PRNG-") - - -def test_list_keys(client): - client.post("/admin/keys", json={"product": "peregrine", "tier": "paid"}, - headers=ADMIN_HDR) - resp = client.get("/admin/keys", headers=ADMIN_HDR) - assert resp.status_code == 200 - assert len(resp.json()) == 1 - - -def test_revoke_key(client): - create = client.post("/admin/keys", json={"product": "peregrine", "tier": "paid"}, - headers=ADMIN_HDR) - key_id = create.json()["id"] - resp = client.delete(f"/admin/keys/{key_id}", headers=ADMIN_HDR) - assert resp.status_code == 200 - # Activation should now fail - key_display = create.json()["key_display"] - act = client.post("/v1/activate", json={ - "key": key_display, "machine_id": "m1", "product": "peregrine" - }) - assert act.status_code == 403 - - -def test_admin_requires_token(client): - resp = client.get("/admin/keys", headers={"Authorization": "Bearer wrong"}) - assert resp.status_code == 401 - - -def test_admin_usage_returns_events(client): - # Create key, activate, report usage - create = client.post("/admin/keys", json={"product": "peregrine", "tier": "paid"}, - headers=ADMIN_HDR) - key_display = create.json()["key_display"] - act = client.post("/v1/activate", json={ - "key": key_display, "machine_id": "m1", "product": "peregrine" - }) - token = act.json()["jwt"] - client.post("/v1/usage", json={"event_type": "cover_letter_generated", - "product": "peregrine"}, - headers={"Authorization": f"Bearer {token}"}) - resp = client.get("/admin/usage", headers=ADMIN_HDR) - assert resp.status_code == 200 - assert len(resp.json()) >= 1 - - -def test_admin_flags_returns_list(client): - create = client.post("/admin/keys", json={"product": "peregrine", "tier": "paid"}, - headers=ADMIN_HDR) - key_display = create.json()["key_display"] - act = client.post("/v1/activate", json={ - "key": key_display, "machine_id": "m1", "product": "peregrine" - }) - token = act.json()["jwt"] - client.post("/v1/flag", json={"flag_type": "content_violation", "product": "peregrine"}, - headers={"Authorization": f"Bearer {token}"}) - resp = client.get("/admin/flags", headers=ADMIN_HDR) - assert resp.status_code == 200 - flags = resp.json() - assert len(flags) == 1 - assert flags[0]["status"] == "open" -``` - -**Step 3: Run to verify failure** - -```bash -conda run -n job-seeker python -m pytest tests/test_public_routes.py tests/test_admin_routes.py -v -``` -Expected: failures on new tests - -**Step 4: Add `/usage` and `/flag` to `app/routes/public.py`** - -```python -# Add these imports at top of public.py -import json as _json -from fastapi import Header - -# Add to router (append after deactivate): - -def _jwt_bearer(authorization: str = Header(...)) -> dict: - try: - token = authorization.removeprefix("Bearer ") - return verify_jwt(token) - except pyjwt.exceptions.PyJWTError as e: - raise HTTPException(status_code=403, detail=str(e)) - - -@router.post("/usage") -def record_usage(req: UsageRequest, payload: dict = Depends(_jwt_bearer)): - from app.routes._db_dep import get_db_path - with get_db(get_db_path()) as conn: - key_row = conn.execute( - "SELECT id FROM license_keys WHERE key_display=?", - (payload.get("sub", ""),), - ).fetchone() - if not key_row: - raise HTTPException(status_code=403, detail="Key not found") - conn.execute( - "INSERT INTO usage_events (id, key_id, machine_id, product, event_type, metadata, created_at) " - "VALUES (?,?,?,?,?,?,?)", - (str(uuid.uuid4()), key_row["id"], payload.get("machine", ""), - req.product, req.event_type, - _json.dumps(req.metadata) if req.metadata else None, _now()), - ) - return {"status": "recorded"} - - -@router.post("/flag") -def record_flag(req: FlagRequest, payload: dict = Depends(_jwt_bearer)): - from app.routes._db_dep import get_db_path - with get_db(get_db_path()) as conn: - key_row = conn.execute( - "SELECT id FROM license_keys WHERE key_display=?", - (payload.get("sub", ""),), - ).fetchone() - if not key_row: - raise HTTPException(status_code=403, detail="Key not found") - conn.execute( - "INSERT INTO flags (id, key_id, machine_id, product, flag_type, details, created_at) " - "VALUES (?,?,?,?,?,?,?)", - (str(uuid.uuid4()), key_row["id"], payload.get("machine", ""), - req.product, req.flag_type, - _json.dumps(req.details) if req.details else None, _now()), - ) - return {"status": "flagged"} -``` - -**Step 5: Complete `app/routes/admin.py`** — add GET keys, DELETE, activations, usage, flags, PATCH flag: - -```python -# Append to app/routes/admin.py - -@router.get("/keys") -def list_keys(authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - rows = conn.execute("SELECT * FROM license_keys ORDER BY created_at DESC").fetchall() - result = [] - for row in rows: - seat_count = conn.execute( - "SELECT COUNT(*) FROM activations WHERE key_id=? AND deactivated_at IS NULL", - (row["id"],), - ).fetchone()[0] - result.append({**dict(row), "active_seat_count": seat_count, "revoked": bool(row["revoked"])}) - return result - - -@router.delete("/keys/{key_id}") -def revoke_key(key_id: str, authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - row = conn.execute("SELECT id FROM license_keys WHERE id=?", (key_id,)).fetchone() - if not row: - raise HTTPException(status_code=404, detail="Key not found") - now = datetime.now(timezone.utc).isoformat() - conn.execute("UPDATE license_keys SET revoked=1 WHERE id=?", (key_id,)) - conn.execute( - "INSERT INTO audit_log (id, entity_type, entity_id, action, created_at) " - "VALUES (?,?,?,?,?)", - (str(uuid.uuid4()), "key", key_id, "revoked", now), - ) - return {"status": "revoked"} - - -@router.get("/activations") -def list_activations(authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - rows = conn.execute( - "SELECT a.*, k.key_display, k.product FROM activations a " - "JOIN license_keys k ON k.id=a.key_id ORDER BY a.activated_at DESC" - ).fetchall() - return [dict(r) for r in rows] - - -@router.get("/usage") -def list_usage(key_id: str | None = None, authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - if key_id: - rows = conn.execute( - "SELECT * FROM usage_events WHERE key_id=? ORDER BY created_at DESC", - (key_id,), - ).fetchall() - else: - rows = conn.execute( - "SELECT * FROM usage_events ORDER BY created_at DESC LIMIT 500" - ).fetchall() - return [dict(r) for r in rows] - - -@router.get("/flags") -def list_flags(status: str = "open", authorization: str = Header(...)): - _require_admin(authorization) - with get_db(get_db_path()) as conn: - rows = conn.execute( - "SELECT * FROM flags WHERE status=? ORDER BY created_at DESC", (status,) - ).fetchall() - return [dict(r) for r in rows] - - -@router.patch("/flags/{flag_id}") -def update_flag(flag_id: str, req: "FlagUpdateRequest", authorization: str = Header(...)): - from app.models import FlagUpdateRequest as FUR - _require_admin(authorization) - with get_db(get_db_path()) as conn: - row = conn.execute("SELECT id FROM flags WHERE id=?", (flag_id,)).fetchone() - if not row: - raise HTTPException(status_code=404, detail="Flag not found") - now = datetime.now(timezone.utc).isoformat() - conn.execute( - "UPDATE flags SET status=?, action_taken=?, reviewed_at=? WHERE id=?", - (req.status, req.action_taken, now, flag_id), - ) - conn.execute( - "INSERT INTO audit_log (id, entity_type, entity_id, action, created_at) " - "VALUES (?,?,?,?,?)", - (str(uuid.uuid4()), "flag", flag_id, f"flag_{req.status}", now), - ) - return {"status": "updated"} -``` - -Add `from app.models import FlagUpdateRequest` to the imports at top of admin.py. - -**Step 6: Run all server tests** - -```bash -conda run -n job-seeker python -m pytest tests/ -v -``` -Expected: all tests pass - -**Step 7: Commit** - -```bash -git add -A -git commit -m "feat: usage/flag endpoints + complete admin CRUD" -``` - ---- - -### Task 6: Docker + infrastructure files - -**Files:** -- Create: `Dockerfile` -- Create: `docker-compose.yml` -- Create: `.env.example` -- Create: `keys/README.md` - -**Step 1: Write `Dockerfile`** - -```dockerfile -FROM python:3.12-slim -WORKDIR /app -COPY requirements.txt . -RUN pip install --no-cache-dir -r requirements.txt -COPY app/ ./app/ -EXPOSE 8600 -CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8600", "--workers", "1"] -``` - -**Step 2: Write `docker-compose.yml`** - -```yaml -services: - license: - build: . - restart: unless-stopped - ports: - - "127.0.0.1:8600:8600" - volumes: - - license_data:/app/data - - ./keys:/app/keys:ro - env_file: .env - -volumes: - license_data: -``` - -**Step 3: Write `.env.example`** - -```bash -# Copy to .env and fill in values — never commit .env -ADMIN_TOKEN=replace-with-long-random-string -JWT_PRIVATE_KEY_PATH=/app/keys/private.pem -JWT_PUBLIC_KEY_PATH=/app/keys/public.pem -JWT_EXPIRY_DAYS=30 -GRACE_PERIOD_DAYS=7 -# Optional: shown to users as a banner on next JWT refresh -SERVER_NOTICE= -``` - -**Step 4: Write `keys/README.md`** - -```markdown -# Keys - -Generate the RSA keypair once on the server, then copy `public.pem` into the Peregrine repo. - -```bash -openssl genrsa -out private.pem 2048 -openssl rsa -in private.pem -pubout -out public.pem -``` - -- `private.pem` — NEVER commit. Stays on Heimdall only. -- `public.pem` — committed to this repo AND to `peregrine/scripts/license_public_key.pem`. -``` - -**Step 5: Write `scripts/issue-key.sh`** - -```bash -#!/usr/bin/env bash -# scripts/issue-key.sh — Issue a CircuitForge license key -# Usage: ./scripts/issue-key.sh [--product peregrine] [--tier paid] [--seats 2] -# [--email user@example.com] [--notes "Beta user"] -# [--trial] [--valid-until 2027-01-01] - -set -euo pipefail - -SERVER="${LICENSE_SERVER:-https://license.circuitforge.com}" -TOKEN="${ADMIN_TOKEN:-}" - -if [[ -z "$TOKEN" ]]; then - echo "Error: set ADMIN_TOKEN env var" >&2 - exit 1 -fi - -PRODUCT="peregrine" -TIER="paid" -SEATS=1 -EMAIL="" -NOTES="" -TRIAL="false" -VALID_UNTIL="null" - -while [[ $# -gt 0 ]]; do - case "$1" in - --product) PRODUCT="$2"; shift 2 ;; - --tier) TIER="$2"; shift 2 ;; - --seats) SEATS="$2"; shift 2 ;; - --email) EMAIL="$2"; shift 2 ;; - --notes) NOTES="$2"; shift 2 ;; - --trial) TRIAL="true"; shift 1 ;; - --valid-until) VALID_UNTIL="\"$2\""; shift 2 ;; - *) echo "Unknown arg: $1" >&2; exit 1 ;; - esac -done - -EMAIL_JSON=$([ -n "$EMAIL" ] && echo "\"$EMAIL\"" || echo "null") -NOTES_JSON=$([ -n "$NOTES" ] && echo "\"$NOTES\"" || echo "null") - -curl -s -X POST "$SERVER/admin/keys" \ - -H "Authorization: Bearer $TOKEN" \ - -H "Content-Type: application/json" \ - -d "{ - \"product\": \"$PRODUCT\", - \"tier\": \"$TIER\", - \"seats\": $SEATS, - \"valid_until\": $VALID_UNTIL, - \"customer_email\": $EMAIL_JSON, - \"source\": \"manual\", - \"trial\": $TRIAL, - \"notes\": $NOTES_JSON - }" | python3 -c " -import json, sys -data = json.load(sys.stdin) -if 'key_display' in data: - print(f'Key: {data[\"key_display\"]}') - print(f'ID: {data[\"id\"]}') - print(f'Tier: {data[\"tier\"]} ({data[\"seats\"]} seat(s))') -else: - print('Error:', json.dumps(data, indent=2)) -" -``` - -```bash -chmod +x scripts/issue-key.sh -``` - -**Step 6: Commit** - -```bash -git add -A -git commit -m "feat: Dockerfile, docker-compose.yml, .env.example, issue-key.sh" -``` - ---- - -### Task 7: Init Forgejo repo + push - -**Step 1: Create repo on Forgejo** - -Using `gh` CLI configured for your Forgejo instance, or via the web UI at `https://git.opensourcesolarpunk.com`. Create a **private** repo named `circuitforge-license` under the `pyr0ball` user. - -```bash -# If gh is configured for Forgejo: -gh repo create pyr0ball/circuitforge-license --private \ - --gitea-url https://git.opensourcesolarpunk.com - -# Or create manually at https://git.opensourcesolarpunk.com and add remote: -cd /Library/Development/devl/circuitforge-license -git remote add origin https://git.opensourcesolarpunk.com/pyr0ball/circuitforge-license.git -``` - -**Step 2: Push** - -```bash -git push -u origin main -``` - -**Step 3: Generate real keypair on Heimdall (do once, after deployment)** - -```bash -# SSH to Heimdall or run locally — keys go in circuitforge-license/keys/ -mkdir -p /Library/Development/CircuitForge/circuitforge-license/keys -cd /Library/Development/CircuitForge/circuitforge-license/keys -openssl genrsa -out private.pem 2048 -openssl rsa -in private.pem -pubout -out public.pem -git add public.pem -git commit -m "chore: add RSA public key" -git push -``` - ---- - -## PART B — Peregrine Client Integration - -**Working directory for all Part B tasks:** `/Library/Development/devl/peregrine/` -**Run tests:** `/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v` - ---- - -### Task 8: `scripts/license.py` + public key - -**Files:** -- Create: `scripts/license_public_key.pem` (copy from license server `keys/public.pem`) -- Create: `scripts/license.py` -- Create: `tests/test_license.py` - -**Step 1: Copy the public key** - -```bash -cp /Library/Development/CircuitForge/circuitforge-license/keys/public.pem \ - /Library/Development/devl/peregrine/scripts/license_public_key.pem -``` - -**Step 2: Write failing tests** - -```python -# tests/test_license.py -import json -import pytest -from pathlib import Path -from unittest.mock import patch, MagicMock -from cryptography.hazmat.primitives.asymmetric import rsa -from cryptography.hazmat.primitives import serialization -import jwt as pyjwt -from datetime import datetime, timedelta, timezone - - -@pytest.fixture() -def test_keys(tmp_path): - """Generate test RSA keypair and return (private_pem, public_pem, public_path).""" - private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) - private_pem = private_key.private_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PrivateFormat.TraditionalOpenSSL, - encryption_algorithm=serialization.NoEncryption(), - ) - public_pem = private_key.public_key().public_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PublicFormat.SubjectPublicKeyInfo, - ) - public_path = tmp_path / "test_public.pem" - public_path.write_bytes(public_pem) - return private_pem, public_pem, public_path - - -def _make_jwt(private_pem: bytes, tier: str = "paid", - product: str = "peregrine", - exp_delta_days: int = 30, - machine: str = "test-machine") -> str: - now = datetime.now(timezone.utc) - payload = { - "sub": "CFG-PRNG-TEST-TEST-TEST", - "product": product, - "tier": tier, - "seats": 1, - "machine": machine, - "iat": now, - "exp": now + timedelta(days=exp_delta_days), - } - return pyjwt.encode(payload, private_pem, algorithm="RS256") - - -def _write_license(tmp_path, jwt_token: str, grace_until: str | None = None) -> Path: - data = { - "jwt": jwt_token, - "key_display": "CFG-PRNG-TEST-TEST-TEST", - "tier": "paid", - "valid_until": None, - "machine_id": "test-machine", - "last_refresh": datetime.now(timezone.utc).isoformat(), - "grace_until": grace_until, - } - p = tmp_path / "license.json" - p.write_text(json.dumps(data)) - return p - - -class TestVerifyLocal: - def test_valid_jwt_returns_tier(self, test_keys, tmp_path): - private_pem, _, public_path = test_keys - token = _make_jwt(private_pem) - license_path = _write_license(tmp_path, token) - from scripts.license import verify_local - result = verify_local(license_path=license_path, public_key_path=public_path) - assert result is not None - assert result["tier"] == "paid" - - def test_missing_file_returns_none(self, tmp_path): - from scripts.license import verify_local - result = verify_local(license_path=tmp_path / "missing.json", - public_key_path=tmp_path / "key.pem") - assert result is None - - def test_wrong_product_returns_none(self, test_keys, tmp_path): - private_pem, _, public_path = test_keys - token = _make_jwt(private_pem, product="falcon") - license_path = _write_license(tmp_path, token) - from scripts.license import verify_local - result = verify_local(license_path=license_path, public_key_path=public_path) - assert result is None - - def test_expired_within_grace_returns_tier(self, test_keys, tmp_path): - private_pem, _, public_path = test_keys - token = _make_jwt(private_pem, exp_delta_days=-1) - grace_until = (datetime.now(timezone.utc) + timedelta(days=3)).isoformat() - license_path = _write_license(tmp_path, token, grace_until=grace_until) - from scripts.license import verify_local - result = verify_local(license_path=license_path, public_key_path=public_path) - assert result is not None - assert result["tier"] == "paid" - assert result["in_grace"] is True - - def test_expired_past_grace_returns_none(self, test_keys, tmp_path): - private_pem, _, public_path = test_keys - token = _make_jwt(private_pem, exp_delta_days=-10) - grace_until = (datetime.now(timezone.utc) - timedelta(days=1)).isoformat() - license_path = _write_license(tmp_path, token, grace_until=grace_until) - from scripts.license import verify_local - result = verify_local(license_path=license_path, public_key_path=public_path) - assert result is None - - -class TestEffectiveTier: - def test_returns_free_when_no_license(self, tmp_path): - from scripts.license import effective_tier - result = effective_tier( - license_path=tmp_path / "missing.json", - public_key_path=tmp_path / "key.pem", - ) - assert result == "free" - - def test_returns_tier_from_valid_jwt(self, test_keys, tmp_path): - private_pem, _, public_path = test_keys - token = _make_jwt(private_pem, tier="premium") - license_path = _write_license(tmp_path, token) - from scripts.license import effective_tier - result = effective_tier(license_path=license_path, public_key_path=public_path) - assert result == "premium" -``` - -**Step 3: Run to verify failure** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_license.py -v -``` -Expected: `FAILED` — `ModuleNotFoundError: No module named 'scripts.license'` - -**Step 4: Write `scripts/license.py`** - -```python -# scripts/license.py -""" -CircuitForge license client for Peregrine. - -Activates against the license server, caches a signed JWT locally, -and verifies tier offline using the embedded RS256 public key. - -All functions accept override paths for testing; production code uses -the module-level defaults. -""" -from __future__ import annotations - -import hashlib -import json -import socket -import threading -import uuid -from datetime import datetime, timedelta, timezone -from pathlib import Path -from typing import Any - -import jwt as pyjwt - -_HERE = Path(__file__).parent -_DEFAULT_LICENSE_PATH = _HERE.parent / "config" / "license.json" -_DEFAULT_PUBLIC_KEY_PATH = _HERE / "license_public_key.pem" -_LICENSE_SERVER = "https://license.circuitforge.com" -_PRODUCT = "peregrine" -_REFRESH_THRESHOLD_DAYS = 5 -_GRACE_PERIOD_DAYS = 7 - - -# ── Machine fingerprint ──────────────────────────────────────────────────────── - -def _machine_id() -> str: - raw = f"{socket.gethostname()}-{uuid.getnode()}" - return hashlib.sha256(raw.encode()).hexdigest()[:32] - - -# ── License file helpers ─────────────────────────────────────────────────────── - -def _read_license(license_path: Path) -> dict | None: - try: - return json.loads(license_path.read_text()) - except (FileNotFoundError, json.JSONDecodeError, OSError): - return None - - -def _write_license(data: dict, license_path: Path) -> None: - license_path.parent.mkdir(parents=True, exist_ok=True) - license_path.write_text(json.dumps(data, indent=2)) - - -# ── Core verify ─────────────────────────────────────────────────────────────── - -def verify_local( - license_path: Path = _DEFAULT_LICENSE_PATH, - public_key_path: Path = _DEFAULT_PUBLIC_KEY_PATH, -) -> dict | None: - """Verify the cached JWT offline. Returns payload dict or None (= free tier). - - Returns dict has keys: tier, in_grace (bool), sub, product, notice (optional). - """ - stored = _read_license(license_path) - if not stored or not stored.get("jwt"): - return None - - if not public_key_path.exists(): - return None - - public_key = public_key_path.read_bytes() - - try: - payload = pyjwt.decode(stored["jwt"], public_key, algorithms=["RS256"]) - # Valid and not expired - if payload.get("product") != _PRODUCT: - return None - return {**payload, "in_grace": False} - - except pyjwt.exceptions.ExpiredSignatureError: - # JWT expired — check grace period - grace_until_str = stored.get("grace_until") - if not grace_until_str: - return None - try: - grace_until = datetime.fromisoformat(grace_until_str) - if grace_until.tzinfo is None: - grace_until = grace_until.replace(tzinfo=timezone.utc) - except ValueError: - return None - if datetime.now(timezone.utc) > grace_until: - return None - # Decode without verification to get payload - try: - payload = pyjwt.decode(stored["jwt"], public_key, - algorithms=["RS256"], - options={"verify_exp": False}) - if payload.get("product") != _PRODUCT: - return None - return {**payload, "in_grace": True} - except pyjwt.exceptions.PyJWTError: - return None - - except pyjwt.exceptions.PyJWTError: - return None - - -def effective_tier( - license_path: Path = _DEFAULT_LICENSE_PATH, - public_key_path: Path = _DEFAULT_PUBLIC_KEY_PATH, -) -> str: - """Return the effective tier string. Falls back to 'free' on any problem.""" - result = verify_local(license_path=license_path, public_key_path=public_key_path) - if result is None: - return "free" - return result.get("tier", "free") - - -# ── Network operations (all fire-and-forget or explicit) ────────────────────── - -def activate( - key: str, - license_path: Path = _DEFAULT_LICENSE_PATH, - public_key_path: Path = _DEFAULT_PUBLIC_KEY_PATH, - app_version: str | None = None, -) -> dict: - """Activate a license key. Returns response dict. Raises on failure.""" - import httpx - mid = _machine_id() - resp = httpx.post( - f"{_LICENSE_SERVER}/v1/activate", - json={"key": key, "machine_id": mid, "product": _PRODUCT, - "app_version": app_version, "platform": _detect_platform()}, - timeout=10, - ) - resp.raise_for_status() - data = resp.json() - stored = { - "jwt": data["jwt"], - "key_display": key, - "tier": data["tier"], - "valid_until": data.get("valid_until"), - "machine_id": mid, - "last_refresh": datetime.now(timezone.utc).isoformat(), - "grace_until": None, - } - _write_license(stored, license_path) - return data - - -def deactivate( - license_path: Path = _DEFAULT_LICENSE_PATH, -) -> None: - """Deactivate this machine. Deletes license.json.""" - import httpx - stored = _read_license(license_path) - if not stored: - return - try: - httpx.post( - f"{_LICENSE_SERVER}/v1/deactivate", - json={"jwt": stored["jwt"], "machine_id": stored.get("machine_id", _machine_id())}, - timeout=10, - ) - except Exception: - pass # best-effort - license_path.unlink(missing_ok=True) - - -def refresh_if_needed( - license_path: Path = _DEFAULT_LICENSE_PATH, - public_key_path: Path = _DEFAULT_PUBLIC_KEY_PATH, -) -> None: - """Silently refresh JWT if it expires within threshold. No-op on network failure.""" - stored = _read_license(license_path) - if not stored or not stored.get("jwt"): - return - try: - payload = pyjwt.decode(stored["jwt"], public_key_path.read_bytes(), - algorithms=["RS256"]) - exp = datetime.fromtimestamp(payload["exp"], tz=timezone.utc) - if exp - datetime.now(timezone.utc) > timedelta(days=_REFRESH_THRESHOLD_DAYS): - return - except pyjwt.exceptions.ExpiredSignatureError: - # Already expired — try to refresh anyway, set grace if unreachable - pass - except Exception: - return - - try: - import httpx - resp = httpx.post( - f"{_LICENSE_SERVER}/v1/refresh", - json={"jwt": stored["jwt"], - "machine_id": stored.get("machine_id", _machine_id())}, - timeout=10, - ) - resp.raise_for_status() - data = resp.json() - stored["jwt"] = data["jwt"] - stored["tier"] = data["tier"] - stored["last_refresh"] = datetime.now(timezone.utc).isoformat() - stored["grace_until"] = None - _write_license(stored, license_path) - except Exception: - # Unreachable — set grace period if not already set - if not stored.get("grace_until"): - grace = datetime.now(timezone.utc) + timedelta(days=_GRACE_PERIOD_DAYS) - stored["grace_until"] = grace.isoformat() - _write_license(stored, license_path) - - -def report_usage( - event_type: str, - metadata: dict | None = None, - license_path: Path = _DEFAULT_LICENSE_PATH, -) -> None: - """Fire-and-forget usage telemetry. Never blocks, never raises.""" - stored = _read_license(license_path) - if not stored or not stored.get("jwt"): - return - - def _send(): - try: - import httpx - httpx.post( - f"{_LICENSE_SERVER}/v1/usage", - json={"event_type": event_type, "product": _PRODUCT, - "metadata": metadata or {}}, - headers={"Authorization": f"Bearer {stored['jwt']}"}, - timeout=5, - ) - except Exception: - pass - - threading.Thread(target=_send, daemon=True).start() - - -def report_flag( - flag_type: str, - details: dict | None = None, - license_path: Path = _DEFAULT_LICENSE_PATH, -) -> None: - """Fire-and-forget violation report. Never blocks, never raises.""" - stored = _read_license(license_path) - if not stored or not stored.get("jwt"): - return - - def _send(): - try: - import httpx - httpx.post( - f"{_LICENSE_SERVER}/v1/flag", - json={"flag_type": flag_type, "product": _PRODUCT, - "details": details or {}}, - headers={"Authorization": f"Bearer {stored['jwt']}"}, - timeout=5, - ) - except Exception: - pass - - threading.Thread(target=_send, daemon=True).start() - - -def _detect_platform() -> str: - import sys - if sys.platform.startswith("linux"): - return "linux" - if sys.platform == "darwin": - return "macos" - if sys.platform == "win32": - return "windows" - return "unknown" -``` - -**Step 5: Run tests to verify they pass** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_license.py -v -``` -Expected: all tests pass - -**Step 6: Commit** - -```bash -cd /Library/Development/devl/peregrine -git add scripts/license.py scripts/license_public_key.pem tests/test_license.py -git commit -m "feat: license.py client — verify_local, effective_tier, activate, refresh, report_usage" -``` - ---- - -### Task 9: Wire `tiers.py` + update `.gitignore` - -**Files:** -- Modify: `app/wizard/tiers.py` -- Modify: `.gitignore` -- Create: `tests/test_license_tier_integration.py` - -**Step 1: Write failing test** - -```python -# tests/test_license_tier_integration.py -import json -import pytest -from pathlib import Path -from datetime import datetime, timedelta, timezone -from unittest.mock import patch -from cryptography.hazmat.primitives.asymmetric import rsa -from cryptography.hazmat.primitives import serialization -import jwt as pyjwt - - -@pytest.fixture() -def license_env(tmp_path): - """Returns (private_pem, public_path, license_path) for tier integration tests.""" - private_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) - private_pem = private_key.private_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PrivateFormat.TraditionalOpenSSL, - encryption_algorithm=serialization.NoEncryption(), - ) - public_pem = private_key.public_key().public_bytes( - encoding=serialization.Encoding.PEM, - format=serialization.PublicFormat.SubjectPublicKeyInfo, - ) - public_path = tmp_path / "public.pem" - public_path.write_bytes(public_pem) - license_path = tmp_path / "license.json" - return private_pem, public_path, license_path - - -def _write_jwt_license(license_path, private_pem, tier="paid", days=30): - now = datetime.now(timezone.utc) - token = pyjwt.encode({ - "sub": "CFG-PRNG-TEST", "product": "peregrine", "tier": tier, - "iat": now, "exp": now + timedelta(days=days), - }, private_pem, algorithm="RS256") - license_path.write_text(json.dumps({"jwt": token, "grace_until": None})) - - -def test_effective_tier_free_without_license(tmp_path): - from app.wizard.tiers import effective_tier - tier = effective_tier( - profile=None, - license_path=tmp_path / "missing.json", - public_key_path=tmp_path / "key.pem", - ) - assert tier == "free" - - -def test_effective_tier_paid_with_valid_license(license_env): - private_pem, public_path, license_path = license_env - _write_jwt_license(license_path, private_pem, tier="paid") - from app.wizard.tiers import effective_tier - tier = effective_tier(profile=None, license_path=license_path, - public_key_path=public_path) - assert tier == "paid" - - -def test_effective_tier_dev_override_takes_precedence(license_env): - """dev_tier_override wins even when a valid license is present.""" - private_pem, public_path, license_path = license_env - _write_jwt_license(license_path, private_pem, tier="paid") - - class FakeProfile: - dev_tier_override = "premium" - - from app.wizard.tiers import effective_tier - tier = effective_tier(profile=FakeProfile(), license_path=license_path, - public_key_path=public_path) - assert tier == "premium" -``` - -**Step 2: Run to verify failure** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_license_tier_integration.py -v -``` -Expected: `FAILED` — `effective_tier() got unexpected keyword argument 'license_path'` - -**Step 3: Update `app/wizard/tiers.py`** — add `effective_tier()` function - -```python -# Add at bottom of app/wizard/tiers.py (after existing functions): - -def effective_tier( - profile=None, - license_path=None, - public_key_path=None, -) -> str: - """Return the effective tier for this installation. - - Priority: - 1. profile.dev_tier_override (developer mode override) - 2. License JWT verification (offline RS256 check) - 3. "free" (fallback) - - license_path and public_key_path default to production paths when None. - Pass explicit paths in tests to avoid touching real files. - """ - if profile and getattr(profile, "dev_tier_override", None): - return profile.dev_tier_override - - from scripts.license import effective_tier as _license_tier - from pathlib import Path as _Path - - kwargs = {} - if license_path is not None: - kwargs["license_path"] = _Path(license_path) - if public_key_path is not None: - kwargs["public_key_path"] = _Path(public_key_path) - return _license_tier(**kwargs) -``` - -**Step 4: Add `config/license.json` to `.gitignore`** - -Open `/Library/Development/devl/peregrine/.gitignore` and add: -``` -config/license.json -``` - -**Step 5: Run tests** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_license_tier_integration.py -v -``` -Expected: `3 passed` - -**Step 6: Run full suite to check for regressions** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` -Expected: all existing tests still pass - -**Step 7: Commit** - -```bash -git add app/wizard/tiers.py .gitignore tests/test_license_tier_integration.py -git commit -m "feat: wire license.effective_tier into tiers.py; add dev_override priority" -``` - ---- - -### Task 10: Settings License tab + app.py startup refresh - -**Files:** -- Modify: `app/pages/2_Settings.py` (add License tab) -- Modify: `app/app.py` (call `refresh_if_needed` on startup) - -**Step 1: Add License tab to `app/pages/2_Settings.py`** - -Find the `_tab_names` list and insert `"🔑 License"` after `"🛠️ Developer"` (or at the end of the list before Developer). Then find the corresponding tab variable assignment block and add: - -```python -# In the tab variables section: -tab_license = _all_tabs[] -``` - -Then add the license tab content block: - -```python -# ── License tab ────────────────────────────────────────────────────────────── -with tab_license: - st.subheader("🔑 License") - - from scripts.license import ( - verify_local as _verify_local, - activate as _activate, - deactivate as _deactivate, - _DEFAULT_LICENSE_PATH, - _DEFAULT_PUBLIC_KEY_PATH, - ) - - _lic = _verify_local() - - if _lic: - # Active license - _grace_note = " _(grace period active)_" if _lic.get("in_grace") else "" - st.success(f"**{_lic['tier'].title()} tier** active{_grace_note}") - st.caption(f"Key: `{_DEFAULT_LICENSE_PATH.exists() and __import__('json').loads(_DEFAULT_LICENSE_PATH.read_text()).get('key_display', '—') or '—'}`") - if _lic.get("notice"): - st.info(_lic["notice"]) - if st.button("Deactivate this machine", type="secondary"): - _deactivate() - st.success("Deactivated. Restart the app to apply.") - st.rerun() - else: - st.info("No active license — running on **free tier**.") - st.caption("Enter a license key to unlock paid features.") - _key_input = st.text_input( - "License key", - placeholder="CFG-PRNG-XXXX-XXXX-XXXX", - label_visibility="collapsed", - ) - if st.button("Activate", disabled=not (_key_input or "").strip()): - with st.spinner("Activating…"): - try: - result = _activate(_key_input.strip()) - st.success(f"Activated! Tier: **{result['tier']}**") - st.rerun() - except Exception as _e: - st.error(f"Activation failed: {_e}") -``` - -**Step 2: Add startup refresh to `app/app.py`** - -Find the startup block (near where `init_db` is called, before `st.navigation`). Add: - -```python -# Silent license refresh on startup — no-op if unreachable -try: - from scripts.license import refresh_if_needed as _refresh_license - _refresh_license() -except Exception: - pass -``` - -**Step 3: Run full test suite** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` -Expected: all tests pass (License tab is UI-only, no new unit tests needed — covered by existing Settings tests for tab structure) - -**Step 4: Commit** - -```bash -git add app/pages/2_Settings.py app/app.py -git commit -m "feat: License tab in Settings (activate/deactivate UI) + startup refresh" -``` - ---- - -### Task 11: Final check + Forgejo push - -**Step 1: Run full suite one last time** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v --tb=short -``` -Expected: all tests pass - -**Step 2: Push Peregrine to Forgejo** - -```bash -cd /Library/Development/devl/peregrine -git push origin main -``` - -**Step 3: Verify Caddy route is ready** - -Add to `/opt/containers/caddy/Caddyfile` on Heimdall (SSH in and edit): - -```caddy -license.circuitforge.com { - reverse_proxy localhost:8600 -} -``` - -Reload Caddy: -```bash -docker exec caddy-proxy caddy reload --config /etc/caddy/Caddyfile -``` - -**Step 4: Deploy license server on Heimdall** - -```bash -# SSH to Heimdall -cd /devl/circuitforge-license # live clone lives here -cp .env.example .env -# Edit .env: set ADMIN_TOKEN to a long random string -# keys/ already has private.pem + public.pem from Task 7 step 3 -docker compose up -d -``` - -**Step 5: Smoke test** - -```bash -# Create a test key -export ADMIN_TOKEN= -./scripts/issue-key.sh --product peregrine --tier paid --email test@example.com -# → Key: CFG-PRNG-XXXX-XXXX-XXXX - -# Test activation from Peregrine machine -curl -X POST https://license.circuitforge.com/v1/activate \ - -H "Content-Type: application/json" \ - -d '{"key":"CFG-PRNG-XXXX-XXXX-XXXX","machine_id":"test","product":"peregrine"}' -# → {"jwt":"eyJ...","tier":"paid",...} -``` - ---- - -## Summary - -| Task | Repo | Deliverable | -|------|------|-------------| -| 1 | license-server | Repo scaffold + DB schema | -| 2 | license-server | `crypto.py` + test keypair fixture | -| 3 | license-server | Pydantic models | -| 4 | license-server | `/v1/activate`, `/v1/refresh`, `/v1/deactivate` | -| 5 | license-server | `/v1/usage`, `/v1/flag`, full admin CRUD | -| 6 | license-server | Docker + Caddy + `issue-key.sh` | -| 7 | license-server | Forgejo push + real keypair | -| 8 | peregrine | `scripts/license.py` + public key | -| 9 | peregrine | `tiers.py` wired + `.gitignore` updated | -| 10 | peregrine | License tab in Settings + startup refresh | -| 11 | both | Deploy to Heimdall + smoke test | diff --git a/docs/plans/2026-02-26-dual-gpu-design.md b/docs/plans/2026-02-26-dual-gpu-design.md deleted file mode 100644 index 860a17a..0000000 --- a/docs/plans/2026-02-26-dual-gpu-design.md +++ /dev/null @@ -1,257 +0,0 @@ -# Peregrine — Dual-GPU / Dual-Inference Design - -**Date:** 2026-02-26 -**Status:** Approved — ready for implementation -**Scope:** Peregrine (reference impl; patterns propagate to future products) - ---- - -## Goal - -Replace the fixed `dual-gpu` profile (Ollama + vLLM hardwired to GPU 0 + GPU 1) with a -`DUAL_GPU_MODE` env var that selects which inference stack occupies GPU 1. Simultaneously -add a first-run download size warning to preflight so users know what they're in for before -Docker starts pulling images and models. - ---- - -## Modes - -| `DUAL_GPU_MODE` | GPU 0 | GPU 1 | Research backend | -|-----------------|-------|-------|-----------------| -| `ollama` (default) | ollama + vision | ollama_research | `ollama_research` | -| `vllm` | ollama + vision | vllm | `vllm_research` | -| `mixed` | ollama + vision | ollama_research + vllm (VRAM-split) | `vllm_research` → `ollama_research` fallback | - -`mixed` requires sufficient VRAM on GPU 1. Preflight warns (not blocks) when GPU 1 has -< 12 GB free before starting in mixed mode. - -Cover letters always use `ollama` on GPU 0. Research uses whichever GPU 1 backend is -reachable. The LLM router's `_is_reachable()` check handles this transparently — the -fallback chain simply skips services that aren't running. - ---- - -## Compose Profile Architecture - -Docker Compose profiles used to gate which services start per mode. -`DUAL_GPU_MODE` is read by the Makefile and passed as a second `--profile` flag. - -### Service → profile mapping - -| Service | Profiles | -|---------|---------| -| `ollama` | `cpu`, `single-gpu`, `dual-gpu-ollama`, `dual-gpu-vllm`, `dual-gpu-mixed` | -| `vision` | `single-gpu`, `dual-gpu-ollama`, `dual-gpu-vllm`, `dual-gpu-mixed` | -| `ollama_research` | `dual-gpu-ollama`, `dual-gpu-mixed` | -| `vllm` | `dual-gpu-vllm`, `dual-gpu-mixed` | -| `finetune` | `finetune` | - -User-facing profiles remain: `remote`, `cpu`, `single-gpu`, `dual-gpu`. -Sub-profiles (`dual-gpu-ollama`, `dual-gpu-vllm`, `dual-gpu-mixed`) are injected by the -Makefile and never typed by the user. - ---- - -## File Changes - -### `compose.yml` - -**`ollama`** — add all dual-gpu sub-profiles to `profiles`: -```yaml -profiles: [cpu, single-gpu, dual-gpu-ollama, dual-gpu-vllm, dual-gpu-mixed] -``` - -**`vision`** — same pattern: -```yaml -profiles: [single-gpu, dual-gpu-ollama, dual-gpu-vllm, dual-gpu-mixed] -``` - -**`vllm`** — change from `[dual-gpu]` to: -```yaml -profiles: [dual-gpu-vllm, dual-gpu-mixed] -``` - -**`ollama_research`** — new service: -```yaml -ollama_research: - image: ollama/ollama:latest - ports: - - "${OLLAMA_RESEARCH_PORT:-11435}:11434" - volumes: - - ${OLLAMA_MODELS_DIR:-~/models/ollama}:/root/.ollama # shared — no double download - - ./docker/ollama/entrypoint.sh:/entrypoint.sh - environment: - - OLLAMA_MODELS=/root/.ollama - - DEFAULT_OLLAMA_MODEL=${OLLAMA_RESEARCH_MODEL:-llama3.2:3b} - entrypoint: ["/bin/bash", "/entrypoint.sh"] - profiles: [dual-gpu-ollama, dual-gpu-mixed] - restart: unless-stopped -``` - -### `compose.gpu.yml` - -Add `ollama_research` block (GPU 1). `vllm` stays on GPU 1 as-is: -```yaml -ollama_research: - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["1"] - capabilities: [gpu] -``` - -### `compose.podman-gpu.yml` - -Same addition for Podman CDI: -```yaml -ollama_research: - devices: - - nvidia.com/gpu=1 - deploy: - resources: - reservations: - devices: [] -``` - -### `Makefile` - -Two additions after existing `COMPOSE` detection: - -```makefile -DUAL_GPU_MODE ?= $(shell grep -m1 '^DUAL_GPU_MODE=' .env 2>/dev/null | cut -d= -f2 || echo ollama) - -# GPU overlay: matches single-gpu, dual-gpu (findstring gpu already covers these) -# Sub-profile injection for dual-gpu modes: -ifeq ($(PROFILE),dual-gpu) - COMPOSE_FILES += --profile dual-gpu-$(DUAL_GPU_MODE) -endif -``` - -Update `manage.sh` usage block to document `dual-gpu` profile with `DUAL_GPU_MODE` note: -``` -dual-gpu Ollama + Vision on GPU 0; GPU 1 mode set by DUAL_GPU_MODE - DUAL_GPU_MODE=ollama (default) ollama_research on GPU 1 - DUAL_GPU_MODE=vllm vllm on GPU 1 - DUAL_GPU_MODE=mixed both on GPU 1 (VRAM-split; see preflight warning) -``` - -### `scripts/preflight.py` - -**1. `_SERVICES` — add `ollama_research`:** -```python -"ollama_research": ("ollama_research_port", 11435, "OLLAMA_RESEARCH_PORT", True, True), -``` - -**2. `_LLM_BACKENDS` — add entries for both new backends:** -```python -"ollama_research": [("ollama_research", "/v1")], -# vllm_research is an alias for vllm's port — preflight updates base_url for both: -"vllm": [("vllm", "/v1"), ("vllm_research", "/v1")], -``` - -**3. `_DOCKER_INTERNAL` — add `ollama_research`:** -```python -"ollama_research": ("ollama_research", 11434), # container-internal port is always 11434 -``` - -**4. `recommend_profile()` — unchanged** (still returns `"dual-gpu"` for 2 GPUs). -Write `DUAL_GPU_MODE=ollama` to `.env` when first setting up a 2-GPU system. - -**5. Mixed-mode VRAM warning** — after GPU resource section, before closing line: -```python -dual_gpu_mode = os.environ.get("DUAL_GPU_MODE", "ollama") -if dual_gpu_mode == "mixed" and len(gpus) >= 2: - if gpus[1]["vram_free_gb"] < 12: - print(f"║ ⚠ DUAL_GPU_MODE=mixed: GPU 1 has only {gpus[1]['vram_free_gb']:.1f} GB free") - print(f"║ Running ollama_research + vllm together may cause OOM.") - print(f"║ Consider DUAL_GPU_MODE=ollama or DUAL_GPU_MODE=vllm instead.") -``` - -**6. Download size warning** — profile-aware block added just before the closing `╚` line: - -``` -║ Download sizes (first-run estimates) -║ Docker images -║ ollama/ollama ~800 MB (shared by ollama + ollama_research) -║ searxng/searxng ~300 MB -║ app (Python build) ~1.5 GB -║ vision service ~3.0 GB [single-gpu and above] -║ vllm/vllm-openai ~10.0 GB [vllm / mixed mode only] -║ -║ Model weights (lazy-loaded on first use) -║ llama3.2:3b ~2.0 GB → OLLAMA_MODELS_DIR -║ moondream2 ~1.8 GB → vision container cache [single-gpu+] -║ Note: ollama + ollama_research share the same model dir — no double download -║ -║ ⚠ Total first-run: ~X GB (models persist between restarts) -``` - -Total is summed at runtime based on active profile + `DUAL_GPU_MODE`. - -Size table (used by the warning calculator): -| Component | Size | Condition | -|-----------|------|-----------| -| `ollama/ollama` image | 800 MB | cpu, single-gpu, dual-gpu | -| `searxng/searxng` image | 300 MB | always | -| app image | 1,500 MB | always | -| vision service image | 3,000 MB | single-gpu, dual-gpu | -| `vllm/vllm-openai` image | 10,000 MB | vllm or mixed mode | -| llama3.2:3b weights | 2,000 MB | cpu, single-gpu, dual-gpu | -| moondream2 weights | 1,800 MB | single-gpu, dual-gpu | - -### `config/llm.yaml` - -**Add `vllm_research` backend:** -```yaml -vllm_research: - api_key: '' - base_url: http://host.docker.internal:8000/v1 # same port as vllm; preflight keeps in sync - enabled: true - model: __auto__ - supports_images: false - type: openai_compat -``` - -**Update `research_fallback_order`:** -```yaml -research_fallback_order: - - claude_code - - vllm_research - - ollama_research - - github_copilot - - anthropic -``` - -`vllm` stays in the main `fallback_order` (cover letters). `vllm_research` is the explicit -research alias for the same service — different config key, same port, makes routing intent -readable in the YAML. - ---- - -## Downstream Compatibility - -The LLM router requires no changes. `_is_reachable()` already skips backends that aren't -responding. When `DUAL_GPU_MODE=ollama`, `vllm_research` is unreachable and skipped; -`ollama_research` is up and used. When `DUAL_GPU_MODE=vllm`, the reverse. `mixed` mode -makes both reachable; `vllm_research` wins as the higher-priority entry. - -Preflight's `update_llm_yaml()` keeps `base_url` values correct for both adopted (external) -and Docker-internal routing automatically, since `vllm_research` is registered under the -`"vllm"` key in `_LLM_BACKENDS`. - ---- - -## Future Considerations - -- **Triple-GPU / 3+ service configs:** When a third product is active, extract this pattern - into `circuitforge-core` as a reusable inference topology manager. -- **Dual vLLM:** Two vLLM instances (e.g., different model sizes per task) follows the same - pattern — add `vllm_research` as a separate compose service on its own port. -- **VRAM-aware model selection:** Preflight could suggest smaller models when VRAM is tight - in mixed mode (e.g., swap llama3.2:3b → llama3.2:1b for the research instance). -- **Queue optimizer (1-GPU / CPU):** When only one inference backend is available and a batch - of tasks is queued, group by task type (all cover letters first, then all research briefs) - to avoid repeated model context switches. Tracked separately. diff --git a/docs/plans/2026-02-26-dual-gpu-plan.md b/docs/plans/2026-02-26-dual-gpu-plan.md deleted file mode 100644 index 08f84b0..0000000 --- a/docs/plans/2026-02-26-dual-gpu-plan.md +++ /dev/null @@ -1,811 +0,0 @@ -# Dual-GPU / Dual-Inference Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Add `DUAL_GPU_MODE=ollama|vllm|mixed` env var that gates which inference service occupies GPU 1 on dual-GPU systems, plus a first-run download size warning in preflight. - -**Architecture:** Sub-profiles (`dual-gpu-ollama`, `dual-gpu-vllm`, `dual-gpu-mixed`) are injected alongside `--profile dual-gpu` by the Makefile based on `DUAL_GPU_MODE`. The LLM router requires zero changes — `_is_reachable()` naturally skips backends that aren't running. Preflight gains `ollama_research` as a tracked service and emits a size warning block. - -**Tech Stack:** Docker Compose profiles, Python (preflight.py), YAML (llm.yaml, compose files), bash (Makefile, manage.sh) - -**Design doc:** `docs/plans/2026-02-26-dual-gpu-design.md` - -**Test runner:** `conda run -n job-seeker python -m pytest tests/ -v` - ---- - -### Task 1: Update `config/llm.yaml` - -**Files:** -- Modify: `config/llm.yaml` - -**Step 1: Add `vllm_research` backend and update `research_fallback_order`** - -Open `config/llm.yaml`. After the `vllm:` block, add: - -```yaml - vllm_research: - api_key: '' - base_url: http://host.docker.internal:8000/v1 - enabled: true - model: __auto__ - supports_images: false - type: openai_compat -``` - -Replace `research_fallback_order:` section with: - -```yaml -research_fallback_order: -- claude_code -- vllm_research -- ollama_research -- github_copilot -- anthropic -``` - -**Step 2: Verify YAML parses cleanly** - -```bash -conda run -n job-seeker python -c "import yaml; yaml.safe_load(open('config/llm.yaml'))" -``` - -Expected: no output (no error). - -**Step 3: Run existing llm config test** - -```bash -conda run -n job-seeker python -m pytest tests/test_llm_router.py::test_config_loads -v -``` - -Expected: PASS - -**Step 4: Commit** - -```bash -git add config/llm.yaml -git commit -m "feat: add vllm_research backend and update research_fallback_order" -``` - ---- - -### Task 2: Write failing tests for preflight changes - -**Files:** -- Create: `tests/test_preflight.py` - -No existing test file for preflight. Write all tests upfront — they fail until Task 3–5 implement the code. - -**Step 1: Create `tests/test_preflight.py`** - -```python -"""Tests for scripts/preflight.py additions: dual-GPU service table, size warning, VRAM check.""" -import pytest -from pathlib import Path -from unittest.mock import patch -import yaml -import tempfile -import os - - -# ── Service table ────────────────────────────────────────────────────────────── - -def test_ollama_research_in_services(): - """ollama_research must be in _SERVICES at port 11435.""" - from scripts.preflight import _SERVICES - assert "ollama_research" in _SERVICES - _, default_port, env_var, docker_owned, adoptable = _SERVICES["ollama_research"] - assert default_port == 11435 - assert env_var == "OLLAMA_RESEARCH_PORT" - assert docker_owned is True - assert adoptable is True - - -def test_ollama_research_in_llm_backends(): - """ollama_research must be a standalone key in _LLM_BACKENDS (not nested under ollama).""" - from scripts.preflight import _LLM_BACKENDS - assert "ollama_research" in _LLM_BACKENDS - # Should map to the ollama_research llm backend - backend_names = [name for name, _ in _LLM_BACKENDS["ollama_research"]] - assert "ollama_research" in backend_names - - -def test_vllm_research_in_llm_backends(): - """vllm_research must be registered under vllm in _LLM_BACKENDS.""" - from scripts.preflight import _LLM_BACKENDS - assert "vllm" in _LLM_BACKENDS - backend_names = [name for name, _ in _LLM_BACKENDS["vllm"]] - assert "vllm_research" in backend_names - - -def test_ollama_research_in_docker_internal(): - """ollama_research must map to internal port 11434 (Ollama's container port).""" - from scripts.preflight import _DOCKER_INTERNAL - assert "ollama_research" in _DOCKER_INTERNAL - hostname, port = _DOCKER_INTERNAL["ollama_research"] - assert hostname == "ollama_research" - assert port == 11434 # container-internal port is always 11434 - - -def test_ollama_not_mapped_to_ollama_research_backend(): - """ollama service key must only update the ollama llm backend, not ollama_research.""" - from scripts.preflight import _LLM_BACKENDS - ollama_backend_names = [name for name, _ in _LLM_BACKENDS.get("ollama", [])] - assert "ollama_research" not in ollama_backend_names - - -# ── Download size warning ────────────────────────────────────────────────────── - -def test_download_size_remote_profile(): - """Remote profile: only searxng + app, no ollama, no vision, no vllm.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("remote", "ollama") - assert "searxng" in sizes - assert "app" in sizes - assert "ollama" not in sizes - assert "vision_image" not in sizes - assert "vllm_image" not in sizes - - -def test_download_size_cpu_profile(): - """CPU profile: adds ollama image + llama3.2:3b weights.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("cpu", "ollama") - assert "ollama" in sizes - assert "llama3_2_3b" in sizes - assert "vision_image" not in sizes - - -def test_download_size_single_gpu_profile(): - """Single-GPU: adds vision image + moondream2 weights.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("single-gpu", "ollama") - assert "vision_image" in sizes - assert "moondream2" in sizes - assert "vllm_image" not in sizes - - -def test_download_size_dual_gpu_ollama_mode(): - """dual-gpu + ollama mode: no vllm image.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("dual-gpu", "ollama") - assert "vllm_image" not in sizes - - -def test_download_size_dual_gpu_vllm_mode(): - """dual-gpu + vllm mode: adds ~10 GB vllm image.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("dual-gpu", "vllm") - assert "vllm_image" in sizes - assert sizes["vllm_image"] >= 9000 # at least 9 GB - - -def test_download_size_dual_gpu_mixed_mode(): - """dual-gpu + mixed mode: also includes vllm image.""" - from scripts.preflight import _download_size_mb - sizes = _download_size_mb("dual-gpu", "mixed") - assert "vllm_image" in sizes - - -# ── Mixed-mode VRAM warning ──────────────────────────────────────────────────── - -def test_mixed_mode_vram_warning_triggered(): - """Should return a warning string when GPU 1 has < 12 GB free in mixed mode.""" - from scripts.preflight import _mixed_mode_vram_warning - gpus = [ - {"name": "RTX 3090", "vram_total_gb": 24.0, "vram_free_gb": 20.0}, - {"name": "RTX 3090", "vram_total_gb": 24.0, "vram_free_gb": 8.0}, # tight - ] - warning = _mixed_mode_vram_warning(gpus, "mixed") - assert warning is not None - assert "8.0" in warning or "GPU 1" in warning - - -def test_mixed_mode_vram_warning_not_triggered_with_headroom(): - """Should return None when GPU 1 has >= 12 GB free.""" - from scripts.preflight import _mixed_mode_vram_warning - gpus = [ - {"name": "RTX 4090", "vram_total_gb": 24.0, "vram_free_gb": 20.0}, - {"name": "RTX 4090", "vram_total_gb": 24.0, "vram_free_gb": 18.0}, # plenty - ] - warning = _mixed_mode_vram_warning(gpus, "mixed") - assert warning is None - - -def test_mixed_mode_vram_warning_not_triggered_for_other_modes(): - """Warning only applies in mixed mode.""" - from scripts.preflight import _mixed_mode_vram_warning - gpus = [ - {"name": "RTX 3090", "vram_total_gb": 24.0, "vram_free_gb": 20.0}, - {"name": "RTX 3090", "vram_total_gb": 24.0, "vram_free_gb": 6.0}, - ] - assert _mixed_mode_vram_warning(gpus, "ollama") is None - assert _mixed_mode_vram_warning(gpus, "vllm") is None - - -# ── update_llm_yaml with ollama_research ────────────────────────────────────── - -def test_update_llm_yaml_sets_ollama_research_url_docker_internal(): - """ollama_research backend URL must be set to ollama_research:11434 when Docker-owned.""" - from scripts.preflight import update_llm_yaml - - llm_cfg = { - "backends": { - "ollama": {"base_url": "http://old", "type": "openai_compat"}, - "ollama_research": {"base_url": "http://old", "type": "openai_compat"}, - "vllm": {"base_url": "http://old", "type": "openai_compat"}, - "vllm_research": {"base_url": "http://old", "type": "openai_compat"}, - "vision_service": {"base_url": "http://old", "type": "vision_service"}, - } - } - - with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f: - yaml.dump(llm_cfg, f) - tmp_path = Path(f.name) - - ports = { - "ollama": { - "resolved": 11434, "external": False, "env_var": "OLLAMA_PORT" - }, - "ollama_research": { - "resolved": 11435, "external": False, "env_var": "OLLAMA_RESEARCH_PORT" - }, - "vllm": { - "resolved": 8000, "external": False, "env_var": "VLLM_PORT" - }, - "vision": { - "resolved": 8002, "external": False, "env_var": "VISION_PORT" - }, - } - - try: - # Patch LLM_YAML to point at our temp file - with patch("scripts.preflight.LLM_YAML", tmp_path): - update_llm_yaml(ports) - - result = yaml.safe_load(tmp_path.read_text()) - # Docker-internal: use service name + container port - assert result["backends"]["ollama_research"]["base_url"] == "http://ollama_research:11434/v1" - # vllm_research must match vllm's URL - assert result["backends"]["vllm_research"]["base_url"] == result["backends"]["vllm"]["base_url"] - finally: - tmp_path.unlink() - - -def test_update_llm_yaml_sets_ollama_research_url_external(): - """When ollama_research is external (adopted), URL uses host.docker.internal:11435.""" - from scripts.preflight import update_llm_yaml - - llm_cfg = { - "backends": { - "ollama": {"base_url": "http://old", "type": "openai_compat"}, - "ollama_research": {"base_url": "http://old", "type": "openai_compat"}, - } - } - - with tempfile.NamedTemporaryFile(mode="w", suffix=".yaml", delete=False) as f: - yaml.dump(llm_cfg, f) - tmp_path = Path(f.name) - - ports = { - "ollama": {"resolved": 11434, "external": False, "env_var": "OLLAMA_PORT"}, - "ollama_research": {"resolved": 11435, "external": True, "env_var": "OLLAMA_RESEARCH_PORT"}, - } - - try: - with patch("scripts.preflight.LLM_YAML", tmp_path): - update_llm_yaml(ports) - result = yaml.safe_load(tmp_path.read_text()) - assert result["backends"]["ollama_research"]["base_url"] == "http://host.docker.internal:11435/v1" - finally: - tmp_path.unlink() -``` - -**Step 2: Run tests to confirm they all fail** - -```bash -conda run -n job-seeker python -m pytest tests/test_preflight.py -v 2>&1 | head -50 -``` - -Expected: all FAIL with `ImportError` or `AssertionError` — that's correct. - -**Step 3: Commit failing tests** - -```bash -git add tests/test_preflight.py -git commit -m "test: add failing tests for dual-gpu preflight additions" -``` - ---- - -### Task 3: `preflight.py` — service table additions - -**Files:** -- Modify: `scripts/preflight.py:46-67` (`_SERVICES`, `_LLM_BACKENDS`, `_DOCKER_INTERNAL`) - -**Step 1: Update `_SERVICES`** - -Find the `_SERVICES` dict (currently ends at the `"ollama"` entry). Add `ollama_research` as a new entry: - -```python -_SERVICES: dict[str, tuple[str, int, str, bool, bool]] = { - "streamlit": ("streamlit_port", 8501, "STREAMLIT_PORT", True, False), - "searxng": ("searxng_port", 8888, "SEARXNG_PORT", True, True), - "vllm": ("vllm_port", 8000, "VLLM_PORT", True, True), - "vision": ("vision_port", 8002, "VISION_PORT", True, True), - "ollama": ("ollama_port", 11434, "OLLAMA_PORT", True, True), - "ollama_research": ("ollama_research_port", 11435, "OLLAMA_RESEARCH_PORT", True, True), -} -``` - -**Step 2: Update `_LLM_BACKENDS`** - -Replace the existing dict: - -```python -_LLM_BACKENDS: dict[str, list[tuple[str, str]]] = { - "ollama": [("ollama", "/v1")], - "ollama_research": [("ollama_research", "/v1")], - "vllm": [("vllm", "/v1"), ("vllm_research", "/v1")], - "vision": [("vision_service", "")], -} -``` - -**Step 3: Update `_DOCKER_INTERNAL`** - -Add `ollama_research` entry: - -```python -_DOCKER_INTERNAL: dict[str, tuple[str, int]] = { - "ollama": ("ollama", 11434), - "ollama_research": ("ollama_research", 11434), # container-internal port is always 11434 - "vllm": ("vllm", 8000), - "vision": ("vision", 8002), - "searxng": ("searxng", 8080), -} -``` - -**Step 4: Run service table tests** - -```bash -conda run -n job-seeker python -m pytest tests/test_preflight.py::test_ollama_research_in_services tests/test_preflight.py::test_ollama_research_in_llm_backends tests/test_preflight.py::test_vllm_research_in_llm_backends tests/test_preflight.py::test_ollama_research_in_docker_internal tests/test_preflight.py::test_ollama_not_mapped_to_ollama_research_backend tests/test_preflight.py::test_update_llm_yaml_sets_ollama_research_url_docker_internal tests/test_preflight.py::test_update_llm_yaml_sets_ollama_research_url_external -v -``` - -Expected: all PASS - -**Step 5: Commit** - -```bash -git add scripts/preflight.py -git commit -m "feat: add ollama_research to preflight service table and LLM backend map" -``` - ---- - -### Task 4: `preflight.py` — `_download_size_mb()` pure function - -**Files:** -- Modify: `scripts/preflight.py` (add new function after `calc_cpu_offload_gb`) - -**Step 1: Add the function** - -After `calc_cpu_offload_gb()`, add: - -```python -def _download_size_mb(profile: str, dual_gpu_mode: str = "ollama") -> dict[str, int]: - """ - Return estimated first-run download sizes in MB, keyed by component name. - Profile-aware: only includes components that will actually be pulled. - """ - sizes: dict[str, int] = { - "searxng": 300, - "app": 1500, - } - if profile in ("cpu", "single-gpu", "dual-gpu"): - sizes["ollama"] = 800 - sizes["llama3_2_3b"] = 2000 - if profile in ("single-gpu", "dual-gpu"): - sizes["vision_image"] = 3000 - sizes["moondream2"] = 1800 - if profile == "dual-gpu" and dual_gpu_mode in ("vllm", "mixed"): - sizes["vllm_image"] = 10000 - return sizes -``` - -**Step 2: Run download size tests** - -```bash -conda run -n job-seeker python -m pytest tests/test_preflight.py -k "download_size" -v -``` - -Expected: all PASS - -**Step 3: Commit** - -```bash -git add scripts/preflight.py -git commit -m "feat: add _download_size_mb() pure function for preflight size warning" -``` - ---- - -### Task 5: `preflight.py` — VRAM warning, size report block, DUAL_GPU_MODE default - -**Files:** -- Modify: `scripts/preflight.py` (three additions to `main()` and a new helper) - -**Step 1: Add `_mixed_mode_vram_warning()` after `_download_size_mb()`** - -```python -def _mixed_mode_vram_warning(gpus: list[dict], dual_gpu_mode: str) -> str | None: - """ - Return a warning string if GPU 1 likely lacks VRAM for mixed mode, else None. - Only relevant when dual_gpu_mode == 'mixed' and at least 2 GPUs are present. - """ - if dual_gpu_mode != "mixed" or len(gpus) < 2: - return None - free = gpus[1]["vram_free_gb"] - if free < 12: - return ( - f"⚠ DUAL_GPU_MODE=mixed: GPU 1 has only {free:.1f} GB free — " - f"running ollama_research + vllm together may cause OOM. " - f"Consider DUAL_GPU_MODE=ollama or DUAL_GPU_MODE=vllm." - ) - return None -``` - -**Step 2: Run VRAM warning tests** - -```bash -conda run -n job-seeker python -m pytest tests/test_preflight.py -k "vram" -v -``` - -Expected: all PASS - -**Step 3: Wire size warning into `main()` report block** - -In `main()`, find the closing `print("╚═...═╝")` line. Add the size warning block just before it: - -```python - # ── Download size warning ────────────────────────────────────────────── - dual_gpu_mode = os.environ.get("DUAL_GPU_MODE", "ollama") - sizes = _download_size_mb(profile, dual_gpu_mode) - total_mb = sum(sizes.values()) - print("║") - print("║ Download sizes (first-run estimates)") - print("║ Docker images") - print(f"║ app (Python build) ~{sizes.get('app', 0):,} MB") - if "searxng" in sizes: - print(f"║ searxng/searxng ~{sizes['searxng']:,} MB") - if "ollama" in sizes: - shared_note = " (shared by ollama + ollama_research)" if profile == "dual-gpu" and dual_gpu_mode in ("ollama", "mixed") else "" - print(f"║ ollama/ollama ~{sizes['ollama']:,} MB{shared_note}") - if "vision_image" in sizes: - print(f"║ vision service ~{sizes['vision_image']:,} MB (torch + moondream)") - if "vllm_image" in sizes: - print(f"║ vllm/vllm-openai ~{sizes['vllm_image']:,} MB") - print("║ Model weights (lazy-loaded on first use)") - if "llama3_2_3b" in sizes: - print(f"║ llama3.2:3b ~{sizes['llama3_2_3b']:,} MB → OLLAMA_MODELS_DIR") - if "moondream2" in sizes: - print(f"║ moondream2 ~{sizes['moondream2']:,} MB → vision container cache") - if profile == "dual-gpu" and dual_gpu_mode in ("ollama", "mixed"): - print("║ Note: ollama + ollama_research share model dir — no double download") - print(f"║ ⚠ Total first-run: ~{total_mb / 1024:.1f} GB (models persist between restarts)") - - # ── Mixed-mode VRAM warning ──────────────────────────────────────────── - vram_warn = _mixed_mode_vram_warning(gpus, dual_gpu_mode) - if vram_warn: - print("║") - print(f"║ {vram_warn}") -``` - -**Step 4: Wire `DUAL_GPU_MODE` default into `write_env()` block in `main()`** - -In `main()`, find the `if not args.check_only:` block. After `env_updates["PEREGRINE_GPU_NAMES"]`, add: - -```python - # Write DUAL_GPU_MODE default for new 2-GPU setups (don't override user's choice) - if len(gpus) >= 2: - existing_env: dict[str, str] = {} - if ENV_FILE.exists(): - for line in ENV_FILE.read_text().splitlines(): - if "=" in line and not line.startswith("#"): - k, _, v = line.partition("=") - existing_env[k.strip()] = v.strip() - if "DUAL_GPU_MODE" not in existing_env: - env_updates["DUAL_GPU_MODE"] = "ollama" -``` - -**Step 5: Add `import os` if not already present at top of file** - -Check line 1–30 of `scripts/preflight.py`. `import os` is already present inside `get_cpu_cores()` as a local import — move it to the top-level imports block: - -```python -import os # add alongside existing stdlib imports -``` - -And remove the local `import os` inside `get_cpu_cores()`. - -**Step 6: Run all preflight tests** - -```bash -conda run -n job-seeker python -m pytest tests/test_preflight.py -v -``` - -Expected: all PASS - -**Step 7: Smoke-check the preflight report output** - -```bash -conda run -n job-seeker python scripts/preflight.py --check-only -``` - -Expected: report includes the `Download sizes` block near the bottom. - -**Step 8: Commit** - -```bash -git add scripts/preflight.py -git commit -m "feat: add DUAL_GPU_MODE default, VRAM warning, and download size report to preflight" -``` - ---- - -### Task 6: `compose.yml` — `ollama_research` service + profile updates - -**Files:** -- Modify: `compose.yml` - -**Step 1: Update `ollama` profiles line** - -Find: -```yaml - profiles: [cpu, single-gpu, dual-gpu] -``` -Replace with: -```yaml - profiles: [cpu, single-gpu, dual-gpu-ollama, dual-gpu-vllm, dual-gpu-mixed] -``` - -**Step 2: Update `vision` profiles line** - -Find: -```yaml - profiles: [single-gpu, dual-gpu] -``` -Replace with: -```yaml - profiles: [single-gpu, dual-gpu-ollama, dual-gpu-vllm, dual-gpu-mixed] -``` - -**Step 3: Update `vllm` profiles line** - -Find: -```yaml - profiles: [dual-gpu] -``` -Replace with: -```yaml - profiles: [dual-gpu-vllm, dual-gpu-mixed] -``` - -**Step 4: Add `ollama_research` service** - -After the closing lines of the `ollama` service block, add: - -```yaml - ollama_research: - image: ollama/ollama:latest - ports: - - "${OLLAMA_RESEARCH_PORT:-11435}:11434" - volumes: - - ${OLLAMA_MODELS_DIR:-~/models/ollama}:/root/.ollama - - ./docker/ollama/entrypoint.sh:/entrypoint.sh - environment: - - OLLAMA_MODELS=/root/.ollama - - DEFAULT_OLLAMA_MODEL=${OLLAMA_RESEARCH_MODEL:-llama3.2:3b} - entrypoint: ["/bin/bash", "/entrypoint.sh"] - profiles: [dual-gpu-ollama, dual-gpu-mixed] - restart: unless-stopped -``` - -**Step 5: Validate compose YAML** - -```bash -docker compose -f compose.yml config --quiet -``` - -Expected: no errors. - -**Step 6: Commit** - -```bash -git add compose.yml -git commit -m "feat: add ollama_research service and update profiles for dual-gpu sub-profiles" -``` - ---- - -### Task 7: GPU overlay files — `compose.gpu.yml` and `compose.podman-gpu.yml` - -**Files:** -- Modify: `compose.gpu.yml` -- Modify: `compose.podman-gpu.yml` - -**Step 1: Add `ollama_research` to `compose.gpu.yml`** - -After the `ollama:` block, add: - -```yaml - ollama_research: - deploy: - resources: - reservations: - devices: - - driver: nvidia - device_ids: ["1"] - capabilities: [gpu] -``` - -**Step 2: Add `ollama_research` to `compose.podman-gpu.yml`** - -After the `ollama:` block, add: - -```yaml - ollama_research: - devices: - - nvidia.com/gpu=1 - deploy: - resources: - reservations: - devices: [] -``` - -**Step 3: Validate both files** - -```bash -docker compose -f compose.yml -f compose.gpu.yml config --quiet -``` - -Expected: no errors. - -**Step 4: Commit** - -```bash -git add compose.gpu.yml compose.podman-gpu.yml -git commit -m "feat: assign ollama_research to GPU 1 in Docker and Podman GPU overlays" -``` - ---- - -### Task 8: `Makefile` + `manage.sh` — `DUAL_GPU_MODE` injection and help text - -**Files:** -- Modify: `Makefile` -- Modify: `manage.sh` - -**Step 1: Update `Makefile`** - -After the `COMPOSE_OVERRIDE` variable, add `DUAL_GPU_MODE` reading: - -```makefile -DUAL_GPU_MODE ?= $(shell grep -m1 '^DUAL_GPU_MODE=' .env 2>/dev/null | cut -d= -f2 || echo ollama) -``` - -In the GPU overlay block, find: -```makefile -else - ifneq (,$(findstring gpu,$(PROFILE))) - COMPOSE_FILES := -f compose.yml $(COMPOSE_OVERRIDE) -f compose.gpu.yml - endif -endif -``` - -Replace the `else` branch with: -```makefile -else - ifneq (,$(findstring gpu,$(PROFILE))) - COMPOSE_FILES := -f compose.yml $(COMPOSE_OVERRIDE) -f compose.gpu.yml - endif -endif -ifeq ($(PROFILE),dual-gpu) - COMPOSE_FILES += --profile dual-gpu-$(DUAL_GPU_MODE) -endif -``` - -**Step 2: Update `manage.sh` — profiles help block** - -Find the profiles section in `usage()`: -```bash - echo " dual-gpu Ollama + Vision + vLLM on GPU 0+1" -``` - -Replace with: -```bash - echo " dual-gpu Ollama + Vision on GPU 0; GPU 1 set by DUAL_GPU_MODE" - echo " DUAL_GPU_MODE=ollama (default) ollama_research on GPU 1" - echo " DUAL_GPU_MODE=vllm vllm on GPU 1" - echo " DUAL_GPU_MODE=mixed both on GPU 1 (VRAM-split)" -``` - -**Step 3: Verify Makefile parses** - -```bash -make help -``` - -Expected: help table prints cleanly, no make errors. - -**Step 4: Verify manage.sh help** - -```bash -./manage.sh help -``` - -Expected: new dual-gpu description appears in profiles section. - -**Step 5: Commit** - -```bash -git add Makefile manage.sh -git commit -m "feat: inject DUAL_GPU_MODE sub-profile in Makefile; update manage.sh help" -``` - ---- - -### Task 9: Integration smoke test - -**Goal:** Verify the full chain works for `DUAL_GPU_MODE=ollama` without actually starting Docker (dry-run compose config check). - -**Step 1: Write `DUAL_GPU_MODE=ollama` to `.env` temporarily** - -```bash -echo "DUAL_GPU_MODE=ollama" >> .env -``` - -**Step 2: Dry-run compose config for dual-gpu + dual-gpu-ollama** - -```bash -docker compose -f compose.yml -f compose.gpu.yml --profile dual-gpu --profile dual-gpu-ollama config 2>&1 | grep -E "^ [a-z]|image:|ports:" -``` - -Expected output includes: -- `ollama:` service with port 11434 -- `ollama_research:` service with port 11435 -- `vision:` service -- `searxng:` service -- **No** `vllm:` service - -**Step 3: Dry-run for `DUAL_GPU_MODE=vllm`** - -```bash -docker compose -f compose.yml -f compose.gpu.yml --profile dual-gpu --profile dual-gpu-vllm config 2>&1 | grep -E "^ [a-z]|image:|ports:" -``` - -Expected: -- `ollama:` service (port 11434) -- `vllm:` service (port 8000) -- **No** `ollama_research:` service - -**Step 4: Run full test suite** - -```bash -conda run -n job-seeker python -m pytest tests/ -v -``` - -Expected: all existing tests PASS, all new preflight tests PASS. - -**Step 5: Clean up `.env` test entry** - -```bash -# Remove the test DUAL_GPU_MODE line (preflight will re-write it correctly on next run) -sed -i '/^DUAL_GPU_MODE=/d' .env -``` - -**Step 6: Final commit** - -```bash -git add .env # in case preflight rewrote it during testing -git commit -m "feat: dual-gpu DUAL_GPU_MODE complete — ollama/vllm/mixed GPU 1 selection" -``` diff --git a/docs/plans/2026-02-26-email-classifier-benchmark-design.md b/docs/plans/2026-02-26-email-classifier-benchmark-design.md deleted file mode 100644 index 23ba35f..0000000 --- a/docs/plans/2026-02-26-email-classifier-benchmark-design.md +++ /dev/null @@ -1,132 +0,0 @@ -# Email Classifier Benchmark — Design - -**Date:** 2026-02-26 -**Status:** Approved - -## Problem - -The current `classify_stage_signal()` in `scripts/imap_sync.py` uses `llama3.1:8b` via -Ollama for 6-label email classification. This is slow, requires a running Ollama instance, -and accuracy is unverified against alternatives. This design establishes a benchmark harness -to evaluate HuggingFace-native classifiers as potential replacements. - -## Labels - -``` -interview_scheduled offer_received rejected -positive_response survey_received neutral -``` - -## Approach: Standalone Benchmark Script (Approach B) - -Two new files; nothing in `imap_sync.py` changes until a winner is chosen. - -``` -scripts/ - benchmark_classifier.py — CLI entry point - classifier_adapters.py — adapter classes (reusable by imap_sync later) - -data/ - email_eval.jsonl — labeled ground truth (gitignored — contains email content) - email_eval.jsonl.example — committed example with fake emails - -scripts/classifier_service/ - environment.yml — new conda env: job-seeker-classifiers -``` - -## Adapter Pattern - -``` -ClassifierAdapter (ABC) - .classify(subject, body) → str # one of the 6 labels - .name → str - .model_id → str - .load() / .unload() # explicit lifecycle - -ZeroShotAdapter(ClassifierAdapter) - # uses transformers pipeline("zero-shot-classification") - # candidate_labels = list of 6 labels - # works for: DeBERTa, BART-MNLI, BGE-M3-ZeroShot, XLM-RoBERTa - -GLiClassAdapter(ClassifierAdapter) - # uses gliclass library (pip install gliclass) - # GLiClassModel + ZeroShotClassificationPipeline - # works for: gliclass-instruct-large-v1.0 - -RerankerAdapter(ClassifierAdapter) - # uses FlagEmbedding reranker.compute_score() - # scores (email_text, label_description) pairs; highest = predicted label - # works for: bge-reranker-v2-m3 -``` - -## Model Registry - -| Short name | Model | Params | Adapter | Default | -|------------|-------|--------|---------|---------| -| `deberta-zeroshot` | MoritzLaurer/DeBERTa-v3-large-zeroshot-v2.0 | 400M | ZeroShot | ✅ | -| `deberta-small` | cross-encoder/nli-deberta-v3-small | 100M | ZeroShot | ✅ | -| `gliclass-large` | knowledgator/gliclass-instruct-large-v1.0 | 400M | GLiClass | ✅ | -| `bart-mnli` | facebook/bart-large-mnli | 400M | ZeroShot | ✅ | -| `bge-m3-zeroshot` | MoritzLaurer/bge-m3-zeroshot-v2.0 | 600M | ZeroShot | ✅ | -| `bge-reranker` | BAAI/bge-reranker-v2-m3 | 600M | Reranker | ❌ (`--include-slow`) | -| `deberta-xlarge` | microsoft/deberta-xlarge-mnli | 750M | ZeroShot | ❌ (`--include-slow`) | -| `mdeberta-mnli` | MoritzLaurer/mDeBERTa-v3-base-mnli-xnli | 300M | ZeroShot | ❌ (`--include-slow`) | -| `xlm-roberta-anli` | vicgalle/xlm-roberta-large-xnli-anli | 600M | ZeroShot | ❌ (`--include-slow`) | - -## CLI Modes - -### `--compare` (live IMAP, visual table) -Extends the pattern of `test_email_classify.py`. Pulls emails via IMAP, shows a table: -``` -Subject | Phrase | llama3 | deberta-zs | deberta-sm | gliclass | bart | bge-m3 -``` -- Phrase-filter column shows BLOCK/pass (same gate as production) -- `llama3` column = current production baseline -- HF model columns follow - -### `--eval` (ground-truth evaluation) -Reads `data/email_eval.jsonl`, runs all models, reports per-label and aggregate metrics: -- Per-label: precision, recall, F1 -- Aggregate: macro-F1, accuracy -- Latency: ms/email per model - -JSONL format: -```jsonl -{"subject": "Interview invitation", "body": "We'd like to schedule...", "label": "interview_scheduled"} -{"subject": "Your application", "body": "We regret to inform you...", "label": "rejected"} -``` - -### `--list-models` -Prints the registry with sizes, adapter types, and default/slow flags. - -## Conda Environment - -New env `job-seeker-classifiers` — isolated from `job-seeker` (no torch there). - -Key deps: -- `torch` (CUDA-enabled) -- `transformers` -- `gliclass` -- `FlagEmbedding` (for bge-reranker only) -- `sentence-transformers` (optional, for future embedding-based approaches) - -## GPU - -Auto-select (`device="cuda"` when available, CPU fallback). No GPU pinning — models -load one at a time so VRAM pressure is sequential, not cumulative. - -## Error Handling - -- Model load failures: skip that column, print warning, continue -- Classification errors: show `ERR` in cell, continue -- IMAP failures: propagate (same as existing harness) -- Missing eval file: clear error message pointing to `data/email_eval.jsonl.example` - -## What Does Not Change (Yet) - -- `scripts/imap_sync.py` — production classifier unchanged -- `scripts/llm_router.py` — unchanged -- `staging.db` schema — unchanged - -After benchmark results are reviewed, a separate PR will wire the winning model -into `classify_stage_signal()` as an opt-in backend in `llm_router.py`. diff --git a/docs/plans/2026-02-26-email-classifier-benchmark-plan.md b/docs/plans/2026-02-26-email-classifier-benchmark-plan.md deleted file mode 100644 index ff84b35..0000000 --- a/docs/plans/2026-02-26-email-classifier-benchmark-plan.md +++ /dev/null @@ -1,1334 +0,0 @@ -# Email Classifier Benchmark Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Build a benchmark harness that evaluates 9 HuggingFace classifiers against our 6 email labels in two modes: live IMAP visual table (`--compare`) and labeled-JSONL metrics (`--score`). - -**Architecture:** Standalone scripts (`benchmark_classifier.py` + `classifier_adapters.py`) in a new isolated conda env (`job-seeker-classifiers`). Three adapter types (ZeroShot NLI, GLiClass, Reranker) normalize each model's output to our 6 labels. IMAP fetching uses stdlib only — no dependency on `imap_sync.py` or LLMRouter. - -**Tech Stack:** Python 3.11, `transformers` zero-shot pipeline, `gliclass`, `FlagEmbedding`, `torch` (CUDA auto-select), `pytest`, `unittest.mock` - ---- - -## Labels constant (referenced throughout) - -```python -LABELS = [ - "interview_scheduled", "offer_received", "rejected", - "positive_response", "survey_received", "neutral", -] -``` - ---- - -### Task 1: Conda environment - -**Files:** -- Create: `scripts/classifier_service/environment.yml` - -**Step 1: Create the environment file** - -```yaml -name: job-seeker-classifiers -channels: - - pytorch - - nvidia - - conda-forge - - defaults -dependencies: - - python=3.11 - - pip - - pip: - - torch>=2.1.0 - - transformers>=4.40.0 - - accelerate>=0.26.0 - - sentencepiece>=0.1.99 - - protobuf>=4.25.0 - - gliclass>=0.1.0 - - FlagEmbedding>=1.2.0 - - pyyaml>=6.0 - - tqdm>=4.66.0 - - pytest>=8.0.0 -``` - -**Step 2: Create the environment** - -```bash -conda env create -f scripts/classifier_service/environment.yml -``` - -Expected: env `job-seeker-classifiers` created successfully. - -**Step 3: Verify torch + CUDA** - -```bash -conda run -n job-seeker-classifiers python -c "import torch; print(torch.cuda.is_available())" -``` - -Expected: `True` (if GPU available). - -**Step 4: Commit** - -```bash -git add scripts/classifier_service/environment.yml -git commit -m "feat: add job-seeker-classifiers conda env for HF classifier benchmark" -``` - ---- - -### Task 2: Data directory + gitignore + example scoring file - -**Files:** -- Modify: `.gitignore` -- Create: `data/email_score.jsonl.example` - -**Step 1: Update .gitignore** - -Add to `.gitignore`: -``` -data/email_score.jsonl -data/email_compare_sample.jsonl -``` - -**Step 2: Create the example scoring file** - -Create `data/email_score.jsonl.example` with fake-but-realistic emails: - -```jsonl -{"subject": "Interview Invitation — Senior Engineer", "body": "Hi Alex, we'd love to schedule a 30-min phone screen. Are you available Thursday at 2pm? Please reply to confirm.", "label": "interview_scheduled"} -{"subject": "Your application to Acme Corp", "body": "Thank you for your interest in the Senior Engineer role. After careful consideration, we have decided to move forward with other candidates whose experience more closely matches our current needs.", "label": "rejected"} -{"subject": "Offer Letter — Product Manager at Initech", "body": "Dear Alex, we are thrilled to extend an offer of employment for the Product Manager position. Please find the attached offer letter outlining compensation and start date.", "label": "offer_received"} -{"subject": "Quick question about your background", "body": "Hi Alex, I came across your profile and would love to connect. We have a few roles that seem like a great match. Would you be open to a brief chat this week?", "label": "positive_response"} -{"subject": "Company Culture Survey — Acme Corp", "body": "Alex, as part of our evaluation process, we invite all candidates to complete our culture fit assessment. The survey takes approximately 15 minutes. Please click the link below.", "label": "survey_received"} -{"subject": "Application Received — DataCo", "body": "Thank you for submitting your application for the Data Engineer role at DataCo. We have received your materials and will be in touch if your qualifications match our needs.", "label": "neutral"} -{"subject": "Following up on your application", "body": "Hi Alex, I wanted to follow up on your recent application. Your background looks interesting and we'd like to learn more. Can we set up a quick call?", "label": "positive_response"} -{"subject": "We're moving forward with other candidates", "body": "Dear Alex, thank you for taking the time to interview with us. After thoughtful consideration, we have decided not to move forward with your candidacy at this time.", "label": "rejected"} -``` - -**Step 3: Commit** - -```bash -git add .gitignore data/email_score.jsonl.example -git commit -m "feat: add scoring JSONL example and gitignore for benchmark data files" -``` - ---- - -### Task 3: ClassifierAdapter ABC + compute_metrics() - -**Files:** -- Create: `scripts/classifier_adapters.py` -- Create: `tests/test_classifier_adapters.py` - -**Step 1: Write the failing tests** - -Create `tests/test_classifier_adapters.py`: - -```python -"""Tests for classifier_adapters — no model downloads required.""" -import pytest - - -def test_labels_constant_has_six_items(): - from scripts.classifier_adapters import LABELS - assert len(LABELS) == 6 - assert "interview_scheduled" in LABELS - assert "neutral" in LABELS - - -def test_compute_metrics_perfect_predictions(): - from scripts.classifier_adapters import compute_metrics, LABELS - gold = ["rejected", "interview_scheduled", "neutral"] - preds = ["rejected", "interview_scheduled", "neutral"] - m = compute_metrics(preds, gold, LABELS) - assert m["rejected"]["f1"] == pytest.approx(1.0) - assert m["__accuracy__"] == pytest.approx(1.0) - assert m["__macro_f1__"] == pytest.approx(1.0) - - -def test_compute_metrics_all_wrong(): - from scripts.classifier_adapters import compute_metrics, LABELS - gold = ["rejected", "rejected"] - preds = ["neutral", "interview_scheduled"] - m = compute_metrics(preds, gold, LABELS) - assert m["rejected"]["recall"] == pytest.approx(0.0) - assert m["__accuracy__"] == pytest.approx(0.0) - - -def test_compute_metrics_partial(): - from scripts.classifier_adapters import compute_metrics, LABELS - gold = ["rejected", "neutral", "rejected"] - preds = ["rejected", "neutral", "interview_scheduled"] - m = compute_metrics(preds, gold, LABELS) - assert m["rejected"]["precision"] == pytest.approx(1.0) - assert m["rejected"]["recall"] == pytest.approx(0.5) - assert m["neutral"]["f1"] == pytest.approx(1.0) - assert m["__accuracy__"] == pytest.approx(2 / 3) - - -def test_compute_metrics_empty(): - from scripts.classifier_adapters import compute_metrics, LABELS - m = compute_metrics([], [], LABELS) - assert m["__accuracy__"] == pytest.approx(0.0) - - -def test_classifier_adapter_is_abstract(): - from scripts.classifier_adapters import ClassifierAdapter - with pytest.raises(TypeError): - ClassifierAdapter() -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py -v -``` - -Expected: `ModuleNotFoundError: No module named 'scripts.classifier_adapters'` - -**Step 3: Create scripts/classifier_adapters.py with ABC + metrics** - -```python -"""Classifier adapters for email classification benchmark. - -Each adapter wraps a HuggingFace model and normalizes output to LABELS. -Models load lazily on first classify() call; call unload() to free VRAM. -""" -from __future__ import annotations - -import abc -from collections import defaultdict -from typing import Any - -LABELS: list[str] = [ - "interview_scheduled", - "offer_received", - "rejected", - "positive_response", - "survey_received", - "neutral", -] - -# Natural-language descriptions used by the RerankerAdapter. -LABEL_DESCRIPTIONS: dict[str, str] = { - "interview_scheduled": "scheduling an interview, phone screen, or video call", - "offer_received": "a formal job offer or employment offer letter", - "rejected": "application rejected or not moving forward with candidacy", - "positive_response": "positive recruiter interest or request to connect", - "survey_received": "invitation to complete a culture-fit survey or assessment", - "neutral": "automated ATS confirmation or unrelated email", -} - - -def _cuda_available() -> bool: - try: - import torch - return torch.cuda.is_available() - except ImportError: - return False - - -def compute_metrics( - predictions: list[str], - gold: list[str], - labels: list[str], -) -> dict[str, Any]: - """Return per-label precision/recall/F1 + macro_f1 + accuracy.""" - tp: dict[str, int] = defaultdict(int) - fp: dict[str, int] = defaultdict(int) - fn: dict[str, int] = defaultdict(int) - - for pred, true in zip(predictions, gold): - if pred == true: - tp[pred] += 1 - else: - fp[pred] += 1 - fn[true] += 1 - - result: dict[str, Any] = {} - for label in labels: - denom_p = tp[label] + fp[label] - denom_r = tp[label] + fn[label] - p = tp[label] / denom_p if denom_p else 0.0 - r = tp[label] / denom_r if denom_r else 0.0 - f1 = 2 * p * r / (p + r) if (p + r) else 0.0 - result[label] = { - "precision": p, - "recall": r, - "f1": f1, - "support": denom_r, - } - - result["__macro_f1__"] = ( - sum(v["f1"] for v in result.values() if isinstance(v, dict)) / len(labels) - ) - result["__accuracy__"] = sum(tp.values()) / len(predictions) if predictions else 0.0 - return result - - -class ClassifierAdapter(abc.ABC): - """Abstract base for all email classifier adapters.""" - - @property - @abc.abstractmethod - def name(self) -> str: ... - - @property - @abc.abstractmethod - def model_id(self) -> str: ... - - @abc.abstractmethod - def load(self) -> None: - """Download/load the model into memory.""" - - @abc.abstractmethod - def unload(self) -> None: - """Release model from memory.""" - - @abc.abstractmethod - def classify(self, subject: str, body: str) -> str: - """Return one of LABELS for the given email.""" -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py -v -``` - -Expected: 6 tests pass. - -**Step 5: Commit** - -```bash -git add scripts/classifier_adapters.py tests/test_classifier_adapters.py -git commit -m "feat: ClassifierAdapter ABC + compute_metrics() with full test coverage" -``` - ---- - -### Task 4: ZeroShotAdapter - -**Files:** -- Modify: `scripts/classifier_adapters.py` -- Modify: `tests/test_classifier_adapters.py` - -**Step 1: Add failing tests** - -Append to `tests/test_classifier_adapters.py`: - -```python -def test_zeroshot_adapter_classify_mocked(): - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import ZeroShotAdapter - - mock_pipeline = MagicMock() - mock_pipeline.return_value = { - "labels": ["rejected", "neutral", "interview_scheduled"], - "scores": [0.85, 0.10, 0.05], - } - - with patch("scripts.classifier_adapters.pipeline", mock_pipeline): - adapter = ZeroShotAdapter("test-zs", "some/model") - adapter.load() - result = adapter.classify("We went with another candidate", "Thank you for applying.") - - assert result == "rejected" - call_args = mock_pipeline.return_value.call_args - assert "We went with another candidate" in call_args[0][0] - - -def test_zeroshot_adapter_unload_clears_pipeline(): - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import ZeroShotAdapter - - with patch("scripts.classifier_adapters.pipeline", MagicMock()): - adapter = ZeroShotAdapter("test-zs", "some/model") - adapter.load() - assert adapter._pipeline is not None - adapter.unload() - assert adapter._pipeline is None - - -def test_zeroshot_adapter_lazy_loads(): - """classify() loads the model automatically if not already loaded.""" - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import ZeroShotAdapter - - mock_pipe_factory = MagicMock() - mock_pipe_factory.return_value = MagicMock(return_value={ - "labels": ["neutral"], "scores": [1.0] - }) - - with patch("scripts.classifier_adapters.pipeline", mock_pipe_factory): - adapter = ZeroShotAdapter("test-zs", "some/model") - adapter.classify("subject", "body") # no explicit load() call - - mock_pipe_factory.assert_called_once() -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py::test_zeroshot_adapter_classify_mocked -v -``` - -Expected: `AttributeError` — ZeroShotAdapter not defined. - -**Step 3: Add import shim + ZeroShotAdapter to classifier_adapters.py** - -Add after the `_cuda_available()` helper: - -```python -# Lazy import shim — lets tests patch 'scripts.classifier_adapters.pipeline' -try: - from transformers import pipeline # type: ignore[assignment] -except ImportError: - pipeline = None # type: ignore[assignment] -``` - -Add after `ClassifierAdapter`: - -```python -class ZeroShotAdapter(ClassifierAdapter): - """Wraps any transformers zero-shot-classification pipeline.""" - - def __init__(self, name: str, model_id: str) -> None: - self._name = name - self._model_id = model_id - self._pipeline: Any = None - - @property - def name(self) -> str: - return self._name - - @property - def model_id(self) -> str: - return self._model_id - - def load(self) -> None: - from transformers import pipeline as _pipeline # noqa: PLC0415 - device = 0 if _cuda_available() else -1 # 0 = first GPU, -1 = CPU - self._pipeline = _pipeline( - "zero-shot-classification", - model=self._model_id, - device=device, - ) - - def unload(self) -> None: - self._pipeline = None - - def classify(self, subject: str, body: str) -> str: - if self._pipeline is None: - self.load() - text = f"Subject: {subject}\n\n{body[:600]}" - result = self._pipeline(text, LABELS, multi_label=False) - return result["labels"][0] -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py -v -``` - -Expected: 9 tests pass. - -**Step 5: Commit** - -```bash -git add scripts/classifier_adapters.py tests/test_classifier_adapters.py -git commit -m "feat: ZeroShotAdapter — wraps transformers zero-shot-classification pipeline" -``` - ---- - -### Task 5: GLiClassAdapter - -**Files:** -- Modify: `scripts/classifier_adapters.py` -- Modify: `tests/test_classifier_adapters.py` - -**Step 1: Add failing tests** - -Append to `tests/test_classifier_adapters.py`: - -```python -def test_gliclass_adapter_classify_mocked(): - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import GLiClassAdapter - - mock_pipeline_instance = MagicMock() - mock_pipeline_instance.return_value = [[ - {"label": "interview_scheduled", "score": 0.91}, - {"label": "neutral", "score": 0.05}, - {"label": "rejected", "score": 0.04}, - ]] - - with patch("scripts.classifier_adapters.GLiClassModel") as _mc, \ - patch("scripts.classifier_adapters.AutoTokenizer") as _mt, \ - patch("scripts.classifier_adapters.ZeroShotClassificationPipeline", - return_value=mock_pipeline_instance): - adapter = GLiClassAdapter("test-gli", "some/gliclass-model") - adapter.load() - result = adapter.classify("Interview invitation", "Let's schedule a call.") - - assert result == "interview_scheduled" - - -def test_gliclass_adapter_returns_highest_score(): - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import GLiClassAdapter - - mock_pipeline_instance = MagicMock() - mock_pipeline_instance.return_value = [[ - {"label": "neutral", "score": 0.02}, - {"label": "offer_received", "score": 0.88}, - {"label": "rejected", "score": 0.10}, - ]] - - with patch("scripts.classifier_adapters.GLiClassModel"), \ - patch("scripts.classifier_adapters.AutoTokenizer"), \ - patch("scripts.classifier_adapters.ZeroShotClassificationPipeline", - return_value=mock_pipeline_instance): - adapter = GLiClassAdapter("test-gli", "some/model") - adapter.load() - result = adapter.classify("Offer letter enclosed", "Dear Alex, we are pleased to offer...") - - assert result == "offer_received" -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py::test_gliclass_adapter_classify_mocked -v -``` - -Expected: `AttributeError` — GLiClassAdapter not defined. - -**Step 3: Add gliclass import shims + GLiClassAdapter** - -Add import shims near top of `scripts/classifier_adapters.py` (after the pipeline shim): - -```python -try: - from gliclass import GLiClassModel, ZeroShotClassificationPipeline # type: ignore - from transformers import AutoTokenizer -except ImportError: - GLiClassModel = None # type: ignore - ZeroShotClassificationPipeline = None # type: ignore - AutoTokenizer = None # type: ignore -``` - -Add class after `ZeroShotAdapter`: - -```python -class GLiClassAdapter(ClassifierAdapter): - """Wraps knowledgator GLiClass models via the gliclass library.""" - - def __init__(self, name: str, model_id: str) -> None: - self._name = name - self._model_id = model_id - self._pipeline: Any = None - - @property - def name(self) -> str: - return self._name - - @property - def model_id(self) -> str: - return self._model_id - - def load(self) -> None: - if GLiClassModel is None: - raise ImportError("gliclass not installed — run: pip install gliclass") - device = "cuda:0" if _cuda_available() else "cpu" - model = GLiClassModel.from_pretrained(self._model_id) - tokenizer = AutoTokenizer.from_pretrained(self._model_id) - self._pipeline = ZeroShotClassificationPipeline( - model, - tokenizer, - classification_type="single-label", - device=device, - ) - - def unload(self) -> None: - self._pipeline = None - - def classify(self, subject: str, body: str) -> str: - if self._pipeline is None: - self.load() - text = f"Subject: {subject}\n\n{body[:600]}" - # threshold=0.0 ensures all labels are scored; we pick the max. - results = self._pipeline(text, LABELS, threshold=0.0)[0] - return max(results, key=lambda r: r["score"])["label"] -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py -v -``` - -Expected: 11 tests pass. - -**Step 5: Commit** - -```bash -git add scripts/classifier_adapters.py tests/test_classifier_adapters.py -git commit -m "feat: GLiClassAdapter — wraps gliclass zero-shot pipeline" -``` - ---- - -### Task 6: RerankerAdapter - -**Files:** -- Modify: `scripts/classifier_adapters.py` -- Modify: `tests/test_classifier_adapters.py` - -**Step 1: Add failing tests** - -Append to `tests/test_classifier_adapters.py`: - -```python -def test_reranker_adapter_picks_highest_score(): - from unittest.mock import MagicMock, patch - from scripts.classifier_adapters import RerankerAdapter, LABELS - - mock_reranker = MagicMock() - # Scores for each label pair — "rejected" (index 2) gets the highest - mock_reranker.compute_score.return_value = [0.1, 0.05, 0.85, 0.05, 0.02, 0.03] - - with patch("scripts.classifier_adapters.FlagReranker", return_value=mock_reranker): - adapter = RerankerAdapter("test-rr", "BAAI/bge-reranker-v2-m3") - adapter.load() - result = adapter.classify( - "We regret to inform you", - "After careful consideration we are moving forward with other candidates.", - ) - - assert result == "rejected" - pairs = mock_reranker.compute_score.call_args[0][0] - assert len(pairs) == len(LABELS) - - -def test_reranker_adapter_descriptions_cover_all_labels(): - from scripts.classifier_adapters import LABEL_DESCRIPTIONS, LABELS - assert set(LABEL_DESCRIPTIONS.keys()) == set(LABELS) -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py::test_reranker_adapter_picks_highest_score -v -``` - -Expected: `AttributeError` — RerankerAdapter not defined. - -**Step 3: Add FlagEmbedding import shim + RerankerAdapter** - -Add import shim in `scripts/classifier_adapters.py`: - -```python -try: - from FlagEmbedding import FlagReranker # type: ignore -except ImportError: - FlagReranker = None # type: ignore -``` - -Add class after `GLiClassAdapter`: - -```python -class RerankerAdapter(ClassifierAdapter): - """Uses a BGE reranker to score (email, label_description) pairs.""" - - def __init__(self, name: str, model_id: str) -> None: - self._name = name - self._model_id = model_id - self._reranker: Any = None - - @property - def name(self) -> str: - return self._name - - @property - def model_id(self) -> str: - return self._model_id - - def load(self) -> None: - if FlagReranker is None: - raise ImportError("FlagEmbedding not installed — run: pip install FlagEmbedding") - self._reranker = FlagReranker(self._model_id, use_fp16=_cuda_available()) - - def unload(self) -> None: - self._reranker = None - - def classify(self, subject: str, body: str) -> str: - if self._reranker is None: - self.load() - text = f"Subject: {subject}\n\n{body[:600]}" - pairs = [[text, LABEL_DESCRIPTIONS[label]] for label in LABELS] - scores: list[float] = self._reranker.compute_score(pairs, normalize=True) - return LABELS[scores.index(max(scores))] -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py -v -``` - -Expected: 13 tests pass. - -**Step 5: Commit** - -```bash -git add scripts/classifier_adapters.py tests/test_classifier_adapters.py -git commit -m "feat: RerankerAdapter — scores (email, label_description) pairs via BGE reranker" -``` - ---- - -### Task 7: MODEL_REGISTRY + --list-models + CLI skeleton - -**Files:** -- Create: `scripts/benchmark_classifier.py` -- Create: `tests/test_benchmark_classifier.py` - -**Step 1: Write failing tests** - -Create `tests/test_benchmark_classifier.py`: - -```python -"""Tests for benchmark_classifier — no model downloads required.""" -import pytest - - -def test_registry_has_nine_models(): - from scripts.benchmark_classifier import MODEL_REGISTRY - assert len(MODEL_REGISTRY) == 9 - - -def test_registry_default_count(): - from scripts.benchmark_classifier import MODEL_REGISTRY - defaults = [k for k, v in MODEL_REGISTRY.items() if v["default"]] - assert len(defaults) == 5 - - -def test_registry_entries_have_required_keys(): - from scripts.benchmark_classifier import MODEL_REGISTRY - from scripts.classifier_adapters import ClassifierAdapter - for name, entry in MODEL_REGISTRY.items(): - assert "adapter" in entry, f"{name} missing 'adapter'" - assert "model_id" in entry, f"{name} missing 'model_id'" - assert "params" in entry, f"{name} missing 'params'" - assert "default" in entry, f"{name} missing 'default'" - assert issubclass(entry["adapter"], ClassifierAdapter), \ - f"{name} adapter must be a ClassifierAdapter subclass" - - -def test_load_scoring_jsonl(tmp_path): - from scripts.benchmark_classifier import load_scoring_jsonl - import json - f = tmp_path / "score.jsonl" - rows = [ - {"subject": "Hi", "body": "Body text", "label": "neutral"}, - {"subject": "Interview", "body": "Schedule a call", "label": "interview_scheduled"}, - ] - f.write_text("\n".join(json.dumps(r) for r in rows)) - result = load_scoring_jsonl(str(f)) - assert len(result) == 2 - assert result[0]["label"] == "neutral" - - -def test_load_scoring_jsonl_missing_file(): - from scripts.benchmark_classifier import load_scoring_jsonl - with pytest.raises(FileNotFoundError): - load_scoring_jsonl("/nonexistent/path.jsonl") -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_benchmark_classifier.py -v -``` - -Expected: `ModuleNotFoundError: No module named 'scripts.benchmark_classifier'` - -**Step 3: Create benchmark_classifier.py with registry + skeleton** - -```python -#!/usr/bin/env python -""" -Email classifier benchmark — compare HuggingFace models against our 6 labels. - -Usage: - # List available models - conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --list-models - - # Score against labeled JSONL - conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score - - # Visual comparison on live IMAP emails - conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --compare --limit 20 - - # Include slow/large models - conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score --include-slow -""" -from __future__ import annotations - -import argparse -import json -import sys -from pathlib import Path -from typing import Any - -sys.path.insert(0, str(Path(__file__).parent.parent)) - -from scripts.classifier_adapters import ( - LABELS, - ClassifierAdapter, - GLiClassAdapter, - RerankerAdapter, - ZeroShotAdapter, - compute_metrics, -) - -# --------------------------------------------------------------------------- -# Model registry -# --------------------------------------------------------------------------- - -MODEL_REGISTRY: dict[str, dict[str, Any]] = { - "deberta-zeroshot": { - "adapter": ZeroShotAdapter, - "model_id": "MoritzLaurer/DeBERTa-v3-large-zeroshot-v2.0", - "params": "400M", - "default": True, - }, - "deberta-small": { - "adapter": ZeroShotAdapter, - "model_id": "cross-encoder/nli-deberta-v3-small", - "params": "100M", - "default": True, - }, - "gliclass-large": { - "adapter": GLiClassAdapter, - "model_id": "knowledgator/gliclass-instruct-large-v1.0", - "params": "400M", - "default": True, - }, - "bart-mnli": { - "adapter": ZeroShotAdapter, - "model_id": "facebook/bart-large-mnli", - "params": "400M", - "default": True, - }, - "bge-m3-zeroshot": { - "adapter": ZeroShotAdapter, - "model_id": "MoritzLaurer/bge-m3-zeroshot-v2.0", - "params": "600M", - "default": True, - }, - "bge-reranker": { - "adapter": RerankerAdapter, - "model_id": "BAAI/bge-reranker-v2-m3", - "params": "600M", - "default": False, - }, - "deberta-xlarge": { - "adapter": ZeroShotAdapter, - "model_id": "microsoft/deberta-xlarge-mnli", - "params": "750M", - "default": False, - }, - "mdeberta-mnli": { - "adapter": ZeroShotAdapter, - "model_id": "MoritzLaurer/mDeBERTa-v3-base-mnli-xnli", - "params": "300M", - "default": False, - }, - "xlm-roberta-anli": { - "adapter": ZeroShotAdapter, - "model_id": "vicgalle/xlm-roberta-large-xnli-anli", - "params": "600M", - "default": False, - }, -} - -# --------------------------------------------------------------------------- -# Helpers -# --------------------------------------------------------------------------- - -def load_scoring_jsonl(path: str) -> list[dict[str, str]]: - """Load labeled examples from a JSONL file for benchmark scoring.""" - p = Path(path) - if not p.exists(): - raise FileNotFoundError( - f"Scoring file not found: {path}\n" - f"Copy data/email_score.jsonl.example → data/email_score.jsonl and label your emails." - ) - rows = [] - with p.open() as f: - for line in f: - line = line.strip() - if line: - rows.append(json.loads(line)) - return rows - - -def _active_models(include_slow: bool) -> dict[str, dict[str, Any]]: - return {k: v for k, v in MODEL_REGISTRY.items() if v["default"] or include_slow} - - -# --------------------------------------------------------------------------- -# Subcommands -# --------------------------------------------------------------------------- - -def cmd_list_models(_args: argparse.Namespace) -> None: - print(f"\n{'Name':<20} {'Params':<8} {'Default':<20} {'Adapter':<15} Model ID") - print("-" * 100) - for name, entry in MODEL_REGISTRY.items(): - adapter_name = entry["adapter"].__name__ - default_flag = "yes" if entry["default"] else "(--include-slow)" - print(f"{name:<20} {entry['params']:<8} {default_flag:<20} {adapter_name:<15} {entry['model_id']}") - print() - - -def cmd_score(_args: argparse.Namespace) -> None: - raise NotImplementedError("--score implemented in Task 8") - - -def cmd_compare(_args: argparse.Namespace) -> None: - raise NotImplementedError("--compare implemented in Task 9") - - -# --------------------------------------------------------------------------- -# CLI -# --------------------------------------------------------------------------- - -def main() -> None: - parser = argparse.ArgumentParser( - description="Benchmark HuggingFace email classifiers against our 6 labels." - ) - parser.add_argument("--list-models", action="store_true", help="Show model registry and exit") - parser.add_argument("--score", action="store_true", help="Score against labeled JSONL") - parser.add_argument("--compare", action="store_true", help="Visual table on live IMAP emails") - parser.add_argument("--score-file", default="data/email_score.jsonl", help="Path to labeled JSONL") - parser.add_argument("--limit", type=int, default=20, help="Max emails for --compare") - parser.add_argument("--days", type=int, default=90, help="Days back for IMAP search") - parser.add_argument("--include-slow", action="store_true", help="Include non-default heavy models") - parser.add_argument("--models", nargs="+", help="Override: run only these model names") - - args = parser.parse_args() - - if args.list_models: - cmd_list_models(args) - elif args.score: - cmd_score(args) - elif args.compare: - cmd_compare(args) - else: - parser.print_help() - - -if __name__ == "__main__": - main() -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_benchmark_classifier.py -v -``` - -Expected: 5 tests pass. - -**Step 5: Smoke-test --list-models** - -```bash -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --list-models -``` - -Expected: table of 9 models printed without error. - -**Step 6: Commit** - -```bash -git add scripts/benchmark_classifier.py tests/test_benchmark_classifier.py -git commit -m "feat: benchmark_classifier skeleton — MODEL_REGISTRY, --list-models, CLI" -``` - ---- - -### Task 8: --score mode - -**Files:** -- Modify: `scripts/benchmark_classifier.py` -- Modify: `tests/test_benchmark_classifier.py` - -**Step 1: Add failing tests** - -Append to `tests/test_benchmark_classifier.py`: - -```python -def test_run_scoring_with_mock_adapters(tmp_path): - """run_scoring() returns per-model metrics using mock adapters.""" - import json - from unittest.mock import MagicMock - from scripts.benchmark_classifier import run_scoring - - score_file = tmp_path / "score.jsonl" - rows = [ - {"subject": "Interview", "body": "Let's schedule", "label": "interview_scheduled"}, - {"subject": "Sorry", "body": "We went with others", "label": "rejected"}, - {"subject": "Offer", "body": "We are pleased", "label": "offer_received"}, - ] - score_file.write_text("\n".join(json.dumps(r) for r in rows)) - - perfect = MagicMock() - perfect.name = "perfect" - perfect.classify.side_effect = lambda s, b: ( - "interview_scheduled" if "Interview" in s else - "rejected" if "Sorry" in s else "offer_received" - ) - - bad = MagicMock() - bad.name = "bad" - bad.classify.return_value = "neutral" - - results = run_scoring([perfect, bad], str(score_file)) - - assert results["perfect"]["__accuracy__"] == pytest.approx(1.0) - assert results["bad"]["__accuracy__"] == pytest.approx(0.0) - assert "latency_ms" in results["perfect"] - - -def test_run_scoring_handles_classify_error(tmp_path): - """run_scoring() falls back to 'neutral' on exception and continues.""" - import json - from unittest.mock import MagicMock - from scripts.benchmark_classifier import run_scoring - - score_file = tmp_path / "score.jsonl" - score_file.write_text(json.dumps({"subject": "Hi", "body": "Body", "label": "neutral"})) - - broken = MagicMock() - broken.name = "broken" - broken.classify.side_effect = RuntimeError("model crashed") - - results = run_scoring([broken], str(score_file)) - assert "broken" in results -``` - -**Step 2: Run tests — expect FAIL** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_benchmark_classifier.py::test_run_scoring_with_mock_adapters -v -``` - -Expected: `ImportError` — `run_scoring` not defined. - -**Step 3: Implement run_scoring() and cmd_score()** - -Add `import time` at the top of `benchmark_classifier.py`. Then add `run_scoring()`: - -```python -def run_scoring( - adapters: list[ClassifierAdapter], - score_file: str, -) -> dict[str, Any]: - """Run all adapters against a labeled JSONL. Returns per-adapter metrics.""" - import time - rows = load_scoring_jsonl(score_file) - gold = [r["label"] for r in rows] - results: dict[str, Any] = {} - - for adapter in adapters: - preds: list[str] = [] - t0 = time.monotonic() - for row in rows: - try: - pred = adapter.classify(row["subject"], row["body"]) - except Exception as exc: - print(f" [{adapter.name}] ERROR on '{row['subject'][:40]}': {exc}", flush=True) - pred = "neutral" - preds.append(pred) - elapsed_ms = (time.monotonic() - t0) * 1000 - metrics = compute_metrics(preds, gold, LABELS) - metrics["latency_ms"] = round(elapsed_ms / len(rows), 1) - results[adapter.name] = metrics - adapter.unload() - - return results -``` - -Replace the `cmd_score` stub: - -```python -def cmd_score(args: argparse.Namespace) -> None: - active = _active_models(args.include_slow) - if args.models: - active = {k: v for k, v in active.items() if k in args.models} - - adapters = [ - entry["adapter"](name, entry["model_id"]) - for name, entry in active.items() - ] - - print(f"\nScoring {len(adapters)} model(s) against {args.score_file} …\n") - results = run_scoring(adapters, args.score_file) - - # Summary table - col = 12 - print(f"{'Model':<22}" + f"{'macro-F1':>{col}} {'Accuracy':>{col}} {'ms/email':>{col}}") - print("-" * (22 + col * 3 + 2)) - for name, m in results.items(): - print( - f"{name:<22}" - f"{m['__macro_f1__']:>{col}.3f}" - f"{m['__accuracy__']:>{col}.3f}" - f"{m['latency_ms']:>{col}.1f}" - ) - - # Per-label F1 breakdown - print("\nPer-label F1:") - names = list(results.keys()) - print(f"{'Label':<25}" + "".join(f"{n[:11]:>{col}}" for n in names)) - print("-" * (25 + col * len(names))) - for label in LABELS: - row_str = f"{label:<25}" - for m in results.values(): - row_str += f"{m[label]['f1']:>{col}.3f}" - print(row_str) - print() -``` - -**Step 4: Run tests — expect PASS** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_benchmark_classifier.py -v -``` - -Expected: 7 tests pass. - -**Step 5: Commit** - -```bash -git add scripts/benchmark_classifier.py tests/test_benchmark_classifier.py -git commit -m "feat: --score mode with macro-F1, accuracy, latency, and per-label F1 table" -``` - ---- - -### Task 9: --compare mode (stdlib IMAP + table output) - -**Files:** -- Modify: `scripts/benchmark_classifier.py` - -**Step 1: Add IMAP fetch helpers** - -Add after the `_active_models()` helper in `benchmark_classifier.py`: - -```python -import email as _email_lib -import imaplib -from datetime import datetime, timedelta - -_BROAD_TERMS = [ - "interview", "opportunity", "offer letter", - "job offer", "application", "recruiting", -] - - -def _load_imap_config() -> dict[str, Any]: - import yaml - cfg_path = Path(__file__).parent.parent / "config" / "email.yaml" - with cfg_path.open() as f: - return yaml.safe_load(f) - - -def _imap_connect(cfg: dict[str, Any]) -> imaplib.IMAP4_SSL: - conn = imaplib.IMAP4_SSL(cfg["host"], cfg.get("port", 993)) - conn.login(cfg["username"], cfg["password"]) - return conn - - -def _decode_part(part: Any) -> str: - charset = part.get_content_charset() or "utf-8" - try: - return part.get_payload(decode=True).decode(charset, errors="replace") - except Exception: - return "" - - -def _parse_uid(conn: imaplib.IMAP4_SSL, uid: bytes) -> dict[str, str] | None: - try: - _, data = conn.uid("fetch", uid, "(RFC822)") - raw = data[0][1] - msg = _email_lib.message_from_bytes(raw) - subject = str(msg.get("subject", "")).strip() - body = "" - if msg.is_multipart(): - for part in msg.walk(): - if part.get_content_type() == "text/plain": - body = _decode_part(part) - break - else: - body = _decode_part(msg) - return {"subject": subject, "body": body} - except Exception: - return None - - -def _fetch_imap_sample(limit: int, days: int) -> list[dict[str, str]]: - cfg = _load_imap_config() - conn = _imap_connect(cfg) - since = (datetime.now() - timedelta(days=days)).strftime("%d-%b-%Y") - conn.select("INBOX") - - seen_uids: dict[bytes, None] = {} - for term in _BROAD_TERMS: - _, data = conn.uid("search", None, f'(SUBJECT "{term}" SINCE {since})') - for uid in (data[0] or b"").split(): - seen_uids[uid] = None - - sample = list(seen_uids.keys())[:limit] - emails = [] - for uid in sample: - parsed = _parse_uid(conn, uid) - if parsed: - emails.append(parsed) - try: - conn.logout() - except Exception: - pass - return emails -``` - -**Step 2: Replace cmd_compare stub** - -```python -def cmd_compare(args: argparse.Namespace) -> None: - active = _active_models(args.include_slow) - if args.models: - active = {k: v for k, v in active.items() if k in args.models} - - print(f"Fetching up to {args.limit} emails from IMAP …") - emails = _fetch_imap_sample(args.limit, args.days) - print(f"Fetched {len(emails)} emails. Loading {len(active)} model(s) …\n") - - adapters = [ - entry["adapter"](name, entry["model_id"]) - for name, entry in active.items() - ] - model_names = [a.name for a in adapters] - - col = 22 - subj_w = 50 - print(f"{'Subject':<{subj_w}}" + "".join(f"{n:<{col}}" for n in model_names)) - print("-" * (subj_w + col * len(model_names))) - - for row in emails: - short_subj = row["subject"][:subj_w - 1] if len(row["subject"]) > subj_w else row["subject"] - line = f"{short_subj:<{subj_w}}" - for adapter in adapters: - try: - label = adapter.classify(row["subject"], row["body"]) - except Exception as exc: - label = f"ERR:{str(exc)[:8]}" - line += f"{label:<{col}}" - print(line, flush=True) - - for adapter in adapters: - adapter.unload() - print() -``` - -**Step 3: Run full test suite** - -```bash -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_benchmark_classifier.py tests/test_classifier_adapters.py -v -``` - -Expected: all 13 tests pass. - -**Step 4: Commit** - -```bash -git add scripts/benchmark_classifier.py -git commit -m "feat: --compare mode — stdlib IMAP fetch + side-by-side model label table" -``` - ---- - -### Task 10: First real benchmark run - -No code changes — first live execution. - -**Step 1: Create your labeled scoring file** - -```bash -cp data/email_score.jsonl.example data/email_score.jsonl -``` - -Open `data/email_score.jsonl` and replace the fake examples with at least 10 real emails from your inbox. Format per line: - -```json -{"subject": "actual subject", "body": "first 600 chars of body", "label": "one_of_six_labels"} -``` - -Valid labels: `interview_scheduled`, `offer_received`, `rejected`, `positive_response`, `survey_received`, `neutral` - -**Step 2: Run --score with default models** - -```bash -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score -``` - -Models download on first run (~400–600MB each) — allow a few minutes. - -**Step 3: Run --compare on live IMAP** - -```bash -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --compare --limit 15 -``` - -**Step 4: Run slow models (optional)** - -```bash -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score --include-slow -``` - -**Step 5: Capture results (optional)** - -```bash -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score \ - > docs/plans/2026-02-26-benchmark-results.txt 2>&1 -git add docs/plans/2026-02-26-benchmark-results.txt -git commit -m "docs: initial HF classifier benchmark results" -``` - ---- - -## Quick Reference - -```bash -# Create env (once) -conda env create -f scripts/classifier_service/environment.yml - -# List models -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --list-models - -# Score against labeled data (5 default models) -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score - -# Live IMAP visual table -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --compare --limit 20 - -# Single model only -conda run -n job-seeker-classifiers python scripts/benchmark_classifier.py --score --models deberta-zeroshot - -# Run all tests (job-seeker env — mocks only, no downloads) -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_classifier_adapters.py tests/test_benchmark_classifier.py -v -``` diff --git a/docs/plans/2026-03-02-public-mirror-design.md b/docs/plans/2026-03-02-public-mirror-design.md deleted file mode 100644 index 7b5d38b..0000000 --- a/docs/plans/2026-03-02-public-mirror-design.md +++ /dev/null @@ -1,229 +0,0 @@ -# Public Mirror Strategy — Design - -**Date:** 2026-03-02 -**Scope:** Peregrine (initial); pattern applies to all future CircuitForge products -**Status:** Approved — ready for implementation planning - ---- - -## Summary - -Publish Peregrine to GitHub and Codeberg as push-mirrored community hubs. Full BSL 1.1 -codebase, no MIT carve-outs. Git hooks enforcing safety + commit format committed to the -repo so every clone gets them automatically. Issue templates and a CONTRIBUTING.md make -the project approachable for external contributors. FossHub added when a Windows installer -exists. - ---- - -## License - -**Whole repo: BSL 1.1.** No MIT exception — including `scrapers/`. The original rationale -for making scrapers MIT (community maintenance) is equally served by BSL 1.1: contributors -can fix broken scrapers, submit PRs, and run the tool at home for free. Making scrapers MIT -would allow competitors to lift CF-authored scraper code into a competing commercial product -without a license, which is not in CircuitForge's interest. - -The `LICENSE` file at repo root covers the full codebase. No `LICENSE-MIT` file needed. -CONTRIBUTING.md explains what BSL means practically for contributors. - -BSL converts to MIT after 4 years per the standard BSL 1.1 terms. - ---- - -## Mirror Sync - -Forgejo has built-in **push mirror** support (Settings → Mirror → Push mirrors). Every push -to the primary Forgejo repo auto-replicates within seconds — no CI/CD overhead, no cron job. - -Two mirrors: -- `github.com/CircuitForge/peregrine` -- `codeberg.org/CircuitForge/peregrine` - -Both under the `CircuitForge` org (consistent branding; not the personal `pyr0ball` account). -GitHub and Codeberg orgs to be created if not already present. - ---- - -## README Canonical-Source Banner - -A prominent notice near the top of the README: - -``` -> **Primary development** happens at [git.opensourcesolarpunk.com](https://git.opensourcesolarpunk.com/pyr0ball/peregrine). -> GitHub and Codeberg are push mirrors. Issues and PRs are welcome on either platform. -``` - ---- - -## CONTRIBUTING.md - -Sections: - -1. **License** — BSL 1.1 overview. What it means: self-hosting for personal non-commercial - use is free; commercial SaaS use requires a paid license; converts to MIT after 4 years. - Link to full `LICENSE`. - -2. **CLA** — One-sentence acknowledgment in bold: - *"By submitting a pull request you agree that your contribution is licensed under the - project's BSL 1.1 terms."* No separate CLA file or signature process — the PR template - repeats this as a checkbox. - -3. **Dev setup** — Docker path (recommended) and conda path, pointing to - `docs/getting-started/installation.md`. - -4. **PR process** — GH and Codeberg PRs are reviewed and cherry-picked to Forgejo; Forgejo - is the canonical merge target. Contributors do not need a Forgejo account. - -5. **Commit format** — `type: description` (or `type(scope): description`). Valid types: - `feat fix docs chore test refactor perf ci build`. Hooks enforce this — if your commit is - rejected, the hook message tells you exactly why. - -6. **Issue guidance** — link to templates; note that security issues go to - `security@circuitforge.tech`, not GitHub Issues. - ---- - -## Git Hooks (`.githooks/`) - -Committed to the repo. Activated by `setup.sh` via: - -```sh -git config core.hooksPath .githooks -``` - -`setup.sh` already runs on first clone; hook activation is added there so no contributor -has to think about it. - -### `pre-commit` - -Blocks the commit if any staged file matches: - -**Exact path blocklist:** -- `config/user.yaml` -- `config/server.yaml` -- `config/llm.yaml` -- `config/notion.yaml` -- `config/adzuna.yaml` -- `config/label_tool.yaml` -- `.env` -- `demo/data/*.db` -- `data/*.db` -- `data/*.jsonl` - -**Content scan** (regex on staged diff): -- `sk-[A-Za-z0-9]{20,}` — OpenAI-style keys -- `Bearer [A-Za-z0-9\-_]{20,}` — generic bearer tokens -- `api_key:\s*["\']?[A-Za-z0-9\-_]{16,}` — YAML key fields with values - -On match: prints the offending file/pattern, aborts with a clear message and hint to use -`git restore --staged ` or add to `.gitignore`. - -### `commit-msg` - -Reads `$1` (the commit message temp file). Rejects if: -- Message is empty or whitespace-only -- First line does not match `^(feat|fix|docs|chore|test|refactor|perf|ci|build)(\(.+\))?: .+` - -On rejection: prints the required format and lists valid types. Does not touch the message -(no auto-rewriting). - ---- - -## Issue Templates - -Location: `.github/ISSUE_TEMPLATE/` (GitHub) and `.gitea/ISSUE_TEMPLATE/` (Codeberg/Forgejo). - -### Bug Report (`bug_report.md`) - -Fields: -- Peregrine version (output of `./manage.sh status`) -- OS and runtime (Docker / conda-direct) -- Steps to reproduce -- Expected behaviour -- Actual behaviour (with log snippets) -- Relevant config (redact keys) - -### Feature Request (`feature_request.md`) - -Fields: -- Problem statement ("I want to do X but currently...") -- Proposed solution -- Alternatives considered -- Which tier this might belong to (free / paid / premium / ultra) -- Willingness to contribute a PR - -### PR Template (`.github/pull_request_template.md`) - -Fields: -- Summary of changes -- Related issue(s) -- Type of change (feat / fix / docs / ...) -- Testing done -- **CLA checkbox:** `[ ] I agree my contribution is licensed under the project's BSL 1.1 terms.` - -### Security (`SECURITY.md`) - -Single page: do not open a GitHub Issue for security vulnerabilities. Email -`security@circuitforge.tech`. Response target: 72 hours. - ---- - -## GitHub-Specific Extras - -**CI (GitHub Actions)** — `.github/workflows/ci.yml`: -- Trigger: push and PR to `main` -- Steps: checkout → set up Python 3.11 → install deps from `requirements.txt` → - `pytest tests/ -v` -- Free for public repos; gives contributors a green checkmark without needing local conda - -**Repo topics:** `job-search`, `ai-assistant`, `privacy`, `streamlit`, `python`, -`open-core`, `neurodivergent`, `accessibility`, `bsl` - -**Releases:** Mirror Forgejo tags. Release notes auto-generated from conventional commit -subjects grouped by type. - ---- - -## FossHub (Future — Windows RC prerequisite) - -When a signed Windows installer (`.msi` or `.exe`) is ready: - -1. Submit via FossHub publisher portal (`https://www.fosshub.com/contribute.html`) -2. Requirements: stable versioned release, no bundled software, no adware -3. FossHub gives a trusted, antivirus-clean download URL — important for an app running on - users' personal machines -4. Link FossHub download from README and from `circuitforge.tech` downloads section - -No action needed until Windows RC exists. - ---- - -## File Map - -``` -peregrine/ -├── .githooks/ -│ ├── pre-commit # sensitive file + key pattern blocker -│ └── commit-msg # conventional commit format enforcer -├── .github/ -│ ├── workflows/ -│ │ └── ci.yml # pytest on push/PR -│ ├── ISSUE_TEMPLATE/ -│ │ ├── bug_report.md -│ │ └── feature_request.md -│ └── pull_request_template.md -├── .gitea/ -│ └── ISSUE_TEMPLATE/ # mirrors .github/ISSUE_TEMPLATE/ for Forgejo/Codeberg -├── CONTRIBUTING.md -└── SECURITY.md -``` - ---- - -## Out of Scope - -- Forgejo mirror configuration (done via Forgejo web UI, not committed to repo) -- GitHub/Codeberg org creation (manual one-time step) -- Windows installer build pipeline (separate future effort) -- `circuitforge-core` extraction (deferred until second product) diff --git a/docs/plans/2026-03-03-feedback-button-design.md b/docs/plans/2026-03-03-feedback-button-design.md deleted file mode 100644 index 95bed8d..0000000 --- a/docs/plans/2026-03-03-feedback-button-design.md +++ /dev/null @@ -1,185 +0,0 @@ -# Feedback Button — Design - -**Date:** 2026-03-03 -**Status:** Approved -**Product:** Peregrine (`PRNG`) - ---- - -## Overview - -A floating feedback button visible on every Peregrine page that lets beta testers file -Forgejo issues directly from the UI. Supports optional attachment of diagnostic data -(logs, recent listings) and screenshots — all with explicit per-item user consent and -PII masking before anything leaves the app. - -The backend is intentionally decoupled from Streamlit so it can be wrapped in a -FastAPI route when Peregrine moves to a proper Vue/Nuxt frontend. - ---- - -## Goals - -- Zero-friction bug reporting for beta testers -- Privacy-first: nothing is sent without explicit consent + PII preview -- Future-proof: backend callable from Streamlit now, FastAPI/Vue later -- GitHub support as a config option once public mirrors are active - ---- - -## Architecture - -### Files - -| File | Role | -|---|---| -| `scripts/feedback_api.py` | Pure Python backend — no Streamlit imports | -| `app/feedback.py` | Thin Streamlit UI shell — floating button + dialog | -| `app/components/screenshot_capture.py` | Custom Streamlit component using `html2canvas` | -| `app/app.py` | One-line addition: inject feedback button in sidebar block | -| `.env` / `.env.example` | Add `FORGEJO_API_TOKEN`, `FORGEJO_REPO` | - -### Config additions (`.env`) - -``` -FORGEJO_API_TOKEN=... -FORGEJO_REPO=pyr0ball/peregrine -# GITHUB_TOKEN= # future — filed when public mirror is active -# GITHUB_REPO= # future -``` - ---- - -## Backend (`scripts/feedback_api.py`) - -Pure Python. No Streamlit dependency. All functions return plain dicts or bytes. - -### Functions - -| Function | Signature | Purpose | -|---|---|---| -| `collect_context` | `(page: str) → dict` | Page name, app version (git describe), tier, LLM backend, OS, timestamp | -| `collect_logs` | `(n: int = 100) → str` | Tail of `.streamlit.log`; `mask_pii()` applied before return | -| `collect_listings` | `(n: int = 5) → list[dict]` | Recent jobs from DB — `title`, `company`, `url` only | -| `mask_pii` | `(text: str) → str` | Regex: emails → `[email redacted]`, phones → `[phone redacted]` | -| `build_issue_body` | `(form, context, attachments) → str` | Assembles final markdown issue body | -| `create_forgejo_issue` | `(title, body, labels) → dict` | POST to Forgejo API; returns `{number, url}` | -| `upload_attachment` | `(issue_number, image_bytes, filename) → str` | POST screenshot to issue assets; returns attachment URL | -| `screenshot_page` | `(port: int) → bytes` | Server-side Playwright fallback screenshot; returns PNG bytes | - -### Issue creation — two-step - -1. `create_forgejo_issue()` → issue number -2. `upload_attachment(issue_number, ...)` → attachment auto-linked by Forgejo - -### Labels - -Always applied: `beta-feedback`, `needs-triage` -Type-based: `bug` / `feature-request` / `question` - -### Future multi-destination - -`feedback_api.py` checks both `FORGEJO_API_TOKEN` and `GITHUB_TOKEN` (when present) -and files to whichever destinations are configured. No structural changes needed when -GitHub support is added. - ---- - -## UI Flow (`app/feedback.py`) - -### Floating button - -A real Streamlit button inside a keyed container. CSS injected via -`st.markdown(unsafe_allow_html=True)` applies `position: fixed; bottom: 2rem; -right: 2rem; z-index: 9999` to the container. Hidden entirely when `IS_DEMO=true`. - -### Dialog — Step 1: Form - -- **Type selector:** Bug / Feature Request / Other -- **Title:** short text input -- **Description:** free-text area -- **Reproduction steps:** appears only when Bug is selected (adaptive) - -### Dialog — Step 2: Consent + Attachments - -``` -┌─ Include diagnostic data? ─────────────────────────────┐ -│ [toggle] │ -│ └─ if on → expandable preview of exactly what's sent │ -│ (logs tailed + masked, listings title/company/url) │ -├─ Screenshot ───────────────────────────────────────────┤ -│ [📸 Capture current view] → inline thumbnail preview │ -│ [📎 Upload screenshot] → inline thumbnail preview │ -├─ Attribution ──────────────────────────────────────────┤ -│ [ ] Include my name & email (shown from user.yaml) │ -└────────────────────────────────────────────────────────┘ -[Submit] -``` - -### Post-submit - -- Success: "Issue filed → [view on Forgejo]" with clickable link -- Error: friendly message + copy-to-clipboard fallback (issue body as text) - ---- - -## Screenshot Component (`app/components/screenshot_capture.py`) - -Uses `st.components.v1.html()` with `html2canvas` loaded from CDN (no build step). -On capture, JS renders the visible viewport to a canvas, encodes as base64 PNG, and -returns it to Python via the component value. - -Server-side Playwright (`screenshot_page()`) is the fallback when the JS component -can't return data (e.g., cross-origin iframe restrictions). It screenshots -`localhost:` from the server — captures layout/UI state but not user session -state. - -Both paths return `bytes`. The UI shows an inline thumbnail so the user can review -before submitting. - ---- - -## Privacy & PII Rules - -| Data | Included? | Condition | -|---|---|---| -| App logs | Optional | User toggles on + sees masked preview | -| Job listings | Optional (title/company/url only) | User toggles on | -| Cover letters / notes | Never | — | -| Resume content | Never | — | -| Name + email | Optional | User checks attribution checkbox | -| Screenshots | Optional | User captures or uploads | - -`mask_pii()` is applied to all text before it appears in the preview and before -submission. Users see exactly what will be sent. - ---- - -## Future: FastAPI wrapper - -When Peregrine moves to Vue/Nuxt: - -```python -# server.py (FastAPI) -from scripts.feedback_api import build_issue_body, create_forgejo_issue, upload_attachment - -@app.post("/api/feedback") -async def submit_feedback(payload: FeedbackPayload): - body = build_issue_body(payload.form, payload.context, payload.attachments) - result = create_forgejo_issue(payload.title, body, payload.labels) - if payload.screenshot: - upload_attachment(result["number"], payload.screenshot, "screenshot.png") - return {"url": result["url"]} -``` - -The Streamlit layer is replaced by a Vue `` component that POSTs -to this endpoint. Backend unchanged. - ---- - -## Out of Scope - -- Rate limiting (beta testers are trusted; add later if abused) -- Issue deduplication -- In-app issue status tracking -- Video / screen recording diff --git a/docs/plans/2026-03-03-feedback-button-plan.md b/docs/plans/2026-03-03-feedback-button-plan.md deleted file mode 100644 index 7c53195..0000000 --- a/docs/plans/2026-03-03-feedback-button-plan.md +++ /dev/null @@ -1,1136 +0,0 @@ -# Feedback Button — Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Add a floating feedback button to Peregrine that lets beta testers file Forgejo issues directly from the UI, with optional PII-masked diagnostic data and screenshot attachments. - -**Architecture:** Pure Python backend in `scripts/feedback_api.py` (no Streamlit dep, wrappable in FastAPI later) + thin Streamlit shell in `app/feedback.py`. Floating button uses CSS `position: fixed` targeting via `aria-label`. Screenshots via server-side Playwright (capture) and `st.file_uploader` (upload). - -**Tech Stack:** Python `requests`, `re`, `playwright` (optional), Streamlit 1.54 (`@st.dialog`), Forgejo REST API v1. - ---- - -## Task 1: Project setup — env config + Playwright dep - -**Files:** -- Modify: `.env.example` -- Modify: `requirements.txt` - -**Step 1: Add env vars to `.env.example`** - -Open `.env.example` and add after the existing API keys block: - -``` -# Feedback button — Forgejo issue filing -FORGEJO_API_TOKEN= -FORGEJO_REPO=pyr0ball/peregrine -FORGEJO_API_URL=https://git.opensourcesolarpunk.com/api/v1 -# GITHUB_TOKEN= # future — enable when public mirror is active -# GITHUB_REPO= # future -``` - -**Step 2: Add playwright to requirements.txt** - -Add to `requirements.txt`: - -``` -playwright>=1.40 -``` - -**Step 3: Install playwright and its browsers** - -```bash -conda run -n job-seeker pip install playwright -conda run -n job-seeker playwright install chromium --with-deps -``` - -Expected: chromium browser downloaded to playwright cache. - -**Step 4: Add FORGEJO_API_TOKEN to your local `.env`** - -Open `.env` and add: -``` -FORGEJO_API_TOKEN=your-forgejo-api-token-here -FORGEJO_REPO=pyr0ball/peregrine -FORGEJO_API_URL=https://git.opensourcesolarpunk.com/api/v1 -``` - -**Step 5: Commit** - -```bash -git add requirements.txt .env.example -git commit -m "chore: add playwright dep and Forgejo env config for feedback button" -``` - ---- - -## Task 2: Backend — PII masking + context collection - -**Files:** -- Create: `scripts/feedback_api.py` -- Create: `tests/test_feedback_api.py` - -**Step 1: Write failing tests** - -Create `tests/test_feedback_api.py`: - -```python -"""Tests for the feedback API backend.""" -import pytest -from unittest.mock import patch, MagicMock -from pathlib import Path - - -# ── mask_pii ────────────────────────────────────────────────────────────────── - -def test_mask_pii_email(): - from scripts.feedback_api import mask_pii - assert mask_pii("contact foo@bar.com please") == "contact [email redacted] please" - - -def test_mask_pii_phone_dashes(): - from scripts.feedback_api import mask_pii - assert mask_pii("call 555-123-4567 now") == "call [phone redacted] now" - - -def test_mask_pii_phone_parens(): - from scripts.feedback_api import mask_pii - assert mask_pii("(555) 867-5309") == "[phone redacted]" - - -def test_mask_pii_clean_text(): - from scripts.feedback_api import mask_pii - assert mask_pii("no sensitive data here") == "no sensitive data here" - - -def test_mask_pii_multiple_emails(): - from scripts.feedback_api import mask_pii - result = mask_pii("a@b.com and c@d.com") - assert result == "[email redacted] and [email redacted]" - - -# ── collect_context ─────────────────────────────────────────────────────────── - -def test_collect_context_required_keys(): - from scripts.feedback_api import collect_context - ctx = collect_context("Home") - for key in ("page", "version", "tier", "llm_backend", "os", "timestamp"): - assert key in ctx, f"missing key: {key}" - - -def test_collect_context_page_value(): - from scripts.feedback_api import collect_context - ctx = collect_context("MyPage") - assert ctx["page"] == "MyPage" - - -def test_collect_context_timestamp_is_utc(): - from scripts.feedback_api import collect_context - ctx = collect_context("X") - assert ctx["timestamp"].endswith("Z") -``` - -**Step 2: Run to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -v 2>&1 | head -30 -``` - -Expected: `ModuleNotFoundError: No module named 'scripts.feedback_api'` - -**Step 3: Create `scripts/feedback_api.py` with mask_pii and collect_context** - -```python -""" -Feedback API — pure Python backend, no Streamlit imports. -Called directly from app/feedback.py now; wrappable in a FastAPI route later. -""" -from __future__ import annotations - -import os -import platform -import re -import subprocess -from datetime import datetime, timezone -from pathlib import Path - -import requests -import yaml - -_ROOT = Path(__file__).parent.parent -_EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}") -_PHONE_RE = re.compile(r"(\+?1[\s\-.]?)?\(?\d{3}\)?[\s\-.]?\d{3}[\s\-.]?\d{4}") - - -def mask_pii(text: str) -> str: - """Redact email addresses and phone numbers from text.""" - text = _EMAIL_RE.sub("[email redacted]", text) - text = _PHONE_RE.sub("[phone redacted]", text) - return text - - -def collect_context(page: str) -> dict: - """Collect app context: page, version, tier, LLM backend, OS, timestamp.""" - # App version from git - try: - version = subprocess.check_output( - ["git", "describe", "--tags", "--always"], - cwd=_ROOT, text=True, timeout=5, - ).strip() - except Exception: - version = "dev" - - # Tier from user.yaml - tier = "unknown" - try: - user = yaml.safe_load((_ROOT / "config" / "user.yaml").read_text()) or {} - tier = user.get("tier", "unknown") - except Exception: - pass - - # LLM backend from llm.yaml - llm_backend = "unknown" - try: - llm = yaml.safe_load((_ROOT / "config" / "llm.yaml").read_text()) or {} - llm_backend = llm.get("provider", "unknown") - except Exception: - pass - - return { - "page": page, - "version": version, - "tier": tier, - "llm_backend": llm_backend, - "os": platform.platform(), - "timestamp": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"), - } -``` - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py::test_mask_pii_email \ - tests/test_feedback_api.py::test_mask_pii_phone_dashes \ - tests/test_feedback_api.py::test_mask_pii_phone_parens \ - tests/test_feedback_api.py::test_mask_pii_clean_text \ - tests/test_feedback_api.py::test_mask_pii_multiple_emails \ - tests/test_feedback_api.py::test_collect_context_required_keys \ - tests/test_feedback_api.py::test_collect_context_page_value \ - tests/test_feedback_api.py::test_collect_context_timestamp_is_utc -v -``` - -Expected: 8 PASSED. - -**Step 5: Commit** - -```bash -git add scripts/feedback_api.py tests/test_feedback_api.py -git commit -m "feat: feedback_api — mask_pii + collect_context" -``` - ---- - -## Task 3: Backend — log + listing collection - -**Files:** -- Modify: `scripts/feedback_api.py` -- Modify: `tests/test_feedback_api.py` - -**Step 1: Write failing tests** - -Append to `tests/test_feedback_api.py`: - -```python -# ── collect_logs ────────────────────────────────────────────────────────────── - -def test_collect_logs_returns_string(tmp_path): - from scripts.feedback_api import collect_logs - log = tmp_path / ".streamlit.log" - log.write_text("line1\nline2\nline3\n") - result = collect_logs(log_path=log, n=10) - assert isinstance(result, str) - assert "line3" in result - - -def test_collect_logs_tails_n_lines(tmp_path): - from scripts.feedback_api import collect_logs - log = tmp_path / ".streamlit.log" - log.write_text("\n".join(f"line{i}" for i in range(200))) - result = collect_logs(log_path=log, n=10) - assert "line199" in result - assert "line0" not in result - - -def test_collect_logs_masks_pii(tmp_path): - from scripts.feedback_api import collect_logs - log = tmp_path / "test.log" - log.write_text("user foo@bar.com connected\n") - result = collect_logs(log_path=log) - assert "foo@bar.com" not in result - assert "[email redacted]" in result - - -def test_collect_logs_missing_file(tmp_path): - from scripts.feedback_api import collect_logs - result = collect_logs(log_path=tmp_path / "nonexistent.log") - assert "no log file" in result.lower() - - -# ── collect_listings ────────────────────────────────────────────────────────── - -def test_collect_listings_safe_fields_only(tmp_path): - """Only title, company, url — no cover letters, notes, or emails.""" - from scripts.db import init_db, insert_job - from scripts.feedback_api import collect_listings - db = tmp_path / "test.db" - init_db(db) - insert_job(db, { - "title": "CSM", "company": "Acme", "url": "https://example.com/1", - "source": "linkedin", "location": "Remote", "is_remote": True, - "salary": "", "description": "great role", "date_found": "2026-03-01", - }) - results = collect_listings(db_path=db, n=5) - assert len(results) == 1 - assert set(results[0].keys()) == {"title", "company", "url"} - assert results[0]["title"] == "CSM" - - -def test_collect_listings_respects_n(tmp_path): - from scripts.db import init_db, insert_job - from scripts.feedback_api import collect_listings - db = tmp_path / "test.db" - init_db(db) - for i in range(10): - insert_job(db, { - "title": f"Job {i}", "company": "Acme", "url": f"https://example.com/{i}", - "source": "linkedin", "location": "Remote", "is_remote": False, - "salary": "", "description": "", "date_found": "2026-03-01", - }) - assert len(collect_listings(db_path=db, n=3)) == 3 -``` - -**Step 2: Run to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "collect_logs or collect_listings" -v 2>&1 | head -20 -``` - -Expected: all FAIL with `ImportError` or similar. - -**Step 3: Add functions to `scripts/feedback_api.py`** - -Append after `collect_context`: - -```python -def collect_logs(n: int = 100, log_path: Path | None = None) -> str: - """Return last n lines of the Streamlit log, with PII masked.""" - path = log_path or (_ROOT / ".streamlit.log") - if not path.exists(): - return "(no log file found)" - lines = path.read_text(errors="replace").splitlines() - return mask_pii("\n".join(lines[-n:])) - - -def collect_listings(db_path: Path | None = None, n: int = 5) -> list[dict]: - """Return the n most-recent job listings — title, company, url only.""" - import sqlite3 - from scripts.db import DEFAULT_DB - path = db_path or DEFAULT_DB - conn = sqlite3.connect(path) - conn.row_factory = sqlite3.Row - rows = conn.execute( - "SELECT title, company, url FROM jobs ORDER BY id DESC LIMIT ?", (n,) - ).fetchall() - conn.close() - return [{"title": r["title"], "company": r["company"], "url": r["url"]} for r in rows] -``` - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "collect_logs or collect_listings" -v -``` - -Expected: 6 PASSED. - -**Step 5: Commit** - -```bash -git add scripts/feedback_api.py tests/test_feedback_api.py -git commit -m "feat: feedback_api — collect_logs + collect_listings" -``` - ---- - -## Task 4: Backend — issue body builder - -**Files:** -- Modify: `scripts/feedback_api.py` -- Modify: `tests/test_feedback_api.py` - -**Step 1: Write failing tests** - -Append to `tests/test_feedback_api.py`: - -```python -# ── build_issue_body ────────────────────────────────────────────────────────── - -def test_build_issue_body_contains_description(): - from scripts.feedback_api import build_issue_body - form = {"type": "bug", "title": "Test", "description": "it broke", "repro": ""} - ctx = {"page": "Home", "version": "v1.0", "tier": "free", - "llm_backend": "ollama", "os": "Linux", "timestamp": "2026-03-03T00:00:00Z"} - body = build_issue_body(form, ctx, {}) - assert "it broke" in body - assert "Home" in body - assert "v1.0" in body - - -def test_build_issue_body_bug_includes_repro(): - from scripts.feedback_api import build_issue_body - form = {"type": "bug", "title": "X", "description": "desc", "repro": "step 1\nstep 2"} - body = build_issue_body(form, {}, {}) - assert "step 1" in body - assert "Reproduction" in body - - -def test_build_issue_body_no_repro_for_feature(): - from scripts.feedback_api import build_issue_body - form = {"type": "feature", "title": "X", "description": "add dark mode", "repro": "ignored"} - body = build_issue_body(form, {}, {}) - assert "Reproduction" not in body - - -def test_build_issue_body_logs_in_collapsible(): - from scripts.feedback_api import build_issue_body - form = {"type": "other", "title": "X", "description": "Y", "repro": ""} - body = build_issue_body(form, {}, {"logs": "log line 1\nlog line 2"}) - assert "
" in body - assert "log line 1" in body - - -def test_build_issue_body_omits_logs_when_not_provided(): - from scripts.feedback_api import build_issue_body - form = {"type": "bug", "title": "X", "description": "Y", "repro": ""} - body = build_issue_body(form, {}, {}) - assert "
" not in body - - -def test_build_issue_body_submitter_attribution(): - from scripts.feedback_api import build_issue_body - form = {"type": "bug", "title": "X", "description": "Y", "repro": ""} - body = build_issue_body(form, {}, {"submitter": "Jane Doe "}) - assert "Jane Doe" in body - - -def test_build_issue_body_listings_shown(): - from scripts.feedback_api import build_issue_body - form = {"type": "bug", "title": "X", "description": "Y", "repro": ""} - listings = [{"title": "CSM", "company": "Acme", "url": "https://example.com/1"}] - body = build_issue_body(form, {}, {"listings": listings}) - assert "CSM" in body - assert "Acme" in body -``` - -**Step 2: Run to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "build_issue_body" -v 2>&1 | head -20 -``` - -**Step 3: Add `build_issue_body` to `scripts/feedback_api.py`** - -Append after `collect_listings`: - -```python -def build_issue_body(form: dict, context: dict, attachments: dict) -> str: - """Assemble the Forgejo issue markdown body from form data, context, and attachments.""" - _TYPE_LABELS = {"bug": "🐛 Bug", "feature": "✨ Feature Request", "other": "💬 Other"} - lines: list[str] = [ - f"## {_TYPE_LABELS.get(form.get('type', 'other'), '💬 Other')}", - "", - form.get("description", ""), - "", - ] - - if form.get("type") == "bug" and form.get("repro"): - lines += ["### Reproduction Steps", "", form["repro"], ""] - - if context: - lines += ["### Context", ""] - for k, v in context.items(): - lines.append(f"- **{k}:** {v}") - lines.append("") - - if attachments.get("logs"): - lines += [ - "
", - "App Logs (last 100 lines)", - "", - "```", - attachments["logs"], - "```", - "
", - "", - ] - - if attachments.get("listings"): - lines += ["### Recent Listings", ""] - for j in attachments["listings"]: - lines.append(f"- [{j['title']} @ {j['company']}]({j['url']})") - lines.append("") - - if attachments.get("submitter"): - lines += ["---", f"*Submitted by: {attachments['submitter']}*"] - - return "\n".join(lines) -``` - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "build_issue_body" -v -``` - -Expected: 7 PASSED. - -**Step 5: Commit** - -```bash -git add scripts/feedback_api.py tests/test_feedback_api.py -git commit -m "feat: feedback_api — build_issue_body" -``` - ---- - -## Task 5: Backend — Forgejo API client - -**Files:** -- Modify: `scripts/feedback_api.py` -- Modify: `tests/test_feedback_api.py` - -**Step 1: Write failing tests** - -Append to `tests/test_feedback_api.py`: - -```python -# ── Forgejo API ─────────────────────────────────────────────────────────────── - -@patch("scripts.feedback_api.requests.get") -@patch("scripts.feedback_api.requests.post") -def test_ensure_labels_uses_existing(mock_post, mock_get): - from scripts.feedback_api import _ensure_labels - mock_get.return_value.ok = True - mock_get.return_value.json.return_value = [ - {"name": "beta-feedback", "id": 1}, - {"name": "bug", "id": 2}, - ] - ids = _ensure_labels( - ["beta-feedback", "bug"], - "https://example.com/api/v1", {"Authorization": "token x"}, "owner/repo" - ) - assert ids == [1, 2] - mock_post.assert_not_called() - - -@patch("scripts.feedback_api.requests.get") -@patch("scripts.feedback_api.requests.post") -def test_ensure_labels_creates_missing(mock_post, mock_get): - from scripts.feedback_api import _ensure_labels - mock_get.return_value.ok = True - mock_get.return_value.json.return_value = [] - mock_post.return_value.ok = True - mock_post.return_value.json.return_value = {"id": 99} - ids = _ensure_labels( - ["needs-triage"], - "https://example.com/api/v1", {"Authorization": "token x"}, "owner/repo" - ) - assert 99 in ids - - -@patch("scripts.feedback_api._ensure_labels", return_value=[1, 2]) -@patch("scripts.feedback_api.requests.post") -def test_create_forgejo_issue_success(mock_post, mock_labels, monkeypatch): - from scripts.feedback_api import create_forgejo_issue - monkeypatch.setenv("FORGEJO_API_TOKEN", "testtoken") - monkeypatch.setenv("FORGEJO_REPO", "owner/repo") - monkeypatch.setenv("FORGEJO_API_URL", "https://example.com/api/v1") - mock_post.return_value.status_code = 201 - mock_post.return_value.raise_for_status = lambda: None - mock_post.return_value.json.return_value = {"number": 42, "html_url": "https://example.com/issues/42"} - result = create_forgejo_issue("Test issue", "body text", ["beta-feedback", "bug"]) - assert result["number"] == 42 - assert "42" in result["url"] - - -@patch("scripts.feedback_api.requests.post") -def test_upload_attachment_returns_url(mock_post, monkeypatch): - from scripts.feedback_api import upload_attachment - monkeypatch.setenv("FORGEJO_API_TOKEN", "testtoken") - monkeypatch.setenv("FORGEJO_REPO", "owner/repo") - monkeypatch.setenv("FORGEJO_API_URL", "https://example.com/api/v1") - mock_post.return_value.status_code = 201 - mock_post.return_value.raise_for_status = lambda: None - mock_post.return_value.json.return_value = { - "uuid": "abc", "browser_download_url": "https://example.com/assets/abc" - } - url = upload_attachment(42, b"\x89PNG", "screenshot.png") - assert url == "https://example.com/assets/abc" -``` - -**Step 2: Run to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "label or issue or attach" -v 2>&1 | head -20 -``` - -**Step 3: Add Forgejo API functions to `scripts/feedback_api.py`** - -Append after `build_issue_body`: - -```python -def _ensure_labels( - label_names: list[str], base_url: str, headers: dict, repo: str -) -> list[int]: - """Look up or create Forgejo labels by name. Returns list of IDs.""" - _COLORS = { - "beta-feedback": "#0075ca", - "needs-triage": "#e4e669", - "bug": "#d73a4a", - "feature-request": "#a2eeef", - "question": "#d876e3", - } - resp = requests.get(f"{base_url}/repos/{repo}/labels", headers=headers, timeout=10) - existing = {lb["name"]: lb["id"] for lb in resp.json()} if resp.ok else {} - ids: list[int] = [] - for name in label_names: - if name in existing: - ids.append(existing[name]) - else: - r = requests.post( - f"{base_url}/repos/{repo}/labels", - headers=headers, - json={"name": name, "color": _COLORS.get(name, "#ededed")}, - timeout=10, - ) - if r.ok: - ids.append(r.json()["id"]) - return ids - - -def create_forgejo_issue(title: str, body: str, labels: list[str]) -> dict: - """Create a Forgejo issue. Returns {"number": int, "url": str}.""" - token = os.environ.get("FORGEJO_API_TOKEN", "") - repo = os.environ.get("FORGEJO_REPO", "pyr0ball/peregrine") - base = os.environ.get("FORGEJO_API_URL", "https://git.opensourcesolarpunk.com/api/v1") - headers = {"Authorization": f"token {token}", "Content-Type": "application/json"} - label_ids = _ensure_labels(labels, base, headers, repo) - resp = requests.post( - f"{base}/repos/{repo}/issues", - headers=headers, - json={"title": title, "body": body, "labels": label_ids}, - timeout=15, - ) - resp.raise_for_status() - data = resp.json() - return {"number": data["number"], "url": data["html_url"]} - - -def upload_attachment( - issue_number: int, image_bytes: bytes, filename: str = "screenshot.png" -) -> str: - """Upload a screenshot to an existing Forgejo issue. Returns attachment URL.""" - token = os.environ.get("FORGEJO_API_TOKEN", "") - repo = os.environ.get("FORGEJO_REPO", "pyr0ball/peregrine") - base = os.environ.get("FORGEJO_API_URL", "https://git.opensourcesolarpunk.com/api/v1") - headers = {"Authorization": f"token {token}"} - resp = requests.post( - f"{base}/repos/{repo}/issues/{issue_number}/assets", - headers=headers, - files={"attachment": (filename, image_bytes, "image/png")}, - timeout=15, - ) - resp.raise_for_status() - return resp.json().get("browser_download_url", "") -``` - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "label or issue or attach" -v -``` - -Expected: 4 PASSED. - -**Step 5: Run full test suite to check for regressions** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -v -``` - -Expected: all PASSED. - -**Step 6: Commit** - -```bash -git add scripts/feedback_api.py tests/test_feedback_api.py -git commit -m "feat: feedback_api — Forgejo label management + issue filing + attachment upload" -``` - ---- - -## Task 6: Backend — server-side screenshot capture - -**Files:** -- Modify: `scripts/feedback_api.py` -- Modify: `tests/test_feedback_api.py` - -**Step 1: Write failing tests** - -Append to `tests/test_feedback_api.py`: - -```python -# ── screenshot_page ─────────────────────────────────────────────────────────── - -def test_screenshot_page_returns_none_without_playwright(monkeypatch): - """If playwright is not installed, screenshot_page returns None gracefully.""" - import builtins - real_import = builtins.__import__ - def mock_import(name, *args, **kwargs): - if name == "playwright.sync_api": - raise ImportError("no playwright") - return real_import(name, *args, **kwargs) - monkeypatch.setattr(builtins, "__import__", mock_import) - from scripts.feedback_api import screenshot_page - result = screenshot_page(port=9999) - assert result is None - - -@patch("scripts.feedback_api.sync_playwright") -def test_screenshot_page_returns_bytes(mock_pw): - """screenshot_page returns PNG bytes when playwright is available.""" - from scripts.feedback_api import screenshot_page - fake_png = b"\x89PNG\r\n\x1a\n" - mock_context = MagicMock() - mock_pw.return_value.__enter__ = lambda s: mock_context - mock_pw.return_value.__exit__ = MagicMock(return_value=False) - mock_browser = mock_context.chromium.launch.return_value - mock_page = mock_browser.new_page.return_value - mock_page.screenshot.return_value = fake_png - result = screenshot_page(port=8502) - assert result == fake_png -``` - -**Step 2: Run to verify they fail** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "screenshot" -v 2>&1 | head -20 -``` - -**Step 3: Add `screenshot_page` to `scripts/feedback_api.py`** - -Append after `upload_attachment`. Note the `try/except ImportError` for graceful degradation: - -```python -def screenshot_page(port: int | None = None) -> bytes | None: - """ - Capture a screenshot of the running Peregrine UI using Playwright. - Returns PNG bytes, or None if Playwright is not installed. - """ - try: - from playwright.sync_api import sync_playwright - except ImportError: - return None - - if port is None: - port = int(os.environ.get("STREAMLIT_PORT", os.environ.get("STREAMLIT_SERVER_PORT", "8502"))) - - try: - with sync_playwright() as p: - browser = p.chromium.launch() - page = browser.new_page(viewport={"width": 1280, "height": 800}) - page.goto(f"http://localhost:{port}", timeout=10_000) - page.wait_for_load_state("networkidle", timeout=10_000) - png = page.screenshot(full_page=False) - browser.close() - return png - except Exception: - return None -``` - -Also add the import at the top of the try block to satisfy the mock test. The import at the function level is correct — do NOT add it to the module level, because we want the graceful degradation path to work. - -**Step 4: Run tests to verify they pass** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -k "screenshot" -v -``` - -Expected: 2 PASSED. - -**Step 5: Run full backend test suite** - -```bash -conda run -n job-seeker pytest tests/test_feedback_api.py -v -``` - -Expected: all PASSED. - -**Step 6: Commit** - -```bash -git add scripts/feedback_api.py tests/test_feedback_api.py -git commit -m "feat: feedback_api — screenshot_page with Playwright (graceful fallback)" -``` - ---- - -## Task 7: UI — floating button + feedback dialog - -**Files:** -- Create: `app/feedback.py` - -No pytest tests for Streamlit UI (too brittle for dialogs). Manual verification in Task 8. - -**Step 1: Create `app/feedback.py`** - -```python -""" -Floating feedback button + dialog — thin Streamlit shell. -All business logic lives in scripts/feedback_api.py. -""" -from __future__ import annotations - -import os -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).parent.parent)) - -import streamlit as st - -# ── CSS: float the button to the bottom-right corner ───────────────────────── -# Targets the button by its aria-label (set via `help=` parameter). -_FLOAT_CSS = """ - -""" - - -@st.dialog("Send Feedback", width="large") -def _feedback_dialog(page: str) -> None: - """Two-step feedback dialog: form → consent/attachments → submit.""" - from scripts.feedback_api import ( - collect_context, collect_logs, collect_listings, - build_issue_body, create_forgejo_issue, - upload_attachment, screenshot_page, - ) - from scripts.db import DEFAULT_DB - - # ── Initialise step counter ─────────────────────────────────────────────── - if "fb_step" not in st.session_state: - st.session_state.fb_step = 1 - - # ═════════════════════════════════════════════════════════════════════════ - # STEP 1 — Form - # ═════════════════════════════════════════════════════════════════════════ - if st.session_state.fb_step == 1: - st.subheader("What's on your mind?") - - fb_type = st.selectbox( - "Type", ["Bug", "Feature Request", "Other"], key="fb_type" - ) - fb_title = st.text_input( - "Title", placeholder="Short summary of the issue or idea", key="fb_title" - ) - fb_desc = st.text_area( - "Description", - placeholder="Describe what happened or what you'd like to see...", - key="fb_desc", - ) - if fb_type == "Bug": - st.text_area( - "Reproduction steps", - placeholder="1. Go to...\n2. Click...\n3. See error", - key="fb_repro", - ) - - col_cancel, _, col_next = st.columns([1, 3, 1]) - with col_cancel: - if st.button("Cancel"): - _clear_feedback_state() - st.rerun() - with col_next: - if st.button( - "Next →", - type="primary", - disabled=not st.session_state.get("fb_title", "").strip() - or not st.session_state.get("fb_desc", "").strip(), - ): - st.session_state.fb_step = 2 - st.rerun() - - # ═════════════════════════════════════════════════════════════════════════ - # STEP 2 — Consent + attachments - # ═════════════════════════════════════════════════════════════════════════ - elif st.session_state.fb_step == 2: - st.subheader("Optional: attach diagnostic data") - - # ── Diagnostic data toggle + preview ───────────────────────────────── - include_diag = st.toggle( - "Include diagnostic data (logs + recent listings)", key="fb_diag" - ) - if include_diag: - with st.expander("Preview what will be sent", expanded=True): - st.caption("**App logs (last 100 lines, PII masked):**") - st.code(collect_logs(100), language=None) - st.caption("**Recent listings (title / company / URL only):**") - for j in collect_listings(DEFAULT_DB, 5): - st.write(f"- {j['title']} @ {j['company']} — {j['url']}") - - # ── Screenshot ──────────────────────────────────────────────────────── - st.divider() - st.caption("**Screenshot** (optional)") - col_cap, col_up = st.columns(2) - - with col_cap: - if st.button("📸 Capture current view"): - with st.spinner("Capturing page…"): - png = screenshot_page() - if png: - st.session_state.fb_screenshot = png - else: - st.warning( - "Playwright not available — install it with " - "`playwright install chromium`, or upload a screenshot instead." - ) - - with col_up: - uploaded = st.file_uploader( - "Upload screenshot", - type=["png", "jpg", "jpeg"], - label_visibility="collapsed", - key="fb_upload", - ) - if uploaded: - st.session_state.fb_screenshot = uploaded.read() - - if st.session_state.get("fb_screenshot"): - st.image( - st.session_state["fb_screenshot"], - caption="Screenshot preview — this will be attached to the issue", - use_container_width=True, - ) - if st.button("🗑 Remove screenshot"): - st.session_state.pop("fb_screenshot", None) - st.rerun() - - # ── Attribution consent ─────────────────────────────────────────────── - st.divider() - submitter: str | None = None - try: - import yaml - _ROOT = Path(__file__).parent.parent - user = yaml.safe_load((_ROOT / "config" / "user.yaml").read_text()) or {} - name = (user.get("name") or "").strip() - email = (user.get("email") or "").strip() - if name or email: - label = f"Include my name & email in the report: **{name}** ({email})" - if st.checkbox(label, key="fb_attr"): - submitter = f"{name} <{email}>" - except Exception: - pass - - # ── Navigation ──────────────────────────────────────────────────────── - col_back, _, col_submit = st.columns([1, 3, 2]) - with col_back: - if st.button("← Back"): - st.session_state.fb_step = 1 - st.rerun() - - with col_submit: - if st.button("Submit Feedback", type="primary"): - _submit(page, include_diag, submitter, collect_context, - collect_logs, collect_listings, build_issue_body, - create_forgejo_issue, upload_attachment, DEFAULT_DB) - - -def _submit(page, include_diag, submitter, collect_context, collect_logs, - collect_listings, build_issue_body, create_forgejo_issue, - upload_attachment, db_path) -> None: - """Handle form submission: build body, file issue, upload screenshot.""" - with st.spinner("Filing issue…"): - context = collect_context(page) - attachments: dict = {} - if include_diag: - attachments["logs"] = collect_logs(100) - attachments["listings"] = collect_listings(db_path, 5) - if submitter: - attachments["submitter"] = submitter - - fb_type = st.session_state.get("fb_type", "Other") - type_key = {"Bug": "bug", "Feature Request": "feature", "Other": "other"}.get( - fb_type, "other" - ) - labels = ["beta-feedback", "needs-triage"] - labels.append( - {"bug": "bug", "feature": "feature-request"}.get(type_key, "question") - ) - - form = { - "type": type_key, - "description": st.session_state.get("fb_desc", ""), - "repro": st.session_state.get("fb_repro", "") if type_key == "bug" else "", - } - - body = build_issue_body(form, context, attachments) - - try: - result = create_forgejo_issue( - st.session_state.get("fb_title", "Feedback"), body, labels - ) - screenshot = st.session_state.get("fb_screenshot") - if screenshot: - upload_attachment(result["number"], screenshot) - - _clear_feedback_state() - st.success(f"Issue filed! [View on Forgejo]({result['url']})") - st.balloons() - - except Exception as exc: - st.error(f"Failed to file issue: {exc}") - - -def _clear_feedback_state() -> None: - for key in [ - "fb_step", "fb_type", "fb_title", "fb_desc", "fb_repro", - "fb_diag", "fb_upload", "fb_attr", "fb_screenshot", - ]: - st.session_state.pop(key, None) - - -def inject_feedback_button(page: str = "Unknown") -> None: - """ - Inject the floating feedback button. Call once per page render in app.py. - Hidden automatically in DEMO_MODE. - """ - if os.environ.get("DEMO_MODE", "").lower() in ("1", "true", "yes"): - return - if not os.environ.get("FORGEJO_API_TOKEN"): - return # silently skip if not configured - - st.markdown(_FLOAT_CSS, unsafe_allow_html=True) - if st.button( - "💬 Feedback", - key="__feedback_floating_btn__", - help="Send feedback or report a bug", - ): - _feedback_dialog(page) -``` - -**Step 2: Verify the file has no syntax errors** - -```bash -conda run -n job-seeker python -c "import app.feedback; print('OK')" -``` - -Expected: `OK` - -**Step 3: Commit** - -```bash -git add app/feedback.py -git commit -m "feat: floating feedback button + two-step dialog (Streamlit shell)" -``` - ---- - -## Task 8: Wire into app.py + manual verification - -**Files:** -- Modify: `app/app.py` - -**Step 1: Add import and call to `app/app.py`** - -Find the `with st.sidebar:` block near the bottom of `app/app.py` (currently ends with `st.caption(f"Peregrine {_get_version()}")`). - -Add two lines — the import near the top of the file (after the existing imports), and the call in the sidebar block: - -At the top of `app/app.py`, after `from scripts.db import ...`: -```python -from app.feedback import inject_feedback_button -``` - -At the end of the `with st.sidebar:` block, after `st.caption(...)`: -```python - inject_feedback_button(page=st.session_state.get("__current_page__", "Unknown")) -``` - -To capture the current page name, also add this anywhere early in the sidebar block (before the caption): -```python - # Track current page for feedback context - try: - _page_name = pg.pages[st.session_state.get("page_index", 0)].title - except Exception: - _page_name = "Unknown" - inject_feedback_button(page=_page_name) -``` - -> **Note on page detection:** Streamlit's `st.navigation` doesn't expose the current page via a simple API. If `pg.pages[...]` doesn't resolve cleanly, simplify to `inject_feedback_button()` with no argument — the page context is a nice-to-have, not critical. - -**Step 2: Verify app starts without errors** - -```bash -bash /Library/Development/CircuitForge/peregrine/manage.sh restart -bash /Library/Development/CircuitForge/peregrine/manage.sh logs -``` - -Expected: no Python tracebacks in logs. - -**Step 3: Manual end-to-end verification checklist** - -Open http://localhost:8502 and verify: - -- [ ] A "💬 Feedback" pill button appears fixed in the bottom-right corner -- [ ] Button is visible on Home, Setup, and all other pages -- [ ] Button is NOT visible in DEMO_MODE (set `DEMO_MODE=1` in `.env`, restart, check) -- [ ] Clicking the button opens the two-step dialog -- [ ] Step 1: selecting "Bug" reveals the reproduction steps field; "Feature Request" hides it -- [ ] "Next →" is disabled until title + description are filled -- [ ] Step 2: toggling diagnostic data shows the masked preview (no real emails/phones) -- [ ] "📸 Capture current view" either shows a thumbnail or a warning about Playwright -- [ ] Uploading a PNG via file picker shows a thumbnail -- [ ] "🗑 Remove screenshot" clears the thumbnail -- [ ] Attribution checkbox shows the name/email from user.yaml -- [ ] Submitting files a real issue at https://git.opensourcesolarpunk.com/pyr0ball/peregrine/issues -- [ ] Issue has correct labels (beta-feedback, needs-triage, + type label) -- [ ] If screenshot provided, it appears as an attachment on the Forgejo issue -- [ ] Success message contains a clickable link to the issue - -**Step 4: Commit** - -```bash -git add app/app.py -git commit -m "feat: wire feedback button into app.py sidebar" -``` - ---- - -## Done - -All tasks complete. The feedback button is live. When moving to Vue/Nuxt, `scripts/feedback_api.py` is wrapped in a FastAPI route — no changes to the backend needed. - -**Future tasks (not in scope now):** -- GitHub mirroring (add `GITHUB_TOKEN` + `GITHUB_REPO` env vars, add `create_github_issue()`) -- Rate limiting (if beta users abuse it) -- In-app issue status tracking diff --git a/docs/plans/2026-03-05-digest-parsers-design.md b/docs/plans/2026-03-05-digest-parsers-design.md deleted file mode 100644 index c09926e..0000000 --- a/docs/plans/2026-03-05-digest-parsers-design.md +++ /dev/null @@ -1,242 +0,0 @@ -# Digest Email Parsers — Design - -**Date:** 2026-03-05 -**Products:** Peregrine (primary), Avocet (bucket) -**Status:** Design approved, ready for implementation planning - ---- - -## Problem - -Peregrine's `imap_sync.py` can extract leads from digest emails, but only for LinkedIn — the -parser is hardcoded inline with no extension point. Adzuna and The Ladders digest emails are -unhandled. Additionally, any digest email from an unknown sender is silently dropped with no -way to collect samples for building new parsers. - ---- - -## Solution Overview - -Two complementary changes: - -1. **`peregrine/scripts/digest_parsers.py`** — a standalone parser module with a sender registry - and dispatcher. `imap_sync.py` calls a single function; the registry handles dispatch. - LinkedIn parser moves here; Adzuna and Ladders parsers are built against real IMAP samples. - -2. **Avocet digest bucket** — when a user labels an email as `digest` in the Avocet label UI, - the email is appended to `data/digest_samples.jsonl`. This file is the corpus for building - and testing new parsers for senders not yet in the registry. - ---- - -## Architecture - -### Production path (Peregrine) - -``` -imap_sync._scan_unmatched_leads() - │ - ├─ parse_digest(from_addr, body) - │ │ - │ ├─ None → unknown sender → fall through to LLM extraction (unchanged) - │ ├─ [] → known sender, nothing found → skip - │ └─ [...] → jobs found → insert_job() + submit_task("scrape_url") - │ - └─ continue (digest email consumed; does not reach LLM path) -``` - -### Sample collection path (Avocet) - -``` -Avocet label UI - │ - └─ label == "digest" - │ - └─ append to data/digest_samples.jsonl - │ - └─ used as reference for building new parsers -``` - ---- - -## Module: `peregrine/scripts/digest_parsers.py` - -### Parser interface - -Each parser function: - -```python -def parse_(body: str) -> list[dict] -``` - -Returns zero or more job dicts: - -```python -{ - "title": str, # job title - "company": str, # company name - "location": str, # location string (may be empty) - "url": str, # canonical URL, tracking params stripped - "source": str, # "linkedin" | "adzuna" | "theladders" -} -``` - -### Dispatcher - -```python -DIGEST_PARSERS: dict[str, tuple[str, Callable[[str], list[dict]]]] = { - "jobalerts@linkedin.com": ("linkedin", parse_linkedin), - "noreply@adzuna.com": ("adzuna", parse_adzuna), - "noreply@theladders.com": ("theladders", parse_theladders), -} - -def parse_digest(from_addr: str, body: str) -> list[dict] | None: - """ - Dispatch to the appropriate parser based on sender address. - - Returns: - None — no parser matched (not a known digest sender) - [] — parser matched, no extractable jobs found - [dict, ...] — one dict per job card extracted - """ - addr = from_addr.lower() - for sender, (source, parse_fn) in DIGEST_PARSERS.items(): - if sender in addr: - return parse_fn(body) - return None -``` - -Sender matching is a substring check, tolerant of display-name wrappers -(`"LinkedIn "` matches correctly). - -### Parsers - -**`parse_linkedin`** — moved verbatim from `imap_sync.parse_linkedin_alert()`, renamed. -No behavior change. - -**`parse_adzuna`** — built against real Adzuna digest email bodies pulled from the -configured IMAP account during implementation. Expected format: job blocks separated -by consistent delimiters with title, company, location, and a trackable URL per block. - -**`parse_theladders`** — same approach. The Ladders already has a web scraper in -`scripts/custom_boards/theladders.py`; URL canonicalization patterns from there apply here. - ---- - -## Changes to `imap_sync.py` - -Replace the LinkedIn-specific block in `_scan_unmatched_leads()` (~lines 561–585): - -**Before:** -```python -if _LINKEDIN_ALERT_SENDER in parsed["from_addr"].lower(): - cards = parse_linkedin_alert(parsed["body"]) - for card in cards: - # ... LinkedIn-specific insert ... - known_message_ids.add(mid) - continue -``` - -**After:** -```python -from scripts.digest_parsers import parse_digest # top of file - -cards = parse_digest(parsed["from_addr"], parsed["body"]) -if cards is not None: - for card in cards: - if card["url"] in existing_urls: - continue - job_id = insert_job(db_path, { - "title": card["title"], - "company": card["company"], - "url": card["url"], - "source": card["source"], - "location": card["location"], - "is_remote": 0, - "salary": "", - "description": "", - "date_found": datetime.now().isoformat()[:10], - }) - if job_id: - submit_task(db_path, "scrape_url", job_id) - existing_urls.add(card["url"]) - new_leads += 1 - print(f"[imap] digest ({card['source']}) → {card['company']} — {card['title']}") - known_message_ids.add(mid) - continue -``` - -`parse_digest` returning `None` falls through to the existing LLM extraction path — all -non-digest recruitment emails are completely unaffected. - ---- - -## Avocet: Digest Bucket - -### File - -`avocet/data/digest_samples.jsonl` — gitignored. An `.example` entry is committed. - -Schema matches the existing label queue (JSONL on-disk schema): - -```json -{"subject": "...", "body": "...", "from_addr": "...", "date": "...", "account": "..."} -``` - -### Trigger - -In `app/label_tool.py` and `app/api.py`: when a `digest` label is applied, append the -email to `digest_samples.jsonl` alongside the normal write to `email_score.jsonl`. - -No Peregrine dependency — if the file path doesn't exist the `data/` directory is created -automatically. Avocet remains fully standalone. - -### Usage - -When a new digest sender appears in the wild: -1. Label representative emails as `digest` in Avocet → samples land in `digest_samples.jsonl` -2. Inspect samples, write `parse_(body)` in `digest_parsers.py` -3. Add the sender string to `DIGEST_PARSERS` -4. Add fixture test in `peregrine/tests/test_digest_parsers.py` - ---- - -## Testing - -### `peregrine/tests/test_digest_parsers.py` - -- Fixture bodies sourced from real IMAP samples (anonymized company names / URLs acceptable) -- Each parser: valid body → expected cards returned -- Each parser: empty / malformed body → `[]`, no exception -- Dispatcher: known sender → correct parser invoked -- Dispatcher: unknown sender → `None` -- URL canonicalization: tracking params stripped, canonical form asserted -- Dedup within digest: same URL appearing twice in one email → one card - -### `avocet/tests/test_digest_bucket.py` - -- `digest` label → row appended to `digest_samples.jsonl` -- Any other label → `digest_samples.jsonl` not touched -- First write creates `data/` directory if absent - ---- - -## Files Changed / Created - -| File | Change | -|------|--------| -| `peregrine/scripts/digest_parsers.py` | **New** — parser module | -| `peregrine/scripts/imap_sync.py` | Replace inline LinkedIn block with `parse_digest()` call | -| `peregrine/tests/test_digest_parsers.py` | **New** — parser unit tests | -| `avocet/app/label_tool.py` | Append to `digest_samples.jsonl` on `digest` label | -| `avocet/app/api.py` | Same — digest bucket write in label endpoint | -| `avocet/tests/test_digest_bucket.py` | **New** — bucket write tests | -| `avocet/data/digest_samples.jsonl.example` | **New** — committed sample for reference | - ---- - -## Out of Scope - -- Avocet → Peregrine direct import trigger (deferred; bucket is sufficient for now) -- `background_tasks` integration for digest re-processing (not needed with bucket approach) -- HTML digest parsing (all three senders send plain-text alerts; revisit if needed) diff --git a/docs/plans/2026-03-05-digest-parsers-plan.md b/docs/plans/2026-03-05-digest-parsers-plan.md deleted file mode 100644 index d4e5e8f..0000000 --- a/docs/plans/2026-03-05-digest-parsers-plan.md +++ /dev/null @@ -1,897 +0,0 @@ -# Digest Email Parsers Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Extract job listings from LinkedIn, Adzuna, and The Ladders digest emails into Peregrine leads, with an Avocet bucket that collects digest samples for future parser development. - -**Architecture:** New `peregrine/scripts/digest_parsers.py` exposes a `parse_digest(from_addr, body)` dispatcher backed by a sender registry. `imap_sync.py` replaces its inline LinkedIn block with one dispatcher call. Avocet's two label paths (`label_tool.py` + `api.py`) append digest-labeled emails to `data/digest_samples.jsonl`. Adzuna and Ladders parsers are built from real IMAP samples fetched in Task 2. - -**Tech Stack:** Python stdlib only — `re`, `json`, `pathlib`. No new dependencies. - ---- - -### Task 1: Create `digest_parsers.py` with dispatcher + LinkedIn parser - -**Files:** -- Create: `peregrine/scripts/digest_parsers.py` -- Create: `peregrine/tests/test_digest_parsers.py` - -**Context:** -`parse_linkedin_alert()` currently lives inline in `imap_sync.py`. We move it here (renamed -`parse_linkedin`) and wrap it in a dispatcher. All other parsers plug into the same registry. - -Run all tests with: -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py -v -``` - ---- - -**Step 1: Write the failing tests** - -Create `peregrine/tests/test_digest_parsers.py`: - -```python -"""Tests for digest email parser registry.""" -import pytest -from scripts.digest_parsers import parse_digest, parse_linkedin - -# ── LinkedIn fixture ────────────────────────────────────────────────────────── -# Mirrors the plain-text format LinkedIn Job Alert emails actually send. -# Each job block is separated by a line of 10+ dashes. -LINKEDIN_BODY = """\ -Software Engineer -Acme Corp -San Francisco, CA - -View job: https://www.linkedin.com/comm/jobs/view/1111111111/?refId=abc&trackingId=xyz - --------------------------------------------------- -Senior Developer -Widget Inc -Remote - -View job: https://www.linkedin.com/comm/jobs/view/2222222222/?refId=def -""" - -LINKEDIN_BODY_EMPTY = "No jobs matched your alert this week." - -LINKEDIN_BODY_NO_URL = """\ -Software Engineer -Acme Corp -San Francisco, CA - --------------------------------------------------- -""" - - -def test_dispatcher_linkedin_sender(): - cards = parse_digest("LinkedIn ", LINKEDIN_BODY) - assert cards is not None - assert len(cards) == 2 - - -def test_dispatcher_unknown_sender_returns_none(): - result = parse_digest("noreply@randomboard.com", LINKEDIN_BODY) - assert result is None - - -def test_dispatcher_case_insensitive_sender(): - cards = parse_digest("JOBALERTS@LINKEDIN.COM", LINKEDIN_BODY) - assert cards is not None - - -def test_parse_linkedin_returns_correct_fields(): - cards = parse_linkedin(LINKEDIN_BODY) - assert cards[0]["title"] == "Software Engineer" - assert cards[0]["company"] == "Acme Corp" - assert cards[0]["location"] == "San Francisco, CA" - assert cards[0]["source"] == "linkedin" - - -def test_parse_linkedin_url_canonicalized(): - """Tracking params stripped; canonical jobs/view// form.""" - cards = parse_linkedin(LINKEDIN_BODY) - assert cards[0]["url"] == "https://www.linkedin.com/jobs/view/1111111111/" - assert "refId" not in cards[0]["url"] - assert "trackingId" not in cards[0]["url"] - - -def test_parse_linkedin_empty_body_returns_empty_list(): - assert parse_linkedin(LINKEDIN_BODY_EMPTY) == [] - - -def test_parse_linkedin_block_without_url_skipped(): - cards = parse_linkedin(LINKEDIN_BODY_NO_URL) - assert cards == [] -``` - -**Step 2: Run tests to verify they fail** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py -v -``` -Expected: `ImportError: cannot import name 'parse_digest'` - ---- - -**Step 3: Write `digest_parsers.py`** - -Create `peregrine/scripts/digest_parsers.py`: - -```python -"""Digest email parser registry for Peregrine. - -Each parser extracts job listings from a known digest sender's plain-text body. -New parsers are added by decorating with @_register(sender_substring, source_name). - -Usage: - from scripts.digest_parsers import parse_digest - - cards = parse_digest(from_addr, body) - # None → unknown sender (fall through to LLM path) - # [] → known sender, nothing extractable - # [...] → list of {title, company, location, url, source} dicts -""" -from __future__ import annotations - -import re -from typing import Callable - -# ── Registry ────────────────────────────────────────────────────────────────── - -# Maps sender substring (lowercased) → (source_name, parse_fn) -DIGEST_PARSERS: dict[str, tuple[str, Callable[[str], list[dict]]]] = {} - - -def _register(sender: str, source: str): - """Decorator to register a parser for a given sender substring.""" - def decorator(fn: Callable[[str], list[dict]]): - DIGEST_PARSERS[sender.lower()] = (source, fn) - return fn - return decorator - - -def parse_digest(from_addr: str, body: str) -> list[dict] | None: - """Dispatch to the appropriate parser based on sender address. - - Returns: - None — no parser matched (caller should use LLM fallback) - [] — known sender, no extractable jobs - [dict, ...] — one dict per job card with keys: - title, company, location, url, source - """ - addr = from_addr.lower() - for sender, (source, parse_fn) in DIGEST_PARSERS.items(): - if sender in addr: - return parse_fn(body) - return None - - -# ── Shared helpers ───────────────────────────────────────────────────────────── - -_LINKEDIN_SKIP_PHRASES = { - "promoted", "easily apply", "apply now", "job alert", - "unsubscribe", "linkedin corporation", -} - - -# ── LinkedIn Job Alert ───────────────────────────────────────────────────────── - -@_register("jobalerts@linkedin.com", "linkedin") -def parse_linkedin(body: str) -> list[dict]: - """Parse LinkedIn Job Alert digest email body. - - Blocks are separated by lines of 10+ dashes. Each block contains: - Line 0: job title - Line 1: company - Line 2: location (optional) - 'View job: ' → canonicalized to /jobs/view// - """ - jobs = [] - blocks = re.split(r"\n\s*-{10,}\s*\n", body) - for block in blocks: - lines = [ln.strip() for ln in block.strip().splitlines() if ln.strip()] - - url = None - for line in lines: - m = re.search(r"View job:\s*(https?://\S+)", line, re.IGNORECASE) - if m: - raw_url = m.group(1) - job_id_m = re.search(r"/jobs/view/(\d+)", raw_url) - if job_id_m: - url = f"https://www.linkedin.com/jobs/view/{job_id_m.group(1)}/" - break - if not url: - continue - - content = [ - ln for ln in lines - if not any(p in ln.lower() for p in _LINKEDIN_SKIP_PHRASES) - and not ln.lower().startswith("view job:") - and not ln.startswith("http") - ] - if len(content) < 2: - continue - - jobs.append({ - "title": content[0], - "company": content[1], - "location": content[2] if len(content) > 2 else "", - "url": url, - "source": "linkedin", - }) - return jobs - - -# ── Adzuna Job Alert ─────────────────────────────────────────────────────────── - -@_register("noreply@adzuna.com", "adzuna") -def parse_adzuna(body: str) -> list[dict]: - """Parse Adzuna job alert digest email body. - - TODO: implement after reviewing samples in avocet/data/digest_samples.jsonl - See Task 3 in docs/plans/2026-03-05-digest-parsers-plan.md - """ - return [] - - -# ── The Ladders Job Alert ────────────────────────────────────────────────────── - -@_register("noreply@theladders.com", "theladders") -def parse_theladders(body: str) -> list[dict]: - """Parse The Ladders job alert digest email body. - - TODO: implement after reviewing samples in avocet/data/digest_samples.jsonl - See Task 4 in docs/plans/2026-03-05-digest-parsers-plan.md - """ - return [] -``` - -**Step 4: Run tests to verify they pass** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py -v -``` -Expected: all 8 tests PASS - -**Step 5: Commit** - -```bash -git add scripts/digest_parsers.py tests/test_digest_parsers.py -git commit -m "feat: digest parser registry + LinkedIn parser (moved from imap_sync)" -``` - ---- - -### Task 2: Fetch digest samples from IMAP - -**Files:** -- Create: `avocet/scripts/fetch_digest_samples.py` - -**Context:** -We need real Adzuna and Ladders email bodies to write parsers against. This one-off script -searches the configured IMAP account by sender domain and writes results to -`data/digest_samples.jsonl`. Run it once; the output file feeds Tasks 3 and 4. - ---- - -**Step 1: Create the fetch script** - -Create `avocet/scripts/fetch_digest_samples.py`: - -```python -#!/usr/bin/env python3 -"""Fetch digest email samples from IMAP into data/digest_samples.jsonl. - -Searches for emails from known digest sender domains, deduplicates against -any existing samples, and appends new ones. - -Usage: - conda run -n job-seeker python scripts/fetch_digest_samples.py - -Reads config/label_tool.yaml for IMAP credentials (first account used). -""" -from __future__ import annotations - -import imaplib -import json -import sys -from pathlib import Path - -import yaml - -ROOT = Path(__file__).parent.parent -CONFIG = ROOT / "config" / "label_tool.yaml" -OUTPUT = ROOT / "data" / "digest_samples.jsonl" - -# Sender domains to search — add new ones here as needed -DIGEST_SENDERS = [ - "adzuna.com", - "theladders.com", - "jobalerts@linkedin.com", -] - -# Import shared helpers from avocet -sys.path.insert(0, str(ROOT)) -from app.imap_fetch import _decode_str, _extract_body, entry_key # noqa: E402 - - -def _load_existing_keys() -> set[str]: - if not OUTPUT.exists(): - return set() - keys = set() - for line in OUTPUT.read_text().splitlines(): - try: - keys.add(entry_key(json.loads(line))) - except Exception: - pass - return keys - - -def main() -> None: - cfg = yaml.safe_load(CONFIG.read_text()) - accounts = cfg.get("accounts", []) - if not accounts: - print("No accounts configured in config/label_tool.yaml") - sys.exit(1) - - acc = accounts[0] - host = acc.get("host", "imap.gmail.com") - port = int(acc.get("port", 993)) - use_ssl = acc.get("use_ssl", True) - username = acc["username"] - password = acc["password"] - folder = acc.get("folder", "INBOX") - days_back = int(acc.get("days_back", 90)) - - from datetime import datetime, timedelta - import email as _email_lib - - since = (datetime.now() - timedelta(days=days_back)).strftime("%d-%b-%Y") - - conn = (imaplib.IMAP4_SSL if use_ssl else imaplib.IMAP4)(host, port) - conn.login(username, password) - conn.select(folder, readonly=True) - - known_keys = _load_existing_keys() - found: list[dict] = [] - seen_uids: dict[bytes, None] = {} - - for sender in DIGEST_SENDERS: - try: - _, data = conn.search(None, f'(FROM "{sender}" SINCE "{since}")') - for uid in (data[0] or b"").split(): - seen_uids[uid] = None - except Exception as exc: - print(f" search error for {sender!r}: {exc}") - - print(f"Found {len(seen_uids)} candidate UIDs across {len(DIGEST_SENDERS)} senders") - - for uid in seen_uids: - try: - _, raw_data = conn.fetch(uid, "(RFC822)") - if not raw_data or not raw_data[0]: - continue - msg = _email_lib.message_from_bytes(raw_data[0][1]) - entry = { - "subject": _decode_str(msg.get("Subject", "")), - "body": _extract_body(msg)[:2000], # larger cap for parser dev - "from_addr": _decode_str(msg.get("From", "")), - "date": _decode_str(msg.get("Date", "")), - "account": acc.get("name", username), - } - k = entry_key(entry) - if k not in known_keys: - known_keys.add(k) - found.append(entry) - except Exception as exc: - print(f" fetch error uid {uid}: {exc}") - - conn.logout() - - if not found: - print("No new digest samples found.") - return - - OUTPUT.parent.mkdir(exist_ok=True) - with OUTPUT.open("a", encoding="utf-8") as f: - for entry in found: - f.write(json.dumps(entry) + "\n") - - print(f"Wrote {len(found)} new samples to {OUTPUT}") - - -if __name__ == "__main__": - main() -``` - -**Step 2: Run the fetch script** - -``` -cd /Library/Development/CircuitForge/avocet -conda run -n job-seeker python scripts/fetch_digest_samples.py -``` - -Expected output: `Wrote N new samples to data/digest_samples.jsonl` - -**Step 3: Inspect the samples** - -``` -# View first few entries — look at from_addr and body for Adzuna and Ladders format -conda run -n job-seeker python -c " -import json -from pathlib import Path -for line in Path('data/digest_samples.jsonl').read_text().splitlines()[:10]: - e = json.loads(line) - print('FROM:', e['from_addr']) - print('SUBJECT:', e['subject']) - print('BODY[:500]:', e['body'][:500]) - print('---') -" -``` - -Note down: -- The exact sender addresses for Adzuna and Ladders (update `DIGEST_PARSERS` in `digest_parsers.py` if different from `noreply@adzuna.com` / `noreply@theladders.com`) -- The structure of each job block in the body (separator lines, field order, URL format) - -**Step 4: Commit** - -```bash -cd /Library/Development/CircuitForge/avocet -git add scripts/fetch_digest_samples.py -git commit -m "feat: fetch_digest_samples script for building new parsers" -``` - ---- - -### Task 3: Build and test Adzuna parser - -**Files:** -- Modify: `peregrine/scripts/digest_parsers.py` — implement `parse_adzuna` -- Modify: `peregrine/tests/test_digest_parsers.py` — add Adzuna fixtures + tests - -**Context:** -After running Task 2, you have real Adzuna email bodies in `avocet/data/digest_samples.jsonl`. -Inspect them (see Task 2 Step 3), identify the structure, then write the test fixture from -a real sample before implementing the parser. - ---- - -**Step 1: Write a failing Adzuna test** - -Inspect a real Adzuna sample from `data/digest_samples.jsonl` and identify: -- How job blocks are separated (blank lines? dashes? headers?) -- Field order (title first? company first?) -- Where the job URL appears and what format it uses -- Any noise lines to filter (unsubscribe, promo text, etc.) - -Add to `peregrine/tests/test_digest_parsers.py`: - -```python -from scripts.digest_parsers import parse_adzuna - -# Replace ADZUNA_BODY with a real excerpt from avocet/data/digest_samples.jsonl -# Copy 2-3 job blocks verbatim; replace real company names with "Test Co" etc. if desired -ADZUNA_BODY = """ - -""" - -def test_dispatcher_adzuna_sender(): - # Update sender string if real sender differs from noreply@adzuna.com - cards = parse_digest("noreply@adzuna.com", ADZUNA_BODY) - assert cards is not None - assert len(cards) >= 1 - -def test_parse_adzuna_fields(): - cards = parse_adzuna(ADZUNA_BODY) - assert cards[0]["title"] # non-empty - assert cards[0]["company"] # non-empty - assert cards[0]["url"].startswith("http") - assert cards[0]["source"] == "adzuna" - -def test_parse_adzuna_url_no_tracking(): - """Adzuna URLs often contain tracking params — strip them.""" - cards = parse_adzuna(ADZUNA_BODY) - # Adjust assertion to match actual URL format once you've seen real samples - for card in cards: - assert "utm_" not in card["url"] - -def test_parse_adzuna_empty_body(): - assert parse_adzuna("No jobs this week.") == [] -``` - -**Step 2: Run tests to verify they fail** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py::test_parse_adzuna_fields -v -``` -Expected: FAIL (stub returns `[]`) - -**Step 3: Implement `parse_adzuna` in `digest_parsers.py`** - -Replace the stub body of `parse_adzuna` based on the actual email structure you observed. -Pattern to follow (adapt field positions to match Adzuna's actual format): - -```python -@_register("noreply@adzuna.com", "adzuna") # update sender if needed -def parse_adzuna(body: str) -> list[dict]: - jobs = [] - # Split on whatever delimiter Adzuna uses between blocks - # e.g.: blocks = re.split(r"\n\s*\n{2,}", body) # double blank line - # For each block, extract title, company, location, url - # Strip tracking params from URL: re.sub(r"\?.*", "", url) or parse with urllib - return jobs -``` - -If Adzuna sender differs from `noreply@adzuna.com`, update the `@_register` decorator -**and** the `DIGEST_PARSERS` key in the registry (they're set by the decorator — just change -the decorator argument). - -**Step 4: Run all digest tests** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py -v -``` -Expected: all tests PASS - -**Step 5: Commit** - -```bash -cd /Library/Development/CircuitForge/peregrine -git add scripts/digest_parsers.py tests/test_digest_parsers.py -git commit -m "feat: Adzuna digest email parser" -``` - ---- - -### Task 4: Build and test The Ladders parser - -**Files:** -- Modify: `peregrine/scripts/digest_parsers.py` — implement `parse_theladders` -- Modify: `peregrine/tests/test_digest_parsers.py` — add Ladders fixtures + tests - -**Context:** -Same approach as Task 3. The Ladders already has a web scraper in -`scripts/custom_boards/theladders.py` — check it for URL patterns that may apply here. - ---- - -**Step 1: Write failing Ladders tests** - -Inspect a real Ladders sample from `avocet/data/digest_samples.jsonl`. Add to test file: - -```python -from scripts.digest_parsers import parse_theladders - -# Replace with real Ladders body excerpt -LADDERS_BODY = """ - -""" - -def test_dispatcher_ladders_sender(): - cards = parse_digest("noreply@theladders.com", LADDERS_BODY) - assert cards is not None - assert len(cards) >= 1 - -def test_parse_theladders_fields(): - cards = parse_theladders(LADDERS_BODY) - assert cards[0]["title"] - assert cards[0]["company"] - assert cards[0]["url"].startswith("http") - assert cards[0]["source"] == "theladders" - -def test_parse_theladders_empty_body(): - assert parse_theladders("No new jobs.") == [] -``` - -**Step 2: Run tests to verify they fail** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py::test_parse_theladders_fields -v -``` -Expected: FAIL - -**Step 3: Implement `parse_theladders`** - -Replace the stub. The Ladders URLs often use redirect wrappers — canonicalize to the -`theladders.com/job/` form if possible, otherwise just strip tracking params. - -**Step 4: Run all digest tests** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_parsers.py -v -``` -Expected: all tests PASS - -**Step 5: Commit** - -```bash -git add scripts/digest_parsers.py tests/test_digest_parsers.py -git commit -m "feat: The Ladders digest email parser" -``` - ---- - -### Task 5: Update `imap_sync.py` to use the dispatcher - -**Files:** -- Modify: `peregrine/scripts/imap_sync.py` - -**Context:** -The LinkedIn-specific block in `_scan_unmatched_leads()` (search for -`_LINKEDIN_ALERT_SENDER`) gets replaced with a generic `parse_digest()` call. -The existing behavior is preserved — only the dispatch mechanism changes. - ---- - -**Step 1: Add the import** - -At the top of `imap_sync.py`, alongside other local imports, add: - -```python -from scripts.digest_parsers import parse_digest -``` - -**Step 2: Find the LinkedIn-specific block** - -Search for `_LINKEDIN_ALERT_SENDER` in `imap_sync.py`. The block looks like: - -```python -if _LINKEDIN_ALERT_SENDER in parsed["from_addr"].lower(): - cards = parse_linkedin_alert(parsed["body"]) - for card in cards: - ... - known_message_ids.add(mid) - continue -``` - -**Step 3: Replace with the generic dispatcher** - -```python -# ── Digest email — dispatch to parser registry ──────────────────────── -cards = parse_digest(parsed["from_addr"], parsed["body"]) -if cards is not None: - for card in cards: - if card["url"] in existing_urls: - continue - job_id = insert_job(db_path, { - "title": card["title"], - "company": card["company"], - "url": card["url"], - "source": card["source"], - "location": card["location"], - "is_remote": 0, - "salary": "", - "description": "", - "date_found": datetime.now().isoformat()[:10], - }) - if job_id: - submit_task(db_path, "scrape_url", job_id) - existing_urls.add(card["url"]) - new_leads += 1 - print(f"[imap] digest ({card['source']}) → {card['company']} — {card['title']}") - known_message_ids.add(mid) - continue -``` - -**Step 4: Remove the now-unused `parse_linkedin_alert` import/definition** - -`parse_linkedin_alert` was defined in `imap_sync.py`. It's now `parse_linkedin` in -`digest_parsers.py`. Delete the old function from `imap_sync.py`. Also remove -`_LINKEDIN_ALERT_SENDER` constant if it's no longer referenced. - -**Step 5: Run the full test suite** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` -Expected: all existing tests still pass; no regressions - -**Step 6: Commit** - -```bash -git add scripts/imap_sync.py -git commit -m "refactor: imap_sync uses digest_parsers dispatcher; remove inline LinkedIn parser" -``` - ---- - -### Task 6: Avocet digest bucket - -**Files:** -- Modify: `avocet/app/label_tool.py` -- Modify: `avocet/app/api.py` -- Create: `avocet/tests/test_digest_bucket.py` -- Create: `avocet/data/digest_samples.jsonl.example` - -**Context:** -When either label path (`_do_label` in the Streamlit UI or `POST /api/label` in the FastAPI -app) assigns the `digest` label, the full email record is appended to -`data/digest_samples.jsonl`. This is the sample corpus for building future parsers. - ---- - -**Step 1: Write failing tests** - -Create `avocet/tests/test_digest_bucket.py`: - -```python -"""Tests for digest sample bucket write behavior.""" -import json -import pytest -from pathlib import Path -from unittest.mock import patch, MagicMock - - -# ── Helpers ─────────────────────────────────────────────────────────────────── - -def _read_bucket(tmp_path: Path) -> list[dict]: - bucket = tmp_path / "data" / "digest_samples.jsonl" - if not bucket.exists(): - return [] - return [json.loads(line) for line in bucket.read_text().splitlines() if line.strip()] - - -SAMPLE_ENTRY = { - "subject": "10 new jobs for you", - "body": "Software Engineer\nAcme Corp\nRemote\nView job: https://example.com/123", - "from_addr": "noreply@adzuna.com", - "date": "Mon, 03 Mar 2026 09:00:00 +0000", - "account": "test@example.com", -} - - -# ── api.py bucket tests ─────────────────────────────────────────────────────── - -def test_api_digest_label_writes_to_bucket(tmp_path): - from app.api import _append_digest_sample - data_dir = tmp_path / "data" - _append_digest_sample(SAMPLE_ENTRY, data_dir=data_dir) - rows = _read_bucket(tmp_path) - assert len(rows) == 1 - assert rows[0]["from_addr"] == "noreply@adzuna.com" - - -def test_api_non_digest_label_does_not_write(tmp_path): - from app.api import _append_digest_sample - data_dir = tmp_path / "data" - # _append_digest_sample should only be called for digest; confirm it writes when called - # Confirm that callers gate on label == "digest" — tested via integration below - _append_digest_sample(SAMPLE_ENTRY, data_dir=data_dir) - rows = _read_bucket(tmp_path) - assert len(rows) == 1 # called directly, always writes - - -def test_api_digest_creates_data_dir(tmp_path): - from app.api import _append_digest_sample - data_dir = tmp_path / "nonexistent" / "data" - assert not data_dir.exists() - _append_digest_sample(SAMPLE_ENTRY, data_dir=data_dir) - assert data_dir.exists() - - -def test_api_digest_appends_multiple(tmp_path): - from app.api import _append_digest_sample - data_dir = tmp_path / "data" - _append_digest_sample(SAMPLE_ENTRY, data_dir=data_dir) - _append_digest_sample({**SAMPLE_ENTRY, "subject": "5 more jobs"}, data_dir=data_dir) - rows = _read_bucket(tmp_path) - assert len(rows) == 2 -``` - -**Step 2: Run tests to verify they fail** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/test_digest_bucket.py -v -``` -Expected: `ImportError: cannot import name '_append_digest_sample'` - ---- - -**Step 3: Add `_append_digest_sample` to `api.py`** - -In `avocet/app/api.py`, add this helper (near the top, after the imports and `_DATA_DIR` -constant): - -```python -_DIGEST_SAMPLES_FILE = _DATA_DIR / "digest_samples.jsonl" - - -def _append_digest_sample(entry: dict, data_dir: Path | None = None) -> None: - """Append a digest-labeled email to the sample corpus.""" - target_dir = data_dir if data_dir is not None else _DATA_DIR - target_dir.mkdir(parents=True, exist_ok=True) - bucket = target_dir / "digest_samples.jsonl" - record = { - "subject": entry.get("subject", ""), - "body": entry.get("body", ""), - "from_addr": entry.get("from_addr", entry.get("from", "")), - "date": entry.get("date", ""), - "account": entry.get("account", entry.get("source", "")), - } - with bucket.open("a", encoding="utf-8") as f: - f.write(json.dumps(record) + "\n") -``` - -Then in `post_label()` (around line 127, after `_append_jsonl(_score_file(), record)`): - -```python - if req.label == "digest": - _append_digest_sample(match) -``` - -**Step 4: Add the same write to `label_tool.py`** - -In `avocet/app/label_tool.py`, add a module-level constant after `_SCORE_FILE`: - -```python -_DIGEST_SAMPLES_FILE = _ROOT / "data" / "digest_samples.jsonl" -``` - -In `_do_label()` (around line 728, after `_append_jsonl(_SCORE_FILE, row)`): - -```python - if label == "digest": - _append_jsonl( - _DIGEST_SAMPLES_FILE, - { - "subject": entry.get("subject", ""), - "body": (entry.get("body", ""))[:2000], - "from_addr": entry.get("from_addr", ""), - "date": entry.get("date", ""), - "account": entry.get("account", ""), - }, - ) -``` - -(`_append_jsonl` already exists in label_tool.py at line ~396 — reuse it.) - -**Step 5: Create the example file** - -Create `avocet/data/digest_samples.jsonl.example`: - -```json -{"subject": "10 new Software Engineer jobs for you", "body": "Software Engineer\nAcme Corp\nSan Francisco, CA\n\nView job: https://www.linkedin.com/jobs/view/1234567890/\n", "from_addr": "LinkedIn ", "date": "Mon, 03 Mar 2026 09:00:00 +0000", "account": "example@gmail.com"} -``` - -**Step 6: Update `.gitignore` in avocet** - -Verify `data/digest_samples.jsonl` is gitignored. Open `avocet/.gitignore` — it should -already have `data/*.jsonl`. If not, add: - -``` -data/digest_samples.jsonl -``` - -**Step 7: Run all avocet tests** - -``` -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v -``` -Expected: all tests PASS - -**Step 8: Commit** - -```bash -cd /Library/Development/CircuitForge/avocet -git add app/api.py app/label_tool.py tests/test_digest_bucket.py data/digest_samples.jsonl.example -git commit -m "feat: digest sample bucket — write digest-labeled emails to digest_samples.jsonl" -``` - ---- - -## Summary - -| Task | Repo | Commit message | -|------|------|----------------| -| 1 | peregrine | `feat: digest parser registry + LinkedIn parser (moved from imap_sync)` | -| 2 | avocet | `feat: fetch_digest_samples script for building new parsers` | -| 3 | peregrine | `feat: Adzuna digest email parser` | -| 4 | peregrine | `feat: The Ladders digest email parser` | -| 5 | peregrine | `refactor: imap_sync uses digest_parsers dispatcher; remove inline LinkedIn parser` | -| 6 | avocet | `feat: digest sample bucket — write digest-labeled emails to digest_samples.jsonl` | - -Tasks 1, 2, and 6 are independent and can be done in any order. -Tasks 3 and 4 depend on Task 2 (samples needed before implementing parsers). -Task 5 depends on Tasks 1, 3, and 4 (all parsers should be ready before switching imap_sync). diff --git a/docs/plans/2026-03-07-circuitforge-hooks-design.md b/docs/plans/2026-03-07-circuitforge-hooks-design.md deleted file mode 100644 index 1bafe37..0000000 --- a/docs/plans/2026-03-07-circuitforge-hooks-design.md +++ /dev/null @@ -1,161 +0,0 @@ -# CircuitForge Hooks — Secret & PII Scanning Design - -**Date:** 2026-03-07 -**Scope:** All CircuitForge repos (Peregrine first; others on public release) -**Status:** Approved, ready for implementation - -## Problem - -A live Forgejo API token was committed in `docs/plans/2026-03-03-feedback-button-plan.md` -and required emergency history scrubbing via `git-filter-repo`. Root causes: - -1. `core.hooksPath` was never configured — the existing `.githooks/pre-commit` ran on zero commits -2. The token format (`FORGEJO_API_TOKEN=`) matched none of the hook's three regexes -3. No pre-push safety net existed - -## Solution - -Centralised hook repo (`circuitforge-hooks`) shared across all products. -Each repo activates it with one command. The heavy lifting is delegated to -`gitleaks` — an actively-maintained binary with 150+ built-in secret patterns, -native Forgejo/Gitea token detection, and a clean allowlist system. - -## Repository Structure - -``` -/Library/Development/CircuitForge/circuitforge-hooks/ -├── hooks/ -│ ├── pre-commit # gitleaks --staged scan (fast, every commit) -│ ├── commit-msg # conventional commits enforcement -│ └── pre-push # gitleaks full-branch scan (safety net) -├── gitleaks.toml # shared base config -├── install.sh # wires core.hooksPath in the calling repo -├── tests/ -│ └── test_hooks.sh # migrated + extended from Peregrine -└── README.md -``` - -Forgejo remote: `git.opensourcesolarpunk.com/pyr0ball/circuitforge-hooks` - -## Hook Behaviour - -### pre-commit -- Runs `gitleaks protect --staged` — scans only the staged diff -- Sub-second on typical commits -- Blocks commit and prints redacted match on failure -- Merges per-repo `.gitleaks.toml` allowlist if present - -### pre-push -- Runs `gitleaks git` — scans full branch history not yet on remote -- Catches anything committed with `--no-verify` or before hooks were wired -- Same config resolution as pre-commit - -### commit-msg -- Enforces conventional commits format (`type(scope): subject`) -- Migrated unchanged from `peregrine/.githooks/commit-msg` - -## gitleaks Config - -### Shared base (`circuitforge-hooks/gitleaks.toml`) - -```toml -title = "CircuitForge secret + PII scanner" - -[extend] -useDefault = true # inherit all 150+ built-in rules - -[[rules]] -id = "cf-generic-env-token" -description = "Generic KEY= in env-style assignment" -regex = '''(?i)(token|secret|key|password|passwd|pwd|api_key)\s*[=:]\s*['\"]?[A-Za-z0-9\-_]{20,}['\"]?''' -[rules.allowlist] -regexes = ['api_key:\s*ollama', 'api_key:\s*any'] - -[[rules]] -id = "cf-phone-number" -description = "US phone number in source or config" -regex = '''\b(\+1[\s\-.]?)?\(?\d{3}\)?[\s\-.]?\d{3}[\s\-.]?\d{4}\b''' -[rules.allowlist] -regexes = ['555-\d{4}', '555\.\d{4}', '5550', '1234567890', '0000000000'] - -[[rules]] -id = "cf-personal-email" -description = "Personal email address in source/config (not .example files)" -regex = '''[a-zA-Z0-9._%+\-]+@(gmail|yahoo|icloud|hotmail|outlook|proton)\.(com|me)''' -[rules.allowlist] -paths = ['.*\.example$', '.*test.*', '.*docs/.*'] - -[allowlist] -description = "CircuitForge global allowlist" -paths = [ - '.*\.example$', - 'docs/reference/.*', - 'gitleaks\.toml$', -] -regexes = [ - 'sk-abcdefghijklmnopqrstuvwxyz', - 'your-forgejo-api-token-here', -] -``` - -### Per-repo override (e.g. `peregrine/.gitleaks.toml`) - -```toml -[extend] -path = "/Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml" - -[allowlist] -regexes = [ - '\d{10}\.html', # Craigslist listing IDs (10-digit, look like phone numbers) -] -``` - -## Activation Per Repo - -Each repo's `setup.sh` or `manage.sh` calls: - -```bash -bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh -``` - -`install.sh` does exactly one thing: - -```bash -git config core.hooksPath /Library/Development/CircuitForge/circuitforge-hooks/hooks -``` - -For Heimdall live deploys (`/devl//`), the same line goes in the deploy -script / post-receive hook. - -## Migration from Peregrine - -- `peregrine/.githooks/pre-commit` → replaced by gitleaks wrapper -- `peregrine/.githooks/commit-msg` → copied verbatim to hooks repo -- `peregrine/tests/test_hooks.sh` → migrated and extended in hooks repo -- `peregrine/.githooks/` directory → kept temporarily, then removed after cutover - -## Rollout Order - -1. `circuitforge-hooks` repo — create, implement, test -2. `peregrine` — activate (highest priority, already public) -3. `circuitforge-license` (heimdall) — activate before any public release -4. All subsequent repos — activate as part of their public-release checklist - -## Testing - -`tests/test_hooks.sh` covers: - -- Staged file with live-format token → blocked -- Staged file with phone number → blocked -- Staged file with personal email in source → blocked -- `.example` file with placeholders → allowed -- Craigslist URL with 10-digit ID → allowed (Peregrine allowlist) -- Valid conventional commit message → accepted -- Non-conventional commit message → rejected - -## What This Does Not Cover - -- Scanning existing history on new repos (run `gitleaks git` manually before - making any repo public — add to the public-release checklist) -- CI/server-side enforcement (future: Forgejo Actions job on push to main) -- Binary files or encrypted secrets at rest diff --git a/docs/plans/2026-03-07-circuitforge-hooks-plan.md b/docs/plans/2026-03-07-circuitforge-hooks-plan.md deleted file mode 100644 index 81952f7..0000000 --- a/docs/plans/2026-03-07-circuitforge-hooks-plan.md +++ /dev/null @@ -1,705 +0,0 @@ -# CircuitForge Hooks Implementation Plan - -> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. - -**Goal:** Create the `circuitforge-hooks` repo with gitleaks-based secret/PII scanning, activate it in Peregrine, and retire the old hand-rolled `.githooks/pre-commit`. - -**Architecture:** A standalone git repo holds three hook scripts (pre-commit, commit-msg, pre-push) and a shared `gitleaks.toml`. Each product repo activates it with `git config core.hooksPath`. Per-repo `.gitleaks.toml` files extend the base config with repo-specific allowlists. - -**Tech Stack:** gitleaks (Go binary, apt install), bash, TOML config - ---- - -### Task 1: Install gitleaks - -**Files:** -- None — binary install only - -**Step 1: Install gitleaks** - -```bash -sudo apt-get install -y gitleaks -``` - -If not in apt (older Ubuntu), use the GitHub release: -```bash -GITLEAKS_VERSION=$(curl -s https://api.github.com/repos/gitleaks/gitleaks/releases/latest | python3 -c "import sys,json; print(json.load(sys.stdin)['tag_name'])") -curl -sSfL "https://github.com/gitleaks/gitleaks/releases/download/${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION#v}_linux_x64.tar.gz" | sudo tar -xz -C /usr/local/bin gitleaks -``` - -**Step 2: Verify** - -```bash -gitleaks version -``` -Expected: prints version string e.g. `v8.x.x` - ---- - -### Task 2: Create repo and write gitleaks.toml - -**Files:** -- Create: `/Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml` - -**Step 1: Scaffold repo** - -```bash -mkdir -p /Library/Development/CircuitForge/circuitforge-hooks/hooks -mkdir -p /Library/Development/CircuitForge/circuitforge-hooks/tests -cd /Library/Development/CircuitForge/circuitforge-hooks -git init -``` - -**Step 2: Write gitleaks.toml** - -Create `/Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml`: - -```toml -title = "CircuitForge secret + PII scanner" - -[extend] -useDefault = true # inherit all 150+ built-in gitleaks rules - -# ── CircuitForge-specific secret patterns ──────────────────────────────────── - -[[rules]] -id = "cf-generic-env-token" -description = "Generic KEY= in env-style assignment — catches FORGEJO_API_TOKEN=hex etc." -regex = '''(?i)(token|secret|key|password|passwd|pwd|api_key)\s*[=:]\s*['"]?[A-Za-z0-9\-_]{20,}['"]?''' -[rules.allowlist] -regexes = [ - 'api_key:\s*ollama', - 'api_key:\s*any', - 'your-[a-z\-]+-here', - 'replace-with-', - 'xxxx', -] - -# ── PII patterns ────────────────────────────────────────────────────────────── - -[[rules]] -id = "cf-phone-number" -description = "US phone number committed in source or config" -regex = '''\b(\+1[\s\-.]?)?\(?\d{3}\)?[\s\-.]?\d{3}[\s\-.]?\d{4}\b''' -[rules.allowlist] -regexes = [ - '555-\d{4}', - '555\.\d{4}', - '5550\d{4}', - '^1234567890$', - '0000000000', - '1111111111', - '2222222222', - '9999999999', -] - -[[rules]] -id = "cf-personal-email" -description = "Personal webmail address committed in source or config (not .example files)" -regex = '''[a-zA-Z0-9._%+\-]+@(gmail|yahoo|icloud|hotmail|outlook|proton)\.(com|me)''' -[rules.allowlist] -paths = [ - '.*\.example$', - '.*test.*', - '.*docs/.*', - '.*\.md$', -] - -# ── Global allowlist ────────────────────────────────────────────────────────── - -[allowlist] -description = "CircuitForge global allowlist" -paths = [ - '.*\.example$', - 'docs/reference/.*', - 'gitleaks\.toml$', -] -regexes = [ - 'sk-abcdefghijklmnopqrstuvwxyz', - 'your-forgejo-api-token-here', - 'your-[a-z\-]+-here', -] -``` - -**Step 3: Smoke-test config syntax** - -```bash -cd /Library/Development/CircuitForge/circuitforge-hooks -gitleaks detect --config gitleaks.toml --no-git --source . 2>&1 | head -5 -``` -Expected: no "invalid config" errors. (May report findings in the config itself — that's fine.) - -**Step 4: Commit** - -```bash -cd /Library/Development/CircuitForge/circuitforge-hooks -git add gitleaks.toml -git commit -m "feat: add shared gitleaks config with CF secret + PII rules" -``` - ---- - -### Task 3: Write hook scripts - -**Files:** -- Create: `hooks/pre-commit` -- Create: `hooks/commit-msg` -- Create: `hooks/pre-push` - -**Step 1: Write hooks/pre-commit** - -```bash -#!/usr/bin/env bash -# pre-commit — scan staged diff for secrets + PII via gitleaks -set -euo pipefail - -HOOKS_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -BASE_CONFIG="$HOOKS_REPO/gitleaks.toml" -REPO_ROOT="$(git rev-parse --show-toplevel)" -REPO_CONFIG="$REPO_ROOT/.gitleaks.toml" - -if ! command -v gitleaks &>/dev/null; then - echo "ERROR: gitleaks not found. Install with: sudo apt-get install gitleaks" - echo " or: https://github.com/gitleaks/gitleaks#installing" - exit 1 -fi - -CONFIG_ARG="--config=$BASE_CONFIG" -[[ -f "$REPO_CONFIG" ]] && CONFIG_ARG="--config=$REPO_CONFIG" - -if ! gitleaks protect --staged $CONFIG_ARG --redact 2>&1; then - echo "" - echo "Commit blocked: secrets or PII detected in staged changes." - echo "Review above, remove the sensitive value, then re-stage and retry." - echo "If this is a false positive, add an allowlist entry to .gitleaks.toml" - exit 1 -fi -``` - -**Step 2: Write hooks/commit-msg** - -Copy verbatim from Peregrine: - -```bash -#!/usr/bin/env bash -# commit-msg — enforces conventional commit format -set -euo pipefail - -RED='\033[0;31m'; YELLOW='\033[1;33m'; NC='\033[0m' - -VALID_TYPES="feat|fix|docs|chore|test|refactor|perf|ci|build|security" -MSG_FILE="$1" -MSG=$(head -1 "$MSG_FILE") - -if [[ -z "${MSG// }" ]]; then - echo -e "${RED}Commit rejected:${NC} Commit message is empty." - exit 1 -fi - -if ! echo "$MSG" | grep -qE "^($VALID_TYPES)(\(.+\))?: .+"; then - echo -e "${RED}Commit rejected:${NC} Message does not follow conventional commit format." - echo "" - echo -e " Required: ${YELLOW}type: description${NC} or ${YELLOW}type(scope): description${NC}" - echo -e " Valid types: ${YELLOW}$VALID_TYPES${NC}" - echo "" - echo -e " Your message: ${YELLOW}$MSG${NC}" - echo "" - echo -e " Examples:" - echo -e " ${YELLOW}feat: add cover letter refinement${NC}" - echo -e " ${YELLOW}fix(wizard): handle missing user.yaml gracefully${NC}" - echo -e " ${YELLOW}security: rotate leaked API token${NC}" - exit 1 -fi -exit 0 -``` - -Note: added `security` to VALID_TYPES vs the Peregrine original. - -**Step 3: Write hooks/pre-push** - -```bash -#!/usr/bin/env bash -# pre-push — scan full branch history not yet on remote -# Safety net: catches anything committed with --no-verify or before hooks were wired -set -euo pipefail - -HOOKS_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" -BASE_CONFIG="$HOOKS_REPO/gitleaks.toml" -REPO_ROOT="$(git rev-parse --show-toplevel)" -REPO_CONFIG="$REPO_ROOT/.gitleaks.toml" - -if ! command -v gitleaks &>/dev/null; then - echo "ERROR: gitleaks not found. Install with: sudo apt-get install gitleaks" - exit 1 -fi - -CONFIG_ARG="--config=$BASE_CONFIG" -[[ -f "$REPO_CONFIG" ]] && CONFIG_ARG="--config=$REPO_CONFIG" - -if ! gitleaks git $CONFIG_ARG --redact 2>&1; then - echo "" - echo "Push blocked: secrets or PII found in branch history." - echo "Use git-filter-repo to scrub, then force-push." - echo "See: https://github.com/newren/git-filter-repo" - exit 1 -fi -``` - -**Step 4: Make hooks executable** - -```bash -chmod +x hooks/pre-commit hooks/commit-msg hooks/pre-push -``` - -**Step 5: Commit** - -```bash -cd /Library/Development/CircuitForge/circuitforge-hooks -git add hooks/ -git commit -m "feat: add pre-commit, commit-msg, and pre-push hook scripts" -``` - ---- - -### Task 4: Write install.sh - -**Files:** -- Create: `install.sh` - -**Step 1: Write install.sh** - -```bash -#!/usr/bin/env bash -# install.sh — wire circuitforge-hooks into the calling git repo -# Usage: bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh -set -euo pipefail - -HOOKS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/hooks" && pwd)" - -if ! git rev-parse --git-dir &>/dev/null; then - echo "ERROR: not inside a git repo. Run from your product repo root." - exit 1 -fi - -git config core.hooksPath "$HOOKS_DIR" -echo "CircuitForge hooks installed." -echo " core.hooksPath → $HOOKS_DIR" -echo "" -echo "Verify gitleaks is available: gitleaks version" -``` - -**Step 2: Make executable** - -```bash -chmod +x install.sh -``` - -**Step 3: Commit** - -```bash -git add install.sh -git commit -m "feat: add install.sh for one-command hook activation" -``` - ---- - -### Task 5: Write tests - -**Files:** -- Create: `tests/test_hooks.sh` - -**Step 1: Write tests/test_hooks.sh** - -```bash -#!/usr/bin/env bash -# tests/test_hooks.sh — integration tests for circuitforge-hooks -# Requires: gitleaks installed, bash 4+ -set -euo pipefail - -HOOKS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/hooks" -PASS_COUNT=0 -FAIL_COUNT=0 - -pass() { echo " PASS: $1"; PASS_COUNT=$((PASS_COUNT + 1)); } -fail() { echo " FAIL: $1"; FAIL_COUNT=$((FAIL_COUNT + 1)); } - -# Create a temp git repo for realistic staged-content tests -setup_temp_repo() { - local dir - dir=$(mktemp -d) - git init "$dir" -q - git -C "$dir" config user.email "test@example.com" - git -C "$dir" config user.name "Test" - git -C "$dir" config core.hooksPath "$HOOKS_DIR" - echo "$dir" -} - -run_pre_commit_in() { - local repo="$1" file="$2" content="$3" - echo "$content" > "$repo/$file" - git -C "$repo" add "$file" - bash "$HOOKS_DIR/pre-commit" 2>&1 - echo $? -} - -echo "" -echo "=== pre-commit hook tests ===" - -# Test 1: blocks live-format Forgejo token -echo "Test 1: blocks FORGEJO_API_TOKEN=" -REPO=$(setup_temp_repo) -echo 'FORGEJO_API_TOKEN=4ea4353b88d6388e8fafab9eb36662226f3a06b0' > "$REPO/test.env" -git -C "$REPO" add test.env -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:1"; then pass "blocked FORGEJO_API_TOKEN"; else fail "should have blocked FORGEJO_API_TOKEN"; fi -rm -rf "$REPO" - -# Test 2: blocks OpenAI-style sk- key -echo "Test 2: blocks sk- pattern" -REPO=$(setup_temp_repo) -echo 'api_key = "sk-abcXYZ1234567890abcXYZ1234567890"' > "$REPO/config.py" -git -C "$REPO" add config.py -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:1"; then pass "blocked sk- key"; else fail "should have blocked sk- key"; fi -rm -rf "$REPO" - -# Test 3: blocks US phone number -echo "Test 3: blocks US phone number" -REPO=$(setup_temp_repo) -echo 'phone: "5107643155"' > "$REPO/config.yaml" -git -C "$REPO" add config.yaml -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:1"; then pass "blocked phone number"; else fail "should have blocked phone number"; fi -rm -rf "$REPO" - -# Test 4: blocks personal email in source -echo "Test 4: blocks personal gmail address in .py file" -REPO=$(setup_temp_repo) -echo 'DEFAULT_EMAIL = "someone@gmail.com"' > "$REPO/app.py" -git -C "$REPO" add app.py -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:1"; then pass "blocked personal email"; else fail "should have blocked personal email"; fi -rm -rf "$REPO" - -# Test 5: allows .example file with placeholders -echo "Test 5: allows .example file with placeholder values" -REPO=$(setup_temp_repo) -echo 'FORGEJO_API_TOKEN=your-forgejo-api-token-here' > "$REPO/config.env.example" -git -C "$REPO" add config.env.example -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:0"; then pass "allowed .example placeholder"; else fail "should have allowed .example file"; fi -rm -rf "$REPO" - -# Test 6: allows ollama api_key placeholder -echo "Test 6: allows api_key: ollama (known safe placeholder)" -REPO=$(setup_temp_repo) -printf 'backends:\n - api_key: ollama\n' > "$REPO/llm.yaml" -git -C "$REPO" add llm.yaml -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:0"; then pass "allowed ollama api_key"; else fail "should have allowed ollama api_key"; fi -rm -rf "$REPO" - -# Test 7: allows safe source file -echo "Test 7: allows normal Python import" -REPO=$(setup_temp_repo) -echo 'import streamlit as st' > "$REPO/app.py" -git -C "$REPO" add app.py -RESULT=$(cd "$REPO" && bash "$HOOKS_DIR/pre-commit" 2>&1; echo "EXIT:$?") -if echo "$RESULT" | grep -q "EXIT:0"; then pass "allowed safe file"; else fail "should have allowed safe file"; fi -rm -rf "$REPO" - -echo "" -echo "=== commit-msg hook tests ===" - -tmpfile=$(mktemp) - -echo "Test 8: accepts feat: message" -echo "feat: add gitleaks scanning" > "$tmpfile" -if bash "$HOOKS_DIR/commit-msg" "$tmpfile" &>/dev/null; then pass "accepted feat:"; else fail "rejected valid feat:"; fi - -echo "Test 9: accepts security: message (new type)" -echo "security: rotate leaked API token" > "$tmpfile" -if bash "$HOOKS_DIR/commit-msg" "$tmpfile" &>/dev/null; then pass "accepted security:"; else fail "rejected valid security:"; fi - -echo "Test 10: accepts fix(scope): message" -echo "fix(wizard): handle missing user.yaml" > "$tmpfile" -if bash "$HOOKS_DIR/commit-msg" "$tmpfile" &>/dev/null; then pass "accepted fix(scope):"; else fail "rejected valid fix(scope):"; fi - -echo "Test 11: rejects non-conventional message" -echo "updated the thing" > "$tmpfile" -if bash "$HOOKS_DIR/commit-msg" "$tmpfile" &>/dev/null; then fail "should have rejected"; else pass "rejected non-conventional"; fi - -echo "Test 12: rejects empty message" -echo "" > "$tmpfile" -if bash "$HOOKS_DIR/commit-msg" "$tmpfile" &>/dev/null; then fail "should have rejected empty"; else pass "rejected empty message"; fi - -rm -f "$tmpfile" - -echo "" -echo "=== Results ===" -echo " Passed: $PASS_COUNT" -echo " Failed: $FAIL_COUNT" -[[ $FAIL_COUNT -eq 0 ]] && echo "All tests passed." || { echo "FAILURES detected."; exit 1; } -``` - -**Step 2: Make executable** - -```bash -chmod +x tests/test_hooks.sh -``` - -**Step 3: Run tests (expect failures — hooks not yet fully wired)** - -```bash -cd /Library/Development/CircuitForge/circuitforge-hooks -bash tests/test_hooks.sh -``` - -Expected: Tests 1-4 should PASS (gitleaks catches real secrets), Tests 5-7 may fail if allowlists need tuning — note any failures for the next step. - -**Step 4: Tune allowlists in gitleaks.toml if any false positives** - -If Test 5 (`.example` file) or Test 6 (ollama) fail, add the relevant pattern to the `[allowlist]` or `[rules.allowlist]` sections in `gitleaks.toml` and re-run until all 12 pass. - -**Step 5: Commit** - -```bash -git add tests/ -git commit -m "test: add integration tests for pre-commit and commit-msg hooks" -``` - ---- - -### Task 6: Write README and push to Forgejo - -**Files:** -- Create: `README.md` - -**Step 1: Write README.md** - -```markdown -# circuitforge-hooks - -Centralised git hooks for all CircuitForge repos. - -## What it does - -- **pre-commit** — scans staged changes for secrets and PII via gitleaks -- **commit-msg** — enforces conventional commit format -- **pre-push** — scans full branch history as a safety net before push - -## Install - -From any CircuitForge product repo root: - -```bash -bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh -``` - -On Heimdall live deploys (`/devl//`), add the same line to the deploy script. - -## Per-repo allowlists - -Create `.gitleaks.toml` at the repo root to extend the base config: - -```toml -[extend] -path = "/Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml" - -[allowlist] -regexes = [ - '\d{10}\.html', # example: Craigslist listing IDs -] -``` - -## Testing - -```bash -bash tests/test_hooks.sh -``` - -## Requirements - -- `gitleaks` binary: `sudo apt-get install gitleaks` -- bash 4+ - -## Adding a new rule - -Edit `gitleaks.toml`. Follow the pattern of the existing `[[rules]]` blocks. -Add tests to `tests/test_hooks.sh` covering both the blocked and allowed cases. -``` - -**Step 2: Create Forgejo repo and push** - -```bash -# Create repo on Forgejo -curl -s -X POST "https://git.opensourcesolarpunk.com/api/v1/user/repos" \ - -H "Authorization: token 4ea4353b88d6388e8fafab9eb36662226f3a06b0" \ - -H "Content-Type: application/json" \ - -d '{ - "name": "circuitforge-hooks", - "description": "Centralised git hooks for CircuitForge repos — gitleaks secret + PII scanning", - "private": false, - "auto_init": false - }' | python3 -c "import json,sys; r=json.load(sys.stdin); print('Created:', r.get('html_url','ERROR:', r))" - -# Add remote and push -cd /Library/Development/CircuitForge/circuitforge-hooks -git add README.md -git commit -m "docs: add README with install and usage instructions" -git remote add origin https://git.opensourcesolarpunk.com/pyr0ball/circuitforge-hooks.git -git push -u origin main -``` - ---- - -### Task 7: Activate in Peregrine - -**Files:** -- Create: `peregrine/.gitleaks.toml` -- Modify: `peregrine/manage.sh` (add install.sh call) -- Delete: `peregrine/.githooks/pre-commit` (replaced by gitleaks wrapper) - -**Step 1: Write peregrine/.gitleaks.toml** - -```toml -# peregrine/.gitleaks.toml — per-repo allowlists extending the shared base config -[extend] -path = "/Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml" - -[allowlist] -description = "Peregrine-specific allowlists" -regexes = [ - '\d{10}\.html', # Craigslist listing IDs (10-digit paths, look like phone numbers) - '\d{10}\/', # LinkedIn job IDs in URLs - 'localhost:\d{4,5}', # port numbers that could trip phone pattern -] -``` - -**Step 2: Activate hooks in Peregrine** - -```bash -cd /Library/Development/CircuitForge/peregrine -bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh -``` - -Expected output: -``` -CircuitForge hooks installed. - core.hooksPath → /Library/Development/CircuitForge/circuitforge-hooks/hooks -``` - -Verify: -```bash -git config core.hooksPath -``` -Expected: prints the absolute path to `circuitforge-hooks/hooks` - -**Step 3: Add install.sh call to manage.sh** - -In `peregrine/manage.sh`, find the section that runs setup/preflight (near the top of the `start` command handling). Add after the existing setup checks: - -```bash -# Wire CircuitForge hooks (idempotent — safe to run every time) -if [[ -f "/Library/Development/CircuitForge/circuitforge-hooks/install.sh" ]]; then - bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh --quiet 2>/dev/null || true -fi -``` - -Also add a `--quiet` flag to `install.sh` to suppress output when called from manage.sh: - -In `circuitforge-hooks/install.sh`, modify to accept `--quiet`: -```bash -QUIET=false -[[ "${1:-}" == "--quiet" ]] && QUIET=true - -git config core.hooksPath "$HOOKS_DIR" -if [[ "$QUIET" == "false" ]]; then - echo "CircuitForge hooks installed." - echo " core.hooksPath → $HOOKS_DIR" -fi -``` - -**Step 4: Retire old .githooks/pre-commit** - -The old hook used hand-rolled regexes and is now superseded. Remove it: - -```bash -cd /Library/Development/CircuitForge/peregrine -rm .githooks/pre-commit -``` - -Keep `.githooks/commit-msg` until verified the new one is working (then remove in a follow-up). - -**Step 5: Smoke-test — try to commit a fake secret** - -```bash -cd /Library/Development/CircuitForge/peregrine -echo 'TEST_TOKEN=abc123def456ghi789jkl012mno345' >> /tmp/leak-test.txt -git add /tmp/leak-test.txt 2>/dev/null || true -# Easier: stage it directly -echo 'BAD_TOKEN=abc123def456ghi789jkl012mno345pqr' > /tmp/test-secret.py -cp /tmp/test-secret.py . -git add test-secret.py -git commit -m "test: this should be blocked" 2>&1 -``` -Expected: commit blocked with gitleaks output. Clean up: -```bash -git restore --staged test-secret.py && rm test-secret.py -``` - -**Step 6: Commit Peregrine changes** - -```bash -cd /Library/Development/CircuitForge/peregrine -git add .gitleaks.toml manage.sh -git rm .githooks/pre-commit -git commit -m "chore: activate circuitforge-hooks, add .gitleaks.toml, retire old pre-commit" -``` - -**Step 7: Push Peregrine** - -```bash -git push origin main -``` - ---- - -### Task 8: Run full test suite and verify - -**Step 1: Run the hooks test suite** - -```bash -bash /Library/Development/CircuitForge/circuitforge-hooks/tests/test_hooks.sh -``` -Expected: `All tests passed. Passed: 12 Failed: 0` - -**Step 2: Run Peregrine tests to confirm nothing broken** - -```bash -cd /Library/Development/CircuitForge/peregrine -/devl/miniconda3/envs/job-seeker/bin/pytest tests/ -v --tb=short -q 2>&1 | tail -10 -``` -Expected: all existing tests still pass. - -**Step 3: Push hooks repo final state** - -```bash -cd /Library/Development/CircuitForge/circuitforge-hooks -git push origin main -``` - ---- - -## Public-release checklist (for all future repos) - -Add this to any repo's pre-public checklist: - -``` -[ ] Run: gitleaks git --config /Library/Development/CircuitForge/circuitforge-hooks/gitleaks.toml - (manual full-history scan — pre-push hook only covers branch tip) -[ ] Run: bash /Library/Development/CircuitForge/circuitforge-hooks/install.sh -[ ] Add .gitleaks.toml with repo-specific allowlists -[ ] Verify: git config core.hooksPath -[ ] Make repo public on Forgejo -``` diff --git a/docs/plans/email-sync-testing-checklist.md b/docs/plans/email-sync-testing-checklist.md deleted file mode 100644 index eb29479..0000000 --- a/docs/plans/email-sync-testing-checklist.md +++ /dev/null @@ -1,106 +0,0 @@ -# Email Sync — Testing Checklist - -Generated from audit of `scripts/imap_sync.py`. - -## Bugs fixed (2026-02-23) - -- [x] Gmail label with spaces not quoted for IMAP SELECT → `_quote_folder()` added -- [x] `_quote_folder` didn't escape internal double-quotes → RFC 3501 escaping added -- [x] `signal is None` in `_scan_unmatched_leads` allowed classifier failures through → now skips -- [x] Email with no Message-ID re-inserted on every sync → `_parse_message` returns `None` when ID missing -- [x] `todo_attached` missing from early-return dict in `sync_all` → added -- [x] Body phrase check truncated at 800 chars (rejection footers missed) → bumped to 1500 -- [x] `_DONT_FORGET_VARIANTS` missing left single quotation mark `\u2018` → added - ---- - -## Unit tests — phrase filter - -- [x] `_has_rejection_or_ats_signal` — rejection phrase at char 1501 (boundary) -- [x] `_has_rejection_or_ats_signal` — right single quote `\u2019` in "don't forget" -- [x] `_has_rejection_or_ats_signal` — left single quote `\u2018` in "don't forget" -- [x] `_has_rejection_or_ats_signal` — ATS subject phrase only checked against subject, not body -- [x] `_has_rejection_or_ats_signal` — spam subject prefix `@` match -- [x] `_has_rejection_or_ats_signal` — `"UNFORTUNATELY"` (uppercase → lowercased correctly) -- [x] `_has_rejection_or_ats_signal` — phrase in body quoted thread (beyond 1500 chars) is not blocked - -## Unit tests — folder quoting - -- [x] `_quote_folder("TO DO JOBS")` → `'"TO DO JOBS"'` -- [x] `_quote_folder("INBOX")` → `"INBOX"` (no spaces, no quotes added) -- [x] `_quote_folder('My "Jobs"')` → `'"My \\"Jobs\\""'` -- [x] `_search_folder` — folder doesn't exist → returns `[]`, no exception -- [x] `_search_folder` — special folder `"[Gmail]/All Mail"` (brackets + slash) - -## Unit tests — message-ID dedup - -- [x] `_get_existing_message_ids` — NULL message_id in DB excluded from set -- [x] `_get_existing_message_ids` — empty string `""` excluded from set -- [x] `_get_existing_message_ids` — job with no contacts returns empty set -- [x] `_parse_message` — email with no Message-ID header returns `None` -- [x] `_parse_message` — email with RFC2047-encoded subject decodes correctly -- [x] No email is inserted twice across two sync runs (integration) - -## Unit tests — classifier & signal - -- [x] `classify_stage_signal` — returns one of 5 labels or `None` -- [x] `classify_stage_signal` — returns `None` on LLM error -- [x] `classify_stage_signal` — returns `"neutral"` when no label matched in LLM output -- [x] `classify_stage_signal` — strips `…` blocks -- [x] `_scan_unmatched_leads` — skips when `signal is None` -- [x] `_scan_unmatched_leads` — skips when `signal == "rejected"` -- [x] `_scan_unmatched_leads` — proceeds when `signal == "neutral"` -- [x] `extract_lead_info` — returns `(None, None)` on bad JSON -- [x] `extract_lead_info` — returns `(None, None)` on LLM error - -## Integration tests — TODO label scan - -- [x] `_scan_todo_label` — `todo_label` empty string → returns 0 -- [x] `_scan_todo_label` — `todo_label` missing from config → returns 0 -- [x] `_scan_todo_label` — folder doesn't exist on IMAP server → returns 0, no crash -- [x] `_scan_todo_label` — email matches company + action keyword → contact attached -- [x] `_scan_todo_label` — email matches company but no action keyword → skipped -- [x] `_scan_todo_label` — email matches no company term → skipped -- [x] `_scan_todo_label` — duplicate message-ID → not re-inserted -- [x] `_scan_todo_label` — stage_signal set when classifier returns non-neutral -- [x] `_scan_todo_label` — body fallback (company only in body[:300]) → still matches -- [x] `_scan_todo_label` — email handled by `sync_job_emails` first not re-added by label scan - -## Integration tests — unmatched leads - -- [x] `_scan_unmatched_leads` — genuine lead inserted with synthetic URL `email://domain/hash` -- [x] `_scan_unmatched_leads` — same email not re-inserted on second sync run -- [x] `_scan_unmatched_leads` — duplicate synthetic URL skipped -- [x] `_scan_unmatched_leads` — `extract_lead_info` returns `(None, None)` → no insertion -- [x] `_scan_unmatched_leads` — rejection phrase in body → blocked before LLM -- [x] `_scan_unmatched_leads` — rejection phrase in quoted thread > 1500 chars → passes filter (acceptable) - -## Integration tests — full sync - -- [x] `sync_all` with no active jobs → returns dict with all 6 keys incl. `todo_attached: 0` -- [x] `sync_all` return dict shape identical on all code paths -- [x] `sync_all` with `job_ids` filter → only syncs those jobs -- [x] `sync_all` `dry_run=True` → no DB writes -- [x] `sync_all` `on_stage` callback fires: "connecting", "job N/M", "scanning todo label", "scanning leads" -- [x] `sync_all` IMAP connection error → caught, returned in `errors` list -- [x] `sync_all` per-job exception → other jobs still sync - -## Config / UI - -- [x] Settings UI field for `todo_label` (currently YAML-only) -- [x] Warn in sync summary when `todo_label` folder not found on server -- [x] Clear error message when `config/email.yaml` is missing -- [x] `test_email_classify.py --verbose` shows correct blocking phrase for each BLOCK - -## Backlog — Known issues - -- [x] **The Ladders emails confuse the classifier** — promotional/job alert emails from `@theladders.com` are matching the recruitment keyword filter and being treated as leads. Fix: add a sender-based skip rule in `_scan_unmatched_leads` for known job board senders (similar to how LinkedIn Alert emails are short-circuited before the LLM classifier). Senders to exclude: `@theladders.com`, and audit for others (Glassdoor alerts, Indeed digest, ZipRecruiter, etc.). - ---- - -## Performance & edge cases - -- [x] Email with 10 000-char body → truncated to 4000 chars, no crash -- [x] Email with binary attachment → `_parse_message` returns valid dict, no crash -- [x] Email with multiple `text/plain` MIME parts → first part taken -- [x] `get_all_message_ids` with 100 000 rows → completes in < 1s