From fe2f27c2a2d4c915595978383ba0a20db97b3aa3 Mon Sep 17 00:00:00 2001 From: pyr0ball Date: Sun, 15 Mar 2026 04:16:22 -0700 Subject: [PATCH] docs: add Jobgether integration design spec --- ...2026-03-15-jobgether-integration-design.md | 159 ++++++++++++++++++ 1 file changed, 159 insertions(+) create mode 100644 docs/superpowers/specs/2026-03-15-jobgether-integration-design.md diff --git a/docs/superpowers/specs/2026-03-15-jobgether-integration-design.md b/docs/superpowers/specs/2026-03-15-jobgether-integration-design.md new file mode 100644 index 0000000..12024e7 --- /dev/null +++ b/docs/superpowers/specs/2026-03-15-jobgether-integration-design.md @@ -0,0 +1,159 @@ +# Jobgether Integration Design + +**Date:** 2026-03-15 +**Status:** Approved +**Scope:** Peregrine — discovery pipeline + manual URL import + +--- + +## Problem + +Jobgether is a job aggregator that posts listings on LinkedIn and other boards with `company = "Jobgether"` rather than the actual employer. This causes two problems: + +1. **Misleading listings** — Jobs appear to be at "Jobgether" rather than the real hiring company. Meg sees "Jobgether" as employer throughout the pipeline (Job Review, cover letters, company research). +2. **Broken manual import** — Direct `jobgether.com` URLs return HTTP 403 when scraped with plain `requests`, leaving jobs stuck as `title = "Importing…"`. + +**Evidence from DB:** 29+ Jobgether-sourced LinkedIn listings with `company = "Jobgether"`. Actual employer is intentionally withheld by Jobgether's business model ("on behalf of a partner company"). + +--- + +## Decision: Option A — Filter + Dedicated Scraper + +Drop Jobgether listings from other scrapers entirely and replace with a direct Jobgether scraper that retrieves accurate company names. Existing Jobgether-via-LinkedIn listings in the DB are left as-is for manual review/rejection. + +**Why not Option B (follow-through):** LinkedIn→Jobgether→employer is a two-hop chain where the employer is deliberately hidden. Jobgether blocks `requests`. Not worth the complexity for unreliable data. + +--- + +## Components + +### 1. Jobgether company filter — `config/blocklist.yaml` + +Add `"jobgether"` to the `companies` list in `config/blocklist.yaml`. The existing `_is_blocklisted()` function in `discover.py` already performs a partial case-insensitive match on the company field and applies to all scrapers (JobSpy boards + all custom boards). No code change required. + +```yaml +companies: + - jobgether +``` + +This is the correct mechanism — it is user-visible, config-driven, and applies uniformly. Log output already reports blocklisted jobs per run. + +### 2. URL handling in `scrape_url.py` + +Three changes required: + +**a) `_detect_board()`** — add `"jobgether"` branch returning `"jobgether"` when `"jobgether.com"` is in the URL. Must be added before the `return "generic"` fallback. + +**b) dispatch block in `scrape_job_url()`** — add `elif board == "jobgether": fields = _scrape_jobgether(url)` to the `if/elif` chain (lines 208–215). Without this, the new `_detect_board()` branch silently falls through to `_scrape_generic()`. + +**c) `_scrape_jobgether(url)`** — Playwright-based scraper to bypass 403. Extracts: +- `title` — job title from page heading +- `company` — actual employer name (visible on Jobgether offer pages) +- `location` — remote/location info +- `description` — full job description +- `source = "jobgether"` + +Playwright errors (`playwright.sync_api.Error`, `TimeoutError`) are not subclasses of `requests.RequestException` but are caught by the existing broad `except Exception` handler in `scrape_job_url()` — no changes needed to the error handling block. + +**URL slug fallback for company name (manual import path only):** Jobgether offer URLs follow the pattern: +``` +https://jobgether.com/offer/{24-hex-hash}-{title-slug}---{company-slug} +``` +When Playwright is unavailable, parse `company-slug` using: +```python +m = re.search(r'---([^/?]+)$', parsed_path) +company = m.group(1).replace("-", " ").title() if m else "" +``` +Example: `/offer/69b42d9d24d79271ee0618e8-customer-success-manager---resware` → `"Resware"`. + +This fallback is scoped to `_scrape_jobgether()` in `scrape_url.py` only; the discovery scraper always gets company name from the rendered DOM. `_scrape_jobgether()` does not make any `requests` calls — there is no `raise_for_status()` — so the `requests.RequestException` handler in `scrape_job_url()` is irrelevant to this path; only the broad `except Exception` applies. + +**Pre-implementation checkpoint:** Confirm that Jobgether offer URLs have no tracking query params beyond UTM (already covered by `_STRIP_PARAMS`). No `canonicalize_url()` changes are expected but verify before implementation. + +### 3. `scripts/custom_boards/jobgether.py` + +Playwright-based search scraper following the same interface as `theladders.py`: + +```python +def scrape(profile: dict, location: str, results_wanted: int = 50) -> list[dict] +``` + +- Base URL: `https://jobgether.com/remote-jobs` +- Search strategy: iterate over `profile["titles"]`, apply search/filter params +- **Pre-condition — do not begin implementation of this file until live URL inspection is complete.** Use browser dev tools or a Playwright `page.on("request")` capture to determine the actual query parameter format for title/location filtering. Jobgether may use URL query params, path segments, or JS-driven state — this cannot be assumed from the URL alone. +- Extraction: job cards from rendered DOM (Playwright `page.evaluate()`) +- Returns standard job dicts: `title, company, url, source, location, is_remote, salary, description` +- `source = "jobgether"` +- Graceful `ImportError` handling if Playwright not installed (same pattern as `theladders.py`) +- Polite pacing: 1s sleep between title iterations +- Company name comes from DOM; URL slug parse is not needed in this path + +### 4. Registration + config + +**`discover.py` — import block (lines 20–22):** +```python +from scripts.custom_boards import jobgether as _jobgether +``` + +**`discover.py` — `CUSTOM_SCRAPERS` dict literal (lines 30–34):** +```python +CUSTOM_SCRAPERS: dict[str, object] = { + "adzuna": _adzuna.scrape, + "theladders": _theladders.scrape, + "craigslist": _craigslist.scrape, + "jobgether": _jobgether.scrape, # ← add this line +} +``` + +**`config/search_profiles.yaml` (and `.example`):** +Add `jobgether` to `custom_boards` for any profile that includes `Remote` in its `locations` list. Jobgether is a remote-work-focused aggregator; adding it to location-specific non-remote profiles is not useful. Do not add a `custom_boards` key to profiles that don't already have one unless they are remote-eligible. +```yaml +custom_boards: + - jobgether +``` + +--- + +## Data Flow + +``` +discover.py + ├── JobSpy boards → _is_blocklisted(company="jobgether") → drop → DB insert + ├── custom: adzuna → _is_blocklisted(company="jobgether") → drop → DB insert + ├── custom: theladders → _is_blocklisted(company="jobgether") → drop → DB insert + ├── custom: craigslist → _is_blocklisted(company="jobgether") → drop → DB insert + └── custom: jobgether → (company = real employer, never "jobgether") → DB insert + +scrape_url.py + └── jobgether.com URL → _detect_board() = "jobgether" + → _scrape_jobgether() + ├── Playwright available → full job fields from page + └── Playwright unavailable → company from URL slug only +``` + +--- + +## Implementation Notes + +- **Slug fallback None-guard:** The regex `r'---([^/?]+)$'` returns a wrong value (not `None`) if the URL slug doesn't follow the expected format. Add a logged warning and return `""` rather than title-casing garbage. +- **Import guard in `discover.py`:** Wrap the `jobgether` import with `try/except ImportError`, setting `_jobgether = None`, and gate the `CUSTOM_SCRAPERS` registration with `if _jobgether is not None`. This ensures the graceful ImportError in `jobgether.py` (for missing Playwright) propagates cleanly to the caller rather than crashing discovery. + +## Out of Scope + +- Retroactively fixing existing `company = "Jobgether"` rows in the DB (left for manual review/rejection) +- Jobgether authentication / logged-in scraping +- Pagination beyond `results_wanted` cap +- Dedup between Jobgether scraper and other boards (existing URL dedup in `discover.py` handles this) + +--- + +## Files Changed + +| File | Change | +|------|--------| +| `config/blocklist.yaml` | Add `"jobgether"` to `companies` list | +| `scripts/discover.py` | Add import + entry in `CUSTOM_SCRAPERS` dict literal | +| `scripts/scrape_url.py` | Add `_detect_board` branch, dispatch branch, `_scrape_jobgether()` | +| `scripts/custom_boards/jobgether.py` | New file — Playwright search scraper | +| `config/search_profiles.yaml` | Add `jobgether` to `custom_boards` | +| `config/search_profiles.yaml.example` | Same |