419 lines
16 KiB
Markdown
419 lines
16 KiB
Markdown
# Digest Scrape Queue — Design Spec
|
|
|
|
**Date:** 2026-03-19
|
|
**Status:** Approved — ready for implementation planning
|
|
|
|
---
|
|
|
|
## Goal
|
|
|
|
When a user clicks the 📰 Digest chip on a signal banner, the email is added to a persistent digest queue accessible via a dedicated nav tab. The user browses queued digest emails, selects extracted job links to process, and queues them through the existing discovery pipeline as `status='pending'` jobs in `staging.db`.
|
|
|
|
---
|
|
|
|
## Decisions Made
|
|
|
|
| Decision | Choice |
|
|
|---|---|
|
|
| Digest tab placement | Separate top-level nav tab "📰 Digest", between Interviews and Apply |
|
|
| Storage | New `digest_queue` table in `staging.db`; unique on `job_contact_id` |
|
|
| Table creation | In `scripts/db.py` `init_db()` — canonical schema location, not `dev-api.py` |
|
|
| Link extraction | On-demand, backend regex against HTML-stripped plain-text body — no background task needed |
|
|
| Extraction UX | Show ranked link list; job-likely pre-checked, others unchecked; user ticks and submits |
|
|
| After queueing | Entry stays in digest list for reference; `[✕]` removes explicitly |
|
|
| Failure handling | Digest chip dismisses signal optimistically regardless of `POST /api/digest-queue` success |
|
|
| Duplicate protection | `UNIQUE(job_contact_id)` in table; `POST /api/digest-queue` returns `{ created: false }` on duplicate (no 409) |
|
|
| Mobile nav | Digest tab does NOT appear in mobile bottom tab bar (all 5 slots occupied; deferred) |
|
|
| URL validation | Non-http/https schemes and blank URLs skipped silently in `queue-jobs`; validation deferred to pipeline |
|
|
|
|
---
|
|
|
|
## Data Model
|
|
|
|
### New table: `digest_queue`
|
|
|
|
Added to `scripts/db.py` `init_db()`:
|
|
|
|
```sql
|
|
CREATE TABLE IF NOT EXISTS digest_queue (
|
|
id INTEGER PRIMARY KEY,
|
|
job_contact_id INTEGER NOT NULL REFERENCES job_contacts(id),
|
|
created_at TEXT DEFAULT (datetime('now')),
|
|
UNIQUE(job_contact_id)
|
|
)
|
|
```
|
|
|
|
`init_db()` is called at app startup and by `dev-api.py` startup — adding the `CREATE TABLE IF NOT EXISTS` there is safe and idempotent.
|
|
|
|
---
|
|
|
|
## Backend
|
|
|
|
### New endpoints in `dev-api.py`
|
|
|
|
#### `GET /api/digest-queue`
|
|
|
|
Returns all queued entries joined with `job_contacts`. `body` is HTML-stripped via `_strip_html()` before returning (display only — extraction uses a separate raw read, see `extract-links`):
|
|
|
|
```python
|
|
SELECT dq.id, dq.job_contact_id, dq.created_at,
|
|
jc.subject, jc.from_addr, jc.received_at, jc.body
|
|
FROM digest_queue dq
|
|
JOIN job_contacts jc ON jc.id = dq.job_contact_id
|
|
ORDER BY dq.created_at DESC
|
|
```
|
|
|
|
Response: array of `{ id, job_contact_id, created_at, subject, from_addr, received_at, body }`.
|
|
|
|
---
|
|
|
|
#### `POST /api/digest-queue`
|
|
|
|
Body: `{ job_contact_id: int }`
|
|
|
|
- Verify `job_contact_id` exists in `job_contacts` → 404 if not found
|
|
- `INSERT OR IGNORE INTO digest_queue (job_contact_id) VALUES (?)`
|
|
- Returns `{ ok: true, created: true }` on insert, `{ ok: true, created: false }` if already present
|
|
- Never returns 409 — the `created` field is the duplicate signal
|
|
|
|
---
|
|
|
|
#### `POST /api/digest-queue/{id}/extract-links`
|
|
|
|
Extracts and ranks URLs from the entry's email body. No request body.
|
|
|
|
**Important:** this endpoint reads the **raw** `body` from `job_contacts` directly and runs `URL_RE` against it **before** any HTML stripping. `_strip_html()` calls `BeautifulSoup.get_text()`, which extracts visible text only — it does not preserve `href` attribute values. A URL that appears only as an `href` target (e.g., `<a href="https://greenhouse.io/acme/1">Click here</a>`) would be lost after stripping. Running the regex on raw HTML captures those URLs correctly because `URL_RE`'s character exclusion class (`[^\s<>"')\]]`) stops at `"`, so it cleanly extracts href values without matching surrounding markup.
|
|
|
|
```python
|
|
# Fetch raw body from DB — do NOT strip before extraction
|
|
row = db.execute(
|
|
"SELECT jc.body FROM digest_queue dq JOIN job_contacts jc ON jc.id = dq.job_contact_id WHERE dq.id = ?",
|
|
(digest_id,)
|
|
).fetchone()
|
|
if not row:
|
|
raise HTTPException(404, "Digest entry not found")
|
|
return {"links": extract_links(row["body"] or "")}
|
|
```
|
|
|
|
**Extraction algorithm:**
|
|
|
|
```python
|
|
import re
|
|
from urllib.parse import urlparse
|
|
|
|
JOB_DOMAINS = {
|
|
'greenhouse.io', 'lever.co', 'workday.com', 'linkedin.com',
|
|
'ashbyhq.com', 'smartrecruiters.com', 'icims.com', 'taleo.net',
|
|
'jobvite.com', 'breezy.hr', 'recruitee.com', 'bamboohr.com',
|
|
'myworkdayjobs.com', 'careers.', 'jobs.',
|
|
}
|
|
|
|
FILTER_PATTERNS = re.compile(
|
|
r'(unsubscribe|mailto:|/track/|pixel\.|\.gif|\.png|\.jpg'
|
|
r'|/open\?|/click\?|list-unsubscribe)',
|
|
re.I
|
|
)
|
|
|
|
URL_RE = re.compile(r'https?://[^\s<>"\')\]]+', re.I)
|
|
|
|
def _score_url(url: str) -> int:
|
|
parsed = urlparse(url)
|
|
hostname = parsed.hostname or ''
|
|
path = parsed.path.lower()
|
|
if FILTER_PATTERNS.search(url):
|
|
return -1 # exclude
|
|
for domain in JOB_DOMAINS:
|
|
if domain in hostname or domain in path:
|
|
return 2 # job-likely
|
|
return 1 # other
|
|
|
|
def extract_links(body: str) -> list[dict]:
|
|
if not body:
|
|
return []
|
|
seen = set()
|
|
results = []
|
|
for m in URL_RE.finditer(body):
|
|
url = m.group(0).rstrip('.,;)')
|
|
if url in seen:
|
|
continue
|
|
seen.add(url)
|
|
score = _score_url(url)
|
|
if score < 0:
|
|
continue
|
|
# Title hint: last line of text immediately before the URL (up to 60 chars)
|
|
start = max(0, m.start() - 60)
|
|
hint = body[start:m.start()].strip().split('\n')[-1].strip()
|
|
results.append({'url': url, 'score': score, 'hint': hint})
|
|
results.sort(key=lambda x: -x['score'])
|
|
return results
|
|
```
|
|
|
|
Response: `{ links: [{ url, score, hint }] }` — `score=2` means job-likely (pre-check in UI), `score=1` means other (unchecked).
|
|
|
|
---
|
|
|
|
#### `POST /api/digest-queue/{id}/queue-jobs`
|
|
|
|
Body: `{ urls: [string] }`
|
|
|
|
- 404 if digest entry not found
|
|
- 400 if `urls` is empty
|
|
- Non-http/https URLs and blank strings are skipped silently (counted as `skipped`)
|
|
|
|
Calls `insert_job` from `scripts/db.py`. The actual signature is `insert_job(db_path, job)` where `job` is a dict. The `status` field is **not** passed — the schema default of `'pending'` handles it:
|
|
|
|
```python
|
|
from scripts.db import insert_job
|
|
from datetime import datetime
|
|
|
|
queued = 0
|
|
skipped = 0
|
|
for url in body.urls:
|
|
if not url or not url.startswith(('http://', 'https://')):
|
|
skipped += 1
|
|
continue
|
|
result = insert_job(DB_PATH, {
|
|
'url': url,
|
|
'title': '',
|
|
'company': '',
|
|
'source': 'digest',
|
|
'date_found': datetime.utcnow().isoformat(),
|
|
})
|
|
if result:
|
|
queued += 1
|
|
else:
|
|
skipped += 1 # duplicate URL — insert_job returns None on UNIQUE conflict
|
|
return {'ok': True, 'queued': queued, 'skipped': skipped}
|
|
```
|
|
|
|
---
|
|
|
|
#### `DELETE /api/digest-queue/{id}`
|
|
|
|
Removes entry from `digest_queue`. Does not affect `job_contacts`.
|
|
Returns `{ ok: true }`. 404 if not found.
|
|
|
|
---
|
|
|
|
## Frontend Changes
|
|
|
|
### Chip handler update (`InterviewCard.vue` + `InterviewsView.vue`)
|
|
|
|
When `newLabel === 'digest'`, the handler fires a **third call** after the existing reclassify + dismiss calls. Note: `sig.id` is `job_contacts.id` — this is the correct value for `job_contact_id` (the `StageSignal.id` field maps directly to the `job_contacts` primary key):
|
|
|
|
```typescript
|
|
// After existing reclassify + dismiss calls:
|
|
if (newLabel === 'digest') {
|
|
fetch('/api/digest-queue', {
|
|
method: 'POST',
|
|
headers: { 'Content-Type': 'application/json' },
|
|
body: JSON.stringify({ job_contact_id: sig.id }), // sig.id === job_contacts.id
|
|
}).catch(() => {}) // best-effort; signal already dismissed optimistically
|
|
}
|
|
```
|
|
|
|
Signal is removed from local array optimistically before this call (same as current dismiss behavior).
|
|
|
|
---
|
|
|
|
### New store: `web/src/stores/digest.ts`
|
|
|
|
```typescript
|
|
import { defineStore } from 'pinia'
|
|
import { ref } from 'vue'
|
|
|
|
export interface DigestEntry {
|
|
id: number
|
|
job_contact_id: number
|
|
created_at: string
|
|
subject: string
|
|
from_addr: string | null
|
|
received_at: string
|
|
body: string | null
|
|
}
|
|
|
|
export interface DigestLink {
|
|
url: string
|
|
score: number // 2 = job-likely, 1 = other
|
|
hint: string
|
|
}
|
|
|
|
export const useDigestStore = defineStore('digest', () => {
|
|
const entries = ref<DigestEntry[]>([])
|
|
|
|
async function fetchAll() {
|
|
const res = await fetch('/api/digest-queue')
|
|
entries.value = await res.json()
|
|
}
|
|
|
|
async function remove(id: number) {
|
|
entries.value = entries.value.filter(e => e.id !== id)
|
|
await fetch(`/api/digest-queue/${id}`, { method: 'DELETE' })
|
|
}
|
|
|
|
return { entries, fetchAll, remove }
|
|
})
|
|
```
|
|
|
|
---
|
|
|
|
### New page: `web/src/views/DigestView.vue`
|
|
|
|
**Layout — collapsed entry (default):**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────┐
|
|
│ ▸ TechCrunch Jobs Weekly │
|
|
│ From: digest@techcrunch.com · Mar 19 │
|
|
│ [Extract] [✕] │
|
|
└─────────────────────────────────────────────┘
|
|
```
|
|
|
|
**Layout — expanded entry (after Extract):**
|
|
|
|
```
|
|
┌─────────────────────────────────────────────┐
|
|
│ ▾ LinkedIn Job Digest │
|
|
│ From: jobs@linkedin.com · Mar 18 │
|
|
│ [Re-extract] [✕] │
|
|
│ ┌──────────────────────────────────────┐ │
|
|
│ │ ☑ Senior Engineer — Acme Corp │ │ ← score=2, pre-checked
|
|
│ │ greenhouse.io/acme/jobs/456 │ │
|
|
│ │ ☑ Staff Designer — Globex │ │
|
|
│ │ lever.co/globex/staff-designer │ │
|
|
│ │ ─── Other links ────────────────── │ │
|
|
│ │ ☐ acme.com/blog/engineering │ │ ← score=1, unchecked
|
|
│ │ ☐ linkedin.com/company/acme │ │
|
|
│ └──────────────────────────────────────┘ │
|
|
│ [Queue 2 selected →] │
|
|
└─────────────────────────────────────────────┘
|
|
```
|
|
|
|
**After queueing:**
|
|
|
|
Inline confirmation replaces the link list:
|
|
```
|
|
✅ 2 jobs queued for review, 1 skipped (already in pipeline)
|
|
```
|
|
|
|
Entry remains in the list. `[✕]` removes it.
|
|
|
|
**Empty state:**
|
|
```
|
|
🦅 No digest emails queued.
|
|
When you mark an email as 📰 Digest, it appears here.
|
|
```
|
|
|
|
**Component state (per entry, keyed by `DigestEntry.id`):**
|
|
|
|
```typescript
|
|
const expandedIds = ref<Record<number, boolean>>({})
|
|
const linkResults = ref<Record<number, DigestLink[]>>({})
|
|
const selectedUrls = ref<Record<number, Set<string>>>({})
|
|
const queueResult = ref<Record<number, { queued: number; skipped: number } | null>>({})
|
|
const extracting = ref<Record<number, boolean>>({})
|
|
const queuing = ref<Record<number, boolean>>({})
|
|
```
|
|
|
|
`selectedUrls` uses `Set<string>`. Toggling a URL uses the spread-copy pattern to trigger Vue 3 reactivity — same pattern as `expandedSignalIds` in `InterviewCard.vue`:
|
|
|
|
```typescript
|
|
function toggleUrl(entryId: number, url: string) {
|
|
const prev = selectedUrls.value[entryId] ?? new Set()
|
|
const next = new Set(prev)
|
|
next.has(url) ? next.delete(url) : next.add(url)
|
|
selectedUrls.value = { ...selectedUrls.value, [entryId]: next }
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
### Router + Nav
|
|
|
|
Add to `web/src/router/index.ts`:
|
|
```typescript
|
|
{ path: '/digest', component: () => import('../views/DigestView.vue') }
|
|
```
|
|
|
|
**AppNav.vue changes:**
|
|
|
|
Add `NewspaperIcon` to the Heroicons import (already imported from `@heroicons/vue/24/outline`), then append to `navLinks` after `Interviews`:
|
|
|
|
```typescript
|
|
import { NewspaperIcon } from '@heroicons/vue/24/outline'
|
|
|
|
const navLinks = [
|
|
{ to: '/', icon: HomeIcon, label: 'Home' },
|
|
{ to: '/review', icon: ClipboardDocumentListIcon, label: 'Job Review' },
|
|
{ to: '/apply', icon: PencilSquareIcon, label: 'Apply' },
|
|
{ to: '/interviews', icon: CalendarDaysIcon, label: 'Interviews' },
|
|
{ to: '/digest', icon: NewspaperIcon, label: 'Digest' }, // NEW
|
|
{ to: '/prep', icon: LightBulbIcon, label: 'Interview Prep' },
|
|
{ to: '/survey', icon: MagnifyingGlassIcon, label: 'Survey' },
|
|
]
|
|
```
|
|
|
|
`navLinks` remains a static array. The badge count is rendered as a separate reactive expression in the template alongside the Digest link — keep `navLinks` as-is and add the digest store separately:
|
|
|
|
```typescript
|
|
// In AppNav.vue <script setup>
|
|
import { useDigestStore } from '@/stores/digest'
|
|
const digestStore = useDigestStore()
|
|
```
|
|
|
|
In the template, inside the `v-for="link in navLinks"` loop, add a badge overlay for the Digest entry:
|
|
```html
|
|
<span v-if="link.to === '/digest' && digestStore.entries.length > 0" class="nav-badge">
|
|
{{ digestStore.entries.length }}
|
|
</span>
|
|
```
|
|
|
|
The Digest nav item does **not** appear in the mobile bottom tab bar (`mobileLinks` array) — all 5 slots are occupied. Deferred to a future pass.
|
|
|
|
`DigestView.vue` calls `digestStore.fetchAll()` on `onMounted`.
|
|
|
|
---
|
|
|
|
## Required Tests (`tests/test_dev_api_digest.py`)
|
|
|
|
All tests follow the same isolated DB pattern as `test_dev_api_interviews.py`: use `importlib.reload` + FastAPI `TestClient`, seed fixtures directly into the test DB.
|
|
|
|
| Test | Setup + assertion |
|
|
|---|---|
|
|
| `test_digest_queue_add` | Seed a `job_contacts` row; POST `{ job_contact_id }` → 200, `created: true`, row in DB |
|
|
| `test_digest_queue_add_duplicate` | Seed + POST twice → second returns `created: false`, no error, only one row in DB |
|
|
| `test_digest_queue_add_missing_contact` | POST nonexistent `job_contact_id` → 404 |
|
|
| `test_digest_queue_list` | Seed `job_contacts` + `digest_queue`; GET → entries include `subject`, `from_addr`, `body` |
|
|
| `test_digest_extract_links` | Seed `job_contacts` with body containing a `greenhouse.io` URL and a tracker URL; seed `digest_queue`; POST to `/extract-links` → greenhouse URL present with `score=2`, tracker URL absent |
|
|
| `test_digest_extract_links_filters_trackers` | Same setup; assert unsubscribe and pixel URLs excluded from results |
|
|
| `test_digest_queue_jobs` | Seed `digest_queue`; POST `{ urls: ["https://greenhouse.io/acme/1"] }` → `queued: 1, skipped: 0`; row exists in `jobs` with `source='digest'` and `status='pending'` |
|
|
| `test_digest_queue_jobs_skips_duplicates` | POST `{ urls: ["https://greenhouse.io/acme/1", "https://greenhouse.io/acme/1"] }` — same URL twice in a single call → `queued: 1, skipped: 1`; one row in DB |
|
|
| `test_digest_queue_jobs_skips_invalid_urls` | POST `{ urls: ["", "ftp://bad", "https://good.com/job"] }` → `queued: 1, skipped: 2` |
|
|
| `test_digest_queue_jobs_empty_urls` | POST `{ urls: [] }` → 400 |
|
|
| `test_digest_delete` | Seed + DELETE → 200; second DELETE → 404 |
|
|
|
|
---
|
|
|
|
## Files
|
|
|
|
| File | Action |
|
|
|---|---|
|
|
| `scripts/db.py` | Add `digest_queue` table to `init_db()` |
|
|
| `dev-api.py` | Add 4 new endpoints; add `extract_links()` + `_score_url()` helpers |
|
|
| `web/src/stores/digest.ts` | New Pinia store |
|
|
| `web/src/views/DigestView.vue` | New page |
|
|
| `web/src/router/index.ts` | Add `/digest` route |
|
|
| `web/src/components/AppNav.vue` | Import digest store; add Digest nav item + reactive badge; desktop nav only |
|
|
| `web/src/components/InterviewCard.vue` | Third call in digest chip handler |
|
|
| `web/src/views/InterviewsView.vue` | Third call in digest chip handler |
|
|
| `tests/test_dev_api_digest.py` | New test file — 11 tests |
|
|
|
|
---
|
|
|
|
## What Stays the Same
|
|
|
|
- Existing reclassify + dismiss two-call path for digest chip — unchanged
|
|
- `insert_job` in `scripts/db.py` — called as-is, no modification needed
|
|
- Job Review UI — queued jobs appear there as `status='pending'` automatically
|
|
- Signal banner dismiss behavior — optimistic, unchanged
|
|
- `_strip_html()` helper in `dev-api.py` — reused for `GET /api/digest-queue` response body
|