Circuit-Forge/magpie
3
0
Fork 0
Code Issues 11 Pull requests Projects Releases Packages Wiki Activity Actions
magpie by Circuit Forge LLC — Cross-product data gathering and collection tool (reserved)
4 commits 2 branches 0 tags 409 KiB
Python 55.5%
Vue 22.5%
JavaScript 7.5%
TypeScript 5.5%
CSS 5.4%
Other 3.6%
Find a file
pyr0ball a6ea0b9c58 feat(#7,#10): signal crawler -- Reddit + Lemmy community monitoring 
Implements the full signal detection pipeline:

Backend:
- app/services/lemmy/client.py: async Lemmy API v3 client, community@instance
  addressing, integer cursor dedup, normalised post dicts
- app/services/scraper.py: platform-agnostic scraper; Reddit (.json API,
  fullname cursor) + Lemmy (integer ID cursor); keyword/regex/all match modes,
  min_score gate, NormalizedPost shape, upsert dedup via UNIQUE post_id
- app/api/endpoints/signals.py: CRUD for signal_rules + signals queue;
  POST /signals/scrape manual trigger; scrape-state viewer
- migrations 010-012: signal_rules, signals, signal_scrape_state tables
- scheduler: interval job every 30 min (scraper_enabled=True in config)
- Fixed migration collision: 007_signal_rules.sql → 010, 008 → 011, 009 → 012

Frontend:
- SignalsView.vue: signal feed with status filter (new/saved/dismissed),
  keyword chips, score/comment counts, save/dismiss actions, rules editor panel
- api.ts: SignalRule, Signal types + signalRules/signals API methods
- Nav: Signals as default landing route (replaces /campaigns default)

Closes #7 (signal extraction), closes #10 (Lemmy JSON crawler)
2026-04-22 11:00:14 -07:00
app feat(#7,#10): signal crawler -- Reddit + Lemmy community monitoring 2026-04-22 11:00:14 -07:00
frontend feat(#7,#10): signal crawler -- Reddit + Lemmy community monitoring 2026-04-22 11:00:14 -07:00
mcp feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
scripts feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
.env.example feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
.gitignore feat(#7,#10): signal crawler -- Reddit + Lemmy community monitoring 2026-04-22 11:00:14 -07:00
docker-compose.yml feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
Dockerfile feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
LICENSE Initial commit 2026-03-10 20:52:31 -07:00
manage.sh feat(#7,#10): signal crawler -- Reddit + Lemmy community monitoring 2026-04-22 11:00:14 -07:00
pyproject.toml feat: scaffold Magpie — campaign scheduler + social posting platform 2026-04-21 16:51:33 -07:00
README.md Initial commit 2026-03-10 20:52:31 -07:00

README.md

magpie

magpie by Circuit Forge LLC — Cross-product data gathering and collection tool (reserved)