[MGPE] Magpie -- Federated pipeline crystallization layer #19

New issue

Open

opened 2026-03-10 20:50:26 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-03-10 20:50:26 -07:00

Owner

[MGPE] Magpie

Federated crystallization layer -- aggregates cross-user, cross-product approved pipeline runs into shared deterministic workflows.

What Magpie Is

The pipeline module in cf-core handles local crystallization: after a user approves the same LLM-discovered workflow N times, it promotes to a deterministic CrystallizedWorkflow stored in ~/.config/circuitforge/pipeline/. This removes the LLM from the hot path for known tasks.

Magpie is the federated layer on top of that. When 50 osprey users each independently approve the same FTB IVR DTMF sequence, that is 50 independent crystallization signals. Magpie collects those (PII-free -- only hashed input patterns + workflow steps), identifies convergence, and distributes a shared CrystallizedWorkflow back to the fleet -- so the next osprey user never has to discover it from scratch.

User A (osprey) -> PipelineRun + approval --+
User B (osprey) -> PipelineRun + approval --+--> Magpie ingestion (PII-stripped)
User C (osprey) -> PipelineRun + approval --+         |
                                               Global crystallizer
                                                       |
                                          Shared CrystallizedWorkflow
                                                       |
                                       <-- distributed to all fleet instances

Relationship to cf-core pipeline module

The local pipeline module (cf-core#33) is the prerequisite. Magpie is the opt-in federation layer on top:

Local (cf-core, MIT): records runs, crystallizes for the individual user
Magpie (BSL 1.1): aggregates approved runs across users, crystallizes globally, distributes shared workflows

Products call pipeline.run() the same way regardless -- the registry checks local workflows first, then pulls from Magpie's shared registry if connected.

Privacy Model

Raw inputs are never sent to Magpie -- only input_hash (normalized feature vector hash)
Workflow steps contain only deterministic actions (DTMF tones, form field rules, template parameters) -- no user content
Contribution is opt-in per user, per product, per task type
Users can inspect every record contributed before it leaves their device
Magpie stores no PII (personally identifiable information), by architecture

Per-Product Use Cases

Product	Task type	Magpie crystallizes to
`osprey`	`ivr_navigate`	DTMF sequence + timing per agency (shared across all osprey users)
`falcon`	`form_fill`	Field ruleset per form schema (shared across all falcon users)
`kiwi`	`receipt_parse`	Deterministic parser per merchant (shared across all kiwi users)
`snipe`	`trust_score`	New fraud signal rule added to shared aggregator
`peregrine`	`cover_letter`	Template + scoring rules per industry/role niche

Tier Split

Tier	What you get
Free	Local-only crystallization via cf-core pipeline module
Paid	Opt-in contribution to + pull from Magpie shared registry
Premium	Priority distribution, advanced analytics on contributed workflows

Repository

https://git.opensourcesolarpunk.com/Circuit-Forge/magpie

Design Doc

circuitforge-plans/magpie/2026-04-07-magpie-design.md

Prerequisite

circuitforge-core#33 (pipeline crystallization engine) must ship before Magpie can ingest anything.

Design Pillars

Privacy * Safety * Accessibility -- co-equal, non-negotiable.
Neurodivergent and adaptive needs users are a primary audience.

License

Discovery pipeline: MIT | AI features / federation layer: BSL 1.1 (converts to MIT after 4 years)

## [MGPE] Magpie > Federated crystallization layer -- aggregates cross-user, cross-product approved pipeline runs into shared deterministic workflows. ### What Magpie Is The `pipeline` module in cf-core handles **local crystallization**: after a user approves the same LLM-discovered workflow N times, it promotes to a deterministic `CrystallizedWorkflow` stored in `~/.config/circuitforge/pipeline/`. This removes the LLM from the hot path for known tasks. Magpie is the **federated layer** on top of that. When 50 osprey users each independently approve the same FTB IVR DTMF sequence, that is 50 independent crystallization signals. Magpie collects those (PII-free -- only hashed input patterns + workflow steps), identifies convergence, and distributes a shared `CrystallizedWorkflow` back to the fleet -- so the next osprey user never has to discover it from scratch. ``` User A (osprey) -> PipelineRun + approval --+ User B (osprey) -> PipelineRun + approval --+--> Magpie ingestion (PII-stripped) User C (osprey) -> PipelineRun + approval --+ | Global crystallizer | Shared CrystallizedWorkflow | <-- distributed to all fleet instances ``` ### Relationship to cf-core pipeline module The local pipeline module (cf-core#33) is the prerequisite. Magpie is the opt-in federation layer on top: - **Local (cf-core, MIT):** records runs, crystallizes for the individual user - **Magpie (BSL 1.1):** aggregates approved runs across users, crystallizes globally, distributes shared workflows Products call `pipeline.run()` the same way regardless -- the registry checks local workflows first, then pulls from Magpie's shared registry if connected. ### Privacy Model - Raw inputs are **never** sent to Magpie -- only `input_hash` (normalized feature vector hash) - Workflow steps contain only deterministic actions (DTMF tones, form field rules, template parameters) -- no user content - Contribution is **opt-in** per user, per product, per task type - Users can inspect every record contributed before it leaves their device - Magpie stores no PII (personally identifiable information), by architecture ### Per-Product Use Cases | Product | Task type | Magpie crystallizes to | |---------|-----------|------------------------| | `osprey` | `ivr_navigate` | DTMF sequence + timing per agency (shared across all osprey users) | | `falcon` | `form_fill` | Field ruleset per form schema (shared across all falcon users) | | `kiwi` | `receipt_parse` | Deterministic parser per merchant (shared across all kiwi users) | | `snipe` | `trust_score` | New fraud signal rule added to shared aggregator | | `peregrine` | `cover_letter` | Template + scoring rules per industry/role niche | ### Tier Split | Tier | What you get | |------|--------------| | **Free** | Local-only crystallization via cf-core pipeline module | | **Paid** | Opt-in contribution to + pull from Magpie shared registry | | **Premium** | Priority distribution, advanced analytics on contributed workflows | ### Repository https://git.opensourcesolarpunk.com/Circuit-Forge/magpie ### Design Doc `circuitforge-plans/magpie/2026-04-07-magpie-design.md` ### Prerequisite circuitforge-core#33 (pipeline crystallization engine) must ship before Magpie can ingest anything. ### Design Pillars Privacy * Safety * Accessibility -- co-equal, non-negotiable. Neurodivergent and adaptive needs users are a primary audience. ### License Discovery pipeline: MIT | AI features / federation layer: BSL 1.1 (converts to MIT after 4 years)

pyr0ball added the

priority:backlog

status:concept

labels 2026-03-10 20:50:26 -07:00

pyr0ball referenced this issue

2026-04-05 21:37:29 -07:00

[OSPY] Osprey — Customer service — IVR queues, complaint letters, disputes #4

pyr0ball changed title from ~~[MGPE] Magpie — Cross-product data gathering and collection tool (reserved)~~ to [MGPE] Magpie -- Federated pipeline crystallization layer

2026-04-07 10:55:58 -07:00