feat(resources): cf-orch GPU VRAM orchestration — Plan A core #1

Merged

pyr0ball merged 21 commits from feature/cforch-core-orchestration into main

2026-03-31 10:43:53 -07:00

pyr0ball commented

2026-03-30 22:49:49 -07:00

Owner

Summary

Adds circuitforge_core.resources — the cf-orch (CircuitForge Orchestration) VRAM lease management system
Coordinator service tracks VRAM leases per (node_id, gpu_id) pair with asyncio lock; evicts lower-priority holders when VRAM is exhausted
Agent service polls nvidia-smi, accepts /evict calls (SIGTERM → grace → SIGKILL)
8 public GPU profiles (2 GB → 24 GB + CPU-only 16/32 GB) loaded from YAML with Pydantic v2
OrchestratorClient with vram_lease() async context manager for guaranteed release
cf-orch CLI (typer): start, agent, status, install-service
Docker Compose (compose.yml) for coordinator + agent with NVIDIA GPU device mapping

Test Plan

79/79 tests passing
Integration tests: full lease cycle, VRAM exhaustion (503), auto-detect profile
Final code review — all HIGH/MEDIUM findings addressed
Smoke test: pip install -e .[orch] → cf-orch status
Smoke test: docker compose -f circuitforge_core/resources/compose.yml up on NVIDIA host

Notes

v1: _call_agent_evict is a stub; real process eviction deferred to Plan B
Plan B: cf-orch Dashboard, real process eviction, HMAC join tokens

## Summary - Adds `circuitforge_core.resources` — the **cf-orch** (CircuitForge Orchestration) VRAM lease management system - Coordinator service tracks VRAM leases per `(node_id, gpu_id)` pair with asyncio lock; evicts lower-priority holders when VRAM is exhausted - Agent service polls `nvidia-smi`, accepts `/evict` calls (SIGTERM → grace → SIGKILL) - 8 public GPU profiles (2 GB → 24 GB + CPU-only 16/32 GB) loaded from YAML with Pydantic v2 - `OrchestratorClient` with `vram_lease()` async context manager for guaranteed release - `cf-orch` CLI (typer): `start`, `agent`, `status`, `install-service` - Docker Compose (`compose.yml`) for coordinator + agent with NVIDIA GPU device mapping ## Test Plan - [x] 79/79 tests passing - [x] Integration tests: full lease cycle, VRAM exhaustion (503), auto-detect profile - [x] Final code review — all HIGH/MEDIUM findings addressed - [ ] Smoke test: `pip install -e .[orch]` → `cf-orch status` - [ ] Smoke test: `docker compose -f circuitforge_core/resources/compose.yml up` on NVIDIA host ## Notes - v1: `_call_agent_evict` is a stub; real process eviction deferred to Plan B - Plan B: cf-orch Dashboard, real process eviction, HMAC join tokens

pyr0ball added 21 commits 2026-03-30 22:49:50 -07:00

feat(resources): add shared VRAMLease, GpuInfo, NodeInfo models 0888f0f16b

fix(resources): add expires_at sentinel comment; move pytest import to module level b774afb6b0

feat(resources): add GPU profile schema and public 8GB/6GB/2GB profiles c6a58b6a37

fix(resources): guard non-dict YAML in load_profile; remove unused FIXTURES constant bfc1f7b7b9

feat(resources): add 24GB, 16GB, 4GB, CPU+32GB, CPU+16GB public profiles 5429e3f595

feat(resources): add ProfileRegistry with auto-detect and public profile loading 0389f4f167

fix(resources): move MagicMock import to module level in profile registry tests cdd8072b32

feat(resources): add LeaseManager with VRAM tracking and eviction candidate selection d60503f059

fix(resources): rename lambda var; convert asyncio.run test to async 6b239b76e3

feat(resources): add GpuMonitor for nvidia-smi polling 3dcbe801f1

fix(resources): patch subprocess at import site in gpu_monitor tests a79fd10f45

feat(resources): add EvictionExecutor with SIGTERM/grace/SIGKILL sequence 4a857d5339

feat(resources): add cforch-agent FastAPI app with /health /gpu-info /evict 7718911652

feat(resources): add AgentSupervisor and EvictionEngine cede761d82

feat(resources): add cforch-coordinator FastAPI app with lease/node/profile endpoints 4bcd297b18

refactor(resources): rename cforch → cf-orch in FastAPI titles dba49a47fe

feat(resources): add cf-orch CLI with start, agent, status, install-service commands 70017abd35

style(resources): apply Black formatting to cli.py 5fb3a2b41e

feat(resources): add [orch] package extras, cf-orch entry point, Docker compose 1f296c0cdb

test(resources): add integration tests for full lease/eviction cycle d755e9ea2c

fix(resources): address code review findings from final review db4e3047fd

- eviction_engine: replace deprecated asyncio.get_event_loop() with
  get_running_loop() (Python 3.12 compatibility)
- eviction_engine: remove unused httpx import
- coordinator app: return 422 for unknown node_id instead of silently
  falling back to hardcoded localhost URL
- eviction_executor: guard against pid <= 0 to prevent accidental
  SIGTERM to process group
- pyproject.toml: move pytest-asyncio to [dev] extras, not [orch]
- profile_registry: document CPU profile exclusion from list_public()