Circuit-Forge/cf-node-bootstrap
6
0
Fork 0
Code Issues 10 Pull requests Projects Releases Packages Wiki Activity Actions

Add GPU capability checks + per-branch conda env setup to install menu #1

New issue
Open
opened 2026-07-18 09:52:45 -07:00 by pyr0ball · 0 comments
pyr0ball commented 2026-07-18 09:52:45 -07:00
Owner
Copy link

Problem

Discovered while dispatching a transcription/diarization job across the cluster:

  • Nodes report GPU presence but not driver/CUDA compatibility. Heimdall's cf-voice env has torch built for CUDA 13, but the installed driver (560.35.05) only supports up to ~CUDA 12.6 — torch.cuda.is_available() silently returns False and falls back to CPU with no warning to the user.
  • Muninn (RTX 3090, most VRAM in the cluster) has no conda, no ffmpeg, and no NFS mount configured yet — not usable for dispatch despite having the best hardware.
  • Sif has conda but is missing ffmpeg and lacks a matching env for GPU/audio workloads.

Ask

  1. GPU capability check in the install menu: detect driver version vs. installed/about-to-install torch CUDA build, and warn/block on mismatch instead of silently falling back to CPU.
  2. Per-cf-core-branch conda env setup: each cf-core branch (or dependent product needing GPU/torch) should get its own conda env during bootstrap, rather than sharing one env across mismatched CUDA requirements (this is what caused the immediate issue — pyannote.audio pulled in an unpinned newer torch and silently broke CUDA availability on cf-voice-like envs).
  3. Extend the node bootstrap checklist to include: conda install, ffmpeg, NFS mount verification, so nodes like Muninn can actually be dispatched to.

Context

Muninn is intended as a lead dispatch node for cf-orch (per infra notes) but is currently not bootstrapped enough to receive jobs.

## Problem Discovered while dispatching a transcription/diarization job across the cluster: - Nodes report GPU presence but not driver/CUDA compatibility. Heimdall's `cf-voice` env has torch built for CUDA 13, but the installed driver (560.35.05) only supports up to ~CUDA 12.6 — `torch.cuda.is_available()` silently returns `False` and falls back to CPU with no warning to the user. - Muninn (RTX 3090, most VRAM in the cluster) has no conda, no ffmpeg, and no NFS mount configured yet — not usable for dispatch despite having the best hardware. - Sif has conda but is missing ffmpeg and lacks a matching env for GPU/audio workloads. ## Ask 1. **GPU capability check in the install menu**: detect driver version vs. installed/about-to-install torch CUDA build, and warn/block on mismatch instead of silently falling back to CPU. 2. **Per-cf-core-branch conda env setup**: each cf-core branch (or dependent product needing GPU/torch) should get its own conda env during bootstrap, rather than sharing one env across mismatched CUDA requirements (this is what caused the immediate issue — pyannote.audio pulled in an unpinned newer torch and silently broke CUDA availability on cf-voice-like envs). 3. Extend the node bootstrap checklist to include: conda install, ffmpeg, NFS mount verification, so nodes like Muninn can actually be dispatched to. ## Context Muninn is intended as a lead dispatch node for cf-orch (per infra notes) but is currently not bootstrapped enough to receive jobs.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: Circuit-Forge/cf-node-bootstrap#1
No description provided.
Delete branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?