eval: LLMcalc VRAM estimation logic — reference for hardware module model-fit checks #64

New issue

Closed

opened 2026-06-15 12:16:58 -07:00 by pyr0ball · 0 comments

pyr0ball commented

2026-06-15 12:16:58 -07:00

Owner

Source: https://github.com/Raskoll2/LLMcalc (fork: TheAhmadOsman/LLMcalc)
License: NOT SPECIFIED — cannot copy code; reference only
Stack: Python CLI

What it does

Takes a HuggingFace model ID, fetches param count, calculates VRAM requirements at every quantization level (fp16, int8, int4, etc.), detects local GPU VRAM and RAM, estimates tokens/sec throughput. Supports -b (bandwidth), -n (GPU count), -v (VRAM) overrides.

Why relevant for cf-core

cf_core.hardware detects available VRAM but doesn't cross-reference model requirements. The missing capability: "can this hardware actually run model X at quality level Y?"

LLMcalc's formula is the reference implementation. We cannot copy the code (no license) but the algorithm is standard:

# Implement cleanly in cf_core.hardware
vram_gb = (params * bytes_per_param(quant)) + kv_cache_gb(ctx_len, heads) + overhead_gb
fits = vram_gb <= available_vram_gb

Application points

Avocet preflight: verify a checkpoint fits available VRAM before benchmarking
cf-orch worker assignment: match model to GPU worker based on VRAM fit at requested quant level
Peregrine/Kiwi onboarding wizard: "your GPU has X GB — here are models that will run well"

Action

Implement cf_core.hardware.model_vram_estimate(hf_model_id, quant_level) using HuggingFace API for param count and the standard formula. LLMcalc is the reference for the formula, not a dependency to add.

**Source:** https://github.com/Raskoll2/LLMcalc (fork: TheAhmadOsman/LLMcalc) **License:** NOT SPECIFIED — cannot copy code; reference only **Stack:** Python CLI ## What it does Takes a HuggingFace model ID, fetches param count, calculates VRAM requirements at every quantization level (fp16, int8, int4, etc.), detects local GPU VRAM and RAM, estimates tokens/sec throughput. Supports `-b` (bandwidth), `-n` (GPU count), `-v` (VRAM) overrides. ## Why relevant for cf-core `cf_core.hardware` detects available VRAM but doesn't cross-reference model requirements. The missing capability: "can this hardware actually run model X at quality level Y?" LLMcalc's formula is the reference implementation. We cannot copy the code (no license) but the algorithm is standard: ```python # Implement cleanly in cf_core.hardware vram_gb = (params * bytes_per_param(quant)) + kv_cache_gb(ctx_len, heads) + overhead_gb fits = vram_gb <= available_vram_gb ``` ## Application points - **Avocet preflight:** verify a checkpoint fits available VRAM before benchmarking - **cf-orch worker assignment:** match model to GPU worker based on VRAM fit at requested quant level - **Peregrine/Kiwi onboarding wizard:** "your GPU has X GB — here are models that will run well" ## Action Implement `cf_core.hardware.model_vram_estimate(hf_model_id, quant_level)` using HuggingFace API for param count and the standard formula. LLMcalc is the reference for the formula, not a dependency to add.