Question 1

How accurate is this?

Accepted Answer

The page computes from your hardware: output speed ≈ memory bandwidth ÷ model bytes read per second, times an efficiency factor (≈ 0.8 Apple, 0.65 NVIDIA, 0.6 AMD). Real runs usually land at 60–85% of the estimate; for an exact number, measure once on your machine with the LM Studio Panel and write "measured ÷ theoretical ceiling" back into the efficiency field.

Question 2

What is the efficiency factor? Why not 100%?

Accepted Answer

Efficiency is "what fraction of the theoretical ceiling real runs reach" — the page's only empirical number rather than a hard spec. The theory assumes every bandwidth byte is saturated, but real runs also read KV cache and activations, compute attention and sampling, and wait on engine scheduling — together dragging measurements to 60–85% of theory. Family defaults: Apple unified memory 0.8 (short decode path, more efficient); NVIDIA consumer cards 0.65 (CUDA backend, conservative); AMD 0.6 (Vulkan/ROCm backends vary widely); custom defaults to 0.7 (middle ground). Editable — measure tok/s once with the LM Studio Panel, write "measured ÷ theoretical ceiling" back, and later estimates fit your machine better. Example: M4 Max measures Llama-8B Q4 at 60 tok/s against a 70 ceiling → efficiency ≈ 60÷70 ≈ 0.86.

Question 3

Why does output speed follow memory bandwidth, not compute?

Accepted Answer

Generating a token (decode) does one "weights × vector" multiply per step — compute to spare; the bottleneck is moving tens of GB of weights from memory into the compute units each time. That's a bandwidth problem. Compute governs the prefill phase instead: the longer your prompt, the more it shapes time-to-first-token.

Question 4

Why do MoE models (e.g. Qwen3-30B-A3B) size memory at 30B but speed at 3B?

Accepted Answer

Every expert of a mixture-of-experts model lives in memory, but each token activates only a small expert subset. So memory follows total params while speed only reads the activated weights — exactly why MoE models "look big and run fast".

Question 5

What is KV cache? Why do longer contexts squeeze memory?

Accepted Answer

To "remember" context, the model stores per-layer attention intermediates — growing linearly with context length and with layer/KV-head counts. GQA (Qwen3-8B has just 8 KV heads) keeps KV far below classic multi-head attention; post-2026 hybrid-linear models (Qwen3.8) use full attention in only 1/4 of layers, shrinking KV further. Size KV first for long contexts, then pick the quant.

Question 6

Why are some models tagged "partial-layer KV" or "MoE"?

Accepted Answer

Two tags flag non-dense math. "MoE": memory at total params, speed at active params (Qwen3-Next-80B-A3B, gpt-oss). "Partial-layer KV": hybrid-linear or sliding-window architectures (Qwen3.8, gpt-oss, Gemma 3) where only some layers' KV grows with context and the rest use fixed-size states/windows — this page counts only the growing part, so their long-context footprint runs much smaller than same-size dense.

Question 7

What happens when it doesn't fit?

Accepted Answer

Discrete GPUs can spill weights into system RAM (CPU+GPU hybrid) — much slower but runnable; Apple unified-memory machines have no separate VRAM, so oversize means a smaller model or a lower quant. The page judges in four bands: ≤60% totally fine, ≤85% runs, ≤98% borderline, >98% doesn't fit (the ceiling is 98%, not 100% — the OS needs headroom).

Local LLM Speed Calculator

Your machine

Model

Why local-model speed is computable

Params × quants

The three numbers that decide "does it run"

The MoE "value illusion"