What actually runs on a 24 GB GPU

A 24 GB graphics card is an interesting size in local inference right now. It is one slot, one power cable or none at all, and it sits under a desk without a conversation about the electricity bill. It is also almost exactly the size where the useful open models stop fitting.

So I measured where the edge actually is, on one card, with one set of flags, over several weeks. What follows is what fit, what did not, how fast each one is when the context is empty and when it is nearly full, and which of the numbers turn out not to mean what they look like.

The card

Everything below ran on an NVIDIA RTX PRO 4000 Blackwell SFF Edition, the single slot variant.

The 70 W is not a setting. It cannot be raised:

# nvidia-smi -pl 75
Provided power limit 75.00 W is not a valid power limit which should be
between 60.00 W and 70.00 W for GPU 00000000:01:00.0

The card has no auxiliary power connector, so it draws only what the PCIe slot supplies. The full height RTX PRO 4000 Blackwell is a 140 W part; the difference is physical. That matters because it changes which model wins. On a card that is power bound rather than compute bound, the thing that decides throughput is active parameters per token, not total parameter count. A mixture of experts model that routes 3B of its 30B parameters per token does its work inside the envelope and draws about 52 W. A dense 27B pins the meter at 70 W and decodes at roughly half the speed.

Two constraints then, and they pull in different directions: 23.4 GiB of memory sets what can be loaded at all, and 70 W sets how fast it runs once loaded.

What fits

Every model below was served with vLLM at --gpu-memory-utilization 0.90, fp8 KV cache, eager mode, except Bonsai which needs llama.cpp.

Released is the first appearance of the base model on Hugging Face, not of the quantisation, which in several cases followed months later. Origin is who trained the base, not who published the file I ran: several of these quantisations come from independent contributors rather than the original lab.

What does not fit

Both loaded their weights and then had nothing left for a cache. The 31B names its own ceiling in the error: an estimated maximum model length of 2,512 tokens. A model you can only talk to in paragraphs is not deployable.

Qwen3.6 35B A3B is the frustrating one. It routes 3B active parameters per token, exactly the profile that wins on a power bound card, and both the NVFP4 format and the architecture are supported by the runtime. It fails on size alone. Every published quantisation currently sits between 21.8 and 23.7 GiB. A build at or below 18 GiB would be the most promising thing to try on this card.

The working ceiling is about 19 GiB of weights, measured twice rather than estimated once. Above that the model loads and then cannot serve.

Context: how much, and for how many people

Capacity comes free from the vLLM startup log, which reports the KV pool it managed to build. That single number gives the whole context and concurrency curve, because the pool is shared: max context per stream is the pool divided by the number of concurrent requests.

Gemma 4 12B holds four times the context of the Qwen3.8 27B while being half its size on disk. That is sliding window attention: most of its layers keep only a short window of keys and values, so the cache does not grow the way full attention forces it to.

Speed when the context is already full

Every throughput number published for a local model is measured on an empty context, and that is not how anyone uses one. An agent re-sends a growing transcript on every turn. So I measured decode speed with the context pre-filled to 15k, 30k, 45k and 60k tokens.

The models do not degrade the same way. They do not even preserve their ranking.

Ministral 3 14B is the fastest model in the set on an empty context and the second slowest by 60k. If you benchmark at zero depth and deploy into an agent loop, that is the mistake you make.

The two flat models are the two mixture of experts models, and I think the reason is that they are already bound by streaming expert weights rather than by reading the cache, so the growing cache costs them almost nothing on top. The steepest decline belongs to the one dense model in the set with conventional full attention over its whole context. Gemma 4 12B sits between the two, losing some speed but far less than Ministral, which is consistent with its sliding window keeping most of the cache small. That is an inference from five models on one card rather than a proof, but the grouping is clean.

Prefill behaves differently again, and here the spread is much wider:

A cold 60k turn costs six seconds on Gemma 4 26B A4B and twenty two on Qwen3.8 27B before a single token appears. Both engines cache prefixes, so in a steady conversation only the new tokens are prefilled and these numbers apply to the first turn or to a cache miss. I measured that gap by accident: one context that had already been prefilled returned first token in 0.11 s where the identical cold context took 15.77 s, about 145 times faster.

Does it matter what is in the context?

The obvious worry with a depth benchmark is that the filler is fake. If lorem ipsum behaves differently from real traffic, the curve above is decoration.

It does not. At exactly 30,000 tokens:

Under one percent apart on the dense model and about two on the mixture of experts one, which is inside the run to run noise. Decode at depth is bounded by streaming the cache back, and the cache does not care whether it holds Latin, English, Python or escaped JSON.

The control that makes this a real answer rather than a sloppy one is that all four were padded to the same token count, not the same character count. Those four texts tokenise at very different ratios: lorem ipsum is Latin and fragments badly, code is punctuation dense, tool call JSON is full of escaped quotes. Matched by length they would have differed by thousands of tokens, and the benchmark would have measured the tokeniser rather than the engine. Each filler was trimmed against the server's own tokeniser until it landed on the target.

So lorem ipsum is a fair stand in for production traffic, provided you count tokens.

Temperature zero, and why "deterministic" is not quite true

Temperature zero is greedy decoding: no sampling, take the highest scoring token every step. Same weights, same prompt, same winner.

What breaks it is arithmetic rather than sampling. vLLM batches requests continuously, so a request shares kernels with whatever else is in flight. Change the batch and matrix shapes change, a different kernel or split is chosen, and floating point addition is not associative, so the same numbers summed in a different order differ in the last bits. Usually invisible. But when the top two tokens are nearly tied, a last bit decides which wins, and the answers diverge from there.

Five sequential repeats and four concurrent ones per model, counting distinct answers:

Every dense model was bit stable across all nine runs. Both mixture of experts models were not, and Qwen3 Coder could not reproduce itself even one request at a time. Five models is a small sample, so treat that as a pointer rather than a law, but routing tokens to experts plainly adds a second way for batch composition to change what each kernel sees.

The two Qwen3.8 quantisations produced the identical output hash, which is worth stating plainly: INT4 and W4A16 of the same model are the same model. There is nothing to gain from sweeping that axis.

Making it actually deterministic

Three knobs, measured on Gemma 4 12B.

Temperature is free. Decode held at 40.7, 40.7, 40.6 and 40.6 tokens per second at temperature 0, 0.2, 0.7 and 1.0. Sampling is not where the time goes.

Determinism survives further up the range than expected. Temperatures 0.2 and 0.7 both returned a single distinct answer across nine runs. Only at 1.0 did a second variant appear, twice in five. On a short prompt the distribution is peaked enough that sampling rarely changes the winner, which is a property of the prompt as much as the setting.

A fixed seed restores determinism even at temperature 1.0. Passing seed: 42 collapsed both 0.7 and 1.0 back to one answer across all nine runs, at no cost.

VLLM_BATCH_INVARIANT=1 is the real fix. It swaps in kernels whose reduction order does not depend on batch shape, so identical input gives bit identical logits whether it runs alone or beside seven others. Gemma 4 26B A4B, which produced two answers under concurrency by default, became reproducible with it on. The cost on Gemma 4 12B was 37.4 tokens per second against 40.7, about eight percent.

It is not universally available. Enabling it on Qwen3 Coder 30B A3B, the least reproducible model in the set and therefore the one that needed it most, crashed the engine before it served a token:

RuntimeError: expected scalar type Half but found BFloat16

Disabling prefix caching changed nothing and cost nothing, because Gemma was already stable.

So: deterministic by default if the model is dense, fixable with a flag if it is a mixture of experts whose dtype the kernels support, and otherwise not fixable in this build at all.

What this does not measure

None of the above says whether a model is any good. Throughput, capacity and reproducibility are the properties you can measure cheaply and compare honestly across models; quality is not, and a number that looks objective usually is not.

The depth curves are single stream. They do not describe behaviour under concurrency, where the pool is divided and the scheduler starts making decisions. Tokens per second should also not be compared too confidently across rows, because these models tokenise differently and a token is not a constant unit of work.

And the whole exercise is five models on one card with one set of flags. The groupings are clean enough to act on and too small to generalise from.

The jokes

Somewhere in the middle of this I asked all of them for their best joke at temperature zero, mostly as a smoke test, and three returned the same scarecrow gag byte for byte across two quantisations and two inference engines. A fourth, trained in France with no shared ancestry, offered it as well. That says less about the models than about what is written on the internet under the heading of "best joke".

It got out of hand, so it lives on its own page: 24 professions and industries, five models, every answer precomputed.

Method

One model served at a time, since 24 GiB holds exactly one of these. Capacity read from the vLLM startup log. Depth measured by pre-filling the context to a target token count verified against the server's own tokeniser, then timing a short generation. Determinism measured by hashing completions and counting distinct values. Harness and raw output are in the repo alongside the earlier sweep this builds on.