vLLM Is 39% of the Speed of Light. PyTorch Con Is About the Rest

Cohere published a serving note that is more useful than another “we fused the kernel” thread. For their North Mini Code model at 8K context, an H100’s HBM bandwidth is 3.35 TB/s. They put the speed-of-light ceiling at about 470 tokens per second. vLLM, on that setup, delivers 185 tok/s. That is 39% of the ceiling. The rest of the post is Cohere’s megakernel engine trying to eat the other 61%.

PyTorch’s homepage is already advertising the October 20–21 North America conference in San Jose. The vLLM session list is not subtle: KV cache, disaggregated serving, hardware portability, kernel optimization, MoE inference, attention, production serving. Nicolò Lucchesi (Mistral, vLLM maintainer) is on the bill with Amazon Web Services. Red Hat’s vLLM Omni lead is on it too.

We already ran vLLM versus TensorRT-LLM versus ONNX Runtime. That was a chooser. This week is a measurement. 39% is not an insult. It is the number that tells you what “fast enough” still leaves on the table.

What 39% actually means

Speed of light here is not marketing. It is “if you were limited only by HBM, how many tokens could you push.” Real serving is never only HBM. You pay for kernel launch, for attention that is not perfectly fused, for sampling, for the CPU side of the scheduler, for fragmentation in the KV cache.

185 versus 470 is a gap you can feel on a coding model. It is also a gap that might not matter on a 8B chat bot behind a 50-ms network. If your product is a coding agent that streams into a terminal, 185 tok/s is the difference between a colleague and a spinner. If your product is a batch classifier, you care about throughput at high concurrency, which is a different curve than Cohere’s single-stream SoL.

Cohere’s point in building a megakernel is that a pile of small CUDA graphs still leaves money on the floor. vLLM’s point, historically, is that paged attention and a good scheduler beat a custom engine that only the authors can compile. Both can be true. The 39% figure is the first one you can quote without joining either church.

Do not treat 185 as “vLLM is slow.” Treat it as “this model, this context, this GPU, this stack.” Change the model and the percentage moves. Change the batch and the percentage moves. Change from H100 to whatever you actually rent and the percentage becomes a different religion.

Why the conference agenda looks like a serving conference

PyTorch Con NA is still a training conference in the program description: compiler, distributed, device portability, CI. The vLLM block is the tell that the Python ML job has moved. People who used to file issues about DataLoader workers now file issues about KV blocks.

Disaggregated serving is the session title that will age well. Prefill on one set of GPUs, decode on another. vLLM has been living in that design space. If you still run a single replica that does both, October is when your architecture starts to look old in hallway chat.

Hardware portability is the other title that is not optional. Nvidia buying Hugging Face already made the “we might want a second vendor” conversation less theoretical. vLLM talking TPU and non-CUDA backends in public is how Python teams keep a door open. SemiAnalysis’s TPU inference note from this period puts SGLang-JAX on one side of that door and vLLM’s TorchAX backend on the other. You do not need to pick a TPU this week. You need to know the serving engine you standardized on has a story when the GPU quote comes back stupid.

What to do in Python this month, not in October

If you already serve with vLLM, log tokens per second the way Cohere did: one model, one context length, one GPU class, decode-only. Put the SoL number next to it even if you have to compute SoL yourself from HBM and bytes per token. A dashboard without a ceiling is how teams celebrate a 10% win that was 3% of the remaining gap.

If you are still on Transformers generate() in a FastAPI worker, stop using this article as trivia. Move. vLLM or SGLang. The 39% conversation is a luxury for people who already left the naive path.

Pin versions. PyTorch’s site currently lists stable as 2.13.0, Python 3.10 or later. Serving stacks break on silent CUDA wheels more often than they break on research code. Whatever you measure this week, write the wheel versions in the same note as the tok/s.

Do not rewrite your engine because Cohere’s megakernel looks pretty. Megakernels are a research and production bet for a lab that owns the model. vLLM is the engine your intern can upgrade. The conference sessions are how that intern’s upgrade path stays funded.

Training people still have a job

Machine-learning on this site is not only inference. Fine-tunes still happen. The serving number changes the fine-tune, though. If decode is the product, you start caring about KV size, about whether your architecture is MoE, about whether you trained a model whose attention pattern is hostile to paged cache.

The vLLM attention and MoE sessions in San Jose are for people who train. If your next run is “same architecture, bigger.” that is a serving tax you will pay every day after the checkpoint lands. If you can change the architecture, the 61% is a design input, not a compiler ticket.

Paged attention made a class of models cheap to serve. The remaining class is the one Cohere is trying to fuse into one kernel. You probably do not work at Cohere. You probably do work on a model that is closer to Mini Code than to a 7B instruct toy. Measure it.

A boring experiment that beats a blog table

Take the model you actually ship. 8K context if that is your default, not because Cohere used 8K. One H100 if you have one, or whatever GPU is on the invoice. vLLM defaults. Then vLLM with the flags you were scared to touch: chunked prefill, prefix caching, a real max concurrency.

Write four numbers: single-stream tok/s, tok/s at the concurrency you run in prod, p95 latency, KV memory. If you cannot fill those four, you do not have an inference stack. You have a demo.

Compare only against yourself last month. Cohere’s 470 SoL is a ceiling for their byte math, not yours. Steal the method.

PyTorch Con will be full of graphs that look like this paragraph with better fonts. Go if you can. If you cannot, the homework is the same. 39% is a snapshot of a popular engine on one lab’s model. The useful part is admitting the other 61% exists, and deciding whether your product lives in the 39 or in the chase.

One more Python-specific trap. People will read 185 tok/s and try to “fix” it with torch.compile on the training box, or by switching dtypes on a model they do not own. Serving speed is mostly memory movement and kernel shape. If you did not write the attention, you are choosing flags, not writing CUDA. Read the vLLM docs for the version you pinned. Then stop.

If you are presenting to a VP in October, do not take Cohere’s 39% as your slide. Take your four numbers and one sentence: we are at X percent of a bandwidth ceiling we computed the same way. If X is 39, you are in famous company. If X is 15, you have a config bug. If X is 80, you should be suspicious of the ceiling math.

KV cache is where most teams actually lose the 61%. Not in a missing megakernel. In a max sequence they never trimmed, in prefix caching they never turned on, in a batch size that looks good in a notebook and fragments in production. The San Jose sessions on KV cache and disaggregated serving are aimed at that boredom. Prefill is a different hardware shape than decode. If you still run both on the same replica “because it is simpler,” measure the GPU idle during prefill. Simpler is often just an expensive queue.

Python still owns this stack. The engine is a pip package. The configs are YAML or CLI flags. The dashboard is whatever you already use for latency. You do not need a new language to chase the rest of the ceiling. You need a written SoL, a pinned vLLM, and a refusal to celebrate kernel folklore without a before/after on the model you ship.

If you go to San Jose, bring that before/after, not a question about which session to attend. The agenda already told you: KV cache, disaggregated serving, kernels, MoE. The hallway will be people arguing megakernels. Your job is to know whether your replica is starving on prefill, on decode, or on a queue you pretended was a model problem.

vLLM remains the default I would put in a new Python serving repo tomorrow. The Cohere note does not change that. It changes the conversation you have after the default is in production, when someone asks why the GPU is only half as fast as the brochure.