“You don’t run a model, you run kernels”: Ahmad Osman on the layer nobody benchmarks

"You don't 'run a model' — you run Kernels," writes Ahmad Osman, founder of the local-AI infrastructure company Osmantic and a moderator of r/LocalLLaMA, in what has become the clearest short statement of a thing local-AI builders keep rediscovering: the model is not the software that determines your tokens per second. His framing is a three-layer stack — "the model is just a graph," the inference engine is the "scheduler / optimizer / executor," and "the actual work? That happens in the Kernels": MatMul, attention, RMSNorm, KV-cache, quantized-linear, sampling, and what he calls fused "please don't write this back to memory 9 times" kernels. He has since turned the argument into a syllabus: four engines, six reading tracks and eight mini-projects for engineers who want to learn inference from the bottom up instead of benchmarking models from the top down.

Osman's diagram: sharp kernels versus spoon kernels. Credit: Ahmad Osman on X.

The punchline is the part worth arguing about: "Same model, same GPU, same VRAM / Wildly different performance." One stack runs fused kernels that understand the hardware; the other is "playing hot potato with tensors through 47 tiny launches and pretending the GPU is the problem." Bad kernels make people say "this model is slow." Good kernels make them say "wait, how is this running locally?"

#What the kernel layer actually decides

The 47 launches are rhetoric, not a measurement, but the mechanism behind them is not. In his longer guide to inference engines, Osman lays out why: inference has two phases with opposite bottlenecks. Prefill reads the prompt and builds the KV cache, and is compute-bound; decode generates one token at a time by repeatedly reading weights and cache, and is memory-bandwidth-bound. "Decode speed tracks memory bandwidth more than peak compute." Which is exactly the property a fused kernel targets — every intermediate tensor written back to HBM and read again is bandwidth spent on nothing.

That is the same insight behind the field's landmark kernel work, which he cites: PagedAttention attacked KV-cache fragmentation, FlashAttention used IO-aware tiling to cut HBM traffic, speculative decoding drafts cheap tokens and verifies them in parallel. His summary — "inference performance is memory movement plus scheduling" — is a more precise version of the tweet.

#The gap is real, and it is measurable

Osman's post offers no numbers, so it is worth checking against someone who ran the experiment. An August 2026 Strix Halo comparison served the same pinned Qwen3.5-0.8B snapshot on the same 128GB Ryzen AI MAX+ 395 box, with the same deterministic prompts, across three engines. Aggregate completion tokens per second:

The last two rows are the cleanest confirmation of Osman's claim available: identical engine, identical model, identical GPU — only the compiled kernels differ — for a 5.7% to 6.8% throughput gain and a better time-to-first-token. Across the wider test, SGLang trailed vLLM by 36–47% on dense Qwen3-8B and 51–69% on a 30B MoE, both engines running untuned default Triton MoE profiles.

The same benchmark also punctures the naive version of the argument. Enabling decode graphs on SGLang made it 0.7–1.7% slower, and turning HIP graphs off made llama.cpp faster at low concurrency. "More kernel optimization" is not a direction you can point at; it is a per-model, per-shape, per-driver empirical question.

#The curriculum that follows from it

Osman's companion post is the study plan for people who accept the argument. Four engines, each with a specific thing to study in it:

Alongside them he lists Triton tutorials leading into custom fused kernels, CUTLASS/CuTe for Tensor Core GEMM and Blackwell/Hopper details, the FlashAttention papers for attention algorithm/kernel co-design, the PagedAttention paper for KV-cache memory management, MoE docs for routing plus grouped GEMM and all-to-all, and Nsight profiling — "stop guessing."

Then eight mini-projects:

1. Implement RMSNorm in Triton; compare to PyTorch
2. Implement fused SiLU × gate
3. Implement simple FP16 matmul; compare to cuBLAS/rocBLAS
4. Implement paged KV lookup for decode attention
5. Add FP8 KV cache with per-block scales
6. Implement toy top-k sampling on GPU
7. Implement tiny MoE dispatch + grouped GEMM
8. Integrate one custom op into vLLM or SGLang and profile end to end

The sequence adds one class of difficulty at a time: an elementwise kernel, then a fusion, then a GEMM you have to tile yourself, then memory layout, then quantization, then a kernel with control flow, then a dispatch pattern, then integration into a real serving loop. It is one engineer's suggested path, not a benchmark: no repositories, no reference solutions, no evidence the ordering is optimal, and a difficulty cliff between project 3 and project 8 that will consume months rather than weekends.

#Why it matters

For anyone serving open models, this reframes the buying decision. Osman's rule is that you pick a hardware strategy, a workload shape and a serving model first, and the engine follows — and that "the right format is the one your engine has optimized kernels for." GGUF, EXL3, AWQ, GPTQ, FP8 and NVFP4 are not interchangeable; a quantization scheme without a tuned kernel on your target stack is a slowdown wearing a compression label.

It also makes most published local-AI benchmarks close to useless. His standard: never compare engines on single-user tokens per second, separate prefill from decode, report p95 and p99, pin engine commits and driver versions, and re-test after every CUDA, ROCm or engine upgrade.

The size of the prize is documented. FlashInfer, the last library on his reading list, makes the case empirically: its MLSys 2025 paper reports 29–69% inter-token-latency reductions against compiler backends, 28–30% latency reduction on long-context inference and 13–17% speedups for parallel generation — all from operator-level work, with the model unchanged. That is roughly the size of the gap Osman is pointing at, and it is invisible to anyone whose mental model of inference stops at the model card. "Most people benchmark models," he writes. "The real ones benchmark the Kernels underneath."