Agent-generated kernels cut Qwen-Image serving latency 42.3%

An agentic kernel-development framework cut end-to-end latency by 42.3% on Qwen-Image and 15.2% on FLUX.2 in a live serving stack, according to "Agentic Kernels in Production," an article by Brian Li, who works on model performance at Baseten. The models were served with SGLang on NVIDIA B300 GPUs, and Li writes that the optimizations behind those numbers — the profiling, the proposals, the kernel code and the integration into the engine — were "identified, proposed, and implemented entirely by our agentic framework."

Per-step denoise time by configuration. Qwen-Image FP8 falls from 245.6 ms to 141.8 ms; FLUX.2 FP8 from 137.1 ms to 116.2 ms. Credit: Brian Li, "Agentic Kernels in Production".

#The gap between KernelBench and production

Li's premise is that agents are already good at writing kernels — benchmarks like KernelBench measure that — and that this is not the hard part. Four things break on the way to a deployment. The winning kernel configuration depends on the production workload, because tile shapes, warp-specialization strategy and CTA configuration respond differently to real tensor shapes, batch sizes and sequence lengths. A faster microbenchmark does not imply a faster model, since CUDA graph capture and multi-stream execution can erase kernel-level gains or turn them into regressions. Optimizing kernels one at a time misses the cheaper structural wins — fusing operations, deleting redundant work, removing pipeline bubbles. And wiring a kernel into a serving engine, rather than a standalone torch model, means slotting it into interconnected execution paths without breaking the runtime around it.

#Two layers and a memory

The framework's architecture. Credit: Brian Li, "Agentic Kernels in Production".

The stack splits in two. A model-level layer profiles the whole workload, finds where time goes, and proposes restructuring — fusion, removing redundant work, reducing intermediate materialization. A per-kernel layer takes the generated and performance-critical kernels from the trace, explores several implementations in parallel, and iterates on the strongest candidate. Kernels that pass correctness and end-to-end checks are kept as reusable candidates, and lessons from both wins and dead ends go into an evolving knowledge base alongside workload constraints and integration findings, so each run starts from accumulated experience.

#What the agent actually changed

Three optimizations applied to both models:

The prepacking change stops the FP8 path from repeatedly repacking constant weight scales into DeepGEMM's format through sequences of small launches; scales are packed at model load time instead, and Li notes outputs stay bit-identical. The QKV fusion merges three FP8 projections into one GEMM and folds bias, QK normalization, RoPE and the buffer writes into a single Triton epilogue.

Then the model-specific work. On Qwen-Image, two standalone bias additions were eating roughly 11% of FP8 step time; folding each into the next fused operation took 5.2% off. A classifier-free-guidance modulation cache — reusing the timestep-only image and text modulation branches across the conditional and unconditional passes, which are identical at the same timestep — took another 2.1% (FP8) and 3.1% (NVFP4). On FLUX.2, a Python contiguity guard had been keeping the single-stream block off the production fused QK-normalization-and-RoPE kernel; the replacement is 2× faster and eliminates 48 of 60 cosine/sine cache concatenations per step, worth 2.3% FP8 and 4.0% NVFP4. A fused SwiGLU-plus-quantization kernel added 2.3%/3.8%, and a gated residual-normalization fusion 1.2%/2.3%.

#Why it matters

Baseten published its own human-engineered numbers for the same two models in May — 2.3x on FLUX.2 [dev] and 1.6x on Qwen-Image over SGLang on B200s, via quantization, attention kernels and runtime work. The claim here is that an agent found another 42.3% on top of a stack that had already been tuned by a performance team, which is a different proposition from topping a kernel leaderboard.

Li says the framework is model- and engine-agnostic and that LLM work has started, where kernels are more mature and headroom is thinner: early results show up to 5.5% higher tok/s on MiniMax M3 and GLM-5.2 running on vLLM. The end state he sketches is per-deployment kernels that keep evolving toward whatever the real traffic looks like.

The caveat is a large one: these are first-party numbers from a vendor's own stack, published without a reproducible harness, traffic profile or third-party check, and the per-kernel speedup table is printed identically for both models, which reads like a copy-paste. What can be checked from the charts is at least internally consistent — 245.6 ms to 141.8 ms per denoise step is 42.3%.