Chinese labs converge on one architecture: 3:1 linear attention and a 2,048-token budget

Two Chinese labs shipped frontier open-weight models within a day of each other, and their configs read like near-copies. Z.ai released GLM-5.3-Flash, a 320B-parameter multimodal MoE with 18B active parameters under an MIT license; Alibaba's Qwen team released Qwen3.8-Flash-Next, a 125B model with 6B active parameters billed as a preview of the Qwen4 architecture. The two teams designed them independently and landed on the same recipe.

Elie Bakouch, who trains LLMs at Prime Intellect, put the pattern bluntly: "every chinese frontier model now uses linear attention (except deepseek)," they all use sparse attention with similar indexer and compression designs "(except kimi)," they all use "fancy" residuals — mHC, attention residual, gated residual — and "they all use Muon."

The two architectures side by side, with the shared components circled. Credit: Elie Bakouch on X.

#The shared recipe

Three of every four attention layers are linear in both models. Linear layers compress all history into a fixed-size recurrent state instead of a KV cache that grows with the text, so per-token compute stays flat as context grows. The remaining quarter does precise long-range retrieval — and neither model lets those layers see the whole context. Both attach a small learned indexer that compresses history 4× before scoring it, then caps attention at 2,048 tokens, the pattern DeepSeek introduced as DSA in V3.2-Exp.

Both also abandon the single residual stream transformers have carried since 2017, widening it into four gated branches. Z.ai uses mHC (Manifold-Constrained Hyper-Connections), a DeepSeek design; Qwen wrote its own Gated Residual, ablated it against mHC and reported the two roughly equal. Both train with Muon, and both apply the same refinement of splitting fused projection matrices into their independent transformations before Muon orthogonalizes them.

The efficiency payoff is the point. Z.ai reports the Flash architecture cuts attention computation 3.01× and KV cache 4.44× versus GLM-5.3, while nearly halving both active parameters (18B vs 32B) and layer count (45 vs 92); it lists the model at $0.15 per million input tokens and $0.50 output, with a 1M-token context. Qwen says training Qwen3.8-Flash-Next took roughly one-ninth the compute of Qwen3.7-Plus, and credits QSA with up to 7.6× prefill and 4.9× decoding speedups at 1M tokens.

#Where they split, and who dissents

The clean disagreement is positional encoding. GLM-5.3-Flash drops rotary embeddings in its sparse layers entirely (qk_rope_head_dim = 0), letting position flow implicitly through the recurrent linear layers. Qwen tried NoPE too and kept RoPE: per its technical report, NoPE showed no measurable difference during pre-training, but after post-training the variant often failed to stop generating — a useful reminder that loss curves can hide defects that only surface after tuning.

The convergence is also a story about borrowing. KDA came from Moonshot's Kimi Linear; the indexer pattern and mHC came from DeepSeek. Kimi K3 uses linear attention without the sparse indexer; DeepSeek uses compressed sparse attention without linear layers. The real dissenter is MiniMax, which tested linear and sliding-window attention at scale during M2 development and found severe deficits in multi-hop reasoning past 32K context after SFT. M2 shipped full softmax attention in every layer, and M3 uses MiniMax Sparse Attention — block-sparse softmax, no linear layers at all.

#Why it matters

Open weights plus published technical reports mean an architecture idea now propagates across labs in weeks rather than model generations, and the shared destination is a cheaper long-context transformer rather than a bigger one. Whether the trade holds is the open question: MiniMax's ablations say a 3:1 linear hybrid costs reasoning, Z.ai's and Qwen's say it does not, and both sides are arguing from internal evidence.

Z.ai's own evaluation puts GLM-5.3-Flash at 84.3 on Terminal Bench 2.1 against 85.0 for Claude Opus 4.8 and 87.4 for GPT-5.6 Terra. Vendor-reported, with harnesses that differ per benchmark. Credit: Z.ai.

Those numbers are the lab's own, and the independent read is more cautious: Artificial Analysis scores GLM-5.3-Flash 57 on its Intelligence Index — strong intelligence per dollar, slow and verbose — with vision trailing Gemini 3.7 Flash. Semi Fundamental's field guide to Chinese labs frames the whole architectural push as an explicit trade: cheaper tokens for slightly worse capability on the hardest tasks.