OpenAI circuit-sparsity: Are MoEs a dead end?

OpenAI’s research into circuit-sparsity suggests that Mixture of Experts (MoEs) are a crude approximation of sparsity used to appease hardware kernels and may be a dead end.