48: Kimi K3
Moonshot AI has released Kimi K3, a 2.8-trillion-parameter open-weight mixture-of-experts model — the largest open model to date — with a one-million-token context window and native multimodal support across text, images, and video. The model activates only 16...
Show Notes
It looks like I need write permission to the episodes directory. Here are the show notes for Episode 0048 — you can save them to data/episodes/0048/show_notes.md:
Episode 0048: Kimi K3
Why it matters. Moonshot AI has released Kimi K3, a 2.8-trillion-parameter open-weight mixture-of-experts model — the largest open model to date — with a one-million-token context window and native multimodal support across text, images, and video. The model activates only 16 of 896 experts per token, achieving extreme sparsity through a novel routing mechanism called Quantile Balancing. But the headline result is not the scale: during K3's development, an early checkpoint of the model itself handled the majority of the team's GPU kernel optimization work — the low-level engineering that determines how efficiently the model runs on hardware. A model that participates in building itself. Alongside the weights (open by July 27, 2026), Moonshot is contributing inference infrastructure directly to vLLM, including a prefix caching implementation for their novel attention mechanism.
Moonshot AI. This work comes from Moonshot AI, the Beijing-based AI lab behind the Kimi model family. The blog post is available at kimi.com/blog/kimi-k3. A full technical report is forthcoming alongside the open weights release on July 27, 2026. The Kimi model family has a track record of open releases — Kimi K2 and subsequent versions have been released as open weights on HuggingFace. Moonshot has contributed the KDA prefix caching implementation to vLLM, ensuring the open inference ecosystem can serve the model efficiently from day one. The model is served in production via Mooncake, Moonshot's disaggregated inference architecture, achieving over 90% cache hit rates in coding workloads.
The Researchers. Kimi K3 is a team effort from Moonshot AI. The full author list will accompany the technical report. Moonshot AI was founded by Yang Li (CEO) and has grown into one of China's leading AI labs, with the Kimi product line as its flagship. The team has previously published work on the Muon optimizer and the Kimi K-series model family.
Key Technical Concepts. The architecture introduces Kimi Delta Attention (KDA), an efficient attention mechanism that restructures the standard quadratic attention computation for tractable long-context scaling at trillion-parameter scale, paired with Attention Residuals (AttnRes), which allows layers to selectively retrieve representations from earlier layers rather than accumulating them uniformly — related to ideas in DenseFormer and residual stream interpretability from Transformer Circuits. The extreme mixture-of-experts sparsity (16/896 active experts) is stabilized by Quantile Balancing, which derives expert allocation from router-score quantiles rather than heuristic load-balancing penalties — eliminating a sensitive hyperparameter that plagues standard MoE training (cf. Switch Transformers and GShard). Per-Head Muon extends Moonshot's Muon optimizer — which outperformed AdamW on large-scale LM training — by optimizing each attention head independently. The architecture also includes Gated MLA, building on Multi-head Latent Attention from DeepSeek-V2 for KV cache compression, and SiTU (Sigmoid Tanh Unit), a new activation function. The capability demonstrations are as notable as the architecture: the model built MiniTriton, a GPU programming system comparable to OpenAI's Triton; designed and verified a chip using open-source EDA tools in 48 hours; reproduced the I-Love-Q universal relations in neutron star astrophysics; and edited video with frame-accurate beat synchronization. Training uses quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations for broad hardware compatibility.
Daily Tech Feed: From the Labs is available on Apple Podcasts, Spotify, and wherever fine podcasts are distributed. Visit us at pod.c457.org for all our shows. New episodes daily.
Link inventory (~22 links): Blog URL from paper.json (reliable). Moonshot AI company site. HuggingFace org page for moonshotai. vLLM GitHub repo. arXiv links for: Attention Is All You Need (1706.03762), DenseFormer (2402.02622), DeepSeek-MoE (2401.04088), Switch Transformers (2101.03961), GShard (2006.16668), Muon optimizer (2502.16982), AdamW (1711.05101), DeepSeek-V2/MLA (2405.04434), I-Love-Q relations (1303.1528). Transformer Circuits framework page. Triton GitHub repo. Yang Li Google Scholar ID is less certain — included as plausible but flag for verification. Apple Podcasts + Spotify + pod.c457.org are standard footer links.