Why Kimi Built a World-Class LLM｜Decoding Zhilin Yang's GTC Talk

Hello, this is Ryuta Hamamoto from TIMEWELL.

In 2026, the phrase "open-weight, yet world-class" started to travel with the name of one Chinese startup: Moonshot AI (月之暗面), the maker of Kimi. Kimi K2.5 stands shoulder to shoulder with proprietary frontier models on many benchmarks, and the 2.8-trillion-parameter Kimi K3 released in July is described as top-tier in coding and agentic tasks.

What caught my attention is that all of it ships in a form where you can download the weights wholesale. It is not a black-box API. Anyone can load it onto their own servers. And yet the performance is world-class. What exactly are these people doing?

A large part of the answer, close to the core, was laid out at NVIDIA GTC 2026 in March. The speaker was Moonshot AI's co-founder and CEO, Zhilin Yang, in person. The session was titled "How We Scaled Kimi K2.5." In this article I will follow that roughly 39-minute talk from the top and put "why Kimi is strong" into technical words. I explain the jargon as it comes up, so readers who are not deep into the internals of AI will not be left behind. At the end, I share what I took away from this talk, in the context of AI adoption at Japanese companies.

From NVIDIA GTC 2026, "How We Scaled Kimi K2.5" (Zhilin Yang). All slide images below are from the same session.

The answer, up front: they changed how they scale, along three axes

Let me give the conclusion first. The source of Kimi's strength, as Yang framed it in this talk, comes down to just three axes: token efficiency, long context, and agent swarms. He called it "Kimi's Scaling Strategy."

Almost all recent progress in AI can be explained by scaling. Increase the training data (tokens), increase the model parameters, increase the compute, and performance improves. Ever since Kaplan et al. described the "scaling laws" in 2020,¹ the industry has believed in this law and kept getting bigger. What I find interesting about Yang is that he goes one step beyond this plain "just make it bigger" scaling and designs three separate axes along which to scale.

Three pillars: Token Efficiency, Long Context, Agent Swarms — each acting on a different layer (training, inference, execution).

Yang translated all three into the language of agents. This is the crux of his thinking, so let me be careful here. Token efficiency means extracting more intelligence from the same data. In agent terms, it means giving the agent a stronger prior, which makes its trial-and-error search more efficient. Long context means increasing the amount of information the model can handle at once. This lets an agent "keep running for days or even weeks." The third, agent swarms, is a new learning paradigm where many agents run at the same time and split up a complex task.

In short, his goal is clear. Build a swarm of agents, each with a very long context and a strong prior, and optimize the whole thing with reinforcement learning. The three axes are independent levers toward that end. Let me take them one at a time.

If you want to map out how far your own organization can actually use frontier AI, taking the AI literacy check first will make the later analysis feel like it is about your own company.

Axis 1: Token efficiency — running the "forbidden" Muon at a trillion parameters

Let me start with the first axis, token efficiency. Personally, this was the part of the talk that hit me hardest.

For a long time, training large language models has used the Adam optimizer (more precisely AdamW) as the industry standard. Adam was proposed by Kingma and Ba in 2014,² and for over a decade it has held the position of "just use Adam and you won't go wrong." An optimizer is the heart of training that decides how the model's weights get updated.

Yang's team touched this. They used a newer optimizer called Muon. It originally came from Keller Jordan, who published it on a personal blog in December 2024.³ The idea is to take the weight matrices of a neural network's hidden layers and orthogonalize them with a method called Newton-Schulz iteration before updating. It sounds hard, but roughly it means "cleaning up the direction of the update before you step," which yields lean training. Strictly speaking it is not a second-order optimizer per se; it is best understood as a matrix-orthogonalization method.

The problem is that Muon was thought to "work on small models but fall apart when you scale up." Moonshot took this head-on and published a paper in February 2025 titled "Muon is Scalable for LLM Training."⁴ The conclusion is striking: applied to large-scale training, Muon reaches the same performance with about 2x less compute than AdamW. The keys to scaling are two: add weight decay, and match the per-parameter update scale to Adam. That alone lets you scale up without re-tuning hyperparameters.

The Muon slide. Weight decay, consistent RMS updates, and a distributed Muon implementation are the keys to scaling.

Here is the point Yang stressed, and I love it. He said, "Token efficiency is not just about efficiency. It is about raising the ceiling of intelligence itself." The reason is this. The amount of high-quality data in the world is finite, and we are already hitting the "data wall." Suppose high-quality tokens top out at 5 trillion. If token efficiency doubles, it is as if you now have 10 trillion tokens. Because data is finite, improving efficiency directly improves the total amount of intelligence you can extract. It is not about cutting infrastructure cost; it is about pushing the frontier. That he is willing to say this is exactly why they invest so seriously in a mundane area like optimizers.

The "divergence" that bared its teeth at a trillion parameters, and QK-Clip

But when they tried to scale Muon further, they hit a new wall: training instability.

When Muon runs on large models, the attention logits (the values that go into softmax) rapidly explode past 1,000.⁵ Normally these values sit between a few dozen and under 100. Jumping past 1,000 makes the training loss diverge and stop converging. It would be pointless to find an efficient optimizer that cannot be used at scale.

At scale, the max logit diverges past 1,000. On the right, the loss blows up.

Moonshot's remedy is QK-Clip, and it is rather elegant. For every attention head in the network, compute the max logit during the forward pass. Then, only for heads whose max exceeds a threshold, shrink the query and key projection weights so the values do not run away — "clipping" them. In formula terms, with threshold τ and maximum S_max, you just multiply the weights by γ = min(1, τ/S_max).

Algorithm 1: the Muon step plus a QK-Clip step equals MuonClip. Only the heads above the threshold get shrunk.

The interesting thing is that this clipping does not disturb convergence at all. The loss curves before and after clipping overlap exactly; you cannot see the difference. Yet the max logit alone rises to 100, sticks there, and after a certain number of steps naturally comes back down. The network itself finds a way to keep the maximum in a range where it does not run away. Muon plus this clip is what they call MuonClip. Using MuonClip, they succeeded in stably training a trillion-parameter model. That is Kimi K2.⁵

Kimi K2 is a MoE (mixture-of-experts) model with about 1.04 trillion total parameters, 32B active, selecting 8 of 384 experts, 61 layers, and a 128K context length. Pretraining runs to 15.5 trillion tokens, and thanks to MuonClip it completed with no loss spikes (sudden jumps in loss). It was released open-weight in July 2025, and the day after release it became the most-downloaded model on Hugging Face. In Yang's words, "the first example of large-scale Muon training in the history of machine learning" — a line that carried a quiet pride.

Axis 2: Long context — "folding" full attention out to 1M tokens

The second axis is long context. For an agent to keep running for days, it needs to handle that much memory. But full attention, the Transformer standard, has a weakness: the compute and memory cost balloon as context grows.

Yang started here with a little-known figure: a comparison of Transformers and LSTMs (the recurrent networks that once dominated machine translation). At the same parameter count and token count, a Transformer reaches a lower loss. That is why the Transformer became the de facto standard. Everyone knows that part. What he pointed to was a different axis. The further back a token's position in the context, the more a Transformer's loss keeps dropping, whereas an LSTM saturates after a certain length. In other words, a Transformer has the property of "getting smarter the longer the context it captures." Very long tasks — understanding an entire codebase, or writing a Linux kernel from scratch — do not work without this property.

So Moonshot built a new attention architecture called Kimi Linear, whose technical report came out at the end of October 2025.⁶ At its core is a linear attention mechanism called Kimi Delta Attention (KDA), which extends an existing method, Gated DeltaNet (GDN), with a "finer-grained forgetting gate."

Let me break this down. Many linear-attention methods control how much of the past to forget with a single scalar (one number) per head. That tends to go to extremes — "forget almost everything" or "keep almost everything" — and it cannot do the nimble thing of discarding only the unnecessary parts of a long context. KDA replaces that forgetting rate with a channel-wise diagonal gate (Diag(α)). Some channels forget slowly and hold long-term memory; others forget quickly and take in new information. That division of labor lifts the model's expressiveness a notch.

Kimi Linear mixes KDA and full attention (MLA) at 3:1, balancing efficiency and expressiveness.

The other trick is the mixing ratio. Kimi Linear alternates linear-attention (KDA) layers and full-attention (MLA) layers at a 3-to-1 ratio. All linear loses accuracy; all full is heavy. Three-to-one was the sweet spot between long-context capability and efficiency. On the implementation side it is built to compute chunk-wise in parallel while keeping the math exact rather than approximate.

The results are clear. Under the same training recipe, it beat full attention (MLA) on short-context, long-context, and reinforcement-learning tasks — this is a point Moonshot itself claims as "for the first time" at the top of the paper. In numbers, it cut KV-cache (the memory that holds past keys/values) usage by up to 75%, and raised decoding speed by up to 6.3x at a massive 1-million-token context. On the 128K-token RULER benchmark it scored 84.3, above full-attention MLA (81.3).

Left: Pareto-optimal on performance vs. decode acceleration. Right: the speed gap widens as context grows (up to 6.3x).

"Fold full attention, keep the cost of length down without losing intelligence." That is how they pushed the long-context axis. Incidentally, the KDA kernel, vLLM integration, and model weights are all open-sourced. Here too, the "open, yet frontier" stance is thorough.

Axis 3: Agent swarms — solving with a "swarm," not a single agent

The third axis, agent swarms, is where the thinking shifts. Until now, an AI agent has basically been one model working through a task step by step with tools. What Yang's team introduced is a way to solve a complex task that one agent cannot, using a "swarm" of many agents in parallel.

The blueprint is simple. An orchestrator (or main agent) sits at the center, spawns a group of sub-agents as needed, assigns them subtasks, and aggregates the returned results. Yang compared it to a human company. A CEO decomposes a task and distributes it to departments, moving the whole organization toward one goal. AI researchers, web developers, physicists, fact-checkers — agents with different roles advance their own parts in parallel.

The orchestrator uses create_subagent/assign_task to spawn role-specific sub-agents, run them in parallel, and aggregate the results (arXiv:2602.02276).

The effect is dramatic. Looking at execution time against task complexity, a swarm finishes in far less time than a single agent. Run 100 sub-agents, or more, and tasks that never finished in a realistic time now fit inside a window that produces real economic value. Parallelize the inputs to read hundreds or thousands of sources at once, write a 100-page literature review in parallel, run 10 kinds of data analysis at the same time. That kind of scaling becomes possible.

Three reward designs to run the swarm well

What is interesting is how they make the swarm smart with reinforcement learning. Kimi K2.5 uses a framework called PARL (Parallel-Agent Reinforcement Learning) and combines three rewards.

Three rewards. The instantiation reward prevents serial collapse; the finish reward prevents spawning junk tasks; the outcome reward measures goal completion.

The first is the instantiation reward. Left alone, the model takes the easy route and reverts to "processing serially with one agent" (serial collapse). This reward prevents that and encourages parallel execution. The second is the finish reward. With the instantiation reward alone, a different cheat emerges: "spawn a bunch of sub-agents to hack the reward." Getting flooded with meaningless subtasks is a problem, so the more the assigned subtasks are actually completed, the more it is rewarded. The third is the outcome reward, the final signal for whether the overall task succeeded. These three are summed, and the auxiliary rewards are annealed toward zero as training progresses. It is a well-crafted design, I think.

In practice, on BrowseComp — a benchmark for searching across the web — the swarm scored 78.4% versus 60.6% for a single agent. It directs up to 100 sub-agents and cuts execution time by up to 4.5x. To the question of "how smart can you make one agent," they answered on a different axis: "how do you bind a swarm together."

Where the three axes converge: Kimi K2.5 and how to build a multimodal model

Token efficiency (MuonClip), long context (Kimi Linear), and agent swarms. Multiplying these three axes gives the star of the talk: Kimi K2.5.⁷ It was released at the end of January 2026, and the arXiv technical report is titled "Kimi K2.5: Visual Agentic Intelligence."

Here, the recipe for the "world-class multimodal performance" that surprised so many people is revealed. The key is early fusion: training vision (images and video) and text together from the very start.

Many earlier open models first trained heavily on text and bolted on vision afterward. For example, train text for 20 trillion tokens, then add roughly 2 trillion tokens of vision post-training on top. Kimi K2.5 stopped doing that. From day one of training, at 0% progress, it mixes vision tokens and text tokens. In preliminary experiments, this early fusion beat late fusion, and moreover, "keep the vision ratio modest and mix it in early" gave the best result.

Early fusion (fused from the 0% mark, at a 10%:90% vision ratio) is best on visual knowledge, visual reasoning, OCR, and code. K2.5 is positioned as the first open model to natively fuse vision and text.

Why does early fusion work? Yang's explanation is suggestive. An ability like "watch a video and write code" only emerges when vision and text are fused into one brain. Keep the two brains separate and it never happens. Indeed, Kimi K2.5 reads a video and generates a website that reproduces its feel — a feat that is only possible because vision and text are dissolved into the same representation space.

What is more surprising is that the two modalities lift each other. Do reinforcement learning on vision-only tasks (counting objects, visual QA, containing no text tasks at all) and even text reasoning performance improved. Specifically, MMLU-Pro, a text-centric benchmark, improved from 84.7% to 86.4%. Conversely, with a strong text base, they used zero visual SFT (supervised fine-tuning) data — they call it "zero-vision SFT" — and with only text SFT plus joint vision-text reinforcement learning, they reached near state-of-the-art on vision tasks. The long-standing belief that adding vision hurts text was overturned head-on.

For reference, Kimi K2.5's benchmarks include AIME 2025 at 96.1, GPQA-Diamond at 87.6, and SWE-Bench Verified at 76.8. The fact that an open-weight model sits at this level is the substance behind the phrase "open, yet world-class." The puzzle I started with finally shows its face here: multimodal strength was not the dexterity of a bolt-on, but the product of a design philosophy from day one. This lens also changes how you read a comparison of AI agent tools.

And onto the next generation: attention in the depth direction, too — Attention Residuals

Near the end, Yang said, "yesterday we published the technical report for our next-generation architecture." It is Attention Residuals (AttnRes).⁸ He introduced it as a stepping stone for the next model, not yet in Kimi K2.5.

The starting point is the residual connection, another mature fundamental. Until ResNet arrived in 2015, deeply stacked neural networks could not be trained. Add layers and the gradients vanished or exploded, destabilizing training. ResNet's residual connection solved that and let you stack layers freely. Its proposer, Kaiming He, made it a tutorial at ICML 2016. That is what Yang called "Kaiming's tutorial from ten years ago."

Here Yang draws on a famous analogy Ilya Sutskever gave at NeurIPS 2024: "a residual connection is an LSTM rotated 90 degrees." An LSTM is a recurrent mechanism that takes the previous step's state along the time direction and produces the next. A residual connection takes the previous layer's output along the depth direction and produces the next layer's output. Only the orientation differs; the structure is much the same. If so, Yang asks, "what happens if you rotate attention by 90 degrees?"

A residual connection is an LSTM rotated along depth. So rotate attention along depth, and each layer can select over all previous layers with input-dependent weights (starting from Ilya Sutskever's observation at NeurIPS 2024).

A standard residual connection sums the outputs of all layers with "fixed, equal weights." As a result, hidden states grow without control along depth, and each layer's contribution gets diluted (the paper calls this PreNorm dilution). Attention Residuals replaces that fixed addition with softmax attention along the depth direction. Each layer selectively aggregates the outputs of all prior layers with "learned, input-dependent weights" to produce its own output. Picture enriching a single information highway with attention. The paper frames it as the "Duality of Time and Depth."

For practical efficiency, they also prepared a variant called Block AttnRes that splits layers into blocks and applies this attention only to each block's output. The result: in scaling-law experiments, Block AttnRes matched the loss of a baseline trained with 1.25x the compute. In other words, about a 25% boost in token efficiency. On benchmarks too, on Kimi Linear (48B/3B) it lifted GPQA-Diamond by +7.5, Minerva Math by +3.6, and HumanEval by +3.1, with the biggest gains on heavy reasoning and code-generation tasks.

The talk's undercurrent: rebuild mature fundamentals on a scaling ladder

Once you get here, the real theme of the talk comes into view. What Yang closed with was not a boast about individual techniques, but a view of how research itself is done.

Lined up, the structure is clear. Against 2014's Adam, MuonClip. Against 2017's attention, Kimi Linear. Against 2015's residual connections, Attention Residuals. What they are doing is rebuilding, one after another, the fundamentals that had become the industry's "givens." And each is designed as a drop-in replacement. "If you train a Transformer LLM, it will be strictly better with MuonClip than with Adam," Yang said flatly.

Why is this possible now? His answer is one I like. "Ten years ago, it was mostly about publishing a new idea in a paper. But the lack of experimental rigor made it hard to draw solid conclusions. Today we have a scaling ladder. We can train models at various scales and measure them on a whole suite of benchmarks. So we can draw confident, solid conclusions." In other words, precisely because large-scale training infrastructure and benchmark suites are now in place, you can question classic fundamentals and overwrite them with evidence. And these improvements compound. Token efficiency, long context, depth-direction efficiency. Stack them up and the model gets much better.

And all of it is published open-weight. Throughout the talk, Yang kept returning to "democratizing intelligence." Open models can be deployed freely on local machines or in the cloud, and you can access every single bit of the weights. It is not a black box. The slide he borrowed from NVIDIA CEO Jensen Huang was symbolic: open models are rapidly closing the gap with proprietary ones and reaching the frontier. For the founder of the only Chinese LLM company to stand on NVIDIA's main stage at GTC, there could hardly be a better venue.

From K2.5 to K3 — the preview became the main feature

That this talk centered on K2.5 makes sense once you look at the timeline. K2.5 was released at the end of January 2026. The talk was on March 18. The Attention Residuals technical report was posted to arXiv on March 16, two days earlier. So Yang could preview AttnRes as "the next-generation architecture we published yesterday," even though it was not yet in any model.

That preview became the main feature in Kimi K3, released in July 2026. Per the official blog, K3 is a 2.8-trillion-parameter open-weight MoE built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes).⁹ Its optimizer is Per-Head Muon (applying Muon independently per attention head), a further evolution of MuonClip, and with additions like Quantile Balancing and Gated MLA it claims roughly 2.5x the overall scaling efficiency of K2. A 1M-token context, native image support. The three axes and the next-generation architecture laid out at GTC crystallized directly into K3, a real product.

Put differently, if you want to know "why K3 is world-class," watch this March GTC talk. The finished product, K3, stands on the extension of K2.5's design philosophy, with exactly the architecture that was previewed. Treating this talk as "an explanation of K3" is, if anything, quite accurate in direction. The name is K2.5, but what is being described is their way of building itself.

Hamamoto's take — what should Japanese companies take from this talk?

From here, this is my own view. I received this talk in three ways, from the standpoint of AI adoption in Japan.

The first is a shift in perspective: "efficiency is not about cost, but about intelligence." AI-adoption discussions at Japanese companies still tilt toward "how do we hold down API usage fees." That matters, of course, but Yang's argument goes a level higher. Improving token efficiency increases the total intelligence you can extract from finite high-quality data. Pushing efficiency raises the ceiling of achievable intelligence. This framing is suggestive when you think about how to treat your own finite asset: internal data. Extracting maximum value from limited in-house data comes down precisely to the efficient design of models and pipelines.

The second is the strategic meaning of open weights. A world-class multimodal model is arriving in a form where you can download the weights wholesale. For Japanese companies — especially manufacturing, finance, and the public sector, which prize data sovereignty and domestic storage — this is a fact you cannot ignore. Instead of pouring confidential internal data into a black-box API, running a frontier model under your own control (on-premises or in a domestic cloud) is becoming realistic. Our own insistence on domestic-server enterprise AI with ZEROCK (/en/zerock) is precisely because we anticipated this shift. The assumption that "frontier equals depending on an overseas API" is no longer a given.

The third is the idea of agent swarms. From "hand everything to one smart AI" to "bind a swarm of role-specific agents." This idea maps directly onto how companies design work. Rather than trying to build one giant automation, run small specialist agents in parallel and have an orchestrator bind them. It is almost the same structure as a human org chart. If so, the companies that are good at AI will not be the ones that bought a good model, but the ones that can decompose and distribute tasks well. That is less about technical prowess and more about the design of work and the running of an organization.

As an aside, what encouraged me most in this talk was the single point that they "questioned mature fundamentals and rebuilt them." Adam, attention, residual connections — givens that no one had seriously tried to replace for nearly a decade. A young company that got its hands on a new verification environment, the scaling ladder, is overturning them one after another. The frontier is not decided by huge capital or proprietary data alone; it moves on the product of "a willingness to question the givens" and "an environment to verify it rigorously." I believe the same holds for AI adoption in Japan. Before the flashy model selection, which of your own operational givens can you question? That is where the winning path lies.

If you want to take stock of how far you have woven generative AI and agents into your strategy, measure your position with the AI literacy check. If you want to go further and design "how to decompose your operations and where to automate first" together, you can reach us through our AI-adoption support program WARP or an individual consultation. Do not let the moves at the frontier end as someone else's news; connect them to your own decisions. That is the biggest homework I took from this talk.

I also wrote before about how few Japanese firms appear on NVIDIA's list of AI-native companies, in an earlier article. Now that a company like Kimi is proving "open, yet world-class," I take an optimistic view: this is precisely the moment when the tools for Japanese companies to catch up are, if anything, falling into place.

---

References

This article is structured by cross-checking the content of Zhilin Yang's NVIDIA GTC 2026 session "How We Scaled Kimi K2.5" (S81695) against publicly available primary sources (arXiv papers, the Moonshot AI official blog, GitHub, and Hugging Face). Slide images are quoted from the same session. Figures and proper nouns are based on each primary source; where the spoken talk and the paper differ (for example, "+24% token efficiency" is a "1.25x compute advantage" in the paper), the primary source wording takes precedence.

Footnotes

1. Kaplan et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361 (2020). https://arxiv.org/abs/2001.08361 ↩

2. Kingma & Ba, "Adam: A Method for Stochastic Optimization," arXiv:1412.6980 (2014). https://arxiv.org/abs/1412.6980 ↩

3. Keller Jordan, "Muon: An optimizer for hidden layers in neural networks" (2024). https://kellerjordan.github.io/posts/muon/ ↩

4. Liu et al. (Moonshot AI, Kimi Team), "Muon is Scalable for LLM Training," arXiv:2502.16982 (2025). https://arxiv.org/abs/2502.16982 ↩

5. Kimi Team, "Kimi K2: Open Agentic Intelligence," arXiv:2507.20534 (2025). https://arxiv.org/abs/2507.20534 ↩ ↩²

6. Kimi Team, "Kimi Linear: An Expressive, Efficient Attention Architecture," arXiv:2510.26692 (2025). https://arxiv.org/abs/2510.26692 ↩

7. Kimi Team, "Kimi K2.5: Visual Agentic Intelligence," arXiv:2602.02276 (2026). https://arxiv.org/abs/2602.02276 ↩

8. Kimi Team, "Attention Residuals," arXiv:2603.15031 (2026). https://arxiv.org/abs/2603.15031 / https://github.com/MoonshotAI/Attention-Residuals ↩

9. Moonshot AI, "Kimi K3" official technical blog (2026). https://www.kimi.com/blog/kimi-k3 ↩