Kimi K3 on AMD MI355X: Better Cost per Token than NVIDIA B300

Open language models have made a leap over the past few months: DeepSeek V4-Pro and GLM5.2 have closed in on the closed flagships, and the fresh Kimi K3 lays claim to Fable/Sol-level intelligence. But the smarter a model gets, the bigger it is. Kimi K3 carries 2.8 trillion parameters — that's more than 1.5 TB of VRAM before you even allocate the KV cache for a million-token context. Even a node of eight NVIDIA B200s can't hold the whole thing.

Betting on memory

That leaves two paths: deploy the model on a B300 node (288 GB of VRAM per GPU) or tie up two B200 nodes at once in TP16 mode. But that same 288 GB is available on the AMD MI355X too — and the chip is on average roughly 2.4 times cheaper than the B300 and 1.7 times cheaper than the B200. AMD's main weakness is software support: slower kernels and lagging compatibility with inference frameworks. For Kimi K3, however, ROCm support landed on release day, so most of the work was already done.

The numbers

On a “1024 tokens in, 400 out” benchmark, a node of eight MI355Xs delivered 952 tok/s and 118 tok/s single-stream. That's 3.8 times the aggregate per-node throughput of the two-node B200 pairing. The B300 node stays faster in aggregate throughput by about 1.65 times, but at 2.4 times the price it is soundly beaten by the MI355X on performance per dollar.

What had to be fixed

The model ran “out of the box,” but reaching today's numbers took some tuning. Speculative decoding via a third-party draft crashed immediately on ROCm with a NameError: name 'top_k_renorm_prob' is not defined error: the CUDA build has the needed kernel, but in the ROCm version it simply isn't defined. The instinct is that you need to write a custom kernel, but here a single PyTorch function was enough — sort, zero out the excess, normalize. That gave roughly a 2.2x speedup single-stream.

A separate headache was time to first token: a cold warm-up on 172K tokens took about 51 s versus 23 s on the B300. The cause was a fallback to slow generic Triton attention: the fast AITER MLA kernel is designed for 4, 8, or a multiple of 16 heads, while K3 at TP8 yields 12 per rank. The fix turned out to be laughably simple: pad the 12 heads to 16 with zeros, run the fast kernel, and pull the needed 12 back out. Warm-up sped up by 2–3x.

The authors' takeaway is terse: taking the lead in cost per token on AMD required almost no trickery and not a single custom kernel — there were even fewer bugs than with GLM5.2. Isn't it time to declare the “moat” around CUDA dried up?

Source: Hacker News