Cracking Open the Apple Neural Engine: Why the GPU Beats the NPU on Transformers

Three years ago, Eileen Yoon gave up on building an open driver for the Apple Neural Engine (ANE), having reached a disappointing conclusion: the block is simply good for very little. Its architecture turned out to be too “opinionated” to build a general-purpose accelerator around, and even macOS uses the ANE mostly to generate the shrunken thumbnails in Finder. Now the author is returning to the subject — not to make the accelerator useful, but to fully document its internal workings before the standard NPU disappears for good.

The timing is symbolic: for the M5 chip (2025), Apple touted “LLM performance” as the headline feature and moved the ANE cores inside the GPU cores. In Yoon's view, this marks the beginning of the end for the standalone neural module, which debuted all the way back in the A11 Bionic in 2017.

Built for convolutional networks

The ANE was designed for the dense convolutional networks of the image-processing era — with predictable data-reuse patterns. And that very assumption is what transformers broke, especially autoregressive decoding. The compute core itself is general-purpose: 16 cores of 128 FP16 multiply-accumulate lanes each (or 256 INT8), for a total of 2,048 parallel lanes. Multiply-accumulate on its own says nothing about the task — the nature of the operation (convolution, matrix multiplication, dot product) is determined by how the operands are laid out and scheduled onto the array. Nonlinear activations like tanh are implemented with a 33-value piecewise-linear table.

The ANE has no instruction set: every operation is compiled ahead of time into a stream of task descriptors, and the driver merely loads them into memory and “rings the doorbell.” In essence, a descriptor is a serialized dump of the configuration registers for one pass through a fixed data path. The scheme for feeding tasks resembles NVIDIA GPUs' pushbuffer. The author's conclusion: the ANE is a functionally rigid streaming engine, not a processor that executes arbitrary instructions.

It all comes down to memory

The key constraint is not compute, but data movement. At 11 TOP/s and 68 GB/s of DRAM bandwidth, the “inflection point” of the roofline model demands at least 162 operations for every byte read from memory. Below that threshold, faster compute doesn't buy you a single extra token per second. The ANE has its own hierarchy of local memory: private per-core memory for convolution kernels (16×64 KiB), a shared 2 MiB L2, and three DMA engines. What's more, the path for weights is one-way — load only — and weights cannot be fetched from L2, only from DRAM: Apple clearly never expected that weight tensors would have to be shuttled around frequently.

The measurements settle the matter: on the ANE, weight reads deliver only 38 GB/s and tile reads 59 GB/s, whereas the GPU manages 77.7 GB/s. Worse still, weight and tile requests are strictly serialized, and their times add up. To decode a single transformer token — where you have to push all of the model's weights through for the sake of one token — whoever has the higher read bandwidth wins, and that's the GPU. Hence Apple's decision to fold the ANE inside the graphics core.

Source: Hacker News