A Transformer Scored 44% on ARC-AGI-1 for 67 Cents

The ARC-AGI benchmark, conceived by François Chollet as a test of “fluid intelligence,” has resisted assault for six years now. Mithil Vakde, a graduate of the Indian Institute of Technology Bombay, has shown that a respectable result requires neither giant language models nor mountains of synthetic data.

His small transformer (eight layers) is trained from scratch right during the test and scores 44% on the public ARC-AGI-1 split and 7% on the harder ARC-2. All the training and inference fit into an hour and a half on a single RTX 5090, and the total cost of renting the card comes to 67 cents. The code is open, and the result, according to the author, matches the numbers of the much-hyped TRM and HRM models — at incomparably lower cost.

What's under the hood

Each “input-output” pair is turned into a sequence of tokens on which the model is trained autoregressively. Training runs simultaneously on the training and test puzzles, but the labels (the correct answers to the test pairs) stay hidden — so this is not “training on the test” but transductive meta-learning. The largest contribution to quality, judging by the ablations, comes from the well-chosen representations: three-dimensional RoPE and a separate learnable embedding for each task. Remove either of them and accuracy drops to roughly 25%.

Compared with the previous version, Vakde brought the architecture up to date (SwiGLU instead of GELU, RMSNorm instead of LayerNorm), swapped the AdamW optimizer for NorMuon, sharply cut the number of augmentations, and switched to supervised training on the output tokens only. Curiously, the test loss got worse in the process while the final score went up — the author has no explanation for this yet.

A quarrel with the industry

Much of the note is an argument against prevailing approaches. Vakde considers data efficiency (sample efficiency) to be AI's main unsolved problem, and he reproaches researchers for their infatuation with recursion: in his view, HRM and TRM have not proven through ablations that recurrent loops contribute anything, and the claimed “7 million parameters” are disingenuous, since hundreds of millions of weights are hidden away in the embeddings.

Language models come in for their share too. Their progress on ARC, by the author's observation, is explained by fine-tuning on data resembling the benchmark's tasks rather than by any growth in abstract reasoning: base models still score a few percent on ARC-2. Hence his proposal — ban any pretraining and force models to learn from scratch after submission, which would automatically rule out synthetic data.

Vakde honestly admits that he himself did not expect 45% from “just a transformer with the most obvious representation,” and he is baffled that in six years, and with a million-dollar prize pool, no one had thought of it before. A bar of 65%, he is convinced, is achievable within the same architecture — and he invites anyone who wishes to improve the open code (only, he asks, without expanding the training set).

Source: Hacker News