44% on ARC-AGI for 67 cents: a from-scratch transformer that rivals LLMs

An independent researcher has trained a small transformer from scratch that scores 44% on the ARC-AGI-1 reasoning benchmark, using roughly 90 minutes on a single RTX 5090 for about 67 cents of compute. The result matches recent specialized systems like TRM and HRM and outperforms many far larger language models, and it also reaches 7% on the harder ARC-2. All of the code is open source. The work is pitched as an attack on sample efficiency — squeezing generalization out of only ~1,000 puzzles — which the author argues is the central open problem in AI today.

The method converts each input-output grid pair into a token sequence that a small transformer learns autoregressively at test time, using 3D RoPE for positional encoding and a learned per-task embedding to share structure across puzzles. Ablations single these two representation choices out as the biggest performance drivers: swapping 3D RoPE for 1D or dropping the per-task embedding both collapse the score to about 24%. Gains over the author’s prior model came from a modernized architecture (SwiGLU, RMSNorm, eight layers), more diverse data, and cheaper training via far fewer augmentations, the NorMuon optimizer, and FlashAttention/FlexAttention kernels. A counterintuitive finding: training only on output tokens (making the setup supervised) raises the score from 40% to 44% and stabilizes training even though test loss gets worse — a caution to researchers who optimize purely for low validation loss on small datasets.

The author is careful about the leakage objections that dogged the earlier viral result, filtering the 773 ARC-1 puzzles duplicated inside ARC-2 to avoid contamination and arguing that learning from unlabeled eval inputs is legitimate transductive metalearning rather than ‘training on test.’ The broader claim is provocative: a plain transformer with an obvious representation captures a surprising amount of ARC performance without synthetic data or heavy inductive biases, and the author believes 65% is reachable within the same framework. He invites others to push the score up or cut costs 10x with hand-written GPU kernels — while asking that they not expand the training data.