ARC-AGI

ARC-AGI is a collection of puzzles used to test computer programs. Each puzzle consists of small colored grids, similar to squared sheets of paper with colored-in boxes. You see two or three example pairs: on the left a grid before, on the right the same grid after. From these examples you have to figure out on your own the rule by which the grid was changed. Then you get a new grid and are supposed to apply the rule to it. For humans, most of these tasks are easy; for machines, they were astonishingly hard for a long time.

Why colorful little boxes reveal so much

The Frenchman François Chollet, at the time a researcher at Google, published ARC in 2019. His goal was to build a test that couldn’t be passed through memorization. With many other tests for language models, that is exactly the problem. The questions are somewhere on the internet, and the programs have read the entire internet as training material. A good result then proves little.

The ARC puzzles are therefore newly invented and partly kept secret. Each task has its own rule that does not appear in any other task. Memorization is thus of no help. What does help is the ability to derive a pattern from two examples and transfer it to a new case. This is exactly the ability meant when one speaks of true generalization.

That is why ARC-AGI is considered a kind of litmus test in the field. For years, the best programs solved barely a fifth of the tasks, while humans on average managed over 75 percent. This gap was a strong argument against the claim that modern AI systems could already think. When it narrows, that is genuine news.

From example pair to the correct rule

The grids are at most 30 by 30 boxes in size and use ten colors. The rules draw on knowledge that even small children already have. That things are objects, that one can count, that shapes can be mirrored or rotated. One rule might be: color every enclosed area yellow. Another: place the smallest object in the center of the largest.

A program receives the example pairs as number grids and must generate the output grid box by box. Only an exactly correct solution counts; there are no partial points. Usually two attempts per task are allowed. Some of the puzzles remain locked away with the provider so that no one can train on them in advance.

Starting in 2024, systems made big leaps that compute in intermediate steps before giving an answer. This is called test-time computing: the model tries out several solution paths before committing to one. OpenAI’s model o3 achieved over 85 percent on the first ARC version in late 2024. However, some runs cost thousands of dollars in compute time for a single task. The organizers' response was ARC-AGI-2, a harder version with tasks that demand significantly more combinatorial steps.

ARC-AGI in headlines and prize money

You most often encounter the term in reports about new models. When OpenAI, Google, or Anthropic present a system, the ARC score often appears in the results table. It is frequently cited because it is considered particularly hard to game. For investors, it is an indicator of how much real progress lies behind an announcement.

There is also the ARC Prize, a public competition with prize money in the millions. Participants must disclose their solution and may only use limited computing power. This is meant to prevent the outcome from being decided purely by budget.

A common misconception deserves clarification. A high ARC score does not mean that a machine possesses general intelligence. The test measures a single, narrowly defined ability on abstract grids. Chollet himself emphasizes that ARC only shows what systems still cannot do. If they pass the test, a new one is needed.