Harness

A language model, by itself, can do only one thing: it receives text and produces text in response. It cannot look anything up on the internet on its own, cannot open a file, and cannot launch a program. For real work to come out of it anyway, a program is built around it. This program is called a harness. It presents the task to the model, passes on the answer, carries out desired actions, and sends the result back into the model. The English name comes from the harness of a horse pulling a cart: the power lies in the animal, but only the harness makes it usable.

Why two companies achieve different results with the same model

In benchmarks, meaning standardized test series, confusing numbers often turn up. The same model reaches 40 percent solved tasks with one provider and 70 percent with another. The difference then usually lies not in the model but in the harness. How often is it allowed to correct a mistake? Does it get to see the program’s error message? Is it allowed to run tests before submitting? Each of these decisions shifts the result.

For companies, this is good news. Training a top model of one’s own costs hundreds of millions of euros. Building a better harness costs development time. That is why there are many companies that own no model of their own but still sell a sought-after product. Their value lies in the scaffold, not in the engine.

Conversely, this also means: anyone reading benchmark numbers should ask under what conditions they were produced. A result without information about the harness is hard to compare. Reputable publications therefore disclose how many attempts were allowed and which tools were available.

The loop of thinking, acting, and observing

A harness almost always works as a loop. First, it formulates the task for the model, often supplemented with rules and background information. The model responds, for instance with a request to open a particular file. The harness carries out this step and sends the result back. Then the next round begins, until the task is completed or a limit is reached.

A harness also includes safety boundaries. It specifies which commands are allowed and which are not. It limits how many steps a run may take at most, so that costs do not explode. And it decides which parts of the conversation so far the model is still allowed to see, since its capacity for taking in information is limited.

A common misconception is that a harness is simply a particularly long instruction text. The text is only one part of it. What matters is the program logic: error handling, retries, tool integration, logging. Related but not identical is the term agent. The agent is the acting system as a whole; the harness is the machinery that holds it together.

From coding assistant to safety testing

Harnesses are most visible in coding tools. When an assistant in the development environment independently modifies files, runs tests, and fixes errors, a harness is behind it. Research assistants that issue multiple search queries one after another and write a report at the end also work this way.

In research, there is additionally the evaluation harness, meaning the test setup. It automatically poses thousands of test questions, collects the answers, and compares them with the correct solutions. Without such test setups, the leaderboards of AI models could not be created at all.

The term also plays a role in safety investigations. Testing bodies and providers test what a model could do with as strong a harness as possible. The reasoning behind this: a model is as dangerous as the best scaffold someone can build around it. That is why in reports on AI risks you often read that a model was tested under certain harness conditions.