Red Teaming

Red teaming is a test in which a group of people deliberately tries to make a system fail. They slip into the role of attackers and search for every gap they can find. The term comes from the military: in exercises, the “red team” played the enemy, while the “blue team” played the defense. Later, computer security adopted it, and today it is used above all by the AI industry. With language models, meaning programs that answer questions with text, the team specifically tries to coax the program into giving answers it is actually supposed to refuse. What is found here is not a technical defect, but behavior the developers did not want.

Why manufacturers prefer to attack their own systems first

An AI model is not programmed line by line. It learns from huge amounts of text and, in doing so, develops behavior that no one planned in advance. That’s why you can’t just look at the code and check whether it behaves properly. You have to try it out. And because a model reacts differently to an infinite number of possible phrasings, a few spot checks are not enough.

The second reason is sequencing. If the company’s own team finds a weakness, it costs work time. If a user finds it after launch, it’s on social media the next day. Large providers therefore have their models attacked for months before release. For particularly powerful models, external experts, such as chemists or security researchers, additionally check whether dangerous knowledge can be extracted.

By now, red teaming has also become a matter of regulation. The EU’s AI legal framework and agreements with governments require providers of large models to systematically test and document risks. A report on attack tests carried out is thus part of the paperwork before market launch.

The attackers' tricks

The best-known approach is circumventing the rules through an invented framing story. A direct request for instructions on something forbidden gets rejected. But if you wrap the same question as a novel scene, as an assignment for chemistry class, or as a request to an “unrestricted version,” the model sometimes answers anyway. Such attempts are called jailbreaks, after the English word for breaking out of prison.

Other attacks approach things differently. You can hide an instruction in a foreign language or in strange spelling, because the safeguards are trained less strongly there. You can ask harmless partial questions whose answers together add up to the forbidden information. Or you write a command into a webpage that the AI later reads, and the program mistakes it for a user request. This last case is called prompt injection and particularly affects AI systems that independently research on the internet.

Every hit found is logged. From these cases, training examples are created that teach the model to refuse such requests in the future. In addition, attacks today often run automatically: a second program generates thousands of attack variants and tries them out. This doesn’t replace humans, though, since truly new ideas mostly come from them.

Red teaming in product announcements and model reports

When a company introduces a new language model, a technical report usually accompanies it. It contains a section on safety in which red teaming is explicitly mentioned: how many experts were involved, which risk areas were examined, which weaknesses remain. Anyone who reads announcements about new AI models will almost inevitably come across it.

The field also exists as a profession. Companies advertise positions for red teamers, and there are providers who sell such tests as a service. Some companies also pay rewards when outsiders report a vulnerability. Related but not the same is the task of testing bodies that evaluate models according to fixed criteria. Red teaming is open and creative; a checklist-based test is not.

A common misconception: red teaming does not make a model safe, only less easy to attack. New workarounds keep appearing after every release, often within days. The test is therefore not a conclusion but a process that continues throughout a model’s entire lifespan.