Guard Model

Programs like ChatGPT answer almost any question they are asked. But some questions they should not answer, such as how to build a weapon. To reliably prevent this, providers deploy a second, smaller program alongside it. This program reads along with every question and every answer and classifies it: harmless or problematic. That is exactly what a guard model is. It doesn’t write any text itself, but merely judges texts that others have written.

Why a second program plays the bouncer

A chatbot also learns during training to refuse dangerous requests. But this refusal doesn’t hold very firmly. Users keep finding phrasings that get around it. A popular trick is disguising the request as roleplay: “Pretend you’re a chemist in a novel.” The actual program occasionally falls for this.

A guard model is more robust against such tricks because it has a much narrower task. It is not meant to be helpful, only to judge. It therefore cannot be talked into things through politeness or stories. Experts call this principle defense in depth: if one safeguard fails, the next one kicks in.

For companies, there’s an additional practical reason. Laws and contracts require that a provider be able to prove which content it filters. A separate checking program delivers logs for this with clear categories. One can also change its rules without having to retrain the large language program. A hospital needs different limits than a gaming forum.

From text to risk category

Technically, a guard model is a sorter. It receives a text and returns an assessment, often with a category such as violence, self-harm, hate, or instructions for crimes. Some systems additionally provide a numerical value between 0 and 1 indicating how confident the model is. Once a defined threshold is reached, the block kicks in.

It learned this from many sample texts that people had previously rated by hand. From tens of thousands of such examples, it derives patterns. Well-known examples are Llama Guard from Meta and the filtering services from OpenAI and Google. Such models are deliberately small, often only a fraction of the size of the main model. This way, the check costs hardly any computing time and only delays the response by milliseconds.

Checks usually happen at two points. First the user’s input, then the chatbot's finished response. The second step is important because a harmless question sometimes triggers a problematic answer. If the verdict is negative, a standard refusal appears instead of the generated text. A common misconception is that a guard model works with a list of forbidden words. It evaluates the meaning of a sentence, and that is exactly why it is sometimes wrong.

Guard models in products and headlines

Anyone using a chatbot only notices the effect of the guard model. The message “I can’t help with that” often doesn’t come from the language program itself, but from the checker in front of it. Content moderation tools on social networks also use such classifiers to flag insults. In customer service systems, they prevent a bot from giving legal or medical advice.

In the news, guard models mostly show up when they fail. Reports on jailbreaks describe how someone got around the check. Conversely, there is criticism of overly strict filters that block medical questions or historical topics. Experts speak of false positives, i.e., content blocked without reason. Every provider must weigh both types of errors against each other.

For companies integrating AI themselves, guard models have meanwhile become a product of their own. Cloud providers sell them as an add-on service with adjustable categories. They should not be confused with guardrails in the broader sense: those are all protective measures combined, including fixed program rules and human oversight. The guard model is just one component of that, but the one most frequently mentioned.