Safety Classifier

A safety classifier is a program that sorts content. It receives a text or an image and returns an assessment: harmless or problematic. Often it also names a category for this, such as violence, hate, or instructions for weapons. Such programs run in the background of chat programs, without users seeing them. They check what the user enters, and often also what the system answers. You can imagine them as a gatekeeper standing at two doors: one at the entrance, one at the exit.

The second ring of defense behind the model

Large language models learn from huge amounts of text and are then trained to refuse dangerous requests. But this training is never perfect. Clever users get around it with tricks, for example by disguising their request as a role-play or as a translation task. Such tricks are called jailbreaks in the industry. A separate classifier can step in when the model itself has already been deceived.

For providers, this is also a legal and economic question. If a chatbot gives dangerous advice to a teenager, headlines, lawsuits, and trouble with authorities loom. The EU legislator requires providers under the AI Act to actively reduce risks. Classifiers are visible proof that something has been done. That’s why they regularly appear in safety reports from companies like OpenAI, Anthropic, or Google.

The distinction from the actual model is important. The model is supposed to give good answers. The classifier is only supposed to judge whether something is allowed. This separation makes it possible to change rules quickly without having to retrain the large model.

From example catalog to threshold

A classifier is trained with examples. Humans collect thousands of texts and mark each one as permitted or as a violation of a certain rule. The program searches these examples for patterns. Afterward, it can also assess new, never-before-seen texts. It is usually much smaller than the actual language model, so that it runs quickly and cheaply.

The result is not a simple yes-or-no answer, but a number between 0 and 1. It expresses how certain the program is. The operators then set a threshold above which content gets blocked. A low threshold blocks a lot, including harmless content. A high threshold lets more through, including dangerous content. This setting is a trade-off, not a technical truth.

This is precisely where the typical errors arise. A false positive occurs when a harmless question is blocked, such as a nurse’s question about medication dosages. A false negative is the reverse case: dangerous content slips through. Both cannot be pushed to zero at the same time. Modern systems therefore combine several classifiers and let humans review borderline cases.

Where these filters take effect in everyday life

Most often you encounter them in chatbots. When an answer breaks off mid-sentence or is replaced by a standard notice, a classifier has often intervened. Image generators, too, check every request and every finished image. That’s why they sometimes refuse subjects that actually seem harmless.

The same approach lies behind content moderation on social networks. Instagram, TikTok, and YouTube automatically have millions of posts assessed before a human even sees them. Spam filters in email programs are, at their core, nothing different either. Companies, in turn, buy such filters as building blocks when they build their own AI products. In technical texts, they are then often called guardrails.

In the news, safety classifiers usually appear in two contexts. Either someone has outsmarted them and shows what a model reveals anyway. Or there is criticism that they are set too strictly and suppress legitimate topics. Both debates ultimately revolve around the same threshold.