Refusal Detection

Programs like ChatGPT don’t answer every question. Sometimes they decline, for example with the sentence: “I’m sorry, but I can’t help with that.” Such a refusal is called a refusal in technical jargon, from the English word for denial. Refusal detection is the task of automatically finding and counting such refusals in response texts. A human recognizes them immediately. A computer needs a rule or its own small checking program for this, because the refusal can appear in thousands of different phrasings.

Why companies count their own refusals

Anyone operating a chatbot must avoid two errors at the same time. First, the system must not output dangerous instructions, for example on building weapons. Second, it must not constantly block harmless questions. The second case is called over-cautiousness or over-refusal. Both annoy users or harm the provider, just in different ways.

Without measurement, nothing here can be controlled. Providers therefore send large test sets of questions through the model, often tens of thousands. Afterward, someone must check how many responses were refusals. Doing this by hand would be far too expensive. This exact counting is what refusal detection takes over.

The results later end up in model cards and safety reports, i.e. the accompanying documents for new AI versions. There it might state, for example, that a model refuses in 98 percent of cases for critical requests and only in one percent of cases for harmless requests. Such figures are also of interest to regulatory authorities. And to companies that have to decide which model to purchase.

From keyword list to evaluation model

The simplest approach is a list of typical phrasings. One searches the response text for phrases like “I cannot” or “I’m not allowed to.” This is fast and costs almost nothing. However, it is also error-prone, because language is flexible.

An example illustrates the problem. The sentence “I can’t tell you who will win the race, but here are the rules” contains the phrase “I can’t.” Nevertheless, it is not a refusal but a helpful answer. Conversely, there are refusals without any typical keyword at all, for example when the model simply, politely changes the subject. Both types of errors distort the statistics.

For this reason, a second AI model is usually used today as an evaluator. It is presented with the question and the answer and decides: refusal or not. This procedure is called LLM-as-a-Judge, meaning a language model acting as arbiter. Sometimes a small, specially trained classifier is used instead, a program that has only learned this one yes-no decision. An important distinction: refusal detection only checks whether a refusal occurred, not whether the refusal was justified. Whether it was warranted is decided in a separate step.

Refusals in benchmarks and everyday use

The topic is most visible in comparison tests, so-called benchmarks. Test collections like XSTest deliberately consist of questions that sound dangerous but are harmless. A classic example is the question of how to kill a cake, meant as a play on words. Models that refuse here are considered overly cautious. Without automatic refusal detection, such tests could not be evaluated.

Detection also runs during operation. Providers continuously monitor how often their product declines requests. If the rate rises sharply after an update, that is a warning sign. Companies that integrate a chatbot into their customer service also measure this: every unnecessary refusal is a call to human support.

In media coverage, refusals often appear without the technical term. When it is said that a new model has become “finicky” or another has become “more relaxed,” what usually lies behind this are measured refusal rates. A common misconception, by the way, is that a refusal is always an error. In cases of genuine risk, it is the desired behavior.