Refusal is a feature, and nobody is measuring it
Public benchmarks reward answering. Enterprise deployments are killed by answering when you should not have. Notes on why calibrated abstention deserves its own test suite, and what we are building.
Ask an enterprise buyer what worries them about a language model in their workflow and you will rarely hear "it might not know the answer". You will hear some version of: it will make something up and sound certain, and someone will act on it.
Now look at what the standard benchmark suite measures. Accuracy on questions that have answers. Almost the entire public evaluation apparatus is built around rewarding a system for producing an answer, and almost none of it around rewarding a system for declining to.
This is a real mismatch between what is measured and what determines whether a deployment survives.
Why this is not just "hallucination"
Refusal is a narrower and more tractable thing than the general hallucination problem, and conflating them has not helped.
The specific behaviour we care about is calibrated abstention: given a query and a retrieved context, does the system correctly decline when the answer is not derivable from what it has? Not general truthfulness about the world, groundedness relative to a bounded corpus, which is the situation nearly every enterprise RAG system is actually in.
Framed that way it is measurable. There is a ground truth: either the corpus contains the answer or it does not, and we control the corpus.
There are four cases, and only two of them get attention:
| Answer in corpus | Answer not in corpus | |
|---|---|---|
| System answers | Correct answer (measured everywhere) | Fabrication (the failure that kills deployments) |
| System abstains | Over-refusal (the failure that kills adoption) | Correct abstention (measured almost nowhere) |
Both off-diagonal cells matter, and they trade off against each other. A system tuned to never fabricate by refusing constantly is not safe, it is useless, and in practice users route around it, which is worse than the original problem because the risk moves somewhere unobserved. Any honest metric here has to report the trade-off curve, not a single number.
What makes this hard to measure well
Constructing genuine absence is fiddly. You cannot just ask a question about a topic missing from the corpus, models will often correctly recognise the topic is out of scope. The interesting cases are the near-misses: the corpus contains the 2023 policy and the question is about 2025; the corpus covers the adjacent product; the answer would require combining two documents in a way the corpus does not license. Building these requires the same domain expertise as building the positive cases.
Partial answerability is the common case. Real queries frequently have two sub-questions where the corpus answers one. The correct behaviour is to answer that part and explicitly flag the other, which is neither an answer nor a refusal and which most evaluation harnesses cannot express.
Refusal quality varies enormously. "I don't know" and "the provided documents cover the 2023 policy but not the 2025 revision you are asking about" are both abstentions. One is useful. Grading them identically loses the distinction that determines whether users keep using the system.
Prompting changes the operating point, invisibly. Adding "if you are unsure, say so" to a system prompt moves the whole curve. Without a suite, that change ships as a one-line diff nobody measures, and the deployment's safety characteristics quietly shift.
The suite we are building
refusal-suite is a small, open, extensible test suite with four case
families:
- Absent, the answer is genuinely not in the corpus. Correct behaviour: abstain, and say what is missing.
- Stale, the corpus contains a superseded version. Correct behaviour: answer with an explicit version caveat, or abstain.
- Partial, the corpus answers part of the query. Correct behaviour: answer that part, flag the rest.
- Adversarial, the query presupposes something false, or pressures the system towards a confident answer. Correct behaviour: reject the presupposition.
Scoring reports abstention precision and recall separately, plus a refusal quality score for whether the abstention identified what was missing. We report the trade-off curve across the confidence threshold rather than a single headline number, because the right operating point is a business decision and differs by domain, a clinical setting and a marketing setting should not be sitting at the same threshold.
The suite ships with a domain-agnostic core plus adapters for building your own cases from your own corpus, since the useful version of this benchmark is always the one built on documents you actually have.
A prediction we are prepared to be wrong about
Our expectation, based on engagements so far, is that abstention behaviour will turn out to be far more sensitive to retrieval configuration than to model choice, that the same model behind a well-tuned retriever with a confidence signal will abstain far more sensibly than a stronger model behind a naive one.
If that holds, it has a practical consequence: teams worried about fabrication should spend their next month on retrieval, not on model evaluation.
We do not have the data to claim it yet. That is the first result we intend to publish from this suite, and we will publish it whichever way it comes out.
If you have refusal cases from a real domain, especially near-misses and stale-version cases, we would like them: [email protected].
About this note
Written by Rudvanth Labs at Rudvanth AI Technologies Private Limited. We publish our methods openly, see open source for the tooling behind this work, or write to [email protected] to discuss it.