DEV Community

Cover image for Does your AI feature ever say "I don't know"? (Friday AI question)
M.T.Ramkrushna
M.T.Ramkrushna

Posted on

Does your AI feature ever say "I don't know"? (Friday AI question)

Friday AI Question. This week's question is one I think most teams never ask out loud:

When your AI feature doesn't know the answer, what does it actually do?

Pick one (and tell me in the comments):

  • A) It always answers. Something is better than nothing.
  • B) It refuses only on hard-coded rules (blocked topics, missing permissions).
  • C) It abstains when the evidence is weak: says "I couldn't find this", shows what it searched, and offers a next step.
  • D) C, plus our evals score "I don't know" separately from wrong answers.

My lean: C on day one, D as soon as you have an eval set. Here's why.

The scoreboard problem

OpenAI's September 2025 post Why language models hallucinate argues that models hallucinate partly because standard training and evaluation reward guessing over acknowledging uncertainty.

Their analogy is a multiple-choice test: if you don't know, a wild guess might be right, while leaving it blank guarantees zero. When models are graded only on accuracy, guessing wins.

They split answers into three buckets: accurate, errors, and abstentions. And they show it with SimpleQA numbers from the GPT-5 system card:

Metric gpt-5-thinking-mini OpenAI o4-mini
Abstention rate 52% 1%
Accuracy rate 22% 24%
Error rate 26% 75%

o4-mini looks slightly better on accuracy (24% vs 22%). But its error rate, its rate of confidently wrong answers, is 75% vs 26%.

If your only metric is "% correct", you'd ship the model that is wrong three times as often.

Why this matters for your feature, not just for leaderboards

Your internal evals are a scoreboard too. If your RAG or support bot eval only asks "was the answer right?", every "I don't know" counts as a failure, and every prompt tweak that makes the model guess more will look like an improvement.

The post's suggested fix: penalize confident errors more than uncertainty, and give partial credit for appropriate expressions of uncertainty. OpenAI's Model Spec says the same thing in product terms: it's better to indicate uncertainty or ask for clarification than to give confident information that may be incorrect.

What option C looks like in practice

  • Give the model permission to abstain. Say explicitly in the instructions that "I couldn't find this in the provided documents" is a valid answer.
  • Abstain on evidence, not vibes. If retrieval returns nothing relevant, don't hand the model an empty context and hope.
  • Make abstention useful. Show what was searched, suggest a rephrase, or hand off to a human.
  • Log abstentions. A cluster of "I don't know" on the same topic is a content gap you can fix.

What option D adds

Split your eval results into three numbers instead of one: correct, wrong, abstained. Then decide what a wrong answer costs in your product compared to an abstention. For a medical or billing bot, a wrong answer is much worse than "I don't know". For a brainstorming tool, maybe not.

Once you track errors separately, you can see whether a change made the model smarter, or just bolder.

Your turn

A, B, C or D? And if you track abstentions: what abstention rate do you consider healthy?

References

Top comments (11)

Collapse
 
izgorodin profile image
Edward Izgorodin •

D, with one more split for features that answer from a store that changes over time. Some confident wrong answers are not guesses at all: retrieval found a record, the model quoted it faithfully, and the record had been replaced since. A similarity threshold will not catch it, because the old record often scores as high as the new one, both being about the same thing. If the eval counts it as its own class, a bad week tells you whether to fix retrieval or to fix how replacements are written, and the answer itself can say the useful thing: there are two records, and the newer one disagrees.

Collapse
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

Good point: stale-but-faithful answers are a separate failure class. I'd store a version or updated_at on each chunk and have retrieval flag when a newer record supersedes the one it cites, so the answer can surface both instead of quoting the old one.

Collapse
 
izgorodin profile image
Edward Izgorodin •

A version or updated_at tells you which record is newer, but not which one it replaces. Two records about the same customer can both be current, one about the plan and one about the address, and a newer timestamp on the second says nothing about the first. What lets retrieval flag the stale one without guessing is a link written at update time, saying this record replaces that one, so the old record can be shown as replaced instead of being ranked against the new one by score or date.

Thread Thread
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

That replaces-link is the right fix: a timestamp only orders records, it can't say which one supersedes which. I'd write a supersedes_id at update time and have retrieval drop or flag any chunk that a newer record points to before ranking.

Thread Thread
 
izgorodin profile image
Edward Izgorodin •

Dropping before ranking is the right default for questions about the current state. For questions about what it used to be, the same link is what lets the old record come back on purpose, so I would keep a replaced record out of ranking rather than out of the store. And if a newer record is linked by mistake, a flag shows the reviewer which record was hidden and by what, which a silent drop does not.

Collapse
 
recallrundev profile image
RecallRun •

D, mostly. The thing I'd add to C is that "evidence is weak" needs a number picked from labelled data, not a feeling.

I tried a top-1 cosine threshold on the FastAPI docs (MiniLM, 2,808 chunks) with 30 answerable and 30 unanswerable questions. At 0.40 it refused 24 of the 30 unanswerable ones and still answered 29 of 30 answerable.

The unrelated questions (boiling eggs, tides) were easy: all of them scored below 0.30, under the weakest real question at 0.39. The hard part was near-domain stuff like rate limiting or Prometheus metrics. Those scored up to 0.51 and overlapped the lowest answerable queries, so no single threshold separates them. That's where your answer-level check has to take over.

One trap: "retrieval returned nothing" doesn't work as the abstain signal with BM25. It returned candidates for all 60 queries, because words like "how" and "do" always match something.

Collapse
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

That overlap in the near-domain questions is the useful finding. When no single threshold separates them, I'd route the 0.35-0.55 band to a second check, like asking the model whether the retrieved chunk actually answers the question, and refuse only if that fails too.

Collapse
 
jes_p_d4f61f7aa2ec405a418 profile image
Dev_PJS •

C, with one amendment from hard experience: the abstention has to show its work or it decays into laziness with good PR. 'I couldn't find this' plus what was searched, where, and what would settle it — that's a useful answer. A bare 'I don't know' is just a refusal wearing the costume. On scoring: D matters more than it looks. In my world the expensive failure was never silence; it was a confident number that turned out to be invented, steering real money before anyone rechecked it. An eval that lumps abstentions with wrong answers trains exactly that failure — it teaches the model that guessing pays better than honesty. Score 'didn't know and said so' as its own class, and price a confident wrong answer at ten times an abstention. The behaviour you want shows up fast once the scoreboard stops punishing it.

Collapse
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

Pricing a confident wrong answer at ~10x an abstention is a great way to make the eval reflect real cost. I'd also have the abstention carry a structured reason (what was searched, what's missing) so you can grade that it was justified, not just that it refused.

Collapse
 
codingwithjiro profile image
Elmar Chavez •

Interesting. Thank you for showing the numbers. I got more in depth knowledge of AI. I think that old reward system has to go. Of course, bots would rather guess (little chance than no chance at all) for the reward. Decisions which require utmost care (i.e. hospital bills) are something the system should not hand off to AI entirely.

Collapse
 
m_t_ramkrushna profile image
M.T.Ramkrushna •

Agreed. The fix I'd start with is routing by stakes: high-cost decisions like billing get an abstain-and-escalate path to a human, and the eval scores a correct "I don't know" higher than a confident guess. That flips the incentive so guessing stops being the best strategy.