An unanswered question can look answerable
The 2018 paper Know What You Don’t Know introduced the dataset that became SQuAD 2.0. It added more than 50,000 adversarially written unanswerable questions to an existing reading-comprehension dataset. The questions were designed to remain relevant to their passages, and the passages contained plausible but incorrect answer candidates. A superficial match was insufficient. Systems had to answer when supported and abstain when the paragraph did not establish an answer. Rajpurkar, Jia and Liang, 2018
This was an extractive reading-comprehension benchmark, not a test of today’s commercial AI search products. Its enduring distinction is between a passage that mentions the right subject and a passage that establishes the requested fact.
Retrieval adds a separate sufficiency test
The December 2023 version of Benchmarking Large Language Models in Retrieval-Augmented Generation evaluated six language models using an English and Chinese benchmark called RGB. Retrieval-augmented generation means providing external material to help a model answer. RGB tested four abilities: tolerating irrelevant material, rejecting insufficient evidence, combining information and handling false information. Chen and colleagues, 2023
For its insufficient-evidence test, the external documents were selected from material that did not contain the answer. The authors reported difficulties with that task among the evaluated models. This is a result from those historical models and benchmark conditions, not a current failure rate for ChatGPT, Claude or any visibility platform.
The two studies examine different systems and tasks. Their results should not be combined into one accuracy figure. Together, they provide a reason to include unanswerable cases when evaluating a tool that claims to answer from documents.
Make the missing fact visible
Consider this fictional source passage:
The Alder lamp includes a two-year warranty covering manufacturing defects. Keep your purchase receipt when requesting service.
Now compare two questions against that same passage:
Scroll horizontally to read all columns. Use the arrow keys when the table is focused.
| Question | What the passage establishes | An evidence-bounded response |
|---|---|---|
| How long is the warranty? | A two-year duration | The stated duration is two years. |
| Can a second owner use the warranty? | No transfer rule is stated | This passage does not establish whether the warranty transfers. |
Neither the receipt requirement nor the two-year duration supplies the transfer rule. An evaluator should check for the missing relationship, not merely the presence of warranty-related words. This worked example is ours; it is not a benchmark question, model output or observed customer interaction.
Original fictional example. No model was run and no real warranty policy is described.
Score the response and the evidence separately
For a later evaluation, pair questions with a fixed, inspectable source packet. Include supported questions and plausible questions for which that packet lacks a required fact. Save the full answer and the exact source version, and judge whether each substantive claim follows from the permitted evidence.
That protocol is a proposal informed by the studies, not a test we ran. Its scope matters. A system restricted to a supplied packet should identify a gap in that packet. An open-web assistant may legitimately find another source that answers the question; evaluate that source rather than requiring refusal merely because your page is incomplete.
A timeout or empty response also needs its own category. Neither demonstrates that the assistant recognized insufficient evidence. Conversely, a completed response that explains a genuine evidence gap can be useful even without a definitive answer.
A citation does not settle answerability
For teams reviewing AI answers about their business, the next question after opening a citation is whether the source establishes the particular fact being asserted. Related material may support part of a response while leaving another claim unresolved. The warranty example shows how small that gap can be.
Recognizing the gap is an assessable behavior. These historical studies support testing it; they do not establish which current product does it best or that adding a sentence to your site will change an assistant’s answer.
Sources and scope
Published by Gist, which sells AI visibility software. Primary papers checked November 13, 2026; research dates are 2018 and December 2023. This is a reading of those studies with an original fictional example. We ran no new models, used no customer dataset and make no current-engine ranking or visibility-uplift claim.