Moving the relevant passage changed results in a 2023 study
Nelson F. Liu and colleagues' Lost in the Middle used 2,655 questions from the NaturalQuestions dataset in experiments with 10, 20 or 30 documents. An answer-bearing Wikipedia passage appeared among distractors. Performance was often stronger when that passage was near the beginning or end of the input than in the middle. November 20, 2023 version.
These were supplied-text experiments, not tests of how a live search product retrieves, orders or cites a website.
RULER tested more than finding one hidden fact
Cheng-Ping Hsieh and colleagues' RULER benchmark tested 17 models across 13 task configurations. Tasks required finding and combining facts or following relationships. Almost all models performed worse with longer input. August 6, 2024 version.
The authors caution that these synthetic tasks still need validation against realistic uses.
Scroll horizontally to read all columns. Use the arrow keys when the table is focused.
| What the research varied | What the result can inform | What it does not establish |
|---|---|---|
| Position of relevant information in supplied text | Whether the tested model remained effective across those positions | Where a live assistant placed your page's text |
| Context length and task complexity | Whether a capacity claim was accompanied by useful task performance in the benchmark | That every longer input harms every current model |
| Finding facts, following linked facts or combining information | Which task the evaluation rewarded | Brand discovery, commercial recommendations or citation gains |
Gist has not rerun these benchmarks. The table describes historical scope.
Define the task before evaluating a document-reading claim
Consider an original fictional example: a team supplies three policy documents. One contains a standard cancellation rule, another contains an exception for a particular service, and a third contains an old version of that exception. The team asks which rule applies to a specified booking date.
Finding the word “cancellation” would not complete the task. A useful answer needs the applicable version, the exception and the booking date. If the team wants citations, it should also check whether the linked passage supports the stated rule. This example is an evaluation design, not an observed model failure or a result from either paper.
A review sheet can make those expectations explicit:
Scroll horizontally to read all columns. Use the arrow keys when the table is focused.
| Check | Evidence the team would retain |
|---|---|
| Exact task | The question, booking date and expected scope of the answer |
| Supplied material | Document versions and the passages containing the rule and exception |
| Result | Original answer, relevant quoted or linked support and any unavailable response |
| Interpretation | Whether the rule was correct, whether the source supported it and what remained uncertain |
Use material the team is authorized to test, and agree on a correct answer before judging the output. If a provider offers a demonstration, ask how its records distinguish input that was supplied, content that was retrieved and a source that was cited. Missing records should stay unknown instead of becoming a story about why the model behaved as it did.
For website visibility work, retain that separation. A public page, a retrieved passage and a cited URL describe different observations. Neither study establishes a page-editing tactic for gaining citations.
Gist publishes this article and sells AI visibility software. The same evidence requirement applies to our claims: name the system, task and observed outcome before saying what a large context window delivered.