Among questions of the same context length, how strongly are evidence scope, evidence dispersion, and reasoning depth associated with model performance?
FindingAcross questions, greater scope, dispersion and reasoning depth are each associated with lower scores after accounting for context length, source dataset and model. Notably, doubling reasoning depth is associated with a score drop comparable to that of doubling context length.
How does the effect of context length on performance vary with evidence scope, evidence dispersion, and reasoning depth?
FindingAs context length increases, performance degrades sharply on questions whose evidence is widely dispersed, whereas it remains relatively stable on questions with localised evidence. This pattern holds across all evaluated models.
Are model failures associated with particular combinations of scope, dispersion, and depth, even when no individual property is extreme?
FindingDifficulty compounds across properties: questions with moderately high values on all three properties are harder than those with an extreme value on just one. This pattern is broadly consistent with an additive combination of the three properties.
For a context of \(L\) tokens with merged, disjoint evidence intervals \(E = \{e_1, \dots, e_N\}\), \(e_i = [a_i, b_i]\), we measure how much evidence is needed, how widely it is spread, and how many dependent steps it takes to reach the answer.
How much of the context is evidence?
\[ \mathrm{Scope}(E) = \frac{S(E)}{L}, \qquad S(E) = \sum_{i=1}^{N} \ell_i \]The fraction of context tokens that belong to the supporting evidence.
How spread out is the evidence?
\[ \mathrm{Disp}(E) = \frac{H(E) - S(E)}{L - S(E)} \]The share of all non-evidence text that lies inside the evidence span \(H(E) = b_N - a_1 + 1\). It is 0 for a single block and 1 when the evidence spans the full context.
How many dependent steps are needed?
\[ \mathrm{Depth} = \max_{\text{path}\,\to\,\text{answer}} |\text{path}| \]The number of nodes on the longest dependency path in an evidence-to-answer reasoning graph. Parallel lookups do not add depth.
A 100K-token context. Click the bar to add evidence, drag blocks to move them, drag their edges to resize, and right-click to remove.
Each derivation is a directed acyclic graph whose nodes are atomic operations (lookups, arithmetic, comparisons, aggregations) and whose edges are direct dependencies. The two release years are read independently, so both sit at level 1; comparing them is one further step.
Graphs are generated conditioned on the gold answer and evidence, validated and pruned automatically, and checked by two human auditors on 40 questions.
Long-context Understanding and Reasoning from Contextual Hints: naturally occurring long-context questions, each with an answer, verified supporting evidence and a reasoning graph.
26 candidate sources, 323,040 questions.
English, 32K–256K tokens, natural contexts, reasoning-heavy tasks, not solvable without the context.
Use reference evidence, or retrieve and expand passages, then keep only questions where a verifier answers correctly 3/3 times from the evidence alone.
Answer-conditioned generation, validation and pruning, plus a human audit.
6,550 questions from 17 sources, annotated with length, scope, dispersion and depth.
| Source | Questions | Avg. context | Avg. question | Avg. answer | Metric |
|---|---|---|---|---|---|
| SciTrek (subsampled from 5,491) | 1,000 | 82,536 | 132 | 18 | F1 |
| HotpotQA (extended) | 1,663 | 93,580 | 65 | 5 | F1 |
| DocFinQA | 1,994 | 136,392 | 46 | 4 | F1 |
| LongMemEval | 226 | 114,241 | 89 | 4 | F1 |
| LongBench-Pro | 181 | 111,334 | 214 | 22 | F1 |
| ∞Bench (code-debug) | 356 | 117,695 | 142 | 1 | EM |
| ∞Bench (longbook-qa) | 87 | 152,110 | 39 | 3 | F1 |
| ∞Bench (longdiag-qa) | 130 | 106,145 | 68 | 3 | F1 |
| ∞Bench (longbook-choice) | 162 | 146,943 | 95 | 1 | EM |
| DocQA-RL | 57 | 47,062 | 169 | 8 | F1 |
| LV-Eval | 60 | 187,699 | 83 | 4 | F1 |
| LongBench v2 | 141 | 111,760 | 224 | 1 | EM |
| CorpusQA | 99 | 93,748 | 261 | 6 | F1 |
| L-Eval | 18 | 46,507 | 138 | 31 | F1 |
| Loong (paper-cite-relation) | 215 | 82,319 | 304 | 73 | LLM judge |
| Loong (paper-cite-chain) | 97 | 80,050 | 262 | 63 | LLM judge |
| Loong (financial) | 64 | 135,183 | 106 | 16 | LLM judge |
Lengths are average token counts. Full filtering criteria and thresholds are in Appendices B and C of the paper.
We ran four models (GPT-5.6 Luna, Gemini 3.8 Flash, Qwen3.8-27B and Qwen3.5-4B) on all 6,550 questions with their full documents, three answers per question. To separate each property's effect from the others, we fit a regression that compares questions within the same task and model. Effects are in percentage points (pp) of score: −2 pp means a score of 80% drops to 78%.
Take two questions from the same task that differ in one property, one at the easier end (25th percentile) and one at the harder end (75th percentile), with everything else the same. Each bar shows how much lower the harder question is predicted to score. All three properties lower scores, and scope and depth matter about as much as context length itself.
Fractional-logit generalised linear model (GLM) with task and model intercepts, fit on log-transformed, standardised predictors (Equation 4 in the paper). The three properties are jointly significant (Wald test \(\chi^2_3 = 73.0\), Holm-adjusted \(p = 4.0 \times 10^{-15}\)) and each is individually significant (adjusted \(p \le 4.0 \times 10^{-3}\)). “Easier → harder end” is the within-task interquartile contrast (25th to 75th percentile), averaged over all questions. Confidence intervals are clustered by question.
What happens when the document gets longer? Each row shows the score drop from a shorter to a longer document in the same task (25th to 75th percentile of length), for questions low (hollow dot) and high (filled dot) on one property. The drop is about twice as large when evidence is broad or scattered, and this holds for all four models. For reasoning depth, the difference is too uncertain to call.
“Difference” is the extra drop for questions high on the property, with a 95% confidence interval. Averaged over all 6,550 questions (Table 2).
Each property gets its own model, which adds a length × property interaction to the RQ1 regression (Equation 5 in the paper). For scope and dispersion, both the difference and the interaction coefficient are significant after Holm adjustment over six tests (adjusted \(p = 5.5 \times 10^{-3}\) and \(1.0 \times 10^{-2}\)). For depth they are not (adjusted \(p = 0.18\)).
Within each task, we split questions into low, mid and high thirds on two properties, and show how each group scores compared with the average for its task and model. Blue groups are easier than average and red groups are harder. Scores fall steadily as both properties rise, so several moderate demands together can make a question hard. Hover over or tap a cell for details.
Exploratory analysis without formal significance tests (Figure 6, Table 12).
Groups are within-task terciles. “Gap from additive prediction” is the group's observed score minus the out-of-fold prediction of the RQ1 regression, which assumes the properties' effects simply add up. A gap near zero means the effects add up; a negative gap means the group is harder than that. Intervals are individual 95% intervals, not corrected for comparing many cells.
Takeaway. Context length is an incomplete measure of long-context difficulty. We recommend that long-context benchmarks report evidence scope, evidence dispersion and reasoning depth alongside length.
@misc{qin2026beyond,
title = {Beyond Context Length: Characterising the Difficulty of Long-Context Reasoning},
author = {Qin, Shiwen and Barker, Cameron and Gouk, Henry and Crowley, Elliot J. and Cohen, Shay B. and Lapata, Mirella and Li, Miao},
year = {2026},
note = {Preprint}
}