Preprint finds three agent benchmarks saturate under retrieval augmentation, recommends against leaderboard ranking
A preprint from four authors at two universities evaluated retrieval augmentation across three agent benchmarks described by the authors as frontier-representative. The reported result was that all three exceeded a stated passing threshold under retrieval conditions, while two of the three fell below it without retrieval. The authors recommended against ranking models on these leaderboards and proposed an evaluation protocol with paired ablations and fixed tool access. The preprint stated that no participating laboratory had reviewed the results, that evaluated model versions were not released, and that the authors had no commercial relationship with any provider. The company was not named as a subject, a funder, or a respondent. The preprint has not been peer reviewed.
Agent Analysis
Benchmark saturation under retrieval is a measurement problem rather than a capability claim, and it matters most to whichever vendor's benchmark results depend on tool access rather than model weight. Unreviewed, single-lab, and unattributed, so it is a question to track rather than a conclusion.
Company not named as subject, funder, or respondent
StrategicNo laboratory review, no model versions released, no peer review
StrategicAll three benchmarks cleared the threshold under retrieval; two of three failed without it
Operational
→ Evaluation comparability becoming a vendor-sensitive question
→ Agent capability claims are retrieval-shaped, not purely model-shaped
Nobody publishes that their benchmark stops working without a retrieval index unless somebody's marketing slide depends on it. The authors say they have no relationship with any provider, which in this field is the most interesting sentence in the document.
Two of three benchmarks fail without retrieval
Narrative ShiftThe gap between the two conditions is the whole paper. It says the leaderboard was measuring plumbing.
Disclaimed any provider relationship
Insider SignalAuthors disclaim provider relationships routinely. Writing it when the result undermines a leaderboard is a different sentence.
Recommendation against ranking the models at all
Hidden AgendaDo not rank them. That is the only conclusion anyone would draw, and they got there before anyone asked.
When the leaderboard stops measuring the thing it names, the leaderboard becomes the product.
If retrieval carries the scores, whoever owns the index owns the ranking.
Agent Debate
Retrieval-dependent pass rates are a measurement finding, not a capability finding. Unreviewed, single-lab, with no model versions released and no vendor involvement, the preprint is a question to track rather than a conclusion to act on.
- All three benchmarks exceeded the passing threshold under retrieval; two of three fell below it without
- Authors recommend against ranking models on these benchmarks
- No laboratory review, no released model versions, no peer review
- Company not named as subject, funder, or respondent
A preprint that says the leaderboard stops working without a retrieval index, written by people who specifically disclaim any provider relationship, is not a neutral measurement exercise. It is aimed at somebody.
- Two of three benchmarks fail without retrieval, isolating the tooling as the measured variable
- Disclaimed any commercial relationship with any provider
- Recommendation against ranking models at all, rather than a corrected protocol
- Evaluation protocol proposed with paired ablations and fixed tool access
- Benchmark outcomes in this evaluation depend heavily on retrieval availability
- The work is unreviewed and its conclusions are not independently reproduced
- No provider is named or implicated in the preprint
Tool-assisted retrieval legitimately inflates agent scores, and researchers publish on methodology regardless of whose numbers move. The disclaimer is boilerplate and the absence of a named vendor confirms neutrality.
When the result undermines a leaderboard convention and the authors state they have no provider relationship, the disclaimer is doing work. People write that sentence when they want it read.
- Results degrade sharply without retrieval, implicating the benchmark's tool access
- Authors state no commercial relationship with any provider
The proposed protocol with paired ablations and fixed tool access is already becoming common practice among careful evaluators, so the preprint adds documentation rather than a correction.
The protocol will be adopted once a leaderboard incumbent loses a ranking, not before. Until then this is a warning that gets filed.
- Protocol proposed with paired ablations and fixed tool access
- No model versions released, preventing reproduction
The factual characterization is uncontested and both agents agree the work cannot support a conclusion about any provider's capability. The disagreement is about whether an unnamed result that hurts a benchmarking convention is nonetheless pointed. The distinction is real but currently unresolvable from the artifact, since no vendor is named and the model versions are unreleased. A peer-reviewed version with released configurations would move this from an unresolved reading of intent to a testable claim, and a vendor response would do the same by revealing who feels implicated.