The Evidence-Saturation Point of LLMs — Toward an Open k Registry for Local Models*

Research
1

I have posted a small pilot paper on the evidence-saturation point k^* of language models.

The basic question is practical: for a given model, task type and context format, how many evidence fragments should actually be injected into the prompt?

More context is not monotonically better. It may increase cost, distraction and epistemic contamination without improving correctness. A correctness-only sweep can therefore make capable models appear saturation-free while degradation remains visible on another reliability axis.

In the first task battery, the correctness-optimal k ranged from k=1 to the full evidence set across task types for the same model. This suggests that a single context profile per model is insufficient; k^* should be calibrated per model, served backend, task type and reliability objective.

The paper, benchmark code and datasets are public. I would be particularly interested in results from people running local models.

A potentially useful next step would be an open registry in which operators publish measured k-profiles for their models, tasks, quantizations and context formats. I cannot afford to benchmark the entire local-model ecosystem myself, but the protocol was deliberately designed to be small and reproducible.

Criticism, replications and measured k-curves are welcome.

IMG_3436
IMG_34361990×795 159 KB

https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7038059

PS: I have now created an open Evidence-k Registry on Hugging Face, seeded with the 35 task-level measurements from the paper:

Contributions from local-model operators are welcome. Each entry should document the model revision, quantisation, served backend, task type, metric axis and reproducibility source.

2

I encountered a closely related problem while developing SEMA, although in an explicit relational-memory setting rather than an LLM prompt.

During retrieval, query fragments climb through the graph toward the learned contexts that contain them. Initially, allowing more reachable evidence seemed beneficial, but common fragments behaved like hubs: they reached too many contexts, contributed mostly noise, and diluted the genuinely discriminative parts of the query.

I addressed this with an adaptive saturation mechanism rather than a fixed global k. Each region expands only until the system can decide whether it yields a concrete, discriminative vote or has become saturated. Fan-out is capped at √N, votes are weighted by inverse context frequency, ambiguous matches must clear a contrastive noise margin, and saturated regions abstain instead of adding weak evidence. A context is accepted only after pooled evidence exceeds what any single region could contribute alone, requiring actual cross-region corroboration.

A useful consequence is that common boilerplate is not merely discarded: it is marked as scaffolding, allowing later evidence to be evaluated without the preamble diluting the payload.

So I strongly agree that the useful evidence count is task- and objective-dependent. My experience suggests that k may be better treated not only as a calibrated retrieval parameter, but as an endogenous stopping decision: continue adding evidence only while it increases discrimination or independent corroboration, and stop when additional evidence primarily increases reach rather than information.

3

Your concerns are genuine. The alignment between your concerns and research interests and my own is striking. Please feel free to stay in touch.

4

Thank you - this is exactly the kind of adjacent implementation I hoped this discussion might surface.

I agree that what you describe is not merely another fixed-k heuristic. It turns the evidence budget into an endogenous stopping decision based on marginal discrimination and independent corroboration.

I see the two approaches as complementary. In the paper, k^* is an offline calibration primitive for a specified model, task, context format and reliability objective. Your mechanism is an online policy that decides, for the individual query, whether expanding another region still contributes information. A calibrated k^* could therefore act as a prior or safety boundary, while the adaptive mechanism determines the realised k per instance.

Your treatment of common boilerplate as scaffolding rather than decision-bearing evidence is particularly interesting. It preserves structural context without allowing high-frequency fragments to dilute the discriminative payload.

This also suggests that an open registry may eventually need to record more than a single point estimate: for adaptive systems, the realised k-distribution, stopping criterion and reliability objective may be the more meaningful profile.

I would be very interested in a SEMA contribution or short design note for the registry, even if it becomes the first case requiring an explicit “adaptive stopping” category.

5

I like your distinction between offline calibration and online adaptation.

One thing I’d add is that, in SEMA, the stopping criterion is not based solely on diminishing returns. It is also driven by structural convergence. As evidence accumulates, independent derivation paths begin to reinforce the same higher-level contexts. Once the hypothesis stabilizes, exploring additional regions rarely changes the outcome—it mostly adds computational cost or semantic noise.

In that sense, SEMA doesn’t try to find the optimal amount of evidence beforehand. It tries to recognize when the semantic structure has become self-consistent. I suspect this notion of convergence may generalize beyond graph-based systems and could even be useful for LLM retrieval or context assembly.

6

This discussion is important to me as well. Like you, I worked on SEMA as an independent researcher. However, when it came to generating the practical artifact—the code—I encountered and resolved numerous scattered issues that aren’t documented in the initial research; “saturation” was one of them. Feel free to contribute however you’d like.

7

After looking through the implementation, I think SEMA would be an excellent first adaptive-stopping case for the registry. However, the instrumentation is probably best implemented on your side because the relevant information is deeply embedded in the consensus climb and saturation logic. I would be happy to help define the measurement schema and interpret the resulting profile.

8

Sema code allows you to use its explanation mechanism—specifically the “inspectRationale” parameter—for these instrumentation issues. I’ve used this a lot for troubleshooting. It might need some improvements for this specific case, though; I’ll look into it.

9

That sounds ideal. If you expose the saturation reasons and the distinction between candidate breadth and actually contributing evidence, I can help map the output to a registry schema and interpret the resulting profile.

10

I will expose the data you need in `inspectRationale`. I’ll let you know as soon as I’ve updated it.

11

All set. I’ve significantly improved the machinery tracking for the climb, and the data you need will likely be available via inspectRationale.

13

Appreciate the paper — the blind-axis finding will become probe #7 in my characterization battery. Was running six axes (correctness, determinism, format compliance, injection resistance, speed, knowledge). You showed me all six can certify a model as flawless while contamination sits underneath at 0.05. Adding DESi markers. Thank you.

The thing you wanted but couldn’t get: I run Qwen3-30B-A3B IQ3_XXS on Apple Silicon — Metal GPU, ngl=99 — fixed T=0, verified byte-identical across 16,900+ deterministic runs. Zero counterexamples. One quant, one backend across all k values. I poll the GPU die directly over IOKit. Your §9.3 caveat about provider routing noise doesn’t just not apply here — on this hardware, your protocol finally gets the measurement environment it was designed for.

14

Correction and withdrawal of the evidence-k paper

I owe the community an apology.

After additional blind testing and independent adversarial review, I have concluded that the central claims of my evidence-k paper are not supported.

The experiments measured behavior under a specific benchmark, using small synthetic tasks and heuristic scoring. I overstated what those results demonstrated. They do not establish a general or transferable evidence-saturation point (k), nor do they justify the broader conclusions I drew about context calibration and governance.

The paper therefore does not provide the evidence I claimed it did.

I am withdrawing its conclusions.

I sincerely apologize to everyone who spent time reading, discussing, reviewing, or evaluating this work. I presented claims with more confidence than the evidence justified, and I am sorry for wasting your time.

Thank you to everyone who challenged the work critically. Those criticisms ultimately revealed methodological flaws that I had failed to recognize.

Steffen Rentschler