Hmm… this is not rigorous, but I ran a small experiment under roughly similar conditions and was able to reproduce a similar pattern. I still do not really understand why, though:
My current answer would be:
The non-monotonic result may be real under your benchmark, but I would not yet interpret it as a general memory defect in Qwen2.5-3B, or as proof of inverse scaling.
“Reconstruction” can combine several different operations:
- accessing the supplied information;
- selecting the correct first item;
- keeping multiple items in the requested order;
- copying their exact surface form;
- maintaining a long autoregressive output;
- stopping at the correct point;
- passing the parser/scorer.
A model can therefore perform similarly on retrieval while performing differently on full reconstruction.
A compact default route
I would use this sequence before trying to name the cause:
- Fix the exact checkpoints and revisions.
- Use the same explicitly specified deterministic generation settings for all three models.
- Inspect one complete raw example before relying on aggregate scores.
- Split the reconstruction result into item, start, order, format, rollout, and stop components.
- Vary input load and required output load independently.
- Check whether the model ranking survives a few meaning-preserving prompt variants.
The most reusable example would contain:
- exact model IDs and revisions;
- one complete input and expected output;
- the raw output from each model;
- the scorer or exact scoring rule;
- generation settings, library/runtime versions, token counts, and stop reasons.
That single example would already distinguish many otherwise similar-looking failure modes.
What I observed in a small synthetic check
This was only a small synthetic sanity check, not a reproduction of your benchmark.
I used the official Qwen2.5 Instruct checkpoints:
Qwen/Qwen2.5-0.5B-InstructQwen/Qwen2.5-1.5B-InstructQwen/Qwen2.5-3B-Instruct
with a simple exact-copy task containing sequentially labelled records such as R001, R002, and R003.
Under some prompt conditions, I also obtained a non-monotonic result: the 1.5B model was relatively stable, while the 3B model had worse exact reconstruction.
The interesting part was the shape of the failure. In several 3B cases, the output did not become random and did not immediately stop. Instead, it began at R002 or R003 rather than R001, and then copied the following records accurately from that later position.
Small changes to the system/user roles, clause order, wording, or record markers recovered many of those cases.
For my synthetic examples, the most useful description was therefore not “general memory loss,” but:
a prompt-conditioned failure to select the intended output start
Your raw outputs may show a different pattern, so I would treat this only as one candidate failure class.
A small diagnostic scorecard
| Component |
Useful observation |
| Item access |
Does every required item appear anywhere? |
| Start |
Which item or marker is generated first? |
| Order |
Are the recovered item IDs in the required sequence? |
| Format |
Is the mismatch only wording, whitespace, markers, or layout? |
| Rollout |
Are later items omitted, duplicated, substituted, or repeated? |
| Stop |
Did generation end at EOS, a token limit, an external stop rule, or unfinished output? |
| Evaluation |
Did the raw output fail, or only the parser/scorer result? |
A strict full-string exact-match score is still useful. I would keep it, but add these diagnostic measurements beside it.
A quick interpretation guide:
- Each item works when requested alone, but the full reconstruction fails: sequencing or long-output rollout becomes more likely than simple information access.
- The output starts from item 2 or 3 and is otherwise correct: start selection or ordering becomes more likely.
- All items appear but the score is low: inspect parsing, normalization, order, and formatting.
- A small prompt or marker change reverses the ranking: prompt–checkpoint interaction becomes important.
- The 3B model remains worse across deterministic runs, multiple reasonable prompts, and more than one runtime: evidence for a task-specific checkpoint difference becomes stronger.
What is still unknown, and why I would not choose a root cause yet
I am interpreting “reconstruction” as producing multiple previously supplied items, possibly in their original order and wording. If your definition is different, some of the branches below would change.
Several details are currently unknown:
| Unknown |
Why it changes the interpretation |
| Exact model repositories |
Official, converted, fine-tuned, and quantized checkpoints can differ. |
| Base or Instruct |
Instruct models introduce chat-template and alignment behavior. |
| Reconstruction target |
Verbatim copying, semantic reconstruction, ordered extraction, and summarization are different tasks. |
| Complete raw outputs |
Aggregate scores do not reveal omission, wrong order, repetition, format drift, or early stopping. |
| Scorer |
Exact match, normalized match, item parsing, substring matching, and model grading measure different things. |
| Generation settings |
Sampling, penalties, EOS handling, beams, and output limits affect exact-copy behavior. |
| Rendered prompt |
Human-readable messages are not necessarily the final token sequence given to the model. |
| Input/output token counts |
“More information” may also mean more records, distractors, similar prefixes, and required output tokens. |
| Runtime and batching |
Backend, dtype, quantization, padding, and batching can add variables. |
| Stop reason |
EOS, output-budget exhaustion, and application stop strings are different failure modes. |
Because these variables are unknown, I do not think the aggregate ranking alone identifies a root cause.
A compact record for one reproducible sample could look like:
model_id:
revision:
base_or_instruct:
runtime:
library_versions:
dtype:
quantization:
attention_backend:
batch_size:
messages:
rendered_prompt:
input_token_count:
expected_output:
raw_generated_token_ids:
raw_decoded_output:
scorer_input:
score_breakdown:
generation_kwargs:
generated_token_count:
termination_reason:
A controlled generation baseline
For a diagnostic baseline, I would remove optional decoding behavior:
generation_kwargs = {
"do_sample": False,
"num_beams": 1,
"repetition_penalty": 1.0,
"no_repeat_ngram_size": 0,
"max_new_tokens": <SAME_VALUE_FOR_ALL_MODELS>,
"return_dict_in_generate": True,
"output_scores": True,
}
I would also keep the following identical:
- EOS token IDs;
- stop strings or stopping criteria;
- batch size;
- prompt/template;
- dtype;
- quantization status;
- runtime/backend;
- truncation policy.
This is not a claim that greedy decoding is always the best application setting. It is simply a lower-variance diagnostic condition.
Transformers can inherit generation behavior from a checkpoint’s generation_config.json. The current official Qwen2.5 Instruct repositories do not store completely identical defaults:
At the time of checking, all three stored sampling settings, while the repetition penalty was 1.1 for 0.5B/1.5B and 1.05 for 3B.
That difference does not naturally explain why 3B would be worst—the 3B penalty is lower—but silently inheriting checkpoint-specific defaults makes the comparison less controlled.
For an exact-copy task, repetition_penalty=1.0 is also a useful control because the desired output intentionally repeats tokens that already occurred in the prompt. The handling of prompt tokens by repetition penalties has been discussed in Transformers issue #36642. I would treat that issue as an implementation check, not as evidence that it caused this result.
Useful logs include:
model.generation_config
explicit generate() keyword arguments
resolved eos_token_id
resolved pad_token_id
input token count
generated token count
termination reason
Testing the scorer before running a larger experiment
A low-cost option is to pass artificial outputs through the scorer first.
Useful fixtures include:
A. Perfect match
B. Only the first item is missing
C. Only a middle item is missing
D. All items are present, but in the wrong order
E. All items are present with explanatory text around them
F. Only whitespace or line breaks differ
G. One item is duplicated
H. The output is correct until it is truncated at the end
I. Record IDs have prefix collisions, such as R001 and R0010
J. The output starts from R002 and is otherwise a perfect suffix
K. The content is correct, but the marker format changed
L. The expected first item appears later in the output
I would keep two scoring layers.
Strict result
- raw full-string exact match;
- exact format match.
Diagnostic result
- item precision and recall;
- first generated item;
- whether the intended first item appears anywhere;
- recovered item-ID sequence;
- order accuracy or sequence edit distance;
- duplicate count;
- extra-item count;
- format-only mismatch;
- generated-token count;
- termination reason.
The strict metric does not need to be weakened or discarded. The diagnostic layer explains why it failed.
The Inspect scorer documentation is a useful example of treating whole-output exact matching, substring inclusion, location-aware matching, extraction, and multiple scorers as separate operations.
Where the record syntax is deterministic, a simple parser is probably easier to audit than an LLM judge.
Separating input load from required output load
“Amount of provided information” may increase several variables simultaneously:
- total input tokens;
- number of records;
- number of distractors;
- number of requested output items;
- required output tokens;
- number of similar record prefixes;
- distance between relevant items;
- number of competing continuation patterns.
A small two-by-two design can separate some of them:
|
Short required output |
Long required output |
| Short input |
Baseline |
Mainly sequencing/rollout stress |
| Long input |
Mainly access/selection stress |
Mixed condition similar to the original task |
Input-load option
Keep the requested output to one item while increasing:
- distractor count;
- total context length;
- target position;
- density of similar records.
A decline here would be more consistent with access, selection, or positional effects.
Output-load option
Keep the input fixed while requesting:
- 1 item;
- 2 items;
- 4 items;
- 8 items;
- the complete set.
If individual items remain answerable but the full set degrades, ordering, exact copying, long rollout, or stopping become more plausible.
Several existing benchmarks provide useful experimental vocabulary:
- RULER extends simple needle retrieval with multiple needles, tracing, and aggregation.
- Sequential-NIAH studies extraction of multiple sequential items and their order.
- LongProc combines dispersed-information integration with structured long-form output.
- LongGenBench focuses on long-context generation rather than only short-answer retrieval.
These are analogous tasks, not evidence that your benchmark has the same mechanism. Their value here is the separation between accessing information and producing a long ordered reconstruction.
Position, distance, order, similarity, and marker options
The following controls can be varied independently where practical:
| Control |
Keep fixed |
Change |
Possible information |
| Absolute position |
Item content and count |
Beginning, middle, or end placement |
Position sensitivity |
| Relative distance |
Relevant item set |
Cluster or spread relevant items |
Integration across separated information |
| Input order |
Item set |
Shuffle record order |
Dependence on presentation order |
| Requested order |
Input |
Original, reverse, or sorted order |
Ordering versus access |
| Similarity density |
Target records |
Add similar distractors |
Candidate interference |
| Marker style |
Values and order |
Numeric, word, or random IDs |
Prompt/tokenization interaction |
| Prefix uniqueness |
Record count |
Colliding or unique prefixes |
Parser and candidate ambiguity |
| Output schema |
Requested facts |
Lines, JSON, TSV, or delimited blocks |
Schema initiation and adherence |
Lost in the Middle is relevant to absolute-position effects, although it primarily studies retrieval rather than exact reconstruction.
A simple marker comparison could be:
Condition A:
R001
R002
R003
Condition B:
ALPHA
BRAVO
CHARLIE
Condition C:
7f3a
c918
2d6e
If the ranking changes sharply across marker systems while the underlying records remain identical, prompt/tokenization sensitivity would become a useful part of the interpretation.
Chat-template and output-start diagnostics
For Instruct checkpoints, visible chat messages are converted into model-specific control-token sequences. The Hugging Face chat-template guide explains the roles of chat templates, add_generation_prompt, assistant prefilling, and special-token handling.
The official Qwen examples use apply_chat_template before generation. I would retain:
original messages
rendered chat-template text
final input token IDs
That helps identify differences that are invisible when only the natural-language prompt is reported.
A small prompt-robustness check might compare:
- system instruction plus user data;
- everything in one user message;
- instruction before the records;
- instruction after the records;
- alternative record markers;
- an explicit output-start phrase.
The purpose is not to search indefinitely for a prompt that makes every model succeed. It is to check whether the 3B-worst ranking is stable under reasonable, meaning-preserving variations.
A useful report would contain:
- score by model and prompt variant;
- median and range;
- how often the ranking is preserved;
- failure-type counts.
Prompt-format sensitivity is documented in Sclar et al., where meaning-preserving formatting changes sometimes caused substantial model-specific performance differences. That does not prove prompt sensitivity caused this result, but it makes a small robustness check reasonable.
Optional prefill diagnostic
If the visible failure is “the output starts from the wrong record,” a prefix test can separate initiation from later reconstruction:
- no assistant prefill;
- prefill only the output format;
- prefill the first record marker, for example
R001.
The chat-template documentation describes continue_final_message=True for continuing an assistant prefix. It should not be combined with add_generation_prompt=True.
Possible interpretations:
- Format prefix alone recovers the output: output-mode initiation is a candidate.
R001 specifically is needed: first-item/start selection is a candidate.- The start is corrected but later records still degrade: sequencing or rollout is also involved.
- One-item queries still fail: information access needs more attention.
Prefilling is a diagnostic option, not necessarily the intended benchmark condition.
Optional token-level diagnostic
If the effect is stable, Transformers can expose generation scores:
outputs = model.generate(
**inputs,
return_dict_in_generate=True,
output_scores=True,
**generation_kwargs,
)
At the first divergence, candidates might include:
- the intended first marker;
- later record markers;
- newline or schema tokens;
- EOS.
For multi-token markers, comparing only the first token can be misleading. A more complete diagnostic is the teacher-forced log probability of each candidate prefix.
Runtime, tokenization, decoding, and stopping branches
A simple baseline could be:
official checkpoint
Transformers
batch size 1
no quantization
deterministic generation
raw token IDs retained
Deployment-specific optimizations can then be added one at a time.
If behavior changes only in a batch
Compare:
- batch size 1;
- equal-length batched prompts;
- the actual variable-length batch.
For decoder-only generation, padding direction can matter. A relevant implementation discussion is Transformers issue #34842.
If the first records disappear
Record token counts after applying the chat template, and log:
tokenizer.model_max_length
truncation enabled or disabled
truncation_side
actual retained input IDs
A truncated first record could resemble wrong-start behavior while actually being absent from the model input.
If the raw output and displayed result differ
Save these separately:
- generated token IDs;
- direct tokenizer decode;
- decode after special-token removal;
- text after application parsing;
- exact scorer input;
- displayed output.
This separates model behavior from decoding, parsing, or interface behavior.
If outputs end early
Classify termination as:
EOS
max_new_tokens or length limit
custom stop string
runtime interruption
unfinished or repetitive generation
A completion cut off by the output budget is different from a model selecting EOS, and neither automatically indicates retrieval failure.
In my synthetic checks, changing Transformers versions, KV-cache use, attention implementations, quantization, output limits, or early-EOS assumptions did not by themselves explain the observed wrong-start pattern. That only narrows my local observation; it does not exclude those variables in your setup.
Outcome-to-interpretation map
| Observation |
More consistent with |
Does not establish |
| No required items appear |
Access failure, truncation, task misunderstanding, or derailment |
A general architectural memory defect |
| Each item works when queried separately |
Individual access is available |
Correct full ordered reconstruction |
| Output starts at item 2/3 and then copies correctly |
Start selection, order, or prompt-conditioned continuation |
That item 1 was absent from the model’s internal state |
| All items appear in the wrong order |
Sequencing failure |
Retrieval failure |
| Content is correct but wording changes |
Verbatim-fidelity or schema-following failure |
Loss of semantic information |
| Correct beginning followed by omission/duplication |
Long-output rollout or state tracking |
Initial retrieval failure |
| Output reaches the token limit |
Output-budget confound |
Early EOS or forgetting |
| Correct raw output receives a low score |
Parser, normalization, or scorer behavior |
Model failure under every useful metric |
| Prompt variants reverse the ranking |
Prompt–checkpoint interaction |
Stable parameter-size scaling |
R001 prefill recovers the result |
Output initiation or start selection |
Perfect later rollout |
| Repeated greedy runs reproduce the same failure |
Stable behavior under that setup |
Generality across prompts and runtimes |
| 3B remains worse across prompts, runtimes, and score decompositions |
Stronger task-specific checkpoint evidence |
Universal inverse scaling |
| Base and Instruct results differ |
Template/alignment behavior may contribute |
That architecture is irrelevant |
| Base and Instruct retain the same robust ranking |
Checkpoint-family or pretraining factors become more plausible |
A known single root cause |
Related references and the limits of the analogy
These are not exact matches for your benchmark, but they cover nearby components.
Retrieval beyond one item
- RULER evaluates configurable long-context tasks beyond simple in-context recall.
- Sequential-NIAH focuses on multiple sequential items and their order.
- Lost in the Middle shows that information use can depend on input position.
Retrieval versus long reconstruction
- LongProc combines dispersed information with deterministic long structured output.
- LongGenBench evaluates long-context generation rather than only short retrieval answers.
These support the narrower point that locating information and producing a long ordered reconstruction are not the same measurement.
Verbatim fidelity
These are closer to exact copying, but they do not identify the cause of your Qwen result.
Prompt and generation behavior
Evaluation design
The common lesson is not one proposed root cause. It is that access, sequential extraction, verbatim fidelity, output initiation, long rollout, stopping, and scoring are easier to reason about when recorded separately.
How narrowly I would state the scaling result
A cautious statement would be:
Under this benchmark and these settings, these three Qwen2.5 checkpoints produced a non-monotonic reconstruction result.
I would not yet turn it into:
- larger models have less reconstruction memory;
- Qwen2.5-3B has a general memory defect;
- inverse scaling has been demonstrated;
- one architectural component is responsible.
The checkpoints are related, but they are not a one-variable architecture experiment in which only parameter count changes.
Their official configurations differ:
They differ in hidden size, layer count, and attention-head structure, and their instruction-tuned behavior can interact differently with the benchmark prompt.
A robust non-monotonic result would still be useful. It would simply be safest to report it first as a property of these checkpoints on this task family and setup.
A minimal follow-up grid
A relatively small follow-up could use:
- one fixed input instance;
- one official revision of each Instruct checkpoint;
- batch size 1;
- explicit greedy generation with no repetition penalty;
- one sufficiently large common output budget;
- retained raw token IDs and raw decoded text;
- the current prompt plus two meaning-preserving variants;
- requests for 1 item, 2 items, and the full set;
- strict exact match plus item/start/order/stop diagnostics.
That grid would help separate:
individual information access
vs.
wrong first-item selection
vs.
ordering failure
vs.
long-output deterioration
vs.
format/scorer mismatch
vs.
a stable checkpoint-specific difference
My current best guess is not necessarily that the 3B model “forgets more.” A plausible alternative is that reconstruction combines information access with a prompt-conditioned decision about where and how to begin a long exact continuation.
My synthetic examples support that possibility, but a complete raw example and scorer from your benchmark would be needed to determine whether the same explanation applies.