Distinguish between thinking and responding during generation

[stulcrad]

Hi, I am building a custom logits processor that forces the generation of models to exactly follow the input text, plus classify parts of the text using special inline tags for precise text classification using generative models.
I successfully implemented the processor for models that generate the output directly without any chain of thought, deep reasoning, or thinking.
Now I would like to follow up on this by doing a two-phase generation, where I let the model reason however it wants, and then start the verbatim generation of the input text plus the classified spans. However, I am now thinking about how to divide this generation, first I thought about checking for the end-of-thinking token during generation, but it seems naive, as Ollama api or vLLM divide these, and you can simply pick the info you need from the data structure.
I need the possibility to access the logits during generation, which Ollama does not allow, and vLLM added this after I created my processor to fit HF, so I was wondering if there is a way to divide the two phases using only the HF library.

[John6666]

so I was wondering if there is a way to divide the two phases using only the HF library.

Yeah. That approach seems feasible:

---

Yes — I think this should be possible with Hugging Face Transformers alone, without giving up access to the logits.

The most direct route is probably a phase-aware wrapper around your existing LogitsProcessor:

Generation phase	Wrapper behavior
Reasoning	Return the scores unchanged
Optional transition after reasoning	Allow or consume the model-specific answer prefix
Final response	Delegate to your existing verbatim-copy/tagging processor

So using an end-of-thinking marker is not inherently naive. Reasoning-aware inference engines also need some model-specific way to determine when reasoning has ended. The more fragile version would be searching the entire decoded conversation for a hard-coded string. The safer version is to inspect the newly generated token IDs for the current assistant turn, using the actual checkpoint and chat template.

The main implementation issue may be slightly different from the visible question: after detecting the boundary, your existing processor must begin at the start of the final response, rather than treating all preceding reasoning tokens as part of the constrained output.

In other words:

prompt
├── unconstrained reasoning
├── reasoning-end delimiter
├── optional newline / answer prefix / control token
└── constrained verbatim copy with inline tags

The last region is what the existing processor should normally interpret as its generated prefix.

Recommended default route

I would first test this in the narrowest possible configuration:

batch size = 1
num_beams = 1
do_sample = False
torch.compile = off
assisted/speculative decoding = off

Then:

1. Generate once without the custom processor and inspect the raw token IDs around the reasoning/final boundary.
2. Add a wrapper that detects the boundary but does not modify the scores.
3. After the boundary, temporarily force a short known sequence such as OK.
4. Once that switch works, connect the existing processor.
5. Verify that its copy position and tag state begin from the start of the final response.
6. Add sampling, batching, beams, compilation, or serving one at a time.

A wrapper would have roughly this shape:

class PhaseAwareProcessor(LogitsProcessor):
    def __init__(
        self,
        inner_processor,
        prompt_length,
        reasoning_end_ids,
    ):
        self.inner_processor = inner_processor
        self.prompt_length = prompt_length
        self.reasoning_end_ids = reasoning_end_ids

    def __call__(self, input_ids, scores):
        generated_ids = input_ids[:, self.prompt_length:]

        final_start = find_final_start(
            generated_ids,
            self.reasoning_end_ids,
        )

        if final_start is None:
            # Reasoning phase: no constraint.
            return scores

        final_prefix_ids = generated_ids[:, final_start:]

        # This adapter depends on what the existing processor expects.
        inner_input_ids = build_inner_history(
            prompt_ids=input_ids[:, :self.prompt_length],
            final_prefix_ids=final_prefix_ids,
        )

        return self.inner_processor(inner_input_ids, scores)

That is only the architecture, not drop-in code. In particular, build_inner_history() depends on whether your processor reconstructs its state from the token history, derives its copy position from sequence length, or stores mutable state internally.

Which route applies?

• There is a stable reasoning-end token or token sequence: use the wrapper route above.
• The reasoning end is ambiguous, but the final answer has a stable start token: switch on the answer-start token instead.
• There are fixed newlines or control tokens between reasoning and the answer: add a short TRANSITION phase.
• There is no stable model-native boundary: consider an explicit final-answer delimiter or two-pass generation.
• The existing processor cannot be reset or rebased: refactor its state contract, or run the constrained answer as a second generation pass.
• You need rollback, custom cache handling, or several decoding-policy changes: a custom generation loop may be a better boundary than LogitsProcessor.

The exact branch depends mainly on:

• the model/checkpoint and tokenizer revisions;
• the rendered chat template;
• the raw generated token IDs around the boundary;
• the Transformers version;
• and how the existing processor calculates its current copy/tag state.

Why Ollama, vLLM, and similar APIs can return separate reasoning and content fields

The model commonly produces one autoregressive token sequence containing both regions. A parser or serving layer then presents that sequence as separate fields such as:

{
  "reasoning_content": "...",
  "content": "..."
}

Hugging Face describes this separation in its response parsing documentation, and Transformers Serve can expose reasoning separately from final content through its reasoning support.

These are related but distinct layers:

Layer	Responsibility
Chat template	Produces the model-specific prompt and control-token format
Model generation	Produces the token stream
Reasoning/response parser	Divides generated output into reasoning, content, tool calls, and so on
LogitsProcessor	Changes which tokens can be selected during generation

A parser can identify or display the regions after or during decoding, but it does not automatically turn your custom constraint on at the appropriate generation step.

This is why a token-level phase detector inside a wrapper is a reasonable approach even when Transformers does not expose the exact same response object as another inference server.

How to inspect the actual model protocol

The boundary is checkpoint- and template-specific. I would inspect the rendered prompt and the newly generated token IDs before choosing a delimiter.

For example:

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device)

prompt_length = inputs["input_ids"].shape[-1]

output = model.generate(
    **inputs,
    do_sample=False,
    num_beams=1,
    max_new_tokens=512,
)

generated_ids = output[0, prompt_length:]

print(generated_ids.tolist())
print(tokenizer.decode(generated_ids, skip_special_tokens=False))

Then compare:

print(tokenizer.convert_tokens_to_ids("</think>"))
print(tokenizer.encode("</think>", add_special_tokens=False))

Possible results include:

• one dedicated special token;
• one ordinary token;
• a multi-token sequence;
• a different delimiter;
• both reasoning-start and reasoning-end markers;
• a stable answer-start marker;
• a delimiter that only appears under certain chat-template settings;
• no explicit delimiter in some outputs.

The relevant background is in the Transformers chat-template guide. add_generation_prompt, continue_final_message, and model-specific template arguments can change which tokens already exist before generation begins.

Only inspect the newly generated assistant portion:

generated_ids = input_ids[:, prompt_length:]

Searching the entire conversation could incorrectly match:

• a literal </think> inside the user input;
• an earlier assistant reasoning trace;
• an opening marker prefilled by the chat template;
• an example of the format included in the prompt.

A related Transformers proposal, MaxThinkingTokensLogitsProcessor, discusses several of these edge cases. It has a different purpose—forcing a reasoning span to end after a token budget—and should not be treated as a ready-made solution here, but its discussion is useful for boundary-testing cases.

Why the existing processor may need to be rebased

Suppose the current processor assumes that constrained generation starts immediately after the prompt.

It might calculate its position as:

copy position
= total generated length
- inserted tag-token count

Once unconstrained reasoning is inserted before the tagged copy, that calculation no longer represents the copy position.

There are several possible processor designs.

1. History-derived processor

The processor reconstructs its state from the constrained-output prefix on every call.

This is the easiest case. The wrapper can provide a virtual history containing:

original prompt + final-response prefix

while excluding the reasoning prefix from the inner parser’s view.

2. Length-derived processor

The processor assumes:

generated token count == constrained-output token count

It needs either:

• a final_start_position;
• an offset subtracted from the generated length;
• or a separate final-prefix argument.

3. Mutable stateful processor

The processor stores values such as:

self.copy_position
self.open_tag
self.current_label

It needs an explicit reset when the final phase begins.

Mutable state also becomes more complicated with batching and beam search, because physical rows may not correspond to one permanent logical sequence.

4. Prefix parser that expects valid constrained syntax from token 1

If the processor parses every generated token as tagged output, passing <think>... into it may put its automaton into a dead or invalid state before the final response begins.

In that case, it should only receive the final-phase prefix.

5. Input-dependent constraint processor

Your valid output language depends on the exact source text, rather than only on a fixed JSON schema or grammar.

Conceptually, the cleanest inner contract may be:

inner_processor(
    source_ids=source_ids,
    final_prefix_ids=final_prefix_ids,
    scores=scores,
)

The outer wrapper can adapt that contract to the standard LogitsProcessor(input_ids, scores) interface.

The important point is not that one of these designs is universally correct. It is that the phase boundary and the constraint-state boundary should be explicit rather than inferred from the total sequence length.

A small verification sequence

Separating the tests makes it easier to tell whether a failure belongs to the model protocol, the phase detector, or the existing constraint automaton.

Test 1 — Observe only

Run without the custom processor and record:

• rendered prompt tokens;
• prompt length;
• generated token IDs;
• decoded output with special tokens preserved;
• reasoning-start and reasoning-end positions;
• tokens between the reasoning end and visible final answer.

Test 2 — Detect only

Add the wrapper, but always return the scores unchanged.

Record:

generation step
current phase
last few token IDs
detected reasoning-end position
detected final-start position

This tests the detector independently of the constraint.

Test 3 — Dummy switch

After the detected final boundary, force a short known continuation such as:

OK

This tests:

• whether the switch occurs;
• whether it occurs at the intended token;
• whether reasoning before the switch remains unconstrained;
• whether the processor actually changes the next-token scores.

Test 4 — Reconnect the existing processor

At the first constrained step, verify:

• the copy position is zero;
• no tag is already open;
• the first allowed copied token matches the beginning of the source;
• the reasoning prefix is absent from the inner syntax state;
• at least one next token is allowed.

Test 5 — Boundary controls

Useful cases include:

Case	What it checks
User input contains the literal text </think>	Detection is limited to the current generated span
Previous assistant turn contains reasoning	Old boundaries are ignored
Closing marker is multiple tokens	Suffix matching works
Model never emits the expected marker	Missing-marker policy works
Marker appears twice	State machine does not restart incorrectly
Marker is followed by newlines	Transition handling works
EOS appears before final output	Failure is reported clearly
Final copy does not fit in max_new_tokens	Token-budget policy works

Test 6 — Add generation modes incrementally

A useful order is:

1. greedy decoding, batch size 1;
2. sampling, batch size 1;
3. ordinary batching;
4. beam search;
5. compilation;
6. assisted/speculative decoding;
7. serving or continuous batching.

Transformers can return both model logits and processed generation scores through the options documented in generation utilities. That can help verify whether the wrapper changed the intended token candidates.

Transition tokens and final-answer start

The reasoning-end position and the constrained-answer start do not always have to be identical.

A model might naturally produce:

</think>\n\n

or:

</think><answer>

or another control token before the visible final response.

If the wrapper enables the existing processor immediately after </think>, but the processor only permits the first copied source token, it may suppress a prefix that the model was trained to emit.

A slightly richer state machine can make this explicit:

REASONING
    ↓ reasoning-end marker
TRANSITION
    ↓ expected answer prefix consumed
FINAL
    ↓ constrained output complete
DONE

The TRANSITION state could:

• allow a known newline sequence;
• force a fixed answer prefix;
• wait for a stable response-start token;
• or be omitted entirely when the answer begins immediately.

This should be determined from raw generations for the actual checkpoint rather than assumed globally.

Tokenization issues specific to verbatim copying and inline tags

A character or syntax boundary does not necessarily align with a tokenizer boundary.

Potential boundaries include:

reasoning delimiter | final response
plain copied text   | opening tag
closing tag         | copied text
whitespace          | punctuation

Subword tokenizers can sometimes represent the same text through more than one token sequence, and one token can cover characters on both sides of a conceptual boundary.

This is a general constrained-decoding issue discussed in work such as automata-based constraints for language-model decoding.

Since your direct-generation processor already works, it may already handle the important source/tag boundaries. The new boundary introduced by this design is the transition from reasoning into the constrained output.

Cases worth checking include:

• leading whitespace in the source;
• a newline between reasoning and the source copy;
• punctuation at the first copied position;
• Unicode characters;
• tags adjacent to text without whitespace;
• tokenizer-added special tokens;
• a source prefix that tokenizes differently depending on the preceding delimiter.

If the processor enforces one pre-tokenized source path, that may be exactly what you want for strict token-level copying. If the requirement is character-level identity, it may need to accept every token path that decodes to the permitted character continuation.

Batching, beams, streaming, and serving

Ordinary batching

Different sequences can finish reasoning at different steps.

A single flag such as:

self.reasoning_finished = True

would cause the first completed row to activate constraints for every row.

For an initial correct implementation, determining each row’s phase from its own generated suffix is often simpler than maintaining mutable row state.

Beam search

Beam search adds several issues:

• beam rows can be duplicated;
• beam rows can be reordered;
• one logical sequence can have several candidate continuations;
• scores supplied to a processor may be log-softmax scores rather than ordinary raw logits.

A hard allowed/disallowed mask usually transfers more easily than a processor that interprets absolute score values.

Unless beam search is already required, it seems reasonable to treat it as a later compatibility step.

Streaming

A textual delimiter may be buffered or split by a streamer or detokenizer. That does not necessarily reflect how it appears in input_ids.

Inside a LogitsProcessor, token IDs are the more direct generation signal.

Continuous batching and serving

In continuous batching, requests may be inserted, reordered, or removed over time. A physical batch row is not necessarily a persistent request identity.

Current Transformers has separate machinery for processor compatibility in continuous-batching paths. A custom processor that works in ordinary generate() should therefore be validated separately before being moved into a serving loop.

Transformers v5 considerations

The basic idea is not tied to an obsolete v4-only API. Current Transformers still supports custom logits processors through the standard generation interface.

The v5 changes most likely to matter are around the surrounding contract:

• apply_chat_template() returns a BatchEncoding;
• prompt length should therefore be taken from inputs["input_ids"];
• tokenizer special-token storage changed;
• generation configuration belongs in model.generation_config;
• cache defaults can be model-specific;
• advanced generation paths continue to change between patch versions.

The Transformers v5 migration guide documents the larger API changes.

For a reproducible result, it would be useful to keep these fixed:

transformers version
torch version
model revision
tokenizer revision
dtype
generation arguments

The simple single-pass wrapper is less exposed to cache changes than a two-pass design that tries to reuse past_key_values.

I would first validate without:

torch.compile
continuous batching
assisted/speculative decoding
custom serving loops

and then add those paths only if needed.

Related implementations

The exact constraint here—verbatim source copying with inline tags—is specialized, but several projects implement a closely related phase split.

vLLM

The feature request Only apply guided/structured grammar after reasoning asks for unrestricted reasoning followed by constrained output after </think> or inside an answer region.

The corresponding implementation work is in vLLM PR #12955.

This is not drop-in Transformers code, but it is a strong precedent for switching a structured-decoding engine at a reasoning boundary.

A related vLLM bug report, generation stopping after </think>, is also a useful reminder that boundary detection and constraint activation should be tested independently.

SGLang

SGLang received essentially the same request in issue #4055: disable structured-output constraints during reasoning, then apply them to the final answer.

Its linked implementation work is PR #4984.

Outlines

Outlines PR #1711 explores support for reasoning models by delaying the structured-output processor until an end-thinking token is generated.

It is useful as a design reference rather than a universal finished solution. Its discussion also exposes complications around batches and delimiter assumptions.

Litelines

Litelines contains a Transformers example using allow_preamble=True: free-form text is generated first, followed by schema-constrained JSON.

That is not exactly the same as detecting a model-native reasoning span, but it is a concrete example of delayed constraint activation inside model.generate().

Thinking Before Constraining

Thinking Before Constraining studies a closely related strategy: generation remains free until trigger tokens occur, after which structured decoding is enabled.

Its InWriting implementation may also be useful as a code-level reference.

This is recent work and does not prove that a specific custom processor will work unchanged, but it supports the general phase-switch formulation.

CRANE

CRANE separates reasoning from the constrained final region through an explicit delimiter-aware grammar.

That points to a useful fallback: if the model-native reasoning protocol is unstable, an explicit answer delimiter can be made part of the generation contract.

Alternative routes

1. Explicit final-answer delimiter

Use this when:

• the model does not emit a reliable native reasoning-end token;
• but it reliably follows an explicit final-answer marker.

For example:

Do any private working first. When ready to produce the required tagged copy,
emit <FINAL> and then output only the constrained result.

The actual marker should be selected with the tokenizer and prompt format in mind. It should also be detected only in the newly generated assistant span.

2. Detect final-answer start instead of reasoning end

Sometimes the end of reasoning is ambiguous while the beginning of the answer is stable.

In that case:

wait for answer-start token
→ reset/rebase inner processor
→ enable constraint

may be more reliable than trying to classify every possible reasoning termination.

3. Two-pass generation

Pass 1:

generate an unconstrained analysis, plan, or classification draft

Pass 2:

provide the source and the draft
generate under the existing verbatim/tagging constraint

Advantages:

• the second pass has a clean constrained-output boundary;
• the existing processor may need little modification;
• no in-call phase state is needed.

Trade-offs:

• additional inference cost;
• a longer second-pass context;
• possible dependence on errors in the draft;
• the result is not identical to one continuous autoregressive sequence.

A recent related approach is Draft-Conditioned Constrained Decoding.

4. Custom generation loop

Transformers supports custom generation methods.

This is the appropriate escalation point if the design needs:

• rollback;
• direct cache manipulation;
• different sampling policies in each phase;
• more than one model call inside the decoding loop;
• nonstandard stopping behavior;
• or state that cannot be expressed cleanly through a processor.

It is more flexible, but it also has a larger maintenance and compatibility surface than a wrapper.

Failure policies and evaluation

It is useful to define failure behavior explicitly instead of allowing an invalid state to silently produce arbitrary output.

Condition	Possible policy
Reasoning-end marker never appears	Stop after a reasoning budget, return a structured failure, or fall back to two-pass generation
EOS occurs before the final phase	Report that no constrained answer was produced
Marker appears more than once	Accept only the first valid boundary in the active reasoning state
No token is allowed by the inner processor	Log the state and terminate or invoke a defined fallback
Final copy cannot fit in the remaining token budget	Reserve a minimum final budget or reject the generation early
Source copying completes with an open tag	Permit only a legal closure, or report invalid processor state
User text contains delimiter-like content	Restrict matching to the active generated reasoning span

For this task, I would keep these measurements separate:

• exact source-copy rate;
• tag syntax validity;
• span-boundary accuracy;
• classification accuracy;
• reasoning-boundary detection rate;
• empty-allowed-token rate;
• incomplete-output rate;
• reasoning token count;
• final-output token count;
• latency.

A valid tagged copy does not necessarily imply correct classifications, and allowing free reasoning does not necessarily improve them.

A useful comparison set would be:

1. the current direct constrained generation;
2. free reasoning followed by the same processor in one call;
3. free reasoning followed by an unconstrained final answer;
4. optionally, a two-pass draft followed by constrained generation.

That separates the effect of reasoning from the effect of hard format enforcement.

So my default recommendation would be:

Use one generate() call with a phase-aware wrapper, provided that the checkpoint has a stable token-level phase boundary and the existing processor can be initialized at the actual start of the final response.

If the first condition fails, use an explicit answer marker or two-pass generation. If the second fails, refactor the processor to accept a final-start offset or final-phase prefix before adding more generation complexity.