For now, I can’t say anything definite without seeing the relevant parts of the code, but:
Direct answers first
| Question |
My current answer |
| 1. Can Ollama reuse the KV cache between Retell turns? |
Ollama does have internal prompt caching, but that is not the same thing as an application-controlled, persistent conversation KV handle. I do not see a supported session/KV identifier in the current /api/chat contract that you can safely carry between arbitrary WebSocket turns. Whether reuse occurs depends on the exact token prefix, Ollama version, runner/slot assignment, context shifting, and concurrent requests. Measure it rather than assuming either “full reuse” or “full re-prefill.” |
| 2. Is there a 7B–8B model that will reliably stay below one second? |
I would not promise that from the model name and GPU alone. Your own early turns already approach that range, which suggests the more valuable question is why latency grows across the call. Queueing, superseded requests, cache eligibility, reconnect-time duplicate work, and CPU/GPU placement can all produce that slope. Benchmark the exact production prompt and measure P95/P99 time to the first speakable sentence, not only an isolated first token. |
| 3. Is there a better pattern than inline ticket markers? |
Yes. Separate spoken generation from action proposal and action commit. Let the model propose typed data, validate it against the transcript and application state, then perform an idempotent durable operation. Native tool calling or JSON Schema can help with syntax, but neither guarantees that the action is semantically correct or executed exactly once. |
| 4. Is Retell’s timeout configurable? |
The reconnect condition I can find in the current public protocol is not a documented three-second first-token timeout. With auto_reconnect=true, Retell documents a ping_pong exchange every two seconds, closes the current connection after five seconds without one, and reconnects up to two times. The error Max ping pong reconnection count reached therefore points first toward that heartbeat path. This does not prove that no other internal timeout exists, but I would separate the two mechanisms before treating three seconds as a hard model deadline. |
Relevant primary references:
The default route I would take
Before changing models or rebuilding the architecture, I would take one failed call and do the following:
-
Retrieve Retell’s view of the call.
Save:
disconnection_reasonpublic_log_urllatency.llmlatency.llm_websocket_network_rttlatency.e2e
Retell defines its llm latency as the time until the first speakable chunk, and for a custom LLM this includes the WebSocket round trip. That is not necessarily the same number as the local Ollama first-token timestamp.
-
Build one monotonic timeline across all layers.
Key every event by:
call_id- a locally assigned
socket_generation
- Retell
response_id
- the local Ollama request ID
- the logical action ID, if any
-
Run three small controls.
- Slow LLM, healthy heartbeat
- Fast dummy response, blocked heartbeat
- Superseded-response cancellation probe
-
Make heartbeat handling independent of inference.
The WebSocket receive loop must remain able to receive and answer ping_pong while RAG, prompt construction, Ollama prefill, streaming, or action execution are in progress.
-
Make “latest response wins” apply to the whole application.
A new Retell response_id should supersede not only outbound text, but also any obsolete RAG, prompt-building, LLM, TTS-preparation, or action-proposal work associated with the previous response.
-
Only then analyze cache, context length, and model choice.
Use Ollama’s own timing fields to distinguish prompt evaluation from queueing and application delay.
-
Move ticket creation out of the speech stream.
Treat the LLM output as an action proposal, not as the transaction itself.
The first code paths I would inspect
| Area |
What I would look for |
| WebSocket receive loop |
Does it continue receiving while the LLM is generating? |
ping_pong branch |
Can it respond immediately, or is it behind RAG/Ollama work in the same coroutine? |
| Ollama client |
Sync client inside async def, blocking iterator, or genuinely asynchronous streaming? |
| Response ownership |
What happens when a higher response_id arrives? |
| Cancellation |
Does canceling the local task stop server-side Ollama compute, or only stop reading chunks? |
| Disconnect cleanup |
Are every socket-owned task and outbound stream terminated? |
call_details preload |
Is it once per call or once per connection? Can two connections share one in-progress preload future? |
| WebSocket writes |
Are multiple tasks writing concurrently, or is there one serialized outbound writer? |
| Ticket execution |
Can retry, reconnect, interruption, or duplicate generation perform the same mutation twice? |
Why I would investigate the heartbeat path before assuming a three-second token deadline
The documented Retell behavior is fairly specific:
- with
auto_reconnect=true, Retell sends ping_pong events every two seconds;
- your server is expected to send corresponding events back;
- after five seconds without
ping_pong, Retell closes the current connection;
- it opens a new connection up to two times.
See the current LLM WebSocket specification.
That matches the wording of Max ping pong reconnection count reached much more directly than a first-token deadline does.
A plausible failure sequence is therefore:
larger prompt or queued request
-> longer period before the Ollama stream yields
-> WebSocket receive loop cannot process ping_pong
-> no pong for five seconds
-> Retell reconnects
-> duplicate initialization or stale inference remains active
-> the next response is even slower
This only applies if some relevant work is blocking or starving the receive path. It is not possible to confirm without the handler code and a timeline.
A common Python trap is that an async def endpoint does not make a synchronous library call non-blocking. A synchronous Ollama iterator, database client, embedding client, or heavy preprocessing call can still monopolize the event-loop thread.
The websockets asyncio FAQ gives the general explanation: if a coroutine does not yield control, other connection work does not get scheduled. An async Ollama client or asyncio.to_thread() may be part of the fix, but the more important contract is architectural:
receive task:
always receive protocol events
immediately route ping_pong
response task:
perform RAG and LLM generation
outbound writer task:
serialize all writes to Retell
Using a single outbound writer is also useful because it prevents heartbeat, generated text, tool events, and cleanup messages from racing through concurrent WebSocket writes.
There may be other clocks in the path:
- Retell’s application-level
ping_pong
- WebSocket control-frame ping/pong
- Uvicorn timeouts
- reverse-proxy read or idle timeouts
- load-balancer connection limits
- application deadlines
The close code, Retell disconnection_reason, and public_log_url should identify which layer ended the connection. Retell’s disconnection guide distinguishes, for example, error_llm_websocket_lost_connection from runtime and payload errors.
I would also cheaply exclude the cases in Retell’s custom LLM troubleshooting guide:
- accidentally sending
end_call=true;
- hosting on a platform that does not support persistent WebSockets;
- an infrastructure timeout imposed by the hosting provider.
Three controls that should separate the main hypotheses
Control A: deliberately slow inference, responsive heartbeat
For a test call, replace the model response with an intentional delay longer than the suspected deadline:
async def delayed_response():
await asyncio.sleep(6)
return "This is a delayed test response."
During those six seconds, continue receiving and immediately echoing Retell ping_pong events.
Interpretation:
- Connection survives: the “approximately three-second first-token timeout” hypothesis becomes much weaker; heartbeat starvation, production queueing, or another path becomes more likely.
- Connection still reconnects: capture the Retell public log and exact close reason. There may be another deadline, an incorrect pong implementation, or a different infrastructure timeout.
Control B: fast response, deliberately blocked heartbeat
Return a fixed response immediately, but in an isolated test intentionally block the WebSocket event loop for more than five seconds.
Interpretation:
- The same reconnect error appears: this provides a clean reproduction of the heartbeat path.
- A different close occurs: compare the close reason and protocol log.
Do not run the blocking variant against production calls.
Control C: superseded-response cancellation
- Start a deliberately long generation A.
- Send or reproduce a newer Retell
response_id.
- Cancel A locally and close its response body/stream.
- Immediately start a very short generation B.
- Observe:
- when B actually begins;
- GPU utilization;
- Ollama queueing;
- runner/debug logs;
- whether A continues consuming compute.
This matters because stopping iteration on the client is not necessarily proof that server-side compute has stopped.
There is a recent open Ollama report, streaming handlers may retain goroutines and runner semaphore slots after client disconnect. It was reported against current development code, so it should not be treated as proof of your production cause. It is, however, a good reason to test cancellation explicitly on your exact version rather than assuming it works.
An older ollama-python issue also noted that simply breaking out of a streaming loop was not sufficient for aborting generation: ollama-python issue #210. That issue is closed and may not represent current behavior, but the underlying verification question remains useful.
A timeline that should make the failure visible
For one failed call, I would record monotonic timestamps for events like these:
| Field |
Example meaning |
call_id |
Retell call identity |
socket_generation |
First connection, first reconnect, second reconnect |
response_id |
Retell response currently requested |
event |
ping_received, pong_sent, rag_start, ollama_first_chunk, etc. |
t_monotonic_ns |
Local monotonic clock |
task_id |
Local task or coroutine identity |
prompt_tokens |
Rendered input token count |
ollama_request_id |
Local correlation ID |
superseded_by |
Newer response that invalidated this work |
cancel_requested_at |
Local cancellation time |
runner_stopped_at |
When compute actually stopped, if observable |
Useful events include:
socket_open
config_sent
call_details_received
preload_started
preload_finished
ping_received
pong_queued
pong_sent
response_required_received
response_task_started
rag_started
embedding_started
embedding_finished
rag_finished
prompt_rendered
ollama_http_started
ollama_response_headers
ollama_first_chunk
first_token
first_complete_sentence
response_chunk_sent
response_complete_sent
response_superseded
cancel_requested
ollama_stream_closed
ollama_runner_stopped
socket_close_started
socket_closed
action_proposed
action_validated
action_commit_started
action_committed
action_result_sent
Then compare four latency views:
Retell e2e latency
Retell llm latency
Retell llm_websocket_network_rtt
local response_required -> first speakable chunk
Ollama’s final streaming response can provide:
load_durationprompt_eval_countprompt_eval_durationeval_counteval_duration
See Ollama API usage metrics.
A useful residual is:
local time to first chunk
- model load time
- prompt evaluation time
It is not a perfect queue metric, but if that residual is several seconds, prompt prefill alone cannot explain the observed latency.
Also note that Retell says it starts streaming at the first sentence. Its custom LLM guidance explicitly treats latency as:
time to first token
+ time to produce a speakable sentence
See Retell’s current integration guide. This means two models with similar raw TTFT may still have different voice-agent latency because one produces a complete first sentence more quickly.
Call state, socket state, and response state should probably be separate
I would avoid using “the WebSocket” as the owner of all call state.
A safer conceptual model is:
Call state, keyed by call_id
durable memory
authentication state
retrieved case data
completed business actions
once-per-call preload future
Socket generation
connection object
heartbeat task
outbound writer
close reason
connection-local cleanup
Response state, keyed by response_id
prompt
retrieval job
Ollama request
partial spoken output
superseded/canceled status
A reconnect should replace the socket generation, not create a new logical call.
A new response_id should replace the current response work, not reset durable call state.
Retell’s current protocol says that when a new response is needed, it sends a new auto-incrementing response_id and discards previous responses. That is a protocol-level “latest response wins” rule. I would mirror it inside the application:
if incoming_response_id > active_response_id:
mark previous response superseded
cancel previous RAG/prompt/LLM work
reject any late chunks from the previous response
prevent previous action proposals from committing
Late-result rejection is needed even when cancellation is supported, because cancellation is cooperative and a job may finish during the race.
About the two WebSocket connections
I could not find a current Retell protocol statement describing a permanent primary/backup pair. The current documentation I found describes closing the current connection and opening a replacement after heartbeat failure.
That does not mean your observed two connections are not expected behavior. I would simply avoid assigning them “primary” and “backup” semantics until the public log confirms the timeline.
Log:
call_id
socket_generation
socket_open_time
socket_close_time
call_details_time
first_response_required_time
preload_start/end
If both sockets overlap, the important question is not their name but whether work owned by generation 1 remains active after generation 2 starts.
Preload should probably be single-flight per call
Instead of only checking whether _pre_contexto already exists, consider a once-per-call shared future:
async def get_call_preload(call_state):
if call_state.preload_task is None:
call_state.preload_task = asyncio.create_task(load_call_context())
return await call_state.preload_task
That closes the window where two sockets both observe “not loaded yet” and start duplicate work.
If preload fails, store an explicit failed state or controlled retry policy rather than allowing every reconnect to launch another attempt.
What Ollama cache reuse probably does and does not mean here
The statement “Ollama processes the full KV cache on every call” is too broad for current Ollama.
Ollama has internal prompt caching, and recent release notes explicitly mention improved prompt caching and better KV reuse by decoupling it from context shift: Ollama releases.
At the same time, internal prompt caching is not equivalent to a public, portable, caller-controlled KV session.
The current /api/chat interface is message-based. I do not see a supported field that says:
continue from this exact persistent KV state
There was an older context mechanism around /api/generate, but its deprecation and lack of a direct replacement have been discussed in Ollama issue #10576. That issue is not documentation for current cache internals, but it supports the distinction between:
- replaying messages and benefiting from internal prefix reuse;
- owning an explicit inference session state.
keep_alive=-1 is not the same thing
keep_alive controls how long the model remains loaded. It avoids a model reload, but it does not promise that a particular conversation’s KV state remains assigned to the next request. See the Ollama FAQ.
Test the actual cache behavior
Use the same model and fixed runtime conditions, then compare:
-
Exact repeat
- identical messages
- identical options
- identical template
-
Append-only conversation
- exact previous messages
- one user message appended
-
Early-prefix mutation
- change one token near the beginning of the system prompt
-
RAG appended late
- preserve the fixed system/history prefix
- append dynamic retrieval near the end
-
RAG inserted early
- insert dynamic retrieval directly after the system prompt
For every condition, record:
- fully rendered prompt;
- token IDs if available;
prompt_eval_count;prompt_eval_duration;- debug-log cache information;
- runner/slot identity if observable.
Prefix caches generally depend on exact token identity, not semantic equivalence. A useful conceptual reference is vLLM’s prefix-caching design, although vLLM is a different runtime and its implementation should not be projected onto Ollama.
Things that may invalidate an otherwise reusable prefix include:
- timestamps;
- request identifiers;
- a rewritten summary near the beginning;
- reordered tool schemas;
- dynamic RAG inserted before the stable history;
- different chat templates or special tokens;
- minor normalization differences.
Context shifting and truncation
I would log the fully rendered prompt and actual token count instead of relying only on the estimated budget.
When the prompt approaches the configured context limit, verify the exact behavior of your current Ollama version and runner:
- which messages or tokens are removed;
- whether context shifting occurs;
- whether the stable prefix is retained;
- whether cache reuse changes after the shift.
Recent Ollama changes around prompt caching and context shift are another reason to record the exact Ollama version in every benchmark.
Queueing and resource placement may explain more than model speed
The default maximum parallel request count per model is currently documented as one:
OLLAMA_NUM_PARALLEL=1
Ollama queues work when a requested model cannot be loaded or processed immediately. It also documents that increasing parallelism multiplies context-memory requirements:
required memory scales with
OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH
See Ollama’s concurrency documentation.
That creates several possible failure modes:
- a superseded generation still owns the single runner slot;
- preload or warmup overlaps the first real response;
- embedding and generation contend for the GPU;
- the 8B, 14B, and embedding models do not all remain resident under real pressure;
- the model partially falls back to CPU;
- increasing parallelism reduces queueing but creates enough KV pressure to worsen placement or cache locality.
Around each slow turn, save:
/api/ps output;ollama ps;- model
size_vram;
- actual context allocation;
- CPU/GPU split;
- number of active and queued local requests;
- GPU utilization and memory;
- which model is loading or unloading.
A simple matrix for the RAG contention question
| Condition |
Purpose |
| No RAG |
Baseline generation |
| Cached RAG result |
Removes embedding and vector-query work |
| CPU embedding |
Separates GPU contention |
| GPU embedding |
Current production path |
| Generation single-flight |
Removes overlap with stale/preload requests |
| Preload disabled |
Tests reconnect/startup interaction |
The reported average 40 ms embedding overhead is probably not enough by itself to explain a four-second tail. The 300–700 ms spikes are more relevant, but they may still be a symptom of broader GPU queueing rather than an isolated embedding problem.
I would also gate retrieval by turn type. Greetings, acknowledgements, spelling confirmations, yes/no confirmations, and replies based entirely on already collected structured state often do not need fresh retrieval.
Model choice and speculative decoding
I would not interpret the current data as evidence that an 8B model categorically cannot support the application.
The same model reportedly reaches approximately 0.8–1.5 seconds during early turns. That makes these explanations at least as interesting as model size:
- growing prompt prefill;
- changing cache eligibility;
- old requests accumulating;
- response supersession;
- reconnect-time duplicate work;
- model/embedding contention;
- partial CPU placement;
- first-token versus first-sentence differences.
A useful benchmark should reproduce the production path:
same model build and quantization
same chat template
same system prompt
same RAG placement
same tool schemas
same context lengths
same concurrency
same Ollama options
same first-sentence stopping criterion
Measure at least:
- P50/P95/P99 raw first token;
- P50/P95/P99 first complete sentence;
- prompt tokens;
- prompt-evaluation duration;
- generated tokens per second;
- queue delay;
- cache condition.
A model can have a fast median and still be unusable if the P99 crosses the heartbeat or conversation deadline.
Speculative decoding
The current statement “Ollama does not support this natively” is no longer universally true.
The current Ollama Modelfile reference documents draft_num_predict when a separate draft model or embedded MTP tensors are available.
However:
- availability is model/backend dependent;
- it may not apply to the exact Llama 3.1 model being used;
- speculative decoding primarily improves autoregressive decoding;
- it does not remove target-model prompt prefill;
- it may improve time to a complete first sentence more than raw prefill latency.
So I would investigate it only after determining whether the slow component is:
queue
model load
prompt prefill
or output decoding
Hosted fallback
A hosted model may be a valid product decision, especially if a strict call-completion SLO matters more than complete locality.
It does not need to be an all-or-nothing migration. Possible boundaries include:
- local primary with a hosted circuit-breaker for overloaded calls;
- local generation but hosted structured extraction;
- local handling for privacy-sensitive flows and hosted handling for low-risk intents;
- human transfer when the local latency budget is exhausted.
Those choices depend on privacy, consent, contractual, and operational constraints rather than benchmark speed alone.
A safer action layer than inline markers
The current marker asks one generation to produce two incompatible streams:
natural language intended to be spoken immediately
machine-readable data intended to mutate external state
That creates predictable failure modes:
- the stream ends halfway through the marker;
- the marker is spoken aloud;
- it is generated twice;
- arguments are copied from RAG rather than the caller;
- a superseded response still produces a side effect;
- the ticket is created, but the confirmation speech is interrupted;
- a retry creates a duplicate ticket.
I would separate the lifecycle into:
1. Spoken response
2. Action proposal
3. Validation
4. Durable commit
5. Action result
6. Spoken confirmation
Option 1: two-pass extraction
The conversational model produces only speech.
After enough evidence exists, a second short request extracts:
{
"action": "create_ticket",
"ticket_type": "technical_support",
"summary": "The caller cannot access the student portal.",
"priority": "normal",
"source_utterance_ids": [12, 14]
}
The extractor should receive:
- the relevant caller utterances;
- already collected structured state;
- a narrow schema;
- no unrelated RAG text unless required.
The second pass can run asynchronously, but the application should not tell the caller that the ticket was successfully created until the commit succeeds.
Option 2: native tool calling
Ollama currently supports tool calling, and structured outputs can enforce a JSON Schema.
These improve the syntax boundary. They do not establish:
- that the ticket is warranted;
- that its summary is grounded in the caller’s words;
- that the caller is authorized;
- that it was not already created;
- that the operation should execute after interruption;
- that a retry is safe.
Treat it as a transaction
A minimal durable action record might contain:
logical_action_id
call_id
source_response_id
action_type
normalized_arguments
evidence_utterance_ids
status:
proposed
validated
in_progress
committed
failed
external_resource_id
attempt_count
created_at
updated_at
The idempotency key should be created by the application for the logical operation, not improvised from the model’s text.
A timeout during ticket creation means “outcome unknown,” not necessarily “failed.” Retrying with the same logical action ID should return or discover the already created ticket when possible.
Retell supports tool_call_invocation and tool_call_result events, but the protocol describes them primarily as bookkeeping and transcript-weaving events. They are useful for observability; I would not treat the Retell transcript as the authoritative transaction database.
A relevant cross-framework failure example is LiveKit issue #3702: a tool completed, interruption prevented its result from being preserved in conversation history, and the next turn executed the operation again. It is not evidence about Retell, but it demonstrates why business state cannot live only in the conversational transcript.
Spoken text and committed state are different
Track separately:
- text generated by the model;
- text actually sent to Retell;
- text actually spoken before interruption;
- action proposed;
- action committed;
- action result acknowledged by the caller.
If a caller interrupts after hearing only the first part of a response, the next prompt should not assume they heard the entire generated answer.
Conversely, an already committed ticket must remain committed even if its confirmation sentence was interrupted or omitted from the next LLM context.
A compact decision tree
Start with one terminated call
|
|-- Does Get Call report error_llm_websocket_lost_connection?
| |
| |-- Yes:
| | inspect public_log_url and ping/pong timeline
| |
| |-- No:
| follow the reported disconnection reason first
|
|-- Was ping received but pong delayed by several seconds?
| |
| |-- Yes:
| | isolate receive/heartbeat from inference
| |
| |-- No:
| check proxy/runtime close, corrupt payload,
| end_call=true, and undocumented-provider behavior
|
|-- Is Ollama prompt_eval_duration itself large?
| |
| |-- Yes:
| | inspect prompt length, exact-prefix reuse,
| | context shifting, placement, and model
| |
| |-- No:
| inspect queueing, stale requests, HTTP delay,
| RAG/preload, and application scheduling
|
|-- Does a superseded generation continue consuming GPU?
| |
| |-- Yes:
| | add real cancellation, late-result rejection,
| | and runner-health safeguards
| |
| |-- No:
| move to cache and resource-contention tests
|
|-- Does exact-repeat or append-only reduce prompt evaluation?
| |
| |-- Yes:
| | preserve a stable early prefix and move dynamic
| | material later where the prompt format permits
| |
| |-- No:
| record the exact version/model/backend and
| investigate cache eligibility or slot assignment
|
|-- Can a ticket be committed twice after retry/interruption?
| |
| |-- Yes or unknown:
| add a durable logical action ID and idempotent commit
What your 47-call result currently establishes
The sample is useful and the later-turn concentration is a real signal.
I would phrase the inference slightly more cautiously:
The failures correlate strongly with later turns, but later turn number is a proxy for several changing variables, not only context length.
By turn seven, all of these may have increased:
- prompt length;
- number of response supersessions;
- number of interruptions;
- probability of a context shift;
- amount of dynamic RAG variation;
- number of reconnects;
- number of stale jobs;
- chance of duplicate preload;
- GPU queue pressure.
A very informative comparison would be:
- a long prompt on turn one;
- a short prompt after many turns;
- a long synthetic conversation with no interruptions;
- a short conversation with repeated response supersession.
That separates “token count” from “accumulated lifecycle state.”
Practical order of implementation
If I had to prioritize the work, I would use this order:
- Correlate Retell Get Call/public logs with local monotonic logs.
- Make
ping_pong independent of generation.
- Implement response-scoped cancellation and late-result rejection.
- Make preload single-flight and once per
call_id.
- Measure Ollama load, queue, prompt evaluation, placement, and cache behavior.
- Gate RAG and test CPU versus GPU embedding.
- Move ticket creation to a durable idempotent action layer.
- Only then compare models, serving runtimes, speculative decoding, or hosted fallback.
This route preserves most of the existing stack and should identify whether the real limit is model prefill, application scheduling, or accumulated stale work before you make a larger migration.
