Hi. From what I can tell after looking into this, quite a few different problems seem to be getting tangled together, so I think stepping back and reconsidering the overall architecture—rather than continuing to patch individual edge cases—is probably the right move:
Short answer
I do not think the practical choice is “one huge prompt” versus “a rigid FSM.”
The publicly documented systems I could find point more toward a hybrid:
- An LLM interprets natural language and proposes typed actions.
- An external controller owns the authoritative dialogue/workflow state.
- Deterministic code validates transitions and business preconditions.
- Tools perform lookups or side effects under explicit lifecycle rules.
- The LLM verbalizes the resulting state naturally.
A compact way to state it is:
Let the model interpret and propose; let the runtime validate and commit.
There may still be generated prompts, node-specific prompts, summaries, retrieved instructions, or dynamically selected tools behind vendor dashboards. However, the public documentation for systems such as Pipecat Flows, Bland Conversational Pathways, Retell Conversation Flow, and Rasa CALM does not look like “a single secret super-prompt” is the whole mechanism.
The safest default architecture for your case would probably look roughly like this:
Audio / STT events
|
v
Turn finalization
|
v
LLM or SLM
proposes typed command(s)
|
v
Deterministic controller
- validates commands
- owns workflow state
- manages detours and return targets
- decides which tool operations are legal
|
v
Tool / operation layer
- lookup
- confirmation
- retry
- stale-result rejection
- idempotent commit
|
v
Response generation
|
v
TTS / spoken-output tracking
This does not require throwing away your existing Sarvam STT → streaming LLM → Sarvam TTS pipeline. It means moving responsibility for workflow correctness out of the prompt and into an explicit runtime contract.
The first architectural split I would make
I would keep the main business flow separate from reusable conversation-repair flows.
For example:
| Main booking flow |
Reusable repair/navigation behavior |
| Collect date |
Answer a temporary question |
| Collect time |
Correct a previously supplied value |
| Collect party size |
Clarify an ambiguous request |
| Check availability |
Repeat something |
| Present options |
Cancel or restart |
| Confirm selection |
Temporarily switch tasks |
| Commit booking |
Recover from a tool/system error |
That prevents every booking node from acquiring its own versions of:
- “If the user asks a question…”
- “If the user changes their mind…”
- “If the user corrects a previous answer…”
- “If the user interrupts…”
- “If the API fails…”
Rasa’s conversation-pattern model is one public example of this separation. It treats correction, clarification, interruption, cancellation, restart, handoff, repetition, silence, and system errors as reusable patterns rather than duplicating them inside every business flow.
Bland exposes a similar idea through Global Nodes: a restaurant-hours question can temporarily interrupt reservation collection, be answered, and then automatically return to the previous reservation node.
The important part is not either product. It is the reusable pattern:
main flow
|
+-- temporary repair / Q&A / clarification
|
+-- explicit outcome:
- resume the same step
- resume an earlier step
- start a different flow
- cancel the original flow
A practical default route
A low-risk migration path would be:
- Keep the current audio pipeline.
- Pick one representative flow, such as booking.
- Represent its state as typed data outside the LLM context.
- Ask the LLM for typed commands rather than asking it to “manage the whole conversation.”
- Validate and apply those commands in normal code.
- Expose only the tools legal in the current state.
- Generate the user-facing reply from the resulting state.
- Add stronger operation semantics only where there is an actual side effect.
For example, instead of expecting the model to remember all booking rules from prose, it might propose:
{
"commands": [
{
"type": "set_slot",
"slot": "party_size",
"value": 5
},
{
"type": "start_temporary_flow",
"flow": "restaurant_question",
"arguments": {
"question": "What time do you close?"
}
}
]
}
The controller would then decide:
- whether
party_size is currently editable;
- whether changing it invalidates an earlier availability result;
- whether the restaurant question can be answered locally or requires a tool;
- where to return after the question;
- whether another LLM response should be generated immediately.
This preserves the model’s ability to understand “There will be five of us, but before that, when do you close?” without giving it ownership of the booking transaction.
One important Pipecat branch
Your post says you use Pipecat SmartTurn V3, but it is not clear whether the rest of the pipeline is also built with Pipecat. The recommendation changes slightly, so I would keep both branches open:
- If the complete cascaded pipeline uses Pipecat: Pipecat Flows is a concrete implementation candidate.
- If Pipecat is only being used for SmartTurn: the same controller can live in your FastAPI application; Pipecat Flows can still be useful as a design reference.
- If the architecture is mixed: keep the business controller framework-neutral and adapt audio/pipeline events at the boundary.
SmartTurn belongs to the turn-detection side of the system. Its model card does not make it the owner of business state, slot dependencies, transaction commits, or the semantic return target after a detour.
Direct answers to the questions in the post
Are vendors generating a much larger prompt?
Possibly in some places, but that cannot be established from their public interfaces, and a larger prompt alone would not explain the documented workflow behavior.
A dashboard can reasonably compile configuration into some combination of:
- role or policy instructions;
- node-local task messages;
- retrieved business information;
- available tool schemas;
- state projected into the current context;
- transition conditions;
- repair/global flows;
- runtime hooks or webhooks;
- response templates.
So “prompt compilation” may be one piece without being the orchestration layer itself.
Are hierarchical prompts or dynamic injection used?
There are clear public examples of node-specific instructions and dynamically changing tools/context.
Pipecat Flows defines conversations as graphs where each node focuses the model on one task and only the tools needed for that task. It transitions by changing the LLM context and tools during the session.
That is dynamic injection, but it is being driven by an explicit flow runtime rather than by the LLM independently deciding what the authoritative state is.
Are FSMs, graphs, planners, or behavior trees used?
Workflow graphs and deterministic state machines are publicly documented. Planners and multi-agent routing also exist, but I would not make them the default for a booking workflow.
A useful complexity ladder is:
| Situation |
Smallest likely useful mechanism |
| Mostly linear flow |
Typed state plus deterministic validator |
| Temporary detours |
Return stack |
| Nested flows |
Dialogue stack or hierarchical state machine |
| Many conditional substates |
Workflow graph or statechart |
| Complex corrections and references |
Revision-aware or dataflow-style state |
| Long-running side effects |
Durable operation state |
| Truly simultaneous independent goals |
Scheduler or parallel states |
| Distinct specialists with independent responsibilities |
Multi-agent routing |
A flat FSM becomes awkward when every combination of slot values, corrections, detours, and tool states becomes a separate state. That does not mean abandoning state machines; it often means using hierarchy, a stack, orthogonal state variables, or a graph instead of enumerating the Cartesian product.
How is drift prevented while retaining natural conversation?
Usually by making the model flexible about interpretation and wording, but constrained about effects.
The model can remain free to:
- infer that “five of us” means
party_size = 5;
- recognize a temporary question;
- extract several slots from one utterance;
- paraphrase confirmations naturally;
- decide that clarification is needed.
The controller remains responsible for:
- required fields;
- legal transitions;
- which tools are currently available;
- whether confirmation is required;
- whether a result is stale;
- whether a write may be committed;
- where a temporary flow returns.
Rasa’s LLM command generators are a useful public example. A user turn can produce a sequence such as SetSlot(...) plus StartFlow(...), while the deterministic Flow Policy owns flow execution and state transitions.
How are interruptions and topic changes recovered?
I would not model all of them as one generic “resume” operation.
They are different events:
| Event |
Likely handling |
| User talks over the bot |
Stop or attenuate output; decide what generation/tool work is cancelled |
| Temporary business question |
Push a temporary flow and return afterward |
| Correction |
Revise a value and invalidate dependent results |
| New task |
Start or switch flow |
| Cancellation |
Explicitly cancel the current flow |
| Clarification |
Pause progression until ambiguity is resolved |
| Partial information |
Apply supplied slots and compute what remains missing |
| User continues an incomplete utterance |
Merge/replace transcript according to turn semantics |
The return behavior should be an explicit controller decision, not only a sentence in the prompt.
Is tool calling central?
For external facts and actions, yes, but tool calling by itself is not an orchestration architecture.
A tool schema tells the model how to request an operation. It does not necessarily guarantee:
- that the operation is legal in the current business state;
- that the arguments are still current;
- that the user confirmed it;
- that an old asynchronous result should still be accepted;
- that the operation will not be performed twice;
- that a cancelled generation cannot still trigger the effect.
The controller and backend still need to enforce those properties.
What probably sits behind the vendor dashboards?
From the public documentation, a reasonable description is:
Configuration is projected into local prompts, graph nodes, tool definitions, runtime conditions, and state transitions.
That is different from claiming knowledge of any vendor’s private internal implementation.
Why these edge cases become coupled
There appear to be at least four separate timelines in a real-time voice agent.
| Timeline |
Typical state |
Commit boundary |
| Audio/perception |
VAD, interim STT, final STT, turn completion, interruption |
What user input is accepted as a turn |
| LLM/context |
messages, active instructions, tool calls, tool results |
What becomes part of model context |
| Dialogue/workflow |
active flow, step, slots, return target, allowed actions |
What the controller accepts as current dialogue state |
| Business transaction |
lookup, tentative selection, confirmation, booking write |
What changes an external system |
A fifth useful timeline is output delivery:
| Output stage |
Meaning |
| Generated |
The model produced the text |
| Queued |
The text/audio is waiting to be sent |
| Partially spoken |
The user heard only part of it |
| Fully spoken |
The user heard the whole response |
| Context-committed |
The system recorded it as assistant history |
These stages can disagree during barge-in, cancellation, network delay, or TTS queueing.
That is why “the user interrupted, so resume naturally” is underspecified. It could mean:
- resume the audio sentence;
- restart generation;
- return to the previous workflow step;
- continue an asynchronous lookup;
- preserve a partially delivered confirmation;
- cancel a pending write.
Those should not all share one implicit state flag.
Pipecat’s context-management documentation explicitly distinguishes LLM-generated frames from TTS text frames, with the latter intended to represent what was actually spoken. That is an important architectural distinction even if the exact delivery/interrupt behavior still needs testing in a specific pipeline.
A minimal controller contract
The controller does not have to be a large framework. It can begin as a small typed service in the FastAPI application.
A minimal state might contain:
session_id
active_flow
current_step
return_stack
slots
slot_revisions
state_revision
pending_confirmation
allowed_commands
pending_tool_operations
committed_operations
cancelled_operations
active_generation_id
next_response_owner
last_spoken_output
Possible internal commands might include:
SetSlot
ClearSlot
StartFlow
StartTemporaryFlow
ResumeFlow
CancelFlow
AnswerKnowledgeQuestion
RequestClarification
RequestConfirmation
InvokeLookup
CommitOperation
Handoff
The important distinction is:
LLM output:
"Here are the commands I believe match the user's utterance."
Controller:
"Here are the commands that are legal and will actually be applied."
A simplified application rule could look like:
def apply_command(command, state):
if command.based_on_revision != state.revision:
return reject("stale command")
if command.type not in state.allowed_commands:
return reject("command not allowed in current state")
if not business_preconditions_hold(command, state):
return reject("business precondition failed")
new_state = transition(state, command)
new_state = invalidate_dependent_data(state, new_state, command)
new_state.revision += 1
return accept(new_state)
The LLM does not need to see every internal field. The controller can project a smaller view into the prompt:
Current task: restaurant booking
Current step: collect contact details
Known values:
- date: 2026-07-20
- time: 19:00
- party size: 5
Missing required values:
- email
Allowed next actions:
- set_email
- answer_restaurant_question
- cancel_booking
- request_human
That is more compact and more reliable than asking the model to infer the same state from a long transcript plus many exception rules.
State revision and dependency invalidation
Suppose availability was checked for:
date = July 20
time = 19:00
party_size = 5
Then the user says:
Actually, make that eight people.
The correct update is not only:
party_size = 8
It may also require:
availability_result = invalid
selected_option = invalid
pending_confirmation = invalid
The controller should know which derived data depends on which inputs.
A useful operation record might be:
{
"operation_id": "availability-123",
"type": "check_availability",
"based_on_state_revision": 17,
"arguments": {
"date": "2026-07-20",
"time": "19:00",
"party_size": 5
},
"status": "running"
}
If the result returns when the current state revision is 18 and the party size is now 8, the controller can mark it stale rather than allowing it to overwrite current state.
Read operations versus writes
Not every tool needs heavy transaction machinery.
A useful distinction is:
Read-only lookup
- request ID
- based-on revision
- stale-result check
- timeout/cancellation policy
Important write
- operation ID
- current-state validation
- explicit confirmation where appropriate
- backend-side authorization/validation
- idempotency key
- committed/failed/unknown status
- audit record
Booking creation should not rely only on the LLM deciding that a tool call “looks right.”
The general idempotency principle is described well in Temporal’s Activity documentation: retryable operations should be designed so repeated execution does not accidentally repeat the side effect. You do not need Temporal to apply that principle.
Mapping the examples from the post to reusable patterns
“If the user asks a question while I am collecting their email…”
This can be represented as a temporary detour:
collect_email
|
+-- push restaurant_question
|
+-- answer question
|
+-- pop and resume collect_email
The return target should be explicit. It can be a stack entry rather than a prompt instruction such as “remember to continue.”
“If the user changes the topic…”
“Topic change” can mean several different things:
Temporary question
-> push temporary flow, answer, return
New independent request
-> start another flow
Replacement request
-> cancel or suspend current flow, start new flow
Correction to current request
-> revise slot, invalidate dependent results
Ambiguous switch
-> ask a short clarification
Treating all of those as one intent called topic_change would likely move the ambiguity rather than solve it.
“If the user provides partial information…”
The model can emit several commands from one turn:
User:
"Tomorrow at seven, for five people."
Commands:
SetSlot(date, tomorrow)
SetSlot(time, 19:00)
SetSlot(party_size, 5)
The controller then computes missing required fields.
It does not need another prompt rule for every combination of supplied and missing fields.
“If the user interrupts…”
It helps to split acoustic interruption from semantic interruption.
Acoustic interruption:
- stop or duck TTS
- decide whether generation remains valid
- track what was actually delivered
- accept the new user turn
Semantic interruption:
- temporary detour
- correction
- cancellation
- task switch
- clarification
A caller speaking over “Your reservation is confirmed…” is not equivalent to a caller asking about parking while the system is collecting an email.
Corrections inside detours
A stack is particularly useful when detours can be nested:
booking flow
-> restaurant question
-> clarification
-> return to restaurant question
-> return to booking flow
Rasa’s Flow Policy publicly documents a last-in-first-out dialogue stack, where newly started flows can complete and then return control to the previous flow.
For more complex corrections and references, the SMCalFlow paper is useful background. It represents dialogue state as a dataflow graph and includes explicit operators for reference and revision. That is probably more machinery than a basic booking bot initially needs, but it provides vocabulary for cases where users revise or refer back to earlier values.
How much machinery is actually necessary?
I would start with the smallest mechanism that expresses the required invariants.
Typed state plus validator
Use this when:
- the flow is mostly linear;
- detours are shallow;
- there are few concurrent operations;
- correction rules are simple.
This is probably the default starting point.
Return stack
Add this when:
- temporary Q&A is common;
- clarification can interrupt collection;
- nested flows must return to their caller.
Hierarchical statechart or workflow graph
Add this when:
- a flat state list is becoming a Cartesian product;
- several substates share the same transitions;
- history or nested states are important;
- there are distinct reusable subflows.
Revision-aware/dataflow state
Consider this when:
- users frequently refer to earlier values;
- corrections need to propagate through derived values;
- multiple candidate plans coexist;
- reconstructing the intended state from a flat slot dictionary becomes difficult.
Durable workflow or operation ledger
Consider this when:
- operations last longer than a conversational turn;
- retries and process restarts matter;
- writes affect money, reservations, healthcare, or other important systems;
- “unknown whether committed” is a meaningful state.
Planner or multi-agent architecture
I would reserve this for cases where there are genuinely independent goals or specialists.
For a normal booking flow, adding a router LLM, planner LLM, worker LLM, reviewer LLM, and response LLM could increase latency and introduce more ownership ambiguity without solving the underlying state problem.
A local deterministic controller plus one model pass capable of emitting multiple commands is a simpler default.
Pipecat-specific route and current caveats
As of July 13, 2026, the latest Pipecat release is v1.5.0, and Flows is included in Pipecat.
If your full pipeline is Pipecat-based
Pipecat Flows is close to the orchestration layer you are describing:
- conversations are graphs of nodes;
- each node focuses the LLM on one task;
- each node exposes only relevant tools;
- context and tools can change during the session;
- flow logic is separated from pipeline mechanics.
The Flows examples are worth inspecting for reservation, intake, handoff, and related patterns.
Cross-node values can be stored in flow_manager.state.
However, I would treat that dictionary as application state storage, not automatically as:
- a revision system;
- an idempotency system;
- a transaction log;
- a dependency graph;
- an audit trail.
Those semantics still need to be designed where required.
Context strategy still matters
Flows currently offers the following context strategies:
APPEND, which is the default;RESET;RESET_WITH_SUMMARY, which is deprecated in favor of Pipecat’s native LLMSummarizeContextFrame.
A graph does not automatically prevent prompt growth if every node appends more instructions to the same context.
The context policy should therefore be explicit:
- preserve history when it remains relevant;
- reset when old node instructions could conflict;
- summarize when long-term facts matter but exact wording does not;
- keep authoritative business state outside the summary.
Function lifecycle still matters
Pipecat’s function-calling documentation exposes several independent controls:
- whether a tool is cancelled on interruption;
- synchronous versus asynchronous function calls;
- model-directed cancellation of asynchronous calls;
- whether parallel calls are grouped;
- whether an LLM run occurs after a result;
- intermediate versus final results.
These settings illustrate why “the graph moved to another node” and “the old operation no longer has effects” are not necessarily the same event.
Generated versus spoken output
Pipecat’s context-management documentation describes assistant context being updated from TTSTextFrame, representing the text passed through TTS, rather than treating every generated token as necessarily spoken.
That is the right distinction to preserve, especially during interruption.
Recent open reports are useful boundary tests
These reports do not establish that Pipecat is causing the behavior in your system. They are useful examples of lifecycle boundaries a production design should make explicit.
At the time of writing:
- Issue #4997 reports a tool call from an interrupted/discarded generation still executing.
- Issue #4936 reports a timed-out function handler continuing to run, with potential duplicate-side-effect risk.
- Issue #4979 reports duplicate responses after a multi-worker handoff; PR #4995 is an open proposed fix based on an explicit “no response” outcome.
- Issue #4996 reports already-played TTS text being lost from assistant context during interruption; PR #5008 is an open proposed fix.
The general invariant I would take from these examples is:
At any time, explicitly identify the owner of:
- the active generation;
- the next audible response;
- each tool operation;
- each workflow transition;
- each irreversible side effect.
If you only use SmartTurn
There is no need to migrate the whole system to Pipecat merely to get this architecture.
The FastAPI service can own:
ConversationState
CommandValidator
FlowStack
OperationRegistry
ResponseCoordinator
SmartTurn, Silero, Sarvam STT, and Sarvam TTS can remain adapters around that controller.
What the public vendor designs suggest—and what they do not prove
Bland
Bland Conversational Pathways publicly exposes:
- nodes;
- conditions;
- pathway labels;
- global nodes;
- webhook/API interactions;
- automatic return from a global node to the previous node.
Its reservation example specifically describes answering a restaurant-hours question during reservation collection and then returning to the original reservation step.
That supports the general graph-plus-local-prompt pattern. It does not reveal the complete private runtime, model prompts, training data, evaluation stack, or fallback logic.
Retell
Retell Function Nodes separate several choices:
- whether to wait for the function result;
- whether to speak while it executes;
- whether interruptions are blocked;
- when the transition occurs;
- whether transition conditions depend on the result.
That is another example of lifecycle decisions being explicit runtime configuration rather than one general prompt instruction.
Rasa CALM
Rasa publicly separates:
The exact Rasa framework may or may not suit your latency, licensing, or deployment needs, but the component boundaries are useful to compare against a custom implementation.
Research background
Task-Oriented Dialogue as Dataflow Synthesis is useful if correction, reference, and revision become more complex than a normal slot-filling flow.
I would use it as a design reference rather than jumping directly to implementing the full formalism.
Optional controls for deciding whether the architecture is improving
These do not need to become a large test project immediately. They can be treated as executable invariants.
| Invariant |
What it protects |
| No booking write before confirmation |
Premature side effects |
| Same operation ID cannot commit twice |
Retry/duplicate protection |
| A slot correction invalidates dependent results |
Stale availability or pricing |
| Temporary Q&A returns to the same missing field |
Detour recovery |
| Cancelled flow cannot later commit a tool result |
Zombie operations |
| Only one component owns the next response |
Duplicate speech |
| Context reflects delivered output, or records uncertainty |
Repetition after barge-in |
| A stale async result cannot overwrite current state |
Race conditions |
| Required fields cannot be skipped |
Workflow drift |
There are three useful validation layers.
Controller-level
Feed text commands or user turns into the controller without STT/TTS.
This isolates:
- slot updates;
- transition rules;
- return-stack behavior;
- stale-result rejection;
- operation idempotency.
Pipeline-level text mode
Run the real LLM and tool pipeline while bypassing audio.
This isolates:
- prompt/context behavior;
- command generation;
- function selection;
- multi-turn state;
- response ownership.
Audio-level
Add:
- STT;
- VAD;
- SmartTurn;
- TTS;
- barge-in;
- partial delivery;
- telephony/network behavior.
Pipecat’s built-in Evals framework supports text and audio modes, multi-turn assertions, function-call checks, interruptions, and timing budgets. It can be used directly if the complete agent runs in Pipecat, or simply used as a reference for the kinds of events worth asserting in a custom harness.
The point is not to enumerate every possible utterance. It is to preserve architectural properties across varied utterances.
Bottom line
I would not begin by trying to discover the exact hidden prompt used by Bland, Retell, or Vapi, because their public documentation already suggests that prompt text is only one layer.
I would also not begin with a large planner or multi-agent system.
The default I would choose is:
Natural-language interpretation
+
typed commands
+
deterministic workflow state
+
explicit repair/return behavior
+
validated and idempotent side effects
+
separate audio/output lifecycle
If your full pipeline already uses Pipecat, implementing one representative booking flow with Pipecat Flows is a reasonable route.
If Pipecat is only providing SmartTurn, the same pattern can be implemented in FastAPI without changing the rest of your stack.
Either way, the main shift is from asking the prompt to remember how the entire business process works to letting the model interpret the user while a runtime explicitly owns what has been accepted, what remains pending, what may execute, and where the conversation returns next.