Guide
Gemini Live API Async Function Calling for Real-Time Voice Agents
One of the biggest usability problems in real-time voice agents is tool latency. If the model needs to query CRM, orders, databases, search, or ticketing systems and every function call blocks the conversation, the user experiences several seconds of silence. Gemini Live API supports asynchronous function calling in its cascaded architecture. A function can be marked `NON_BLOCKING`, allowing the live conversation to continue while the tool executes in the background. This does not mean the model can invent tool-dependent facts. It means developers need explicit task state, timeouts, cancellation, concurrency controls, result correlation, permissions, and idempotency.
# Gemini Live API Async Function Calling for Real-Time Voice Agents
## Article Summary
One of the biggest usability problems in real-time voice agents is tool latency. If the model needs to query CRM, orders, databases, search, or ticketing systems and every function call blocks the conversation, the user experiences several seconds of silence. Gemini Live API supports asynchronous function calling in its cascaded architecture. A function can be marked `NON_BLOCKING`, allowing the live conversation to continue while the tool executes in the background. This does not mean the model can invent tool-dependent facts. It means developers need explicit task state, timeouts, cancellation, concurrency controls, result correlation, permissions, and idempotency.
---
## 1. Why tool latency hurts voice agents
Text users may tolerate a few seconds. Voice users notice silence immediately.
A normal live interaction can include speech understanding, CRM, databases, search, model generation, and audio output. External systems have unpredictable latency.
If every tool call blocks the session, natural conversation disappears.
## 2. Blocking function calls
A traditional flow is:
```text
user asks
β model calls get_refund_status
β conversation pauses
β backend waits four seconds
β result returns
β model resumes
```
The user hears a long gap.
That may be acceptable for some transactional workflows, but it is poor conversational design.
## 3. What NON_BLOCKING changes
With asynchronous function calling:
```text
user request
β model starts tool call
βββ tool executes in background
βββ conversation continues
β result returns
β model incorporates result
```
The agent can ask a clarification or explain the process while waiting.
## 4. Asynchronous does not mean speculative
If the refund status has not returned, the agent must not claim that the refund is complete.
Separate state into:
```text
known
pending
unknown
```
Only confirmed data should be presented as fact.
## 5. Good non-blocking tools
Useful candidates include customer lookup, order status, tickets, inventory, shipping, knowledge retrieval, web search, recommendations, and other independent read operations.
Several independent read calls can also execute in parallel.
## 6. Poor non-blocking candidates
Be cautious with payments, deletes, production changes, irreversible operations, and any eligibility decision that is required before downstream logic can continue.
These often require blocking behavior and explicit approval.
## 7. Recommended architecture
```text
microphone
β Gemini Live session
β tool router
βββ blocking
βββ non-blocking
βββ approval-required
β async task manager
βββ timeout
βββ retry
βββ cancellation
βββ idempotency
β enterprise APIs
β tool result
β live session
```
Add authorization, tracing, cost controls, and audit logs.
## 8. Tool policy belongs in the application
Maintain metadata such as:
```yaml
name: get_refund_status
mode: non_blocking
timeout_ms: 5000
retry: 1
idempotent: true
risk: read
requires_confirmation: false
```
For a financial write:
```yaml
name: create_refund
mode: blocking
timeout_ms: 10000
retry: 0
idempotent: true
risk: financial
requires_confirmation: true
```
The model should not classify business risk by itself.
## 9. Conceptual Gemini declaration
Exact syntax should follow the active SDK version, but conceptually:
```python
tool = {
"function_declarations": [
{
"name": "get_order_status",
"description": "Get current order status",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string"}
},
"required": ["order_id"]
},
"behavior": "NON_BLOCKING"
}
]
}
```
The important behavior is that the live session does not have to stop while the function runs.
## 10. Use an async task manager
Do not block a WebSocket callback with synchronous HTTP work.
Instead:
```python
async def execute_tool(call):
task_id = create_task_id(call)
task = asyncio.create_task(run_with_timeout(call))
registry[task_id] = task
```
When it completes:
```python
async def on_tool_done(task_id):
result = await registry[task_id]
await send_tool_result(result)
```
## 11. Every tool needs a timeout
External services may respond in 300 milliseconds, two seconds, eight seconds, or never.
A timed-out task should return structured state instead of remaining pending forever.
## 12. Continue the conversation during delays
Instead of dead air, the agent can say:
> βThe customer system is taking a little longer. I can confirm a few details while it finishes.β
This is one of the main user-experience benefits of asynchronous tools.
## 13. Limit parallelism
A model may want to call CRM, orders, tickets, payments, email, and analytics at once.
Set a practical limit such as:
```text
max_parallel_tools = 3
```
Queue the rest.
This protects downstream systems and keeps state manageable.
## 14. Correlate results correctly
Store:
```text
session_id
call_id
tool
arguments
start time
conversation turn
```
When a result returns, match the call ID, confirm that the session is still valid, decide whether the result remains relevant, and only then deliver it back to the live model.
## 15. Handle topic changes
The user may request order A and then immediately correct the request to order B.
If possible, cancel A. If cancellation is impossible, mark its result stale and do not inject it into the active conversation.
Useful states include:
```text
PENDING
RUNNING
COMPLETED
TIMEOUT
FAILED
CANCELLED
STALE
```
## 16. Use idempotency for writes
Async systems create retry ambiguity. A remote write may succeed while the client loses the response.
Use an idempotency key for side-effecting operations such as ticket creation or order changes. The target service should guarantee one execution per key.
## 17. User interruptions must stop current speech
A real voice agent must support barge-in.
When a user interrupts, stop current audio, process the new input, decide whether existing tool tasks remain relevant, cancel or mark them stale, and continue with the updated goal.
## 18. What the agent can do while waiting
Useful actions include asking clarifying questions, collecting identity details, explaining process, and running independent read tools.
The agent should never invent pending results, promise unconfirmed outcomes, or claim that an operation completed before confirmation.
A useful instruction is:
```text
Never state that a tool-dependent fact is confirmed
until the corresponding tool result is received.
```
## 19. Authorization remains mandatory
A model-generated tool call is not authorization.
The application still needs:
```text
user identity
β role
β resource permission
β tool permission
β argument validation
β execution
```
Otherwise the voice agent becomes an authorization bypass.
## 20. Minimize sensitive tool output
If a CRM result contains identity documents, bank information, contact details, and order state while the user asked only for order status, return only the necessary field to the model.
Minimizing model exposure is a basic production principle.
## 21. Observability
Track:
```text
session_id
call_id
tool
mode
start
end
latency
status
retry count
cancellation
token use
business result
```
Important metrics include tool P50/P95 latency, timeout rate, cancellation rate, parallel call count, task success, first-audio latency, and conversation silence time.
## 22. Conversation silence time is a key metric
Backend tool latency alone does not describe user experience.
A tool may take four seconds, but if the agent keeps the conversation useful, the user may experience less than a second of dead air.
Measure silence, not only backend latency.
## 23. Customer-support example
A customer asks why a package has not arrived.
The agent starts shipping lookup while asking whether this is the same order discussed yesterday.
The user responds while the tools execute.
When the result returns, the agent provides the confirmed shipping status.
The backend was slow, but the conversation never fully stopped.
## 24. Blocking versus non-blocking policy
A practical classification is:
```text
READ_FAST
READ_SLOW
WRITE_LOW_RISK
WRITE_HIGH_RISK
FINANCIAL
PRODUCTION
```
Possible defaults:
```text
READ_FAST β NON_BLOCKING
READ_SLOW β NON_BLOCKING + status
WRITE_LOW_RISK β confirmation
WRITE_HIGH_RISK β BLOCKING + confirmation
FINANCIAL β BLOCKING + strong approval
PRODUCTION β BLOCKING + multi-party approval
```
## 25. Launch testing
Test slow responses, timeouts, disconnects, retries, out-of-order results, user interruptions, requirement changes, cancellations, authorization failures, prompt injection, sensitive data, and duplicate writes.
The final business state must remain correct even when the conversation is interrupted.
## Conclusion
Gemini Live API asynchronous function calling addresses a fundamental real-time-agent problem:
> slow tools should not automatically make the conversation slow.
With non-blocking calls, an agent can continue clarification, explanation, and independent work while external systems execute.
But the architecture needs task state, timeouts, cancellation, call IDs, concurrency controls, idempotency, authorization, sensitive-data filtering, and observability.
The central rule is simple: the agent may continue speaking while a tool runs, but it must never pretend to know a tool-dependent result before that result arrives.
For more practical Gemini API, voice-agent, function-calling, and production AI engineering guides, visit **Zyentor Picks**: https://www.zyentorpicks.com/.