Add Tools and Human Transfer to a Vonage + Deepgram Voice Agent

Introduction

The Vonage Voice API and Deepgram guide gets you surprisingly far.

By the end of the guide, you can call a Vonage number, speak to an AI agent, interrupt it while it’s talking, and have a natural voice conversation powered by Deepgram’s Voice Agent API.

But the agent still can’t actually do much for you (it has no tools!).

Ask the agent to do an action, and unless that action information happens to be in the prompt, it has nowhere to look. Give it a task that depends on another system, and now you have another problem: the caller is waiting while that system responds.

Voice makes those details noticeable very quickly. A slow API call doesn’t look like a loading spinner, it sounds like dead air.

This audio specific problem for AI agents was introduced in Don’t Loop the Latency: Where Agent Loops Belong in Voice AI. One of the ideas in his post is to keep the work that happens during a live phone call small and predictable.

The goal in this post is to turn the conversational demo into an agent that can do useful jobs, fail predictably, and -- most importantly -- leave behind enough information to improve it later. In this tutorial, we'll expand on the existing Vonage + Deepgram agent so it can:

• give the agent one tool for checking an order status

give the agent one tool for checking an order status

• put a hard timeout around the lookup

put a hard timeout around the lookup

• handle failures without leaving the caller waiting

handle failures without leaving the caller waiting

• transfer requests the agent can’t handle to a real phone number

transfer requests the agent can’t handle to a real phone number

• save a record of what happened during each call

save a record of what happened during each call

TL;DR The quick start is available on GitHub.

TL;DR The quick start is available on GitHub.

Prerequisites

You’ll need:

• A Vonage API account

A Vonage API account

• A Vonage application with Voice enabled

A Vonage application with Voice enabled

• A Vonage phone number linked to that application

A Vonage phone number linked to that application

• A second phone number you can use as the human support destination

A second phone number you can use as the human support destination

• A Deepgram account with Voice Agent access

A Deepgram account with Voice Agent access

• Node.js 20+

Node.js 20+

• ngrok

ngrok

Give the Agent a Tool

We’ll keep the use case deliberately simple: checking an order status. The caller says:

“Where is order A1001?”

“Where is order A1001?”

Deepgram recognizes that the agent needs outside information and asks our application to call:

getOrderStatus("A1001")

getOrderStatus("A1001")

Our application runs the lookup and sends the result back to Deepgram. The model then turns that result into a natural spoken response.

For this tutorial, the order backend is mocked. You can replace it with your own database or API later; we just need a lookup that can succeed, fail, or take too long:

There are also a couple of special order IDs for testing:

• IDs beginning with SLOW take two seconds to return.

IDs beginning with SLOW take two seconds to return.

• IDs beginning with FAIL simulate a temporary network failure.

IDs beginning with FAIL simulate a temporary network failure.

• Anything else returns not_found.

Anything else returns not_found.

That lets us reproduce the kinds of behavior we’d eventually get from a real database or external API without needing to build a full backend for this tutorial.

Register the Tool With Deepgram

Deepgram’s Voice Agent API supports client-side function calling. We describe the function in the agent’s Settings message:

We don’t give Deepgram an HTTP endpoint for the function. Instead, Deepgram sends our WebSocket a FunctionCallRequest. That means our application stays in control of actually running the tool.

That’s useful because we can decide:

• whether the tool is allowed

whether the tool is allowed

• how long it can run

how long it can run

• whether a failure should be retried

whether a failure should be retried

• what the caller hears if it doesn’t work

what the caller hears if it doesn’t work

The agent prompt stays narrow too:

The prompt tells the model what to do, but we don’t rely on the prompt alone. Before running an order lookup, the application checks that the request is in scope and that the tool hasn’t already been used on that call. You can see how this works in the “Handle the Function Call” section.

Put a Deadline Around the Lookup

This is where building for a phone call starts to differ from building a normal chatbot. Imagine the order service takes six seconds to respond. From the server’s point of view, that’s just a slow request.

From the caller’s point of view, they asked a question and then heard six seconds of silence.

So this example gives the lookup 1500 milliseconds:

The wrapper races the backend call against that deadline:

The important part is that there are three possible outcomes:

Outcome

What it means

What we do

result

The backend answered

Return the result

timeout

It didn’t answer within 1500 ms

Use the fallback

transport_error

The connection failed

Retry once

A not_found response is still a successful lookup. The backend answered; it just didn’t find the order.

That’s different from a timeout.

Limit What Can Happen During the Call

For this example, the rules are intentionally minimal:

Each phone call gets its own policy instance:

A timeout isn’t retried.

We’ve already waited 1.5 seconds. Starting the same slow request again could simply give us another 1.5 seconds of silence.

A transport failure gets one retry because a dropped connection or temporary upstream problem may recover immediately.

Handle the Function Call

When Deepgram sends a FunctionCallRequest, the WebSocket handler applies those rules before running the tool.

The main flow is:

The actual handler also records the tool call and timing, but this is the part that controls what the caller experiences.

Use a Fixed Fallback

If the lookup fails, you don’t want the model inventing its own recovery plan.

The application injects a fixed response:

Deepgram’s InjectAgentMessage lets us put that response directly into the conversation:

The interrupt behavior matters here.

Using the default behavior can cause an injected message to be refused while the caller is speaking. That’s not something you want to discover when the injected message is your failure path.

Also note the field name: it’s message, not content.

Both are small details that can otherwise result in a call going quiet with very little indication of why.

Transfer Unsupported Requests to a Human

An order-status agent shouldn’t try to answer everything.

If someone says:

“I want to dispute a charge.”

“I want to dispute a charge.”

We classify that as a billing request:

For this demo, the classification is deliberately constrained and deterministic:

There’s no second model call here.

If the caller asks for something outside the agent’s job, we know we want the same outcome every time: send the call to a person.

transferToHuman() updates the active Vonage call with a new NCCO.

The NCCO first tells the caller what’s happening, then connects the call to the support number:

Then we transfer the active call:

For local testing, SUPPORT_PHONE_NUMBER can simply be another phone you have nearby.

Call your Vonage number from one phone, ask to dispute a charge, and the other phone should ring.

That’s a much better fallback than having the AI pretend it can solve a billing problem it was never given access to.

Configure the Transfer

To make the real handoff work, add these values to your environment:

You’ll also need the existing configuration:

Use E.164-format numbers for the Vonage and support numbers.

Save What Happened During the Call

The application also writes a small record to SQLite when the call ends.

This isn’t the order database. It’s a record of the voice interaction itself.

A record can tell us:

• what the caller and agent said

what the caller and agent said

• which tool was called

which tool was called

• how long it took

how long it took

• whether the fallback was used

whether the fallback was used

• whether the caller was transferred

whether the caller was transferred

• why the transfer happened

why the transfer happened

• how the call ended

how the call ended

For example, a billing call might end up roughly like:

That is useful even if you never build a complicated evaluation system around it.

If callers keep getting transferred for the same reason, you can see it. If timeouts start increasing, you can see that too. And if you change the prompt, model, or tools later, the agent_version field gives you a simple way to tell which version handled which calls.

How to Run the Agent

Start ngrok first:

Put that HTTPS URL into BASE_URL and into your Vonage application’s Answer and Event webhooks.

Then start the application:

You can sanity-check the answer webhook before making a phone call:

The returned NCCO should contain a WebSocket endpoint similar to:

wss://YOUR-NGROK-URL/socket?callUuid=test123

Try Four Calls

Try testing the main behaviors as separate phone calls.

Call

What to say

What should happen

“Where is order A1001?”

Agent reads back the delivery status

“Where is order SLOW999?”

Lookup times out and the fixed fallback plays

“Where is order XYZ123?”

Agent tells you the order couldn’t be found

“I want to dispute a charge.”

The call transfers to your support phone

Make each one a fresh call. This example allows one order lookup per call.

Afterwards, inspect the records:

You should see the different paths represented in the data:

completed fallback completed handoff

For the last call, handoff_reason should be billing.

Conclusion

The original Vonage + Deepgram tutorial gets a conversational AI agent onto a real phone call. Here, we gave that agent one useful job: look up an order, stop waiting when the backend is too slow, fall back cleanly when something fails, and transfer unsupported requests to a human.

The order backend is mocked, but the pattern is the same if you replace getOrderStatus() with your own database or API.

This is Vince's "don't loop the latency" idea in practice: while the caller is waiting, keep the work small, put deadlines around external calls, and provide a useful escape route.

After the call, those latency constraints disappear. That's where the records we saved, transcripts, tool calls, outcomes, handoff reasons, and agent versions, become useful.

If returns starts showing up repeatedly as a handoff reason, for example, we can use those calls to decide what to build next, add a returns capability to a candidate version, and test it against the saved calls before putting it in front of callers.

That's the beginning of a loop-engineering workflow. But that’s for the next post!