Article
Nov 16, 2026
read
A SignalWire AI agent in a jack-o'-lantern
Adam Kalsey
EVP, Product
Build it free.
Create a space and ship your first call flow in minutes.
Voice AI
AI Agents
Developer Tutorials
Three kids walk up the driveway on Halloween night. A scarecrow with a burning jack-o'-lantern for a head watches them come.
"A witch," it says. "How daring. How utterly unprecedented."
The witch laughs. The vampire next to her asks the pumpkin how old it is. "I was carved on a Tuesday," it says, and goes back to ignoring them as attentively as it can.
The pumpkin is real, and it has a secret hidden inside: a SignalWire AI agent on a video call. A computer in the prop keeps the call open all night, with a webcam for eyes and a speaker for a mouth. The agent sees who's in front of it and talks to them in character. When a small child is in the group, it stays as grand as ever, but nobody in front of it is the joke.
This series shows how I built it. The excerpts in each post show the ideas, and the ai-jackolantern repository has the complete, working code.
What the series covers
The series has nine parts, and each one adds a working piece:
- The system and the call (this post).
The system and the call (this post).
- The agent: a personality and three modes, as contexts and steps.
The agent: a personality and three modes, as contexts and steps.
- Tools: what the AI calls on my server, and why its material lives in code.
Tools: what the AI calls on my server, and why its material lives in code.
- Vision: letting the agent see, and checking the camera before any decision that matters.
Vision: letting the agent see, and checking the camera before any decision that matters.
- The stage: detecting who is standing in front of the pumpkin, in the browser.
The stage: detecting who is standing in front of the pumpkin, in the browser.
- The voice: audio effects in the browser.
The voice: audio effects in the browser.
- Pacing and state: the server keeps score.
Pacing and state: the server keeps score.
- The operator: a person with a phone who can overrule the agent.
The operator: a person with a phone who can overrule the agent.
- Testing it, and running it for a whole night.
Testing it, and running it for a whole night.
By the end of this post, a browser holds a video call to a SignalWire AI agent.
The pieces
The pumpkin runs in four places:
Piece
Runs on
Job
The pumpkin page
The computer in the prop
Holds the call, sends the camera and mic, plays the voice through audio effects, detects who is on stage
The AI agent
SignalWire
Listens, looks, talks, and calls tools
My server
Any host with a public HTTPS address
Serves the agent's configuration and the page, runs the tools, keeps the state of each visit
The phone remote
The operator's phone
Starts and stops the show, adjusts settings, switches modes
The server is one Python process built on our Python SDK (signalwire-sdk). It serves the agent, the pumpkin page, and the page's APIs from one port.
How a call flows
Every time the page dials, the call goes through the same steps:
- The page asks my server for a guest token at /eyes-token.
The page asks my server for a guest token at /eyes-token.
- The page dials the agent's address with that token, over video.
The page dials the agent's address with that token, over video.
- SignalWire fetches the agent's configuration from /pumpkin on my server. The configuration is SWML, our markup for call flows, and the SDK writes it from the Python code.
SignalWire fetches the agent's configuration from /pumpkin on my server. The configuration is SWML, our markup for call flows, and the SDK writes it from the Python code.
- SignalWire runs the ai verb from that SWML: speech recognition, the model, text to speech, and vision.
SignalWire runs the ai verb from that SWML: speech recognition, the model, text to speech, and vision.
- When the model calls a tool, SignalWire posts the request to /pumpkin/swaig on my server, and my code answers.
When the model calls a tool, SignalWire posts the request to /pumpkin/swaig on my server, and my code answers.
Set up SignalWire from code
The page dials an address that belongs to a resource in my SignalWire space. My server creates that resource when it starts, so a new deployment needs no setup in the dashboard. The resource is an SWML webhook, which means SignalWire fetches the call's instructions from a URL when a call arrives. The setup code finds the resource by name, creates it if it's missing, and corrects its URL if it points somewhere else. Running it twice changes nothing.
The agent's URL carries a basic-auth user and password, so only SignalWire can fetch the configuration. I derive the password from the API token with an HMAC, so it comes out the same after every restart. That means a call in progress keeps working when the server reloads.
The server's .env holds SIGNALWIRE_SPACE, SIGNALWIRE_PROJECT_ID, and SIGNALWIRE_API_TOKEN, plus SWML_PROXY_URL_BASE, the server's public HTTPS address, so SignalWire knows where to fetch the agent.
Mint a guest token
Anyone can read what's in a browser page, so the page never gets the project credentials. My server mints a guest token for it instead, and that token can dial one address and expires in two hours.
The response also tells the page to dial the address's video channel, so the agent can see. The token route sits behind a password (EYES_PASSWORD), because each token can start a billed call.
Dial from the browser
The page uses our Browser SDK v4 (@signalwire/js), which takes a token provider, an object with authenticate() and refresh() methods. My provider fetches from /eyes-token, and the SDK calls refresh() before a token expires, so one call can outlive many tokens.
The dial sends the page's processed microphone stream and the camera, which I open at 640 by 480 and 10 frames a second. That's enough for the vision model, and it keeps the work light for the hidden computer. The pumpkin has no screen, so it receives audio only.
Keep the call up all night
A call that runs for hours will drop now and then, so the page dials in a loop if the call drops. A call that ran at least 30 seconds redials at once. One that failed right away backs off, from 5 seconds up to 60, so a broken agent can't spin the loop. If a dial hasn't started within 30 seconds, the page rebuilds the SDK client in place. A reload would need another click on the page before Chrome plays audio.
Traps to avoid
Don't put project credentials in the page. Anyone can open the page's source. Sure, you're the only one running the page and no one else knows the URL, but have good habits from the start. Mint a short-lived guest token on your server, limited to the one address the page may call, and put a password on the token route.
Don't hand the SDK only your processed audio. Once you pass inputAudioStream, the SDK sends only the streams you give it and ignores device settings. Pass the camera as inputVideoStream too, or the call goes out with no video and the agent is blind.
Next in this series
The call connects, but the agent on the other end has no personality yet. In part 2 I give it one, with a base prompt and three modes built as contexts and steps. All of it can be checked without placing a call.












