- Voice AI
- AI Appointment Setters
How AI Voice Agents Work: Speech, Brains and Booking
Josh Pocock
8 min read
An AI voice agent is software that answers or places phone calls and holds a real conversation with the caller. Under the hood it is a chain of separate systems: one that hears, one that thinks, one that acts and one that speaks.
AI voice agents work by turning the caller's speech into text, sending that text to a language model that follows your instructions and knowledge base, letting the model use tools like your calendar, then turning its reply back into speech fast enough to feel like a normal call.
Here is each stage of the pipeline, what breaks at each one and how a good agent gets tested before it talks to your leads.
The pipeline in one picture
Every voice agent call runs the same loop, many times per conversation:
| Stage | What it does | What it needs from you |
|---|---|---|
| Speech-to-text | Converts the caller's audio into words | Nothing, though industry terms may need tuning |
| Language model | Decides what to say or do next | Instructions, a knowledge base, rules for hand-off |
| Tools | Checks the calendar, looks up or updates the CRM | Access to your calendar and CRM |
| Text-to-speech | Turns the reply into a spoken voice | A voice choice and pronunciation fixes |
| Turn-taking | Decides when the caller is done talking | Tuning for your callers |
The caller hears one voice. Behind it, each of these parts can come from a different provider. On StrideOps we can swap between 23 AI providers, which matters when one model handles a certain accent or task well and another struggles with it.
Stage 1: Speech-to-text
The first job is listening. Speech-to-text (also called automatic speech recognition) streams the caller's audio and writes out the words as they speak.
Streaming matters. A system that waits for the full sentence before transcribing adds delay to every turn. A streaming system has most of the sentence written by the time the caller stops.
What goes wrong here
- Names and addresses. "Nguyen" or "Schuylkill Road" can come out wrong, so a good agent reads back anything it will write to the CRM.
- Industry words. Product names, treatment names and local street names may need to be added as hints.
- Noise. A caller on a job site or in a car produces messier transcripts, and the agent should ask again instead of guessing.
Stage 2: The language model, instructions and knowledge base
Once the words are text, a large language model decides what to do next. It reads three things on every turn: the conversation so far, the instructions you gave it, and any facts it pulled from your knowledge base.
Instructions are the agent's job description. They say who it represents, what it is trying to do (usually book an appointment), which questions to ask and in what order, and what it must never do. Our agents never give medical, legal, financial, insurance or rate advice, and never promise coverage, approvals or savings. Those rules live here.
The knowledge base holds your business facts: services, service area, hours, what happens at the first appointment, answers to common questions. When a caller asks "do you work in Plano?" the agent looks it up instead of making something up.
The script itself, meaning the openers, qualifying questions and objection lines, sits inside the instructions. We cover writing one in detail in our AI appointment setter script guide, so this post sticks to the machinery.
What goes wrong here
The main risk is the model answering a question it should not. If a caller asks something outside the knowledge base, the right behavior is "I'll get that answer from the team," not a confident guess. Tight instructions and a clear hand-off rule fix most of this.
Stage 3: Tools that let the agent act
A voice agent that can only talk is a fancy answering machine. Tools let it do real work during the call.
Common tools for an appointment-setting agent:
- Calendar availability. The agent checks open slots and offers two specific times.
- Booking. Once the caller picks, the agent writes the appointment to your calendar.
- CRM lookup. The agent checks whether this caller is an existing customer or an open lead, so it does not ask questions you already have answers to.
- CRM update. After the call, the agent logs a summary, the answers it collected and the outcome.
In practice those tools are the systems you already run on. The booking is written to your real calendar, and the lookup and the call summary go to the CRM your team already works in.
Google Calendar
GoHighLevel
HubSpot
Salesforce
Twilio
The model decides when to call a tool. "Tuesday at 3 works" triggers a booking. "Do you have anything Saturday?" triggers an availability check. The tool returns a result, and the model turns it into a sentence.
What goes wrong here
Tools add delay, since each lookup takes time. They can also fail: a calendar sync that is out of date can double-book a slot. A good build confirms the booking actually saved before telling the caller it did.
Stage 4: Text-to-speech
The model's reply is text. Text-to-speech turns it into audio, and modern voices sound close to human, with natural pacing and emphasis.
Like transcription, this streams. The agent starts speaking the first words of the reply while the rest is still being generated, which cuts the wait.
What goes wrong here
Pronunciation is the common problem. Your business name, a street name or a product might come out oddly. Most voice systems let you add pronunciation rules. Test every proper noun in your knowledge base out loud before going live.
Latency, turn-taking and interruptions
This is where voice agents feel either natural or robotic.
Latency
Latency is the gap between the caller finishing a sentence and the agent starting to answer. Each stage above adds a bit: transcription, the model thinking, any tool calls, then speech generation. Long pauses make callers say "hello?" or hang up. StrideOps agents respond in under 2 seconds.
Turn-taking
The agent has to guess when the caller is done talking. People pause mid-sentence ("I'm looking for... um... a roof inspection"). If the agent jumps in at every pause, it interrupts. If it waits too long, it feels slow. Tuning the end-of-turn detection takes real care in every build.
Interruptions
Callers talk over agents, same as they do with people. A good agent stops speaking when the caller starts (often called barge-in), listens, and responds to what they said instead of finishing its prepared line. An agent that keeps talking over "wait, wait, I already did that" loses the caller fast.
Hand-off to a human
No agent should try to handle everything. Write the exits down before launch:
- The caller asks for a person.
- The question falls outside the knowledge base.
- The caller is upset or has a complaint.
- The request needs a licensed professional, such as a quote or rate question.
A hand-off can be a live transfer to your team or a booked callback with a summary attached. Either way, your person should see the transcript so the caller does not repeat themselves. We compare where each side does better in AI vs human appointment setters.
Consent and disclosure on outbound calls
When an agent places calls instead of answering them, the rules change. The FCC ruled in February 2024 (FCC 24-17) that the TCPA's limits on "artificial or prerecorded voice" calls cover AI-generated voices, so those calls need the called party's prior express consent.
Our agents only call leads who opted in: people who filled out a form, asked for a callback or are existing customers who agreed to hear from you. Put consent for calls and texts on every form, and confirm the wording with your own counsel. Disclosure that the caller is speaking with an AI is set to match the rules in your market.
How voice agents get tested before going live
You do not want your first real lead to be the test. We run every StrideOps agent through simulated calls before launch, playing the difficult caller on purpose:
- Talk over the agent mid-sentence.
- Give a messy address and see if it reads it back correctly.
- Ask a question that is not in the knowledge base.
- Ask for a person and confirm the hand-off works.
- Pick a time, then change your mind.
- Ask "am I talking to a robot?"
After launch, the testing moves to real transcripts. Full optimization takes 30 to 60 days of real conversations, because real callers find edge cases no test script predicts.
FAQ
How do AI voice agents understand what callers say?
They use speech-to-text to turn the caller's audio into words in real time. A language model then reads those words, along with your instructions and knowledge base, and decides how to respond.
Can callers tell they are talking to an AI voice agent?
Modern voices sound natural, but the agent should say it is an AI assistant anyway. Disclosure is set to match the rules in your market, and the agent hands off to a person on request.
Can an AI voice agent book appointments directly?
Yes, when it has a calendar tool. It checks open slots during the call, offers specific times and writes the booking to your calendar once the caller confirms.
What happens when an AI voice agent doesn't know the answer?
A well-built agent says it will get the answer from your team, then hands off with a summary. It should never guess, especially on price, coverage or anything regulated.
See a voice agent built for your business
Our done-for-you AI appointment setters handle the full pipeline, from voice to calendar to hand-off, and most clients are live in 2 to 3 weeks. Book a Stride Scaling Call.
Done for you
Want this running in your business?
We build and run AI that answers, follows up and books calls, all reporting into one CRM. Most clients are live in 2 to 3 weeks.