A caller rings a kitchen shop about a cast iron skillet that arrived cracked, hoping to get their money back. The voice agent politely replies, "Let me process that refund for you now." While the payment provider processes the transaction in the background, everyday life intervenes: the caller murmurs a quiet "mm-hmm," the evening news buzzes on a nearby television, or the caller suddenly remembers something and shouts, "Wait, wrong card!"
Real-time voice agents inhabit a messy world of background noise and spontaneous interruptions. When I set out to test these scenarios, I wanted to answer two practical questions: Which of these everyday sounds might trick an agent into issuing a duplicate refund? And second, what causes an agent to process a refund successfully, yet drop into complete silence without ever telling the caller it worked?
To find out, I built an automated refund desk on PolyAI's Dialog-RSN-1 API (pmckeetx/refund-voice-agent) and ran a scripted caller through 75 measured calls. Passive backchannels turned out to be completely harmless: a soft "mm-hmm" spoken over the agent never changed the outcome across 20 test runs. The real breakdown happened when the caller asked a natural follow-up question over the holding line: "Sorry, did that go through?"
Here is what that interaction looked like on the starting client implementation on 10 November 2026. This naive client suppresses follow-up agent responses once the caller speaks, and it mistakenly keys refunds on ephemeral tool calls rather than the underlying order (run as caller.py --before in the repository, producing R-4417-1):
(caller says refund.wav)
Caller: Hi, I need a refund on order four thousand four hundred seventeen. The cast iron skillet arrived cracked.
(issue_refund 4417 to 4242: refunded)
(caller says did-it-go-through.wav)
(barge-in: stopped the reply after 1.7 s)
Agent: Let me process that refund- [cut off]
Caller: Sorry, did that go through?
Agent: It's still processing. I'll let you know as soon as it goes through.
refund_id order amount card status told calls
R-4417-1 4417 32.00 4242 pending no 1
The money was sent, but the call fell silent. Across 10 out of 10 test runs, the agent promised to confirm the refund as soon as it finished, but never spoke another word. In the real world, this customer wakes up tomorrow, assumes the request was lost, and calls back to demand a refund they have already received.
The bug is subtle. The language model answered the caller's immediate question before the payment gateway returned. When the tool result finally arrived, the client saw that the user had spoken in the interim, decided not to interrupt with an unprompted reply, and abandoned the confirmation entirely.
Solving this requires a client-side ledger that tracks both the state of the transaction and whether the caller has actually heard the confirmation. By keying refunds directly to the order ID, duplicate tool calls resolve to the existing transaction, and card switches update the pending record instead of adding a second charge. Whenever the gateway finishes and no subsequent response has started, the client explicitly prompts the model for confirmation, and automated tests verify that the ledger only marks a refund told when the customer has heard the outcome.
What you'll build
In this guide, you will build a terminal-based voice agent for a fictional store, Copperline Kitchen Supply. This agent enforces strict financial idempotency: it pays exactly one refund per order regardless of how many times the model triggers the tool. If the caller blurts out "wait, wrong card" while the transaction is still pending, the agent diverts the pending refund to the new card rather than creating a duplicate payout.
The agent also ensures every transaction is accompanied by an audible confirmation, even if the caller interrupts with a question mid-process or talks directly over the agent's first attempt to speak. The project includes a scripted caller that simulates acoustic environments and phone calls in real time, allowing you to observe edge cases without touching a physical microphone, along with a test suite that reconciles your SQLite ledger against the audio heard by the caller.
How it works
When a caller requests a refund on Dialog-RSN-1, the model emits an issue_refund call. The client receives the response.function_call_arguments.done event before the audible holding line begins playing; across 72 test runs, the spoken holding audio lagged the event by a median of 0.12 seconds. The client kicks off the background refund immediately upon receiving that event.
Even if the caller barges in and cancels the spoken holding phrase, the tool execution itself has already been dispatched. In 10 test runs where the caller interrupted the agent mid-sentence, the tool call remained safely recorded in the server's conversation history every time.
This means transaction integrity rests entirely on the client, where three failure points emerge. The model might trigger the refund tool more than once, which the ledger must recognize as the same refund. The caller might speak while the gateway is processing, so the client has to decide when to ask for a confirmation. And the confirmation itself might be cut off by an interruption, leaving a completed refund the caller never heard.
Under the hood, barge_in.py manages audio interruptions: it halts outbound audio as soon as the server detects incoming speech, truncates conversation history to what was actually played out loud, and logs cut-off replies. We will leave that module as-is to focus on the heart of the system: the ledger and turn coordinator.
Prerequisites
To follow along, you will need Python 3.13 and uv, which locks down dependencies to match this environment. You will also need an active Dialog-RSN-1 API key, which you can generate under the API Keys tab at studio.poly.ai.
Calls run $0.02 per connected minute plus tokens, new accounts receive launch credits, and sign-up tiers support two concurrent sessions (limits and pricing, verified 10 November 2026). Each automated test call in this walkthrough finishes in under a minute.
The scripted testing suite operates entirely without a physical microphone or speaker setup. However, if you would like to test the live voice agent interactively using agent.py, keep a microphone and a pair of headphones ready.
Set up the project
Start by cloning the repository and generating your local environment configuration:
git clone https://github.com/pmckeetx/refund-voice-agent.git
cd refund-voice-agent
cp .env.example .env
Open .env in your editor and supply your API key:
# Your Dialog-RSN-1 API key: sign in at https://studio.poly.ai and create one in the API Keys tab.
POLY_API_KEY=
# Optional: the full realtime endpoint. Defaults to wss://api.studio.poly.ai/v1/realtime.
# DIALOG_REALTIME_URL=
Leave DIALOG_REALTIME_URL commented out unless your key is for a different PolyAI host. Next, install project dependencies and execute the offline test suite, which runs without requiring network access or API credentials:
uv sync
uv run pytest
...
tests/test_agent.py ...... [ 20%]
tests/test_audio.py .... [ 33%]
tests/test_ledger.py ........ [ 60%]
tests/test_live.py sss [ 70%]
tests/test_refunds.py ......... [100%]
======================== 27 passed, 3 skipped in 22.07s ========================
The three skipped tests represent live calls. When enabled, they run scripted audio against the real-time API using your credentials, taking roughly 90 seconds:
POLYAI_LIVE=1 uv run pytest -m live -v
...
tests/test_live.py::test_one_refund_and_the_caller_heard_it[mm-hmm] PASSED [ 33%]
tests/test_live.py::test_one_refund_and_the_caller_heard_it[did-it-go-through] PASSED [ 66%]
tests/test_live.py::test_one_refund_and_the_caller_heard_it[missed-confirmation] PASSED [100%]
================= 3 passed, 27 deselected in 84.89s (0:01:24) ==================
Setting POLYAI_LIVE=1 activates live integration, while -m live limits test collection to those scenarios. If you encounter an HTTP 401 error, verify that an existing POLY_API_KEY in your shell environment is not overriding your .env value.
Step 1: Give every refund a key
Our ledger consists of a single SQLite table, and it uses order_id as its primary key, making it structurally impossible to insert multiple refunds against the same order.
Here is the schema defined in ledger.py:
SCHEMA = """
CREATE TABLE IF NOT EXISTS refunds (
order_id TEXT PRIMARY KEY, -- the key: what the refund is for, not who asked
refund_id TEXT NOT NULL UNIQUE,
amount_cents INTEGER NOT NULL,
card_last4 TEXT NOT NULL,
status TEXT NOT NULL DEFAULT 'pending', -- pending until the payment provider settles it
session_id TEXT NOT NULL, -- the call that asked first
call_ids TEXT NOT NULL, -- every tool call that asked for it, as a JSON list
accepted INTEGER NOT NULL DEFAULT 0, -- has the payment provider accepted it, to this card?
told INTEGER NOT NULL DEFAULT 0 -- has the caller heard a reply that names its card?
)
"""
It is tempting to key transactions by the tool call's call_id because each call generates a unique identifier. However, that design is precisely why duplicate payouts happen: if the model decides to invoke the tool a second time, it generates a fresh call_id, and a naive insert creates a duplicate record. A refund is an action on an order, meaning the order must serve as the primary key. We still record session_id and all associated call_ids as an audit trail, while status, accepted, and told track provider state and user comprehension.
Warning: Don't key a side effect on
call_id. It identifies the request, not the thing being done, so a repeat of the same request looks new.
Writing to this ledger is handled by a single transaction method, Ledger.refund():
def refund(self, order_id: str, card_last4: str, session_id: str, call_id: str) -> tuple[str, dict]:
"""Record a refund for an order. Returns what happened, and the row as it now stands."""
amount = ORDERS[order_id]["amount_cents"]
self.db.execute("BEGIN IMMEDIATE") # the check and the write happen under one lock
try:
added = self.db.execute(
"INSERT INTO refunds (order_id, refund_id, amount_cents, card_last4, session_id, call_ids) "
"VALUES (?, ?, ?, ?, ?, ?) ON CONFLICT (order_id) DO NOTHING",
(order_id, f"R-{order_id}", amount, card_last4, session_id, json.dumps([call_id])),
).rowcount
if added:
outcome = "refunded"
else:
row = self.row(order_id)
self.db.execute("UPDATE refunds SET call_ids = ? WHERE order_id = ?",
(json.dumps(json.loads(row["call_ids"]) + [call_id]), order_id))
if row["card_last4"] == card_last4:
outcome = "already_refunded"
elif row["status"] == "pending":
self.db.execute("UPDATE refunds SET card_last4 = ?, accepted = 0, told = 0 WHERE order_id = ?",
(card_last4, order_id))
outcome = "card_changed"
else:
outcome = "settled" # the money has gone: moving it is a new refund and a reversal, not an update
self.db.execute("COMMIT")
except BaseException:
self.db.execute("ROLLBACK")
raise
return outcome, self.row(order_id)
The write uses SQLite's ON CONFLICT (order_id) DO NOTHING, so the database itself decides whether this is the first refund for the order. Evaluating rowcount reveals whether a new row was added or an existing record caught the conflict. Relying on an in-memory dictionary or simple checks invites race conditions the moment two tasks execute concurrently. Wrapping the block in BEGIN IMMEDIATE guarantees that the conflict check, read, and subsequent update occur under the same write lock, and a test in tests/test_ledger.py races two connections at one order to prove it.
When an order row already exists, the method determines the appropriate business response. A repeat request on the same card is marked already_refunded. A switch to a different card while the status is still pending updates the target card in place and resets both accepted and told, because both the provider and the customer need to hear about the new card. If the refund has already cleared, the method returns settled, as modifying settled funds requires an explicit reversal and new transaction. Notice also that the refund amount is sourced directly from internal store records (ORDERS), never from model parameters.
Step 2: Answer a repeat call from the ledger
Because the language model only understands tool outcomes through returned values, our tool response must state the ledger's ground truth directly.
The tool definition is declared in agent.py:
TOOLS = [{
"type": "function",
"name": "issue_refund",
"description": "Refund an order in full to one of the caller's cards.",
"parameters": {
"type": "object",
"properties": {
"order_id": {"type": "string", "description": "The order number, like 4417."},
"card_last4": {"type": "string", "description": "The last four digits of the card to refund to."},
"say_while_working": {
"type": "string",
"poly_spoken": True, # the server speaks this argument while the tool runs
"description": "A sentence to say while the refund goes through, like 'Let me process that refund for you now.'",
},
},
"required": ["order_id", "card_last4", "say_while_working"],
},
}]
The say_while_working parameter is tagged with "poly_spoken": True, instructing the server to speak this phrase aloud while the tool runs in the background. We make this field required on purpose. As highlighted in the Dialog-RSN-1 overview, models tend to omit optional spoken arguments, which leaves callers waiting in awkward silence while the external payment gateway completes its work.
The execution handler validates the inputs, consults the ledger, and formats the response:
async def issue_refund(self, args: dict, call_id: str) -> dict:
order_id, card = str(args.get("order_id", "")).strip(), str(args.get("card_last4", "")).strip()
if order_id not in ORDERS:
return {"status": "error", "note": f"There's no order {order_id} for this caller. Ask them to check the number."}
if card not in CARDS:
return {"status": "error", "note": f"The caller has no card ending {card}. Ask which card they want."}
outcome, row = self.ledger.refund(order_id, card, self.session_id, call_id)
self.log("", f"(issue_refund {order_id} to {card}: {outcome})")
if not row["accepted"]: # new, a new card, or a repeat of one the provider never accepted
await submit_refund(row)
self.ledger.mark_accepted(order_id)
return {"status": outcome, "order_id": order_id, "refund_id": row["refund_id"], "note": self.note_for(outcome, row)}
Any transaction that has not yet been accepted (whether it is brand new, transferred to a replacement card, or retried after the provider failed the first time) is forwarded to the payment gateway through submit_refund(), mocked here as a four-second delay. Subsequent calls for an already accepted refund return immediately. In production, each request includes provider_key(row) as an idempotency key so the payment provider can reject duplicates on its end.
The model learns what to say next via note_for():
def note_for(self, outcome: str, row: dict) -> str:
"""What the model is told, worded from the ledger's row, so its next line says what actually happened."""
if outcome == "refunded":
return f"Refunded {describe(row)}. Tell the caller it's done."
if outcome == "already_refunded":
when = "earlier in this call" if row["session_id"] == self.session_id else "on an earlier call"
return f"This refund was already issued {when}: {describe(row)}. Tell the caller it's done; don't refund again."
if outcome == "card_changed":
return f"Still one refund, now {describe(row)} instead. Tell the caller."
return f"This refund already settled, {describe(row)}, and can't be moved. Tell the caller a person will call back."
The helper function describe(row) converts the database entry into natural phrases like "32 dollars to the Visa ending 4242," anchoring the agent's statements directly to the database state. If an order was previously refunded, the note indicates whether the event occurred in the current session or a previous call, providing clear context when a customer calls back.
Across 10 test calls where callers asked "Sorry, did that go through?", the model never triggered issue_refund a second time. However, when the caller interrupted with "Wait, wrong card. Can you put it on my Mastercard instead?", the model reliably invoked issue_refund with the new details in 5 out of 5 runs. While a naive client would have charged the customer twice, our ledger flagged the transition as card_changed, kept a single database row, and prompted the agent to confirm: "That's done. I've put the 32 dollars back to your Mastercard ending 8812."
Step 3: Confirm once the result is in
Now let us revisit the silence bug from our opening example.
In the initial implementation, the client skipped sending response.create after receiving a tool result if the caller had spoken in the meantime. The reasoning seemed sound: avoid talking over a caller who has already moved on to a new question. While that works for read-only database queries, it is disastrous for mutations. When the caller asked "did that go through?", the model answered "It's still processing." When the gateway finally finished, the client saw that the user had spoken, suppressed the follow-up response, and never closed the conversational loop.
To fix this, the client must trigger a reply whenever no new turn has started since the tool result entered history. The event handler tracks response counters, records which responses contained tool calls, and logs mapping identifiers:
async def handle(self, event: dict):
match event["type"]:
case "response.created":
self.responses += 1
...
case "response.function_call_arguments.done":
self.with_calls.add(self.responses)
self.spawn(self.run_tool(event)) # the call is final: start the refund before the holding line plays
await super().handle(event)
The tool executes inside a dedicated task started as soon as argument streaming completes. When finished, run_tool() appends the outcome to the history and coordinates the response:
async def run_tool(self, call: dict):
try:
result = await self.issue_refund(json.loads(call["arguments"]), call["call_id"])
except Exception as error: # every call gets an output, or the model waits for it forever
self.log("", f"(issue_refund failed: {error})")
result = {"status": "error", "note": "The refund system failed, so the refund isn't confirmed. Don't call "
"issue_refund again. Tell the caller the team will email them once it's sorted."}
since = self.responses
await self.send({
"type": "conversation.item.create",
"item": {"type": "function_call_output", "call_id": call["call_id"], "output": json.dumps(result)},
})
if result.get("refund_id"):
self.owed.setdefault(result["order_id"], since) # from the first result the caller hasn't heard
self.reminded.pop(result["order_id"], None)
await self.ask_for_reply(since)
async def ask_for_reply(self, since: int):
"""Ask the model to speak about the result, unless a reply has already started since it went into history."""
while self.responding: # one response at a time: let the holding line, or the answer to the caller, finish
await asyncio.sleep(0.05)
if self.responses == since:
await self.send({"type": "response.create"}) # sent while the caller talks, it's held for their turn
A tool output event alone does not trigger speech. Appending function_call_output merely updates the conversation log; an explicit response.create event is required to prompt an agent response.
The since variable captures the response count at the exact moment the output was recorded, and owed preserves this marker until the caller hears the result. The ask_for_reply() method waits for any active turn to conclude (such as the holding line or the answer to an interruption), avoiding conversation_already_has_active_response errors. If no newer turn has begun, the client issues response.create. If the caller is speaking when that event arrives, the API reference says the server holds it and answers their turn with a single reply, which is what happened in all 5 mm-hmm-gap runs on 10 November.
Running the original problem scenario through the updated client demonstrates the fix:
...
Caller: Sorry, did that go through?
Agent: It's still processing. I'll let you know as soon as it goes through.
Agent: Yes, it's done. I've refunded 32 dollars back to your Visa ending 4242.
refund_id order amount card status told calls
R-4417 4417 32.00 4242 pending yes 1
The caller heard the confirmation in 5 out of 5 runs, compared to 0 out of 10 on the older client.
Handling exceptions inside run_tool() protects this flow. Every tool invocation receives an explicit payload so the model is never left waiting indefinitely. In a run on 10 November with the payment provider down, the caller was promptly told: "The refund system failed, so it's not confirmed yet. Our team will email you once it's sorted." The row remained stored with accepted and told set to zero, an actionable record for support staff, and if the caller asks again, step 2's check submits it again.
Step 4: Tell the caller what the ledger says
Just because the server sent an audio stream does not mean the user actually heard it. An interruption can cut a sentence off halfway through.
To bridge this gap, the client sets the ledger's told flag only when a reply the caller actually heard names the refunded card:
async def heard(self, item_id: str, text: str, whole: bool):
await super().heard(item_id, text, whole)
for order_id, since in list(self.owed.items()):
n = self.response_of.get(item_id, 0)
if n <= since or n in self.with_calls:
continue # a holding line, or a reply that started before the result was in history
row = self.ledger.row(order_id)
if mentions(text, row["card_last4"]):
self.ledger.mark_told(order_id)
del self.owed[order_id]
elif self.reminded.get(order_id, 0) < 2: # twice at most, so a model that won't say it can't loop
self.spawn(self.remind(row))
The heard() hook fires after every spoken reply, receiving the exact text delivered to the caller: either the full sentence or the truncated fragment captured when an interruption occurred. Holding lines and earlier responses are filtered out, and mentions() confirms the specific last four card digits made it across the wire.
If an interruption truncates the confirmation, remind() puts the ledger's word into history and asks for a reply:
async def remind(self, row: dict):
"""The refund is real, but the caller didn't hear it. Put the ledger's word in history and ask for a reply."""
self.reminded[row["order_id"]] = self.reminded.get(row["order_id"], 0) + 1
note = (f"Refund {row['refund_id']} is done: {describe(row)}. The caller hasn't heard that yet. In your next "
"reply, tell them it's done, with the amount and the card's last four digits.")
self.log("", f"(reminder: {row['refund_id']} not heard yet)")
since = self.responses
await self.send({"type": "conversation.item.create",
"item": {"type": "message", "role": "system", "content": [{"type": "input_text", "text": note}]}})
await self.ask_for_reply(since)
In early iterations, I tried injecting this reminder purely as a passive role: system message. When a caller interrupted the confirmation, the system message landed in the history before the agent replied to the caller's new turn. Across 10 out of 10 runs, the model answered the caller's immediate question and completely ignored the passive instruction to confirm the refund.
What proved reliable was requesting a dedicated response once that turn finished. For instance, when a caller interrupted 1.2 seconds into the confirmation with "Okay, thanks. How long does it take to show up?", the system handled the conversational detour before circling back:
Agent: I've refunded 32 dollars back to your Visa- [cut off]
(reminder: R-4417 not heard yet)
Caller: Okay, thanks. How long does it take to show up?
Agent: I don't have information about that. Would you like me to transfer you to someone who can tell you how long it takes?
(reminder: R-4417 not heard yet)
Agent: It's done, 32 dollars back to your Visa ending 4242. I don't have information about how long it takes to show up. Would you like me to transfer you to someone who can tell you?
That second response successfully confirmed the refund in 5 out of 5 runs. Generating the reminder directly from ledger data avoids the ambiguity of truncated transcript text, which often lacks both the card digits and confirmation status.
Step 5: Test the ledger against what the caller heard
Every conversational failure we have examined boils down to a discrepancy between two sources of truth: the database records and the audio that reached the caller's ear.
We can capture that invariant in a single evaluation function inside ledger.py:
def mismatches(rows: list[dict], heard: list[str]) -> list[str]:
"""Where the ledger and the caller disagree. heard is every reply the caller heard, holding lines left out."""
problems = [f"order {order} refunded {n} times" for order, n in Counter(r["order_id"] for r in rows).items() if n > 1]
for row in rows:
if not any(mentions(reply, row["card_last4"]) for reply in heard):
problems.append(f"{row['refund_id']} is in the ledger, but the caller never heard it was done")
elif not row["told"]:
problems.append(f"{row['refund_id']} was heard, but the ledger says it wasn't")
confirmations = [reply for reply in heard if any(mentions(reply, card) for card in CARDS)]
if confirmations and not any(mentions(confirmations[-1], row["card_last4"]) for row in rows):
problems.append(f"the caller last heard {confirmations[-1]!r}, but no refund in the ledger goes to that card")
return problems
An empty list signifies a fully reconciled call. The assertions catch duplicate payouts, unconfirmed transactions, and mismatched card details. The heard log originates from the test harness, which discards intermediate holding lines so phrases like "let me refund your Visa ending 4242" spoken before execution cannot masquerade as completed confirmations.
Offline integration tests simulate calls against a mock server in tests/fake_realtime.py. The mock synchronizes against caller audio duration, streaming responses at normal conversational cadences while allowing test suites to run ten times faster than real time.
Here is how we verify idempotent re-issuance in tests/test_refunds.py:
async def test_a_reissued_call_is_the_same_refund(monkeypatch, tmp_path):
steps = [("tool", "issue_refund", VISA, LINE, 2000), ("tool", "issue_refund", VISA, "Let me check on that.", 1500),
("say", DONE, 2500)]
async with FakeRealtime(steps, ["I need a refund on order 4417.", "Sorry, did that go through?"]) as fake:
room, ledger = await call(fake, monkeypatch, tmp_path, "did-it-go-through")
outputs = [json.loads(e["item"]["output"]) for e in fake.sent("conversation.item.create")
if e["item"]["type"] == "function_call_output"]
assert sorted(o["status"] for o in outputs) == ["already_refunded", "refunded"] # one refund, and one repeat
assert len(ledger.rows()) == 1
assert_ledger_matches_heard(room, ledger)
The steps script a repeat call that the real model didn't make on 10 November, because a test that only covers what one model did on one day isn't much of a test. Comparing it against a client keyed naively on call_id shows why conversational testing alone is insufficient:
uv run pytest tests/test_refunds.py -k "reissued or keyed_on" -s -v
...
tests/test_refunds.py::test_a_reissued_call_is_the_same_refund (caller says refund.wav)
Caller: I need a refund on order 4417.
(issue_refund 4417 to 4242: refunded)
(caller says did-it-go-through.wav)
(barge-in: stopped the reply after 1.7 s)
Agent: Let me process that refund for yo- [cut off]
Caller: Sorry, did that go through?
(issue_refund 4417 to 4242: already_refunded)
Agent: Let me check on that.
Agent: That's done: 32 dollars back to your Visa ending 4242.
PASSED
tests/test_refunds.py::test_a_ledger_keyed_on_call_id_pays_out_twice (caller says refund.wav)
Caller: (caller)
(issue_refund 4417 to 4242: refunded)
(caller says did-it-go-through.wav)
(barge-in: stopped the reply after 1.7 s)
Agent: Let me process that refund for yo- [cut off]
Caller: (caller)
(issue_refund 4417 to 4242: refunded)
Agent: Let me check on that.
Agent: That's done: 32 dollars back to your Visa ending 4242.
PASSED
In both runs, the spoken dialogue sounds completely natural. However, on the call-keyed implementation, mismatches() identifies ["order 4417 refunded 2 times"]. Listening to the audio alone would never reveal the duplicate charge.
Our test suite applies this reconciliation across multiple edge cases, covering mid-flight questions, ignored reminders, card changes, and upstream payment errors.
Run it
You can execute specific interaction scripts against a live server using the scripted caller:
uv run caller.py --script did-it-go-through
The script streams the interaction in real time and prints the resulting ledger status upon completion. Use --script to test different conversational branches: mm-hmm and tv play a backchannel and a newsreader under the holding line, wrong-card and did-it-go-through test direct interruptions, mm-hmm-gap says "mm-hmm" just after the line while the refund is still running, and missed-confirmation talks over the final confirmation. Passing --before runs these scenarios against the legacy client for direct comparison.
To benchmark consistency over time, measure.py executes batches against isolated ledgers:
uv run measure.py --script did-it-go-through --runs 5 --audit
did-it-go-through, 5 runs on api.studio.poly.ai
...
Caller: Sorry, did that go through?
Agent: It's still processing. I'll let you know as soon as it goes through.
Agent: Yes, it's done. I've refunded 32 dollars back to your Visa ending 4242.
RUN 5: {"speech_started": 2, "speech_stopped": 2, "cut_after_s": 0.74, ... "calls_in_history": ["function_call"], "rows": [["4417", "4242", "told"]], "mismatches": [], ...}
5 runs: more than one call 0, refunded twice 0, no refund 0, caller never told 0, any mismatch 0
reply cut a median 0.76 s after the caller started (0.74 to 0.79, 5 runs)
The --audit flag queries the server using conversation.item.retrieve to verify that tool invocations persist in remote history.
Here are the aggregated results from test runs conducted on api.studio.poly.ai on 10 November 2026:
| Script | Runs | What happened | Caller never told |
|---|---|---|---|
mm-hmm, 0.8 s into the holding line |
10 | Nothing: no speech_started, the line played on |
0 |
mm-hmm-confirmation, 1.2 s into the confirmation |
10 | Nothing: the confirmation played to the end | 0 |
mm-hmm-gap, after the line, refund still running |
5 | A turn ("Mm-hmm."), answered by the confirmation | 0 |
tv, 0.8 s into the holding line |
10 | Not a barge-in, but transcribed as the caller's turn every time | 0 |
wrong-card, 0.5 s into the line |
5 | Line cut, a second issue_refund to 8812, one row |
0 |
did-it-go-through, 0.8 s into the line |
10 | Line cut, "still processing", no second call | 10 |
did-it-go-through, fixed client |
5 | "still processing", then "Yes, it's done" | 0 |
missed-confirmation |
5 | Confirmation cut, the reminder ignored | 5 |
missed-confirmation, fixed client |
5 | The second ask told them | 0 |
did-it-go-through --before |
5 | As above, and one row: the model never asked twice | 5 |
mm-hmm-gap --before |
5 | As above | 0 |
Not a single run resulted in a duplicate refund. On the legacy client running did-it-go-through, the holding line was cut a median 0.76 seconds after the caller started talking, timed on the client, so that includes a network round trip.
To test the system yourself with a live microphone, start the interactive service:
uv run agent.py
Ask for a refund on order 4417, interrupt the agent while it works to ask if the transaction went through, and press Ctrl+C to inspect the final ledger summary.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| The agent says "still processing" and never confirms |
response.create skipped because the caller spoke while the tool ran |
Ask unless a reply has started since the result (step 3) |
| The caller talked over the confirmation and never heard it again | A reminder note with no response.create after it |
Ask for a reply once the caller's turn has been answered (step 4) |
Two rows for one order, with different call_ids |
The ledger is keyed on call_id
|
Key on the order and keep call_ids as a column |
sqlite3.IntegrityError: UNIQUE constraint failed: refunds.order_id |
A plain INSERT on the keyed table |
INSERT ... ON CONFLICT (order_id) DO NOTHING, then read the row |
An error event with code conversation_already_has_active_response
|
response.create sent while a response was still open |
Wait for response.done before sending it |
The server refused the connection (HTTP 401). |
Wrong key, or a POLY_API_KEY in your shell beats .env
|
echo $POLY_API_KEY, then unset POLY_API_KEY
|
Before you ship
While this implementation uses a simulated four-second sleep to represent the payment provider, a production deployment should pass provider_key(row) (a composite of the refund ID and card details) directly to the gateway as an idempotency key. This ensures payment providers like Stripe deduplicate repeated requests upstream. Stripe's v1 API (consulted 10 November 2026) replays the first response for a key, raises an error if the same key comes back with different parameters, and can prune keys once they're 24 hours old. Because of those limitations, your database ledger remains your primary source of truth, with provider keys serving as secondary protection. Writing the ledger entry before initiating network calls guarantees that an unexpected crash leaves an actionable record for reconciliation rather than untracked money movement. A card change that lands while the first submission is still with the provider needs that first request cancelled, which the stand-in doesn't model.
Our example assumes full refunds keyed strictly by order ID. If your store supports partial refunds, expand the key to include line item identifiers or amount markers to prevent distinct partial claims from blocking one another. High-value transactions should similarly pass through an explicit human review step before hitting the payment gateway, which fits naturally into the ledger's pending state.
Any order marked as untold when a caller hangs up should automatically queue an asynchronous confirmation via SMS or email. During our testing, background television dialogue was transcribed as direct caller input in all 10 runs; while our agent handled it gracefully, tools that trigger destructive actions should always prompt for explicit voice confirmation before running. Finally, have the agent say it's an AI at the beginning of the call, and if you record calls, tell the caller and, where the law requires it, get their consent.
Where to go next
You now have a refund desk where the database ledger is the source of truth: orders refund once regardless of model repetition, card switches update pending transactions safely, and confirmations are delivered even when callers interrupt mid-flight.
To expand this setup, try introducing an order cancellation tool like cancel_order with its own idempotency constraints within the same SQLite database. For deeper dives into real-time voice architectures, consult the Dialog-RSN-1 overview on turn-taking mechanics, along with the API reference on managing barge-in events and response lifecycles. All source code is available in the pmckeetx/refund-voice-agent repository, and you can obtain an API key at studio.poly.ai.


Top comments (0)