Answering in under a breath: voice AI agents for a Mexican e-commerce marketplace

The situation

A marketplace in Mexico sold electronics, home goods and fashion from thousands of sellers, with deliveries handled by several carriers. Its contact centre took calls in Spanish, and most of them were about the same few things: where an order was, when it would arrive, how to change the address, how to start a return. During sales seasons, queues grew long enough that callers hung up and tried again, which made the queues longer.

Why now. On 29 October 2026, Deepgram announced it is acquiring Daily, the company behind Pipecat, a widely used open-source framework for real-time voice agents¹. Both companies said Pipecat will stay under its BSD licence and move to an independent foundation whose steering committee includes Deepgram's competitors². For a team choosing a voice agent stack, that settled the main worry about building on an open framework. On the legal side, Mexico's new personal data protection law took effect in March 2025³⁴, and legal commentary notes a right to object to automated processing that significantly affects people without human involvement³. Callers would always need a route to a person.

What had to be true

• Natural Mexican Spanish, including regional accents, background noise and code-switching with English product names.
• Conversational speed: replies quick enough that silence never feels like a dropped call.
• Grounded answers only: order status from the systems, never from the model's imagination.
• A person on request, with the conversation so far passed across.
• Clear data rules for recordings and transcripts, and no lock-in to one speech vendor.

Architecture

The data flow, end to end

The hard part: latency you can feel

In text chat, a two-second delay is fine. On a phone call it feels like the line has dropped. The delay a caller hears is the sum of several stages: detecting that they stopped speaking, finishing speech recognition, the model's first tokens, any tool calls, and the first audio from speech synthesis. Each stage looked acceptable alone; together they were too slow.

The team set a latency budget per stage and traced every call end to end, so each turn showed exactly where the time went, in the same way distributed tracing shows which service in a request is slow⁵. That made the fixes obvious: stream everything instead of waiting for whole sentences; prefetch the caller's recent orders as soon as they are verified, so the first tool call is usually instant; use a smaller, faster model for routine turns; and play a short natural acknowledgement while a slow carrier API responds.

Turn-taking was the other half. Callers interrupt, pause mid-sentence and talk over the agent. The framework's interruption handling let the agent stop speaking when the caller started, and tuning silence detection for Spanish speech patterns cut the most awkward moments. See model routing and prompt caching for latency.

How it ran

Key decisions and trade-offs

What I'd do again, and differently

Again: build the evaluation set from real call recordings before choosing providers. Speech models that tested well on clean audio struggled with real calls. Differently: trace latency per stage from the first prototype, not after the first complaints. Related: evaluating AI agents, an agent-ready travel marketplace, today's tech news and where I work.

If you are building voice AI for customers in Mexico or elsewhere in Latin America, see my AI architecture work, or analyse your challenge and bring the result to a call.