Testing Philosophy

If you’re here reading this, you’ve probably faced the unpredictable behavior of voice agents. A conversation goes well during testing, then a real caller asks the same thing differently and the agent struggles. Voice agents are different from regular (text-based) agents as there is limited time for reasoning before a pause starts to feel awkward. The agent also needs to handle the parts of a conversation that words alone don’t capture, like tone, timing, background noise, interruptions, and sometimes even a laugh. Getting an agent to feel natural is hard to pin down. It might answer correctly but pause too long, or sound friendly while missing what the caller asked. “Feels human” is a moving goalpost. To test it, you need to describe the specific behaviors that matter to your users. The possible scenarios can run into thousands, and some problems will only appear in production. You can still catch many issues with a useful set of tests. Much of our approach to testing voice agents comes from Anthropic’s Demystifying evals for AI agents. When we were building tests for our own agents, it helped us put clear names to the pieces and understand how they fit together. We highly recommend reading it! You’ll recognize several of the ideas below, adapted for voice agents.

​

Why we call them tests

We use the word tests because it gives us a familiar starting point from tools like pytest and JUnit: put a system in a situation, observe what it does, and compare that with what you expected. In the broader AI world, you’ll usually see this called an eval, but they mean the same thing for us. The Anthropic blog also uses them interchangeably.

​

Anatomy of a test

The pytest guide to the anatomy of a test describes four steps: arrange, act, assert, and cleanup. The same structure helps explain a voice test. Prepare the situation, have the conversation, judge the evidence, and reset any test data that could affect the next attempt. In Egma, the scenario and persona shape the simulated caller. Together, “change an appointment” and “a frustrated caller who speaks Spanish” give the simulator a goal and a way to approach the conversation. It talks to your agent through the selected connection. Egma records evidence from the conversation, including the transcript and available tool calls and metrics. Graders use the evidence they receive to assess whether the agent met your criteria. For the appointment test, you might expect the agent to explain that the requested time is unavailable, offer an available alternative, and ask for confirmation before booking. Several different conversations could meet those expectations. You usually want to check the behavior without requiring the agent to repeat an exact sentence.

​

From a test to a run

To start a run, choose a test suite, an agent, and one of that agent’s connections. Each test in the suite already selects one or more personas. Egma runs each test once for every selected persona, producing a separate simulation for each pair. For example, a suite with three tests and two personas per test produces six simulations. They all use the same agent and connection. Graders assess the evidence from each simulation.

​

Definitions

​

Regression and hill-climbing suites

We think about tests in two groups:

• Regression suites cover behavior that already works. Run them after changes to see whether you’ve broken something your users depend on.
• Hill-climbing suites cover behavior you want to improve. You expect some failures today and use the results to track progress with each release.

These describe how you use your suites. Both are ordinary test suites in Egma.

​

Start with an everyday caller

We recommend using the write-egma-tests skill to draft tests from your agent’s prompts and tools. Start with an everyday caller persona while you cover the main situations your agent needs to handle. Review the generated tests to make sure their expectations match your product. Add more personas when you want to explore differences in language or caller behavior. As you add flows and capabilities, add tests for them too. Turn failures from production into regression tests so you can check that the same problem stays fixed.