AI agents were shipping untested.
Teams building AI agents on FutureAGI had no systematic way to test them before going live. They ran a few manual chats, declared the agent "tested," and shipped — hoping production traffic wouldn't surface something they'd missed.
It always did. Edge cases hit users first. By the time a failure was caught, it had already happened in production, and there was no way to reproduce the exact conditions that caused it.
Three gaps defined the problem space:
- ·Only the happy path was tested. Teams validated the scenario where everything worked. The moments where things went wrong — ambiguous input, unexpected tool calls, mid-conversation failures — never got tested.
- ·Only the final answer was graded. If the agent's last response was correct, it passed. Nobody was looking at what happened in between.
- ·Failures hit production before they were caught. There was no pre-production environment for agent behaviour, only real users experiencing real failures.
four things we learned from the research.
Sessions with ML engineers, PMs, and QA teams who built and maintained agents revealed four patterns that shaped every design decision that followed.
An agent makes tool calls, branches on intermediate outputs, and handles multi-turn logic. Testing it like a chatbot — one input, one output — misses 90% of what can go wrong.
The final answer could be correct even when the reasoning path was broken. Grading only the output let entire categories of failure go undetected.
When something went wrong in production, teams had no way to replay the exact conversation state that caused it — every debug attempt was starting from scratch.
The bar for "done testing" was three or four conversations where the agent happened to work. Nobody called this inadequate — it was just how it was done.
The insight that unlocked the design: testing an agent isn't testing a response — it's testing a decision tree. The tool had to reflect that.
what the market already offered.
Three tools had the clearest overlap with what we were building. Each had something worth learning from — and a gap that confirmed our direction.
Strongest in the category. Scenario-based testing with structured evaluation.
No clear path from a detected failure to a filed bug. The loop ended at the result.
Good evaluation tooling and metric depth for output quality.
No simulation layer — testing was still manual prompt-response, not scenario-driven.
Strong tracing and logging — great for post-hoc analysis of production failures.
Observability-first, not simulation-first. Catches failures after they've happened.
The gap none of them closed: a failure → diagnosis → fix loop that didn't require engineering involvement at every step.
scenarios as a graph, not a flat list.
The most important design decision came early: how should a scenario be structured?
The obvious answer was a flat list of test cases. Simple, familiar, easy to build. But a flat list assumes every test starts from the same place — and agent behaviour doesn't work that way. Real conversations branch. An ambiguous user input in step 2 changes everything that happens in steps 3 through 7.
A flat list of scenarios treats an agent like a function. A scenario graph treats it like what it actually is — a decision tree with branching paths and compounding failures.
We chose the scenario graph as the core object. Each node is a conversation state. Edges represent branches — what the agent does next, depending on what happened before. This let teams model the full shape of an agent's behaviour, not just a checklist of expected inputs and outputs.
The second big call: global nodes. Some scenarios need to be reachable from anywhere in a conversation — error handling, unexpected input, emergency exits. Rather than duplicating them in every branch, global nodes could be defined once and referenced everywhere.
five stages. one continuous cycle.
The simulation product wasn't a single screen or tool — it was a full workflow. Every stage fed the next, and the whole loop was designed to be repeatable without engineering involvement.
five screens, the full simulation experience.
Screen · 01agent definition — two-step modal, one clear mental model
Setting up an agent for simulation required two distinct types of information: what the agent is (basic info) and how it runs (configuration). These were split into a two-step modal to reduce cognitive overload on a first-time setup.
- ·Step 1 · Basic Info: agent name, description, and the system prompt that defines its behaviour.
- ·Step 2 · Configuration: voice provider selection — ElevenLabs, Retell AI, Vapi, or Custom — with provider-specific fields revealed dynamically.
scenario creation — upload or build, persona library always on hand
Teams could either upload existing scenario files or build from scratch. The persona library — with controls for gender, age, accent, and personality type — was accessible at every point in scenario creation so teams could assign the right synthetic user to each branch.
review flow — the graph before you run it
Before launching a simulation, teams saw a full-page review: the scenario graph visualised as a flow, and a table mapping each persona to the situation it would encounter. This screen served a critical function — it made implicit assumptions explicit before they ran 200 synthetic conversations on a broken setup.
run simulation — named, tracked, repeatable
Every simulation run was given a name and linked to a saved evaluation set. That made runs referenceable — not just "the run we did on Tuesday" but a named, versioned artifact you could return to, compare against, and discuss with the team.
results & eval library — from score to fix in one view
Results surfaced a distribution bar showing how runs performed across the full score range — not just an average that hid the outliers. The eval library organised metrics into Chat, Audio, and Quality categories so teams only saw what was relevant to their agent type.
- ·Distribution bar: shows the shape of results across all runs, making outliers visible without burying them in a mean.
- ·Fix my agent CTA: from any failed run, one click surfaces the exact conversation state that caused the failure and jumps directly to the relevant agent config.
shipped to beta. working as intended.
The simulation product launched to three internal teams in beta. The early signals confirmed the core design decisions held up under real use.
what designing for AI taught me about design.
- —The hardest part wasn't the UI — it was the mental model. Before a single screen could be designed, there had to be agreement on what a "scenario" actually was. Getting to "a scenario is a graph, not a list" took three rounds of working sessions with the ML team. The design only worked because the model did.
- —Designing for failure is different from designing for success. Most product design is about making the happy path fast and clear. Simulation design is the opposite — every surface has to make failure legible, reproducible, and actionable. That requires different instincts.
- —The "Fix my agent" CTA was the most-loved feature in beta — and it wasn't in the original brief. It came from watching a PM try to action a failed result and realising there was no clear next step. Sometimes the most impactful design work is noticing what's missing from the spec.
- —Non-technical users were the unlock. The brief was for ML engineers. But the beta success came from PMs and QA running simulations independently — because the graph visualisation made the agent's behaviour understandable without reading the code. Designing for the person who doesn't have the context is harder, and more valuable.