AI agents were shipping untested.
Teams building AI agents on FutureAGI had no systematic way to test them before going live. They ran a few manual chats, declared the agent "tested," and shipped — hoping production traffic wouldn't surface something they'd missed.
It always did. Edge cases hit users first. By the time a failure was caught, it had already happened in production, and there was no way to reproduce the exact conditions that caused it.
Three gaps defined the problem space:
- ·Only the happy path was tested. Teams validated the scenario where everything worked. The moments where things went wrong — ambiguous input, unexpected tool calls, mid-conversation failures — never got tested.
- ·Only the final answer was graded. If the agent's last response was correct, it passed. Nobody was looking at what happened in between.
- ·Failures hit production before they were caught. There was no pre-production environment for agent behaviour, only real users experiencing real failures.
four things we learned from the research.
Sessions with ML engineers, PMs, and QA teams who built and maintained agents revealed four patterns that shaped every design decision that followed.
An agent makes tool calls, branches on intermediate outputs, and handles multi-turn logic. Testing it like a chatbot — one input, one output — misses 90% of what can go wrong.
The final answer could be correct even when the reasoning path was broken. Grading only the output let entire categories of failure go undetected.
When something went wrong in production, teams had no way to replay the exact conversation state that caused it — every debug attempt was starting from scratch.
The bar for "done testing" was three or four conversations where the agent happened to work. Nobody called this inadequate — it was just how it was done.
The insight that unlocked the design: testing an agent isn't testing a response — it's testing a decision tree. The tool had to reflect that.
five tools. all missing the same thing.
We audited five products with the closest overlap to what we were building. Each solved a real problem — but every one of them stopped at the boundary where pre-production simulation should have started.
Routes LLM calls across providers, handles caching and load balancing, logs production traffic with cost tracking.
Infrastructure-layer tool. Tells you what happened in production — not what will happen before you ship. No simulation, no synthetic conversations, no pre-deployment testing.
Side-by-side prompt comparison, parameter tuning, model switching. Excellent for iterating on a single prompt quickly.
Single-turn, single-prompt focus. No way to test multi-turn agent behaviour, branching logic, or realistic customer conversations at scale.
Strong tracing and logging for post-hoc analysis. Dataset management and eval scoring for captured production data.
Observability-first, not simulation-first. Catches failures after they've hit users — not before. No synthetic persona system, no pre-production test runs.
Version control for prompts, A/B testing across prompt versions, usage analytics and cost monitoring across deployments.
Prompt management, not agent testing. Tracks what changed between versions but can't simulate how an agent behaves in a real conversation once deployed.
Hallucination detection, RAG quality evaluation, output scoring across safety and quality dimensions for production traffic.
Evaluation-first, production-focused. Monitors quality of outputs that have already been generated — no simulation layer for testing agent behaviour before deployment.
Every tool in this space answers the question "what happened?" — after deployment, after failures, after users were affected. FutureAGI Simulation answers a different question: "what will happen?" — before a single real call goes live. Synthetic customers, realistic scenarios, pre-production stress testing, and a failure → diagnosis → fix loop that non-technical users can run without writing a line of code.
scenarios as a graph, not a flat list.
The most important design decision came early: how should a scenario be structured?
The obvious answer was a flat list of test cases. Simple, familiar, easy to build. But a flat list assumes every test starts from the same place — and agent behaviour doesn't work that way. Real conversations branch. An ambiguous user input in step 2 changes everything that happens in steps 3 through 7.
A flat list of scenarios treats an agent like a function. A scenario graph treats it like what it actually is — a decision tree with branching paths and compounding failures.
We chose the scenario graph as the core object. Each node is a conversation state. Edges represent branches — what the agent does next, depending on what happened before. This let teams model the full shape of an agent's behaviour, not just a checklist of expected inputs and outputs.
The second big call: global nodes. Some scenarios need to be reachable from anywhere in a conversation — error handling, unexpected input, emergency exits. Rather than duplicating them in every branch, global nodes could be defined once and referenced everywhere.
five stages. one continuous cycle.
The simulation product wasn't a single screen or tool — it was a full workflow. Every stage fed the next, and the whole loop was designed to be repeatable without engineering involvement.
three rounds, five screens.
The design went through three distinct rounds before the beta build. Each round surfaced something the previous one couldn't — from structure to legibility to the failure-to-fix handoff. What follows is where those rounds landed: the five screens that made up the full simulation experience.
What we tried: Flat list of test cases as the primary scenario structure — one input, one expected output per row. Familiar format borrowed from unit testing.
Why it changed: Teams couldn't model branching conversations. A single ambiguous input in step 2 changed everything after — the list couldn't capture that. Moved to a scenario graph.
What we tried: Scenario graph as a standalone visualisation, with personas selected separately before running. Two-step flow: build graph, then pick who runs it.
Why it changed: Users lost the connection between who the persona was and how the graph read. Merged persona context into the run configuration so the agent's "audience" was always visible alongside the scenarios.
What we tried: Eval results page showed pass/fail scores with trace logs accessible via a side panel. Users could read what failed, then manually navigate back to agent config to fix it.
Why it changed: The gap between diagnosis and action was too wide. Added the "Fix my agent" CTA — one click from a failed run to the exact config field that caused the failure. Became the most-loved feature in beta.
agent definition — two-step modal, one clear mental model
Setting up an agent for simulation required two distinct types of information: what the agent is (basic info) and how it runs (configuration). These were split into a two-step modal to reduce cognitive overload on a first-time setup.
- ·Step 1 · Basic Info: agent name, description, and the system prompt that defines its behaviour.
- ·Step 2 · Configuration: voice provider selection — ElevenLabs, Retell AI, Vapi, or Custom — with provider-specific fields revealed dynamically.
scenario creation — upload or build, persona library always on hand
Teams could either upload existing scenario files or build from scratch. The persona library — with controls for gender, age, accent, and personality type — was accessible at every point in scenario creation so teams could assign the right synthetic user to each branch.
review flow — the graph before you run it
Before launching a simulation, teams saw a full-page review: the scenario graph visualised as a flow, and a table mapping each persona to the situation it would encounter. This screen served a critical function — it made implicit assumptions explicit before they ran 200 synthetic conversations on a broken setup.
run simulation — named, tracked, repeatable
Every simulation run was given a name and linked to a saved evaluation set. That made runs referenceable — not just "the run we did on Tuesday" but a named, versioned artifact you could return to, compare against, and discuss with the team.
results & eval library — from score to fix in one view
Results surfaced a distribution bar showing how runs performed across the full score range — not just an average that hid the outliers. The eval library organised metrics into Chat, Audio, and Quality categories so teams only saw what was relevant to their agent type.
- ·Distribution bar: shows the shape of results across all runs, making outliers visible without burying them in a mean.
- ·Fix my agent CTA: from any failed run, one click surfaces the exact conversation state that caused the failure and jumps directly to the relevant agent config.
shipped to beta. working as intended.
The simulation product launched to three internal teams in beta. The early signals confirmed the core design decisions held up under real use.
what designing for AI taught me about design.
- —The hardest part wasn't the UI — it was the mental model. Before a single screen could be designed, there had to be agreement on what a "scenario" actually was. Getting to "a scenario is a graph, not a list" took three rounds of working sessions with the ML team. The design only worked because the model did.
- —Designing for failure is different from designing for success. Most product design is about making the happy path fast and clear. Simulation design is the opposite — every surface has to make failure legible, reproducible, and actionable. That requires different instincts.
- —The "Fix my agent" CTA was the most-loved feature in beta — and it wasn't in the original brief. It came from watching a PM try to action a failed result and realising there was no clear next step. Sometimes the most impactful design work is noticing what's missing from the spec.
- —Non-technical users were the unlock. The brief was for ML engineers. But the beta success came from PMs and QA running simulations independently — because the graph visualisation made the agent's behaviour understandable without reading the code. Designing for the person who doesn't have the context is harder, and more valuable.