Nine versions was never a target. It is what happens when an agent ships to customers who did not build it, and each version teaches something the one before could not. This is what those versions taught, written as patterns rather than as a changelog.
What it does
CoTester is TestGrid’s testing agent. It takes a plain-English description of what someone wants tested and turns it into steps a runner can execute against a real web or mobile application. Underneath that sentence sit four separate jobs: interpret what the user actually wants, understand the application under test, resolve the intent to the right on-screen targets, and emit steps that are valid for the platform they will run on.
Each of those can fail on its own, and to the user every one of them looks the same: the agent got it wrong. Treating generation as a single step hides where the errors come from. Splitting it into four was the most useful decision in the whole pipeline, because it turned a vague complaint into a specific diagnosis.
Why generated steps break
The first version produced steps that read well and did not run. A model that has seen a great deal of test code will produce something that looks like test code, and looking right is a different property from being executable on a specific platform with a specific set of supported actions. Closing that gap was mostly not a prompting problem. It meant fine-tuning the model on structured, directly executable output, then extending it to hold up across multi-step workflows and long sequences of interactions, where a small drift early on compounds by the end.
The second failure was noise. A page serialised naively is mostly things the model does not need: wrappers, hidden nodes, duplicated attributes. Feeding all of it in makes the model slower and worse at picking the element that matters. Dedicated element parsers for web, Android and iOS, and careful preparation of page data before it reaches the model, did more for reliability than any change to the model itself. On top of that, an agentic element layer lets elements curated for a particular screen take priority during generation, so the steps align with what actually matters on that page rather than everything that happens to be on it.
The third failure was the DOM simply not being enough. Some targets have no stable attribute to anchor to. Giving the agent vision-language understanding and inline image input meant it could reason from what a screen looks like, not only from what the markup says. The same idea reached back into the design stage: a Figma upload into the knowledge base lets tests be generated from a design before a build exists.
Checking its own work
The change customers felt most was making the agent verify and repair its own output before anything executes. Self-validation checks generated steps against what the platform can run and what the application actually exposes; auto-correction repairs the ones that fail instead of handing the failure to the user. Broken and flaky output reaching customers dropped sharply once that was in place.
The lesson underneath it: an agent’s error rate is half the story. What it does with its own errors is the other half, and that is the half users notice.
Running it for many tenants
An agent that works for one customer is a demo. The platform runs as a secure, multi-tenant service: conversational memory, a knowledge base per customer, and behaviour that can be tailored per client and per application, exposed through clean APIs the product consumes. Multi-tenancy forces discipline early. Authentication and request-level security controls; agent configuration that is encrypted and versioned; staged rollout so a new version reaches live customers gradually rather than all at once.
Every interaction is instrumented for usage, latency and cost. That telemetry is not for dashboards. It is the input to model selection and agent strategy: once you can see what each conversation costs and how long it takes, decisions about which model handles which step stop being opinions. That has its own write-up in Cutting cost per conversation in a production LLM agent.
What nine versions taught
- Ship the validation before the capability. A feature that produces impressive output some of the time is worse in production than a narrower one that checks itself.
- Prepare the input; the model is rarely the bottleneck. Most of the reliability came from what reached the model, not from the model.
- Telemetry is a design tool. Cost and latency per conversation shaped the architecture more than any benchmark did.
- Roll out in stages. Real customers find the case the test suite did not, and you want a small blast radius when they do.
- Feedback is the product. The agent that exists now is the one those nine rounds of customer feedback taught us to build.