Cost per conversation is the number that decides whether an agent is a business or a demo. Per-token pricing is an input. What matters is what one complete exchange with a user costs end to end, including every retry, every validation pass, and every step the user never sees.

Measure before you optimise

On CoTester the instrumentation came first, and it changed what we chose to optimise. Every interaction records usage per step, latency per step, and the cost that follows from both. Three things fell out of measuring at the level of a conversation rather than a single call:

  • The expensive part was rarely the obvious call. Long, noisy inputs cost more than clever outputs.
  • Retries and self-correction loops are real cost. A step that fails validation and runs again is two steps.
  • Latency and cost are correlated but not the same. The slowest step and the most expensive step were often different steps.

Without per-step attribution you end up tuning the model when you should be tuning the input.

Where model selection actually pays

Not every step needs the most capable model. An agent’s pipeline is a sequence of jobs of different difficulty: interpreting intent is hard; formatting an already validated step into the platform’s schema is not. Routing each job to the cheapest model that meets its accuracy bar is where most of the saving lives, and telemetry is what tells you the bar is actually met.

Two cautions. A cheaper model that fails validation more often can cost more once retries are counted. And selection has to be re-checked whenever models change, so it should be a configuration decision driven by measured numbers, not a constant in the code.

The input is the biggest lever

The largest reduction in cost per conversation came from what the model never saw. Preparing page data properly, with dedicated parsers for web and mobile that strip wrappers, hidden nodes and duplicated attributes, cuts tokens on every single request. It also improves accuracy, because the signal-to-noise ratio goes up. Cheaper and better at the same time is rare; this was one of those cases.

Knowledge bases fit the same pattern. Retrieving the relevant slice for a customer or an application beats carrying everything in context on every turn.

Fine-tuning as a cost decision

Fine-tuning is usually framed as an accuracy tool. It is also a cost tool. A model tuned to emit structured, directly executable output needs fewer tokens of instruction per call, produces fewer malformed results that trigger retries, and lets a smaller model do work that previously needed a larger one. The training cost is paid once; the inference saving is paid on every conversation.

What to watch afterwards

  • Cost per conversation, tracked per tenant, per application and per version. A regression shows up in one of those before it shows in the total.
  • Retry rate per step. It is the early warning for both cost and quality.
  • Latency per step. Users notice a slow answer; they cannot see an expensive one, so you have to.

The pattern generalises beyond testing agents: instrument at the unit the business cares about, attribute cost to individual steps, then let the numbers choose the model and shape the input. The rest of that story, including why the agent checks its own work before anything runs, is in the CoTester write-up.