What Did That Agent Run Cost? Token and Latency Accounting in OpenSearch UI

Every model call your agent makes is already a span with a token count. A few PPL queries turn those spans into a bill.

For most teams, the cost of last Tuesday's agent runs shows up weeks later on the model provider's invoice, as one number for the whole account. That number can't tell you which agent spent the money, which service called it, or which run got stuck in a loop. When you instrument an agent with OpenTelemetry, each step it takes becomes a span: one for the run as a whole, and one for every model call and tool call inside it. Each model-call span records which model answered, how many input and output tokens it used, and how long it took. Those spans travel the same path as the rest of your application traces, through an OpenTelemetry Collector and Amazon OpenSearch Ingestion into an Amazon OpenSearch Service domain. Once they land there, OpenSearch UI can query them. This post walks through the PPL queries that turn those spans into cost and latency per model, per run, and per service, and that find the loops that spend tokens without producing answers.

Where the numbers come from

In the observability maturity model from the previous post, agents observed as a workload make up the third rung, and this post zooms in on it. Instrumented with the OpenTelemetry GenAI semantic conventions, an agent emits one trace per run. A root span marks the agent invocation, and child spans mark each model call, tool call, and retrieval. The attribute gen_ai.operation.name says which is which: invoke_agent for the agent, chat for a model call, and execute_tool for a tool. Model-call spans also carry gen_ai.request.model and the token counts in gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. Every span also records its own durationInNanos. Auto-instrumentation for Amazon Bedrock, OpenAI, Anthropic, LangChain, and more than 20 other libraries sets these attributes for you. An OpenTelemetry Collector then sends the spans through Amazon OpenSearch Ingestion into the otel-v1-apm-span* indexes, as Implement Agent Observability describes. For a step-by-step setup, the OpenSearch Observability Stack getting-started guide walks through instrumenting an agent with the Python SDK and viewing its traces.

OpenSearch UI already does some of the arithmetic. In an observability workspace, Discover includes an Agent Traces page. A metrics bar across the top shows trace count, average duration, error rate, and token usage for the time range and query you select. Below it, the Traces tab lists the input and output tokens for every run. That view answers how much in total. Who spent it, and why, takes a query. The Agent Traces page fetches all of its data with PPL, so the same query panel accepts your own. Totals by model come first:

source = otel-v1-apm-span*
| where isnotnull(`attributes.gen_ai.request.model`)
| stats avg(durationInNanos) as avg_ns,
        sum(`attributes.gen_ai.usage.input_tokens`) as input_tokens,
        sum(`attributes.gen_ai.usage.output_tokens`) as output_tokens
    by `attributes.gen_ai.request.model`
| eval avg_ms = avg_ns / 1000000

Durations are stored in nanoseconds, so the last line converts the average to milliseconds. To turn tokens into dollars, add one more step. Providers price input and output tokens separately, usually per million tokens, and the rates differ by model, so the calculation runs one model at a time:

source=otel-v1-apm-span*
| where `attributes.gen_ai.request.model` = 'claude-sonnet-4.5'
| stats sum(`attributes.gen_ai.usage.input_tokens`) as input_tokens, sum(`attributes.gen_ai.usage.output_tokens`) as output_tokens
| eval est_cost_usd = input_tokens / 1000000.0 * 3.00 + output_tokens / 1000000.0 * 15.00

Swap in the model you run and the rates you actually pay, whether that is your provider's current price list or a negotiated rate. Treat the result as an estimate, since it leaves out discounts, cached-token pricing, and anything billed outside tokens. Its value is timing: it arrives while the run is still fresh rather than at month end.

Before trusting any of these sums, check where your instrumentation records tokens. Some libraries roll a run's totals up onto the root agent span as well as recording them on each model call, and summing across every span then counts the same tokens twice. Adding where `attributes.gen_ai.operation.name` = 'chat' to a query keeps the sums on model calls only. My test data is shaped this way, so the queries in this post filter on chat spans. Other instrumentation records a run's tokens only on the invoke_agent span, and in that case you need to filter on that span instead.

Which run, and whose

Per-model totals show which model costs the most. Per-run totals show which requests do, and grouping by trace ID ranks the runs themselves:

source = otel-v1-apm-span*
| where `attributes.gen_ai.operation.name` = 'chat'
| stats sum(`attributes.gen_ai.usage.input_tokens`) as input_tokens,
        sum(`attributes.gen_ai.usage.output_tokens`) as output_tokens,
        count() as model_calls
    by traceId
| sort - input_tokens
| head 10

Because the query aggregates, the Agent Traces page shows the result on its Visualization tab as a chart rather than a list. To read one run in detail, note its trace ID and switch to the Traces tab next to Visualization, which lists every agent run with its duration, span count, and input and output tokens. Selecting that run's row opens the trace details, where the trace tree lists the tokens beside every operation and the timeline shows where the time went. Where du -sh on a full disk points you at the directory to clean up, this query points you at the run to open. A run that made 40 model calls when most make 4 has usually been retrying a failing tool and resending its whole context to the model on every attempt.

With the same pattern, you can charge spend back to whoever owns it. Every span carries the name of the service that emitted it, so grouping by serviceName instead of traceId gives tokens per service. For showback by team or cost center, add a resource attribute such as a team name in your OpenTelemetry SDK or Collector configuration. You set it once. From then on it rides along on every span the service emits and becomes one more field to group by.

Loops that spend quietly

More often than not, an expensive run is a broken one. An agent can call the same tool over and over, getting an error or an unhelpful result each time and resending its whole context to the model between attempts. It can still return a plausible answer with a normal status code, so nothing in the response looks wrong. The trace shows the repetition plainly, and one query finds it across every run:

source = otel-v1-apm-span*
| where `attributes.gen_ai.operation.name` = 'execute_tool'
| stats count() as tool_calls by traceId, `attributes.gen_ai.tool.name`
| where tool_calls > 5
| sort - tool_calls

The threshold of 5 is arbitrary, so set it from what normal runs look like in your own data, and check it on a schedule so a loop surfaces while it is still running. The same tool spans also show which tools are slow. Grouping durationInNanos by gen_ai.tool.name with avg and max separates a tool that is slow every time from one that only occasionally hangs, and the second kind is often what sends an agent into its retries.

Bringing it together

An agent's cost and latency do not have to be reconstructed from an invoice. They are attributes on spans the agent already emits, stored in Amazon OpenSearch Service with the rest of your telemetry. The Agent Traces page in OpenSearch UI shows the totals. A handful of PPL queries break them down by model, by run, by service, and by tool, put a dollar estimate on them, and find the loops that spend without answering. That turns the third rung of the maturity model into numbers you can act on, and the same queries can set a token and cost budget for the investigators at the top of the ladder.