Skip to navigation

Traces lists every distributed trace that the gateway records. Use it when you need to debug a specific request instead of looking at aggregate metrics. This topic shows you how to find a trace, inspect it, and check routing decisions.

Prerequisites

Before you begin, make sure you have the following:

  • A route that has received requests.
  • Access to the organization that owns the route.

Find a trace

To find a trace, do the following:

  1. In the sidebar, select Traces to see the current namespace, or select Admin and then Traces to see your whole organization.
  2. Optionally, select a time range: 24H, 7D, or 30D. To look at a specific period, select Custom, set a Start and an End (or Until now), and then select Apply. A custom range can span up to 30 days, and times are in your time zone.
  3. Optionally, add one or more filters: Protocol, Route, Connection, User, or Status. To find a specific operation, enter text in Filter by operation.
  4. To change which columns appear, click View.

The page shows summary cards for total traces, P95 latency, and error rate over the selected range. A table follows, with the following columns:

ColumnWhat it shows
Trace IDThe unique identifier of the trace.
TimestampWhen the request arrived.
ProtocolThe type of traffic: HTTP, LLM, or MCP.
OperationWhat the request did. For LLM traffic, this is chat followed by a model name, for example chat claude-sonnet-5.
RouteThe route that received the request.
ConnectionThe connection that handled the request.
UserThe user the request belongs to.
StatusThe result, such as 200.
DurationHow long the request took.

Filter traces by span attribute

The console doesn’t have a control for filtering by span attribute. Use the admin API instead: add the attribute’s full name as a query parameter on the traces list request.

curl -G "$FABRIC_API/traces" \
-H "fg-tenant-api-key: $FABRIC_API_KEY" \
--data-urlencode "fabric.metadata.user=12345" \
--data-urlencode "http.request.method=GET"

The request follows these rules:

  • Name: Any query parameter with a dot in its name is an attribute filter. A name is dot-separated segments of letters, digits, underscores, and hyphens, for example http.request.method. A name that doesn’t match gets 400. Names that start with fabric.metadata. also accept spaces and other characters, but not = or control characters.
  • Operator: A filter matches when the attribute equals the value. There are no other operators. To match any of several values, repeat the parameter, for example ?http.request.method=GET&http.request.method=POST.
  • Several attributes: The trace must match every attribute. The attributes can be on different spans of the trace, because the match is per trace, not per span. For example, a request’s fabric.metadata.user and a plugin’s attribute can be on different spans, and one filter matches both.
  • Other filters: The filters in the console still apply, and so does the time range. The attribute filter doesn’t search other organizations’ spans.

These attribute names map to the filters in the console:

AttributeSame as
fabric.route.nameRoute
fabric.service.nameConnection
fabric.consumerUser
fabric.protocolProtocol

http.response.status_code takes a status class such as 5xx or an exact code such as 502. Unlike the Status filter, which uses the gateway’s own response code, it matches any span in the trace, so you can find a trace where the upstream returned 502.

fabric.metadata.NAME matches the metadata that clients send in the x-fabric-metadata header. The header is a JSON object. The gateway records it on the trace and removes it before it forwards the request to the provider.

Filtering by attribute can be slower than the other filters on a large time range, so narrow the range first. The summary cards respect the same filters.

Inspect a trace

Select a row in the traces table to open its span view.

Each trace has one or more spans. For example, an HTTP request through an imported connection has a gateway span and a nested span for the request to the upstream. Select a span to see its attributes, including the following:

  • Plugin execution time and invocation count
  • Upstream attempt count, body-read count, and duration
  • Route match type and route name
  • Tenant ID
  • HTTP method and status code
  • The originating user agent

The detail view also shows the request and response bodies. For agent traffic, the request body includes the system prompt and the tool definitions that the agent sends with each request.

See what a request cost

For an LLM request, the span’s attributes include what Fabric uses to work out its cost:

  • Tokens: gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, and gen_ai.usage.total_tokens, plus gen_ai.usage.reasoning_tokens and the cache counts, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens.
  • Models: gen_ai.request.model is the model the client asked for, and gen_ai.response.model is the model that answered. They can differ when a routing plugin changes the model.
  • Cost: gen_ai.usage.cost_usd is the estimated cost in US dollars, and gen_ai.usage.cost_status says how reliable it is: exact, partial, unpriced, dropped, or unavailable. The cost can appear a few seconds after the response.

These are the numbers that spending budgets add up. For details, see Control spend with budgets.

Check how plugins handled a request

When you attach a routing plugin to a route, the trace shows the extra requests that the plugin makes:

  • A request that uses the Adaptive Model Router has its own request to the decision service, followed by the request to the provider. The decision changes the model identifier in the provider request.
  • A request on a route without a routing plugin shows only the request to the provider.

To confirm which model answered, check the model field in the response body. The Operation column can show the model that the client requested instead of the model that the plugin chose.

For more information about the plugins, see Route LLM requests across models and providers.

Next steps

⌂ 换网址