[TakGN]

Why

needle's OpenTelemetry tracing is unusable in v1.0.0. Running the v1.0.0 image against a local collector showed two bugs:

1. Root sampling never fires. otelhttp v0.70 names the server span with the semconv default, GET / (method plus route, and needle sets no route). The root sampler only samples spans whose name starts with needle, so NEEDLE_SAMPLING_FRACTION samples nothing at any value.
2. A traced request is one opaque span. With NEEDLE_TRACING_AUTH and a sampled traceparent, needle exports a single GET / span. Nothing shows which router answered, how long each took, or what a routing cutoff would cost.

This is the needle side of ipni/infra#163 (tracing via Alloy and Tempo). It also makes ipni/workers#18 useful: route-origin's Envoy head-samples 5%, strips the client's traceparent/tracestate, and passes its own traceparent to needle from a router needle egress span. With this PR, needle can join that trace.

What changes

Sampling (tracing.go). The sampler is ParentBased(requestSampler{TraceIDRatioBased(NEEDLE_SAMPLING_FRACTION)}). The root sampler applies the fraction only to the request's HTTP server span. Any other root span is dropped, so background DHT and address-book work stays out of traces, as before. A request continues its caller's trace only when needle trusts the caller's traceparent:

• NEEDLE_TRACING_TRUST_TRACEPARENT=true (new, default off). needle trusts every request's traceparent. This is for a needle that is reachable only through a proxy that replaces client headers, which is what Envoy does in Trace 5% of route-origin's requests workers#18. needle then samples exactly what Envoy sampled: a sampled traceparent joins Envoy's trace, and an unsampled one is dropped.
• NEEDLE_TRACING_AUTH (kept). A request whose Authorization header matches still continues its traceparent. This is on-demand tracing of a single request by someone who can reach needle directly. Behind Envoy, Envoy has already replaced that header, so it no longer does anything there.
• Any other request starts its own trace, sampled at the fraction, and keeps the caller's traceparent as a span link (otelhttp.WithPublicEndpointFn). v1.0.0 deleted the header instead. An untrusted client therefore cannot force sampling, and so cannot control how much needle exports.

Span names (tracing.go). The request span is named needle <method> <route> using boxo's route templates, for example needle GET /routing/v1/providers/{cid}, and gets http.route. Names never include the CID, peer ID or IPNS name, and unknown methods become HTTP. Status comes from otelhttp: ERROR on 5xx, which covers "every router failed". A lookup that finds no records is not an error.

Router spans (server_routers.go). Every router a request asks gets a child span needle router <op> <router>, for example needle router providers delegated:cid.contact or needle router providers dht. Span metrics built per span_name therefore give per-router latency directly. The spans reuse the NEEDLE_ROUTER_TRACE measurements (routerTrace) instead of measuring twice. routerTrace ends each span at that router's own finish time, once the request is done, because exclusive records are only known then. Attributes on each router span:

• needle.router.records
• needle.router.exclusive_records: what the request would have lost without this router.
• needle.router.first_record_ms, plus a first record event.
• needle.router.errors
• needle.router.outcome: the metrics' cancelled reason is split into timeout (routing timeout), client_cancelled and closed (records limit reached, or another router already answered). The other values are exhausted, cut (DHT tail budget) and error.
• needle.router.found (IPNS only).

ERROR is set on a failed call or on error results. Not found and cancellation are not errors.

Paths that skip manyIter get the same span:

• a single router (traceSingle/tracedIter);
• the DHT when no delegated endpoint serves the operation (tracedRouter). With auto endpoints that covers /peers, /ipns and the closest-peers endpoint;
• IPNS get and put.

The request span also gets needle.routing.outcome, needle.routing.routers, needle.routing.last_router and needle.routing.tail_ms. One limit: on a routing timeout or a client disconnect, the response ends without waiting for the routers, so the request span can close before these are set. The router spans still record what happened.

DHT library spans (server_routers.go, tracing.go). kad-dht starts a span for every peer a lookup queries: in a local run, about 230 spans per provider lookup. libp2pRouter now calls the DHT under a context detached from the trace, so those spans become roots and the sampler drops them. The DHT router span covers the lookup as a whole. Without this, the 1 s tail-sampling rule in ipni/infra#163 would keep most DHT-backed traces with hundreds of spans each. If anyone wants the DHT internals back, a flag would be a small follow-up.

Cached address book and /dnsaddr (server_cached_router.go, dnsaddr.go).

• The DHT router span counts needle.addr_book.hits, misses, unused, lookups_dispatched, and, for /peers, negative_hits and lookups_joined. The counts are passed through the context by a small spanCounters helper.
• Each DNS query a request waits on gets a needle dnsaddr lookup span: dns.question.name, result, address count, and whether the query was shared with another request's in-flight lookup. At most 16 such spans per request.
• Cache hits and throttled lookups are counted on the request span.

OTEL_* variables (tracing.go).

• The resource is built with resource.New(..., WithAttributes(needle's), WithFromEnv()), so OTEL_SERVICE_NAME and OTEL_RESOURCE_ATTRIBUTES now win. v1.0.0 merged service.name=needle over them.
• service.instance.id defaults to <hostname>:<API port>, which tells apart instances that share a host network. host.name is added too.
• OTEL_EXPORTER_OTLP_ENDPOINT (or ..._TRACES_ENDPOINT) alone now selects the OTLP exporter. boxo's OTEL_TRACES_EXPORTER defaults to empty, not otlp as in the spec.
• With no exporter configured, needle samples nothing, so there is no span overhead.

Other files.

• main.go, server.go: the new flag, and withTracing replaces withTracingAndDebug.
• docs/tracing.md: rewritten around sampling and proxies, plus a reference for the span names and attributes.
• docs/environment-variables.md: documents NEEDLE_SAMPLING_FRACTION, NEEDLE_TRACING_TRUST_TRACEPARENT, NEEDLE_TRACING_AUTH and the OTEL_* variables.
• CHANGELOG.md: entries under Unreleased.

Tests

tracing_test.go uses tracetest. It covers:

• The sampler decision: sampled and unsampled parents, no parent at fraction 0 and 1, background roots dropped, and roughly half kept at 0.5.
• Span names for every route, with no CID in any name.
• Trust: untrusted traceparent → own trace plus link; proxy trust; NEEDLE_TRACING_AUTH with matching and wrong values.
• Request span status for 200, 404 and 500.
• Router spans through boxo's real handler: parentage, records, exclusive records, errors, the failed-call outcome, and that an empty router is not an error.
• The timeout outcome, the all-routers-failed 500, single-router and DHT-only spans, untraced, address-book counters, OTEL_SERVICE_NAME/OTEL_RESOURCE_ATTRIBUTES precedence, and endpoint-only exporter selection.

TestCombineRouters now expects the DHT-only branch to be wrapped in tracedRouter.

go test -race ./..., go vet ./... and gofmt are clean.

End-to-end check

I ran the built binary against a local otel/opentelemetry-collector-contrib with a file exporter and read the exported spans.

Instance A: as behind Envoy. OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 (HTTP), NEEDLE_TRACING_TRUST_TRACEPARENT=true, NEEDLE_SAMPLING_FRACTION=0.

• A sampled traceparent on /providers joins that trace, and the request span's parent is the remote span. The trace has 6 spans: the request, cid.contact (17 records, 16 exclusive, first record at 221 ms, exhausted), the DHT (136 records, 121 exclusive, timeout after the 25 s routing timeout, with address-book counts) and 3 /dnsaddr lookups. Before the DHT detach it had 232.
• No traceparent, and an unsampled traceparent: nothing exported.
• /peers, /ipns and /dht/closest/peers with a sampled traceparent: request span plus needle router peers dht, needle router ipns_get dht (found=false) and needle router closest dht.

Instance B: direct. OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317 with OTEL_EXPORTER_OTLP_PROTOCOL=grpc, OTEL_SERVICE_NAME=needle-b, NEEDLE_SAMPLING_FRACTION=1, NEEDLE_TRACING_AUTH set.

• No traceparent: sampled as its own root, with needle.routing.outcome=exhausted and last_router=dht.
• Untrusted sampled traceparent: a new trace ID, with a link to the sent one.
• Matching Authorization: joins the sent trace.
• Client gave up at 1 s (curl --max-time 1): DHT span outcome=client_cancelled, cid.contact exhausted.
• Resources showed service.name=needle-b, and distinct service.instance.ids for the two instances.

Rollout

Nothing changes in production until needle is released and ipni/workers sets needle's OTEL environment: OTEL_EXPORTER_OTLP_ENDPOINT at the local Alloy collector, NEEDLE_TRACING_TRUST_TRACEPARENT=true, and a small NEEDLE_SAMPLING_FRACTION for requests that arrive without Envoy's header. That is a separate ipni/workers change after a release, and this PR does not make it.

Not in this PR

• No dependency bumps were needed. Everything uses the otel v1.45 and otelhttp v0.70 already pinned, so dependabot chore(deps): bump google.golang.org/grpc from 1.81.1 to 1.83.2 #1, chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc from 1.44.0 to 1.45.0 #3, chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp from 1.44.0 to 1.45.0 #5 and chore(deps): bump go.opentelemetry.io/otel/exporters/otlp/otlptrace from 1.44.0 to 1.45.0 #6 stay separate.
• Pre-existing and left alone: once manyIter.Next has returned false, manyIter.Close returns early, so on a timeout it neither waits for the routers nor closes the underlying iterators. That is why the request-level needle.routing.* attributes can be missing on timeouts. Changing Close affects the streaming contract, so it is worth its own change.

🤖 Generated with Claude Code