Distributed tracing in .NET with OpenTelemetry

The alert fires at 14:07. The checkout API is returning 504s. You open its logs: "upstream timeout after 30 s". Checkout calls pricing first, so you open pricing: a normal afternoon, nothing at 14:07. Then inventory, which checkout calls after pricing: "command timeout" on a database call, hundreds of times, starting at 14:05. Three services, three log streams, three clocks, and no way to say which checkout request hit which inventory timeout. The incident channel fills with screenshots of grep output.

Every one of those services was logging correctly. What none of them did was tell the next one which request it was working on. That is the whole problem distributed tracing solves, and the fix is smaller than the outage suggests: one header, one .NET type, and an exporter.

A trace is a tree of timed spans

A span is one unit of work with a name, a start time, a duration, a bag of attributes, and two identifiers: the id of the trace it belongs to and the id of its parent span. A trace is the tree you get when every span in a request knows its parent. Draw the spans on a shared time axis and you have a waterfall: the root at the top, each callee indented under its caller, each bar as long as the work took.

The tree exists only if the parent id crosses process boundaries. The W3C Trace Context standard defines the carrier for HTTP: a traceparent header with a version, the 16-byte trace id, the 8-byte id of the calling span, and a flags byte that carries the sampling decision.

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
             ver trace-id                         parent span id   flags

When checkout calls inventory with that header, inventory's server span becomes a child of checkout's client span, and the timeout inventory logs carries the same trace id as the 504. Without the header each service starts a fresh root, and you get three one-node trees that no query can join. That is the shape of the incident above: not missing logs, missing edges.

One request, three services

Three logs, or one trace

14:07:02, checkout returned 504

Without propagation

every service starts a new root

traceparent: absent

• checkout-api trace 7c1e9a...
  504 upstream timeout after 30 s parent: none
• pricing trace a90b2f...
  (nothing at 14:07) parent: none
• inventory trace 3f4d81...
  command timeout, 412 times parent: none

three ids, three clocks, no edges between them

With traceparent on every hop

one trace id, one time axis

trace 4bf92f...

• POST /checkout 30.0 s
• GET /price 40 ms
• SELECT prices 9 ms
• POST /reserve 29.4 s
• SELECT stock 6 ms
• UPDATE stock timeout

0 s15 s30 s

checkout-api pricing inventory status = error

the timeout is a child of the request that returned 504

The .NET primitive is Activity, not a vendor SDK

The tracing primitive lives in the base class library. System.Diagnostics.Activity is the span, and ActivitySource (added in .NET 5) creates them. The framework already uses both: ASP.NET Core starts a server Activity for every request and reads an incoming traceparent into it, and HttpClient writes the current Activity into the outbound traceparent header. OpenTelemetry does not replace these types. Its SDK listens to the sources you name, enriches the activities, and exports them over OTLP to whatever backend you run. Microsoft's distributed tracing overview describes that split, and the OpenTelemetry .NET getting started guide shows the wiring. The minimal production setup is one statement:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService("checkout-api"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddSource("Checkout")
        .AddOtlpExporter());

AddAspNetCoreInstrumentation and AddHttpClientInstrumentation give you the request and outbound-call spans, which is enough to connect the three services in the incident. AddSource opts in a named ActivitySource; a source that is never registered produces nothing, which is the most common reason a manual span "does not show up". AddOtlpExporter reads its endpoint from the OTEL_EXPORTER_OTLP_ENDPOINT environment variable by default, so the same binary ships to every environment.

Manual spans around the work that matters

Automatic instrumentation shows you the edges between services. It does not know that the 400 ms inside POST /reserve was one stock check, one reservation write, and one call to a warehouse adapter. For that you start your own activities:

using System.Diagnostics;

internal static class Tracing
{
    public static readonly ActivitySource Source = new("Checkout", "1.0.0");
}

public async Task ReserveAsync(Order order, CancellationToken ct)
{
    using var activity = Tracing.Source.StartActivity("inventory.reserve");
    activity?.SetTag("order.id", order.Id);
    activity?.SetTag("order.line_count", order.Lines.Count);

    try
    {
        await _inventory.ReserveAsync(order, ct);
        activity?.AddEvent(new ActivityEvent("reservation.confirmed"));
    }
    catch (Exception ex)
    {
        activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
        throw;
    }
}

StartActivity returns null when no listener wants this source, so the null-conditional calls are the intended pattern, not defensive noise. Tags (OpenTelemetry calls them attributes) are key-value facts about the whole span: identifiers, sizes, outcomes. Events are timestamped markers inside the span: "lock acquired", "retry 2 started". Put an identifier in a tag when someone will search for it; a trace backend is built to look up spans by id, which a metrics label is not. Keep secrets and personal data out, because spans are exported and retained exactly like logs.

A trace is not a log with a fancier id. It is the call graph of one request, rebuilt from spans that each recorded who their parent was.

Carrying context across a queue

HTTP propagation is automatic. A message broker is not, and it is where most traces break: the publisher's span ends when the message is acked, the consumer starts an hour later on another machine, and nothing carries the trace id in between except you. The fix is to put traceparent in the message headers and restore it on the other side. The OpenTelemetry API ships a propagator that already speaks the W3C format:

using OpenTelemetry;
using OpenTelemetry.Context.Propagation;

private static readonly TextMapPropagator Propagator =
    Propagators.DefaultTextMapPropagator;

// Publisher: write traceparent (and tracestate) into the message headers.
using var publish = Tracing.Source.StartActivity(
    "orders publish", ActivityKind.Producer);
var headers = new Dictionary<string, string>();
Propagator.Inject(
    new PropagationContext(publish?.Context ?? default, Baggage.Current),
    headers,
    (carrier, key, value) => carrier[key] = value);
await _bus.PublishAsync(message, headers, ct);

// Consumer: read it back and parent the new span on it.
var parent = Propagator.Extract(default, message.Headers,
    (carrier, key) => carrier.TryGetValue(key, out var value)
        ? new[] { value }
        : Array.Empty<string>());
using var consume = Tracing.Source.StartActivity(
    "orders process", ActivityKind.Consumer, parent.ActivityContext);

ActivityKind.Producer and Consumer tell the backend to draw the gap between them as an asynchronous hop rather than a slow call. One rule changes for batch consumers: a span that processes 200 messages cannot have 200 parents. Extract each message's context and pass them as ActivityLink values to StartActivity instead, so the batch span stands on its own and still points back at every trace it touched.

Sampling: decide at the root, then obey the flag

At 50 requests per second with ten spans each, that is 500 spans per second. Export all of them. Sampling exists for the point where the export bill or the backend's ingest rate hurts, not before, and the trace you drop is always the one the incident needed.

When you do sample, the decision is made once at the root and travels in the traceparent flags byte, which is why the SDK's default sampler is parent-based with an always-on root: children follow their parent, so a trace is never half-recorded. Swap the root for new TraceIdRatioBasedSampler(0.1) to keep one trace in ten. That is head-based sampling: cheap, but blind to the outcome, because the root has not finished when it decides. Keeping only slow or failed traces needs tail-based sampling, which means a collector that buffers whole traces before deciding. Start head-based; add a collector when you can name the traces you are losing.

What the waterfall shows

Once the three services share a trace, the incident reads differently. The inventory timeout is a child span under POST /checkout, so the 504 and the timeout are one story with one id. Pricing's span is 40 ms long and finished long before anything went wrong, so its silence was correct. And the root shows checkout giving up at 30 s while inventory's UPDATE kept waiting on the database, which is the cancellation bug you fix next.

Waterfalls expose designs, not just failures. Child spans in a staircase, each starting when the last one ended, are serial calls that could run in parallel. Forty identical tiny children under one parent are the N+1 problem at service scale. A cluster of repeated client spans with growing gaps between them is a retry storm that a circuit breaker should have stopped after the second try. None of those patterns is visible in a log, because a log has no time axis.

Put the trace id in every log line

The last step is to let a log line lead to its trace. ASP.NET Core's logging already attaches TraceId and SpanId as scope values on every event written inside a request, so a JSON console or an OpenTelemetry log exporter with scopes enabled emits them on each line. Search the logs for the failed order, copy the trace id, open the waterfall. This is the join that structured logging sets up and tracing completes: logs hold the application facts, the trace holds the time and the shape. If you run Aspire locally, its dashboard shows these traces without any backend to set up, which makes the propagation gap visible on the first run.

Practice reading a broken trace

In the System Design Studio, the "Observe the Shortener" and "Observe the Video Pipeline" challenges ask for tracing at every tier of a design, so the habit forms before the first 504. Katabench Labs take the same idea into running services: the Labs overview shows the current hands-on lab, a transactional outbox on real PostgreSQL and RabbitMQ, and the Labs guide shows how each check runs against the live services rather than scanning your source.