Table of Contents

OpenTelemetry

NATS.Net has built-in distributed tracing and metrics support through System.Diagnostics.Activity and System.Diagnostics.Metrics.Meter, the standard .NET APIs for OpenTelemetry. Activities are created automatically for publish and subscribe operations, trace context is propagated through message headers so send and receive spans are linked across services, and a set of standard messaging metrics is emitted when a meter listener is attached.

The activity source name and the meter name are both NATS.Net.

Setting Up Tracing

To collect traces, register a listener for the NATS.Net activity source. You can use the OpenTelemetry SDK with an exporter (Jaeger, Zipkin, OTLP, etc.) or a plain ActivityListener for lightweight scenarios:

// The NATS.Net client uses System.Diagnostics.Activity for tracing.
// No additional packages are needed to enable tracing — just add an
// ActivityListener or configure an OpenTelemetry TracerProvider that
// listens to the "NATS.Net" source.

// Using OpenTelemetry SDK (install OpenTelemetry and an exporter):
//
//   using var tracerProvider = Sdk.CreateTracerProviderBuilder()
//       .AddSource("NATS.Net")         // listen for NATS activities
//       .AddSource("MyApp")             // listen for your own activities
//       .AddOtlpExporter()              // export to Jaeger, Zipkin, etc.
//       .Build();

// Or using a plain ActivityListener (no extra packages):
using ActivityListener listener = new ActivityListener
{
    ShouldListenTo = source => source.Name == "NATS.Net",
    Sample = (ref ActivityCreationOptions<ActivityContext> _) =>
        ActivitySamplingResult.AllDataAndRecorded,
    ActivityStarted = activity =>
        Console.WriteLine($"Started: {activity.OperationName}"),
    ActivityStopped = activity =>
        Console.WriteLine($"Stopped: {activity.OperationName}"),
};
ActivitySource.AddActivityListener(listener);

Setting Up Metrics

Metrics are emitted through the same NATS.Net name. No measurements are recorded until a listener subscribes; the runtime cost is a single boolean check per operation when no listener is attached:

// The NATS.Net client emits metrics through System.Diagnostics.Metrics.Meter
// under the same "NATS.Net" name used for activities. Metrics are opt-in:
// nothing is recorded until something subscribes to the meter.

// Using the OpenTelemetry SDK and the NATS.Client.OpenTelemetry package:
//
//   using var meterProvider = Sdk.CreateMeterProviderBuilder()
//       .AddNatsClientInstrumentation()  // or .AddMeter("NATS.Net")
//       .AddOtlpExporter()
//       .Build();

// Or using a plain MeterListener (no extra packages):
using System.Diagnostics.Metrics.MeterListener meterListener = new()
{
    InstrumentPublished = (instrument, listener) =>
    {
        if (instrument.Meter.Name == "NATS.Net")
            listener.EnableMeasurementEvents(instrument);
    },
};
meterListener.SetMeasurementEventCallback<long>((inst, value, tags, _) =>
    Console.WriteLine($"{inst.Name}: {value}"));
meterListener.SetMeasurementEventCallback<double>((inst, value, tags, _) =>
    Console.WriteLine($"{inst.Name}: {value}"));
meterListener.Start();

Automatic Trace Context Propagation

When you publish a message, the client injects the current trace context into the message headers. When a subscriber reads the message, the receive activity is automatically parented to the send activity, giving you end-to-end traces across services with no extra code:

await using NatsConnection nats = new NatsConnection();

// Publish and subscribe — activities are created automatically
await using var sub = await nats.SubscribeCoreAsync<string>("orders.new");
await nats.PublishAsync("orders.new", "order-123");

// The message carries trace context in its headers, so the
// receive activity is automatically linked to the send activity.
await foreach (NatsMsg<string> msg in sub.Msgs.ReadAllAsync())
{
    Console.WriteLine($"Received: {msg.Data}");
    break;
}

Starting Custom Activities

You can start child activities under a message's trace context using the StartActivity extension method. This is useful for tracking processing work that happens after a message is received:

await using NatsConnection nats = new NatsConnection();

await using var sub = await nats.SubscribeCoreAsync<string>("work.items");
await nats.PublishAsync("work.items", "item-456");

await foreach (NatsMsg<string> msg in sub.Msgs.ReadAllAsync())
{
    // Start a child activity under the message's trace context
    using Activity? activity = msg.StartActivity("ProcessWorkItem");

    // The activity is linked to the original publish span
    Console.WriteLine($"Processing: {msg.Data}");
    break;
}

The StartActivity method is available on both NatsMsg<T> and INatsJSMsg<T> for JetStream messages.

Filtering

Use NatsInstrumentationOptions.Default.Filter to skip telemetry for specific requests. When the filter returns false, no activity is created:

// Filter lets you skip telemetry for specific subjects
NatsInstrumentationOptions.Default.Filter = context =>
{
    // Skip internal/health-check subjects
    if (context.Subject.StartsWith("_INBOX."))
        return false;

    return true;
};

When the NATS.Client.OpenTelemetry package is installed, FilterSubjects builds the predicate from NATS subject patterns (* matches one token, > matches one or more trailing tokens) instead of writing the matching by hand. Include patterns allow-list subjects; exclude patterns drop them and win over include. The predicate is combined (logical AND) with any filter already set:

Sdk.CreateTracerProviderBuilder()
    .AddNatsClientInstrumentation(options => options.FilterSubjects(
        include: ["orders.>"],
        exclude: ["orders.internal.>"]))
    .Build();

Enriching Activities

Use NatsInstrumentationOptions.Default.Enrich to add custom tags to every activity:

// Enrich lets you add custom tags to every activity
NatsInstrumentationOptions.Default.Enrich = (activity, context) =>
{
    activity.SetTag("app.environment", "production");

    if (context.QueueGroup is not null)
        activity.SetTag("app.queue_group", context.QueueGroup);
};

The activity passed as the first argument is also Activity.Current for the duration of the callback, on the receive path as well as the send path, so helpers that read the ambient activity work inside Enrich. That holds for the callback only: receive activities are created on the connection's read loop and are kept off the ambient context otherwise, and the read loop's own context is put back when the callback returns.

Filter, Enrich and SpanDestinationNameFormatter all run inside publish and inside message construction on receive, so none of them is allowed to break the operation it is instrumenting. A Filter that throws means the operation is not collected, an Enrich that throws is ignored and leaves the activity with whatever it set before throwing, and a formatter that throws falls back to the default span name.

Span Names and Redacting Subjects

Span names are <destination> <operation>, for example orders.new publish. The destination defaults to the first two tokens of the subject, so orders.new.eu.de becomes orders.new, and inbox subjects are collapsed to the constant inbox before anything else runs.

If you need spans but not subjects in their names, set NatsInstrumentationOptions.Default.SpanDestinationNameFormatter:

// Span names carry the first two subject tokens by default. A formatter replaces
// that truncation rather than running after it, so it is up to you to keep the
// names it returns low cardinality.
NatsInstrumentationOptions.Default.SpanDestinationNameFormatter = subject =>
    subject.StartsWith("tenant.", StringComparison.Ordinal) ? "tenant" : subject;

The formatter replaces the default two-token truncation rather than running after it, so a formatter that returns the subject unchanged produces full-subject span names and it is up to you to keep them low cardinality. It never sees inbox subjects, which are already collapsed.

The formatter only affects span names. Subjects also appear in tags, which the formatter does not touch:

Tag Contents
messaging.destination.name Delivered subject, inbox collapsed
messaging.destination_publish.name Delivered subject, inbox collapsed
messaging.nats.message.subject Delivered subject, inbox collapsed
messaging.destination.template Subscription subject, inbox collapsed
messaging.nats.message.reply_to Reply-to subject, inbox collapsed

To remove or rewrite those, use Enrich, which runs after the tags are set:

// The formatter does not touch tags. Enrich runs after the client has set them,
// so SetTag overwrites the ones carrying subjects.
NatsInstrumentationOptions.Default.Enrich = (activity, _) =>
{
    activity.SetTag("messaging.destination.name", "redacted");
    activity.SetTag("messaging.nats.message.subject", "redacted");
};

Metrics carry no subject at all, so nothing needs redacting there.

Baggage Propagation

W3C Baggage lets you attach arbitrary key/value context (a tenant ID, a correlation ID, etc.) to a trace and have it flow across service boundaries alongside trace context.

Baggage propagation is off by default. Baggage can carry sensitive (PII) or high-cardinality data, and message headers count against size limits (core headers toward the max_payload, 1MB by default; JetStream caps the header block at 64KB), so it needs to be an explicit opt-in:

// Baggage propagation is off by default because baggage can carry
// sensitive or high-cardinality data. Opt in explicitly:
NatsInstrumentationOptions.Default.PropagateBaggage = true;

// Optionally allow-list which baggage keys cross the NATS boundary:
NatsInstrumentationOptions.Default.BaggageKeyFilter = key => key is "tenant.id" or "correlation.id";

// Received baggage is restored onto the receive activity and also
// exposed on the Filter/Enrich callback context:
NatsInstrumentationOptions.Default.Enrich = (activity, context) =>
{
    if (context.Baggage is { } baggage)
    {
        foreach (var entry in baggage)
        {
            Console.WriteLine($"baggage: {entry.Key}={entry.Value}");
        }
    }
};

When enabled, baggage is written to the message as a W3C baggage header on publish and, on receive, extracted from that header, restored onto the receive activity's Activity.Baggage, and exposed via NatsInstrumentationContext.Baggage to the Filter and Enrich callbacks. Child activities created with StartActivity also inherit the restored baggage.

By default the send-side baggage is read from the current send activity's Activity.Baggage. If your application keeps baggage elsewhere — for example OpenTelemetry's Baggage.Current API — use NatsInstrumentationOptions.Default.BaggageSource to bridge it (requires the OpenTelemetry.Api package):

NatsInstrumentationOptions.Default.BaggageSource = () => Baggage.Current.GetBaggage();

When PropagateBaggage is enabled and the source has baggage, NATS.Net owns the baggage header: it overwrites any existing value, or removes the header entirely if BaggageKeyFilter rejects every key. If the source has no baggage, an application-set baggage header passes through untouched.

Like trace context, injecting baggage requires a send activity to exist — there must be a listener on the NATS.Net source that isn't filtered out.

Note

Independent of this feature, the ambient DistributedContextPropagator also writes Activity baggage during trace-context injection: the legacy propagator (the default up to System.Diagnostics.DiagnosticSource 9) writes a non-standard Correlation-Context header, and the W3C propagator (the default from System.Diagnostics.DiagnosticSource 10 / .NET 10) writes an unfiltered W3C baggage header. Enabling PropagateBaggage makes NATS.Net take ownership of the baggage header (so BaggageKeyFilter applies); applications that want strict control over the wire format should configure DistributedContextPropagator.Current (e.g. CreateNoOutputPropagator() or a custom propagator).

Semantic Conventions

NATS.Net follows the OpenTelemetry Semantic Conventions for Messaging. The following attributes are set on activities:

Attribute Example Description
messaging.system nats Always nats
messaging.operation publish / subscribe / request / receive Operation type, matching the value the metrics use for the same operation
messaging.destination.name orders.new Subject name
messaging.nats.message.reply_to inbox Reply-to subject, if the message has one
messaging.client_id 42 NATS client ID
server.address localhost Server host
server.port 4222 Server port
network.protocol.name nats Protocol name
network.transport tcp Transport protocol
network.peer.address localhost Remote host
network.peer.port 4222 Remote port
network.local.address 127.0.0.1 Local IP

Span kind is Producer for publish and request, Consumer for receive, and Client for subscribe, which produces nothing and is a control plane call to the server.

A receive activity is created for every message the client takes off the wire, whether it is handed to the application (core subscriptions, JetStream consume and fetch, service endpoints), consumed by the client itself (key/value watchers, object store reads), or a status frame the client handles internally (no-responder 503 replies, JetStream heartbeats and flow control). It ends when whatever owns the message is done with it, which for a subscription is the read that hands the message to the application.

messaging.client.consumed.messages is narrower: it counts only messages delivered to the application, so receive span counts and that counter are expected to disagree.

Receive activities include additional attributes:

Attribute Example Description
messaging.destination.template orders.* Subject the subscription was created with
messaging.destination.temporary false true when the subscription is on this connection's inbox
messaging.destination_publish.name orders.new Subject the message was published to
messaging.nats.message.subject orders.new Delivered subject
messaging.message.body.size 1024 Message body size in bytes
messaging.message.envelope.size 1280 Total message size in bytes
messaging.consumer.group.name workers Queue group, omitted when the subscription joined none

Every subject-valued tag has inbox subjects collapsed to inbox; see Span Names and Redacting Subjects.

Service Endpoint Spans

A service endpoint is the server side of a request, so its receive activity is the span the handling belongs to and not just the delivery. It stays open until the handler returns, so its duration covers handler time, and it is Activity.Current while the handler runs, so activities the handler starts nest under it without any wiring.

An exception out of the handler is recorded on that span as an exception event and sets its status to Error. That includes NatsSvcEndpointException, which is also the way an endpoint reports an error back to the caller.

Replies Carrying Stored Trace Context

A receive activity normally takes its parent from the trace context carried in the message headers, and falls back to the request it is a reply to when the message carries none.

Messages returned verbatim from storage are the exception. A JetStream direct get replays the stored message headers, including the traceparent written when the value was published, which may be hours old and belong to a trace the backend has long since closed. When a reply carries a trace id different from the request waiting on it, the receive span is parented from the request and the message's context is recorded as a span link instead. The read stays in one trace and the connection to the write is still visible.

A responder that continues the caller's trace shares its trace id, so ordinary request/reply is unaffected.

Metrics

The following instruments are exposed on the NATS.Net meter:

Name Type Unit Description
messaging.client.published.messages Counter {message} Messages published by the client
messaging.client.consumed.messages Counter {message} Messages received by the client
messaging.client.operation.duration Histogram s Duration of publish, request, and subscribe operations
nats.client.active_subscriptions UpDownCounter {subscription} Active NatsSubBase instances. Under SharedInbox request/reply mode each in-flight RequestAsync registers a transient reply subscription with the shared inbox muxer and is included here; Direct mode uses a reply task and is not counted
nats.client.reconnects Counter {reconnect} Successful reconnects since process start
nats.client.sent.bytes Counter By Bytes sent in published messages (body + headers)
nats.client.received.bytes Counter By Bytes received in consumed messages (body + headers)
nats.client.messages.dropped Counter {message} Messages evicted from a full subscription channel (slow consumer)

All instruments carry these tags:

Tag Example Description
messaging.system nats Always nats
messaging.operation publish / receive / subscribe / request / reconnect Operation type
server.address localhost Server host
server.port 4222 Server port
network.protocol.name nats Protocol name
network.transport tcp Transport protocol

messaging.client.operation.duration adds error.type (full exception type name) when the operation fails.

messaging.client.consumed.messages and nats.client.received.bytes count only messages delivered to the application, per the OTel definition ("messages delivered to the application"). NATS status and control frames consumed internally by the client are excluded: no-responder 503 replies and JetStream heartbeats, flow-control, and protocol notifications. The two counters stay consistent, so received.bytes / consumed.messages reflects average delivered message size.

nats.client.received.bytes is a NATS-specific counter with no semantic convention to match, and the bytes of an excluded status frame were physically on the wire, so it could be argued either way. Deliberately excluding them keeps it aligned with consumed.messages, which is what makes the ratio above meaningful. Use nats.client.messages.dropped and the connection's own statistics if you need wire-level accounting.

Histogram Buckets

messaging.client.operation.duration ships advisory bucket boundaries (0.005s to 10s) through InstrumentAdvice, which the OpenTelemetry SDK applies by default, so no view is required for sensible latency buckets. To override them, add a view on the meter provider (this needs the OpenTelemetry SDK package, not just OpenTelemetry.Api):

Sdk.CreateMeterProviderBuilder()
    .AddNatsClientInstrumentation()
    .AddView(
        "messaging.client.operation.duration",
        new ExplicitBucketHistogramConfiguration
        {
            Boundaries = [0.001, 0.005, 0.01, 0.05, 0.1, 0.5, 1, 5],
        })
    .Build();
Edit this page