Tail-Based Distributed-Trace Sampling

26 Oct 2026 observabilitydistributed systems

Tail-based distributed-trace sampling decides whether to keep a trace after its spans have arrived, rather than deciding separately when each span is created. The collector temporarily buffers spans by trace ID, waits long enough to observe the trace, then retains or discards the trace as a whole.

Why decide at the tail?

A distributed trace is the set of spans—records of timed operations—belonging to one request as it moves between services. A trace might include an API gateway, an application service, a database call, and a downstream HTTP request. Each span normally carries the same trace ID, which lets systems associate those records.

Sampling means keeping only some traces to control storage, network traffic, and query cost. The simpler approach, head-based sampling, makes the decision at the start of a request. For example, a service might keep 1% of requests and discard the rest. That is cheap, but it can throw away the one trace that contains a five-second database timeout or an error in a rarely used downstream service.

Tail sampling can make a better decision because it can inspect evidence from the entire trace:

• Keep traces containing an error.
• Keep traces slower than one second.
• Keep traces from a particular customer or endpoint.
• Keep a small random sample of ordinary, successful traces.

Without tail sampling, each service would have to decide independently, and a trace could arrive at the backend with important spans missing. A trace-wide decision also avoids storing a confusing mixture in which some spans from one request were sampled and others were not.

The collector builds a temporary trace

The sampling component usually runs in a telemetry collector, a service that receives, processes, and forwards telemetry. As spans arrive, it uses their trace IDs as keys:

1. A span arrives from an instrumented service.
2. The collector places it in the buffer for that trace ID.
3. More spans from other services join the same buffered trace.
4. The collector evaluates the accumulated trace against sampling policies.
5. It forwards every span for a kept trace, or drops every span for a rejected trace.

The spans do not necessarily arrive in the order the request ran. Network delays, retries, and batching can make a child span arrive before its parent. Grouping by trace ID lets the collector assemble the trace despite that ordering.

The collector cannot normally know the exact moment a trace is finished. Instead, it uses a decision wait, a bounded period during which it expects more spans. When that period expires, the collector decides using the spans it has received. A longer wait captures slower or later-arriving spans but consumes more memory and delays forwarding. A shorter wait reduces resource use but increases the chance that a kept trace is incomplete.

This means “complete trace” has an operational meaning: all spans that arrived before the collector’s deadline, not a guarantee that every service successfully emitted its span.

Policies inspect the whole trace

After the wait, the sampler evaluates fields and measurements across the buffered spans. A policy might match an error status on any span, a duration on the root span, a service name, an HTTP route, or an attribute such as a tenant ID. Several policies can be combined, with rules for which one wins when a trace matches more than one.

A common arrangement is:

Keep all traces with errors, keep slow traces, and sample a small percentage of the remainder.

The decision is generally one of three outcomes: keep, drop, or defer while more spans may still arrive. Once the collector has decided, it can remember that result for late spans. Otherwise, a span arriving after the decision could be forwarded by itself and recreate the inconsistency tail sampling was meant to prevent.

Buffering is the hard resource problem

Tail sampling requires the collector to receive spans that head sampling would have discarded immediately. It must also hold them temporarily. A busy system can therefore create a large number of in-progress traces, especially when requests are slow or spans are delayed.

A practical collector puts explicit bounds around this work:

• Maximum traces in memory: limits how many trace buffers may exist at once.
• Maximum spans or bytes per trace: prevents one unusually large trace from consuming the process.
• Decision timeout: releases buffers that never appear to finish.
• Eviction rules: discard the oldest or least useful incomplete traces when capacity is reached.
• Rate limits and backpressure: slow or reject incoming telemetry when downstream processing cannot keep up.

These limits create trade-offs. Dropping a buffer because the collector is full can lose an error trace, while accepting unlimited data can cause out-of-memory failure. Tail sampling is therefore not a way to avoid ingestion load; it is a way to spend that load selectively and bound its cost.

Scaling without breaking trace grouping

A single collector instance may not have enough memory or network capacity to buffer all incoming traces. Multiple collector instances can share the work, but all spans for a given trace must reach the same instance. A load balancer or routing layer typically hashes the trace ID so that related spans are consistently assigned together.

The deployment often separates responsibilities:

• Ingest collectors receive telemetry and route it by trace ID.
• Sampling collectors buffer and evaluate complete trace candidates.
• Exporters send kept traces to the tracing backend.

This separation lets each tier scale for its own bottleneck. It also makes failure behavior important: if a sampling instance disappears, the trace buffers in its memory disappear with it. Some systems use durable or disk-backed queues for transport, but those protect queued data and do not remove the need for a bounded decision buffer.

What the setting or incident usually means

When you see a configuration named tail_sampling, decision_wait, or a maximum trace count, it is describing this buffer-and-decide process. An incident saying that the collector is “dropping spans due to queue or memory pressure” may mean the sampler hit one of its bounds, not that the application stopped producing telemetry.

Tail sampling works best when trace IDs are preserved across service boundaries, clocks and instrumentation are reliable enough to identify slow or failed operations, and the collector is sized for the unsampled incoming volume. The result is a smaller backend dataset containing complete, deliberately chosen traces—while the collector remains protected by explicit limits on how long and how much it buffers.