What is sampling in tracing and what are the trade-offs?

Deep dive

Common strategies:

• Head-based sampling: decide at request start (cheap, may miss errors).
• Tail-based sampling: decide after completion (captures slow/error traces).
• Rate-limited sampling: keep N traces/sec for baseline visibility.

Trade-offs:

• Cost vs coverage.
• Debuggability of rare issues.
• Consistency across services.

Examples

Tail-based policy:

Keep all traces with error=true or duration > 2s
Sample 1% of the rest

Common pitfalls

• Sampling too aggressively and losing error traces.
• Mixed sampling decisions across services (broken traces).
• No sampling for burst traffic, leading to cost spikes.

Interview follow-ups

• When would you prefer tail-based sampling?
• How do you ensure error traces are retained?
• How do you sample for low-volume services?