Back to guidesGuide

Canary disk-backed tail sampling storage in the OpenTelemetry Collector without breaking incidents

Turn tail sampling memory pressure into a measurable canary: disk, RSS, late spans, queues, and trace parity before global rollout.

A practical guide to validating disk-backed tail sampling in the OpenTelemetry Collector when a long decision window needs more state than pod memory can safely hold.

Created: August 9, 2026

Published: August 9, 2026

Estimated time45 min
LevelAdvanced
Before you startAn OpenTelemetry Collector running on Kubernetes or Linux with internal metrics enabled.
PlatformsLinux
WhatsAppXLinkedIn

Linux

Run the Collector canary as a process or systemd service, persist tail sampling state on local disk, and validate RSS, decision latency, and exporter queue health.

otelcol-contribsystemd or shellcurl
Create state directory
sudo install -d -o otelcol -g otelcol /var/lib/otelcol/tailsampling
Start canary
otelcol --config ./collector-canary.yaml --set=service.telemetry.metrics.address=0.0.0.0:8888

Content locked

This guide requires both steps before full content is available.

  • Click “Like” on this guide.
  • Share on WhatsApp, X, LinkedIn, or copy the link.

Access is automatically unlocked as soon as both steps are completed.