Back to guidesGuide

Build golden signals for Kubernetes HTTP APIs without cardinality sprawl

A practical flow for measuring traffic, errors, latency, and saturation with Prometheus, Grafana, and a label contract that does not burn the TSDB.

Define a label contract, create RED/USE recording rules, validate cardinality with promtool, and roll out Kubernetes API dashboards without turning every route into a series factory.

Created: July 7, 2026

Published: July 7, 2026

Estimated time40 min
LevelIntermediate
Before you startPrometheus scraping HTTP metrics or OpenTelemetry exporting compatible histograms.
PlatformsLinux / Docker
WhatsAppXLinkedIn

Linux

Local validation with kubectl, promtool, and PromQL queries before publishing recording rules and dashboards.

kubectlpromtoolcurlPrometheus access
Open Prometheus and validate base queries
kubectl -n platform-observability port-forward svc/prometheus 9090:9090
promtool query instant http://localhost:9090 'sum by (service, route) (rate(http_server_request_duration_seconds_count{job="checkout-api"}[5m]))'
promtool query instant http://localhost:9090 'topk(20, count by (__name__, service, route, method, status_code)({service="checkout-api"}))'
promtool check rules rules/api-golden-signals.yaml
promtool test rules rules/api-golden-signals.test.yaml
Check that rules load
promtool check rules rules/api-golden-signals.yaml
Run fixtures before rollout
promtool test rules rules/api-golden-signals.test.yaml

Content locked

This guide requires both steps before full content is available.

  • Click “Like” on this guide.
  • Share on WhatsApp, X, LinkedIn, or copy the link.

Access is automatically unlocked as soon as both steps are completed.