Back to guidesGuide

Monitor controller-runtime without melting the Kubernetes API server

A practical workflow to catch LIST storms, stale caches, reconcile queues, and control-plane alerts before your operator becomes the incident. A generous little gift from automation.

Learn how to monitor Kubernetes controllers built with controller-runtime using Prometheus, workqueue metrics, API server signals, promtool rules, and canary validation.

Created: August 27, 2026

Published: August 27, 2026

Estimated time45 min
LevelAdvanced
Before you startPrometheus scraping the API server and custom controllers
PlatformsLinux / Docker
WhatsAppXLinkedIn

Linux

Diagnose from a Linux workstation with kubectl, curl against Prometheus, and promtool.

kubectlcurlpromtool
Check controller state
kubectl get pods -n platform -l app=checkout-operator -o wide
Inspect recent errors
kubectl logs deploy/checkout-operator -n platform --tail=120 | egrep -i 'reconcile|watch|cache|timeout|throttle|error'
Query reconcile queues
curl -G 'http://localhost:9090/api/v1/query' --data-urlencode 'query=max by (controller)(workqueue_depth{job="checkout-operator"})'
Test rules
promtool test rules tests/controller-runtime-cache-pressure.test.yaml

Content locked

This guide requires both steps before full content is available.

  • Click “Like” on this guide.
  • Share on WhatsApp, X, LinkedIn, or copy the link.

Access is automatically unlocked as soon as both steps are completed.