← worklog

2021-06-01

Observability stack rollout with kube-prometheus-stack

PrometheusGrafanaKubernetesAzure

Starting point

No centralized metrics. Teams had ad-hoc scripts polling endpoints and sending Slack messages when things looked wrong.

Stack

  • kube-prometheus-stack Helm chart as the base — Prometheus Operator, Grafana, AlertManager, node-exporter, kube-state-metrics
  • Custom PrometheusRule resources per team, packaged in their service Helm charts
  • AlertManager routes to PagerDuty (P1/P2) and Slack (P3 and below)
  • Grafana with team-scoped folders via provisioning configmaps

Dashboard approach

One standard SLO dashboard per service: error rate, latency p50/p95/p99, throughput, and saturation. Built as a Grafana dashboard JSON template, parameterized by service name. Teams clone and customize.

Lessons

AlertManager routing trees get complex fast. Keep it flat: one matcher per route, no nested routes unless you really need per-receiver inhibition. The complexity cost isn't worth the flexibility until you're at 50+ alert rules.