Kubernetes Monitoring with Prometheus and Grafana: 5 Best Proven Steps

Operating distributed container workloads across ephemeral nodes makes tracking resource consumption, bottleneck identification, and failure diagnosis complex. Without centralized time-series metrics, operations teams face delayed incident responses, unpredicted pod evictions, and unoptimized cloud infrastructure spend. Implementing robust Kubernetes Monitoring with Prometheus and Grafana delivers cluster-wide observability, giving platform engineers deep visibility into CPU saturation, memory pressure, network throughput, and application health.

While native cloud platform tools provide basic metrics, configuring a dedicated stack for Kubernetes Monitoring with Prometheus and Grafana offers granular control over metric retention policies, multi-tenant metric scraping, custom alerting pipelines, and dynamic dashboard creation. This comprehensive technical guide walks through a 5-step, production-hardened blueprint for deploying, configuring, and managing enterprise monitoring on Azure Kubernetes Service (AKS) using the official kube-prometheus-stack Helm operator.

1. Observability Architecture: Prometheus vs Cloud-Native Metrics

Observability in modern container orchestration relies on three pillars: metrics, logs, and distributed traces. Prometheus serves as the metric collection engine, operating on a pull-based scraping mechanism that polls standardized /metrics HTTP endpoints exposed by underlying Kubernetes infrastructure and individual container workloads.

Deploying an enterprise architecture for Kubernetes Monitoring with Prometheus and Grafana requires understanding how each underlying component interacts across the platform:

Component Name	Core Architectural Role	Default Endpoint / Port	Data Flow Type
Prometheus Operator	Manages the lifecycle of Prometheus instances using declarative CRDs	N/A	Control Plane Automation
Prometheus Server	Scrapes, indexes, and stores time-series metric databases (TSDB)	`http://<service>:9090/metrics`	Pull-based Metric Collection
Node Exporter	Collects hardware-level host metrics (disk I/O, memory, CPU, kernel)	`http://<node-ip>:9100/metrics`	DaemonSet per Node
Kube-State-Metrics	Listens to the Kubernetes API server and generates cluster object metrics	`http://<service>:8080/metrics`	API Metadata Translation
Grafana	Visual analytics platform rendering dynamic dashboards and graphs	`http://<service>:3000`	Query Interface (PromQL)
Alertmanager	Handles alert deduplication, grouping, rate-limiting, and routing	`http://<service>:9093`	Alert Dispatch Pipeline

Figure 1: Full-stack Kubernetes observability pipeline connecting container pods to the Prometheus engine, Grafana dashboards, and Alertmanager triggers.

Configuring Kubernetes Monitoring with Prometheus and Grafana ensures your platform team eliminates blind spots across both the physical cloud infrastructure nodes and the ephemeral containerized microservices running inside them.

2. Step 1: Deploying Kubernetes Monitoring with Prometheus and Grafana via Helm

Deploying Prometheus manually using individual Kubernetes Deployment and ConfigMap manifests quickly becomes unmaintainable across multi-node production clusters. The enterprise standard approach utilizes the kube-prometheus-stack Helm chart, which bundles the Prometheus Operator, Grafana, Alertmanager, and essential custom resource definitions (CRDs).

2.1 Prepare Storage Class and Namespace

Before configuring Kubernetes Monitoring with Prometheus and Grafana, establish an isolated monitoring namespace and confirm dynamic Persistent Volume Claim (PVC) provisioning support on Azure:

#!/usr/bin/env bash
set -euo pipefail

# 1. Create a dedicated monitoring namespace
kubectl create namespace monitoring --dry-run=client -o yaml | kubectl apply -f -

# 2. Add and update the official Prometheus Community Helm repository
helm repo add prometheus-community [https://prometheus-community.github.io/helm-charts](https://prometheus-community.github.io/helm-charts)
helm repo update

2.2 Configure Production `values.yaml`

To ensure data persistence during node reboots or pod rescheduling, create a custom values file named prometheus-custom-values.yaml to specify persistent storage, retention duration, and secure authentication credentials:

# prometheus-custom-values.yaml
prometheus:
  prometheusSpec:
    retention: 15d
    retentionSize: "40Gi"
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: managed-csi
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi
    resources:
      requests:
        cpu: "500m"
        memory: "1Gi"
      limits:
        cpu: "2000m"
        memory: "4Gi"

grafana:
  enabled: true
  adminPassword: "DevStackSecureAdmin2026!"
  persistence:
    enabled: true
    storageClassName: managed-csi
    size: 10Gi
  service:
    type: ClusterIP

alertmanager:
  enabled: true
  alertmanagerSpec:
    storage:
      volumeClaimTemplate:
        spec:
          storageClassName: managed-csi
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 10Gi

2.3 Execute Helm Deployment

Install the full stack to initialize Kubernetes Monitoring with Prometheus and Grafana using your custom values configuration:

# Deploy the Helm chart into the monitoring namespace
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --values prometheus-custom-values.yaml

Verify that all pods, DaemonSets, and storage claims have transitioned into the Running and Bound state:

kubectl get pods,pvc -n monitoring -o wide

3. Step 2: Accessing the Grafana Visual Dashboard & Prometheus UI

By default, the services deployed by the Prometheus Operator are assigned internal ClusterIP network definitions. This prevents accidental exposure of cluster metrics to the public internet while securing access through local port forwarding or authenticated internal ingress.

3.1 Port-Forwarding Grafana Locally

To securely access the visualization layer for Kubernetes Monitoring with Prometheus and Grafana without exposing a public Azure LoadBalancer:

# Forward port 3000 from the Grafana service to your local machine
kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Open http://localhost:3000 in your browser and enter the administrative credentials:

Username: admin
Password: DevStackSecureAdmin2026! (or the custom password defined in your values file)

3.2 Pre-Loaded Enterprise Dashboards

Establishing Kubernetes Monitoring with Prometheus and Grafana automatically loads pre-configured production dashboards into the interface:

Kubernetes / Compute Resources / Cluster: Provides high-level visibility into total cluster CPU cores, RAM consumption, and network saturation.
Kubernetes / Compute Resources / Namespace (Pods): Pinpoints runaway containers and displays real-time CPU throttling across specific namespaces.
Node Exporter / Use Method / Node: Delivers deep insights into worker node disk read/write IOPS, memory dirty pages, and socket allocations.

4. Step 3: Custom Application Metrics for Kubernetes Monitoring with Prometheus and Grafana

Collecting infrastructure metrics is only half the battle. To monitor microservices, Prometheus uses a declarative Custom Resource Definition called ServiceMonitor. A ServiceMonitor tells Prometheus which Kubernetes services to target, which endpoints to scrape, and the polling frequency.

4.1 Deploying a Sample Microservice with `/metrics` Endpoint

Create an application manifest app-deployment.yaml that exposes metrics in Prometheus format:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-microservice
  namespace: default
  labels:
    app.kubernetes.io/name: api-microservice
    app.kubernetes.io/part-of: devstack-cloud
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-microservice
  template:
    metadata:
      labels:
        app: api-microservice
    spec:
      containers:
        - name: web-api
          image: [mcr.microsoft.com/oss/nginx/nginx:1.21.6](https://mcr.microsoft.com/oss/nginx/nginx:1.21.6)
          ports:
            - name: http
              containerPort: 80
          resources:
            requests:
              cpu: "100m"
              memory: "128Mi"
            limits:
              cpu: "500m"
              memory: "512Mi"
---
apiVersion: v1
kind: Service
metadata:
  name: api-service
  namespace: default
  labels:
    app.kubernetes.io/name: api-microservice
spec:
  type: ClusterIP
  ports:
    - name: http
      port: 80
      targetPort: 80
  selector:
    app: api-microservice

Apply the deployment:

kubectl apply -f app-deployment.yaml

4.2 Creating the `ServiceMonitor` Custom Resource

Define app-servicemonitor.yaml to configure dynamic scraping:

apiVersion: [monitoring.coreos.com/v1](https://monitoring.coreos.com/v1)
kind: ServiceMonitor
metadata:
  name: api-servicemonitor
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: api-microservice
  namespaceSelector:
    matchNames:
      - default
  endpoints:
    - port: http
      path: /metrics
      interval: 15s
      scrapeTimeout: 10s

Apply the ServiceMonitor manifest:

kubectl apply -f app-servicemonitor.yaml

The Prometheus Operator watches for resources with the label release: kube-prometheus-stack, automatically reconfiguring the Prometheus targets without requiring a manual service restart.

5. Step 4: Configuring Prometheus Alerting Rules (`PrometheusRule`)

Configuring automated alert rules is an essential phase when implementing Kubernetes Monitoring with Prometheus and Grafana across production clusters. Rather than managing complex flat configuration files, the Prometheus Operator allows teams to declare alert conditions natively using the PrometheusRule Custom Resource.

Create a file named cluster-alert-rules.yaml:

apiVersion: [monitoring.coreos.com/v1](https://monitoring.coreos.com/v1)
kind: PrometheusRule
metadata:
  name: devstack-cluster-alerts
  namespace: monitoring
  labels:
    release: kube-prometheus-stack
spec:
  groups:
    - name: node-and-workload.rules
      rules:
        - alert: KubernetesPodCrashLooping
          expr: increase(kube_pod_container_status_restarts_total[1h]) > 5
          for: 5m
          labels:
            severity: critical
          annotations:
            summary: "Pod {{ $labels.pod }} is crash looping"
            description: "Pod {{ $labels.pod }} in namespace {{$labels.namespace }} has restarted more than 5 times in the last hour."

        - alert: HighNodeMemorySaturation
          expr: (node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / node_memory_MemTotal_bytes > 0.85
          for: 10m
          labels:
            severity: warning
          annotations:
            summary: "Node memory utilization above 85%"
            description: "Node {{ $labels.instance }} has exceeded 85% memory capacity for over 10 minutes."

        - alert: ContainerCPUThrottlingHigh
          expr: increase(container_cpu_cfs_throttled_periods_total[5m]) / increase(container_cpu_cfs_periods_total[5m]) > 0.25
          for: 5m
          labels:
            severity: warning
          annotations:
            summary: "Container CPU throttling high for {{ $labels.pod }}"
            description: "Pod {{ $labels.pod }} container {{$labels.container }} is experiencing more than 25% CPU throttling."

Apply the alerting manifest:

kubectl apply -f cluster-alert-rules.yaml

A complete setup of Kubernetes Monitoring with Prometheus and Grafana evaluates these expressions against real-time incoming metrics and automatically dispatches alerts to Alertmanager for downstream routing to Webhooks, Slack channels, or PagerDuty schedules.

6. Step 5: Production Hardening, Retention & Scaling Best Practices

Operating Kubernetes Monitoring with Prometheus and Grafana at scale requires adherence to enterprise reliability and security principles:

Implement Remote Storage (Thanos / Cortex): For retention requirements exceeding 30 days, avoid expanding local PVC storage indefinitely. Integrate Thanos or Azure Managed Prometheus to offload cold time-series chunks directly to cheap Azure Blob Storage.
Tune Metric Scrape Intervals: High scrape frequencies (e.g., 5s) increase CPU load on worker nodes. Standardize on 15s for critical production services and 30s–60s for batch or background workloads.
Enforce Resource Quotas on Monitoring Namespace: Prevent monitoring tools from consuming cluster compute during traffic spikes by applying dedicated LimitRanges and ResourceQuotas to the monitoring namespace.
Protect Ingress with Azure Entra ID: Never expose Grafana directly to the public web with basic authentication. Use OAuth2 Proxy or an Ingress Controller configured with Azure Active Directory (Entra ID) Single Sign-On (SSO).

Troubleshooting Common Prometheus & Grafana Issues

When maintaining production-grade Kubernetes Monitoring with Prometheus and Grafana, resolving misconfigurations quickly prevents data gaps and blind spots.

Issue / Symptom	Root Cause	Immediate Remediation
Prometheus Pod Stuck in `Pending`	PVC cannot bind due to Azure CSI storage class mismatch	Verify `storageClassName: managed-csi` in `values.yaml` and check `kubectl describe pvc -n monitoring`.
Custom Metrics Not Appearing	Missing matching release label on `ServiceMonitor`	Ensure `labels: release: kube-prometheus-stack` is present on your `ServiceMonitor` metadata.
Grafana Dashboard Panels Empty	Prometheus datasource connection timeout or incorrect DNS	Verify Grafana datasource URL points to `http://kube-prometheus-stack-prometheus.monitoring:9090`.
High Memory Usage on Prometheus Pod	High metric cardinality caused by dynamic label keys (e.g., user IDs)	Sanitize application metric labels to remove high-cardinality keys before exporting.

Conclusion: Mastering Enterprise Kubernetes Observability

Setting up Kubernetes Monitoring with Prometheus and Grafana shifts operational strategy from reactive debugging to automated, proactive cluster management.By standardizing on Kubernetes Monitoring with Prometheus and Grafana, engineering teams achieve end-to-end cluster observability, minimize production downtime, and scale containerized microservices with complete operational confidence.