DEV Community

Florient DOGBE
Florient DOGBE

Posted on

HPA Not Scaling: Diagnosing metrics-server, Thresholds, and Stabilization Behavior

#kubernetes #openshift #autoscaling #keda

A HorizontalPodAutoscaler stuck at one replica while your pods are visibly saturated is one of the most common — and most misdiagnosed — incidents in a Kubernetes platform. The usual reaction is to lower the CPU target, add replicas manually, or blame "the autoscaler". None of that fixes the root cause.

In practice, an HPA that does not scale almost always falls into one of four buckets: the metrics pipeline is broken, the workload has no resource requests, the target threshold is mathematically unreachable, or the scaling behavior (stabilization window, policies) is deliberately throttling the decision. This article walks through each of them with real commands, then shows how to move past CPU/memory with custom metrics and KEDA.

Context: why the HPA looks broken when it is not

The HPA controller runs inside kube-controller-manager and reconciles every 15 seconds by default (--horizontal-pod-autoscaler-sync-period). On each loop it:

  1. Fetches the current metric value for the pods selected by the target workload's selector.
  2. Computes a desired replica count with a simple ratio.
  3. Applies the behavior policies and stabilization windows.
  4. Writes the result to the scale subresource of the Deployment / DeploymentConfig / StatefulSet.

The formula is intentionally boring:

desiredReplicas = ceil(currentReplicas × (currentMetricValue / desiredMetricValue))
Enter fullscreen mode Exit fullscreen mode

With a 10% tolerance by default: if the ratio is between 0.9 and 1.1, the HPA does nothing at all. That tolerance alone explains a lot of "it should have scaled" tickets.

Every step above can fail silently from the user's point of view. The HPA object stays Active, kubectl get hpa prints something, and nobody reads the conditions. So start there.

Step 1: read the HPA conditions properly

kubectl get hpa is almost useless for diagnosis. It shows a target column that displays <unknown> on failure, but nothing about why.

kubectl get hpa -n payments
# NAME       REFERENCE             TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
# api-hpa    Deployment/api        <unknown>/70%   1         10        1          3h
Enter fullscreen mode Exit fullscreen mode

The real information lives in the conditions and events:

kubectl describe hpa api-hpa -n payments
Enter fullscreen mode Exit fullscreen mode

Focus on three conditions:

Condition Meaning when False
AbleToScale The controller cannot reach the scale subresource, or a stabilization window / cooldown is blocking it
ScalingActive Metrics are missing, unreadable, or the workload has no requests set
ScalingLimited True means the desired count was clamped by minReplicas / maxReplicas

Typical failure messages and what they actually mean:

ScalingActive  False  FailedGetResourceMetric
  unable to get metrics for resource cpu: no metrics returned from resource metrics API
Enter fullscreen mode Exit fullscreen mode

→ metrics-server is absent, unhealthy, or the pods are too young (no metric window yet).

ScalingActive  False  FailedGetResourceMetric
  missing request for cpu in container app of Pod api-7d9...
Enter fullscreen mode Exit fullscreen mode

→ The container has no resources.requests.cpu. A utilization-based HPA is mathematically impossible without a request, because utilization is usage / request.

AbleToScale  True  ReadyForNewScale
ScalingLimited True  TooFewReplicas
  the desired replica count is less than the minimum replica count
Enter fullscreen mode Exit fullscreen mode

→ Nothing is wrong; the HPA simply wants fewer pods than minReplicas.

Also check the structured status, which is easier to parse in automation:

kubectl get hpa api-hpa -n payments -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\t"}{.message}{"\n"}{end}'
Enter fullscreen mode Exit fullscreen mode

Step 2: verify the metrics pipeline

Is the metrics API actually served? 

kubectl get apiservices v1beta1.metrics.k8s.io
# NAME                    SERVICE                      AVAILABLE   AGE
# v1beta1.metrics.k8s.io  openshift-monitoring/...     True        210d

kubectl top nodes
kubectl top pods -n payments
Enter fullscreen mode Exit fullscreen mode

If kubectl top fails, the HPA will never work for resource metrics. On vanilla Kubernetes, metrics-server may simply not be installed:

kubectl -n kube-system get deploy metrics-server
kubectl -n kube-system logs deploy/metrics-server --tail=50
Enter fullscreen mode Exit fullscreen mode

Common metrics-server failure: TLS verification against kubelet serving certificates signed by a cluster-internal CA.

unable to fully scrape metrics: unable to extract connection information for node
x509: cannot validate certificate for 10.0.3.14 because it doesn't contain any IP SANs
Enter fullscreen mode Exit fullscreen mode

The correct fix is to enable kubelet serving certificate rotation and approve the CSRs, not to disable verification. If you must run in a lab, the insecure flag exists — but never in production:

# metrics-server args (lab only for --kubelet-insecure-tls)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: metrics-server
          args:
            - --cert-dir=/tmp
            - --secure-port=10250
            - --kubelet-preferred-address-types=InternalIP,Hostname
            - --metric-resolution=15s
            # - --kubelet-insecure-tls   # NEVER in production
Enter fullscreen mode Exit fullscreen mode

On OpenShift 4, metrics-server is managed by the Cluster Monitoring Operator (it replaced Prometheus Adapter as the resource metrics provider in recent releases). You do not install it yourself. If oc adm top pods fails, investigate the monitoring stack instead:

oc get clusteroperator monitoring
oc -n openshift-monitoring get pods -l app.kubernetes.io/name=metrics-server
oc adm top pods -n payments
Enter fullscreen mode Exit fullscreen mode

Are metrics fresh enough?

The HPA ignores metrics older than --horizontal-pod-autoscaler-cpu-initialization-period (5 min default) for freshly started pods, and treats pods that are not Ready as unknown. A deployment that restarts constantly will keep the HPA in a permanent "unknown metrics" state.

kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/payments/pods" | jq '.items[] | {name:.metadata.name, cpu:.containers[].usage.cpu}'
Enter fullscreen mode Exit fullscreen mode

Step 3: fix requests and thresholds

No request, no utilization HPA 

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: payments
spec:
  replicas: 1
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: app
          image: quay.io/acme/api:1.14.2
          ports:
            - containerPort: 8080
          resources:
            requests:
              cpu: 200m
              memory: 256Mi
            limits:
              memory: 512Mi
Enter fullscreen mode Exit fullscreen mode

Note the deliberate absence of a CPU limit: CFS throttling caps usage below the HPA target and can make a saturated pod look idle. If you must set a CPU limit for quota or compliance reasons, keep it well above the request so throttling does not mask load.

Sidecars poison container-less targets

With type: Resource, the HPA sums requests across all containers in the pod. A service mesh sidecar requesting 100m next to an app requesting 100m halves your apparent utilization. Use type: ContainerResource to scope the metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: ContainerResource
      containerResource:
        name: cpu
        container: app
        target:
          type: Utilization
          averageUtilization: 70
Enter fullscreen mode Exit fullscreen mode

An unreachable threshold

If your app is I/O bound and never exceeds 25% CPU, a 70% target means the HPA will never scale up no matter how slow the responses are. Check the real distribution before choosing a target:

kubectl top pods -n payments --containers --sort-by=cpu
Enter fullscreen mode Exit fullscreen mode

If CPU is not the bottleneck, stop using CPU as the trigger. Jump to the custom metrics section.

Step 4: understand behavior and stabilization windows

This is where "the HPA refuses to scale" is actually correct behavior. Defaults in autoscaling/v2:

  • Scale up: stabilization window 0s, policies allow +100% or +4 pods per 15s, whichever is higher.
  • Scale down: stabilization window 300s, policy allows -100% per 15s.

The scale-down stabilization window makes the controller pick the maximum desired replica count observed over the last 5 minutes. So after a spike, you wait 5 minutes before the first pod is removed. That is intentional and usually right.

The symptom people hit: someone set a long scale-up stabilization window to "avoid flapping", and now scale-up takes minutes.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 2
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      selectPolicy: Max
      policies:
        - type: Percent
          value: 100
          periodSeconds: 15
        - type: Pods
          value: 4
          periodSeconds: 15
    scaleDown:
      stabilizationWindowSeconds: 300
      selectPolicy: Min
      policies:
        - type: Pods
          value: 1
          periodSeconds: 60
Enter fullscreen mode Exit fullscreen mode

Read selectPolicy carefully. selectPolicy: Min on scaleUp is a classic foot-gun: it picks the most restrictive policy, so a Pods: 1 / 60s policy will add one pod per minute even during a traffic surge. And selectPolicy: Disabled on scaleUp turns the HPA into a scale-down-only controller — perfectly valid YAML, completely silent failure mode.

Watch the decisions live:

kubectl get events -n payments --field-selector involvedObject.name=api-hpa --sort-by=.lastTimestamp
# ... SuccessfulRescale  New size: 4; reason: cpu resource utilization above target
Enter fullscreen mode Exit fullscreen mode

Also check for conflicts: a GitOps controller (Argo CD, Flux) that owns spec.replicas in Git will fight the HPA every sync. Either remove replicas from the manifest or set an ignore rule.

# Argo CD Application excerpt
spec:
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas
Enter fullscreen mode Exit fullscreen mode

Step 5: move to custom metrics with KEDA

CPU is a proxy, rarely the actual signal. Queue depth, request rate, p99 latency, or in-flight connections describe saturation far better. KEDA (the Custom Metrics Autoscaler Operator on OpenShift) generates a standard HPA under the hood and feeds it external metrics.

Install on OpenShift via OperatorHub, then create the controller:

apiVersion: keda.sh/v1alpha1
kind: KedaController
metadata:
  name: keda
  namespace: openshift-keda
spec:
  watchNamespace: ""
  operator:
    logLevel: info
  metricsServer:
    logLevel: "0"
Enter fullscreen mode Exit fullscreen mode

A Prometheus-driven ScaledObject scaling on HTTP request rate:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: api-scaler
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicaCount: 2
  maxReplicaCount: 30
  pollingInterval: 15
  cooldownPeriod: 300
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleUp:
          stabilizationWindowSeconds: 0
          policies:
            - type: Percent
              value: 100
              periodSeconds: 15
        scaleDown:
          stabilizationWindowSeconds: 300
          policies:
            - type: Pods
              value: 2
              periodSeconds: 60
  triggers:
    - type: prometheus
      metadata:
        serverAddress: https://thanos-querier.openshift-monitoring.svc:9091
        query: |
          sum(rate(http_requests_total{namespace="payments",service="api"}[2m]))
        threshold: "50"
        namespace: payments
      authenticationRef:
        name: keda-thanos-auth
Enter fullscreen mode Exit fullscreen mode

Authentication to Thanos uses a dedicated ServiceAccount token, never a static credential:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: keda-thanos
  namespace: payments
---
apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: keda-thanos-auth
  namespace: payments
spec:
  boundServiceAccountToken:
    - parameter: bearerToken
      serviceAccountName: keda-thanos
Enter fullscreen mode Exit fullscreen mode

Grant the minimum read access:

oc adm policy add-cluster-role-to-user cluster-monitoring-view \
  -z keda-thanos -n payments
Enter fullscreen mode Exit fullscreen mode

Verify KEDA produced a working HPA:

oc get scaledobject api-scaler -n payments
oc get hpa keda-hpa-api-scaler -n payments
oc describe scaledobject api-scaler -n payments | grep -A10 Conditions
oc -n openshift-keda logs deploy/keda-operator --tail=100 | grep -i error
Enter fullscreen mode Exit fullscreen mode

A frequent KEDA gotcha: with minReplicaCount: 0, the ScaledObject scales to zero and the underlying HPA is removed while idle. If you then run kubectl get hpa and see nothing, that is scale-to-zero working, not a bug.

Never combine a hand-written HPA and a ScaledObject on the same Deployment — two controllers writing the same scale subresource produce oscillation. KEDA refuses with ScaledObject doesn't have correct scaleTargetRef or reports ownership conflicts.

Security best practices

  • Scope RBAC for metric readers. cluster-monitoring-view is cluster-wide read on metrics. Prefer a namespaced Role when your Prometheus setup allows it, and always use a dedicated ServiceAccount per ScaledObject rather than default.
  • Prefer bound ServiceAccount tokens (boundServiceAccountToken in TriggerAuthentication) over long-lived Secrets. They are short-lived, audience-scoped, and auto-rotated. If an external system (SQS, Kafka SASL) forces a static credential, store it in a Secret referenced by TriggerAuthentication and rotate it, or use a Secrets Store CSI driver.
  • Never disable kubelet TLS verification to "fix" metrics-server. Enable RotateKubeletServerCertificate and approve the serving CSRs instead; --kubelet-insecure-tls lets anything on the node network impersonate a kubelet and feed your autoscaler fabricated metrics.
  • Treat maxReplicas as a blast-radius control. An autoscaler driven by an attacker-influenced metric (public request rate, unauthenticated queue) is a cost-amplification vector. Pair maxReplicas with a ResourceQuota and a LimitRange in the namespace.
  • Protect the metric source. If a ScaledObject queries a Prometheus endpoint reachable without auth, anyone able to push series can drive your scaling. Query Thanos Querier with the tenancy port (9092) when you need per-namespace enforcement.
  • Audit HPA mutations. kubectl edit hpa raising maxReplicas from 10 to 500 is a legitimate-looking change with a real financial impact. Keep HPAs and ScaledObjects in Git with review, and disable direct edits via RBAC in production namespaces.

Conclusion: key takeaways

An HPA pinned at one replica is a diagnosis problem, not an autoscaling problem. Work the chain in order:

  1. Read the conditions, not the TARGETS column. ScalingActive=False with FailedGetResourceMetric points to metrics; ScalingLimited=True points to your bounds.
  2. Validate the metrics pipeline with kubectl top. No top, no HPA. On OpenShift, that means checking the monitoring cluster operator, not installing metrics-server by hand.
  3. Set CPU requests, and scope the metric with ContainerResource when sidecars are present. Beware CPU limits that throttle a pod into looking idle.
  4. Audit behavior. Default scale-down stabilization is 300s by design. A non-zero scaleUp.stabilizationWindowSeconds, a selectPolicy: Min, or Disabled is usually the actual culprit.
  5. Check for an HPA/GitOps conflict on spec.replicas, and never run two controllers against one scale subresource.
  6. Graduate to KEDA when CPU is not the bottleneck — queue depth and request rate are honest signals, and KEDA still gives you the full behavior block through advanced.horizontalPodAutoscalerConfig.

Finally, test it. Generate load, watch kubectl get hpa -w and the SuccessfulRescale events, and record how long the first new pod takes to become Ready. An autoscaler you have never observed scaling is an autoscaler you cannot trust during an incident.

Top comments (0)