Unable to get CPU for container discarding data: missing cpu usage metric

[khteh]

I added the following to the 1.8+ manifest and deploy it to EKS.

imagePullPolicy: Always
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP

Yet, I see the following errors:

I0123 02:04:30.824820       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0123 02:04:31.420786       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0123 02:04:31.448142       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/01/23 02:04:31 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/01/23 02:04:31 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0123 02:04:31.493567       1 serve.go:96] Serving securely on [::]:443
E0123 02:04:34.290945       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-1: no metrics known for pod
E0123 02:04:34.290964       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 02:05:04.301881       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-1: no metrics known for pod
E0123 02:05:04.301998       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 03:11:31.654059       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-10-0-1-40.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-nlp" in pod default/iconverse-nlp-1 on node "10.0.1.40", discarding data: missing cpu usage metric, unable to fully scrape metrics from source kubelet_summary:ip-10-0-0-168.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-converse" in pod default/iconverse-converse-0 on node "10.0.0.168", discarding data: missing cpu usage metric]
E0123 03:11:35.994020       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 03:12:06.018005       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-converse-0: no metrics known for pod
E0123 04:31:31.563306       1 manager.go:102] unable to fully collect metrics: unable to fully scrape metrics from source kubelet_summary:ip-10-0-2-172.ap-southeast-1.compute.internal: unable to get CPU for container "iconverse-nlp" in pod default/iconverse-nlp-0 on node "10.0.2.172", discarding data: missing cpu usage metric

$ k --kubeconfig=$KUBECONFIG get hpa
NAME                 REFERENCE                        TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
iconverse-converse   StatefulSet/iconverse-converse   <unknown>/50%   2         5         2          4h
$ k --kubeconfig=$KUBECONFIG describe hpa iconverse-converse
Error from server (NotFound): the server could not find the requested resource

This is my hpa manifest:

apiVersion: autoscaling/v1
kind: HorizontalPodAutoscaler
metadata:
  name: myapplication
  namespace: default 
spec:
  maxReplicas: 5
  minReplicas: 2
  scaleTargetRef:
    apiVersion: apps/v1 
    kind: StatefulSet
    name: myapplication
  metrics:
  - type: Resource
    resource:
      name: cpu
      targetAverageUtilization: 50
  - type: Resource
    resource:
      name: memory
      targetAverageUtilization: 50

Any insight and advice is appreciated.

[wang510918]

maybe you didnt provide ca.crt and ca.key to the node of your cluster

[khteh]

Can you provide more details please? Thanks.

[mabushey]

There are 200 forks, does anyone know which (if any) of them fix metric server?

[zouyee]

#212 (comment)

[khteh]

Thanks @zouyee. I changed from using "args" to "commands" and I don't see the 401 Unauthorized now. However, kubectl logs -f metrics-server... -n kube-system still shows "no metrics known for pod":

$ k logs -f metrics-server-68df9fbc9f-dgr8v -n kube-system
I0312 03:55:41.841800       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
W0312 03:55:42.433339       1 authentication.go:166] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
W0312 03:55:42.439873       1 authentication.go:210] cluster doesn't provide client-ca-file in configmap/extension-apiserver-authentication in kube-system, so client certificate authentication to extension api-server won't work.
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/03/12 03:55:42 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0312 03:55:42.488139       1 serve.go:96] Serving securely on [::]:443
E0312 03:55:46.554516       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-0: no metrics known for pod
E0312 03:55:46.554540       1 reststorage.go:144] unable to fetch pod metrics for pod default/iconverse-nlp-1: no metrics known for pod
  E0312 05:08:42.634201       1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:ip-192-168-84-18.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-connector" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-connector-0 on node "192.168.84.18", discarding data: missing cpu usage metric], unable to fully scrape metrics from source kubelet_summary:ip-192-168-22-244.ap-southeast-1.compute.internal: [unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-converse" in pod default/iconverse-converse-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-fluentd" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-admin" in pod default/iconverse-admin-0 on node "192.168.22.244", discarding data: missing cpu usage metric, unable to get CPU for container "iconverse-ui" in pod default/iconverse-ui-0 on node "192.168.22.244", discarding data: missing cpu usage metric]]

[NKUCodingCat]

Same here in K8s 1.13.5

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl version
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.3", GitCommit:"721bfa751924da8d1680787490c54b9179b1fed0", GitTreeState:"clean", BuildDate:"2019-02-01T20:08:12Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl logs metrics-server-68d85f76bb-8bb9p -n kube-system
I0505 02:06:44.758400       1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)
[restful] 2019/05/05 02:06:45 log.go:33: [restful/swagger] listing is available at https://:443/swaggerapi
[restful] 2019/05/05 02:06:45 log.go:33: [restful/swagger] https://:443/swaggerui/ is mapped to folder /swagger-ui/
I0505 02:06:45.133211       1 serve.go:96] Serving securely on [::]:443
E0505 02:07:14.671030       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.je5voly0": no metrics known for node
E0505 02:07:14.671049       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.t2mzj8lv": no metrics known for node
E0505 02:07:14.671053       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.4q1e2xzl": no metrics known for node
E0505 02:07:14.671056       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.4iqiwkgz": no metrics known for node
E0505 02:07:14.671060       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.vrdnkjkv": no metrics known for node
E0505 02:07:14.671063       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.kkvwkpqv": no metrics known for node
E0505 02:07:14.671066       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.3wjheffw": no metrics known for node
E0505 02:07:14.671069       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.hxqlgl0o": no metrics known for node
E0505 02:07:14.671072       1 reststorage.go:129] unable to fetch node metrics for node "kube-worker.r85r2n3n": no metrics known for node
E0505 02:07:19.380532       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-gl8v6: no metrics known for pod
E0505 02:07:19.380547       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-s6bzs: no metrics known for pod
E0505 02:07:19.380551       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-launcher-1-6b4bd7d8cc-nxpvn: no metrics known for pod
E0505 02:07:19.380554       1 reststorage.go:144] unable to fetch pod metrics for pod default/kser-se-wrapper-1-64c978ccbc-q5btm: no metrics known for pod

cjingtao-a01:metrics-server-0.3.2 cjingtao$ KUBECONFIG=/Users/cjingtao/Documents/GitHub/deploy/Server-End\ Complex/dev/Dev-CD.kubeconfig.yml kubectl get pods metrics-server-68d85f76bb-8bb9p -n kube-system -o yaml
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2019-05-05T02:06:41Z"
  generateName: metrics-server-68d85f76bb-
  labels:
    k8s-app: metrics-server
    pod-template-hash: 68d85f76bb
  name: metrics-server-68d85f76bb-8bb9p
  namespace: kube-system
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: metrics-server-68d85f76bb
    uid: 6c661bc4-6eda-11e9-a6de-020061cb48ef
  resourceVersion: "5736885"
  selfLink: /api/v1/namespaces/kube-system/pods/metrics-server-68d85f76bb-8bb9p
  uid: 6c6927a2-6eda-11e9-a6de-020061cb48ef
spec:
  containers:
  - command:
    - /metrics-server
    - --kubelet-insecure-tls
    - --kubelet-preferred-address-types=InternalIP
    image: k8s.gcr.io/metrics-server-amd64:v0.3.1
    imagePullPolicy: Always
    name: metrics-server
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /tmp
      name: tmp-dir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: metrics-server-token-nrdwl
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: kube-worker.t2mzj8lv
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: metrics-server
  serviceAccountName: metrics-server
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: tmp-dir
  - name: metrics-server-token-nrdwl
    secret:
      defaultMode: 420
      secretName: metrics-server-token-nrdwl
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:41Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:44Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:44Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-05-05T02:06:41Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://d49d30c912365bca3fb467f38a8e79a1c13c1f61d0a3edcc136d8ef6a5d7555e
    image: k8s.gcr.io/metrics-server-amd64:v0.3.1
    imageID: docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:78938f933822856f443e6827fe5b37d6cc2f74ae888ac8b33d06fdbe5f8c658b
    lastState: {}
    name: metrics-server
    ready: true
    restartCount: 0
    state:
      running:
        startedAt: "2019-05-05T02:06:44Z"
  hostIP: 10.92.101.22
  phase: Running
  podIP: 10.36.0.5
  qosClass: BestEffort
  startTime: "2019-05-05T02:06:41Z"

[tx19980520]

my yaml :

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-server
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: metrics-server
  namespace: kube-system
  labels:
    k8s-app: metrics-server
spec:
  selector:
    matchLabels:
      k8s-app: metrics-server
  template:
    metadata:
      name: metrics-server
      labels:
        k8s-app: metrics-server
    spec:
      serviceAccountName: metrics-server
      volumes:
      # mount in tmp so we can safely use from-scratch images and/or read-only containers
      - name: tmp-dir
        emptyDir: {}
      containers:
      - name: metrics-server
        image: registry.cn-shanghai.aliyuncs.com/rancher/metrics-server:v0.3.3
        imagePullPolicy: Always
        command:
        - /metrics-server
        - --kubelet-insecure-tls
        - --kubelet-preferred-address-types=InternalIP
        volumeMounts:
        - name: tmp-dir
          mountPath: /tmp

It's same

E0730 01:53:08.101704       1 reststorage.go:147] unable to fetch pod metrics for pod istio-system/istio-pilot-58974b9b89-jx4vw: no metrics known for pod
E0730 01:53:08.105228       1 reststorage.go:147] unable to fetch pod metrics for pod istio-system/istio-egressgateway-5bfd96bbb8-8lq9q: no metrics known for pod
E0730 01:53:08.108672       1 reststorage.go:147] unable to fetch pod metrics for pod istio-system/istio-ingressgateway-64ccb6df5b-2jkz8: no metrics known for pod

I run top:

Error from server (NotFound): podmetrics.metrics.k8s.io "default/curl-test-774fdd546c-62gs4" not found

I have used the command mentioned in v0.3.1, but not work and who can tell me how to let pod export metrics, like springboot actuator?

same issue in EKS cluster with version 1.13

[pabclsn]

same here

[tfluehmann]

same here:

... discarding data: missing cpu usage metric, unable to get CPU for container "xyz" on node "1.1.1.1", discarding data: missing cpu usage metric]

[kmdhali]

did u get the workaround ? I have the same issue, nothing worked so far .."unable to fetch pod metrics for pod kube-system/kube-scheduler-ubuntu03: no metrics known for pod"

[haimari]

same here:

unable to fetch pod metrics for pod kube-system/kube-dns-64b69465b4-msj4m: no metrics known for pod

[serathius]

Looked through all logs provided in this issue. Looks like the problem is missing cpu metric in summary API exposed by Kubelet. This could be caused by two things:

• Fresh container (created < 15s ago), that cAdvisor didn't have chance to calculate CPU jet.
• Kubelet failing to expose metric

Please provide me with:

• K8s version
• Kubernetes distro (EKS, GKE, kubeadm)
• Name of pod that metric is missing
• Results from running kubectl get --raw /api/v1/nodes/<node-name>/proxy/stats/summary with node that the pod is currently running