Get kube-apiserver’s metrics manually

Posted on by Yuki Nakamura

TL;DR

In this article, I will explain how to get kube-apiserver’s metrics via a curl command from a pod via the following command.

curl \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  \
https://kubernetes.default.svc/metrics
  1. Monitoring a Kubernetes cluster with Prometheus
  2. Preparation
    1. Kubernetes cluster
    2. kube-apiserver Pod and Service
  3. Get kube-apiserver’s metrics
    1. Create a ClusterRole and a ServiceAccount
    2. Create a Pod from which we will send requests
    3. Send requests to kube-apiserver
      1. Get a shell of the running pod
      2. Send requests to kube-apiserver via the service named kubectl
    4. Sample Metrics
      1. Number of metrics
      2. Latency
      3. Kubernetes features’ enablement
    5. Delete Kubernetes objects
  4. Wrap up
    1. References

Monitoring a Kubernetes cluster with Prometheus

Monitoring a Kubernetes cluster with Prometheus is useful for building dashboards and alerts. However, not many DevOps engineers may understand how Prometheus gets metrics from a Kubernetes Cluster. So let me explain the mechanism!

Kubernetes components emit metrics in Prometheus format via HTTP endpoints, from which Prometheus scrapes metrics.

Example of Kubernetes components that emit metrics:

  • kube-apiserver
  • kube-scheduler
  • kube-controller-manager
  • kube-proxy
  • kubelet
The components of a Kubernetes cluster

Preparation

Kubernetes cluster

Please prepare Kubernetes that you can use freely for learning. Here’s my Kubernetes’ version.

kubectl version
Client Version: v1.28.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.2

kube-apiserver Pod and Service

Please check if kube-apiserver is running in the kube-system namespace as a pod, and it’s exposed through a service named kubernetes in the default namespace.

kubectl get svc kubernetes -n default                                        
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   10.96.0.1    <none>        443/TCP   2d11h

kubectl get svc kubernetes -n default -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2023-10-07T02:39:40Z"
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "191"
  uid: e3e615c4-ca53-41bc-8255-a6208f76a277
spec:
  clusterIP: 10.96.0.1
  clusterIPs:
  - 10.96.0.1
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 6443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Get kube-apiserver’s metrics

We will send requests to kube-apiserver from a pod that has the permission to access metrics api.

Access kube-apiserver’s metrics api from a pod

Create a ClusterRole and a ServiceAccount

Create a ClusterRole and a ServiceAccount that can access metrics api of kube-apiserver.

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-role
rules:
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metrics-scraper-sa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics-role
subjects:
- kind: ServiceAccount
  name: metrics-scraper-sa
  namespace: default
EOF

Create a Pod from which we will send requests

Create a pod that assumes ServiceAccount you just created above.

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: metrics-scraper
  namespace: default
spec:
  serviceAccount: metrics-scraper-sa
  containers:
  - command:
    - tail
    - -f
    - /dev/null
    image: alpine/curl
    name: metrics-scraper
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
EOF

Send requests to kube-apiserver

Get a shell of the running pod

kubectl exec -it metrics-scraper -- sh

Send requests to kube-apiserver via the service named kubectl

Send a request with cacert and Bearer token that was injected the pod.

curl \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  \
https://kubernetes.default.svc/metrics

You can see metrics of kube-apiserver in a Prometheus format!

Sample Metrics

Let’s take a look of some metrics of kube-apiserver. For more information, please see the official document.

Number of metrics

You can check number of metrics of kube-apiserver.

curl \
--silent \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  \
https://kubernetes.default.svc/metrics | grep -v '^#' | wc -l

19148

Latency

You can check Response latency distribution in seconds for each verb, dry run value, group, version, resource, subresource, scope and component.

curl \
--silent \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  \
https://kubernetes.default.svc/metrics | grep apiserver_request_duration_seconds_bucket | head -5

apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.005"} 6
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.025"} 6
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.05"} 6
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.1"} 6
apiserver_request_duration_seconds_bucket{component="",dry_run="",group="",resource="",scope="",subresource="/healthz",verb="GET",version="",le="0.2"} 6

Kubernetes features’ enablement

You can check the data about the stage and enablement of a k8s feature.

curl \
--silent \
--cacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
-H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"  \
https://kubernetes.default.svc/metrics | grep kubernetes_feature_enabled | head -5 

# HELP kubernetes_feature_enabled [BETA] This metric records the data about the stage and enablement of a k8s feature.
# TYPE kubernetes_feature_enabled gauge
kubernetes_feature_enabled{name="APIListChunking",stage="BETA"} 1
kubernetes_feature_enabled{name="APIPriorityAndFairness",stage="BETA"} 1
kubernetes_feature_enabled{name="APIResponseCompression",stage="BETA"} 1

Delete Kubernetes objects

For cleaning up, please delete Kubernetes objects you created in this tutorial.

kubectl delete pod metrics-scraper
kubectl delete sa metrics-scraper-sa
kubectl delete clusterrolebindings metrics-role-binding
kubectl delete clusterrole metrics-role

Wrap up

We get kube-apiserver’s metrics via a curl command from a pod. It should now be clear how Prometheus gets the apiserver metrics!

References

Unknown's avatar

Published by Yuki Nakamura