k8s practice: heapster+influxdb+grafana to implement kubernetes cluster monitoring

Environmental Instructions:

host name Operating System Version ip docker version kubelet version To configure Remarks
master Centos 7.6.1810 Docker 18.09.6 V1.14.2 2C2G master host
node01 Centos 7.6.1810 Docker 18.09.6 V1.14.2 2C2G node Node
node02 Centos 7.6.1810 Docker 18.09.6 V1.14.2 2C2G node Node


The k8s cluster deployment is detailed in: Centos7.6 Deploy k8s(v1.14.2) Cluster
k8s learning materials are detailed in: Basic concepts, kubectl commands, and data sharing


1. Introduction

1. Introduction to heapster

Heapster is a native cluster monitoring scheme for Kubernetes, and Kubelet itself includes an agent called cAdvisor that collects resource consumption for all individual containers running on the entire node and on the node.Heapster runs on a node as a pod, exposing the service through a common KubernetesService, enabling external access through a stable IP address.It collects data from all cAdvisors in the cluster and exposes it through a single address.

Heapster groups data by Pod, stores them in a preconfigured backend, and visualizes them.Heapster currently supports backends such as InfluxDB (shown through Grafana), Google Cloud Monitoring, and so on.

2. Introduction to lnfluxDB and Grafana

lnfluxDB is an open source temporal database for storing application metrics and other monitoring data.Grafana is an open source data analysis and visualization suite with a gorgeous web console that allows users to visualize data stored in InfluxDB and discover how application resource usage changes over time.

2. heapster Installation

1. Install files and mirror Downloads

Installation File Download:

[root@master ~]# git clone https://github.com/kubernetes-retired/heapster.git


[root@master ~]# wget https://github.com/kubernetes-retired/heapster/archive/master.zip
[root@master ~]# unzip master.zip 

You can download the installation file in either way. This article takes the second way

Mirror Download and Label

[root@node02 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-grafana-amd64:v5.0.4
[root@node02 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-influxdb-amd64:v1.5.2
[root@node02 ~]# docker pull registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4

[root@node02 ~]# docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4 k8s.gcr.io/heapster-amd64:v1.5.4 
[root@node02 ~]# docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-influxdb-amd64:v1.5.2 k8s.gcr.io/heapster-influxdb-amd64:v1.5.2
[root@node02 ~]# docker image tag registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-grafana-amd64:v5.0.4 k8s.gcr.io/heapster-grafana-amd64:v5.0.4

[root@node02 ~]# docker rmi registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-amd64:v1.5.4 registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-influxdb-amd64:v1.5.2 registry.cn-hangzhou.aliyuncs.com/google_containers/heapster-grafana-amd64:v5.0.4

Note: Each node needs to execute the above commands

2. Installation File Modification

[root@master ~]# cd heapster-master/deploy/kube-config/
[root@master kube-config]# pwd
[root@master kube-config]# ll
//Total dosage 0
drwxr-xr-x 2 root root  27 11 Month 30 2018 google
drwxr-xr-x 2 root root  68 10 February 2215:00 influxdb
drwxr-xr-x 2 root root  32 10 February 2215:02 rbac
drwxr-xr-x 2 root root  38 11 Month 30 2018 standalone
drwxr-xr-x 2 root root 170 11 Month 30 2018 standalone-test
drwxr-xr-x 2 root root 145 11 Month 30 2018 standalone-with-apiserver
[root@master kube-config]# cd influxdb/
[root@master influxdb]# ll
//Total usage 12
-rw-r--r-- 1 root root 2294 10 February 2214:51 grafana.yaml
-rw-r--r-- 1 root root 1162 10 February 2215:00 heapster.yaml
-rw-r--r-- 1 root root  997 10 February 2214:51 influxdb.yaml
[root@master influxdb]# cd ../rbac/
[root@master rbac]# ll
//Total usage 4
-rw-r--r-- 1 root root 263 10 February 2215:02 heapster-rbac.yaml

Modify the files grafana.yaml, influxdb.yaml, heapster.yaml, and heapster-rbac.yaml, respectively

Modify grafana.yaml, port type is NodePort, nodePort is 30011, accessible at http://NodeIp:30011

Modify influxdb.yaml, port type is NodePort, nodePort is 30012, grafana configuration data source will use

Modify the source and sink parameters in heapster.yaml

Source:Specify the data acquisition source

Sorce parameter Explain
inClusterConfig Use Kube config in the service account associated with heapster's namespace (default: true)
kubeletPort Specifies the port to use for kubelet, default 10255
kubeletHttps Whether to use https to connect to kubelets (default: false)
insecure Whether to use a security certificate (default: false)
auth Security certification
useServiceAccount Whether to use K8S security token (default: false)

sink: Specify the backend data store

sink parameter Explain
user InfluxDB user, default root
pw InfluxDB password, default root
db Database name, default k8s
retention Default duration of infloxDB retention policy, default value 0, infinite
secure Secure connection to InfluxDB (default: false)
insecuressl Ignore SSL certificate validity (default: false)
withfields Use InfluxDB fields (default: false)
cluster_name Cluster name for different cubernete clusters (default)
disable_counter_metrics Disable receive counter metrics to flow into the database (default: false)
concurrency Number of concurrencies (default: 1)

Modify heapster-rbac.yaml to cluster-admi

3. Perform installation

[root@master kube-config]# pwd
[root@master kube-config]# kubectl apply -f influxdb/
deployment.extensions/monitoring-grafana created
service/monitoring-grafana created
serviceaccount/heapster created
deployment.extensions/heapster created
service/heapster created
deployment.extensions/monitoring-influxdb created
service/monitoring-influxdb created
[root@master kube-config]# kubectl apply -f rbac/heapster-rbac.yaml 
clusterrolebinding.rbac.authorization.k8s.io/heapster created

4. Resource Viewing

[root@master kube-config]# kubectl get all -n kube-system -o wide |grep -e monitor -e heapster    

3. Grafana Configuration

1. Sign in to grafana

Landing address:

2. Configure DataSource

url is

3. Import Templates

Template Download

Download address: https://grafana.com/api/dashboards/3649/revisions/1/download ,



Similarly import kubernetes-node-statistics

4. View Grafana

V. Resource Deletion

[root@master ~]# kubectl delete -n kube-system  ClusterRoleBinding heapster               [root@master ~]# kubectl get all -n kube-system -o wide |grep -e monitor -e heapster |awk '{print $1}'|xargs kubectl delete  -n kube-system
[root@master ~]# rm -rf heapster-master master.zip 

All profiles in this article have been uploaded to github: https://github.com/loong576/heapster-influxdb-grafana/archive/master.zip

Tags: Linux InfluxDB Docker Kubernetes github

Posted on Fri, 08 Nov 2019 19:13:10 -0500 by chewydence