IBM Cloud Kubernetes Service - k8s external cloud storage performance test

There are many ways to store k8s data: emptyDir (temporary storage), hostpath (semi persistent storage), persistent storage (file, block, object storage, etc.). Almost all public cloud vendors support cloud storage service integration k8s hosting. The k8s system itself adopts two concepts, persistent volume (PV) and Persistent volume claim (PVC) implements the life cycle management of storage persistence (application, reservation, deletion, discarding, etc.). Pod applies for data volume to the cluster system and uses it according to the demand. If the pod is released and the application disappears, the data volume will be separated and unbound accordingly, and the data will remain. In use, pod and the application have no awareness of the storage back-end infrastructure. This design is very good to understand the application and data. As for the bottom layer of storage resources, ceph, portwx, elastic cloud storage, cloud object storage and so on are selected, which depends on the specific business needs and the support of manufacturers.

Let's cut into today's topic. Since there are many back-end options, users are trying a k8s cluster product. Storage persistence and application level read-write ability are a must. Here we take ibm cloud kubernetes service as an example to see three common test methods and performance comparison of external storage on IBM cloud.

It is roughly divided into three steps:

  1. Create K8S cluster and use it through kubectl connection (deployed in Tokyo, Japan)
  2. Confirm and install IBM cloud storage k8s supply plug-ins (CFS, CBS, COS deployed in Tokyo)
  3. dbench(fio) tests the above three storage volumes

1. k8s cluster creation, connection and use
land cloud.ibm.com Search for the keyword "kubernetes" in the catalog and enter the service creation wizard. The standard tariff will be used in this test (there is only one worker node in the free cluster, which cannot meet the test requirements). It can be noted that the online version of ibm k8s is very new. Select the latest stable version 1.17.6

Next, select the classic infrastructure (covering the whole). vpc infrastructure is also fully ready in the six regions (Tokyo, Sydney, Frankfurt, London, Washington, Dallas). Considering the worker cost of vpc high availability area cluster, it will be more cost-effective to open a set of classic test in a single region.

Finally, we set up the worker pool, and select three 2c*4g virtual machines as the working nodes. If you want to be safe, you can turn on the node disk encryption (ibm has nothing to say), define the cluster name, and click create

About five or six minutes later, the cluster initialization is completed, and it enters the "Normal" state

Click to create a cluster and enter "access". Refer to the steps to install IBM cloud cli on your notebook (k8s related tools such as kubectl and docker will be installed together if the current system does not have them)










danws-MacBook-Pro:k8s-playground danw$ ibmcloud login --sso -a cloud.ibm.com -r jp-tok -g danw-group
danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks cluster config --cluster bro62jit0fjj42dqsh7g
danws-MacBook-Pro:k8s-playground danw$ export KUBECONFIG=/Users/danw/.bluemix/plugins/container-service/clusters/bro62jit0fjj42dqsh7g/kube-config-tok05-demo-k8s-tok05.yml

You can also add the cluster configuration file to ~ /. bash_profile,

Test connection succeeded!

danws-MacBook-Pro:k8s-playground danw$ k get nodes -o wide
NAME           STATUS   ROLES    AGE   VERSION       INTERNAL-IP    EXTERNAL-IP     OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
10.193.33.31   Ready    <none>   26m   v1.17.6+IKS   10.193.33.31   165.192.96.2    Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4
10.193.33.33   Ready    <none>   26m   v1.17.6+IKS   10.193.33.33   165.192.96.12   Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4
10.193.33.44   Ready    <none>   27m   v1.17.6+IKS   10.193.33.44   165.192.96.7    Ubuntu 18.04.4 LTS   4.15.0-101-generic   containerd://1.3.4

2. Install the IBM Cloud storage plug-in
a. IBM Cloud File Storage
The default file reload class (storage plug-in) is pre installed during cluster creation, and can be used directly

danws-MacBook-Pro:k8s-playground danw$ k get storageclasses | grep file
default                    ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-bronze           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-bronze-gid       ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-custom           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-gold (default)   ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-gold-gid         ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-retain-bronze    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-custom    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-gold      ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-retain-silver    ibm.io/ibmc-file   Retain          Immediate           false                  37m
ibmc-file-silver           ibm.io/ibmc-file   Delete          Immediate           false                  37m
ibmc-file-silver-gid       ibm.io/ibmc-file   Delete          Immediate           false                  37m

Here we choose "silver" (hourly, 4 IOPS/GB)

danws-MacBook-Pro:k8s-playground danw$ kubectl describe storageclass ibmc-file-bronze
Name:            ibmc-file-bronze
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"ibmc-file-bronze"},"parameters":{"billingType":"hourly","classVersion":"2","iopsPerGB":"2","sizeRange":"[20-12000]Gi","type":"Endurance"},"provisioner":"ibm.io/ibmc-file","reclaimPolicy":"Delete"}

Provisioner:           ibm.io/ibmc-file
Parameters:            billingType=hourly,classVersion=2,iopsPerGB=2,sizeRange=[20-12000]Gi,type=Endurance
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

b. IBM Cloud Block Storage
By default, the "block storage" plug-in is not pre installed in the cluster. You need to install it first. For official online documents, please refer to the specific steps.

https://cloud.ibm.com/docs/containers?topic=containers-block_storage#install_block

First confirm the status of cluster work nodes

danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks worker ls --cluster demo-k8s-tok05
OK
ID                                                       Public IP       Private IP     Flavor              State    Status   Zone    Version
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000001b3   165.192.96.7    10.193.33.44   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-0000022e   165.192.96.2    10.193.33.31   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527
kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000003fe   165.192.96.12   10.193.33.33   u3c.2x4.encrypted   normal   Ready    tok05   1.17.6_1527

Block storage and subsequent COS plug-in installation need to rely on helm chart. Here we first install helm v3. Here, the helm installation process has been omitted. Please refer to the following connection:
https://github.com/helm/helm/releases/tag/v3.2.4

danws-MacBook-Pro:k8s-playground danw$ helm version
version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"}

danws-MacBook-Pro:k8s-playground danw$ helm repo add iks-charts https://icr.io/helm/iks-charts
"iks-charts" has been added to your repositories


danws-MacBook-Pro:k8s-playground danw$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "istio.io" chart repository
...Successfully got an update from the "iks-charts" chart repository
Update Complete. ⎈ Happy Helming!⎈

danws-MacBook-Pro:k8s-playground danw$ helm search repo iks-charts
NAME                                    	CHART VERSION	APP VERSION	DESCRIPTION
iks-charts/ibm-block-storage-attacher   	1.1.4        	           	A Helm chart for installing ibmcloud block stor...
iks-charts/ibm-iks-cluster-autoscaler   	1.1.7        	           	A Helm chart for installing the IBM Cloud clust...
iks-charts/ibm-object-storage-plugin    	1.0.7        	1.0.7      	A Helm chart for installing ibmcloud object sto...
iks-charts/ibm-worker-recovery          	1.10.71      	           	IBM Autorecovery system allows automatic recove...
iks-charts/ibmcloud-alb-metrics-exporter	1.0.25       	25.0       	A Helm chart to deploy IBM Cloud ALB metrics ex...
iks-charts/ibmcloud-backup-restore      	1.0.1        	           	A Helm chart for taking backup of pvc data and ...
iks-charts/ibmcloud-block-storage-plugin	1.7.0        	           	A Helm chart for installing ibmcloud block stor...
iks-charts/ibmcloud-data-shield         	1.18.731     	1.0        	Protect data in use with IBM Cloud Data Shield
iks-charts/ibmcloud-image-enforcement   	0.2.10       	           	A Helm chart to install IBM Container Image Sec...
iks-charts/strongswan                   	2.6.3        	20.05.15   	A strongSwan IPSec VPN service to securely conn...

danws-MacBook-Pro:k8s-playground danw$ helm install 1.7.0 iks-charts/ibmcloud-block-storage-plugin -n default

danws-MacBook-Pro:k8s-playground danw$ kubectl get pods -n kube-system | grep ibmcloud-block-storage
ibmcloud-block-storage-driver-792wk                   1/1     Running   0          62s
ibmcloud-block-storage-driver-m4phr                   1/1     Running   0          62s
ibmcloud-block-storage-driver-n9pxv                   1/1     Running   0          62s
ibmcloud-block-storage-plugin-665f4b5fb5-4fn9t        1/1     Running   0          61s

danws-MacBook-Pro:k8s-playground danw$ kubectl get storageclasses | grep block
ibmc-block-bronze          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-custom          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-gold            ibm.io/ibmc-block   Delete          Immediate           true                   2m50s
ibmc-block-retain-bronze   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-custom   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-gold     ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-retain-silver   ibm.io/ibmc-block   Retain          Immediate           true                   2m50s
ibmc-block-silver          ibm.io/ibmc-block   Delete          Immediate           true                   2m50s

Note: in the next part, we will discuss COS pvc

3. Using dbench to test the two types of pvc
github is an open source container for fio test image, which can be used directly. clone repo first

danws-MacBook-Pro:dbench danw$ git clone https://github.com/leeliu/dbench.git

⚠️ : here is a small thunder. dbench image is currently moved to this repo "ndrpnt/dbench:1.0.0 "

Please refer to the following link for the script file content of fio,
https://github.com/leeliu/dbench/blob/master/docker-entrypoint.sh
Although the script has been packed into the image, users can adjust some parameters through the pod environment variables, such as quick test, test file size, data offset, etc.

edit dbench.yaml Because of the limited computing resources of the three worker nodes (2c4g) in the cluster, we are worried that the simultaneous execution of fio test by the three kinds of storage will affect the I/O efficiency of each other, so we can change yaml to mount pvc separate tests of different storage types.

danws-MacBook-Pro:dbench danw$ ls -lhrt *-dbench*
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cfs-dbench.yaml
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cbs-dbench.yaml
-rw-r--r--  1 danw  staff   977B Jun 24 00:27 cos-dbench.yaml

Example: cfs-dbench.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: cfs-dbench
spec:
  template:
    spec:
      containers:
      - name: cfs-dbench
        image: ndrpnt/dbench:1.0.0
        imagePullPolicy: Always
        env:
          - name: DBENCH_MOUNTPOINT
            value: /data
          - name: DBENCH_QUICK
            value: "no"
          - name: FIO_SIZE
            value: 10G
          - name: FIO_OFFSET_INCREMENT
            value: 256M
          - name: FIO_DIRECT
            value: "1"
        volumeMounts:
        - name: cfs-vol
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: cfs-vol
        persistentVolumeClaim:
          claimName: cfs-pvc
  backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cfs-pvc
  labels:
    billingType: "hourly"
    region: jp-tok
    zone: tok05
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 500Gi
  storageClassName: ibmc-file-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME               READY   STATUS    RESTARTS   AGE
cfs-dbench-m6ksp   1/1     Running   0          7m20s
danws-MacBook-Pro:dbench danw$ k top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
10.193.33.31   1223m        63%    714Mi           25%
10.193.33.33   117m         6%     1204Mi          42%
10.193.33.44   153m         7%     1102Mi          38%
danws-MacBook-Pro:dbench danw$ k logs -f job/cfs-dbench
Working dir: /data

Testing Read IOPS...
fio: posix_fallocate fails: Not supported
read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64
fio-2.17-45-g06cb
Starting 1 process
read_iops: Laying out IO file(s) (1 file(s) / 10240MiB)
...

File storage test results:

Next, let's look at block storage,
Example: cbs-dbench.yaml


apiVersion: batch/v1
kind: Job
metadata:
  name: cbs-dbench
spec:
  template:
    spec:
      containers:
      - name: cbs-dbench
        image: ndrpnt/dbench:1.0.0
        imagePullPolicy: Always
        env:
          - name: DBENCH_MOUNTPOINT
            value: /data
          - name: DBENCH_QUICK
            value: "no"
          - name: FIO_SIZE
            value: 10G
          - name: FIO_OFFSET_INCREMENT
            value: 256M
          - name: FIO_DIRECT
            value: "1"
        volumeMounts:
        - name: cbs-vol
          mountPath: /data
      restartPolicy: Never
      volumes:
      - name: cbs-vol
        persistentVolumeClaim:
          claimName: cbs-pvc
  backoffLimit: 4
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: cbs-pvc
  labels:
    billingType: "hourly"
    region: jp-tok
    zone: tok05
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 500Gi
  storageClassName: ibmc-block-silver
danws-MacBook-Pro:dbench danw$ k get po
NAME               READY   STATUS      RESTARTS   AGE
cbs-dbench-dv59g   1/1     Running     0          102s
cfs-dbench-m6ksp   0/1     Completed   0          20m

danws-MacBook-Pro:dbench danw$ k get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                  AGE
cbs-pvc   Bound    pvc-ce517510-d529-4f66-a5b9-a0d2d9a58502   500Gi      RWO            ibmc-block-silver             11m
cfs-pvc   Bound    pvc-55ad15de-2deb-4bbd-a805-d45e14831ed1   500Gi      RWX            ibmc-file-silver              30m

danws-MacBook-Pro:dbench danw$ k top nodes
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
10.193.33.31   1173m        61%    718Mi           25%
10.193.33.33   98m          5%     1206Mi          42%
10.193.33.44   160m         8%     1104Mi          38%

Block storage test results:

After the test, clear the exit resource

danws-MacBook-Pro:dbench danw$ k delete -f cbs-dbench.yaml
job.batch "cbs-dbench" deleted
persistentvolumeclaim "cbs-pvc" deleted
danws-MacBook-Pro:dbench danw$ k delete -f cfs-dbench.yaml
job.batch "cfs-dbench" deleted
persistentvolumeclaim "cfs-pvc" deleted

Summary: the file and block storage of 500G * 4 IOPS tier, and the IOPS (read+write) tested at the end all meet the promised 2000, and the bandwidth is 128MiB/S respectively, which indicates that the I/O consumption of k8s container virtualization is not as large as expected. It is worth mentioning that the file and block storage of IBM Cloud are fast in the cluster, and the corresponding storage class, including dynamic PV and k8s volume yaml definition support is good. Of course, different business attributes, different structure of data, data heat and cold, access requirements determine the choice of persistent storage. How to plan k8s storage for your business can be seen in the following link.
https://cloud.ibm.com/docs/containers?topic=containers-storage_planning

Next, we will discuss the mounting of COS and the fio test.

Tags: Kubernetes github Ubuntu VPC

Posted on Thu, 25 Jun 2020 23:31:23 -0400 by kamurj