There are many ways to store k8s data: emptyDir (temporary storage), hostpath (semi persistent storage), persistent storage (file, block, object storage, etc.). Almost all public cloud vendors support cloud storage service integration k8s hosting. The k8s system itself adopts two concepts, persistent volume (PV) and Persistent volume claim (PVC) implements the life cycle management of storage persistence (application, reservation, deletion, discarding, etc.). Pod applies for data volume to the cluster system and uses it according to the demand. If the pod is released and the application disappears, the data volume will be separated and unbound accordingly, and the data will remain. In use, pod and the application have no awareness of the storage back-end infrastructure. This design is very good to understand the application and data. As for the bottom layer of storage resources, ceph, portwx, elastic cloud storage, cloud object storage and so on are selected, which depends on the specific business needs and the support of manufacturers.
Let's cut into today's topic. Since there are many back-end options, users are trying a k8s cluster product. Storage persistence and application level read-write ability are a must. Here we take ibm cloud kubernetes service as an example to see three common test methods and performance comparison of external storage on IBM cloud.
It is roughly divided into three steps:
- Create K8S cluster and use it through kubectl connection (deployed in Tokyo, Japan)
- Confirm and install IBM cloud storage k8s supply plug-ins (CFS, CBS, COS deployed in Tokyo)
- dbench(fio) tests the above three storage volumes
1. k8s cluster creation, connection and use
land cloud.ibm.com Search for the keyword "kubernetes" in the catalog and enter the service creation wizard. The standard tariff will be used in this test (there is only one worker node in the free cluster, which cannot meet the test requirements). It can be noted that the online version of ibm k8s is very new. Select the latest stable version 1.17.6
Next, select the classic infrastructure (covering the whole). vpc infrastructure is also fully ready in the six regions (Tokyo, Sydney, Frankfurt, London, Washington, Dallas). Considering the worker cost of vpc high availability area cluster, it will be more cost-effective to open a set of classic test in a single region.
Finally, we set up the worker pool, and select three 2c*4g virtual machines as the working nodes. If you want to be safe, you can turn on the node disk encryption (ibm has nothing to say), define the cluster name, and click create
About five or six minutes later, the cluster initialization is completed, and it enters the "Normal" state
Click to create a cluster and enter "access". Refer to the steps to install IBM cloud cli on your notebook (k8s related tools such as kubectl and docker will be installed together if the current system does not have them)
danws-MacBook-Pro:k8s-playground danw$ ibmcloud login --sso -a cloud.ibm.com -r jp-tok -g danw-group danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks cluster config --cluster bro62jit0fjj42dqsh7g danws-MacBook-Pro:k8s-playground danw$ export KUBECONFIG=/Users/danw/.bluemix/plugins/container-service/clusters/bro62jit0fjj42dqsh7g/kube-config-tok05-demo-k8s-tok05.yml
You can also add the cluster configuration file to ～ /. bash_profile,
Test connection succeeded!
danws-MacBook-Pro:k8s-playground danw$ k get nodes -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME 10.193.33.31 Ready <none> 26m v1.17.6+IKS 10.193.33.31 165.192.96.2 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4 10.193.33.33 Ready <none> 26m v1.17.6+IKS 10.193.33.33 165.192.96.12 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4 10.193.33.44 Ready <none> 27m v1.17.6+IKS 10.193.33.44 165.192.96.7 Ubuntu 18.04.4 LTS 4.15.0-101-generic containerd://1.3.4
2. Install the IBM Cloud storage plug-in
a. IBM Cloud File Storage
The default file reload class (storage plug-in) is pre installed during cluster creation, and can be used directly
danws-MacBook-Pro:k8s-playground danw$ k get storageclasses | grep file default ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-bronze ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-bronze-gid ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-custom ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-gold (default) ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-gold-gid ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-retain-bronze ibm.io/ibmc-file Retain Immediate false 37m ibmc-file-retain-custom ibm.io/ibmc-file Retain Immediate false 37m ibmc-file-retain-gold ibm.io/ibmc-file Retain Immediate false 37m ibmc-file-retain-silver ibm.io/ibmc-file Retain Immediate false 37m ibmc-file-silver ibm.io/ibmc-file Delete Immediate false 37m ibmc-file-silver-gid ibm.io/ibmc-file Delete Immediate false 37m
Here we choose "silver" (hourly, 4 IOPS/GB)
danws-MacBook-Pro:k8s-playground danw$ kubectl describe storageclass ibmc-file-bronze Name: ibmc-file-bronze IsDefaultClass: No Annotations: kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"labels":{"kubernetes.io/cluster-service":"true"},"name":"ibmc-file-bronze"},"parameters":{"billingType":"hourly","classVersion":"2","iopsPerGB":"2","sizeRange":"[20-12000]Gi","type":"Endurance"},"provisioner":"ibm.io/ibmc-file","reclaimPolicy":"Delete"} Provisioner: ibm.io/ibmc-file Parameters: billingType=hourly,classVersion=2,iopsPerGB=2,sizeRange=[20-12000]Gi,type=Endurance AllowVolumeExpansion: <unset> MountOptions: <none> ReclaimPolicy: Delete VolumeBindingMode: Immediate Events: <none>
b. IBM Cloud Block Storage
By default, the "block storage" plug-in is not pre installed in the cluster. You need to install it first. For official online documents, please refer to the specific steps.
https://cloud.ibm.com/docs/containers?topic=containers-block_storage#install_block
First confirm the status of cluster work nodes
danws-MacBook-Pro:k8s-playground danw$ ibmcloud ks worker ls --cluster demo-k8s-tok05 OK ID Public IP Private IP Flavor State Status Zone Version kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000001b3 165.192.96.7 10.193.33.44 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527 kube-bro62jit0fjj42dqsh7g-demok8stok0-default-0000022e 165.192.96.2 10.193.33.31 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527 kube-bro62jit0fjj42dqsh7g-demok8stok0-default-000003fe 165.192.96.12 10.193.33.33 u3c.2x4.encrypted normal Ready tok05 1.17.6_1527
Block storage and subsequent COS plug-in installation need to rely on helm chart. Here we first install helm v3. Here, the helm installation process has been omitted. Please refer to the following connection:
https://github.com/helm/helm/releases/tag/v3.2.4
danws-MacBook-Pro:k8s-playground danw$ helm version version.BuildInfo{Version:"v3.2.4", GitCommit:"0ad800ef43d3b826f31a5ad8dfbb4fe05d143688", GitTreeState:"clean", GoVersion:"go1.13.12"} danws-MacBook-Pro:k8s-playground danw$ helm repo add iks-charts https://icr.io/helm/iks-charts "iks-charts" has been added to your repositories danws-MacBook-Pro:k8s-playground danw$ helm repo update Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "istio.io" chart repository ...Successfully got an update from the "iks-charts" chart repository Update Complete. ⎈ Happy Helming!⎈ danws-MacBook-Pro:k8s-playground danw$ helm search repo iks-charts NAME CHART VERSION APP VERSION DESCRIPTION iks-charts/ibm-block-storage-attacher 1.1.4 A Helm chart for installing ibmcloud block stor... iks-charts/ibm-iks-cluster-autoscaler 1.1.7 A Helm chart for installing the IBM Cloud clust... iks-charts/ibm-object-storage-plugin 1.0.7 1.0.7 A Helm chart for installing ibmcloud object sto... iks-charts/ibm-worker-recovery 1.10.71 IBM Autorecovery system allows automatic recove... iks-charts/ibmcloud-alb-metrics-exporter 1.0.25 25.0 A Helm chart to deploy IBM Cloud ALB metrics ex... iks-charts/ibmcloud-backup-restore 1.0.1 A Helm chart for taking backup of pvc data and ... iks-charts/ibmcloud-block-storage-plugin 1.7.0 A Helm chart for installing ibmcloud block stor... iks-charts/ibmcloud-data-shield 1.18.731 1.0 Protect data in use with IBM Cloud Data Shield iks-charts/ibmcloud-image-enforcement 0.2.10 A Helm chart to install IBM Container Image Sec... iks-charts/strongswan 2.6.3 20.05.15 A strongSwan IPSec VPN service to securely conn... danws-MacBook-Pro:k8s-playground danw$ helm install 1.7.0 iks-charts/ibmcloud-block-storage-plugin -n default danws-MacBook-Pro:k8s-playground danw$ kubectl get pods -n kube-system | grep ibmcloud-block-storage ibmcloud-block-storage-driver-792wk 1/1 Running 0 62s ibmcloud-block-storage-driver-m4phr 1/1 Running 0 62s ibmcloud-block-storage-driver-n9pxv 1/1 Running 0 62s ibmcloud-block-storage-plugin-665f4b5fb5-4fn9t 1/1 Running 0 61s danws-MacBook-Pro:k8s-playground danw$ kubectl get storageclasses | grep block ibmc-block-bronze ibm.io/ibmc-block Delete Immediate true 2m50s ibmc-block-custom ibm.io/ibmc-block Delete Immediate true 2m50s ibmc-block-gold ibm.io/ibmc-block Delete Immediate true 2m50s ibmc-block-retain-bronze ibm.io/ibmc-block Retain Immediate true 2m50s ibmc-block-retain-custom ibm.io/ibmc-block Retain Immediate true 2m50s ibmc-block-retain-gold ibm.io/ibmc-block Retain Immediate true 2m50s ibmc-block-retain-silver ibm.io/ibmc-block Retain Immediate true 2m50s ibmc-block-silver ibm.io/ibmc-block Delete Immediate true 2m50s
Note: in the next part, we will discuss COS pvc
3. Using dbench to test the two types of pvc
github is an open source container for fio test image, which can be used directly. clone repo first
danws-MacBook-Pro:dbench danw$ git clone https://github.com/leeliu/dbench.git
⚠️ : here is a small thunder. dbench image is currently moved to this repo "ndrpnt/dbench:1.0.0 "
Please refer to the following link for the script file content of fio,
https://github.com/leeliu/dbench/blob/master/docker-entrypoint.sh
Although the script has been packed into the image, users can adjust some parameters through the pod environment variables, such as quick test, test file size, data offset, etc.
edit dbench.yaml Because of the limited computing resources of the three worker nodes (2c4g) in the cluster, we are worried that the simultaneous execution of fio test by the three kinds of storage will affect the I/O efficiency of each other, so we can change yaml to mount pvc separate tests of different storage types.
danws-MacBook-Pro:dbench danw$ ls -lhrt *-dbench* -rw-r--r-- 1 danw staff 977B Jun 24 00:27 cfs-dbench.yaml -rw-r--r-- 1 danw staff 977B Jun 24 00:27 cbs-dbench.yaml -rw-r--r-- 1 danw staff 977B Jun 24 00:27 cos-dbench.yaml
Example: cfs-dbench.yaml
apiVersion: batch/v1 kind: Job metadata: name: cfs-dbench spec: template: spec: containers: - name: cfs-dbench image: ndrpnt/dbench:1.0.0 imagePullPolicy: Always env: - name: DBENCH_MOUNTPOINT value: /data - name: DBENCH_QUICK value: "no" - name: FIO_SIZE value: 10G - name: FIO_OFFSET_INCREMENT value: 256M - name: FIO_DIRECT value: "1" volumeMounts: - name: cfs-vol mountPath: /data restartPolicy: Never volumes: - name: cfs-vol persistentVolumeClaim: claimName: cfs-pvc backoffLimit: 4 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cfs-pvc labels: billingType: "hourly" region: jp-tok zone: tok05 spec: accessModes: - ReadWriteMany resources: requests: storage: 500Gi storageClassName: ibmc-file-silver
danws-MacBook-Pro:dbench danw$ k get po NAME READY STATUS RESTARTS AGE cfs-dbench-m6ksp 1/1 Running 0 7m20s danws-MacBook-Pro:dbench danw$ k top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 10.193.33.31 1223m 63% 714Mi 25% 10.193.33.33 117m 6% 1204Mi 42% 10.193.33.44 153m 7% 1102Mi 38% danws-MacBook-Pro:dbench danw$ k logs -f job/cfs-dbench Working dir: /data Testing Read IOPS... fio: posix_fallocate fails: Not supported read_iops: (g=0): rw=randread, bs=4096B-4096B,4096B-4096B,4096B-4096B, ioengine=libaio, iodepth=64 fio-2.17-45-g06cb Starting 1 process read_iops: Laying out IO file(s) (1 file(s) / 10240MiB) ...
File storage test results:
Next, let's look at block storage,
Example: cbs-dbench.yaml
apiVersion: batch/v1 kind: Job metadata: name: cbs-dbench spec: template: spec: containers: - name: cbs-dbench image: ndrpnt/dbench:1.0.0 imagePullPolicy: Always env: - name: DBENCH_MOUNTPOINT value: /data - name: DBENCH_QUICK value: "no" - name: FIO_SIZE value: 10G - name: FIO_OFFSET_INCREMENT value: 256M - name: FIO_DIRECT value: "1" volumeMounts: - name: cbs-vol mountPath: /data restartPolicy: Never volumes: - name: cbs-vol persistentVolumeClaim: claimName: cbs-pvc backoffLimit: 4 --- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: cbs-pvc labels: billingType: "hourly" region: jp-tok zone: tok05 spec: accessModes: - ReadWriteOnce resources: requests: storage: 500Gi storageClassName: ibmc-block-silver
danws-MacBook-Pro:dbench danw$ k get po NAME READY STATUS RESTARTS AGE cbs-dbench-dv59g 1/1 Running 0 102s cfs-dbench-m6ksp 0/1 Completed 0 20m danws-MacBook-Pro:dbench danw$ k get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE cbs-pvc Bound pvc-ce517510-d529-4f66-a5b9-a0d2d9a58502 500Gi RWO ibmc-block-silver 11m cfs-pvc Bound pvc-55ad15de-2deb-4bbd-a805-d45e14831ed1 500Gi RWX ibmc-file-silver 30m danws-MacBook-Pro:dbench danw$ k top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 10.193.33.31 1173m 61% 718Mi 25% 10.193.33.33 98m 5% 1206Mi 42% 10.193.33.44 160m 8% 1104Mi 38%
Block storage test results:
After the test, clear the exit resource
danws-MacBook-Pro:dbench danw$ k delete -f cbs-dbench.yaml job.batch "cbs-dbench" deleted persistentvolumeclaim "cbs-pvc" deleted danws-MacBook-Pro:dbench danw$ k delete -f cfs-dbench.yaml job.batch "cfs-dbench" deleted persistentvolumeclaim "cfs-pvc" deleted
Summary: the file and block storage of 500G * 4 IOPS tier, and the IOPS (read+write) tested at the end all meet the promised 2000, and the bandwidth is 128MiB/S respectively, which indicates that the I/O consumption of k8s container virtualization is not as large as expected. It is worth mentioning that the file and block storage of IBM Cloud are fast in the cluster, and the corresponding storage class, including dynamic PV and k8s volume yaml definition support is good. Of course, different business attributes, different structure of data, data heat and cold, access requirements determine the choice of persistent storage. How to plan k8s storage for your business can be seen in the following link.
https://cloud.ibm.com/docs/containers?topic=containers-storage_planning
Next, we will discuss the mounting of COS and the fio test.