Embrace blue-green deployment and promote smooth upgrade of AKS cluster Version (Part 2)

In the first part of this series of articles, we have introduced the basic idea of AKS blue-green deployment and how to deploy relevant resources and integrate the application gateway with AKS. Please click here Look back.

In this article, we will further introduce how to deploy applications, how to deploy new AKS clusters, and how to switch AKS versions based on the content of the previous article.

It's not too late. Let's start now!

Application deployment

Let's deploy a demo application to verify that the application gateway and AKS cluster have been successfully integrated. Please copy and save the following YAML source code as deployment_aspnet.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: aspnetapp
spec:
  replicas: 3
  selector:
    matchLabels:
      app: aspnetapp
  template:
    metadata:
      labels:
        app: aspnetapp
    spec:
      containers:
        - name: aspnetapp
          # Sample ASP.Net application from Microsoft which can get private IP.
          image: mcr.microsoft.com/dotnet/core/samples:aspnetapp
          ports:
          - containerPort: 80
---

apiVersion: v1
kind: Service
metadata:
  name: aspnetapp
spec:
  selector:
    app: aspnetapp
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
spec:
  rules:
  - http:
      paths:
      - path: /
        backend:
          serviceName: aspnetapp
          servicePort: 80

Run the following command to deploy this application:

kubectl apply -f deployment_aspnet.yaml

Check the Pod list to confirm that the application deployment has been run:

kubectl get po -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP            NODE                                NOMINATED NODE   READINESS GATES
aad-pod-identity-mic-787c5958fd-kmx9b   1/1     Running   0          177m   10.240.0.33   aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-mic-787c5958fd-nkpv4   1/1     Running   0          177m   10.240.0.63   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-mhp86              1/1     Running   0          177m   10.240.0.4    aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-nmi-sjpvw              1/1     Running   0          177m   10.240.0.35   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-xnfxh              1/1     Running   0          177m   10.240.0.66   aks-nodepool1-94448771-vmss000002   <none>           <none>
agic-ingress-azure-84967fc5b6-cqcn4     1/1     Running   0          111m   10.240.0.79   aks-nodepool1-94448771-vmss000002   <none>           <none>
aspnetapp-68784d6544-j99qg              1/1     Running   0          96    10.240.0.75   aks-nodepool1-94448771-vmss000002   <none>           <none>
aspnetapp-68784d6544-v9449              1/1     Running   0          96    10.240.0.13   aks-nodepool1-94448771-vmss000000   <none>           <none>
aspnetapp-68784d6544-ztbd9              1/1     Running   0          96    10.240.0.50   aks-nodepool1-94448771-vmss000001   <none>           <none>

You can see that the application's pods are running normally. Note that their IP addresses are 10.240.0.13, 10.240.0.50 and 10.240.0.75.

The application gateway backend can see the above IP addresses:

az network application-gateway show-backend-health \
 -g $RESOURCE_GROUP \
 -n $APP_GATEWAY \
 --query backendAddressPools[].backendHttpSettingsCollection[].servers[][address,health]
-o tsv
10.240.0.13     Healthy
10.240.0.50     Healthy
10.240.0.75     Healthy

Run the following command to check the IP address of the front end:

az network public-ip show -g $RESOURCE_GROUP -n $APPGW_IP --query ipAddress -o tsv

Then use the browser to access this IP address, and you can see:

Refresh several times, and you will find that the Host name and Server IP address will display three host names and IP in turn, which are the three Pod names and Intranet IP of the previously deployed Pod. It shows that the Pod integration between application gateway and AKS has been successfully realized.

Deploy AKS new cluster

Create a new version of AKS cluster

In the second AKS subnet, create a new AKS cluster. Our previous AKS version uses the current default version 1.19.11. The new AKS cluster uses 1.20.7, and other parameters remain unchanged.

Variables declaring the name of the new AKS cluster:

AKS_NEW=new

Get the ID of the subnet where the new cluster is located:

NEW_AKS_SUBNET_ID=$(az network vnet subnet show -g $RESOURCE_GROUP --vnet-name $VNET_NAME --name $NEW_AKS_SUBNET --query id -o tsv)

Create a new AKS cluster:

az aks create -n $AKS_NEW \
-g $RESOURCE_GROUP \
-l $AZ_REGION \
--generate-ssh-keys \
--network-plugin azure \
--enable-managed-identity \
--vnet-subnet-id $NEW_AKS_SUBNET_ID \
--kubernetes-version 1.20.7

The new AKS cluster still uses Helm to install application gateway kubernetes ingress.

Install Pod Identify on the new AKS cluster

Connecting to AKS cluster:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NEW

To install AAD Pod Identify:

kubectl create serviceaccount --namespace kube-system tiller-sa
kubectl create clusterrolebinding tiller-cluster-rule --clusterrole=cluster-admin --serviceaccount=kube-system:tiller-sa
helm repo add aad-pod-identity https://raw.githubusercontent.com/Azure/aad-pod-identity/master/charts
helm install aad-pod-identity aad-pod-identity/aad-pod-identity

Helm install Application Gateway Ingress Controller:

helm repo add application-gateway-kubernetes-ingress https://appgwingress.blob.core.windows.net/ingress-azure-helm-package/
helm repo update

Deploy apps on the new AKS cluster

First, install the same application for the new AKS cluster:
kubectl apply -f deployment_aspnet.yaml

After the application is deployed, list the Pod:

kubectl get po -o=custom-columns=NAME:.metadata.name,\
podIP:.status.podIP,NODE:.spec.nodeName,\
READY-true:.status.containerStatuses[*].ready

NAME                                    podIP          NODE                                READY-true
aad-pod-identity-mic-644c7c9f6-cqkxr   10.241.0.25   aks-nodepool1-20247409-vmss000000   true
aad-pod-identity-mic-644c7c9f6-xpwlt   10.241.0.67   aks-nodepool1-20247409-vmss000002   true
aad-pod-identity-nmi-k2c8s             10.241.0.35   aks-nodepool1-20247409-vmss000001   true
aad-pod-identity-nmi-vqqzq             10.241.0.66   aks-nodepool1-20247409-vmss000002   true
aad-pod-identity-nmi-xvcxm             10.241.0.4    aks-nodepool1-20247409-vmss000000   true
aspnetapp-5844845bdc-82lcw             10.241.0.33   aks-nodepool1-20247409-vmss000000   true
aspnetapp-5844845bdc-hskvg             10.241.0.43   aks-nodepool1-20247409-vmss000001   true
aspnetapp-5844845bdc-qzt7f             10.241.0.84   aks-nodepool1-20247409-vmss000002   true

In the actual production operation process, after the application is deployed, do not associate it with the existing application gateway, but log in remotely. Test it through intranet IP access:

kubectl run -it --rm aks-ssh --image=mcr.microsoft.com/aks/fundamental/base-ubuntu:v0.0.11

After the container is started, it will directly enter the container. Let's visit the above three intranet IP S: 10.241.0.33, 10.241.0.43 and 10.241.0.84. For example:

root@aks-ssh:/# curl http://10.241.0.33
root@aks-ssh:/# curl http://10.241.0.43
root@aks-ssh:/# curl http://10.241.0.84

You can return the content normally when you see it. This can demonstrate that the new environment has passed the test. Finally, associate the new AKS cluster with the existing application gateway.

Switch between AKS clusters of different versions

The application gateway switches to integrate with the new version of AKS
Execute the following command to install AGIC:

helm install agic application-gateway-kubernetes-ingress/ingress-azure -f helm_agic.yaml

After a few seconds, run:

kubectl get po -o=custom-columns=NAME:. metadata.name,podIP:.status.podIP,NODE:.spec.nodeName,READY-true:.status.containerStatuses[*].ready
NAME                                    podIP          NODE                                READY-true
aad-pod-identity-mic-644c7c9f6-cqkxr   10.241.0.25   aks-nodepool1-20247409-vmss000000   true
aad-pod-identity-mic-644c7c9f6-xpwlt   10.241.0.67   aks-nodepool1-20247409-vmss000002   true
aad-pod-identity-nmi-k2c8s             10.241.0.35   aks-nodepool1-20247409-vmss000001   true
aad-pod-identity-nmi-vqqzq             10.241.0.66   aks-nodepool1-20247409-vmss000002   true
aad-pod-identity-nmi-xvcxm             10.241.0.4    aks-nodepool1-20247409-vmss000000   true
agic-ingress-azure-84967fc5b6-6x4dd    10.241.0.79   aks-nodepool1-20247409-vmss000002   true
aspnetapp-5844845bdc-82lcw             10.241.0.33   aks-nodepool1-20247409-vmss000000   true
aspnetapp-5844845bdc-hskvg             10.241.0.43   aks-nodepool1-20247409-vmss000001   true
aspnetapp-5844845bdc-qzt7f             10.241.0.84   aks-nodepool1-20247409-vmss000002   true

You can see that the agic ingress azure - * Pod has been running normally.

First use the command line to check that the backend of the application gateway has been updated to a new Pod:

az network application-gateway show-backend-health \
-g $RESOURCE_GROUP \
-n $APP_GATEWAY \
--query backendAddressPools[].backendHttpSettingsCollection[].servers[][address,health]
-o tsv
10.241.0.33     Healthy
10.241.0.43     Healthy
10.241.0.84     Healthy

Go back to the browser and refresh the public IP of the application gateway. You can see that the Host name and IP in the displayed content have been switched to the new back end:

Version rollback

If the new AKS cluster fails, we need to switch back to the old AKS cluster. At this time, you only need to go back to the old AKS cluster, reinstall AGIC and re associate the application gateway with the application Pod in the old AKS cluster.

To do this, first run:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_OLD

Then execute:

helm uninstall agic
helm install agic application-gateway-kubernetes-ingress/ingress-azure -f helm_agic.yaml

You can soon see that AGIC's Pod has been running:

kubectl get po -o wide
NAME                                    READY   STATUS    RESTARTS   AGE    IP            NODE                                NOMINATED NODE   READINESS GATES
aad-pod-identity-mic-787c5958fd-kmx9b   1/1     Running   0          2d1h   10.240.0.33   aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-mic-787c5958fd-nkpv4   1/1     Running   1          2d1h   10.240.0.63   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-mhp86              1/1     Running   0          2d1h   10.240.0.4    aks-nodepool1-94448771-vmss000000   <none>           <none>
aad-pod-identity-nmi-sjpvw              1/1     Running   0          2d1h   10.240.0.35   aks-nodepool1-94448771-vmss000001   <none>           <none>
aad-pod-identity-nmi-xnfxh              1/1     Running   0          2d1h   10.240.0.66   aks-nodepool1-94448771-vmss000002   <none>           <none>
agic-ingress-azure-84967fc5b6-nwbh4     1/1     Running   0          8s     10.240.0.70   aks-nodepool1-94448771-vmss000002   <none>           <none>
aspnetapp-68784d6544-j99qg              1/1     Running   0          2d     10.240.0.75   aks-nodepool1-94448771-vmss000002   <none>           <none>
aspnetapp-68784d6544-v9449              1/1     Running   0          2d     10.240.0.13   aks-nodepool1-94448771-vmss000000   <none>           <none>
aspnetapp-68784d6544-ztbd9              1/1     Running   0          2d     10.240.0.50   aks-nodepool1-94448771-vmss000001   <none>     

Look at the back end of the application gateway:

az network application-gateway show-backend-health \
 -g $RESOURCE_GROUP \
 -n $APP_GATEWAY \
 --query backendAddressPools[].backendHttpSettingsCollection[].servers[][address,health]
-o tsv
10.240.0.13     Healthy
10.240.0.50     Healthy
10.240.0.75     Healthy

You can see that the same application gateway backend has recovered the IP of the old AKS cluster.

Apply availability test during version switching

We use successive HTTP requests to verify that the service is not interrupted during the handover.

Open another command line window and execute:

while(true); \

do curl -s http://139.217.117.86/ |ts '[%Y-%m-%d %H:%M:%S]' | grep 10.24; \

sleep 0.1; done

[2021-08-03 16:35:09] 10.240.0.13                        <br />

[2021-08-03 16:35:10] 10.240.0.50                        <br />

[2021-08-03 16:35:11] 10.240.0.13                        <br />

[2021-08-03 16:35:12] 10.240.0.75                        <br />

[2021-08-03 16:35:12] 10.240.0.50                        <br />

[2021-08-03 16:35:13] 10.240.0.13                        <br />

[2021-08-03 16:35:14] 10.240.0.75                        <br />

You can see that the private IP address of pod in the old AKS cluster is returned in turn.

Go back to the previous AKS operation window, switch to the new AKS cluster, and execute the command to delete and install AGIC again:

az aks get-credentials --resource-group $RESOURCE_GROUP --name $AKS_NEW

Re execution:

helm uninstall agic

In the second window, you will find that the IP of the old AKS cluster is still returned. At this time, we only delete the new AKS cluster, and the application gateway and the old AKS cluster are running normally.

Then execute on the new AKS cluster:

helm install agic application-gateway-kubernetes-ingress/ingress-azure -f helm_agic.yaml

In the second window, you will find that the IP address of the new AKS cluster is directly replaced from a certain line without any interruption:

[2021-08-03 16:42:08] 10.240.0.13                        <br />

[2021-08-03 16:42:09] 10.240.0.50                        <br />

[2021-08-03 16:42:09] 10.240.0.75                        <br />

[2021-08-03 16:42:10] 10.240.0.13                        <br />

[2021-08-03 16:42:11] 10.240.0.50                        <br />

[2021-08-03 16:42:11] 10.240.0.75                        <br />

[2021-08-03 16:42:12] 10.241.0.33                        <br />

[2021-08-03 16:42:13] 10.241.0.33                        <br />

[2021-08-03 16:42:13] 10.241.0.43                        <br />

[2021-08-03 16:42:15] 10.241.0.43                        <br />

[2021-08-03 16:42:15] 10.241.0.84                        <br />

[2021-08-03 16:42:16] 10.241.0.84                        <br />

Thus, it is verified that the external services of the application gateway are always running normally during the handover process. Through this operation, the old and new AKS clusters can be retained at the same time and can be switched in real time.

summary

Taking common Web applications as an example, the above demonstrates that the new AKS cluster can realize stable version upgrade through blue-green deployment.

In addition to Web applications, applications of other types and scenarios can refer to and switch between AKS cluster and upstream integration, so as to realize real-time switching and rollback.

Tags: Kubernetes Azure

Posted on Sun, 05 Dec 2021 06:34:33 -0500 by MishieMoo