Based on Kubernetes Gitlab CICD

Original text: https://i4t.com/4373.html

In most cases, the construction project will take up a lot of system resources. If gitlab itself is used to run the construction task, obviously gitlab's performance will be greatly reduced. The most important function of gitlab ci is to manage the construction status of each project. Therefore, it is much better to leave the waste of resources such as running the construction task to an independent gitlab runner. More importantly, gitlab runner can be installed on different machines, even our own machine, so it will not affect gitlab itself at all.

Since gitlab 8.0, gitlab CI has been integrated in gitlab. We just need to add a. gitlab-ci.yaml file to the project, and then run a Runner to achieve continuous integration.

What is Runner?
Gitlab Runner is an open source project for running your job and sending the results to gitlab. It is used in conjunction with gitlab Ci, an open source continuous integration service for coordination that comes with gitlab.

Gitlab Runner is written in Go and can be run as a binary without language specific requirements
It runs on GNU/Linux, Mac OS and Windows operating systems. If you want to use Docker, Gitlab Runner needs at least Docker v1.13.0

Project illustration of this article

Gitlab installation

gitlab officially provides Helm for quick installation in Kubernetes cluster, but in the process of using it, we found that there are many other additional configurations in the Chart package provided by Helm, so we customize the way to install here, that is, we define some resource manifest files ourselves.

Gitlab mainly involves three applications: Redis, Postgresql and gitlab core program

Deployment services can also run on containers without affecting gitlab's cicd. I use k8s to install gitlab, and I don't use persistent storage. If you want to use persistent storage, please refer to prometheus building Articles

Create namespace

kubectl create namespace kube-ops

First, you need to deploy the Redis service (gitlab Redis. Yaml)

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  template:
    metadata:
      name: redis
      labels:
        name: redis
    spec:
      containers:
      - name: redis
        image: sameersbn/redis
        imagePullPolicy: IfNotPresent
        ports:
        - name: redis
          containerPort: 6379
        volumeMounts:
        - mountPath: /var/lib/redis
          name: data
        livenessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - redis-cli
            - ping
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: redis
  namespace: kube-ops
  labels:
    name: redis
spec:
  ports:
    - name: redis
      port: 6379
      targetPort: redis
  selector:
    name: redis

Database Postgresql deployment (gitlab Postgresql. Yaml)

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  template:
    metadata:
      name: postgresql
      labels:
        name: postgresql
    spec:
      containers:
      - name: postgresql
        image: sameersbn/postgresql:10
        imagePullPolicy: IfNotPresent
        env:
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: DB_EXTENSION
          value: pg_trgm
        ports:
        - name: postgres
          containerPort: 5432
        volumeMounts:
        - mountPath: /var/lib/postgresql
          name: data
        livenessProbe:
          exec:
            command:
            - pg_isready
            - -h
            - localhost
            - -U
            - postgres
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - pg_isready
            - -h
            - localhost
            - -U
            - postgres
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: postgresql
  namespace: kube-ops
  labels:
    name: postgresql
spec:
  ports:
    - name: postgres
      port: 5432
      targetPort: postgres
  selector:
    name: postgresql

The configuration of gitlab application is as follows (gitlab.yaml)

Instead of using the official image warehouse, we use a third party
http://www.damagehead.com/doc...

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  template:
    metadata:
      name: gitlab
      labels:
        name: gitlab
    spec:
      containers:
      - name: gitlab
        image: sameersbn/gitlab:11.8.1
        imagePullPolicy: IfNotPresent
        env:
        - name: TZ
          value: Asia/Shanghai
        - name: GITLAB_TIMEZONE
          value: Beijing
        - name: GITLAB_SECRETS_DB_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_SECRET_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_SECRETS_OTP_KEY_BASE
          value: long-and-random-alpha-numeric-string
        - name: GITLAB_ROOT_PASSWORD
          value: admin321
        - name: GITLAB_ROOT_EMAIL
          value: 381493251@qq.com
        - name: GITLAB_HOST
          value: 10.4.82.135:30004
        - name: GITLAB_PORT
          value: "80"
        - name: GITLAB_SSH_PORT
          value: "22"
        - name: GITLAB_NOTIFY_ON_BROKEN_BUILDS
          value: "true"
        - name: GITLAB_NOTIFY_PUSHER
          value: "false"
        - name: GITLAB_BACKUP_SCHEDULE
          value: daily
        - name: GITLAB_BACKUP_TIME
          value: 01:00
        - name: DB_TYPE
          value: postgres
        - name: DB_HOST
          value: postgresql
        - name: DB_PORT
          value: "5432"
        - name: DB_USER
          value: gitlab
        - name: DB_PASS
          value: passw0rd
        - name: DB_NAME
          value: gitlab_production
        - name: REDIS_HOST
          value: redis
        - name: REDIS_PORT
          value: "6379"
        ports:
        - name: http
          containerPort: 80
        - name: ssh
          containerPort: 22
        volumeMounts:
        - mountPath: /home/git/data
          name: data
        livenessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 180
          timeoutSeconds: 5
        readinessProbe:
          httpGet:
            path: /
            port: 80
          initialDelaySeconds: 5
          timeoutSeconds: 1
      volumes:
      - name: data
        emptyDir: {}

---
apiVersion: v1
kind: Service
metadata:
  name: gitlab
  namespace: kube-ops
  labels:
    name: gitlab
spec:
  type: NodePort
  ports:
    - name: http
      port: 80
      targetPort: http
      nodePort: 30004
    - name: ssh
      port: 22
      targetPort: ssh
  selector:
    name: gitlab


################
GITLAB_HOST = The domain name here is us Git clone Code domain name, I use it directly here svc+port Demonstration

Next, we need to create a namespace in advance. We have already created the namespace

kubectl apply -f .
deployment.apps/postgresql created
service/postgresql created
deployment.apps/redis created
service/redis created
deployment.apps/gitlab created
service/gitlab created

Check pod and svc health

[root@abcdocker gitlab]# kubectl get pod,svc -n kube-ops
NAME                              READY   STATUS    RESTARTS   AGE
pod/gitlab-57cf47cd7d-cfpmn       0/1     Running   1          5m14s
pod/postgresql-7c6bf8974d-r9djh   1/1     Running   0          5m19s
pod/redis-c45ffd79b-75vbd         1/1     Running   0          5m17s

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                    AGE
service/gitlab       NodePort    10.254.212.182   <none>        80:30004/TCP,22:8530/TCP   5m11s
service/postgresql   ClusterIP   10.254.255.216   <none>        5432/TCP                   5m18s
service/redis        ClusterIP   10.254.83.148    <none>        6379/TCP                   5m15s

Here I use the nodePort method, and then we can directly access any IP+30004 port of the node (if the pod does not report an error, wait patiently for a while to access it)

User name: root
Password: admin321
The user name and password are configured in the gitlab configuration file!

Next, create a project to demonstrate


Next, add the commit code
How to submit the code is not covered in detail here. The REDME of the project also prompts us

[root@abcdocker tmp]# git clone http://10.4.82.135:30004/root/abcdocker.git
//Cloning to 'abcdocker'
Username for 'http://10.4.82.135:30004': root
Password for 'http://root@10.4.82.135:30004':
warning: You seem to have cloned an empty version library.

wget http://down.i4t.com/abcdocker-gitlab-demo.tar.gz

#git code submission
git add *
git add .* (Remember to submit hidden files)
git commit -m "abcdocker"
git push -u origin master

Gitlab Runner installation

gitlab runner can be installed in many ways. I will install it in k8s here.

Official document address: https://docs.gitlab.com/runne...

First, we need to check the status of the k8s cluster

[root@abcdocker ~]# kubectl cluster-info
Kubernetes master is running at https://10.4.82.139:8443
CoreDNS is running at https://10.4.82.139:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Next, we open the gitlab address

As shown in the figure below, the left represents the runner status, and the right represents the configuration runner information

Next, configure the gitlab runner resource list (runner configmap. Yaml)

apiVersion: v1
data:
  REGISTER_NON_INTERACTIVE: "true"
  REGISTER_LOCKED: "false"
  METRICS_SERVER: "0.0.0.0:9100"
  CI_SERVER_URL: "http://gitlab.kube-ops.svc.cluster.local/ci"
  RUNNER_REQUEST_CONCURRENCY: "4"
  RUNNER_EXECUTOR: "kubernetes"
  KUBERNETES_NAMESPACE: "kube-ops"
  KUBERNETES_PRIVILEGED: "true"
  KUBERNETES_CPU_LIMIT: "1"
  KUBERNETES_CPU_REQUEST: "500m"
  KUBERNETES_MEMORY_LIMIT: "1Gi"
  KUBERNETES_SERVICE_CPU_LIMIT: "1"
  KUBERNETES_SERVICE_MEMORY_LIMIT: "1Gi"
  KUBERNETES_HELPER_CPU_LIMIT: "500m"
  KUBERNETES_HELPER_MEMORY_LIMIT: "100Mi"
  KUBERNETES_PULL_POLICY: "if-not-present"
  KUBERNETES_TERMINATIONGRACEPERIODSECONDS: "10"
  KUBERNETES_POLL_INTERVAL: "5"
  KUBERNETES_POLL_TIMEOUT: "360"
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-cm
  namespace: kube-ops
  • Ci server URL is the address of gitlab. If gitlab writes the ip address of the host directly on the host, the container is in the form of svc name. Namespace. svc.cluster.local (if you install according to my documents, you don't need to modify other configurations)

If the defined gitlab domain name is not resolved through the Internet DNS, but mapped through / etc/hosts, then we need to add the corresponding hosts in the Pod of Runner (I use the ip + port here). We need to specify a script through the -- pre clone script parameter to add hosts information, that is, to add the value of the environment variable Runner ﹣ pre ﹣ clone ﹣ script in ConfigMap

RUNNER_PRE_CLONE_SCRIPT = "echo 'xx.xx.xxx.xx git.i4t.com' >> /etc/hosts"

Note: after adding new options to ConfigMap, you need to delete Gitlab ci Runner Pod
Because we use envFrom to inject these environment variables instead of env(envfrom helps reduce the manifest file by placing environment variables in ConfigMaps or Secrets)

If we want to add other options, we can run gitlab CI multi runner register -- help command in Pod to see all available options, just add env variable for the configured flag

gitlab-runner@gitlab-ci-runner-0:/$ gitlab-ci-multi-runner register --help
[...]
--kubernetes-cpu-limit value                          The CPU allocation given to build containers (default: "1") [$KUBERNETES_CPU_LIMIT]
--kubernetes-memory-limit value                       The amount of memory allocated to build containers (default: "4Gi") [$KUBERNETES_MEMORY_LIMIT]
--kubernetes-service-cpu-limit value                  The CPU allocation given to build service containers (default: "1") [$KUBERNETES_SERVICE_CPU_LIMIT]
--kubernetes-service-memory-limit value               The amount of memory allocated to build service containers (default: "1Gi") [$KUBERNETES_SERVICE_MEMORY_LIMIT]
--kubernetes-helper-cpu-limit value                   The CPU allocation given to build helper containers (default: "500m") [$KUBERNETES_HELPER_CPU_LIMIT]
--kubernetes-helper-memory-limit value                The amount of memory allocated to build helper containers (default: "3Gi") [$KUBERNETES_HELPER_MEMORY_LIMIT]
--kubernetes-cpu-request value                        The CPU allocation requested for build containers [$KUBERNETES_CPU_REQUEST]
...
--pre-clone-script value                              Runner-specific command script executed before code is pulled [$RUNNER_PRE_CLONE_SCRIPT]
[...]

Establish

[root@abcdocker gitlab]# kubectl apply -f runner-configmap.yaml
configmap/gitlab-ci-runner-cm created

[root@abcdocker gitlab]# kubectl get configmaps  -n kube-ops
NAME                  DATA   AGE
gitlab-ci-runner-cm   19     12s

We also need to configure a small script to register, run, and cancel the gitlab ci runner. Deregistration is triggered only when the Pod is normally terminated by Kubernetes (TERM signal). If the Pod (SIGKILL signal) is forcibly terminated, the Runner will not log off itself. This kind of Runner must be cleaned up manually (binary installation is not affected by this on non K8s)

(runner-scripts-cm.yaml)

apiVersion: v1
data:
  run.sh: |
    #!/bin/bash
    unregister() {
        kill %1
        echo "Unregistering runner ${RUNNER_NAME} ..."
        /usr/bin/gitlab-ci-multi-runner unregister -t "$(/usr/bin/gitlab-ci-multi-runner list 2>&1 | tail -n1 | awk '{print $4}' | cut -d'=' -f2)" -n ${RUNNER_NAME}
        exit $?
    }
    trap 'unregister' EXIT HUP INT QUIT PIPE TERM
    echo "Registering runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner register -r ${GITLAB_CI_TOKEN}
    sed -i 's/^concurrent.*/concurrent = '"${RUNNER_REQUEST_CONCURRENCY}"'/' /home/gitlab-runner/.gitlab-runner/config.toml
    echo "Starting runner ${RUNNER_NAME} ..."
    /usr/bin/gitlab-ci-multi-runner run -n ${RUNNER_NAME} &
    wait
kind: ConfigMap
metadata:
  labels:
    app: gitlab-ci-runner
  name: gitlab-ci-runner-scripts
  namespace: kube-ops

Establish

[root@abcdocker gitlab]# kubectl apply -f runner-scripts-cm.yaml
configmap/gitlab-ci-runner-scripts created

We need to create a gitlab? CI? Token, and then we use gitlab ci runner token to create a Kubernetes secret object. base64 transcoding of token is required in advance

[root@abcdocker gitlab]# echo BMxb1ezMiTYFxtZTsVxP|base64 -w0
Qk14YjFlek1pVFlGeHRaVHNWeFAK

The token here is the screenshot of our gitlab runner. base64 is only needed in k8s environment

Following chart

Next, use the token above to create a Sercret object (gitlab CI token secret. Yaml)

[root@abcdocker gitlab]# cat gitlab-ci-token-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: gitlab-ci-token
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
data:
  GITLAB_CI_TOKEN: Qk14YjFlek1pVFlGeHRaVHNWeFAK

Establish

[root@abcdocker gitlab]# kubectl apply -f gitlab-ci-token-secret.yaml
secret/gitlab-ci-token created

Next, create the controller image that actually runs the Runner. Here, use Statefulset. When running, try to unregister all runners with the same name. When the node is lost (that is, NodeLost event), this is particularly useful. Then try to register yourself and start running. When the Pod is stopped normally, the Runner will run the unregister command to try to cancel itself, so gitlab can no longer use the Runner. This is done through the hooks in the kubernetes Pod life cycle

runner-statefulset.yaml

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: gitlab-ci-runner
  namespace: kube-ops
  labels:
    app: gitlab-ci-runner
spec:
  updateStrategy:
    type: RollingUpdate
  replicas: 2
  serviceName: gitlab-ci-runner
  template:
    metadata:
      labels:
        app: gitlab-ci-runner
    spec:
      volumes:
      - name: gitlab-ci-runner-scripts
        projected:
          sources:
          - configMap:
              name: gitlab-ci-runner-scripts
              items:
              - key: run.sh
                path: run.sh
                mode: 0755
      serviceAccountName: gitlab-ci
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        supplementalGroups: [999]
      containers:
      - image: gitlab/gitlab-runner:latest
        name: gitlab-ci-runner
        command:
        - /scripts/run.sh
        envFrom:
        - configMapRef:
            name: gitlab-ci-runner-cm
        - secretRef:
            name: gitlab-ci-token
        env:
        - name: RUNNER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        ports:
        - containerPort: 9100
          name: http-metrics
          protocol: TCP
        volumeMounts:
        - name: gitlab-ci-runner-scripts
          mountPath: "/scripts"
          readOnly: true
      restartPolicy: Always

Above, we named a service account of gitlab CI. Here, we need to create a new RBAC file (runner-rbac.yaml)

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-ci
  namespace: kube-ops
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
rules:
  - apiGroups: [""]
    resources: ["*"]
    verbs: ["*"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: gitlab-ci
  namespace: kube-ops
subjects:
  - kind: ServiceAccount
    name: gitlab-ci
    namespace: kube-ops
roleRef:
  kind: Role
  name: gitlab-ci
  apiGroup: rbac.authorization.k8s.io

Creation completed

[root@abcdocker gitlab]# kubectl apply -f runner-rbac.yaml
serviceaccount/gitlab-ci created
role.rbac.authorization.k8s.io/gitlab-ci created
rolebinding.rbac.authorization.k8s.io/gitlab-ci created
[root@abcdocker gitlab]# kubectl apply -f runner-statefulset.yaml
statefulset.apps/gitlab-ci-runner created

Now let's examine what we created

[root@abcdocker gitlab]# kubectl get pod,svc,cm -n kube-ops
NAME                              READY   STATUS    RESTARTS   AGE
pod/gitlab-57cf47cd7d-cfpmn       1/1     Running   1          47h
pod/gitlab-ci-runner-0            1/1     Running   0          2m51s
pod/gitlab-ci-runner-1            1/1     Running   0          119s
pod/postgresql-7c6bf8974d-r9djh   1/1     Running   0          47h
pod/redis-c45ffd79b-75vbd         1/1     Running   0          47h

NAME                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                    AGE
service/gitlab       NodePort    10.254.212.182   <none>        80:30004/TCP,22:8530/TCP   47h
service/postgresql   ClusterIP   10.254.255.216   <none>        5432/TCP                   47h
service/redis        ClusterIP   10.254.83.148    <none>        6379/TCP                   47h

NAME                                 DATA   AGE
configmap/gitlab-ci-runner-cm        19     150m
configmap/gitlab-ci-runner-scripts   1      78m


#At present, pod gitlab-ci-runner 0 and 1 are in normal operation state

At this point, we can see that the Runners in gitlab have added these two pod nodes

Here we can also change some configuration of Runner, such as adding tag tag, etc

gitlab CICD configuration

First, we need to have a project. Next, we will demonstrate the configuration of gitlab

#Project Download
wget http://down.i4t.com/gitlab-ci-k8s-demo.tar.gz

You need to configure gitlab yourself, create a new project and upload it. Detailed steps will not be introduced here

If Auto DevOps is checked in the project, please cancel it. It will affect our cicd

Project introduction
This environment uses the go environment, just for demonstration. Java and php are built in much the same way. The first demo can be done through my template

This actually refers to a script format similar to Jenkins pipeline

Introduction to gitlab-ci.yaml

image:
  name: golang:1.10.3-stretch    #The global image here is the global image address when there is no image defined in the following stage. But when we specify a mirror, we do not use global variables
  entrypoint: ["/bin/sh", "-c"]

# The problem is that to be able to use go get, one needs to put
# the repository in the $GOPATH. So for example if your gitlab domain
# is mydomainperso.com, and that your repository is repos/projectname, and
# the default GOPATH being /go, then you'd need to have your
# repository in /go/src/mydomainperso.com/repos/projectname
# Thus, making a symbolic link corrects this.
before_script:    #In our go environment, the default go path is under our go directory, so we need to create an src under the go directory. Equivalent to an address in our project. Then we will add the project address ln to our go path address
  - mkdir -p "/go/src/git.qikqiak.com/${CI_PROJECT_NAMESPACE}"
  - ln -sf "${CI_PROJECT_DIR}" "/go/src/git.i4t.com/${CI_PROJECT_PATH}"
  - cd "/go/src/git.i4t.com/${CI_PROJECT_PATH}/"

stages:             ##Here, stages represents a project. One or more tasks can be created under each name and executed at the same time. Before that, the next stage will be executed after the stage is executed
  - test
  - build
  - release
  - review
  - deploy

test1:
  stage: test       #In this case, the stage is used to match the task tag, and test1 and test2 belong to the formation operation, not the serial operation
  script:
    - make test

test2:
  stage: test
  script:
    - sleep 3
    - echo "We did it! Something else runs in parallel!"

compile:      #This build phase performs a compile task
  stage: build           #When we finish the test above, we will start to perform the build operation
  script:      #Execute a specific command through script. Here is the make build operation
    # Add here all the dependencies, or use glide/govendor/...
    # to get them automatically.
    - make build     #The build here is the script in our build Makefile. Of course, you can also modify other commands
  artifacts:     #Here is to mount a value of our build. You can go to the following job to get it. We can also get this file on the browser and download it directly. It's equivalent to exposing the file of app
    paths:
      - app

image_build:       #At the release stage, I have done an imageBuild operation (i.e. image build). You need the docker command to use docker build
  stage: release
  image: docker:latest   #The image here is the image using docker, which provides a docker command. When we build a docker, we need to submit all files to a docker daemon for construction. So we need a service to specify the services that docker depends on. Now we refer to an image of docker in docker. This service is actually a service that our docker image depends on. If we also depend on other images, just add them below
  variables:          #The configuration here is to specify a host of docker
    DOCKER_DRIVER: overlay
    DOCKER_HOST: tcp://localhost:2375
  services:      #This is 2375, which represents the port exposed by our service. It is equivalent to associating the client of our docker with the service. In this way, when we execute the command of docker build, we will call our docker daemon to perform a build operation
    - name: docker:17.03-dind
      command: ["--insecure-registry=registry.i4t.com"]   #You can write private warehouse here, but for demonstration, I will change it to public warehouse directly. If you don't write warehouse address here, 509 error will appear
  script:   #This is to execute a real script to build an operation
    - docker info
    - docker login -u "${CI_REGISTRY_USER}" -p "${CI_REGISTRY_PASSWORD}" registry.i4t.com   #The user name and password here are not easy to write, so they are written as variables here. Add configuration in gitlab project
    - docker build -t "${CI_REGISTRY_IMAGE}:latest" .
    - docker tag "${CI_REGISTRY_IMAGE}:latest" "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}"    #It should be noted that the ci'commit'ref'name here is actually the version number of gitlab commit. It has been used in jenkins many times before, and it will not be described in detail (please click i4t.com to find jenkins document)
    - test ! -z "${CI_COMMIT_TAG}" && docker push "${CI_REGISTRY_IMAGE}:latest"
    - docker push "${CI_REGISTRY_IMAGE}:${CI_COMMIT_REF_NAME}"

deploy_review:        #Here is a preview environment, equivalent to the development environment. 
  image: registry.cn-beijing.aliyuncs.com/abcdocker/k8s:kubectl     #The image here is a kubectl command. In fact, the image has a built-in binary kubectl command. However, we know that if we want to use the kubectl command, we need to provide some relevant configurations of apserver before we can connect to our kubernetes cluster
  stage: review
  only:
    - branches   #Our job task will only be executed if it is submitted to the branch specified by our gitlab. If you submit a tag, it will not trigger our task
  except:
    - tags
  environment:    #Here we define an environment and a url in it. In this way, when we trigger the review job, we can access it through the following domain name
    name: dev
    url: https://dev-gitlab-k8s-demo.i4t.com if traifik is installed here, it can be directly bound to host or configured through dns, or svc
    on_stop: stop_review    #Here we add a callback to do one thing when we stop our dev environment (the stop? Review here is shown below)
  script:   #The command here is to replace some operations of the regular image version number. The deployment and other related files need to be submitted to gitlab
    - kubectl version
    - cd manifests/
    - sed -i "s/__CI_ENVIRONMENT_SLUG__/${CI_ENVIRONMENT_SLUG}/" deployment.yaml ingress.yaml service.yaml
    - sed -i "s/__VERSION__/${CI_COMMIT_REF_NAME}/" deployment.yaml ingress.yaml service.yaml
    - |
      if kubectl apply -f deployment.yaml | grep -q unchanged; then
          echo "=> Patching deployment to force image update."
          kubectl patch -f deployment.yaml -p "{\"spec\":{\"template\":{\"metadata\":{\"annotations\":{\"ci-last-updated\":\"$(date +'%s')\"}}}}}"
      else
          echo "=> Deployment apply has changed the object, no need to force image update."
      fi
    - kubectl apply -f service.yaml || true
    - kubectl apply -f ingress.yaml
    - kubectl rollout status -f deployment.yaml
    - kubectl get all,ing -l ref=${CI_ENVIRONMENT_SLUG}

stop_review:   #Here is the stop [review] referenced in the review above. One thing to do before you stop
  image: registry.cn-beijing.aliyuncs.com/abcdocker/k8s:kubectl
  stage: review
  variables:
    GIT_STRATEGY: none
  when: manual
  only:         #Except for our master branch and tag, other branches can perform a branches job task
    - branches
  except:
    - master
    - tags
  environment:
    name: dev   #The environment here is also dev
    action: stop   #The action to be executed, here represents the stop action to be executed
  script:     #The command executed here is to delete the image and related configuration in the previously built project
    - kubectl version  
    - kubectl delete ing -l ref=${CI_ENVIRONMENT_SLUG}    #Here, match and delete through ref tag (the environment configuration here is all referenced in our manifests directory)
    - kubectl delete all -l ref=${CI_ENVIRONMENT_SLUG}

deploy_live:      #Here is the real deployment phase, and the real deployment (here is the definition of a deployment "live" task)
  image: registry.cn-beijing.aliyuncs.com/abcdocker/k8s:kubectl  #The kubectl command is still used here
  stage: deploy
  environment:    #An online environment labeled live is defined here
    name: live
    url: https://live-gitlab-k8s-demo.i4t.com this is an online address after the deployment
  only:
    - tags    #Only when we submit a tag can we perform the job task
  when: manual   #The parameters here represent manual execution
  script:   #The following command operation is the same as the command configured in the previous dev environment, which is actually an operation to replace the image
    - kubectl version
    - cd manifests/
    - sed -i "s/__CI_ENVIRONMENT_SLUG__/${CI_ENVIRONMENT_SLUG}/" deployment.yaml ingress.yaml service.yaml
    - sed -i "s/__VERSION__/${CI_COMMIT_REF_NAME}/" deployment.yaml ingress.yaml service.yaml
    - kubectl apply -f deployment.yaml
    - kubectl apply -f service.yaml
    - kubectl apply -f ingress.yaml
    - kubectl rollout status -f deployment.yaml
    - kubectl get all,ing -l ref=${CI_ENVIRONMENT_SLUG}

Now we need to configure kubectl. By default, kubectl needs a certificate to connect to the k8s cluster. By default, / root/.kube/config is used on the server. gitlab also has the location to add certificates. Let's add the configuration

Click Add cluster

Let's talk about it briefly

Kubernetes cluster name can be written freely
 The API URL here is actually the API server address, which can be viewed through kubectl cluster info

CA TOken can be obtained from / root/.kube/config or the following methods

Obtain certificate token and other related configurations

#In addition to using / root/.kube/config, you can also get

#First, create a namespace

kubectl create ns gitlab

#Because the rbac permission will be involved in the operation, a rbac file will be created here. We bind the permissions of cluster to ServerAccount
[root@yzsjhl82-135 ~]# cat gitlabdemo-sa.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab
  namespace: gitlab

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: gitlab
subjects:
  - kind: ServiceAccount
    name: gitlab
    namespace: gitlab
roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: cluster-admin

Cluster admin is a built-in role, which has all the permissions of the cluster

Then we create

[root@abcdocker ~]# kubectl create ns gitlab
[root@abcdocker ~]# kubectl apply -f gitlabdemo-sa.yaml
serviceaccount/gitlab unchanged
clusterrolebinding.rbac.authorization.k8s.io/gitlab created

[root@abcdocker ~]# kubectl get sa -n gitlab
NAME      SECRETS   AGE
default   1         74m
gitlab    1         3m10s

The serviceaccount we created is actually a secret. Next, we get the token and ca

[root@abcdocker ~]# kubectl get secret -n gitlab
NAME                  TYPE                                  DATA   AGE
default-token-pmlvw   kubernetes.io/service-account-token   3      112m
gitlab-token-5cgx2    kubernetes.io/service-account-token   3      40m

[root@abcdocker ~]# kubectl get secrets gitlab-token-5cgx2 -n gitlab -o yaml
apiVersion: v1
data:
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUR4akNDQXE2Z0F3SUJBZ0lVR29Balg2cGdQVUJkL1ZEU2RrbzlvOFR5ckQwd0RRWUpLb1pJaHZjTkFRRUwKQlFBd2FERUxNQWtHQTFVRUJoTUNRMDR4RURBT0JnTlZCQWdUQjBKbGFVcHBibWN4RURBT0JnTlZCQWNUQjBKbAphVXBwYm1jeEREQUtCZ05WQkFvVEEyczRjekVTTUJBR0ExVUVDeE1KTkZCaGNtRmthV2R0TVJNd0VRWURWUVFECkV3cHJkV0psY201bGRHVnpNQ0FYRFRFNU1EZ3lOakV6TURNd01Gb1lEekl4TVRrd09EQXlNVE13TXpBd1dqQm8KTVFzd0NRWURWUVFHRXdKRFRqRVFNQTRHQTFVRUNCTUhRbVZwU21sdVp6RVFNQTRHQTFVRUJ4TUhRbVZwU21sdQpaekVNTUFvR0ExVUVDaE1EYXpoek1SSXdFQVlEVlFRTEV3azBVR0Z5WVdScFoyMHhFekFSQmdOVkJBTVRDbXQxClltVnlibVYwWlhNd2dnRWlNQTBHQ1NxR1NJYjNEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURLNm02UXd2dUwKYWw1S1h2aVFlbDJXL0pFTm5OSVVsalNmUHhraVE2NUloUStLaEVaeEJ3S0hZczNzMFdYMDFVQUtVeDNVVmMxSgo2V3pQL3MvbGlCaC8xVmxwOXpNRFR1OEhjbWdKVW01VnBmTTFhQ1d4eXk2bHZXVWl4V1FadVNCSjFZT2NJN3R4ClZYbktxR3FnS2RuTEJyamJGSWNtdnJFV01RMmRUcjBOWUZuaVd5UkRmQXlpdW11NEtnZEVZSmlIYXFXNm1xYVMKK1d3czFaWEZzRHl6SzNvQWxmTTZ4ZEhXMk9OQ0JadFovaTFtMTBsV3FTS0hsZlBRU1VOcDRMaURHWElaTFpKeApOQ1Z5T2lINFI4T1RaM3dhME9yOVhEeVEzMmIrNkhYa01FMTAvaFc5bnRlanRCZXExVzNjMTVOeVRzemJYYWQzCjJyZ3ZvZ1JtcFNwUkFnTUJBQUdqWmpCa01BNEdBMVVkRHdFQi93UUVBd0lCQmpBU0JnTlZIUk1CQWY4RUNEQUcKQVFIL0FnRUNNQjBHQTFVZERnUVdCQlErQ3RaTWZrbGV1MmtCY0hFc2RzcVMxRVJ5YkRBZkJnTlZIU01FR0RBVwpnQlErQ3RaTWZrbGV1MmtCY0hFc2RzcVMxRVJ5YkRBTkJna3Foa2lHOXcwQkFRc0ZBQU9DQVFFQUF4Wlo5R0NOCmdJMzVWaDNvdE9IUkw4eXJSSEw5b05qbzdVQnZEU2JBZkp2dDd1WGx3WmF0bXBZTUN1MEpVUnJnMWM0Rng3cWUKUHdOQVNnQUFuVHZKZmdEYlZRWDlTM25CcytJYzdTUE5wdXVacTloZzhmSHkyaHRYZ0RiL3lOMkdpNlBBK29zTwpWdWRTYXI5S0tMcVJzNnBuU2tPOEJkbmtPR2llUks5WnJIOVlnS3VodFVhbmZGSEhJeTE0ZXJqQ0s3YUV0S2FOCmxJQVNhV2crYkNnK29JdklqcjdyRzZoU1VFZWkwdVJGbHg5OXN0OVBSeUVaVTZ2MVpETEt6bjY0MUp1elREeDkKTVhLdUtKQUF1anUyc1c0VjZ2VEowR1pGUGNobkxrZSs1NC9QSnVsLytzVExRS1puVE9EMm9KdW1VMEFiVnFMRwpZa1VLUWJ2dUhzMW9sUT09Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
  namespace: Z2l0bGFi
  token: ZXlKaGJHY2lPaUpTVXpJMU5pSXNJbXRwWkNJNklpSjkuZXlKcGMzTWlPaUpyZFdKbGNtNWxkR1Z6TDNObGNuWnBZMlZoWTJOdmRXNTBJaXdpYTNWaVpYSnVaWFJsY3k1cGJ5OXpaWEoyYVdObFlXTmpiM1Z1ZEM5dVlXMWxjM0JoWTJVaU9pSm5hWFJzWVdJaUxDSnJkV0psY201bGRHVnpMbWx2TDNObGNuWnBZMlZoWTJOdmRXNTBMM05sWTNKbGRDNXVZVzFsSWpvaVoybDBiR0ZpTFhSdmEyVnVMVFZqWjNneUlpd2lhM1ZpWlhKdVpYUmxjeTVwYnk5elpYSjJhV05sWVdOamIzVnVkQzl6WlhKMmFXTmxMV0ZqWTI5MWJuUXVibUZ0WlNJNkltZHBkR3hoWWlJc0ltdDFZbVZ5Ym1WMFpYTXVhVzh2YzJWeWRtbGpaV0ZqWTI5MWJuUXZjMlZ5ZG1salpTMWhZMk52ZFc1MExuVnBaQ0k2SWpFNFptWXlabVpqTFdWak1UVXRNVEZsT1MxaE5UZGxMVFV5TlRRd01HUTNZakk0TkNJc0luTjFZaUk2SW5ONWMzUmxiVHB6WlhKMmFXTmxZV05qYjNWdWREcG5hWFJzWVdJNloybDBiR0ZpSW4wLmxacC02bGdnXzUzcnJiYjI1Y0xQWnlXSlpLQ0JQYUJSc0IybDAzMmNHSEFzSWpLdmJOemVpQnJ4M2U4SGNoVjVxNGJ5NDljem53dmkyeXAzZDJ0TGpZOTZTTUt3UDRQQUJjcFozci1YM2NMUFEzTGZrUkhXZlJyQktNY3BYbm1mRmRnZUJ4LXFTRS00ZFo3LTF3ZzNwTDV1ZkZWaEo1OTlZbHBqXzU4OFpxc0c2YTk4WjR1NDVyeTlpSHd2QV9KRDVDcGRvU1FObDk1ZGw2dDlra0c0NHZET2N1cG5WQkIxS1ByeXJsY3FyMlozVjJ4MEVHQWRxSjdUU0JwZWxSaTIyVmFHcGRyMmdhUjV4dTU4cEVKUjdDYl9BUnBHRVZaZ0JvZF9SWmtWU0FtTkdGVVV2YWZSdHBNZ3RLVTM4RkMzZ1drVVBleVZMY2o4Yk9DQTdscndnUQ==
kind: Secret
metadata:
  annotations:
    kubernetes.io/service-account.name: gitlab
    kubernetes.io/service-account.uid: 18ff2ffc-ec15-11e9-a57e-525400d7b284
  creationTimestamp: "2019-10-11T10:51:36Z"
  name: gitlab-token-5cgx2
  namespace: gitlab
  resourceVersion: "6743518"
  selfLink: /api/v1/namespaces/gitlab/secrets/gitlab-token-5cgx2
  uid: 18f0217c-ec15-11e9-8945-525400a79155
type: kubernetes.io/service-account-token

We have seen ca.crt and token above, but we can't use them. You also need to use base64 for transcoding

echo "xxxx"|base64 -d

#Need to copy all, including -- BEGIN
#Generate token in the same way

Then we click ADD

If we click the add prompt to report an error, we need to add an operation that allows the hook

Error graph

Solution
Admin area = > Settings = > network, then click the "expand" button to the right of Outbound requests

The addition is completed as shown in the figure below (now we have access to our cluster)

In the. gitlab-ci.yaml file, we refer to many environment variables (such as the docker account password file, etc.) which can't be displayed in clear text. Therefore, variables are used in the configuration file. Now we need to configure this variable on gitlab

Add the image address below, and the account password of the container warehouse will be OK
CI_REGISTRY_USER
CI_REGISTRY_PASSWORD
CI_REGISTRY_IMAGE

gitlab ci variable document https://docs.gitlab.com/ee/ci...

We have submitted git code before. If we do not submit the test, we can decompress the following files, and submit them once

wget http://down.i4t.com/abcdocker-gitlab-demo.tar.gz
 Code file, including gitlab ci script and this demonstration environment

Let's submit a code here for demonstration

In our Kube OPS namespace, gitlab will generate the runner image, which contains the image defined by our ci script and the runner image

Next we can see the ci build steps in gitlab

Build note that without the. gitlab-ci.yaml file, gitlab does not trigger the CI script

Click below to see the detailed construction steps

You can see the detailed construction steps, which are the same as Jenkins pipeline

In the build phase, we added a variable so that we can download the packed file after the build is completed

When the build is complete, it will appear below, and you can also build repeatedly

Next, we can check the status of pod and svc

It can be accessed by using ingress or by nodeport. Here I use the nodeport mode for access demonstration

We can modify it according to our own needs

Summary
1. I use the commit version number to ensure the uniqueness of the pod version number
2. Only the master that submits the code here will be punished.
3. My image warehouse uses Alibaba cloud

If you have any questions about the document, please comment below. I will revise it in time and answer questions In a hurry, you can click Add group at the bottom right, group @ administrator
21:54:47, October 15, 2019

Tags: Linux GitLab Docker Redis Kubernetes

Posted on Fri, 08 Nov 2019 04:21:17 -0500 by bkanmani