jinkens+gitlab implement CI/CD for k8s cluster

I. environmental preparation

  • K8s cluster environment (here I am k8s cluster of three sets);
  • A single docker server is mainly used to upload images to the private warehouse. Jenkins and gitlab are also deployed in this server;
  • There are 4 servers in the above environment, all of which point to the same private warehouse for sharing docker image;
  • Server IP is, 20.3, 20.4, 20.5 (the first three IP are nodes in K8s cluster)

Jenkins is deployed in the way of war package. It needs to use tomcat environment and refer to the blog for deployment;
For other environment deployment, please refer to the following blog:
Tomcat installation and optimized configuration
Detailed configuration of Docker installation
Introduction and installation of K8s (Kubernetes).

Note: all package files used below can be found in my network disk link download.

II. Deployment of registry private warehouse

Any node that can run the docker container can be deployed. Here I choose

1. Run private warehouse container

[root@jenkins ~]# docker run -tid --name registry --restart=always -p 5000:5000 -v /data/registry:/var/lib/registry registry

2. Configure each server to point to the private warehouse

#Select any node to do the following
[root@jenkins ~]# vim /usr/lib/systemd/system/docker.service 
#Fix the following configuration item to specify the listening address of its private warehouse
ExecStart=/usr/bin/dockerd -H unix:// --insecure-registry
#Send the modified profile to other nodes
[root@jenkins ~]# scp /usr/lib/systemd/system/docker.service
[root@jenkins ~]# scp /usr/lib/systemd/system/docker.service
[root@jenkins ~]# scp /usr/lib/systemd/system/docker.service

3. Restart the docker service for each node to make the changes take effect

The following operations need to be configured at each node in turn.

[root@jenkins ~]# systemctl daemon-reload
[root@jenkins ~]# systemctl restart docker

III. deploy Jenkins service and start it

Before deploying Jenkins, deploy tomcat service by yourself. Refer to the blog: Tomcat installation and optimized configuration.

The source package and plug-ins needed for the deployment of Jenkins can be found in my online disk link download.

[root@jenkins src]# cd /usr/local/tomcat/webapps/
#Delete all contents in the original web page directory (based on the actual situation, it's better to make a backup)
[root@jenkins webapps]# rm -rf * 
[root@jenkins webapps]# rz       #Upload Jenkins' war package
[root@jenkins webapps]# ls
#Specify Jenkins' home directory
[root@jenkins webapps]# vim ../bin/catalina.sh    #Edit this file

#!/bin/sh    #Just add it at the beginning of the file
export CATALINA_OPTS="-DJENKINS_HOME=/data/jenkins"
export JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.ClassicPluginStrategy.noBytecodeTransformer=true"
#Start tomcat
[root@jenkins webapps]# cd ../bin/
[root@jenkins bin]# ./catalina.sh start
#Make sure the port is listening
[root@jenkins bin]# netstat -anpt | grep 8080
tcp6       0      0 :::8080                 :::*                    LISTEN      58560/java          

IV. configure Jenkins service

In order to realize the CI/CD of Jenkins, there must be three plug-ins: GitLab, GitLab HOOK and GitLab authentication. It is not easy to download these plug-ins in China. There are packaged plug-ins in the online disk link above me. Just unzip them to the corresponding directory.

1. Configure Jenkins web interface

When the browser accesses the IP + port / Jenkins of the following tomcat services, such as, it can see the following interface (it is better to disconnect the external network before accessing, otherwise, after entering the password, the plug-in will be installed online, which is slow and likely to fail):

Check the password according to the prompts and paste it to the corresponding location, as follows, check the password:

[root@jenkins webapps]# cat /data/jenkins/secrets/initialAdminPassword

After entering the password, wait for a moment, and then click as follows:

Create corresponding user:

After successful login, it is as follows:

Now close the browser page and add a plug-in to Jenkins:

2. Configure the required plug-ins

[root@jenkins src]# rm -rf /data/jenkins/plugins/   #Delete the original empty directory
[root@jenkins src]# tar zxf plugins.tar.gz -C /data/jenkins/
#Restart tomcat to take effect
[root@jenkins src]# /usr/local/tomcat/bin/catalina.sh stop
[root@jenkins src]# /usr/local/tomcat/bin/catalina.sh start

Visit Jenkins's web interface again:

You can see that it has become a Chinese page, indicating that the plug-in configuration is effective, as follows:

V. deployment of gitlab services

After thinking about it, I still don't want to take a screenshot to write it. Please refer to my previous blog to deploy gitlab. I also watched that blog deployment (don't operate after installing gitlab, you need to see the following precautions, change the monitoring port), and finally make sure that you can clone the remote library to the local. Here I deploy gitlab and Jenkins on the same server. Reference blog: Gitlab installation and application of continuous integration

matters needing attention!!!

After installing gitlab, you need to change its listening port (to prevent port conflicts), as follows:

[root@jenkins src]# vim /etc/gitlab/gitlab.rb     #The changes are as follows
external_url ''
unicorn['listen'] = ''       #Native IP
unicorn['port'] = 3000

After you change the listening port, you can follow that blog for the next operation (but the port is different from that blog, for example, to query whether the port is listening again, you should see port 90 instead of port 80. To visit the browser page, you need to add port 90).

Finally, ensure that there is a local clone library, as follows:

[root@jenkins ~]# ls -d teset01/

Vi. configure Jenkins to create a task

Paste the following shell script content into the web interface, as follows:

#In the script, is the IP address of the private warehouse, and is the IP address of the master in the k8s cluster
backupcode="/data/backcode/$JOB_NAME/$BUILD_NUMBER"   #The default variables of Jenkins are referenced here
mkdir -p $backupcode
chmod 644 "$JENKINS_HOME"/workspace/"$JOB_NAME"/*
rsync -acP   "$JENKINS_HOME"/workspace/"$JOB_NAME"/*  $backupcode
echo From > "$JENKINS_HOME"/workspace/Dockerfile
echo COPY ./"$JOB_NAME"/* /usr/share/nginx/html/ >> "$JENKINS_HOME"/workspace/Dockerfile
docker rmi        
docker build -t /"$JENKINS_HOME"/workspace/.
docker push
ssh root@ kubectl delete deployment nginx
ssh root@ kubectl apply -f /root/nginx.yaml

After the script is filled in, it is as follows:

After the script is filled in, do not save it first. Copy the Jenkins address as follows:

7. Enable anonymous access of Jenkins

VIII. Go back to gitlab and enable sending web hook to yourself

The following operations are performed in the new database. I forgot to cut the diagram of the step of switching to the new database:

After saving, pull down the page to see the newly added web hook. Click the following to test:

If status code 200 is returned, the configuration is correct, as follows:

IX. configure the master node of the Jenkins password free login k8s cluster

The jenkins server is configured as follows:

[root@jenkins ~]# ssh-copy-id root@
#The IP above is the master node IP in the K8s cluster

IX. test continuous CI/CD effect

1. Running nginx resource object in k8s cluster

#Upload the required image to the private warehouse
[root@jenkins ~]# docker tag nginx:latest
[root@jenkins ~]# docker push 
#Running nginx resource object on the master node
[root@master ~]# vim nginx.yaml     #Write yaml file

apiVersion: extensions/v1beta1
kind: Deployment
  name: nginx
  replicas: 2
        name: nginx
        - name: nginx
          - containerPort: 80

apiVersion: v1
kind: Service
    name: nginx
  name: nginx
  type: NodePort
  - port: 80
    targetPort: 80
    nodePort: 31234
    name: nginx
[root@master ~]# kubectl apply -f nginx.yaml     #Execute yaml file
[root@master ~]# kubectl get pod | grep nginx    #Make sure the container is working properly
nginx-76645bc84f-rbgtl   1/1     Running   0          68s
nginx-76645bc84f-s6xp6   1/1     Running   0          68s

2. Client access nginx

3. Update and iterate test the version in gitlab

#Upload files to gitlab in the local library
[root@jenkins ~]# cd teset01/
[root@jenkins teset01]# git config --global user.name "test"
[root@jenkins teset01]# git config --global user.email "test@test.com"
[root@jenkins teset01]# echo "test ....." > index.html
[root@jenkins teset01]# git add *
[root@jenkins teset01]# git commit -m "Test CI/CD" 
[root@jenkins teset01]# git push origin master 

After the above operations, you can see the successful construction information in Jenkins's new task, as follows:

Visit the homepage of nginx again and find that it has become the content we submitted on gitlab, as follows:

Test complete.

Tags: jenkins Nginx Docker GitLab

Posted on Wed, 04 Dec 2019 13:45:08 -0500 by GOKU