Working with Jenkins, Docker, Git, and GitHub — Part II

I had previously written an article where I gave a glimpse into how Jenkins can be used to automate the product development, deployment, monitoring, and maintenance cycle. This article is going to be along the same lines, but with a different system being set up.

In this article, we can see the integration of Jenkins, Docker, Git, and GitHub with the help of the following objectives:

  1. Create a Docker container image that has Jenkins installed using Dockerfile. When we launch this image, it should automatically start the Jenkins service in the container.
  2. Create a job chain of job1, job2, job3, and job4 using the build pipeline plugin in Jenkins.
  3. Job1: Set up a local Git repository. When the developers push the local repository to GitHub, Jenkins Job 1 must automatically pull the GitHub repository to the localhost.
  4. Job 2: Job 2 must launch specific containers depending on the type of code in each file. For example, if the code is in PHP, then Jenkins must start the container that has PHP installed in it during the respective Dockerfile creation and build.
  5. Test if the code file is working or not. If the code is not working, then an email notification is sent to the developer with error messages.
  6. Job 3: If the container running the code collapses then Job 3 must automatically start the container again with the code file.

To accomplish these objectives, the basic pre-requisites we need are:

  1. A RHEL8 OS launched as a VM. (I use Oracle Virtual Box)
  2. Installation of Git (In the RHEL8 VM or bare metal OS) and Docker (in RHEL8 VM)

Creation of Docker Container Images using Dockerfile

The first objective is to create all the container images that can come into use for us. Currently, the type of code files I am considering are HTML (.html), and Python (.py) files. Hence, we need to create four Dockerfile for launching Jenkins, HTML, and Python files each.

There are multiple methods to create custom Docker container images, but the method we will use is the creation of Dockerfiles. To create a Dockerfile, we first need to set up a workspace from the command line.

# mkdir test-systemd
# cd test-systemd
# gedit Dockerfile
Figure 1: Dockerfile contents for Jenkins container image

The commands used are for the following purpose:

  1. FROM centos - It references the base OS for the new image.
  2. RUN yum install wget -y - Installation of wget command (Used to install Jenkins)
  3. RUN yum install net-tools -y - Installation of net-tools which form the base of Linux OS networking
  4. RUN wget -0 /etc/yum.repos.d/jenkins.repo https://pkg.jenkins.io/redhat-stable/jenkins.repo
  5. RUN rpm --import https://pkg.jenkins.io/redhat-stable/jenkins.io.key - Commands 4. and 5. are meant for the installation of Jenkins. (Refer to the link https://www.jenkins.io/download/ )
  6. RUN yum install java -y && yum install jenkins -y - The installation of Jenkins and Java as Jenkins is dependant on Java for its working.
  7. RUN yum install git -y - Installation of Git to perform commands by interacting with GitHub repositories
  8. RUN yum install gedit -y - Installation of text editor Gedit for performing minor changes in the configuration files of Jenkins to enable email services. (Can be ignored)
  9. RUN yum install openssh-clients -y — The installation allows us to use remote access of other systems from within the container and vice-versa.
  10. RUN echo -e "jenkins ALL=(ALL) NOPASSWD:ALL" /etc/sudoers - This allows Jenkins to have maximum access to perform commands by modifying the sudoers file.

After saving the Dockerfile, we use docker build to save the container image.

# docker build -t jenkins-os:v3 .
# docker run -dit -p 8888:8080 -v /task2:/task2 --name jen-os jenkins-os:v3
# docker exec jen-os sudo cat /var/lib/jenkins/.jenkins/secrets/initialAdminPassword

After we create the docker container image under the name, jenkins-os:v3, we start a container with this image. The options -dit means that there is an interactive terminal running in the background once the container is started. To allow exposure to the outside world, the port 8888 is exposed using PATing and a volume is mounted from the host system to ensure data persistence.

When we launch Jenkins in the web browser using the URL <container IP>:<Port>, we see the installation page of Jenkins, where we can use the initialAdminPassword to login and setup the admin user account. Once we have logged in as the admin user, we need to install several Plugins in the Manage Plugins section of Manage Jenkins option. The plugins which must be installed are:

  1. Github Plugin
  2. Build Pipeline Plugin
  3. Delivery Pipeline Plugin
  4. Email Extension Plugin
  5. SSH Plugin
  6. Publish Over SSH Plugin
  7. SSH Agent Plugin
  8. SSH2 Easy Plugin

Once the plugins are installed, reload Jenkins on the Web UI and to update the list of installed plugins.

We need to enable the communication between the host RHEL8 VM and the Jenkins container using SSH. To achieve this, we need to set up the credentials of RHEL8 as a remote SSH host in the Configure System section of Manage Jenkins.

Figure 2: Setting up communication between RHEL8 and Jenkins container using SSH remote hosts

Since our set up also involves pulling data from Github, we can configure the Webhooks for the GitHub repository as well. This configuration is done on both Jenkins as well as GitHub.

  1. In Jenkins, we set up a Webhook as per the screenshot given below, in the Configure Systems section of Manage Jenkins.
Figure 3: Setting up Webhooks in Jenkins

2. In GitHub, go to Settings and set up the Webhook as shown below.

Figure 4: Setting up GitHub Webhooks in the GitHub Repository

Once all the settings are finished, we can proceed to the jobs in Jenkins.

The Dockerfile and the container image for the containers for Python files can be created similarly. The contents of the Dockerfile is as shown below.

Figure 5: Dockerfile contents for Python container image

Jenkins Jobs

Job 1 asks us to pull the content from the online GitHub repository. The process of using both Git and GitHub repositories and pulling the content from the GitHub repository has been discussed in a previous article, and we will be following the same procedure here.

Figure 6: Jenkins Job 1 (Part 1)
Figure 7: Jenkins Job 1 (Part 2)
Figure 8: Jenkins Job 1 (Part 3)
Figure 9: Jenkins Job 1 (Part 4)

In Figure 8, we have the main shellcode that Job 1 will execute during a successful build. The code essentially copies only the files with the extensions .html and .py into the given directory task2.

Figure 10: Jenkins Job 2 (Part 1)
Figure 11: Jenkins Job 2 (Part 2)
Figure 12: Jenkins Job 2 (Part 3)

In Job 2, we execute shell commands remotely on the host system using SSH. There are two build shell scripts, one for each type of file being handled. The first build shell is involved in launching the container for .html files. The second shell script is involved in loading the .py files and launching the required container.

Figure 13: Jenkins Job 3 (Part 1)
Figure 14: Jenkins Job 3 (Part 2)
Figure 15: Jenkins Job 3 (Part 3)
Figure 16: Jenkins Job 3 (Part 4)

In Jenkins Job 3, we are testing the file(s) to see if they are executed successfully. Hence, we write a remote shell script under Builds for the .html file(s) (Figure 14) and the .py file(s) (Figure 15). Also, the Plugins we have installed for Email Notifications allow us to send emails if the build of Job 2 is unsuccessful.

Figure 17: Jenkins Job 4 (Part 1)
Figure 18: Jenkins Job 4 (Part 2)
Figure 19: Jenkins Job 4 (Part 4)

In Job 4, the containers are monitored to see if they stop running due to any unexpected reason. If they have stopped, then they are restarted again. This monitoring has been configured using a periodic build, where the Job is built every 15 minutes as shown in Figure 16.

Outputs

Based on the set up we have completed, we can see the following outputs

Figure 20: The containers running after the Jobs are executed
Figure 21: Workspace in Jenkins after pulling files from the GitHub repository
Figure 21: Contents of the .html files Docker container
Figure 23: HTML file sample (index.html)
Figure 24: The contents of the Docker container for .py file
Figure 25: Successful execution of .py file
Figure 26: Console Output of Job 1
Figure 27: Partial Console Output of Job 2
Figure 28: Partial Console Output of Job 3
Figure 29: Partial Console Output of Job 4

ECE Undergrad | ML, AI and Data Science Enthusiast | Avid Reader | Keen to explore different domains in Computer Science