A Practical Guide To Building A Standard Continuous Integration Pipeline with Jenkins

Visit my website: Practical Cloud for more posts and projects on Devops and Cloud engineering.

Continuous Integration is a devops methodology whereby the integration of code changes from multiple contributors into a single software project is automated.

The Version Control system is the crux of continuous integration. Developers frequently commit to a shared repository using a version control system such as Git.

A continuous integration service automatically builds and runs unit tests on the new code changes to immediately surface any errors.

A key benefit of continuous integration is that One of the key benefits of integrating regularly is that you can detect errors quickly and locate them more easily. As each change introduced is typically small, pinpointing the specific change that introduced a defect can be done quickly.

To-Do

In this project we will use python to build a simple HTTP server. The application code will be pushed into our remote GitHub repository. This will immediately trigger our Jenkins pipeline which runs static tests on the code using SonarQube, builds a container image of the application using Docker, conatinerizes and briefly runs the application. After that the application image will be pushed into our DockerHub repository.

Here is our reference architecture:

Stage 1: Create EC2 Instances

We will be using three servers for this demonstration, our Jenkins server and Sonarqube server.

1. Create The Jenkins Server:

• On the AWS EC2 dashboard, click on launch instance, select the "ubuntu 22.04 LTS" AMI. Also make sure your instance type is at least t2.medium.
• On the security group section, allow inbound traffic on port 80 and 8080
• Scroll down to the userdata section and add the following bash script:

#!/bin/bash

# Install jenkins
wget -q -O - https://pkg.jenkins.io/debian-stable/jenkins.io.key |sudo gpg --dearmor -o /usr/share/keyrings/jenkins.gpg
sudo sh -c 'echo deb [signed-by=/usr/share/keyrings/jenkins.gpg] http://pkg.jenkins.io/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
curl -fsSL https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key | sudo tee \
  /usr/share/keyrings/jenkins-keyring.asc > /dev/null
echo deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] \
  https://pkg.jenkins.io/debian-stable binary/ | sudo tee \
  /etc/apt/sources.list.d/jenkins.list > /dev/null
sudo apt update
sudo apt-get install openjdk-11-jdk -y
sudo apt install -y openjdk-17-jre
sudo apt install jenkins -y

# Install Docker
sudo apt-get update -y 
sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu jammy stable" -y
apt-cache policy docker-ce
sudo apt install docker-ce -y
sudo usermod -aG docker jenkins

• Visit the instance IP address on port 8080 and create a default admin user account.
• Remember to select "Install suggested plugins".

2. Create The SonarQube Server

• Follow through with similar steps as in "1" above with the following exceptions.
• Allow inbound traffic on port 9000.
• Add the following bash script to the userdata section:

#!/bin/bash

apt update
apt install -y unzip
apt install -y openjdk-17-jre
cd /opt
wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-10.3.0.82913.zip?_gl=1*re8o0g*_gcl_au*MTQxNjEwNjE0NS4xNzAxODYxNTUw*_ga*MTc0MDc0NTE4Mi4xNzAxODYxNTUx*_ga_9JZ0GZ5TC6*MTcwMjAzMzQ3Ni40LjEuMTcwMjAzMzUwMi4zNC4wLjA.
unzip *
rm 'sonarqube-10.3.0.82913.zip?_gl=1*re8o0g*_gcl_au*MTQxNjEwNjE0NS4xNzAxODYxNTUw*_ga*MTc0MDc0NTE4Mi4xNzAxODYxNTUx*_ga_9JZ0GZ5TC6*MTcwMjAzMzQ3Ni40LjEuMTcwMjAzMzUwMi4zNC4wLjA.'
chown -R ubuntu sonarqube-10.3.0.82913

• Once the instance has initialzed, connect to the instance.
• cd to the "/opt/sonarqube-10.3.0.82913/bin" directory.
• Run the command: sudo ./sonar.sh console.
• Your SonarQube server should operational (This might take some time).
• Visit the instance public IP on port 9000 to login.
• The default username and password is "admin".

Now you should have all two servers up and running.

Stage 2: Create The Application Code

• On your local server (or on another EC2 instance), clone this repository.
• Should look like this:

• In the Jenkinsfile, change all occurences of "kelvinskell" to your own DockerHub Username.

Stage 3: Configure The SonarQube Server

• On the brwoser, log in to the sonarqube server.
• Select "Create a local project"
• Click on "next"
• Select "Global settings" and Create your project.
• On the next page, select jenkins.
• On the next page, select GitHub as your devops platform and on "4", select other.
• Note down your project key, you will be needing it later.
• In the Jenkinsfile, under the 'SonarQube Analysis' stage, replace the project key with your own project key: -Dsonar.projectKey=Python-Server-Scan ==> -Dsonar.projectKey=<Your-OWN-PROJECT-KEY
• Click on "Account" > "My Account" > "Generate token".
• Give it a name and click on "generate".

• Save this token for later, we will be using it in the Jenkins configuration stage.

Stage 4: Configure The Jenkins Server

• On the browser, login to the Jenkins server
• Click on "Manage Jenkins" > "Plugins" > "Available plugins"
• Install the following plugins: Docker, Docker Commons, Docker pipeline, SonarQube Scanner, Sonar quality gates, SSH2 Easy.

• Restart the Jenkins server.
• Go to "Manage Jenkins" > "credentials" > "add credentials"
• select secret text, give your credential an ID and paste the SonarQube project token we copied in step 3.
• At this point, we will also add our DockerHub password. This password will be used by Jenkins to login to DockerHub.
• Click on "Add credentials" once more.
• select "secret text", your credentials ID should be "DockerHubPass"
• Input your DockerHub password.

• Go to "Manage Jenkins" > "System".
• Scroll down to the "SonarQube servers" section and click on "add sonarqube".
• give it the name "SonarQube Server", enter the server url and credential ID for the secret token.

• Go to "Manage Jenkins", "Tools".
• Scroll down to the "SonarQube Scanner" section and click on add.

• The SonarQube scanner has to be named "SonarScanner".
• Click on "save".

Stage 5: Configure The Jenkins Pipeline

• Go to Dashboard > New item.
• Select pipeline and name it.

• Since we want this pipeline to be triggered whenever we push to our GitHub repository, we will check the box beside "GitHub hook trigger for Git Scm polling".

• Under the "pipeline" section, select "pipeline script from Scm".
• Enter your GitHub repository url.
• Make sure to configure your SSH key-pair with GitHub. Here is a great guide on how to get that done.

• For the GitHub hook to work properly, unselect the "Lightweight checkout" option.
• Click on "save".

Stage 6: Run The Pipeline

Now, we are done configuring this pipeline. It is time to test our work.
You can either trigger the pipeline by making a change and pushing to your GitHub repository. If the web hook trigger is properly configured, this will automatically start the pipeline.

Alternatively, you can simply click on "Build now" to execute the pipeline.

If everything is configured as they should, you will get this output:

A Final Note

If you made it through this tutorial, congrats on achieving this milestone. Hopefully, this has taught you a lot about the processes and mechanisms involved in building a continuous integration pipeline.

If you want me to write another tutorial on a continuous deployment pipeline to kubernetes (Minikube or EKS), please indicate in the comments.

Connect with me on LinkedIn.