Building Immutable Infrastructure with Packer and GitLab CI

What is Packer?

Packer is an Open Source tool that can be used to create identical images for different platforms, what it makes the process of creation and deployment of your infrastructure to be simple, as it uses a single configuration file.

Builders

Builders are Packer components that are responsible of creating a machine and generate correspondent image. There are builders for Docker, Vagrant, VirtualBox, VMWare, etc.

Check the documentation here for more information.

Provisioners

Provisioners are tools that can be used to install software and configure it in generated images. Shell scripts and tools like Ansible, Chef and Puppet can be used for provisioning.

Check the documentation here for more information.

Installation

Packer can be installed on Windows, Mac and Linux following instructions in the documentation. If you're using Linux, Packer is available from the repositories of some distributions, in other case you can follow the installation instructions to add the official repository to the system configuration.

If you are building a Docker image, your must install Docker first on your system.

Templates

For configuring Packer on what instructions to run and what plugins to use for building the image, you must use a template. There are two formats, JSON, that is the template that has been used historically, and HCL, that stands for HashiCorp Configuration Language, the recommended format since version 1.7.0. The template file must be named using .pkr.json or .pkr.hcl extensions.

Using Packer for Vagrant

What is Vagrant?

Vagrant is a tool for building and managing virtual environments. It works with VirtualBox, Docker, VMWare as providers and provisioning can be done using shell scripts and tools like Ansible, Chef and Puppet.

Using JSON template

{
 "provisioners": [
 {
 "type": "shell",
 "execute_command": "echo 'vagrant' | {{.Vars}} sudo -S -E bash '{{.Path}}'",
 "script": "scripts/install.sh"
 }
 ],
 "builders": [
 {
 "communicator": "ssh",
 "source_path": "ubuntu/focal64",
 "provider": "virtualbox",
 "type": "vagrant",
 "box_name": "python-dev"
 }
 ]
}

In the code snippet above, there are two arrays in the JSON object, builders of vagrant type as a Vagrant box is being built, and provisioners of shell type as a shell script is being used for provisioning.

In builders array:

• communicator is set to ssh as an SSH connection is required to access the virtual environment created with Vagrant
• source_path is equal to ubuntu/focal64. This is the based image that will be used for building the final image for the infrastructure.
• virtualbox is used as provider for Vagrant.
• type is set to vagrant.
• python-dev will be the name of the box created.

In provisioners array:

• type of provisioner is shell as a Bash script is being used for provisioning.
• execute_command is the set of instructions to run when the provisioning process is started.
• script is the path of the Bash script.

Using HCL template

source "vagrant" "autogenerated_1" {
 add_force = true
 box_name = "python-dev"
 communicator = "ssh"
 provider = "virtualbox"
 source_path = "ubuntu/focal64"
}

build {
 sources = ["source.vagrant.autogenerated_1"]

 provisioner "shell" {
 execute_command = "echo 'vagrant' | {{ .Vars }} sudo -S -E bash '{{ .Path }}'"
 script = "scripts/setup.sh"
 }

}

HCL is the recommended template since version 1.7.0 and it contains the same information as in the JSON template.

If you already have a JSON template but want to use HCL instead, you can run the following command to generate .pkr.hcl file from .pkr.json file:

$ packer hcl2_upgrade template.pkr.json

Building image

For building the image, run any of the following commands:

$ packer build config.pkr.json
$ packer build config.pkr.hcl

Above command will create an output_vagrant directory where package.box will be stored and a Vagrantfile will be created.

Running Virtual Env

Before starting the virtual environment, edit the Vagrantfile in the output_vagrant directory, removing the following lines:

config.vm.define "source", autostart: false do |source|
    source.vm.box = "ubuntu/focal64"
    config.ssh.insert_key = false
 end

Start your virtual environment running:

$ vagrant up
$ vagrant ssh

Using Packer for Docker

packer {
 required_plugins {
 docker = {
 version = ">= 0.0.7"
 source = "github.com/hashicorp/docker"
 }
 }
}

variable "login_username" { 
 type = string
 default = "username"
}

variable "login_password" { 
 type = string
 default = "password"
}

source "docker" "ubuntu" {
 image = "ubuntu:xenial"
 commit = true
}

build {
 sources = [
 "source.docker.ubuntu"
 ]

 provisioner "shell" {
 script = "./scripts/install.sh"
 }

 post-processors {
 post-processor "docker-tag" {
 repository = "user/ubuntu-docker"
 tags = ["latest"]
 only = ["docker.ubuntu"]
 }

 post-processor "docker-push" {
 login = true
 login_username = var.login_username
 login_password = var.login_password
 }
 }
}

For building a Docker image using Packer, a plugin is required and can be obtained from GitHub. The plugin being used must be specified in the required_plugins block in the template.

Post-processors

Post-processors are instructions run after the image is created and provisioned. In the code snippet above, the Docker image is published on Docker Hub after the image is built.

Two post-processors are defined, docker-tag, where repository and tags are set, and docker-push, where authentication details are specified, including username and password (or access token) at Docker Hub.

login_username and login_password in the docker-push post-processor are set after getting values from the command line when running packer build. These variables are set in the template:

variable "login_username" { 
 type = string
 default = "username"
}

variable "login_password" { 
 type = string
 default = "password"
}

Where type is set to string and a default value is assigned. The default value will be replaced later.

Building image

For building the Docker image run:

packer build -var login_username=”USER” -var login_password=”PASSWORD” ubuntu-docker.pkr.hcl

Image will be built and published on Docker Hub.

GitLab CI

You can use GitLab CI for automating the building process.

At first you must:

• Create a GitLab repository.
• Upload your .pkr.hcl template.
• Upload your scripts directory. This contains your provisioning script, a shell script.

Configure GitLab CI

• Go to Setting --> CI/CD in your repository.
• Click on Expand in the Variables section.
• Add the CI_REGISTRY_USER variable. The value will be your Docker Hub username.
• Add the CI_REGISTRY_PASSWORD. The value will be your Docker Hub password or access token.
• Create the .gitlab-ci.yml file in your repository. It will have two stages, test and deploy, and two jobs, test-job and deploy-job.

image:
 name: docker:latest

before_script:
 - echo "Fetching packer"
 - wget https://releases.hashicorp.com/packer/1.7.8/packer_1.7.8_linux_amd64.zip
 - unzip packer_1.7.8_linux_amd64.zip
 - chmod +x packer
 - ./packer init .

stages:
 - test
 - deploy

test-job:
 stage: test
 script:
 - echo "Validating Packer template"
 - ./packer validate ./ubuntu-docker.pkr.hcl

deploy-job:
 stage: deploy
 services:
 - docker:dind
 script:
 - echo "Deploying Docker image..."
 - ./packer build -var login_username=$CI_REGISTRY_USER -var login_password=$CI_REGISTRY_PASSWORD ./ubuntu-docker.pkr.hcl
 - echo "Docker image successfully published."

Latest Docker image of Docker is required and before running any of the jobs, Packer must be installed as specified in before_script. To validate the Packer template, the following command is run in the test-job:

packer validate ubuntu.docker.pkr.hcl

In the deploy-job, the values of login_username and login_password are assigned from the variables in the configuration of the repository, CI_REGISTRY_USER and CI_REGISTRY_PASSWORD.

Once you commit that file, the pipeline is run and when it finishes your Docker image will be available from Docker Hub.