Nowadays, if you want to minimize human errors and maintain a consistent process for how software is released then you are going to rely on Continuous integration and continuous deployment (CI/CD). It's really hard to imagine how much productivity they bring into the plate.
In this tutorial, we are going to take entire AWS instance backup using tools like Packer and see how it solves our problem and make our life much easier.
Amazon Machine Image (AMI)
An Amazon Machine Image (AMI) is a special type of virtual appliance that is used to create a virtual machine within the Amazon Elastic Compute Cloud ("EC2"). It serves as the basic unit of deployment for services delivered using EC2. -- Wikipedia
An AMI includes the following:
- A template for the root volume for the instance (for example, an operating system, an application server, and applications)
- Launch permissions that control which AWS accounts can use the AMI to launch instances.
- A block device mapping that specifies the volumes to attach to the instance when it's launched.
What is Packer ?
Packer is a tool for building identical machine images for multiple platforms from a single source configuration.
Image Source : https://www.hashicorp.com/
Packer is lightweight, runs on every major operating system, and is highly performant, creating machine images for multiple platforms in parallel. Packer comes out of the box with support for many platforms.
To know more about Packer, visit : https://developer.hashicorp.com/packer
Project Structure
GitHub Repository : https://github.com/mukulmantosh/Packer-Exercises
- .github - Workflow files for GitHub Actions
- packer - Contains HCL2 Packer templates, Shell Scripts etc.
- Dockerfile - Building Docker Image
- main.py - FastAPI Routes handling two endpoints
- requirements.txt - listing all the dependencies for a specific Python project
Let's Begin
I have used Amazon Linux 2 with arm64 architecture as our base AMI.
The custom AMI name is FastAPI_Base_Image
. It's a clean AMI without any OS/Software dependencies.
If you are not sure how to create an AMI, follow this link : https://docs.aws.amazon.com/toolkit-for-visual-studio/latest/user-guide/tkv-create-ami-from-instance.html
Dockerfile
I will create a container from the Dockerfile which is taking Python 3.9 as the base image and followed with python dependencies installation and starting the uvicorn server.
The image is already hosted in DockerHub.
URL : https://hub.docker.com/r/mukulmantosh/packerexercise
We have compiled for three architectures. Thanks to Docker Buildx.
amd64
arm64
arm/v7
Packer Template
packer/build.pkr.hcl
variable "ami_name" {
type = string
description = "The name of the newly created AMI"
default = "fastapi-nginx-ami-{{timestamp}}"
}
variable "security_group" {
type = string
description = "SG specific for Packer"
default = "sg-064ad8064cf203657"
}
variable "tags" {
type = map(string)
default = {
"Name" : "FastAPI-NGINX-AMI-{{timestamp}}"
"Environment" : "Production"
"OS_Version" : "Amazon Linux 2"
"Release" : "Latest"
"Creator" : "Packer"
}
}
source "amazon-ebs" "nginx-server-packer" {
ami_name = var.ami_name
ami_description = "AWS Instance Image Created by Packer on {{timestamp}}"
instance_type = "c6g.medium"
region = "ap-south-1"
security_group_id = var.security_group
tags = var.tags
run_tags = var.tags
run_volume_tags = var.tags
snapshot_tags = var.tags
source_ami_filter {
filters = {
name = "FastAPI_Base_Image"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["self"]
}
ssh_username = "ec2-user"
}
build {
sources = [
"source.amazon-ebs.nginx-server-packer"
]
provisioner "shell" {
inline = [
"sudo yum update -y",
]
}
provisioner "shell" {
script = "./scripts/build.sh"
pause_before = "10s"
timeout = "300s"
}
provisioner "file" {
source = "./scripts/fastapi.conf"
destination = "/tmp/fastapi.conf"
}
provisioner "shell" {
inline = ["sudo mv /tmp/fastapi.conf /etc/nginx/conf.d/fastapi.conf"]
}
error-cleanup-provisioner "shell" {
inline = ["echo 'update provisioner failed' > packer_log.txt"]
}
}
User variables allow your templates to be further configured with variables from the command-line, environment variables, Vault, or files. This lets you parameterize your templates so that you can keep secret tokens, environment-specific data, and other types of information out of your templates. This maximizes the portability of the template.
Builders create machines and generate images from those machines for various platforms (EC2, GCP, Azure, VMware, VirtualBox) etc. Packer also has some builders that perform helper tasks, like running provisioners.
Provisioners use built-in and third-party software to install and configure the machine image after booting. Provisioners prepare the system, so you may want to use them for the following use cases:
- installing packages
- patching the kernel
- creating users
- downloading application code
Post-processors run after builders and provisioners. Post-processors are optional, and you can use them to upload artifacts, re-package files, and more.
You can optionally create a single specialized provisioner called an error-cleanup-provisioner. This provisioner will not run unless the normal provisioning run fails. If the normal provisioning run does fail, this special error provisioner will run before the instance is shut down. This allows you to make last minute changes and clean up behaviors that Packer may not be able to clean up on its own.
The amazon-ebs Packer builder is able to create Amazon AMIs backed by EBS volumes for use in EC2.
source "amazon-ebs"
This builder builds an AMI by launching an EC2 instance from a source AMI, provisioning that running machine, and then creating an AMI from that machine. This is all done in your own AWS account. The builder will create temporary keypairs, security group rules, etc. that provide it temporary access to the instance while the image is being created. This simplifies configuration quite a bit.
The builder does not manage AMIs. Once it creates an AMI and stores it in your account, it is up to you to use, delete, etc. the AMI.
To know more, visit this link : https://developer.hashicorp.com/packer/plugins/builders/amazon/ebs
In the “source_ami_filter” section, We are filtering based on the base AMI which we created earlier.
source_ami_filter {
filters = {
name = "FastAPI_Base_Image"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["self"]
}
most_recent - Selects the newest created image when true.
owners - You may specify one or more AWS account IDs, "self" (which will use the account whose credentials you are using to run Packer)
We are using a Packer function called “timestamp” to generate UNIX timestamp, which helps to get a unique AMI name on every build.
By default the AMI’s you create will be private. If you want to share the AMI’s with other accounts you can make use of the “ami_users” option in packer.
If you want to build images in multi-region, you can specify the below code in the source section.
ami_regions = ["us-west-2", "us-east-1", "eu-central-1"]
In the provisioner section we will be updating the OS along-with installing scripts and copy nginx configuration.
packer/scripts/build.sh
Installing Docker, NGINX, and pulling latest application image from DockerHub and starting the container.
#!/bin/bash
sudo yum install jq -y
sudo yum install -y git
sudo yum install -y docker
sudo usermod -a -G docker ec2-user
sudo systemctl enable docker.service
sudo systemctl start docker.service
sudo amazon-linux-extras install nginx1 -y
sudo systemctl enable nginx.service
sudo systemctl start nginx.service
IMAGE_TAG=`curl -L -s 'https://hub.docker.com/v2/repositories/mukulmantosh/packerexercise/tags'|jq '."results"[0]["name"]' | bc`
sudo docker pull mukulmantosh/packerexercise:$IMAGE_TAG
sudo docker run -d --name fastapi --restart always -p 8080:8080 mukulmantosh/packerexercise:$IMAGE_TAG
packer/scripts/fastapi.conf
Copy the configuration to NGINX configuration folder. So, NGINX will proxy the request to backend.
upstream fastapi {
server 127.0.0.1:8080;
}
server {
listen 80;
location / {
proxy_pass http://fastapi;
proxy_set_header X-Forwarded-For
$proxy_add_x_forwarded_for;
proxy_set_header Host $host;
proxy_redirect off;
}
}
Building Template
Before you begin to build, make sure you have setup the following keys in your system and aws-cli is installed in your machine.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
There are two commands which you need to run before you execute build.
packer fmt build.pkr.hcl
The packer fmt Packer command is used to format HCL2 configuration files to a canonical format and style
packer validate build.pkr.hcl
The packer validate Packer command is used to validate the syntax and configuration of a template
Starting the Build
packer build build.pkr.hcl
The packer build command takes a template and runs all the builds within it in order to generate a set of artifacts.
You can see the new AMI has been successfully created and tag has been assigned.
You must have observed in the packer template, that we are using a custom security group. By default, Packer creates security group which access port 22 (0.0.0.0) from anywhere.
This posses security risk and to minimize that, I created a custom security group (Packer_SG) which allows only My IP.
variable "security_group" {
type = string
description = "SG specific for Packer"
default = "sg-064ad8064cf203657"
}
You can add more security by taking leverage of Session Manager Connections.
Session Manager Connections
Support for the AWS Systems Manager session manager lets users manage EC2 instances without the need to open inbound ports, or maintain bastion hosts.
GitHub Actions
GitHub Actions is a continuous integration and continuous delivery (CI/CD) platform that allows you to automate your build, test, and deployment pipeline.
Self-hosted runners
For our setup we will be using self-hosted Github runners.
Self-hosted runners offer more control of hardware, operating system, and software tools than GitHub-hosted runners provide. With self-hosted runners, you can create custom hardware configurations that meet your needs with processing power or memory to run larger jobs, install software available on your local network, and choose an operating system not offered by GitHub-hosted runners. Self-hosted runners can be physical, virtual, in a container, on-premises, or in a cloud.
Don't know how to setup ? Follow the below link :
As from security standpoint, we will make sure "Packer_SG" security group allow inbound port 22 for Github Action IP.
Execute Pipeline
Before proceeding, make sure to create the secrets which will be required in the build process.
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
.github/workflows/build-packer.yml
name: Packer
on:
push:
branches: main
jobs:
packer:
runs-on: self-hosted
name: packer
steps:
- name: Checkout Repository
uses: actions/checkout@v2
- name: Configure AWS Credentials
uses: aws-actions/configure-aws-credentials@v1-node16
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ap-south-1
# validate templates
- name: Validate Template
uses: hashicorp/packer-github-actions@master
with:
command: validate
arguments: -syntax-only
target: build.pkr.hcl
working_directory: ./packer
# build artifact
- name: Build Artifact
uses: hashicorp/packer-github-actions@master
with:
command: build
arguments: "-color=false -on-error=abort"
target: build.pkr.hcl
working_directory: ./packer
On inspecting the YAML file, you can clearly observe that we will be validating packer templates and then followed by building the artifact.
Let me make a small change in main branch. So, the pipeline will get triggered.
You can see now, the new AMI is created.
Vault
HashiCorp Vault tightly controls access to secrets and encryption keys by authenticating against trusted sources of identity such as Active Directory, LDAP, Kubernetes, Cloud Foundry, and cloud platforms. Vault enables fine grained authorization of which users and applications are permitted access to secrets and keys.
To know more about Vault, visit this link :
The reason we are using Vault over here is to create dynamic user credentials.
This helps us to avoid setting up environment variables for
export AWS_ACCESS_KEY_ID="XXXXXXXXXXXXXXX"
export AWS_SECRET_ACCESS_KEY="XXXXXXXXXXXXXXX"
Putting this keys in local machine, might expose some risks. So, I would recommend trying out AWS Secrets Engine.
AWS Secrets Engine
The AWS secrets engine generates AWS access credentials dynamically based on IAM policies. This generally makes working with AWS IAM easier, since it does not involve clicking in the web UI. Additionally, the process is codified and mapped to internal auth methods (such as LDAP). The AWS IAM credentials are time-based and are automatically revoked when the Vault lease expires.
I have already setup Vault in my local machine.
Follow the below link for setting up Vault.
You can either setup in your local machine or use HashiCorp Cloud.
Let's now begin by enabling the AWS secrets engine in our Vault server which is running locally.
Now, click on Configuration to setup our credentials.
Provide the AWS credentials and region which will be used to create user and attach role to them.
Next, I will modify lease time to 15 minutes. So, once the user is created it will be deleted automatically after 15 minutes.
Click on Save.
I have configured the AWS credential. Now, I will create the role which is going to be attached to the new user.
Policy Document
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ec2:AttachVolume",
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CopyImage",
"ec2:CreateImage",
"ec2:CreateKeypair",
"ec2:CreateSecurityGroup",
"ec2:CreateSnapshot",
"ec2:CreateTags",
"ec2:CreateVolume",
"ec2:DeleteKeyPair",
"ec2:DeleteSecurityGroup",
"ec2:DeleteSnapshot",
"ec2:DeleteVolume",
"ec2:DeregisterImage",
"ec2:DescribeImageAttribute",
"ec2:DescribeImages",
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeRegions",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSnapshots",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeVolumes",
"ec2:DetachVolume",
"ec2:GetPasswordData",
"ec2:ModifyImageAttribute",
"ec2:ModifyInstanceAttribute",
"ec2:ModifySnapshotAttribute",
"ec2:RegisterImage",
"ec2:RunInstances",
"ec2:StopInstances",
"ec2:TerminateInstances"
],
"Resource": "*"
}
]
}
I would recommend follow defense in depth and principle of least privilege.
Most of them don't encourage that policy document should contain delete permissions.
I came across an interesting article for tightening your policy document and make it more secure. So, it won't interfere with other instances.
Please checkout the below link :
Now, I will click on Generate Credentials.
Now, it's going to create a IAM user which is valid for 15 minutes (900 seconds)
You can see below, the new user is appearing in the IAM User section.
The PackerRole with assigned permissions are also being reflected.
Now, we are going to make sure that Packer should generate this credentials automatically.
Let's begin by editing the build.pkr.hcl file.
You need to add this line before closing of the source block.
vault_aws_engine {
name = "PackerRole"
}
Next, you need to setup environment variables.
Windows :
set VAULT_ADDR=http://127.0.0.1:8200
set VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX
Linux :
export VAULT_ADDR=http://127.0.0.1:8200
export VAULT_TOKEN=XXXXXXXXXXXXXXXXXXXXX
Once, we are done setting up our environment variables. We need to validate everything is working as expected by running the validate
command.
packer validate build.pkr.hcl
If you receive this message The configuration is valid
then you are good to proceed.
To initiate the build run the below command :
packer build build.pkr.hcl
- Note : Make sure before your begin build. The security group Packer_SG allows inbound access to port 22 from MyIP, as you are running the build from local machine.
Observe the message : You're using Vault-generated AWS credentials
This is going to pick the credentials from Vault, which is going to dynamically create a new user and attach the role.
The user will get automatically deleted based on the expiry specified.
Once, the build is complete. You will find the new image appearing in the AMI section.
Final Destination
Congratulations !!! You did it 🏆🏆🏆
If you liked this tutorial 😊, make sure to share across your friends and colleagues.
Top comments (0)