Building a virtualbox image on AWS EC2 with packer - this sounds straightforward, how hard could it be?
Indeed, it's relatively simple because there are existing tools and tutorials our there, but I still ran into a few hiccups during my experiment and I think it's worth nothing that down.
Why
Some background first. I need to build a VirtualBox image as a part of the release. There are few different approaches. In the past I did this with Vagrant and VirtualBox in my home-made Jenkins CI. So naturally I would dig in this way.
But now we're using CircleCI and it's hard to get VirtualBox running inside CircleCI environment.
I also checked AWS, AWS doesn't support nested virtualization unless you're using an expensive bare-metal machine. GCP supports nested virtualization but at that point I decided not to mess with it.
Chances are that we may need to build a AWS AMI too in the future, so after some digging I decide to use Packer with its amazon-ebs builder to build an AMI and then utilize aws-cli to export AMI to VMDK.
Packer
Using Packer to build an AMI is quite easy. Here's an example hcl file:
variable "ami_name" {
type = string
default = "${env("CUSTOM_AMI_NAME")}"
}
variable "region" {
type = string
default = "ap-northeast-1"
}
source "amazon-ebs" "ubuntu" {
ami_name = "${var.ami_name}"
instance_type = "c4.2xlarge"
ssh_pty = true
region = "${var.region}"
source_ami_filter {
filters = {
name = "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*"
root-device-type = "ebs"
virtualization-type = "hvm"
}
most_recent = true
owners = ["099720109477"]
}
launch_block_device_mappings {
device_name = "/dev/sda1"
volume_size = 40
volume_type = "gp2"
delete_on_termination = true
}
ssh_username = "ubuntu"
}
build {
sources = ["source.amazon-ebs.ubuntu"]
provisioner "shell" {
script = "./server_setup.sh"
}
}
There are few things we need to pay attention here in the bootstrap script.
First, you'll want to wait until cloud-init is finished to proceed your script. Otherwise there will be random failures.
# Wait for cloud-init to finish
while [ ! -f /var/lib/cloud/instance/boot-finished ]; do echo 'Waiting for cloud-init...'; sleep 1; done
Second, if you build the AMI and export to VMDK, import it in your VirtualBox now, you'll find that there's no network interface in your virtual machine. Because those AMIs on AWS are optimized for cloud environments, we need to tweak a little bit after your bootstrap is finished.
# Remove cloud-init to speed up the boot in virtualbox
sudo touch /etc/cloud/cloud-init.disabled
sudo apt remove cloud-init --yes
sudo rm -rf /etc/cloud/; sudo rm -rf /var/lib/cloud/
# Setup network in virtualbox
sudo rm /etc/netplan/50-cloud-init.yaml
sudo bash -c "cat > /etc/netplan/config.yaml" <<'EOF'
network:
version: 2
renderer: networkd
ethernets:
id0:
match:
name: en*
dhcp4: yes
EOF
sudo bash -c "echo '@reboot /usr/sbin/netplan apply' | crontab -"
Finally, you can convert AMI to VMDK on S3 when packer finished the building process.
export CUSTOM_AMI_NAME=your-ami-name
packer build example.pkr.hcl
AMI_ID=$(aws ec2 describe-images --owners self --filters "Name=name,Values=${CUSTOM_AMI_NAME}" | grep ImageId | cut -f4 -d'"')
aws ec2 export-image --image-id ${AMI_ID} --disk-image-format VMDK --s3-export-location S3Bucket=bucketname,S3Prefix=prefixname/
Top comments (0)