AWS Graviton processors continue to deliver price performance for cloud workloads. Migrating Kubernetes applications can be tricky, but AWS and Docker provide tools to make the process easier.
You can experiment with multi-architecture clusters to better understand the price performance benefits, and select the best hardware for the task. Multi-architecture clusters also ease migration when some software components are not available for the Arm architecture.
Refer to the The Insider’s Guide to Building a Multi-Arch Infrastructure for more information on the benefits of multi-architecture Kubernetes clusters.
Multi-architecture Kubernetes clusters
A multi-architecture Kubernetes cluster runs workloads on multiple hardware architectures, typically arm64 and amd64 (using Docker terminology).
To learn more about multi-architecture Kubernetes you can create a cluster in Amazon EKS and gain some practical experience with arm64 and amd64 nodes. This will also help you understand multi-architecture container images.
Before you begin
You will need docker
, eksctl
and kubectl
on your local computer to try out the steps below.
Your local computer can be Windows, Linux, or macOS. The instructions generally assume Linux.
Multipass from Canonical is a great tool for Linux virtual machines if you use macOS.
You can refer to the Kubectl install guide and the AWS EKS CLI install guide for installation details. The guides provide details for Arm Linux machines, but links to installation information about other platforms are also provided.
There are plenty of places to find Docker installation instructions. One option is the Docker install guide.
You will also need an AWS account.
To create an account, go to https://aws.amazon.com and click on Create an AWS Account in the top right corner. Follow the instructions to register. See the Creating an AWS account documentation for full instructions.
Make sure to configure your access key ID and secret access key, which are used to sign programmatic requests that you make to AWS. Refer to AWS Credentials for a quick summary of how to run aws configure
.
Multi-architecture containers
Before you can try out a multi-architecture Kubernetes cluster, you need a multi-architecture container image.
Multi-architecture container images are the easiest way to deploy applications, and hide the underlying hardware architecture. Building multi-architecture images is slightly more complex compared to building single-architecture images.
Docker provides two ways to create multi-architecture images:
-
docker buildx
builds both architectures at the same time -
docker manifest
builds each architecture separately and joins them together into a multi-architecture image
You can read more about how to create container images in the article Using Docker manifest to create multi-arch images on AWS Graviton processors
Below is a simple Go application you can use to learn about multi-architecture Kubernetes clusters.
Use your favorite text editor to create a file named hello.go
with the contents below:
// Copyright 2022 Google LLC
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package main
import (
"fmt"
"log"
"net/http"
"os"
"runtime"
)
func handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello from image NODE:%s, POD:%s, CPU PLATFORM:%s/%s",
os.Getenv("NODE_NAME"), os.Getenv("POD_NAME"), runtime.GOOS, runtime.GOARCH)
}
func main() {
http.HandleFunc("/", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
Next, create a file named go.mod
with the following two lines:
module example.com/arm
go 1.21
Create a third file named Dockerfile
with the contents:
ARG T
#
# Build: 1st stage
#
FROM golang:1.21-alpine as builder
ARG TARCH
WORKDIR /app
COPY go.mod .
COPY hello.go .
RUN GOARCH=${TARCH} go build -o /hello && \
apk add --update --no-cache file && \
file /hello
#
# Release: 2nd stage
#
FROM ${T}alpine
WORKDIR /
COPY --from=builder /hello /hello
RUN apk add --update --no-cache file
CMD [ "/hello" ]
Docker buildx
With the three text files created, you can build the container using docker buildx
.
Before you start, log in to Docker Hub:
docker login
Run the commands below to build the image, and make sure to substitute your own Docker ID instead of jasonrandrews
.
docker buildx create --use --name builder
docker buildx build --platform linux/amd64,linux/arm64 -t jasonrandrews/go-arch-x:1.0 --push .
When the commands are complete, you will see the new image in your Docker Hub account. Note, this is the go-arch-x
repository.
Docker manifest
You can also use docker manifest
to create a multi-architecture image from two single-architecture images. This is an alternative way to to build the multi-architecture image, but you can also use it to learn how to deploy a container image that only supports one architecture.
Substitute your Docker ID instead of jasonrandrews
and run these commands on an amd64 machine:
# Build for amd64
docker build -t jasonrandrews/go-arch:amd64 --build-arg TARCH=amd64 --build-arg T=amd64/ .
docker push jasonrandrews/go-arch:amd64
Substitute your Docker ID instead of jasonrandrews
and run these commands on an arm64 machine:
# Build for arm64
docker build -t jasonrandrews/go-arch:arm64 --build-arg TARCH=arm64 --build-arg T=arm64v8/ .
docker push jasonrandrews/go-arch:arm64
After building individual containers for each architecture, merge them into a single image by running the commands below on either architecture:
docker manifest create jasonrandrews/go-arch:1.0 \
--amend jasonrandrews/go-arch:arm64 \
--amend jasonrandrews/go-arch:amd64
docker manifest push --purge jasonrandrews/go-arch:1.0
You now have two single-architecture images in your Docker Hub account and the combined multi-architecture image in the same repository. Note, this is the go-arch
repository.
In the sections below, you will learn how to deploy the images in Amazon EKS.
Create a multi-architecture Kubernetes cluster
You can use eksctl
to create a cluster with multiple architectures.
First, use a text editor to create a text file cluster.yaml
with the contents below:
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
name: cluster1
region: us-east-1
nodeGroups:
- name: ng-1
instanceType: t2.micro
desiredCapacity: 2
volumeSize: 80
- name: ng-2
instanceType: t4g.small
desiredCapacity: 2
volumeSize: 80
The instance types above take advantage of free tier and the T4g free trial.
Run the eksctl
command with the yaml file to create an EKS cluster:
eksctl create cluster -f cluster.yaml
The cluster includes two amd64 nodes and two arm64 nodes.
It takes 15-20 minutes to launch the cluster so take a break while it is created.
When the cluster is ready check the nodes by running kubectl
on your local machine:
kubectl get nodes
The output shows four nodes, but doesn't print the architecture.
NAME STATUS ROLES AGE VERSION
ip-192-168-11-18.ec2.internal Ready <none> 106s v1.27.5-eks-43840fb
ip-192-168-27-254.ec2.internal Ready <none> 71s v1.27.5-eks-43840fb
ip-192-168-32-244.ec2.internal Ready <none> 106s v1.27.5-eks-43840fb
ip-192-168-61-31.ec2.internal Ready <none> 72s v1.27.5-eks-43840fb
To get the architecture run:
kubectl get node -o jsonpath='{.items[*].status.nodeInfo.architecture}'
The output shows four nodes, two of each architecture.
arm64 amd64 amd64 arm64
The Kubernetes cluster is now ready to use, and you can deploy the Go app.
Create a service to deploy
To deploy the application create a text file named hello-service.yaml
with the contents:
apiVersion: v1
kind: Service
metadata:
name: hello-service
labels:
app: hello
tier: web
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
selector:
app: hello
tier: web
Deploy the service by running:
kubectl apply -f hello-service.yaml
The output is:
service/hello-service created
Deploy the amd64 application
You can deploy the amd64 container first to demonstrate a single-architecture container.
Create a text file named amd64-deployment.yaml
with the contents below. Make sure to use your Docker ID instead of jasonrandrews
in the file. Notice that the container image is go-arch:amd64
. The amd64 image will only run on amd64 nodes. The nodeSelector is used to make sure the continer is only scheduled on amd64.
apiVersion: apps/v1
kind: Deployment
metadata:
name: amd-deployment
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
tier: web
template:
metadata:
labels:
app: hello
tier: web
spec:
containers:
- name: hello
image: jasonrandrews/go-arch:amd64
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: 300m
nodeSelector:
kubernetes.io/arch: amd64
Deploy the amd64 container by running:
kubectl apply -f amd64-deployment.yaml
You can see the running deployment.
kubectl get pods
The output shows 1 pod running:
NAME READY STATUS RESTARTS AGE
amd-deployment-7d4d44889d-vzhpd 1/1 Running 0 9s
To get the endpoint of the service run:
kubectl get services
The output prints the IP address of the load balancer.
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
hello-service LoadBalancer 10.100.188.181 a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com 80:30489/TCP 70s
kubernetes ClusterIP 10.100.0.1 <none> 443/TCP 14m
To confirm the amd64 application is running use curl
and substitute the endpoint of your load balancer:
curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
The output displays the hello message from the application and confirms the architecture is amd64.
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:amd-deployment-7d4d44889d-vzhpd, CPU PLATFORM:linux/amd64
Save the address of your load balancer (the http address) and use it for the curl
commands in the following sections.
Deploy the arm64 application
Create a text file named arm64-deployment.yaml
with the contents below. Again, substitute your Docker ID and notice that the nodeSelector is now arm64.
apiVersion: apps/v1
kind: Deployment
metadata:
name: arm-deployment
labels:
app: hello
spec:
replicas: 1
selector:
matchLabels:
app: hello
tier: web
template:
metadata:
labels:
app: hello
tier: web
spec:
containers:
- name: hello
image: jasonrandrews/go-arch:arm64
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: 300m
nodeSelector:
kubernetes.io/arch: arm64
Deploy the arm64 container by running:
kubectl apply -f arm64-deployment.yaml
You can see the running deployment.
kubectl get pods
Now, the output shows both deployments, amd64 and arm64.
NAME READY STATUS RESTARTS AGE
amd-deployment-7d4d44889d-vzhpd 1/1 Running 0 3m1s
arm-deployment-5996f6b85d-wp8cv 1/1 Running 0 10s
Run the curl
command a few times and you will see a mix of amd64 and arm64 messages printed, depending on which node services the request.
curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com
Deploy the multi-architecture application
You can also deploy the multi-architecture image.
Create a text file named multi-arch-deployment.yaml
with the contents below. Again, substitute your Docker ID. The image is the multi-architecture image created with docker buildx
and 6 replicas are specified.
apiVersion: apps/v1
kind: Deployment
metadata:
name: multi-arch-deployment
labels:
app: hello
spec:
replicas: 6
selector:
matchLabels:
app: hello
tier: web
template:
metadata:
labels:
app: hello
tier: web
spec:
containers:
- name: hello
image: jasonrandrews/go-arch-x:1.0
imagePullPolicy: Always
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
resources:
requests:
cpu: 300m
Deploy the multi-architecture container by running:
kubectl apply -f multi-arch-deployment.yaml
You can see the running deployment.
kubectl get pods
Now, the output shows all three deployments:
NAME READY STATUS RESTARTS AGE
amd-deployment-7d4d44889d-vzhpd 1/1 Running 0 4m24s
arm-deployment-5996f6b85d-wp8cv 1/1 Running 0 93s
multi-arch-deployment-547684cd44-7w6vf 1/1 Running 0 7s
multi-arch-deployment-547684cd44-chcc8 1/1 Running 0 7s
multi-arch-deployment-547684cd44-kw966 1/1 Running 0 7s
multi-arch-deployment-547684cd44-r4789 1/1 Running 0 7s
multi-arch-deployment-547684cd44-vqwc7 1/1 Running 0 7s
multi-arch-deployment-547684cd44-w4x8d 1/1 Running 0 7s
To test the appliation, run a loop of curl
commands and see the output messages from the application.
for i in $(seq 1 10); do curl -w '\n' http://a5d50ef4f76f3414a9f7e6e91ada5447-1826701750.us-east-1.elb.amazonaws.com; done
The output will show a variety of amd64 and arm64 messages:
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:multi-arch-deployment-547684cd44-kw966, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-chcc8, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-27-254.ec2.internal, POD:arm-deployment-5996f6b85d-wp8cv, CPU PLATFORM:linux/arm64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-32-244.ec2.internal, POD:multi-arch-deployment-547684cd44-w4x8d, CPU PLATFORM:linux/amd64
Hello from image NODE:ip-192-168-61-31.ec2.internal, POD:multi-arch-deployment-547684cd44-r4789, CPU PLATFORM:linux/arm64
You have now deployed single-architecture images for amd64 and arm64 as well as six copies of a multi-architecture image.
You can use the same techniques to incrementally migrate applications from amd64 to arm64 and invesigate the price performance provided by AWS Graviton processors.
Delete the clusters
When you are done, make sure to delete all AWS resources.
You can clean up with a single eckctl
command.
eksctl delete cluster -n cluster1
Top comments (0)