After relocating to South Korea, I began provisioning resources in the Korea Central and Korea South regions, moving away from the United Kingdom South. I quickly noticed that specific SKUs, and sometimes even entire resources or services, were unavailable in my new region. While this was manageable for my Proof of Concepts (POCs) and personal projects, such limitations could pose significant challenges for organizations operating in the region, particularly when compliance or security requirements necessitate keeping data within South Korea.
These regional limitations, coupled with the varied performance metrics offered by different Cloud Service Providers (CSPs), the potential benefits of partnership programs (like discounts or free resource allocations), and the strategic advantage of minimizing vendor lock-in, are compelling more companies to adopt multi-cloud environments. However, this shift doesn’t come free and bring with it multiple challenges, such as the increased difficulty in monitoring and the need for engineers to possess a broader range of skills to integrate these diverse resources effectively.
In this article, I aim to explore the deployment of Kubernetes clusters on AWS and then use Azure Arc to manage these resources from a unified control plane along the other native Azure resources.
What is Azure Arc?
Before we delve deeper into the demonstration, let’s clarify what Azure Arc is and how it functions. Azure Arc was introduced in April 2021, initially focusing on managing databases, with a particular emphasis on PostgreSQL servers. However, its support was quickly extended to include a wider range of resources, such as Virtual Machines, VMware vSphere, System Center VMM, and Kubernetes Clusters, both on-premises and from other Cloud Service Providers. Each resource type has its specific requirements for integration. For instance, to connect an external Kubernetes cluster to Azure Arc, the cluster must be running Kubernetes version 1.13 or later. Additionally, there must be connectivity to the cluster through ports 443 and 9418.
The core functionality of Azure Arc involves “projecting” your external resources into Azure. This is achieved by installing Azure Arc agents on these external resources. These agents facilitate communication with Azure, enabling management tasks such as patching, policy enforcement, and tagging across your infrastructure. Once connected, these resources appear in the Azure portal alongside your native Azure resources. Additionally, Azure Arc enhances monitoring capabilities by integrating with Azure Monitor, providing insights for all cloud-based resources registered under its umbrella.
From a financial point of view, this service is available at no additional cost. This is probably a strategic decision by Microsoft that aims to encourage customers to leverage its services over competitors’ or on-premise solutions, especially for those already utilizing other Microsoft services.
Deploying Kubernetes Clusters on Different Clouds
After setting up a new Kubernetes cluster on AWS, the first step is to grant your user account with the necessary permissions. This is done by assigning the AmazonEKSClusterAdminPolicy access policy to your user.
Next, it’s time to create a Node Group. This Node Group will be linked to your EKS cluster and hosts the virtual machines, essentially the backbone where your application pods will reside and run.
$ aws eks list-nodegroups --cluster-name eks-training-dev-krc --region us-east-1
{
"nodegroups": [
"eks-training-dev-krc"
]
}
$ aws eks describe-cluster --name eks-training-dev-krc --region us-east-1
{
"cluster": {
"name": "eks-training-dev-krc",
"arn": "arn:aws:eks:us-east-1:370515815379:cluster/eks-training-dev-krc",
"createdAt": "2024-03-12T10:59:16.159000+09:00",
"version": "1.29",
"endpoint": "https://2281EF2EDCF25F5B4BC63704E5DE0468.gr7.us-east-1.eks.amazonaws.com",
"roleArn": "arn:aws:iam::370515815379:role/EKSClusterRole",
"resourcesVpcConfig": {
"subnetIds": [
"subnet-08b3a655cf510da0a",
"subnet-09470b12b7dad0427",
"subnet-01964a49f044d69a3"
],
"securityGroupIds": [],
"clusterSecurityGroupId": "sg-070ba0ce70c3c4ae9",
"vpcId": "vpc-0b3737b03c3980dc1",
"endpointPublicAccess": true,
"endpointPrivateAccess": true,
"publicAccessCidrs": [
"0.0.0.0/0"
]
}
...
}
}
With your cluster and node groups in place, the next crucial step is generating a config file. This file will contains all the information required to access the Kubernetes cluster, such as cluster, user, server, and certificate.
$ aws eks update-kubeconfig --region us-east-1 --name eks-training-dev-krc
Added new context arn:aws:eks:us-east-1:xxxxxxxxxxxx:cluster/eks-training-dev-krc to C:\Users\Ivan\.kube\config
Moving to Azure’s part of the setup, you’ll need to enrich your Azure CLI with several extensions crucial for Kubernetes management via Azure Arc:
az extension add --name connectedk8s
az extension add --name k8s-extension
az extension add --name k8s-configuration
az extension add --name customlocation
Additionally, you must register the necessary Azure resource providers to support Azure Arc-enabled Kubernetes features:
az provider register --namespace Microsoft.Kubernetes
az provider register --namespace Microsoft.KubernetesConfiguration
az provider register --namespace Microsoft.ExtendedLocation
Now that we have provisioned out Kubernetes cluster on AWS, we can move forward and connect it to Azure via Azure Arc. The first step is to go to the Azure Arc service page, select Kubernetes cluster, and the click Add a Kubernetes cluster with Azure Arc.
Note: the Create a Kubernetes cluster with Azure Arc is focused on the creation of Kubernetes clusters on-premises which is out of scope of this article.
Next, select the resource group where the subcription and resource is going to be added, and give it a name and region. The region in neccessary as Azure doesn’t know the region o the target Cloud Service provider where the resource is provisioned. You will also need to specify how to connect to the resource; this options are the same across all resoruce types and are the following:
- Public endpoint: The resource has direct access to the internet without any intermediary.
- Proxy server: The traffic to and from the Azure Arc agent to Azure services are routed through a proxy server. This method is common in enterprise environments where direct Internet access is restricted for security reasons.
- Private endpoint: The traffic is routed through the Azure backbone network rather than the public Internet.
After that, you will be able to define some tags, but the magic happens in the scirpt section.
This script will need to be executed on the machine with the EKS clutter context and will take approximatively 10 minutes to run. First it will run a series of prerequisite checks, then it will provision several resources on Azure, and the start of the Azure Arc agent installation on your Kubernetes cluster.
$ az connectedk8s connect --name "eks-training-dev-krc" --resource-group "aks-training-dev-krc-infra" --location "koreacentral" --correlation-id "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" --tags "Datacenter=Demo City=Demo StateOrDistrict=Demo CountryOrRegion=Demo Scope=Private"
D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\cryptography/hazmat/backends/openssl/backend.py:27: UserWarning: You are using cryptography on a 32-bit Python on a 64-bit Windows Operating System. Cryptography will be significantly faster if you switch to using a 64-bit Python.
This operation might take a while...
The required pre-checks for onboarding have succeeded.
Azure resource provisioning has begun.
Azure resource provisioning has finished.
Starting to install Azure arc agents on the Kubernetes cluster.
{
"agentPublicKeyCertificate": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
"agentVersion": null,
"connectivityStatus": "Connecting",
"distribution": "eks",
"id": "/subscriptions/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/resourceGroups/aks-training-dev-krc-infra/providers/Microsoft.Kubernetes/connectedClusters/eks-training-dev-krc",
"identity": {
"principalId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"tenantId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
"type": "SystemAssigned"
},
"infrastructure": "aws",
"kubernetesVersion": null,
"lastConnectivityTime": null,
"location": "koreacentral",
"managedIdentityCertificateExpirationTime": null,
"name": "eks-training-dev-krc",
"offering": null,
"provisioningState": "Succeeded",
"resourceGroup": "aks-training-dev-krc-infra",
"systemData": {
"createdAt": "2024-04-01T01:13:12.973246+00:00",
"createdBy": "porta.ivan@outlook.com",
"createdByType": "User",
"lastModifiedAt": "2024-04-01T01:13:12.973246+00:00",
"lastModifiedBy": "demo@outlook.com",
"lastModifiedByType": "User"
},
"tags": {
"Datacenter": "Demo City=Demo StateOrDistrict=Demo CountryOrRegion=Demo Scope=Private"
},
"totalCoreCount": null,
"totalNodeCount": null,
"type": "microsoft.kubernetes/connectedclusters"
}
When you inspect your Kubernetes cluster after running the script, you’ll notice several new resource types. These resources enable a range of functionalities from secure communication to Git repositories configuration for GitOps.
$ kubectl api-resources
NAME SHORTNAMES APIVERSION NAMESPACED KIND
...
connectedclusters arc.azure.com/v1beta1 true ConnectedCluster
...
arccertificates clusterconfig.azure.com/v1beta1 true ArcCertificate
azureclusteridentityrequests azidentityreq clusterconfig.azure.com/v1beta1 true AzureClusterIdentityRequest
azureextensionidentities clusterconfig.azure.com/v1beta1 true AzureExtensionIdentity
configsyncstatuses clusterconfig.azure.com/v1beta1 true ConfigSyncStatus
customlocationsettings clusterconfig.azure.com/v1beta1 true CustomLocationSettings
extensionconfigs ec clusterconfig.azure.com/v1beta1 true ExtensionConfig
extensionevents clusterconfig.azure.com/v1beta1 true ExtensionEvents
gitconfigs clusterconfig.azure.com/v1beta1 true GitConfig
healthstates clusterconfig.azure.com/v1beta1 true HealthState
...
You’ll also see multiple new pods within the azure-arc namespace. These pods use the resource types prefiously listed to perform specific activities like managing the cluster metadata and ensures it is synchronized with Azure Arc, collection of logs from across the cluster, and more.
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
azure-arc cluster-metadata-operator-7fd8878c5d-k95w4 2/2 Running 0 10m
azure-arc clusterconnect-agent-6774cfff98-qdvlr 3/3 Running 0 10m
azure-arc clusteridentityoperator-64d97957c9-p7qpt 2/2 Running 0 10m
azure-arc config-agent-56445bc8-w2z46 2/2 Running 0 10m
azure-arc controller-manager-6996d658f4-5z862 2/2 Running 0 10m
azure-arc extension-events-collector-6bb86644dc-wqg8m 2/2 Running 0 10m
azure-arc extension-manager-6cf5c5965d-d4czf 3/3 Running 0 10m
azure-arc flux-logs-agent-7b8b4d7fcb-wsw6n 1/1 Running 0 10m
azure-arc kube-aad-proxy-697b664cfc-tnswg 2/2 Running 0 10m
azure-arc logcollector-9b8f89d57-thhh4 1/1 Running 0 10m
azure-arc metrics-agent-68689d8c5d-6vx64 2/2 Running 0 10m
azure-arc resource-sync-agent-d7dc9449b-th46d 2/2 Running 0 10m
...
Several of these pods will also internally communicate with each other and the Azure services via the following services.
$ kubectl get services --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
azure-arc extension-events-collector ClusterIP 10.100.108.124 <none> 8082/TCP 14m
azure-arc extension-manager-svc ClusterIP None <none> 8081/TCP 14m
azure-arc flux-logs-agent ClusterIP 10.100.243.81 <none> 80/TCP 14m
azure-arc kube-aad-proxy ClusterIP 10.100.249.91 <none> 443/TCP,8080/TCP 14m
azure-arc logcollector ClusterIP 10.100.216.251 <none> 24224/TCP 14m
...
The deployment also creates several Kubernetes secrets in the azure-arc namespace, which store sensitive information such as private keys, certificates, and tokens necessary for secure communication and operation within the Azure Arc environment:
$ kubectl get secrets --all-namespaces
NAMESPACE NAME TYPE DATA AGE
azure-arc-release sh.helm.release.v1.azure-arc.v1 helm.sh/release.v1 1 12m
azure-arc azure-arc-connect-privatekey Opaque 1 12m
azure-arc azure-identity-certificate Opaque 3 12m
azure-arc connectproxy-agent-identity-request-token Opaque 1 12m
azure-arc identity-request-2a051a512c1afcd426dd4090206c017a675c0f002bf329cc3165a7ba3abdcc97-token Opaque 1 12m
azure-arc kube-aad-proxy-certificate kubernetes.io/tls 2 12m
Non-confidential configuration data required for the operation of Azure Arc like identity requests, Azure-related settings, Fluent Bit are stored in the following Config-Maps.
$ kubectl get configmaps --all-namespaces
NAMESPACE NAME DATA AGE
azure-arc-release kube-root-ca.crt 1 17m
azure-arc azidentityreqleaderid-lock 0 13m
azure-arc azure-clusterconfig 59 13m
azure-arc azure-fluentbit-collector-config 5 13m
azure-arc azure-fluentbit-config 5 13m
azure-arc azure-telegraf-config 1 13m
azure-arc extension-immutable-values 0 13m
azure-arc extensioncontrollerleaderid-lock 0 13m
azure-arc gitconfig-immutable-values 0 13m
azure-arc gitconfigleaderid-lock 0 13m
azure-arc kube-root-ca.crt 1 13m
azure-arc rsync-datastore 2 13m
azure-arc rsync-datastore-1 1 13m
azure-arc rsync-datastore-10 1 13m
azure-arc rsync-datastore-11 1 13m
azure-arc rsync-datastore-12 1 13m
azure-arc rsync-datastore-13 1 13m
azure-arc rsync-datastore-14 1 13m
azure-arc rsync-datastore-15 1 13m
azure-arc rsync-datastore-16 1 13m
azure-arc rsync-datastore-17 1 13m
azure-arc rsync-datastore-18 1 13m
azure-arc rsync-datastore-19 1 13m
azure-arc rsync-datastore-2 1 13m
azure-arc rsync-datastore-20 1 13m
azure-arc rsync-datastore-21 1 13m
azure-arc rsync-datastore-22 1 13m
azure-arc rsync-datastore-23 1 13m
azure-arc rsync-datastore-24 1 13m
azure-arc rsync-datastore-25 1 13m
azure-arc rsync-datastore-26 1 13m
azure-arc rsync-datastore-27 1 13m
azure-arc rsync-datastore-28 1 13m
azure-arc rsync-datastore-29 1 13m
azure-arc rsync-datastore-3 1 13m
azure-arc rsync-datastore-4 1 13m
azure-arc rsync-datastore-5 1 13m
azure-arc rsync-datastore-6 1 13m
azure-arc rsync-datastore-7 1 13m
azure-arc rsync-datastore-8 1 13m
azure-arc rsync-datastore-9 1 13m
...
Finally, you can complete the Kubernetes cluster setup for Azure Arc.
Before being able to inspect the EKS cluster resources from the azure portal, we will need to create a Service Account token within your EKS cluster and add it to the Azure portal.
$ kubectl create serviceaccount azure-user -n default
Next, you need to assign the necessary permissions to your new service account. In this case I liked the azure-user service account to the cluster-admin role, which grants it administrative access to the entire cluster.
$ kubectl create clusterrolebinding azure-user-binding --clusterrole cluster-admin --serviceaccount default:azure-user
clusterrolebinding.rbac.authorization.k8s.io/azure-user-binding created
Then, I’ll create a secret that holds the service account token.
$ kubectl apply -f - <<EOF
apiVersion: v1
kind: Secret
metadata:
name: azure-user-secret
annotations:
kubernetes.io/service-account.name: azure-user
type: kubernetes.io/service-account-token
EOF
secret/azure-user-secret created
Finally, extract and decrypt the token from the secret you just created.
TOKEN=$(kubectl get secret azure-user-secret -o jsonpath='{$.data.token}' | base64 -d | sed 's/$/\n/g')
With the token in hand, you can complete the connection between Azure Arc and your Kubernetes cluster.
After validating the token, you will be able to freely view and manage your cluster resources directly from the Azure portal.
References
-Azure Arc: https://learn.microsoft.com/en-us/azure/azure-arc/kubernetes/
Top comments (0)