Deploy Airbyte on AKS

What is Airbyte?

Airbyte is an open-source data movement infrastructure for building extract and load (EL) data pipelines. It is designed for versatility, scalability, and ease-of-use.

In simple terms, it is a platform to unanimously fetch your data from various platforms and use it for your data analysis purposes.

What is AKS?

Azure Kubernetes Service (AKS) is a managed Kubernetes service that you can use to deploy and manage containerized applications.

What is Kubernetes?

Kubernetes is a portable, extensible, open source platform for managing containerized workloads and services, that facilitates both declarative configuration and automation.

Why to use AKS (K8s) for Airbyte?

To understand this, let's first understand the available options to deploy Airbyte.

So, as suggested by Airbyte, we can use Virtual machine to run Airbyte locally in VM. It is a fantastic idea for keep the VM running all the time to run Airbyte locally on it. But the issues comes where we have multiple data pipelines with loads and loads on data getting fetched via Airbyte. It can make the VM go out of service. And to make it run all the time with such load, it will incur lot of cost. So, here comes the scaling and cost as an issue.

Thus comes the better option to deploy it over K8s in Azure using Azure Kubernetes Service for easy scaling!

Even Airbyte too suggest K8s as the best option to deploy it over. But they don't have enough documents for AKS and thus we are here to understand it better.

Using Helm Charts

A Helm Chart is a collection of templates and settings that describe a set of Kubernetes resources.

We will use Helm Charts to deploy Airbyte on AKS. Airbyte has predefined Helm charts with some specific general values for Airbyte. You can even alter those values. We will see later in the article, how we can update the helm chart as per our usage.

Let's start with the actual work

Assuming that you've already created the AKS using UI or IaC, let's move ahead to connect with it and deploy Airbyte.

Steps to connect with AKS locally

1. Sign in to Azure in your terminal.
2. Set your specific Azure subscription where you have created your AKS cluster.

az account set --subscription <subscription_id>

1. Download the credentials to your cluster.

az aks get-credentials --resource-group <aks_rg_name> --name <aks_cluster_name> --overwrite-existing

Using Helm to deploy Airbyte

• Install Helm in your system.
• Run command

helm repo add airbyte https://airbytehq.github.io/airbyte/charts

You can get all the helm charts present via the ArtifactHub platform.

• Now we will install Airbyte on our AKS cluster.

helm install airbyte airbyte/airbyte --namespace <namespace_name> --create-namespace --kubeconfig '<path_to_downloaded_cluster_credential_file>'

• After running the above command, you can check your AKS cluster, in the namespace you can see your deployed Airbyte namespace.

• To connect with your Airbyte locally, use below command.

kubectl port-forward pod/<airbyte_webapp_pod_name> 8080:8080 --namespace <namespace_name>

You can get the running pod name from AKS cluster --> Workloads --> Pods.

** Note that the pod name can be different every time the cluster scales up or down.

How to modify Airbyte pods configuration as per your need?

To modify, you just need to pull helm chart locally and alter the values file and push it again as an update to the cluster. Or you can use your altered Helm chart folder to directly push to your new cluster.

Steps to follow:

1. Pull Helm chart locally

helm pull --untar airbyte/airbyte

1. Open the unzipped airbyte folder.
2. From the list of files and folders, open values.yaml and update the configurations as per your need.
3. Use the local helm chart folder to install Airbyte on AKS.

helm install airbyte <path_to_local_helm_chart> --namespace <namespace_name> --create-namespace --kubeconfig '<path_to_downloaded_cluster_credential_file>'

Or say you have airbyte already functional in your AKS, then you can even simply upgrade the values.yaml file present in AKS.

helm upgrade --install airbyte airbyte/airbyte -f <path_to_values.yaml_file_present_in_local> --namespace <namespace_name>

Conclusion

You are set to scale Airbyte in AKS!