The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

Table of Contents

1. Introduction
2. Prerequisites
3. Setting Up EKS with Karpenter
   • Creating an EKS Cluster
   • Installing Karpenter
4. Deploying Airbyte on EKS
   • Configuring Airbyte Resources
   • Deploying Airbyte
5. Upgrading Airbyte OSS Version
6. Troubleshooting Common Errors

Introduction

Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. Deploying Airbyte on Amazon Elastic Kubernetes Service (EKS) provides scalability and reliability, while Karpenter enhances cluster efficiency by dynamically provisioning nodes. This guide walks you through setting up a self-hosted Airbyte on EKS using Karpenter.

Prerequisites

• AWS Account: An active AWS account with permissions to create EKS clusters and associated resources.
• AWS CLI: Installed and configured with your AWS credentials.
• kubectl: Installed for Kubernetes cluster management.
• Helm: Installed for deploying applications on Kubernetes.
• eksctl: Installed for EKS cluster management.

Setting Up EKS with Karpenter

Creating an EKS Cluster

First, create an EKS cluster using eksctl.

eksctl create cluster \
  --name airbyte-cluster \
  --region us-west-2 \
  --version 1.25 \
  --with-oidc \
  --zones us-west-2a,us-west-2b \
  --nodegroup-name standard-workers \
  --node-type t3.medium \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 3 \
  --managed

• --with-oidc: Enables IAM roles for service accounts.
• --zones: Specify availability zones.
• --node-type: Instance type for worker nodes.

Installing Karpenter

1. Create AWS IAM Resources for Karpenter

Karpenter requires specific IAM roles and permissions.

   aws cloudformation deploy \
     --stack-name karpenter-iam \
     --template-file https://karpenter.sh/docs/getting-started/cloudformation.yaml \
     --capabilities CAPABILITY_NAMED_IAM

1. Install Karpenter Controller

Add the Helm repository and install Karpenter.

   helm repo add karpenter https://charts.karpenter.sh
   helm repo update

   helm install karpenter karpenter/karpenter \
     --namespace karpenter --create-namespace \
     --version v0.17.2 \
     --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<AWS_ACCOUNT_ID>:role/KarpenterControllerRole \
     --set settings.aws.clusterName=airbyte-cluster \
     --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-airbyte-cluster \
     --set settings.aws.interruptionQueueName=KarpenterInterruptionQueue

1. Create a Provisioner

Karpenter uses provisioners to manage node provisioning.

   apiVersion: karpenter.sh/v1alpha5
   kind: Provisioner
   metadata:
     name: default
   spec:
     requirements:
       - key: "node.kubernetes.io/instance-type"
         operator: In
         values: ["t3.medium", "t3.large"]
     provider:
       subnetSelector:
         karpenter.sh/discovery: airbyte-cluster
       securityGroupSelector:
         karpenter.sh/discovery: airbyte-cluster

Apply the provisioner:

   kubectl apply -f provisioner.yaml

Deploying Airbyte on EKS

Configuring Airbyte Resources

1. Create a Namespace

   kubectl create namespace airbyte

1. Set Up Persistent Storage

Airbyte requires persistent storage for configurations and data.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: airbyte-pvc
     namespace: airbyte
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi

Apply the PVC:

   kubectl apply -f pvc.yaml

Deploying Airbyte

1. Add Airbyte Helm Repository

   helm repo add airbyte https://airbytehq.github.io/helm-charts
   helm repo update

1. Install Airbyte

   helm install airbyte airbyte/airbyte \
     --namespace airbyte \
     --set persistence.enabled=true \
     --set persistence.existingClaim=airbyte-pvc

1. Verify Deployment

   kubectl get pods -n airbyte

Ensure all pods are in the Running state.

Upgrading Airbyte OSS Version

1. Check for New Versions

Visit the Airbyte GitHub Releases page to find the latest version.

1. Update Helm Chart

Update the Helm repository and list available chart versions.

   helm repo update
   helm search repo airbyte -l

1. Upgrade Airbyte

   helm upgrade airbyte airbyte/airbyte \
     --namespace airbyte \
     --set image.tag=<NEW_VERSION_TAG>

Replace <NEW_VERSION_TAG> with the new version number.

1. Verify Upgrade

   kubectl rollout status deployment/airbyte-server -n airbyte

Troubleshooting Common Errors

Pods Stuck in Pending State

• Cause: Insufficient resources or misconfigured Karpenter provisioner.
• Solution: Check Karpenter logs and ensure the provisioner allows for the required instance types.

   kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter

PVC Not Bound

• Cause: Storage class issues or lack of storage resources.
• Solution: Verify the storage class and ensure that the AWS EBS CSI driver is installed.

Airbyte Server CrashLoopBackOff

• Cause: Misconfiguration or insufficient memory.
• Solution: Increase resource limits in the Helm chart values.

   resources:
     limits:
       memory: "2Gi"
     requests:
       memory: "1Gi"

Karpenter Not Provisioning Nodes

• Cause: IAM permissions or subnet/security group selectors misconfigured.
• Solution: Verify IAM roles and Karpenter's provisioner configurations.

Deploying Airbyte on EKS with Karpenter allows for a scalable and efficient data integration platform. By following this guide, you should have a self-hosted Airbyte instance up and running, with the ability to upgrade and troubleshoot as needed.

Feel free to reach out in the comments below if you have any questions or run into issues!