DEV Community

Cover image for The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter
AWS Community Builders profile image Luqman Bello
Luqman Bello for AWS Community Builders

Posted on • Edited on

The Ultimate Comprehensive Guide to Setting Up Self-Hosted Airbyte on EKS Using Karpenter

#karpenter #eks #aws #kubernetes

Table of Contents

  1. Introduction
  2. Prerequisites
  3. Setting Up EKS with Karpenter
  4. Deploying Airbyte on EKS
  5. Upgrading Airbyte OSS Version
  6. Monitoring and Maintenance
  7. Troubleshooting Common Errors

Introduction

Airbyte is an open-source data integration platform that enables you to consolidate your data from various sources to data warehouses, lakes, and databases. This 2024 guide walks you through setting up a production-ready, self-hosted Airbyte instance on Amazon EKS (Elastic Kubernetes Service) using Karpenter for dynamic node provisioning. The setup includes best practices for security, scalability, and maintenance.

Prerequisites

  • AWS Account: An active AWS account with appropriate permissions
  • AWS CLI: Version 2.13.0 or later
  • kubectl: Version 1.28 or later
  • Helm: Version 3.12 or later
  • eksctl: Version 0.163.0 or later
  • Basic knowledge: Kubernetes, AWS, and command-line operations

Setting Up EKS with Karpenter

Creating an EKS Cluster

Create a modern EKS cluster using eksctl with the latest supported version:

eksctl create cluster \
  --name airbyte-cluster \
  --region us-west-2 \
  --version 1.28 \
  --with-oidc \
  --zones us-west-2a,us-west-2b \
  --nodegroup-name standard-workers \
  --node-type t3.large \
  --nodes 2 \
  --nodes-min 1 \
  --nodes-max 4 \
  --managed \
  --addons aws-ebs-csi-driver,vpc-cni,coredns,kube-proxy \
  --node-volume-size 100 \
  --external-dns-access
Enter fullscreen mode Exit fullscreen mode

Key changes from previous versions:

  • Updated to EKS 1.28 (latest stable)
  • Added essential EKS add-ons
  • Increased default node size to t3.large for better performance
  • Added larger root volume for containers
  • Enabled external DNS access for better integration

Installing Karpenter

  1. Create AWS IAM Resources for Karpenter

Using the latest Karpenter IAM setup:

   export CLUSTER_NAME="airbyte-cluster"
   export AWS_ACCOUNT_ID="$(aws sts get-caller-identity --query Account --output text)"
   export AWS_REGION="us-west-2"

   curl -fsSL https://karpenter.sh/v0.32.0/getting-started/getting-started-with-eksctl/cloudformation.yaml  > cloudformation.yaml

   aws cloudformation deploy \
     --stack-name "karpenter-${CLUSTER_NAME}" \
     --template-file cloudformation.yaml \
     --capabilities CAPABILITY_NAMED_IAM \
     --parameter-overrides "ClusterName=${CLUSTER_NAME}"
Enter fullscreen mode Exit fullscreen mode
  1. Install Karpenter Controller 
   helm repo add karpenter https://charts.karpenter.sh
   helm repo update

   helm install karpenter karpenter/karpenter \
     --namespace karpenter --create-namespace \
     --version v0.32.0 \
     --set serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::${AWS_ACCOUNT_ID}:role/karpenter-controller-${CLUSTER_NAME} \
     --set settings.aws.clusterName=${CLUSTER_NAME} \
     --set settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
     --set settings.aws.interruptionQueueName=karpenter-${CLUSTER_NAME} \
     --set controller.resources.requests.cpu=1 \
     --set controller.resources.requests.memory=1Gi \
     --set controller.resources.limits.cpu=1 \
     --set controller.resources.limits.memory=1Gi
Enter fullscreen mode Exit fullscreen mode
  1. Create a Modern Provisioner

Create provisioner.yaml:

   apiVersion: karpenter.sh/v1beta1
   kind: NodePool
   metadata:
     name: default
   spec:
     template:
       spec:
         requirements:
           - key: "kubernetes.io/arch"
             operator: In
             values: ["amd64"]
           - key: "kubernetes.io/os"
             operator: In
             values: ["linux"]
           - key: "karpenter.sh/capacity-type"
             operator: In
             values: ["on-demand"]
           - key: "node.kubernetes.io/instance-type"
             operator: In
             values: ["t3.large", "t3.xlarge", "m5.large", "m5.xlarge"]
         nodeClassRef:
           name: default
     limits:
       resources:
         cpu: 100
         memory: 100Gi
     disruption:
       consolidationPolicy: WhenUnderutilized
       consolidateAfter: 30s
     weight: 10
---
apiVersion: karpenter.sh/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  roleClassName: default
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: ${CLUSTER_NAME}
  tags:
    karpenter.sh/discovery: ${CLUSTER_NAME}
Enter fullscreen mode Exit fullscreen mode

Apply the configuration:

   kubectl apply -f provisioner.yaml
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte on EKS

Configuring Airbyte Resources

  1. Create Namespace and Storage Class 
   kubectl create namespace airbyte

   cat <<EOF | kubectl apply -f -
   apiVersion: storage.k8s.io/v1
   kind: StorageClass
   metadata:
     name: airbyte-storage
   provisioner: ebs.csi.aws.com
   parameters:
     type: gp3
     encrypted: "true"
   reclaimPolicy: Retain
   allowVolumeExpansion: true
   volumeBindingMode: WaitForFirstConsumer
   EOF
Enter fullscreen mode Exit fullscreen mode
  1. Set Up Persistent Storage 
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: airbyte-pvc
     namespace: airbyte
   spec:
     storageClassName: airbyte-storage
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 50Gi
Enter fullscreen mode Exit fullscreen mode

Deploying Airbyte

  1. Add Airbyte Helm Repository 
   helm repo add airbyte https://airbytehq.github.io/helm-charts
   helm repo update
Enter fullscreen mode Exit fullscreen mode
  1. Create Custom Values File

Create airbyte-values.yaml:

   global:
     serviceAccountName: airbyte-admin
     database:
       secretName: airbyte-db-secrets
     logs:
      storage:
        enabled: true

   webapp:
     resources:
       limits:
         cpu: "1"
         memory: 2Gi
       requests:
         cpu: "0.5"
         memory: 1Gi

   server:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   worker:
     resources:
       limits:
         cpu: "2"
         memory: 4Gi
       requests:
         cpu: "1"
         memory: 2Gi

   persistence:
     enabled: true
     storageClass: airbyte-storage
     size: 50Gi

   metrics:
     enabled: true
     serviceMonitor:
       enabled: true
Enter fullscreen mode Exit fullscreen mode
  1. Install Airbyte 
   helm install airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version 0.50.0
Enter fullscreen mode Exit fullscreen mode
  1. Verify Deployment 
   kubectl get pods -n airbyte
   kubectl get svc -n airbyte
Enter fullscreen mode Exit fullscreen mode

Upgrading Airbyte OSS Version

  1. Backup Before Upgrade 
   # Backup using Velero or similar tool
   velero backup create airbyte-backup \
     --include-namespaces airbyte
Enter fullscreen mode Exit fullscreen mode
  1. Update Helm Repository 
   helm repo update
   helm search repo airbyte/airbyte --versions
Enter fullscreen mode Exit fullscreen mode
  1. Upgrade Airbyte 
   helm upgrade airbyte airbyte/airbyte \
     --namespace airbyte \
     --values airbyte-values.yaml \
     --version <NEW_VERSION> \
     --timeout 15m
Enter fullscreen mode Exit fullscreen mode

Monitoring and Maintenance

  1. Set Up Monitoring 
   # Install Prometheus and Grafana
   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   helm repo update

   helm install prometheus prometheus-community/kube-prometheus-stack \
     --namespace monitoring \
     --create-namespace
Enter fullscreen mode Exit fullscreen mode
  1. Configure Log Aggregation

Use AWS CloudWatch Logs:

   kubectl apply -f https://raw.githubusercontent.com/aws/containers-roadmap/master/preview-programs/cloudwatch-logs/cloudwatch-logs-configmap.yaml
Enter fullscreen mode Exit fullscreen mode

Troubleshooting Common Errors

Pod Scheduling Issues

  • Symptom: Pods stuck in Pending state
  • Solution: Check Karpenter logs and node pool configuration: 
  kubectl logs -n karpenter -l app.kubernetes.io/name=karpenter -f
Enter fullscreen mode Exit fullscreen mode

Database Connection Issues

  • Symptom: Database connection errors in Airbyte logs
  • Solution: Verify secrets and connection strings: 
  kubectl get secrets -n airbyte
  kubectl logs -n airbyte deployment/airbyte-server
Enter fullscreen mode Exit fullscreen mode

Resource Constraints

  • Symptom: OOMKilled or CPU throttling
  • Solution: Adjust resource limits in values.yaml: 
  resources:
    limits:
      memory: "4Gi"
      cpu: "2"
    requests:
      memory: "2Gi"
      cpu: "1"
Enter fullscreen mode Exit fullscreen mode

Karpenter Node Provisioning Issues

  • Symptom: Nodes not being provisioned
  • Solution: Check NodePool and EC2NodeClass configurations: 
  kubectl describe nodepool default
  kubectl describe ec2nodeclass default
Enter fullscreen mode Exit fullscreen mode

This guide provides a robust foundation for running Airbyte on EKS with Karpenter. Remember to regularly check for updates and security patches for all components. For production environments, consider implementing additional security measures and backup strategies.

For questions, issues, or suggestions, please leave a comment below!

Top comments (2)

Collapse
 
xyz_oblivion profile image
oblivion •

Hello, I am getting the next message:

Invalid template path https://karpenter.sh/docs/getting-started/cloudformation.yaml
Enter fullscreen mode Exit fullscreen mode

Could you please suggests?

Collapse
 
aaber profile image
AAber •

raw.githubusercontent.com/aws/karp...

Read next

siddhantkcode profile image

Querying S3 logs in a flash with DuckDB 🦆

Siddhant Khare -

jaysaadana profile image

Celebrating Open Source in 2024: Top Projects and Milestones

Jay Saadana -

benbpyle profile image

First Impressions of AWS DSQL with Lambda and Rust

Benjamen Pyle -

omahn profile image

Data inconsistency in AWS Amazon Aurora Postgres solved with Local Write Forwarding?

Paul Elliott -