Deploying ByConity with Kubernetes: A Step-by-Step Guide

Introduction:

ByConity is a powerful open-source data warehouse developed by ByteDance. It's designed for high-performance computing and utilizes a computing-storage separation architecture. For those who don't know, ByConity's origin can be traced back to ClickHouse DB. Initially, ByteDance was implementing ClickHouse as their main Data warehouse. As the business grew and enhanced performance was needed, the team decided to fork ClickHouse to start ByConity as we know it today.
If you want to harness its capabilities within your Kubernetes cluster, this step-by-step guide will walk you through the deployment process. Whether you want to set up a local environment for testing or deploy it in your self-built Kubernetes cluster, we've got you covered.

Why Kubernetes?
Deploying ByConity with Kubernetes offers several advantages. Most tech enterprises require agility, cost-effectiveness, and superior performance to harness real-time data. Kubernetes applications exhibit versatility, capable of running seamlessly across diverse environments, from public clouds to isolated on-premise setups. The Kubernetes operator pattern simplifies intricate analytic infrastructure management, transforming it into easily controllable assets.
I like to think of Kubernetes as if it's an orchestra to understand how it works...
Imagine Kubernetes as the conductor of the orchestra, responsible for managing and coordinating various musical instruments (containers) to produce a harmonious symphony (your application). Let's break it down:

1. Conductor (Kubernetes Master): Kubernetes has a conductor, which is like the conductor of an orchestra. This conductor is the brain behind the operation, making high-level decisions and coordinating the entire performance.
2. Musicians (Containers): Each musician in the orchestra corresponds to a container within Kubernetes. Each musician (container) has a specific role and plays a part in the overall composition.
3. Sheet Music (Pods): Kubernetes groups containers together in units called "Pods." A Pod is like a piece of sheet music that defines which instruments (containers) should play together in harmony. The conductor (Kubernetes) ensures that the right combinations of instruments (containers) are playing together.
4. Stage (Nodes): The stage represents the physical or virtual machines where the musicians (containers) perform. Kubernetes manages these stages (nodes) and assigns musicians (containers) to them based on availability and resource requirements.
5. Instruments (Images): Musicians use instruments to create music. In Kubernetes, these instruments are represented by container images. Each musician (container) uses a specific instrument (image) to perform its part. The Kubernetes orchestra ensures efficient resource allocation, scalability, and high availability for your data warehouse. It allows you to easily manage and scale your data infrastructure, optimizing resource utilization and reducing operational complexity. This approach aligns well with modern DevOps practices, streamlining deployment, automation, and monitoring. By using Kubernetes, you can build a robust, flexible, and cost-effective data warehouse with ByConity. If you'd like to learn more about Kubernetes visit https://kubernetes.io

What Can I use it for?
Here are some use cases for deploying ByConity with Kubernetes

• Live data statistics dashboard
• System link monitoring
• ETL calculation
• Real-time data access
• Behavior log analysis And much more! Now, let's start our deployment method. For this post, we're going to assume you have some kind of experience with Kubernetes. There are a few things we need to have ready first before we begin. How to Deploy ByConity Locally Before diving into the full deployment process, have all your prereqs ready to go. Prerequisites:
• A managed Kubernetes Service like Azure, Google, or Digital Ocean. I'll be using an AWS EKS cluster in this example. Choose the one that fits your or your team/organization's needs.
• Install and set up kubectl in your local environment.
  • a command-line utility used for interacting with Kubernetes clusters.
• Install Helm in your local environment. a package manager for Kubernetes that simplifies the deployment and management of containerized applications, services, and resources in Kubernetes clusters.
• Install Kind and Docker. Steps:
• Navigate to the ByConity Deployment Documentation and head over to Deploy ByConity in Kubernetes. In this post, we're going to be using the local deployment demo version. If you have your own self-built Kubernetes cluster, you'll find those instructions on the same doc.

1. Start by cloning the ByConity deployment code from GitHub: git clone git@github.com:ByConity/byconity-deploy.git cd byconity-deploy

git clone git@github.com:ByConity/byconity-deploy.git
cd byconity-deploy

Open up your IDE and let's check out the .yaml file we'll be working with. In this example, we'll use values.yaml in the k8s folder(examples -> k8s -> values.yaml). This YAML-formatted file already has default values for ByConity. If you decide to deploy your own self-built cluster, you can declare variables to be passed into your templates. Afterwards, replace the storageClassName.

1. Use Kind to configure a local Kubernetes cluster. Note that Kind is not intended for production use.

kind create cluster --config examples/kind/kind-byconity.yaml

1. Test to ensure the local Kind cluster is ready:

kubectl cluster-info

1. Initialize the ByConity demo cluster:

# Install with fdb CRD first
helm upgrade --install --create-namespace --namespace byconity -f ./examples/k8s/values.yaml byconity ./chart/byconity --set fdb.enabled=false

1. Then install the FBD Cluster

# Install with fdb cluster
helm upgrade --install --create-namespace --namespace byconity -f ./examples/k8s/values.yaml byconity ./chart/byconity

1. Wait until all the pods are ready:

kubectl -n byconity get po

1. Now you can try it out:

kubectl -n byconity exec -it sts/byconity-server -- bash
root@byconity-server-0:/# clickhouse client
172.16.1.1 :)

Let's test out a query

1. To delete or stop ByConity from your Kubernetes cluster, you can use the following command:

helm uninstall --namespace byconity byconity

By following these steps, you can deploy ByConity with Kubernetes and harness its powerful data warehousing capabilities within your own environment.
Join our community on Discord to meet developers and stay updated with the latest releases.