What are they?
Virtual clusters are fully working Kubernetes clusters that run on top of other Kubernetes clusters. Compared to fully separate "real" clusters, virtual clusters reuse worker nodes and networking of the host cluster. They have their own control plane and schedule all workloads into a single namespace of the host cluster. Like virtual machines, virtual clusters partition a single physical cluster into multiple separate ones.
Official documentation
Why do we need them?
To partition the cluster into multiple virtual clusters which can be provided to the tenant teams for isolation. They provide following advantages:
- Ease of use - each tenant gets a cluster of their own!
- Customization - ability to install and use (and test) different cluster level resources (CRDs)
- Isolation - much stricter isolation than traditional namespace based multi tenancy
- Manageability - operators need not handle (too) many clusters
- Lightweight and full-fledged - based on the popular
k3sdistro and sqlite for DB (instead of etcd)
How to use them
Download vcluster cli
brew install vclusterInstall the cluster under a namespace
team-abc
❯ time vcluster create -n team-abc vcluster-abc
info Detected local kubernetes cluster kind. Will deploy vcluster with a NodePort & sync real nodes
info Create vcluster vcluster-abc...
done √ Successfully created virtual cluster vcluster-abc in namespace team-abc
info Waiting for vcluster to come up...
warn vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating
warn vcluster is waiting, because vcluster pod vcluster-abc-0 has status: ContainerCreating
warn vcluster is waiting, because vcluster pod vcluster-abc-0 has
...
info Starting proxy container...
done √ Switched active kube context to vcluster_vcluster-abc_team-abc_kind-macbook
- Use `vcluster disconnect` to return to your previous kube context
- Use `kubectl get namespaces` to access the vcluster
vcluster create -n team-abc vcluster-abc 0.57s user 0.49s system 0% cpu 2:18.09 total
- Check the new cluster
❯ kubectl cluster-info
Kubernetes control plane is running at https://127.0.0.1:10754
CoreDNS is running at https://127.0.0.1:10754/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.
❯ kubectl ns
default
kube-system
kube-public
kube-node-lease
- Deploy workloads
❯ kubectl create ns test-ns
namespace/test-ns created
❯ kubectl create deployment test-dep --image=nginx --replicas=2 -n test-ns
deployment.apps/test-dep created
❯ kubectl get all -n test-ns
NAME READY STATUS RESTARTS AGE
pod/test-dep-574f5c6754-x4l45 0/1 ContainerCreating 0 51s
pod/test-dep-574f5c6754-7tlr5 0/1 ContainerCreating 0 51s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/test-dep 0/2 2 0 51s
NAME DESIRED CURRENT READY AGE
replicaset.apps/test-dep-574f5c6754 2 2 0 51s
- Check back from host cluster
❯ vcluster disconnect
❯ k get pods -n team-abc
NAME READY STATUS RESTARTS AGE
coredns-5df468b6b7-dtmn6-x-kube-system-x-vcluster-abc 1/1 Running 0 7m38s
test-dep-574f5c6754-7tlr5-x-test-ns-x-vcluster-abc 1/1 Running 0 2m47s
test-dep-574f5c6754-x4l45-x-test-ns-x-vcluster-abc 1/1 Running 0 2m47s
vcluster-abc-0 2/2 Running 0 9m30s
House keeping
❯ vcluster list
NAME NAMESPACE STATUS CONNECTED CREATED AGE
vcluster-abc team-abc Running 2022-06-26 21:38:38 +0530 IST 10m44s
❯ vcluster delete vcluster-abc
info Stopping docker proxy...
info Delete vcluster vcluster-abc...
done √ Successfully deleted virtual cluster vcluster-abc in namespace team-abc
done √ Successfully deleted virtual cluster pvc data-vcluster-abc-0 in namespace team-abc
Get the kubeconfig of a vcluster
❯ vcluster list
NAME NAMESPACE STATUS CONNECTED CREATED AGE
vcluster-abc team-abc Running True 2022-06-26 21:51:11 +0530 IST 5m29s
❯ vcluster disconnect
# Look for a secret named vc-<cluster-name> in the <namespace> used
❯ kubectl get secret vc-vcluster-abc -n team-abc --template={{.data.config}} | base64 -D
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJkakNDQVIyZ0F3SUJBZ0lCQURBS0JnZ3Foa2pPUFFRREFqQWpNU0V3SHdZRFZRUUREQmhyTTNNdGMyVnkKZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3SGhjTk1qSXdOakkyTVRZeU1qQTFXaGNOTXpJd05qSXpNVFl5TWpBMQpXakFqTVNFd0h3WURWUVFEREJock0zTXRjMlZ5ZG1WeUxXTmhRREUyTlRZeU5qQTFNalV3V1RBVEJnY3Foa2pPClBRSUJCZ2dxaGtqT1BRTUJCd05DQUFTVThuZGdKaWlNTFFHTEI0VmxSSjlQNm9LdTdGbkU4ZTM2OXl0eDhUYnUKeFJxNTg3NVE4SE1zRU9SMjl4a2puUmN5dUNrcVpOTTRHeDYzLzM1bW5TSjBvMEl3UURBT0JnTlZIUThCQWY4RQpCQU1DQXFRd0R3WURWUjBUQVFIL0JBVXdBd0VCL3pBZEJnTlZIUTRFRmdRVU5PKzlHb1M5Z1lib0NMWjU3ck9GCmdndFFDZlF3Q2dZSUtv=
...
TODO
There are a lot more features to be explored and will do in another post
- pause/resume
- applying manifests on init
- passing down limits and quotas
- extensions/plugins
- using different distributions
What is impressive
- Tailor made for testing CRDs
- Probably well suited for CI use-cases to create clusters on the fly
- Very useful in creating throw-away clusters (for DEV/R&D purposes)
Top comments (0)