Resource quotas prevent resource contention issues by rationing available resources (cpu, memory, gpu ..). Without these controls one rogue application can potentially make the cluster crawl.
Resource quotas provide constraints that limit resource consumption.
They can be applied only at the namespace level, which means they can be applied to computing resources and limit the number of objects inside the namespace.
In Kubernetes quotas are implemented with a ResourceQuota object. It can be used to limit/control the following objects:
- ConfigMaps
- Persistent Volume Claims (PVCs)
- Pods
- Secrets
- Services
Create quotas
- Check:
Apply quota to a namespace
❯ kubectl create -f compute-resources.yaml -n test-ns
resourcequota/compute-resources created
❯ kubectl create -f object-counts.yaml -n test-ns
resourcequota/object-counts created
Get quota details and status
❯ kubectl get resourcequotas -n test-ns
NAME AGE REQUEST LIMIT
compute-resources 22s requests.cpu: 0/10, requests.memory: 0/40Gi, requests.nvidia.com/gpu: 0/2 limits.cpu: 0/15, limits.memory: 0/60Gi
object-counts 11s configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 0/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4
Dele quotas
❯ kubectl delete resourcequota/compute-resources resourcequota/object-counts
resourcequota "compute-resources" deleted
resourcequota "object-counts" deleted
Tests
❯ for i in $(seq 1 5); do
kubectl create deployment test-dep-${i} --image=alpine --replicas=1 -n test-ns
kubectl expose deployment test-dep-${i} --type=NodePort --port=80 -n test-ns
done
deployment.apps/test-dep-1 created
service/test-dep-1 exposed
deployment.apps/test-dep-2 created
service/test-dep-2 exposed
deployment.apps/test-dep-3 created
service/test-dep-3 exposed
deployment.apps/test-dep-4 created
service/test-dep-4 exposed
deployment.apps/test-dep-5 created
Error from server (Forbidden): services "test-dep-5" is forbidden: exceeded quota: object-counts, requested: services.nodeports=1, used: services.nodeports=4, limited: services.nodeports=4
Side effect:
Even the pods are also not created as there are no memory, cpu requirements given
❯ kubectl get deployments.apps test-dep-1 -n test-ns
NAME READY UP-TO-DATE AVAILABLE AGE
test-dep-1 0/1 0 0 5m17s
❯ kubectl get rs test-dep-1-85d7f5dcdb -n test-ns
NAME DESIRED CURRENT READY AGE
test-dep-1-85d7f5dcdb 1 0 0 5m44s
❯ kubectl describe rs test-dep-1-85d7f5dcdb -n test-ns | tail -2
Warning FailedCreate 4m40s replicaset-controller Error creating: pods "test-dep-1-85d7f5dcdb-jh62r" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
Warning FailedCreate 118s (x7 over 4m39s) replicaset-controller (combined from similar events): Error creating: pods "test-dep-1-85d7f5dcdb-6679m" is forbidden: failed quota: compute-resources: must specify limits.cpu,limits.memory,requests.cpu,requests.memory
We add resource limits:
...
spec:
containers:
- name: nginx
image: nginx:1.14.2
ports:
- containerPort: 80
resources:
limits:
memory: '1Gi'
cpu: '800m'
requests:
memory: '256Mi'
cpu: '100m'
Rerun the deployments (replicaCount=3)
❯ for i in $(seq 1 5); do
sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f -
done
deployment.apps/nginx-dep-1 created
deployment.apps/nginx-dep-2 created
deployment.apps/nginx-dep-3 created
deployment.apps/nginx-dep-4 created
deployment.apps/nginx-dep-5 created
❯ k get deployments.apps -n test-ns
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-dep-1 3/3 3 3 102s
nginx-dep-2 3/3 3 3 102s
nginx-dep-3 3/3 3 3 102s
nginx-dep-4 3/3 3 3 101s
nginx-dep-5 3/3 3 3 101s
❯ k get resourcequotas -n test-ns
NAME AGE REQUEST LIMIT
compute-resources 22m requests.cpu: 1500m/10, requests.memory: 3840Mi/40Gi, requests.nvidia.com/gpu: 0/2 limits.cpu: 12/15, limits.memory: 15Gi/60Gi
object-counts 22m configmaps: 1/20, persistentvolumeclaims: 0/20, pods: 15/50, replicationcontrollers: 0/20, secrets: 1/20, services: 0/20, services.loadbalancers: 0/4, services.nodeports: 0/4
Let us push beyond cpu limits
❯ for i in $(seq 6 10); do
sed -e "s/nginx-dep/nginx-dep-${i}/" /tmp/dep.yaml | kubectl apply -f -
deployment.apps/nginx-dep-6 created
deployment.apps/nginx-dep-7 created
deployment.apps/nginx-dep-8 created
deployment.apps/nginx-dep-9 created
deployment.apps/nginx-dep-10 created
done
❯ k get rs -n test-ns
NAME DESIRED CURRENT READY AGE
...
nginx-dep-6-7fbd56f7d8 3 3 3 7m35s
nginx-dep-7-7fbd56f7d8 3 0 0 3m51s
nginx-dep-8-7fbd56f7d8 3 0 0 3m39s
nginx-dep-9-7fbd56f7d8 3 0 0 3m39s
It creates 6 deployments and stalls for deployment-7 .. 10
❯ kubectl describe rs nginx-dep-7-7fbd56f7d8 -n test-ns | tail -2
Warning FailedCreate 4m14s replicaset-controller Error creating: pods "nginx-dep-7-7fbd56f7d8-bv7hb" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15
Warning FailedCreate 92s (x7 over 4m13s) replicaset-controller (combined from similar events): Error creating: pods "nginx-dep-7-7fbd56f7d8-54shl" is forbidden: exceeded quota: compute-resources, requested: limits.cpu=800m, used: limits.cpu=14400m, limited: limits.cpu=15
Setting boundary values for cpu and memory
Use the LimitRange resource
Top comments (1)
Namespaces create virtual clusters within a physical Kubernetes cluster to help users avoid resource naming conflicts and manage capacity, among others. Prayer to separate two people