Resource quota
When several users or teams share a cluster with a fixed number of nodes, there is a concern that one team could use more than its fair share of resources.
Resource quotas are a tool for administrators to address this concern.
A resource quota, defined by a ResourceQuota object, provides constraints that limit aggregate resource consumption per namespace. It can limit the quantity of objects that can be created in a namespace by type, as well as the total amount of compute resources that may be consumed by resources in that namespace.
Resource quotas work like this:
Different teams work in different namespaces. This can be enforced with RBAC.
The administrator creates one ResourceQuota for each namespace.
Users create resources (pods, services, etc.) in the namespace, and the quota system tracks usage to ensure it does not exceed hard resource limits defined in a ResourceQuota.
If creating or updating a resource violates a quota constraint, the request will fail with HTTP status code 403 FORBIDDEN with a message explaining the constraint that would have been violated.
If quota is enabled in a namespace for compute resources like cpu and memory, users must specify requests or limits for those values; otherwise, the quota system may reject pod creation. Hint: Use the LimitRanger admission controller to force defaults for pods that make no compute resource requirements.
Explanation of resource_quota.yaml file
This is a YAML file containing two Kubernetes objects.
apiVersion: v1
kind: Namespace
metadata:
name: mynamespace
---
apiVersion: v1
kind: List
items:
- apiVersion: v1
kind: ResourceQuota
metadata:
name: user-compute-quota
namespace: mynamespace
spec:
hard:
requests.cpu: "1"
requests.memory: 1Gi
limits.cpu: "2"
limits.memory: 2Gi
- apiVersion: v1
kind: ResourceQuota
metadata:
name: user-object-quota
namespace: mynamespace
spec:
hard:
configmaps: "10"
persistentvolumeclaims: "4"
replicationcontrollers: "20"
secrets: "10"
services: "10"
services.loadbalancers: "2"
The first object is of kind "Namespace" and has a metadata field with the name "mynamespace". This is creating a namespace named "mynamespace" in Kubernetes.
The second object is of kind "List" and contains a list of two "ResourceQuota" objects. The first "ResourceQuota" object has a metadata field with the name "user-compute-quota" and namespace set to "mynamespace". The "spec" field defines resource usage limits for the namespace, such as CPU and memory requests and limits. Specifically, this quota allows for a maximum of 1 CPU and 1Gi of memory for requests and a maximum of 2 CPUs and 2Gi of memory for limits.
The second "ResourceQuota" object has a metadata field with the name "user-object-quota" and namespace set to "mynamespace". The "spec" field defines limits on the number of objects of different types that can be created in the namespace. Specifically, this quota allows for a maximum of 10 configmaps, 4 persistentvolumeclaims, 20 replicationcontrollers, 10 secrets, 10 services, and 2 load balancer services.
Steps to create ‘Resource Quota’ objects and observe their working
- Create resource quota objects:
kubectl apply -f resource_quota.yaml
- Verify created resource quota objects:
kubectl get quota -n mynamespace
NAME AGE REQUEST LIMIT
user-compute-quota 79s requests.cpu: 0/1, requests.memory: 0/1Gi limits.cpu: 0/2, limits.memory: 0/2Gi
user-object-quota 44s configmaps: 1/10, persistentvolumeclaims: 0/4, replicationcontrollers: 0/20, secrets: 0/10, services: 0/10, services.loadbalancers: 0/2
- Create deployment (dep_without_quota.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
namespace: mynamespace
labels:
app: nginx
spec:
replicas: 3
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: k8s-demo
image: nginx
ports:
- name: nginx-port
containerPort: 8080
kubectl create -f dep_without_quota.yaml
- Verify deployment
kubectl get deploy -n mynamespace
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 0/3 0 0 31s
nothing in ready state.
- Explore reason
kubectl get rs -n mynamespace
NAME DESIRED CURRENT READY AGE
nginx-deployment-64ff9dbcdf 3 0 0 2m51s
kubectl describe rs nginx-deployment-64ff9dbcdf
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-wkr8s" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-mhn6c" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-77n7m" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-t5m7q" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-j6974" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-ldtx8" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m9s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-jzfqg" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m8s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-xrp62" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 3m8s replicaset-controller Error creating: pods "nginx-deployment-64ff9dbcdf-s29ht" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
Warning FailedCreate 25s (x7 over 3m6s) replicaset-controller (combined from similar events): Error creating: pods "nginx-deployment-64ff9dbcdf-9gxvc" is forbidden: failed quota: user-compute-quota: must specify limits.cpu for: k8s-demo; limits.memory for: k8s-demo; requests.cpu for: k8s-demo; requests.memory for: k8s-demo
since quota is not specified in the deployment file, it fails to provision the pods.
- Create deployment (dep_with_quota.yaml)
apiVersion: apps/v1
kind: Deployment
metadata:
name: helloworld-deployment
namespace: mynamespace
labels:
app: nginx-ssl
spec:
replicas: 3
selector:
matchLabels:
app: nginx-ssl
template:
metadata:
labels:
app: nginx-ssl
spec:
containers:
- name: k8s-demo
image: nginx:1.16
ports:
- name: nginxssl-port
containerPort: 8081
resources:
requests:
cpu: 200m
memory: 0.5Gi
limits:
cpu: 400m
memory: 1Gi
kubectl create -f dep_with_quota.yaml
- Verify deployment
kubectl get rs -n mynamespace
NAME DESIRED CURRENT READY AGE
helloworld-deployment-7986cbf64d 3 2 2 45s
nginx-deployment-64ff9dbcdf 3 0 0 8m52s
only 2 out of 3 are in ready state.
- Explore reason
kubectl get rs -n mynamespace
NAME DESIRED CURRENT READY AGE
helloworld-deployment-7986cbf64d 3 2 2 45s
nginx-deployment-64ff9dbcdf 3 0 0 8m52s
kubectl describe rs helloworld-deployment-7986cbf64d -n mynamespace
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 59s replicaset-controller Created pod: helloworld-deployment-7986cbf64d-k46hw
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-cxjqx" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Normal SuccessfulCreate 59s replicaset-controller Created pod: helloworld-deployment-7986cbf64d-r68vf
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-sx2vj" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-zx6pq" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-khchm" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-z8j4s" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-7rqj4" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-sb8ml" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 59s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-pxsdv" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 58s replicaset-controller Error creating: pods "helloworld-deployment-7986cbf64d-jdnfp" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
Warning FailedCreate 39s (x8 over 57s) replicaset-controller (combined from similar events): Error creating: pods "helloworld-deployment-7986cbf64d-pjms2" is forbidden: exceeded quota: user-compute-quota, requested: limits.memory=1Gi,requests.memory=512Mi, used: limits.memory=2Gi,requests.memory=1Gi, limited: limits.memory=2Gi,requests.memory=1Gi
failed to provision 3rd pod as it exceeded quota
- How much quota exhausted:
kubectl describe quota user-compute-quota -n mynamespace
Name: user-compute-quota
Namespace: mynamespace
Resource Used Hard
-------- ---- ----
limits.cpu 800m 2
limits.memory 2Gi 2Gi
requests.cpu 400m 1
requests.memory 1Gi 1Gi
And they consumed resources as specified in resources specs of the helloworld deployment.
- Create a limit ranger object:
apiVersion: v1
kind: LimitRange
metadata:
name: limits-quota
namespace: mynamespace
spec:
limits:
- default:
cpu: 200m
memory: 512Mi
defaultRequest:
cpu: 100m
memory: 256Mi
type: Container
kubectl create -f limit_range.yaml
- Now retry dep_without_quota.yaml
kubectl create -f dep_without_quota.yaml
- Verify again dep_without_quota replicas/pods:
kubectl get deploy -n mynamespace
NAME READY UP-TO-DATE AVAILABLE AGE
nginx-deployment 3/3 3 3 13s
All 3 are in ready states
- Finally check the consumption of quota
kubectl describe quota user-compute-quota -n mynamespace
Name: user-compute-quota
Namespace: mynamespace
Resource Used Hard
-------- ---- ----
limits.cpu 600m 2
limits.memory 1536Mi 2Gi
requests.cpu 300m 1
requests.memory 768Mi 1Gi
All 3 pods got scheduled using default limits specified in limit_range.yaml This is because the deployment ‘nginx-deployment’ does not have resources spec specified hence the pods created out of it use default specs from LimitRange object.
Folks, if you like my content, would you consider following me on linked in at: https://www.linkedin.com/in/hitesh-pattanayak-52290b160/
Top comments (0)