I'm currently reading the Manning Publication Kubernetes in Action in a book club. Of the four of us who gather every week to discuss the chapter of the week (we just finished Chapter 8), we all seem to use different languages and frameworks for building backends. I have the most experience with Django, so I have been trying to apply my Django knowledge as I learn Kubernetes concepts.
Prior art
When I went Googling for "django in kubernetes", I clicked on this Medium article, which is the first in a series of articles that describes how to setup Django applications in Minikube, and then in AWS using KOPS. I worked the series of articles and was able to successfully set up a functioning Django application in Kubernetes using minikube (I haven't touched on KOPS yet). I was also able to run my integration test suite against minikube.local
(the host name I have selected for my minikube's ingress) and all the tests passed. Before I go any further, I owe a big thanks to @MarkGituma for providing this excellent series. Thank you!
Background
I have been working on an open source project that combines several different technologies that I really enjoy working with. Here are a few that my project uses:
- Django (DRF, celery, beat, channels)
- Vue.js
- Postgres
- Redis
- Quasar Framework (an awesome Vue.js framework)
- Docker (and docker-compose)
- VuePress for documentation
- Cypress (for integration and e2e testing)
- GitLab (GitLab CI, gitlab-runner debugging GitLab CI jobs and saving on GitLab CI/CD minutes)
- AWS Elastic Container Service (ECS)
- CloudFormation (Infrastructure as Code tool from AWS)
The project repo can be found here: https://gitlab.com/verbose-equals-true/django-postgres-vue-gitlab-ecs.
The project name, Verbose Equals True
, is both a reminder and encouragement for me to be as verbose as possible when writing documentation and code comments in order to help those who may wish to adopt the patterns or techniques I have used to develop the project (most of which were borrowed from other projects).
One of my goals for this project is to create an excellent developer experience. To get going, you need to create an .env
file from a template (.env.template
) and then run docker-compose up
. That's it!
This will set you up with a development environment that provides:
- Frontend hot-reloading
- Backend hot-reloading
- Monitoring and utilities (flower, mailhog, redis-commander)
- nginx for making both the frontend and backend available on
localhost
- a default admin user account with credentials automatically populated in the frontend login form
- Simple instructions for running unit and integration tests locally
The project documentation site can also be brought up easily with docker-compose -f compose/docs.yml up
.
Minikube
I really like using ECS and CloudFormation, but I want to learn more about GCP, GKE and other tools for Infrastructure as Code (Terraform and Pulumi are high on the list of tools that I want to try out). I also want to avoid vendor lock-in. ECS is great, but I won't be able to use it if I want to run my application on GCP, Linode, Azure, Digital Ocean or Alicloud. These other providers all offer managed Kubernetes solutions (I think they do, or at least most certainly will at some point in the future).
Minikube allows you to run a single-node Kubernetes cluster inside of a virtual machine on your laptop. It has been a great tool for learning how Kubernetes works and for getting comfortable with kubectl
, the Kubernetes CLI tool that is used to interact with a Kubernetes cluster.
Quickstart
Here's an overview of of how to start an application on minikube. A more detailed explanation can be found in the next section.
Project setup
Here is how to setup the entire application from a fresh minikube Kubernetes cluster.
Prepare minikube
Delete any existing minikube cluster and then start a new one:
minikube delete
minikube start
Edit /etc/hosts
Get the minikube Kubernetes cluster IP with the following command:
minikube ip
192.168.99.108
Edit your /etc/hosts
file and add an entry to map minikube.local
(or any other domain you want to use) to the result of minikube ip
:
sudo vim /etc/hosts
127.0.0.1 localhost
127.0.1.1 a1
192.168.99.108 minikube.local <-- this is the line you need to add
# The following lines are desirable for IPv6 capable hosts
::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
Enable ingress addon
Then enable the ingress adddon:
minikube addons enable ingress
Build docker images
Next, set your local docker CLI to point to the minikube docker daemon:
eval $(minikube docker-env)
Next, build the frontend and backend containers with the following command:
docker-compose -f compose/minikube.yml build frontend backend
Configure Kubernetes resources
k apply -f kubernetes/postgres/
k apply -f kubernetes/redis/
k apply -f kubernetes/django/
k apply -f kubernetes/channels/
k apply -f kubernetes/celery/
k apply -f kubernetes/beat/
k apply -f kubernetes/frontend/
k apply -f kubernetes/flower/
k apply -f kubernetes/ingress.yml
Check that you can visit minikube.local
in your browser.
The rest of this article will be copied from the documentation site of Verbose Equals True (which is hosted on GitLab pages). It takes things one step at a time and provides more details than the previous Quickstart
section.
(This page discusses setting up my Django/Vue application in minikbue: https://verbose-equals-true.gitlab.io/django-postgres-vue-gitlab-ecs/topics/minikube/.)
Step-by-Step Guide to setting up a Django application in minikube
Minikube is a tool for running a single-node Kubernetes cluster inside of a virtual machine. It is a popular tool for developing Kubernetes applications locally.
This topic will cover using minikube
to set up the project Kubernetes locally.
By the end of this guide, you will be able to:
Navigate to
http://minikube.local
in your browser and interact with the application running in minikube in the same way that you would with the application running using docker-compose for local development.Run Cypress tests against the application running in minikube to verify that everything is working correctly.
I'll be following this great guide to get started, making changes and additions where necessary.
Getting started
Start minikube
To get started, bring up minikube
with
minikube start
Optionally, run minikube delete
, and then minikube start
to start with a clean cluster.
I'll be using the following alias to use kubectl
:
alias k='kubectl'
Building Images
We will need to build two images from our code:
- The
backend
image that will run the Django server, Django Channels, Celery and Beat - The
frontend
image that will contains nginx for serving our Quasar frontend application.
Both of these images will need environment variables. We will use docker-compose
to easily manage the building and environment variable management. Read this article for more information. You don't absolutely have to user docker-compose to build the images, but it should keep things straightforward and easy to understand.
Remember that that the docker CLI, like kubectl
, send requests to a REST API. When we run minikube start
, this configures kubectl
to send commands to the Kubernetes API server that is running inside of the minikube virtual machine. Similarly, we need to tell our docker CLI that we want to send API calls that the docker CLI command makes to the docker daemon running in the minikube VM, not the docker daemon on our local machine (even though the files from which we build our images are on our local machine and not on the minikube VM's file system). We can configure our docker CLI to point to the minikube VM with the following command:
eval $(minikube docker-env)
Now run docker ps
and you will see many different containers that Kubernetes uses internally.
To point the docker CLI back at your local docker daemon, run:
eval $(minikube docker-env -u)
Let's look at what the command is doing:
$(minikube docker-env)
results in the following output:
export DOCKER_TLS_VERIFY="1"
export DOCKER_HOST="tcp://192.168.99.100:2376"
export DOCKER_CERT_PATH="/home/brian/.minikube/certs"
# Run this command to configure your shell:
# eval $(minikube docker-env)
Notice that the DOCKER_HOST
is pointing to the minikube VM on docker's default port 2376
. eval
executes these commands, setting the environment variables in the current shell by using export
. If you switch to another shell, you will need to rerun this command if you want to run docker commands against minikube's docker daemon.
With these environment variables set, let's build the Django container image with the following command:
docker-compose -f compose/minikube.yml build backend
Here's the backend
service defined in compose/minikube.yml
:
backend:
image: backend:1
build:
context: ../backend/
dockerfile: scripts/dev/Dockerfile
kubernetes/django/deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
name: django-backend
labels:
app: django-backend
spec:
replicas: 1
selector:
matchLabels:
app: django-backend
template:
metadata:
labels:
app: django-backend
spec:
containers:
- name: django-backend-container
imagePullPolicy: IfNotPresent
image: backend:1
command: ["./manage.py", "runserver", "0.0.0.0:8000"]
ports:
- containerPort: 8000
Note: the pod template in this deployment definition does not have any environment variables. We will need to add environment variables for sensitive information such as the Postgres username and password. We will add these shortly.
There is one line in the above resource definition that makes everything work with minikube and the docker images we have just built: imagePullPolicy: IfNotPresent
. This line tells Kubernetes to pull the image (from Docker Hub, or another registry if specified) only if the image is not present locally. If we didn't set the imagePullPolicy
to IfNotPresent
, Kubernetes would try to pull the image from docker hub, which would probably fail, resulting in an ErrImagePull
.
Don't configure the deployment yet!
We would run the following command to configure this deployment.
kubectl apply -f kubernetes/django/deployment.yml
We haven't created the secrets that Django needs yet for access to the Postgres database, save this file and we will come back to it after we configure Postgres in our minikube Kubernetes cluster.
Postgres
Using Postgres in our minikube cluster will involve the following resources:
- secrets
- persistent volume
- persistent volume claim
- deployment
- service
Secrets
Secrets should be base64 encoded because they can contain either strings or raw bytes. Here's an example of how we can encode my-secret-string
with base64 encdoding:
echo -n "my-secret-string" | base64
bXktc2VjcmV0LXN0cmluZw==
We will use bXktc2VjcmV0LXN0cmluZw==
in our secrets.yml
file. We shouldn't commit any sensitive information in secrets files. base64 encdoing is not encrypted, the value can be decoded read as my-secret-string
:
echo -n "bXktc2VjcmV0LXN0cmluZw==" | base64 -d
my-secret-string
Choose a username and password for your Postgres database and enter both of them as base64-encoded values:
kubernetes/postgres/secrets.yml
apiVersion: v1
kind: Secret
metadata:
name: postgres-credentials
type: Opaque
data:
user: YnJpYW4=
password: cGFzc3dvcmQx
You can open the minikube dashboard with minikube dashboard
and view the secret values after you send this file to the kubernetes API with:
k apply -f kubernetes/postgres/secrets.yml
Persistent Volume
Next, we need to configure a volume to persist data that will be stored in the postgres database.
In minikube, since we are only using a single-node cluster, it is OK to use a hostPath
volume:
kubernetes/postgres/volume.yml
kind: PersistentVolume
apiVersion: v1
metadata:
name: postgres-pv
labels:
type: local
spec:
storageClassName: manual
capacity:
storage: 2Gi
accessModes:
- ReadWriteOnce
hostPath:
path: /data/postgres-pv
Persistent Volumes are not namespaced in Kubernetes
Persistent Volume Claim
Next we will make a persistent volume claim that we can reference in the postgres deployment:
kubernetes/postgres/volume_claim.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: postgres-pvc
labels:
type: local
spec:
storageClassName: manual
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2Gi
volumeName: postgres-pv
The storageClassName
is arbitrary; it only needs to be the same value in order for the PVC to get access to the storage it needs.
Deployment
Now we can create the Postgres deployment. This will use our secrets and persistent volumes:
kubernetes/postgres/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: postgres-deployment
spec:
replicas: 1
selector:
matchLabels:
app: postgres-container
template:
metadata:
labels:
app: postgres-container
tier: backend
spec:
containers:
- name: postgres-container
image: postgres:9.6.6
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
ports:
- containerPort: 5432
volumeMounts:
- name: postgres-volume-mount
mountPath: /var/lib/postgresql/data
volumes:
- name: postgres-volume-mount
persistentVolumeClaim:
claimName: postgres-pvc
Service
Finally, we can create a service that will allow us to access the Postgres database from pods in our Django deployment (which we will come back to next):
kubernetes/postgres/service.yml
kind: Service
apiVersion: v1
metadata:
name: postgres
spec:
selector:
app: postgres-container
ports:
- protocol: TCP
port: 5432
targetPort: 5432
Redis
Next, let's configure a redis server in our minikube cluster. This is similar to the guestbook example from the Kubernetes documentation, but we will only have a single-node redis cluster, not a master-slave setup.
Deployment
kubernetes/redis/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: redis
labels:
deployment: redis
spec:
selector:
matchLabels:
pod: redis
replicas: 1
template:
metadata:
labels:
pod: redis
spec:
containers:
- name: master
image: redis
resources:
requests:
cpu: 100m
memory: 100Mi
ports:
- containerPort: 6379
Service
kubernetes/redis/service.yml
apiVersion: v1
kind: Service
metadata:
name: redis
spec:
selector:
pod: redis
ports:
- protocol: TCP
port: 6379
targetPort: 6379
Configure the redis deployment and service with the following command:
k apply -f kubernetes/redis/
Django Webserver
Deployment
Next let's come back to the deployment that will serve requests for our Django API. As mentioned earlier, this needs to be configured with some additional environment variables. Some of these environment variables will be added explicitly, and some will be added automatically by Kubernetes for simple and easy service discovery.
Here's the full deployment definition for our Django deployment:
kubernetes/django/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: django
spec:
replicas: 1
selector:
matchLabels:
app: django-container
template:
metadata:
labels:
app: django-container
spec:
containers:
- name: backend
imagePullPolicy: IfNotPresent
image: backend:11
command: ["./manage.py", "runserver", "0.0.0.0:8000"]
livenessProbe:
httpGet:
path: /healthz
port: 8000
readinessProbe:
# an http probe
httpGet:
path: /readiness
port: 8000
initialDelaySeconds: 10
timeoutSeconds: 5
ports:
- containerPort: 8000
env:
- name: DJANGO_SETTINGS_MODULE
value: 'backend.settings.minikube'
- name: SECRET_KEY
value: "my-secret-key"
- name: POSTGRES_NAME
value: postgres
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
# I'm not sure that we need these volumes, but they were included in the tutorial referenced at the beginning of this guide.
volumeMounts:
- name: postgres-volume-mount
mountPath: /var/lib/busybox
volumes:
- name: postgres-volume-mount
persistentVolumeClaim:
claimName: postgres-pvc
Let's notice the additions to our Django deployment. First, we see an array of environment variables:
-
DJANGO_SETTINGS_MODULE
: this tells Django which settings module to use. It is set tobackend.settings.minikube
, which means that we are using the settings filebackend/settings/minikube.py
-
SECRET_KEY
: Django needs a secret key to start (this should also be configured as a secret...) -
POSTGRES_NAME
: we are using the defaultpostgres
database -
POSTGRES_USER
andPOSTGRES_PASSWORD
: these environment variables that we are
Let's look at the minikube.py
settings file:
backend/settings/minikube.py
from .development import * # noqa
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': os.environ.get('POSTGRES_NAME', 'kubernetes_django'), # noqa
'USER': os.environ.get('POSTGRES_USER', 'postgres'), # noqa
'PASSWORD': os.environ.get('POSTGRES_PASSWORD', 'postgres'), # noqa
'HOST': os.environ.get('POSTGRES_SERVICE_HOST', 'postgres'), # noqa
'PORT': os.environ.get('POSTGRES_SERVICE_PORT', 5432), # noqa
}
}
Notice that in the DATABASES
section we see the Postgres name, user and password environment variables that we added to the deployment's pod template.
POSTGRES_SERVICE_HOST
and POSTGRES_SERVICE_PORT
are added automatically. Kubernetes adds a set of environment variables for all services in the namespace that include the service IP and the service port of the service. Environment variables are one of two ways to do this type of simple service discovery.
Also, take note of the addition of the livenessProbe
and readinessProbe
keys in the container definition of the pod template. These tell kubelet to send HTTP requests to /healthz
and /readiness
which are used to evaluate the health and readiness of the Django deployment, respectively. We will come back to these to see exactly how they work by sabotaging our Django deployment in different ways.
See this article as a reference for how health checks have been implemented using Django middleware.
Service
Now that we have a deployment for our Django webserver, let's create a service that will allow us to reach it:
kubernetes/django/service.yml
kind: Service
apiVersion: v1
metadata:
name: kubernetes-django-service
spec:
selector:
app: django-container
ports:
- protocol: TCP
port: 8000
targetPort: 8000
type: NodePort
This needs to do two things: match the django-container
label that is present in the Django deployment pod template, and specify port 8000
that our Django webserver is listening on, and that the pod has configured with containerPort: 8000
.
Migration Job
We are almost ready to apply our Django deployment and service, but before we do that we need migrate our database by running ./manage.py migrate
. The migration should be ran once, and it must run successfully. This type of task can be handled by a Kubernetes Job.
kubernetes/django/migration.yml
apiVersion: batch/v1
kind: Job
metadata:
name: django-migrations
spec:
template:
spec:
containers:
- name: django
image: backend:2
command: ['python', 'manage.py', 'migrate']
env:
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
- name: POSTGRES_NAME
value: postgres
- name: DJANGO_SETTINGS_MODULE
value: 'backend.settings.minikube'
restartPolicy: Never
backoffLimit: 5
Configure the job by running the following command:
k apply -f kubernetes/django/migration.yml
Now let's inspect our pods
k get pods
NAME READY STATUS RESTARTS AGE
django-migrations-lphgb 0/1 Completed 0 9s
postgres-deployment-57df8f899f-8fzmj 1/1 Running 0 53s
The Django migration file has a status of Completed
, which should mean that the migrations have completed succesffully. Let's verify this by inspecting the pod logs:
Now let's look at the Job's pod logs:
k logs django-migrations-lphgb
loading minikube settings...
Operations to perform:
Apply all migrations: accounts, admin, auth, contenttypes, sessions, social_django
Running migrations:
Applying contenttypes.0001_initial... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0001_initial... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying auth.0010_alter_group_name_max_length... OK
Applying auth.0011_update_proxy_permissions... OK
Applying accounts.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying admin.0003_logentry_add_action_flag_choices... OK
Applying sessions.0001_initial... OK
Applying social_django.0001_initial... OK
Applying social_django.0002_add_related_name... OK
Applying social_django.0003_alter_email_max_length... OK
Applying social_django.0004_auto_20160423_0400... OK
Applying social_django.0005_auto_20160727_2333... OK
Applying social_django.0006_partial... OK
Applying social_django.0007_code_timestamp... OK
Applying social_django.0008_partial_timestamp... OK
We can see that our database migrations did indeed run successfully. Now we can configure the Django service and deployment with the following command:
k apply -f kubernetes/django/
deployment.apps/django created
job.batch/django-migrations unchanged
service/kubernetes-django-service created
Visit the Django admin panel by running the following command:
minikube service kubernetes-django-service
and then navigate to /admin
, and you should see the Django admin login page. Let's create a default user. I have a management command which we can run:
k exec django-59fc87fd6f-7slzl -it -- ./manage.py create_default_user
loading minikube settings...
Creating default user
Default user created:
email: 'admin@company.com'
password: 'password'
You could also replace my create_default_user
command with createsuperuser
and create a user that way.
Login with your user to verify that everything is working properly.
Frontend
Building the frontend image
Now that the Django backend is working, let's take a look at the front end client that is built with Vue and Quasar Framework and served with nginx. As we did with the backend, we will build the frontend container with the compose/minikube.py
file. Let's look at the frontend service definition in that file:
compose/minikube.yml
version: '3.7'
services:
frontend:
image: frontend:1
build:
context: ../
dockerfile: nginx/minikube/Dockerfile
args:
- DOMAIN_NAME=minikube.local
- GOOGLE_OAUTH2_KEY=google123
- GITHUB_KEY=github123
- WS_PROTOCOL=ws
- HTTP_PROTOCOL=http
Make sure that your current shell has the correct environment variables set for the DOCKER_HOST
by running:
eval $(minikube docker-env)
Build the image with the following command:
docker-compose -f compose/minikube.yml build frontend
Notice that we set DOMAIN_NAME
to be minikube.local
. We will use this address to access both the frontend and backend service once we configure an Ingress for our minikube Kubernetes cluster.
Deployment
kubernetes/fronend/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: frontend-deployment
labels:
app: frontend
spec:
replicas: 1
selector:
matchLabels:
app: frontend-container
template:
metadata:
labels:
app: frontend-container
spec:
containers:
- name: frontend
imagePullPolicy: IfNotPresent
image: frontend:1
Service
For now let's finish by setting up a service for the frontend client:
kubernetes/fronend/service.yml
kind: Service
apiVersion: v1
metadata:
name: kubernetes-frontend-service
spec:
selector:
app: frontend-container
ports:
- nodePort: 30002
protocol: TCP
port: 80
targetPort: 80
type: NodePort
In this service the nodePort
is set explicitly, but doesn't have to be as is the case with the Django service.
Configure the frontend deployment and service with the following command:
k apply -f kubernetes/fronent/
Now let's take a look at the frontend site by visiting it in the browser. Run the following:
minikube service kubernetes-frontend-service
Or, since we know that the nodePort
is 30002
, go to <minikube ip>
:30002.
Ingress
Enable Ingress Addon in Minikibe
minikube addons enable ingress
Define Ingress Resource for services
With the Ingress enabled, we can add an Ingress
resource:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: minikube-ingress
spec:
rules:
- host: minikube.local
http:
paths:
- path: /api/
backend:
serviceName: kubernetes-django-service
servicePort: 8000
- path: /admin/
backend:
serviceName: kubernetes-django-service
servicePort: 8000
- path: /static/
backend:
serviceName: kubernetes-django-service
servicePort: 8000
- path: /
backend:
serviceName: kubernetes-frontend-service
servicePort: 80
Configure the Ingress resource with the following command:
k apply -f kubernetes/ingress.yml
ingress.extensions/minikube-ingress created
/etc/hosts
Also, we need to add an entry to /etc/hosts
so that requests to minikube.local
will be forwarded to the minikube ip
:
192.168.99.106 minikube.local
Now you navigate to http://minikube.local
in your browser and you should be able to login through the frontend Vue/Quasar app.
Celery
Next, let's add a deployment for Celery. This deployment will be very similar to our Django webserver deployment, but the command will be different. Also, this deployment does not need a service since it only process background tasks; it does not handle API requests. Instead, the celery workers only watch the redis queue for jobs to perform. Here is the deployment:
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: celery-worker
labels:
deployment: celery-worker
spec:
replicas: 1
selector:
matchLabels:
pod: celery-worker
template:
metadata:
labels:
pod: celery-worker
spec:
containers:
- name: celery-worker
image: backend:11
command: ["celery", "worker", "--app=backend.celery_app:app", "--loglevel=info"]
env:
- name: DJANGO_SETTINGS_MODULE
value: 'backend.settings.minikube'
- name: SECRET_KEY
value: "my-secret-key"
- name: POSTGRES_NAME
value: postgres
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
We still need to configure a readinessProbe
and livenessProbe
for the celery worker containers, but for now let's inspect the logs to see if celery is ready start working on tasks:
k logs celery-worker-6d9fffdddf-gsp4r
loading minikube settings...
/usr/local/lib/python3.7/site-packages/celery/platforms.py:801: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!
Please specify a different user using the --uid option.
User information: uid=0 euid=0 gid=0 egid=0
uid=uid, euid=euid, gid=gid, egid=egid,
-------------- celery@celery-worker-6d9fffdddf-gsp4r v4.3.0 (rhubarb)
---- **** -----
--- * *** * -- Linux-4.15.0-x86_64-with-debian-10.1 2019-09-15 18:24:51
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app: backend:0x7fd25e93da90
- ** ---------- .> transport: redis://10.97.206.254:6379/1
- ** ---------- .> results: redis://10.97.206.254/1
- *** --- * --- .> concurrency: 2 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
-------------- [queues]
.> celery exchange=celery(direct) key=celery
[tasks]
. core.tasks.debug_task
. core.tasks.send_test_email_task
. debug_periodic_task
[2019-09-15 18:24:51,686: INFO/MainProcess] Connected to redis://10.97.206.254:6379/1
[2019-09-15 18:24:51,692: INFO/MainProcess] mingle: searching for neighbors
[2019-09-15 18:24:52,716: INFO/MainProcess] mingle: all alone
[2019-09-15 18:24:52,723: WARNING/MainProcess] /usr/local/lib/python3.7/site-packages/celery/fixups/django.py:202: UserWarning: Using settings.DEBUG leads to a memory leak, never use this setting in production environments!
warnings.warn('Using settings.DEBUG leads to a memory leak, never '
[2019-09-15 18:24:52,723: INFO/MainProcess] celery@celery-worker-6d9fffdddf-gsp4r ready.
Beat
Let's look at the logs of our celery beat pod.
k logs celery-beat-7f4cd559bc-9jnmp
loading minikube settings...
celery beat v4.3.0 (rhubarb) is starting.
Stale pidfile exists - Removing it.
__ - ... __ - _
LocalTime -> 2019-09-15 18:42:46
Configuration ->
. broker -> redis://10.97.206.254:6379/1
. loader -> celery.loaders.app.AppLoader
. scheduler -> celery.beat.PersistentScheduler
. db -> celerybeat-schedule
. logfile -> [stderr]@%INFO
. maxinterval -> 5.00 minutes (300s)
[2019-09-15 18:42:46,483: INFO/MainProcess] beat: Starting...
[2019-09-15 18:42:46,495: INFO/MainProcess] Scheduler: Sending due task debug_periodic_task (debug_periodic_task)
[2019-09-15 18:43:00,000: INFO/MainProcess] Scheduler: Sending due task debug_periodic_task (debug_periodic_task)
[2019-09-15 18:44:00,035: INFO/MainProcess] Scheduler: Sending due task debug_periodic_task (debug_periodic_task)
Remember
We never want to scale this deployment; it should always have only one replica in order to ensure that scheduled tasks only fire once. Try scaling this pod and you will see that duplicates of scheduled tasks are sent to the queue.
We can see the results of these tasks in the logs of our celery deployment:
[2019-09-15 18:43:00,006: INFO/MainProcess] Received task: debug_periodic_task[f45ff2e0-dfb8-41f4-84d8-32f66e872c07]
[2019-09-15 18:43:00,010: WARNING/ForkPoolWorker-2] Periodic task complete
[2019-09-15 18:43:00,010: INFO/ForkPoolWorker-2] Task debug_periodic_task[f45ff2e0-dfb8-41f4-84d8-32f66e872c07] succeeded in 0.0009783900022739545s: None
[2019-09-15 18:44:00,048: INFO/MainProcess] Received task: debug_periodic_task[69a30165-f052-4ac4-8900-67d7bce8246b]
[2019-09-15 18:44:00,051: WARNING/ForkPoolWorker-2] Periodic task complete
[2019-09-15 18:44:00,051: INFO/ForkPoolWorker-2] Task debug_periodic_task[69a30165-f052-4ac4-8900-67d7bce8246b] succeeded in 0.000996144997770898s: None
There's a better way to look at the results of our celery tasks: flower
. Let's set this up next.
Flower
Let's configure flower with a simple deployment and service:
kubernetes/flower/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: flower
labels:
deployment: flower
spec:
replicas: 1
selector:
matchLabels:
pod: celery-flower
template:
metadata:
labels:
pod: celery-flower
spec:
containers:
- name: flower
image: mher/flower
ports:
- containerPort: 5555
env:
- name: CELERY_BROKER_URL
value: redis://$(REDIS_SERVICE_HOST)/1
resources:
limits:
cpu: 100m
memory: 100Mi
kubernetes/flower/service.yml
apiVersion: v1
kind: Service
metadata:
name: flower-service
spec:
selector:
pod: celery-flower
ports:
- port: 5555
type: NodePort
Websockets
Next, let's add a deployment and service for Django Channels.
kubernetes/channels/deployment.yml
apiVersion: apps/v1beta2
kind: Deployment
metadata:
name: django-channels
spec:
replicas: 1
selector:
matchLabels:
app: django-channels-container
template:
metadata:
labels:
app: django-channels-container
spec:
containers:
- name: backend
imagePullPolicy: IfNotPresent
image: backend:14
command: ["daphne", "backend.asgi:application", "--bind", "0.0.0.0", "--port", "9000"]
livenessProbe:
httpGet:
path: /healthz
port: 9000
readinessProbe:
httpGet:
path: /readiness
port: 9000
initialDelaySeconds: 20
timeoutSeconds: 5
ports:
- containerPort: 9000
env:
- name: DJANGO_SETTINGS_MODULE
value: 'backend.settings.minikube'
- name: SECRET_KEY
value: "my-secret-key"
- name: POSTGRES_NAME
value: postgres
- name: POSTGRES_USER
valueFrom:
secretKeyRef:
name: postgres-credentials
key: user
- name: POSTGRES_PASSWORD
valueFrom:
secretKeyRef:
name: postgres-credentials
key: password
kubernetes/channels/service.yml
kind: Service
apiVersion: v1
metadata:
name: kubernetes-django-channels-service
spec:
selector:
app: django-channels-container
ports:
- protocol: TCP
port: 9000
targetPort: 9000
type: NodePort
Configure the Django channels deployment and service with the following command:
k apply -f kubernetes/channels/
Cypress tests against the minikube cluster
Now that we have implemented all parts of our application in minikube, let's run our tests against the cluster. Run the following command to open Cypress:
$(npm bin)/cypress open --config baseUrl=http://minikube.local
Click Run all specs
and make sure there are no errors in the test results.
Next Steps
Helm
Helm is a convenient way to package Kubernetes applications. The next topic will cover installaing and configuring Helm, and then packaging this application in a Helm chart and deploying everything to our minikube cluster with just one command.
GKE
Now that everything is working locally, the next topic will cover deploying this application to a GKE cluster and implementing monitoring.
GitLab
Implement CI/CD with GitLab CI and an attached Kubernetes cluster, review apps and other GitLab features.
Thanks for reading!
Top comments (4)
Dude.. I hope you've discovered kustomize since you wrote this.
kubernetes.io/docs/tasks/manage-ku...
Great post with good explanation! Is it a coincidence that our tech stack is exactly the same including Quasar! Now containerisation is out of the way, we were gradually switching to K8s after outgrowing Docker-Compose (which is not production tool anyway) and your post is incredibly helpful! You mentioned that your next topic would cover Helm chart deployment - did you manage to release it?
Thank you @geowat ! Also, good choice on your tech stack ;) I haven't added any Helm charts to this project yet. I have mostly been doing ECS and docker swarm for container orchestration in my recent projects. For k8s, I'm also not sure which tool is best for CI/CD and IaC. I know that Helm is a way to do this, but there is also CDK, cdk8s, TerraForm, CDK for Terraform (I'm not making that up) and Pulumi. These options almost all sound better than writing Go-formatted YAML templates which is what Helm does I believe. This does depend on how complex the template logic/application is, I suppose. These options would probably all be fine at least for getting started.
Have you considered docker swarm? If you are familiar with docker-compose, then it might be worth exploring. I usually use docker swarm for single-node projects and ECS with Fargate for something that requires auto-scaling. I did look into Helm for only managing dependencies like postgres and redis, but I'm typically using those as managed services from AWS.
Very nice article, lots of learning, keep posting. I also created a full course on Kubernetes in 10hr on YouTube, please have a look and motivate me. Thanks
youtu.be/toLAU_QPF6o