Background tasks are processes that run in the background so your server is ready to take further requests.
Some common use cases for background tasks:
- Web Scraping
- Heavy data processing
- Image processing
- Multiple 3rd party API calls
- Building Slack-bots (Slack requires a 3 second response)
Basically, if the task takes a long time to complete (e.g. >30 seconds) you probably want to offload it to a background task.
If you have your Django project set up it's dead easy to get a background tasks up and running.
Step 1: Getting Started
We want to install Django Q via:
pip install django-q
Django Q allows us to set up some queues to run in the background. Be sure to run the migrations as well to generate the job models (which you'll see in your admin panel)
python manage.py migrate
It requires a broker though...
Step 2: Setting up Redis
We'll use redis. This is basically the communication channel between your django instance (normal django) and your django worker (this is your background task/worker).
You can check if your redis server is running by running:
redis-cli ping
If not you'll need to follow the redis instructions to set up on your system.
Step 3: Setting up the Cluster
The standard django-q configuration will work for us locally but with a couple of changes so we can run this in production as well. Add the following to your base settings.py
file:
REDIS_HOST = os.environ.get("REDIS_HOST", "localhost")
REDIS_PORT = os.environ.get("REDIS_PORT", 6379)
REDIS_DB = os.environ.get("REDIS_DB", 0)
REDIS_PASSWORD = os.environ.get("REDIS_PASSWORD", None)
Q_CLUSTER = {
"name": "hello-django",
"workers": 8,
"recycle": 500,
"timeout": 60,
"compress": True,
"cpu_affinity": 1,
"save_limit": 250,
"queue_limit": 500,
"label": "Django Q",
"redis": {
"host": REDIS_HOST,
"port": REDIS_PORT,
"db": REDIS_DB,
"password": REDIS_PASSWORD,
},
}
You can play around with the number of workers
- the default is 8
and these are the number of instances that can run in parallel with each other.
We can now actually start running the cluster. Have 2 terminal windows and in 1 have your standard python manage.py runserver
running and in the other run:
python manage.py qcluster
You now have your cluster running.
Step 4: Creating our Task
Create a standard API to make a simple POST
request and let's add the task:
from django_q.tasks import async_task
from rest_framework.views import APIView
import time
...
class SimpleTask(APIView):
def task(self, s = 5):
print("⏳ Running task")
# We could call some long running code here
time.sleep(s)
print("⌛ Task finished")
def post(self, request):
# We can run the task in the background
async_task(self.task, s=5)
return Response({
"message": "Task started"
})
We use async_task
to wrap the task. The code will essentially see this and send it to the redis server (which will send it to the cluster) and continue to the response.
You can place anything in the task
and the response will still be returned in the same amount of time.
Step 5: Deploying to Production
Having this run in a safe local environment is one thing. Deploying it to a production environment is another.
We'll be using Railway (other than them I'd recommend Fly.io or Render - the principles are the same).
We want to add a railway.json
file in our Django App with the following:
{
"$schema": "https://railway.app/railway.schema.json",
"build": {
"builder": "NIXPACKS"
},
"deploy": {
"restartPolicyType": "ON_FAILURE",
"restartPolicyMaxRetries": 10
}
}
In Railway you want to create 1 Redis database, 2 Django Apps. In the first Django App you want the Deploy (Start Command) to be:
python manage.py migrate && gunicorn NAMEOFYOURAPP.wsgi
And in the second:
python manage.py migrate && python manage.py qcluster
You'll want to update the environment variables for both applications to match the REDIS
environment variables (see Step 2), for this I recommend using Railway's shared variables. The end result looks something like:
Think of this mirroring your terminals (the standard Django instance and the new Django qcluster instance)
To verify this is all working, make a request to your SimpleTask
(you can find the URL if you click on the api
app) and take a look at the Observability tab and you should see your background task running 🎉
Top comments (1)
Thanks for sharing!