Brief intro on Celery

What is celery?

Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation but supports scheduling as well.

If we want to execute some piece of code in python asynchronously or If we want to schedule code to execute in the future. We can use celery.

Consider below scenario:
Suppose, User uploads an excel of 1000 rows to create a database record for each row. It is a time-consuming task. The user needs to wait until the task gets completed. It is not ideal to make the user wait and the browser will send timeout error if the response is taking too long. In this case, we should execute the time-consuming task asynchronously.

How it works?

In this post, I am focusing on the internal structure of celery. Refer to tutorial on how to implement the celery task.

Celery client - Publisher

• When we execute the Task.apply_async()/task.delay() celery internally uses kombu which will serialize the task to JSON like message and send it to the Broker.

{
    "task": "myapp.tasks.Task",
    "id": "54086c5e-6193-4575-8308-dbab76798756",
    "args": [4, 4],
    "kwargs": {}
}

RabbitMQ - Broker

• Celery needs the broker which follows the AMQP(Advanced Message Queuing Protocol) like RabbitMQ, Redis, etc.,
• This broker will take the message and place it in the Queue. Whenever a worker is free (consumer) it gives the task to it.

Celery worker - Consumer

• Celery worker takes the message and extracts the required information like task name, args, kwargs, etc., and executes it.
• If the result backend is present then the task result will store in it.