Message Queues

What is Message Queue?

Message Queue allows application to communicate by sending messages without communicating directly. It stores messages temporarily in a simple format until it has been delivered to the receiving application. This model consists of following components:

1. Message: It is the data that will be send/receive between the applications.
2. Queue: Messages are stored in a queue so that they can be processed in the specific sorted order when the other application is ready to receive them.
3. Producer: Producer is the application that sends the message.
4. Consumer: Consumer is the receiving application.

Why need Message queue?

Let’s understand it with the most common example, a mail delivery mechanism.

1. There is a small business which serves for the customer service, in the beginning they have small volume of mail and it’s easy to check every piece at the door step.
2. As the business increases the customer support increases, hence mail volume grows. Mail delivery will become disruptive so it is a good idea to have an unattended mailbox where the mail can be dropped off.
3. As the business grows over the time, multiple mailboxes becomes the need to handle the volume of mail.
4. Sometime in future it may require to setup a Post Office box to receive all of the inbound mail and distribute the accumulated mail to various mailboxes by the company.
5. So the post office box is the great solution to avoid loosing the mail during the peak time. It will get the mail for the sender and hold it until it is dispatched to the intended Mailbox.
6. Similarly, the message queues are designed to hold the data temporarily when the consumer is busy or it is peak time of traffic and consumer cannot handle the volume. Message Queues offers following benefits to the application architecture.

Reliability

Use of message queues can make application more reliable. During downtime of consumer, Message Queue will hold the data until the consumer is ready to receive the message again. This will increase the reliability and overcome the chances of losing message.

Decoupling

Another advantage of using message queue is decoupling. Producer can send messages without bothering if consumer is available, therefore applications are decoupled.

Scalability

Adding more Mailbox in Mail Delivery example increases the capacity of holding Mail. Similarly, adding more message queues can increase scalability of application during peak time.

There are three types of popular queues (list is not in any specific order):

1. Apache Kafka
2. AWS SQS
3. RabbitMQ

Apache Kafka

Apache Kafka is a distributed data store that’s ideal for processing real-time data which is known as streaming data. Streaming data is a non-stop generation of data from countless number of sources. Kafka will records the streams using a fault-tolerant approach and it offers high throughput to manage real time data transfers.

The major advantage of Kafka is its ability to parallelize topics using partitions. Each partition can be hosted over different machines. Therefore, consumers can access topics in parallel. You can also allow multiple consumers to access multiple partitions which makes scaling even easier. This unique feature significantly increases Kafka’s throughput for processing message queues.

AWS SQS

Amazon Simple Queue Service (SQS) is said to be a fully managed message queuing service. Therefore, using SQS allows to decouple and scale micro-services, distributed systems, and serverless applications. Being a managed service it eliminates the complexity and overhead of the managing and operating message-oriented middleware. The developers can easily focus on business logic and other development tasks.

Amazon SQS offers two types of queues i.e. standard queues and FIFO queues. Both ensure that message reaches a single consumer and offer similar functionality.

RabbitMQ

RabbitMQ supports many messaging techniques including pub-sub, point-to-point, and request-reply. The service is also portable, reliable, and user-friendly, however it’s not as scalable as the other message brokers. The overhead of mirroring and the latency from the central node significantly decreases the throughput of RabbitMQ.

The major advantage of RabbitMQ is the support for complex routing, because of this it is used by many major companies. It is preferred where the message security is crucial or if more control on messages is required. It offers several built-in plugins and also gives the option to use a custom plugin.

Final Words

Every message queues has its own pros and cons, and its selection totally depend on your use case and requirements. For example, companies that do not want to manage MQs and serve more focus on application development will choose AWS SQS. Whereas, Apache Kafka can be a choice where large stream of data is required to be processed in parallel.

Reference & Further Reading

1. Amazon SQS | Message Queuing Service | AWS
2. Amazon SQS vs Apache Kafka — The Iron.io Blog
3. Introduction to message queuing — IBM Documentation
4. Kafka vs RabbitMQ vs AWS SNS/SQS: Which Broker to Choose? | Aspecto