Never underestimate the power of best practice

I see best practices as special recipes created after doing the same thing over and over again by mixing various ingredients in various ways until you get an outcome easily reproducible (standard) and of superior quality.

Let’s look at the following scenario of an endless loop and options to solve it by applying the right ingredients:

Use case prerequisite

• The message published on the queue is invalid
• The exception thrown by the receiver that picked up the message for processing, is not caught in the catch block

Use case description

1. A requestor publishes an invalid message to a message channel (queue).

2. There is one listener / receiver to this queue, that will get the message.

   a) In case of a valid message, an acknowledgement (ACK) is sent back saying that it is now in safe hands.
   b) In our case the invalid message with incorrect error handling leads to an uncaught exception and a message locked “in-flight (a message awaiting ACK or NACK) until the Default Lock TTL (time-to-live) expires (for example: 2 min)

3. After the 2 minutes (Lock TTL), the invalid message is made visible to other receivers. In our case, the same receiver will pick up the message and resume from 2.b)

4. The poisonous message will be kept on the queue for 7 days (TTL = 7 days) which gives us a nicely hidden infinite loop.

What do you do when you see that the message keeps being released on the queue and the receiver logic gets executed at a speed that only makes you happy when you are looking for fast performing systems? You of course try to solve the puzzle...

As you probably noticed from the above prerequisites there were already two major issues:

• Message validation was not applied before publish to the queue
• Always catch ALL exceptions, no matter what!

To err is human, hence you should always design your solution for failure.

Several suggestions:

• Always use a Dead Letter Queue assigned to your processing queue so that the invalid messages are moved out to the Dead Letter Queue after a given number of retries. This way you at least shorten the life of the infinite loop.
• Setup monitoring for your application / services in relation to the non-functional requirements: do you expect a load on your queue of approx. 10 messages per minute and you end up processing 2000, then most probably this requires your attention
• Review your configuration settings, default is good, but custom case by case might be even better
• Unit test your code, including error handling

Technology used: Anypoint MQ (MuleSoft’s cloud messaging service) and MuleESB