It hurts to say but reactivity is not an easy world, fellows.
Even not being an easy world, the benefits from reactive architecture are worthy. Awesome, life is beautiful again, but take it easy, young boy.
In order to reach the reactivity's holy grail there's only one way, one truth and one queue. Ok, I'm lying, it doesn't have to be one queue and it doesn't have to be a single truth. But there is no other way, as the reactive manifesto claims, there has to have asynchronous messages:
Message Driven: Reactive Systems rely on asynchronous message-passing to establish a boundary between components that ensures loose coupling, isolation and location transparency.
But additional complexity comes due this asynchronous approach. It's not easy to detect, avoid and handle stuff like infinite loops, dead locks and race conditions.
Polyglot flows
I can see, at least, forty two languages here. I dare you to do the same. Depending on the system constrains an IPC protocol may fits better than others. Sometimes, there are premises that exclude some possibilities and limit the options such as network latency, hardware configuration, synchronicity and number or actors involved. Therefore it's not unusual to have more than one IPC protocol, or even several ones, in the same flow with a set of distinct systems and different behavior. This flow, the one with several IPC protocols involved, is also known, at least by me, as polyglot flow.
With all of this being said, new problems raise: how can we test these polyglot flows and their several IPC protocols, asynchronous property and flowing messages? How can we keep track of all of this? The feeling I have is that we are always creating things harder to be tested. Take a breath, is does not have to be that hard. You are not alone.
Mock! Mock them all!
One way to test it is to mock everything! Let's say you have a reactive system. Inside its project, you can write unit tests and component tests, mocking them all and pretend that the message is really flowing back and forth from/to where it
should come/go. But, at some point, you have make your test scope higher. At some point, pretending is not enough, you have to make sure that the message is properly flowing by. You have to check whether the configuration is properly set and you have to check the result in your database, file system, broker or something similar. Furthermore, to write this kind of tests, you got to have some programming skills, what may move away non-coders from testing it. If we think this through, anyone who understands how the flow works should be able to test it.
Give me tools!
Given that to integrate different systems, sources/destinations and formats is indispensable, we can get as conclusion that we need to take this seriously, use automatic tools to make this job for us and minimize human error as much as we can. Some say that one of the reasons of SOA (Service-Oriented Architecture) fall was exactly the lack of mature tools. But it was some years ago, a lot of time has passed and the community has learned with it's previous mistake. Has it?
Nowadays, we can find several awesome tools in the community, some of them, the most known ones, will be mentioned:
Postman
Postman is a really good application. There's a good reason for being proudly announced that it's used by 5 million developers and more than 100,000 companies to access 130 million APIs every month and in every earth in our Solar system in our Galaxy! It's a lot, don't you think? It doesn't take that effort to think why such good numbers: it's really easy to use it, including by non-coders, all you have to do is to click in two or three buttons and voilà, the magic happens! It's easily added to your CI pipeline and provides you nice features like running several requisitions sequentially one after another.
But (there is always a but), despite of its deeply HTTP universe exploration it limits itself to this. Even though HTTP based REST is the largely most adopted IPC in microservices world, reactive systems are not about synchronous communication, as mentioned in forget HTTP in microservices, in is REST the better option?, in Data is no longer at REST, REST is the new SOAP, and in REST turns microservices back to monoliths. I hope you're convinced by this time. It's exactly the opposite, such communications deteriorate the whole system because they make your system more coupled. Therefore, Postman loses a lot of points for not being able to handle several IPC protocols. I believe that, and this is completely personal, that not strongly emphasizing its tests functionalities makes it lose some points as well.
Mountebank
This one is able to handle more IPC protocols other than HTTP. To be exact it can work with: SMTP, TCP and HTTP/HTTPS. This last one receives a especial focus, of course. Given that it's able to understand TCP flows, one may work with every other TCP based protocol, but one would have to build some layers on top of it to make it work properly. Mountebank real power is its capacity to expose a HTTP API and to provide stubs and proxies to tests executions. Through imposters, it's possible to simulate HTTP responses, to parse messages and use predicates and assertions to make sure that the message is correct.
But (I said there was always a but), being able to handle raw TCP message is not enough, it's still up to you to understand how the intended protocol works and build it over TCP. Come on, is it serious? It's not tempting for non-coders. As it acts only under demand it has to be used ensemble with other testing tool, one that is able to trigger the message and start the test.
Apache Camel
It handles several several several IPC protocols. Uhay! Amazing! Finally. And more: it triggers the message, you don't have to, necessarily, use it with other testing tools. Since you control it by coding, you can do whatever you want to: nest tests, combine executions, test several systems at one time…
However (you know, it's just a fancy but), it's not tempting for non-coders. Actually, it's not tempting for coders either. Given that you have to code you test scenario, it requires deep programming knowledge. As I said before, everyone who understand how the flow flows should be able to test it. These tests should not (and cannot) be a responsibility exclusive to the coder. Besides that, as mentioned in breaking the camel's back:
Most of Apache Camel's connectors are implemented in a synchronous and/or blocking fashion, leading to inefficient use of resources and limited scalability due to high contention for shared resources.
Also, fan-in/fan-out patterns tend to be either poorly supported, requiring knowing all inputs/outputs at creation; or, lowest-common-denominator - limiting throughput to the slowest producer/consumer.
We need better tools!
There is an obvious need for better tooling. What can be done given what is known for reactive architectures: asynchronous, responsive, resilient and elastic?
Let me introduce one thing to you all, folks.
Enqueuer to the rescue!
As incredible as it sounds, there was no testing tool able:
to assert that whenever one thing happens other things happen in a suitable time and format. This is where the reactivity lies on;
to be used by non-coders;
to test isolated application and whole systems at one time;
to trigger events by itself;
to mock responses;
to chain and to nest events;
to natively support several well known IPC protocols. To name a few: amqp, mqtt, http, sqs, 0mq, Kafka, TCP etc; and
to be easily added to a CI pipeline.
This problem no longer exists, fellas. Enqueuer \nqr\ is a CLI framework, open source born to fulfill our need! It simulates one requisition and inspects how the system reacts by analyzing its outputs messages. It's a mix of the before mentioned tools.
Show me how to do it
In order to illustrate how enqueuer works and how to use it, I will use a fictitious example with a polyglot flow:
When your e-commerce http endpoint is hit, you have to send an information to a credit card processing RESTful API and notify users about this through MQTT events.
Now, you have three options to test this polyglot flow:
Shamelessly write no test at all; (It happens more often than we would like to admit. You know)
Write a component test for each one of these cases separately in the codebase itself, mock them all, handle new dependencies, figure out details and deal with debugging them when they fail; or
Use enqueuer and have it all tested right out of the box.
By unanimity, you got to choose number 3. Given that we want to test stuff, we have to isolated the test target, be it a single component or a set of systems, by doing this, we make sure that whenever an error occurs we'll be certain that the target is the one to blame.
After isolating it, we'll trigger the event firing a hit in the exposed endpoint and evaluate if every expected output is fulfilled. The good news is that this is what Enqueuer does. So, start by installing it:
$npm install -g enqueuer
Create a configuration and a file to describe the test scenario file like these:
There are a few things that deserves special attention here. Looking at the test description file we can see that enqueuer will hit a (POST) in http://e-commerce.com:8080/endpoint. e-commerce.com:8080 is where the system under test is supposed to be running. There are two expected subscriptions. The first one is a http-server. So enqueuer will run a HTTP server and listen to a /output1 call. Once this endpoint gets hit, enqueuer asserts that it got hit and response mocking stuff back. As you can see in response: {status: 123…
The second subscription claims that there has to be a message sent to output2 MQTT topic. Even thought we can not guarantee it, and it's a good thing not being able to do so, this message is probably sent by the system under test. In the field onMessageReceived we pass javascript code value to assert something. In the example, we are creating a UserNotified test and assigning a true value to it. Of course, we can make deeper analysis there. Given that it is a javascript code snippet, you can do whatever your imagination want to - Since that this flow is about HTTP and MQTT IPC protocols, these are the ones that the description file is worried about, but enqueuer is able to handle other protocols as well. Currently all of the following are supported: Amqp , File, Http, Kafka, Mqtt, Sqs, ZeroMq, Stomp, Uds, Tcp, Udp etc.
Now we are able to run it and see its report:
$enqueuer --config-file enqueuer.yml
Through the report we see that 10 tests were ran. Some of them were automatically created by enqueuer, such as eCommerceExample.startEvent.publisher.tests.Published and some of them were created by us, such as eCommerceExample.subscriptions[1].tests.User Notified. All of them are passing. There are other interesting things: execution time, beginning time, end time and these kinds of things. I'm sure you got it.
Conclusion
The most knows tools may not be enough to test polyglot flows. Enqueuer is here to supply this lack of responsibility. As we've seen, it's possible to test properly the whole flow and several IPC protocols with no deep programming skill.
In a near future, other examples will be published and how to test them with enqueuer.
Top comments (0)