Comparing Two RESTful APIs With JMeter

JMeter is commonly used for performance testing, but it can do more than that. In this post, we explore how to compare two RESTful APIs.

Comparing API Results

Sometimes, you might need to compare the results of two RESTful API responses.

To do this, you should first create a Thread Group and add a Simple Logic Controller.

Within the Simple Controller, you can now add the two HTTP requests that you want to compare.
You should have a structure similar to the following:

For simplicity, let’s assume both endpoints return a JSON structure with a unique identifier, and it’s enough to compare if all the identifiers are present and in order.

A JSON extractor can be used to run JSON Path expressions against a JSON response.
A variety of options in the JSON extractor:

• Variable Names: this is a required field and needed for later
• JSON Path expression: again a required field, here we describe the value we’d like to extract
• Match No: you can adjust which values to extract,
  • -1 to extract all
  • 0 for random extraction
  • N (any number) to extract the result at position N
• Compute concatenation var: this will concatenate results by using , as separator

As you can see we will set the Variable name to v1id, we try to extract id's with $..id, match any result by setting matching no. to -1 and concatenate the results.

You should repeat the same step for the second HTTP request, but be aware to change the variable name or you will overwrite the results.

To Compare the two results we can add a JSR223Sampler (groovy sampler) to the Simple Controller

Add the following code to the sampler to compare the results of the JSON extractor

if(vars.get("v1id_ALL") != vars.get("v2id_ALL")){
    log.error("got: "+vars.get("v2id_ALL")+" \n want: "+vars.get("v1id_ALL"))
    SampleResult.setSuccessful(false)
}

the sampler should be last in the simple controller to not be executed before the HTTP requests

Within a View Result Tree Listener you can see the sampler failing if id's don't match. You can also see a error message in the jmeter log file if you want to see the exact result.

But there is more we can compare.
We might be interested in the response time, maybe our new API should be faster than the previous version.
To do this we can add a JSR223Sampler to each HTTP requests with the following code

vars.put("responseTimeV1", String.valueOf(prev.getTime()));

And we can then again compare the two variables in the sampler of the simple controller.

You can do more with JMeter than just the traditional performance testing. You could even add functional tests within the above test plan. This allows you to manage all your tests with one tool, making your test setup hopefully simpler by using less tools.