Written by Alexander Pruss
This blog post is part 1 of a series on consumer-driven contract testing. The other parts are: writing contract tests, integrating them into build pipelines and lessons learned.
Over the past year, we have had the opportunity to introduce consumer-driven contract testing at one of our larger customers, idealo. There were a lot of lessons learned and pitfalls discovered (some avoided, some not) along the way. I decided to write this blog sometime between the fifth and fiftieth time that I thought to myself, "Wow, I wish I'd known that before I started."
This is the first of four blog entries describing our journey with contract testing, both conceptually and with practical examples. The four parts are:
Part 1: Introduction to Consumer-Driven Contract Testing
Part 3: Integrating Contract Tests into Build Pipelines with Pact Broker and Jenkins
Part 4: Lessons Learned
TL;DR: Contract testing will keep the APIs you produce and consume from breaking unexpectedly, without any manual intervention needed. This makes integration testing and developing new features (including breaking features!) much easier.
1.1 - What is contract testing?
On a broad level, contract testing can be thought of as testing (and therefore guaranteeing) the communication layer between services. Contract testing tests that any pair of dependent services can properly send and decode messages between each other, but doesn’t test the services’ internal logic. As such, contract testing exists somewhere on the boundary between integration testing and end-to-end testing.
Let's get some definitions out of the way. There are two main types of actors in contract testing, Providers and Consumers. The Provider is an application responsible for publishing an API; a Consumer of the Provider is another application using (consuming) said API. The Consumers of a Provider will always have basic expectations of the API, such as:
What endpoints can I use?
What input do the endpoints take?
What does the output look like?
Or, more simply,
- How does the API work?
Consumer-driven contract testing is a way to formalize these expectations into a contract between each Consumer-Provider pair. Once this contract is established, contract testing ensures, automatically, that the contract doesn't break unexpectedly.
1.2 - Why contract testing
So what we have so far is that contract testing allows you to define contracts and test them. Brilliant. Before you close this tab, let me offer up some motivation.
Contract testing will make your life easier in four main ways. Contract testing:
Helps Providers make changes without being scared of accidentally breaking their Consumers;
Lets Consumers know that the APIs they consume won't suddenly break;
Allows Consumers to develop against API definitions before the Provider API has actually been developed;
Makes integrating and testing a service in a microservice landscape easier; and
Serves as an efficient communication tool between Provider and Consumer teams.
Or again, more simply,
- Contract testing lets everyone relax and be assured that the APIs won't up and die.
Let’s expound on that a little.
1.3 - Benefits for Providers and Consumers - Relax and be assured that the APIs won’t up and die
Having APIs not just up and die seems to obviously be A Good Thing™. But there are some less obvious benefits that come from preventing errors from unexpected API changes.
It’s easy for API errors to result in teams blocking each other’s progress. If a Provider team pushes a breaking change, new updates from the Consumer team can’t be deployed until that’s fixed. The contrapositive is also annoying, where a Provider team can’t push their change until they’ve made sure each Consumer team is ready for it. Or even more simply, the Provider is afraid to make a change because they think they impact a Consumer somewhere using it, but they aren’t sure how to make sure. Hopefully they figure out something better than to see if Production goes down or not.
Problems like this do more than just directly slow down development. Having a team’s development be blocked by another team’s breaking changes can result in rising tensions. Rising tensions result in worse communication, and worse communication increases the chance of further miscommunicated breaking changes. One way to get out of a negative feedback loop like this is with good management - or, failing that, by preventing these problems in the first place by using contract testing!
Put simply, contract testing lets Provider teams easily know that changes being made are safe, and lets Consumer teams work without worrying about having the rug pulled out from under them. Moreover, if an API change is breaking, contract testing allows Providers to know which of their Consumers are affected and when it’s safe to push the change.
1.4 - Easier integration testing
If you're developing in a microservice landscape, chances are any given one of your services has dependencies on other services. Suppose, for instance, you want to write tests for a Pet Service that speaks to a Cat Service. How do we want to test the parts of our code that depend on responses from the Cat Service?
Well if there’s a Cat Service deployed in a testing environment, you could always write and run integration tests of the Pet Service against this live, running Cat Service. But if you do this, your integration tests are only as stable as your dependencies are. If the testing instance of Cat Service ever acts up, all the impacted Pet Service integration tests will fail, even though there's nothing wrong with the Pet Service itself. Having false positive test results like this is not only incredibly annoying, but also dangerous, as this train developers to ignore failing tests. If the last hundred times your integration tests failed was because of a flaky Cat Service run by a different team, it's easy to ignore the one time your test failure actually indicates a problem.
An alternative would be to instead run your integration tests against mocks or stubs of your external dependencies. This way you’ve cut your testing dependency on Cat Service, and any errors you find are actually your fault. But then you face a new problem; how do you know your mock is actually accurately representing your external dependencies? Suppose Pet Service depends on the POST /meow endpoint, but those flaky Cat Service devs change it to POST /purr without letting you know. Now your tests are returning a false negative! Everything looks great, you deploy your application, and everything falls apart.
The reason this falls apart is that our stubs are making assumptions on the behavior of the Cat Service. If this assumption proves to be incorrect, then the stubs are useless. But if we had a way to formalize our assumptions on the Cat Service, and automatically test that those assumptions are still true, then we’d be able to safely run integration tests against stubs. And that’s exactly what contract testing does.
So contract testing lets you do integration testing against mocks while ensuring that your mocks are actually accurate representations of your external dependencies. Neat.
1.5 - Peace in our time
In addition to all of the above, there are any number of small benefits contract testing brings to the table. Chief among them, in my experience, was that contract testing made inter-team communication much easier and friendlier - easier because it was now clear when the teams had to collaborate on changes, and friendlier because fewer things went wrong!
1.6 - Types of Contract Testing
That’s pretty much it for the big picture of contract testing. Before we start a deep dive into the technical side, though, it’d helpful to briefly consider what different kinds of contract testing exist. There’s a good in-depth dive on Martin Fowler’s website which is worth a look.
In practice, contract testing can be consumer-driven or provider-driven. If it’s consumer-driven, then the Consumers define their expectations and the Provider checks that they’re fulfilled; if it’s provider-driven, then the Provider defines the contract, and Consumers check that they’re compatible with it.
Due to a variety of reasons, I would highly recommend the consumer-driven approach over the provider-driven approach. With consumer-driven contracts:
It’s much easier to properly integrate contract tests into your CI/CD;
You have more beneficial side-effects, such as each Provider being shown how it is used by its Consumers;
It’s easier for Consumers to communicate change requests;
It’s easier for Consumers to develop against requested API changes before they’re actually live;
There’s better tooling.
The two big frameworks we found for helping out with contract testing are Pact (www.pact.io) and Spring Cloud Contract (https://cloud.spring.io/spring-cloud-contract/). Pact is an open-source language-agnostic consumer-driven language-agnostic framework that comes with some very helpful features, including the Pact Broker for orchestrating contract tests. Spring Cloud Contract, on the other hand, began its life as a stub-runner and was then adapted into a provider-driven framework. We would definitely recommend Pact over Spring Cloud Contract.
Contract Testing as a process can be applied to any kind of API. In this blog series, however, we’ll be focusing on HTTP APIs, and on RESTful APIs in particular.
Contract testing brings a lot of benefits but it’s not trivially easy to do properly. Especially the details of how to set up contract testing properly in your CI can get complicated. To find about more about that, stay tuned for the next part of this blog series.
Top comments (0)