Update: Consumer-Driven Contract Tests: Lessons Learned

Written by Kristine Jetzke

This blog post is part 4 of a series on consumer-driven contract testing. The other parts are: introduction, writing contract tests and integrating them into build pipelines.

In the previous blog posts of this series we gave an introduction to contract testing, showed how to write contract tests using the Pact framework in Spring Boot and how to integrate them into build and deployment pipelines with the help of the Pact Broker.

We will conclude the series in this last part with the most important lessons we have learned.

Lesson 1: Get everyone on board!

Contract testing is a concern that crosses team boundaries, so it's essential to get all the teams involved on the same page as soon as possible. Otherwise you can quickly run into trouble.

Image you have three teams A, B and C, where B and C consume A's API. These three teams agree to do provider-driven contract testing with Spring Cloud Contract. But on the other side of the building, there are three other teams, teams X, Y and Z, where X and Y consume Z's API. They agree to do consumer-driven contract testing with Pact. Everything works fine for those six teams. But what happens if team A needs to start consuming team Z's API? Team A says "We're doing provider-driven contract testing and we're the consumer in this case. We don't need to write any contracts!". Team Z says "We're doing consumer-driven contract testing. We need all our consumers to provide contracts. Otherwise we don't know when we break something."

This an easy recipe for conflict and frustration, and not a situation you want to get into. One team would either need to introduce an additional approach (and technology stack), or it would need to completely switch to the different approach. This would mean, de facto, that all affected teams would also need to switch to the new approach. This would very likely lead to a lot of discussions, extra work and general dissatisfaction.

So our first lesson is: Agree on the general approach and - if possible - even the same technology before introducing contract testing. Spend as much time as needed here. Don't stop the discussions until everyone agrees on the same approach. Whatever you choose, the contract testing framework will be a central part of the whole environment, shared by all teams. If you need to change it later, all teams will be affected.

Lesson 2: Don't underestimate the learning curve

Contract testing is a new kind of testing. Just because you have lots of experience with unit tests, integration tests and all other kinds of tests doesn't mean that you know how to write valuable and maintainable contract tests. You'll quickly run into questions without easy answers:

• How much mocking is good?
• Did we cross the line to functional testing?
• How many variations to include in a contract? E.g. every possible enum value in a POST payload? Every possible value for a query parameter? What if it's numeric?

We were very frustrated after some months because the contract tests had to be updated more frequently than expected, often blocked deployment, and even allowed a breaking change to slip through one time. Over time we learned and developed patterns and best practices (mentioned in part 2 of this blog post series) that helped us write good contract tests. But it took time.

So our second lesson is: Don't underestimate the time it takes for the tests to be stable and to actually add value. Writing the initial tests is very easy, which might mislead you into thinking that you've mastered contract tests. But the real challenge comes in handling how the APIs and contracts change over time.

Lesson 3: You still need to talk

This is hardly surprising, but I still think it makes sense to point it out: contract testing won't replace talking to each other. At least initially, you might even need to communicate more between the teams while you set everything up. Later on, contract testing is a good aid in helping you communicate with each other e.g. for a new API feature the consumer team can just provide the contract instead of documenting the expected format in a ticket. But they still should discuss it beforehand with the provider.

Our third lesson is: Tools don't replace talking to each other, they can just help you do it in a more efficient way.

Lesson 4: Pact is pretty good

We initially used Spring Cloud Contract but didn't like it for various reasons:

• We actually wanted to do consumer-driven contract testing and Spring Cloud Contract is not really made for that. Every time a consumer wanted to add or change a contract, they had to open a pull request in the provider's repository and wait for them to merge it before they could start integrating it. With Pact the consumer can (if they want to) finish their complete implementation before the provider even starts.
• We had several technical issues. One prominent issue was that it was not possible to always load the latest provider stub which you need to run the consumer tests. This issue has been addressed in the meantime but it took at least three different people to open three different GitHub issue over a time span of half a year until the maintainer finally agreed to fix it.
• I personally didn't like that the contracts were written in Groovy files which had to manually be kept in sync with the consumer code. Also, not a lot of options existed for additional setup and assertions on the provider side.

So we moved to Pact and didn't regret it. Some things we especially like about it:

• The Pact API seems quite mature in comparison and you have more flexibility e.g. it's possible to set up provider states. That the Pact files are generated, rather than written by hand in Groovy, is a nice benefit because it automatically ensures that the contracts you generate can be consumed by your own code.
• The documentation is very good, especially regarding underlying concepts and best practices. It's sometimes hard to find it though.
• The community support is quite good as well. A slack channel exists and most of our questions were answered very promptly. Some issues and pull requests were being taken care of immediately, while others took a few weeks.

Our fourth lesson is: There are not a lot of existing contract testing frameworks. Of the ones we investigated, Pact was the best choice. But its main selling point is the Pact Broker. If you don't plan on using the broker, you can still use Pact to write your tests but you just might not need it. You could get similar results by just using json schema or regular unit test frameworks.

Lesson 5: The Pact Broker is awesome BUT…!

The Pact Broker is really awesome because it

• handles the whole verification workflow in a clever way
• supports different deployment models (e.g. continuous deployment, deployment to multiple environments, manual deployment) out-of-the-box
• is easy to set up and maintain
• is easy to integrate into the build and deployment process via webhooks and the CLI

But you don't get this for free.

UPDATE 03/2021: The following section has become somewhat outdated since the time of writing in November 2018. The community around Pact has grown a lot in the last several years. And most importantly: You no longer need to host your own pact broker. You can now use a managed version of the pact broker via pactflow.io.

First of all, the Pact Broker is an additional piece of infrastructure that needs to be understood and maintained. Whether that cost is worth it depends on your organization. In our environment it is definitely worth it because there are lots of teams and services. Services are consumed by multiple other teams and new consumers join on a regular basis. However, in my opinion the broker might be too much overhead if the services managed by it only have a small number of consumers each and are either continuously deployed or deployed at similar intervals. ¹

Second, the pact broker is not 100% bug-free. We found a bug that allowed the can-i-deploy command to return true (i.e. says it's safe to deploy) even though the contract was not verified. This bug cost us time because initially we thought it was our mistake, and spent some time investigating. Filing the bug and describing the scenario took additional time. All in all we already filed more than a handful of bugs and pull requests in the one year we've been using Pact. It's to be expected that software contains bugs but I was still surprised about the nature of some of the bugs we found. I would have expected them to be found earlier (because we're not the first people using pact) or before release (by the person accepting the pull request ).

This brings us to my main concern with the Pact Broker: its bus factor. It seems to be 1 - or at least very close to it. This becomes obvious if you look at the top four Pact Broker committers this year, depicted below in comparison to Spring Boot:

You can see that one person is responsible for almost all commits.

If you need the features that the Pact Broker provides and your only alternative is to develop something in-house yourself, I would still highly recommend Pact. Just be aware of the fact that there is no strong community around the broker at the moment.

Our fifth - and final - lesson is: The Pact Broker is a really great tool but it comes with a trade-off. Be sure that you really need its functionality before introducing it.

That's it for this blog post series! We introduced contract testing, talked in detail about how to implement it and shared our most important lessons. Overall, we'd say that contract testing is an important testing strategy, but not one to be applied blindly.

We're always happy to compare our experience with yours. Do you have any insights, stories or questions? Feel free to get in touch with us.

---

¹ A simpler approach without the broker is to commit the pacts to the consumer repositories and use the @PactUrl to fetch them in the provider. ↩
² Source: https://github.com/pact-foundation/pact_broker/graphs/contributors?from=2018-01-01&to=2018-11-04&type=c
³ Source: https://github.com/spring-projects/spring-boot/graphs/contributors?from=2018-01-01&to=2018-11-01&type=c