This PoC shows a step by step implementation of contract testing using Pact.
First of all a couple of definitions:
Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a "contract".
Consumer driven contract testing is a type of contract testing that ensures that a provider is compatible with the expectations that the consumer has of it. For an HTTP API (and other synchronous protocols), this would involve checking that the provider accepts the expected requests, and that it returns the expected responses. For a system that uses message queues, this would involve checking that the provider generates the expected message.
So let's try to implement this flow:
rogervinas / contract-testing-with-pact
🤝 Contract Testing with Pact
- 1) Consumer defines the "contract" with the Provider
- 2) Consumer tests the "contract" using a provider mock
- 3) Consumer publishes the "contract"
- 4) Provider tests the "contract" using a consumer mock
- 5) Provider verifies or refutes the "contract"
- 6+7) Deploy only when the "contract" is verified
- Implementation Details
- Want to know more?
1) Consumer defines the "contract" with the Provider
For the "Sample API Client" we will use Kotlin and Ktor client.
To define POST /thing
endpoint we specify the following "pact":
@ExtendWith(PactConsumerTestExt::class)
@TestInstance(PER_CLASS)
class SampleApiClientContractTest {
@Pact(
provider = "Sample API Server",
consumer = "Sample API Client"
)
fun create(builder: PactDslWithProvider): V4Pact {
return builder
.given("Initial State")
.uponReceiving("Create Thing")
.path("/thing")
.method("POST")
.headers(mapOf("Content-Type" to "application/json"))
.body(
"""
{
"name": "Foo",
"value": 123.45,
"date": "2022-10-13"
}
""".trimIndent()
)
.willRespondWith()
.status(201)
.headers(mapOf("Content-Type" to "application/json"))
.body(
"""
{
"id": 123
}
""".trimIndent()
)
.toPact(V4Pact::class.java)
}
}
Note that:
- We use just a junit5 test with
PactConsumerTestExt
extension. - We are defining the "contract" and not defining test methods yet. We will create them in next step.
- In this first example we are using fixed JSON expectations, but we can use also PactDslJsonBody DSL which allows us to specify regex and type matchers to each field:
// The request can be specified as:
.body(
PactDslJsonBody()
.stringMatcher("name", "\\w+", "Foo")
.decimalType("value", 123.45)
.localDate("date", "yyyy-MM-dd", LocalDate.of(2022, 10, 13))
)
// And the response:
.body(
PactDslJsonBody()
.integerType("id", 123)
)
To define GET /thing/{id}
endpoint we will specify two "pacts":
- One for the case "thing exists":
@Pact(
provider = "Sample API Server",
consumer = "Sample API Client"
)
fun getExistingThing(builder: PactDslWithProvider): V4Pact {
return builder
.given("Thing 123 exists")
.uponReceiving("Get Thing 123 when it exists")
.path("/thing/123")
.method("GET")
.willRespondWith()
.status(200)
.headers(mapOf("Content-Type" to "application/json"))
.body(
"""
{
"name": "Foo",
"value": 123.45,
"date": "2022-10-13"
}
""".trimIndent()
)
.toPact(V4Pact::class.java)
}
- Another one for the case "thing does not exist":
@Pact(
provider = "Sample API Server",
consumer = "Sample API Client"
)
fun getNonExistingThing(builder: PactDslWithProvider): V4Pact {
return builder
.given("Initial State")
.uponReceiving("Get Thing 123 when it does not exist")
.path("/thing/123")
.method("GET")
.willRespondWith()
.status(404)
.toPact(V4Pact::class.java)
}
2) Consumer tests the "contract" using a provider mock
If we want to focus on "test first" we need an empty implementation of the client just to make it compile:
data class SampleThing(
val name: String,
val value: Double,
@JsonFormat(pattern = "yyyy-MM-dd") val date: LocalDate
)
data class SampleThingId(
val id: Int
)
interface SampleApiClient {
suspend fun create(thing: SampleThing): SampleThingId?
suspend fun get(thingId: SampleThingId): SampleThing?
}
class SampleApiKtorClient(
private val serverUrl: String
) : SampleApiClient {
override suspend fun create(thing: SampleThing)
: SampleThingId? {
TODO("Not yet implemented")
}
override suspend fun get(thingId: SampleThingId)
: SampleThing? {
TODO("Not yet implemented")
}
}
Now we create tests for the three "pacts" defined in the previous step:
@Test
@PactTestFor(
providerName = "Sample API Server",
pactMethod = "create",
providerType = SYNCH
)
fun `should create thing`(mockServer: MockServer) {
val client = SampleApiKtorClient(mockServer.getUrl())
val thingId = runBlocking {
client.create(
SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
)
}
assertThat(thingId)
.isEqualTo(SampleThingId(123))
}
@Test
@PactTestFor(
providerName = "Sample API Server",
pactMethod = "getExistingThing",
providerType = SYNCH
)
fun `should get thing 123 when it exists`(
mockServer: MockServer
) {
val client = SampleApiKtorClient(mockServer.getUrl())
val thing = runBlocking {
client.get(SampleThingId(123))
}
assertThat(thing).isEqualTo(
SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
)
}
@Test
@PactTestFor(
providerName = "Sample API Server",
pactMethod = "getNonExistingThing",
providerType = SYNCH
)
fun `should not get thing 123 when it does not exist`(mockServer: MockServer) {
val client = SampleApiKtorClient(mockServer.getUrl())
val thing = runBlocking {
client.get(SampleThingId(123))
}
assertThat(thing).isNull()
}
Note that:
- In
@PactTestFor
annotationpactMethod
should match the name of the method annotated with@Pact
. - We pass to the client the
MockServer
'surl
. This MockServer will mock the provider based on the "contract", no need for us to configure it! ✨magic✨ - Just for documentation, we specify the provider as a synchronous provider (HTTP)
Once we have a final implementation of the client wrapping a Ktor client ...
class SampleApiKtorClient(
private val serverUrl: String
) : SampleApiClient {
private val client = HttpClient(CIO) {
install(ContentNegotiation) {
jackson {
registerModule(JavaTimeModule())
}
}
}
override suspend fun create(thing: SampleThing)
: SampleThingId? {
val response = client.post("$serverUrl/thing") {
contentType(ContentType.Application.Json)
setBody(thing)
}
return when(response.status) {
HttpStatusCode.Created -> response.body<SampleThingId>()
else -> null
}
}
override suspend fun get(thingId: SampleThingId)
: SampleThing? {
val response = client.get("$serverUrl/thing/${thingId.id}")
return when(response.status) {
HttpStatusCode.OK -> response.body<SampleThing>()
else -> null
}
}
}
... if we execute tests on SampleApiClientContractTest:
- Tests will be executed against a provider mock.
- The "contract" will be saved locally as
build/pacts/build/pacts/Sample API Client-Sample API Server.json
. We can generate it in another directory using@PactDirectory
annotation orpact.rootDir
system property.
3) Consumer publishes the "contract"
The consumer "contract" is now saved locally as a JSON file, but it should be published to a PactBroker, so it can be shared with the provider.
To test it locally:
- Start a local instance of PactBroker with a sqlite database:
docker compose up -d
Go to http://localhost:9292, you will see a "contract" example that comes by default:
Publish the consumer "contract" using
pactPublish
gradle task:
cd ./sample-api-client
./gradlew pactPublish
> Task :pactPublish
Publishing 'Sample API Client-Sample API Server.json' ...
OK
- Go back to http://localhost:9292, you will see the consumer "contract":
4) Provider tests the "contract" using a consumer mock
For the "Sample API Server" we will use Kotlin and Spring Boot with WebFlux.
Here we have two options to test the "contract":
- Test it against only the API layer using a WebFluxTest.
- Test it against the whole Application using a SpringBootTest.
Using @WebFluxTest
We start with this:
@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactFolder("../sample-api-client/build/pacts")
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {
@Autowired
private lateinit var webTestClient: WebTestClient
@BeforeEach
fun beforeEach(context: PactVerificationContext) {
context.target = WebTestClientSpring6Target(webTestClient)
}
@TestTemplate
fun pactVerificationTestTemplate(
context: PactVerificationContext
) {
context.verifyInteraction()
}
@State("Initial State")
fun `initial state`() {
// TODO
}
@State("Thing 123 exists")
fun `thing 123 exists`() {
// TODO
}
}
Note that:
-
@WebFluxTest
is a standard Spring Boot "slice test" where only components needed to testSampleApiController
will be started. - A
WebTestClient
is the standard way to test controllers on a@WebFluxTest
, but in this case we will not use it directly, we will just pass it to thePactVerificationContext
. - Just temporarily we use
@PactFolder
annotation to read the "contract" from the local directory wheresample-api-client
has generated it. No need for a PactBroker yet. - We use
@Provider
annotation to specify that we are executing tests for the "Sample API Server" provider. - We have to create as many methods annotated with
@State
as states the "contract" expects. We leave them empty for now but we will have to properly set the state there. - Finally
PactVerificationSpring6Provider
junit5 extension andpactVerificationTestTemplate
method annotated with junit5's@TestTemplate
will create tests dynamically following the "contract". Again ✨magic✨
If we create an empty SampleApiController
to make this test compile:
@RestController
class SampleApiController
And then we try to execute the test:
cd sample-api-server
./gradlew test
> Task :test
Create Thing - FAILED
1) Verifying a pact between Sample API Client and Sample API Server - Create Thing: has status code 201
1.1) status: expected status of 201 but was 404
1.2) body: $ Actual map is missing the following keys: id
Get Thing 123 when it does not exist - PASSED
Get Thing 123 when it exists - FAILED
1) Verifying a pact between Sample API Client and Sample API Server - Get Thing 123 when it exists: has status code 200
1.1) status: expected status of 200 but was 404
1.2) body: $ Actual map is missing the following keys: date, name, value
Turns out that with an empty controller we pass one of the tests 😜
Once we have a final implementation of the controller ...
@RestController
class SampleApiController(
private val repository: SampleRepository
) {
@PostMapping(
"/thing",
consumes = [APPLICATION_JSON_VALUE],
produces = [APPLICATION_JSON_VALUE]
)
@ResponseStatus(CREATED)
suspend fun create(@RequestBody thing: SampleThing)
= repository.save(thing)
@GetMapping(
"/thing/{id}",
produces = [APPLICATION_JSON_VALUE]
)
suspend fun get(@PathVariable("id") id: Int) =
when (val thing = repository.get(SampleThingId(id))) {
null -> ResponseEntity.notFound().build()
else -> ResponseEntity.ok(thing)
}
}
... we can implement the contract test as:
@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactFolder("../sample-api-client/build/pacts")
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {
@Autowired
private lateinit var webTestClient: WebTestClient
@MockkBean
private lateinit var repository: SampleRepository
@BeforeEach
fun beforeEach(context: PactVerificationContext) {
context.target = WebTestClientSpring6Target(webTestClient)
}
@TestTemplate
fun pactVerificationTestTemplate(
context: PactVerificationContext
) {
context.verifyInteraction()
}
@State("Initial State")
fun `initial state`() {
every { repository.save(any()) } returns SampleThingId(123)
every { repository.get(any()) } returns null
}
@State("Thing 123 exists")
fun `thing 123 exists`() {
every {
repository.get(SampleThingId(123))
} returns SampleThing(
"Foo", 123.45, LocalDate.of(2022, 10, 13)
)
}
}
Note that:
- We mock a
SampleRepository
becauseSampleApiController
needs one. - In the state "Initial State":
-
SampleRepository
mock will returnnull
wheneverget
is called with anySampleThingId
because it is supposed to be empty. -
SampleRepository
mock will returnSampleThingId(123)
wheneversave
is called with anySampleThing
to simulate saving it.
-
- In the state "Thing 123 exists":
-
SampleRepository
mock will returnnull
wheneverget
is called withSampleThingId(123)
because is supposed to be stored there.
-
Now if we execute the test everything should be 🟩
Finally, in a real scenario we will use @PactBroker
instead of @PactFolder
in order to retrieve the "contract" from a PactBroker:
@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {
// ...
}
Note that:
-
PactBroker url can be set directly on the
@PactBroker(url=xxx)
annotation or via thepactbroker.url
system property.
You can review the final implementation in SampleApiControllerContractTest.
Using @SpringBootTest
We can also test the "contract" against the whole application using a SpringBootTest:
@SpringBootTest(webEnvironment = DEFINED_PORT)
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiServerContractTest {
@TestTemplate
fun pactVerificationTestTemplate(
context: PactVerificationContext
) {
context.verifyInteraction()
}
@State("Initial State")
fun `initial state`() {
// TODO set "Initial State" state
}
@State("Thing 123 exists")
fun `thing 123 exists`() {
// TODO set "Thing 123 exists" state
}
}
And a little extra code if we want to start the application using a random port:
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiServerContractTest {
@LocalServerPort
private var port = 0
@BeforeEach
fun beforeEach(context: PactVerificationContext) {
context.target = HttpTestTarget("localhost", port)
}
// ...
}
Finally, we only need to set the expected state using the @State
annotated methods. How we do that will be different on each case. For example if we are using a real database that would mean to insert/delete rows from a table, etc.
Just for this PoC we can cheat a little and implement an in-mem SampleRepository
this way with a convenient reset
method:
@Repository
class SampleRepository {
private val sampleThingIdNext = AtomicInteger(1)
private val sampleThings
= mutableMapOf<SampleThingId, SampleThing>()
fun save(thing: SampleThing): SampleThingId {
val thingId = SampleThingId(
sampleThingIdNext.getAndIncrement()
)
sampleThings[thingId] = thing
return thingId
}
fun get(thingId: SampleThingId) = sampleThings[thingId]
fun reset(nextId: Int) {
sampleThingIdNext.set(nextId)
sampleThings.clear()
}
}
So we can implement the state methods this way:
@Autowired
private lateinit var repository: SampleRepository
@State("Initial State")
fun `initial state`() {
// clear repository and set next id = 123
repository.reset(123)
}
@State("Thing 123 exists")
fun `thing 123 exists`() {
// clear repository and set next id = 123
repository.reset(123)
// save a thing that will get id = 123
repository.save(
SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
)
}
You can review the final implementation in SampleApiServerContractTest.
5) Provider verifies or refutes the "contract"
To publish the result of the contract tests automatically to a PactBroker we need to:
- Use
@PactBroker
annotation. - Set PactBroker url directly on
@PactBroker(url=xxx)
annotation or viapactbroker.url
system property. - Set system property
pact.verifier.publishResults=true
. - Set system property
pact.provider.version
(we can use same version as the gradle project).
To test it locally:
- Start a local instance of PactBroker with a sqlite database:
docker compose up -d
- Publish the consumer "contract" using pact gradle plugin:
cd ./sample-api-client
./gradlew pactPublish
- Make one of the provider tests fail, for example commenting these lines on the SampleApiControllerContractTest:
@State("Thing 123 exists")
fun `thing 123 exists`() {
//every {
// repository.get(SampleThingId(123))
//} returns SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
}
- Execute provider SampleApiControllerContractTest (from IDE or command line):
cd ./sample-api-server
./gradlew test --tests '*SampleApiControllerContractTest'
Go to http://localhost:9292, and you will see the "contract" not verified (in red):
Fix the provider test and execute it again.
Go back to http://localhost:9292 and you will see the "contract" verified (in green):
6+7) Deploy only when the "contract" has been verified
Both for the consumer and the provider we can execute canIDeploy
gradle task that will check if the "contract" has been verified in PactBroker, failing if not:
- No contract published ❌
./gradlew canIDeploy
> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯ Request to path '/matrix/...'
failed with response 400
- Provider has neither verified nor refuted the contract ❌
./gradlew canIDeploy
> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯
There is no verified pact between version 1.0 of
Sample API Client and the latest version of
Sample API Server (no such version exists)
- Provider has refuted the contract ❌
./gradlew canIDeploy
> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯
The verification for the pact between version 1.0 of
Sample API Client and the latest version of
Sample API Server (1.0) failed
- Provider has verified the contract ✅
./gradlew canIDeploy
> Task :canIDeploy
Computer says yes \o/
All required verification results are published and successful
Implementation Details
Some, I hope useful, implementation details of this PoC:
- We use au.com.dius.pact gradle plugin both for the consumer and the provider.
- We use au.com.dius.pact.consumer:junit5 dependency for the consumer.
- We use au.com.dius.pact.provider:spring6 dependency for the provider.
- All these system properties are available.
- Properties used in this PoC for the consumer:
-
project.extra["pacticipant"] = "Sample API Client"
andproject.extra["pacticipantVersion"] = version
so we do not need to pass them everytime in thecanIDeploy
task. - Test tasks'
systemProperties["pact.writer.overwrite"] = true
so the contract is always overwritten. -
project.extra["pactbroker.url"] = project.properties["pactbroker.url"] ?: "http://localhost:9292"
so: - We can override it if needed (using
./gradlew -Ppactbroker.url=http://xxx
). - We can use the same in the
publish
andbroker
configuration sections.
-
- Properties used in this PoC for the provider:
-
project.extra["pacticipant"] = "Sample API Server"
andproject.extra["pacticipantVersion"] = version
so we do not need to pass them everytime in thecanIDeploy
task. -
project.extra["pactbroker.url"] = project.properties["pactbroker.url"] ?: "http://localhost:9292"
so: - We can override it if needed (using
./gradlew -Ppactbroker.url=http://xxx
). - We can use it in the
broker
configuration section. - We can use it in test tasks'
systemProperties["pactbroker.url"]
used by@PactBroker
annotation as default. - Test tasks':
-
systemProperties["pact.provider.version"] = version
to specify the provider version (it does not get it automatically from the gradle project, like it does for the consumer 🤷). -
systemProperties["pact.verifier.publishResults"] = "true"
to always publish results back to the PactBroker (in a real example we would disable it locally and enable only in CI though).
-
ALso Github Actions CI is enabled for this repo and executes a complete flow (you can execute it locally too):
- Start PactBroker
- Will run
docker compose up -d
to start a local PactBroker.
- Will run
- Sample API Client check
- Will run
./gradlew check
to execute tests and generate the "contract" locally.
- Will run
- Sample API Client Pact publish
- Will run
./gradlew pactPublish
to publish the "contract" to the local PactBroker.
- Will run
- Sample API Server check
- Will run
./gradlew check
that: - Will download the "contract" from the local PactBroker.
- Will execute tests accordingly.
- Will publish the result of the tests back to the local PactBroker.
- Will run
- Sample API Client can deploy?
- Will run
./gradlew canIDeploy
that will connect to the local PactBroker and be successful if provider has verified the "contract".
- Will run
- Sample API Server can deploy?
- Will run
./gradlew canIDeploy
that will connect to the local PactBroker and be successful if provider has verified the "contract".
- Will run
- Stop PactBroker
- Will run
docker compose down
to stop the local PactBroker.
- Will run
Want to know more?
Some interesting documentation at Pact, including:
-
Pact Workshops - from 0 to Pact in ~2 hours - hands-on labs for:
- Ruby
- JS
- Golang
- JVM
- Android
- .NET
- CI/CD
-
Pact Broker Docs - including:
- Pact Broker Client CLI
- Pact Broker Webhooks to trigger builds every time a pact is changed, published or verified.
That's all! Happy coding! 💙
Top comments (0)