Roger Viñas Alcon

Posted on Oct 21, 2022 • Edited on Dec 7, 2023

🤝 Contract Testing with Pact

#contracttesting #pact #kotlin #showdev

This PoC shows a step by step implementation of contract testing using Pact.

First of all a couple of definitions:

Contract testing is a technique for testing an integration point by checking each application in isolation to ensure the messages it sends or receives conform to a shared understanding that is documented in a "contract".

Consumer driven contract testing is a type of contract testing that ensures that a provider is compatible with the expectations that the consumer has of it. For an HTTP API (and other synchronous protocols), this would involve checking that the provider accepts the expected requests, and that it returns the expected responses. For a system that uses message queues, this would involve checking that the provider generates the expected message.

So let's try to implement this flow:

rogervinas / contract-testing-with-pact

🤝 Contract Testing with Pact

1) Consumer defines the "contract" with the Provider
2) Consumer tests the "contract" using a provider mock
3) Consumer publishes the "contract"
4) Provider tests the "contract" using a consumer mock
- Using @WebFluxTest
- Using @SpringBootTest
5) Provider verifies or refutes the "contract"
6+7) Deploy only when the "contract" is verified
Implementation Details
Want to know more?

1) Consumer defines the "contract" with the Provider

For the "Sample API Client" we will use Kotlin and Ktor client.

To define POST /thing endpoint we specify the following "pact":

@ExtendWith(PactConsumerTestExt::class)
@TestInstance(PER_CLASS)
class SampleApiClientContractTest {
  @Pact(
    provider = "Sample API Server", 
    consumer = "Sample API Client"
  )
  fun create(builder: PactDslWithProvider): V4Pact {
    return builder
      .given("Initial State")
      .uponReceiving("Create Thing")
      .path("/thing")
      .method("POST")
      .headers(mapOf("Content-Type" to "application/json"))
      .body(
        """
        {
          "name": "Foo",
          "value": 123.45,
          "date": "2022-10-13"
        }
        """.trimIndent()
      )
      .willRespondWith()
      .status(201)
      .headers(mapOf("Content-Type" to "application/json"))
      .body(
        """
        {
          "id": 123
        }
        """.trimIndent()
      )
      .toPact(V4Pact::class.java)
  }
}

Note that:

We use just a junit5 test with PactConsumerTestExt extension.
We are defining the "contract" and not defining test methods yet. We will create them in next step.
In this first example we are using fixed JSON expectations, but we can use also PactDslJsonBody DSL which allows us to specify regex and type matchers to each field:

// The request can be specified as:
.body(
  PactDslJsonBody()
    .stringMatcher("name", "\\w+", "Foo")
    .decimalType("value", 123.45)
    .localDate("date", "yyyy-MM-dd", LocalDate.of(2022, 10, 13))
)

// And the response:
.body(
  PactDslJsonBody()
    .integerType("id", 123)
)

To define GET /thing/{id} endpoint we will specify two "pacts":

One for the case "thing exists":

@Pact(
  provider = "Sample API Server", 
  consumer = "Sample API Client"
)
fun getExistingThing(builder: PactDslWithProvider): V4Pact {
return builder
  .given("Thing 123 exists")
  .uponReceiving("Get Thing 123 when it exists")
  .path("/thing/123")
  .method("GET")
  .willRespondWith()
  .status(200)
  .headers(mapOf("Content-Type" to "application/json"))
  .body(
    """
      {
        "name": "Foo",
        "value": 123.45,
        "date": "2022-10-13"
      }
      """.trimIndent()
  )
  .toPact(V4Pact::class.java)
}

Another one for the case "thing does not exist":

@Pact(
  provider = "Sample API Server",
  consumer = "Sample API Client"
)
fun getNonExistingThing(builder: PactDslWithProvider): V4Pact {
return builder
  .given("Initial State")
  .uponReceiving("Get Thing 123 when it does not exist")
  .path("/thing/123")
  .method("GET")
  .willRespondWith()
  .status(404)
  .toPact(V4Pact::class.java)
}

2) Consumer tests the "contract" using a provider mock

If we want to focus on "test first" we need an empty implementation of the client just to make it compile:

data class SampleThing(
  val name: String,
  val value: Double,
  @JsonFormat(pattern = "yyyy-MM-dd") val date: LocalDate
)

data class SampleThingId(
  val id: Int
)

interface SampleApiClient {
  suspend fun create(thing: SampleThing): SampleThingId?
  suspend fun get(thingId: SampleThingId): SampleThing?
}

class SampleApiKtorClient(
  private val serverUrl: String
) : SampleApiClient {

  override suspend fun create(thing: SampleThing)
    : SampleThingId? {
    TODO("Not yet implemented")
  }

  override suspend fun get(thingId: SampleThingId)
    : SampleThing? {
    TODO("Not yet implemented")
  }
}

Now we create tests for the three "pacts" defined in the previous step:

@Test
@PactTestFor(
  providerName = "Sample API Server", 
  pactMethod = "create", 
  providerType = SYNCH
)
fun `should create thing`(mockServer: MockServer) {
  val client = SampleApiKtorClient(mockServer.getUrl())
  val thingId = runBlocking {
    client.create(
      SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
    )
  }
  assertThat(thingId)
    .isEqualTo(SampleThingId(123))
}

@Test
@PactTestFor(
  providerName = "Sample API Server", 
  pactMethod = "getExistingThing", 
  providerType = SYNCH
)
fun `should get thing 123 when it exists`(
  mockServer: MockServer
) {
  val client = SampleApiKtorClient(mockServer.getUrl())
  val thing = runBlocking { 
    client.get(SampleThingId(123)) 
  }
  assertThat(thing).isEqualTo(
    SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
  )
}

@Test
@PactTestFor(
  providerName = "Sample API Server", 
  pactMethod = "getNonExistingThing", 
  providerType = SYNCH
)
fun `should not get thing 123 when it does not exist`(mockServer: MockServer) {
  val client = SampleApiKtorClient(mockServer.getUrl())
  val thing = runBlocking {
    client.get(SampleThingId(123))
  }
  assertThat(thing).isNull()
}

Note that:

In @PactTestFor annotation pactMethod should match the name of the method annotated with @Pact.
We pass to the client the MockServer's url. This MockServer will mock the provider based on the "contract", no need for us to configure it! ✨magic✨
Just for documentation, we specify the provider as a synchronous provider (HTTP)

Once we have a final implementation of the client wrapping a Ktor client ...

class SampleApiKtorClient(
  private val serverUrl: String
) : SampleApiClient {

  private val client = HttpClient(CIO) {
    install(ContentNegotiation) {
      jackson {
        registerModule(JavaTimeModule())
      }
    }
  }

  override suspend fun create(thing: SampleThing)
    : SampleThingId? {
    val response = client.post("$serverUrl/thing") {
      contentType(ContentType.Application.Json)
      setBody(thing)
    }
    return when(response.status) {
      HttpStatusCode.Created -> response.body<SampleThingId>()
      else -> null
    }
  }

  override suspend fun get(thingId: SampleThingId)
    : SampleThing? {
    val response = client.get("$serverUrl/thing/${thingId.id}")
    return when(response.status) {
      HttpStatusCode.OK -> response.body<SampleThing>()
      else -> null
    }
  }
}

... if we execute tests on SampleApiClientContractTest:

Tests will be executed against a provider mock.
The "contract" will be saved locally as build/pacts/build/pacts/Sample API Client-Sample API Server.json. We can generate it in another directory using @PactDirectory annotation or pact.rootDir system property.

3) Consumer publishes the "contract"

The consumer "contract" is now saved locally as a JSON file, but it should be published to a PactBroker, so it can be shared with the provider.

To test it locally:

Start a local instance of PactBroker with a sqlite database:

docker compose up -d

Go to http://localhost:9292, you will see a "contract" example that comes by default:
Publish the consumer "contract" using pactPublish gradle task:

cd ./sample-api-client
./gradlew pactPublish

> Task :pactPublish
Publishing 'Sample API Client-Sample API Server.json' ... 
OK

Go back to http://localhost:9292, you will see the consumer "contract":

4) Provider tests the "contract" using a consumer mock

For the "Sample API Server" we will use Kotlin and Spring Boot with WebFlux.

Here we have two options to test the "contract":

Test it against only the API layer using a WebFluxTest.
Test it against the whole Application using a SpringBootTest.

Using @WebFluxTest

We start with this:

@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactFolder("../sample-api-client/build/pacts")
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {

  @Autowired
  private lateinit var webTestClient: WebTestClient

  @BeforeEach
  fun beforeEach(context: PactVerificationContext) {
    context.target = WebTestClientSpring6Target(webTestClient)
  }

  @TestTemplate
  fun pactVerificationTestTemplate(
    context: PactVerificationContext
  ) {
    context.verifyInteraction()
  }

  @State("Initial State")
  fun `initial state`() {
      // TODO
  }

  @State("Thing 123 exists")
  fun `thing 123 exists`() {
      // TODO
  }  
}

Note that:

@WebFluxTest is a standard Spring Boot "slice test" where only components needed to test SampleApiController will be started.
A WebTestClient is the standard way to test controllers on a @WebFluxTest, but in this case we will not use it directly, we will just pass it to the PactVerificationContext.
Just temporarily we use @PactFolder annotation to read the "contract" from the local directory where sample-api-client has generated it. No need for a PactBroker yet.
We use @Provider annotation to specify that we are executing tests for the "Sample API Server" provider.
We have to create as many methods annotated with @State as states the "contract" expects. We leave them empty for now but we will have to properly set the state there.
Finally PactVerificationSpring6Provider junit5 extension and pactVerificationTestTemplate method annotated with junit5's @TestTemplate will create tests dynamically following the "contract". Again ✨magic✨

If we create an empty SampleApiController to make this test compile:

@RestController
class SampleApiController

And then we try to execute the test:

cd sample-api-server
./gradlew test

> Task :test

Create Thing - FAILED
1) Verifying a pact between Sample API Client and Sample API Server - Create Thing: has status code 201
  1.1) status: expected status of 201 but was 404
  1.2) body: $ Actual map is missing the following keys: id

Get Thing 123 when it does not exist - PASSED

Get Thing 123 when it exists - FAILED
1) Verifying a pact between Sample API Client and Sample API Server - Get Thing 123 when it exists: has status code 200
  1.1) status: expected status of 200 but was 404
  1.2) body: $ Actual map is missing the following keys: date, name, value

Turns out that with an empty controller we pass one of the tests 😜

Once we have a final implementation of the controller ...

@RestController
class SampleApiController(
  private val repository: SampleRepository
) {

  @PostMapping(
    "/thing", 
    consumes = [APPLICATION_JSON_VALUE], 
    produces = [APPLICATION_JSON_VALUE]
  )
  @ResponseStatus(CREATED)
  suspend fun create(@RequestBody thing: SampleThing)
    = repository.save(thing)

  @GetMapping(
    "/thing/{id}",
    produces = [APPLICATION_JSON_VALUE]
  )
  suspend fun get(@PathVariable("id") id: Int) =
    when (val thing = repository.get(SampleThingId(id))) {
      null -> ResponseEntity.notFound().build()
      else -> ResponseEntity.ok(thing)
    }
}

... we can implement the contract test as:

@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactFolder("../sample-api-client/build/pacts")
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {

  @Autowired
  private lateinit var webTestClient: WebTestClient

  @MockkBean
  private lateinit var repository: SampleRepository

  @BeforeEach
  fun beforeEach(context: PactVerificationContext) {
    context.target = WebTestClientSpring6Target(webTestClient)
  }

  @TestTemplate
  fun pactVerificationTestTemplate(
    context: PactVerificationContext
  ) {
    context.verifyInteraction()
  }

  @State("Initial State")
  fun `initial state`() {
    every { repository.save(any()) } returns SampleThingId(123)
    every { repository.get(any()) } returns null
  }

  @State("Thing 123 exists")
  fun `thing 123 exists`() {
    every {
      repository.get(SampleThingId(123))
    } returns SampleThing(
      "Foo", 123.45, LocalDate.of(2022, 10, 13)
    )
  }
}

Note that:

We mock a SampleRepository because SampleApiController needs one.
In the state "Initial State":
- SampleRepository mock will return null whenever get is called with any SampleThingId because it is supposed to be empty.
- SampleRepository mock will return SampleThingId(123) whenever save is called with any SampleThing to simulate saving it.
In the state "Thing 123 exists":
- SampleRepository mock will return null whenever get is called with SampleThingId(123) because is supposed to be stored there.

Now if we execute the test everything should be 🟩

Finally, in a real scenario we will use @PactBroker instead of @PactFolder in order to retrieve the "contract" from a PactBroker:

@WebFluxTest(controllers = [SampleApiController::class])
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiControllerContractTest {
   // ... 
}

Note that:

PactBroker url can be set directly on the @PactBroker(url=xxx) annotation or via the pactbroker.url system property.

You can review the final implementation in SampleApiControllerContractTest.

Using @SpringBootTest

We can also test the "contract" against the whole application using a SpringBootTest:

@SpringBootTest(webEnvironment = DEFINED_PORT)
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiServerContractTest {

  @TestTemplate
  fun pactVerificationTestTemplate(
    context: PactVerificationContext
  ) {
    context.verifyInteraction()
  }

  @State("Initial State")
  fun `initial state`() {
    // TODO set "Initial State" state
  }

  @State("Thing 123 exists")
  fun `thing 123 exists`() {
    // TODO set "Thing 123 exists" state
  }
}

And a little extra code if we want to start the application using a random port:

@SpringBootTest(webEnvironment = RANDOM_PORT)
@Provider("Sample API Server")
@PactBroker
@ExtendWith(PactVerificationSpring6Provider::class)
class SampleApiServerContractTest {

  @LocalServerPort
  private var port = 0

  @BeforeEach
  fun beforeEach(context: PactVerificationContext) {
    context.target = HttpTestTarget("localhost", port)
  }

  // ...
}

Finally, we only need to set the expected state using the @State annotated methods. How we do that will be different on each case. For example if we are using a real database that would mean to insert/delete rows from a table, etc.

Just for this PoC we can cheat a little and implement an in-mem SampleRepository this way with a convenient reset method:

@Repository
class SampleRepository {

  private val sampleThingIdNext = AtomicInteger(1)
  private val sampleThings
    = mutableMapOf<SampleThingId, SampleThing>()

  fun save(thing: SampleThing): SampleThingId {
    val thingId = SampleThingId(
      sampleThingIdNext.getAndIncrement()
    )
    sampleThings[thingId] = thing
    return thingId
  }

  fun get(thingId: SampleThingId) = sampleThings[thingId]

  fun reset(nextId: Int) {
    sampleThingIdNext.set(nextId)
    sampleThings.clear()
  }
}

So we can implement the state methods this way:

@Autowired
private lateinit var repository: SampleRepository

@State("Initial State")
fun `initial state`() {
  // clear repository and set next id = 123
  repository.reset(123)
}

@State("Thing 123 exists")
fun `thing 123 exists`() {
  // clear repository and set next id = 123
  repository.reset(123)
  // save a thing that will get id = 123
  repository.save(
    SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
  )
}

You can review the final implementation in SampleApiServerContractTest.

5) Provider verifies or refutes the "contract"

To publish the result of the contract tests automatically to a PactBroker we need to:

Use @PactBroker annotation.
Set PactBroker url directly on @PactBroker(url=xxx) annotation or via pactbroker.url system property.
Set system property pact.verifier.publishResults=true.
Set system property pact.provider.version (we can use same version as the gradle project).

To test it locally:

Start a local instance of PactBroker with a sqlite database:

docker compose up -d

Publish the consumer "contract" using pact gradle plugin:

cd ./sample-api-client
./gradlew pactPublish

Make one of the provider tests fail, for example commenting these lines on the SampleApiControllerContractTest:

@State("Thing 123 exists")
fun `thing 123 exists`() {
  //every {
  //  repository.get(SampleThingId(123))
  //} returns SampleThing("Foo", 123.45, LocalDate.of(2022, 10, 13))
}

Execute provider SampleApiControllerContractTest (from IDE or command line):

cd ./sample-api-server
./gradlew test --tests '*SampleApiControllerContractTest'

Go to http://localhost:9292, and you will see the "contract" not verified (in red):
Fix the provider test and execute it again.
Go back to http://localhost:9292 and you will see the "contract" verified (in green):

6+7) Deploy only when the "contract" has been verified

Both for the consumer and the provider we can execute canIDeploy gradle task that will check if the "contract" has been verified in PactBroker, failing if not:

No contract published ❌

./gradlew canIDeploy

> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯ Request to path '/matrix/...' 
failed with response 400

Provider has neither verified nor refuted the contract ❌

./gradlew canIDeploy 

> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯

There is no verified pact between version 1.0 of 
Sample API Client and the latest version of 
Sample API Server (no such version exists)

Provider has refuted the contract ❌

./gradlew canIDeploy

> Task :canIDeploy FAILED
Computer says no ¯\_(ツ)_/¯

The verification for the pact between version 1.0 of 
Sample API Client and the latest version of 
Sample API Server (1.0) failed

Provider has verified the contract ✅

./gradlew canIDeploy

> Task :canIDeploy
Computer says yes \o/ 

All required verification results are published and successful

Implementation Details

Some, I hope useful, implementation details of this PoC:

We use au.com.dius.pact gradle plugin both for the consumer and the provider.
We use au.com.dius.pact.consumer:junit5 dependency for the consumer.
We use au.com.dius.pact.provider:spring6 dependency for the provider.
All these system properties are available.
Properties used in this PoC for the consumer:
- project.extra["pacticipant"] = "Sample API Client" and project.extra["pacticipantVersion"] = version so we do not need to pass them everytime in the canIDeploy task.
- Test tasks' systemProperties["pact.writer.overwrite"] = true so the contract is always overwritten.
- project.extra["pactbroker.url"] = project.properties["pactbroker.url"] ?: "http://localhost:9292" so:
- We can override it if needed (using ./gradlew -Ppactbroker.url=http://xxx).
- We can use the same in the publish and broker configuration sections.
Properties used in this PoC for the provider:
- project.extra["pacticipant"] = "Sample API Server" and project.extra["pacticipantVersion"] = version so we do not need to pass them everytime in the canIDeploy task.
- project.extra["pactbroker.url"] = project.properties["pactbroker.url"] ?: "http://localhost:9292" so:
- We can override it if needed (using ./gradlew -Ppactbroker.url=http://xxx).
- We can use it in the broker configuration section.
- We can use it in test tasks' systemProperties["pactbroker.url"] used by @PactBroker annotation as default.
- Test tasks':
- systemProperties["pact.provider.version"] = version to specify the provider version (it does not get it automatically from the gradle project, like it does for the consumer 🤷).
- systemProperties["pact.verifier.publishResults"] = "true" to always publish results back to the PactBroker (in a real example we would disable it locally and enable only in CI though).

ALso Github Actions CI is enabled for this repo and executes a complete flow (you can execute it locally too):

Start PactBroker
- Will run docker compose up -d to start a local PactBroker.
Sample API Client check
- Will run ./gradlew check to execute tests and generate the "contract" locally.
Sample API Client Pact publish
- Will run ./gradlew pactPublish to publish the "contract" to the local PactBroker.
Sample API Server check
- Will run ./gradlew check that:
- Will download the "contract" from the local PactBroker.
- Will execute tests accordingly.
- Will publish the result of the tests back to the local PactBroker.
Sample API Client can deploy?
- Will run ./gradlew canIDeploy that will connect to the local PactBroker and be successful if provider has verified the "contract".
Sample API Server can deploy?
- Will run ./gradlew canIDeploy that will connect to the local PactBroker and be successful if provider has verified the "contract".
Stop PactBroker
- Will run docker compose down to stop the local PactBroker.

Want to know more?

Some interesting documentation at Pact, including:

Pact Workshops - from 0 to Pact in ~2 hours - hands-on labs for:
- Ruby
- JS
- Golang
- JVM
- Android
- .NET
- CI/CD
Pact Broker Docs - including:
- Pact Broker Client CLI
- Pact Broker Webhooks to trigger builds every time a pact is changed, published or verified.

That's all! Happy coding! 💙

DEV Community

🤝 Contract Testing with Pact

rogervinas / contract-testing-with-pact

🤝 Contract Testing with Pact

1) Consumer defines the "contract" with the Provider

2) Consumer tests the "contract" using a provider mock

3) Consumer publishes the "contract"

4) Provider tests the "contract" using a consumer mock

Using @WebFluxTest

Using @SpringBootTest

5) Provider verifies or refutes the "contract"

6+7) Deploy only when the "contract" has been verified

Implementation Details

Want to know more?

Top comments (0)

Read next

The Last Saree: Connoisseurship in the Age of AI

11 Practical Ways to Bring Side Income as a Developer 💰

💡Only 20% of Developers are Happy at Work?

Experimenting with "following" tab — will give more details once more fully launched. Folks have been asking for something like this for a while.