I heard about Contract Tests in detail at the Ministry of Testing Meetup. Abhi Nandan made an excellent introduction talk Contract testing in Polyglot Microservices.
I liked the idea and decided to try this approach on my current project. I must say that the devil is in the details. In theory, everything is simple until you try the integration in practice on an existing project.
It is the same with algorithms: when you are watching how the developer on YouTube solves the task, everything seems straightforward. You immediately hit a snag while sitting down to solve the problem alone.
Today I want to focus on practical examples, as much as possible close to real-world conditions. I encourage you to try it on your own to gain experience and see if these approaches are worth applying to your projects. We will use the realword project on GitHub, an Exemplary Medium.com clone powered by different frameworks and tools.
I like using this project for the demo because it's much more complex than HelloWorld
, where everything works most of the time. True-to-life problems give your extensive experience.
What is so unique in contract tests?
Contract tests assert that inter-application messages conform to a shared understanding documented in a contract. Without contract testing, the only way to ensure that applications will work correctly together is by using expensive and brittle integration tests.
You can hear opinions: Contract tests often make sense when you have a microservice architecture with 10-50 services.
I'm afraid I have to disagree with that. If you have at least one Provider (back-end with API) and Consumer (front-end), it already makes sense to try this testing approach.
I suggest starting exploration with bi-directional contract tests.
Bi-Directional Contract Testing is a type of static contract testing.
Teams generate a consumer contract from a mocking tool (such as Pact or Wiremock) and API providers verify a provider contract (such as an OAS) using a functional API testing tool (such as Postman). Pactflow then statically compares the contracts down to the field level to ensure they remain compatible.
Why is it better to start with BDCT? Bi-Directional Contract Testing (BDCT) allows you to use existing Provider Open API. If your current project has a back-end service with Open API specification, you are lucky to leverage it as a Provider contract.
1. Provider Contract
Let's configure publishing of OAS spec from realworld
app to pactflow.
Requirements:
- Create a free account on Pactflow.io;
- Install Docker on your machine.
We want to be able to publish Open API spec to Pactflow locally and from CI/CD. To do that we will need the following: API key from Pactflow, docker
and make
commands installed.
Consider the Makefile:
# Makefile
PACTICIPANT ?= "realworld-openapi-spec"
## ====================
## Pactflow Provider Publishing
## ====================
PACT_CLI="docker run --rm -v ${PWD}:/app -w "/app" -e PACT_BROKER_BASE_URL -e PACT_BROKER_TOKEN pactfoundation/pact-cli"
OAS_FILE_PATH?=api/openapi.yml
OAS_FILE_CONTENT_TYPE?=application/yaml
REPORT_FILE_PATH?=api/README.md
REPORT_FILE_CONTENT_TYPE?=text/markdown
VERIFIER_TOOL?=newman
# Export all variable to sub-make if .env exists
ifneq (,$(wildcard ./.env))
include .env
export
endif
default:
cat ./Makefile
ci: ci-test publish_pacts can_i_deploy
# Run the ci target from a developer machine with the environment variables
# set as if it was on CI.
# Use this for quick feedback when playing around with your workflows.
fake_ci:
@CI=true \
GIT_COMMIT=`git rev-parse --short HEAD` \
GIT_BRANCH=`git rev-parse --abbrev-ref HEAD` \
make ci
## =====================
## Build/test tasks
## =====================
ci-test:
@echo "\n========== STAGE: CI Tests ==========\n"
## =====================
## Pact tasks
## =====================
publish_pacts:
@echo "\n========== STAGE: publish provider contract (spec + results) - success ==========\n"
PACTICIPANT=${PACTICIPANT} \
"${PACT_CLI}" pactflow publish-provider-contract \
/app/${OAS_FILE_PATH} \
--provider ${PACTICIPANT} \
--provider-app-version ${GIT_COMMIT} \
--branch ${GIT_BRANCH} \
--content-type ${OAS_FILE_CONTENT_TYPE} \
--verification-exit-code=0 \
--verification-results /app/${REPORT_FILE_PATH} \
--verification-results-content-type ${REPORT_FILE_CONTENT_TYPE} \
--verifier ${VERIFIER_TOOL}
deploy: deploy_app record_deployment
can_i_deploy:
@echo "\n========== STAGE: can-i-deploy? 🌉 ==========\n"
"${PACT_CLI}" broker can-i-deploy --pacticipant ${PACTICIPANT} --version ${GIT_COMMIT} --to-environment test
deploy_app:
@echo "\n========== STAGE: deploy ==========\n"
@echo "Deploying to test"
record_deployment:
"${PACT_CLI}" broker record-deployment --pacticipant ${PACTICIPANT} --version ${GIT_COMMIT} --environment test
## =====================
## Misc
## =====================
.env:
cp -n .env.example .env || true
Run make .env
to create .env
file. You will need to update two variables:
PACT_BROKER_BASE_URL
PACT_BROKER_TOKEN
[Optional] We may try to publish the contract from the local machine using make fake-ci
. This target under the hood executes the following stages:
-
ci-test
runs functional tests (skipped in our case) -
publish_pacts
uses pactflow docker image to publish current OAS to Pactflow broker.GIT_COMMIT
andGIT_BRANCH
environment variables are used to set the version tag and the branch on the broker side. -
can_i_deploy
verifies that the published contract is compatible with dependent consumers. Hence we are safe to deploy.
And the last piece. Let's create a GitHub workflow to publish OAS on each push to the repository:
# .github/workflows/pact.yml
name: Pact CI
on:
pull_request:
push:
branches:
- main
paths-ignore:
- '*.md'
concurrency:
# For pull requests, cancel all currently-running jobs for this workflow
# https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }}
cancel-in-progress: true
env:
PACT_BROKER_BASE_URL: https://drakulavich.pactflow.io
PACT_BROKER_TOKEN: ${{ secrets.PACT_BROKER_TOKEN }}
GIT_COMMIT: ${{ github.sha }}
jobs:
verify:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action@v4
- name: Tests
run: GIT_BRANCH=${GITHUB_REF_NAME_SLUG} make ci
deploy:
runs-on: ubuntu-latest
needs:
- verify
if: ${{ github.ref == 'refs/heads/main' }}
steps:
- uses: actions/checkout@v3
- name: Inject slug/short variables
uses: rlespinasse/github-slug-action@v4
- name: 🚀 Record deployment on Pactflow
run: GIT_BRANCH=${GITHUB_REF_NAME_SLUG} make deploy
GitHub will use the same Makefile
. Please, have a look at the Deploy job from the workflow above. We are executing make deploy
to explicitly mark the deployed version for the test environment on the Pactflow broker.
We will provide the environment variable to access Pactflow differently. Don't forget to add the secret PACT_BROKER_TOKEN
in Settings — Secrets — New repository secret.
Ok. We are ready to commit the files! If everything is fine, the pipeline should be green.
Open your Pactflow workspace and check out the Provider Contract. You should be able to open Open API Swagger spec.
So, we finished with the first part. We have CI/CD for Provider Contract. Let's move on to the second part.
2. Consumer contract
We will use ts-redux-react-realworld-example-app project which implements the front-end in TS for realword app using React/Redux.
If you want to follow the same approach, I suggest you to check pact-workshop-js for better understanding.
1) We'll start by installing @pact-foundation/pact
dependency:
npm install --save-dev @pact-foundation/pact
2) Create directory src/services/consumer/
3) Create src/services/consumer/apiPactProvider.ts
Here we'll describe the pact specification format and name of our pacticipants (participants).
// src/services/consumer/apiPactProvider.ts
import path from 'path';
import { PactV3, MatchersV3, SpecificationVersion } from '@pact-foundation/pact';
export const { eachLike, like } = MatchersV3;
export const provider = new PactV3({
consumer: 'ts-redux-react-realworld-example-app',
provider: 'realworld-openapi-spec',
logLevel: 'warn',
dir: path.resolve(process.cwd(), 'pacts'),
spec: SpecificationVersion.SPECIFICATION_VERSION_V2,
});
4) Create src/services/consumer/api.pact.spec.ts
Let's start with something tiny and achievable. We would like to check /tags
endpoint.
// src/services/consumer/api.pact.spec.ts
describe('API Pact tests', () => {
describe('getting all tags', () => {
test('tags exist', async () => {
// set up Pact interactions
// Execute provider interaction
});
});
});
From pact-workshop-js you will learn that each Pact test contains two parts:
- Setup Pact interactions.
- Execute the interaction using the API client.
The first part might look like:
// src/services/consumer/api.pact.spec.ts
// ...
const tagsResponse = {
tags: ['reactjs', 'angularjs'],
};
// set up Pact interactions
await provider.addInteraction({
states: [{ description: 'tags exist' }],
uponReceiving: 'get all tags',
withRequest: {
method: 'GET',
path: '/tags',
},
willRespondWith: {
status: 200,
headers: {
'Content-Type': 'application/json',
},
body: like(tagsResponse),
},
});
// ...
We created a stub for the response tagsResponse
. Then we added interaction to the provider object from src/services/consumer/apiPactProvider.ts
. The essential parts relate to the expected request and how the provider will reply.
In the second part we will add the test execution:
// src/services/consumer/api.pact.spec.ts
// ...
await provider.executeTest(async (mockService) => {
axios.defaults.baseURL = mockService.url;
// make request to Pact mock server
const tags = await getTags();
expect(tags).toStrictEqual(tagsResponse);
});
// ...
Ultimately we need to change the URL for the API client and execute the request after it. In our case we have to alter axios baseURL
, because all API interactions explicitly use axios. For example, getTags()
implementation:
export async function getTags(): Promise<{ tags: string[] }> {
return guard(object({ tags: array(string) }))((await axios.get('tags')).data);
}
We want to verify that response is the same as a prepared stub. After collecting all parts inside the test file we will get the following:
// src/services/consumer/api.pact.spec.ts
import axios from 'axios';
import { provider, like } from './apiPactProvider';
import { getTags } from '../conduit';
describe('API Pact tests', () => {
describe('getting all tags', () => {
test('tags exist', async () => {
const tagsResponse = {
tags: ['reactjs', 'angularjs'],
};
// set up Pact interactions
await provider.addInteraction({
states: [{ description: 'tags exist' }],
uponReceiving: 'get all tags',
withRequest: {
method: 'GET',
path: '/tags',
},
willRespondWith: {
status: 200,
headers: {
'Content-Type': 'application/json',
},
body: like(tagsResponse),
},
});
await provider.executeTest(async (mockService) => {
axios.defaults.baseURL = mockService.url;
// make request to Pact mock server
const tags = await getTags();
expect(tags).toStrictEqual(tagsResponse);
});
});
});
});
Let's run make fake-ci
to check how is going:
Computer says yes \o/
CONSUMER | C.VERSION | PROVIDER | P.VERSION | SUCCESS? | RESULT#
-------------------------------------|--------------------|------------------------|------------|----------|--------
ts-redux-react-realworld-example-app | d981433+1663657371 | realworld-openapi-spec | 95dbd23... | true | 1
Hooray! It's a huge step towards contract testing.
Now let's try to test another endpoint GET /user
. We will start with the same ideas: create a stub, prepare interactions and execute API client call:
// src/services/consumer/api.pact.spec.ts
// ...
describe('getting current user', () => {
test('user exists', async () => {
const tUser: User = {
email: 'jake@jake.jake',
token: 'jwt.token.here',
username: 'jake',
bio: 'I work at statefarm',
image: 'https://i.stack.imgur.com/xHWG8.jpg',
};
const userResponse = {
user: tUser,
};
// set up Pact interactions
await provider.addInteraction({
states: [{ description: 'user has logged in' }],
uponReceiving: 'get user',
withRequest: {
method: 'GET',
path: '/user',
},
willRespondWith: {
status: 200,
headers: {
'Content-Type': 'application/json',
},
body: like(userResponse),
},
});
await provider.executeTest(async (mockService) => {
axios.defaults.baseURL = mockService.url;
// make request to Pact mock server
const user = await getUser();
expect(user).toStrictEqual(tUser);
});
});
});
// ...
If you run make fake-ci
, you will get an error:
Computer says no ¯_(ツ)_/¯
Follow the generated report link and you will see the details:
Request Authorization header is missing but is required by the spec file
We missed Authorization header that is required to check logged in user. Let's add the stub (1), headers to the expected request (2) and axios defaults (3):
// src/services/consumer/api.pact.spec.ts
// (1)
const authToken = {
Authorization: 'Token xxxxxx.yyyyyyy.zzzzzz',
};
// ...
uponReceiving: 'get user',
withRequest: {
method: 'GET',
path: '/user',
headers: authToken, // (2)
},
// ...
await provider.executeTest(async (mockService) => {
axios.defaults.baseURL = mockService.url;
axios.defaults.headers.Authorization = authToken.Authorization; // (3)
After next make fake-ci
attempt you will get the new error:
FAIL src/services/consumer/api.pact.spec.ts
● Console
console.error
Error: Cross origin http://localhost forbidden
I found the fix for it on StackOverflow. We need to add the following line on the top of the test file:
axios.defaults.adapter = require('axios/lib/adapters/http');
Next make fake-ci
should be successful.
Summary
We have learned how to configure Provider and Consumer for bi-directional contract testing. We used RealWorld project to practice integration with Pactflow broker. And finally, we wrote a couple of consumer tests for ts-redux-react-realworld-example-app. We also learned how to debug the tests using make fake-ci
command. All of the examples are available on GitHub. Feel free to try and post your questions if you face the issues.
Materials
- RealWorld repo with Open API spec (provider contract)
- ts-redux-react-realworld-example-app with pact tests (consumer)
- pact js workshop
Top comments (0)