I wrote many blog posts about unit and integration testing. But today I want to tell you something beyond it. And this is E2E-testing. Though it's important to test each service's behaviour distinctly. But it's also crucial to verify the business scenario validity on the whole running system. In this article, I'm telling you what is E2E-testing, why is it so important and how you can implement it within your release pipeline. You'll learn how to run E2E-tests on each new pull request before merging the changes to the master
branch.
The code examples are in Java but the proposed solution is applicable to any programming language. You can find the source code of the whole project by this link.
Domain
We're going to develop a system for gaining upcoming messages with additional data. Take a look at the schema below.
The message processing algorithm is simple:
- User sends a message via REST API.
-
API-Service
transfers it to RabbitMQ. -
Gain-Service
updates data for future gaining in Redis, if the message contains something valuable. And then puts additional data to the message itself and then transfers it to RabbitMQ again.
Testing
Unit testing
How can we validate the system's behaviour? There are several options. The simplest one is unit testing. Take a look at the diagram below. I pointed out the testing areas with pale green and blue ovals.
Unit tests have several advantages:
- They are fast to run.
- Easily integrated into CI/CD pipeline.
- Can be run in parallel (if properly written).
Though they also have a problem. Unit tests do not check interactions with real external services (e.g. Redis, RabbitMQ). It's about verifying business logic but not the actual production scenario.
I wrote a longread about unit testing patterns and best practices. Go check it out, it's really awesome.
Integration testing
We need to extend the perspective. So, integration tests can come in handy, right? Have a look at the next diagram below.
In this case, we do check interactions with external services. Though one problem remains. A business operation involves several components of communication. Even if each module is tested properly, how can we verify the correctness of a multi-service request (i.e. a business scenario)? For example, if the API-service
puts a breaking change to the format of the output message, then the gain-service
won't be able to proceed with enrichment successfully. Though the API-service
integration and unit tests would pass.
To overcome this issue we need something beyond integration tests.
I wrote an article explaining integration tests deeply. You should check it out.
E2E testing
The idea of E2E testing is straightforward. We're considering the whole system as a black box that accepts some data and returns the computed result (either synchronously or asynchronously). Take a look at the schema below.
Well, that sounds reasonable and trustworthy. But how can we implement it? Where do we begin? Let's start to deconstruct this problem step by step.
Releasing strategies
Firstly, let's clarify the release pipeline of individual services. That'll help us to understand the whole E2E testing approach. Take a look at the schema below.
Here is the flow step by step:
- A developer push changes to the
feature/task
branch. - Then makes pull request from
feature/task
tomaster
branch. - During the CI pipeline the pull request is being built (i.e. unit tests and integration tests execution).
- If the pipeline is green, the changes are merged to the
master
branch. - When the pull request is merged, the resulting artefact is published to the Docker Hub.
- When the release is triggered (e.g. on a scheduled basis), the
deploy
stage pulls the required Docker image (latest by default) and runs it in the specified environment.
So, how can we put E2E tests within the stated process? Actually, there are several ways.
Synchronous releasing strategy
That's the easiest approach to understand. No matter how many services we have, the release pipeline deploys each of them within a single job. In this case, we just need to run E2E tests right before deploying artefacts to production. Take a look at the schema below describing the process.
The algorithm is:
- Trigger release
- Pull all services' images from the Docker Hub (latest by default).
- Run E2E tests with the pulled images (I'll explain the approach to you later in the article).
- If tests succeed, deploy the pulled images.
Despite its simplicity, this approach has a significant obstacle. You cannot update a single microservice
isolated. It means that different modules have to be released all at once. Though in reality, some microservices have to be updated more frequently than others. But here you have to choose a release trigger that satisfies (at least partially) every service's requirements.
Asynchronous releasing strategy
This one means updating each service like an isolated functionality. Each module can be deployed accordirng to its own rules.
Here is an example of an asynchronous releasing strategy. Take a look at the schema below.
As you can see, the diagram is similar to a single module release pipeline that we've seen before. Though there are slight differences. Now there is the E2E-tests
stage that runs both during pull request build and right before deploying to production.
Why do we need to run E2E-tests
again if they have already been completed on the pull request pipeline? Take a look at the picture below to understand the problem.
We deployed API-Service
immediately after the PR merge. But we delayed the Gain-Service
release by one day. So, if E2E-tests run only during pull request build, there is a chance that some other services have been already updated. But we verified the correctness only with previous versions because during the pull request build the newest releases have not been promoted yet.
If you stick with the asynchronous releasing strategy, you have to run E2E-tests right before deploying to production as well as during pull request build.
In this article, we're looking at asynchronous releasing strategy as preferred for microservices.
Establishing the process
Well, that all sounds promising. But how do we establish this scenario? I can say that's not as complex as it seems. Take a look at the example of running E2E-tests for the API-Service
below.
There are two parts. Running E2E-tests during pull request build and right before deploying the artefact to production. Let's go through each scenario step by step.
Pull request build
- Firstly, unit tests and integration tests are run. These two steps are usually combined with the building artefact itself.
- Then the current version of
API-Service
is being built and saved locally as a Docker image. We don't push it to the hub because the proposed changes might not be correct (we haven't run E2E-tests to check it yet). Though some CI providers don't allow building Docker images locally to reuse them later. In that case, you can specify a tag that won't be used in production. For example,dev-CI_BUILD_ID
. - Then we pull a Docker image containing E2E-tests themselves. As we see later, it's a simple application. So, it's convenient to keep in Docker Hub as well.
- And finally, it's time to run E2E tests. The app that contains tests should be configurable to run with different Docker images of services (in this case,
API-Service
andGain-Service
). Here we put theAPI_SERVICE_IMAGE
as the one that we've built locally in step 2.
All other services should have the default Docker image as the latest tag. That'll give us an opportunity to run E2E tests in any repository by overriding the current service image version.
If all verifications are passed, the PR is accepted to be merged. After the merge, the new version of API-Service
is pushed to Docker Hub with the latest
tag.
E2E-tests running before the deploy stage
- Unit tests and integrations tests are run the same way.
- The latest version of the
E2E-tests
images is pulled from the Docker Hub. - E2E-tests are run with the tags of
latest
for all the services.
The
API-Service
has been already pushed to Docker Hub with thelatest
tag on the pull request merge. Therefore, there is no need to specify the particular image version on the E2E-tests run.
Code Implementation
Let's start implementing the E2E tests. You can check out the source code by this link.
I'm using Spring Boot Test as the framework for E2E tests. But you can apply any technology you like.
I placed all modules (including
e2e-tests
) within a single mono-repository for the sake of simplicity. Anyway, the approach I'm describing to you is comprehensive. So, you can apply it to multi-repositories microservices as well.
Let's start with the E2ESuite
. This one will contain all configurations and act as a superclass for all the test cases. Take a look at the code example below.
@ContextConfiguration(initializers = Initializer.class)
@SpringBootTest(webEnvironment = RANDOM_PORT)
@Import({
TestRedisFacade.class,
TestRabbitListener.class,
TestRestFacade.class
})
public class E2ESuite {
private static final Network SHARED_NETWORK = Network.newNetwork();
private static GenericContainer<?> REDIS;
private static RabbitMQContainer RABBIT;
private static GenericContainer<?> API_SERVICE;
private static GenericContainer<?> GAIN_SERVICE
}
Firstly, we have to declare Docker containers to run within the Testcontainers environment. Here we've got Redis and RabbitMQ that are part of the infrastructure. Whilst API_SERVICE
and GAIN_SERVICE
are the custom services implementing the business logic.
The
@Import
annotation is used to add custom classes to the Spring Context that are used for testing purposes. Their implementation is trivial. So, you can find it by the repository link above. Though@ContextConfiguration
is important. We'll get to this soon.
Also, SHARED_NETWORK
is crucial. You see, the containers should communicate with each other because that's the purpose of the E2E scenario. But also we have to be able to send HTTP requests to API-Service
to invoke the business logic. To achieve both of these goals we bound all the containers with a single network and forward the API-Service
HTTP port to open access for the client. Take a look at the schema below describing the process.
Now we need to initialize and start the containers somehow. Besides, we also have to specify the correct properties to connect our E2E-tests
application to the recently started Docker containers. In this case, the @ContextConfiguration
annotation can come in handy. It provides the initializers
parameter which represents callbacks invoked in Spring Context initializing stage. Here we've put the inner class Initializer
. Take a look at the code example below.
static class Initializer implements
ApplicationContextInitializer<ConfigurableApplicationContext> {
@Override
public void initialize(ConfigurableApplicationContext context) {
final var environment = context.getEnvironment();
REDIS = createRedisContainer();
RABBIT = createRabbitMQContainer(environment);
Startables.deepStart(REDIS, RABBIT).join();
final var apiExposedPort = environment.getProperty("api.exposed-port", Integer.class);
API_SERVICE = createApiServiceContainer(environment, apiExposedPort);
GAIN_SERVICE = createGainServiceContainer(environment);
Startables.deepStart(API_SERVICE, GAIN_SERVICE).join();
setPropertiesForConnections(environment);
}
...
}
Let's deconstruct this functionality step by step. Redis container is created first. Take a look at the code snippet below.
private GenericContainer<?> createRedisContainer() {
return new GenericContainer<>("redis:5.0.14-alpine3.15")
.withExposedPorts(6379)
.withNetwork(SHARED_NETWORK)
.withNetworkAliases("redis")
.withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger("Redis")));
}
At the moment of writing, there is no distinct container for Redis in the Testcontainers library. So, I'm using a generic one. The most important attributes are network
and network aliases
. Their presence makes a container reachable for the other ones within the same network. We're also exposing the 6379
port (the default Redis port) because the E2E test case will connect to Redis during the execution.
Also, I'd like you to pay attention to the log consumer
. You see, when the E2E scenario fails, it's not always obvious why. Sometimes to understand the source of the problem you have to dig into containers' logs. Thankfully the log consumer
allows us to forward a container's logs to any SLF4J logger instance. In this project, containers' logs are forwarded to regular text files (you can find the Logback configuration in the repository). Though it's much better to transfer logs to external logging facility (e.g. Kibana).
Next comes RabbitMQ. Take a look at the container initialization below.
private RabbitMQContainer createRabbitMQContainer(Environment environment) {
return new RabbitMQContainer("rabbitmq:3.7.25-management-alpine")
.withNetwork(SHARED_NETWORK)
.withNetworkAliases("rabbit")
.withQueue(
environment.getProperty("queue.api", String.class)
)
.withQueue(
environment.getProperty("queue.gain", String.class)
)
.withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger("Rabbit")));
}
The idea is similar to Redis container instantiation. But here we also called withQueue
method (which is part of the RabbitMQContainer
class) to specify default topics on RabbitMQ start. API-Service
sends messages to queue.api
topic and Gain-Service
sends messages to queue.gain
topic (those properties are configurable). So, it's convenient to create the required topics on the application start.
Then there is an interesting line of code.
Startables.deepStart(REDIS, RABBIT).join();
The deepStart
method accepts varargs of containers to start and returns CompletableFuture
. We need those containers to start before API-Service
and Gain-Service
. So, we call the join
method to wait until containers are ready to accept requests.
You can also start all the containers with the single
deepStart
method invocation and specify the order by calling thedependsOn
method on the container itself. It's more performant but harder to read through. So, I'm leaving the simpler example.
And now we can start our custom containers.
final var apiExposedPort = environment.getProperty("api.exposed-port", Integer.class);
API_SERVICE = createApiServiceContainer(environment, apiExposedPort);
First of all, let's deep dive into the createApiServiceContainer
method. Take a look at the code snipped below.
private GenericContainer<?> createApiServiceContainer(
Environment environment,
int apiExposedPort
) {
final var apiServiceImage = environment.getProperty(
"image.api-service",
String.class
);
final var queue = environment.getProperty(
"queue.api",
String.class
);
return new GenericContainer<>(apiServiceImage)
.withEnv("SPRING_RABBITMQ_ADDRESSES", "amqp://rabbit:5672")
.withEnv("QUEUE_NAME", queue)
.withExposedPorts(8080)
.withNetwork(SHARED_NETWORK)
.withNetworkAliases("api-service")
.withCreateContainerCmdModifier(
cmd -> cmd.withHostConfig(
new HostConfig()
.withNetworkMode(SHARED_NETWORK.getId())
.withPortBindings(new PortBinding(
Ports.Binding.bindPort(apiExposedPort),
new ExposedPort(8080)
))
)
)
.waitingFor(
Wait.forHttp("/actuator/health")
.forStatusCode(200)
)
.withImagePullPolicy(new AbstractImagePullPolicy() {
@Override
protected boolean shouldPullCached(DockerImageName imageName,
ImageData localImageData) {
return true;
}
})
.withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger("API-Service")));
}
There some things I want to point out.
The withEnv
method just sets a regular environment variable. Those are used to configure API-Service
. You have probably noticed that the RabbitMQ
URL is amqp://rabbit:5672
. Because rabbit
is the corresponding container's name in the internal network (we specified it as a network alias on the container's instantiation). That is what makes RabbitMQ
reachable by the API-Service
.
The waitingFor
clause is more interesting. Testcontainers has to know somehow that a container is ready to accept connections. API-Service
exposes the /actuator/health
HTTP path that returns a 200
code, if the instance is prepared.
The withCreateContainerCmdModifier
combined with the withExposedPorts
method binds the internal container's port 8080
to the apiExposedPort
(specified by environment variable before E2E tests start).
The withImagePullPolicy
defines the rule for retrieving images directly from the Docker Hub. By default, Testcontainers checks the image's presence locally. If it finds one, it does not pull anything from the remote server. The behaviour is suitable for testing particular images. But if you specify the one with the latest
tag, there is a chance that the library won't pull the most relevant version. In this case, Testcontainers always pull images from the remote Docker Hub.
Take a look at the Gain-Service
container declaration below.
private GenericContainer<?> createGainServiceContainer(Environment environment) {
final var gainServiceImage = environment.getProperty(
"image.gain-service",
String.class
);
final var apiQueue = environment.getProperty(
"queue.api",
String.class
);
final var gainQueue = environment.getProperty(
"queue.gain",
String.class
);
return new GenericContainer<>(gainServiceImage)
.withNetwork(SHARED_NETWORK)
.withNetworkAliases("gain-service")
.withEnv("SPRING_RABBITMQ_ADDRESSES", "amqp://rabbit:5672")
.withEnv("SPRING_REDIS_URL", "redis://redis:6379")
.withEnv("QUEUE_INPUT_NAME", apiQueue)
.withEnv("QUEUE_OUTPUT_NAME", gainQueue)
.waitingFor(
Wait.forHttp("/actuator/health")
.forStatusCode(200)
)
.withImagePullPolicy(new AbstractImagePullPolicy() {
@Override
protected boolean shouldPullCached(DockerImageName imageName,
ImageData localImageData) {
return true;
}
})
.withLogConsumer(new Slf4jLogConsumer(LoggerFactory.getLogger("Gain-Service")));
}
As you can see, the initialization is similar to API-Service
. So, let's go further.
When API-Service
and Gain-Service
containers are ready, we can start them. Take a look at the code snippet below.
Startables.deepStart(API_SERVICE, GAIN_SERVICE).join();
setPropertiesForConnections(environment);
We have already discussed the idea of Startables.deepStart
. Though setPropertiesForConnections
requires some explanations. This method sets URLs of the started container as the properties for the E2E test cases. So, test suites can verify the results. Take a look at the procedure implementation below.
private void setPropertiesForConnections(ConfigurableEnvironment environment) {
environment.getPropertySources().addFirst(
new MapPropertySource(
"testcontainers",
Map.of(
"spring.rabbitmq.addresses", RABBIT.getAmqpUrl(),
"spring.redis.url", format(
"redis://%s:%s",
REDIS.getHost(),
REDIS.getMappedPort(6379)
),
"api.host", API_SERVICE.getHost()
)
)
);
}
Here we've specified connections for the RabbitMQ and Redis. Also, we stored the API-Service
host to send HTTP requests.
OK, let's do the test cases. We're writing a single E2E scenario. Take a look at the bullet list below.
- A client sends the message that contains both
msisdn
andcookie
values to theAPI-Service
- The message with no modifications should be transmitted to RabbitMQ eventually.
- A client sends the message that contains only the
cookie
value to theAPI-Service
. - The enriched message with a determined
msisdn
value should be transmitted to RabbitMQ eventually.
Take a look at the test suite below.
class GainTest extends E2ESuite {
@Test
void shouldGainMessage() {
rest.post(
"/api/message",
Map.of(
"some_key", "some_value",
"cookie", "cookie-value",
"msisdn", "msisdn-value"
),
Void.class
);
await().atMost(FIVE_SECONDS)
.until(() -> getGainQueueMessages().contains(Map.of(
"some_key", "some_value",
"cookie", "cookie-value",
"msisdn", "msisdn-value"
)));
rest.post(
"/api/message",
Map.of(
"another_key", "another_value",
"cookie", "cookie-value"
),
Void.class
);
await().atMost(FIVE_SECONDS)
.until(() -> getGainQueueMessages().contains(Map.of(
"another_key", "another_value",
"cookie", "cookie-value",
"msisdn", "msisdn-value"
)));
}
}
First of all, we send a message with cookie
and msisdn
. Then we check that the message is transferred further as-is. The next step is to send another message with omitted msisdn
but present cookie
value. Finally, the message with enriched msisdn
value should be pushed to RabbitMQ by Gain-Service
eventually.
If you run the test locally, it may take a while. Anyway, it takes time to download the required images and start the corresponding containers. But the test should pass successfully.
Running in CI environment
Well, that all sounds great. But how do we run E2E tests during the CI pipeline?
Firstly, we should pack E2E tests as the Docker image. Take a look at the Dockerfile below.
FROM openjdk:17-alpine
WORKDIR /app
COPY . /app
CMD ["/app/gradlew", ":e2e-tests:test"]
So, tests are compiled and run on the container's start.
Tests are not part of the compiled artefact (in this case, the
.jar
file). That's why we copy the whole directory with the code itself.
Next comes the YAML configuration for the GitHub Actions pipeline. The result playbook is quite long. So, I'm showing it to you in small parts.
We're going to run the test cases on each pull request and each merge to the master
branch.
name: Java CI
on:
push:
branches: [ "master" ]
pull_request:
branches: [ "master" ]
The whole pipeline consists of 3 jobs:
-
build
compiles all the services (API-Service
andGain-Service
), and runs unit and integration tests. -
build-dev-images
packs all the components (includingE2E-tests
) as the Docker images and pushes them to the Docker Hub with thedev-$CI_BUILD_NUM
tag. -
e2e-tests
runs E2E tests for the images pushed on thebuild-dev-images
job. -
build-prod-images
packs all the components as the Docker images and pushes them to the Docker Hub with thelatest
tag. Runs only in themaster
branch after successfully passing thee2e-tests
job.
Let's look at each job distinctly.
build
That's the most trivial one. Moreover, GitHub can generate this one for you.
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up JDK 17
uses: actions/setup-java@v3
with:
java-version: '17'
distribution: 'temurin'
- name: Build with Gradle
uses: gradle/gradle-build-action@0d13054264b0bb894ded474f08ebb30921341cee
with:
arguments: :gain-service:build :api-service:build
build-dev-images
This one is tricky. Firstly, we have to store DOCKERHUB_USERNAME
and DOCKERHUB_TOKEN
as the repository secrets to push Docker-built images. Then we should push the artefacts. And finally, we have to forward the calculated dev
tag to the next job. Take a look at the implementation below.
jobs:
...
build-dev-images:
needs:
- build
runs-on: ubuntu-latest
outputs:
image_tag: ${{ steps.env.outputs.image_tag }}
steps:
- name: Checkout
uses: actions/checkout@v3
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2
- name: Login to DockerHub
uses: docker/login-action@v2
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Define images tags
id: env
run: |
export IMAGE_TAG_ENV=dev-${{ github.run_number }}
echo "IMAGE_TAG=$IMAGE_TAG_ENV" >> "$GITHUB_ENV"
echo "::set-output name=image_tag::$IMAGE_TAG_ENV"
- name: Build and push E2E-tests
uses: docker/build-push-action@v3
with:
file: "./Dockerfile_e2e_tests"
push: true
tags: kirekov/e2e-tests:${{ env.IMAGE_TAG }}
- name: Build and push API-Service
uses: docker/build-push-action@v3
with:
file: "./Dockerfile_api_service"
push: true
tags: kirekov/api-service:${{ env.IMAGE_TAG }}
- name: Build and push Gain-Service
uses: docker/build-push-action@v3
with:
file: "./Dockerfile_gain_service"
push: true
tags: kirekov/gain-service:${{ env.IMAGE_TAG }}
I want you to pay attention to these lines of code.
jobs:
...
build-dev-images:
...
outputs:
image_tag: ${{ steps.env.outputs.image_tag }}
steps:
...
- name: Define images tags
id: env
run: |
export IMAGE_TAG_ENV=dev-${{ github.run_number }}
echo "IMAGE_TAG=$IMAGE_TAG_ENV" >> "$GITHUB_ENV"
echo "::set-output name=image_tag::$IMAGE_TAG_ENV"
The export IMAGE_TAG_ENV=dev-${{ github.run_number }}
line sets the dev
tag with the generated build number to the IMAGE_TAG_ENV
environment variable.
The echo "IMAGE_TAG=$IMAGE_TAG_ENV" >> "$GITHUB_ENV"
line makes ${{ env.IMAGE_TAG }}
variable available. It is used to specify the Docker tag on image publishing in the next steps.
The echo "::set-output name=image_tag::$IMAGE_TAG_ENV"
saves image_tag
variable as the output. So, the next job can reference it to run the specified version of E2E tests.
The pushing to Docker Hub itself is implemented with docker/build-push-action
. Take a look at the code snippet below.
- name: Build and push E2E-tests
uses: docker/build-push-action@v3
with:
file: "./Dockerfile_e2e_tests"
push: true
tags: kirekov/e2e-tests:${{ env.IMAGE_TAG }}
Building and pushing API-Service
and Gain-Service
is similar.
e2e-tests
And now it's time to run E2E tests. Take a look at the configuration below.
jobs:
...
e2e-tests:
needs:
- build-dev-images
runs-on: ubuntu-latest
container:
image: kirekov/e2e-tests:${{needs.build-dev-images.outputs.image_tag}}
volumes:
- /var/run/docker.sock:/var/run/docker.sock
steps:
- name: Run E2E-tests
run: |
cd /app
./gradlew :e2e-tests:test
The container.image
specifies the version of E2E tests to run. The ${{needs.build-dev-images.outputs.image_tag}}
variable references to the one exposed by the build-dev-images
job on the previous step.
The volumes: /var/run/docker.sock:/var/run/docker.sock
is crucial. Because e2e-tests
images uses Testcontainers library to run another Docker containers. Mounting docker.sock
as the volume implements Docker Wormhole pattern. You can read more about it by this link.
build-prod-images
This step is almost the same as the build-dev-images
. You can find it in the repository.
Conclusion
As the result, we have configured the CI environment to run unit tests, integration tests, and E2E-tests for multiple business components (i.e. Gain-Service
and API-Service
) and external services (i.e. RabbitMQ, Redis). Testcontainers allows us to build comprehensive and solid pipelines. What's more exciting is that you don't have to own dedicated servers for E2E testing. Pure CI pipelines are sufficient!
I hope you liked the E2E-testing approach I proposed. If you have any questions or suggestions, please leave your comments down below. Besides, you can always text me directly. I'll be happy to discuss the topic. Thanks for reading!
Resources
- Repository with the source code
- Apache Spark, Hive, and Spring Boot Testing Guide
- Spring Boot Testing — Testcontainers and Flyway
- Spring Boot Testing — Data and Services
- Spring Data JPA — Clear Tests
- A Deep Dive into Unit Testing
- Getting Integration Testing Right
- SLF4J
- Logback
- Kibana
- Patterns for running tests inside a Docker container
Top comments (0)