Testing Amazon S3 using s3Ninja

Introduction

AWS is now a common service being used in many web applications, being used for file uploads, storage, container registries, hosting, as a container orchestrator platform, among many other things.

As its usage grows, developers also have increasingly higher needs of testing the integration of such complex services with their applications, and this can be no easy task, as these services require a lot of configuration and are very complex on its own.

Thanks to the increase of containerization and popularity of docker, we are now also seeing a trend where the community is providing and maintaining docker images that facilitate the work of developers needing to test the integration of their application services with AWS services.

We will look at one of the most popular images, scireum/s3-ninja, how we can configure it, and how to setup a simple IT for one of the services in our application that depends on Amazon S3.

Basics of S3Ninja

Upon starting the docker image, here is what we see:

As stated in the image above:

S3 ninja emulates the S3 API for development and testing purposes.
It is however not intended as production system as it neither provides scalability nor replication or proper security.

Accessing the running container in a web browser, and if we navigate to, for instance, http://localhost:32782/ui/api we can see the supported API and what type of methods we can test.

While the container is running, we can configure, programatically, from the Java side, a "mock AmazonS3 client", by exposing the URL of the running container, and we can then use this mock client to perform our integration testing.

Configuring the mock S3 ninja

To configure the mock client, we need to spin up the docker container that has the S3Ninja image, and then use it to configure the client:

@Testcontainers
@ExtendWith({SpringExtension.class, MockitoExtension.class})
@SpringBootTest
class TestingAServiceThatRequiresUsingAmazonS3Test {

//Container configuration using the Testcontainers dependency to manage docker containers from test contexts
    @Container
    private static GenericContainer s3Ninja = new GenericContainer<>("scireum/s3ninja").withExposedPorts(9444);

(...)

private void setupAmazonS3ninja(String bucketName) {
        final AmazonS3 httpAmazonS3 = AmazonS3ClientBuilder.standard()
            .withEndpointConfiguration(new AwsClientBuilder.EndpointConfiguration(
                "http://" + s3Ninja.getContainerIpAddress() + ":" + s3Ninja.getMappedPort(9444) + "/",
                Region.EU_North_1.getFirstRegionId()
            )).build();

//Inject s3Ninja into the service for testing purposes  
  someServiceThatRequiresUsingS3.setupAmazonS3client(httpAmazonS3);
    }

The basic scaffold above leverages the Testcontainers maven dependency that automatically manages docker containers for use from a test context, and below that, we call a public method from our Spring managed service whose sole purpose is to "inject" the mock client into the service.

Here it's passed via an explicit method, and not using for example, constructor injection, since the mock S3 client is a regular dependency, not managed by Spring. By using a method like this, we can control between using real clients or mocked ones from a test context, while still retaining the flexibility to parameterize it for production usage.

With this simple setup, we can now write an actual test.

Writing a test

Leveraging the above scaffold, writing a test becomes very easy:

@Test
    void correct_setup_returns_good_response_from_service() {
        setupAmazonS3ninja("mytestBucketName");

        Optional<SomeResponseDTO>response = someServiceThatRequiresUsingS3.callAmethodThatUsesS3ClientInternally();

        assertTrue(response.isPresent());
    }

As we can see, once we setup the mock S3 client, from the running docker container, we can now effectively unit test our service method against a fixed s3 client that allows us to test exactly the business logic we need.

Conclusion

This post describes one possible approach to have a basic configuration to test services that rely on Amazon S3, by using S3 Ninja.

Thanks for reading!