Parallel Unit Tests in Android

**TL;DR* Our unit tests were taking a lot of time to run and there was no way to shard them and run in parallel. We found a way to use JUnit Categories and custom gradle command to shard our tests and reduce test execution time by half.*

Unit tests are the fundamental tests in our app testing strategy. But the more tests we write, the more time it takes to execute them and for a code to merge.

Android apps have various layers and we can test each layer using different tools. But we can categorize these tests into 3 buckets.

1. Unit Test (fast, runs on JVM)
2. UI test (slow, runs using an emulator)
3. Robolectric test (moderate, runs on JVM with simulated android env)

JVM tests usually run super fast but on android, all unit test runs Sequentially, the more tests we write, the longer it will take to execute them.

With project Nitrogen, Robolectric tests are now supported in the Android X testing library. Robolectric tests run on JVM and are faster then espresso tests. But since they depend on simulated android env they are slower than vanilla JUnit tests.

Combining Robolectric and JUnit tests makes test execution slow and can impact build time and developer productivity. Gradle has support for executing tests in parallel using a custom setting like this:

tasks.withType(Test) {
        maxParallelForks = Runtime.runtime.availableProcessors()
    }

But this option is rigid and can cause a crash in our build machine.

Modern CI systems come with support for parallel builds. To get most out of our CI, we need a way to Shard our tests and execute them in parallel. For ex: we can divide our unit tests into 2 parts (Robolectric and pure JVM) and run them in parallel cutting test execution time in half.

Sharding is splitting the test suite into multiple threads, so they run as a group.

Unfortunately Android and Gradle don't have an easy way to shard or break tests. In this post, we will discuss 2 different approaches we can use to achieve this.

Command line

Command-line parameters are the most simple option available with gradle, when it comes to executing 1 test, all tests in a class or a package.

Let's take an example test class

package com.dexterapps.testsharding

class ConverterUtilTest {

    @Test
    fun testConvertFahrenheitToCelsius() {
        val actual = ConverterUtil.convertCelsiusToFahrenheit(100F)
        val expected = 212f

        assertEquals("Conversion from celsius to fahrenheit failed",
            expected.toDouble(), actual.toDouble(), 0.001)
    }
    @Test
    fun testConvertCelsiusToFahrenheit() {
     // test code here
    }
}

Executing Single Test

We can execute single test like testConvertFahrenheitToCelsius with this command

 ./gradlew :ProjectName:testVariantNameUnitTest --tests com.dexterapps.testsharding.ConverterUtilTest.testConvertFahrenheitToCelsius

Relpace :ProjectName:testVariantNameUnitTest with yours. Like :app:testDebugUnitTest or the variant you will like to test.

Executing all tests in class

To run all tests in a class ConverterUtilTest we can run

./gradlew :ProjectName:testVariantNameUnitTest --tests com.dexterapps.testsharding.ConverterUtilTest

Executing all tests in a package

To run all tests in a package com.dexterapps.testsharding we can

./gradlew :ProjectName:testVariantNameUnitTest --tests com.dexterapps.testsharding.*

Command line options work for a lot of cases but as a downside, each engineer would need to set up there own Android Studio run config or shell script.

This approach doesn't Shard the tests. Let's say if we want to separate Robolectric and JVM tests, we need to use specific packages, forcing everyone to put tests in diff package from where they belong canonically.

Can we do better? Yes, we can, Let's look at another approach that gives us the flexibility we need.

JUnit Categories

JUnit 4.12 introduced a nifty feature Categories. Categories provide us with a mechanism to label and group tests and run these tests either by including or excluding the categories.

JUnit categories are simple but there is no direct support for it in Gradle or Gradle Android. We need to write a custom Gradle code to make it work. Let's look at the code.

You can download all the code discussed in this post from
https://github.com/pranayairan/android-unit-test-sharding

Marker Interface

To represent the categories or label tests, we need to create marker interfaces. This simple interfaces will be will use to classify our tests and run them in parallel.

interface RobolectricTests

interface UnitTests

interface FastTests

interface SlowTests

Note: JUnit categories can take any class name as a category, it is
not required to create custom interfaces. We can use any predefine
classes as well to categorize tests.

Category Example

Once we have marker interfaces, it is trivial to add them as categories. To categorize a test annotate it with @Category annotation and add interface name. Let's look at some code.

 @Test
 @Category(UnitTests::class)
 fun testConvertFahrenheitToCelsius() {
     val actual = ConverterUtil.convertCelsiusToFahrenheit(100F)
     val expected = 212f

     assertEquals("Conversion from celsius to fahrenheit failed",
         expected.toDouble(), actual.toDouble(), 0.001)
 }

We can add many categories to single method or add categories at class level.

 @Category(FastTests::class)  
 class ConverterUtilTest {
    @Test  
    @Category(FastTests::class,UnitTests::class)  
    fun testConvertCelsiusToFahrenheit() {
    }
 }

@Category annotation is part of JUnit experiemental package
org.junit.experimental.categories.Category

Running the category tests

There is no easy way to run category tests on Android. We can add support for Categories in the Android gradle plugin by writing custom gradle code.

Let's look at the code to execute tests with category robolectric

if (project.gradle.startParameter?.taskRequests?.args[0]?.remove("--robolectric")) {
    for (Project project : subprojects) {
        project.tasks
                .withType(Test)
                .configureEach {
                    useJUnit {
                        includeCategories 'com.dexterapps.testsharding.RobolectricTests'
                        //excludeCategories if we want to exclude any test
                    }
                }
    }
}

We need to add this code to root build.gradle file to make it work for all modules. We can also enable it for specific module. To enable for specific module check this code

Let's walk through this code line by line.

1. First, we check if a gradle command is executed with the --robolectric parameter. If yes we remove the parameter and proceed. We need to call remove because --robolectric is a custom parameter and gradle doesn't understand it.
2. If we find this parameter we will instruct JUnit to include all tests with Category com.dexterapps.testsharding.RobolectricTests and ignore other tests. We can also use excludeCategories to do the reverse.

To run Unit tests with @Category(RobolectricTests::class) using following command.

./gradlew :app:testDebugUnitTest --robolectric

Similarly we can run other category tests with

./gradlew :app:testDebugUnitTest --unit
or
./gradlew :app:testDebugUnitTest --fasttest

Results

JUnit Categories enabled us to divide and run multiple test jobs in parallel. Before Categories our test execution time was 5 min. With Categories and parallel jobs, it takes only 3 min (Unit Test 2 min, Robolectric 3 min) giving us ~40% savings in test execution time.

Wrap up

The concept of test sharding is not new in android. UI testing frameworks like Spoon or Flank supported it for a long time. But sharding for a unit testing is non-existing.

This post covered the use of JUnit Categories and custom gradle code to break unit tests into different categories and run them in parallel. We achieved ~40% improvement in test execution time and improved developer productivity.

I created a sample android app that shard unit tests into 3 different categories. You can find the complete source code at https://github.com/pranayairan/android-unit-test-sharding

I would like to give shot out to my Colleague Martin Feldsztejn who wrote the custom gradle command to get categories to work in android. I would also like to thanks Sriram Santosh for proofreading this and providing his feedback

Post originally published on https://prandroid.dev/Parallel-unit-tests-in-android/