Improving Compose snapshot tests with Paparazzi

#android #compose #testing #snapshottesting

Update:

Since writing this article I've moved on to yet another improved way of doing this. Instead of generating code with KotlinPoet, it automatically runs Paparazzi tests for each Compose Preview. Check it out in this article:
https://proandroiddev.com/no-emulator-needed-for-screenshot-tests-of-compose-previews-on-ci-d292fa0eb26e

Background

I previously wrote about snapshot testing Compose with Shot here:
https://medium.com/proandroiddev/oh-snap-966bd786b7a4

One problem with these tests is they take too long to run. Starting an actual emulator and running the tests on this emulator in our CI environment takes around 30 minutes. This is too long to include it in our continuous integration pipeline which runs on every pull request. Instead we have a nightly job that runs the snapshot tests and reports to our Teams channel if any of them fail.

This approach has some major drawbacks

You are no longer "forced" to fix your failing snapshot tests before you merge
You might discover failing tests too late - even after the changed code has been released
Someone has to take the time to check the Teams channel every morning and fix the tests

Paparazzi to the rescue

Paparazzi is a library which can render your application screens without a physical device or emulator. Not having to deal with the emulator is a massive time saver.

Back when we initially implemented our snapshot tests, Paparazzi was not an option because it did not support Compose. This all changed recently in the 1.0.0 release.

Paparazzi, like Shot, is quite easy to set up. A Paparazzi test looks something like this:



class AlertViewPaparazziTest : PaparazziTest() {

    @get:Rule
    val paparazziRule: Paparazzi = Paparazzi(
        theme = "android:Theme.MaterialComponents.Light.NoActionBar",
        deviceConfig = DeviceConfig.NEXUS_5.copy(softButtons = false, screenHeight = 1),
        renderingMode = SessionParams.RenderingMode.V_SCROLL
    )

    @Test
    fun alert() {
        paparazziRule.snapshot { AlertPreview() }
    }
}

Again like Shot, Paparazzi has one gradle task for recording:
$ ./gradlew sample:recordPaparazziDebug
And one for verifying:
$ ./gradlew sample:verifyPaparazziDebug

Unfortunately Paparazzi does not (yet) support taking a screenshot with the exact size of the Composable: https://github.com/cashapp/paparazzi/issues/383
The screenHeight = 1 and renderingMode = V_SCROLL is a trick to get the minimum height needed to take a screenshot of the Composable. Thanks https://github.com/hrach!
There is no similar trick for the width (that I know of, please let me know if you know one!), so we always get the full width of the device which leads to some screenshots looking like this (if you click it you see the full device width):

Though it would be nicer without the full width, this is not a big problem. The main point is to compare the current situation to the previously recorded snapshot and see that there are no unintended changes. This works just as well with a full width.

Rewrite all the tests

To really compare Paparazzi with Shot and see if it solved our problems, we had to rewrite all our snapshot tests. This would be quite boring and repetitive, so we decided to try to generate them instead of writing them manually.

We always write one snapshot test per @Preview annotation. Because of this, all our snapshot tests look pretty much the same,
Our idea was something like this:

Find every file that has at least one @Preview annotation
Create a snapshot test file for it
Create one test per @Preview in that file

To do this we created a Kotlin program with a main function and used another library, also from Square, KotlinPoet (https://github.com/square/kotlinpoet) to generate the code.

Disclaimer: This code is still quite rough, but you get the idea and it does what we needed. I am sure it could be improved further.



/**
 * Finds all files in the components module which have Compose previews
 * and generates Paparazzi screenshot tests for them.
 *
 * The generated tests can then be used to record screenshots with
 * ./gradlew components:recordPaparazziInternalDebug
 *
 * To verify that the current implementation matches the recorded screenshots
 * ./gradlew components:verifyPaparazziInternalDebug
 */
fun main() {
    val path = System.getProperty("user.dir") ?: error("Can't get user dir")

    // Paparazzi does not currently work in the app module: https://github.com/cashapp/paparazzi/issues/107
    // For now this is hardcoded to only check files in the components module.
    // If we pull our compose files out of the app module to a separate module this code has to be updated.
    File(path).walk().filter { it.path.contains("/components/src/main/java") && it.extension == "kt" }.forEach {
        if (it.readText().contains("@Preview")) {
            processFileWithPreviews(it)
        }
    }
}

/**
 * Reads the given file, finds the names of all the functions annotated with @Preview
 * and uses them to generate a Paparazzi test file with one test for each preview.
 */
private fun processFileWithPreviews(file: File) {
    val lines = file.readLines()
    val previewNames = mutableListOf<String>()
    var saveNextFunctionName = false
    var packageName = ""
    lines.forEachIndexed { i, line ->
        if (i == 0) {
            packageName = line.split(" ").last()
        }
        if (line.contains("@Preview")) {
            saveNextFunctionName = true
        }
        if (saveNextFunctionName && line.startsWith("fun ")) {
            previewNames += line.split(" ")[1].removeSuffix("()")
            saveNextFunctionName = false
        }
    }
    val pathString = file.path.replace("src/main", "src/test").split("java").first() + "java"
    val testFilePath = pathString.toPath()
    generatePaparazziTest(packageName, file.nameWithoutExtension + "PaparazziTest", testFilePath, previewNames)
}

fun generatePaparazziTest(packageName: String, fileName: String, path: Path, previewNames: List<String>) {
    val classBuilder = TypeSpec.classBuilder(fileName)
        .superclass(PaparazziTest::class)
        .addAnnotation(
            AnnotationSpec.builder(Suppress::class)
                // KotlinPoet does not let us remove redundant public modifiers or Unit return types for the functions,
                // but we don't mind for generated code as long as the tests work
                .addMember("\"RedundantVisibilityModifier\", \"RedundantUnitReturnType\"")
                .build()
        )
    previewNames.forEach {
        classBuilder.addFunction(
            FunSpec.builder(it.removeSuffix("Preview").usLocaleDecapitalize())
                .addStatement("paparazziRule.snapshot { $it() }")
                .addAnnotation(Test::class)
                .build()
        )
    }
    val testFile = FileSpec.builder(packageName, fileName)
        .addType(classBuilder.build())
        .addFileComment("AUTO-GENERATED FILE by generate_paparazzi_tests.kt\nDO NOT MODIFY")
        .build()
    val nioPath = path.toNioPath()
    testFile.writeTo(nioPath)
}

Since we always use the same device specs and Paparazzi rule, we created a parent class and the program above generates sub-classes for that:



open class PaparazziTest {

    @get:Rule
    val paparazziRule: Paparazzi = Paparazzi(
        theme = "android:Theme.MaterialComponents.Light.NoActionBar",
        deviceConfig = DeviceConfig.NEXUS_5.copy(softButtons = false, screenHeight = 1),
        renderingMode = SessionParams.RenderingMode.V_SCROLL
    )
}

As you can see from one of the comments, Paparazzi does not yet support running tests in the app module: https://github.com/cashapp/paparazzi/issues/107

In our project the composables live in two modules: :components for reusable components and :app for the screens and single use views. For now, our program only generates paparazzi tests for the :components module. We would have to move all our composables out of the app module which can turn into a bigger refactor. This recent issue gives some hope that it could be fixed soon and we might not need to do that: https://github.com/cashapp/paparazzi/issues/524

Add it to continuous integration

After all the tests had been generated, we recorded the snapshots (60 of them in our :components module) and pushed them to our source control. We then added a new step to our CI pipeline which runs on every pull request. Just a simple gradle step that runs the verifyPaparazzi task. It is super fast:

Final thoughts

Paparazzi makes snapshot test verification fast enough to run in our CI pipeline on every pull request
Recording screenshots locally when adding new UI or changing existing UI is much faster without an emulator
Not having to deal with an emulator seems to solve a problem we had in the past where screenshots would be slightly different if they were recorded on a Mac with Apple Silicon or one with Intel
It is unfortunate that we can't use Paparazzi in the app module (yet)

We have decided to move forward with Paparazzi and will wait for the app module issue to be fixed or move our composables out of the app module.

Please let me know if you have any feedback or questions in the comments or on Twitter.

I leave you with an image of some of our components recorded by Paparazzi: