Running Android Instrumented Tests on CI - from Bitrise.io to GitHub Actions

It's almost 2020 and it remains a challenge to run Android Instrumented tests on CI, especially for opensource projects and small teams:

• Setting up and managing an in-house device farm is expensive and requires long-term investment and on-going infrastructure support - not a viable option for most teams.
• Cloud-based device farms such as Firebase Test Lab are a much more cost-effective solution for most teams as they take care of managing the infrastructure while offering simple APIs for integrating with existing CI pipelines. However, for opensource projects with a large amount of tests, free plans offered by these services likely won't cover the needs, while paying for such service is usually hard to justify for an opensource effort.
• For small teams who are not quite ready to invest in a fully-fledged cloud-based testing service, running tests on the Android Emulator within a docker container has been the most viable and common approach. But today most cloud-based CI services are still lacking hardware acceleration support from the host VM, which is the no.1 blocker for running tests on modern Android Emulators (especially on recent API levels) on CI.

I recently released FlowBinding which has over 160 instrumented tests across 10 library modules. I wanted to be able to run all these tests on each PR / merge into master without worrying about things like usage limits and quotas. So Firebase Test Lab is not an option as the free plan has a daily limit of 10 test runs on virtual devices and 5 test runs on physical devices. Instead I needed something that checks these boxes:

• Free for opensource projects.
• Supports configuring the Android Emulator system images used - API level, target: (default or google_apis), arch / ABI (x86 or x86_64).
• Supports running emulators in headless mode (as of Emulator 29.2.7 Canary this is equivalent to running emulator -no-window).
• Supports running modern x86 and x86_64 based emulators with hardware acceleration (KVM) enabled for better performance.
• Provides enough RAM for running both Gradle and an instance of Emulator.

In this post I'll share my journey on finding the best solution for running Android Emulators on CI for opensource projects.

Hardware-accelerated Emulators

Out of those requirements mentioned earlier, enabling hardware acceleration support on the host VM has been the most difficult one to meet.

The old ARM-based emulators were slow. But more importantly ARM-based system images are no longer supported by Google, as only x86 and x86_64 images are provided for recent API levels.

The modern Intel Atom (x86 and x86_64) emulators intend to provide much better performance with GPU hardware acceleration. However, installation of special software (HAXM on Mac & Windows, QEMU on Linux) is required for enabling hardware acceleration support. This presents a challenge on CI as to be able to run hardware accelerated emulators within a docker container, KVM must be enabled by the host VM which isn't the case for most cloud-based CI providers due to infrastructural limits.

This means currently we are not able to run hardware-accelerated emulators on some of the most popular cloud-based CI services such as CircleCI and Travis.

Google wants to help

It's worth to note that recently Google has been putting more efforts into offering better support for running Emulators on CI.

• Earlier this year the Android Emulator Team added support for running the emulator engine in headless mode to reduce some of the implicit expectations on the host machine running the emulator in a CI environment. This feature has been unified with the -no-window mode in a recent release.
• Google recently published a blog post highlighting some of the new tools they've been working on to improve deployability and debuggability of the Emulators on CI. They also opensourced a bunch of scripts for running Emulators in (Docker) containers.
• There was a dedicated session titled Emulator in a Continuous Integration (CI) Environment at Android Developer Summit 2019.
• There is also Project Nitrogen which is a larger effort on testing introduced back at Google IO 2018, but there hasn't been any public announcement on its progress this year.

It's encouraging to see Google's interest and commitment to continuously improve the experience of running Emulators on CI, but as of today the KVM dependency is still the main hurdle that prevents most users from being able to leverage these new tools and improvements.

Bitrise.io

bitrise.io is a CI/CD platform dedicated for Mobile. It has its own ecosystem including an online Workflow Editor and an extensive library of workflow steps, which probably appeal to new teams / projects starting out with basic Mobile CI needs.

But what got me excited were the provided steps for running Android UI tests (both physical and virtual devices), and KVM support in the host environment.

Using Bitrise Steps

Bitrise provides the Android Build for UI Testing and Virtual Device Testing for Android steps which use Firebase Test Lab to run tests for the chosen module / build variant with no limit on device hours / no. of test executions. But there are a couple of limitations:

• Only 1 build variant from 1 module can be run for each step. This means if you have multiple library modules with instrumented tests, you'll have to manually configure a Android Build for UI Testing or Virtual Device Testing for Android step for each module (or setup parallel workflows for running these in parallel if you wish to pay for additional containers).
• Running tests in a library module doesn't work unless an app APK is also present. The workaround is to also run app:assembleDebug for your library module tests.

You could also use the AVD Manager and Wait for Android emulator steps to spin up an Emulator on Bitrise locally and then run all your tests with Gradle, but you don't have full control over the Emulator configs and the specific SDK components needed for your workflow.

Custom workflow

Since Bitrise's host VMs have KVM enabled, we can easily setup a custom Workflow to control exactly what SDK components we want to download / update, how we want to configure and start an Emulator by providing with a custom shell script and running tests with regular Gradle commands.

Here's a repo showing how to setup such custom workflow on Bitrise.

At this point I was finally able to run Android instrumented tests on CI for FlowBinding on every PR!

Getting greedy

Being able to run unlimited instrumented tests for free on CI is not something to be taken for granted. But as more modules and tests were being added to FlowBinding, build time for the Bitrise workflow also significantly increased.

When I first integrated the Bitrise workflow, FlowBinding had 2 library modules with 9 instrumented tests; the entire Bitrise workflow took about 10 mins (this includes running a custom script to configure and spin up an Emulator, and running the tests with ./gradlew connectedCheck).

By the time I was ready to publish the first version of FlowBinding, there were 10 library modules with a total of 160 instrumented tests; the entire Bitrise workflow was taking over 30 mins.

While the number of modules and tests are unlikely to significantly increase for this project, an above 30 mins build time on CI for each PR is still quite an inconvenience.

Besides the long build time, I also wasn't very comfortable with the recommended way of configuring workflows with the online editor and having to rely on those built-in steps provided by Bitrise. Relying on the platform and ecosystem makes it harder to port our pipelines to potentially better alternatives in the future.

Given these concerns and dissatisfactions, I was set out to continue the search for better experience for running instrumented tests on CI.

GitHub Actions

GitHub Actions recently added support for CI/CD, and is free for public repositories. As a result many opensource projects on GitHub are starting to migrate their CI/CD workflows to GitHub Actions which has first-class integration with GitHub itself and strong infrastructure support from Azure.

The first thing I tried was running a hardware-accelerated Emulator within a docker container on a Linux / Ubuntu VM. Unfortunately (and perhaps unsurprisingly) KVM is not enabled on the host machines.

But GitHub Actions also offer free macOS machines for public repositories, and these macOS VMs also have HAXM (Hardware Accelerated Execution Manager) installed. This is promising as we should be able to setup a GitHub workflow that installs the required SDK components and spins up a hardware-accelerated Emulator directly from the host machine, and finally run our tests on the Emulator!

Custom action on macOS

Setting up a workflow that runs on macOS though is not as easy as doing the same with Docker, as our base environment is no longer a docker image with all the tools installed and configured and we'll need to port our scripts to work in the macOS environment. The workflow would need to do the following:

• Install / update the required Android SDK components including build-tools, platform-tools, platform (for the required API level), emulator and system-images (for the required API level).
• Create a new instance of AVD with various configuration options such as the API level, CPU architecture / ABI, and device profile used.
• Launch a new instance of Emulator from the AVD created.
• Wait until the Emulator is booted and ready for use.
• Finally execute a Gradle command to run the tests - e.g. ./gradlew connectedCheck.

We can package this process into a custom GitHub Action and provide a bunch of configuration options for the consumer.

A workflow that uses the custom action (Android Emulator Runner) to run instrumented tests on API 29 may look something like this:

jobs:
  test:
    runs-on: macOS-latest
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: 29
        script: ./gradlew connectedCheck

Outcome

It was time to run the custom GitHub Action with FlowBinding, and the result was promising!

The entire GitHub workflow took around 20 mins, while the test suites can now be run against multiple API-levels in parallel (maximum concurrent macOS jobs for public repo is 5 per user/org).

This is more than 30% improvements over the previous solution with Bitrise.

On paper the specs of the build machines provided by GitHub Actions and Bitrise are similar - 2-core CPU / 7GB RAM. So I'm not really sure what the cause of the improvement is. One possibility is being able to run everything directly on the host VM without the virtualization overhead in a docker-based environment, but that depends on many factors.

Leveraging build matrix

One of my favourite features of GitHub Actions is Build Matrix, which provides an easy way to run the same job across multiple configurations.

For example, to run instrumented tests on API 21, 23, 29 for both x86 and x86_64 ABIs, we can define the values for each configuration option under matrix and reference these values in the inputs of the action:

jobs:
  test:
    runs-on: macOS-latest
    strategy:
      matrix:
        api-level: [21, 23, 29]
        arch: [x86, x86_64]
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.api-level }}
        arch: ${{ matrix.arch }}
        script: ./gradlew connectedCheck

The workflow will run the job with all 6 configurations:

• API 21, x86
• API 21, x86_64
• API 23, x86
• API 23, x86_64
• API 29, x86
• API 29, x86_64

We can also filter out certain configurations generated by the build matrix. If we only want to run the job with (API 21, x86) and (API 29, x86_64), we can easily define the configurations we don't want with exclude:

jobs:
  test:
    runs-on: macOS-latest
    strategy:
      matrix:
        api-level: [21, 29]
        arch: [x86, x86_64]
        exclude:
          - api-level: 21
            arch: x86_64
          - api-level: 29
            arch: x86
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.api-level }}
        arch: ${{ matrix.arch }}
        script: ./gradlew connectedCheck

This will only generate 2 jobs:

• API 21, x86
• API 29, x86_64

You can even go nuts (please don't) and run your tests against all combinations of the configurations available, which when executed will generate a total number of 9 API levels (API 21 to 29) x 2 targets (default, google_apis) x 2 CPU / ABI (x86, x86_64) = 36 jobs.

🙃

Fun fact - while playing with build matrix I discovered that there is no system image available for android-27;google_apis;x86_64.

Update - a response from Google re. the missing system image:

For ‘android-27;google_apis;x86_64’, there may not be a good reason for it missing. That was in a transitional period where we just moved to x86_64 kernels with x86 32-bit userspace, and we needed to manage the extra complexity from there and never got around to making the x86_64 userspace version. We should have a x86 64-bit userspace version of it at some point, though, but it's not currently on our list of stuff to do (Sorry).

Build Cache?

One of the things I love the most about CircirCI is its native support for caching. Since we are able to cache the entire ~/.gradle directly to get incredibly fast incremental builds with CircleCI, I wondered if I could do something similar to my Github workflow to make it even faster.

Turned out there is this actions/cache provided by GitHub for caching dependencies and build outputs.

We can cache the ~/.gradle/caches directory by adding the following step to the job:

- uses: actions/cache@v1
  with:
    path: ~/.gradle/caches
    key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}-${{ hashFiles('**/gradle/wrapper/gradle-wrapper.properties') }}
    restore-keys: ${{ runner.os }}-gradle-

Unfortunately this won't work right now as there's a 200 MB limit on the cache size which isn't enough for Gradle, although they are planning to increase the limit to 2GB in the future.

Update: per-cache size limit has been increased to 2GB so the caching config above should now work.

Journey never ends

While the custom GitHub Action has been working really well for FlowBinding since the migration, I'm still hoping to find a more generic and future-proof solution where everything runs within a docker container rather than directly on a macOS VM. It's important for me to be able to easily try out / migrate to other CI/CD providers in the future, and using a workflow that runs things directly on a macOS VM instead of a standard Docker environment would likely make it harder to do so.

I recently came across a less-known cloud-based CI provider with native KVM support, and is completely free for opensource projects. In a future article I'll explore and share my experience with this product and compare it with my current CI workflow.

Stay tuned!

Featured in Android Weekly #390.