Exploring Cirrus CI for Android

I recently wrote about my experience using various cloud-based CI services for running Android instrumented tests on CI for opensource projects, and how I landed on a solution using a custom GitHub Action.

As mentioned in that article, most cloud-based CI providers today don't have KVM enabled in the host VMs which is required for running hardware-accelerated x86 / x86_64 emulators in a docker environment.

I recently came across Cirrus CI, a less-known cloud-based CI provider with native KVM support and it's completely free for opensource projects. Despite having a positive experience with GitHub Workflow using my custom android-emulator-runner action that runs on macOS VMs, I was keen to try out KVM-enabled containers provided by Cirrus CI as I'd still be more comfortable with a solution based on a standard Linux/docker environment which Google has been putting more effort into recently.

In this article I'm going to share some of my experiences with Cirrus CI - specifically, how I migrated FlowBinding's instrumented tests from GitHub Action to Cirrus CI task, along with some of the features provided by Cirrus CI you might find useful for Android in general.

Cirrus CI Overview

Cirrus CI doesn't have a fancy product / marketing website like some of the other providers such as CircleCI. The main entry point is cirrus-ci.org which covers feature descriptions, pricing information, examples, and comprehensive documentations.

Features

A list of features can be found here. For Android (and opensource projects in particular) some of the more interesting ones are:

• Free for opensource projects (public repositories on GitHub)
• Provides Linux, Windows, macOS and FreeBSD containers
• Provides KVM-enabled containers
• Containers on community clusters (free plans) can use a maximum of 8.0 CPUs and up to 24 GB of memory
• Supports local and remote build cache
• Supports build matrix

Here's a comparison with other CI services found on Cirrus CI's website:

Pricing

Cirrus CI's pricing models can be found here.

As mentioned before, Cirrus CI is free for public repositories. There is a per-user concurrency limit of 8 Linux VMs, but this is quite generous for a free plan and should be more than enough for most projects.

Commercial plans for private repositories are available at $10/seat/month which is again quite affordable compared to some of the other more popular services, but for my projects I haven't needed to upgrade.

Installation

Cirrus CI currently only supports repositories hosted on GitHub. To get started, go to the Cirrus CI Application on GitHub Marketplace and setup a plan for your account or organization.

That's basically all you need to do to setup Cirrus CI for a project. Everything else (except a couple of security options) is configured in a .cirrus.yml file in your project's root directory.

Writing CI Tasks

Task(s) are the building block of a .cirrus.yml configuration file. A task defines a sequence of instructions to run and the environment to execute these instructions in. The equivalent concepts of tasks and instructions in GitHub Actions are jobs and steps respectively. Here's a simple task that assembles a debug APK:

assemble_task:
  container:
    image: reactivecircus/android-sdk:latest
  assemble_script:
    ./gradlew assembleDebug

Tasks in .cirrus.yml have the task suffix. In this case assemble_task is the only task we define in the config.

Script is one of the supported instructions. Similarly scripts need to have the script suffix e.g. assemble_script.

container is a field where we can define the docker image used for the task. reactivecircus/android-sdk is a docker image with the minimum Android SDK components required for building Android projects.

A comprehensive guide for writing tasks can be found here.

It's worth noting that code completion support from IntelliJ IDEA / Android Studio is available when writing .cirrus.yml, thanks to the YMAL plugin bundled in the IDE and the supported schema store.

Dashboard

Cirrus CI's build dashboard is hosted at cirrus-ci.com. Once logged in with your GitHub account you'll see a list of builds for all your projects using Cirrus CI. Individual tasks and build results are available for each build.

While the UIs don't look as pretty as some of the more popular products, overall I found them effective and responsive.

Running Instrumented Tests

Now let's look at how we can write a task that runs Android instrumented tests on hardware-accelerated emulators.

We'll start by converting the following GitHub workflow that uses the android-emulator-runner action for running instrumented tests on macOS:

jobs:
  test:
    runs-on: macOS-latest
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: 28
        arch: x86_64
        target: default
        emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim
        script: ./gradlew connectedDebugAndroidTest

Converted task in .cirrus.yml:

connected_check_task:
  name: Run Android instrumented tests
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  env:
    API_LEVEL: 28
    TARGET: default
    ARCH: x86_64
  container:
    image: reactivecircus/android-emulator-28:latest
    kvm: true
    cpu: 8
    memory: 24G
  create_device_script:
    echo no | avdmanager create avd --force --name "api-${API_LEVEL}" --abi "${TARGET}/${ARCH}" --package "system-images;android-${API_LEVEL};${TARGET};${ARCH}"
  start_emulator_background_script:
    $ANDROID_HOME/emulator/emulator -avd "api-${API_LEVEL}" -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -camera-back none
  wait_for_emulator_script:
    adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 3; done; input keyevent 82'
  disable_animations_script: |
    adb shell settings put global window_animation_scale 0.0
    adb shell settings put global transition_animation_scale 0.0
    adb shell settings put global animator_duration_scale 0.0
  run_instrumented_tests_script:
    ./gradlew connectedDebugAndroidTest

name is a field where we can define a custom name for the task.

only_if is a keyword that controls whether the task should be executed. $CIRRUS_BRANCH == "master" || CIRRUS_PR != "" here means we only want to run this task on commits on the master branch or any pull request.

Environment variables can be specified in the env block. Here we are only specifying the configurations of the system image used for creating the emulator. Other common environment variables such as JAVA_TOOL_OPTIONS or GRADLE_OPTS can also be specified here.

container defines the host VM used for running the task and its configurations. We use reactivecircus/android-emulator-28 as the docker image which has the minimum SDK components for running hardware-accelerated emulator on API 28.

The most interesting bit here is kvm: true where we enable KVM for the container to take advantage of native virtualization when running the modern x86 / x86_64 emulators.

We've also given the container the maximum amount of CPU (8) and memory (24G) as the container will be running an Android Emulator instance as well as Gradle commands, both of which are known to be processor and memory intensive.

create_device_script and wait_for_emulator_script create a new AVD instance, launch an Emulator, and wait until it's fully booted and ready for use. These are not needed in the GitHub workflow as they are taken care of by the android-emulator-runner Github Action.

We then add the disable_animations_script to disable the global animations on the emulator. This is again run by default when using the android-emulator-runner GitHub Action.

Finally in the run_instrumented_script we define the Gradle task for running all Android tests in all modules.

Here's the build summary for FlowBinding which has 160 tests across 10 modules:

The build took ~20 mins which is almost identical to the GitHub Actions equivalence. Note that there's an extra 1-2 mins of scheduling period for KVM-enabled containers due to the additional virtualization layer.

Build matrix

With GitHub Actions we can leverage build matrix to run the tests across multiple API levels:

jobs:
  test:
    runs-on: macOS-latest
    strategy:
      matrix:
        api-level: [21, 22, 23, 24, 25, 26, 27, 28]
    steps:
    - name: checkout
      uses: actions/checkout@v2

    - name: run tests
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.api-level }}
        script: ./gradlew connectedDebugAndroidTest

We can do the same with Cirrus CI using matrix modification:

connected_check_task:
  name: Run Android instrumented tests (API $API_LEVEL)
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  env:
    matrix:
      - API_LEVEL: 21
      - API_LEVEL: 22
      - API_LEVEL: 23
      - API_LEVEL: 24
      - API_LEVEL: 25
      - API_LEVEL: 26
      - API_LEVEL: 27
      - API_LEVEL: 28
  container:
    image: reactivecircus/android-emulator-${API_LEVEL}:latest
    kvm: true
    cpu: 8
    memory: 24G
...

We've defined a matrix of API_LEVEL values so the same connected_check_task will be executed for each of these values. Note that we are able to specify the name of the docker image dynamically because these images are named in the same reactivecircus/android-emulator-${API_LEVEL} pattern. The repository for these images can be found here (all x86 images for API 21 and above are available).

With GitHub Actions you can exclude some of the configurations generated by the build matrix. This is not currently supported by Cirrus CI.

Build cache (local)

Many Gradle tasks are incremental and cacheable, which can significantly improve build times.

The local Gradle build cache is located at ~/.gradle/cache and we are able to get fast incremental builds on CI when tasks outputs are available in this directory.

With CircleCI saving and restoring the local Gradle cache directory is easy:

jobs:
  build:
    executor: android
    steps:
      - checkout
      - restore_cache:
          key: gradle-{{ checksum "buildSrc/dependencies.gradle" }}-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}
      - run:
          name: Assemble
          command: ./gradlew assemble
      - save_cache:
          key: gradle-{{ checksum "buildSrc/dependencies.gradle" }}-{{ checksum "gradle/wrapper/gradle-wrapper.properties" }}
          paths:
            - ~/.gradle/cache

The cache will only be invalidated when buildSrc/dependencies.gradle or gradle/wrapper/gradle-wrapper.properties changes which is usually when we update our library dependencies or Gradle version. By doing this we can avoid re-downloading all the dependencies on each build on CI.

Cirrus CI also provides a similar cache instruction that works similarly:

connected_check_task:
  name: Run Android instrumented tests
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  env:
    API_LEVEL: 28
    TARGET: default
    ARCH: x86_64
  container:
    image: reactivecircus/android-emulator-28:latest
    kvm: true
    cpu: 8
    memory: 24G
  gradle_cache:
    folder: ~/.gradle/caches
    fingerprint_script: cat buildSrc/dependencies.gradle && cat gradle/wrapper/gradle-wrapper.properties
  ...
  cleanup_script:
    - rm -rf ~/.gradle/caches/[0-9].*
    - rm -rf ~/.gradle/caches/transforms-1
    - rm -rf ~/.gradle/caches/journal-1
    - rm -rf ~/.gradle/caches/jars-3/*/buildSrc.jar
    - find ~/.gradle/caches/ -name "*.lock" -type f -delete

The Cirrus CI doc recommends using a technique for avoiding re-upload of the cache when the build outputs are unchanged - adding a cleanup_script that removes the non-deterministic files generated on each build at the end of the task. This improvement is based on the expectation that re-generating these non-deterministic files and always executing a few tasks on every build has much less impact than uploading hundred of Mb of the cache files on every build.

But there's one area where CircleCI performs significantly better - the time taken for downloading and uploading these cache archives.

For roughly 500Mb of cache archive, with CircleCI it only takes about 15 seconds to download (restore) it, and 25 seconds to upload it.

With Cirrus CI, downloading ~500Mb of cache archive takes nearly 1 minute while uploading takes over 1 minute.

Build cache (remote)

Gradle also has built-in support for HTTP remote build cache which works similarly as a local build cache except the build outputs come from a remote server rather than the local ~/.gradle/cache directory.

Cirrus CI is the first CI service I've used that provides native support for Gradle's remote HTTP build cache. This can be configured by adding the following to your settings.gradle file:

ext.isCiServer = System.getenv().containsKey("CIRRUS_CI")
ext.isMasterBranch = System.getenv()["CIRRUS_BRANCH"] == "master"
ext.buildCacheHost = System.getenv().getOrDefault("CIRRUS_HTTP_CACHE_HOST", "localhost:12321")

buildCache {
  local {
    enabled = !isCiServer
  }
  remote(HttpBuildCache) {
    url = "http://${buildCacheHost}/"
    enabled = isCiServer
    push = isMasterBranch
  }
}

Note that if your project uses the buildSrc directory, the build cache configuration should also be applied to buildSrc/settings.gradle. This can be done by putting the build cache configuration above into a separate gradle/buildCacheSettings.gradle file and applying it to both settings.gradle and buildSrc/settings.gradle.

Optimizing script execution order

Before looking at the final build time improvement, let's take a look at a technique for optimizing your CI pipeline when running instrumented tests on an Emulator.

Here's a visualization of our current connected_check_task:

We create a new AVD instance, launch an Emulator in the background, wait until the Emulator is fully booted, and finally run the ./gradlew connectedDebugAndroidTest task.

Is there anything else we could be doing while waiting for the Emulator to come online (usually takes about 1 minute)?

As mentioned earlier many Gradle tasks are incremental, this means that once a task has been executed, its outputs will be stored in the build directory within the sub-project (module) and running the same task again (without changing its inputs) won't actually execute the task itself. Instead the outputs of the task will be immediately loaded from the build directory.

The assemble<BuildVariant>AndroidTest task that generates the test APKs is incremental.

Let's first run it locally with no build cache:

./gradlew clean assembleDebugAndroidTest --no-build-cache
...
BUILD SUCCESSFUL in 59s
936 actionable tasks: 934 executed, 2 up-to-date

The build took about 1 minute with 934 out of 936 tasks executed.

If we run it again without cleaning the build directory:

./gradlew assembleDebugAndroidTest
...
BUILD SUCCESSFUL in 3s
914 actionable tasks: 914 up-to-date

The build completes in 3 seconds and no tasks were executed as they are all up-to-date.

Now since the connectedDebugAndroidTest Gradle task we run at the end effectively depends on the assembleDebugAndroidTest task for generating the test APKs, by first running the assembleDebugAndroidTest task explicitly while waiting for the Emulator to boot, the connectedDebugAndroidTest task will be significantly faster as all of its sub-tasks would be up-to-date at that point.

Final result

Here's the final .cirrus.yml file:

connected_check_task:
  name: Run Android instrumented tests (API $API_LEVEL)
  timeout_in: 30m
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  env:
    matrix:
      - API_LEVEL: 21
      - API_LEVEL: 22
      - API_LEVEL: 23
      - API_LEVEL: 24
      - API_LEVEL: 25
      - API_LEVEL: 26
      - API_LEVEL: 27
      - API_LEVEL: 28
    JAVA_TOOL_OPTIONS: -Xmx6g
    GRADLE_OPTS: -Dorg.gradle.daemon=false -Dkotlin.incremental=false -Dkotlin.compiler.execution.strategy=in-process
  container:
    image: reactivecircus/android-emulator-${API_LEVEL}:latest
    kvm: true
    cpu: 8
    memory: 24G
  create_device_script:
    echo no | avdmanager create avd --force --name "api-${API_LEVEL}" --abi "${TARGET}/${ARCH}" --package "system-images;android-${API_LEVEL};${TARGET};${ARCH}"
  start_emulator_background_script:
    $ANDROID_HOME/emulator/emulator -avd "api-${API_LEVEL}" -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -camera-back none
  assemble_instrumented_tests_script:
    ./gradlew assembleDebugAndroidTest
  wait_for_emulator_script:
    adb wait-for-device shell 'while [[ -z $(getprop sys.boot_completed) ]]; do sleep 3; done; input keyevent 82'
  disable_animations_script: |
    adb shell settings put global window_animation_scale 0.0
    adb shell settings put global transition_animation_scale 0.0
    adb shell settings put global animator_duration_scale 0.0
  run_instrumented_tests_script:
    ./gradlew connectedDebugAndroidTest

With remote build cache enabled, I managed to cut the build time to ~15 mins for FlowBinding. This is a significant improvement compared to the ~20 mins build time with GitHub Actions (mostly due to the unlimited build cache provided by Cirrus CI).

Other Features

After having a positive experience migrating the instrumented tests to run on Cirrus CI, I decided to explore some of the other features provided by the service and see how they work in broader Android CI use cases.

Task dependency

By default all tasks defined in .cirrus.yml run in parallel. Task execution dependencies can be configured by using the depends_on keyword. In the following example, the publish_artifacts_task will only be executed after both the assemble_task and connected_check_task have completed successfully.

assemble_task:
  ...

connected_check_task:
  ...

publish_artifacts_task:
  depends_on:
    - assemble
    - connected_check
  deploy_snapshot_script:
    ...

Encrypted secrets

Sometimes a CI task might need access to some secrets. For example, building a release APK may require the encryption key for the release keystore, while publishing a library to Maven Central requires the Sonatype Nexus credentials.

Cirrus CI supports encrypted variables which can be safely added to the .cirrus.yml file.

To use a secret in a CI task, go to the Cirrus CI settings page and follow the instruction to generate the encrypted variable:

Then add the generated ENCRYPTED[xxx] as a regular environment variable in the env block of a task:

publish_artifacts_task:
  depends_on:
    - assemble_and_check
    - connected_check
  env:
    SONATYPE_NEXUS_USERNAME: ENCRYPTED[abcd]
    SONATYPE_NEXUS_PASSWORD: ENCRYPTED[1234]
  container:
    image: reactivecircus/android-sdk:latest
  deploy_snapshot_script:
    ./gradlew clean androidSourcesJar androidJavadocsJar uploadArchives --no-daemon --no-parallel

Artifacts

Cirrus CI supports storing build artifacts and making them available for download from the dashboard via the artifacts instruction. The following example produces the JUnit XML artifacts for the unit_tests_task:

unit_tests_task:
  container:
    image: reactivecircus/android-sdk:latest
    cpu: 8
    memory: 24G
  unit_test_script:
    ./gradlew test
  always:
    junit_artifacts:
      path: "**/test-results/**/*.xml"
      type: text/xml
      format: junit

Conditionally skipping builds

Sometimes you may want to skip a CI task if the there's no change in the source code (e.g. changing the README.md file). For example we might want to only run the connected_check_task if there are changes to the Kotlin, XML or Gradle sources in the commit or pull request. This can be implemented with the skip keyword and the changesInclude function:

connected_check_task:
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  skip: "!changesInclude('.cirrus.yml', '*.gradle', '*.gradle.kts', '**/*.gradle', '**/*.gradle.kts', '*.properties', '**/*.properties', '**/*.kt', '**/*.xml')"
  ...

Note that this is also supported by GitHub Actions via the path filters.

Manual trigger

A task can also be triggered manually from the Cirrus CI dashboard or the GitHub Checks page. This is often useful in deployment (e.g. publishing a new APK) where we need to be able to control when to trigger the task.

Manual task can be configured by adding trigger_type: manual to the task:

publish_artifacts_task:
  trigger_type: manual
  depends_on:
    - assemble_and_check
    - connected_check
  ...

Delayed execution - Required PR Labels

Tasks such as instrumented tests require special execution environment and have a longer feedback cycle, which make them more expensive to run. Therefore when a new PR is created we might want to run the fast checks (assemble and unit tests) immediately, and only trigger the expensive checks after an initial review.

Cirrus CI provides a feature called Required PR Labels. Tasks can specify a list of required_pr_labels to only get triggered once these labels have been added to the pull request.

In the following example the connected_check_task will only be run after the initial-review label has been added to the pull request.

connected_check_task:
  required_pr_labels: initial-review
  only_if: $CIRRUS_BRANCH == "master" || CIRRUS_PR != ""
  ...

On the GitHub Checks page, tasks which are waiting for PR labels are considered "neutral":

To trigger these tasks, apply the required initial-review label to the PR:

The connected_check_task will now be triggered.

Build notification

A major feature missing from Cirrus CI is notification. When a build fails, the status will be reflected on the GitHub Checks page, but Cirrus CI doesn't have built-in mechanism for sending an Email notification.

This can be worked around by adding a GitHub Action that sends emails when a GitHub Check Suite completes, but you'll need to provide your own SMTP server information.

Cirrus CI Android Templates

If you've made it this far, chances are you want to try out Cirrus CI for your own projects. To help you get started, I've created some cirrusci-android-templates for common use cases such as building APKs, running instrumented tests, and publishing library artifacts.

Summary

Overall I have been really impressed with Cirrus CI. I was initially attracted by the KVM-enabled containers for running instrumented tests. Not only was I able to migrate FlowBinding to use Cirrus CI for running instrumented tests, it was also a lot of fun discovering and trying a few other cool features along the way.

It's clear that the Cirrus CI Team understands the unique challenges when it comes to doing CI for Android. Having used quite a few CI solutions for Android including Jenkins, Buddybuild, Buildkite, CircleCI, Bitrise, and GitHub Actions, I'd like to highlight some of the key competitive advantages Cirrus CI brings to the table and why you might want to consider using it for Android:

• Excellent documentations and examples
• Completely free for opensource projects with very generous concurrency limit
• KVM-enabled containers for running hardware-accelerated Emulators
• Powerful VMs - up to 8.0 CPUs and 24 GB of RAM for Linux containers
• Built-in support for Gradle build cache (both local and remote)
• First-class GitHub integration with features such as Required PR Labels

There are also a few things that could use some improvements:

• Being a smaller player in the market means less efforts have been put into marketing, branding, and UX of the product especially the build dashboard
• Downloading and uploading cache archives takes noticeably longer than some of the other services
• Currently only GitHub is supported, although BitBucket support is planed
• There's no built-in support for Email notifications

None of these was a deal breaker for running instrumented tests. But due to the slower download and upload speeds for build cache, I have yet to migrate the rest my CI pipelines from CircleCI.

What's Next

Cirrus Lab (the team behind Cirrus CI) recently announced Cirrus Emulators, a cloud service that provides hardware-accelerated Android Emulators which can be integrated with any CI services using a CLI. I'll share my experience with it once the product is publicly available.

Thanks for reading!

Thanks to Fedor Korotkov for the review.