Around the World in 8 remote cache nodes

Nowadays, distributed teams across different regions or countries are common in organizations. Depending on the bandwidth and latency, we could experience a degraded experience in our builds when consuming the cache entries.

This article shares the results of measuring builds in the project nowinandroid using different Remote Cache nodes.

The Local environment
To simulate the "local" build, we are using a GitHub Action runner with:

• 2-core CPU (x86_64)
• 7 GB of RAM

We are not able to choose the runner location in GitHub Action. Following the doc, the supported Azure regions are:

• East US (eastus)
• East US 2 (eastus2)
• West US 2 (westus2)
• Central US (centralus)
• South Central US (southcentralus)

So we assume our "local" builds will be executed somewhere in the US.

The Project
The project used for this experiment is nowinandroid(commit id), build stack components at that point:

• Gradle 8.2
• AGP 8.0.2
• KGP 1.8.22

Cache Nodes
We created a Remote Cache node for each scenario defined in different GCP zones.

Methodology

• Each scenario is configured pointing to one specific Remote cache node.
• Each scenario executes 11 builds, the first build populates the cache. Then, 10 builds are executed in parallel reusing the cache entries.
• The task executed in this experiment is "test". After populating the cache, the build gets 1859 tasks from the remote cache.
• All builds are tagged in the Build Scan, including the scenario and execution number.
• Using Gradle Enterprise API, we fetch and aggregate the build information.

Results
We start analyzing the results of measuring the median for the task ":core:analytics:compileDemoReleaseJavaWithJavac":

We don't observe a significant difference using different locations for the remote cache.
However, the output of this task is small (8.4 KiB). Let's see what happens when we analyze the median of a task with a bigger output size like ":app:mergeProdDebugResources" (950.1 KiB):

We observed a considerable difference across remote cache node locations once the task increased the output size.

Finally, instead of analyzing single tasks, we aggregated the results by task type. AGP task "MergeResources":

For KGP, these are the results of the median aggregated task "KotlinCompile":

Complete results table grouped by task type (ms):

Final Words

Caching continues to be the most simple and effective mechanism to optimize the build execution of Android projects.
Highly modularized projects enjoy the savings provided by reusing the outputs of the tasks, not only in terms of duration but reducing the pressure on the environment's resources executing the build.

Cache Node Replication improves the experience of teams in regions far from the cache origin by creating closer cache components for the user. There are different resources and articles explaining this strategy:

• https://docs.gradle.com/enterprise/helm-admin/current/#replication
• https://gradle.com/blog/managing-a-remote-build-cache-at-scale-with-local-build-observability-at-cash-app/
• https://dev.to/thedancercodes/gradle-build-cache-node-replication-in-android-14bc

Happy Building!