Benchmarking Android Automated Test Device (ATD) Emulators

2022-04-17

5 min read

TL;DR: We measured these emulators to be roughly ~20% faster in a similar configuration.

With the release of Android Studio Dolphin Google announced a new class of Android emulator images - Automated Test Devices, or ATD for short. They come with a couple of key differences compared to the standard images:

Various standard apps like Browser, Music etc are not installed
System UI is disabled
Many background services are disabled
Hardware rendering is disabled

The changes make sure that the emulator is spending as little time as possible on things other than running your tests. Hardware rendering being disabled might seem odd at first, but in most cases these images will be used in CI machines without GPUs where there will be an additional software GPU emulation layer (swiftshader) anyway, so it makes sense to do software rendering already there in the emulator itself.

Read the release notes about ATD images here: https://developer.android.com/studio/preview/features#gmd-atd

Benchmarks

Mandatory disclaimer: this is by no means a scientific experiment but should give a good enough glimpse into what performance to expect from these images. As always, your mileage may wary, depending on your app, tests, host machine performance, weather conditions, specific emulator versions used, etc.

Methodology

We used the Pixel2, NexusLowRes device profiles from emulator.wtf with both ATD and non-ATD variants. In addition we also added api 23 emulators for comparison as those have typically been the fastest. The benchmark project itself is a fork of the sunflower sample.

I ran the size=medium suite of tests, without any sharding, Android Test Orchestrator enabled with clearing package data between each tests. Each combination was run for a total of 3 times to average out some variance. For a fair result the test times shown include all steps necessary to run them including apk install times as these could vary between emulator images.

Apples-to-apples comparison

First lets compare the vanilla Pixel2 version=30 configuration vs the ATD configuration:

A side-by-side comparison of Pixel2 API 30 (25m test duration) vs Pixel2 API 30 Atd (20m test duration)

The ATD version performs really well - just by switching out a model we’re able to slice 5min from our test runtime - roughly a 20% improvement.

Comparing to other fast images

Lets see how ATD fares against the fastest non-ATD image we have - the NexusLowRes model at API 23. It’s an image with very small mdpi 360x640 resolution to waste minimal time pushing pixels.

A side-by-side comparison of NexusLowRes API 23 (20m test duration) vs Pixel2 API 30 Atd (20m test duration)

The performance of these two images are really similar - but with ATD we don’t have to make any compromises - we can run the full 1080p resolution of the Pixel2 model and we can use the much newer Android instead of testing on Android 6.0 Marshmallow.

Similarly, using the NexusLowResAtd model with api 30 gives us comparable results to Pixel2Atd:

A side-by-side comparison of NexusLowRes API 30 (20m test duration) vs Pixel2 API 30 Atd (20m test duration)

You can really see disabling hardware rendering reduces the impact of the emulator screen resolution on durations by a large margin.

All variants tested

Here’s a graph showing all the combinations we tested:

A side-by-side comparison of 6 configurations: NexusLowRes API 23 (20m), NexusLowRes API 30 (21m), NexusLowRes API 30 ATD (20m), Pixel2 API 23 (24m) Pixel2 API 30 (25m) and Pixel2 API 30 ATD (20m)

Start using ATD images

Using ATD images in emulator.wtf

If you’re already using emulator.wtf then switching is easy - switch to one of the *Atd images available when running your tests:

ew-cli --device model=Pixel2Atd,version=30 [other test options]

You can list all available combinations, including ATD ones, with ew-cli --models.

Installing ATD images locally

The official notes mention them in the context of Gradle-Managed Devices and Android Dolphin preview but you can actually get started with the images without having neither - they’re typical Emulator images so they can be installed like any other emulator images.

Note that at the time of writing the images are in the preview (3) channel, they’re only visible in the IDE on Dolphin and later. When creating a new emulator image open the x86 Images tab and select a Google APIs ATD image. At the time of writing there are only 30 (R) and api 31 (S) images available.

Android Studio virtual device creation dialog showing ATD images — *ATD image among other emulator images*

If you’re not on Dolphin you can still install these from the commandline. Here’s an example to display the list of available atd images, assuming that your Android SDK is at $ANDROID_HOME:

$ cd $ANDROID_HOME && ./cmdline-tools/latest/bin/sdkmanager --list --channel=3 | grep atd | cut -d ' ' -f 3
system-images;android-30;aosp_atd;arm64-v8a
system-images;android-30;aosp_atd;x86
system-images;android-30;google_atd;arm64-v8a
system-images;android-30;google_atd;x86
system-images;android-31;aosp_atd;x86_64
system-images;android-31;google_atd;x86_64

We can then proceed to install one of those. I’m interested in Android R and I’m on an Intel machine so I went with x86 variant:

$ ./cmdline-tools/latest/bin/sdkmanager "system-images;android-30;google_atd;x86" --channel=3
[=======================================] 100% Unzipping... x86/data/system/disp

BONUS: What’s in these ATD images anyway?

I wanted to poke around a bit so I ran one of these locally in my Android Studio. What’s interesting is that the device doesn’t render anything except a solid color - although the color does change depending on what screen the app was on. Layout inspector only rendered a single 1x1 pixel view. The tests themselves ran fine though.

Android studio screenshot with ATD emulator on the right side and layout inspector open on the bottom — *The tiniest Layout Inspector view I've ever seen*

Out of interest I captured adb shell ps and adb shell pm list packages outputs for both ATD and non-ATD variants of Pixel 2.

On ATD we have roughly 250 processes running and around 100 packages installed:

$ adb shell ps | wc -l
253
$ adb shell pm list packages | wc -l
109

On a vanilla image we’ve got around 270 processes and 150 packages (a +50% bump over ATD!):

$ adb shell ps | wc -l
271
$ adb shell pm list packages | wc -l
155

When diffing the list we see many apps like Calendar, Camera2, Dialer, Contacts etc being removed from ATD as expected. Instead of SystemUI there’s a com.android.fakesystemapp package instead. What’s fascinating is that there are package-level references to specific hardware configurations in the ATD image, like com.android.internal.emulation.pixel_5 or com.android.internal.emulation.pixel_2_xl.

The exact diff of those lists can be seen in this gist.