Benchmarking Android Automated Test Device (ATD) Emulators
TL;DR: We measured these emulators to be roughly ~20% faster in a similar configuration.
With the release of Android Studio Dolphin Google announced a new class of Android emulator images - Automated Test Devices, or ATD for short. They come with a couple of key differences compared to the standard images:
- Various standard apps like Browser, Music etc are not installed
- System UI is disabled
- Many background services are disabled
- Hardware rendering is disabled
The changes make sure that the emulator is spending as little time as possible on things other than running your tests. Hardware rendering being disabled might seem odd at first, but in most cases these images will be used in CI machines without GPUs where there will be an additional software GPU emulation layer (swiftshader) anyway, so it makes sense to do software rendering already there in the emulator itself.
Read the release notes about ATD images here: https://developer.android.com/studio/preview/features#gmd-atd
Mandatory disclaimer: this is by no means a scientific experiment but should give a good enough glimpse into what performance to expect from these images. As always, your mileage may wary, depending on your app, tests, host machine performance, weather conditions, specific
emulatorversions used, etc.
We used the
NexusLowRes device profiles from emulator.wtf with both ATD and non-ATD variants. In addition we also added api 23 emulators for comparison as those have typically been the fastest. The benchmark project itself is a fork of the sunflower sample.
I ran the
size=medium suite of tests, without any sharding, Android Test Orchestrator enabled with clearing package data between each tests. Each combination was run for a total of 3 times to average out some variance. For a fair result the test times shown include all steps necessary to run them including apk install times as these could vary between emulator images.
First lets compare the vanilla Pixel2 version=30 configuration vs the ATD configuration:
The ATD version performs really well - just by switching out a model we’re able to slice 5min from our test runtime - roughly a 20% improvement.
Comparing to other fast images
Lets see how ATD fares against the fastest non-ATD image we have - the
NexusLowRes model at API 23. It’s an image with very small mdpi 360x640 resolution to waste minimal time pushing pixels.
The performance of these two images are really similar - but with ATD we don’t have to make any compromises - we can run the full 1080p resolution of the Pixel2 model and we can use the much newer Android instead of testing on Android 6.0 Marshmallow.
Similarly, using the
NexusLowResAtd model with api 30 gives us comparable results to
You can really see disabling hardware rendering reduces the impact of the emulator screen resolution on durations by a large margin.
All variants tested
Here’s a graph showing all the combinations we tested:
Start using ATD images
Using ATD images in emulator.wtf
If you’re already using emulator.wtf then switching is easy - switch to one of the
*Atd images available when running your tests:
ew-cli --device model=Pixel2Atd,version=30 [other test options]
You can list all available combinations, including ATD ones, with
Installing ATD images locally
The official notes mention them in the context of Gradle-Managed Devices and Android Dolphin preview but you can actually get started with the images without having neither - they’re typical Emulator images so they can be installed like any other emulator images.
Note that at the time of writing the images are in the preview (
3) channel, they’re only visible in the IDE on Dolphin and later. When creating a new emulator image open the x86 Images tab and select a Google APIs ATD image. At the time of writing there are only 30 (R) and api 31 (S) images available.
If you’re not on Dolphin you can still install these from the commandline. Here’s an example to display the list of available atd images, assuming that your Android SDK is at
$ cd $ANDROID_HOME && ./cmdline-tools/latest/bin/sdkmanager --list --channel=3 | grep atd | cut -d ' ' -f 3 system-images;android-30;aosp_atd;arm64-v8a system-images;android-30;aosp_atd;x86 system-images;android-30;google_atd;arm64-v8a system-images;android-30;google_atd;x86 system-images;android-31;aosp_atd;x86_64 system-images;android-31;google_atd;x86_64
We can then proceed to install one of those. I’m interested in Android R and I’m on an Intel machine so I went with
$ ./cmdline-tools/latest/bin/sdkmanager "system-images;android-30;google_atd;x86" --channel=3 [=======================================] 100% Unzipping... x86/data/system/disp
BONUS: What’s in these ATD images anyway?
I wanted to poke around a bit so I ran one of these locally in my Android Studio. What’s interesting is that the device doesn’t render anything except a solid color - although the color does change depending on what screen the app was on. Layout inspector only rendered a single 1x1 pixel view. The tests themselves ran fine though.
Out of interest I captured
adb shell ps and
adb shell pm list packages outputs for both ATD and non-ATD variants of Pixel 2.
On ATD we have roughly 250 processes running and around 100 packages installed:
$ adb shell ps | wc -l 253 $ adb shell pm list packages | wc -l 109
On a vanilla image we’ve got around 270 processes and 150 packages (a +50% bump over ATD!):
$ adb shell ps | wc -l 271 $ adb shell pm list packages | wc -l 155
When diffing the list we see many apps like Calendar, Camera2, Dialer, Contacts etc being removed from ATD as expected. Instead of SystemUI there’s a
com.android.fakesystemapp package instead. What’s fascinating is that there are package-level references to specific hardware configurations in the ATD image, like
The exact diff of those lists can be seen in this gist.
- Android Studio Dolphin release notes, Run tests using Automated Test Devices https://developer.android.com/studio/preview/features#gmd-atd
- Android Developers Blog, What’s New in Scalable Automated Testing https://android-developers.googleblog.com/2021/10/whats-new-in-scalable-automated-testing.html
- emulator.wtf Sunflower benchmarks Google Sheet https://docs.google.com/spreadsheets/d/1rcg0_iHYIzQm537m8Aj3LJW0bK1W3NvQx8LtVHGKpCc/edit?usp=sharing