Dynamic Testing with the Monkey Program
Not quite the Turing test, but if your app can pass the monkey test, you deserve a pat on the back.
Testing is so easy a monkey can do it, literally. Despite the lack of testing tools for Android, I have to admit that monkey is pretty cool. If you're not familiar with Android's monkey, it's a testing tool that comes with the Android SDK and simulates a monkey (or perhaps a child) using an Android device. Imagine a monkey sitting at keyboard and flailing away, get the idea? What better way to flush out those hidden ANRs?
Running monkey is as simple as starting the emulator (or connecting your development device to your development machine) and launching the monkey script. I hate to admit this, but by running monkey on a daily basis we've repeatedly found defects that probably would've escaped a normal QA pass and would've been very challenging to troubleshoot if found in the field. Or worse yet, caused users to stop using our app.
Here we talk about a few best practices for using monkey in your development process.
First, use monkey.
Second, create your own monkey shell script or batch file that wraps Android's monkey script. This is to ensure that all the developers on your team are running monkey with the same parameters. If you're a team of one, this helps with predictability (discussed below).
Next, you'll want to configure monkey so that it runs long enough to catch defects and not so long that it's a productivity killer. In our development process, we configured monkey to run for a total of 50,000 events. This took about 40 minutes to run on a Samsung Galaxy Tab. Not too bad, but I would've liked it to be in the 30 minute range. Obviously, faster tablets will have a higher throughput.
Monkey is random, so when we first started running monkey every developer was getting different results and we were unable to reproduce defects. We then figured out that monkey allows you to set the seed for it's random number generator. So, configure your wrapper script to set monkey's seed. This will ensure uniformity and predictability across monkey runs in your development team.
Lastly, once you gain confidence in your app with a specific seed value, change it, you'll never know what monkey will find.
Below is a monkey script wrapper, and a description of its arguments.
- -p package name will ensure that monkey only targets the package specified
- --throttle is the delay between events
- -s is the seed value
- -v is the verbose option
- 50000 is the number of event monkey will simulate.
# Utility script to run monkey
# See: http://developer.android.com/guide/developing/tools/monkey.html
adb shell monkey -p package.name.here --throttle 100 -s 43686 -v 50000 | tee tmp/monkey.log
There are many more configuration options for monkey, we deliberately chose not to mess around with what types of events monkey generates because we appreciate the pain. For example, the seed value we chose causes monkey to disable Wifi about halfway through the run. This was really frustrating at first because we felt like we weren't getting the coverage we wanted. Turns out, monkey did us a favor by disabling Wifi and then relentlessly playing with our app. After discovering and fixing a few defects, we soon had complete confidence that our app operated as expected with no network connection.
You probably don't want to run it on your production phone as the device you run it on is not usable by you while the monkey is running; you can do nothing but watch it mess up your settings and otherwise be annoying.
Official documentation suggests running it under "adb shell" as above. You can also run it in "terminal emulator" but only on a rooted device, as in this mode it needs 'su' privilege to generate synthetic events to other programs.