How to speed up the execution of Android UI tests as part of the CI system

Question

I want all unit and instrument tests (Espresso) to run after each commit / merge in the main develop branch. Single tests are fast enough to allow this, but UI tests are not - 150 completely mocking UI tests take ~ 1 hour to run on a single device. Shazam FORK library does an excellent job with this 150 tests on all connected devices. The current solution is the local machine running Jenkins. Connecting 4 devices to it reduces the time to run UI tests to ~ 15 minutes. It is not perfect, but bearable.

Ideally, I would like to find a cloud-based CI system that allows me to run UI tests with Fork , so local Jenkins can be dropped and not supported internally.

I tried AWS Device Farm and Firebase Test Lab , but both use their own systems to run the tests. It seems that they do not give the opportunity to outline a single set of tests on multiple devices. They seem like great tools to run the entire test suite on different devices at the same time, but that’s not what I want for the CI solution (split the test suite into multiple devices at the same time).

I tried BuddyBuild , but internally they used the Firebase Test Lab, so that didn't help for my case either.

I mostly thought about solutions in these areas:

• find a way to run Fork in a cloud solution
• find another way to outline one set of tests on multiple devices

Any suggestions are welcome! How do you solve this problem?

Answer 1

Congrats! Too much automation is a big problem. It looks like you have already made great progress by speeding up execution time. I would say that 15 minutes is a pretty reasonable number, but here are a few alternative approaches to get this number even lower:

• Create a smaller and faster test suite for tests with the highest priority for each transaction, while a longer complete test suite runs continuously against the latest merged commits. This is a fairly common pattern in the industry. You can use the built-in Android @Small, @Medium and @Large annotations or package names to divide the tests up.

• Optimize your test cases to test several features at once. Essentially, instead of testing each combination, such as A1, A2, B1, B2, instead, you should only test A1, B2. See pair testing the Wikipedia page for a more detailed explanation.

• If possible for your application, try using emulators. Over the past couple of years, productivity has improved significantly. Using Intel Hardware Accelerated Execution Manager (HAXM) drivers, I found that tests were faster and more reliable than on most physical devices. It can also facilitate your task in the cloud.

• Check how much time each individual test takes to complete, corresponding to its priority. A test with a low priority of run time can be transferred to a less frequently performed test.

• Make sure that tests are performed to identify sleeping or other slow areas of test execution for improvement.

• Look for tests that have never been run. This may be another candidate for the transition to less frequent testing.

• If you are looking for alternative ways to test shards, you can topple your own by running parallel scripts on devices that target a subset of your tests using the adb shell am instrument . Although, this would create many separate test reports, and you would need to evenly divide the tests.

Answer 2

You can take a look at the Flank open source project , which seemed to be built with almost the same purpose: https://medium.com/walmartlabs/flank-smart-test-runner-for-firebase-cf65e1b1eca7

Flank is a Firebase test lab for massively scaling your automated Android tests. Run large test suites in parallel on many devices / versions / configurations simultaneously. Flank can be easily used in a CI environment where Gradle (or similar) first creates APK: s, and then Flank is used to run tests.

Answer 3

I would also vote for the aforementioned proposal, especially for point 1 “Create a smaller and faster test suite for the tests with the highest priority to run on each commit, while a longer complete test suite works constantly against the latest merged commits. This is a fairly common pattern in the industry. You can use the built-in Android @Small, @Medium and @Large annotations or package names to share tests. Optimize your test cases to test multiple functions on "

Using annotations and managing execution through TestNG frameworks (via Jenkins) would be one way to partition a large set of sections - merging the results would also be less painful if managing annotations divides your tests into logical groups (hope this helps)