Mobile Apps provide unique challenges for development and especially for software testing. This article describes some of the key challenges in functional and performance testing of mobile apps and provides suggested resources for improving the mobile application performance testing.

One reason mobile apps are unique is that mobile devices are unique. There is a wide range of devices for both Android and iOS operating systems ranging from phones that are several years old, to tablets that are much larger and brand new. Android devices, in particular, are produced by multiple manufacturers and some of these devices remain running on older versions of the operating system with distinct features.

Another reason that mobile apps are unique is that mobile devices are mobile. Apps are then used with a wide range of environments. The quality of internet connections then varies widely from high quality WIFI within your home, to low quality cellular content in a subway tunnel.

Additionally, mobile app users are often multitasking and running several different apps at the same time, receiving notifications from various services, playing music, etc.

There are three main types of mobile applications: Standard, Network and Hybrid. A standard app is one that runs entirely on the phone regardless of having an internet connection. For example, perhaps a puzzle game that does not require the internet to play the game. The other extreme is a network app will only work when it has a full internet connection. An example might be a weather radar app which has to download real time information to be useful. In contrast, a hybrid app is one that uses the internet, but also has some functionality without the internet. Most apps are in this category. For example, the app could download information from the internet (videos, messages, etc), and then the user can watch this downloaded content while offline.

These different types of apps then have different testing requirements. Bugs may occur as different apps are running and using resources on the phone.

These different types of apps and different mobile configurations increase the complexity of testing an app. At a minimum, a mobile device farm is needed to allow a range of devices to be tested. The device farm can be a set of actual devices, simulators, or remote devices available over the web. It is impossible to take mobile application performance test for all devices, display sizes, operating systems, etc, so some creativity is needed in order to effectively test a mobile device.

Developers perform the first set of testing as they develop their app, however, a separate set of software testers should also perform the end-to-end testing of the app under realistic conditions. For example, a developer may test on the simulator available in the development environment, or on their own phone. Often, this also means that the developer is on the same network as the backend server that supports the content available for the app. We at Iterators had one client with an app that included GPS location as part of their app and a back end server that stored all of the information. We reported several bugs even related to the login feature and the developers had trouble replicating these bugs because they had local access to the backend server, while we were at a different location and were not able to reliably access the server through the app.

In general, it is very helpful to have a set of independent software testers in addition to the developers. Software Testers provide a different perspective and should look at the software from the user’s perspective. Developers and Product Owners tend to use the “happy path” because they know how the software was designed. The testing process for software should include testing a variety of conditions, not just using the software as intended, in order to uncover issues that could arise from users attempting to use the software. This is important for ensuring user satisfaction and successful software usage.

Performance testing is very important for most mobile apps. Apps that do not use a network connection should still be tested on a variety of phones and tablets that have different operating systems and on older devices that have slower processors. Apps that do use a network connection have the additional need to test on variety of network connections, including dropped connections.

Performance testing can be completed as part of functional testing. Software testers should pay attention to the responsiveness of the system as the testing is completed under different network conditions.

If the app involves the use of a backend server, then the server performance should also be tested. This should consider the number of users that are expected during the rollout. The expected number of users can be simulated using API performance testing software that can track the response time for typical user interactions. Another common problem is to consider the performance of the production server rather than a staging server. Often, projects use a staging server that has a different configuration than the production server. It is important to try this out before the launch so that the production server is ready to go at launch time.

Development environments often have the option to monitor network traffic and the interactions can be monitored directly. Software testers can do this as long the developer provides a debug build of the app. Note that the debug build may impact the performance of the software too, and this can provide a good stress testing of the performance of the app.

A strategy must start with objectives. Is your app going to be used worldwide, or by a niche set of users? Will there be thousands of users simultaneously or only a small chance of simultaneous users? Will the demand be relatively even each day, or are there peak demand needs, such as introducing a sale or a new download? The second component is to look at the architecture of your design to look for potential issues. Perhaps your app has a niche market and your content is provided by a single server running the backend of your app. In this case, it is definitely useful to test from a remote location so that the testing is not done on the same network as the server itself.

A smaller app could then benefit from an initial triage with actual software testers. We at Iterators, LLC have tested several apps and websites with performance issues. We observe these issues as we also perform functional testing for the app. Sometimes this can be sufficient for apps expected with light usage on the backend and the key metric is to make sure that the server responds in a reasonable amount of time.

Apps expected to have more requests sent to the backend require additional planning and tools to complete the testing. The next section describes available tools that can create traffic from multiple users and then track the response times.

The following introduces several mobile application performance testing tools with links to their websites for more information. When it comes to a mobile performance testing approach, you should research these to determine which fits your needs whether that means you are looking for a free tool that you are willing to invest time learning to use, or a more expensive tool that is preconfigured to be easier to use.

Postman is a standard tool for functional and performance testing of APIs. This tool enables users to create performance tests that can simulate various loads on the backend server and measure their effect on the overall system's performance.

Blazemeter and JMeter are other useful tools for API performance testing. These tools are often compared to each other, so research which is best for your needs. JMeter is also a good option for taking load testing.

Akami Cloudtest also provides API testing features for your backend server.

Apptim is designed more specifically for mobile apps and includes several aspects of mobile app performance. This includes checking battery usage, screen rendering time, nonresponsiveness

Eggplant is a major vendor with several performance tools for mobile apps. This tool can be used as part of a Continuous Integration / Continuous Deployment (CI/CD) pipeline.

Tricentis is another major supplier of testing tools including Performance testing.

Mobile apps are unique in that they are designed for use on small, portable devices. This can make performance testing challenging, as you need to take into account factors such as screen size and battery life. Who should test a mobile app? Who needs to performance test mobile apps? We hope we've answered these questions.