Combinatorial Approach to Testing Mobile Applications: An Experimental Investigation

Brandi Amstutz Advisor: Dr. Sergiy Vilkomir This research is supported by the National Science Foundation REU Site Award 1262933

Introduction

Importance of Research

Mobile testing

is a critical step in the development process where errors in an application can be discovered and corrected to increase reliability and functionality. Traditional software testing strategies can be modified and implemented to solve the challenges associated with mobile device selection. The amount of devices required to exhaustively represent today’s mobile market is extremely high due to a wide range of variability between available devices.

This investigative research analyzes the efficiency of mobile testing based on a combinatorial approach to device selection.

Introduction

Research Questions Addressed

 Which device characteristics have the potential to influence application reliability?

Both hardware and software components have the potential to influence an application’s functionality. ▫ Hardware components considered for this approach include:    Mobile device type Screen resolution Hardware manufacturer ▫ Software attributes included:  Android OS version  RAM memory

Introduction

Research Questions Addressed

 How should device selection be executed to ensure an optimal set of devices?

This approach suggests parameter value specification based on user profile data reflecting the current mobile market. Less popular parameter values can be grouped to reduce the total possible values without compromising proportionality Other (tvdpi and xxhigh) 5,9% Low 10% Huawei; 1,2% Pantech; 1,8% Motorola; 2,4% Google; 2,5% Xiaomi; 2,5% Amazon; 3,6% Other; 9,5% LG; 6,1% Samsung; 55,9% Extra High 24,6% Medium 23% HTC; 6,3% Sony; 8,1% High 36% Figure 1. Screen Resolution Distribution Figure 2. Device Manufacturer Distribution Source: Unity 3D Mobile Statistics

Introduction

Research Questions Addressed

 How many devices are sufficient for mobile testing?

Exhaustive testing strategies are limited by resource constraints and contain redundancy between device attributes. This research suggests a set of five devices selected based on device parameterization and combinatorial testing techniques for maximum coverage

Materials and Methods: Step One

Device Parameterization

Based on user profile data we suggest to use the following device characteristics and their parameter values:

Type

Smartphone Tablet

Device Manufacturer

HTC Motorola Samsung Other

Android OS Version Screen Resolution

Honeycomb (3.0-3.2) Ice Cream Sandwich (4.0) Jelly Bean (4.1-4.3) Other Low less than 426 dp x 320 dp Medium at least 470 dp x 320 dp High at least 640 dp x480 dp Extra High at least 960 dp x 720 dp

Table 1.

Selected Parameter Values

RAM Memory

< 768 MB ≥ 768 MB

Materials and Methods: Step Two

Device Selection

Through Each-Choice coverage techniques and consideration of current market statistics we suggest the following two sets of selected devices: Table 2. Selected Device Sets Based on user profile data the following devices were included due to their high level of popularity: Set A: Galaxy S2 i9100 and Galaxy Tab 2, 7.0

Set B: Galaxy S2 i9100 and Galaxy Tab 10.1

All other devices have been added to provide total coverage of device characteristics

Materials and Methods: Step Three

Application Selection

Three applications with similar functionality were selected for testing. The following applications used provide access to search, book and review travel accommodations ranging from car rental to discount vacation packages: Hotwire Travelocity BookIt.com

Materials and Methods: Step Four

Device Access

Perfecto Mobile cloud services were used for remote access to devices for testing purposes. Tests were recorded and screen shots were saved for later analysis.

http://www.perfectomobile.com/ Remote testing procedure using SaaS cloud-based device access:

User provides test input Input is sent to devices selected by user Audio & video of live test execution is recorded Test results & video are sent to user for analysis

Materials and Methods: Step Four

Device Access

Figure 3. Perfecto Mobile Dashboard

Materials and Methods: Step Five

Additional Devices

For comparison, two sets of random devices equivalent in size to the Each-Choice sets were selected. The same testing procedures were executed on the ten additional devices: Table 3. Random Device Sets

Results

Screenshot Analysis

Screenshots were analyzed from all sets of devices for fault identification

Figure 4 (left)

illustrates Fault H5 for the Hotwire application. The expected result is that the number of rooms should update on the main page when a user selects a new number of desired hotel rooms from the drop down menu. The fault occurs when the room quantity is not updated appropriately

Results

Screenshot Analysis

Figure 5 (right)

fault T1 for illustrates the Travelocity.com

application.

The expectation is that the Travelocity menu bar with their gnome icon is displayed at the top of the home screen.

The fault occurs when the menu bar is missing from view.

Results

Screenshot Analysis

Expected Fault Figure 6 (left)

illustrates fault B1 of the Bookit.com application.

The expectation is that a menu bar with current query's information and reference number is displayed at the top of the screen. The fault occurs when this menu bar is missing from view.

Results

Fault Detection

Fault detection totals are as follows:  Hotwire: 7  2 device independent  5 device dependent  Travelocity: 7  1 device independent  6 device dependent  Bookit.com: 4  1 device independent  3 device dependent

Results

Fault Detection

Table 4. Fault Detection In 7 out of 12 cases (or 58%) the effectiveness of our approach was greater than the results of random testing.

In 4 out of 12 cases (or 33%) the effective ness of our approach was equivalent to the results of random testing.

Conclusion

Conclusions

The quantity of application faults detected by the selected set of devices was equal to or greater than the quantity detected by the random set of devices. Additionally, the random set of devices did not yield a fault that the selected set was not able to identify.

This suggests that the device attributes selected were suitable based on their expected influence on application reliability.

Combinatorial testing strategies provided sufficient parameter value coverage with a test suite of five devices.

This research suggests that this small set of devices is effective for mobile testing while also reasonable enough for resource constraints.

Questions?