Mobile Functional Testing 101

Author: Bernard Lelchuk
Reviewer: uTest

Background: As you may have noticed, the use of mobile applications has skyrocketed over the past few years. And while most mobile companies are lagging behind Apple’s success, the market itself has nevertheless become a multi-billion dollar playground. While for years software manufacturers have been trying to find out how to make money from this market, Apple seems to have the early lead. Their iTunes AppStore is yielding them annual revenues in the billions - [1] and growing. They finally presented a viable business model for the mobile software market. As one might expect, Apple’s success has prompted many competitors to open their own mobile application stores.

Market: According to a recent report published on the Wireless Expertise website, “the global mobile app market – including games – will be worth $4.66 billion in 2009, rising to $16.60 billion, in 2013.”[2]

This of course would help explain the sudden entrance of Microsoft, Google, Research in Motion (RIM) and Palm, along with mobile vendors like Verizon and AT&T into the mobile applications market. As I like to say, they are trying to catch the fast-riding “Mobile App Train.”

This brings us to the topic on mobile testing, or more formerly known as mobile app testing. However, before we discuss the testing implications of this trend, I’d like to share with you how I got into mobile testing in the first place.  It is my hope that this story will encourage other testers to consider extending their careers by hopping on the Mobile App Train.

My Break into Mobile testing
I’m a big fan of mobile devices and other sorts of gadgets, and I always have been. And so after several years in IT and music production, I started searching for a new career path that could feed this interest. It was four years ago that I joined WorldMate – a great startup that develops mobile travel applications, and it was there that I got my first big break as a tester, and evidently a mobile tester.

As a long-time beta tester, the QA role was not all that new to me and I was glad to turn it into a profession. Of course, the best part about this particular job was the ever expanding mobile cabinet! The first time I opened it, my jaw literally dropped. It was full of hundreds of mobile phones. You could see classics as the Palm III, V/Vx, m5xx, Tungsten & Zire devices, alongside the newest Nokia, Sony Ericsson, HP, BlackBerry, and HTC devices.

Is any tester with a mobile phone also a mobile tester?
The short answer is YES – basically anyone with a mobile phone can be a mobile tester. However, one should have the following knowledge, skills and enthusiasm in order to translate them to mobile testing:

Know your phone! Okay, so you know the make and model name, such as a Nokia 5800, BlackBerry Bold, or Sony Ericsson W910. But do you really know what your phone capabilities are?

  • What platform is it running (Windows Mobile, Symbian, J2ME, RIM, Android, etc)?
  • Do you know its limitations (RAM, Storage Memory, Maximum application size, etc)?
  • Do you know what type of connection can you use (Edge, 3G, CDMA, Wifi, etc)?

If you answered “No” to at least one of above, I suggest taking a test drive on your phone.

  • Navigate through ALL of its menus and sub menus.
  • Click ALL keyboard and  buttons. Learn how they react in different conditions – e.g. in different applications, screen layouts (portrait, landscape), etc.
  • Use its basic applications such as calendar, messaging, contacts and phone and learn the phone’s behavior. Many users may overlook this as something unimportant, however this basic functionality will tell you the most essential skills of your phone. You may find it hard to believe, however this basic functionality differs largely between brands, platforms, vendors and models.
  • Use the phone extensively! This will reveal its weakness spots and will help you gain a lot of confidence.

Second, get to know other phones and compare them with yours. Once you know your device, do the same with different brands/models if possible and identify similarities and differences between them in regards to functionality and overall behavior. Ultimately, you will want to translate this into to testing needs:

  • What type of testing should be taken for each type of device (software/hardware)?
  • Which tools and dedicated software should you test with – e.g. sync tools, file explorers, RAM/Storage memory usage tools?
  • How applications are installed into device – sync tools, manually on device, OTA (over the air)?
  • Install 3rd party applications and try to identify any similarities between them.
  • Screen and keyboard layouts, RAM and storage usage, etc.

Know the Mobile Market:

The mobile market consists of 3 types of players:

  • OS manufactures
  • Devices manufactures
  • Companies that do both

There are several leading OS manufactures that hold their own Mobile Platforms (OS), while other device manufactures focus on delivering the actual devices, some focus on one specific platform while others will develop devices on multiple platforms. Here's a graph of all available Mobile Platforms in the market. Are you familiar with all of them or at least the most popular ones and can point them out? If not then I'd suggest you to get familiar with them all.

Mobile platforms are only half of the equation. You also need to be familiar with device manufactures – each has its own style, features, usability & popularity.

Take a look at the table bellow and see if you're familiar with these brands. If not, then it's a good place to start:


Mobile Testing Tips:

  • Carriers – Test your application via different carriers (e.g. Vodafone, O2, Orange, AT&T, Cingular, etc). You may find some differences when running the SAME application on different carriers.
  • Resolutions – Get familiar with different resolutions, e.g. 240×260, 320×240, 128×128 (square devices) and identify differences and UI bugs. For more advance testers, plan a test matrix which covers all possible/available resolutions for the tested devices/platforms.
  • Connections/airplane Mode – Test your application with different connection types (Edge, 3G, CDMA,Wifi) and also in airplane mode on supported devices. Also test GPS location connectivity.
  • Device Settings - Change the device settings: Date and Time, Languages (Advanced regional settings), Connections, Shortcuts – anything you have control over – and learn how each and combined settings impact the device.
  • Interrupts – Test how the device and applications behave on interrupts, e.g. Incoming/outgoing Call, SMS, MMS, Video-Call, Low battery, Low/No connectivity.
  • Local Device Storage Vs. Memory Card – Install/Uninstall and test your application on both local device storage and memory card. You’ll be amazed by applications behavior when installed on each storage option.
  • Device input entry modes – Physical keyboard, Virtual keyboard/Touch screen.


Additional test areas one should focus when approaching mobile testing:

  • Installation/Un-installation and product activations
  • UI Testing
  • File formats and conversions
  • Device and application backup
  • Device-PC sync
  • Stability
  • Connectivity and data traffic – Use sniffers to monitor traffic and find bottlenecks or data scrambling (mainly done by specific carriers which plant their data in incoming/outgoing XMLs)


[1] http://www.apple.com/pr/library/2009/07/21results.html

[2] http://www.wirelessexpertise.com/admin/upload/pdf/5MobileAppStoresWirelessExpertiseReport.pdf. "The future of Mobile Application Storefronts", Wireless Expertise

utest Help Taxonomy:

Print

Thanks for contacting uTest