Testing on SmartPhones

1. About Smartphones

Mobile Phone: A mobile phone is more frequently called a cellular phone or cellphone. Most mobile phones provide voice communications, Short Message Service (SMS), Multimedia Message Service (MMS).

PDA: Personal Digital Assistant (PDA) is a handheld computer for managing contacts, appointments and tasks. Wireless PDAs may also offer e-mail and Web browsing, and data are synchronized between the PDA and desktop computer via USB or wireless.

Smartphone: A smartphone is considered to be the combination of the traditional PDA and cellular phone, with a bigger focus on the cellular phone part. Smartphones allow users to store information, e-mail, install programs, along with using a mobile phone in one device.

Some Popular Examples:

a. The popular Apple iPhone is a Smartphone and is designed and marketed by Apple Inc. The first iPhone was introduced on January 9, 2007. An iPhone functions as a camera phone, including text messaging and visual voice-mail, a portable media player, and an Internet client, with e-mail, web browsing, and Wi-Fi connectivity.

iPhone Features: FaceTime, Camera, Mail, Photos, Messages, App Store, Retina display, Folders, safari, AirPrint, Maps + Compass, iTunes Store, iMovie (from app store), Multitasking, Game center, iPod, AirPlay, Keyboard, Accessibility, iBooks (from app store), HD Video Recording, Phone, Home Screen, Voice Control, Search, Find My iPhone.

b. The iPad (PDA) is a tablet computer designed, developed and marketed by Apple primarily as a platform for audio-visual media including books, periodicals, movies, music, games, and web content. Apple released the iPad in April 2010. The iPad runs the same operating system as the iPod Touch and iPhone. Without modification, it will only run programs approved by Apple and distributed via its online store.

iPad Features: Safari, Mail, Photos, Videos, You Tube, iPod, iTunes, App Store, Maps, Notes, Calendar, Contacts, Home Screen, Multitasking, Folders, Airprint, Airplay, game Center, Spotlight search, Accessibility, Find My iPad, iBooks (from app store).

c. iPod (such as the newer iTouch are PDA's) is a portable media player designed and marketed by Apple and launched on October 23, 2001.

iPod Features: FaceTime, Music, mail, Voice Control, AirPrint, Find My iPod touch, Retina display, Movies and TV shows, Safari, Maps, AirPlay, HD Video Recording, App Store, Photos, You Tube, Voice Memos, Game center, iTunes, Home Screen, Nike _ iPod, Accessibility, iBooks (from app store).

d. The HP iPAQ Mobile Messenger is a Smartphone.

e. The LG Prada is a cellular phone with a touch screen — but its not a smartphone

f. The RIM BlackBerry 8800 is considered a smartphone

g. The Palm Treo 700p is a PDA phone

h. The Motorola Q is also considered to be a PDA phone

2. Operating systems for Smartphones

a. Android operating system: Android is a mobile operating system initially developed by Android Inc. Android was purchased by Google in 2005. Android operating system is currently being run by mobile phones built by a variety of different companies, including Samsung, Acer, Motorola, and HTC.

b. iPhone OS: iOS (known as iPhone OS prior to June 2010) is Apple's mobile operating system. Iphone OS is the mobile phone version of Mac OSX.

c. Palm: The palm OS (also known as Garnet OS) is a mobile operating system that was designed by Palm, Inc., in 1996. In 2009, the main licensee of Palm OS, Palm, Inc., switched from Palm OS to webOS for their forthcoming devices. HP webOS is a proprietary mobile operating system running on the Linux kernel, initially developed by Palm and purchased by Hewlett Packard (HP) in 2010.

d. Blackberry OS: Blackberry OS is developed and designed by Canadian company Research In Motion (RIM) since 1999. One bad side is that it limits one to viewing MS Office documents only.

e. Windows Phone 7 series (Windows Mobile): Windows Mobile is a mobile operating system developed by Microsoft that was for use in Smartphones and mobile devices. Windows Phone 7 mobile operating system is the successor to its Windows Mobile platform. It launched in Europe, Singapore and Australia on October 21, 2010 with Asia to follow in 2011.

f. Symbian: Symbian is an operating system for smartphones and mobile devices designed by Symbian Ltd. Nokia acquired Symbian Software Limited in 2008. Symbian OS was made available in 2010 as open source software.

3. TESTING on Smartphones

1. Installation testing: Tester’s need to focus on installation & uninstallation of the app (remember, user will install the app through Market place).

2. Requirement Functionality Testing: Test for the application functionality according to designed test cases.

3. Widget testing: Test the widget by adding widget, using widget as per functionality i.e. music widget should play music, deleting widget.

4. Phone Interrupt testing: Test the application behavior on phone call, SMS & other notifications.

5. Stress testing (monkey test): There is a command in android where it generates series of events at different frequencies. Make sure our app passes this monkey test. The command used for generating event is:

$ adb shell monkey [options] <event-count>

(http://developer.android.com/guide/developing/tools/monkey.html)

6. UI testing: All mobile platforms have certain submission guidelines to follow before the application can be available commercially. Apple AppStore is very strict on guidelines. Most of the applications submitted are rejected due to small errors on AppStore. We need to verify (UI test) that application is meeting guidelines:


a. Apple Guidelines
b. Android Guidelines

c. Screen Resolution for Cellphones

7. Compatibility Test: Developers do the unit testing on the emulators. Testers need to test on actual environment i.e. devices. There are multiple devices of different resolutions & having various OS. We need to check for compatibility on maximum devices.

8. Following are some basic scenarios which should be considered while testing mobile application:

a. UI of the app is as per the screen size of the device, no text/control should be cutting off.

b. Text should be readable.

c. Suspend/Resume (Call/SMS/Alarm) when app is running.

d. If app has timer/sound then Suspend/Resume when timer/sound is active.

e. Behavior of app on Flip/Slider close.

f. Behavior of app when memory of the device is almost full (very critical).

g. Behavior of when network is not available.

h. Backup and restore

i. Behavior of app if app is running for longer period of time

j. Behavior of app when keys are pressed randomly.

k. If app support app/port directed SMS then behavior on receiving such SMS.

9. Performance tests

a. System tests (booting the phone, or resuming from standby)

b. User interface tests (rotating the screen, displaying a menu)

c. Network (downloading a Web page or application)

d. Media (starting a video)

e. Phone (calling a contact)

f. Application (displaying the day’s calendar, viewing the photo folder)

g. File system (reading an mp3 file)

10. Testing Checklist for Mobile Applications

http://mobileappstesting.com/2009/10/23/testing-checklist-for-mobile-applications/

11. iPhone Testing: Applications for the iPhone are written in Objective-C. Objective-C is an object-oriented programming language which is very similar the C programming language.

To write iPhone applications the iPhone SDK (software development kit) is needed which uses Xcode (Mac OS X development software) as the development environment for the programming and debugging, and an iPhone simulator to test the program.

Testing: As with all software, thorough testing is essential before release. However, all faults will not be found as they may be in rarely used sections of the application, but there is a limit to how much time is spent on finding faults because removing a high percentage of faults may only improve the application by a very small percentage.

The interfaces and the main functionality must be tested in various ways to make sure that the chance of faults that would cause a failure in the system is low. This testing will be carried out in three different ways, first, the system will be tested on a simulator, then by experienced iPhone users and finally by inexperienced iPhone users. The use of inexperienced and experienced users will test the interface for ease of use.

Resources Required:

iPhone SDK – needed for iPhone API incorporation

iPhone Simulator – needed to test application

iPhone Developer Membership – needed to test application on an iPhone

Apple Mac – iPhone Simulator and SDK only designed to work on Mac OS

4. Why testing should be done on actual device?

a. Resource availability: Simulator may have access to more or less resources, and the speed at which those resources become available is liable to be very different.

b. Program performance: The simulator may run at very different speeds to the device.

c. Communications availability: If app communicates over 3g/GPRS/WAN/Bluetooth, simulators tend to simulate in such a way that they always work, whereas in the real world you get outages, performance drops, etc.

d. Hardware exceptions: how does your code handle low battery, power-offs, insufficent memory, etc.

e. Given you're only dealing with a service, you are possibly not worried about the user experience so much, but you really need to carry out usability testing on the device in its target environment, rather than at your desk.

f. Some of the UI elements looks different in simulator and in emulator.

Issues with different Simulators:

One of the guy has posted following points on a testing forum:

1. Issues with Blackberry Simulator

a. Blackberry simulator is extremely slow

b. mouse does not work while working on Blackberry simulator

c. it crashes a number of times

d. impossible to close the application (Have to kill it via task manager)

2. Issues with Android Simulator

a. Android is definitely faster than Blackberry simulator - but the smaller screen of the phone makes things a little impossible to test the entire application

b. We cannot zoom in once we have the modal window

c. In the original zoomed out size, we can hardly see anything for testing a specific feature. I was trying to search the net to extend the simulator for better testing- but do not really think that’s possible

3. Simulator for Ipad and Iphone

a. They are extremely slow.

5. Problems occur while doing Testing on Mobile Applications:

a. Mobile software is written on an emulator and eventually runs on the actual device.

b. There are less or more subtle differences between emulators and devices (and in particular across devices from different vendors) in terms of supported APIs, hardware and Java Virtual Machine implementation details.

c. Uploading and testing the application on each and every device is a tedious, time consuming process.

6. Tools used

1. Simulator [Blackberry]

2. Simulator [iPhone]

3. Simulator [iPhone]

4. Demo [Tool]

5. Simulator [Windows Mobile]





Related Testing Topics: 
a. Black Box testing
b. Agile Testing