Tuesday, October 30, 2012

Designing Test Cases with 100% Coverage in a nutshell

I, in the recent past worked in a project which used the TMAP process for creating test cases.
It was definitely one of the best ways of creating test cases whose coverage was simply put 100%. We had used both White box and Black box techniques for deriving the test cases.

So though would share my experience with you all.



Introduction:

Testing is the most new kid in the block. This term is perhaps not very true, but true in the sense of implementation and recognition of testing as a practice in most of the organizations.
The earliest mention of software testing is found in 1967 in IBM white paper Evaluation of the Functional Testing of Control Programs William Elmendorf calls for a disciplined approach to software testing. Soon after this Software Engineering was introduced by a study group founded by NATO.
Looking back at the way this field has covered it has indeed made headway. Testing has diversified in the last decade into many domains and technologies. This decade we might witness the thinning of the line between White and Black box testing methodologies.
Almost Every aspect of testing has undergone changes except test design methodology. When we speak about test case design techniques two things that come to mind are Equivalence Class Partitioning and Boundary Value Analysis. These are well proven methodologies but they suffice only the functional part; what about the coverage? Can Traceability Matrix take care of that?

Traditional Test Case Design Techniques

I think now the focus is more on trying to uncover defects. In most of the cases in complex and big application this has been one of the pain points. Testers who write these cases fail to look at the coverage part which comes to highlight only when making the traceability matrix.
In General Test case design in broadly classified in two categories:
Black box (functional)
White box (structural)
Other
Specification derived tests
Equivalence partitioning
Boundary value analysis
State-transition testing
Branch testing
Condition testing
Data definition-use testing
Internal boundary value testing
Error guessing

Test Case Design using White box techniques

Test case design technique could use few techniques from White Box testing for deriving a very robust test suite.
Cyclomatic complexity for long has been used in white box testing only, but this can be effective used by testers to come to a conclusion to a minimum number to test cases to be written for a particular functionality as well as go on to derive different scenarios and then subsequently derive test cases from these scenarios.
  • It is a more effective way to detect defects than e.g. ad-hoc test cases.
  • The tests are reproducible because the order and content of the test execution are described in detail.
  • The standardized method ensures that the test process is independent of the individual who specifies and executes the test cases.
  • The standardized method ensures that the test specifications are transferable and maintainable.
  • It becomes easier to plan and manage the test process because the processes of test specification and execution can be split up into clearly definable blocks.
Example:
The below flowchart is for a program that accepts 3 numbers from the User and prints the highest.
 




Now the Cyclomatic Complexity of the above flowchart is 4 which is the min. number of required test cases.
Looking at the above flowchart the different paths from the above scenario are:
  1. 1,2,4
  2. 1,2,5
  3. 1,3,6
  4. 1,3,7
Now that we have the scenario we can further break down the scenarios to individual test cases using Equivalence and Boundary value analysis and State Transition testing methodologies.
Now applying other techniques on the above scenarios we have the following test cases:
Case 1:
  1. Enter A as the greatest number.
  2. Try to enter a character as the value of A
  3. Enter A as a negative value
  4. Enter A as 5.3 and B as 5.2
  5. Enter A as 5.3 and C as 5.2
  6. Enter all numbers of the same value
  7. Enter a very large value number for A
  8. Enter a very small value number for A like 0.1
  9. Try to enter symbols or garbage characters as value of A
  10. Try to enter the value of A as 0

Conclusion

From the above example we can make out that applying Branch Coverage along with traditional methods of test case design give a very comprehensive and exhaustive suit of test cases which can cover functional as well as non-functional and all other aspects of test case design. Of course the skill of the tester in thinking out the test data also is very crucial for test case designing.

References:

Android tools and commands



Android tools and commands


dmtracedump:
Generates graphical call-stack diagrams from trace log files. The tool uses the Graphviz Dot utility to create the graphical output, so you need to install Graphviz before running dmtracedump. For more information on using dmtracedump, see Profiling with Traceview and dmtracedump.

Draw 9-patch:
Allows you to easily create a NinePatch graphic using a WYSIWYG editor. It also previews stretched versions of the image, and highlights the area in which content is allowed.

Hierarchy Viewer (hierarchyviewer):
Lets you debug and optimize an Android application's user interface.

layoutopt:
Lets you quickly analyze your application's layouts in order to optimize them for efficiency.

mksdcard:
Helps you create a disk image that you can use with the emulator, to simulate the presence of an external storage card (such as an SD card).

Monkey:
Runs on your emulator or device and generates pseudo-random streams of user events such as clicks, touches, or gestures, as well as a number of system-level events. You can use the Monkey to stress-test applications that you are developing, in a random yet repeatable manner.

monkeyrunner:
Provides an API for writing programs that control an Android device or emulator from outside of Android code.

ProGuard:
Shrinks, optimizes, and obfuscates your code by removing unused code and renaming classes, fields, and methods with semantically obscure names.
sqlite3:
Lets you access the SQLite data files created and used by Android applications.

traceview:
Provides a graphical viewer for execution logs saved by your application.

Android Emulator (emulator):
A QEMU-based device-emulation tool that you can use to design, debug, and test your applications in an actual Android run-time environment.

hprof-conv:
Converts the HPROF file that is generated by the Android SDK tools to a standard format so you can view the file in a profiling tool of your choice.



Androidscreencast:
Desktop app to control an android device remotely using mouse and keyboard. Should work on Windows/Linux/MacOS with any android device.

Sensor Simulator Description:
Sensor simulator is a java standalone application that simulates sensor data and transmits them to the Android emulator.

Video Capture:
This tool allows you to capture what is displayed on your android device as a .avi video movie, useful for recording demonstrations or problem reporting.

re-signer tool:
If you want to test e.g. with Robotium against an existing APK file and you don’t have access to the source code, you need to resign the APK file with your own signature. With the re-signer tool you are now able to resign APK files over a graphical user interface. All you need is the re-sign.jar file and an installed Android SDK.

Cert-Checker:
To check the certificate from an APK file to see, if it is valid or not. Just Drag & Drop the APK file in the window and the tool will show you the SHA1 fingerprint of the APK file.

SeeTest:
In general, the process of working with SeeTest includes Connecting the physical mobile device (with USB/WiFi to your computer; note that On-Site Cloud configuration is also available) or the emulator, Recording a script and then Editing it. Once the script is ready, you play it, receive a report and can then export the test code to any testing framework you regularly work with (such as QTP, TestComplete, RFT, C#, JUnit, Python, Perl) and run the tests (as well as edit the test and view a report) from that framework.

Sikuli:
Sikuli is a visual technology to automate and test graphical user interfaces (GUI) using images (screenshots).





Listing of adb Commands

The table below lists all of the supported adb commands and explains their meaning and usage.

Category
Command
Description
Comments
Options
-d
Direct an adb command to the only attached USB device.
-e
Direct an adb command to the only running emulator instance.
Returns an error if more than one emulator instance is running.
-s 
Direct an adb command a specific emulator/device instance, referred to by its adb-assigned serial number (such as "emulator-5556").
If not specified, adb generates an error.
General
devices
Prints a list of all attached emulator/device instances.
help
Prints a list of supported adb commands.

version

Debug
logcat [
Prints log data to the screen.

bugreport

jdwp
Prints a list of available JDWP processes on a given device.
You can use the forward jdwp: port-forwarding specification to connect to a specific JDWP process. For example:
adb forward tcp:8000 jdwp:472
jdb -attach localhost:8000

Data
install 

pull  
Copies a specified file from an emulator/device instance to your development computer.


push  
Copies a specified file from your development computer to an emulator/device instance.


Ports and Networking
forward  
Forwards socket connections from a specified local port to a specified remote port on the emulator/device instance.
Port specifications can use these schemes:

tcp:
local:
dev:
jdwp:

ppp  [parm]...
Run PPP over USB.


— the tty for PPP stream. For example dev:/dev/omap_csmi_ttyl.
[parm]... &mdash zero or more PPP/PPPD options, such as defaultroute, local, notty, etc.

Note that you should not automatically start a PDP connection.
Scripting
get-serialno
Prints the adb instance serial number string.
get-state
Prints the adb state of an emulator/device instance.
wait-for-device
Blocks execution until the device is online — that is, until the instance state is device.
You can prepend this command to other adb commands, in which case adb will wait until the emulator/device instance is connected before issuing the other commands. Here's an example:

adb wait-for-device shell getprop

Note that this command does not cause adb to wait until the entire system is fully booted. For that reason, you should not prepend it to other commands that require a fully booted system. As an example, the install requires the Android package manager, which is available only after the system is fully booted. A command such as

adb wait-for-device install .apk

Server
start-server
Checks whether the adb server process is running and starts it, if not.

kill-server
Terminates the adb server process.

Shell
shell
Starts a remote shell in the target emulator/device instance.
shell []
Issues a shell command in the target emulator/device instance and then exits the remote shell.

Fastboot Commands:

fastboot devices – lists which devices in fastboot mode are currently attached to your computer
fastboot boot – boots a rom stored on your pc specified by the filename
fastboot flash – flashes a rom stored on your pc, partition can be one of {boot, recovery, system, userdata}