Automated Testing Methodology

Commonly Used Methodologies:

Record/Playback Method

· The Test Tool's Recording Mechanism records keystrokes, mouse actions, verification lists, etc.

Functional Decomposition Method

· The Application is broken down into Business Functions

· Automated Scripts are developed using the Tool's Scripting Language to perform those functions

· Data-Driven Process using Input & Verification Data Files

Test-Plan Driven Method

· The Test Cases are broken down into 'generic' testing actions

· Scripts are developed using the Tool's Scripting Language to perform these 'generic' actions

· Input File controls the processing as well as providing the Input and Verification data

Record/Playback Method

Advantages:

· Easy to Use

· Tester just starts recording and executes the Manual Test Case

Disadvantages:

· Reliability

The tester can make errors, which are then recorded, and have to be corrected

Failure on replay due to timing issues

Failure on replay due to events that occur that were not recorded (pop-ups, messages, etc.)

· Maintenance

Scripts contain hard-coded data that must be updated if the Application or the Data changes

Scripts must be enhanced or corrected after recording

Scripts & verification need to be re-recorded if the Application processing changes

Functional Decomposition Method

Business Functionality is broken down into its fundamental operations.

Test Case Example: "Post a Payment and Verify Account Data is Updated Correctly"

This could be broken down into the following operations:

Navigation: Access Payment Screen from Main Menu

Post a Payment

Verify Current Balance Updated

Navigation: Return to Main Menu

Navigation: Access Account Record

Verify Account Data Updated (Balance, Next Payment Date, etc.)

Navigation: Return to Main Menu

Using this breakdown, we can extract Business Functionality:

Navigation Routine(s)

Payment Posting & Verification Routine

Account Data Verification Routine

Routines can be Data-Driven, using Input and Verification files.

Numerous Test Cases can be automated by simply adding Input and Verification files for each.

Functional Decomposition Method

Architecture:

Driver Scripts

· Perform Initialization (if required), then call Test Case Scripts in the desired order

· Can be arranged to account for Test Case dependencies (If Test 1 fails, skip to Test 4, etc.)

Test Case Scripts

· Call Business Function Scripts to perform the required Test Case actions and verifications

Business Function Scripts

· Perform specific Business Functions within the application

· Call Subroutine Scripts and User-Defined Functions tp perform specific actions

Subroutine Scripts

· Perform application-specific tasks required by two or more Business Function Scripts

User-Defined Functions

Perform general, specific tasks that can be used by any number of scripts

Navigation, Enter Data using Input File, Verify Data using Verification File, etc.

Advantages:

Maintenance

· Modular design: If a Business Function changes, only 1 or 2 scripts must be modified

· Complex Test Cases can be constructed by calling Business Function Scripts from a Main Routine

· Data-Driven: Script can be used for many Test Cases by using different input/verification files

Reliability

· Tester error is eliminated, as is "scripter error" after the script has been properly coded and tested

· Unexpected events (pop-ups, messages, etc.) can be anticipated and coded for

Disadvantages:

Maintenance

· Each Business Function requires a script. There may be hundreds of Business Functions

· Changes in Test Cases require updates to several sets of input/verification files for each Test Case

· Format of input/verification records must be strictly adhered to or the tests will fail

· Testers must maintain the input/verification records as well as the Test Case documentation

Totally Data-Driven Method

Testing Activity is broken down into it's fundamental actions.

Examples: Data Entry

Select item in list_box/combo_box

Set a radio_button on or off

Set check_box on or off

Set a spin_control to a value

Enter text in an edit_box or field

Actions

Press a push_button or key

Select a menu_item

Select a tab

Verification

Verify correct Screen or Window is displayed

Verify Data (corollary to Data Entry)

Verify Field or Object Attributes

Each Testing Action is Associated with a Key-Word

Each Key-Word is Associated with a Utility Script

Data Entry Key-Word: "Enter:" Utility Script: Enter()

Action Key-Word: "Action:" Utility Script: Action()

Verify Window Key-Word: "Verify:" Utility Script: Verify()

Verify Data Key-Word: "Verify_Data:" Utility Script: Ver_Data()

Verify Attributes Key-Word: "Verify_Attributes:" Utility Script: Ver_Attr()

A Spreadsheet can be used for Input to this process:

Key-Words are placed in Column-1

Parameters or Field/Object names are placed in Column-2

Data or Field/Object names are placed in Column-3

Column 4 is used for comments

How it Works:

Spreadsheet is saved as a tab-delimited (text) file

A Controller script reads and processes the tab-delimited file

Switch/Case is used to match on the Column-1 Key-Word

A "list" is created from the remaining columns (Column-2~Column-3|Column-2~Column-3|etc.)

This continues until a blank line is reached

The Controller script then calls the Utility Script associated with the Key-Word, passing it the created list.

The Utility Script processes the input list from the Controller

Splits the input list into an array

Processes the Column-2 & Column-3 data in much the same way as the Controller Script

Returns to Controller script when finished

A Driver script processes multiple Test Cases, calling the Controller script

The File Name of the tab-delimited text file representing the Test Case is passed to the Controller

Architecture:

Driver Scripts

· Perform Initialization (if required), then call the Controller Script passing it the Test Case file name

· Also can be arranged to account for Test Case dependencies (If Test 1 fails, skip to Test 4, etc.)

Controller Script

· Calls Utility Scripts associated with Key-Words to perform the Test Case actions and verifications

Utility Scripts

· Perform specific Testing tasks required: Data Entry, Actions, Data Verification, etc.

· Call Subroutine Scripts and User-Defined Functions to perform specific actions

Business Function Utility Scripts

· Perform required application-specific tasks which may be a combination of Testing tasks

· Are associated with application-specific Key-Words and are parameterized within the Spreadsheet

User-Defined Functions

· Perform general tasks that can be used by any number of scripts

Architecture using TestDirector: TestDirector acts as the Driver

Single Test-Case Drivers call the Controller Script, passing the name of the tab-delimited

File Structure:

C:\

· AUT ------------- Application Under Test - Main Dir

· APP_Init ---------- --- Initialization Scripts

· DLL_Lib --------- --- DLLs to import (if required)

· Exp_Res --------- --- Common Expected Results Dir

· Fcn_Lib ----------- --- Application-Specific Functions

· GUI_Lib ---------- --- GUI Files for the Application

· Parameters ------ --- Parameter Files (set variables represented in the spreadsheet)

· Scripts ------------ --- Application-Specific Scripts

· AUT_Util ---- ---------- Application-Specific Utility Scripts

· Bus_Fcn ------ ---------- Application-Specific Business Function Scripts

· Drivers ------- ---------- Application-Specific Driver Scripts

· Recorded ---- ---------- Application-Specific Recorded Scripts

· TestData --------- --- Tab-Delimited Files

· TestPlan ---------- --- Spreadsheets / Workbooks

· TestRept --------- --- Customized Test Reports

· ToolKit_Fcn -- General ToolKit Function Libraries

· ToolKit_Util -- General Utility Scripts

Using TestDirector as the Driver:

To use Mercury Interactive's TestDirector, a slightly different architecture must be applied. TestDirector is designed to manage and call each different test or test case. Therefore we insert a Test Case Driver between TestDirector and the Controller Script:

TestDirector

· Calls the Test Case Driver script for each Test Case

· May also call a Test Case Script composed of Business Functions

· Additionally may call a recorded script, or any type of script

Test Case Driver

· Calls the Controller script, passing to it the Test Case name

· Test Case Driver folder contains the Spreadsheet file and the tab-delimited file

· Test Case Driver folder may also contain any other input/verification files required

From this point, the architecture is the same: The Controller script calls Utility / Business Function Scripts associated with Key-Words Utility Scripts perform the specific Testing Tasks delineated within the Spreadsheet

Business Function Scripts perform application-specific Testing Tasks User-defined functions perform generic tasks that can be used by any script as required.

Test Case Driver

Test Case Driver Folder contains all files & data relevant to the Test Case:

C:\ Mercury

AUT--------- Application Under Test - Main Dir

TestCases----- --- Test Case Driver Scripts (Called by TestDirector)

TC0001---- -------- Test Case Driver Script Folder

db (Folder)

exp (Folder)

Header (Test Properties)

Script (WinRunner TSL Script)

Script.bak (Script Backup Copy)

TC001.xls (Spreadsheet Test Case)

TC001.txt (Tab-Delimited File)

Results (Folder)

TC001R.txt (Customized Test Results)

Summary:

The Keyword-Driven method allows the Tester to create Test Cases in a Spreadsheet and have these be used as Automated Test Cases without having to use or learn the Test Tool. Automated Script development and maintenance is reduced, as only a few dozen scripts will be needed rather than hundreds. Test Case (Spreadsheet) maintenance is neither increased nor reduced, as the Tester must currently update the manual test cases anyhow when changes or revisions must be made. Only one "Test Tool Expert" will be required. The whole testing staff will not have to learn to use the test tool (unless they want to). Business Function Scripts and Recorded Scripts can still be used under this architecture if required. Parameterized input as variable-data allows Test Case (Spreadsheet) to be re-used in multiple-database environments. Pre-written "ToolKit" Functions and Utility Scripts greatly reduces "ramp-up" time.