Smart Way to Identify Test Cases

Article written by Sreenivas Mothukuru

Software testing is an important activity during software development life cycle. In general, Testing process consists of a) identifying test cases, b) executing test cases and c) analyzing test cases. Test execution and Test Analysis largely depend on the set of test cases. Identifying test cases for testing is always a challenging job. For any application under test (AUT) the key factors which either make or break software application is identifying correct set of cases at right time. As complexity and size of software system grows, scope of testing will increase proportionally.

Let us look at a few questions ...

1) Why do some software/hardware applications fail in fields such as medical, aeronautical, banking, insurance, etc.?
2) What causes software failures (logical errors, inadequate validation, interaction faults*, integration issues, not handling exceptions, etc.)?
3) Is there a better way to anticipate such failure scenarios in the early stages?
4) Is there a way to choose minimal set of relevant cases which covers most of the combinations irrespective of Test Engineer’s efficiency? 

In general, defects in any software can be detected by understanding interacting parameters and the importance of the interacting parameters.

*Interaction faults (Example of interaction fault):
pressure < 10 (1-way interaction = testing with pressure values less than 10 catches)
pressure < 10 & volume > 300 (2-way interaction = testing with pressure and volume in pairs)
pressure < 10 & volume > 300 & velocity = 5 (3-way interaction)‏

In real time 1-way interaction is very rarely seen in software applications. Pair wise testing is most commonly seen in any application and using pair wise testing we can find about 50% to 90% of defects. Also, the more complex failures are seen in 4 (to 6)-way interaction.

FDA Medical Devices:

~66% of defects are detected when medical devices were tested based on 1-way interaction.
~98% by 2-way interaction
~99% by 3-way interaction
~99% by 4-way interaction

Web applications (Green):

~41% of defects are detected when web applications were tested based on 1-way interaction.
~70% by 2-way interaction
~88% by 3-way interaction
~94% by 4-way interaction
~95% by 5-way interaction
~99% by 6-way interaction

NASA Distributed Database (Light Blue):

~66% of defects are detected when NASA distributed database was tested based on 1-way interaction.
~93% by 2-way interaction
~98% by 3-way interaction
~99% by 4-way interaction
~99% by 5-way interaction
~99% by 6-way interaction

Check below examples.

Example 1: 

What is the maximum number of test cases you can imagine when you are asked to test an application on OS (XP, OS X, RHL), Browser (IE, Firefox), Protocol (IPv4, IPv6), CPU (Intel, AMD) and DBMS (MySQL, Sybase, Oracle)?

3 OSs:  XP, OS X, RHL
2 Browsers: IE, Firefox
2 Protocols: IPv4, IPv6
2 CPUs: Intel, AMD
3 DBMSs: MySQL, Sybase, Oracle

Answer: 72 test cases (3 * 2 * 2 * 2 * 3) considering the number of parameters as 5 (OS, Browsers, Protocols, CPU, DBMS) and the degree of interaction coverage as 5. In this example, degree of interaction coverage 5 means all combinations with values within 5 parameters are important to test the application. Based on the knowledge of interacting parameters (within application) and the values within parameters we can reduce the scope of testing.

72 test cases (100% of exhaustive cases) - degree of interaction coverage as 5.

36 test cases (50% of exhaustive cases) - degree of interaction coverage as 4.

18 test cases (25% of exhaustive cases) - degree of interaction coverage as 3.

10 test cases (14% of exhaustive cases) - degree of interaction coverage as 2.

Example: Degree of interaction coverage as 2.

Example 2: 

What is the maximum number of test cases you can imagine to test Andriod Smart Phone application with below configuration?

Total possible test cases: 1,72,800 (3 x 3 x 4 x 3 x 5 x 4 x 4 x 5 x 4)

Based on the domain knowledge of smartphones, interaction parameters (within application) and the values within each parameter we can intellegently narrow down testing by choosing appropriate test cases.

1,72,800 test cases (100% of exhaustive cases) - degree of interaction coverage as 9.

9168 test cases (5.3% of exhaustive cases) - degree of interaction coverage as 6.

2532 test cases (1.5% of exhaustive cases) - degree of interaction coverage as 5.

625 test cases (0.4% of exhaustive cases) - degree of interaction coverage as 4.

137 test cases (0.08% of exhaustive cases) - degree of interaction coverage as 3.

29 test cases (0.02% of exhaustive cases) - degree of interaction coverage as 2.

Example 3:
How to test below system with several "ON" and "OFF" switches?

If a similar system has 34 switches then the possible test cases are 2³⁴ (i.e. 1.7 x 10¹⁰⁾.

Increase in interactions within the system demands more and more testing. If all defects/faults are triggered by the interaction of 't' or fewer variables, then testing all t-way combinations can provide strong assurance. But, is it practical / reasonable to test all combinations for such system involving more parameters and variables considering limitations such as strict timelines, limited resources, etc.? 

What if a Test engineer knows a fact that any combination of 4 switches is very critical (out of 34) and rest all switches work fine in any combination. If this is the case then testing 2³⁴ test cases doesn't make any sense! With proper domain knowledge a Test Engineer can well manage and adjust testing scope. Also, it would be very helpful if there is any tool which can intellegent choose minimal set of relevant cases which covers most of the combinations? 

Kuhn, D. Richard & Kacker, Raghu N did extensive research on Combinatorial Testing. If you would like your QA/Testing team to work intelligently while coming up with test cases then I suggest you understand Combinatorial Testing.

Follow below steps:

- Go to http://csrc.nist.gov/groups/SNS/acts/documents/comparison-report.html#acts

- Read guide: http://csrc.nist.gov/groups/SNS/acts/documents/acts_user_guide_v2_r1.1.pdf

- To request a copy of ACTS software, send an email to Rick Kuhn - kuhn@nist.gov.  

Note: Images or examples or content shown in this article are copied from the excellent material composed by Kuhn, D. Richard & Kacker, Raghu N.