Tuesday, 18 October 2011

Smart Way to Identify Test Cases

Article written by Sreenivas Mothukuru

Software testing is an important activity during software development life cycle. In general, Testing process consists of a) identifying test cases, b) executing test cases and c) analyzing test cases. Test execution and Test Analysis largely depend on the set of test cases. Identifying test cases for testing is always a challenging job. For any application under test (AUT) the key factors which either make or break software application is identifying correct set of cases at right time. As complexity and size of software system grows, scope of testing will increase proportionally.

Let us look at a few questions ...
1) Why do some software/hardware applications fail in fields such as medical, aeronautical, banking, insurance, etc.?
2) What causes software failures (logical errors, inadequate validation, interaction faults*, integration issues, not handling exceptions, etc.)?
3) Is there a better way to anticipate such failure scenarios in the early stages?

4) Is there a way to choose minimal set of relevant cases which covers most of the combinations irrespective of Test Engineer’s efficiency? 
In general, defects in any software can be detected by understanding interacting parameters and the importance of the interacting parameters.

*Interaction faults (Example of interaction fault):

pressure < 10 (1-way interaction = testing with pressure values less than 10 catches)
pressure < 10 & volume > 300 (2-way interaction = testing with pressure and volume in pairs)

pressure < 10 & volume > 300 & velocity = 5 (3-way interaction)‏
In real time 1-way interaction is very rarely seen in software applications. Pair wise testing is most commonly seen in any application and using pair wise testing we can find about 50% to 90% of defects. Also, the more complex failures are seen in 4 (to 6)-way interaction.
FDA Medical Devices:
~66% of defects are detected when medical devices were tested based on 1-way interaction.
~98% by 2-way interaction
~99% by 3-way interaction
~99% by 4-way interaction
Web applications (Green):
~41% of defects are detected when web applications were tested based on 1-way interaction.
~70% by 2-way interaction
~88% by 3-way interaction
~94% by 4-way interaction
~95% by 5-way interaction
~99% by 6-way interaction

NASA Distributed Database (Light Blue):
~66% of defects are detected when NASA distributed database was tested based on 1-way interaction.
~93% by 2-way interaction
~98% by 3-way interaction
~99% by 4-way interaction
~99% by 5-way interaction
~99% by 6-way interaction


Check below examples.

Example 1: 
What is the maximum number of test cases you can imagine when you are asked to test an application on OS (XP, OS X, RHL), Browser (IE, Firefox), Protocol (IPv4, IPv6), CPU (Intel, AMD) and DBMS (MySQL, Sybase, Oracle)?

3 OSs:  XP, OS X, RHL
2 Browsers: IE, Firefox
2 Protocols: IPv4, IPv6
2 CPUs: Intel, AMD
3 DBMSs: MySQL, Sybase, Oracle


Answer: 72 test cases (3 * 2 * 2 * 2 * 3) considering the number of parameters as 5 (OS, Browsers, Protocols, CPU, DBMS) and the degree of interaction coverage as 5. In this example, degree of interaction coverage 5 means all combinations with values within 5 parameters are important to test the application. Based on the knowledge of interacting parameters (within application) and the values within parameters we can reduce the scope of testing.

72 test cases (100% of exhaustive cases) - degree of interaction coverage as 5.
36 test cases (50% of exhaustive cases) - degree of interaction coverage as 4.
18 test cases (25% of exhaustive cases) - degree of interaction coverage as 3.
10 test cases (14% of exhaustive cases) - degree of interaction coverage as 2.


Example: Degree of interaction coverage as 2.

Example 2: 
What is the maximum number of test cases you can imagine to test Andriod Smart Phone application with below configuration?

Total possible test cases: 1,72,800 (3 x 3 x 4 x 3 x 5 x 4 x 4 x 5 x 4)

Based on the domain knowledge of smartphones, interaction parameters (within application) and the values within each parameter we can intellegently narrow down testing by choosing appropriate test cases.

1,72,800 test cases (100% of exhaustive cases) - degree of interaction coverage as 9.
9168 test cases (5.3% of exhaustive cases) - degree of interaction coverage as 6.
2532 test cases (1.5% of exhaustive cases) - degree of interaction coverage as 5.
625 test cases (0.4% of exhaustive cases) - degree of interaction coverage as 4.
137 test cases (0.08% of exhaustive cases) - degree of interaction coverage as 3.
29 test cases (0.02% of exhaustive cases) - degree of interaction coverage as 2.

Example 3:
How to test below system with several "ON" and "OFF" switches?
If a similar system has 34 switches then the possible test cases are 234 (i.e. 1.7 x 1010).

Increase in interactions within the system demands more and more testing. If all defects/faults are triggered by the interaction of 't' or fewer variables, then testing all t-way combinations can provide strong assurance. But, is it practical / reasonable to test all combinations for such system involving more parameters and variables considering limitations such as strict timelines, limited resources, etc.? 

What if a Test engineer knows a fact that any combination of 4 switches is very critical (out of 34) and rest all switches work fine in any combination. If this is the case then testing 234 test cases doesn't make any sense! With proper domain knowledge a Test Engineer can well manage and adjust testing scope. Also, it would be very helpful if there is any tool which can intellegent choose minimal set of relevant cases which covers most of the combinations? 

Kuhn, D. Richard & Kacker, Raghu N did extensive research on Combinatorial Testing. If you would like your QA/Testing team to work intelligently while coming up with test cases then I suggest you understand Combinatorial Testing.
Follow below steps:
- To request a copy of ACTS software, send an email to Rick Kuhn - kuhn@nist.gov.  

Note: Images or examples or content shown in this article are copied from the excellent material composed by Kuhn, D. Richard & Kacker, Raghu N.

Friday, 14 October 2011

Who You Are Makes A Difference

Article written by Sreenivas Mothukuru

The article "Who You Are Makes A Difference" from the book titled "Chicken Soup for the Soul" (presented by Bala, Vice President, UOL on 5 years completion) inspired me to acknowledge my colleagues at workplace and friends.

If you are inspired / motivated by your colleagues, team members, boss, etc. please take some time and acknowledge him / her in your own words (using below template). The purpose of this is to "acknowledge the worth and importance of others, Communicate with compassion, acknowledgment and love, Empower and support dreams and Make a difference where we live, work and learn".


Template:
-------------------------------------------------------------------------------------------------------------------
"Who You Are Makes A Difference"

-------------------------------------------------------------------------------------------------------------------
Hey <Name>,


"I admire/appreciate you because ..............."

Thanks,
<Your Name>

The purpose of this is to:

1)      Acknowledge the worth and importance of others
2)      Communicate with compassion, acknowledgment and love
3)      Empower and support dreams
4)      Make a difference where we live, work and learn
-------------------------------------------------------------------------------------------------------------------
Short Story on "Who You Are Makes A Difference":
A teacher in New York decided to honor each of her seniors in high school by telling them the difference they each made. Using a process developed by Helice Bridges of Del Mar, California, she called each student to the front of the class, one at a time. First she told them how the student made a difference to her and the class. Then she presented each of them with a blue ribbon imprinted with gold letters which read, "Who I Am Makes a Difference."

Afterwards the teacher decided to do a class project to see what kind of impact recognition would have on a community. She gave each of the students three more ribbons and instructed them to go out and spread this acknowledgment ceremony. Then they were to follow up on the results, see who honored whom and report back to the class in about a week.

One of the boys in the class went to a junior executive in a nearby company and honored him for helping him with his career planning. He gave him a blue ribbon and put it on his shirt. Then he gave him two extra ribbons, and said, "We're doing a class project on recognition, and we'd like you to go out, find somebody to honor, give them a blue ribbon, then give them the extra blue ribbon so they can acknowledge a third person to keep this acknowledgment ceremony going. Then please report back to me and tell me what happened."

Later that day the junior executive went in to see his boss, who had been noted, by the way, as being kind of a grouchy fellow. He sat his boss down and he told him that he deeply admired him for being a creative genius. The boss seemed very surprised. The junior executive asked him if he would accept the gift of the blue ribbon and would he give him permission to put it on him. His surprised boss said, "Well, sure.". The junior executive took the blue ribbon and placed it right on his boss's jacket above his heart. As he gave him the last extra ribbon, he said, "Would you do me a favor? Would you take this extra ribbon and pass it on by honoring somebody else? The young boy who first gave me the ribbons is doing a project in school and we want to keep this recognition ceremony going and find out how it affects people."

That night the boss came home to his 14-year-old son and sat him down. He said, "The most incredible thing happened to me today. I was in my office and one of the junior executives came in and told me he admired me and gave me a blue ribbon for being a creative genius. Imagine. He thinks I'm a creative genius.". Then he put this blue ribbon that says "Who I Am Makes A Difference" on my jacket above my heart. He gave me an extra ribbon and asked me to find somebody else to honor. As I was driving home tonight, I started thinking about whom I would honor with this ribbon and I thought about you. I want to honor you. "My days are really hectic and when I come home I don't pay a lot of attention to you. Sometimes I scream at you for not getting good enough grades in school and for your bedroom being a mess, but somehow tonight, I just wanted to sit here and, well, just let you know that you do make a difference to me. Besides your mother, you are the most important person in my life. You're a great kid and I love you!".

The startled boy started to sob and sob, and he couldn't stop crying. His whole body shook. He looked up at his father and said through his tears, "I was planning on committing suicide tomorrow, Dad, because I didn't think you loved me. Now I don't need to." 
-------------------------------------------------------------------------------------------------------------------
- Sreenivas Mothukuru