Alternative strategies of Software Testing

Nobody denies the fact that every software should be tested and the best option to make sure that is to test all the features of it. That's no brainer, and for that one doesn't need to be expert in QA or software expert. But, often, the reality on the ground may not allow the QA team to go through that "best" route to make the software "bug free". So, what are the alternatives that we have to release a software with an optimal level of testing, when the reality doesn't let you to execute the complete test suite? I have broadly touched that in another post how much testing is enough for software with few tips on how to justify the QA test strategy to perform partial testing. Let's take a look into it from different perspective to derive some kind of strategy that would allow QA team to react at the time of necessity.

There could be number of scenarios that would demand to react differently. However, I would like to focus on three test strategies that I found helpful based on my experience and also probably cover much of the testing ground:

Risk Weighted Testing

This testing strategy actually goes back to the basic question on why in first place we need a QA team and what's the purpose of testing. To me, the QA process exists in Systems Development Life Cycle (SDLC) to reduce the risk of slipping bugs into the production environment i.e. to the end users. I know this would immediate stir up controversy but there's a reason I am saying that. You ask the question to any software professional (including QA professional) that if they can proclaim a software is "completely" bug free after the QA release. I can guarantee you that nobody would dare to claim. The reason is, QA process makes a software relatively bug free and make it usable to an extent that is paid for by the end users. Having said all of that, the best strategy, at the time of time and cost pressure, to effectively conduct the testing would be go with Risk Weighted Testing (RWT) approach. In this strategy, the risk associated with each functionalities would be calculated in number of dimensions and rank them to create an ordered list. That ranked list would be used in pragmatic fashion depending on the budget, time and resource constraints. If the time is the constraint and only few days are given to the QA team, then QA team should start from the top ranked functionality and go through the list sequentially. On the other hand, if QA team is asked to take as minimum functionalities as possible to test, they should draw a line where it would cover the most risk with least resource (can use Pareto Principle)

The primary challenge of this strategy is to create the risk weighted list of functionalities. There are several techniques and tools can be used and here I'm explaining a few of them:

Brainstorming: the most efficient and cost effective method is to have software developers, business analysts, testers, and end users in a conference room and brainstorm the list. People often misuse the term Brainstorming by just having a controlled discussion but if brainstorming is done properly, it can create dramatic result. 

Delphi method: if you have access to group of experts, you can use Delphi method where the experts can take each functionality of the software and risk weight them through a rigorous round of refinement. This is comparatively costly than Brainstorming

Analogy technique: using the past experience within the same company, the potential risks of functionality can be identified. This is the least expensive but often most effective if there are high number of reference projects available with expert personnel to make that analogy.

There also exist quantitative approaches like Monte-Carlo simulation, PERT distributions, etc. to use to create the risk ranked list if appropriate resources are available

Change Oriented Testing

In one of my project, we had introduced a bug into a perfectly working functionality while implementing an enhancement by modifying a common piece of code. Apparently an unrelated functionality, though it proved not to be so unrelated,  got broken due to that common code change which was done for right purpose i.e. to achieve code reusability and with right intention. Unfortunately, the bug introduced due to that enhancement was slipped through all the way to production. This provoked me to think on the existing testing strategy which proved to be insufficient to provide a good solid test coverage (and obviously everyone takes their best shot to squeeze QA cycle as much as they could do). To avoid that kind of situation, this Change Oriented Test (COT) strategy where Development team and QA team would work hand in hand to identify what piece of code or component are changed in the software with the potential impact areas (often labeled as: functionality).  The basic and foundation could be a simple Change-Functionality Impact Matrix that would capture changes as rows and functionalities as columns with X-marked intersection as impact. QA team can't develop this matrix alone by themselves but with the help of Development team a fairly accurate matrix can be created where it would be possible to identify the impact areas and then subsequently plan for the testing to cover the impacted functionalities.

To make it successful, the below steps should be continued to be in place in the SDLC:

- In every release Development team will create a mapping of code components to requirements and hand over to QA team ahead of testing cycle

- The release note would have one section explaining the changes along with the published impact matrix

The downside of this strategy is that if an automated tool (and embedded in IDE) isn't used (I didn't use any such tool and had done in spreadsheet which proved to be expensive)

Usage Driven Testing

The approach of testing a computer software or systems isn't something novel but what I found is that this isn't used explicitly in the Software Development Life Cycle (SDLC) by the Quality Assurance (QA) team. At least I haven't seen that in any Test Strategy document or Test Plan QA team is specified a concrete plan on studying and analyzing end users behavior and psychology to come up with test plan, test scenario, and test data.

The fundamental idea behind this strategy is to focus on the features or areas that the users would use most. The automatic next question would be - "how you would know before you release the software?". Yes, I don't have crystal ball to forecast that but it may not be impossible to get an idea of it, through -

- Surveying among the business users on their priority ranking of every features of the software. Then draw a line at, for the sake of the discussion, third quantile and test all the features thoroughly that fall within that bucket while leaving the rest to only smoke test kind of

- Another approach could be shadowing users day to day activity to come up with the list of mostly usage. The limitation of that is you can only do that if you're automating existing business process, enhancing existing software, rebuilding a software etc.

- The last, and least preferred with highest risk, option could be use expert judgement and pick your list. But this approach is better than to release software by random testing or half done testing with no strategy

The way, in some, TV shows mention the disclaimer at the beginning of an apparently risky techniques as "Do not try this at home", I would like to reiterate this at conclusion that all of the above testing strategies should be considered only in a situation where the full testing cycle can not be performed and the senior management understands and make the conscious call to go with those one of alternate routes to meet business need e.g. customer need, time to market, beat the competitors, etc.

The great debacle of healthcare.gov

This is the first time in history when the president of the United States of America, or probably for any head of state around the world, is blamed for the quality of a software and had to stood up to defend a mere software. Though nobody should extract fun out of that fiasco, but I kind of appreciate the fact that how technology has taken the epicenter of every ones life in this country. The talk shows, news anchors, and congressional hearing, everyone is talking on how software development works, how much testing was done or not done, how the integration of subsystems works within software etc. This is all I do for my living so I couldn't hold my temptation to chip in and have my two cents on it.

I want to make it clear that this healthcare.gov is no way a simple system. I would rather say that this is an integrated platform of very complex systems in terms of the volume of work, need of the sheer amount of coordination with all the stakeholders and expectation of a real time communication among various existing government IT systems. Just to get an idea of how vast was the volume of work, check out this information: it consists of 6 complex systems, 55 contractors were working in parallel, 5 government agencies were involved, used in 36 States, and covers 300 insurers' 4500 insurance plans. All these tells us one thing, to deliver a successful system like that the planning must be very precise, and execution of that should have been close to perfect.

Having said that I would focus on three areas of it and would try to reflect on it based on my experiences that I gathered throughout these years of working in the field of software engineering. Before I jump into my analysis on the issues, let's see the architectural modules of the health care marketplace system. Among other, the system has these significant modules: the eligibility & enrollment module, plan and financial management module, Data Services module, Registration module, Income verification system etc.

Now let's take a look into the detail. The first area of failure, I believe, was proper planning. This system is designed as a real-time system and planned to go live in a big bang approach, which is, in my opinion, single biggest mistake. I would explain the real time system architecture in the later part, but the big bang release idea must have come from political part of the decision making process. This market place system connects to several disparate systems so it would have been a better plan to have it in a phased approach and then slowly ramp up users into the systems. The gradually ramping up of load could've been achieved through either integrating with as few external systems as possible or by rolling out the system to a few selected states and then gradually expand the user base. The other planning failure was, in my opinion, the belittling of the role of testing in software development. The testing of each systems might have been planned right, but the integration testing might have not received the required level of priority. Any plan should focus on the highest risk points and have a mitigation plan around those risks. There shouldn't be any doubt that integration of all these disparate systems should be the highest risk to mitigate for this health care marketplace system.

Second area that I would like to focus on is the architecture and design of this system. This market place platform was designed to work as a real time system which is a bit surprising to me that the healthcare.gov site is designed to be accessed by millions of users while in the back end it would be connecting to several federal government's legacy systems with the expectation that all those systems would respond back within a predefined time. How on earth they thought that every legacy systems of the government e.g. USCIS, IRS, Social Security systems etc., those are designed decades ago, would be able to deliver the expected level of performance that a real time systems, like healthcare.gov, would need to function. Real time system only makes sense when you have full control of all underlying systems or at least you're given the guarantee by the underlying systems to return with the response of a request within a predefined period of time. In this case, the underlying systems for the healthcare.gov neither has full control nor has guaranteed response time from dependent systems, so designing a real time system on top of multiple legacy systems is nothing but a pipe dream. The real time online market place should've started with a bottom up approach i.e. enable the downstream systems to deliver real time response and then open the grand door of healthcare.gov. Again, this may not be acceptable due to the politically set deadline of the Affordable Care Act but, in the other hand, you can't always squeeze or compromise with the software development process. On top of that, adding more people to the team isn't always a solution to get  software faster (for reference, check out the brooks law) but in fact it may end up adding more delay and chaos in the project.

The last area of my analysis is the test strategy of the software and its execution. In my opinion, the focus of the test strategy should've been on the early integration the system and of its testing because that's the biggest risk factor to the successful roll out of it. Though this could've been covered in the planning and architecture and designing phase, but a good solid test strategy has ability to influence the overall planning and cloud work as an early warning indicator of any disaster. If you aren't allowing the QA team to fully test it out after it's integrated, you may be living in a lala land assuming that it would all work fine as designed. I've mentioned in my several older posts that, the greatest mistake made by most of the software professionals is to make assumption and never verifying those assumptions ahead of time. As a side note, I've another post (How much testing is enough) on the importance of testing, and its strategy where I explained how should one decide to go for a confidence or risk based testing approach where I ended the post with my suggestion to  encourage the decision makers to boldly take risks that can be affordable. In this case, the Center for Medicare and Medicaid Services (CMS), the managing organization of this insurance market place platform, had gone through that path of taking risks but in absolute wrong way. They had decided to push the integration testing (the biggest risk factor of this kind of systems) to the very end of the release cycle and had given just two weeks (according to CGI Federal) for integration testing which is, to my point of view, a joke to the people who have a basic understanding of the software development process. Just compare this to the overall development period of the system which is more than two years. If the integration testing gets only two weeks, I don't dare to ask about how much time was given to have load testing and stress testing. From my personal experience of a software project, I remember we had failed to deliver software on time for which we came to know just a few weeks before the go live that we're not going to meet the deadline. The single most mistake, along with few others, was done during the planning of that project which was to push out the final integration to the very end of the release. The integration with the other system was designed on paper and, anyone involved in software development know that most of the time the design always looks shinier on paper but doesn't always glow similarly when you do the real coding (that's why we have Prototype and Proof of Concept model in software development process). Anyway, later when the real integration started, we found that the design on paper didn't work due to some very basic assumptions that went wrong (e.g. the primary key of two systems were named same but used for completely different purposes). which led us to a complete redesign which caused the go live to be pushed out by months. I've got a plan to write about that experience in detail in a separate post in the future.

At the end, I would like to close this with a positive note that after all, these are all fixable. I strongly believe that all these software glitches would be fixed but probably won't be overnight and might need to go through a thorough re-architecting of it. The best would have been to start it with asynchronous real time system in the first phase and then slowly move to synchronous real time while upgrading the other federal government legacy systems, but I don't think that government can afford this at this moment, so they've to swallow this bitter pill and work it right.

Share on Facebook

How much testing is enough for a software?

This is a million dollar question, especially when you're in a release crunch time and you found out that you don't have automated test suites for your, technically speaking , System Under Test (SUT). So, how would you define the testing boundary of a software? Like most other questions in this world, the answer is similar - it depends. It depends on what you're testing, why you're testing and most importantly how you're testing it- manually or in automated fashion. For the remainder of this post let me clarify what I believe is the goal of software testing which is to certify the behaviors of a software by labeling them if they work or they aren't; releasing it to the end user is a whole different ball game.

Let's first take the most important factor i.e. How you're testing. If you happen to have the fully automated test cases and test suits, my immediate response would be - Run 'em all. This is the safest and most cost efficient way to certify the behavior of a software. Like the Microsoft way, for MS Windows they execute the entire test suits and build it every night. If you can afford it do it, why take the chance. In Statistical world, we take sample because getting the entire population of data is unrealistic. Similarly if you don't have that luxury then you pick and choose based on the reality and expectations set for the software. I've explained it later part of this post.

Now the next factor is, when you're testing with the assumption that you don't have automated test coverage (or at least the time frame doesn't allow you to run the full automated test suits) and you have to test the software manually to certify it. If you are under pressure to complete the testing within a specified time frame then my suggestion is - go for targeted test. To determine what and how much to test, follow the Goldilocks' principle - don't over test it nor under test it, test which is JUST RIGHT. You'll always find that you can cover 80% of the software's feature by executing just around 20% of the test cases that you have and you would spend remaining 80% of your resources to cover the rest; check the famous 80-20 rule if you don't take my words - http://en.wikipedia.org/wiki/Pareto_principle. So identify that 20% test cases and run them and if you're asked to release your software before you are able to open the remaining test cases, go ahead and release the software. One important thing to remember, make sure you release it with a confidence factor attached with it - so for instance when you run 80% test cases of it, label it as "QA certified with 80% confidence". I know you can't release it to the external world with that tag but nonetheless you should have that number tagged with it to communicate the risk factor to the management. And most importantly, to cover your back.

The last but definitely not the least factor is- what you're testing. If you're testing the NASA's space flight module, you have no choice but to test it FULL and in that case you're certainly not told to test it by a certain time frame, but the release of it i.e. the mission launch date would be determined by when you're able complete 100% of the testing. Same is true when you're talking about medical equipment software or life supporting systems. But when you're testing a non-mission critical systems and missing a bug in the software won't take the company down (I remember once I logged in to Facebook and I was taken to profile of some one else while I could see certain photo album of mine within that profile; which happened in around 2010 and now in 2013, Facebook turned out as the de-facto platform for social networking - that bug was surely missed out by some test engineer but who did care about that) and you're given a deadline to meet, go ahead boldly with confidence based testing. One more thing about test coverage, there are some cases where you want to test both positive and negative test cases for a feature and there are cases where you're just fine of running only positive test cases (this is applicable for the software that are built to use in house and won't ever go out of the corporate intranet boundary)

The bottom line is, you always want to create every possible test cases for every features and run them all but you need to act based on the reality on the ground and don't hesitate to take calculated risks.