Fighting estimation fatigue - Part 2: Introduction to standard estimation models and methods

Let's first start with the basics of estimation thing and then I will touch upon some of the popular and useful estimation models. This will give a solid foundation for the upcoming posts of this Software Estimation series.

A Google search reveals the “meaning of estimation” as “a rough calculation of the value, number, quantity, or extent of something”. Even though by definition “estimation” is a rough calculation but most of us when we hear the word “estimation” in software development, we internally treat it as “actual” and commitment. Software Estimation is often considered as a black art which is mostly neglected during the inception of an Information Technology Project, nonetheless, almost always is used to formulate the two most important attributes of a project i.e. the cost and the deadline, and often with rigid expectation of high level of accuracy.

Software Estimation is an widely, and in some cases overly, used topic in the field of computer science and software engineering. People were looking for a panacea where one can predict the schedule....A wide range of models have been developed since mid of twentieth century to solve the premise of sizing the software before it is built. Some of them are very effective at their time with a certain programing practice whereas the usefulness of some of them transcended the boundary of technology and programing practices. Here are some of the very popular and widely used software cost estimation models and metrics:

 

Line of Code (LOC/SLOC)

Though Line of Code or Source Line of Code (SLOC) is not a software estimation model in itself, this is the most widely used metric in software estimation models to determine the size e.g. COCOMO, SLIM, SEER-SEM, PRICE-S. Moreover, the most of ad-hoc estimation models used in the software house, and they are the majority in the software estimation landscape, uses this metric as in input to estimate the software development effort. The popularity of the LOC is not because that this gives an accurate picture of the size of the software but due to its simplistic connotation to the direct result of programming work of developing a software product. The primary advantage of SLOC is that it’s easily agreeable by the parties that are involved in the project to consider SLOC for software sizing as, in reality, source code is the apparent building block of the software. Despite of its sheer popularity, the use of LOC in software estimation is the biggest contributor to estimation inaccuracy. The reasons behind this inaccuracy are:

• There’s no standard or single agreed upon method on counting Source Line of Code
• SLOC is language dependent; changing the programming language immediately impacts it
• It is inherently inconceivable to predict SLOC from scope document with a very high level of requirements
• There’s a psychological toll of SLOC as it may incentivize the bad coding practice thus increases the SLOC

 

COCOMO

COCOMO, stands for COnstructive COst MOdel, is one of the early generation software cost estimation model and enjoyed its early popularity in nineteen eighties. It uses Line of Code as a basis of estimation and suffers all the shortcomings that I mentioned above in SLOC. I believe COCOMO is not very effective in this software development age where Object Oriented Programming rules the world and LOC doesn’t make a whole lot of sense in OOP in terms of effort estimation. If you’re still interested (after my all wrath on COCOMO), you can visit http://en.wikipedia.org/wiki/COCOMO and http://www.softstarsystems.com/overview.htm to learn more

 

Function Point Analysis

The obvious limitations on guessing LOC of a to-be-built software paved the way for another popular and somewhat realistic during early phase of a software project , which his called Function Point Analysis (FPA). In FPA, the software size is measured through a construct termed ‘Function Points’ (FP). Function points allow the measurement of software size in standard units, independent of the underlying language in which the software is developed. Instead of counting the lines of code that make up a system, count the number of externals (inputs, outputs, inquiries, and interfaces) that make up the system.

There are five types of externals to count: External inputs, External outputs, External inquiries, External interfaces, and Internal data files. The below Value Adjustment Multiplier (VAM) formula is used to obtain the function point count:

Where Vi is a rating of 0 to 5 for fourteen factors predefined factors.

The primary advantages of using Function Point Analysis model is that it measures the size of the solution rather than the problem and is extremely useful for the transaction processing systems (e.g. MIS applications). Moreover, FPA can be derived directly from the requirements and easily understood by the non-technical user. However, it does not provide an accurate estimate when dealing with command and control software, switching software, systems software or embedded systems. Moreover, FPA isn’t very effective in Object-Oriented software development that uses Use Cases and converting Use Cases into Function Points may be counter intuitive.

 

Use Case Points Method (UUCPM)

Similar in concept to function points, the theoretical basis of the Use Case Points Method, first described by Gustav Karner, is based on use cases as a basic notation for representation of functionality, and uses case points, like function points, measure the size of the system. Once we know the approximate size of a system, we can derive an expected duration for the project if we also know (or can estimate) the team’s rate of progress. In this approach, the first step is to calculate a measure called the unadjusted use case point (UUCP) count. To the UUCP count is applied a technical adjustment factor, as per FPA, albeit the factors themselves have been changed to reflect the different methodology that underpins development with use case. In addition, the UUCPM also defines a set of environmental (project) factors that contribute to a weighting factor that is also used to modify the UUCP measure. Four important formula used in UCP are Unadjusted Use case Points (UUCP), Technical Complexity Factor (TCF), Environment Factor (EF) and Use Case Point (UCP):

Once the number of UCP has been determined, an effort estimate can be calculated by multiplying the number of UCP by a fixed number of hours. 

The primary advantages of UCP model is that it can be automated thus saving the team a great deal of estimating time. Of course, there’s the counter argument that an estimate is only as good as the effort put into it. Additionally, they are a very pure measure of size as it allows separating estimating size from deriving duration. Moreover, by establishing average implementation time per UCP, forecasting is possible for future schedules. In the contrary, the fundamental problem with UCP is that the estimate cannot be arrived at until all of the use cases are written. While use case points may work well for creating a rough, initial estimate of overall project size, they are much less useful in driving the iteration-to-iteration work of a team. A related issue is that the rules for determining what constitutes a transaction are imprecise and since the detail in a use case varies tremendously by the author of the use case, the approach is flawed. Moreover, few Technical Factors do not really have an impact across the overall project, yet, the way they are multiplied with the weight do impact the overall size. The installation ease factor is an example of such. Finally, like most other estimation models, they do not fit into the agile development methodology.

 

User Story Points (USP)

User stories, which help shifting the focus from writing about requirements to talking about them, are the foundational block of Agile development technique. All Agile user stories include a written sentence or two, and more importantly a series of conversations about the desired functionality from the perspective of the user of the system. In Agile world, the method of estimating the size of the software is the User story points which are a unit of measure for expressing the overall size of a user story, feature, or other piece of work.  It tells us how big a story is, relative to others, either in terms of size or complexity by using relative sizing technique. Popular sizing techniques include- Fibonacci series (1, 2, 3, 5, 8, 13, 21, etc.) and T-shirt size (S, M, L, XL, XXL, etc.). Then estimate the size of each user stories with the entire team, usually through planning poker session.

The major advantage of User Story Point is that it is relatively easy and fun to estimate the relative size of a user story. Also the estimated size of the product is the outcome of a consensus of the team, so the ownership is, which has psychological influence to achieve higher throughput. On the contrary, benchmarking of size is challenging as the story points taken by one team cannot be compared with another team’s USP. Also some people may find it hard to get to the duration form the USP as there’s practically no direct relationship between user story point and person hour. If the team is not diverse enough to balance out the skewed sizing due to the biasness of a particular group, the sizing, eventually, could be proved to be useless. A subtle risk exists of inflated sizing by the development team if management has unrealistic expectation to show higher velocity to prove productivity

 

Delphi Method

The Delphi Method is an information gathering technique that was created in the 1950s by the RAND Corporation. The Delphi Method is based on the surveys and makes use of the information of the participants, who are mainly experts. Under this method of software, project specification would be given to a few experts and their opinion taken. The steps taken to get the estimation using are:  (i) Selection of experts, (ii) Briefing the experts about the project, objective of the estimation and overall project scope and clarification, (iii) Collate the estimates (software size and development effort) received from the experts and finally (iv) Convergence of estimates and finalization

The major advantages of Delphi technique are that it’s very simple to administer and also can be derived relatively quicker. Also, it is useful when the organization does not have any in-house experts with the domain knowledge to come out with a quick estimate. On the contrary, the disadvantages primarily come from selecting the wrong experts as well as getting adequate number of experts willing to participate in the estimation. Moreover, it is not possible to determine the causes of variance between the estimated value and the actual values.

 

Heuristic Method

Heuristic methods of estimation are essentially based on the experience that exists within a particular organization where past projects are used to estimate the required resources necessary to deliver future projects. A convenient sub-classification is to divide heuristic methods into ‘top-down’ and ‘bottom-up’ approaches. These approaches are the de-facto methods by which estimates are produced and as such they are implicated in being poor reflections of the actual resources that are required as evidenced by the failure of projects to be estimated accurately.  Top-down approaches to effort estimation may rely on the opinion of an expert whereas the Bottom-up estimation is the process by which time needed to code each identified module is estimated based on the discrete tasks that must be performed, such as analysis, design and project management.

Despite its simplistic approach, the error margin of heuristic method’s estimation is not proven to be worse than any other parametric or algorithmic (COCOMO, UUCP, etc.) estimation models. This doesn’t come free of risk either. The lack of access to historical data will cause high degree of error margin of future project and the underlying assumption of repeatability of organizations success could be proved to be deadly. Moreover, if not conducted systematically, tasks such as integration, quality and configuration could be overlooked

There is another category that usually people don't talk a lot, which is - home grown estimation models. Those models are created by the team members of a software project where they're working for quite a long period of time and those work pretty good for their projects. As those kind of estimation models typically aren't standardize to use for software projects outside of that group, those are usually not made public. To get a flavor of that kind of models, you can check out the JEE Software Development Estimation post where I have posted such a model which I used in 2008 in one of my software project.

In my next post of this series, I would cover the famous "Cone of Uncertainty" in software project management as well as the influence of human psychology in software estimation. Stay tuned!

Alternative strategies of Software Testing

Nobody denies the fact that every software should be tested and the best option to make sure that is to test all the features of it. That's no brainer, and for that one doesn't need to be expert in QA or software expert. But, often, the reality on the ground may not allow the QA team to go through that "best" route to make the software "bug free". So, what are the alternatives that we have to release a software with an optimal level of testing, when the reality doesn't let you to execute the complete test suite? I have broadly touched that in another post how much testing is enough for software with few tips on how to justify the QA test strategy to perform partial testing. Let's take a look into it from different perspective to derive some kind of strategy that would allow QA team to react at the time of necessity.

There could be number of scenarios that would demand to react differently. However, I would like to focus on three test strategies that I found helpful based on my experience and also probably cover much of the testing ground:

Risk Weighted Testing

This testing strategy actually goes back to the basic question on why in first place we need a QA team and what's the purpose of testing. To me, the QA process exists in Systems Development Life Cycle (SDLC) to reduce the risk of slipping bugs into the production environment i.e. to the end users. I know this would immediate stir up controversy but there's a reason I am saying that. You ask the question to any software professional (including QA professional) that if they can proclaim a software is "completely" bug free after the QA release. I can guarantee you that nobody would dare to claim. The reason is, QA process makes a software relatively bug free and make it usable to an extent that is paid for by the end users. Having said all of that, the best strategy, at the time of time and cost pressure, to effectively conduct the testing would be go with Risk Weighted Testing (RWT) approach. In this strategy, the risk associated with each functionalities would be calculated in number of dimensions and rank them to create an ordered list. That ranked list would be used in pragmatic fashion depending on the budget, time and resource constraints. If the time is the constraint and only few days are given to the QA team, then QA team should start from the top ranked functionality and go through the list sequentially. On the other hand, if QA team is asked to take as minimum functionalities as possible to test, they should draw a line where it would cover the most risk with least resource (can use Pareto Principle)

The primary challenge of this strategy is to create the risk weighted list of functionalities. There are several techniques and tools can be used and here I'm explaining a few of them:

Brainstorming: the most efficient and cost effective method is to have software developers, business analysts, testers, and end users in a conference room and brainstorm the list. People often misuse the term Brainstorming by just having a controlled discussion but if brainstorming is done properly, it can create dramatic result. 

Delphi method: if you have access to group of experts, you can use Delphi method where the experts can take each functionality of the software and risk weight them through a rigorous round of refinement. This is comparatively costly than Brainstorming

Analogy technique: using the past experience within the same company, the potential risks of functionality can be identified. This is the least expensive but often most effective if there are high number of reference projects available with expert personnel to make that analogy.

There also exist quantitative approaches like Monte-Carlo simulation, PERT distributions, etc. to use to create the risk ranked list if appropriate resources are available

Change Oriented Testing

In one of my project, we had introduced a bug into a perfectly working functionality while implementing an enhancement by modifying a common piece of code. Apparently an unrelated functionality, though it proved not to be so unrelated,  got broken due to that common code change which was done for right purpose i.e. to achieve code reusability and with right intention. Unfortunately, the bug introduced due to that enhancement was slipped through all the way to production. This provoked me to think on the existing testing strategy which proved to be insufficient to provide a good solid test coverage (and obviously everyone takes their best shot to squeeze QA cycle as much as they could do). To avoid that kind of situation, this Change Oriented Test (COT) strategy where Development team and QA team would work hand in hand to identify what piece of code or component are changed in the software with the potential impact areas (often labeled as: functionality).  The basic and foundation could be a simple Change-Functionality Impact Matrix that would capture changes as rows and functionalities as columns with X-marked intersection as impact. QA team can't develop this matrix alone by themselves but with the help of Development team a fairly accurate matrix can be created where it would be possible to identify the impact areas and then subsequently plan for the testing to cover the impacted functionalities.

To make it successful, the below steps should be continued to be in place in the SDLC:

- In every release Development team will create a mapping of code components to requirements and hand over to QA team ahead of testing cycle

- The release note would have one section explaining the changes along with the published impact matrix

The downside of this strategy is that if an automated tool (and embedded in IDE) isn't used (I didn't use any such tool and had done in spreadsheet which proved to be expensive)

Usage Driven Testing

The approach of testing a computer software or systems isn't something novel but what I found is that this isn't used explicitly in the Software Development Life Cycle (SDLC) by the Quality Assurance (QA) team. At least I haven't seen that in any Test Strategy document or Test Plan QA team is specified a concrete plan on studying and analyzing end users behavior and psychology to come up with test plan, test scenario, and test data.

The fundamental idea behind this strategy is to focus on the features or areas that the users would use most. The automatic next question would be - "how you would know before you release the software?". Yes, I don't have crystal ball to forecast that but it may not be impossible to get an idea of it, through -

- Surveying among the business users on their priority ranking of every features of the software. Then draw a line at, for the sake of the discussion, third quantile and test all the features thoroughly that fall within that bucket while leaving the rest to only smoke test kind of

- Another approach could be shadowing users day to day activity to come up with the list of mostly usage. The limitation of that is you can only do that if you're automating existing business process, enhancing existing software, rebuilding a software etc.

- The last, and least preferred with highest risk, option could be use expert judgement and pick your list. But this approach is better than to release software by random testing or half done testing with no strategy

The way, in some, TV shows mention the disclaimer at the beginning of an apparently risky techniques as "Do not try this at home", I would like to reiterate this at conclusion that all of the above testing strategies should be considered only in a situation where the full testing cycle can not be performed and the senior management understands and make the conscious call to go with those one of alternate routes to meet business need e.g. customer need, time to market, beat the competitors, etc.

Fighiting Estimation Fatigue - Part 1: Software Project Management perspective

Software Estimation is often considered as a black art which is mostly neglected during the inception of an Information Technology Project, nonetheless almost always is used to formulate two most important attributes of a project i.e. the cost and the deadline, and often with a rigid expectation of high level of accuracy to hit the targeted project deadline. 

There is a viscous cycle in the whole estimation thing: when executives ask the Project Manager for estimation, it is often portrayed as, which should always be the true purpose of estimation, which is, to get a "ball park" figure to gauge the depth of the water or to be exact, should the executive be bother about the project and invest her time to materialize the project from its conceptual level. But, as soon as the estimation is handed over for that project, that number is considered as a commitment from the project manager. Later on, if that project deviates from that supposedly “ball park figure” estimation, the project manager is personally held liable for that deviation. 

 

So the project managers are either unwilling to provide an estimation at all or pad the estimation to cover themselves from the uncertainties and unknowns of projects. This is a classic “catch 22” situation for most of the project managers who are managing software development projects or planning to start a career as Project Manager of Information Technology.  Now, the million dollar question would be: how an IT PM should walk on that double edge sword, if you will, to provide true estimation of software project without falling into that “commitment” trap.

I am working on a series of blog posts where I would try to answer that question including overview of some of the well-known, in terms of market share and most commonly used, software cost estimation models such as Function Point Analysis, Use Case Point, and Heuristic method. I would create a framework and guideline to help the IT Project Managers to choose the right estimation model for their software projects. The posts would also have a list of best practices that would help to fight “estimation fatigue” and prepare Project Managers to duck the most common pitfalls where they’re forced to respond with a “ball park figure”, sometime at a water-cooler conversation, at a very early stage of projects. There is an important, but often ignored, characteristic of software estimation which is called “the cone of uncertainty”. I would also touch upon that characteristic in brief by showing how to effectively use that phenomenon to build confidence and earn the trust of the executive sponsors and senior management of your organization.