Software Confidence Index

[Finally I've materialized the concept that I had started in 2009 to measure the confidence index of a software or application. The initial idea of this indexing was mentioned in my another post in 2009 Project Confidence Level and that I've refined it through these last few years to finalize it to an Indexing scheme (I was influenced by the concept of Consumer confidence Index and partly by Credit Scoring scheme as I mentioned in my previous post)]

At the end of a successful software release, the members of the project team get some breathing space to reflect on the completed project and ask some questions specially surrounding the success of the project or lesson learned from it; as we don't like the word “mistake” to label our shortcomings so we sugar coat this as “lesson learned”. These questions are absolutely important to understand the level of success of any project but the irony is that, most of the time, the way these questions are answered are very vague in nature and almost impossible to quantify the success of that project.

Imagine a large conference room, full of hard core technical people, the managers and potentially the sponsors or business users of the delivered software, of the project completion meeting (there are so many fancy names of this kind of meetings - lesson learned, retrospective, project closure etc.) and everyone is eager to know how well or not so well was done. The first question usually is very simple which is most of the time asked by the Project Manager or Technical Manager of the project and that is “how have we done in this project?”. Though the question is so simple but the drama starts when people of different areas start to respond to this apparently simple question. 

The range of answers could be -

• "we've done a great job. The number of defect has gone down greatly …” - the person is most probably the technical lead of the project
• "It was good. The developers had written comparatively more codes in this release (along with LOC, if available)" - the person probably is a senior developer or technical lead
• "The quality of business requirement was better; the developer didn't have much complain about requirement issues in this release" - this answer is certainly from a business systems analyst
• Now the business sponsors or users weigh in - “The software greatly simplifies our day to day work and improves the productivity of our employees” or “the performance of the software is not what we expected so doesn’t add much values in our business process”
• “The throughput of the team has been increased significantly and we were able to deliver more software features compare to our previous release” - this one is probably from the Technical Manager of the project
• And these go on and on ...

There is nothing wrong with above all the answers and they all are right in the perspective of their own business area. But the problem is how do you communicate success or failure to different group of people let’s say the senior management in the company, do you think that they’ve enough time or interest to listen to all the answers of number of lines written, the defect count, application performance or throughput of the development team? And moreover, how do you compare this result with your next or previous project or different software project in your company and in the industry? So, even though the above kind of answers tell us the success or failure of the software but it’s not easy to communicate to everyone across different interest groups, not comparable across different software projects and certainly does not give a single measure point of success or failure that can be projected on a trend graph unambiguously. 

Wouldn’t it be simpler if we have a straightforward way of evaluating the success or failure or comparing with other releases of projects which would be unambiguous to everyone and most importantly, measurable and quantified in a single number or index? My goal is to debunk the myth that software development is something that you can’t compare one with another, even if the software is developed by the same team of programmers, because every software is a creative and an unique piece of work that make them incomparable. Let’s see if all the dimensions (e.g. quality, productivity, reliability, performance, business value, end user satisfaction etc.) of success of a software project can be combined in to an index to measure them; let's call it Software Confidence Index (SCI). After all, we develop software systems which follows a predictive path of execution and can be measured in number so why not the success of the software that we’ve delivered.

So now let’s change the answers that we had seen above in the conference room to a little different way and more unambiguous way:

• "The project was great! The SCI was 85 in this release", or
• "The release went somewhat good. The SCI was 75, we've room to improve in the field ...", or
• "The SCI was 55. It didn't meet our expected goal. Lets talk about what went wrong ..." etc.

And if people have interest in specific facts like defect density or software performance, that can be dug down in that meeting or in a separate meeting with specific interest groups. The advantage of quantifying the success of the software through an index certainly is that it creates a new vocabulary to communicate across the board but on top of that, it can be used as a new goal setting parameter for the entire team involved in the development and implementation of the software.

So now the question would be how this SCI could be computed. The SCI will cover both sides of the aisle of the software product, the technical side and business side. I would like to give equal weight to technical and business factors but this may change depending on the project's importentce on the techincal or business priorities, as an example, for a software company technical factors may get higher weightage whereas for a non-software organizations may focus more on business values rather than technical excellence. This should be determined by the organization's goals and priorities. There is no hard list of facts that should be considered to compute the index but it has to be consistent across the organization so that the comparison among the softwares SCI would make sense. Let’s check out how it can be computed.

The index considers two kind of the factors - Technical factors and Business Factors. The Technical Factors includes such as - code quality (duplicate code, unused variables etc.), defect density, effective code coverage for unit tests, use of design documents and processes, use of standard coding practices (vulnerabilities in the code for security, memory leak etc.), benchmark of load test result etc.

• Each factors have threshold values to provide a point. E.g. if the defect density is less than 2 in 1,000 Lines of Code (LOC) then the point is 5 and if it’s greater than 20 but less than 25 then the point is 1. The point between 1 and 5 is to cover the in between values.
• Each factors have associated weights. The sum of weights should be 10.

The Business factors is a questionnaire and sent to the customers (the end users) as a survey. The sample questions in the survey could be:

• Does the application have all the features delivered those were committed to business?
• The application saves valuable time and simplifies end users day to day job
• It is very easy and intuitive to use the features of the software
• How satisfied are you with the performance of the software?
• How satisfied are you with the response of the IT Team to any problem experienced in the software?
• Overall how satisfied are you with the software?
• How likely are you to recommend this IT Team to others with similar software need?

Each questions have points ranged from 1 to 5 where 5 is Extremely Satisfied or Agree and 1 is Extremely Dissatisfied or Completely Disagree. Similar to the Technical Factors, each business factor questions have weight and the sum of the all weights is 10.For both the Technical Factors and Business Factors, the point would be 0 if the value falls outside of the accepted range.

Once all the factor values are known, then get the Software Confidence Index using the below equation:

           

where T is Technical Factors and B is Business Factors

Here is the snapshot of factors along with their weights and points that are used to calculate the first SCI:

It's a starting point from where the discussion starts moving towards more of a rational and quantitative direction rather than vague and subjective discussion. This index is most effective and useful when the index is captured for a period of time that enables the comparison of historical data. This SCI can be used in many ways e.g. to create benchmark in an organization, setting goal to the software development team, compare the heterogeneous set of software delivered by an organization etc. This isn't a silver bullet to improve the business confidence on the software but it definitely will set the course to improve the business confidence in the developed software.

Follow-up on Project Confidence Level

I had done some modifications on the perspective of the PCL (to see the detail click on PCL) and included the business factors into it. So far it had only considered the technical factors but I came to a realization that to business or the end users it doesn't make any sense of reaching to the highest level of the PCL and not satisfying the business need. I've outlined the end users perspectives into a form of survey questionnaire that would be send out to the end user (i.e. the customer of the software) at the end of the release and then the answers would be taken into consideration to compute the PCL.

I have yet to finalize the weight of the end users feedback into the PCL computation but it can be 50% of the PCL factors.

Example of end user survey questions could be like -

Q- Does the software simplifies your job?
A - 1. Drastically 2. Somewhat 3. Same as before 4. Made worse 5. Made Extremely Difficult

Note: I've presented this idea to my mentor sometime in 2011 (named it as Software Confidence Index) and he had appreciated the idea and encouraged me to continue pursuing it at my work

Software Development Team Management Best Practices

Though management is a skill which is generally considered as a generic skill through which a manager gets the job done by a group of people (though it is preferable to have a team rather than a group of people but I don't want to go into that debate on what makes a group of people "a team" and saving that discussion for future discussion) but there's a subtle (sometime, not so subtle) differences in managing a team to accomplish a software development project compare to a non-software development project in Information Technology.

Sometime what you learn in school of management doesn't always fit to the specific work environment you are in. Here I am summarizing some of the practices that I found very useful, specially, when I applied them in my area of work. You may treat them as best practices or as thumb rules and these would be very helpful if you managed non-software project but would like to have the transition to the software development field or you just have started managing a software development project. 

1. Any issue or questions from the Development team and business users should be documented in some forms of register. Without this there is high chance that you'll loose track of issues and queries and eventually would be dropped off of the final software deliveries - it is a "must have" when you work with geographically distributed team.
2. Always keep the delivery schedule in front of your eyes (literally like hanging the schedule on the wall of your cubicle or office) with minimum information such as the delivery date and/or possibly with the features to be delivered on the targeted date. Again, use the reference number to trace the requirements

3. Always assign number (or ID) to the user requirements and features. You can use any format ranging from word document to excel file or just a text file but each piece of the requirements should be referenced to a number for traceability purpose

4. Make sure you track the changes (it comes as a feature if you use Microsoft Word) of the the requirements and scope of works, or at least track the changes from the base requirements through highlighting the delta. This will reduce the frustration from fishing for a small changes within a large text document more over it will reduce the chance of missing out the subtle changes in requirement and as well as dropping in the actual implementation

5. You should never set priorities of a feature and, even if you have that in your mind, never mention it to users before user expresses their need/want. This is something you don't intercept - keep it with users.You can negotiate with the date later on once user put the date on the table

6. Keep versions of each and everything the team produces - from simple document (e.g. manual, requirement, design artifacts etc.) to the software itself. For documents, keeping the version histories is a must that should include track changes, the summary of changes, who has updated it and when it was updated for each of the version update. For software and software package make sure to give a version number. This will pay you off down the road, could be after few years but it will save you from a potential disaster

7. Never compromises with the IT Security e.g. access management, put control on code review,  loophole in code etc. If you need to compromise it due to a VERY high priority business case, get back to the process soon after that need is met. Otherwise you will put yourself in unnecessary troubles. It also shows your maturity in management

8. When you are talking about deadline with the development team, ask them for the date and negotiate it from their. Try to give them the date developers asked for. Developers take the responsibility of their given deadline. But make sure when you review the status and if they're lagging, never ask them when they can complete it rather, this time, you tell them the date.

9. Measure the completion of tasks numerically. Never take the development status as near completion, almost done, should be done in couple of hours etc. Just ask the the percent of work left

10. Have One-on-One session with all the member of your team. This is very helpful to get to know lot of internals that you otherwise would never get. You should ask at least 3 questions in that meeting - (a) what do you think about your performance and what you could do more; (b) What do you think we should do to improve the team's performance and (c) Open up yourself of anything related or unrelated to work

11. Measure the performance of the team and individuals quantitatively using consistent metrics. Have 360 degree evaluation if feasible

12. Clarify the meaning of any apparently simple word that might have multiple meanings which would have critical or adverse effect on your project deadline. For instance, when a developer tells you that she's going to "reuse" an existing piece of code, don't be shy to ask to explain the word "reuse". This seemingly obvious word caused one of my project to be delayed by five (5) months where the developers meant "copy the code to the new module and modify it for the purpose of the new module" whereas I assumed the literal meaning of that which was "reuse the existing code from the existing module while refactoring the code to meet both modules' need". Sometime the words - "it doesn't work" could have two meanings. One is "it doesn't give out the result as expected" and the other is "it is actually broken or it bombs"

13. Programmers take pride of what they do so don't offend them with your knowledge or or with your idea specially if it's half baked. Brainstorm with the team to come up with solutions and participate with the team not as a boss. The creativity stops sprouting when the boss is in the conference room with an attitude of bossing around. Yes, you've to take the call to stop at a point where the discussion turns into argument (yes that's very common in software development world) but do it without offending one over other

14. This kind of goes against the current stream - Never call the programmers as a "resource". This offends the very nature of humanity (and programmers are human at work). I also have seen where managers call them as "body" like "I need three more bodies for my project". It's very subtle and often the person who you're referring to may not even notice that but eventually it has profound impact on your behavior on how you would treat a member of your team. Treat them as human and I would strongly stand with the fact that the programmers are actually more emotional compare with people of other industries

15. Recognize the excellent performance of individual member of your team. There's a debate on where you should single out a person to reward or recognize and I agree that there's pros and cons of that as it may go against the spirit of team work. But I prefer to recognize a best performer of your team while keeping the team spirit as one of the criteria to select one. It's like mixing of Socialism with Capitalism where you create the sense of common ownership but also reward individuals for their extra mile run that goes beyond the set expectation

16. Remember Newton's first Law of Motion: An object at rest stays at rest and an object in motion stays in motion. Same is true for your team that you manage or lead. If the team is in a habit of failing the deadlines, they would keep it like that until you force it to turn around. So, make sure that every tiny little milestones, deliveries or commitments are met. If you think a deadline won't work, reset it early enough (but don't wait for it to fail). Once your team is in meeting their commitments mode, you can relieve the pressure but keep the monitoring in place so that the hard earned team's state doesn't get rusty