Software Zen

This blog has moved

bhupinder — Wed, 03 Aug 2011 22:53:14 +0000

to Bhupinder Singh’s Blog on Software Engineering

Whitepaper on Software Testing Metrics

bhupinder — Mon, 14 Feb 2011 23:51:58 +0000

You can now download this free whitepaper on Software Testing Metrics that I wrote some time back. Comments are welcome.

Software Metrics- V

bhupinder — Thu, 22 Jul 2010 20:55:53 +0000

A good quality management approach provides warning signs early on so that there is room for timely intervention. For this, it is essential to gather metrics and predict values of critical indicators at different stages in the project so that controlling these parameters during the course of the project will ensure that the final product has the desired quality. If these predictions can be made, then the actual data during the execution of the process can be used to judge whether the process has been effectively applied. A number of metrics therefore need to be measured during the course of software development so that predictable results can be achieved on all the three major considerations- quality, cost and schedule.

Here are the definitions of some of the metrics that pertain to software testing:

	Metric	Definition	Potential Use
1	Defect Density (DD)	= Total Weighted Defects / Size (in Function points or kLOC)	Product Quality
2	Defect Containment Efficiency (%DCE)	= Total number of weighted defects observed before release * 100 / (Total weighted defects reported before release + Total weighted defects reported after release)	Test Process Quality
3	Average Defect Age (ADA)	= Sum of Defect ages of all defects / Total number of defects	Test Process Quality
4	Rework Ratio (RR)	= Effort Spent on Rework / Total Effort	Measure of redundant effort.
5	Defect Detection by Phase	= Number of Weighted Defects detected at the end of each phase	Quality of review and test process during each phase. It is a more detailed version of the Defect Containment Efficiency
6	Defect Rejection %	= Total Number of defects withdrawn/Total number of defects found	Quality of testing team
7	Cost per defect	= Total cost of testing/ Number of weighted defects	Testing Efficiency
8	Defect Removal Efficiency	= Number of defects detected at each phase of testing/ total number of defects	Testing Efficiency
9	Test coverage	= Number of requirements tested/ Total number of baselined requirements in scope	Testing Efficiency
10	Test case execution productivity	= Number of Test cycles executed / Actual Effort for execution	Testing Efficiency
11	Test Effectiveness	= Number of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100	Testing Efficiency

(This is the last post in the series on Software Testing Metrics).
Image Source

Estimating Test Effort

bhupinder — Tue, 13 Jul 2010 20:20:55 +0000

The best way for a ball park estimate for the testing effort is relatively simple- just take one third of the effort required for the complete development of an application. However, sometimes this total effort may not be available, for example, if the development and test vendors are different or the development and testing practices are not integrated. In that situation, the testing team needs do an independent sizing of the application, using a technique like Function Points and then computing the number of test cases.

One of the best known techniques are the ones provided by Caper Jones and David Longstreet. According to Caper Jones,

Total number of test cases = (Count of Function Points) raised to the power of 1.2.

David Longstreet provides a similar formula for computing the number of UAT test cases:

Total number of UAT test cases, which is = 1.2 x (Count of Function Points).

As with every such empirical model, one has to use them as a guide and evaluate them with real data. In my own experiences with web based (Microsoft dot net platform), the numbers are slightly different, and approximate better as follows:

Total Number of test cases = (Function Point Count) raised to the power 1.05
Total number of UAT test cases = 1.35 x (Function Point Count)

Once the count of the test cases has been found, apply the productivity factors for test case authoring and test case execution to arrive at the total testing effort.

The following is the data for a 3000 function point project on which the above formula have been tweaked.

Unit Test Cases                 : 13,000
Integration Test Cases       : 5,000
System Test Cases            : 13,000
UAT:                                : 4,100
Total                                : 35,100

References: Estimating Test Cases and Defects by David Longstreet
Software Estimation Rules of Thumb by Caper Jones (pdf)

Managing by Exception

bhupinder — Thu, 08 Jul 2010 22:06:53 +0000

As a techie who grew through the ranks, starting as a programmer to project lead to project manager and then a program manager, I have had no management training as such. It came naturally, the learning happened either by burning one’s own hands or by the guidance of a knowledgeable manager.

Moving from project lead to project manager was relatively seamless. There were a few changes though. The most significant was the focus I had to bring in laying down project specific processes and ensuring compliance. I also became less hands on as far as programming and architecting was concerned. My reliance on metrics increased. Unknowingly, I also carried forward the habit of micro managing the project, keeping an eye on each and every team member and their activities. This included spending time with individual team members at their desks while they worked, sometimes helping them with debugging their programs. An interesting aside was that I realized while one may grow out of the habit of programming, one almost never loses the ability to debug programs.

This observation came to my help much later. In my role as a program manager, I had to manage multiple projects- often in multiple technologies and across different types of projects.

In one such role I was managing support for about 100 business applications, an identity management application, SAP implementations in ABAP and BW as well as smaller independent projects in JDE Edwards, C++ and so on. It became almost impossible for me to micro manage all these projects. Intuitively, I started devoting more time to project start up, ensuring that the fundamental processes were laid out and the team was committed to follow those, and then getting into project details only when something did not go right.

This was akin to debugging- that is it is sometimes easier to manage what is going wrong rather than keeping a tab on each and every a line of code. In more sophisticanted parlance, it is known as management by exception. Wikipedia defines this as:

a “policy by which management devotes its time to investigating only those situations in which actual results differ significantly from planned results. The idea is that management should spend its valuable time concentrating on the more important items (such as shaping the company’s future strategic course). Attention is given only to material deviations requiring investigation.”

Like any other concept, an over reliance on management by exception can be risky. Handling too many exceptions is not a good sign of the health of the projects either. By concentrating only on exception management, the manager may not be able to invest much time and attention to building processes and grooming talent. The trick is to kill two birds with one stone- that is, while attending to exceptions, the manager has to ensure that he or she identifies the root cause and then endeavors to put checks and balances in the processes to avoid future recurrences.

Software Development is…

bhupinder — Thu, 08 Jul 2010 17:30:21 +0000

Results of a fun poll that I conducted over at Linkedin.

Reverse Engineering for Software Systems

bhupinder — Wed, 30 Jun 2010 19:54:44 +0000

In its most rudimentary form, reverse engineering can be defined as “going backwards through the development cycle”. It has its origins in espionage when rival countries would break apart a product from the opposite country and figure out its mechanism in order to build a similar product. Wikipedia defines reverse engineering as:

Reverse engineering (RE) is the process of discovering the technological principles of a device, object or system through analysis of its structure, function and operation. It often involves taking something (e.g., a mechanical device, electronic component, or software program) apart and analyzing its workings in detail to be used in maintenance, or to try to make a new device or program that does the same thing without utilizing any physical part of the original.

When I started my career at an Indian public sector telecommunication company, one of the projects that I worked on was a versatile multiplexer. At that time the Indian market was still closed and technological know how from the more developed countries was limited. We did not even have a sample of the multiplexer, just a technical brochure with a feature list of the product. I am not sure if that can be called reverse engineering, but our team did manage to build a working product. By the time it hit the market, I had left the company so I do not know how well it fared. However, I had other brushes with reverse engineering, both successful and not-so-successful.

One of them was when I was writing the software for a PC card based POS system. The challenge was that the system had to display a certain message when the system was booting up. Since the PC display had been replaced by a custom hardware display, there was an inordinately long time before the user would see a message. The POS software was written in C on top of MS- DOS. Since I was familiar with assembly code for the Intel 8086, I retrieved the BIOS that came with the PC motherboard and inserted hexadecimal code to initialize and display a message up- front after the BIOS had completed the initializations. This was the easy part. The challenge was that the code worked when I burnt it on a blank ROM, but refused to work when the exact code was added into the BIOS. I had to tear apart the entire assembly code using a dis- assembler and reference many a book on BIOS and DOS internals before hitting upon the source of the problem. The IBM reference manual for the PC XT provided the mnemonics, and I chanced upon the chunk of code that did a sum- check to ensure that the BIOS was correct. Re- computing the sum- check number and replacing it in the right place did the job.

This was an example of a relatively successful tweaking using reverse engineering. My other endeavors have been less successful. A few years later when I was working to re- engineering a software used for manufacture of sophisticated X- Ray rooms, I was impeded not only by my lack of knowledge of COBOL and the network databases used by the legacy system, but also by the complete lack of documentation by the programmers. After a month of unsuccessful attempts at understanding the source code, I and my lead had to switch over to start top down and understand the business scenarios.

Even on a more recent project when re- engineering an enterprise system for a medical chain, I and my team figured that doing a top down analysis was much more useful to build the new application. Later we were informed by the customer that a number of unsuccessful attempts had been made earlier to re- engineer the system, all of them had started by trying reverse engineer the legacy application.

The major challenges I see with reverse engineering are:

The source code, where it is available, is often ‘spaghetti’- and very difficult to decrypt
The cost of reverse engineering is extremely high,and its returns vary from being at best, low to, at worst, unpredictable
Often because of obsolete technology, the technical skills are hard to find
Even if the source code is readable, it may be dependent on hard coded data or implicit assumptions that may not be visible to the person trying to reverse engineer the code
In case of applications, business requirements change so much over a period of time that a lot of the functionality in the code may not even be relevant any longer

A slightly more useful artifact to start the reverse engineering process is the database structure or the ER diagram where it is available. Even there, my experience has been that beyond the list of entities that may help us to understand the system, the details are misleading. The reasons are similar to those listed above- archaic and anachronistic remnants from the past.

To conclude, while reverse engineering sounds very good, and can be successfully used, it is always a procedure of last resort.

Software Metrics- IV

bhupinder — Tue, 22 Jun 2010 20:31:20 +0000

I have often been asked to take charge of projects that are in deep trouble. I am no longer surprised that almost universally, the stage when a project is generally acknowledged to be in trouble is during the late stages of testing and rolling out the first release. Never does it happen during the requirements and design stages at all- for reasons that are obvious- there is nothing tangible that the end users actually see during these phases except perhaps a stream of documents.

While I will write a set of posts on my experience in turning around projects, I would like to illustrate here how metrics can be used to analyze the depth of the problem and also provide a very quick plan for stabilizing the project very quickly.

A simple defect analysis based on the cause of the defects can shed light on what are the key areas that need to be addressed.

In the following diagram from an actual web- based application development project, for example, it is very clear that the process for regression testing needs to be improved as defects because of its inefficiency are 22%. It is also clear that the substantial numbers of defects (23%) are on account of requirements (missed + ambiguous) and another 17% of the defects pertain to insufficient test cases. Addressing these three areas would have an immediate impact on the quality of testing. Given cost and schedule constraints, it may not be required to address all the causes and instead concentrate only on these three factors that address 72% of the defects.

IT- The Future is Here

bhupinder — Tue, 15 Jun 2010 19:26:23 +0000

This article was written 15 years ago when internet services officially started in India. I had expressed a number of fears in this some of which have been happily proved incorrect. However, I find it interesting that there are almost no fundamentally new technological breakthroughs that have come around since the article was written. Some of the concerns raised in the article still hold, particularly its conclusion.

Trivia: The original article was typed on a PC- XT machine using Word Star 7.

I had used email for just over a month then using a corporate account and the browser I was then using were Mosaic and Gopher !

Anyone remember using these?

Read the full article here.

How Unique is Software Development?

bhupinder — Thu, 10 Jun 2010 18:06:56 +0000

In a recent short discussion on “3 common mistakes a project manager makes’ over at Linkedin, I mentioned that the three most common mistakes a manager makes are:

1. Rely on his or her instinct and not measurable parameters
2. Manage people and not activities/ deliverables
3. Not set expectations with all stakeholders

The set of what one perceives to be the most common mistakes is likely to vary from person to person. Nevertheless, the second point came in for a sharp response at the forum:

I have a contention on point # 2 in your response. In any people centric organization, managing people is an integral part of Project Management. I am not sure why you consider it as a mistake. Can you explain?

This is not the first time that I have encountered this criticism- that software is “people centric” and hence the focus should be on managing people and not the deliverable. The root cause of this observation goes much deeper- that somehow, software development is something very unique, that it is not similar to other engineering disciplines, that it should not emulate manufacturing.

I aver that while, like any other discipline, software has its unique aspects, it is still part of the long tradition in engineering. Indeed, over the years, consistent attempts have been made to align it with engineering- establishment of standards, the ISO and CMM models all intend to achieve the same objective.

By stating that software is somehow “people centric”, the real point is lost- where software differs fundamentally from manufacturing is that its work products are intangible- they do not have a tangible physical inventory of items that can be measured discreetly, in that software is closer to process engineering where the basic ingredient undergoes a number of changes throughout its lifecycle. In the absence of such tangible artifacts, it seems that somehow it is people centric because that is the nearest tangible association one can make.

Over the years, there has been an increasing recognition of the engineering or manufacturing aspect within the software industry- where there is the increasing preponderance of “components” (like from Microsoft), frameworks, standards, metrics and processes.

I am sure the the debate will, nevertheless continue, and provide better insights into this still- young discipline.