"To realize that you do not understand is a virtue; not to realize that you do not understand is a defect."

— Lao Tzu (sixth century B.C.E)

Software defects take on many shapes and forms. For example, they may manifest themselves as a poorly interpreted or missed requirement or lay hidden in a seemingly innocuous piece of program code. As we develop our system, we subject it to the rigors of all manner of testing phases and approaches to try to eradicate defects, always though without total success. We can never prove that our software is totally defect-free — only that our testing endeavors have failed to find more.

But what do we do with our defects once we have found them, and how do we prioritize them, as clearly all defects are not created equal? Tracking, prioritizing, and managing defects are critical tasks for effective software development. It therefore makes sense that to effectively handle the process, we need to introduce some degree of formality to it — how much depends entirely on you. As always, I'm a great believer in simplicity of application. Make a process too difficult to administer, and it breaks down; in other words, start with more of less.

That said, in this article, I outline the mechanics of the defect tracking process and discuss the requirements for selecting a defect tracking tool.

The Defect Tracking Process

Integration, system, and user acceptance testing in a System i shop typically is not subjected to the same rigors as that of larger mainframe shops. However, reviewing, documenting, analyzing, prioritizing, and resolving defects in a structured way will greatly help you keep the testing focused, on track, and the correct priorities maintained. In addition, the statistics produced from the process aids the project manager and his team in managing, estimating, and analyzing defects.

The two areas I cover relating to this process are the activities performed by the Review Control Board and the selection and use of a defect tracking tool. Don't be put off by the term Review Control Board, as this is purely a group of people, ideally three, that reviews new and existing defects found as part of a testing cycle. The people who form part of this group may vary, depending on whether the testing is conducted at the integration, system, or user acceptance testing level.

Certainly, the person conducting or responsible for the testing effort should attend the review sessions. Ideally, the team should consist of a system architect, a stakeholder (as in the case of user acceptance testing), and a business analyst (as in the case of system testing). If you work in a small shop, the team might include people who serve multiple roles. My point is, do whatever works best in your environment. Now let's look at an outline of the process, as understanding this will help everything fit into context.

Testing and Recording

During the execution of a test case, the tester may discover several potential defects. The tester will record information relating to the nature of these defects so that someone can investigate them in detail later. At this point, testers should always resist attempting to determine the root cause. Their primary focus should be purely to test the software and record the results from each test case executed. (For a definition of test case, see "Glossary of Terms," below.)

Apart from recording the steps involved in executing the test case, if testers discover anomalies in the test case, they should report the disposition as having an "unconfirmed" status. Figure 1 shows an overview of the process schema, and Figure 2 provides status coding recommendations for recording defects. I strongly recommend that testers report the details relating to the potential defect electronically and in some form of database to maximize inquiry and reporting capabilities (I discuss this later).

Review Sessions

It's important that the Review Control Board conduct a periodic review session. The frequency of the review session will largely depend on such factors as the development life-cycle model, the frequency of testing cycles, and resource availability. The team's primary objectives, however, are to review new unconfirmed defects, prioritize confirmed ones, and defer or reject others.

The process is logically divided between

• reviewing newly reported unconfirmed defects and assigning resources to investigate defects further
• prioritizing newly confirmed defects or reprioritizing previously confirmed ones
• reporting the status of confirmed defects
• determining whether to reject or defer previously investigated defects

In determining the course of action, the team should consider the priority of the issue based on the previously calculated priority and any inter-test case dependencies that may exist, which could impede the progress of the testing.

Priority is based on a combination of likelihood and impact. Typically, you apply a weighting factor to both of these, and a simple multiplication of the two provides a priority. The team can make changes to this priority and/or risk assessment at this time. It is the project manager's responsibility to communicate any issues to the project sponsor that may be cause for concern. Such issues may be but are not limited to

• functional and nonfunctional requirements
• system requirements specification
• technical or architectural design

The project manager should perform an impact analysis based on the results of each Review Control Board session and on subsequent input from both the technical lead and the person designated as the testing lead.

Reporting Benefits

The defect tracking and recording system's testing status is a useful tool for managing the testing process. We can produce various reports and, at a minimum, we should be looking at

• defects confirmed and severity (all defects are not equal and therefore should be viewed and categorized in the context of priority)
• defects fixed by priority
• defects analyzed by root cause (provides a view of where the majority of defects occur, which may be in requirements, design, or coding)
• defects (mean time to fix by priority)

A useful metric relating to test effectiveness is Defect Removal Efficiency. This may be measured as follows:

DRE = Number of defects found in testing/ Number of defects found in testing + number not found within the testing cycle

The number of defects not found is simply the number users discovered — either in the subsequent phase or when the defects are "live." This number tells how effective your testing is, but remember that users won't find all the defects. The closer this number computes to a value of 1.0 the better the DRE. For example, if we discovered 78 defects in our testing phase and trap another 25 in a later testing cycle, we'd have 78/ (78+25) = DRE of 0.757, indicating a poor DRE.

Defect Tracking Software

Several vendors provide software that will help you record and manage defects. For a starting point, Microsoft Project Web Access offers some excellent capabilities that integrate well with the project life cycle. A number of freeware packages such as Bugzilla and Jitterbug are also available, thereby circumventing the need to write your own software.

Should you decide to develop your own defect tracking software, see the recommended attributes in Figure 3. However, I suggest that you start small by using freeware so that you become accustomed to the process. Taking this approach will help you determine which options will work most effectively in your environment.

Take Control of Defect Tracking

Although using a defect tracking package may, on first pass, seem to be yet more project overhead, it provides a way to control the correct prioritization and management of testing defects. It also gives you useful insight into ways to improve the quality of the system development life cycle.

One question that often arises is how applicable is defect tracking in an agile development model. Agile models are both incremental and iterative. You might replace formal test-case writing with less formal methods and spend two to three weeks coding and testing cycles, but you must still pursue the tracking, prioritization, and management of defects. Try using the techniques I have described in this article next time you undertake a new project, and I'm sure you will be pleasantly surprised at the results.

Russ Bartlett has worked in information systems for 39 years. He holds an IEEE CSDP and has experience in all areas of system development, having been responsible for implementing systems in many different organizations, including ERP solutions in major Fortune 500 companies.

Glossary of Terms

Test Case: Defines a particular condition to be tested. Defined by input, with an expected result.

Test Log: A chronological record of related details about the execution of the test cases.

Testing Priority: A means of ranking the importance of a test case based on the likelihood of the event occurring and the affect on the business should the event occur.

— R.B.