Tom Gilb and Kai Gilb's blog
ON THE 33% TEAM EFFECTIVENESS OF AN AGILE INSPECTION
Published
by Tom Gilb
on Wed 18 of Feb., 2009
A Good friend of mine RG, Germany asked how we calculated remaining defects after an Agile Inspection. There was some debate with his colleagues about whether they were 33% effective (single pass, team total) in finding Major Defects, or more like 50% or more.
Here is my email answer.
The form and process for calculating remaining defects is is page 244-245 in the SQC Chapter of the Competitive Engineering book (I'll send that chapter to anyone who asks, tom at Gilb dot com)
and
and in this paper Case 2
Agile SQC Sl…
http://www.gilb.com/tiki-download_file.php?fileId=239
Here are a set of papers on this site on Inspections
AgileCutter5p http://www.gilb.com/tiki-download_file.php?fileId=64
INCOSE SQC… http://www.gilb.com/tiki-download_file.php?fileId=57
Rule Magazine http://www.gilb.com/tiki-download_file.php?fileId=192
Eng.Rev.Pro… http://www.gilb.com/tiki-download_file.php?fileId=143
Course Cert http://www.gilb.com/Inspection+Leader+Certification
The effectiveness will of course vary depending on many factors (discussed in my SI book) but, general data is given by Capers Jones papers on my website. Be careful, when Jones cites Inspection Efficiency, he is misusing english and means effectiveness, more importantly he is referring to the effectiveness of a set of inspections - THE CUMULATIVE EFFECT OF SEVERAL CLASSES OF DOCUMENTATION IE REQUIREMENTS, DESIGN, CODE. He is also referring to Inspections designed as in my book Sw Inspection, designed to maximize effectiveness (example optimum rate of 1 page per hour per engineer checker, specialized roles, 4-5 inspectors etc.).
In the Agile Inspection/SQC we are NOT trying to maximize inspection effectiveness. It is NOT necessary. We are not trying to remove defects - we are trying to measure. So the important point is
1. have we clearly exceeded the Exit level? ( if exit is max 1.0 major /page and you estimate 100 majors, you have clearly failed exit)
2. is the measure reasonably accurate ?(so that we do not accept bad defect ridden work).
How did I arrive at 33% ? Initially I guessed at this approximately ( I knew it was between 5% and 55%), and then we did 2 things that confirmed that 33% was generally the right order of magnitude:
1. we cleaned up the ones we found, predicted that we would find 1/3 of the remaining 2/3s, AND WE DID FIND THE PREDICTED NUMBER EVERY TIME!
2. In a few cases including Case 2 at GE Aero Engines, Cincinnatti Ohio and in Ericsson Germany (900 person real project) we used the 33% to calculate the project delay, as a function of bugs NOT found. Both these calculations were done blind for us of the facts, and were confirmed delays by our clients! This delay prediction could not have worked ( except for compensating errors in assumptions) if 33% was not a good estimate!
So your reality on a given day with given rules, given people, giving timing, giving motivation may vary say 10% to 50%, but 33% (TEAMS TOTAL EFFECTIVENESS, BEST INDIVIDUAL IS HALF THAT) is as good an engineering approximation as you will get, and it works in ENABLING YOU TO REJECT EXIT OF BAD WORK.
here are some of Capers data. ( see Download from others this site, listed below).
So anybody who is still not convinced can easily test by doing a series of inspection. In a classroom we did 4 consecutive days of bug finding code, I predicted the remaining bugs and how many would be found by the class next day after the coder removed the bugs we found. The coder was always sure there were near zero bugs remaining, but he was always wrong, and the predictions were always right! It is natural that people are skeptical, forgive them their ignorance! The only way people are convinced is by this prediction demonstration.
Defect Removal http://www.gilb.com/tiki-download_file.php?fileId=234
Method Effec… http://www.gilb.com/tiki-download_file.php?fileId=250
St Art 07 http://www.gilb.com/tiki-download_file.php?fileId=147m/tiki-download_file.php?fileId=252
Here is my email answer.
The form and process for calculating remaining defects is is page 244-245 in the SQC Chapter of the Competitive Engineering book (I'll send that chapter to anyone who asks, tom at Gilb dot com)
and
and in this paper Case 2
Agile SQC Sl…
http://www.gilb.com/tiki-download_file.php?fileId=239
Here are a set of papers on this site on Inspections
AgileCutter5p http://www.gilb.com/tiki-download_file.php?fileId=64
INCOSE SQC… http://www.gilb.com/tiki-download_file.php?fileId=57
Rule Magazine http://www.gilb.com/tiki-download_file.php?fileId=192
Eng.Rev.Pro… http://www.gilb.com/tiki-download_file.php?fileId=143
Course Cert http://www.gilb.com/Inspection+Leader+Certification
The effectiveness will of course vary depending on many factors (discussed in my SI book) but, general data is given by Capers Jones papers on my website. Be careful, when Jones cites Inspection Efficiency, he is misusing english and means effectiveness, more importantly he is referring to the effectiveness of a set of inspections - THE CUMULATIVE EFFECT OF SEVERAL CLASSES OF DOCUMENTATION IE REQUIREMENTS, DESIGN, CODE. He is also referring to Inspections designed as in my book Sw Inspection, designed to maximize effectiveness (example optimum rate of 1 page per hour per engineer checker, specialized roles, 4-5 inspectors etc.).
In the Agile Inspection/SQC we are NOT trying to maximize inspection effectiveness. It is NOT necessary. We are not trying to remove defects - we are trying to measure. So the important point is
1. have we clearly exceeded the Exit level? ( if exit is max 1.0 major /page and you estimate 100 majors, you have clearly failed exit)
2. is the measure reasonably accurate ?(so that we do not accept bad defect ridden work).
How did I arrive at 33% ? Initially I guessed at this approximately ( I knew it was between 5% and 55%), and then we did 2 things that confirmed that 33% was generally the right order of magnitude:
1. we cleaned up the ones we found, predicted that we would find 1/3 of the remaining 2/3s, AND WE DID FIND THE PREDICTED NUMBER EVERY TIME!
2. In a few cases including Case 2 at GE Aero Engines, Cincinnatti Ohio and in Ericsson Germany (900 person real project) we used the 33% to calculate the project delay, as a function of bugs NOT found. Both these calculations were done blind for us of the facts, and were confirmed delays by our clients! This delay prediction could not have worked ( except for compensating errors in assumptions) if 33% was not a good estimate!
So your reality on a given day with given rules, given people, giving timing, giving motivation may vary say 10% to 50%, but 33% (TEAMS TOTAL EFFECTIVENESS, BEST INDIVIDUAL IS HALF THAT) is as good an engineering approximation as you will get, and it works in ENABLING YOU TO REJECT EXIT OF BAD WORK.
here are some of Capers data. ( see Download from others this site, listed below).
So anybody who is still not convinced can easily test by doing a series of inspection. In a classroom we did 4 consecutive days of bug finding code, I predicted the remaining bugs and how many would be found by the class next day after the coder removed the bugs we found. The coder was always sure there were near zero bugs remaining, but he was always wrong, and the predictions were always right! It is natural that people are skeptical, forgive them their ignorance! The only way people are convinced is by this prediction demonstration.
Defect Removal http://www.gilb.com/tiki-download_file.php?fileId=234
Method Effec… http://www.gilb.com/tiki-download_file.php?fileId=250
St Art 07 http://www.gilb.com/tiki-download_file.php?fileId=147m/tiki-download_file.php?fileId=252
