A couple of years ago I put together some collateral to use when “selling” agile to potential clients. As part of the effort to educate our sales team about agile methods I wanted to provide some strong evidence for our sales team to use.
The only numbers that I had seen at this point were the numbers cited in Agile and Iterative Development: A Manager’s Guide by Craig Larman. To say the least, the numbers were very impressive: Of those polled, 93% saw increases in productivity, 88% increases in quality, 83% improved stakeholder satisfaction, 49% reduced costs, and so on. I dropped these numbers in the collateral and I made every one of the sales members happy.
A few months later I was looking for some deeper numbers and I started to dig deeper for more quantitative data. The first place that I went to look for numbers was the study used that came up with the above numbers (by Shine Technologies in 2003).
When I started digging deeper, I realized that this survey has huge room for error. I’m not an expert on surveys or quantitative methods, but what I read didn’t leave me feeling convinced that it was worth publishing and standing behind the results.
After digging a little deeper, I noticed that these numbers (that I had been advocating) were based on a web-based survey that had only been taken by 131 people. Yes, 131 people and web-based. Ouch.
You can find the survey here.
Now, I don’t want to completely discount the survey and/or the results. This is important information; but I’m just not comfortable presenting these findings as “evidence” that agile works. I’m just not comfortable putting them up on my company website and I’m not comfortable putting them in my presentations. I believe in agile methods, I just don’t believe that this study “proves” that they are a valid alternative.
An interesting thing that I have noticed is that many individuals and many companies have used this survey as justification for agile initiatives. Last week (at Agile 2007) I saw these numbers presented multiple times in slide decks and I’ve seen them on plenty of web sites of vendors that are offering agile services.
Anyway, I just thought that this was an interesting bit to share about how you should always know more about the survey results and numbers that you use to backup any claims. Make sure you can stand behind your numbers and that you find out the information behind the numbers.