A couple of years ago I put together some collateral to use when “selling” agile to potential clients. As part of the effort to educate our sales team about agile methods I wanted to provide some strong evidence for our sales team to use.
The only numbers that I had seen at this point were the numbers cited in Agile and Iterative Development: A Manager’s Guide by Craig Larman. To say the least, the numbers were very impressive: Of those polled, 93% saw increases in productivity, 88% increases in quality, 83% improved stakeholder satisfaction, 49% reduced costs, and so on. I dropped these numbers in the collateral and I made every one of the sales members happy.
A few months later I was looking for some deeper numbers and I started to dig deeper for more quantitative data. The first place that I went to look for numbers was the study used that came up with the above numbers (by Shine Technologies in 2003).
When I started digging deeper, I realized that this survey has huge room for error. I’m not an expert on surveys or quantitative methods, but what I read didn’t leave me feeling convinced that it was worth publishing and standing behind the results.
After digging a little deeper, I noticed that these numbers (that I had been advocating) were based on a web-based survey that had only been taken by 131 people. Yes, 131 people and web-based. Ouch.
You can find the survey here.
Now, I don’t want to completely discount the survey and/or the results. This is important information; but I’m just not comfortable presenting these findings as “evidence” that agile works. I’m just not comfortable putting them up on my company website and I’m not comfortable putting them in my presentations. I believe in agile methods, I just don’t believe that this study “proves” that they are a valid alternative.
An interesting thing that I have noticed is that many individuals and many companies have used this survey as justification for agile initiatives. Last week (at Agile 2007) I saw these numbers presented multiple times in slide decks and I’ve seen them on plenty of web sites of vendors that are offering agile services.
Anyway, I just thought that this was an interesting bit to share about how you should always know more about the survey results and numbers that you use to backup any claims. Make sure you can stand behind your numbers and that you find out the information behind the numbers.
Ben,
I agree with your comment. We deliberately put both the poll numbers and the questions asked so that people can review the results in that light. I am certainly concerned if the survey is being mis-represented as the ‘truth’ about the benefits of Agile Methodologies.
The more fundamental issue is “why aren’t there better surveys?” I find it hard to believe that with the rise of Agile Methodologies that a more rigourous evaluation approach has not been undertaken. I have been approached by a couple of postgrad students who are undertaking research, but I haven’t seen the results yet.
As with any survey, it is important to understand the sample size and the survey methodology. Shine Technologies is happy to provide some information to inform debate, but I hope other sources will hit the market soon to allow a range of results to inform the market.
Cheers,
Mark Johnson
Principal
Shine Technologies
Agreed. I really wish that there were more surveys out there too. Hopefully soon we’ll have some of the bigger names pick these items up and give us some large surveys with some good numbers. I’m sure it’s not too far away.
Actually, the real question to ask is what are the objective results of adopting agile? Opinions are fine, but statistics are better still.