"Joel Foner" asked:
>
>I wonder if the failure rate is any higher or lower
>than software development projects in general?
>How would you define a failed test automation project?
>One that only achieved partial coverage? One that got
>outright cancelled?
>
I believe there's a general accepted notion that shrinkwrapped commercial test automation has a higher failure rate than general software development, sure.
Here's why: The generally accepted definition of "failed" test automation is something like: We paid five figures or more for the software. It came in a box. We did some ... stuff. When you come back in a month, the box is on the shelf, but no one is using it.
That is the general definition Elfriede Dustin uses when she claims some large percentage of test automation projects fail -- I believe she discusses the concept in this talk at GTAC 2008; there is video:
I don't have a study to point to. What I can tell you is the large number of clients I hear from who hired a contractor to develop some test automation, paid him big bucks, he goes away after ~ 3-6 months, then the team hits crunch time, the GUI changes, the testware isn't updated, and the tests become a big mess on the floor.
The call me to clean it up.
As a contractor.
For a 3-6 month contract.
Now I don't mean to be overly judgmental or critical here, but I hope you can understand that when I see a project like that, I see risk for an unhappy client, and that means reputation risk. So far, when I've been discussing these types of assignments, I've always a different opportunity to pursue that I was more excited about.
There's more than one way to do it, but increasingly I'm coming to believe that the testware is a software product to be developed by the programmers as a technical deliverable, as close to the code as possible. Once the programmers develop this high-level, domain-specific language, then testers can create test cases, do exploratory testing, and, in many cases, extend the testware. There are many ways to do it, but i've probably seen the most success with this one.
regards,
--
Matthew Heusser,
Consulting Software Tester/Writer
