Test Automation Success Criteria

I’ve been thinking recently about how to validate the success of an automation project, and actually prove a return on investment to anyone interested. This is important for a number of reasons.

Firstly, it shows the QA team, (who are most likely set in their ways with the traditional manual testing approach), that at least some of their work can be completed in a more efficient manner, and testing coverage can be increased without any extra effort on their part. Secondly, it shows management that automation is worth investing in. It also boosts the morale of the people who actually worked on the development of the automation, and gives them confidence to continue and make even more improvements.

Brett Pettichord defines what I think are 4 excellent items to validate any test automation project against:

  1. The automation runs
  2. The automation does real testing
  3. The automation finds defects
  4. The automation saves time

I believe the fourth item above is probably the most important, (assuming 1 and 2 are satisifed). The whole point of test automation is process improvement, and having the ability to absorb more work without requiring additional resources. Item 3 is important also, any defects should obviously be flagged, if not logged automatically also.

If the four points mentioned are satisfied once your automation project is complete, I believe it can be qualified as a success.