Thursday, December 31, 2015

Return of test investment

During a recent discussion the question was raised on how to make investment in test automation pay itself back. How do we know its worth the effort? Implementing automated acceptance tests can require quite big effort.

Theoretically there are two alternatives to automated tests. You can either test manually or you can disregard testing altogether. Having no testing today should be considered a breach of ethics in IT. As a presented at USENIX OSDI 14 by Ding Yuan et. al. most catastrophic failures could be avoided by testing error handling without any understanding the underlying software design.

Manual testing is time consuming, subject to testing procedure errors and require lots of attention. As manual tests take time to execute, the manual test procedure can easily slow down the release deployment and cause delay. With a manually executed procedure the results will also not be as reliable as with the automated tests. The variation can naturally be beneficial, but most likely the test result may be compromised.

When calculating how much benefit automated testing can bring, the main consideration is often how much effort and time can be saved by the automation. Less emphasis is typically placed on reliability aspect. Freeing up testers time to do more thorough explorative testing must also be considered. Secondary effects are also of importance. Having automated tests cause code and features to become more easy to test. Having automated tests also enable faster releases. Taking secondary effects into account when calculating benefit of test automation can change the equation.

Additionally there is also the consideration on which side do you want your estimate to side, on the side of promoting improved testing or on the side of limiting quality...