With the extensive amount of testing performed nightly on large software projects (like Klocwork), test and verification teams often experience lengthy wait times for the availability of test results of the latest build. As we strive to identify and resolve issues as fast as possible, alternative methods of test execution have to be found.
At Rogue Wave Software, to verify our Klocwork static code analysis product before release, what we have implemented now is a series of Jenkins jobs, triggered automatically by the availability of a new Continuous Integration (CI) build, which launches and executes our Linux-based tests across a farm of virtual machines. It re-uses the bulk of our existing test framework, minimizing the amount of refactoring required. Over 6000 tests that used to take 24 hours now executes completely in under four hours. One of the most intriguing aspects of this solution is its scalability: execution time can be further reduced by adding more VMs to the farm. Also, test results are immediately imported into a web-based test repository as soon as their test suite completes, which means the first results can start being reviewed just minutes after execution.