Supercharging Maven Surefire with SuperTest
This article is authored by Rishabh Arora and Sanidhya Vijaivargia, and Jude Pereira.
Having a large number of unit tests in a project is a challenge in itself, let alone having all of them pass when
mvn test is invoked. Of course, there’s Maven’s Surefire to the rescue, which is capable of running tests in parallel.
However, what happens when tests go rogue?
Let’s define what a “rogue” test is. It’s a test that interferes with static variables and singletons in such a manner which causes other tests that after it to fail. Here’s a simple example of one such rogue test:
So what happens? If
GoodTest runs after
RogueTest, it will always fail. What about using Surefire’s
rerunFailingTestsCount? Will that solve this?
Unfortunately, no. Surefire will rerun
GoodTest within the same Java classpath, where the static variable will still hold the incorrect value. The obvious solution would be to tell Surefire to not reuse forks:
Doing so will isolate all tests run, and everybody will be happy. Well, except for engineers. Why? Simply because forking a new JVM is expensive, and will increase the time taken to run all tests significantly. Moreover, this problem is exaggerated when a test harness (think of black box tests which require a database setup, and pre populated with certain values) is required by thousands of tests. If it takes a second to initialise the test harness, it would take ~16 minutes to run a thousand tests (and we have thousands).
Can be problem be solved within Surefire? Of course, with Surefire’s
This brings down the time for a thousand tests from 16 minutes to 4 minutes. However, this is still inefficient, since we’re making assumptions that are wildly inaccurate:
- Each test takes < 1ms to execute, which is never the case
- The underlying hardware on which
mvn testis running is capable of spawning our test harness fast enough (1 second or less)
Therefore, a balance needs to be struck, between reusing forks and not reusing forks.
SuperTest is a wrapper around Maven’s Surefire Plugin, with advanced rerun capabilities.
- Run all tests just like
mvn testdoes, however, run with
-faeinstructs Surefire to fail the build after all tests are executed, and not on the first failure)
- Collect the tests that have failed, by parsing the Surefire XML reports found under
target/surefire-reports, and rerun the failed tests with a special Maven profile that sets
Since only the failed tests are rerun in isolation, SuperTest brought down our PR build/test/report iteration time down from 50 minutes to 26 minutes (we have a monorepo, and 7000+ tests are executed for each PR). Before SuperTest, we used to run
mvn test optimistically up to five (!) times, and sometimes they would pass.
Therefore, we came up with SuperTest’s slogan:
Test your code, not your patience.
In a nutshell, here’s how SuperTest runs:
Interested in adapting SuperTest to your project? SuperTest is open source and is available here.
- Header image: Web vector created by storyset – www.freepik.com