Supercharging Maven Surefire with SuperTest

This article is authored by Rishabh Arora and Sanidhya Vijaivargia, and Jude Pereira.

Having a large number of unit tests in a project is a challenge in itself, let alone having all of them pass when mvn test is invoked. Of course, there’s Maven’s Surefire to the rescue, which is capable of running tests in parallel.

However, what happens when tests go rogue?

Rogue Tests

Let’s define what a “rogue” test is. It’s a test that interferes with static variables and singletons in such a manner which causes other tests that after it to fail. Here’s a simple example of one such rogue test:

	class State {
	public static boolean someState;
	}
	class GoodTest {
	@Test
	public void checkState() {
	assertFalse(State.someState);
	}
	}
	class RogueTest {
	@Test
	public void playWithState() {
	State.someState = true;
	// Test something that relies on State#someState
	// Whoops! We forgot to reset State#someState to its original value!
	}
	}

view raw Test.java hosted with ❤ by GitHub

So what happens? If GoodTest runs after RogueTest, it will always fail. What about using Surefire’s rerunFailingTestsCount? Will that solve this?

Unfortunately, no. Surefire will rerun GoodTest within the same Java classpath, where the static variable will still hold the incorrect value. The obvious solution would be to tell Surefire to not reuse forks:

<reuseForks>false</reuseForks>

Doing so will isolate all tests run, and everybody will be happy. Well, except for engineers. Why? Simply because forking a new JVM is expensive, and will increase the time taken to run all tests significantly. Moreover, this problem is exaggerated when a test harness (think of black box tests which require a database setup, and pre populated with certain values) is required by thousands of tests. If it takes a second to initialise the test harness, it would take ~16 minutes to run a thousand tests (and we have thousands).

Can be problem be solved within Surefire? Of course, with Surefire’s forkCount option:

<forkCount>4</forkCount>

This brings down the time for a thousand tests from 16 minutes to 4 minutes. However, this is still inefficient, since we’re making assumptions that are wildly inaccurate:

1. Each test takes < 1ms to execute, which is never the case
2. The underlying hardware on which mvn test is running is capable of spawning our test harness fast enough (1 second or less)

Therefore, a balance needs to be struck, between reusing forks and not reusing forks.

Hello SuperTest!

SuperTest is a wrapper around Maven’s Surefire Plugin, with advanced rerun capabilities.

Goals

• Run all tests just like mvn test does, however, run with -fae specified (-fae instructs Surefire to fail the build after all tests are executed, and not on the first failure)
• Collect the tests that have failed, by parsing the Surefire XML reports found under target/surefire-reports, and rerun the failed tests with a special Maven profile that sets reuseForks to false

Since only the failed tests are rerun in isolation, SuperTest brought down our PR build/test/report iteration time down from 50 minutes to 26 minutes (we have a monorepo, and 7000+ tests are executed for each PR). Before SuperTest, we used to run mvn test optimistically up to five (!) times, and sometimes they would pass.

Therefore, we came up with SuperTest’s slogan:

Test your code, not your patience.

In a nutshell, here’s how SuperTest runs:

Interested in adapting SuperTest to your project? SuperTest is open source and is available here.

Credits

• Header image: Web vector created by storyset – www.freepik.com