The JVM Test Suite plugin (plugin id: jvm-test-suite) provides a DSL and API to model multiple groups of automated tests into test suites in JVM-based projects. Test suites are intended to be grouped by their purpose and can have separate dependencies and use different testing frameworks.

For instance, this plugin can be used to define a group of Integration Tests, which might run much longer than unit tests and have different environmental requirements.

The JVM Test Suite plugin is an incubating API and is subject to change in a future release.

Usage

This plugin is applied automatically by the java plugin but can be additionally applied explicitly if desired. The plugin cannot be used without a JVM language plugin applied as it relies on several conventions of the java plugin.

build.gradle.kts
plugins {
    java
    `jvm-test-suite`
}
build.gradle
plugins {
    id 'java'
    id 'jvm-test-suite'
}

The plugin adds the following objects to the project:

  • A testing extension (type: TestingExtension) to the project used to configure test suites.

When used with the Java Plugin:

  • A test suite named test (type: JvmTestSuite).

  • A test SourceSet.

  • Several configurations derived from the test SourceSet name: testImplementation, testCompileOnly, testRuntimeOnly

  • A single test suite target backed by a task named test.

The test task, SourceSet and derived configurations are identical in name and function to those used in prior Gradle releases.

Dependency configurations

Every test suite, whether it’s the built-in test suite or one you declare yourself (e.g. integrationTest), gets the same set of configurations, named after the suite:

Configuration name Role Description

suiteNameImplementation

Declarable

Dependencies required to compile and run this suite. For the built-in test suite only, this configuration additionally extends the main source set’s implementation, so it automatically sees the project’s own production dependencies; every other suite must add a project(…​) dependency itself to reference production code (see Configure dependencies of a test suite to reference project outputs).

suiteNameCompileOnly

Declarable

Compile-time-only dependencies of this suite, not included on its runtime classpath.

suiteNameRuntimeOnly

Declarable

Runtime-only dependencies of this suite.

suiteNameAnnotationProcessor

Declarable

Annotation processors used when compiling this suite.
Configuration name Role Description

suiteNameCompileClasspath (extends suiteNameCompileOnly, suiteNameImplementation)

Resolvable

The classpath used to compile this suite.

suiteNameRuntimeClasspath (extends suiteNameRuntimeOnly, suiteNameImplementation)

Resolvable

The classpath used to run this suite.

Each test suite target also produces an outgoing variant containing its test execution results — see Outgoing Variants below.

Tasks

The JVM Test Suite plugin adds the following task to the project:

test — Test

Depends on: testClasses from the java plugin, and all tasks which produce the test runtime classpath

Runs the tests using the framework configured for the default test suite.

Additional instances of Test tasks will be automatically created for each test suite added via the testing extension.

Configuration

See the TestingExtension class in the API documentation.

Terminology and Modeling

The JVM Test Suite Plugin introduces some modeling concepts backed by new APIs. Here are their definitions.

Test Suite

A test suite is a collection of JVM-based tests.

Test Suite Target

By default, each test suite has a single target. This results in a 1:1:1 relationship between test suite, test suite target and a matching Test task. The name of this default target’s Test task is derived from the suite name.

A suite is not limited to this single, default target, however. You can register additional targets on any suite, including the built-in test suite — see Add an additional target to a test suite below. Each additional target gets its own independently-configurable Test task, sharing the suite’s sources, dependencies and testing framework, so it is possible to run the same suite more than once with different task-level configuration (for example, a different JVM via javaLauncher, different system properties, or different test filters).

Each test suite has some configuration that is common to all tests contained in the suite:

  • Testing framework

  • Sources

  • Dependencies

In the future, other properties may be specified in the test suite which may influence the toolchains selected to compile and run tests.

The Test task of the default target inherits its name from the suite; additional targets are named however you register them. Other properties of the Test task are configurable.

Configuration Examples

Here are several examples to illustrate the configurability of test suites.

Declare an additional test suite

build.gradle.kts
testing {
    suites { (1)
        val test = named<JvmTestSuite>("test") { (2)
            useJUnitJupiter() (3)
        }

        register<JvmTestSuite>("integrationTest") { (4)
            dependencies {
                implementation(project()) (5)
            }

            targets { (6)
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
        }
    }
}

tasks.named("check") { (7)
    dependsOn(testing.suites.named("integrationTest"))
}
build.gradle
testing {
    suites { (1)
        test { (2)
            useJUnitJupiter() (3)
        }

        integrationTest(JvmTestSuite) { (4)
            dependencies {
                implementation project() (5)
            }

            targets { (6)
                all {
                    testTask.configure {
                        shouldRunAfter(test)
                    }
                }
            }
        }
    }
}

tasks.named('check') { (7)
    dependsOn(testing.suites.integrationTest)
}
1 Configure all test suites for this project.
2 Configure the built-in test suite. This suite is automatically created for backwards compatibility. You must specify a testing framework to use in order to run these tests (e.g. JUnit 4, JUnit Jupiter). This suite is the only suite that will automatically have access to the production source’s implementation dependencies, all other suites must explicitly declare these.
3 Declare this test suite uses JUnit Jupiter. The framework’s dependencies are automatically included. It is always necessary to explicitly configure the built-in test suite even if JUnit 4 is desired.
4 Define a new suite called integrationTest. Note that all suites other than the built-in test suite will by convention work as if useJUnitJupiter() was called. You do not have to explicitly configure the testing framework on these additional suites, unless you wish to change to another framework.
5 Add a dependency on the production code of the project to the integrationTest suite targets. By convention, only the built-in test suite will automatically have a dependency on the production code of the project.
6 Configure all targets of this suite. By convention, test suite targets have no relationship to other Test tasks. This example shows that the slower integration test suite targets should run after all targets of the test suite are complete.
7 Configure the check task to depend on all integrationTest targets. By convention, test suite targets are not associated with the check task.

Invoking the check task on the above configured build should show output similar to:

> Task :compileJava
> Task :processResources NO-SOURCE
> Task :classes
> Task :jar
> Task :compileTestJava
> Task :processTestResources NO-SOURCE
> Task :testClasses
> Task :test
> Task :compileIntegrationTestJava
> Task :processIntegrationTestResources NO-SOURCE
> Task :integrationTestClasses
> Task :integrationTest
> Task :check

BUILD SUCCESSFUL in 0s
6 actionable tasks: 6 executed

Note that the integrationTest test suite does not run until after the test test suite completes.

Configure the built-in test suite

build.gradle.kts
testing {
    suites {
        named<JvmTestSuite>("test") {
            useTestNG() (1)

            targets {
                all {
                    testTask.configure { (2)
                        // set a system property for the test JVM(s)
                        systemProperty("some.prop", "value")
                        options { (3)
                            val options = this as TestNGOptions
                            options.preserveOrder = true
                        }
                    }
                }
            }
        }
    }
}
build.gradle
testing { (1)
    suites {
        test {
            useTestNG() (1)

            targets {
                all {
                    testTask.configure { (2)
                        // set a system property for the test JVM(s)
                        systemProperty 'some.prop', 'value'
                        options { (3)
                            preserveOrder = true
                        }
                    }
                }
            }
        }
    }
}
1 Declare the test test suite uses the TestNG test framework.
2 Lazily configure the test task of all targets of the suite; note the return type of testTask is TaskProvider<Test>.
3 Configure more detailed test framework options. The options will be a subclass of org.gradle.api.tasks.testing.TestFrameworkOptions, in this case it is org.gradle.api.tasks.testing.testng.TestNGOptions.

Add an additional target to a test suite

Every suite, including the built-in test suite, can have more than one target. Registering an additional target creates another independently-configurable Test task that shares the suite’s sources, dependencies and testing framework — it does not create a new suite or a new source set.

build.gradle.kts
testing {
    suites {
        named<JvmTestSuite>("test") {
            useJUnitJupiter()

            targets {
                register("otherTest") { (1)
                    testTask.configure {
                        systemProperty("category", "smoke") (2)
                    }
                }
            }
        }
    }
}
build.gradle
testing {
    suites {
        test {
            useJUnitJupiter()

            targets {
                register('otherTest') { (1)
                    testTask.configure {
                        systemProperty 'category', 'smoke' (2)
                    }
                }
            }
        }
    }
}
1 Register otherTest as a second target of the built-in test suite. This creates an otherTest task, alongside the existing test task, that runs the same suite’s tests.
2 Configure the new target’s Test task independently of the suite’s default target — here, setting a system property that only applies when running via otherTest.

This is the officially recommended way to give a suite more than one target: there is no separate "multi-target" API beyond registering additional entries in the suite’s targets container, and no restriction on how many targets a suite — including the built-in test suite — can have. See the upgrade guide for the migration scenario that motivated this pattern: recreating a standalone Test task that previously relied on the deprecated test source set convention.

Configure dependencies of a test suite

build.gradle.kts
testing {
    suites {
        named<JvmTestSuite>("test") { (1)
            dependencies {
                // Note that this is equivalent to adding dependencies to testImplementation in the top-level dependencies block
                implementation("org.assertj:assertj-core:3.21.0") (2)
                annotationProcessor("com.google.auto.value:auto-value:1.9") (3)
            }
        }
    }
}
build.gradle
testing {
    suites {
        test { (1)
            dependencies {
                // Note that this is equivalent to adding dependencies to testImplementation in the top-level dependencies block
                implementation 'org.assertj:assertj-core:3.21.0' (2)
                annotationProcessor 'com.google.auto.value:auto-value:1.9' (3)
            }
        }
    }
}
1 Configure the built-in test test suite.
2 Add the assertj library to the test’s compile and runtime classpaths. The dependencies block within a test suite is already scoped for that test suite. Instead of having to know the global name of the configuration, test suites have a consistent name that you use in this block for declaring implementation, compileOnly, runtimeOnly and annotationProcessor dependencies.
3 Add the Auto Value annotation processor to the suite’s annotation processor classpath so that it will be run when compiling the tests.

Configure dependencies of a test suite to reference project outputs

build.gradle.kts
dependencies {
    api("com.google.guava:guava:30.1.1-jre") (1)
    implementation("com.fasterxml.jackson.core:jackson-databind:2.13.3") (2)
}

testing {
    suites {
        register<JvmTestSuite>("integrationTest") {
            dependencies {
                implementation(project()) (3)
            }
        }
    }
}
build.gradle
dependencies {
    api 'com.google.guava:guava:30.1.1-jre' (1)
    implementation 'com.fasterxml.jackson.core:jackson-databind:2.13.3' (2)
}

testing {
    suites {
        integrationTest(JvmTestSuite) {
            dependencies {
                implementation project() (3)
            }
        }
    }
}
1 Add a production dependency which is used as part of the library’s public API.
2 Add a production dependency which is only used internally and is not exposed as part of this project’s public classes.
3 Configure a new test suite which adds a project() dependency to the suite’s compile and runtime classpaths. This dependency provides access to the project’s outputs as well as any dependencies declared on its api and compileOnlyApi configurations.

Configure a test suite to use test fixtures

Like a project() dependency, a dependency on this project’s own test fixtures is not added automatically to any suite other than the built-in test suite — you add it the same way, using testFixtures(Object) inside the suite’s dependencies {} block:

build.gradle.kts
testing {
    suites {
        register<JvmTestSuite>("integrationTest") {
            dependencies {
                implementation(project()) (1)
                implementation(testFixtures(project())) (2)
            }
        }
    }
}
build.gradle
testing {
    suites {
        integrationTest(JvmTestSuite) {
            dependencies {
                implementation project() (1)
                implementation testFixtures(project()) (2)
            }
        }
    }
}
1 Add a project() dependency so integrationTest can see this project’s production code, same as the previous example.
2 Add a dependency on this project’s own test fixtures.

The testFixtures(project()) dependency resolves against this project’s own testFixturesApiElements/testFixturesRuntimeElements (see the Java Test Fixtures Plugin’s configurations). To use the fixtures of another project instead, replace project() with project(":other-project") in both dependency declarations.

For example, given this fixture in src/testFixtures/java, which builds a handful of reusable Person instances:

src/testFixtures/java/com/acme/Simpsons.java
public class Simpsons {
    private static final Person HOMER = new Person("Homer", "Simpson");
    private static final Person MARGE = new Person("Marjorie", "Simpson");
    private static final Person BART = new Person("Bartholomew", "Simpson");
    private static final Person LISA = new Person("Elisabeth Marie", "Simpson");
    private static final Person MAGGIE = new Person("Margaret Eve", "Simpson");
    private static final List<Person> FAMILY = Arrays.asList(HOMER, MARGE, BART, LISA, MAGGIE);

    public static Person homer() {
        return HOMER;
    }

    public static Person marge() {
        return MARGE;
    }

    public static Person bart() {
        return BART;
    }

    public static Person lisa() {
        return LISA;
    }

    public static Person maggie() {
        return MAGGIE;
    }

    public static List<Person> family() {
        return FAMILY;
    }
}

The integrationTest suite can consume it directly, exactly as it would consume any other suite-local class:

src/integrationTest/java/com/acme/SimpsonsIntegrationTest.java
public class SimpsonsIntegrationTest {
    @Test
    public void homerHasTheRightName() { (1)
        Person homer = Simpsons.homer();

        assertEquals("Homer", homer.getFirstName());
        assertEquals("Simpson", homer.getLastName());
    }

    @Test
    public void familyHasFiveMembers() { (2)
        assertEquals(5, Simpsons.family().size());
    }
}
1 Person comes from the project’s own production code, available because of the implementation(project()) dependency.
2 Simpsons comes from the project’s test fixtures, available because of the implementation(testFixtures(project())) dependency.

Configure source directories of a test suite

build.gradle.kts
testing {
    suites {
        register<JvmTestSuite>("integrationTest") { (1)
            sources { (2)
                java { (3)
                    setSrcDirs(listOf("src/it/java")) (4)
                }
            }
        }
    }
}
build.gradle
testing {
    suites {
        integrationTest(JvmTestSuite) { (1)
            sources { (2)
                java { (3)
                    srcDirs = ['src/it/java'] (4)
                }
            }
        }
    }
}
1 Declare and configure a suite named integrationTest. The SourceSet and synthesized Test tasks will be based on this name.
2 Configure the sources of the test suite.
3 Configure the java SourceDirectorySet of the test suite.
4 Overwrite the srcDirs property, replacing the conventional src/integrationTest/java location with src/it/java.

Configure the Test task for a test suite

build.gradle.kts
testing {
    suites {
        named<JvmTestSuite>("integrationTest") {
            targets {
                all { (1)
                    testTask.configure {
                        setMaxHeapSize("512m") (2)
                    }
                }
            }
        }
    }
}
build.gradle
testing {
    suites {
        integrationTest {
            targets {
                all { (1)
                    testTask.configure {
                        maxHeapSize = '512m' (2)
                    }
                }
            }
        }
    }
}
1 Configure the integrationTest task created by declaring a suite of the same name.
2 Configure the Test task properties.

Test tasks associated with a test suite target can also be configured directly, by name. It is not necessary to configure via the test suite DSL.

Sharing configuration between multiple test suites

There are several ways to share configuration between multiple test suites to avoid the duplication of dependencies or other configuration. Each method is a tradeoff between flexibility and the use of a declarative vs. imperative configuration style.

  1. Use the configureEach method on the suites container to configure every test suite the same way.

  2. Use withType and matching with configureEach to filter test suites and configure a subset of them.

  3. Extract the configuration block to a local variable and apply it only to the desired test suites.

Method 1: Use configureEach

This is the most straightforward way to share configuration across every test suite. The configuration is applied to each test suite.

build.gradle.kts
testing {
    suites {
        withType<JvmTestSuite> { (1)
            useJUnitJupiter()
            dependencies { (2)
                implementation("org.mockito:mockito-junit-jupiter:4.6.1")
            }
        }
        (3)
        register<JvmTestSuite>("integrationTest")
        register<JvmTestSuite>("functionalTest") {
            dependencies { (4)
                implementation("org.apache.commons:commons-lang3:3.11")
            }
        }
    }
}
build.gradle
testing {
    suites {
        configureEach { (1)
            useJUnitJupiter()
            dependencies { (2)
                implementation('org.mockito:mockito-junit-jupiter:4.6.1')
            }
        }
        (3)
        integrationTest(JvmTestSuite)
        functionalTest(JvmTestSuite) {
            dependencies { (4)
                implementation('org.apache.commons:commons-lang3:3.11')
            }
        }
    }
}
1 Configure every JVM test suite
2 Provide the dependencies to be shared by all test suites
3 Define the additional test suites which will be configured upon creation
4 Add dependencies specific to the functionalTest test suite

Method 2: Use withType, matching and configureEach

This method adds filtering commands to only apply the configuration to a subset of the test suites, as identified by name.

build.gradle.kts
testing {
    suites {
        withType(JvmTestSuite::class).matching { it.name in listOf("test", "integrationTest") }.configureEach { (1)
            useJUnitJupiter()
            dependencies {
                implementation("org.mockito:mockito-junit-jupiter:4.6.1")
            }
        }
        register<JvmTestSuite>("integrationTest")
        register<JvmTestSuite>("functionalTest") {
            useJUnit() (2)
            dependencies { (3)
                implementation("org.apache.commons:commons-lang3:3.11")
            }
        }
    }
}
build.gradle
testing {
    suites {
        withType(JvmTestSuite).matching { it.name in ['test', 'integrationTest'] }.configureEach { (1)
            useJUnitJupiter()
            dependencies {
                implementation('org.mockito:mockito-junit-jupiter:4.6.1')
            }
        }

        integrationTest(JvmTestSuite)
        functionalTest(JvmTestSuite) {
            useJUnit() (2)
            dependencies { (3)
                implementation('org.apache.commons:commons-lang3:3.11')
            }
        }
    }
}
1 Configure every JVM test suite matching the given criteria
2 Use a different testing framework for the functionalTest test suite
3 Add dependencies specific to the functionalTest test suite

Method 3: Extract a custom configuration block

This method is the most flexible, but also the most imperative.

build.gradle.kts
testing {
    suites {
        val applyMockito = { suite: JvmTestSuite -> (1)
            suite.useJUnitJupiter()
            suite.dependencies {
                implementation("org.mockito:mockito-junit-jupiter:4.6.1")
            }
        }

        /* This is the equivalent of:
            named<JvmTestSuite>("test") {
                applyMockito(this)
            }
         */
        named<JvmTestSuite>("test", applyMockito)  (2)

        /* This is the equivalent of:
            register<JvmTestSuite>("integrationTest")
            applyMockito(integrationTest.get())
         */
        register<JvmTestSuite>("integrationTest", applyMockito) (3)

        register<JvmTestSuite>("functionalTest") {
            useJUnit()
            dependencies {
                implementation("org.apache.commons:commons-lang3:3.11")
            }
        }
    }
}
build.gradle
testing {
    suites {
        def applyMockito = { suite -> (1)
            suite.useJUnitJupiter()
            suite.dependencies {
                implementation('org.mockito:mockito-junit-jupiter:4.6.1')
            }
        }

        /* This is the equivalent of:
            test {
                applyMockito(this)
            }
         */
        test(applyMockito) (2)

        /* This is the equivalent of:
            integrationTest(JvmTestSuite)
            applyMockito(integrationTest)
         */
        integrationTest(JvmTestSuite, applyMockito) (3)

        functionalTest(JvmTestSuite) {
            useJUnit()
            dependencies {
                implementation('org.apache.commons:commons-lang3:3.11')
            }
        }
    }
}
1 Define a closure which takes a single JvmTestSuite parameter and configures it
2 Apply the closure to a test suite, using the default (test) test suite
3 Alternate means of applying a configuration closure to a test suite outside of its declaration, using the integrationTest test suite

Differences between the test suite dependencies and the top-level dependencies blocks

See differences with custom dependencies blocks.

Differences between similar methods on JvmTestSuite and Test task types

Instances of JvmTestSuite have methods useJUnit() and useJUnitJupiter(), which are similar in name to methods on the Test task type: useJUnit() and useJUnitPlatform(). However, there are important differences.

Unlike the methods on the Test task, JvmTestSuite methods perform two additional pieces of configuration:

  1. JvmTestSuite#useJUnit(), #useJUnitJupiter() and other framework-specific methods automatically apply the relevant testing framework libraries to the compile and runtime classpaths of the suite targets. Note the overloaded methods like JvmTestSuite#useJUnit(String) and #useJUnitJupiter(String) allow you to provide specific versions of the framework dependencies.

  2. JvmTestSuite#useJUnit() and #useJUnitJupiter() automatically configure the suite targets' Test tasks to execute using the specified framework.

An example of this behavior is shown in the first configuration example above, Configuring the built-in test suite.

Unlike the Test task, the aforementioned methods on JvmTestSuite do not accept a closure or action for configuring the framework. These framework configuration options can be set on the individual targets.

Outgoing Variants

Each test suite creates a single outgoing variant containing its test execution results, named testResultsElementsForSuiteName. These variants are designed for consumption by the Test Report Aggregation Plugin.

The attributes will resemble the following. User-configurable attributes are highlighted below the sample.

--------------------------------------------------
Variant testResultsElementsForTest (i)
--------------------------------------------------
Description = Binary results obtained from running all targets in the 'test' Test Suite.

Capabilities
    - org.gradle.sample:list:1.0.2 (default capability)
Attributes
    - org.gradle.category         = verification
    - org.gradle.testsuite.name   = test           (1)
    - org.gradle.verificationtype = test-results

Artifacts
    - build/test-results/test/binary (artifactType = directory)
1 TestSuiteName attribute; value is derived from TestSuite#getName().

This variant is created once per suite, not once per target. If a suite has more than one target, every target’s binary results directory is added as a separate artifact to this same variant, rather than each target getting its own variant — Gradle does not yet have a way for targets to distinguish themselves via attributes. Consumers that need to tell one target’s results apart from another’s currently have to arrange that themselves, for example by adding a distinguishing attribute to the variant by hand.