With cucumber integration tests not counted in total test summary in surefire-plugin 3.5.3

[jamshidCo]

Affected version

3.5.3

Bug description

Hello,
Using JDK:

• openjdk 21.0.5 2024-10-15 LTS
• Apache Maven 3.9.9
• maven surfire plugin

Issue

When running integration tests using cucumber-junit-platform-engine with maven-surefire-plugin version 3.5.3, the tests are executed and the logs are printed, but the final test summary says:

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

This behavior does not happen in version 3.5.2 — in that version, Cucumber integration tests are correctly counted in the final summary — even though Cucumber scenarios are technically not individual JUnit test methods.

But in version 3.5.3, this seems to have changed:

• Tests still run
• Output/logs appear
• But Tests run: 0 appears at the end
  ➤ That can mess with CI pipelines or give misleading reports.

Expected Behavior

The final summary should include the number of Cucumber scenarios executed (like in 3.5.2).

[pzygielo]

• SUREFIRE-2299
• [SUREFIRE-2298] fix xml output with junit 5 nested classes (fix integration with Cucumber and Archunit) #828 (comment)

[juergencodes]

I experienced a similar problem: My maven build was successful though I had a test failure. Since it is in a setup of a custom framework based on JBehave it is interesting to see that others experience similar problems in cucumber (this issue) or e.g. Arch (following the links in SUREFIRE-2229).

I was drilling down a bit deeper and found out the following reproducer that just uses DynamicTest (which probably all the above mentioned frameworks use) and plays with providing test source.

    @TestFactory
    DynamicTest withSource() {
        return test("withSource", URI.create("pom.xml"));
    }

    @TestFactory
    DynamicTest noSource() {
        return test("noSource", null);
    }

    DynamicTest test(final String test, final URI source) {
        return DynamicTest.dynamicTest(test, source, () -> {
            System.out.println("Running '" + test + "'");
            org.junit.jupiter.api.Assertions.assertNull(source);
        });
    }

Running the test (also see attachment for "full" project) with surefire 3.5.2 fails as expected with

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.049 s <<< FAILURE! -- in SurefireReproducerTest
[ERROR] withSource().withSource -- Time elapsed: 0.012 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: <null> but was: <pom.xml>
        at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
        at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at org.junit.jupiter.api.AssertNull.failNotNull(AssertNull.java:50)
        at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:35)
        at org.junit.jupiter.api.AssertNull.assertNull(AssertNull.java:30)
        at org.junit.jupiter.api.Assertions.assertNull(Assertions.java:279)
        at SurefireReproducerTest.lambda$test$0(SurefireReproducerTest.java:20)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   expected: <null> but was: <pom.xml>
[INFO]
[ERROR] Tests run: 2, Failures: 1, Errors: 0, Skipped: 0

Switching to 3.5.3 makes

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.049 s -- in SurefireReproducerTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Mind that both tests are executed (see "Running..." output), but

• No error is reported
• The build does not fail
• The test count dropped from 2 to 1

(just in case it matters, i used Apache Maven 3.8.6)

Full reproducer, just make sure to bump 3.5.2 to 3.5.3
surefire-reproducer.zip

[juergencodes]

I forgot to mention that using a source or not isn't just something I was doing for fun. If you change the assertNull to assertNotNull, everythings works fine even with surefire 3.5.3

[INFO] Running SurefireReproducerTest
Running 'withSource'
Running 'noSource'
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.048 s <<< FAILURE! -- in SurefireReproducerTest
[ERROR] SurefireReproducerTest.noSource()[1] -- Time elapsed: 0.004 s <<< FAILURE!
org.opentest4j.AssertionFailedError: expected: not <null>
        at org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:152)
        at org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
        at org.junit.jupiter.api.AssertNotNull.failNull(AssertNotNull.java:49)
        at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:35)
        at org.junit.jupiter.api.AssertNotNull.assertNotNull(AssertNotNull.java:30)
        at org.junit.jupiter.api.Assertions.assertNotNull(Assertions.java:304)
        at SurefireReproducerTest.lambda$test$0(SurefireReproducerTest.java:20)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1596)

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR]   SurefireReproducerTest.lambda$test$0:20 expected: not <null>
[INFO]
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

I.e. the problem ONLY happens if one provides a source.

[bramklg]

I'm also running into this problem where unit test reports are still properly reported (and failures result in a build failure) but the (cucumber) integration test results are not recorded properly any more, it always says 0 tests executed.

I ran into this after upgrading spring boot from 3.4.4 to 3.4.5 which includes an upgrade of the surefire plugin from 3.5.2 to 3.5.3.