Maven surefire reruns successful tests when JUnitPlatformProvider used

[Yahmus]

Hello!

maven.surefire.version > 3.1.2
junit.version > 5.11.4

We were using Junit 4 in our project. Then we decided to move to Junit 5 version, but there are still many Junit 4 tests in our project. We are using maven surefire to run tests. Since we've migrated to Junit 5, surefire is using org.apache.maven.surefire.junitplatform.JUnitPlatformProvider to run tests. It was org.apache.maven.surefire.junit4.JUnit4Provider
while we were using Junit 4 only.

The thing is - now when I'm running tests with rerunFailingTestsCount

mvn surefire:test -B -Dtest=JUnit4Test -Dsurefire.rerunFailingTestsCount=2

if whole test class fails (in @afterclass for example) - surefire reruns all test class, even successful test methods.

With JUnitPlatformProvider:

[INFO]
[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.null
[ERROR] Run 1: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 2: JUnit4Test.methodThatFailsWholeClass:35
[ERROR] Run 3: JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[INFO]
[ERROR] Tests run: 7, Failures: 1, Errors: 0, Skipped: 0

With JUnit4Provider:

[INFO] Results:
[INFO]
[ERROR] Failures:
[ERROR] JUnit4Test.methodThatFailsWholeClass:35
[INFO]
[ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0

I've prepared sample project where this behaviour can be reproduced - https://github.com/Yahmus/JunitProviderTest
You can run the test class, check the behaviour, then switch the provider by adding

  			<dependencies>   
  				<dependency>   
  					<groupId>org.apache.maven.surefire</groupId>   
  					<artifactId>surefire-junit4</artifactId>   
  					<version>${maven.surefire.version}</version>   
  				</dependency>   
  			</dependencies>

to maven-surefire-plugin in pom.xml and run the test class again.

I've found a little bit similar issue but it doesn't really help #4026
I don't actually understand is this a bug or an expected behaviour? Am I doing smth wrong? Could you please help?

[marcphilipp]

Since that's implemented in Surefire, please raise an issue in their issue tracker.