Gradle modernization #1642

rozza · 2025-03-05T17:11:43Z

This PR comprises of 3 separate commits for the purpose of aiding reviewability.

The commits themselves aren't designed to be standalone (they are not) but rather compose separate areas of work that updating the build has required.

Added library conventions and language based configurations. - Added `doclets` to `buildSrc` for the `javadocs` - Added `conventions` to centralize configuration of gradle plugins - Added `projects` to handle project level conventions JAVA-5758 JAVA-5757

Added library conventions and language based configurations. - Updated to Gradle 8 (JAVA-5756) - Removed `util:taglets` as now in `buildSrc`. - Moved `util:spock` into `driver-core` as the annotation is now shared via `testArtifacts`. - Added optional support (JAVA-5757) JAVA-5756 JAVA-5758 JAVA-5757

Added library conventions and language based configurations. - Refactored `JsonPoweredTestHelper` to handle either local and jar sources - Refactored test cases to use new `JsonPoweredTestHelper.getTestDocuments` method - Refactored test cases to normalize usage of loading test data - Moved scala integration tests into a standardized `integerationTest` directory JAVA-5758

rozza · 2025-03-12T18:11:38Z

Testing builds between `main` and `JAVA-5758` branches.

Output testing

Manual checks

javadoc - ensured taglets output correctly
scaladoc - ensured mongo-scala-driver docs include the bson-scala scaladoc

The following shows how I tested / compared outputted pom.xml files, MANIFEST.MF files and the gradle tasks run when running the check command.

Generating build output for main:

git co main
./gradlew clean publishAllPublicationsToMaven2Repository

mkdir -p ./build/checks
branch_name=$(git rev-parse --abbrev-ref HEAD)
file_prefix="./build/checks/${branch_name}"

# output pom files &> /dev/null
find ./build/repo -name "*.pom" | xargs -I {} cat {} > ${file_prefix}-poms.txt

# output manifests &> /dev/null
find ./build/repo -regex ".*[[:digit:]]\.jar" | sort | xargs -I {} unzip -p {} META-INF/MANIFEST.MF  &> ${file_prefix}-manifests.txt

# check tasks list &> /dev/null
./gradlew check --dry-run | sort -o ${file_prefix}-check-tasks.txt

Generating build output for JAVA-5758:

Note: Here we also clean the previously created build/docs directory caused by publishing the main branch. Its needed to clean up elements-list artifacts that break javadoc.

git co JAVA-5758
rm -rf build/docs
./gradlew clean publishMavenToLocalBuildRepo 

mkdir -p ./build/checks
branch_name=$(git rev-parse --abbrev-ref HEAD)
file_prefix="./build/checks/${branch_name}"

# output pom files &> /dev/null
find ./build/repo -name "*.pom" | xargs -I {} cat {} > ${file_prefix}-poms.txt

# output manifests &> /dev/null
find ./build/repo -regex ".*[[:digit:]]\.jar" | sort | xargs -I {} unzip -p {} META-INF/MANIFEST.MF  &> ${file_prefix}-manifests.txt

# check tasks list &> /dev/null
./gradlew check --dry-run | sort -o ${file_prefix}-check-tasks.txt

Comparing output

Compare pom files ✅

Compare the main branch outputted poms with that of the new gradle build:

diff ./build/checks/main-poms.txt ./build/checks/JAVA-5758-poms.txt

Differences:

All BSON jars now have the url set to bson.org (not just the bson.jar)
All BSON jars use capitalized BSON (previously it was mixed BSON and Bson)
License url updated to use https (previously it was mixed)
developerConnection updated to use https: scm:https://github.com/mongodb/mongo-java-driver.git (git@github support has been removed)
Order of some dependencies may have changed (usually slf4j)
Scala libraries bumped to latest patch.

Compare manifests ✅

Compare the main branch outputted MAINFESTS with that of the new gradle build:

diff ./build/checks/main-manifests.txt ./build/checks/JAVA-5758-manifests.txt

Differences:

Build-Version updated - expected
Bnd-LastModified updated - expected
driver-legacy now has a manifest - side effect of the single publishing config

Compare check tasks ✅

Check any tasks that existed in main but not in JAVA-5758:

diff build/checks/main-check-tasks.txt build/checks/JAVA-5758-check-tasks.txt | grep ^\<  > build/checks/check-tasks-removals.txt

codenarc / compile groovy removed from projects without spock
checkstyle removed from non java projects
util project and all tasks removed

Check any tasks added to JAVA-5758:

diff build/checks/main-check-tasks.txt build/checks/JAVA-5758-check-tasks.txt | grep ^\>  > build/checks/check-tasks-additions.txt

Spotless tasks added to all projects
integrationTest tasks for kotlin
testJar added for test artifacts
buildSrc project added

Evergreen test counts

Env & task	main	JAVA-5758		Findings
latest Sharded Cluster Auth SSL JDK8 Linux test-core	4718	4776	⬆️	All tests for a class are now reported as skipped rather than the first, when `@IgnoreIf` is set on the class. Thanks Nathan for the investigation 🔎
latest Sharded Cluster Auth SSL JDK8 Linux test-legacy	1033	1033
latest Sharded Cluster Auth SSL JDK8 Linux test-reactive	3218	3218
latest Sharded Cluster Auth SSL JDK8 Linux test-sync	4454	4460	⬆️
Load Balancer latest Auth SSL JDK21 Ubuntu	1826	1826
Scala 2.13 JDK17 7.0 Replica Set Ubuntu	944	944
Kotlin: JDK17 7.0 Replica Set Ubuntu	1314	1227	⬇️ ⚠️	Now Fixed - Kotlin extensions had the wrong junit library
JDK8 Linux Unit test bson and crypt	3813	3813

Source Main: Evergreen main build
Source PR: Evergreen patch build

Issues found and fixed by this testing

Scala language dependencies updated to api from implementation to keep the poms the same.
slf4j dependencies updated to optionalAPI from optionalImplementation to keep poms the same.
bson-kotlinx dependencies for datetime and json serialization updated to optionalAPI from optionalImplementation to keep poms the same.
Fixed various mainfest attributes as they differed from previous versions
Fixed driver-kotlin-extensions testing and updated kotlin junit dependency in libs.versions.toml to use junit5.

…to use junit 5

jyemin

LGTM!

.github/workflows/bump-version.sh

buildSrc/src/main/kotlin/project/kotlin.gradle.kts

buildSrc/src/main/kotlin/conventions/testing-spock.gradle.kts

buildSrc/build.gradle.kts

driver-core/build.gradle.kts

jyemin

Why didn't the integration test directory name change require a corresponding change in Gradle scripts?

rozza · 2025-03-19T12:06:23Z

@jyemin static tests will now pass - a side effect of moving integration tests. Also removed the unneeded extra source directory for kotlin integration tests from the kotlin project.

jyemin · 2025-03-19T13:36:32Z

Also removed the unneeded extra source directory for kotlin integration tests from the kotlin project.

Ah, so it was looking in both places, so they were found in either location. Thanks.

jyemin

LGTM

Since the validation of the correctness of changes were largely manual, please re-validate before final merge. :)

NathanQingyangXu

Looks great and I learned a lot from the PR. Thanks.

.github/workflows/bump-version.sh

NathanQingyangXu · 2025-03-21T14:52:27Z

buildSrc/build.gradle.kts

+}
+
+repositories {
+    gradlePluginPortal()


This thread is only for discussion. Did you see scenario where some plugin could only be found in gradlePluginPortal(), not in mavenCentral()? I asked for I struggled on whehter I need to use gradlePluginPortal() in mongo-hibernate project as well.

Just following the official recommendations. I imagine any third party plugins would be on maven but any gradle specific ones might not be.

NathanQingyangXu · 2025-03-21T14:53:47Z

bom/build.gradle.kts

+                // Add multiple versions with Scala suffixes for each Scala-related dependency.
+                scalaVersions!!.forEach { scalaVersion ->
+                    if (scalaVersion != defaultScalaVersion) {
+                        // Replace scala version suffix


super minor. scala version -> Scala version? Don't wanna make a fuss about it. Feel free to ignore.

vbabanin · 2025-03-25T07:39:32Z

bom/build.gradle.kts

+        pom.withXml {
+            val pomXml: Node = asNode()


It looks like the validation isn’t currently being executed, becausepom.withXml {} runs during the configuration phase (before the POM file is written), placing it inside a doLast block causes it to be skipped.

One alternative is to remove pom.WithXml block and parse the generated POM file directly using XmlParser, as we did previously:
val pomXml: Node = XmlParser().parse(destination)

The rest of the logic should not be affected after the change.

vbabanin · 2025-03-25T07:41:57Z

bom/build.gradle.kts

+        pom.withXml {
+            val pomXml: Node = asNode()
+            val dependenciesNode = pomXml.getNode("dependencyManagement").getNode("dependencies")
+            assert(dependenciesNode!!.children().isNotEmpty()) {


These assertions only take effect when the JVM is run with the -ea flag, which could lead to silent failures. We could replace assert() with require() to ensure the validation always runs.

Added buildSrc, library conventions and language based configurations. - Updated to Gradle 8 (JAVA-5756) - Added libs.versions.toml (JAVA-5291) - Removed `util:taglets` as now in `buildSrc`. - Moved `util:spock` into `driver-core` as the annotation is now shared via `testArtifacts`. - Added optional support (JAVA-5757) Migrated tests to work with conventions: - Refactored `JsonPoweredTestHelper` to handle either local and jar based resources - Added test artifacts dependencies convention for use with test artifact jars. - Refactored test cases to use new `JsonPoweredTestHelper.getTestDocuments` method - Refactored test cases to normalize usage of loading test data - Moved scala / kotlin integration tests into a standardized `integerationTest` directory Updated github action and put version in `gradle.properties`. #1642 #1654 JAVA-5291 JAVA-5756 JAVA-5757 JAVA-5758

rozza force-pushed the JAVA-5758 branch 6 times, most recently from 0144d6e to aeda926 Compare March 6, 2025 13:35

rozza changed the base branch from gradle to main March 6, 2025 13:39

rozza changed the base branch from main to gradle March 6, 2025 13:40

rozza marked this pull request as draft March 6, 2025 13:42

rozza force-pushed the JAVA-5758 branch from aeda926 to d730cc3 Compare March 6, 2025 14:46

rozza marked this pull request as ready for review March 6, 2025 14:49

rozza requested a review from katcharov March 6, 2025 14:49

rozza force-pushed the JAVA-5758 branch 4 times, most recently from 1ca551a to d80ee10 Compare March 11, 2025 14:25

rozza changed the title ~~Gradle modernization to precompile build scripts~~ Gradle modernization Mar 11, 2025

rozza removed the request for review from katcharov March 11, 2025 16:30

rozza marked this pull request as draft March 11, 2025 16:30

rozza force-pushed the JAVA-5758 branch 3 times, most recently from 5fa97ac to 3f03dbd Compare March 12, 2025 12:11

rozza added 3 commits March 12, 2025 17:51

Gradle: Added buildSrc (1/3)

266a2ef

Added library conventions and language based configurations. - Added `doclets` to `buildSrc` for the `javadocs` - Added `conventions` to centralize configuration of gradle plugins - Added `projects` to handle project level conventions JAVA-5758 JAVA-5757

rozza force-pushed the JAVA-5758 branch from 3f03dbd to 3311cc4 Compare March 12, 2025 17:52

Fix kotlin junit library version and update driver-kotlin-extensions …

ea08cde

…to use junit 5

rozza marked this pull request as ready for review March 13, 2025 14:37

jyemin self-requested a review March 13, 2025 15:18

rozza requested a review from NathanQingyangXu March 18, 2025 12:57

jyemin approved these changes Mar 18, 2025

View reviewed changes

NathanQingyangXu reviewed Mar 18, 2025

View reviewed changes

.github/workflows/bump-version.sh Show resolved Hide resolved

NathanQingyangXu reviewed Mar 18, 2025

View reviewed changes

buildSrc/src/main/kotlin/project/kotlin.gradle.kts Outdated Show resolved Hide resolved

NathanQingyangXu reviewed Mar 18, 2025

View reviewed changes

buildSrc/src/main/kotlin/conventions/testing-spock.gradle.kts Show resolved Hide resolved

NathanQingyangXu reviewed Mar 18, 2025

View reviewed changes

buildSrc/build.gradle.kts Show resolved Hide resolved

NathanQingyangXu reviewed Mar 19, 2025

View reviewed changes

driver-core/build.gradle.kts Show resolved Hide resolved

rozza added 3 commits March 19, 2025 09:58

Normalize integration test directory naming

6328f9d

Spotless updates and formatting

26527ba

Explicitly include junit 5

c666dfd

rozza requested a review from NathanQingyangXu March 19, 2025 10:16

Fix spotbugs exclude location string

df1e4f5

jyemin reviewed Mar 19, 2025

View reviewed changes

rozza added 2 commits March 19, 2025 11:59

Update detekt baseline now testing integrationTest

62b1c38

Update kotlin convention - integration tests locations now normalized

c72e031

rozza requested a review from jyemin March 19, 2025 12:03

rozza mentioned this pull request Mar 19, 2025

JAVA-5452 #1378

Closed

jyemin approved these changes Mar 19, 2025

View reviewed changes

NathanQingyangXu approved these changes Mar 19, 2025

View reviewed changes

jyemin reviewed Mar 19, 2025

View reviewed changes

.github/workflows/bump-version.sh Show resolved Hide resolved

rozza added 2 commits March 19, 2025 16:31

Fix manifest configuration race

75c9334

Merge branch 'gradle' into JAVA-5758

406e1cd

NathanQingyangXu reviewed Mar 21, 2025

View reviewed changes

rozza merged commit c4053a4 into mongodb:gradle Mar 24, 2025
60 checks passed

rozza deleted the JAVA-5758 branch March 24, 2025 10:30

vbabanin reviewed Mar 25, 2025

View reviewed changes

Gradle modernization #1642

Gradle modernization #1642

Uh oh!

Conversation

rozza commented Mar 5, 2025

Uh oh!

rozza commented Mar 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Testing builds between main and JAVA-5758 branches.

Output testing

Manual checks

Generating build output for main:

Generating build output for JAVA-5758:

Comparing output

Compare pom files ✅

Compare manifests ✅

Compare check tasks ✅

Evergreen test counts

Issues found and fixed by this testing

Uh oh!

jyemin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jyemin left a comment

Choose a reason for hiding this comment

Uh oh!

rozza commented Mar 19, 2025

Uh oh!

jyemin commented Mar 19, 2025

Uh oh!

jyemin left a comment

Choose a reason for hiding this comment

Uh oh!

NathanQingyangXu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

NathanQingyangXu Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

rozza Mar 24, 2025

Choose a reason for hiding this comment

Uh oh!

NathanQingyangXu Mar 21, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

vbabanin Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

vbabanin Mar 25, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rozza commented Mar 12, 2025 •

edited

Loading

Testing builds between `main` and `JAVA-5758` branches.