KAFKA-7197 expand gradle build: include Scala 2.13 #5454

dejan2609 · 2018-08-02T22:47:09Z

Final PR description (June 2019):

include Scala 2.13 into gradle build
handle future milestone and RC versions of Scala in a better way
if no Scala version is specified, default to scala 2.12 (bump from 2.11)
include certain Xlint options (removed by Scala 2.13) for Scala 2.11/2.12 build only
upgrade versions for dependencies:
- scalaLogging: 3.9.0 -->> 3.9.2
- scalatest: 3.0.7 -->> 3.0.8
- scoverage: 1.3.1 -->> 1.4.0

Initial PR description (August 2018):

details:

build expanded in order to use Scala 2.13 milestone version (version 2.13.0-M3 is used)
'scoverage' version upgraded: 1.3.1 -->> 1.4.0-M3

Note: ./gradlew -PscalaVersion=2.13 jar -q returns few errors (well, as expected):

8 errors found

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:compileScala'.
> Compilation failed

Upgrade to Scala 2.13.0-M4 depends on:

Add Scala 2.13.0-M4 support scoverage/scalac-scoverage-plugin#229 Add Scala 2.13.0-M4 support

ijuma · 2018-08-08T04:26:21Z

Thanks for the PR. M5 is due soon, I think we should go straight to that in preparation of RC1. We should probably not build it by default until the final release.

gradle/dependencies.gradle

ijuma · 2018-09-08T16:14:17Z

@dejan2609 2.13.0-M5 has been released. Maybe update this PR to use that?

dejan2609 · 2018-09-29T21:38:24Z

@ijuma Sure, sorry for my late answer (I was traveling). I will try to update PR asap.

dejan2609 · 2018-10-09T12:41:45Z

Update / related PR:
scoverage/scalac-scoverage-plugin#234 Add Scala 2.13.0-M5 support

dejan2609 · 2018-11-19T11:28:25Z

Update 2: related PR mentioned above is solved, I will try to accommodate this PR accordingly.

dejan2609 · 2019-02-17T13:25:43Z

Long overdue, I will try to squeeze this in a next few days.
FYI @ijuma

dejan2609 · 2019-02-26T07:48:16Z

Ok, I have some solution, but I need to hack Xlint options in addition, suppress unused import statements temporarily and so on (after that I will push some solution here):

dejans@BEG-IT-09 MINGW64 /c/Work/IDEA-2019.1/workspace/kafka (trunk)
$ ./gradlew -x test -PscalaVersion=2.13 jar -q
Building project 'core' with Scala version 2.13.0-M5
Building project 'streams-scala' with Scala version 2.13.0-M5
'by-name-right-associative' is not a valid choice for '-Xlint'
'unsound-match' is not a valid choice for '-Xlint'
<=------------> 14% EXECUTING [14s]
> :core:compileScala

dejans@BEG-IT-09 MINGW64 /c/Work/IDEA-2019.1/workspace/kafka (trunk)
$ ./gradlew -x test -PscalaVersion=2.12 jar -q
Building project 'core' with Scala version 2.12.8
Building project 'streams-scala' with Scala version 2.12.8
<=------------> 14% EXECUTING [9s]
> :core:compileScala

dejans@BEG-IT-09 MINGW64 /c/Work/IDEA-2019.1/workspace/kafka (trunk)
$ date
Tue Feb 26 08:25:43 CEST 2019

dejans@BEG-IT-09 MINGW64 /c/Work/IDEA-2019.1/workspace/kafka (trunk)

dejan2609 · 2019-02-26T10:27:29Z

Ok, I pushed commit (after quite a while).
Build passes locally for Scala 2.11 and 2.12 and (unsurprisingly) fails for Scala 2.13 with bunch of errors.

@ijuma now I see that I forgot to comply with your comment above (2.13 should not be listed among defaultScalaVersions yet... I will fix that).

dejan2609 · 2019-02-26T17:19:51Z

Just to sum it up:

2.13 is moved from defaultScalaVersions to availableScalaVersions
only 1 test failed (not related with these changes, maybe test is flaky)
now I see that I could check for a dash instead for a letter M, i.e.
versions.scala.contains('-') is much better than versions.scala.contains('M') because that way both milestones and release candidates would be covered:
-- https://github.com/scala/scala/releases/tag/v2.13.0-M5
-- https://contributors.scala-lang.org/t/2-13-0-rc1-timing-january-2019/2375

Let me know what you think @ijuma @ewencp

dejan2609 · 2019-03-02T06:31:02Z

retest this please

dejan2609 · 2019-03-02T09:42:22Z

retest this please

dejan2609 · 2019-03-02T14:21:25Z

@ijuma please review (note: there are some flaky test).

dejan2609 · 2019-03-13T09:46:11Z

@ijuma please review.

build.gradle

gradle/dependencies.gradle

dejan2609 · 2019-03-18T09:25:18Z

Thanx @ewencp for your review(s) above, I will try to implement/modify based on your suggestions.

dejan2609 · 2019-03-20T14:44:11Z

New commit pushed. @ewencp and @ijuma: feel free to check and review.

I added related comment(s) that could be seen as an overzealous/overkill (will make them short if required or move them into commit message).

dejan2609 · 2019-04-05T21:27:03Z

Update: Scala 2.13.0-RC1 has just been released:

dejan2609 · 2019-04-23T19:50:04Z

Related: scoverage/scalac-scoverage-plugin#250 Add Scala 2.13.0-RC1 support

dejan2609 · 2019-04-29T10:33:36Z

Update: scalac-scoverage-plugin and scalac-scoverage-runtime versions for 2.13.0-RC1 are released.

scala-logging release is pending:

Update scala 2.13.0-M5 to 2.13.0-RC1 lightbend-labs/scala-logging#159 Update scala 2.13.0-M5 to 2.13.0-RC1
https://repo1.maven.org/maven2/com/typesafe/scala-logging/

ijuma

Thanks for the PR. I left some minor comments. Apart from that:

Please rebase as we just upgraded the Gradle version in trunk.
I think we can go straight to 2.13.0-RC1 since it seems like most libraries are already there.

Hopefully we can enable 2.13 by default for Kafka 2.4.

README.md

build.gradle

gradle/dependencies.gradle

README.md

ennru · 2019-06-12T10:08:31Z

Scalatest 3.0.8 and scala-logging 3.9.2 are available for Scala 2.13.0 now. Getting there...

dejan2609 · 2019-06-13T18:18:07Z

Ok, all dependencies are here, but I just digged some warnings that should be suppressed (by-name-right-associative and unsound-match made their way into Scala 2.13 build, I will have to investigate that).

dejan2609 · 2019-06-13T21:34:55Z

Update: switch from if ((versions.baseScala = '2.11') || (versions.baseScala = '2.12')) to if (versions.baseScala in ['2.11','2.12']) did the trick (and it looks better, I guess).

Build works for Scala 2.11 / 2.12 and (unsurprisingly) returns some errors for Scala 2.13: ./gradlew clean -PscalaVersion=2.13 jar -q

151 errors found

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':core:compileScala'.
> Compilation failed

ennru · 2019-06-14T05:05:44Z

Great!
So what would be the plan here:

Will the build updates get merged and the code changes be worked on another PR, or
Should someone else branch this PR and suggest how to adapt to the Scala 2.13 (collection) APIs?

dejan2609 · 2019-06-14T05:52:52Z

Thanx @ennru, I was just about to ask the same question.

@ijuma, @ewencp: what is your take on this ?

dejan2609 · 2019-06-14T05:53:20Z

retest this please

dejan2609 · 2019-06-19T05:18:10Z

Pinging @ijuma and @ewencp again.

ijuma · 2019-06-19T15:26:47Z

@ennru We have to support Scala 2.11, 2.12 and 2.13. I don't expect an immediate code change to adapt to 2.13 collection APIs. Maybe further down the line when we drop 2.11 and consider switching to 2.13 as the recommended version (we'd still have to be compatible with 2.12 at that point).

ijuma · 2019-06-19T15:27:29Z

retest this please

build.gradle

gradle/dependencies.gradle

dejan2609 · 2019-06-21T07:06:33Z

@ijuma solved, rebased, squashed and force-pushed.

ijuma · 2019-06-22T00:35:00Z

@dejan2609 Can you please rebase (unfortunately your branch includes a change that broke one test, it has since been fixed) and update the PR description to match the current reality? I should be able to merge then.

ijuma

Sorry, noticed a couple of additional changes. We should be good to go after that.

README.md

gradle/dependencies.gradle

ijuma

I made a couple of changes and it LGTM. It just needs a rebase and for the PR description to be updated.

…d default Scala versions details/notes: * gradle build now includes Scala 2.13; Scala 2.11/2.12 builds are intact * if no Scala version is specified, default to scala 2.12 (was 2.11) * 'baseScala' version resolving is expanded in order to handle milestone and RC versions of Scala * Scala 2.13 removes some scalac options and hence gradle build will include certain Xlint options ('by-name-right-associative' and 'unsound-match') for Scala 2.11/2.12 only * 'README.md' is changed accordingly * required version upgrades for dependencies: ** scalaLogging: 3.9.0 -->> 3.9.2 ** scalatest: 3.0.7 -->> 3.0.8 ** scoverage: 1.3.1 -->> 1.4.0

dejan2609 · 2019-06-22T08:08:16Z

@ijuma Done as requested, all changes are bundled into one commit.
I took the liberty to alter PR description in a way that list both initial and final state (given a fact that PR exists for a while).

ijuma · 2019-06-22T16:19:28Z

Scala 2.12 is green, Scala 2.11 had 1 unrelated flaky connect test. Merging to master.

ijuma · 2019-06-23T00:05:15Z

While updating Jenkins to build with Scala 2.13, I ran into a number of compiler errors. I'll submit a PR with fixes.

ijuma · 2019-06-24T10:31:24Z

See #6989

Scala 2.13 support was added to build via #5454. This PR adjusts the code so that it compiles with 2.11, 2.12 and 2.13. Changes: * Add `scala-collection-compat` dependency. * Import `scala.collection.Seq` in a number of places for consistent behavior between Scala 2.11, 2.12 and 2.13. * Remove wildcard imports that were causing the Java classes to have priority over the Scala ones, related Scala issue: scala/scala#6589. * Replace parallel collection usage with `Future`. The former is no longer included by default in the standard library. * Replace val _: Unit workaround with one that is more concise and works with Scala 2.13 * Replace `filterKeys` with `filter` when we expect a `Map`. `filterKeys` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `mapValues` with `map` or add a `toMap` as an additional transformation when we expect a `Map`. `mapValues` returns a view that doesn't implement the `Map` trait in Scala 2.13. * Replace `breakOut` with `iterator` and `to`, `breakOut` was removed in Scala 2.13. * Replace to() with toMap, toIndexedSeq and toSet * Replace `mutable.Buffer.--` with `filterNot`. * ControlException is an abstract class in Scala 2.13. * Variable arguments can only receive arrays or immutable.Seq in Scala 2.13. * Use `Factory` instead of `CanBuildFrom` in DecodeJson. `CanBuildFrom` behaves a bit differently in Scala 2.13 and it's been deprecated. `Factory` has the behavior we need and it's available via the compat library. * Fix failing tests due to behavior change in Scala 2.13, "Map.values.map is not strict in Scala 2.13" (scala/bug#11589). * Use Java collections instead of Scala ones in StreamResetter (a Java class). * Adjust CheckpointFile.write to take an `Iterable` instead of `Seq` to avoid unnecessary collection copies. * Fix DelayedElectLeader to use a Map instead of Set and avoid `to` call that doesn't work in Scala 2.13. * Use unordered map for mapping in SimpleAclAuthorizer, mapping of ordered maps require an `Ordering` in Scala 2.13 for safety reasons. * Adapt `ConsumerGroupCommand` to compile with Scala 2.13. * CoreUtils.min takes an `Iterable` instead of `TraversableOnce`, the latter does not exist in Scala 2.13. * Replace `Unit` with `()` in a couple places. Scala 2.13 is stricter when it expects a value instead of a type. * Fix bug in CustomQuotaCallbackTest where we did not necessarily set `partitionRatio` correctly, `forall` can terminate early. * Add a couple of spotbugs exclusions that are needed by code generated by Scala 2.13 * Remove unused variables, simplify some code and remove procedure syntax in a few places. * Remove unused `CoreUtils.JSONEscapeString`. Reviewers: Manikumar Reddy <[email protected]>, José Armando García Sancio <[email protected]>

ijuma reviewed Aug 8, 2018

View reviewed changes

gradle/dependencies.gradle Show resolved Hide resolved

marcospereira mentioned this pull request Feb 21, 2019

Scala 2.13 support lagom/lagom#1240

Closed

francescopellegrini mentioned this pull request Feb 22, 2019

Build with Scala 2.13 embeddedkafka/embedded-kafka#33

Closed

dejan2609 force-pushed the develop branch from 6046a55 to ce4d68e Compare February 26, 2019 10:21

dejan2609 force-pushed the develop branch from ce4d68e to 461467b Compare February 26, 2019 13:29

dejan2609 force-pushed the develop branch from 0309e42 to 49484fd Compare March 1, 2019 20:12

dejan2609 changed the title ~~[WIP] KAFKA-7197 Scala '2.13-milestone' experimental build (gradle build improvement)~~ KAFKA-7197 Scala '2.13-milestone' experimental build (gradle build improvement) Mar 1, 2019

dejan2609 force-pushed the develop branch from 49484fd to 5b29320 Compare March 4, 2019 08:42

ewencp reviewed Mar 17, 2019

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

gradle/dependencies.gradle Outdated Show resolved Hide resolved

gradle/dependencies.gradle Show resolved Hide resolved

marcospereira mentioned this pull request Mar 27, 2019

Play Sprint Plan 2019-03-25 playframework/play-meta#56

Closed

66 tasks

ijuma reviewed Apr 29, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

build.gradle Show resolved Hide resolved

gradle/dependencies.gradle Outdated Show resolved Hide resolved

gradle/dependencies.gradle Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

dejan2609 force-pushed the develop branch from 1273b97 to 49019ca Compare May 5, 2019 18:26

dejan2609 force-pushed the develop branch from 553e22e to befa370 Compare June 13, 2019 17:18

dejan2609 force-pushed the develop branch from befa370 to cd2fbf4 Compare June 13, 2019 21:27

ijuma reviewed Jun 19, 2019

View reviewed changes

build.gradle Outdated Show resolved Hide resolved

ijuma reviewed Jun 19, 2019

View reviewed changes

gradle/dependencies.gradle Show resolved Hide resolved

dejan2609 force-pushed the develop branch from cd2fbf4 to 8bf3122 Compare June 21, 2019 06:54

ijuma reviewed Jun 22, 2019

View reviewed changes

README.md Outdated Show resolved Hide resolved

gradle/dependencies.gradle Outdated Show resolved Hide resolved

ijuma approved these changes Jun 22, 2019

View reviewed changes

dejan2609 changed the title ~~KAFKA-7197 Scala '2.13-milestone' experimental build (gradle build improvement)~~ KAFKA-7197 expand gradle build: include Scala 2.13 Jun 22, 2019

dejan2609 force-pushed the develop branch from bd26e67 to 01b8ae0 Compare June 22, 2019 07:42

ijuma merged commit 5339d2d into apache:trunk Jun 22, 2019

dejan2609 deleted the develop branch June 23, 2019 11:37

ijuma mentioned this pull request Jun 24, 2019

MINOR: Make the build compile with Scala 2.13 #6989

Merged

3 tasks

KAFKA-7197 expand gradle build: include Scala 2.13 #5454

KAFKA-7197 expand gradle build: include Scala 2.13 #5454

Uh oh!

Conversation

dejan2609 commented Aug 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ijuma commented Aug 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ijuma commented Sep 8, 2018

Uh oh!

dejan2609 commented Sep 29, 2018

Uh oh!

dejan2609 commented Oct 9, 2018

Uh oh!

dejan2609 commented Nov 19, 2018

Uh oh!

dejan2609 commented Feb 17, 2019

Uh oh!

dejan2609 commented Feb 26, 2019

Uh oh!

dejan2609 commented Feb 26, 2019

Uh oh!

dejan2609 commented Feb 26, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented Mar 2, 2019

Uh oh!

dejan2609 commented Mar 2, 2019

Uh oh!

dejan2609 commented Mar 2, 2019

Uh oh!

dejan2609 commented Mar 13, 2019

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dejan2609 commented Mar 18, 2019

Uh oh!

dejan2609 commented Mar 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dejan2609 commented Apr 5, 2019

Uh oh!

dejan2609 commented Apr 23, 2019

Uh oh!

dejan2609 commented Apr 29, 2019

Uh oh!

ijuma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ennru commented Jun 12, 2019

Uh oh!

dejan2609 commented Jun 13, 2019

Uh oh!

dejan2609 commented Jun 13, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ennru commented Jun 14, 2019

Uh oh!

dejan2609 commented Jun 14, 2019

Uh oh!

dejan2609 commented Jun 14, 2019

Uh oh!

dejan2609 commented Jun 19, 2019

Uh oh!

ijuma commented Jun 19, 2019

Uh oh!

ijuma commented Jun 19, 2019

Uh oh!

Uh oh!

Uh oh!

dejan2609 commented Jun 21, 2019

dejan2609 commented Aug 2, 2018 •

edited

Loading

ijuma commented Aug 8, 2018 •

edited

Loading

dejan2609 commented Feb 26, 2019 •

edited

Loading

dejan2609 commented Mar 20, 2019 •

edited

Loading

dejan2609 commented Jun 13, 2019 •

edited

Loading