Skip to content

add a Java 9 community build #609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
SethTisue opened this issue Sep 24, 2017 · 64 comments
Closed

add a Java 9 community build #609

SethTisue opened this issue Sep 24, 2017 · 64 comments
Assignees
@retronym @SethTisue

Comments

@SethTisue
Copy link
Member

SethTisue commented Sep 24, 2017 •
edited
Loading

starting, I suppose, with just a stripped-down thing that only attempts to build some basic stuff like the Scala standard modules, then expanding from there

Scala 2.12 only is fine for now, I think. the behemoths are already somewhat pressed for space (and the space can't be increased without rebuilding the nodes), and even just adding the 2.12/jdk9 combo will make the problem, one might say, worse enough

plan:

  • get it running locally on my Mac
  • get it running on Jenkins

re: Jenkins, @retronym did some work, prior to JDK 9 going final/gold/GA, on making it available on our Jenkins nodes; see scala/scala-jenkins-infra#174 for details
@SethTisue SethTisue self-assigned this Sep 24, 2017
@SethTisue
Copy link
Member Author

SethTisue commented Sep 25, 2017 •
edited
Loading

my experience attempting this locally was that all extractions failed with [scalacheck] [error] scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.

but then I made an empty sbt project and ran it with Java 9, and sbt/sbt-launcher-package#143 kicked in and showed the "Copying runtime jar." message. I thought then running dbuild again might work, but same error

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

@SethTisue
Copy link
Member Author

maybe I also need to do something to make sure -Dscala.ext.dirs=... gets set?

helps, but is not sufficient. adding extra.options: ["-Dscala.ext.dirs=/Users/tisue/.sbt/0.13/java9-rt-ext-oracle_corporation_9"] to every projects allows extraction to proceed farther, but then eventually every projects gets an error like:

[scala-parser-combinators] java.lang.AssertionError: assertion failed: not a type: symbol
[scala-parser-combinators]  loaded from class ScalaModulePlugin$ in package with name ScalaModulePlugin$ and classloader sbt.PluginManagement$PluginClassLoader@9b8d3db
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror.scala$reflect$runtime$JavaMirrors$JavaMirror$$classToScala1(JavaMirrors.scala:979)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)
[scala-parser-combinators] 	at scala.reflect.runtime.JavaMirrors$JavaMirror$$anonfun$classToScala$1.apply(JavaMirrors.scala:938)

@SethTisue
Copy link
Member Author

SethTisue commented Sep 25, 2017 •
edited
Loading

an old (2014) ticket on Java 9 support dbuild is lightbend-labs/dbuild#155

@retronym
Copy link
Member

That might be scala/scala-dev#304. The only workaround I know it to refactor the .scala files in the build to avoid using the empty package.

@SethTisue
Copy link
Member Author

SethTisue commented Sep 25, 2017 •
edited
Loading

java.lang.AssertionError: assertion failed: not a type: symbol

this is reproducible outside dbuild simply by switching to Java 9 and then firing up sbt in e.g. the scala-xml or scala-parser-combinators repo

two repos that don't have this problem: utest, scalacheck

as an experiment, I went into scala-parser-combinators and ripped out sbt-scala-module, after which scala-parser-combinatorsJVM/test passed. so the problem is with that plugin somehow

@SethTisue
Copy link
Member Author

the problem is with that plugin somehow

okay, so let's work on that over at scala/sbt-scala-module#33, then come back here later

@SethTisue
Copy link
Member Author

SethTisue commented Sep 25, 2017 •
edited
Loading

a "community build" consisting only of scalacheck (with scala and scala-xml binaries retrieved via aether), at https://github.com/SethTisue/community-builds/blob/wip-2.12.x-jdk9/configs/community.dbuild, is passing locally with the extra.options: ["-Dscala.ext.dirs=... workaround 🎉

so it looks like dbuild works on Java 9, which is good news, and we'll just need to expand the build a step at a time from here (and work around the scala.ext.dirs thing in a more elegant way)

@SethTisue SethTisue mentioned this issue Sep 25, 2017
Closed
@SethTisue
Copy link
Member Author

SethTisue commented Sep 25, 2017 •
edited
Loading

we'll just need to expand the build a step at a time from here

questioning, all the while, how ambitious it's appropriate to get with this how soon! I could see this staying pretty minimal for a long time

until at least:

  • community projects have added Java 9 to their own CI matrices
    • which will probably be happening pretty rapidly over the next year or so... but in the meantime we don't need to get out ahead of the community on this

and probably not even until either or both of these stronger conditions are met:

  • we start work on a version of Scala which requires Java 9
  • significant community projects start requiring Java 9
    • this is what prompted the creation of the scala-2.11.x-jdk8 build — significant libraries started dropping support for Java 6 and/or 7

@retronym
Copy link
Member

retronym commented Sep 25, 2017 •
edited
Loading

Are we able to set the Java version individually per-project? That would allow us more flexibility: we could build problematic projects on Java 8 and run compliant ones on 9.

@cunei
Copy link

cunei commented Sep 25, 2017

Not at the moment; dbuild uses whatever version of Java is in the environment, and makes no attempt to control it.

@SethTisue
Copy link
Member Author

Akka has announced Java 9 support (as of Akka 2.5.6). I'll make Akka a priority next time I do some work on this

@SethTisue SethTisue mentioned this issue Oct 5, 2017
Closed
@SethTisue
Copy link
Member Author

SethTisue commented Oct 17, 2017 •
edited by dwijnand
Loading

the new build has grown from one to five projects.

they are: scala-xml, plus

[info] Project scala-js----------------: SUCCESS (project rebuilt ok)
[info] Project scala-parser-combinators: SUCCESS (unchanged, not rebuilt)
[info] Project scalacheck--------------: SUCCESS (unchanged, not rebuilt)
[info] Project utest-------------------: SUCCESS (project rebuilt ok)

how I got there:

I rebased Jason's backport PR onto 2.10.x (which has other JDK9 stuff merged on it that Jason's PR doesn't include yet)

after much trial and error, my efforts to get dbuild+sbt to use a Scala version with "SNAPSHOT" and/or a SHA in the version number failed, so I publish a 2.10.7 snapshot as 2.10.7 to ~/.m2 locally (no -SNAPSHOT or SHA in the version):

brew install [email protected]
PATH=/usr/local/Cellar/[email protected]/1.9.9/bin:$PATH
export JAVA_HOME=$(/usr/libexec/java_home -v 1.6)
ANT_OPTS="-Xmx2048M -Xss4M -XX:MaxPermSize=256M" ant -Dbuild.release=true fastdist distpack
cd dists/maven/2.10.7
ant deploy.local

and then in my repositories file put:

  maven-local-alt: file:///Users/tisue/.m2/repository/

since as @dwijnand discovered, sbt is hardwired to ignore a normal maven-local entry when resolving Scala snapshots specifically (?!)

the config file changes necessary are in the wip branch in my fork

I'm also not going to bother trying to run this on Jenkins until Scala 2.10.7 and sbt 0.13.17 are published — I think it would be wasted work, it's enough if I just run it locally for now.

@SethTisue
Copy link
Member Author

@retronym might pick this up and run with it for a while. thus, brain dump:

I have pushed my wip fumblings on this to https://github.com/SethTisue/community-builds/commits/wip-2.12.x-jdk9

to get it up on Jenkins, some work on the scala-jenkins-infra repository will be needed:

  • the new build should go on behemoth 1; behemoth 2 already runs out of space a lot, behemoth 1 is much less prone to that. (we have so far avoided just cranking up the disk space on the boxes because doing so requires re-creating them, which Chef is only some help with)
  • an appropriate JDK needs to be available. Jason already did some work to make JDK 9 available there, I think, but it might be out of date? ideally we go through Chef to get stuff installed on our Jenkins nodes, but if doing so is just too much hassle, we're not above just manually installing something; I think we have done that there in the past with JDKs. @adriaanm might be more aware than I am of where we currently stand on that.

adding more Scala modules to the build would require doing something about scala/sbt-scala-module#35, iirc

besides the remaining Scala modules, obvious next additions could include ScalaTest and akka-actor

@SethTisue SethTisue mentioned this issue Oct 23, 2017
Closed
@SethTisue
Copy link
Member Author

sbt 0.13.17-RC2 and 1.1.0 exist now and should make this a bit easier

@SethTisue
Copy link
Member Author

SethTisue commented Mar 2, 2018 •
edited
Loading

what are those great crashing footfalls I hear? it's our THREE mighty behemoths, LARGER than the old ones. what a thrilling and awe-inspiring sight they are! and apparently they have JDK 9 on them already 🤖🤖🤖 💪💪💪 ⓽⓽⓽ 💥💥💥

I created https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build via the Jenkins UI by copying the regular 2.12.x job, adding -jdk9- to the name, and changing the JVM version to 9, and a run is in progress here: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/1/consoleFull

note that's a run of the regular 2.12.x branch of this repo. thought I'd see how that does first. we can also try the wip-2.12.x-jdk9 branch — some of the changes there shouldn't be needed anymore, maybe all of them.

@ashawley
Copy link
Member

It failed. Didn't look like there were any more builds kicked off. Did this end up getting hung up?

@SethTisue
Copy link
Member Author

SethTisue commented Mar 22, 2018 •
edited
Loading

work on 2.12.5 (and the usual thousand other things) interceded. ironically, since this would have caught the big regression in 2.12.5. sigh

yeah so in the first run a bunch of projects failed to even extract, I looked at a sampling and it was usually (always?) because something in the build choked on the unexpected Java version number

so I'll need to start just marching down the list and making pull requests or issues in those projects, starting with the ones with the most downstream dependents

I kicked off a new run just now at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/2/console , perhaps some of the included projects have already been fixed

@ashawley
Copy link
Member

it was usually (always?) because something in the build choked on the unexpected Java version number

Oh, ok. Yeah, that's brutal.

@retronym
Copy link
Member

retronym commented Mar 22, 2018 •
edited
Loading 

[argonaut-shapeless] [error] (coursierResolutions) java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class coursier/shaded/fastparse/core/LowestPriSequencer. Method $anonfun$Sequencer1$1(Ljava/lang/Object;Ljava/lang/Object;)Lscala/Tuple2; at index 39 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
[argonaut-shapeless] [error] (coursierSbtClassifiersResolution) java.lang.IncompatibleClassChangeError: Inconsistent constant pool data in classfile for class coursier/shaded/fastparse/core/LowestPriSequencer. Method $anonfun$Sequencer1$1(Ljava/lang/Object;Ljava/lang/Object;)Lscala/Tuple2; at index 39 is CONSTANT_MethodRef and should be CONSTANT_InterfaceMethodRef
[argonaut-shapeless] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore?

I know this problem, it requires an updated shade plugin, as in playframework/play-ws#173. I'll take this up with coursier and argonaut-shapeless.

Issue: coursier/coursier#822

Workaround: #700

@retronym retronym mentioned this issue Mar 22, 2018
Closed
@retronym
Copy link
Member 

[paradox] [info] 'compiler-interface' not yet compiled for Scala 2.10.6. Compiling...
[paradox:error] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jenkins/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.6.jar(scala/package.class)
[paradox:error] error: error while loading package, Missing dependency 'object java.lang.Object in compiler mirror', required by /home/jenkins/.ivy2/cache/org.scala-lang/scala-library/jars/scala-library-2.10.6.jar(scala/runtime/package.class)
[paradox:error] error: scala.reflect.internal.MissingRequirementError: object java.lang.Object in compiler mirror not found.
[paradox:error] 	at scala.reflect.internal.MissingRequirementError$.signal(MissingRequirementError.scala:16)
[paradox:error] 	at scala.reflect.internal.MissingRequirementError$.notFound(MissingRequirementError.scala:17)
[paradox:error] 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:48)
[paradox:error] 	at scala.reflect.internal.Mirrors$RootsBase.getModuleOrClass(Mirrors.scala:40)

This probably just means that project is building with SBT 0.13.16 or lower, which is yet to update to Scala 2.10.7 to get support of for Java 9.

#701

@SethTisue
Copy link
Member Author

SethTisue commented Mar 23, 2018 •
edited
Loading

This probably just means that project is building with SBT 0.13.16 or lower

that's odd — we override the sbt versions of all included projects (to 0.13.17 and 1.1.1, currently)

@ashawley
Copy link
Member

ashawley commented May 4, 2018

I've opened a PR for ScalaTest:

scalatest/scalatest#1340

@SethTisue
Copy link
Member Author

this is still a back-burner task, but, to at least make sure we're not slipping backwards, I merged current 2.12.x onto https://github.com/SethTisue/community-builds/commits/wip-2.12.x-jdk9 and made sure it still works:

[info] Project cloc-plugin-----------------: SUCCESS (unchanged, not rebuilt)
[info] Project scala-----------------------: SUCCESS (unchanged, not rebuilt)
[info] Project scala-js--------------------: SUCCESS (project rebuilt ok)
[info] Project scala-js-stubs--------------: SUCCESS (project rebuilt ok)
[info] Project scala-parser-combinators----: SUCCESS (unchanged, not rebuilt)
[info] Project scalacheck------------------: SUCCESS (unchanged, not rebuilt)
[info] Project utest-----------------------: SUCCESS (project rebuilt ok)
[info] >>> The dbuild result is----------------: SUCCESS (project rebuilt ok)

as for the full build, I've set a personal to-do item to make gradual progress on at least getting all of the projects extracting. today's increment of progress on that is here: scala/scala-swing#78

@SethTisue
Copy link
Member Author

it didn't, and I couldn't reproduce it outside dbuild, so maybe it's choking on a dbuild-mangled version number

@SethTisue
Copy link
Member Author

gah, of course, it's scala/sbt-scala-module#35 again, which has been pending so long I forgot about it

@SethTisue SethTisue mentioned this issue May 18, 2018
Merged
@SethTisue
Copy link
Member Author

scala/scala-xml#220 should do it

@SethTisue
Copy link
Member Author

that worked. stuff is building now. let's see how far it gets: https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/115/consoleFull

@SethTisue
Copy link
Member Author

SethTisue commented May 19, 2018 •
edited
Loading

sigh, for some reason slick/slick#1908 brings the entire build to a halt

@SethTisue
Copy link
Member Author

do the best you can, run 117!

@hvesalai
Copy link

You need to add --add-modules java.xml.bind -XX:+IgnoreUnrecognizedVMOptions to JVM parameters

@SethTisue
Copy link
Member Author

SethTisue commented May 19, 2018 •
edited
Loading

thanks, that does it (--add-modules=java.xml.bind). next up: run 118

@SethTisue
Copy link
Member Author

SethTisue commented May 19, 2018 •
edited
Loading

latest is run 121

in my local testing, we're now very close to being able to declare victory (as in, enough green projects, not all of them), but there is one more keystone project that is failing and taking out a big swath of the build, namely specs2. I've filed etorreborre/specs2#659 on it

@SethTisue
Copy link
Member Author

oh and there's one other thing I need to fix, which is that I pushed a "fix" for Scala.js that hardcodes /Users/tisue which obviously doesn't work on Jenkins. I think we shouldn't need to be setting scala.ext.dirs explicitly like this, it should be set for us by the sbt launch script, except I guess on Jenkins we have an outdated one.

locally, with Homebrew-installed sbt, if I run sbt on Java 9 and do eval System.getProperty("scala.ext.dirs") I get a path, but if I grab /usr/local/bin/sbt off of Jenkins (it's the sbt-extras one, not sure what version) and run that locally, scala.ext.dirs is null

@SethTisue
Copy link
Member Author

specs2 needs scala-js-stubs, which I haven't fixed yet (still the highest priority needed fix)

scala-collection-compat and sourcecode were both failing because of a too-old sbt-osgi version; I've pushed fixes for both

@SethTisue
Copy link
Member Author

I pushed a scala-js-stubs fix. https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/182/consoleFull has scala-js-stubs green. the rest is still running, I'll swing back by tomorrow and see how it did.

@SethTisue
Copy link
Member Author

119 projects are green now! declaring victory.

@SethTisue
Copy link
Member Author

@SethTisue
Copy link
Member Author

@SethTisue
Copy link
Member Author

@SethTisue
Copy link
Member Author

@SethTisue
Copy link
Member Author

status at https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/274/ : [info] >>> The dbuild result is----------------: FAILED (failed: paradox, akka-actor, slick, jackson-module-scala, http4s-websocket, fs2, kxbmap-configs, zinc)

@SethTisue
Copy link
Member Author

141 green projects now! https://scala-ci.typesafe.com/view/scala-2.12.x/job/scala-2.12.x-jdk9-integrate-community-build/310/console has failed: paradox, slick, scalikejdbc, kxbmap-configs, specs2-more, fs2, github4s, pureconfig, http4s-websocket, scala-debugger, twitter-util, zinc, akka-stream

@SethTisue
Copy link
Member Author

someone want to look into the specs2-more and zinc failures...? or any of the others?

@SethTisue SethTisue mentioned this issue Jul 31, 2018
Closed
@SethTisue
Copy link
Member Author

@SethTisue
Copy link
Member Author

the expected-to-fail list is now kept in this repo itself, here: https://github.com/scala/community-builds/blob/2.12.x/report/Report.scala#L38

@SethTisue
Copy link
Member Author

PR to fix blaze: http4s/blaze#213

we are using a frozen blaze so we won't immediately get the fix once this is merged. but perhaps http4s has moved on and could handle blaze master now, I should try it and see

@SethTisue
Copy link
Member Author

ticket on zinc failure: sbt/zinc#567

milessabin pushed a commit to milessabin/shapeless that referenced this issue Aug 27, 2018
@SethTisue @milessabin
upgrade sbt-api-mappings so build works on JDK 9
f79386f 
this came up in the context of the Scala community build,
at scala/community-build#609
@SethTisue
Copy link
Member Author

ticket on boopickle failure:
suzaku-io/boopickle#105

@SethTisue
Copy link
Member Author

perhaps http4s has moved on and could handle blaze master now, I should try it and see

nope, we had to keep blaze frozen for now. I could re-fork it and cherry-pick
http4s/blaze#213

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@retronym retronym

@SethTisue SethTisue

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@retronym @SethTisue @ashawley @cunei @hvesalai