add parallel collections module to 2.13 build and fix key projects to use it

[SethTisue]

once scala/scala#5603 is merged

as per discussion on that PR

[SethTisue]

I hit merge on 5603 just now, so I can start on this tomorrow if tonight's bootstrap job is green

[SethTisue]

Welcome to Scala 2.13.0-pre-e2a2cba (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_112).
Type in expressions for evaluation. Or try :help.

scala> List(1).par
<console>:12: error: value par is not a member of List[Int]
       List(1).par
               ^

ahhhhhhh

[SethTisue]

[scala-parallel-collections] [error] Uncaught exception when running scala.runtime.ScalaRunTimeTest: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing
[scala-parallel-collections] sbt.ForkMain$ForkError: java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing

🤔

[SethTisue]

pushed a fix for the hamcrest thing (though I don't understand it)

[SethTisue]

fixed by #501

[SethTisue]

reopening because there's still the "and fix key projects to use it" part left to do

[SethTisue]

[info] >>> The dbuild result is------------: FAILED (failed: scala-js, scala-java8-compat, fs2, scala-stm, jackson-module-scala, specs2)

37 projects remain green. scala-js and specs2 are the heavies here.

[SethTisue]

Scala.js: ticket opened at scala-js/scala-js#2811 requesting they make whatever the necessary change is

[SethTisue]

log with current set of failures: https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-integrate-community-build/283/consoleFull (2.13.0-pre-e2a2cba)

[SethTisue]

I'm having trouble seeing what the specs2 errors have to do with parallel collections. maybe there's some other cause.

the last 2.13.x run with green specs2 was 2.13.0-pre-ef63cb9 https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-integrate-community-build/275/consoleFull just a couple days ago. in that build specs2 was at 3eb0e7fa808131608d9cd139736fda6f28aaf219 which is the same SHA as run 283, so nothing changed at specs2's end

the errors look like:

[specs2] Running "compile" in: core
[specs2] [warn] Credentials file /home/jenkins/.sbt/specs2.credentials does not exist
[specs2] [warn] Credentials file /home/jenkins/.sbt/specs2.credentials does not exist
[specs2] [warn] Credentials file /home/jenkins/.sbt/specs2.credentials does not exist
[specs2] [info] Compiling 103 Scala sources to /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/specs2-14bf4f0d841299aa033a7e0ed4192277121cd44f/core/target/scala-2.13.0-pre-e2a2cba/test-classes...
[specs2] [error] /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/specs2-14bf4f0d841299aa033a7e0ed4192277121cd44f/core/src/test/scala/org/specs2/execute/ResultExecutionSpec.scala:4: object junit is not a member of package org
[specs2] [error] import org.junit.Assert
[specs2] [error]            ^

attempting to reproduce outside dbuild with:

++2.13.0-pre-e2a2cba
set every scalaBinaryVersion := "2.12"
core/test:compile

which succeeds, and note also:

specs2(master)> show core/test:libraryDependencies
[info] * org.scala-lang:scala-library:2.13.0-pre-e2a2cba
[info] * org.scala-sbt:test-interface:1.0:optional
[info] * org.mockito:mockito-core:1.9.5:test
[info] * junit:junit:4.12:test
[info] * org.spire-math:kind-projector:0.9.3:plugin->default(compile)

but tests/test:compile gives

[error] /Users/tisue/specs2/tests/src/test/scala/org/specs2/matcher/TraversableMatchersSpec.scala:7: object parallel is not a member of package collection
[error] import scala.collection.parallel.ParSeq
[error]                         ^

an error which does not occur in the Jenkins log, but that's probably because the problem in core happens first.

[SethTisue]

hmm, this looks like a clue:

[specs2] [debug] Rewriting junit:junit:4.12:test to junit:junit:1.0.0-dbuildx24701bb5daadbc815ecca5d5241ab2c5606c244b:test (against: ArtifactLocation(junit:junit:pom,1.0.0-dbuildx24701bb5daadbc815ecca5d5241ab2c5606c244b,,None) )

no such line appears in the green run at https://scala-ci.typesafe.com/view/scala-2.13.x/job/scala-2.13.x-integrate-community-build/267/consoleFull

let's go in with:

./dbuild-0.9.7-RC1/bin/dbuild -d checkout 35138e2fd430397c3bfe1f72af166ae47c51c70c specs2 newdir`
newdir/.dbuild/start
> dbuild-build
> core/test:compile

which reproduces the failure, at least (and the Rewriting junit:junit thing, but note that running dbuild with -d is needed in order to see that)

@cunei I'm totally puzzled by this. do you have any idea why dbuild would rewrite the junit dependency like this, even though it's a Java-based dependency? (and why it would only have started doing this now...?)

[SethTisue]

@cunei perhaps it's not unrelated that when I added scala-parallel-collections to the build, I had to do:

    // I have absolutely no idea why this would only be required in the                                                
    // community build context, but without it, we get                                                                 
    // java.lang.NoClassDefFoundError: org/hamcrest/SelfDescribing                                                     
    extra.commands: ["set libraryDependencies in junit += \"org.hamcrest\" % \"hamcrest-core\" % \"1.3\" % \"test\""]  

maybe somehow the same problem — a JUnit-related Java-based dependency not being found

[SethTisue]

workaround attempt "set libraryDependencies in core += \"junit\" % \"junit\" % \"4.12\" % \"test\"" did not help, dbuild just went ahead and rewrote that one too

[SethTisue]

@cunei the only other recent change I can think of that might somehow possibly be relevant is the switch in Scala version number format — used to be 2.13.0-abcd123-nightly, is now 2.13.0-pre-abcd123 (in the 2.12 series it's "bin" instead of "pre", so that could be relevant to why we're only having this problem in a 2.13 context)

[SethTisue]

OK, I finally have a solid lead on this. I think the problem is that scala-parallel-collection publishes, or dbuild thinks scala-parallel-collections publishes.... something (junit?) that causes a bunch of extra dependencies to show up:

[info] Project scala-continuations
[info]   depends on: scala, scala-parallel-collections, scalacheck
[info] Project scala-parser-combinators
[info]   depends on: scala, scala-parallel-collections, scalacheck

these dependencies should certainly not be there, as none of these projects have been modified to depend on the new module.

and in fact, removing scala-parallel-collections from community.dbuild fixes specs2.

so now it's just a matter of figuring out what's weird about scala-parallel-collections

[SethTisue]

hehehehe gotcha!

[scala-parallel-collections] [info] 	published junit to /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/scala-parallel-collections-27e1c5f46089c911086ad4f5f05e7f74278dce0c/.dbuild/local-publish-repo/junit/junit/1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.part/junit-1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.pom
[scala-parallel-collections] [info] 	published junit to /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/scala-parallel-collections-27e1c5f46089c911086ad4f5f05e7f74278dce0c/.dbuild/local-publish-repo/junit/junit/1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.part/junit-1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.jar
[scala-parallel-collections] [info] 	published junit to /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/scala-parallel-collections-27e1c5f46089c911086ad4f5f05e7f74278dce0c/.dbuild/local-publish-repo/junit/junit/1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.part/junit-1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c-sources.jar
[scala-parallel-collections] [info] 	published junit to /home/jenkins/workspace/scala-2.13.x-integrate-community-build/target-0.9.7-RC1/project-builds/scala-parallel-collections-27e1c5f46089c911086ad4f5f05e7f74278dce0c/.dbuild/local-publish-repo/junit/junit/1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c.part/junit-1.0.0-dbuildx27e1c5f46089c911086ad4f5f05e7f74278dce0c-javadoc.jar

glad that neither I nor dbuild turns out to be insane after all

[SethTisue]

fixed it over at scala/scala-parallel-collections#10

[SethTisue]

scala-js and specs2 are back in, so I'm going to close this; re-additions (and re-removals, surely) will happen gradually over the rest of the 2.13 cycle