Implement SIP-51 (unfreezing the Scala library)

[lrytz]

There is an open PR for sbt to implement SIP-51 (unfreezing the Scala library). The topic was discussed at a recent Scala meeting and will be brought up in the next SIP meeting (Friday Mar 15); everyone seems to be supporting it.

Example:

➜ m cat build.sc
import mill._, scalalib._
object proj extends ScalaModule {
  def scalaVersion = "2.13.8"
  def ivyDeps = Agg(
    ivy"com.softwaremill.sttp.client3::core:3.8.3", // depends on scala-library 2.13.10, ws 1.3.10
    ivy"com.softwaremill.sttp.shared::ws:1.2.7",
  )
}
➜ m mill show proj.runClasspath
[
  "qref:v1:868554b6:/.../maven2/com/softwaremill/sttp/client3/core_2.13/3.8.3/core_2.13-3.8.3.jar",
  "qref:v1:f3ba6af6:/.../maven2/com/softwaremill/sttp/shared/ws_2.13/1.3.10/ws_2.13-1.3.10.jar",
  "qref:v1:438104da:/.../maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar",
  ...
]

ws is upgraded to 1.3.10, but scala-library remains at 2.13.8.

Constraints for Scala 2.13

In the example shown above, sbt willl fail the build and require the user to upgrade scalaVersion to 2.13.10. The reason is interactions between the compiler's compile-time and run-time classpath.

On a high level, the compiler is invoked as java -cp RUNTIME_CLASSPATH scala.tool.nsc.Main -cp COMPILETIME_CLASSPATH Source.scala. Both classpaths contain a scala-library.

• The scala-library on the runtime classpath needs to match the compiler version (also on the runtime classpath) exactly because compiler releases are built with the inliner enabled.
• The scala-library on the runtime classpath cannot be newer because
  • When running the REPL (mill console), REPL lines compiled against a newer library could fail to execute on the older library
  • Macros from the classpath compiled against a newer library could fail to expand on the older library

The implementation for Scala 2.13 in sbt is therefore

• unpin scala-library in dependency resolution
• fail the build if the Scala library is newer than the build's scalaVersion
• use coursier's SameVersion rule to ensure scala-library, scala-reflect and scala-compiler are all at the same version (https://github.com/coursier/sbt-coursier/pull/490/files)
  • this is important for projects that have a library dependency on scala-reflect, for example. Because Scala is built with the inliner, the artifacts need to be at exactly the same version.

For sbt-specific details see the individual commit messages in https://github.com/sbt/sbt/pull/7480/commits.

Scala 3

For Scala 3, unpinning scala-library and updating it in dependency resolution like other libraries is the right solution. Currently mill pins scala-library according to the Scala version:

➜ m cat build.sc
import mill._, scalalib._
object proj extends ScalaModule {
  def scalaVersion = "3.2.0" // depends on scala-library 2.13.8
  def ivyDeps = Agg(
    ivy"com.softwaremill.sttp.client3::core:3.8.3", // depends on scala-library 2.13.10
  )
}
➜ m mill show proj.runClasspath
[
  "qref:v1:feb4b39a:/.../maven2/com/softwaremill/sttp/client3/core_3/3.8.3/core_3-3.8.3.jar",
  "qref:v1:2d688157:/.../maven2/org/scala-lang/scala3-library_3/3.2.0/scala3-library_3-3.2.0.jar",
  "qref:v1:438104da:/.../maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar",
  ...
]

Just as in Scala 2, to support macros and the REPL correctly, the runtime classpath of the Scala 3 compiler needs to have the updated scala-library. A macro on the dependency classpath can be compiled against some new Scala library; expanding it in the compiler with an older Scala library can fail.

Here's a test case for this scenario (it exploits some package private addition that was done in 2.13.9, as an example of what will happen after unfreezing the standard library):

// A.scala, in project a, Scala version 3.2.2 (which depends on scala-library 2.13.10)

import scala.quoted.*
package scala.collection {
  object Exp:
    def m(i: Int) = IterableOnce.checkArraySizeWithinVMLimit(i) // added in 2.13.9
}
object Mac:
  inline def inspect(inline x: Any): Any = ${ inspectCode('x) }
  def inspectCode(x: Expr[Any])(using Quotes): Expr[Any] =
    scala.collection.Exp.m(42)
    x

// B.scala, project b, Scala version 3.2.0 (which uses scala-library 2.13.8)
@main def hubu =
  println(scala.util.Properties.versionString)
  Mac.inspect(println("hai"))

Scala 2.12

Nothing should change for projects using Scala 2.12 or older.

[lrytz]

@lefou are you interested in working on this yourself, or should I try to find someone (maybe me, though I know nothing about mill) to take a look?

[lefou]

@lrytz Good question. First, I need to make myself familiar with what this change actually means. Is there some ETA for when users expect SIP-51 to be usable?

[lrytz]

It took me a long time to get it done for sbt because I didn't anticipate all the interactions I wrote in this ticket, I was only discovering them along the way. Hopefully there should be no surprises when doing the same for mill and the actual diff should not be too big (it's not in sbt).

For timing, I don't know what is the timing for sbt 1.10.0 (which could ship the change). Even when the change is released in the build tools, nothing is going to happen until there is actually some Scala release with a forwards incompatible change in the library, which would be 2.13.15 the earliest (late summer 2024).

[lefou]

use coursier's SameVersion rule to ensure scala-library, scala-reflect and scala-compiler are all at the same version (https://github.com/coursier/sbt-coursier/pull/490/files)

• this is important for projects that have a library dependency on scala-reflect, for example. Because Scala is built with the inliner, the artifacts need to be at exactly the same version.

I wonder whether upcoming version of scala-compiler and scala-reflect should declare their dependencies on scala-library in the pom.xml with a tight version range (e.g. [2.13.14] or [2.13.14,2.13.14]) instead of just a single version (e.g. 2.13.14), which is treated like an upper unbound range. I haven't tested it yet, but coursier should then automatically detect and report mismatches according to its version reconciliation documentation.

[lefou]

So, this is interesting. I took your example project, and overrode some targets defined in ScalaModule to disable the version pinning.

import mill._
import mill.scalalib._

object proj extends ScalaModule {
  def scalaVersion = "2.13.8"
  def ivyDeps = Agg(
    ivy"com.softwaremill.sttp.client3::core:3.8.3", // depends on scala-library 2.13.10, ws 1.3.10
    ivy"com.softwaremill.sttp.shared::ws:1.2.7"
  )
  def scalaLibraryIvyDeps = Agg(
    // same as super, but without the `.force`
    ivy"org.scala-lang:scala-library:${scalaVersion()}"
  )
  def mapDependencies: Task[coursier.Dependency => coursier.Dependency] = T.task {
    // ignore super-setup
    d: coursier.Dependency => d
  }
}

With this setup I can unpin the scala-library used in the compile classpath. It's resolved to version 2.13.10, as expected, due to this version transitively depended on.

> mill show proj.compileClasspath
[
  "qref:v1:868554b6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/client3/core_2.13/3.8.3/core_2.13-3.8.3.jar",
  "qref:v1:f3ba6af6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/shared/ws_2.13/1.3.10/ws_2.13-1.3.10.jar",
  "qref:v1:a94bfc28:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar",
  "qref:v1:0c9ef1ab:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/model/core_2.13/1.5.2/core_2.13-1.5.2.jar",
  "qref:v1:9b3d3f7d:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/shared/core_2.13/1.3.10/core_2.13-1.3.10.jar",
  "ref:v0:c984eca8:/home/lefou/work/opensource/mill/scratch-3075-unfreeze-scala/proj/compile-resources"
]

The compiler is still using the correct scala-library version 2.13.8.

> mill show proj.scalaCompilerClasspath
[
  "qref:v1:8b4e78c6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.13.8/scala-compiler-2.13.8.jar",
  "qref:v1:28f86f0b:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-reflect/2.13.8/scala-reflect-2.13.8.jar",
  "qref:v1:438104da:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar",
  "qref:v1:ad06b5c9:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/jline/jline/3.21.0/jline-3.21.0.jar",
  "qref:v1:c30391bd:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/net/java/dev/jna/jna/5.9.0/jna-5.9.0.jar"
]

And here is the dependency tree:

> mill show proj.ivyDepsTree
├─ com.softwaremill.sttp.client3:core_2.13:3.8.3
│  ├─ com.softwaremill.sttp.model:core_2.13:1.5.2
│  │  └─ org.scala-lang:scala-library:2.13.8 -> 2.13.10
│  ├─ com.softwaremill.sttp.shared:core_2.13:1.3.10
│  │  └─ org.scala-lang:scala-library:2.13.9 -> 2.13.10
│  ├─ com.softwaremill.sttp.shared:ws_2.13:1.3.10
│  │  ├─ com.softwaremill.sttp.model:core_2.13:1.5.2
│  │  │  └─ org.scala-lang:scala-library:2.13.8 -> 2.13.10
│  │  ├─ com.softwaremill.sttp.shared:core_2.13:1.3.10
│  │  │  └─ org.scala-lang:scala-library:2.13.9 -> 2.13.10
│  │  └─ org.scala-lang:scala-library:2.13.9 -> 2.13.10
│  └─ org.scala-lang:scala-library:2.13.10
├─ com.softwaremill.sttp.shared:ws_2.13:1.2.7 -> 1.3.10 (possible incompatibility)
│  ├─ com.softwaremill.sttp.model:core_2.13:1.5.2
│  │  └─ org.scala-lang:scala-library:2.13.8 -> 2.13.10
│  ├─ com.softwaremill.sttp.shared:core_2.13:1.3.10
│  │  └─ org.scala-lang:scala-library:2.13.9 -> 2.13.10
│  └─ org.scala-lang:scala-library:2.13.9 -> 2.13.10
└─ org.scala-lang:scala-library:2.13.8 -> 2.13.10

This all looks good to me. But I'm not 100% sure since you also wrote:

The implementation for Scala 2.13 in sbt is therefore

• fail the build if the Scala library is newer than the build's scalaVersion

Is this sbt specific or should I enforce the use of the 2.13.8 version on the compile classpath and fail the build, unless the user changes scalaVersion to a compatible version?

So, the interesting part starts, if I try to use a version range to let coursier 2.1.9 resolve either a consistent classpath, or fail.

   def scalaLibraryIvyDeps = Agg(
-    ivy"org.scala-lang:scala-library:${scalaVersion()}"
+    ivy"org.scala-lang:scala-library:[${scalaVersion()}]"
   )

My expectation is, that it fails, since the resolved version range is [2.13.8] but the transitive dependency on 2.13.10is outside this range.

Instead, it resolves the following compile classpath:

>  mill show proj.compileClasspath
[
  "qref:v1:868554b6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/client3/core_2.13/3.8.3/core_2.13-3.8.3.jar",
  "qref:v1:f3ba6af6:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/shared/ws_2.13/1.3.10/ws_2.13-1.3.10.jar",
  "qref:v1:438104da:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar",
  "qref:v1:a94bfc28:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar",
  "qref:v1:0c9ef1ab:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/model/core_2.13/1.5.2/core_2.13-1.5.2.jar",
  "qref:v1:9b3d3f7d:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/com/softwaremill/sttp/shared/core_2.13/1.3.10/core_2.13-1.3.10.jar",
  "qref:v1:05364166:/home/lefou/.cache/coursier/v1/https/repo1.maven.org/maven2/org/scala-lang/scala-library/2.13.9/scala-library-2.13.9.jar",
  "ref:v0:c984eca8:/home/lefou/work/opensource/mill/scratch-3075-unfreeze-scala/proj/compile-resources"
]

Instead of a proper resolution or an failure I now see a classpath with the same artifact but two different version. I need to inspect if this comes from Mill or if it's an issue in coursier. It's definitely not correct.

[lefou]

Instead of a proper resolution or an failure I now see a classpath with the same artifact but two different version. I need to inspect if this comes from Mill or if it's an issue in coursier. It's definitely not correct.

@alexarchambault Do you see anything obvious? To me, it looks like an issue in coursier.

We effectively call coursier API with the following parameters:

val rootDeps = List(
  Dependency(
    module = Module(
      organization = Organization(value = "com.softwaremill.sttp.client3"),
      name = ModuleName(value = "core_2.13"),
    ),
    version = "3.8.3"
  ),
  Dependency(
    module = Module(
      organization = Organization(value = "com.softwaremill.sttp.shared"),
      name = ModuleName(value = "ws_2.13"),
    ),
    version = "1.2.7"
  ),
  Dependency(
    module = Module(
      organization = Organization(value = "org.scala-lang"),
      name = ModuleName(value = "scala-library"),
    ),
    version = "[2.13.8]",
  )
)
val forced = Map()

coursier.Resolution()
 .withRootDependencies(rootDeps)
 .withForceVersions(forced)

[lrytz]

whether upcoming version of scala-compiler and scala-reflect should declare their dependencies on scala-library in the pom.xml with a tight version range

That's an interesting, I wasn't aware (https://maven.apache.org/pom.html#dependency-version-requirement-specification).

We'd have to test it and see what happens. For example in a project that has a dependency on scala-reflect:2.13.8 and some other library (sttp.client3::core:3.8.3) that depends on scala-library:2.13.10. It would be wrong to get scala-library:2.13.8, instead scala-reflect would have to be upgraded to 2.13.10.

[lefou]

That's an interesting, I wasn't aware (https://maven.apache.org/pom.html#dependency-version-requirement-specification).

Yeah, since version ranges in Maven never worked that well in older Maven versions (it wasn't able to correctly find good version candidates IIRC), most users found other ways to avoid them, and I think using <properties> to hold the versions and import-scoped dependencies aka BOM-imports is the latest incantation to let them believe they manage it.

[lrytz]

The implementation for Scala 2.13 in sbt is therefore

• fail the build if the Scala library is newer than the build's scalaVersion

Is this sbt specific or should I enforce the use of the 2.13.8 version on the compile classpath and fail the build, unless the user changes scalaVersion to a compatible version?

I think unpinning the scala-library as you did in your experiment is good, it removes a special case.

But then mill should also fail the build if the scala-library on the compiletime classpath is newer than the scala compiler (in the case of Scala 2 only).

Here's an example that simulates what could happen. opportunistic was added in 2.13.4.

// A.scala, in a module using 2.13.4

import scala.language.reflectiveCalls
object AMacro {
  import scala.language.experimental.macros
  import scala.reflect.macros.blackbox.Context
  def m(x: Int): Int = macro impl
  def impl(c: Context)(x: c.Expr[Int]): c.Expr[Int] = {
    import c.universe._
    val ec = (scala.concurrent.ExecutionContext: {def opportunistic: scala.concurrent.ExecutionContextExecutor}).opportunistic
    println(ec)
    c.Expr(q"2 + $x")
  }
}


// B.scala, in a module using 2.13.1

object B extends App {
  println(AMacro.m(33))
  println(scala.util.Properties.versionString)
}

Expanding the macro in the 2.13.1 compiler will crash.

[lefou]

Is it

1. "The compiler needs to be of the exact same version as the library it targets" or
2. "The compiler need to be of at least the same version as the library it targets, or newer"?

[lrytz]

The compiler could be newer, though in reality this probably won't happen, as setting scalaVersion X adds a dependency to scala-library:X.

[lefou]

So, if we need to align the resolved and the used scala versions, my preferred implementation would be to use coursier to detect any mismatch. We can catch and recognize the exception and provide a more actionable error message, possibly also a copy-and-paste-able fix.

But for this, we need to find and fix the root cause for the incorrect resolution I posted above.

[lefou]

cc @lolgab, who has more profound knowledge of our Scala.JS and Scala Native implementations.

[lrytz]

In my simple mental model coursier doesn't know about the compiler's compile-time and run-time classpath. IIUC, the build tool invokes coursier's dependency resolution twice:

• once to get a classpath for scala-compiler:scalaVersion, the build tool will use this as the compiler's runtime classpath
• then to get a classpath for the dependencies, which will be the compiletime classpath

Coursier is just answering these two independent requests. Isn't that how it works?

[lefou]

In my simple mental model coursier doesn't know about the compiler's compile-time and run-time classpath. IIUC, the build tool invokes coursier's dependency resolution twice:

It's exactly as you said!

• once to get a classpath for scala-compiler:scalaVersion, the build tool will use this as the compiler's runtime classpath

This is

> mill show proj.scalaCompilerClasspath

• then to get a classpath for the dependencies, which will be the compiletime classpath

This is

> mill show proj.resolvedIvyDeps

Coursier is just answering these two independent requests. Isn't that how it works?

Sure. But IIUC, you said we need to align the version of the used compiler (e.g. 2.13.8, you called it RUNTIME_CLASSPATH, in Mill it's the scalaCompilerClasspath) with the one used when compiling, which might depend on the transitive dependencies (e.g. 2.13.10, you called it COMPILE_CLASSPATH, in Mill it's resolvedIvyDeps). So, since 2.13.8 is not equal to 2.13.10, we should fail the build asking the user to update to 2.13.10 (or newer).

As you said, we add the scala-library to the classpath with the same version as the compiler. When we pinned them, this was enforced, without any consistency checks. Now, with removed pinning, the effective version can be raised by coursier, due to transitively higher lower bounds, which is the case we want to detect and then fail. Hence, my idea to use a version range in the compile classpath with the exact Scala version we need to comply to.

In Mill jargon this means effectively:

  def ivyDeps = Agg(
    ivy"org.scala-lang:scala-library:[2.13.8],
    ivy"com.softwaremill.sttp.client3::core:3.8.3", // depends on scala-library 2.13.10, ws 1.3.10
    ivy"com.softwaremill.sttp.shared::ws:1.2.7"
  )

In therory, this should force coursier to either resolve to 2.13.8 or fail. It can't resolve to 2.13.8 due to the transitive dependency on (at least) 2.13.10, which is outside the given interval. Coursier should fail, but it doesn't. Instead, it puts both version on the classpath, which is definitely not correct.

>  mill show proj.resolvedIvyDeps
...
  "qref:v1:438104da:.../org/scala-lang/scala-library/2.13.8/scala-library-2.13.8.jar",
  "qref:v1:a94bfc28:.../org/scala-lang/scala-library/2.13.10/scala-library-2.13.10.jar",
...