Scala distribution replacement proposal

[adriaanm]

By @dwijnand

This proposal seeks to address the problem of new users installing Scala, either with their package manager or with the distribution archives, and getting a setup that doesn't match how the majority of the Scala community consumes and develops in Scala.

The proposed way to address this is by:

1. Removing the Scala distribution archives from the Scala website (and, therefore, from the Scala project's build and release setup and procedure docs.)
2. Ensure sbt is appropriately highlighted and documented as the primary way the Scala ecosystem consumes Scala and builds their Scala libraries and apps, with sufficient detail on how to install and get started with sbt for it to replace the Scala distribution for new users learning and starting to develop in Scala.
3. Make strides to ensure that there is a continuation path for the operating system-specific archives present in the Scala distribution, such as the deb, rpm and msi files generated.
4. Provide a replacement solution for the legitimate users of the command-line runners in the Scala distribution (scala, scalac, scaladoc, and scalap).

Note that Scala will continue to be distributed as jars to Maven Central, which is how the majority of the ecosystem consumes Scala. A side-benefit of this change would, therefore, be that it would simplify, and thus reduce the maintainance cost of, the Scala project's build and release procedure.

A secondary action that is also proposed is to:

• Somewhat promote and bring attention to Coursier and its app installing and launching capabilities, as a way to gain access to other tools in the Scala ecosystem, outside of what might already be available with their package manager, in order to promote (a) the command-line runners replacement, (b) other installable apps of primary interest for new users, such as for Creative Scala or Scala 3, perhaps.

Background

Historically the Scala distribution has been available as: (in addition to api docs and sources archives)

• a tarball (.tgz)
• a zip file
• a msi installer file
• a deb package file
• a rpm package file

Within the distribution are the Scala projects jars, and any dependencies:

• scala-library
• scala-reflect
• scala-compiler
• scalap
• jline

as well as the following runner scripts, as both bash and BAT scripts, presented here with added description of what they allow users to do from the command-line):

• scala: a multifaceted Scala runner, that allows:
  • scala Main.scala to run a Scala "script", that is compile and run a Scala source file, possibly in a synthetic main method with a synthetic args,
  • scala Main to run a Scala class with a main method,
  • scala main.jar to run a Scala jar with a Main-Class,
  • scala to run the Scala REPL,
• scalac: the (batch) Scala compiler,
• scaladoc: the Scaladoc tool,
• scalap: the Scala class file disassembler, and lastly
• fsc: the "fast" Scala compiler, which is disrecommended

with optionally the ability to define any:

• Scala compiler options,
• Java system properties,
• Java options, and
• Java classpath (hand-constructed)

Many years ago, when bootstrapping the Scala language and its subsequent ecosystem, that worked well. However, nowadays the majority of Scala users working on Scala projects will mostly never use the command-line runners or the bundled jars in the Scala distribution. A large number of Scala users will instead mostly interact with sbt, the most commonly used build tool in the Scala community, to:

• run their Scala project,
• run the Scala REPL within their project,
• resolve their project's dependencies, defining their project's classpath, and
• compile, test, generate scaladoc, package, and publish or distributed their Scala projects.

A lot of tools in the Scala ecosystem provide an integration with sbt in the form of an sbt plugin which users consume by modifying either their build definition or their user-wide sbt settings. However, not every tool in a Scala user's toolkit is strictly part their project's build. So having every Scala tool be accessed through sbt is an avoidable imposition.

Unfortunately, within the larger JVM community there hasn't been any popular, general-purpose tool for downloading, launching, or bootstrapping tools, which has lead the ecosystem to repeatedly creating and maintaining a tangle of bash, BAT scripts and installation docs, with their related system properties, environment variables, and configuration files.

Within the Scala ecosystem, over time, there have been several, disconnected, projects that seeked to address how to launch, and possibly boostrap and/or install, command-line tools in the Scala ecosystems:

• sbaz, the "Scala Bazaar System" project, a package manager for Scala installations, now dead,
• sbt/launcher, a module of sbt which, while initially built for sbt, was made to be a general-purpose Scala application jar and dependency jar downloader and launcher, but really only used for the sbt project to dynamically download and launch the version of sbt defined by the sbt.version key in project/build.properties
• Conscript, a distribution mechanism for Scala apps, built using sbt/launcher, now dead; and lastly
• Coursier, initially just a pure-Scala replacement dependency manager to sbt's use of Ivy, but which later also developed general-purpose Scala application launching, bootstrapping, and installing capabilities

Some of the most popular tools built in Scala and used by the Scala ecosystem, and how they run, are:

• the Scala runner scripts, as mentioned above, that rely on the existence of the library jars and dependencies on the filesystem,
• sbt, Scala's build tool, which actually has two runners:
  • the official sbt runner scripts in sbt-launcher-package, as both bash and BAT scripts, which noteably relies on the presence of sbt's sbt-launch.jar (which is based on sbt/launcher) on the filesystem,
  • the unofficial, but fairly used, sbt-extras runner script, as a UNIX and Cgywin bash script only, which noteably is used on Travis CI and by some Scala compiler engineers, and also noteably dynamically downloads the sbt's launch jar meaning all it needs is the runner script (and Java),
• The old "Typesafe Zinc", which provided sbt's incremental compiler module for consumption by other build systems (nowadays Zinc refers to the module itself, as an independent project),
• Ammonite, an alternative Scala REPL, which isn't based on any general purpose installers or launcher,
• Almond, which brings Scala support to Jupyter's notebooks and uses Coursier (and Ammonite),
• Scalafmt, a popular code formatter for Scala, that uses Coursier,
• Scalafix, a refactoring and linting tool for Scala, that also uses Coursier, and finally
• Bloop, a Scala build server, that also uses Coursier behind a Python script veneer
• Mill, an alternative build tool for Scala, like Ammonite doesn't use anything general purpose
• Fury, another alternative build tool for Scala, which uses a bash script veneer over Coursier,

Of additional importance sbt's launcher and sbt use a custom, non-standard classloader setup in order, for example, for the sbt launcher to be written in Scala 2.10 but run sbt 1 which is written in scala 2.12. See Jason Zaugg's notes for more details.

Notes and Design Decisions

• scala-runners is a proposed replacement for the runner scripts, based on coursier launch
• what if package maintainers just bundle scala-runners? That would undermine everything.
• ok to drop fsc?
• additional coursier app definitions?
  • scala-repl, for just the official Scala REPL? (or called scalar?)
  • creative-scala, for non-programmer beginners,
  • dotty
  • dotty-repl/dotr
  • ammonite
• the sbt runner will use the latest stable version of Coursier and sbt, at the time of the Scala distribution
• see Rustup for comparisions and inspiration on a language installer, manager and distribution
• no download, usage, "phone home" statistics gathering (see Ammonite's controversy)
• dropping debs and rpms: everything OK to let the Debian/Ubuntu and RedHat communities maintain them externally?
• drop MSI: everything OK to let the Chocolately and/or Scoop communities maintain it externally?
• drop sbt's distribution archives? what about preloaded dependencies?
• coursier-based sbt launching
• there will be overlap with OS package manager packages/formula, e.g. a separate coursier formula
• legal concerns with Lightbend or EPFL or the Apache 2 license with dropping the distribution?

Participants

Lead: Dale Wijnand (Scala contributor, sbt ex-maintainer)

I propose we invite the people we know in the community have exhibited an interest in this space, and might be interested in being involved:

• Alexandre Archambault - Coursier
• Eugene Yokota - sbt
• Jason Zaugg - sbt launcher classloader notes and ideas
• Jorge Vicente Cantero - sbt and Bloop
• Jon Pretty - Fury
• Kenji Yoshida - Conscript
• Li Haoyi - Ammonite and Mill
• Ólafur Páll Geirsson - Scalafmt and Scalafix

Let me know if you have any other suggestions.

[jvican]

Interesting! /CC @heathermiller

[antonkulaga]

I personally even do not bother installing Scala, as sbt is enough to make Scala projects work and for REPL there is ammonite-shell

[retronym]

I've cataloged a few ideas for simplifying the SBT launcher, in particular around its classloader hierarchy. I'll work through these with @eed3si9n and @dwijnand, keeping the context of consolidation in mind.

[jvican]

@retronym I am actually working on a smarter mechanism to cache classloaders for sbt plugins, it builds upon the same concepts you describe in the catalogue. Will submit PR soon.

[curoli]

I routinely download the DEB or RPM files from the website to install Scala on all of my Linux machines. Is it really much work to provide those? I created DEB files of my own apps by SBT Native Packager and it has been quick and easy.

[lrytz]

What are you using the installed Scala distribution for? To do experiments in the REPL?

I definitely see the value of packing a Scala app as deb package so that it can easily be installed. But the Scala distribution is quite limited, as it ships only with the standard library.

[curoli]

Yes, experiments in the REPL.

[dwijnand]

Hey @curoli. It's not so much work, but it is work to maintain. What we're hoping to do is pull the deb/rpm/zip/tgz distribution building out of the build and replace them with (proposed) scala-runners project, which is based on coursier's app launching capabilities). Scala Runners would be directly consumable by users, but my thinking is that it could also be used by package maintainers if they want to re-build the deb/rpm packages (also the Homebrew formula for macOS and Chocolately/Scoop packages for Windows).

[martijnhoekstra]

@curoli have you considered https://github.com/dwijnand/scala-runners for your REPL needs (if you're not on Windows?)

[martijnhoekstra]

Jinx

[dwijnand]

(if you're not on Windows?)

Windows/BAT support is a requirement for scala-runners to be a replacement (opened dwijnand/scala-runners#4)

[ritschwumm]

there's one thing the distribution gives you that none of the other ways of installation provide without glitches: you need connection to the internet exactly once to download it, and from then on it just works, regardless of whether you are connected or not.

with e.g. sbt it always feels like playing the lottery when youi try to develop on the train without connectivity: sometimes it works, sometimes it want to download new stuff and fails... those are the times where i pull our my trusty "normal" distribution and find something i can work on using that instead.

[lihaoyi-databricks]

I personally don't see the point in maintaining a "scala distribution" at all. Someone using Scala can get by just fine with SBT-only, or Mill-only, or Bazel-only, or even Intellij-only or Ammonite-only for small tasks. IMO just stopping the publishings of distribution and pointing people directly at the proper tools would be the greatest benefit for the least amount of work. Each tool already has good-enough integrations for scaladoc/scalac/console/etc.

I may be the exception in that I do not use SBT in any of my open source projects or work projects, but I do recognize it is widely used enough that for most people the SBT onboarding experience is as much a part of the Scala experience as the language itself. As long as it is the de-facto Scala experience, prominently advertising it and supporting it is the right thing to do

I like Coursier as an ivy replacement, but I don't think it's usage as an application-launcher and bootstrap system has been sufficiently proven out in the wild to label it as the "recommended" way of doing things. That opinion might change as coursier-as-a-bootstrap-launcher sees more adoption by people who aren't alex :P

[dwijnand]

there's one thing the distribution gives you that none of the other ways of installation provide without glitches: you need connection to the internet exactly once to download it, and from then on it just works, regardless of whether you are connected or not.

That use case is important, no doubt. My thoughts is scala-runners would support it and platform packages would use scala-runners to construct such "download one thing once" packages.

[dwijnand]

I like Coursier as an ivy replacement, but I don't think it's usage as an application-launcher and bootstrap system has been sufficiently proven out in the wild to label it as the "recommended" way of doing things. That opinion might change as coursier-as-a-bootstrap-launcher sees more adoption by people who aren't alex :P

Yeah, absolutely fair. I guess you can call me an early-adopter fanboy, but the viability of scala-runners depends on how well it works.

[martijnhoekstra]

Someone using Scala can get by just fine with SBT-only, or Mill-only, or Bazel-only, or even Intellij-only or Ammonite-only

I don't agree that this is a reasonable approach for quick REPL experiments.

[ashawley]

One solution isn't writing new Scala tools, but making Scala available on distribution channels that users are already using. The sbt-extras hack and the Apache Spark installation are embarrassing examples of how deficient things are. There's no reason sbt and Scala couldn't be first-class citizens on more distributions.

However, this will require the Scala community to follow the conventions and adopt the tooling of these other projects. The gap was evident recently when a Debian contributor was trying to work on getting Scala updated, but ran in to trouble.

https://contributors.scala-lang.org/t/question-about-building-scala-and-sbt-proper/2811

For example, if Scala and sbt were available on distributions like Debian or Fedora, it would likely trickle to Ubuntu and Red Hat and Amazon Linux, and therefore to the many Cloud, serverless, container orchestrated, or Continuous Integration services. The work with Scala could also pave the way for packaging other Scala tools like Coursier, mill or Spark.

Scala and sbt have long development cycles, so once the pipe is primed, things could work pretty well. As evidence, consider the Macports and Homebrew packages for sbt. They stay updated, and are very reliable in my experience.

Most free and open source projects know this, and bake in these guidelines from the beginning. Scala should, too. It doesn't help that the underlying Java ecosystem sometimes doesn't have the best story here, either. It's getting better, though. Scala should catch up, as well.

[dwijnand]

I don't agree that this is a reasonable approach for quick REPL experiments.

@martijnhoekstra I agree and it's why I built Scala Runners. Have you given it a go? It needs improvements, so I'm looking for feedback. 😄

[martijnhoekstra]

@dwijnand due to dayjob reasons, I run Windows, and after not getting it to work for, like, 5 minutes, I gave up. In theory, it's just what I want though.

@ashawley IMO a standard scala distribution for more distributions is not as useful as it sounds. You are going to want to use a build tool for anything but repl'ing anyway (and repl experiments, at least for me, are often version dependent) and the build tool will download it's own version. I'm struggling to find a use case for it.

A scala library package is even less useful IMO -- linking against a static scala library jar on a fixed classpath is just not how applications tend to run.

a scala build tool like sbt, cbt or mill is much more reasonable to want as package IMO. The bootstrap fo such packages is an issue though, at least for Debian.

[ashawley]

I don't disagree. I even mentioned sbt, mill and Coursier in my comment. Even Adriaan's original proposal mentions supporting "legitimate users of the command-line runners in the Scala". We need to provide a lot of different avenues for people to use Scala, though. We need to argue less about picking a single solution.

[dwijnand]

You want sbt consoleQuick if you don't want the project's classes.

[neontorrent]

I think the sentiment is people want more options instead of less, and a few people here including myself have expressed our use cases.

I would suggest that we provide distributions with

• just scala binaries and libs (like currently)
• toolset including scala bins, libs, sbt, etc.

and let the users decide what they want?

[som-snytt]

Just to note that users don't know what they want, they just want it to work in whatever context they happen to find themselves. I am one of those users.

Edit: to clarify, in the Steve Jobs sense that the visionary provider has to show the way to an outcome that the user could not foresee. I know what I want only in the sense that I know what I do: I download the tarballs for REPL and one-off tests and scripting, and I use sbt, mostly. But, for example, sdkman made a big difference in my workflow for jdk switching. Once upon a time, jdk switching was rare for me. sdkman also mostly works for scala. But there is surely a better way, if only I knew what it was. So I kind of know what my use cases are (but even they are ripe for disruption), but I don't have a visionary sense of the way we're all doing it wrong. Unless we go back to sbaz, which I never experienced first-hand, but I think we should migrate sbt to sbz.

[Jasper-M]

IMHO scala-runners—if it works well—seems to strike the perfect balance between having a native binary and going through sbt to launch a REPL.

[neontorrent]

Just to note that users don't know what they want, they just want it to work in whatever context they happen to find themselves. I am one of those users.

I had a different experience. I know which distribution to get and what to do with it. Other devs in my team have the same experience so I wouldn't say users don't know what they want.

IMHO scala-runners—if it works well—seems to strike the perfect balance between having a native binary and going through sbt to launch a REPL.

runner

That tool looks very nice. I am okay with it if it is the official solution. Does it have dependency management? Sorry I might have missed it because I don't see in the README

[dwijnand]

No, it doesn't. But we've started a conversation about that in dwijnand/scala-runners#3.

[eyalroth]

The proposed way to address this is by:
2. Ensure sbt is appropriately highlighted and documented as the primary way the Scala ecosystem consumes Scala and builds their Scala libraries and apps, with sufficient detail on how to install and get started with sbt for it to replace the Scala distribution for new users learning and starting to develop in Scala.

Please don't; or rather, please advocate the other tools as well.

It's been a while since this ticket was opened, and in that time there have been multiple ocassions where the question of what is Scala's primary build tool is -- or should be -- has been brought up:

1. Scala Users: Who is winning the build war with Scala So Far? (2019)
2. Scala Contributors: Why does Scala need its own build tool (SBT)? (2019)
3. @lihaoyi blog: So, what's wrong with SBT? (2017) (see comments and reddit discussion as well)
4. YCombinator discussion following "Towards Scala 3" blog post (2018)
5. Linkedin blog: Productivity at Scale: How We Improved Build Time by 400% at LinkedIn (2018)

I personally am greatly in favor of Gradle. It's more mature, more popular, better maintained, and most importantly - more familiar to Java developers, which are the potential adopters of Scala. Only downside is that it's currently using "the old Zinc", but integration with "the new Zinc" or Bloop is one of the most voted tickets in their repository and on the milestone of their next release (6.0).

[bjornregnell]

I teach Scala first at Lund University. Compiling in terminal with scalac and see that class-files are created is a second step after gentle scala REPL experiments and before using a build tool. It's good for your conceptual understanding to see what's going on with a one-file object Main and input-output of the compiler.

My students want sudo apt install scala on ubuntu (dual boot or vm with virtualbox is common) to get the REPL and scalac. And many look for scala in app stores on windows and macos and ask me why its not there...

[lrytz]

(I think) we are not suggesting to kill the scala and scalac command line tools, just change the way they work. Today you have to download the distribution (jars + scripts). In the new model, the commands would be scripts / aliases for coursier commands / coursier apps. How exactly you'd install them is to be discussed, but I think it would be easier for us / for contributors to provide a good experience across platforms than today.

[bjornregnell]

to provide a good experience across platforms than today

Yes, that would be really nice!
Something to write in terminal and then scala and scalac just works, and I would like to have sbt as well in the same "package" deal. (cf the magic brew paste which is a bit cryptic but works).
But a person using a platform expect something typical for that platform... That's why I also suggest a "scala app" in the respective main stream app store (apt+snap, ms store, apple app-store).

[Jasper-M]

I don't think you'll find many compilers or related command line tools which are available via the actual app stores.

[bjornregnell]

No, but if so it would make it easy to install'em. Programming für alle :)

[neontorrent]

No, but if so it would make it easy to install'em. Programming für alle :)

I agree. Improving distribution can help newcomers to quickly get hands on. This is definitely something we can do.

Pretty much the only dependency for Scala is JRE. Many linux distributions now have repo with OpenJDK. We can add Scala distribution to the repo and add OpenJDK as dependency.

[eyalroth]

I don't think you'll find many compilers or related command line tools which are available via the actual app stores.

I'd say that the vast majority of software is not available via the app store in the desktop world. Well, at least not on Windows.

The biggest exception that I can think of is games, which are usually available via a game store launcher (Steam, Uplay, Origin, Epic, GoG, etc), but they are an entirely different beast (paid, online multiplayer, extremely large download size, etc).

[bjornregnell]

It's the 19 year old cs students that are tomorrow's devs. Making it as easy as possible for them to start coding with the real scala tools is crucial for adoption. What is "normal" or "common" today is not what's important, but what makes it easy for tomorrow's scala devs. The IntelliJ Scala bundle is a good step, but we also need a one click experience to get the terminal things going for all newbies.

[eyalroth]

It's actually the <9 year old kids that are tomorrow's devs, and I hope their first programming experience will be much simpler than Scala and its ecosystem; be it using an online REPL or via a dedicated learning program, as it has been with Logo's turtle in the past and nowadays with more elaborate apps (I know of CodeMonkey but I'm sure there are more).

[dwijnand]

Making a disruptive change like this late in Scala 2.x and it not carrying over into Scala 3.0 is not a great outcome.

So, while the ideas here are good, it should be revisited for (perhaps/hopefully) Scala 3.1.

scala-runners still exists (and gets lots of use in the Scala Team), but also there have been lots of great improvements in Coursier (cs, cs setup, etc) as well as sbt 1.4's sbtn.

Thank you all for the comments.

[SethTisue]

I agree with Dale that this only makes sense to do if it's a coordinated change made to Scala 2 or 3 both. And it seems unlikely that there's time for big changes in this area for 3.0.

[aorinevo]

Very nice analysis!

[SethTisue]

what's happening instead is scala-cli / SIP-46 https://docs.scala-lang.org/sips/scala-cli.html