Incremental compilation always recompiles all

[andreak]

I always get this:

Pruning sources from previous analysis, due to incompatible CompileSetup.

and a full recompile.

My pom.xml is:

<plugin>
	<groupId>org.apache.maven.plugins</groupId>
	<artifactId>maven-compiler-plugin</artifactId>
	<version>3.8.0</version>
	<configuration>
		<source>${build.jdk.version}</source>
		<target>${build.jdk.version}</target>
		<encoding>UTF-8</encoding>
		<compilerArgs>
			<arg>--add-modules</arg><arg>ALL-SYSTEM</arg>
		</compilerArgs>
	</configuration>
</plugin>

<plugin>
	<groupId>net.alchim31.maven</groupId>
	<artifactId>scala-maven-plugin</artifactId>
	<version>${version.scala-maven-plugin}</version>
	<configuration>
		<compilerPlugins>
			<compilerPlugin>
				<groupId>com.artima.supersafe</groupId>
				<artifactId>supersafe_${version.scalalibs}</artifactId>
				<version>${version.supersafe}</version>
			</compilerPlugin>
		</compilerPlugins>
		<jvmArgs>
			<jvmArg>-Xms64m</jvmArg>
			<jvmArg>-Xss4m</jvmArg>
			<jvmArg>-Xmx2g</jvmArg>
		</jvmArgs>
		<args>
			<arg>-release</arg>
			<arg>${build.jdk.version}</arg>
		</args>
		<scalaVersion>${version.scalalibs}</scalaVersion>
		<recompileMode>incremental</recompileMode>
	</configuration>
</plugin>

version.scalalibs=2.12.7
build.jdk.version=10
version.supersafe=1.1.7
version.scala-maven-plugin=3.4.4

Any hints?

[davidB]

from gatling/gatling#3245

sbt misleadingly issue a WARN message every time it has to full compile, eg on initial run.

Sorry, but there are not support on incremental Mode which use an old zinc/sbt fragment (TBH, I planned to removed it).

[andreak]

No incremental mode?? That's very sad...

[davidB]

• I never use it (it was a contribution from typesafe)
• current mode generate some annoying bug
• nobody want to support it (require to upgrade to new version of zinc)
• some alternatives maven plugin (see README) provide more up-to-date intregration

I keep it for existing user.

[davidB]

see #208

[slandelle]

No incremental mode

IMHO, this feature is poorly named: it's not about incremental compilation, which is perfectly feasible with simple zinc as it can persist analysis between runs.
This feature is actually about nailgun: connecting to a compilation server process.

nobody want to support it (require to upgrade to new version of zinc)

As I successfully upgraded Gatling compiler module to Zinc 1 for Gatling 3, I gave a shot at trying to contribute a zinc upgrade a few months ago.
I gave up exactly because of this nailgun support:

• Zinc is utterly not documented. I was able to figure out how to upgrade to Zinc 1 for basic usage, but I have no idea what to do about
• I personally really don't need it (nor am convinced it's really helpful) so I can't waste days exploring those uncharted lands...

TBH, I planned to removed it

+1000
If you were to drop it, I'd happily contribute a Zinc upgrade.

some alternatives maven plugin (see README) provide more up-to-date intregration

Nah. This other plugin is not stable and there hasn't been activity there for 6 months (not blaming the author, just stating).

[davidB]

@slandelle,

TBH, I planned to removed it

I'll probably not drop zinc.

I'd happily contribute a Zinc upgrade.

Feel free to contribute (or not). TBH I waited from your feedback after release of zinc 1.0 ;-) when I saw some of your comments about it on other project, and I also noticed the lake of documentation and usable sample.

Nah. This other plugin is not stable and there hasn't been activity there for 6 months (not blaming the author, just stating).

scala-maven-plugin has no real activity since a long time, and often I didn't see issue notification. I'm surprise by the usage of this plugin in an ecosystem where sbt is the defacto build tool.

I used maven only for CI or clean build, (IDE for incremental). At work we use sbt for scala project, At Home, currently I mainly code in Rust.

[slandelle]

scala-maven-plugin has no real activity since a long time

It does the job very well for standard usage.

I'm surprise by the usage of this plugin in an ecosystem where sbt is the defacto build tool.

Well, for me, your plugin is irreplaceable. With Gatling, we have more Java users than Scala ones and we'll never want to force sbt complexity onto our users.

At work we use sbt for scala project

Ah, ah, I honestly thought you were doing Scala anymore :P

[jvican]

This feature is actually about nailgun: connecting to a compilation server process.

Bloop is a compilation server that we officially maintain at the Scala Center and that currently has support for Maven, Gradle, sbt and mill. The goal is to provide the fastest developer workflow for all Scala users, no matter what build tool they use. You can check the project here https://github.com/scalacenter/bloop and the docs https://scalacenter.github.io/bloop/. Bloop is the recommended tool to use to compile Scala in build tools.

I've never liked the idea that we're developing our own maven plugin, but it's been good for development as we've been able to move really fast and experiment. Internally, we depend on internals of scala-maven-plugin to be able to reuse as much code as possible from this project, as well as it's configuration (any project depending on this plugin should be able to add maven-bloop and have everything working). There are some rough edges (even though we have some happy maven users) so I would love help improving the maven-side of things in bloop to get a faster compilation for Maven users.

This being said, I would like to find a solution that allows us to replace the implementation of compile, test and run in Maven while reusing as much code from this plugin and the excellent community around it. I think it would be ideal if this plugin can depend on bloop directly and compile/test/run in an external server. This will simplify maintenance for the maintainers of this plugin (no more direct use of zinc and its intricate APIs) and make sure compilation is as fast as possible for Maven users.

I have some ideas to get to this ideal scenario, but I cannot lead this effort because I'm not a Maven user. However, I'm happy to help from the bloop side of the things (I'm working full-time on this project), all I would need is someone that can make some small changes in this plugin to fully integrate with bloop.

Would anyone in this thread be interested in that? Maybe @slandelle?

[andreak]

FWIW; It would be really nice to get incremental compilation of Scala-projects using Maven. We really depend on Maven at work and use many maven-plugins for which there really aren't any good Sbt-replacements. If Bloop can help with that I'm crossing my fingers....

[slandelle]

@jvican

Would anyone in this thread be interested in that? Maybe @slandelle?

Sorry, but I definitively will never have the cycles to work on this, even less become an active maintainer.

I feel like there's some confusion between 2 topics here.

1. Incremental compilation

This can be perfectly achieved with plain Zinc. There's no need for a compile server for his. If Bloop can also work in embedded mode and is better documented than Zinc, fine.

2. Remote server compilation

Lightbend contributed a nailgun support a few years ago, but now there's no one there to maintain it. IMHO, no one uses it. Making things worse: it's what stops upgrading Zinc and getting proper incremental compilation.
Such contributions are poisonous. Maintainers can't count on contributors to help maintaining.

I perfectly see how a compile server helps with long lived client processes such as IDEs and sbt in interactive mode. That's just my opinion, but I really fail to see how regular maven users would accept to install and start a side compile server service.

IMHO, as long as no one is willing to step up and maintain this plugin actively and on the long term, focus should be on core features, possibly reducing scope (dropping nailgun right away).

[jvican]

Agreed some comments here hint at a conflation of both incremental compilation and an external build server (remote server compilation is a misnomer).

Lightbend contributed a nailgun support a few years ago, but now there's no one there to maintain it.

You could see bloop as the successor of that old nailgun integration. Bloop runs in a nailgun server and supports nailgun clients.

That's just my opinion, but I really fail to see how regular maven users would accept to install and start a side compile server service.

If this process is completely automatic to the users, they won't care. For example, Gradle spawns a background process in the background and then the gradle client talks to it. Such mechanism is built-in, transparent to the user and helps improve the performance of the build. I have in mind doing something similar for Maven but only applied to Scala compilation.

IMHO, as long as no one is willing to step up and maintain this plugin actively and on the long term, focus should be on core features, possibly reducing scope (dropping nailgun right away).

The current maven-bloop plugin does all this plugin does but allows the use of bloop, using the newest Zinc (with the far better incremental compilation engine compared to 0.13.x).

[slandelle]

I have in mind doing something similar for Maven but only applied to Scala compilation.

That would be great. No idea if it's possible with maven though, all the more in multi-modules project.

The current maven-bloop plugin does all this plugin does but allows the use of bloop, using the newest Zinc (with the far better incremental compilation engine compared to 0.13.x).

My only concern (but I'm not the owner here) is to avoid one nighter style contribution as I suspect no one but you and the other people working on Bloop will be able to maintain it.

[jvican]

My only concern (but I'm not the owner here) is to avoid one nighter style contribution as I suspect no one but you and the other people working on Bloop will be able to maintain it.

I understand your concern. If this is done it won't be a "nighter style contribution" because:

1. The integration will be independent of compiler internals. You won't need to be an expert to maintain the integration.
2. Bloop will be maintained by the core Scala team as it's becoming more and more used in key infrastructure in our community.
3. Bloop versioning has a strict compatibility policy to ease migration on the build tools side.

A big problem of build tools in the Scala community, IMO, is that they have had to cope with integrating with fragile Scala compiler internals/test engines/Scala-specific logic for basic features. This is problematic because the Scala build tool plugins are often maintained by OSS plugin authors that don't have the time to work on migration such as the Zinc 0.13 -> Zinc 1.0 and we, at both the Scala Center and Lightbend, are only a few engineers overwhelmed with work.

I hope that removing the burden of integrating with compiler internals will be a positive move in our community, it will allow library authors to focus on more important things than migrating the Scala compilation engine and will allow us compiler maintainers to focus on improving compilation performance in a tool that every build tool can then reuse, hence shipping the best user experience to our Scala developers.

That would be great. No idea if it's possible with maven though, all the more in multi-modules project.

To elaborate on my ideas, I want to use the Build Server Protocol (https://github.com/scalacenter/bsp/) to offload compilation from Maven to Bloop. Bloop already implements the protocol and we provide a Java-only library to communicate via BSP, so all we need to do is to tie them together so that when the user does compile in maven it sends a request to the server and gets a response back. This can be done for multi-module builds too (I've been playing with this idea for quite a while in other integrations, such as the bloop-sbt integration, and it's been working nicely so far). Everything is more or less thought out, I just need the time to properly implement it. I will get back to the Maven integration when I'm done with all the other high-priority tasks I'm doing now.

[slandelle]

Thanks @jvican , looks promising!

[He-Pin]

I think this plugin could depend on bloop for increase compiling but no need to install a server? And IIRC, the Scala Gradle plugin is currently using an old Zinc too. And bloop will help all these plugins definitely.

[jvican]

@slandelle 👍 Will keep you posted.

@hepin1989 FTR, Bloop also has a well-maintained gradle plugin.

[davidB]

Hi @jvican ,

I lurk Bloop some time ago (very promising), with the idea to use it in the plugin, someday instead of Zinc. I spent less and less time on this 10y old project, but any contribution are welcome, and I could help (limited time) for the intergation of Bloop.
I had similiar discussion on Gitter with @Andrei-Pozolotin the author of scalor-maven-plugin few months ago ( before #226 ).

PS: Lightbend is not an argument for a long term support, due to my past experience with EPFL / SourceSafe / Typesafe on this pluggin and Scala-IDE (FYI the support of zinc in this plugin was made by someone of EPFL / Typesafe).

[lefou]

The largest project where I used maven and this scala plugin migrated to sbt in the last couple of months, so since then, I use it this plugin no longer on a day-to-day basis, but here is how I used to use this plugin:

Incremental compilation was not useful to me, as the interproject-state needed for efficiency was pretty much always invalid or missing - that's the "fault" of Maven, so I think there is not much to do about.

Compile server via nailgun was the killer feature, as it made compiling scala projects with maven bearable! Instead of 6-8 minutes, I just had to wait 2 minutes, roughly. Consider a 60+ module project, with two (cold) compiler instances for each module...

I also tried to use bloop plugin, which isn't an "integration" at all (at least when I checked), but a generator to use bloop independently of Maven. But IMHO using bloop without build system integration only works for "simple" projects. Building OSGi bundles which need to consume other sub-modules as packaged JARs instead of just a bunch of classes doesn't work with bloop.

(I also tried to write a bloop integration for maven, was convinced in it's Gitter channel to not create a separate project but integrate into bloop repo. I did in a fork. Unfortunately, this was not the right decision for me, I lost some days trying to integrate but there was no support for integration testing and the tight integration with this scala-maven-plugin was a show stopper for me. After that, I had to turn to other work.)

[jvican]

I also tried to use bloop plugin, which isn't an "integration" at all (at least when I checked), but a generator to use bloop independently of Maven

It isn't an integration yet. We're in the middle of turning our plugins into fully working integrations, first starting with sbt and then doing it for other plugins.

But IMHO using bloop without build system integration only works for "simple" projects. Building OSGi bundles which need to consume other sub-modules as packaged JARs instead of just a bunch of classes doesn't work with bloop.

We can make it work whenever there's a full integration with the build system. The reason it's not done yet is because we're focusing on features/properties that affect the most developers (fast developer workflows). We're only a few so we need to prioritize. OSGi is a niche feature, it will eventually be supported, but the focus isn't there.

[jvican]

Hey @davidB, thanks for offering some small help, it's appreciated. @Andrei-Pozolotin has done amazing work, but it's a pity that it isn't compatible with the configuration of this plugin. Would you guys be free to jump on a call to talk about the future of Scala support in Maven?

[davidB]

@jvican

Would you guys be free to jump on a call to talk about the future of Scala support in Maven?

Yes, it depends of the time window (I'm in France, and Friday is often a better day for a call with me, less constraint).
FYI Scala, is no longer my daily job or my favorite language.

[He-Pin]

FYI Scala, is no longer my daily job or my favorite language.

Hm, Rust. Spark is using this plugin, maybe someone of them could do some contribution:).

[SethTisue]

I'm surprise by the usage of this plugin in an ecosystem where sbt is the defacto build tool.

if you participate in Scala OSS and watch conference talks, it's easy to get the impression that sbt use is near-universal.

but if you talk to people who are just quietly using Scala at work, you get a different impression: Maven and Gradle use seems rather common, especially at big companies. (in a Java shop, it can be easier to get a new language in the door than it is to convince co-workers to switch build tools.)

[slandelle]

@davidB I know you don't actively maintain this plugin, but you're still the owner here, so the decision is up to you.
Would you be willing to merge and release a PR for Zinc upgrade at the cost of dropping nailgun support?

Maybe some day someone will contribute (and commit to maintain actively) a Bloop support, which would be great.

[jvican]

A small update as a bloop maintainer: I've been talking to the Gradle guys and looking into a way we can pull of a built-in bloop gradle integration jointly. The next bloop release will be material to enable this to happen and will likely be followed up by a built-in sbt integration too.

If someone is motivated enough to do the work to upgrade to a zinc version in this plugin, I'd encourage them to sync up with me next week and look into how we can do the same integration for Maven. I'd be happy to help out actively, provide support and help with the implementation, which will be very similar to that of the built-in Gradle and bloop integrations, via BSP (Build Server Protocol).

Once the integration is done, I'd be happy to maintain semi-actively the integration and fix any potential issues that we could stumble upon, even though I wouldn't commit to work on feature requests.

[slandelle]

Hi @jvican

Sadly, I'm afraid I have very little time to offer, hence what I had in mind was just to port what I've done for Gatling.

[davidB]

@slandelle Yes, I'm ok to merge and rebase PR to improve the current state, including with drop of nailgun (just need to know if I upgrade the major or the minor version number)

I'm also open to add collaborator into the project.

[davidB]

Note, that the code of the plugin itself should stay in java, to be protected against some scala changes.

[slandelle]

Note, that the code of the plugin itself should stay in java, to be protected against some scala changes.

Yes, of course. Will try to send a PR this week-end.

[slandelle]

PTAL #321 (first take).
Let's discuss there.

[slandelle]

@jvican I'm running into an issue with Mixed compile order. Could you please take a look at #322?

[slandelle]

Closed as of ba7ab93 and 0f01f2f

[seanf]

Thanks @slandelle and @davidB! These issues both mention zinc 1.x: #208 & #226

[slandelle]

@seanf I've closed them too.