Support Zinc pipelining

[bishabosha]

support ThisBuild/usePipelining := true in sbt for Scala 3 projects.

Requirements

• add -Ypickle-java and -Ypickle-write flags, expected by Zinc when pipelining is enabled in sbt.
• presence of -Ypickle-write <destination> arguments requires that TASTy files are written early to that destination (ASAP after pickler)
• apiPhaseCompleted and dependencyPhaseCompleted Zinc callbacks must be called as soon as TASTy is fully written, which will activate early downstream compilation.
• call generatedNonLocalClass callback before TASTy writing, which is necessary for Zinc to run the incremental algorithm early.
• -Ypickle-java flag is a directive to include TASTy for Java sources in the -Ypickle-write destination (not in the final compiler outputs)

Compatibility

All functionality is observable only under -Y flags, so this isn't a breaking change

[goshacodes]

Hi! Can we expect this to be backported to LTS some day?

[WojciechMazur]

Hi! Can we expect this to be backported to LTS some day?

We were considering it, however, the effort to enable pipelining spans over multiple PRs, some of these might be problematic to backport. Right now it seems unlikely it would be backported. We're considering having a new LTS series somewhere in Q4 2025 thus it might be even more likely to be backported.

[bishabosha]

It would need a redesign because it specifically uses a new TASTy feature to implement (Attributes section), meaning its problematic to consume from an older compiler in the same LTS. perhaps like with .betasty it could be achieved with a new file extension, but then older compilers would still be unaware - sbt plugin could enforce a minimum LTS version maybe?.