Handle build description compatibility issue

[neonichu]

Description

We introduced new tools into SwiftPM’s build system to be able to write auxiliary files during the build. Older SwiftPM versions won’t know about this and will fail to load a build description that references them:

❯ swift build
error: invalid tool type in 'tools' map
error: unable to load build file

This seems kinda bad and we need to figure out how to handle this.

rdar://113005515 (Handle build description compatibility issue)

[neonichu]

One idea is to only use the new behavior for packages with tools-version 5.10 (or whatever the version of Swift we're shipping these changes ends up being).

That seems fine in general, but #6573 was important for Windows and making its behavior contingent on the tools-version might be undesirable. cc @compnerd

[compnerd]

Yeah, Windows really needs this, so making that conditional wouldn't work very well. This might make building some projects impossible on Windows (e.g. DocC).

[neonichu]

Maybe what we can do is make this dependent on the tools-version in general, but not for Windows -- that's a little wonky but I think gives us the behavior we want. Once one release has passed, we can probably drop the tools-version check since we don't really need to support switching between several releases of SwiftPM on the same project.

@tomerd wdyt?

[tomerd]

seems pragmatic to make an exception for windows

[neonichu]

I guess there's also swiftlang/sourcekit-lsp#676, though, which won't really be fixed in most cases because packages will use an older tools-version. We could offer an option for LSP to opt-in, but of course that would also pollute the state in that same way.

I wonder if another thing we could do is add dummy versions of the tools to 5.9 (and maybe even 5.8 if we are still making releases for that) so that the build manifests stays compatible. I'm going to experiment with that to see if it is viable. It seems like it should be, since the older releases are writing the files outside of the build, so the only issue is really that they can't load the llbuild manifest.

[BrianHenryIE]

Because of a backup restore, I got this error when trying to run some tests against a build that was made with an August 5.9 nightly, but I was now back on a July 5.9 snapshot and saw error: invalid tool type in 'tools' map (and maybe the rest but the error is gone from my screen).

I deleted the .build folder and everything was fine.

[dnadoba]

I run into this as well.

we don't really need to support switching between several releases of SwiftPM on the same project.

@neonichu We do this actually quite frequently for all our Swift on Server projects. We usually support the latest 3 minor swift versions and at least test on the next minor in addition to main nightly. This makes a total of 5 Swift versions we regularly test locally. I have recently started to set --scratch-path .build/$${SWIFT_VERSION-}/ in some of our scripts, mainly to accelerate local rebuilds. This works surprisingly well. I'm starting to think that it might actually be a good idea that SwiftPM creates a separate build folder for each swift version. It could still create a symbolic links to the current swift version, similar to how .debug and .release links to the folder of the current host architecture.

[rauhul]

+1 to @dnadoba's suggestion, I'm trying to use docker to do linux build on my macOS host and its extremely annoying to hit error: invalid tool type in 'tools' map because the SPM version in the container doesn't match the host