Driver: honour -sdk even on Darwin

[compnerd]

If -resource-dir is passed, it takes precedence. If the user specifies -sdk
that should find content in that SDK. Generalise this to Darwin as well.

Replace this paragraph with a description of your changes and rationale. Provide links to external references/discussions if appropriate.

Resolves SR-NNNN.

[compnerd]

@swift-ci please test macOS platform

[swift-ci]

Build failed
Swift Test OS X Platform
Git Sha - 3d4b363

[jrose-apple]

This is not correct. All builds on Darwin use SDKs, always.

[jrose-apple]

The right answer is to better distinguish compiler resource content from platform content, or to fall back to the SDK if a particular platform isn't in the compiler resource dir.

[compnerd]

@jrose-apple hmm, am I mistaken in that didn't Darwin start pulling the resource content from the SDK? So, that sounds like we would diverge the Darwin and non-Darwin paths. Also, do we want to make -resource-dir a required parameter to the driver then? How about dealing with building Swift code on a Linux host where the ("sysroot") is different from the Swift content ("SDK")? I'm not tied to the current approach, but the current state is significantly simpler flags wise.

[jrose-apple]

Darwin started putting some resource dir content in the SDK in addition to what's in the compiler directory, not replacing it. In particular, the compiler directory still has

• shim headers (questionable)
• API notes (definitely needs to go away)
• migrator content (questionable)
• SwiftPM host resources (definitely belongs in resource dir)
• backwards-deployment static libraries (probably always belongs in resource dir)
• Clang resource dir (definitely belongs in resource dir)
• (non-Darwin only) CF and CDispatch headers (probably belongs in SDK)

Basically, it's not that the resource directory is "the directory where the stdlib can be found"; it's that the directory where the stdlib can be found used to just be the resource directory and now it's both the resource directory and the SDK.

[compnerd]

Okay, that sounds reasonable. (Personally, I think that we should be minimising the content of the resource directory - it doesn't scale well especially as you get into the cross-compilation scenarios where platforms other than Darwin do not have fat binaries.)

The behaviour of -sdk still needs to be different (as in what it is currently). But, we should have a fallback from the SDK to the resource directory and that sounds reasonable. That also raises the question of should the order be:

• SDK
• "sysroot" (quotes because Windows doesn't have the concept of a sysroot, its a collection of system include paths and library search paths)
• toolchain

As to the content, I think what the other platforms currently need is:

• shim headers (probably can be in the resource directory, but IMO belongs in the SDK)
• SPM host resources (belongs in the toolchain - this is build specific AIUI)
• backwards deployment support (some day, but belongs in the SDK - this is target dependent)
• clang resource directory (belongs in the toolchain)
• CF and CDispatch headers (working on removing the CF headers, but belongs in the SDK)

The split that makes sense on non-Darwin platforms IMO is something like this:

• SDK: all the swift content (*.swiftdoc, *.swiftmodule, *.swiftinterface, swiftrt.o, CF/CDispatch headers)
• "sysroot": libc headers and system libraries (e.g. glibc, bionic, WinSDK/msvcrt)
• toolchain: the compiler resource dir and provider of the tools for building/inspecting code

Does that sound reasonable to you?

[jrose-apple]

Darwin isn't going to draw a distinction between "SDK" and "sysroot". Do we really need one for other platforms?

I do agree about minimizing the content of the resource dir, and I think that's the direction we're going with Darwin too. We just haven't gotten there completely yet.

[compnerd]

Yeah, sadly, I think that the other platforms will need that distinction :-(. I don't think that we want to get into the business of trying to repackage the Windows SDK and shipping it (which may have additional constraints). The Linux side, again, the question is, do we really want to get into repackaging the distributions content? Remember that Ubuntu and Fedora and anything else are technically incompatible, so you need a different SDK per Linux distribution and architecture :/

[jrose-apple]

Okay, so next question: without ABI stability, there's no module stability, and without module stability everything in the SDK is tied to the compiler anyway. So why not put it in the compiler resource dir? Even if you have to install it separately.

(I think the main answer so far is "one kind of Linux cross-compiling to another Linux" but I wonder if that just means we should organize the resource dir differently.)

[compnerd]

An interesting idea - so you are proposing that on all the non-Darwin platforms we go the exact opposite of what we really want and expand the resource dir to be as large as possible? That still won't work for the Linux cross-compilation scenario. One more thing ... this also doesn't work on the same Linux distro even! With distributions now providing an alternative of glibc with musl, we need multiple resource directory even for every linux distribution.

Honestly, at that point the difference between resource dir and SDK really becomes nomenclature and flags. I personally would prefer that we don't diverge on this point and that we continue to minimise the compiler resource dir for all targets. If you don't like SDK, that is something we can work around by having another flag. We could restore -sdk to mean -sysroot and add a new -swiftroot flag to be the non-SDK SDK.

I really want to start investigating the possibility of getting ABI stability on other targets, we are pretty close to having Foundation be at parity on Windows, Linux, and android. The work that @millenomi is doing to enable the parity with macOS is getting us much closer to having functional equivalence across the platforms.

[compnerd]

Okay, I want to finish up the discussion here about the way forward, but, in the mean time, I'm going to merge #26388 which effectively reverts #25990 and will perform the cascading search. But, the proper approach for the longer term thing is something to actually pin down so that we can address that.

[jrose-apple]

Yeah, maybe we should bring this to the forums. You've made the case (effectively, again) for three relevant directories, but we need to decide what they are and how they work in all the various scenarios.

[vgorloff]

UPDATE:

Usage of the flags -Xclang-linker and -Xcc helped to solve issue.

---

Can you please also have a look on comment to already merged PR: #25990 (comment)

It is about compile error when using Swift compiler (made during cross-compilation on macOS host).
It happens when -sdk flag points to Android SDK (-sdk /usr/local/ndk/platforms/android-24/arch-arm) and, as result, synthesised path to swiftrt.o points to non-existing location (/usr/local/ndk/platforms/android-24/arch-arm/usr/lib/swift/android/armv7/swiftrt.o)

Thank you!

[shahmishal]

Please update the base branch to main by Oct 5th otherwise the pull request will be closed automatically.

• How to change the base branch: (Link)
• More detail about the branch update: (Link)