Cross-compiling packages which contain macros doesn't work #6950

finagolfin · 2023-09-26T09:31:44Z

Description

@woxiang04 reported in the forum that they couldn't cross-compile the macro examples, and @neonichu said that doesn't work yet. I am unsure what this entails, as presumably the official Swift 5.9 toolchain for macOS can cross-compile code with macros for iOS?

I have not tried this myself, just reporting that error from the forum.

Expected behavior

Macros to work when cross-compiling a package

Actual behavior

According to that forum thread:

compiler plugin not loaded: /data/Downloads/swift-macro-examples/.build/aarch64-unknown-linux-android24/debug/MacroExamplesPlugin; loader error: Exec format error

Steps to reproduce

Build the Swift macro examples with a cross-compilation SDK

Swift Package Manager version/commit hash

5.9

Swift & OS version (output of `swift --version ; uname -a`)

5.9 for Ubuntu 20.04 x86_64 cross-compiling to Android AArch64

The text was updated successfully, but these errors were encountered:

neonichu · 2023-09-26T18:48:31Z

That's correct, macros are not expected to work when cross-compiling today (that's not limited to any particular platform).

Resolves #6950 Resolves rdar://105991372

### Motivation: Not supporting macros in cross-compilation is a major limitation, especially for libraries like https://github.com/apple/swift-mmio. ### Modifications: Added `enum BuildTriple { case buildTools, buildProducts }` and `var buildTriple: BuildTriple` on `ResolvedTarget` and `ResolvedProduct`. Corresponding value is assigned to this property depending on target and product type: `buildTools` for macros, plugins, and their dependencies, `buildProducts` for everything else (the default). Based on this property we can choose between `buildParameters.hostTriple` and `buildParameters.targetTriple` during build plan construction. ### Result: Resolves #6950 Resolves rdar://105991372

### Motivation: Not supporting macros in cross-compilation is a major limitation, especially for libraries like https://github.com/apple/swift-mmio. ### Modifications: Added `enum BuildTriple { case tools, destination }` and `var buildTriple: BuildTriple` on `ResolvedTarget` and `ResolvedProduct`. We're not using "host" and "target" triple terminology in this enum, as that clashes with build system "targets" and can lead to confusion in this context. Corresponding value is assigned to this property depending on target and product type: `tools` for macros, plugins, and their dependencies, `destination` for everything else (the default). Based on this property we can choose between `buildParameters.hostTriple` and `buildParameters.targetTriple` during build plan construction. Additionally, the resolved products and resolved targets graph is now constructed in a way that allows certain targets and products to be built twice: once for host triple, once for target triple if needed. This required modifying build description and build manifest generation to distinguish these products and targets from each other that are built twice. Artifacts built for the host now have `-tools` suffix appended to their names. This cascaded into making changes throughout the code base for build tool plugins and package plugins handling, which constructed their paths in an ad-hoc manner without accounting for possible changes to these names. Also added `CrossCompilationPackageGraphTests` and `CrossCompilationBuildPlanTests` to verify the changes made are applied correctly. ### Result: Resolves #6950 Resolves rdar://105991372

finagolfin · 2024-03-27T09:10:54Z

Should be reopened as this is still being worked on, #7353.

finagolfin · 2024-04-28T12:04:04Z

I'm still seeing issues with cross-compiling packages containing macros. I downloaded the April 25 trunk snapshot ubi9 toolchain, the latest LTS Android NDK 26d, and a trunk Android SDK that my daily Android CI generated, sdk-trunk-aarch64-ndk26d, which works well for macro-free packages.

Using these commands, I consistently see build errors for packages that use macros:

> tar xf swift-DEVELOPMENT-SNAPSHOT-2024-04-25-a-ubi9.tar.gz
> unzip sdk-trunk-aarch64-ndk26d.zip
> tar xf swift-trunk-android-aarch64-2024-04-25-24-ndk26d-sdk.tar.xz
> ln -sf /home/fina/swift-DEVELOPMENT-SNAPSHOT-2024-04-25-a-ubi9/usr/lib/clang/17 ~/swift-trunk-android-aarch64-2024-04-25-24-sdk/usr/lib/swift/clang
> git clone https://github.com/finagolfin/swift-android-sdk.git

I then modified the destination config android-aarch64.json from my repo for my local environment:

> cat swift-android-sdk/android-aarch64.json
{
    "version": 1,
    "target": "aarch64-unknown-linux-android24",
    "toolchain-bin-dir": "/home/fina/swift-DEVELOPMENT-SNAPSHOT-2024-04-25-a-ubi9/usr/bin",
    "sdk": "/home/fina/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/sysroot",
    "extra-cc-flags": [
        "-fPIC"
    ],
    "extra-swiftc-flags": [
        "-resource-dir", "/home/fina/swift-trunk-android-aarch64-2024-04-25-24-sdk/usr/lib/swift",
        "-tools-directory", "/home/fina/android-ndk-r26d/toolchains/llvm/prebuilt/linux-x86_64/bin",
    ],
    "extra-cpp-flags": [
        "-lstdc++"
    ]
}

But it always fails with weird build errors when cross-compiling packages that have macros:

> cd swift-syntax/Examples/
> ~/swift-DEVELOPMENT-SNAPSHOT-2024-04-25-a-ubi9/usr/bin/swift package reset
> ~/swift-DEVELOPMENT-SNAPSHOT-2024-04-25-a-ubi9/usr/bin/swift build --build-tests --destination ~/swift-android-sdk/android-aarch64.json
error: Failed opening '/home/fina/swift-syntax/Examples/.build/aarch64-unknown-linux-android24/debug/index/store/v5/units/AddAsyncMacroTests.swift.o-1VZ0WNZLXJYVM': No such file or directory

I just tried again with the April 27 trunk snapshot that was just tagged, same build error.

@MaxDesiatov, let me know if you're able to reproduce with your cross-compilation SDK bundles for linux.

### Motivation: Not supporting macros in cross-compilation is a major limitation, especially for libraries like https://github.com/apple/swift-mmio. ### Modifications: Added `enum BuildTriple { case tools, destination }` and `var buildTriple: BuildTriple` on `ResolvedTarget` and `ResolvedProduct`. We're not using "host" and "target" triple terminology in this enum, as that clashes with build system "targets" and can lead to confusion in this context. Corresponding value is assigned to this property depending on target and product type: `tools` for macros, plugins, and their dependencies, `destination` for everything else (the default). Based on this property we can choose between `buildParameters.hostTriple` and `buildParameters.targetTriple` during build plan construction. Additionally, the resolved products and resolved targets graph is now constructed in a way that allows certain targets and products to be built twice: once for host triple, once for target triple if needed. This required modifying build description and build manifest generation to distinguish these products and targets from each other that are built twice. Artifacts built for the host now have `-tools` suffix appended to their names. This cascaded into making changes throughout the code base for build tool plugins and package plugins handling, which constructed their paths in an ad-hoc manner without accounting for possible changes to these names. Also added `CrossCompilationPackageGraphTests` and `CrossCompilationBuildPlanTests` to verify the changes made are applied correctly. ### Result: Resolves swiftlang#6950 Resolves rdar://105991372

finagolfin added the bug label Sep 26, 2023

finagolfin changed the title ~~Cross-compiling packages which contain macros doesn't work on linux~~ Cross-compiling packages which contain macros doesn't work Sep 27, 2023

MaxDesiatov mentioned this issue Oct 25, 2023

Failed to cross-compile source code using #Predicate macros. swiftlang/swift-sdk-generator#27

Closed

kabiroberai mentioned this issue Nov 13, 2023

Support building with Xcode kabiroberai/node-swift#14

Merged

MaxDesiatov added a commit that referenced this issue Nov 23, 2023

Support macros when cross-compiling

d4114da

Resolves #6950 Resolves rdar://105991372

MaxDesiatov mentioned this issue Nov 23, 2023

Support macros when cross-compiling #7118

Merged

MaxDesiatov added a commit that referenced this issue Dec 3, 2023

Support macros when cross-compiling

65a79cf

Resolves #6950 Resolves rdar://105991372

MaxDesiatov closed this as completed in #7118 Feb 18, 2024

MaxDesiatov reopened this Mar 27, 2024

MaxDesiatov linked a pull request Apr 9, 2024 that will close this issue

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

Merged

MaxDesiatov closed this as completed in #7353 Apr 17, 2024

finagolfin mentioned this issue Aug 10, 2024

Add macOS toolchain build and test finagolfin/swift-android-sdk#162

Merged

This was referenced Oct 30, 2024

Trunk 6.1 Macro builds appear to incorrectly compile all package dependencies also for the host when cross-compiling? #8078

Open

Cross-compiling tests doesn't work if a package uses macros #8094

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cross-compiling packages which contain macros doesn't work #6950

Cross-compiling packages which contain macros doesn't work #6950

finagolfin commented Sep 26, 2023

neonichu commented Sep 26, 2023 •

edited

Loading

finagolfin commented Mar 27, 2024

finagolfin commented Apr 28, 2024

Cross-compiling packages which contain macros doesn't work #6950

Cross-compiling packages which contain macros doesn't work #6950

Comments

finagolfin commented Sep 26, 2023

Description

Expected behavior

Actual behavior

Steps to reproduce

Swift Package Manager version/commit hash

Swift & OS version (output of swift --version ; uname -a)

neonichu commented Sep 26, 2023 • edited Loading

finagolfin commented Mar 27, 2024

finagolfin commented Apr 28, 2024

Swift & OS version (output of `swift --version ; uname -a`)

neonichu commented Sep 26, 2023 •

edited

Loading