Main merge, more ideas for experimental README changes #215
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…tlang#194) The [atomic library is used in the Swift stdlib](https://github.com/swiftlang/swift/blob/69756635819114a851023aa2e8e6fd87f51d094d/cmake/modules/AddSwift.cmake#L336-L338) for 32-bit architectures. As such, when attempting to compile a binary containing the static Swift stdlib using a Swift SDK for armv7, `-latomic` must be linked. This is not needed for regular cross-compilation, just when passing `--static-swift-stdlib`. I've added `-latomic` for "armv7" and added a test case in `LinuxRecipeTests` to ensure that it is included in the toolset.json file. Now, I can build my binary that contains the statically linked Swift stdlib with this command: ``` swift build --swift-sdk 6.0.3-RELEASE_debian_bookworm_armv7 --static-swift-stdlib ``` This generates a rather large binary when also linking other things such as Foundation and Dispatch, as is to be expected: ``` $ pwd ~/tmp/hummingbird-examples/hello $ du -hs .build/debug/App 123M .build/debug/App ``` But that's fine, because it works perfectly on the target (Raspberry Pi 2) and provides yet another option for compiling binaries using the Swift SDK.
…ts (swiftlang#193) Apparently I didn't test very well from PR swiftlang#188, because when I went to use the Ubuntu Noble Swift SDK, I got this error: ``` error: link command failed with exit code 1 (use -v to see invocation) ld.lld: error: ~/.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_noble_aarch64.artifactbundle/6.0.3-RELEASE_ubuntu_noble_aarch64/aarch64-unknown-linux-gnu/ubuntu-noble.sdk/usr/lib/aarch64-linux-gnu/libm.so:4: cannot find /lib/aarch64-linux-gnu/libm.so.6 inside ~/.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_noble_aarch64/aarch64-unknown-linux-gnu/ubuntu-noble.sdk >>> GROUP ( /lib/aarch64-linux-gnu/libm.so.6 AS_NEEDED ( /lib/aarch64-linux-gnu/libmvec.so.1 ) ) >>> ^ clang: error: linker command failed with exit code 1 (use -v to see invocation) ld.lld: error: ~/.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_noble_aarch64.artifactbundle/6.0.3-RELEASE_ubuntu_noble_aarch64/aarch64-unknown-linux-gnu/ubuntu-noble.sdk/usr/lib/aarch64-linux-gnu/libm.so:4: cannot find /lib/aarch64-linux-gnu/libm.so.6 inside /~/.swiftpm/swift-sdks/6.0.3-RELEASE_ubuntu_noble_aarch64.artifactbundle/6.0.3-RELEASE_ubuntu_noble_aarch64/aarch64-unknown-linux-gnu/ubuntu-noble.sdk >>> GROUP ( /lib/aarch64-linux-gnu/libm.so.6 AS_NEEDED ( /lib/aarch64-linux-gnu/libmvec.so.1 ) ) >>> ^ ``` Turns out, the packages for Ubuntu Noble do not come with a lib/ symlink included, so all I needed was to add a custom step to create that symlink from `ubuntu-noble.sdk/lib` -> `ubuntu-noble.sdk/usr/lib` and all is well. As a part of these changes, I also added a missing cleanup of the target toolchain to remove unused parts of the LinuxRecipe as is done for the [WebAssemblyRecipe](https://github.com/swiftlang/swift-sdk-generator/blob/a0ff972af294243ef9211534f7c9ba6f83672b82/Sources/SwiftSDKGenerator/SwiftSDKRecipes/WebAssemblyRecipe.swift#L120-L125). Doing this reduces the size of `ubuntu-noble.sdk` from 1GB to 721MB, when building the Swift SDK without docker. This can help for the end-to-end tests, and is nice to have for smaller Swift SDK distributions. Finally, I updated the EndToEndTests to provide `linuxDistributionVersion` and set the Swift60_Ubuntu tests to use "24.04" as a way to test this lib/ directory fix. This has me thinking we may want to think about end-to-end tests for each version of Ubuntu, like "20.04", "22.04", and "24.04", for each version of Swift.
* [CI] Add support for GitHub Actions * Add locales locales-all libsqlite3-dev to the GitHub Actions * workflows: Disable tests; concentrate on fixing errors in soundness * workflows: Remove soundness.sh, which now comes from github-workflows Ironically, Utilities/soundness.sh fails the shellcheck soundness check. * workflows: Fix yamllint formatting * workflows: Exempt vendored code from swift format checks * workflows: Exempt ProcessExecutor and its tests from unacceptable language check "kill" in these files refers to a Unix command and signal and cannot be avoided. * Update .github/workflows/pull_request.yml * Unify license headers * Update `license_header_check_project_name` in workflow * Use `.license_header_template` * Fix most of the license checks * Update .licenseignore * Fix `.licenseignore` * workflows: Fix multiline string formatting for swift-format * workflows: Allow swift-format to fix whitespace * workflows: Snapshot default swift-format settings The .swift-format settings file was generated by: swift format dump-configuration > .swift-format We will need to configure or turn off some checks; this allows us to keep track of what settings we have changed. * workflows: Disable invasive swift-format checks swift format complains about some long-standing choices in this repository. For now we will disable these specific complaints. * workflows: Remove soundness check from CI compose files --------- Co-authored-by: Euan Harris <[email protected]> Co-authored-by: Max Desiatov <[email protected]>
* workflows: Re-enable tests job * workflows: Disable tests on Windows Requires NIO support on Windows * workflows: Disable end-to-end tests in GitHub Actions There is a good chance that GitHub Actions will be able to run these tests, but we will disable them for now in order to get the unit tests running. The end-to-end tests could not run with swift-ci, so disabling them here does not reduce CI coverage.
… distributions (swiftlang#195) This fixes the final issue we saw with swiftlang#138 where there was an error generating an `amazonlinux2` Swift SDK for x86_64. This was fixed by removing 32-bit libraries from the sysroot imported from the container for RHEL-based distributions. To test this, I've added EndToEndTests for `amazonlinux2` and `fedora39` on top of the existing RHEL UBI9 tests. All of these depend on pulling from containers to test: ``` 2.1 GiB [#################] /fedora39_aarch64_6.0.3-RELEASE_with-docker.artifactbundle 2.0 GiB [################ ] /fedora39_x86_64_6.0.3-RELEASE_with-docker.artifactbundle 2.0 GiB [################ ] /amazonlinux2_aarch64_6.0.3-RELEASE_with-docker.artifactbundle 1.9 GiB [############### ] /amazonlinux2_x86_64_6.0.3-RELEASE_with-docker.artifactbundle 1.9 GiB [############### ] /rhel_ubi9_x86_64_6.0.3-RELEASE_with-docker.artifactbundle 1.9 GiB [############### ] /rhel_ubi9_aarch64_6.0.3-RELEASE_with-docker.artifactbundle 1.8 GiB [############### ] /fedora39_x86_64_5.10.1-RELEASE_with-docker.artifactbundle 1.8 GiB [############### ] /fedora39_aarch64_5.10.1-RELEASE_with-docker.artifactbundle 1.8 GiB [############## ] /amazonlinux2_aarch64_5.10.1-RELEASE_with-docker.artifactbundle 1.8 GiB [############## ] /amazonlinux2_x86_64_5.10.1-RELEASE_with-docker.artifactbundle 1.7 GiB [############## ] /rhel_ubi9_x86_64_5.10.1-RELEASE_with-docker.artifactbundle 1.7 GiB [############## ] /rhel_ubi9_aarch64_5.10.1-RELEASE_with-docker.artifactbundle 1.7 GiB [############## ] /amazonlinux2_aarch64_5.9.2-RELEASE_with-docker.artifactbundle 1.7 GiB [############# ] /amazonlinux2_x86_64_5.9.2-RELEASE_with-docker.artifactbundle 1.7 GiB [############# ] /rhel_ubi9_x86_64_5.9.2-RELEASE_with-docker.artifactbundle 1.7 GiB [############# ] /rhel_ubi9_aarch64_5.9.2-RELEASE_with-docker.artifactbundle ``` All of these Swift SDKs take 30GB of disk space. --------- Co-authored-by: Max Desiatov <[email protected]>
`mkdir cross-compilation test` -> `mkdir cross-compilation-test`
…lang#205) Make formatting consistent and closer to the widely adopted style.
- Defaulted `swiftVersion` to "6.1-RELEASE".
- Added EndToEndTests for Ubuntu and RHEL for Swift61, including some additional tests for Ubuntu Jammy.
- Fixed missing /lib and /lib64 directories in Ubuntu Noble when building from deb packages. If these directories don't exist, they are symlinked to the correct directories in /usr, which is consistent to what happens when copying from the Docker containers.
- Added the fix for downloading Swift for the target that adds the platform name to the cache file, to avoid corruption of the artifacts:
* <img width="375" alt="image" src="https://github.com/user-attachments/assets/b18efd38-2cbe-44e1-a6fd-2fbb1a13002b" />
ALSO: I did not include EndToEndTests for Fedora39 for Swift 6.1 since the container is non-existent, likely because it's EOL.
…#206) Utilities/soundness.sh was deleted in swiftlang#199 which moved CI to GitHub Actions. Credit to @heckj, who found the same problem in apple/swift-container-plugin#96.
…lang#203) This completes swiftlang#116 and can be used to generate Debian Swift SDKs either by downloading dependencies from mirrors directly or from containers. - Added Debian `bullseye` and `bookworm` to `LinuxDistribution` and modded `SwiftSDKGenerator+Download` to work downloading both Ubuntu and Debian packages from their respective mirrors. - Swift SDKs can be generated from Docker using the `swift:*-bookworm` container for Debian 12, but for Debian 11 you must create the container yourself/manually put the Swift toolchain into it. - Added special handling in `VersionsConfiguration` so that if Debian 11 (bullseye) is selected, the Ubuntu 20.04 toolchain is downloaded. If Debian 12 (bookworm) is selected for Swift 5.9.* or Swift 5.10, the Ubuntu 22.04 toolchain is used instead. https://askubuntu.com/a/445496 - Added tests for the special handling in `LinuxRecipeTests.testItemsToDownloadForDebianTargets()` to ensure the correct toolchain is selected. To use: ```bash $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 11 $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12 $ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12 --with-docker ``` I also added EndToEndTests for all Debian supported combinations, resulting in another 30GB of generated Swift SDKs: ``` 2.1 GiB [####################] /debian_12_x86_64_6.0.3-RELEASE_with-docker.artifactbundle 2.1 GiB [################### ] /debian_12_aarch64_6.0.3-RELEASE_with-docker.artifactbundle 2.0 GiB [################## ] /debian_12_x86_64_5.10.1-RELEASE_with-docker.artifactbundle 1.9 GiB [################# ] /debian_12_x86_64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_12_aarch64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_11_x86_64_6.0.3-RELEASE.artifactbundle 1.9 GiB [################# ] /debian_12_aarch64_5.10.1-RELEASE_with-docker.artifactbundle 1.9 GiB [################# ] /debian_11_aarch64_6.0.3-RELEASE.artifactbundle 1.8 GiB [################ ] /debian_12_x86_64_5.10.1-RELEASE.artifactbundle 1.7 GiB [################ ] /debian_12_aarch64_5.10.1-RELEASE.artifactbundle 1.7 GiB [################ ] /debian_11_x86_64_5.10.1-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_11_aarch64_5.10.1-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_12_x86_64_5.9.2-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_12_aarch64_5.9.2-RELEASE.artifactbundle 1.7 GiB [############### ] /debian_11_x86_64_5.9.2-RELEASE.artifactbundle 1.6 GiB [############### ] /debian_11_aarch64_5.9.2-RELEASE.artifactbundle ``` To make this work properly, I modded the `targetSwift` path in `DownloadableArtifacts` to use the swiftPlatform name in the file name to avoid the issues I saw with the EndToEndTests, where the *.tar.gz files would get corrupted when trying to download a different version of the target Swift on top of the existing file. Now, they look like this: ``` target_swift_5.10.1-RELEASE_debian12_aarch64.tar.gz target_swift_5.9.2-RELEASE_ubuntu22.04_aarch64.tar.gz target_swift_5.10.1-RELEASE_debian12_x86_64.tar.gz target_swift_5.9.2-RELEASE_ubuntu22.04_x86_64.tar.gz target_swift_5.10.1-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_debian12_aarch64.tar.gz target_swift_5.10.1-RELEASE_ubuntu20.04_x86_64.tar.gz target_swift_6.0.3-RELEASE_debian12_x86_64.tar.gz target_swift_5.10-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_5.9.2-RELEASE_ubuntu20.04_aarch64.tar.gz target_swift_6.0.3-RELEASE_ubuntu20.04_x86_64.tar.gz target_swift_5.9.2-RELEASE_ubuntu20.04_x86_64.tar.gz ```
All of the CI jobs are using Swift 5.9 or later at this point.
This seems to be a reasonable limit still readable on most devices, while reducing the amount of reformatting applied by `swift format` checks.
This allows targeting WASI with Embedded Swift, enabling use of a subset of Swift Concurrency in this mode, and also hiding most if not all of unsafe flags in toolsets. This way, users of Embedded Swift can build their packages with a similar `swift build --swift-sdk` command as the rest of users of WASI.
The generator incorrectly recorded absolute paths to Swift SDK metadata in bundle archive metadata, which is fragile and also can't be handled by SwiftPM.
…ng#213) Embedded Swift looks up Clang's compiler-rt static library in a different path than non-embedded. These paths could be normalized in the future, but for now the easiest fix seems to be just to symlink the directory the static library is located in.
…ing less than 6.0
…r artifactbundles to be distributed
MaxDesiatov
approved these changes
May 17, 2025
|
Perfect, thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hi @MaxDesiatov, I've been thinking about this for a while and decided to contribute the ideas back to your branch.
main, probably a good idea to start with.experimental-prefix to success message when running under Swift 5.x. Under 6.0 and later, you'll just get the correct message based on your Swift version:Swift SDK Distributionsection of the README on how to generate checksums.Let me know what you think, maybe some good inclusions or could use some tweaks.