Support Debian 11 and Debian 12 distributions in the generator #203

xtremekforever · 2025-03-24T17:36:32Z

This completes #116 and can be used to generate Debian Swift SDKs either by downloading dependencies from mirrors directly or from containers.

Added Debian bullseye and bookworm to LinuxDistribution and modded SwiftSDKGenerator+Download to work downloading both Ubuntu and Debian packages from their respective mirrors.
Swift SDKs can be generated from Docker using the swift:*-bookworm container for Debian 12, but for Debian 11 you must create the container yourself/manually put the Swift toolchain into it.
Added special handling in VersionsConfiguration so that if Debian 11 (bullseye) is selected, the Ubuntu 20.04 toolchain is downloaded. If Debian 12 (bookworm) is selected for Swift 5.9.* or Swift 5.10, the Ubuntu 22.04 toolchain is used instead. https://askubuntu.com/a/445496
Added tests for the special handling in LinuxRecipeTests.testItemsToDownloadForDebianTargets() to ensure the correct toolchain is selected.

To use:

$ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 11
$ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12
$ swift run swift-sdk-generator make-linux-sdk --linux-distribution-name debian --linux-distribution-version 12 --with-docker

I also added EndToEndTests for all Debian supported combinations, resulting in another 30GB of generated Swift SDKs:

    2.1 GiB [####################] /debian_12_x86_64_6.0.3-RELEASE_with-docker.artifactbundle
    2.1 GiB [################### ] /debian_12_aarch64_6.0.3-RELEASE_with-docker.artifactbundle
    2.0 GiB [##################  ] /debian_12_x86_64_5.10.1-RELEASE_with-docker.artifactbundle
    1.9 GiB [#################   ] /debian_12_x86_64_6.0.3-RELEASE.artifactbundle
    1.9 GiB [#################   ] /debian_12_aarch64_6.0.3-RELEASE.artifactbundle
    1.9 GiB [#################   ] /debian_11_x86_64_6.0.3-RELEASE.artifactbundle
    1.9 GiB [#################   ] /debian_12_aarch64_5.10.1-RELEASE_with-docker.artifactbundle
    1.9 GiB [#################   ] /debian_11_aarch64_6.0.3-RELEASE.artifactbundle
    1.8 GiB [################    ] /debian_12_x86_64_5.10.1-RELEASE.artifactbundle
    1.7 GiB [################    ] /debian_12_aarch64_5.10.1-RELEASE.artifactbundle
    1.7 GiB [################    ] /debian_11_x86_64_5.10.1-RELEASE.artifactbundle
    1.7 GiB [###############     ] /debian_11_aarch64_5.10.1-RELEASE.artifactbundle
    1.7 GiB [###############     ] /debian_12_x86_64_5.9.2-RELEASE.artifactbundle
    1.7 GiB [###############     ] /debian_12_aarch64_5.9.2-RELEASE.artifactbundle
    1.7 GiB [###############     ] /debian_11_x86_64_5.9.2-RELEASE.artifactbundle
    1.6 GiB [###############     ] /debian_11_aarch64_5.9.2-RELEASE.artifactbundle

To make this work properly, I modded the targetSwift path in DownloadableArtifacts to use the swiftPlatform name in the file name to avoid the issues I saw with the EndToEndTests, where the *.tar.gz files would get corrupted when trying to download a different version of the target Swift on top of the existing file. Now, they look like this:

target_swift_5.10.1-RELEASE_debian12_aarch64.tar.gz     target_swift_5.9.2-RELEASE_ubuntu22.04_aarch64.tar.gz
target_swift_5.10.1-RELEASE_debian12_x86_64.tar.gz      target_swift_5.9.2-RELEASE_ubuntu22.04_x86_64.tar.gz
target_swift_5.10.1-RELEASE_ubuntu20.04_aarch64.tar.gz  target_swift_6.0.3-RELEASE_debian12_aarch64.tar.gz
target_swift_5.10.1-RELEASE_ubuntu20.04_x86_64.tar.gz   target_swift_6.0.3-RELEASE_debian12_x86_64.tar.gz
target_swift_5.10-RELEASE_ubuntu20.04_aarch64.tar.gz    target_swift_6.0.3-RELEASE_ubuntu20.04_aarch64.tar.gz
target_swift_5.9.2-RELEASE_ubuntu20.04_aarch64.tar.gz   target_swift_6.0.3-RELEASE_ubuntu20.04_x86_64.tar.gz
target_swift_5.9.2-RELEASE_ubuntu20.04_x86_64.tar.gz

Have at it @MaxDesiatov @euanh

- The Debian 11 configuration does not work since there is no official Swift container for it. But, later a special case will be added.

…ackage mirrors

- When downloading Debian packages, we don't get contrib packages since they are not needed.

…y toolchain

…chain - Otherwise weird things happen from the EndToEndTests when running tests that then need to re-download a new toolchain on top.

…elected

MaxDesiatov · 2025-03-24T17:48:06Z

@swift-ci test

MaxDesiatov · 2025-03-25T09:34:45Z

@swift-ci test

xtremekforever · 2025-03-26T00:39:16Z

I updated the README.md since I realized it didn't reflect the fact that the generator supports Debian 11 and 12 now.

MaxDesiatov · 2025-03-26T08:02:58Z

@swift-ci test

- This is required since Debian 11 has a different layout than other Ubuntu/Debian versions.

xtremekforever · 2025-03-26T13:48:06Z

I found that Debian 11 has a different layout for where libraries are with the /lib and /lib64 directories when building from the container image. /lib64/ld-linux-x86-64.so.2 is symlinked to ../lib/x86_64-linux-gnu/ld-2.31.so in the sysroot, which means that we must copy over the /lib/x86_64-linux-gnu directory from the container as well to get a properly working sysroot for Debian 11.

I've added changes for this that should just apply to Debian 11- and also ran the EndToEndTests for containers to make sure other containers are not affected.

MaxDesiatov · 2025-03-26T14:19:57Z

@swift-ci test

MaxDesiatov · 2025-03-28T15:57:12Z

@swift-ci test linux

xtremekforever · 2025-03-28T21:48:54Z

@swift-ci test linux

Looks like it doesn't like the switch syntax I'm using on 5.8:

/__w/swift-sdk-generator/swift-sdk-generator/Sources/SwiftSDKGenerator/Generator/SwiftSDKGenerator+Copy.swift:86:28: error: expected initial value after '='
          let archSubpath =

Should be a simple fix.

MaxDesiatov · 2025-03-29T14:54:58Z

@swift-ci test

MaxDesiatov

LGTM, thanks!

MaxDesiatov · 2025-03-31T11:36:31Z

I'd prefer to hold off merging this until @euanh has a moment to review it, unless this PR is blocking some subsequent urgent PRs that you plan to submit?

xtremekforever · 2025-03-31T13:59:56Z

I'd prefer to hold off merging this until @euanh has a moment to review it, unless this PR is blocking some subsequent urgent PRs that you plan to submit?

This isn't urgent for me. I am using the stuff in this PR in my https://github.com/xtremekforever/swift-linux-cross-sdks repo, but I'm still working out being able to build Swift SDKs for bullseye for x86_64 and aarch64.

In case you didn't know, in this repo I'm basically working on a workflow for building Swift SDKs for all supported versions of Swift for x86_64, aarch64, and armv7 using the swift-sdk-generator. So to have some "pre-built" Swift SDKs out in the wild. However, there's still some ways to go as I still want to be able to build test binaries with each Swift SDK and test them on real targets for all of the above architectures, which means hooking up some real Raspberry PIs in to run the binaries for me. Lofty goals, long path...but it also helps in discovering issues in the swift-sdk-generator as I use it to generate different Swift SDKs here.

So yeah, @MaxDesiatov there aren't any more PRs for me here yet. I do plan to look at #197 after this PR is in, since I'll eventually need it, but I'm not there yet.

MaxDesiatov · 2025-04-07T16:18:32Z

@swift-ci test

MaxDesiatov · 2025-04-16T10:05:35Z

@euanh friendly ping

euanh · 2025-04-16T10:09:22Z

@swift-ci test

xtremekforever added 14 commits March 3, 2025 14:20

Increase unzip stream limit for larger package lists

f52aa21

Merge branch 'main' into swiftlang#116-support-debian-distributions

8e0ffcf

Add missing --linux-distribution-version to sdkGeneratorArguments

dedcd6d

Add Debian 11/12 as potential distributions for building SDKs for

fd58911

- The Debian 11 configuration does not work since there is no official Swift container for it. But, later a special case will be added.

Initial code to download dependencies for Debian distributions from p…

c28cbc0

…ackage mirrors

Support downloading Debian packages on top of Ubuntu packages

e8aa0c3

- When downloading Debian packages, we don't get contrib packages since they are not needed.

Add support for Debian 11 build from packages

da140b3

Add EndToEndTests for all supported Debian Swift SDK builds

a41ce54

Merge branch 'main' into swiftlang#116-support-debian-distributions

4ebb1ae

Restore missing withContainerImageSuffix

051985a

Update comments for Debian 11 end-to-end tests

ea57421

Support building Swift 5.9 SDKs for Debian Bookworm using Ubuntu Jamm…

72588ac

…y toolchain

Adjust targetSwift path to include swiftPlatform to identify the tool…

f4d0922

…chain - Otherwise weird things happen from the EndToEndTests when running tests that then need to re-download a new toolchain on top.

Add tests for downloading Ubuntu toolchains when Debian targets are s…

5462ab0

…elected

xtremekforever requested a review from MaxDesiatov as a code owner March 24, 2025 17:36

MaxDesiatov requested a review from euanh March 24, 2025 17:38

MaxDesiatov added the enhancement New feature or request label Mar 24, 2025

Merge branch 'main' into swiftlang#116-support-debian-distributions

ad87cf8

Also update README.md to recognize new Debian 11/12 support

9bf533b

Copy /lib and /lib64 for Debian 11 containers

144b407

- This is required since Debian 11 has a different layout than other Ubuntu/Debian versions.

Fix missing syntax for switch statement in 5.8

b199eef

MaxDesiatov approved these changes Mar 31, 2025

View reviewed changes

xtremekforever added 2 commits April 3, 2025 21:54

Merge branch 'main' into swiftlang#116-support-debian-distributions

8f00942

Merge branch 'main' into swiftlang#116-support-debian-distributions

fd400b1

Merge branch 'main' into swiftlang#116-support-debian-distributions

1e59866

euanh approved these changes Apr 16, 2025

View reviewed changes

MaxDesiatov merged commit ed6a85b into swiftlang:main Apr 16, 2025
19 checks passed

xtremekforever deleted the #116-support-debian-distributions branch April 16, 2025 10:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support Debian 11 and Debian 12 distributions in the generator #203

Support Debian 11 and Debian 12 distributions in the generator #203

xtremekforever commented Mar 24, 2025 •

edited

Loading

MaxDesiatov commented Mar 24, 2025

MaxDesiatov commented Mar 25, 2025

xtremekforever commented Mar 26, 2025

MaxDesiatov commented Mar 26, 2025

xtremekforever commented Mar 26, 2025 •

edited

Loading

MaxDesiatov commented Mar 26, 2025

MaxDesiatov commented Mar 28, 2025

xtremekforever commented Mar 28, 2025

MaxDesiatov commented Mar 29, 2025

MaxDesiatov left a comment

MaxDesiatov commented Mar 31, 2025

xtremekforever commented Mar 31, 2025 •

edited

Loading

MaxDesiatov commented Apr 7, 2025

MaxDesiatov commented Apr 16, 2025

euanh commented Apr 16, 2025

Support Debian 11 and Debian 12 distributions in the generator #203

Support Debian 11 and Debian 12 distributions in the generator #203

Conversation

xtremekforever commented Mar 24, 2025 • edited Loading

MaxDesiatov commented Mar 24, 2025

MaxDesiatov commented Mar 25, 2025

xtremekforever commented Mar 26, 2025

MaxDesiatov commented Mar 26, 2025

xtremekforever commented Mar 26, 2025 • edited Loading

MaxDesiatov commented Mar 26, 2025

MaxDesiatov commented Mar 28, 2025

xtremekforever commented Mar 28, 2025

MaxDesiatov commented Mar 29, 2025

MaxDesiatov left a comment

Choose a reason for hiding this comment

MaxDesiatov commented Mar 31, 2025

xtremekforever commented Mar 31, 2025 • edited Loading

MaxDesiatov commented Apr 7, 2025

MaxDesiatov commented Apr 16, 2025

euanh commented Apr 16, 2025

xtremekforever commented Mar 24, 2025 •

edited

Loading

xtremekforever commented Mar 26, 2025 •

edited

Loading

xtremekforever commented Mar 31, 2025 •

edited

Loading