Significant build regressions on `swift:6.0-noble` compared to `5.10-noble`

[MahdiBM]

Description

Absolutely massive Significant build regressions on linux , worsened by swift-testing

EDIT: Please also read my next comment which includes more info.

Environment

• GitHub Actions
• 8 CPU x 16GB RAM c7x EC2 Instance (we use RunsOn)
• Migrating from swift:5.10-noble to swift:6.0-noble
• Tried with and without using a previous .build cache. No behavior difference noticed at all.
• 200K-300K LOC project
• Updated to // swift-tools-version:6.0 in Package.swift
• All targets still have Swift 5 language mode enabled.
• No actual tests migrated to swift-testing just yet.

What Happened?

• Our Tests CI started getting stuck after the toolchain update.
• The CI machines were getting killed by AWS. Considering this has been a common problem specially in the Server Side Swift ecosystem, which also has been brought up for ages and numerous times, my first guess was that the machine is running out of RAM when building the Swift project.
• After a lot of tries, I noticed a c7x-family machine with 64 CPU x 128 GB RAM (8x larger than before) runs the tests as well as they were being run before, on swift 5.10.
• My first guess was that maybe Swift testing is causing an issue, so tried --disable-swift-testing.
• After that, our tests have been running on an only 2x larger machine, and the rest of the things are back to normal in tests CI.
• With this bigger machine, it still almost takes as long as before for the tests CI to complete (we run tests in parallel with 64 workers).
• So this means that not only Swift 6 has a significant build regression, but also swift-testing makes it go from significant to absolutely massive.
• This is borderline unusable for the Swift-Server ecosystem. I hope this problem requires more specific situations to happen that what meets the eye.
• Still trying things out with the deployment CI. even 2x CPU and 4x RAM isn't proving helpful even though I did throw the --disable-swift-testing flag into the mix.

Reproduction

Not sure.

Expected behavior

No regressions. Preferably even faster than previous Swift versions.

Environment

Mentioned above.

Additional information

No response

[grynspan]

Some notes, in no particular order (will update if I think of more):

• @stmontgomery's and my initial reaction is that this is probably not related to Swift Testing since this project hasn't added any Swift Testing tests yet.
• --disable-swift-testing does not affect the build. It did for a while with prerelease Swift 6 toolchains before we added Swift Testing to the toolchain, but it has no effect on the release toolchain. Only one build product is produced for a test target, and it's the same either way. It does affect whether or not a second test process is spun up after building to run Swift Testing tests.
• It is normal to see text output from Swift Testing when you run swift test even if you don't have any tests using it. The output is basically just saying "didn't find any Swift Testing tests, bye now." If you pass --disable-swift-testing, it suppresses spawning the process that looks for Swift Testing tests, which is why you don't see any output from it when you pass that flag.
• Swift Testing is available in Swift 5 language mode so long as you're using a Swift 6 toolchain. Language mode isn't a factor.
• @tayloraswift suggested this might be Ubuntu 24.04 Swift 5.10.1 release toolchain should not have assertions enabled #76091.

[MahdiBM]

To be clear:

• I don't think it was necessarily swift-testing doing something wrong. More-likely the compiler or SwiftPM mishandling it.
• Even without swift-testing there is a significant build time regression, when the build ends at all. The benchmarks below show it properly.

It's very sad to see the [Linux] build situation constantly getting worse despite us asking for faster and less-RAM-hungry builds for ages.
Based on the lots of user reports we consistently have over on the Vapor Discord server, a lot of deployment services have blocking problems with building Swift projects since their builders run out of RAM. As you can imagine, this can be a significant obstacle in deploying server-side Swift apps.
Just ask the SSWG folks @0xTim / @gwynne if you have any doubts.

Anyways, let's head to the benchmarks I did. I likely ran 500+ CI jobs in the past 48 hours ...
Note that we do use a couple of macros. All self-served. One of them lives in an external repo.
Image names are the exact official Docker image names.

Tests CI

• Tests CI machine sizes are compared to what we had for swift:5.10-noble: 8 CPU x 16GB RAM c7x EC2 Instance.
  • Other than "Same", the other machines are of the m7x family which have the higher RAM ratio.
• In the tests below, I tried both with and without --disable-swift-testing. Results are only marginally different.
• There are 3 variables in build times. 1- usage of cache 2- build-step time 3- tests-build/run-time.
  • The table below contains 2 of the variables.
  • If cache-usage and build-step-time have not changed but the run-time has changed, It means it was because of the tests-run/build time.
  • We build the package, then cache .build, and then run the tests in another step.
  • The tests-build/run-step does also build something more. I'm not sure what, but it seems like it does.
  • Build-step command: swift build --explicit-target-dependency-import-check error --build-tests
  • Tests-build/run-step command: swift test --enable-code-coverage --parallel --num-workers 64

Image	Machine Size	w/ cache total	w/ cache build	no cache total	no cache build
5.10-noble	Same	12m 53s	3m 7s	19m 37s	7m 34s
5.10-noble	2x RAM	13m 41s	3m 16s	21m 9s	7m 55s
5.10-noble	4x RAM 2x CPU	9m 19s	2m 40s	16m 37s	6m 18s
6.0-jammy	Same	15m 21s	3m 12s	24m 8s	9m 14s
6.0-jammy	2x RAM	16m 18s	3m 25s	25m 1s	9m 36s
6.0-jammy	4x RAM 2x CPU	9m 5s	2m 17s	17m 15s	6m 15s
6.0-noble	Same	17m 21s	5m 10s	23m 31s	9m 9s
6.0-noble	2x RAM	18m 32s	5m 42s	23m 34s	9m 1s
6.0-noble	4x RAM 2x CPU	9m 47s	2m 59s	18m 42s	6m 43s

(Side note: I didn't know using higher RAM could hurt?! I don't think it's a machine-type problem since the deployment builds below show the expected behavior of some performance improvements when having access to more RAM.)

Analyzing the results (Excluding the bigger 4x RAM 2x CPU machine):

• 30% worse tests-build/run-step performance on 5.10-noble compared to 6.0-jammy.
  • The exact tests-build/run-step numbers are not in the table. It would hover around 7-11 minutes.
• 60+% worse build performance in the build-step w/ cache, on 6.0-noble compared to 6.0-jammy.
• Overall all things considered, 40-50+% worse build performance when moving from 5.10-noble to 6.0-noble.
• A bit of the total time (which is not included in the comparisons above, but visible in the total time) goes back to our "cache .build" step which uses the actions/cache which apparently doesn't properly handle the big 2GB .build directories.

Deployment CI

• Deployment CI machine sizes are compared to what we had for swift:5.10-noble: 4 CPU x 8GB RAM c7x EC2 Instance.
  • Other than "Same", the other machines are of the m7x family which have the higher RAM ratio.
• Compared to the tests builds, deployment builds lack --build-tests, use jemalloc, and only build the certain product that will be deployed.
  • jemalloc might be why the behavior is worse than test builds?!

Image	Machine Size	w/ cache total	w/ cache build	no cache total	no cache build
5.10-noble	Same	4m 19s	2m 37s	13m 27s	9m 1s
5.10-noble	2x RAM	3m 31s	1m 49s	12m 29s	9m 12s
5.10-noble	4x RAM 2x CPU	4m 7s	2m 14s	9m 47s	6m 9s
6.0-jammy	Same	6m 25s	3m 7s	OOM	OOM
6.0-jammy	2x RAM	3m 40s	2m 0s	13m 57s	10m 14s
6.0-jammy	4x RAM 2x CPU	4m 39s	2m 39s	10m 7s	6m 32s
6.0-noble	Same	OOM	OOM	OOM	OOM
6.0-noble	2x RAM	OOM	OOM	OOM	OOM
6.0-noble	4x RAM 2x CPU	6m 12s	4m 10s	11m 6s	7m 21s

Only notable change when I moved our app to Swift 6 compiler, is that we have 3 executable targets which Swift was throwing errors about e.g. using @testable on those, so what I did is that I added 3 more .executableTarget targets, marked existing targets as just .target, and the .executableTargets only have like 5 lines in each to call the original target's entry-point.
This way we can still @testable import the original target.

You may ask: "Didn't you say you needed 8x larger machine to run the tests? How come this time they ran even on the same machine as you had before when using swift:5.10-noble?!"
That's a good question.
I tried multiple times yesterday, result of which is still available in GitHub Actions. I was even live-texting the results over on the Vapor Discord Server in a thread. I also sent a sample of such failing logs to @grynspan.
I also did check and the swift:6.0-noble images from yesterday and today are the same (have matching hashes).
I know there hasn't been any other changes since yesterday to the project, so not exactly sure why the tests are going with no getting-stuck today. I have no complaints about that. Though the deployment CI does still get stuck.

So:
Yesterday I reported that there seem to be 2 problems, one a general build time regression, second one a massive regression when not using --disable-swift-testing. Today I'm unable to reproduce the latter.
Something's really up. And It definitely isn't the Swift code since that hasn't changed.
I don't think it's the AWS machines we use either, those are just standard EC2 instances.
The only thing I can think of is our cache usage. Maybe Swift 6 was using Swift 5's caches and didn't like that at all. But even then, I remember testing both with and without cache. I've set the CI to disable usage of caching when you rerun a job, so it was trivial to test both scenarios.

Worth mentioning, when the build is stuck, I consistently see a sequence of logs like this, containing Write Objects.LinkFileList around the end:

[6944/6948] Wrapping AST for MyLib for debugging
[6946/6950] Compiling MyExec Entrypoint.swift
[6947/6950] Emitting module MyExec
[6948/6951] Wrapping AST for MyExec for debugging
[6949/6951] Write Objects.LinkFileList

[gwynne]

This almost starts to sound like a recurrence of the infamous linker RAM usage problem due to huge command lines with repeated libraries. @al45tair Is there any chance we're still failing to deduplicate linker command lines fully?

[finagolfin]

Is there any chance we're still failing to deduplicate linker command lines fully?

Yes, I pinged you on that last month, but never got a response. A fix was just merged into release/6.0 in #76504, so it will not be released until 6.0.2 or 6.0.3.

@jmschonfeld or @shahmishal, can that fix be prioritized to get into the next patch release?

[finagolfin]

@MahdiBM, thanks for all the build info. Do you do any CI builds of the 6.0 snapshot toolchains before the final release? That would help find and stop build regressions like this when they happen, rather than being surprised on the final release.

If you can, I'd like to know how an earlier 6.0 July 19 snapshot toolchain build for jammy does on these same CI runs of yours. That might help figure out the regression, particularly if you compare it to the next July 21 build of the 6.0 toolchain.

[MahdiBM]

@MahdiBM, thanks for all the build info. Do you do any CI builds of the 6.0 snapshot toolchains before the final release? That would help find and stop build regressions like this when they happen, rather than being surprised on the final release.

@finagolfin this is an "executable" work project, not a public library, so we don't test on multiple Swift versions.
I think it would be possible for us though to do such a thing on the next nightly images. Not a bad idea to catch these kinds of issues.
The only problem is that how reliable the nightly images are? Do they not have assertions and such enabled which makes the build slower? How can I trust the results? Do I need to run current-nightly and next-ver-nightly and compare those?
I can set up a weekly job perhaps 🤔

If you can, I'd like to know how an earlier 6.0 July 19 snapshot toolchain build for jammy does on these same CI runs of yours. That might help figure out the regression, particularly if you compare it to the next July 21 build of the 6.0 toolchain.

We just use docker images in CI. To be clear, the image names above are exact Docker image names that we use (added this explanation to the comment). I haven't tried or figured out manually using nightly images, although I imagine I could just use swiftly to set up the machine with the specific nightly toolchain and there should be little problems. It will make the different benchmarks diverge a bit though in terms of environment / setup and all.
Preferably, I should be able to just try a nightly image if you think that'll be helpful, just let me know what exact nightly image (the image identifier (hash) and tag ?).
Or I can try a newer 6.0 nightly image that does contain the linker RAM usage fix? If such already exists.

[MahdiBM]

Another visible issue is the noble vs jammy difference ... I don't think I could have caught that even if we were running CIs on nightly images, considering Swift 6 just very recently got a noble image.

Not sure where that comes from. Any ideas?

[MahdiBM]

I would guess that even on 5.10, the jammy images would behave better than the nobles. Though haven't tested that.

[gwynne]

My guess is that difference comes from the updated glibc in noble; when going from bionic to jammy we saw a significant improvement in malloc behavior for that reason, I wouldn't be too surprised if noble regressed some.

[finagolfin]

The only problem is that how reliable the nightly images are? Do they not have assertions and such enabled which makes the build slower? How can I trust the results?

I don't know. In my Android CI, the latest Swift release builds the same code about 70-80% faster than the development snapshot toolchains. But you'd be looking for regressions in relative build time, so those branch differences shouldn't matter.

Do I need to run current-nightly and next-ver-nightly and compare those?

I'd simply build with the snapshots of the next release, eg 6.1 once that's branched, and look for regressions in build time with the almost-daily snapshots.

I can set up a weekly job perhaps

The CI tags snapshots and provides official builds a couple times a week: I'd set it up to run whenever one of those drops.

Preferably, I should be able to just try a nightly image if you think that'll be helpful, just let me know what exact nightly image (the image identifier (hash) and tag ?).

I don't use Docker, so don't know anything about its identifiers, but I presume the 6.0 snapshot tag dates I listed should identify the right images.

Or I can try a newer 6.0 nightly image that does contain the linker RAM usage fix? If such already exists.

Not yet. The fix was added to trunk a couple weeks ago, so you could try the Sep. 4 or latest 6.1 snapshot build with it and compare to the Aug. 29 build without it.

You may also want to talk to @ktoso and the SSWG about what kind of toolchain benchmarking exists to catch these issues on linux and what needs to be done to either start or augment it.

[tbkka]

If you can log into the build machine and watch memory usage with top, it should be a lot clearer what's going on. There's a big difference between linker and compiler, as noted above. There's also a big difference between "the compiler uses too much memory" and "the build system is running too many compilers at the same time."

[MahdiBM]

@tbkka I can ssh into the containers since RunsOn provides such a feature, but as you already noticed, I'm not well-versed in compiler/build-system workings. I can use top but not sure what or how to derive conclusions about where the problem is.

It does appear @gwynne was on point though, about linker issues.