Rustup fails with os error 10054 on a new Windows 11 machine

[fh-igd-mueller-roemer]

Verification

• I searched for recent similar issues at https://github.com/rust-lang/rustup/issues?q=is%3Aissue+is%3Aopen%2Cclosed and found no duplicates.
• I am on the latest version of Rustup according to https://github.com/rust-lang/rustup/tags and am still able to reproduce my issue.

Problem

I am on a new Windows 11 machine, with the most current version of rustup (installed two days ago). Running rustup update fails approximately 3 times out of 4 with an os error 10054.

Steps

1. Run rustup update multiple times, oberve how it fails 3 times out of 4

Possible Solution(s)

No response

Notes

I am also encountering the likely related cargo issue rust-lang/cargo#13457 on the same machine. The workaround in rust-lang/cargo#13457 (comment) resolved that issue for me.

Rustup version

rustup 1.27.0 (bbb9276d2 2024-03-08)

Installed toolchains

Default host: x86_64-pc-windows-msvc
rustup home:  <...>\.rustup

stable-x86_64-pc-windows-msvc (default)
rustc 1.77.2 (25ef9e3d8 2024-04-09)

OS version

Windows 11 Enterprise 23H2 22631.3447

[fh-igd-mueller-roemer]

Full output of a failing call to rustup update

info: syncing channel updates for 'stable-x86_64-pc-windows-msvc'
error: could not download file from 'https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256' to '<...>\.rustup\tmp\o2oe59asu0iwsug6_file'
info: checking for self-update
error: could not download file from 'https://static.rust-lang.org/rustup/release-stable.toml' to '<...>\AppData\Local\Temp\rustup-updatevmhcHP\release-stable.toml': failed to make network request: error sending request for url (https://static.rust-lang.org/rustup/release-stable.toml):
error trying to connect: An existing connection was forcibly closed by the remote host. (os error 10054): error trying to connect: An existing connection was forcibly closed by the remote host. (os error 10054): An existing connection was forcibly closed by the remote host. (os error 10054)

[djc]

Does it work if you set a RUSTUP_USE_CURL=1 environment variable?

[fh-igd-mueller-roemer]

It appears to work now, no matter if I set RUSTUP_USE_CURL to 0 or 1. curl https://static.rust-lang.org/dist/channel-rust-stable.toml.sha256 seems to work consistently now, while curl https://index.crates.io/config.json still fails on most attempts. There seems to be some interaction between Windows networking and the servers that Cargo and Rustup use.

[rami3l]

@fh-igd-mueller-roemer Thanks for the report anyway! Looks like a @rust-lang/infra problem though...

[jdno]

Oof. 😮‍💨 Thanks for reporting this, @fh-igd-mueller-roemer.

The reason why it might work sometimes is that we split traffic for static.rust-lang.org between AWS CloudFront and Fastly, while index.rust-lang.org is only served through AWS CloudFront.

Can you test my theory that this is a problem with one of the two by explicitly downloading the file from both?

curl https://cloudfront-static.rust-lang.org/dist/channel-rust-stable.toml.sha256

curl https://fastly-static.rust-lang.org/dist/channel-rust-stable.toml.sha256

I would suspect that the download through Fastly works, while downloading from CloudFront fails. If it's the case that one of them fails, it would also be interesting to test whether the issue exists for both IPv4 and IPv6. You can force the protocol version with the -4 and -6 flags for curl.

(Off-topic: I immediately recognized the background in your profile picture. I used to work for the IGD as well. 😄)

[fh-igd-mueller-roemer]

@jdno You are correct. The Cloudfront-link failed 4 times out of 5, the Fastly-link worked fine on every attempt. The Cloudfront-link also works with -4 added, so we can narrow it down to Cloudfront and IPv6. Oddly enough, Cloudfront works fine on my Windows 10 machine and on an older Windows 11 machine.

(Off-topic: yes, although a different background was used for our new photos - I need to swap this one 😀 We probably crossed paths at some point since I've been at IGD since 2011)

[rami3l]

@jdno curl -4 can solve the sh.rustup.rs issue, but for this situation regarding static.rust-lang.org in particular, is there a reasonable workaround? (For example, is setting the environment variable RUSTUP_UPDATE_ROOT=https://fastly-static.rust-lang.org/rustup a valid workaround? I haven't tested it myself since I cannot reproduce this issue...)

I'm asking for @nopeless (#3797 (comment)) since I don't know anything about the deployment of these websites :|

But if @fh-igd-mueller-roemer's experiences apply, maybe we should have a closer look at CloudFront IPv6.

[fh-igd-mueller-roemer]

@rami3l I don't think setting RUSTUP_UPDATE_ROOT=https://fastly-static.rust-lang.org/rustup is sufficient as a workaround, but that you need to set RUSTUP_DIST_SERVER=https://fastly-static.rust-lang.org as well. If I set either to the Cloudfront equivalent it begins failing again.

Maybe also check in with the people encountering the linked Cargo issue. There were (at least) two more people there with issues on Windows 11

[nopeless]

@fh-igd-mueller-roemer what do you think about the solution listed in #3797?

[ChrisDenton]

Maybe that could be done as a last resort? I.e. if the retries all fail then it tries once more with ipv4 only?

[fh-igd-mueller-roemer]

@nopeless doesn't seem like a particularly clean solution, but it would work until upstream (either Cloudfront or the Windows distribution of curl, depending on who's at fault here) manages to resolve the issue. It could be an environment variable that sets the IP protocol version, since it's only a workaround

[kLiHz]

If my understandings are correct, Rust-lang.org are currently using both Fastly and AWS. I'd like to know if this means rust-lang.org will migrate to a new CDN?

(And speaking of CDN, I wrote a reverse proxy with Cloudflare Workers. I hope this is not illegal by the way. Anyone can deploy their own with this gist, although Cloudflare doesn't support disabling IPv6 support for free plans. Will Cloudflare be considered since its object storage is only priced by requests (edit: and storage size) and there's no bandwidth fee? I'm just a random developer and saying so because I think this might reduce rust-lang.org's running cost to some degree.)

[rami3l]

@kLiHz Thanks for your interest! However, as I mentioned above, the website and its deployment is out of the Rustup Project's control. Maybe the t-infra channel on the official Zulip server would be a better place to continue this discussion?

---

Oops, there was a misclick below...