net/http: deadlock in TestMissingStatusNoPanic on release-branch.go1.15 branch

[cagedmantis]

2505e26/windows-amd64-longtest
1035bb9/windows-amd64-longtest
b35c38a/windows-amd64-longtest
636a6d14/windows-amd64-longtest

May be related to ##42942

CC CC @alexbrainman @fraenkel @neild @bradfitz @golang/release

[fraenkel]

@cagedmantis I think this is the wrong test. While the error occurs on TestCancelRquestWhenSharing after a few seconds, the moire obvious one is TestMissingStatusNoPanice which has been stuck for 14m.

goroutine 17971 [chan receive, 14 minutes]:
net/http_test.TestMissingStatusNoPanic(0xc00060e000)
	C:/workdir/go/src/net/http/transport_test.go:5320 +0x450

[bcmills]

https://storage.googleapis.com/go-build-log/2ba9aff9/windows-amd64-longtest_d50badd6.log (a TryBot on CL 319710)

[gopherbot]

Change https://golang.org/cl/319275 mentions this issue: net/http: prevent infinite wait during TestMissingStatusNoPanic

[bcmills]

Also https://storage.googleapis.com/go-build-log/5200af43/windows-amd64-longtest_e8416262.log (a TryBot on CL 319709).

[fraenkel]

All of the failures are due to TestMissingStatusNoPanic so far. My fix above will just expose the real issue which we currently aren't seeing.

[gopherbot]

Change https://golang.org/cl/320189 mentions this issue: [release-branch.go1.15] net/http: prevent infinite wait during TestMissingStatusNoPanic

[gopherbot]

Change https://golang.org/cl/320190 mentions this issue: [release-branch.go1.16] net/http: prevent infinite wait during TestMissingStatusNoPanic

[fraenkel]

On the plus side, we now see the actual issue(s) in Go 1.15 failures.

[fraenkel]

I see at least one of the issues and may be the actual problem....
Notice its all using the same port.

TestTransportPersistConnReadLoopEOF is failing but not sending a bogus conn to the connc channel.

[fraenkel]

The above is incorrect, the testcase is fine.
The error being reported happens when the system is underload and usually out of resources. One can tweak the TCP settings to reduce the failure.

There is also this https://go.googlesource.com/benchmarks/+/refs/heads/master/http/http.go#42
I guess something got fixed/improved in 1.16+.

[gopherbot]

Closed by merging e3c9537 to release-branch.go1.15.

[neild]

The root cause of the failures is port exhaustion, which hasn't been addressed, but the deadlock in TestMissingStatusNoPanic should be fixed.

I haven't been able to reproduce the port exhaustion failures manually after running go test net a number of times on a windows-amd64-longtest builder today. Possibly I'm not taking the right steps to reproduce the problem, or possibly whatever was causing that builder to run out of ports has been addressed. I'm going to tentatively assume the latter unless these failures show up again.

[fraenkel]

I think it might be the combination of things. One test that didn't finish, holding up others than haven't done their cleanup.
It was a bit weird that it was only occurring in one release.

[dmitshur]

The root cause of the failures is port exhaustion, which hasn't been addressed, but the deadlock in TestMissingStatusNoPanic should be fixed.

The port exhaustion issue is now also fixed (by CL 339829). The windows-amd64-longtest builder is looking healthy on release-branch.go1.15, and I don't think anything more is left here, so closing.

It was a bit weird that it was only occurring in one release.

I think it was an unfortunate coincidence to do with runtime scheduling. On 1.15 the TestCancelRequestWhenSharingConnection test was managing to reach the 32k ephemeral port limit during its 10 seconds, but on 1.16+ it was only using up around 30k~, leaving just barely enough room for the rest of net/http tests to succeed. Rerunning the same test in quick succession would fail on 1.16+ as well.