net/http: connCount underflow panic with MaxConnsPerHost

[andybalholm]

I'm splitting this off from #34941 because it seems to be a separate issue, although with the same symptom.

$ go version
go version go1.14 darwin/amd64

I compiled this binary on macOS, but I'm running it on Linux amd64.

Here is the stack trace:

panic: net/http: internal error: connCount underflow

goroutine 2114 [running]:
net/http.(*Transport).decConnsPerHost(0x1094ca0, 0x0, 0x0, 0xc001f110b5, 0x4, 0xc001a27320, 0x14, 0x0)
		/usr/local/go/src/net/http/transport.go:1391 +0x590
net/http.(*persistConn).closeLocked(0xc0000cd320, 0xb8a620, 0xc000096b60)
		/usr/local/go/src/net/http/transport.go:2584 +0xf5
net/http.(*persistConn).close(0xc0000cd320, 0xb8a620, 0xc000096b60)
		/usr/local/go/src/net/http/transport.go:2574 +0x83
net/http.(*persistConn).readLoop.func1(0xc0000cd320, 0xc0073b4d60)
		/usr/local/go/src/net/http/transport.go:1946 +0x41
net/http.(*persistConn).readLoop(0xc0000cd320)
		/usr/local/go/src/net/http/transport.go:2122 +0x12ae
created by net/http.(*Transport).dialConn
		/usr/local/go/src/net/http/transport.go:1647 +0xc56

The usage is in a forward proxy server (github.com/andybalholm/redwood). We have about 2000 users running through the server I have been watching these errors on the most closely. So the usage is heavy, and probably about as close to random as you are likely to find.

I thought that switching to two Transports (one for HTTP/1.1 and one for HTTP/2) instead of one http.Transport that was configured to automatically switch protocols would get rid of the panics. But it didn't seem to help. (So it seems that the panic doesn't depend on having HTTP/2 enabled on the Transport.)

I had Transport.MaxConnsPerHost set to 8. Setting it to 0 (unlimited) made the panics go away.

[andybalholm]

I just noticed that the binary that stack trace is from was compiled with Go 1.14, not Go 1.14.1 like I had thought originally. But we got the same panics with a binary compiled on Linux with Go 1.14.1.

[andybons]

@bradfitz @kardianos @kevinburke

[bcmills]

@andybalholm, could you try testing with a go build from head, or patching in CL 220905? I wonder if this is another symptom of the bug that was fixed there.

[andybalholm]

Building with

go version devel +620208790e Wed Apr 1 19:32:57 2020 +0000 darwin/amd64

I still get the panic:

panic: net/http: internal error: connCount underflow

goroutine 67124 [running]:
net/http.(*Transport).decConnsPerHost(0x108df80, 0x0, 0x0, 0xc00b2800e4, 0x4, 0xc002061900, 0x17, 0x0)
	/usr/local/go/src/net/http/transport.go:1388 +0x590
net/http.(*persistConn).closeLocked(0xc00af806c0, 0xb847e0, 0xc00006cb60)
	/usr/local/go/src/net/http/transport.go:2586 +0xf5
net/http.(*persistConn).close(0xc00af806c0, 0xb847e0, 0xc00006cb60)
	/usr/local/go/src/net/http/transport.go:2576 +0x83
net/http.(*persistConn).closeConnIfStillIdle(0xc00af806c0)
	/usr/local/go/src/net/http/transport.go:1890 +0xcb
created by time.goFunc
	/usr/local/go/src/time/sleep.go:169 +0x44

[fraenkel]

@bcmills This is an http request issue not http2.

[bcmills]

@andybalholm, could you share some more detail about your usage? (Any unusual Transport customizations, proxies, or anything like that?)

[rsc]

If you have a bunch of these panic traces, are there any that don't have that stack sequence (decConnsPerHost called from closeLocked called from close)?

[andybalholm]

@bcmills Here is my transport:

var dialer = &net.Dialer{
	Timeout:   30 * time.Second,
	KeepAlive: 30 * time.Second,
	DualStack: true,
}

var httpTransport = &http.Transport{
	Proxy:                 http.ProxyFromEnvironment,
	DialContext:           dialer.DialContext,
	TLSHandshakeTimeout:   10 * time.Second,
	ExpectContinueTimeout: 1 * time.Second,
}

The environment doesn't specify a proxy. I call http2.ConfigureTransport(httpTransport) elsewhere in the code. (But not making that call doesn't fix the problem.)