Description
We're running a Go 1.17rc2 httputil.ReverseProxy in front of a Go go1.17rc2 net/http.Server.
We were seeing weird ErrCodeProtocol stream errors, so we stopped using the bundled x/net/http2 and switched both to using x/net/http2 explicitly (with http2.ConfigureTransports and http2.ConfigureServer, so the bundled one isn't used).
Our x/net git version is golang/net@aaa1db6, which is today's current master.
The logs we were getting weren't initially enough to tell us where the problem was:
2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293155; PROTOCOL_ERROR
2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293157; PROTOCOL_ERROR
But after hacking it up locally to add some expvar counters on which paths in http2.Server were returning the protocol error, we found it was here:
// http://tools.ietf.org/html/rfc7540#section-5.1.2
// [...] Endpoints MUST NOT exceed the limit set by their peer. An
// endpoint that receives a HEADERS frame that causes their
// advertised concurrent stream limit to be exceeded MUST treat
// this as a stream error (Section 5.4.2) of type PROTOCOL_ERROR
// or REFUSED_STREAM.
if sc.curClientStreams+1 > sc.advMaxStreams {
if sc.unackedSettings == 0 {
// They should know better.
return streamError(id, ErrCodeProtocol)
}
So, the Go http2 client (Transport) and/or server is getting its concurrent/max streams accounting wrong.
Perhaps interesting: the backend (the http2.Server that the ReverseProxy's Transport is hitting) has extremely long-running http.Handlers (like: days or weeks longs) alongside many short ones. That might explain why this bug hasn't been reported by more people.
I see no recent commits that suggest this is a regression. This server is new, so I also can't say whether it worked previously.
Activity
fraenkel commentedon Aug 11, 2021
possible? #42777
bradfitz commentedon Aug 11, 2021
Ah, yes. That's probably what we're hitting.
http2: don't reuse Transport conns after seeing stream protocol errors
http2: don't reuse Transport conns after seeing stream protocol errors
dmitshur commentedon Aug 13, 2021
CC @neild.
gopherbot commentedon Sep 3, 2021
Change https://golang.org/cl/347033 mentions this issue:
http2: make Transport not reuse conns after a stream protocol error
http2: make Transport not reuse conns after a stream protocol error
gopherbot commentedon Oct 19, 2021
Change https://golang.org/cl/356978 mentions this issue:
[internal-branch.go1.16-vendor] http2: make Transport not reuse conns after a stream protocol error
gopherbot commentedon Oct 21, 2021
Change https://golang.org/cl/357673 mentions this issue:
[internal-branch.go1.17-vendor] http2: make Transport not reuse conns after a stream protocol error
[internal-branch.go1.17-vendor] http2: make Transport not reuse conns…
[internal-branch.go1.16-vendor] http2: make Transport not reuse conns…
http2: make Transport not reuse conns after a stream protocol error
[internal-branch.go1.17-vendor] http2: make Transport not reuse conns…