Skip to content

x/net/http2: Server rejects Transport with ErrCodeProtocol stream error #47635

Open
@bradfitz

Description

@bradfitz

We're running a Go 1.17rc2 httputil.ReverseProxy in front of a Go go1.17rc2 net/http.Server.

We were seeing weird ErrCodeProtocol stream errors, so we stopped using the bundled x/net/http2 and switched both to using x/net/http2 explicitly (with http2.ConfigureTransports and http2.ConfigureServer, so the bundled one isn't used).

Our x/net git version is golang/net@aaa1db6, which is today's current master.

The logs we were getting weren't initially enough to tell us where the problem was:

2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293155; PROTOCOL_ERROR
2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293157; PROTOCOL_ERROR

But after hacking it up locally to add some expvar counters on which paths in http2.Server were returning the protocol error, we found it was here:

https://github.com/golang/net/blob/aaa1db679c0d7765d2b1cb1f92cac8ebf4d94c53/http2/server.go#L1835

// http://tools.ietf.org/html/rfc7540#section-5.1.2
// [...] Endpoints MUST NOT exceed the limit set by their peer. An
// endpoint that receives a HEADERS frame that causes their
// advertised concurrent stream limit to be exceeded MUST treat
// this as a stream error (Section 5.4.2) of type PROTOCOL_ERROR
// or REFUSED_STREAM.
if sc.curClientStreams+1 > sc.advMaxStreams {
        if sc.unackedSettings == 0 {
                // They should know better.
                return streamError(id, ErrCodeProtocol)
        }

So, the Go http2 client (Transport) and/or server is getting its concurrent/max streams accounting wrong.

Perhaps interesting: the backend (the http2.Server that the ReverseProxy's Transport is hitting) has extremely long-running http.Handlers (like: days or weeks longs) alongside many short ones. That might explain why this bug hasn't been reported by more people.

I see no recent commits that suggest this is a regression. This server is new, so I also can't say whether it worked previously.

/cc @neild @bcmills @fraenkel @dmitshur @dsnet @josharian

Activity

fraenkel

fraenkel commented on Aug 11, 2021

@fraenkel
Contributor

possible? #42777

bradfitz

bradfitz commented on Aug 11, 2021

@bradfitz
ContributorAuthor

Ah, yes. That's probably what we're hitting.

added this to the Backlog milestone on Aug 13, 2021
added
NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.
on Aug 13, 2021
dmitshur

dmitshur commented on Aug 13, 2021

@dmitshur
Member

CC @neild.

gopherbot

gopherbot commented on Sep 3, 2021

@gopherbot
Contributor

Change https://golang.org/cl/347033 mentions this issue: http2: make Transport not reuse conns after a stream protocol error

gopherbot

gopherbot commented on Oct 19, 2021

@gopherbot
Contributor

Change https://golang.org/cl/356978 mentions this issue: [internal-branch.go1.16-vendor] http2: make Transport not reuse conns after a stream protocol error

gopherbot

gopherbot commented on Oct 21, 2021

@gopherbot
Contributor

Change https://golang.org/cl/357673 mentions this issue: [internal-branch.go1.17-vendor] http2: make Transport not reuse conns after a stream protocol error

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    NeedsInvestigationSomeone must examine and confirm this is a valid issue and not a duplicate of an existing one.

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

      Development

      No branches or pull requests

        Participants

        @bradfitz@fraenkel@dmitshur@gopherbot

        Issue actions

          x/net/http2: Server rejects Transport with ErrCodeProtocol stream error · Issue #47635 · golang/go