Description
What version of Go are you using (go version
)?
$ go version go version go1.14.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env
)?
go env
Output
$ go env set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\Administrator\AppData\Local\go-build set GOENV=C:\Users\Administrator\AppData\Roaming\go\env set GOEXE=.exe set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=D:\wxgo set GOPRIVATE= set GOPROXY=https://goproxy.io,direct set GOROOT=D:\Go set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=D:\Go\pkg\tool\windows_amd64 set GCCGO=gccgo set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=1 set GOMOD= set CGO_CFLAGS=-g -O2 set CGO_CPPFLAGS= set CGO_CXXFLAGS=-g -O2 set CGO_FFLAGS=-g -O2 set CGO_LDFLAGS=-g -O2 set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessag e-length=0 -fdebug-prefix-map=C:\Users\ADMINI~1\AppData\Local\Temp\go-build74128 8550=/tmp/go-build -gno-record-gcc-switches
What did you do?
- I created a proxy and a restfulapi server
- I used postman made many post requests with body request to server through the proxy(proxy and server run in the same vm,server runed used docker,They use docker0 to connect),and server response body is larger than 4K
- all requests use http/1.1
- OS:Centos7.4
What did you expect to see?
post request sucessed and there is only one persistent connection
What did you see instead?
the following log appears in proxy .
httputil: ReverseProxy read error during body copy: read tcp 172.17.0.1:39810->172.17.0.2:8082: use of closed network connection
many TIME_WAIT conntecions appears .
Everything was ok in any of the the following scenarios:
- Post request without body
- GET request with same response
- Response body less than 4K
- Do not use Keep-Alive
- Server response not write "Content-Length" in header
- Use NGINX instead of my proxy
- Server runed in a vm
PROXY
Output
package mainimport (
"flag"
"net/http"
"net/http/httputil"
"net/url"
"time"
)var burl = flag.String("u", "", "backend address")
func buildProxy(target *url.URL) *httputil.ReverseProxy {
flushInterval := time.Duration(100 * time.Millisecond)
passHost := true
passHostHeader := &passHost
targetQuery := target.RawQuery
proxy := &httputil.ReverseProxy{
Director: func(req *http.Request) {
req.URL.Path = target.Path
req.URL.Host = target.Host
req.URL.Scheme = target.Schemereq.URL.RawPath = target.RawPath if targetQuery == "" || req.URL.RawQuery == "" { req.URL.RawQuery = targetQuery + req.URL.RawQuery } else { req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery } req.RequestURI = "" req.Proto = "HTTP/1.1" req.ProtoMajor = 1 req.ProtoMinor = 1 if passHostHeader != nil && !*passHostHeader { req.Host = req.URL.Host } }, Transport: http.DefaultTransport, FlushInterval: flushInterval, } return proxy
}
func handler(p *httputil.ReverseProxy) func(http.ResponseWriter, *http.Request) {
return func(w http.ResponseWriter, r *http.Request) {p.ServeHTTP(w, r) }
}
func main() {
flag.Parse()
remote, err := url.ParseRequestURI("http://" + *burl)
if err != nil {
panic(err)
}proxy := buildProxy(remote) http.HandleFunc("/", handler(proxy)) err = http.ListenAndServe(":8081", nil) if err != nil { panic(err) }
}
SERVER
Output
package mainimport (
"fmt"
"strconv"
"github.com/gin-gonic/gin"
)type myJson struct {
Name stringjson:"name"
}func createBigString(name string) string {
var bs string for i := 1; i <= 500; i++ { bs += name + "abcdefghij" } return bs
}
func main() {
gin.SetMode(gin.ReleaseMode)
router := gin.Default()router.POST("/test/post", func(c *gin.Context) { var json myJson c.ShouldBindJSON(&json) myName := createBigString(json.Name) lenBody := len(myName) + 11 fmt.Println(lenBody) c.Header("Content-Length", strconv.Itoa(lenBody)) c.JSON(200, gin.H{"name": myName}) }) router.Run(":8082")
}
Activity
[-]ReverseProxy read error during body copy[/-][+]ReverseProxy read error during body copy use of closed network connection[/+][-]ReverseProxy read error during body copy use of closed network connection[/-][+]net/http/httputil:ReverseProxy read error during body copy use of closed network connection[/+][-]net/http/httputil:ReverseProxy read error during body copy use of closed network connection[/-][+]net/http/httputil: ReverseProxy read error during body copy use of closed network connection[/+]dmitshur commentedon Aug 17, 2020
/cc @bradfitz per owners.
coolaj86 commentedon Nov 13, 2020
I've got something that I think could be related to this - also on Windows, but I'm getting a panic as well:
This happened after a local application that communicates with this process had been running for a few weeks successfully.
davecheney commentedon Nov 13, 2020
This looks like memory corruption. Have you tried running your program under the race detector? See https://blog.golang.org/race-detector .
markusthoemmes commentedon Nov 25, 2020
I'm hitting the very same symptoms on Golang 1.15.5.
I also whipped up a quick reproducer that reliably fails on my machine:
`go env` output
Reproducer
markusthoemmes commentedon Dec 7, 2020
Any chance we can get some guidance/help investigating this? @bradfitz @rsc @empijei (per OWNERS)
empijei commentedon Dec 16, 2020
cc/ @FiloSottile this looks like #34902, which you worked on. WDYT?
This also looks a lot like #15527 but I didn't find the race at a first glance. Running the code suggested by @markusthoemmes a few times reliably repros the issue, but even if ran with
-race
the detector does not pick up the race so it's not a trivial one.The offending call seems to be
go/src/net/http/httputil/reverseproxy.go
Line 328 in 75e16f5
which calls
go/src/net/http/httputil/reverseproxy.go
Line 439 in 75e16f5
which encounters this error here:
go/src/net/http/httputil/reverseproxy.go
Line 451 in 75e16f5
If we look at the first snippet, data is copied from
res.Body
torw
.My best guess here is that while we are reading from
res.Body
thistransport.RoundTrip
call might still be writing to the outgoing request.This could be caused by a persistent connection doing some work in the background, but it would require some more in-depth analysis to validate.
duglin commentedon Dec 17, 2020
Not 100% sure it's related but it does appear the reverseproxy has issues with raw tcp stuff - see this: knative/serving#6146 (comment)
25 remaining items