net/http: connection leak with TLS and reading Body

[ssajja]

go version go1.10.1 darwin/amd64
darwin, amd64

package main

import (
"fmt"
"net/http"
"net/url"
"io/ioutil"
"os"
"crypto/tls"
)

func main() {
    for {
        test_func()
    }
}

func test_func() {
    u, _ := url.ParseRequestURI("https://golang.org")
    urlStr := fmt.Sprintf("%v", u)
    req, _ := http.NewRequest("GET", urlStr, nil)

    tr := &http.Transport{TLSClientConfig: &tls.Config{InsecureSkipVerify: true}}
    cli := &http.Client{Transport: tr}
    response, err := cli.Do(req)

    if err != nil {
        fmt.Printf("%s", err)
        os.Exit(1)
    }

    defer response.Body.Close()

    _, err = ioutil.ReadAll(response.Body)
    if err != nil {
        fmt.Printf("%s", err)
        os.Exit(1)
    }
}

Above program results in a large number (observed by checking lsof -p <process_id>) of
TCP connections in established state, eventually running out of file descriptors.

I see this issue only when I use TLS and do a ioutil.ReadAll.
Skipping either one of them results in a single established TCP connection.
The issue also goes away If I do req.Close = true after req, _ := http.NewRequest("GET", urlStr, nil)

[FiloSottile]

I can reproduce. It does not happen if the end of the body is not reached, which suggests the leak happens in the connection pooling logic. Connections are not reused and MaxIdleConnsPerHost is somehow ignored.

Also happens without InsecureSkipVerify, ParseRequestURI, H/2 (GODEBUG=http2client=0) or HTTPS.

It does not happen with DefaultTransport, or calling CloseIdleConnections.

[FiloSottile]

Oh, didn't realize the http.Transport and http.Client are created inside the loop.

http.Transport keeps a pool of open connections for reuse:

By default, Transport caches connections for future re-use.

So if you create a new one every iteration it won't know about all the other ones and it will just hold onto the connection for future reuse (which never happens).

You should make a single Client and reuse it for each iteration:

Transports should be reused instead of created as needed.

The Client's Transport typically has internal state (cached TCP connections), so Clients should be reused instead of created as needed. Clients are safe for concurrent use by multiple goroutines.

[ssajja]

Thank you. That fixed the issue.