crypto/tls: handshake failure negotiating TLS 1.3 with JDK11

[houseofcat]

What version of Go are you using (go version)?

go version go1.13.7 windows/amd64

Does this issue reproduce with the latest release?

Not tested on 1.14.x

What operating system and processor architecture are you using (go env)?

Windows 10 (amd64) and Docker/CentOS7 (amd64)

go env Output

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\thyams\AppData\Local\go-build
set GOENV=C:\Users\thyams\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\thyams\go
set GOPRIVATE=
set GOPROXY=direct
set GOROOT=c:\go
set GOSUMDB=off
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=C:\******\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=C:\Users\thyams\AppData\Local\Temp\go-build099847662=/tmp/go-build -gno-record-gcc-switches

What did you do?

HTTPClient that have default values for the HTTPTransport TLSClientConfig don't appear to be negotiating handshake downwards (using a lower supported protocol version) if receiving client/system/web.api doesn't support the max version we have configured.

Default Example:

	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: false,
		MinVersion:         tls.VersionTLS10,
		MaxVersion:         tls.VersionTLS13, 
	}

Communicating with a client whose max protocol level is tls.VersionTLS12 (771) will trigger a remote error: tls: handshake failure response.

Workaround
If I manipulate the transport.TLSClientConfig

	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: false,
		MinVersion:         tls.VersionTLS10,
		MaxVersion:         tls.VersionTLS12, // Lowered
	}

Communication resumes. I, of course, apologize if I am mistaken in my understanding of the functionality in the HTTPTransport.TLSClientConfig.

What did you expect to see?

I expected to see proper HTTP Status of 200 for a call succesful through curl or postman.

// BuildHTTPTransport creates a basic HTTP transport with default timeouts and can skip TLS verification check.
func BuildHTTPTransport(tlsInsecure bool, proxy func(*http.Request) (*url.URL, error)) *http.Transport {

	transport := &http.Transport{
		Proxy: proxy, // Can Be Nil
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
			DualStack: true,
		}).DialContext,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   100, // Default was 2
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: time.Second,
	}

	transport.TLSClientConfig = &tls.Config{
		InsecureSkipVerify: tlsInsecure,
		MinVersion:         tls.VersionTLS10,
		MaxVersion:         tls.VersionTLS12, // Default is TLS13
	}

	return transport
}

What did you see instead?

remote error: tls: handshake failure

[toothrot]

/cc @bradfitz @rsc

[toothrot]

Also /cc @FiloSottile for crypto/tls.

[FiloSottile]

This is not supposed to happen, and it could be a bug on the server side as well.

Can you give us some details on what the server is? Any logs from it would also help.

Also, to ensure this has nothing to do with net/http, can you check that tls.Dial also fails?

[houseofcat]

I will write a local test to confirm with dial, I have remote server details just not on me. Will update shortly. Thanks guys.

[houseofcat]

The remote system is a Java based web.api, I can get more details from the author later. Here is what I do know off the top of my head. I switch to go version go1.13.7 windows/amd64 about 5 days ago and began dev testing my software today where I identified a problem that had me tearing my hair out. The previous version of my app (Golang 1.12.4) is currently hitting the system in question without this issue. That Java Web Api isn't really in my domain of knowledge - but I think it is a Spring framework, and if I had to guess, they are using OpenJDK 11.

I am now concerned after reading this that it might not be a cut and dry Golang problem:
https://webtide.com/openjdk-11-and-tls-1-3-issues/

There weren't any logs on external system indicating an error and does currently work against with the 1.12.4 version. Client side, we just received a remote error: tls: handshake failure on our end on any attempt to call the HTTPS endpoints to the system in question. HTTP was unaffected (no shocker there). I suspected Tls v1.3 as it was recently introduced and confirmed my TLSClientConfig change resumed communication as previously indicated.

@FiloSottile
I was successfully able to ping pong with Tcp Dialer and a self-signed cert for both server and client with InsecureSkipVerify false. I don't know if this is a good representation of things based on using loopback localhost - everything else should be similar conditions.

package main

import (
	"bufio"
	"crypto/tls"
	"crypto/x509"
	"io/ioutil"
	"log"
	"net"
)

func main() {

	readyToReceive := make(chan struct{}, 1)
	go startServer(readyToReceive)
	<-readyToReceive

	runClient()
}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer(readyToReceive chan struct{}) {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}
	ln, err := tls.Listen("tcp", ":443", config)
	if err != nil {
		log.Println(err)
		return
	}
	defer ln.Close()

	readyToReceive <- struct{}{}
	for {
		conn, err := ln.Accept()
		if err != nil {
			log.Println(err)
			continue
		}
		go handleConnection(conn)
	}
}

func handleConnection(conn net.Conn) {
	defer conn.Close()
	r := bufio.NewReader(conn)
	for {
		msg, err := r.ReadString('\n')
		if err != nil {
			log.Println(err)
			return
		}

		println(msg)

		n, err := conn.Write([]byte("pong\n"))
		if err != nil {
			log.Println(n, err)
			return
		}
	}
}

func runClient() {
	caCert, err := ioutil.ReadFile("server.crt")
	if err != nil {
		log.Fatal(err)
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	log.SetFlags(log.Lshortfile)

	conf := &tls.Config{
		//InsecureSkipVerify: true,
		RootCAs:    caCertPool,
		MinVersion: tls.VersionTLS10,
		MaxVersion: tls.VersionTLS13,
	}

	//conn, err := tls.Dial("tcp", "127.0.0.1:443", conf)
	conn, err := tls.Dial("tcp", "localhost:443", conf)
	if err != nil {
		log.Println(err)
		return
	}
	defer conn.Close()

	n, err := conn.Write([]byte("ping\n"))
	if err != nil {
		log.Println(n, err)
		return
	}

	buf := make([]byte, 100)
	n, err = conn.Read(buf)
	if err != nil {
		log.Println(n, err)
		return
	}

	println(string(buf[:n]))
}

[houseofcat]

If I had to speculate - I think the JDK based system is returning an unrecognizable type of error - it really doesn't support TLS1.3 and throws a weird exception internally - for instance. I think internally Golang attributes this response to a more generic type of error and doesn't attempt to fallback and retry with the next protocol down.

Just a hunch.

[houseofcat]

Starting to play around with NewServeMux and Tls1.2 as the only TLS protocol allowed and what Postman was configured for - I am getting the following:
server.go:3056: http: TLS handshake error from [::1]:61003: EOF

So this is Golang TLS 1.2 (server) and Postman TLS 1.2 (client)

package main

import (
	"crypto/tls"
	"log"
	"net/http"
)

func main() {

	startServer()

}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer() {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.Header().Add("Strict-Transport-Security", "max-age=63072000; includeSubDomains")
		w.Write([]byte("This is an example server.\n"))
	})
	srv := &http.Server{
		Addr:         ":443",
		Handler:      mux,
		TLSConfig:    config,
		TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
	}
	log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key"))
}

[houseofcat]

HttpClient w/ Server

Unable to reproduce it either locally, or golang to golang. No errors other than the above EOF ^ in the example above.

I tested with total mismatch (only v1.2 server, and only v1.3 client) and received the correct error I was expecting from the Golang handshake.

Get https://localhost:443/: remote error: tls: protocol version not supported

package main

import (
	"crypto/tls"
	"crypto/x509"
	"fmt"
	"io/ioutil"
	"log"
	"net"
	"net/http"
	"net/url"
	"time"
)

func main() {

	readyToReceive := make(chan struct{}, 1)
	go startServer(readyToReceive)
	<-readyToReceive

	startClient()
}

// using self-signed from here: https://github.com/denji/golang-tls
func startServer(readyToReceive chan struct{}) {

	log.SetFlags(log.Lshortfile)

	cer, err := tls.LoadX509KeyPair("server.crt", "server.key")
	if err != nil {
		log.Println(err)
		return
	}

	config := &tls.Config{
		Certificates: []tls.Certificate{cer},
		MinVersion:   tls.VersionTLS12,
		MaxVersion:   tls.VersionTLS12,
	}

	mux := http.NewServeMux()
	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
		w.Write([]byte("This is an example server."))
	})
	srv := &http.Server{
		Addr:         ":443",
		Handler:      mux,
		TLSConfig:    config,
		TLSNextProto: make(map[string]func(*http.Server, *tls.Conn, http.Handler), 0),
	}
	readyToReceive <- struct{}{}
	log.Fatal(srv.ListenAndServeTLS("server.crt", "server.key"))
}

func startClient() {
	client := &http.Client{
		Transport: BuildHTTPTransport(nil),
	}

	req, err := http.NewRequest("GET", "https://localhost:443/", nil)
	if err != nil {
		fmt.Println(err)
		return
	}

	for {
		resp, err := client.Do(req)
		if err != nil {
			fmt.Println(err)
			return
		}
		defer resp.Body.Close()

		if resp.StatusCode == http.StatusOK {
			bodyBytes, err := ioutil.ReadAll(resp.Body)
			if err != nil {
				log.Fatal(err)
			}

			fmt.Printf("Client - Response Body: %s\r\n", string(bodyBytes))
		} else {
			fmt.Printf("Client - Response Status: %d\r\n", resp.StatusCode)
		}

		time.Sleep(1 * time.Second)
	}
}

func BuildHTTPTransport(proxy func(*http.Request) (*url.URL, error)) *http.Transport {

	caCert, err := ioutil.ReadFile("server.crt")
	if err != nil {
		log.Fatal(err)
	}
	caCertPool := x509.NewCertPool()
	caCertPool.AppendCertsFromPEM(caCert)

	transport := &http.Transport{
		Proxy: proxy, // Can Be Nil
		DialContext: (&net.Dialer{
			Timeout:   30 * time.Second,
			KeepAlive: 30 * time.Second,
			DualStack: true,
		}).DialContext,
		MaxIdleConns:          100,
		MaxIdleConnsPerHost:   100, // Default was 2
		IdleConnTimeout:       90 * time.Second,
		TLSHandshakeTimeout:   10 * time.Second,
		ExpectContinueTimeout: time.Second,
	}

	transport.TLSClientConfig = &tls.Config{
		RootCAs:    caCertPool,
		MinVersion: tls.VersionTLS10,
		MaxVersion: tls.VersionTLS13,
	}

	return transport
}

[houseofcat]

After doing some more research, I think Java JDK (v11) is a - or perhaps THE- key missing ingredient from my testing.

This was found in another thread that was working in reverse (Golang Servers with Java Clients). Odds are, our Java Web.Api side maybe out of date. I will try and track down what version we are using.

Related to #35722
https://hg.openjdk.java.net/jdk/jdk/rev/b9d1ce20dd4b

[FiloSottile]

It does sound like another OpenJDK issue. Can you update the server and check if it goes away?

Otherwise, it would be useful to get PCAPs of the connection.

(A correctly implemented TLS 1.2 server will automatically negotiate down if the client supports at least a shared protocol version.)

[houseofcat]

@FiloSottile I have started that conversation internally. I would like to ask about the EOF ^ I got above on TLS.

#37808 (comment)

[FiloSottile]

That just means the client closed the connection. Without client side logs or details it's hard to tell, but it might just be that it did not like the certificate. Anyway it's a separate issue so if you think it might be a Go bug, please open a new issue.

[houseofcat]

Fair point, I am going to close this issue - thanks for reviewing it with me Filo. I have updated the title so its more readily searchable in issues - I know I didn't see the other related tickets because of that. If you disprove you can rename back.

[FiloSottile]

Still interested in hearing if this is resolved by an update, if not we might need to work with OpenJDK to report the bug.

[houseofcat]

I wasn't able to update the client from JDK 11, but I did manage to write a little something with JDK 11 web.api with only TLS1.2 enabled and did confirm the issue matched what happens in production for me.

With the world ending, I have been a little busy, so I apologize for the delay.

[FiloSottile]

Take your time, no need to apologize, and take care.

[Jab2870]

I don't know if it is related but I am having a similar issue. I am not a Go developer so my apologies in advance. However, I am playing about with a project to try to learn go and the tls negotiation seems to be failing sometimes. However, if I proxy the requests through Burp, it works fine - presumably because burp then handles the tls negotiation with the server?

I captured both TLS "Client Hello" messages with wireshark, without a proxy on the left and with on the right. I noticed that without the proxy, the TLS version is reported as 1.0 and 1.2 in the same packet. Is this expected behaviour?

[FiloSottile]

@Jab2870 That's intended behavior, yeah, the record layer is a legacy field completely stripped of any semantic meaning. We just send the most compatible value.

If you can share a PCAP of a failed handshake, we can look at it and try to understand what's happening.

(Is the issue happening between a Go client and a JDK11 server? If not, please open a separate issue with details about the server.)

[Jab2870]

@FiloSottile Thanks for your quick response and explanation.

[FiloSottile]

@houseofcat Closing for timeout, but let me know if you get more info.