TLS change in 20.1 Go version can break clients

[jseldess]

@timveil tried upgrading his CC cluster to 20.1, and his app (java + spring boot deployed in GKE) crashed with the following error:

"org.springframework.jdbc.CannotGetJdbcConnectionException: Failed to obtain JDBC Connection; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:82) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:612) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:669) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:700) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:753) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at com.flightchop.web.TurbulenceSearchService.findAircraftReports(TurbulenceSearchService.java:76) ~[classes!/:20.0.1-SNAPSHOT]
	at com.flightchop.web.TurbulenceController.lambda$turbulence$1(TurbulenceController.java:101) ~[classes!/:20.0.1-SNAPSHOT]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[na:na]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
	at java.base/java.lang.Thread.run(Thread.java:834) ~[na:na]
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:689) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:196) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:161) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:128) ~[HikariCP-3.4.2.jar!/:na]
	at org.springframework.jdbc.datasource.DataSourceUtils.fetchConnection(DataSourceUtils.java:158) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.doGetConnection(DataSourceUtils.java:116) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	at org.springframework.jdbc.datasource.DataSourceUtils.getConnection(DataSourceUtils.java:79) ~[spring-jdbc-5.2.4.RELEASE.jar!/:5.2.4.RELEASE]
	... 11 common frames omitted
Caused by: org.postgresql.util.PSQLException: SSL error: extension (5) should not be presented in certificate_request
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:43) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:445) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:139) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:196) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:211) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.Driver.makeConnection(Driver.java:459) ~[postgresql-42.2.10.jar!/:42.2.10]
	at org.postgresql.Driver.connect(Driver.java:261) ~[postgresql-42.2.10.jar!/:42.2.10]
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:138) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:354) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:202) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:473) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool.access$100(HikariPool.java:71) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:719) ~[HikariCP-3.4.2.jar!/:na]
	at com.zaxxer.hikari.pool.HikariPool$PoolEntryCreator.call(HikariPool.java:705) ~[HikariCP-3.4.2.jar!/:na]
	... 4 common frames omitted
Caused by: javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[na:na]
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[na:na]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:307) ~[na:na]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[na:na]
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:254) ~[na:na]
	at java.base/sun.security.ssl.SSLExtensions.<init>(SSLExtensions.java:90) ~[na:na]
	at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestMessage.<init>(CertificateRequest.java:818) ~[na:na]
	at java.base/sun.security.ssl.CertificateRequest$T13CertificateRequestConsumer.consume(CertificateRequest.java:922) ~[na:na]
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) ~[na:na]
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) ~[na:na]
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) ~[na:na]
	at java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:177) ~[na:na]
	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1151) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1062) ~[na:na]
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) ~[na:na]
	at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:41) ~[postgresql-42.2.10.jar!/:42.2.10]
	... 18 common frames omitted

@vladdy identified the likely cause as:

javax.net.ssl.SSLHandshakeException: extension (5) should not be presented in certificate_request is the main cause

And suggested this might have something to do with TLS change in the Go version we're using: https://golang.org/doc/go1.13#tls_1_3

golang/go#35722

We need to document this a 20.1 backward-incompatible change. The workaround in the Go issue might be what we need.

[bdarnell]

Oops. Looks like our internal testing is all on jdk 8 (or insecure mode). And the java patch releases that fix the problem are only a couple of weeks old. So this is going to affect most java users.

Suggested docs (java users may want to tweak the language around "java" vs "openjdk"):

CockroachDB 20.1 is incompatible with some releases of Java, including OpenJDK 11 versions older than 11.0.7 and OpenJDK 13 versions older than 13.0.3. The recommended solution is to upgrade to the latest JDK release in a given branch. If that is not possible, two workarounds are available:

• Set the environment variable GODEBUG=tls13=0 when starting the cockroach server process. Note that this workaround will only work in CockroachDB 20.1.x and will not be available in future releases.
• Add the flag -Djdk.tls.client.protocols=TLSv1.2 when running the Java client processes.

[bdarnell]

In cockroachdb/cockroach#48294 we are discussing making a change in the first 20.1 patch release that would remedy this incompatibility by disabling TLS 1.3 by default.

[bdarnell]

We've decided to disable TLS 1.3 in 20.1.0 instead of waiting for the first patch release, so we don't need to document anything here.

[jseldess]

@bdarnell, does that mean we're releasing another rc?

[bdarnell]

I'm not sure if we're doing to do another RC or if we're just slipping this change into the final release. CC @dt

[dt]

just slipping it into v20.1.0 -- email coming shortly once the build finishes.