Connecting to api server fails with HTTP/2 handshake error

[mikhail-barg]

Describe the bug
I'm getting the following exception while connecting to my K8s cluster:

System.Net.Http.HttpRequestException: 'An HTTP/2 connection could not be established because the server did not complete the HTTP/2 handshake.'

The code:

KubernetesClientConfiguration config = KubernetesClientConfiguration.BuildConfigFromConfigFile(kubeConfFilePath);
Console.WriteLine($"Connecting to {config.CurrentContext}, {config.Host}, {config.Namespace}, {config.Username}");
config.SkipTlsVerify = true;	//does not change anything
using (IKubernetes client = new Kubernetes(config))
{
	V1NamespaceList namespaces = client.CoreV1.ListNamespace(); //this fails

Whole stacktrace is:

   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.<WaitWithCancellationAsync>d__1.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<GetHttp2ConnectionAsync>d__79.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.HttpConnectionPool.<SendWithVersionDetectionAndRetryAsync>d__83.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Threading.Tasks.ValueTask`1.get_Result()
   at System.Runtime.CompilerServices.ConfiguredValueTaskAwaitable`1.ConfiguredValueTaskAwaiter.GetResult()
   at System.Net.Http.RedirectHandler.<SendAsync>d__4.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult()
   at System.Net.Http.HttpClient.<<SendAsync>g__Core|83_0>d.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.Kubernetes.<SendRequestRaw>d__39.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.AbstractKubernetes.<k8s-ICoreV1Operations-ListNamespaceWithHttpMessagesAsync>d__21.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.CoreV1OperationsExtensions.<ListNamespaceAsync>d__15.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at k8s.CoreV1OperationsExtensions.ListNamespace(ICoreV1Operations operations, Nullable`1 allowWatchBookmarks, String continueParameter, String fieldSelector, String labelSelector, Nullable`1 limit, String resourceVersion, String resourceVersionMatch, Nullable`1 timeoutSeconds, Nullable`1 watch, Nullable`1 pretty)
   at K8sApiTest.Program.Main(String[] args) in Z:\kube-test\src\K8sApiTest\Program.cs:line 18

Kubernetes C# SDK Client Version
9.0.38

Server Kubernetes Version
1.25.4

Dotnet Runtime Version
net6

Expected behavior
I expected to be able to connect to the api server

KubeConfig
my kubeconf file looks like this:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: LS0t...
    server: https://<some_ip>:6443
  name: kubernetes-test
contexts:
- context:
    cluster: kubernetes-test
    user: kubernetes-test-admin
  name: kubernetes-test-admin@kubernetes-test
current-context: kubernetes-test-admin@kubernetes-test
kind: Config
preferences: {}
users:
- name: kubernetes-test-admin
  user:
    client-certificate-data: LS0...
    client-key-data: LS0...

Where do you run your app with Kubernetes SDK (please complete the following information):

• OS: Windows 8.1 Prof
• Environment - just a plain app from MS VS
• Cloud - No

Additional context
I doubt that there's a problem with my kubeconf file, because both Lens and KubeNav have no problems connecting to the API server using the same file.

Here are screenshots from wireshark:

My app with the problem:

Kubenav with no problem:

(black rects mask my machine, green ones mask api server)

[tg123]

looks like server side rejected tls1.2

[tg123]

seems windows8 does not support tls1.3, is your kubenav running on same machine?

[mikhail-barg]

@tg123 yes, my kubenav is on the same machine

[mikhail-barg]

My first thought also was the TLS version problem, but as far as I can understand the dumps, SSL handshake completes sucessfully even thoughs it's tls1.2 (I think it's allowed on the api server). If that was not the case, I'd be receiving SSL errors, not HTTP/2, right?

[tg123]

6443 -> you RST, which means server closed the connection
i do not think handshake succ, you can test with browser as well

[mikhail-barg]

Not sure what to test though, but I did open the https://<apiserver-url>:6443 in Chrome, and (after skipping server certificate validation failed warning), I got a (probably expected) response:

{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
    "reason": "Forbidden",
    "details": {},
    "code": 403
}

In the wireshark I see that the connection is done via TLS 1.3, and it seems to be keepalived. And I did run it from the same machine, yes.

Should I check something else?

[mikhail-barg]

Okay, I did the following from a c# program:

HttpClientHandler handler = new HttpClientHandler() {
	ServerCertificateCustomValidationCallback = HttpClientHandler.DangerousAcceptAnyServerCertificateValidator
};

using (HttpClient client = new HttpClient(handler))
{
	using (HttpResponseMessage result = await client.GetAsync("https://<apiserver-url>:6443"))
	{
		Console.WriteLine(result.StatusCode);
		Console.WriteLine(await result.Content.ReadAsStringAsync());
	}
}

and it gives me the same HTTP response

{
	"kind": "Status",
	"apiVersion": "v1",
	"metadata": {},
	"status": "Failure",
	"message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
	"reason": "Forbidden",
	"details": {},
	"code": 403
}

And in wireshark I see that it's done via TLS 1.2

[mikhail-barg]

Ok, I am able to reproduce the problem using the following change to the code above:

using (HttpRequestMessage req = new HttpRequestMessage(HttpMethod.Get, "https://<apiserver-url>:6443") {
	Version = new Version(2, 0)
})
{
	using (HttpResponseMessage result = await client.SendAsync(req))

So switching request to Http/2 causes the same problem as I experience with the client library.
Not sure where to go next then.

[tg123]

anything special with your api server, the sdk assumes it supports http2 and did not have n option to change it at the moment.

[mikhail-barg]

anything special with your api server

don't think so. Just a vanilla K8s, bootstrapped via kubeadm.

The only thing that comes to mind, is that we've set it up with --skip-phases=addon/kube-proxy to use Cilium for CNI, but I don't think it's a culprit.

[mikhail-barg]

I also have no luck in googling any HTTP/2 issues with kube api server, except for this, but it's from K3s, which we don't use.

[mikhail-barg]

OK, some more info. I tried running my original app from Win 11 machine, and it does connect successfully using the same kubeconf file. So it seems that it's really not a HTTP/2 problem.

[tg123]

could you please take a look at api server to see why it does not like windows 8.1

[mikhail-barg]

Yes, but I'm not sure what to look at. Maybe you could give some hints? I think, we have all TLS versions and all ciphers enabled at apiserver.

[tg123]

https://kubernetes.io/docs/tasks/debug/debug-cluster/#control-plane-nodes

LMK if it helps, so we can do something to improve sdk's compatibility