Upgrade request required when running exec command with python-kubernetes client #409

genadic · 2017-12-05T15:37:11Z

I saw this image already appeared in #58 and was solved, but now it's broken again in 4.0

dulek · 2017-12-05T15:40:02Z

I was looking at that as well, so here's some additional info:

#58 was fixed by #120, but @dims code is now gone due to commit #353.

dims · 2017-12-05T16:01:05Z

hmmmmmmm....

dims · 2017-12-05T16:02:00Z

@dulek do you have some time to re-patch please? i thought i had tests in there

dulek · 2017-12-05T17:09:27Z

@DISM: I can try. Those tests were disabled by #259.

@chekolyn

This commit reapplies PR kubernetes-client#120 that was removed by kubernetes-client#353. Below is the original commit message. inspired by the POC from @chekolyn * Adds a new requirement on websocket-client * Add a new class WSClient that uses WebSocketApp from the websocket-client. * Make sure we pass Authorization header * Make sure we honor the SSL settings in configuration * Some of the code will get overwritten when we generate fresh classes from swagger definition. To remind us added a e2e test so we don't lose the changes * Added a new configuration option to enable/disable failures when hostnames in certificates don't match Fixes kubernetes-client#58 Fixes kubernetes-client#409

@chekolyn

This commit reapplies PR kubernetes-client#120 that was removed by kubernetes-client#353. Below is the original commit message. inspired by the POC from @chekolyn * Adds a new requirement on websocket-client * Add a new class WSClient that uses WebSocketApp from the websocket-client. * Make sure we pass Authorization header * Make sure we honor the SSL settings in configuration * Some of the code will get overwritten when we generate fresh classes from swagger definition. To remind us added a e2e test so we don't lose the changes * Added a new configuration option to enable/disable failures when hostnames in certificates don't match Fixes kubernetes-client#58 Fixes kubernetes-client#409

dulek · 2017-12-05T17:41:35Z

#410 is the commit. I was however unable to verify it as connection is now failing with "Reason: Handshake status 404".

mpwusr · 2018-04-30T20:20:08Z

I can run from the command line:
kubectl exec secret-pod --namespace=production cat data/busybox/password
and
kubectl exec secret-pod --namespace=production cat data/busybox/username
and they are working successfully

But when I run the following code in python
I get
"Upgrade request required"
def _get_user_token(self):
user=''
passwd=''
name = 'secret-pod'
namespace = 'production'
exec_cmd = ['cat /data/busybox/username']

    try:
        user = self.core_api_instance.connect_get_namespaced_pod_exec(name="secret-pod",
                                                                      namespace="production",
                                                                      command=["cat /data/busybox/username"],
                                                                      stderr=True, stdin=False,
                                                                      stdout=True, tty=False)
        pprint(user)
    except ApiException as e:
        print("Exception when calling CoreV1Api->get_namespaced_pod_exec: %s\n" % e)

    exec_cmd = ['cat /data/busybox/password']
    try:
        passwd = self.core_api_instance.connect_get_namespaced_pod_exec(name="secret-pod",
                                                                        namespace='production',
                                                                        command=["cat /data/busybox/password"],
                                                                        stderr=True, stdin=False,
                                                                        stdout=True, tty=False)
        pprint(passwd)
    except ApiException as e:
        print("Exception when calling CoreV1Api->get_namespaced_pod_exec: %s\n" % e)

    return {'user': user, 'password': passwd}

Full error messages
send: 'GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fusername%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1\r\nHost: 192.168.42.218:8443\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nauthorization: Bearer heoJMu0hqlrLWNJ2lV9MsaIWp1kPuKvwPhmhcmj9-Kg\r\nAccept: /\r\nUser-Agent: Swagger-Codegen/4.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Cache-Control: no-store
header: Content-Type: application/json
header: Date: Mon, 30 Apr 2018 19:50:05 GMT
header: Content-Length: 139
2018-04-30 15:50:05,528 DEBUG https://192.168.42.218:8443 "GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fusername%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1" 400 139
Exception when calling CoreV1Api->get_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 30 Apr 2018 19:50:05 GMT', 'Content-Length': '139', 'Content-Type': 'application/json', 'Cache-Control': 'no-store'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

send: 'GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fpassword%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1\r\nHost: 192.168.42.218:8443\r\nAccept-Encoding: identity\r\nContent-Type: application/json\r\nauthorization: Bearer heoJMu0hqlrLWNJ2lV9MsaIWp1kPuKvwPhmhcmj9-Kg\r\nAccept: /\r\nUser-Agent: Swagger-Codegen/4.0.0/python\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Cache-Control: no-store
header: Content-Type: application/json
header: Date: Mon, 30 Apr 2018 19:50:05 GMT
header: Content-Length: 139
2018-04-30 15:50:05,536 DEBUG https://192.168.42.218:8443 "GET /api/v1/namespaces/production/pods/secret-pod/exec?command=%5B%27cat+%2Fdata%2Fbusybox%2Fpassword%27%5D&stderr=True&stdin=False&stdout=True&tty=False HTTP/1.1" 400 139
Exception when calling CoreV1Api->get_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Date': 'Mon, 30 Apr 2018 19:50:05 GMT', 'Content-Length': '139', 'Content-Type': 'application/json', 'Cache-Control': 'no-store'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

TopherGopher · 2018-07-06T13:59:40Z

Is there any status update on this? Has anyone figured out a workaround?
I don't mind patching this, but I don't understand why that massive pull request wiped out this logic. Was it because it was bad logic? Or was it just a merge mistake?

TopherGopher · 2018-07-06T19:48:14Z

I found this issue mentioned (and apparently solved) over here: #526

resp = kubernetes.stream.stream(self.core_api_instance.connect_get_namespaced_pod_exec, name, namespace,command=exec_command1, stderr=True, stdin=False, stdout=True, tty=False)

You can then manipulate that pipe using the technique from the example code

My issue is that I'm still getting an error message when I run it inside a pod in the cluster -

>>> resp = k.stream.stream(k8s.connect_get_namespaced_pod_attach, 'datadog-agent-8pkwh', container="datadog-agent", namespace="default", stderr=True, stdin=False, stdout=True, tty=False)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/stream.py", line 32, in stream
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 711, in connect_get_namespaced_pod_attach
    (data) = self.connect_get_namespaced_pod_attach_with_http_info(name, namespace, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/apis/core_v1_api.py", line 808, in connect_get_namespaced_pod_attach_with_http_info
    collection_formats=collection_formats)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 321, in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/client/api_client.py", line 155, in __call_api
    _request_timeout=_request_timeout)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/stream.py", line 27, in _intercept_request_call
    return ws_client.websocket_call(config, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kubernetes/stream/ws_client.py", line 255, in websocket_call
    raise ApiException(status=0, reason=str(e))
kubernetes.client.rest.ApiException: (0)
Reason: hostname '10.100.0.1' doesn't match either of 'ip-10-0-113-207.us-west-2.compute.internal', 'kubernetes', 'kubernetes.default', 'kubernetes.default.svc', 'kubernetes.default.svc.cluster.local', '0e23207807f30610a20a7dc08f349bad.sk1.us-west-2.eks.amazonaws.com'

I'll let you know if I find the fix ;-)

TopherGopher · 2018-07-06T19:48:38Z

The fix was I upgraded from Python 2 to Python 3 and the hostname message went away, and I can confirm the Upgrade message is gone. If no one objects, could we close this issue?

scher200 · 2018-08-06T14:18:58Z

for anyone ending up here Googling: connect_get_namespaced_pod_attach "message":"Upgrade request required"

    resp = stream( api_instance.connect_get_namespaced_pod_attach, NAME, NAMESPACE, stderr=stderr, stdin=stdin, stdout=stdout, tty=tty, _preload_content=False )

read more:
https://github.com/kubernetes-client/python/blob/master/examples/exec.py

mpwusr · 2018-08-06T15:03:22Z

@scher200 you also need to upgrade the ssl on python to make it work, definitely you need kubernetes stream but thats one aspect you also need to upgrade SSL and on my mac its a bit complicated not sure about windows/linux

scher200 · 2018-08-06T18:42:27Z

here it worked out of the box with linux..

mpwusr · 2018-08-06T19:27:57Z

the older openssl versions still cause this issue using Streams

type "openssl version"
I believe you need a version 1.X and above
the 0.98 and earlier versions fail with the same error with streams

I have
OpenSSL 1.0.2o 27 Mar 2018 and it works with streams
the 0.98 and earlier versions fail with the same error with streams

you also may want to do
"which openssl"

and you can set
--openssldir=
to directory for new openssl install

fejta-bot · 2019-04-25T00:54:44Z

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

fejta-bot · 2019-05-25T01:36:37Z

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

kebab-mai-haddi · 2019-05-28T14:08:14Z

Even after applying all the standard configurations and examples, applying the solutions of @scher200 , I am getting a bad request.
My code:

from __future__ import print_function
import time
import kubernetes.client
from kubernetes.client.rest import ApiException
from pprint import pprint
from kubernetes import client, config, stream
stream = stream.stream

# Configure API key authorization: BearerToken
configuration = kubernetes.client.Configuration()
configuration.host = "https://mine.eks.amazonaws.com"
configuration.verify_ssl = False
configuration.api_key['authorization'] = "my_token"
# Uncomment below to setup prefix (e.g. Bearer) for API key, if needed
configuration.api_key_prefix['authorization'] = 'Bearer'
configuration.debug = True

api_instance = kubernetes.client.CoreV1Api(
    kubernetes.client.ApiClient(configuration))
name = 'jupyter-aviral'  # str | name of the PodExecOptions
namespace = 'jhub'  # str | object name and auth scope, such as for teams and projects
# str | Command is the remote command to execute. argv array. Not executed within a shell. (optional)
command = 'echo "hail aviral"'
# bool | Redirect the standard error stream of the pod for this call. Defaults to true. (optional)
stderr = True
# bool | Redirect the standard input stream of the pod for this call. Defaults to false. (optional)
stdin = True
# bool | Redirect the standard output stream of the pod for this call. Defaults to true. (optional)
stdout = True
# bool | TTY if true indicates that a tty will be allocated for the exec call. Defaults to false. (optional)
tty = True

try:
    api_response = stream(api_instance.connect_post_namespaced_pod_exec(
        name, namespace, command=command, stderr=stderr, stdin=stdin, stdout=stdout, tty=tty)
    )
    pprint(api_response)
except ApiException as e:
    print("Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: %s\n" % e)

The traceback is:

/usr/local/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.25.3) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
2019-05-28 19:34:38,092 DEBUG Starting new HTTPS connection (1): 0d6fb20cb6b6acb7fbc1b2c264d5e9a1.yl4.ap-south-1.eks.amazonaws.com:443
/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py:851: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
send: b'POST /api/v1/namespaces/jhub/pods/jupyter-aviral/exec?command=echo+%22hail+aviral%22&stderr=True&stdin=True&stdout=True&tty=True HTTP/1.1\r\nHost: mine.eks.amazonaws.com\r\nAccept-Encoding: identity\r\nContent-Length: 0\r\nAccept: */*\r\nContent-Type: application/json\r\nUser-Agent: Swagger-Codegen/9.0.0/python\r\nauthorization: Bearer my_api_token\r\n\r\n'
reply: 'HTTP/1.1 400 Bad Request\r\n'
header: Audit-Id: 88df8863-61b1-4fe7-9d39-d0e6059ea993
header: Content-Type: application/json
header: Date: Tue, 28 May 2019 14:04:38 GMT
header: Content-Length: 139
2019-05-28 19:34:38,420 DEBUG https://mine.eks.amazonaws.com:443 "POST /api/v1/namespaces/jhub/pods/jupyter-ankita/exec?command=echo+%22hail+aviral%22&stderr=True&stdin=True&stdout=True&tty=True HTTP/1.1" 400 139
Exception when calling CoreV1Api->connect_post_namespaced_pod_exec: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '88df8863-61b1-4fe7-9d39-d0e6059ea993', 'Content-Type': 'application/json', 'Date': 'Tue, 28 May 2019 14:04:38 GMT', 'Content-Length': '139'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

fejta-bot · 2019-06-27T14:50:43Z

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

k8s-ci-robot · 2019-06-27T14:50:51Z

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Pega88 · 2019-10-16T12:17:31Z

I'm having the same issue with a way more simpler setup:
curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/exec?command=ls I am getting the same response:
`{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "Upgrade request required",
"reason": "BadRequest",
"code": 400
}`

When I just ask for my pod details, it does work - so my Auth and Cert should be fine:
curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/log --> logs printing correctly. Would like to /reopen

vennilashanthi · 2020-11-25T10:55:00Z

Hi all,
I am getting the same error which @Pega88 got. Is the issue resolved ? If so, could you please let me know how to get rid of this error ?

z-koki · 2022-09-09T02:24:12Z

I'm having the same issue with a way more simpler setup: curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/exec?command=ls I am getting the same response: `{ "kind": "Status", "apiVersion": "v1", "metadata": {

}, "status": "Failure", "message": "Upgrade request required", "reason": "BadRequest", "code": 400 }`

When I just ask for my pod details, it does work - so my Auth and Cert should be fine: curl -H "Authorization: Bearer $TOKEN" --cacert $CACERT $K8S/api/v1/namespaces/devops/pods/spinnaker-spinnaker-halyard-0/log --> logs printing correctly. Would like to /reopen

This is because a websocket connection is required if you want to exec into a pod.
See https://stackoverflow.com/questions/49250370/kubernetes-pod-exec-api-upgrade-request-required
For convenience, you can use wscat or wssh instead. If you insist on using curl, these parameters worked for me:
--include \ --no-buffer \ --header "Connection: Upgrade" \ --header "Upgrade: websocket" \ --header "Sec-WebSocket-Key: NVwjmQUcWCenfWu98asDmg==" \ --header "Sec-WebSocket-Version: 13" \

dulek mentioned this issue Dec 5, 2017

Reapply "Implementation for /exec using websocket" #410

Closed

cfchase mentioned this issue Jan 11, 2018

Cannot retrieve test results when running apb test ansibleplaybookbundle/ansible-playbook-bundle#190

Closed

lucid-at-dream mentioned this issue Feb 7, 2019

Exec calls #58

Closed

k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Apr 25, 2019

k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels May 25, 2019

k8s-ci-robot closed this as completed Jun 27, 2019

yliaog mentioned this issue May 10, 2021

python2.7.18 call connect_get_namespaced_pod_exec get Hostname doesn't match but call list_namespaced_pod is normal #1436

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade request required when running exec command with python-kubernetes client #409

Upgrade request required when running exec command with python-kubernetes client #409

genadic commented Dec 5, 2017

dulek commented Dec 5, 2017

dims commented Dec 5, 2017

dims commented Dec 5, 2017

dulek commented Dec 5, 2017

dulek commented Dec 5, 2017

mpwusr commented Apr 30, 2018

TopherGopher commented Jul 6, 2018

TopherGopher commented Jul 6, 2018

TopherGopher commented Jul 6, 2018

scher200 commented Aug 6, 2018 •

edited

Loading

mpwusr commented Aug 6, 2018

scher200 commented Aug 6, 2018

mpwusr commented Aug 6, 2018 •

edited

Loading

fejta-bot commented Apr 25, 2019

fejta-bot commented May 25, 2019

kebab-mai-haddi commented May 28, 2019

fejta-bot commented Jun 27, 2019

k8s-ci-robot commented Jun 27, 2019

Pega88 commented Oct 16, 2019 •

edited

Loading

vennilashanthi commented Nov 25, 2020

z-koki commented Sep 9, 2022

Upgrade request required when running exec command with python-kubernetes client #409

Upgrade request required when running exec command with python-kubernetes client #409

Comments

genadic commented Dec 5, 2017

dulek commented Dec 5, 2017

dims commented Dec 5, 2017

dims commented Dec 5, 2017

dulek commented Dec 5, 2017

dulek commented Dec 5, 2017

mpwusr commented Apr 30, 2018

TopherGopher commented Jul 6, 2018

TopherGopher commented Jul 6, 2018

TopherGopher commented Jul 6, 2018

scher200 commented Aug 6, 2018 • edited Loading

mpwusr commented Aug 6, 2018

scher200 commented Aug 6, 2018

mpwusr commented Aug 6, 2018 • edited Loading

fejta-bot commented Apr 25, 2019

fejta-bot commented May 25, 2019

kebab-mai-haddi commented May 28, 2019

fejta-bot commented Jun 27, 2019

k8s-ci-robot commented Jun 27, 2019

Pega88 commented Oct 16, 2019 • edited Loading

vennilashanthi commented Nov 25, 2020

z-koki commented Sep 9, 2022

scher200 commented Aug 6, 2018 •

edited

Loading

mpwusr commented Aug 6, 2018 •

edited

Loading

Pega88 commented Oct 16, 2019 •

edited

Loading