Exec calls

[diegodelemos]

Whenever I try to open an exec call I get:

{
  "kind":"Status",
  "apiVersion":"v1",
  "metadata":{},
  "status":"Failure",
  "message":"Upgrade request required",
  "reason":"BadRequest","code":400
}

Which is related with the fact that requests is not SPDY compatible. Are you going to support this kind of requests in the near future?

[mbohlool]

cc @caesarxuchao

[mbohlool]

realted: http://stackoverflow.com/questions/37349440/upgrade-request-required-when-running-exec-in-kubernetes

I guess if we change code generator to accept headers, it should be possible to pass upgrade headers to the call.

Update: Look like we need an SPDY client for python too.

[diegodelemos]

Is it feasible to have it soon? It would be nice to have a date in order to know if I can use it within my deadline. If not, do you have some workaround in mind? Thank you in advance.

[mbohlool]

I don't have a date yet, the problem is urllib3 does not support SPDY protocol. we need to use something like python-spdylay. Unless somebody own this (or I found the time to look at it in more depth) I can't put a date on it.

[mbohlool]

Just spend a little more time with python-spdylay. Even their client example is not working and it does not seem to be actively developed/supported. If anybody have any idea of how to talk SPDY in python, please let me know.

[mbohlool]

More information here: kubernetes/kubernetes#24668

[diegodelemos]

Moreover, there is kubernetes/kubernetes#7452 which stands that SPDY is deprecated and HTTP/2 might be the right option, but looks like it is still hanging in the air. Anyways, it is SPDY what is working for current versions so we need to somehow find a workaround.

[mbohlool]

A little more digging pointed me to a library called thor. I will give it a try. If you can try it too to see if you can establish a SPDY connection with it, that would be nice. My assumption is SPDY is always secure so there should be a way to specify client-cert and other cert files that we normally set in kubernetes.client.configuration class.

[sebgoa]

You can do exec over websockets instead of SPDY. So would be good to investigate websockets instead of the sdpy route (which will be osbolete anyway).

[mbohlool]

Thanks @sebgoa it look like websockets are supported as a fall-back: https://github.com/kubernetes/kubernetes/blob/master/docs/devel/api-conventions.md#websockets-and-spdy

[mrmcmuffinz]

@mbohlool is there an example that you could provide demonstrating the use of exec via websockets?

[mbohlool]

@mrmcmuffinz I did not have time to do it. That is why I added "Help needed" to this issue so somebody from the community hopefully help.

[mrmcmuffinz]

@mbohlool My apologies, I thought you had some example. Do you happen to know who we can talk to, to help out? Can we work out defining what exactly it takes to get this done?

[mrmcmuffinz]

Adding my stackoverflow question here too, for other people who stumble across this. Also has anyone looked into speaking with the developers of urilib3 to see if they will support http/2? That way there is a minimum impact to the api_client...

[mbohlool]

The ApiException for exec or any call needs SPDY/Websockets is informative. The right way to fix this is to catch that exception in api_client.py and start a websocket call instead. For the first step, try to write a python code that does not use this repo, and directly call into exec REST API using websocket. use kubectl proxy to get rid of authorization part and call into local host for simplification.

[mrmcmuffinz]

I actually tried doing that but I could not figure out how to setup the authentication part of the code for the websocket. I have three pem certificates that were given to me and I'm just not sure how that fits into the picture. Thank you for your response btw.

[mbohlool]

have you try kubectl proxy? that way you should not worry about authentication for now. it will proxy your call from localhost with the right authentication headers to the remote host.

[mrmcmuffinz]

Ohh wow I totally forgot about the kubectl proxy command, if I do that should I switch my url to localhost in the websocket url versus the actual endpoint itself?

Edit:
nvm, I reread again and that is indeed what you suggested I do.

[mbohlool]

Yes. lets have a working code with proxy then think about how to incorporate it into the client-python.

[mrmcmuffinz]

Yeah I think I'm a bit further however I get a connection reset by peer error right now. I think I will need to read up on sockets in python as well as what all I need to do in order to setup the appropriate message for communicating. FYI: I'm using a package called hyper for the actual communication unless you can suggest anything better.

[mrmcmuffinz]

Here is the issue against urllib3 for http/2 suuport urllib3/urllib3#836.

[chekolyn]

@mrmcmuffinz I was wondering how you are testing this. I was doing some basic testing with hyper; but the exec call needs a "SPDY/3.1" upgrade header that hyper doesn't know how to handle.

If you do a connection upgrade with "SPDY/3.1" you will get this exception: https://github.com/Lukasa/hyper/blob/24b3b37d226bedb6199f22b788e066b91dbf892a/hyper/common/conncetion.py#L130

And if you do an h2 upgrade you will get this from kubernetes (since http2 is not implemented yet):
https://github.com/kubernetes/kubernetes/blob/55bee3ad21f025b1416a4e1f10de753f484b66d3/pkg/util/httpstream/spdy/upgrade.go#L48

[mrmcmuffinz]

Hi @chekolyn in all honesty I put it on hold and started looking at other clients. I just wanted to use these python client with py.test for FVT but that kinda went out the door because of this limitation.

Edit:
As for the socket workaround that was another rabbit hole that I did not want to go into right now.

[chekolyn]

OK guys a gave websockets a try and I have a very limited POC

I can confirm websockets works!
(Tested on OpenShift)

# POC for Websocket exec calls to pods:
#
# Channel 0 is STDIN, 1 is STDOUT, 2 is STDERR (if TTY is not requested),
# and 3 is a special error channel. The server only reads from STDIN,
# writes to the other three.

import websocket
import ssl
from subprocess import check_output


def on_message(ws, message):
    print 'message received ..'
    print message


def on_error(ws, error):
    print 'error happened .. '
    print error


def on_close(ws):
    print "### closed ###"


def on_open(ws):

    print 'Opening Websocket connection to the server ... '
    # This session_key I got, need to be passed over websocket header isntad
    # of ws.send.
    #ws.send(session_key)

token = check_output(["oc", "whoami", "-t"])
pod = "router-1-8u540"
exec_command = "uptime".replace(" ", "+")

url = "wss://localhost:8443/api/v1/namespaces/default/pods/%s/exec?command=%s&stderr=true&stdin=true&stdout=true&tty=false" % (pod, exec_command)
header = "Authorization: Bearer " + token



if __name__ == "__main__":
    websocket.enableTrace(False)
    ws = websocket.WebSocketApp(url,
                                on_message = on_message,
                                on_error = on_error,
                                on_close = on_close,
                                header = [header]
                                )
    ws.on_open = on_open
    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})

#    ws.on_message = on_message
#    ws.run_forever(sslopt={"cert_reqs": ssl.CERT_NONE})

[dims]

@chekolyn "oc" implies openshift? does this work in a pure kubernetes environment?

[chekolyn]

@dims I haven't tested it on a pure k8s but I believe It should work, the oc command is just grabbing the current auth token to authenticate via headers (perhaps this is the only portion that it's OpenShift specific in the POC, you can probably remove the check_output for a valid token and it should work too ). The url api exec call is pure kubernetes and websockets calls seem to be working fine.

I'm basically using this authentication strategy in the api call: https://kubernetes.io/docs/user-guide/accessing-the-cluster/#without-kubectl-proxy-post-v13x

[mbohlool]

I still think the way it's implemented is a little hacky (not the implementation itself but the way we integrated it into the api call by checking an specific API endpoint). I would like to keep this issue a little longer so we can look back and clean it up a little. I will take another look tomorrow.

[mrmcmuffinz]

Hi might I bring my two cents. I still think we need to fix the ged and post requests for exec as they both go down the same path. Lastly I have created a dozen or so tests for the product I'm working on but there is one thing that I don't understand, where does one access the stderr, stdout? All we get back is a response message but those elements are not accessible in any specific manner. I think the api is a bit misleading though perhaps I'm misinterpreting it. However that being said this Git issue was opened to add exec support via websockets and @dims did an excellent job at doing that, despite me reporting some post work needed. Which leads me to believe that what you(@mbohol) are asking for is beyond the extent of the original scope for this item.

[mbohlool]

Well I don't mind creating a new issue for post work. The access to stderr and stdout separately was one of my comments on original PR too. The majority part of the task is done so I will leave it to you and @dims if you want to close this and create a new issue, but I don't think our work on exec calls is done until we solve those issues. I would like to have a similar approach to Watch here. Will try something today.

[mrmcmuffinz]

I'm ok with either approach @mbohlool and will provide my feedback along the way as now I'm kinda a "stakeholder" in this. Any idea when we will have a new release with these changes, or at least a git tag for marking this particular milestone?

[mbohlool]

I have a WIP PR that will send out soon. Only thing that is not working is stdin. stdout and stderr is working as well as streaming output. I will do another beta release after that PR.

[mrmcmuffinz]

Ok, please keep me posted.

[mbohlool]

@mrmcmuffinz I sent a partial PR. stdin is not working yet (and I am not sure why). You also show concern about get/post going the same path, what is your concern? as far as I can tell websocket does not have a method like http. so get/post is the same for websocket.

[mrmcmuffinz]

I sent a partial PR. stdin is not working yet
Do you need some review on this or are you fine?

What is your concern for get/post going down the same path?
Yeah I realized later and came to the same conclusion, which was that the websocket protocol does not deal with get and post requests however the intent I believe was to have websockets as a failback mechanism to the rest api and what we have right now is it just default for exec for get/post. I think we should have that separated out such that if it is get, it goes down the get path and vice versa for post. That makes it easier to fix when we the refactor comes into play. At least that is my two cents on that point.

Edit 1:
I looked at some of the code you wrote and have a few questions:

1. I'd like to suggest first if we can agree on below before we come up with some implementation?

File descriptor stderr standard error or stream 2
File descriptor stdout standard out or stream 1
File descriptor stdin standard input or input stream.

2. Can we agree that the way the above std streams get externalize, they should be externalized via one way for both websockets and rest?

3. As for stdout and stderr I would like to be able to access those in the response body via some member, example below, do you think this right?

response.stdout
response.stderr

4. Does it really make sense to exertnalize stdin right now and if it does what does that mean for exec calls?

5. Lastly, how do I access the return code of the exec command I just executed?

[mbohlool]

The normal HTTP call (that I assume you mean that by rest) for exec does not work and will fail for exec call.writing to stdin would allow interaction with pod. It is much like kubctl exec command when you attach to stdin.

[zhenhua]

@mbohlool I meet the following error when trying to execute a command in the pod.

 	Exception when calling connect_post_namespaced_pod_exec: (0)
 	Reason: 'NoneType' object is not iterable

I tried both k8s 1.2 and 1.5 cluster. It seems the websockets is not accessable so it fails when calling WebSocket.connect() method. How to enable websockets on the k8s master node?

[mbohlool]

I don't think the issue is the target cluster (I don't think websocket can be disabled, it is always enabled). Would you share your code (fragment)? This can be a bug in our implementation.

[mrmcmuffinz]

@mbohlool I actually edited my prior reply and not sure if you saw the full message could you kindly please revisit it?

@zhenhua Could you kindly please include apart from the fragment/redacted source code but also the stack trace?

[mbohlool]

@mrmcmuffinz

1. That is what I have in PR, right?
2. what do you mean by rest? you mean normal http call? we don't support that. if you mean a blocking call then this question is the same as 3?
3. Good idea.
4. kubectl let you connect to a container using exec (I think the option is -stdin that Passes stdin to the container/pod). This let you interact with container like the last part of my example (that is not working right now and I disabled it)
5. By return code you mean return code of the command you ran on the container? You should somehow output it and access it from the output (something like echo $?).

[zhenhua]

My test code is straight forward enough. Here it is

def _exec_pod_cmd(cli, pod_name, command):
    try:
        resp = cli.connect_post_namespaced_pod_exec(
                    name=pod_name, namespace=namespace, command=command)
        return resp 
    except ApiException as e:
        print("Exception when calling connect_post_namespaced_pod_exec: %s\n" % e)

config.load_kube_config(config_file=path_to_config)
config.assert_hostname = False
cli = client.CoreV1Api()

resp = _list_namespaced_pod(cli, str("name=%s" % rc_name))
pod_name = resp.items[0].metadata.name
print("pod name %s " % pod_name)
resp = _exec_pod_cmd(cli, pod_name, "pwd")
pprint(resp)

Here is some debug trace:

(Pdb) b 108
Breakpoint 5 at /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py:108
(Pdb) c
> /usr/lib/python2.7/site-packages/kubernetes/client/ws_client.py(108)GET()
-> client = WSClient(configuration, url, headers)
(Pdb) p url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p configuration
<kubernetes.client.configuration.ConfigurationObject object at 0x3012290>
(Pdb) p headers
{'Content-Type': 'application/json', 'Accept': '*/*', 'User-Agent': 'Swagger-Codegen/1.0.0-snapshot/python'}
(Pdb) 

After invoking websocket

(Pdb) b 181
Breakpoint 7 at /usr/lib/python2.7/site-packages/websocket/_app.py:181
(Pdb) c
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) l
176                 self.sock.connect(self.url, header=self.header, cookie=self.cookie,
177                     http_proxy_host=http_proxy_host,
178                     http_proxy_port=http_proxy_port,
179                     http_no_proxy=http_no_proxy, http_proxy_auth=http_proxy_auth,
180                     subprotocols=self.subprotocols,
181 B->                 host=host, origin=origin)
182                 self._callback(self.on_open)
183     
184                 if ping_interval:
185                     event = threading.Event()
186                     thread = threading.Thread(target=self._send_ping, args=(ping_interval, event))
(Pdb) p self.url
'ws://10.13.10.63:8080/api/v1/namespaces/289-ns/pods/test-rc-jd2rp/exec&command=pwd'
(Pdb) p http_proxy_host
None
(Pdb) p self.header
None
(Pdb) n
TypeError: TypeErro...erable",)
> /usr/lib/python2.7/site-packages/websocket/_app.py(181)run_forever()
-> host=host, origin=origin)
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(219)run_forever()
-> except (Exception, KeyboardInterrupt, SystemExit) as e:
(Pdb) p Exception
<type 'exceptions.Exception'>
(Pdb) n
> /usr/lib/python2.7/site-packages/websocket/_app.py(220)run_forever()
-> self._callback(self.on_error, e)
(Pdb) p e
TypeError("'NoneType' object is not iterable",)
(Pdb) 

Let me ask a stupid question: How to access the websocket, port 8443 or port 8080?

[mbohlool]

@mrmcmuffinz: for no 3. It is a good idea, however API spec defined the return type as string. If you want to get stdout and stderr separately you need to call the API with _preload_content=False and read from each channel separately.

resp = api.connect_get_namespaced_pod_exec(name, 'default',
                                           command=exec_command,
                                           stderr=True, stdin=True,
                                           stdout=True, tty=False,
                                           _preload_content=False)
resp.run_forever()
if resp.peek_stdout(): print("STDOUT: ", resp.read_stdout())
if resp.peek_stderr(): print("STDERR: ", resp.read_stderr())

[mbohlool]

@zhenhua, I guess your problem is header. it should not be None. It is fixed as part of my PR.

[zhenhua]

@mbohlool, thanks. I use your mbohlool:exec branch to verify it on my side and it works fine. Only one thing is that, the container parameter of connect_post_namespaced_pod_exec() call is NOT optional, otherwise, it will return an 404 error of 'HTTP/1.1 404 Not Found'.

[mbohlool]

@zhenhua documentation of the parameter says: "Defaults to only container if there is only one container in the pod." I think in case of our examples and e2e tests we have one container but you may have more than one container in your pod. Happy that it worked, I will merge the PR and cut another beta release.

[zhenhua]

@mbohlool You are right. Actually, when creating a pod in my test, there're always have at least two containers, one is registry.access.redhat.com/rhel7/pod-infrastructure:latest and another is my test container. So I have to always specify the container name.

$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

63aa2b961a10 registry.v2.service.xxx.org/dongyue/xxx-base-centos7:20170111 "/bin/sh -c /usr/sbin" 40 minutes ago Up 40 minutes k8s_test-rc.12ee117e_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_1c38b8b6

be4cc79af38c registry.access.redhat.com/rhel7/pod-infrastructure:latest "/pod" 40 minutes ago Up 40 minutes k8s_POD.ae8ee9ac_test-rc-t4052_289-ns_0b32083c-f806-11e6-b075-c81f66cd4467_ffa70144

[ling2yt]

hi,
i test with websocket, but got error message "x509: certificate is valid for 127.0.0.1, not xx.xx.xx.xx"
i use golang like :

wsurl := "wss://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443/api/v1/namespaces/NM/pods/testPod-546cdd8d79-7h8nv/exec?command=ls&container=testPod&stderr=true&stdin=true&stdout=true&tty=false"

u, err := neturl.Parse(wsurl)

rawConn, err := net.Dial("tcp", u.Host)

wsHeaders := http.Header{
        "Authorization":                   {"Bearer "+env_bearer_token},
        "Origin":                   {"https://xx.xx.xx.xx:8080/r/projects/1a92/kubernetes:6443"},
        "Sec-WebSocket-Extensions": {"permessage-deflate; client_max_window_bits, x-webkit-deflate-frame"},

    }

wsConn, resp, err := websocket.NewClient(rawConn, u, wsHeaders, 1024, 1024)

anything wrong ? thanks~

@mbohlool

[lucid-at-dream]

Hi! I'm getting this same error now in version 8.0.1

Here's the offending code:

            v1 = client.CoreV1Api()
            all_pods = [pod.metadata.name for pod in v1.list_namespaced_pod(self.namespace).items]
            pod = next(pod for pod in all_pods if self.name in pod)

            api_response = v1.connect_get_namespaced_pod_exec(
                pod, self.namespace,
                container=self.pod_template.containers[0].name,
                command=['ps', 'axu'],
                stdin=False, stdout=True, stderr=True,
                tty=False
            )

And here's the exception that is being raised:

self = <kubernetes.client.rest.RESTClientObject object at 0x7f870f136cc0>, method = 'GET', url = 'https://****************:443/api/v1/namespaces/elastictests-ozxuefet/pods/test-elastic-cluster-77cf4bfdf9-rsslx/exec'
query_params = [('command', ['ps', 'axu']), ('container', 'elastic-0'), ('stderr', True), ('stdin', False), ('stdout', True), ('tty', False)]
headers = {'Accept': '*/*', 'Content-Type': 'application/json', 'User-Agent': 'Swagger-Codegen/8.0.1/python', 'authorization': 'Bearer f725ca479019a546e359f8a5b7ce9b9d'}, body = None, post_params = {}, _preload_content = True, _request_timeout = None

    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True, _request_timeout=None):
        """
        :param method: http request method
        :param url: http request url
        :param query_params: query parameters in the url
        :param headers: http request headers
        :param body: request json body, for `application/json`
        :param post_params: request post parameters,
                            `application/x-www-form-urlencoded`
                            and `multipart/form-data`
        :param _preload_content: if False, the urllib3.HTTPResponse object will be returned without
                                 reading/decoding response data. Default is True.
        :param _request_timeout: timeout setting for this request. If one number provided, it will be total request
                                 timeout. It can also be a pair (tuple) of (connection, read) timeouts.
        """
        method = method.upper()
        assert method in ['GET', 'HEAD', 'DELETE', 'POST', 'PUT', 'PATCH', 'OPTIONS']
    
        if post_params and body:
            raise ValueError(
                "body parameter cannot be used with post_params parameter."
            )
    
        post_params = post_params or {}
        headers = headers or {}
    
        timeout = None
        if _request_timeout:
            if isinstance(_request_timeout, (int, ) if PY3 else (int, long)):
                timeout = urllib3.Timeout(total=_request_timeout)
            elif isinstance(_request_timeout, tuple) and len(_request_timeout) == 2:
                timeout = urllib3.Timeout(connect=_request_timeout[0], read=_request_timeout[1])
    
        if 'Content-Type' not in headers:
            headers['Content-Type'] = 'application/json'
    
        try:
            # For `POST`, `PUT`, `PATCH`, `OPTIONS`, `DELETE`
            if method in ['POST', 'PUT', 'PATCH', 'OPTIONS', 'DELETE']:
                if query_params:
                    url += '?' + urlencode(query_params)
                if re.search('json', headers['Content-Type'], re.IGNORECASE):
                    if headers['Content-Type'] == 'application/json-patch+json':
                        if not isinstance(body, list):
                            headers['Content-Type'] = \
                                'application/strategic-merge-patch+json'
                    request_body = None
                    if body is not None:
                        request_body = json.dumps(body)
                    r = self.pool_manager.request(method, url,
                                                  body=request_body,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                elif headers['Content-Type'] == 'application/x-www-form-urlencoded':
                    r = self.pool_manager.request(method, url,
                                                  fields=post_params,
                                                  encode_multipart=False,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                elif headers['Content-Type'] == 'multipart/form-data':
                    # must del headers['Content-Type'], or the correct Content-Type
                    # which generated by urllib3 will be overwritten.
                    del headers['Content-Type']
                    r = self.pool_manager.request(method, url,
                                                  fields=post_params,
                                                  encode_multipart=True,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                # Pass a `string` parameter directly in the body to support
                # other content types than Json when `body` argument is provided
                # in serialized form
                elif isinstance(body, str):
                    request_body = body
                    r = self.pool_manager.request(method, url,
                                                  body=request_body,
                                                  preload_content=_preload_content,
                                                  timeout=timeout,
                                                  headers=headers)
                else:
                    # Cannot generate the request from given parameters
                    msg = """Cannot prepare a request message for provided arguments.
                             Please check that your arguments match declared content type."""
                    raise ApiException(status=0, reason=msg)
            # For `GET`, `HEAD`
            else:
                r = self.pool_manager.request(method, url,
                                              fields=query_params,
                                              preload_content=_preload_content,
                                              timeout=timeout,
                                              headers=headers)
        except urllib3.exceptions.SSLError as e:
            msg = "{0}\n{1}".format(type(e).__name__, str(e))
            raise ApiException(status=0, reason=msg)
    
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.rest.ApiException: (400)
E           Reason: Bad Request
E           HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 07 Feb 2019 14:03:23 GMT', 'Content-Length': '139'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Upgrade request required","reason":"BadRequest","code":400}

/root/.local/share/virtualenvs/stratio-atf-4Tz-qxfE/lib/python3.7/site-packages/kubernetes/client/rest.py:222: ApiException

I'm using pipenv to manage the working environment. Here are the packages I have installed and corresponding versions:

In [11]: vs = json.loads(open('Pipfile.lock','r').read())
In [12]: [(i, vs['default'][i]['version']) for i in vs['default']]
Out[12]: 
[('adal', '==1.2.1'),
 ('apipkg', '==1.5'),
 ('asn1crypto', '==0.24.0'),
 ('atomicwrites', '==1.3.0'),
 ('attrs', '==18.2.0'),
 ('cachetools', '==3.1.0'),
 ('certifi', '==2018.11.29'),
 ('cffi', '==1.11.5'),
 ('chardet', '==3.0.4'),
 ('coverage', '==4.5.2'),
 ('cryptography', '==2.5'),
 ('elasticsearch', '==6.3.1'),
 ('execnet', '==1.5.0'),
 ('google-auth', '==1.6.2'),
 ('idna', '==2.8'),
 ('kubernetes', '==8.0.1'),
 ('more-itertools', '==5.0.0'),
 ('oauthlib', '==3.0.1'),
 ('pluggy', '==0.8.1'),
 ('py', '==1.7.0'),
 ('pyasn1', '==0.4.5'),
 ('pyasn1-modules', '==0.2.4'),
 ('pycparser', '==2.19'),
 ('pyjwt', '==1.7.1'),
 ('pytest', '==4.2.0'),
 ('pytest-cov', '==2.6.1'),
 ('pytest-forked', '==1.0.1'),
 ('pytest-xdist', '==1.26.1'),
 ('python-dateutil', '==2.8.0'),
 ('pyyaml', '==3.13'),
 ('requests', '==2.21.0'),
 ('requests-oauthlib', '==1.2.0'),
 ('rsa', '==4.0'),
 ('six', '==1.12.0'),
 ('urllib3', '==1.24.1'),
 ('websocket-client', '==0.54.0')]

It seems like a regression to me, but I may be wrong. Could you guys help me out figuring this out?

I'm using Python v3.7.2

[lucid-at-dream]

nevermind! Found a fix!
For those looking and reaching this issue, I found the fix looking here:
#409
and here
#526