"Union[str, Path]" is incompatible with "Union[str, bytes, int, _PathLike[Any]]"

[sametmax]

I have some code doing:

def extract(path: Union[str, Path]) -> Generator:
    with open(path) as f:

Mypy complains:

 Argument 1 to "open" has incompatible type "Union[str, Path]"; expected "Union[str, bytes, int, _PathLike[Any]]"

But it seems to me that Union[str, Path] is an subset of Union[str, bytes, int, _PathLike[Any]], or am I mistaken ?

[gvanrossum]

What mypy version? What's your full program? I cannot repro, with this example:

from pathlib import Path
from typing import Union, Generator

def extract(path: Union[str, Path]) -> Generator:
    with open(path) as f:
        for line in f:
            yield line

[sametmax]

Sorry I should have given more info up front. I should know better.

Shell session:

➤ mypy analyse.py 
/home/user/.local/lib/mypy/typeshed/stdlib/3.5/pathlib.pyi:15: error: Name 'List' is not defined
analyse.py:6: error: Argument 1 to "open" has incompatible type "Union[str, Path]"; expected "Union[str, bytes, int, _PathLike[Any]]"
user@laptop:~/B/docapost
➤ mypy --version
mypy 0.540

Minimal example:

from typing import Union
from pathlib import Path

def extract(path: Union[str, Path]):
    open(path)

So Python 3.5 64bits on Ubuntu 16.04 installed with --user and mypy 0.540.

I just tried with a mypy from Python 3.6 and the error is not here anymore.

[JelleZijlstra]

I guess it's because there is no PathLike protocol on 3.5, so "_PathLike" there does not include Path.

There might be something weird going on with the error. In typeshed, open is defined as follows:

if sys.version_info >= (3, 6):
    # This class is to be exported as PathLike from os,
    # but we define it here as _PathLike to avoid import cycle issues.
    # See https://github.com/python/typeshed/pull/991#issuecomment-288160993
    class _PathLike(Generic[AnyStr]):
        def __fspath__(self) -> AnyStr: ...

    def open(file: Union[str, bytes, int, _PathLike], mode: str = 'r', buffering: int = -1, encoding: Optional[str] = None,
             errors: Optional[str] = None, newline: Optional[str] = None, closefd: bool = ...) -> IO[Any]: ...
else:
    def open(file: Union[str, bytes, int], mode: str = 'r', buffering: int = -1, encoding: Optional[str] = None,
             errors: Optional[str] = None, newline: Optional[str] = None, closefd: bool = ...) -> IO[Any]: ...

So on 3.5, _PathLike is not part of the accepted type for the file argument, but nevertheless mypy's error mentions _PathLike.

[JelleZijlstra]

To be clear, this means mypy found a real bug in your code—in Python 3.5, open does not accept Path objects.

[sametmax]

To be clear, this means mypy found a real bug in your code—in Python 3.5, open does not accept Path objects.

Well not really. The code is targeting Python 3.6. It just happens that mypy was installed with "python3.5 install --user mypy".
Now I didn't realize that, and I'm pretty sure a lot of people won't. I have 6 versions of Python installed on this machine. One other partition with more, and 2 other laptops. It's quite hard to keep track.
Maybe we should print some info at the beginning of mypy's output, stating for what version of typing + Python + mypy we are checking against. And add a hook for IDE to get the info in a machine friendly format.

So on 3.5, _PathLike is not part of the accepted type for the file argument, but nevertheless mypy's error mentions _PathLike.

Ok so there is some kind of bug. But given my setup, it's probably some weird interaction between all the Pythons stdlib, .local installs and pipsi virtualenvs. So maybe it's just me. Can your reproduce the mistaken _PathLike mention ?

[JelleZijlstra]

You should probably be running mypy with --python-version 3.6 then.

[JelleZijlstra]

And it doesn't repro for me in this setup:

$ mypy --python-version 3.5 bin/pathlike.py 
bin/pathlike.py:5: error: Argument 1 to "open" has incompatible type "Union[str, Path]"; expected "Union[str, bytes, int]"
$ mypy --version
mypy 0.550-dev-8895d18df9ee290ef9f872cd16b58fce8aa14824
$ python --version
Python 3.6.1

[gvanrossum]

The python version used to check against is independent from the Python version used to run mypy. And the default is 3.6. I don't see a bug here.

[JelleZijlstra]

The potential bug is that the error message is incorrect (it talks about PathLike even though there is no PathLike on Python 3.5). But I can't reproduce that bug, so @sametmax any objections to closing this?

[ilevkivskyi]

@sametmax
I am worried about this line in your output:

/home/user/.local/lib/mypy/typeshed/stdlib/3.5/pathlib.pyi:15: error: Name 'List' is not defined

It looks like you might have a broken installation.

[gvanrossum]

Note, `_PathLike` (with `_`) is defined in builtins.pyi for 3.6 only.

[sametmax]

@JelleZijlstra: I'm ok with closing it.

However, I'm expecting many others to make the same mistake of running mypy with the wrong python interpreter, not realizing how important it is. Maybe we should open another issue about printing some warning by mypy when not using explicitly "--python-version".

Particularly, I can't help but notice that some of my editors (e.g: sublime text and vscode) have mypy plugins, yet no settings for choosing "--python-version". The issue may not be very well known.

Mypy could do a check on the command line args and if it fails, start with something like something like:

 warnings.warn( 
    "Using mypy without '--python-version', the code will be implicitly "
    "checked against the python mypy is installed with: CPython 3.5",
    RuntimeWarning
)

Or make the python version a mandatory positional argument. But that would break compat and you probably don't want that.

Not only will it help people to prevent them from making the same mistake that I did (or fix it quickly), but it will also save you time by avoiding other tickets like this one in the future. Indeed, the same mistake could produce very different error messages that would be unlikely to be linked to this current ticket by a future dev encountering them.

In any case, thanks for the help.

[gvanrossum]

"UserWarning: using mypy without '--python-version', the code will be implicitly checked against the python mypy is installed with: 3.5"

But that's not how it works! What makes you think it works that way? It defaults to 3.6 regardless what you run to install it.

[sametmax]

So it defaults to the last stable python version that was released when this particular mypy version went out ?

In that case, the same warning would be useful. A lot of users have several Python versions installed, but I don't think somebody would think about installing mypy twice.