Support new Python 3.5 features

[JukkaL]

Python 3.5 has a bunch of new things mypy doesn't quite support yet:

https://docs.python.org/3/whatsnew/3.5.html

This is an umbrella task for all Python 3.5 related issues (except for PEP 484 related tasks).

[o11c]

Some previous discussion: #487

Issue #705: Matmul operator

WIP PR #880: Parser redesign

[JukkaL]

Also #706 (async and await)

[JukkaL]

In every Python release there are also a bunch of stub changes.

[JukkaL]

I'm going to focus on Python 2 and better pre-3.5 Python 3 support first, so taking away the priority tag. The rationale is that way more people are using older Python releases and they still aren't well supported by mypy. PRs that add new Python 3.5 features are still welcome, though.

[avanov]

Hi @JukkaL

How much time would it take you to resolve the following issue with 3.5 compatibility:

$ mypy .
Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/latest-3/bin/mypy", line 14, in <module>
    from mypy import build
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/build.py", line 19, in <module>
    from typing import Undefined, Dict, List, Tuple, cast, Set, Union
ImportError: cannot import name 'Undefined'

It seems that it is not compatible with the standard typing package of 3.5. I am not sure about the popularity of 3.5, but from my point of view there is no reason to provide compatibility for 3.x branch versions lower than 3.5 exactly because of the fact that mypy is under development and 3.x is still less popular than 2.x. The reason for prioritising support for 3.5 on the other hand is the presence of the typing in the stdlib.

I'm using Python 3.5 in production, btw.

[refi64]

I think a new release would fix this. @avanov I believe this already works in master.

[JukkaL]

Yeah, we need to cut a new release to get at least minimal Python 3.5 compatibility. @avanov Let me know if using github master works for you.

A lot has happened since the last release so preparing release notes is going to be some work. Also there are a few major bugs that I'd like to fix before the release.

[avanov]

@kirbyfan64, @JukkaL thanks for the tip,
I've just checked the master and it does fix the above error, which is good. Still it is unable to handle new async / await syntax.

[avanov]

And there's something wrong with asyncio.futures.Future parsing:

$ mypy -m <my_project_module> --verbose
Traceback (most recent call last):
  File "/home/vagrant/.pyenv/versions/latest-3/bin/mypy", line 6, in <module>
    exec(compile(open(__file__).read(), __file__, 'exec'))
  File "/projects/mypy/scripts/mypy", line 6, in <module>
    main(__file__)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/main.py", line 49, in main
    type_check_only(sources, bin_dir, options)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/main.py", line 92, in type_check_only
    python_path=options.python_path)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/build.py", line 197, in build
    result = manager.process(initial_states)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/build.py", line 379, in process
    next.process()
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/build.py", line 849, in process
    self.type_checker().visit_file(self.tree, self.tree.path)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 371, in visit_file
    self.accept(d)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 473, in accept
    return visitor.visit_class_def(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 866, in visit_class_def
    self.accept(defn.defs)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 514, in accept
    return visitor.visit_block(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 939, in visit_block
    self.accept(s)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 401, in accept
    return visitor.visit_decorator(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1697, in visit_decorator
    e.func.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 370, in accept
    return visitor.visit_func_def(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 431, in visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 464, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 545, in check_func_def
    self.accept_in_frame(item.body)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 391, in accept_in_frame
    answer = self.accept(node, type_context)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 514, in accept
    return visitor.visit_block(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 939, in visit_block
    self.accept(s)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 730, in accept
    return visitor.visit_with_stmt(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1722, in visit_with_stmt
    self.accept(s.body)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 514, in accept
    return visitor.visit_block(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 939, in visit_block
    self.accept(s)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 681, in accept
    return visitor.visit_if_stmt(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1464, in visit_if_stmt
    self.accept(b)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 514, in accept
    return visitor.visit_block(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 939, in visit_block
    self.accept(s)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 378, in accept
    typ = node.accept(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/nodes.py", line 611, in accept
    return visitor.visit_return_stmt(self)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1328, in visit_return_stmt
    'asyncio.futures.Future', s)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1345, in wrap_generic_type
    check_type)
  File "/home/vagrant/.pyenv/versions/3.5.0/envs/latest-3/lib/python3.5/site-packages/mypy/checker.py", line 1358, in count_nested_types
    typ = map_instance_to_supertype(self.named_generic_type(check_type, typ.args),
AttributeError: 'NoneTyp' object has no attribute 'args'

[rowillia]

@JukkaL I totally understand the decision to support 2 before 3.5, but one counterpoint to bring into the discussion would be the folks who are willing to adopt 3.5 are also likely to be the folks who are willing to adopt types. I was pretty bummed to learn I'll have to choose between types and async/await for now.

I think 3.5 will be a banner release for Python 3. The combination of type hinting and async/await made me choose it for my latest project, and I know a lot of other developers are excited about this release for that same reason. It'd be amazing if these two worked well together with Mypy and we could start developing awesome examples of modern Python.

[JukkaL]

@avanov Could you file a separate issue for that crash, and maybe try to come up with a short code sample that reproduces the crash? The latest github master should provide more information in the traceback.

@rowillia I know -- it's a hard decision. I probably won't have any spare bandwidth in the next several weeks, but I might look at 3.5 support (at least async/await) in early 2016. If somebody sends a PR I'll do my best to review it pretty quickly.

[avanov]

@JukkaL I'll do later this week. Thanks for the feedback!

[rowillia]

@JukkaL Do you guys have any docs for your Python 2 plans? Are you going to backport PEP 3107? It'd be great if this used the same format as PyCharm https://www.jetbrains.com/pycharm/help/type-hinting-in-pycharm.html#d302201e159 since they've already got some tooling around supporting it.

EDIT: Sorry @gvanrossum, I'll leave this for 3.5 features. If I have some time I'll look into implementing async/await support.

[gvanrossum]

We're not planning to backport function annotations to Python 2.7.
We're also not introducing a new notation for types, nor are we planning to parse docstrings.
Instead, we're adopting a comment-based notation for Python 2.7. The gist of it is as follows:

def foo(x, y):
    # type: (int, Iterable[int]) -> Dict[int, int]
    """Optional docstring goes here"""

Once we're happy with how this works out we will document it. (Please, if you have more questions about this topic, open a new issue -- this issue should really be reserved for new 3.5 features such as async/await.)

[gvanrossum]

IMO main thing is async def and friends (PEP 492). See python/typing#119

[gvanrossum]

Closing in favor of a more specific task, #1453.