Internal Error w/ aio-pika==1.7.0 mypy=0.550

[xlevus]

Reporting an INTERNAL_ERROR as requested.

The error appears to happen both from within my project and when running directy against aio_pika.

I am not the developer of aio-pika.

$ pip freeze | grep "mypy\|aio-pika" 
aio-pika==1.7.0
flake8-mypy==17.8.0
mypy==0.550

$ mypy -p common --show-traceback                                                                                                                                                               
/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/aio_pika/patterns/rpc.py:186: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.550
Traceback (most recent call last):
  File "/usr/lib64/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib64/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/__main__.py", line 11, in <module>
    main(None)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/main.py", line 66, in main
    res = type_check_only(sources, bin_dir, options)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/main.py", line 119, in type_check_only
    options=options)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/build.py", line 214, in build
    graph = dispatch(sources, manager)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/build.py", line 1935, in dispatch
    process_graph(graph, manager)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/build.py", line 2199, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/build.py", line 2327, in process_stale_scc
    graph[id].type_check_first_pass()
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/build.py", line 1836, in type_check_first_pass
    self.type_checker.check_first_pass()
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 196, in check_first_pass
    self.accept(d)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 690, in accept
    return visitor.visit_class_def(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1185, in visit_class_def
    self.accept(defn.defs)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 750, in accept
    return visitor.visit_block(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1305, in visit_block
    self.accept(s)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 567, in accept
    return visitor.visit_decorator(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 2470, in visit_decorator
    self.check_func_item(e.func, name=e.func.name())
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 591, in check_func_item
    self.check_func_def(defn, typ, name)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 751, in check_func_def
    self.accept(item.body)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 750, in accept
    return visitor.visit_block(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1305, in visit_block
    self.accept(s)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 952, in accept
    return visitor.visit_try_stmt(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 2265, in visit_try_stmt
    self.visit_try_without_finally(s, try_frame=bool(s.finally_body))
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 2300, in visit_try_without_finally
    self.accept(s.body)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 750, in accept
    return visitor.visit_block(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1305, in visit_block
    self.accept(s)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 283, in accept
    stmt.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 794, in accept
    return visitor.visit_assignment_stmt(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1312, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checker.py", line 1406, in check_assignment
    self.infer_variable_type(inferred, lvalue, self.expr_checker.accept(rvalue),
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 2363, in accept
    typ = node.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 1245, in accept
    return visitor.visit_yield_from_expr(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 2486, in visit_yield_from_expr
    subexpr_type = self.accept(e.expr)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 2363, in accept
    typ = node.accept(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/nodes.py", line 1235, in accept
    return visitor.visit_call_expr(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 268, in visit_call_expr
    ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname, object_type)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 505, in check_call_expr_with_callee_type
    object_type=object_type)[0]
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/checkexpr.py", line 563, in check_call
    callee = freshen_function_type_vars(callee)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/expandtype.py", line 46, in freshen_function_type_vars
    fresh = cast(CallableType, expand_type(callee, tvmap)).copy_modified(variables=tvdefs)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/expandtype.py", line 16, in expand_type
    return typ.accept(ExpandTypeVisitor(env))
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/types.py", line 748, in accept
    return visitor.visit_callable_type(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/expandtype.py", line 97, in visit_callable_type
    return t.copy_modified(arg_types=self.expand_types(t.arg_types),
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/expandtype.py", line 132, in expand_types
    a.append(t.accept(self))
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/types.py", line 1424, in accept
    return visitor.visit_forwardref_type(self)
  File "/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/mypy/types.py", line 1509, in visit_forwardref_type
    raise RuntimeError('Internal error: unresolved forward reference')
RuntimeError: Internal error: unresolved forward reference
/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/aio_pika/patterns/rpc.py:186: : note: use --pdb to drop into pdb

$ mypy -p aio_pika 
... <snip> ...
/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/aio_pika/patterns/rpc.py:186: error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.550
/home/USER/.virtualenvs/backoffice-py/lib64/python3.6/site-packages/aio_pika/patterns/rpc.py:186: : note: please use --show-traceback to print a traceback when reporting a bug

[ilevkivskyi]

Hm, this looks important. Could you please provide the code you checked with mypy that triggers this (or some hints about it, if it is private). Are there union types or generics involved? It looks like this is triggered by some assignment statement that looks like this:

x = yield from function_call(...)

In particular could you please tell are there forward references (annotations with types that are defined later in code) involved?

[xlevus]

@ilevkivskyi I am able to reproduce the INTERNAL_ERROR independently from my project with the following:

$ python3 -m venv venv
$ source venv/bin/activate
$ pip install mypy=0.550 aio-pika==1.7.0
$ MYPYPATH=venv/lib/python3.6/site-packages mypy -p aio_pika --show-traceback

I don't actually use the aio_pika.packages.rpc module (this appears to be an addition ~6 hours ago). Given this, and the re-produce-ability outside of my project, I don't expect the error is caused by any of my code.

I'm also not a contributor to aio-pika, and have worked-around this by ignoring the module in my mypy.ini.

[ilevkivskyi]

OK, so it involves type-checking something in site-packages. This is discouraged until PEP 561 is fully implemented, so this is a lower priority. I think I have found the root cause, they write:

P = [TypeVar('P')]
# ... then later
    result = yield from self.execute(func, payload)
# ... and even later
    @asyncio.coroutine
    def execute(self, func: Callable[P, R], payload: P) -> R:
        ...

[gvanrossum]

We got the same exception (though with a somewhat different traceback). The user sent me a repro (update: which I have not managed to confirm):

class BaseClass(object):
    def some_method(self):
        # type: () -> BaseClass
        pass
class SubClass1(BaseClass):
    def some_method(self):
        # type: () -> AnotherNameForSubClass1  # <--- Forward ref to alias causes crash
        pass
AnotherNameForSubClass1 = SubClass1

The traceback ended in:

File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checker.py", line 2101, in visit_return_stmt
self.check_return_stmt(s)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checker.py", line 2132, in check_return_stmt
allow_none_return=allow_none_func_call)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 2365, in accept
typ = node.accept(self)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/nodes.py", line 1245, in accept
return visitor.visit_call_expr(self)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 268, in visit_call_expr
ret_type = self.check_call_expr_with_callee_type(callee_type, e, fullname, object_type)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 505, in check_call_expr_with_callee_type
object_type=object_type)[0]
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 578, in check_call
messages=arg_messages)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 1048, in check_argument_types
actual + 1, i + 1, callee, context, messages)
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 1080, in check_arg
elif not is_subtype(caller_type, callee_type):
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/subtypes.py", line 83, in is_subtype
ignore_declared_variance=ignore_declared_variance))
File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/types.py", line 1415, in accept
    return visitor.visit_forwardref_type(self)
  File "/Users/xxx/src/server/.mypy/venv/lib/python3.5/site-packages/mypy/types.py", line 1500, in visit_forwardref_type
    raise RuntimeError('Internal error: unresolved forward reference')
RuntimeError: Internal error: unresolved forward reference

[ilevkivskyi]

@gvanrossum I have a fix for the original crash, but I cannot reproduce your crash, there should be something else.

[ilevkivskyi]

PR #4375 should fix the original crash.

[gvanrossum]

I've made another issue for the crash my use saw: #4376