Error: Item "None" of "Optional[Popen]" has no attribute "communicate"

[furgerf]

Hi, I'm trying to use mypy for a new project which triggers the following error and asks me to report it:

minion/src/utils.py:46: error: Item "None" of "Optional[Popen]" has no attribute "communicate"
Traceback (most recent call last):
  File "venv/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "<dir>/venv/lib/python3.5/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/main.py", line 92, in main
    res = build.build(sources, options, None, flush_errors, fscache)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 155, in build
    result = _build(sources, options, alt_lib_path, flush_errors, fscache)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 204, in _build
    graph = dispatch(sources, manager)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 2172, in dispatch
    process_graph(graph, manager)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 2469, in process_graph
    process_stale_scc(graph, scc, manager)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 2592, in process_stale_scc
    graph[id].type_check_first_pass()
  File "<dir>/venv/lib/python3.5/site-packages/mypy/build.py", line 1730, in type_check_first_pass
    self.type_checker().check_first_pass()
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 282, in check_first_pass
    self.accept(d)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 393, in accept
    stmt.accept(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/nodes.py", line 608, in accept
    return visitor.visit_func_def(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 704, in visit_func_def
    self._visit_func_def(defn)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 708, in _visit_func_def
    self.check_func_item(defn, name=defn.name())
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 770, in check_func_item
    self.check_func_def(defn, typ, name)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 934, in check_func_def
    self.accept(item.body)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 393, in accept
    stmt.accept(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/nodes.py", line 910, in accept
    return visitor.visit_block(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 1688, in visit_block
    self.accept(s)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 393, in accept
    stmt.accept(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/nodes.py", line 965, in accept
    return visitor.visit_assignment_stmt(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 1696, in visit_assignment_stmt
    self.check_assignment(s.lvalues[-1], s.rvalue, s.type is None, s.new_syntax)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 1730, in check_assignment
    infer_lvalue_type)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 2111, in check_assignment_to_multiple_lvalues
    self.check_multi_assignment(lvalues, rvalue, context, infer_lvalue_type)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checker.py", line 2136, in check_multi_assignment
    rvalue_type = rv_type or self.expr_checker.accept(rvalue)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 3007, in accept
    typ = node.accept(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/nodes.py", line 1418, in accept
    return visitor.visit_call_expr(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 229, in visit_call_expr
    return self.visit_call_expr_inner(e, allow_none_return=allow_none_return)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 271, in visit_call_expr_inner
    callee_type = self.accept(e.callee, type_context, always_allow_any=True)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 3007, in accept
    typ = node.accept(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/nodes.py", line 1370, in accept
    return visitor.visit_member_expr(self)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 1640, in visit_member_expr
    result = self.analyze_ordinary_member_access(e, is_lvalue)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkexpr.py", line 1655, in analyze_ordinary_member_access
    original_type=original_type, chk=self.chk)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkmember.py", line 122, in analyze_member_access
    for subtype in typ.relevant_items()]
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkmember.py", line 122, in <listcomp>
    for subtype in typ.relevant_items()]
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkmember.py", line 115, in analyze_member_access
    original_type=original_type, chk=chk)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkmember.py", line 105, in analyze_member_access
    original_type=original_type, chk=chk)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/checkmember.py", line 310, in analyze_member_var_access
    return msg.has_no_attr(original_type, itype, name, node)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/messages.py", line 501, in has_no_attr
    typ_format, self.format(original_type), member, extra), context)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/messages.py", line 198, in fail
    self.report(msg, context, 'error', file=file, origin=origin)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/messages.py", line 193, in report
    origin_line=origin.get_line() if origin else None)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/errors.py", line 260, in report
    self.add_error_info(info)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/errors.py", line 281, in add_error_info
    self._add_error_info(file, info)
  File "<dir>/venv/lib/python3.5/site-packages/mypy/errors.py", line 263, in _add_error_info
    assert file not in self.flushed_files
AssertionError: 

Python 3.5.3, mypy 0.641 installed via pip in a virtualenv.

Offending code (I've marked line 46 where the error occurs):

def get_output_of_command(command) -> tuple:
  if not isinstance(command, list) or not isinstance(command[0], list):
    # if the command is either not a list (because it's a string) or because its first element is
    # not a list, then we want to wrap the command in a list (to treat it as "one pipe")
    command = [command]

  current_process = None
  for pipe_command in command:
    if not isinstance(pipe_command, list):
      # if the pipe's command isn't a list (because it's a string), we want to wrap it in a list
      pipe_command = [pipe_command]

    assert all([isinstance(part, str) for part in pipe_command])

    if current_process:
      current_process = Popen(pipe_command, stdin=current_process.stdout, stdout=PIPE, stderr=PIPE)
    else:
      current_process = Popen(pipe_command, stdout=PIPE, stderr=PIPE)
  output, error = current_process.communicate() # <--- line 46

  return output.strip().decode("utf-8"), error.strip().decode("utf-8")

Hopefully this is useful to you, have a nice day :-)

[msullivan]

The type error that gets printed is genuine: mypy doesn't track information like whether a list must be non-empty, so it can't tell that the for loop must execute at least once, which is what is required for current_process to be non-None. You can suppress the type error with an assert current_process is not None after the loop.

The bug here is the crash, which happens as mypy tries to print out errors from a file that it has already printed errors from.

Finding the cause here will require more information about the structure of your source code and how you are invoking mypy.

An obvious culprit would be if minion/src/utils.py is somehow getting handed to mypy twice---mypy ought to reject this, but maybe if it somehow considered it as having two different module names?

[furgerf]

I see, thanks for the explanation - that makes sense.

The function gets imported multiple times (but always like this: from utils import get_output_of_command).

The mypy invocation isn't really special, I set the MYPYPATH, and add --ignore-missing-imports since I'm using coloredlogs which seems to be missing type definitions if I understood correctly.

It's fine for me if you want to close the issue, or I can provide further information if desired.

[msullivan]

Could you share what you set MYPYPATH to and what your full command line is?

[msullivan]

I suspect this is a duplicate of #4881, where the file is getting pulled in once via the command line and once via MYPYPATH, with different module names.

Probably you shouldn't need to set MYPYPATH? Do you have an __init__.py in src that could be confusing mypy?

[furgerf]

Sure, I'm calling this from a makefile which executes @MYPYPATH="$$MYPYPATH:minion/src" $(BIN)/mypy minion/src/ --ignore-missing-imports $(ARGS)

So minion/src contains all python scripts (for now), including utils.py. BIN points to the virtual environment and ARGS is empty.

By the way, I need assert current_process, assert command is insufficient, even if this does ensure that current_process isn't None - though I guess that is more of a "missing feature" than a bug.

Edit: Yes I have an __init__.py and no, I do have to set the MYPYPATH.

[ilevkivskyi]

OK, then closing this as a duplicate of #4881. You can open a separate issue for the feature request (if you want of course).

[msullivan]

@furgerf There probably shouldn't be an __init__.py in src, though? Since you presumably don't want a package named src. If you remove that, I think you shouldn't need the MYPYPATH?