Relative import from toplevel cause exception

[aquavitae]

I have a folder structure something like this:

root
    setup.py
    project
        __init__.py
        cli.py
        lib
            __init__.py
            options.py

cli.py contains a relative import, from .lib import options. Running mypy from the root works fine, but if I run it from inside project, I get an exception:

~/root/project $ mypy -m cli
Traceback (most recent call last):
  File "/usr/bin/mypy", line 6, in <module>
    main(__file__)
  File "/usr/lib/python3.6/site-packages/mypy/main.py", line 42, in main
    res = type_check_only(sources, bin_dir, options)
  File "/usr/lib/python3.6/site-packages/mypy/main.py", line 86, in type_check_only
    options=options)
  File "/usr/lib/python3.6/site-packages/mypy/build.py", line 183, in build
    dispatch(sources, manager)
  File "/usr/lib/python3.6/site-packages/mypy/build.py", line 1525, in dispatch
    graph = load_graph(sources, manager)
  File "/usr/lib/python3.6/site-packages/mypy/build.py", line 1631, in load_graph
    ancestor_for=st)
  File "/usr/lib/python3.6/site-packages/mypy/build.py", line 1119, in __init__
    assert id or path or source is not None, "Neither id, path nor source given"
AssertionError: Neither id, path nor source given

Digging into the source code, it looks like the issue is in add_ancestors(). When the relative path in encountered, parent == '.lib', so parent.rsplit('.', 1) == ['', 'lib']. The empty string gets added to ancestors, is eventually used as the id parameter when initializing State, and this assertion then fails.

I'm not certain exactly how the module resolver is supposed to handle relative imports so I'm not clear on what is needed to fix this.

[gvanrossum]

It seems project is your top-level package but you don't show an __init__.py file there. Mypy uses those to figure out where the top-level package is. Does adding that help?

(The assert is still a bug, the code should produce a user-facing error message about a relative import in what it thinks of as a top-level module, cli.py.)

PS. If you really don't want to add an __init__.py file, you could add a __init__.pyi file -- it will have the same effect for mypy but will be ignored when you run the code.

[aquavitae]

Sorry, indentation not quite right in the original post - I've corrected it and made it a little clearer. There is an __init__.py file in project so that's not the issue. In any case, the package itself runs perfectly well and I don't think it would if the top-level __init__.py was missing.

I've also noticed that if I use absolute imports (i.e. from project.lib import options) mypy runs but reports Cannot find module named 'project.lib', so it does look like it's not finding the top-level package.

Can anyone reproduce this? My python version is 3.6.0 and mypy is 0.501.

[gvanrossum]

Sorry to hear it. Can you publish the full source code and exactly how you run mypy to repro this? There's probably something specific in it that causes the breakage.

[aquavitae]

Here is an example project that shows the issue: https://github.com/aquavitae/mypy-test. I hope it is just something I'm doing wrong!

[gvanrossum]

OK, the user error is that in the crashing runs you are attempting to treat cli.py as a top-level module, but a top-level module cannot use a relative import.

The mypy bug is that it doesn't give a reasonable error, but instead just crashes.

[aquavitae]

I didn't realize there was that limitation with relative imports. But surely the code shouldn't run then? Also, if I change it to use an absolute import (from project.lib import options) mypy run inside the project gives the error cli.py:1: error: Cannot find module named 'project.lib'

[gvanrossum]

Indeed, that code won't run from inside the project dir.

Changing it to an absolute import requires Python (and mypy) to be able to find the toplevel module -- which means that you would have to have something like PYTHONPATH=.. or MYPYPATH=.. -- but that's a bad idea, you just shouldn't be running stuff from inside the package.

(I feel that you ought to educate yourself about the ins and outs of this issue independently -- the mypy issue should be purely about the crash, which should be fixed so it produces a regular error message.)

[aquavitae]

Thanks, and you're right - I'll read up more on it.

[cpitclaudel]

I'm running into this issue, and unfortunately I'm not quite following the explanation. I'm well aware of the issue about running a module inside a package, but I'm not seeing the connection to this issue (I'm not trying to run my program, just to typecheck it).

What's the proper way to typecheck a file that uses relative imports?

[FichteFoll]

You typecheck from the project folder (and outside the package) and then select the file you want to have reports about. Just like you can't run something with relative imports from within the package, mypy will refuse to lint something that does this.

The solution is to go up one level and outside of the package.

[cpitclaudel]

Thanks, but I still don't follow: why doesn't mypy chose the appropriate directory on its own?
Additionally, why does the current directory matter?

[gvanrossum]

In order to determine the correct module name of the file you pass it, mypy searches up until it finds a directory that does *not* contain __init__.py. However it doesn't search above the current directory (unless you gave it an absolute path).

[cpitclaudel]

In order to determine the correct module name of the file you pass it, mypy searches up until it finds a directory that does not contain init.py.

Thanks, that makes sense.

However it doesn't search above the current directory (unless you gave it an absolute path).

But I do :/ Should it work in that case?

[cpitclaudel]

To clarify, the context is automatic linting (in an IDE plugin); I'm just trying to get a sense of how we should call mypy so that it works without requiring user configuration.

[cpitclaudel]

I guess this is the same error?

error: INTERNAL ERROR -- please report a bug at https://github.com/python/mypy/issues version: 0.570
Traceback (most recent call last):
  File "/home/clement/.local/bin/mypy", line 11, in <module>
    sys.exit(console_entry())
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/__main__.py", line 7, in console_entry
    main(None)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/main.py", line 80, in main
    type_check_only(sources, bin_dir, options, flush_errors)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/main.py", line 129, in type_check_only
    flush_errors=flush_errors)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 180, in build
    result = _build(sources, options, alt_lib_path, bin_dir, saved_cache, flush_errors)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 266, in _build
    graph = dispatch(sources, manager)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 2112, in dispatch
    process_graph(graph, manager)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 2413, in process_graph
    process_stale_scc(graph, scc, manager)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 2589, in process_stale_scc
    graph[id].semantic_analysis()
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/build.py", line 1947, in semantic_analysis
    self.manager.semantic_analyzer.visit_file(self.tree, self.xpath, self.options, patches)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 294, in visit_file
    self.accept(d)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 3962, in accept
    node.accept(self)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/nodes.py", line 711, in accept
    return visitor.visit_class_def(self)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 675, in visit_class_def
    with self.analyze_class_body(defn) as should_continue:
  File "/usr/lib/python3.5/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 684, in analyze_class_body
    is_protocol = self.detect_protocol_base(defn)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 863, in detect_protocol_base
    sym = self.lookup_qualified(base.name, base)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 3730, in lookup_qualified
    return self.lookup(name, ctx, suppress_errors=suppress_errors)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 3713, in lookup
    self.name_not_defined(name, ctx)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 3914, in name_not_defined
    self.fail(message, ctx)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/semanal.py", line 3942, in fail
    self.errors.report(ctx.get_line(), ctx.get_column(), msg, blocker=blocker)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/errors.py", line 329, in report
    self.add_error_info(info)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/errors.py", line 350, in add_error_info
    self._add_error_info(file, info)
  File "/home/clement/.local/lib/python3.5/site-packages/mypy/errors.py", line 332, in _add_error_info
    assert file not in self.flushed_files
AssertionError: 

[gvanrossum]

That crash seems similar to #4881.

From your messages I can't figure out exactly what it is you're doing so I can't help you further. If all you get is that crash, that obviously is a mypy bug and shouldn't happen. If you get some other failure, please boil it down to a simple example that can be reproduced outside your IDE integration.

[cpitclaudel]

Thanks.

From your messages I can't figure out exactly what it is you're doing so I can't help you further.

Sorry. Here's clarification.

I'm attempting to maintain a mypy plugin for Emacs.

Currently, the plugin always calls mypy from the current directory, on the file being edited. It passes absolute paths to mypy, and uses --shadow-file. This leads to the assertion error reported here.

I'm attempting to understand whether 1. the Emacs extension is doing the right thing, and I just need to wait for or help implement a mypy fix, or 2. the Emacs extension is doing the wrong thing, and I need to change it to run in a different directory.

From your previous statement, I understand that mypy's behavior (raising an assertion error despite being passed full paths) is unexpected. Am I understanding right?

Here's a concrete repro: I create three files __init__.py, main.py and other.py in a new folder, with other.py making a relative import from main.py; then I call mypy with absolute paths from the new folder; things work. I copy other.py to /tmp/other.py, call mypy again with absolute paths but with --shadow-file, and I get this assertion error.

$ cd /scratch/
$ mkdir mypy_2974
$ cd mypy_2974/
$ touch __init__.py
$ echo "def f(): pass" > main.py
$ echo "from .main import f" > other.py
$ mypy other.py # Expected to fail, does fail
Traceback (most recent call last): ...
AssertionError: Neither id, path nor source given
$ mypy /scratch/mypy_2974/other.py # Expected to succeed, does succeed
$ cp /scratch/mypy_2974/other.py /tmp/temp.py
$ mypy --shadow-file /tmp/temp.py /scratch/mypy_2974/other.py /tmp/temp.py # expected to succeed; fails
Traceback (most recent call last): ...
AssertionError: Neither id, path nor source given

[gvanrossum]

Ah, --shadow-file. Your final line uses the context of /tmp/temp.py but the content of other.py. However other.py has a relative import which isn't satisfied in the context of /tmp/temp.py. It's as if you typed just mypy /tmp/temp.py. You can see for yourself whether that gives the same error.

The usage for --shadow-file is just too confusing. I'd expect you to use mypy --shadow-file /scratch/mypy_2974/other.py /tmp/temp.py /scratch/mypy_2974/other.py, where /tmp/temp.py is a temporary modification of other.py.

[FichteFoll]

When I implemented mypy linting for Sublime Text, I used to determine the top level folder (without init.py) manually before starting the lint and launched mypy from that folder. Now, however, the linting framework just launches linter processes from the project dir, so I don't need to do this anymore.

[gvanrossum]

FWIW the best way to think of --shadow-file is this equivalence:

mypy --shadow-file A B C

is equivalent to

cp B A
mypy C

(except for the obvious side effect that afterwards, A is overwritten. ;-)

[cpitclaudel]

. I'd expect you to use mypy --shadow-file /scratch/mypy_2974/other.py /tmp/temp.py /scratch/mypy_2974/other.py

Ah, thanks a lot. It makes plenty of sense then. And passing the full name works great now.

[JukkaL]

It looks like this has been fixed at some point, as I can't reproduce the crash on master (but can using mypy 0.600).