stubtest regression in 0.930 handling local/relative imports?

[daviddrysdale]

Bug Report

The 0.930 version of stubtest doesn't seem to handle local/relative imports (from .local import thing) that worked with 0.921.

To Reproduce

(VIRTUAL) ~/mypytest:python --version
Python 3.9.9
(VIRTUAL) ~/mypytest:tail -n +1 temp/* run_stubtest.py 
==> temp/__init__.py <==
from .local import AN_INT
ANOTHER_INT = AN_INT
==> temp/__init__.pyi <==
ANOTHER_INT: int
==> temp/local.py <==
AN_INT = 1
==> temp/local.pyi <==
AN_INT: int
==> run_stubtest.py <==
import sys
from mypy import stubtest
if __name__ == "__main__":
    sys.exit(stubtest.test_stubs(stubtest.parse_options(["temp"])))
(VIRTUAL) ~/mypytest:pip install mypy==0.921
<snip>
Successfully installed mypy-0.921
(VIRTUAL) ~/mypytest:python run_stubtest.py 
(VIRTUAL) ~/mypytest:pip install mypy==0.930
<snip>
Successfully installed mypy-0.930
(VIRTUAL) ~/mypytest:python run_stubtest.py 
error: temp.AN_INT is not present in stub
Stub:
MISSING
Runtime:
1

Expected Behavior

Expect the behaviour of 0.921 – stubtest passes:

(VIRTUAL) ~/mypytest:pip install mypy==0.921
<snip>
Successfully installed mypy-0.921
(VIRTUAL) ~/mypytest:python run_stubtest.py 

Actual Behavior

stubtest 0.931 shows an error relating to the local import:

(VIRTUAL) ~/mypytest:pip install mypy==0.930
<snip>
Successfully installed mypy-0.930
(VIRTUAL) ~/mypytest:python run_stubtest.py 
error: temp.AN_INT is not present in stub
Stub:
MISSING
Runtime:
1

Your Environment

• Mypy version used: 0.921 and 0.930
• Mypy command-line flags: N/A
• Mypy configuration options from mypy.ini (and other config files): None
• Python version used: 3.9.9
• Operating system and version: MacOS 10.15.7

[hauntsaninja]

Yeah, this is an expected result of #11634

stubtest works by comparing the results of a) statically analysing pyi files, b) importing the actual module and using runtime introspection.

stubtest has to figure out which attributes of the runtime module should have definitions in the stub. If there's an __all__, stubtest will use that. But without that, stubtest has to guess and there isn't a foolproof way when looking at temp.AN_INT at runtime to distinguish between:
a) AN_INT = 0 (definition, ideally stubtest should check)
b) from .local import AN_INT (non re-exported import, ideally stubtest should not check)
bonus case that stubtest ignores, c) from .local import AN_INT as AN_INT (re-exported import, ideally stubtest should check)

In the case temp.AN_INT is a function or class, stubtest uses the __module__ attribute to distinguish between cases a) and b). But if temp.AN_INT is an int or str, there isn't a good way to distinguish between a) and b) at runtime.

The behaviour change between stubtest 0.921 and 0.930 is that (in the absence of __all__ or a __module__ attribute on the object in question) stubtest defaults to checking that the stub has the object in question, as opposed to not checking (if the object is not a module).

This is pretty important for user expectations, see this additional test case that fails with 0.921: https://github.com/python/mypy/pull/11634/files#diff-64c76dab759f5e184794b869cd69f8a5afa97380e693e98b980025d5487dcfc1R662-R665 It was surprising to users that missing CONSTANT from the stubs was not detected (case a, above).
In addition, I tested the change on typeshed, and it seemed like a reasonably high signal change.

I see your project uses stubtest's allowlist functionality, which you could reuse here (--generate-allowlist is handy). You could also define __all__ in your actual Python source.

[hauntsaninja]

It does look somewhat annoying for your project (although a phonenumbers.data.[A-Z0-9]+ isn't the worst). While I'm not going to revert the default to 0.921's behaviour, I could potentially make it configurable.
There might also be room to have some more heuristics here (e.g. looking at attributes on imported submodules).

[daviddrysdale]

Thanks for the detailed explanation!

It sounds like I should look into setting up __all__ definitions for my various submodules to sort this out…

[daviddrysdale]

Quick update: I found a straightforward workaround, which was just to repeat the from .local import AN_INT lines in the stub file as well as the Python file.

That seemed to make sense:

• in the .py file, the definition of temp.AN_INT comes from the import of local.AN_INT
• in the .pyi file, the type declaration of temp.AN_INT comes from the import of local.AN_INT.

The change also meant I could re-enable the latest (0.930) stubtest, with no need for any new config flag – so maybe close this out?

Thanks again!

[hauntsaninja]

Happy to hear and happy new year! :-)