`six.moves` brain is not effective when `six.moves` was already imported in interpreter running astroid

[perrinjerome]

Inference of six.moves attributes differs when six.moves was already imported or not.

Steps to reproduce

import astroid.builder
# import six.moves   #   <<< un-comment this line to see the problem

print(
    next(
        astroid.builder.extract_node(
            """
from six.moves import StringIO
StringIO  #@
"""
        ).infer()
    ).qname()
)

Current behavior

When six.moves is not imported, the node is inferred as _io.StringIO, which is expected, but when six.moves is imported, the node is not inferred correctly. The reproduction snippet prints six.moves.

Expected behavior

This should not depend on the imported modules, the inferred value should be _io.StringIO in all cases.

More information

six.moves is a dynamic module with a custom loader:

>>> import six.moves
>>> six.moves.__spec__
ModuleSpec(name='six.moves', loader=<six._SixMetaPathImporter object at 0x7fa3b2bad9d0>, submodule_search_locations=[])

When building the ast for a module, astroid tries various "spec finders" to find a module spec here:

astroid/astroid/interpreter/_import/spec.py

Lines 377 to 379 in 396f01a

	for finder in _SPEC_FINDERS:
	finder_instance = finder(search_path)
	spec = finder_instance.find_module(

one is ExplicitNamespacePackageFinder which looks in sys.modules:

astroid/astroid/interpreter/_import/spec.py

Line 231 in 396f01a

if util.is_namespace(modname) and modname in sys.modules:

this is where the behavior differs.

util.is_namespace("six.moves") returns False, so an empty module is returned, as we can see, the module is empty:

>>> import astroid
>>> import six.moves
>>> astroid.MANAGER.ast_from_module_name('six.moves').as_string()
'\n\n'

But in the "it works" case, when six.moves was not imported, the module content is present:

>>> import astroid
>>> astroid.MANAGER.ast_from_module_name('six.moves').as_string()[:100]
'import _io\ncStringIO = _io.StringIO\nfilter = filter\nfrom itertools import filterfalse\ninput = input\n'

note that this is not the actual module source code, this was built dynamically by the failed import hooks from the brain here:

astroid/astroid/brain/brain_six.py

Line 230 in 9aa9bfd

manager.register_failed_import_hook(_six_fail_hook)

Suggested fixes

I see two possibilities to fix this:

The first option is to consider that astroid.interpreter._import.util.is_namespace is wrong when evaluating six.moves as a namespace.

Honestly, I don't fully understand what is the definition of a "namespace" here. I guess this is an empty module with submodules search path, so one fix might be to change the condition found_spec.submodule_search_locations is not None into a "simpler" found_spec.submodule_search_locations here:

astroid/astroid/interpreter/_import/util.py

Line 102 in de942f3

and found_spec.submodule_search_locations is not None

in the case of six.moves, found_spec.submodule_search_locations is []. I think that what's the most important in the distinction between a namespace and a "regular" module here is that a namespace comes with submodules, so if it's [] or None it might be equivalent (but once again, I don't really understand this).

The second option is that before deciding that a module is "just an empty namespace", give failed import hooks a chance to populate the module with something. This change would be simply to try all the self._failed_import_hooks before _build_namespace_module in

astroid/astroid/manager.py

Lines 247 to 250 in de942f3

	elif found_spec.type == spec.ModuleType.PY_NAMESPACE:
	return self._build_namespace_module(
	modname, found_spec.submodule_search_locations or []
	)

this branch would become:

            elif found_spec.type == spec.ModuleType.PY_NAMESPACE:
                # before returning an empty namespace module, allow a fail import hook
                # to return a dynamic module instead.
                for hook in self._failed_import_hooks:
                    try:
                        return hook(modname)
                    except AstroidBuildingError:
                        pass
                return self._build_namespace_module(
                    modname, found_spec.submodule_search_locations or []
                )

python -c "from astroid import __pkginfo__; print(__pkginfo__.version)" output

3.2.0-dev0, this is on current git version, de942f3

[perrinjerome]

I made draft pull requests with the two possible fixes, #2410 and #2411