RecursionError when linting scripts containing merge_asof from pandas 0.24

[grobgl]

Pylint crashes with a RecursionError from astroid when linting merge_asof in version 0.24 of pandas and any recent version of pylint (tested 1.9, 2.0, 2.1, 2.2, 2.3). Note that this is not an issue with older versions of pandas.

Steps to reproduce

1. Install pandas 0.24
2. Create file replication.py:

import pandas as pd

pd.merge_asof(pd.DataFrame(), pd.DataFrame(), on='MergeCol')

3. Run pylint replication.py

Current behavior

Pylint crashes as follows:

************* Module replication
replication.py:1:0: C0111: Missing module docstring (missing-docstring)
Traceback (most recent call last):
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/bases.py", line 183, in getattr
    values = self._proxied.instance_attr(name, context)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/scoped_nodes.py", line 2311, in instance_attr
    target=self, attribute=name, context=context
astroid.exceptions.AttributeInferenceError: 'values' not found on <ClassDef.SparseSeries l.38 at 0x1099349e8>.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/bases.py", line 217, in igetattr
    get_attr = self.getattr(name, context, lookupclass=False)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/bases.py", line 195, in getattr
    ) from exc
astroid.exceptions.AttributeInferenceError: 'values' not found on <Instance of pandas.core.sparse.series.SparseSeries at 0x4452257920>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/g.grob/anaconda3/envs/py37/bin/pylint", line 11, in <module>
    sys.exit(run_pylint())
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/__init__.py", line 20, in run_pylint
    Run(sys.argv[1:])
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/lint.py", line 1628, in __init__
    linter.check(args)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/lint.py", line 943, in check
    self._do_check(files_or_modules)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/lint.py", line 1075, in _do_check
    self.check_astroid_module(ast_node, walker, rawcheckers, tokencheckers)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/lint.py", line 1158, in check_astroid_module
    walker.walk(ast_node)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/utils.py", line 1303, in walk
    self.walk(child)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/utils.py", line 1303, in walk
    self.walk(child)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/utils.py", line 1300, in walk
    cb(astroid)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/checkers/logging.py", line 215, in visit_call
    result, name = is_logger_class()
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/pylint/checkers/logging.py", line 197, in is_logger_class
    for inferred in node.func.infer():
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 131, in raise_if_nothing_inferred
    yield next(generator)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 95, in wrapped
    res = next(generator)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/inference.py", line 303, in infer_attribute
    yield from owner.igetattr(self.attrname, context)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/bases.py", line 137, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 131, in raise_if_nothing_inferred
    yield next(generator)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 95, in wrapped
    res = next(generator)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/bases.py", line 137, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 131, in raise_if_nothing_inferred
    yield next(generator)

...

  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/decorators.py", line 95, in wrapped
    res = next(generator)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/inference.py", line 238, in infer_import
    yield self.do_import_module(self.real_name(name))
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/mixins.py", line 100, in do_import_module
    modname, level=level, relative_only=level and level >= 1
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/scoped_nodes.py", line 619, in import_module
    return MANAGER.ast_from_module_name(absmodname)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/manager.py", line 171, in ast_from_module_name
    return self.ast_from_file(found_spec.location, modname, fallback=False)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/manager.py", line 91, in ast_from_file
    return AstroidBuilder(self).file_build(filepath, modname)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/builder.py", line 135, in file_build
    module = self._data_build(data, modname, path)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/builder.py", line 188, in _data_build
    module = builder.visit_module(node, modname, node_file, package)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 158, in visit_module
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 158, in <listcomp>
    newnode.postinit([self.visit(child, newnode) for child in node.body])
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 1032, in visit_classdef
    node, parent, newstyle=newstyle
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 404, in visit_classdef
    [self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 404, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 607, in visit_functiondef
    return self._visit_functiondef(nodes.FunctionDef, node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 597, in _visit_functiondef
    body=[self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 597, in <listcomp>
    body=[self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 653, in visit_if
    [self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 653, in <listcomp>
    [self.visit(child, newnode) for child in node.body],
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 960, in visit_raise
    _visit_or_none(node, "exc", self, newnode),
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 104, in _visit_or_none
    return getattr(visitor, visit)(value, parent, **kws)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 358, in visit_call
    args = [self.visit(child, newnode) for child in node.args]
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 358, in <listcomp>
    args = [self.visit(child, newnode) for child in node.args]
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 170, in visit
    return visit_method(node, parent)
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/rebuilder.py", line 760, in visit_str
    parent,
  File "/Users/g.grob/anaconda3/envs/py37/lib/python3.7/site-packages/astroid/node_classes.py", line 2466, in __init__
    super(Const, self).__init__(lineno, col_offset, parent)
RecursionError: maximum recursion depth exceeded while calling a Python object

pylint --version output

Latest tested:

pylint 2.3.1
astroid 2.2.5
Python 3.7.2 (default, Dec 29 2018, 00:00:04)
[Clang 4.0.1 (tags/RELEASE_401/final)]

[PCManticore]

Thanks for the report @grobgl I can reproduce the issue.

[jetheurer]

Any luck on solving this issue? I'm having a similar problem

[PCManticore]

@jetheurer I haven't been able to find time to work on this issue, but I presume one of the issues is the recursive nature of the rebuilder, which takes Python AST and transforms it to astroid AST. Replacing that recurser would probably solve this issue as well.

[PCManticore]

This should be fixed with the release of astroid 2.4.2. Can you give it a go and let me know if you can still reproduce the issue?

[grobgl]

This should be fixed with the release of astroid 2.4.2

I assume you mean pylint 2.4.2?

Just tried it, it's still failing for me with

pylint --version
pylint 2.4.2
astroid 2.3.1
Python 3.7.4 (default, Aug 13 2019, 15:17:50)
[Clang 4.0.1 (tags/RELEASE_401/final)]

[grobgl]

Just tried with astroid 2.3.2 as well. Still failing with the same error. This is my environment:

astroid                   2.3.2
pandas                    0.24.2
pylint                    2.4.3
python                    3.7.4

[PCManticore]

Alright, thanks for confirming! I haven't tried with Python 3.7.4, maybe that has something to do with it.

[witzatom]

I am pretty sure that 2.5 reintroduced this as a regression.