Maximum recursion depth exceeded during analysis of two files #3836

psass-edfsf · 2020-09-15T14:40:51Z

Steps to reproduce

Ceate a.py:

from b import e, f


def c(a, b, c):
    g = e(a)
    h = f({"a": a, "b": "c"}, {"a": b, "b": "d"}, {"a": c, "b": "e"}, h=1)
    [h["c"].sum()]

Create b.py in the same directory than a.py

from datetime import timedelta

import pandas as pd


def e(y: pd.DataFrame):
    d = (pd.Series(y.index[1:]) - pd.Series(y.index[:-1])).value_counts()
    d = d.index[d.index > timedelta(minutes=1)].min()
    return d


def f(t):
    return t

Run pylint on a.py

Current behavior

************* Module a
a.py:4:12: W0621: Redefining name 'c' from outer scope (line 4) (redefined-outer-name)
Traceback (most recent call last):
  File "/home/user/.local/bin/pylint", line 8, in <module>
    sys.exit(run_pylint())
  File "/home/user/.local/lib/python3.8/site-packages/pylint/__init__.py", line 22, in run_pylint
    PylintRun(sys.argv[1:])
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/run.py", line 349, in __init__
    linter.check(args)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 862, in check
    self._check_files(
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 896, in _check_files
    self._check_file(get_ast, check_astroid_module, name, filepath, modname)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 922, in _check_file
    check_astroid_module(ast_node)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1054, in check_astroid_module
    retval = self._check_astroid_module(
  File "/home/user/.local/lib/python3.8/site-packages/pylint/lint/pylinter.py", line 1099, in _check_astroid_module
    walker.walk(ast_node)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 75, in walk
    self.walk(child)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/utils/ast_walker.py", line 72, in walk
    callback(astroid)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 1990, in visit_assignname
    self._check_name("variable", node.name, node)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 2068, in _check_name
    if match is None and not _should_exempt_from_invalid_name(node):
  File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/base.py", line 2044, in _should_exempt_from_invalid_name
    inferred = utils.safe_infer(node)
  File "/home/user/.local/lib/python3.8/site-packages/pylint/checkers/utils.py", line 1143, in safe_infer
    value = next(infer_gen)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/user/.local/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
(...)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/user/.local/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/home/user/.local/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/home/user/.local/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
RecursionError: maximum recursion depth exceeded while calling a Python object

Expected behavior

************* Module a
a.py:4:12: W0621: Redefining name 'c' from outer scope (line 4) (redefined-outer-name)
a.py:5:8: E1121: Too many positional arguments for function call (too-many-function-args)
a.py:5:8: E1123: Unexpected keyword argument 'h' in function call (unexpected-keyword-arg)
a.py:6:4: W0106: Expression "[h['c'].sum()]" is assigned to nothing (expression-not-assigned)
a.py:1:0: W0611: Unused e imported from b (unused-import)

pylint --version output

pylint 2.6.0
astroid 2.4.2
Python 3.8.2 (default, Jul 16 2020, 14:00:26)
[GCC 9.3.0]

The text was updated successfully, but these errors were encountered:

hippo91 · 2020-09-15T19:16:12Z

@Pierre-Sassoulas i cannot reproduce it either.

psass-edfsf · 2020-09-16T07:23:29Z

I also had difficulty reproducing when using another directory than the original one. I managed to do it by installing the following dependencies:

apipkg==1.5
appdirs==1.4.4
aspy.yaml==1.3.0
astroid==2.4.2
attrs==19.3.0
black==19.10b0
certifi==2020.6.20
cfgv==3.2.0
cftime==1.2.1
chardet==3.0.4
click==7.1.2
colorama==0.4.3
confuse==1.3.0
coverage==5.2.1
cycler==0.10.0
decorator==4.4.2
dicttoxml==1.7.4
distlib==0.3.1
eradicate==1.0
execnet==1.7.1
filelock==3.0.12
flake8==3.8.3
freezegun==0.3.15
identify==1.4.28
idna==2.10
importlib-metadata==1.7.0
iniconfig==1.0.1
ipython-genutils==0.2.0
iso8601==0.1.12
isort==5.4.2
jsonschema==3.2.0
jupyter-core==4.6.3
kiwisolver==1.2.0
lazy-object-proxy==1.4.3
matplotlib==3.3.1
mccabe==0.6.1
more-itertools==8.4.0
mypy==0.782
mypy-extensions==0.4.3
nbformat==5.0.7
netCDF4==1.5.4
nodeenv==1.4.0
numpy==1.19.1
packaging==20.4
pandas==0.25.3
pathspec==0.8.0
Pillow==7.2.0
plotly==3.6.1
pluggy==0.13.1
pre-commit==2.7.1
py==1.9.0
pycodestyle==2.6.0
pyflakes==2.2.0
pylint==2.6.0
pyparsing==2.4.7
pyrsistent==0.16.0
pytest==6.0.1
pytest-cache==1.0
pytest-cov==2.10.1
python-dateutil==2.8.1
pytz==2020.1
PyYAML==5.3.1
regex==2020.7.14
requests==2.24.0
retrying==1.3.3
scipy==1.5.2
six==1.15.0
toml==0.10.1
tqdm==4.48.2
traitlets==4.3.3
typed-ast==1.4.1
typing-extensions==3.7.4.3
urllib3==1.25.10
virtualenv==20.0.31
wcwidth==0.2.5
wrapt==1.12.1
xmltodict==0.12.0
zipp==3.1.0

I wanted to pin down the exact dependencies required without setting them with pip freeze, but the problem started to happen everywhere even after deleting the virtualenv (?!).

hippo91 · 2020-09-19T16:27:32Z

@psass-edfsf in order for us to investigate on this potential issue, could you provide a restricted set of dependencies required to trigger the infinite recursion?
Thanks in advance.

atodorov · 2020-10-13T15:35:58Z

I have also started seeing this in a recent PR on my project which also adds a new checker:
kiwitcms/Kiwi#1993
https://github.com/kiwitcms/Kiwi/runs/1247821987?check_suite_focus=true

My specifics are that pylint executes against a large pool of files (site-packages dir) but for some reason this happens only in CI. Locally starts to happen when I have 3x the amount of packages installed.

pylint==2.6.0
pylint-django==2.3.0
pylint-plugin-utils==0.6
astroid==2.4.2

Note: as far as we can tell disabling the new checker being added in the PR above doesn't cause the recursion issues. This is very puzzling. Do you have any hints on how to investigate this ?

atodorov · 2020-10-13T15:38:00Z

Possible duplicate #3602

atodorov · 2020-10-23T19:41:10Z

I've been able to dig more information around this. For me it happens when executing pylint against site-packages/setuptools/ like so:

pylint --disable=all /home/senko/.virtualenvs/kiwi_django3/lib/python3.6/site-packages/setuptools/

Attached is the full traceback I get - traceback.txt

which was discovered to trigger a recursion loop: pylint-dev/pylint#3836 (comment)

hippo91 · 2020-11-26T08:57:22Z

@atodorov thanks for your feed back. We will investigate this ASAP.

which was discovered to trigger a recursion loop: pylint-dev/pylint#3836 (comment)

timstaley · 2020-12-10T12:21:11Z

Hi,
I've also encountered this and seem to have boiled it down to a smaller reproducible example:

New virtualenv, Python 3.8.0 (Python 3.6 seems unaffected, oddly).
pip install pylint pandas

Drop this into mwe.py

import pandas as pd

pd.merge()

Then pylint mwe.py gives

<many repeats>
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/decorators.py", line 96, in wrapped
    res = next(generator)
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/bases.py", line 136, in _infer_stmts
    for inferred in stmt.infer(context=context):
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/util.py", line 160, in limit_inference
    yield from islice(iterator, size)
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/context.py", line 113, in cache_generator
    for result in generator:
  File "/home/staley/.virtualenvs/testpylint/lib/python3.8/site-packages/astroid/decorators.py", line 132, in raise_if_nothing_inferred
    yield next(generator)
RecursionError: maximum recursion depth exceeded while calling a Python object

pip freeze lists the following

astroid==2.4.2
isort==5.6.4
lazy-object-proxy==1.4.3
mccabe==0.6.1
numpy==1.19.4
pandas==1.1.5
pylint==2.6.0
python-dateutil==2.8.1
pytz==2020.4
six==1.15.0
toml==0.10.2
wrapt==1.12.1

timstaley · 2020-12-10T13:12:39Z

Workaround; turns out this is a genuinely large recursion problem, and can be gotten around by putting the following in your pylintrc file:

init-hook='import sys; sys.setrecursionlimit(3 * sys.getrecursionlimit())'

(3x works for me, YMMV).
h/t https://stackoverflow.com/questions/36496192/pylint-infinite-recursion-in-astriod-package & @Exhor 🖖

Perhaps there's some way to make the Pylint error more helpful, and point to a relevant FAQ section or something?

Pierre-Sassoulas · 2020-12-10T13:38:06Z

Maybe we could change the recursion limit directly inside pylint? I encountered this error a lot lately while working with pandas as a dependency. Genuine recursion error would still be caught after 3x the system limit. Does that sound reasonable @hippo91 @AWhetter ?

AWhetter · 2020-12-10T16:44:26Z

Is it worth trying to figure out why it's recursing so much first?
But yes if we think this is good reason for us to need to raise it then let's do that. I think it makes sense to do it only when pylint is run on the command line and not when run as a library, because then the host application should be responsible for it rather than us.

atodorov · 2020-12-10T18:47:56Z

Is it worth trying to figure out why it's recursing so much first?

Definitely worth trying to figure out IMO.

hippo91 · 2020-12-12T11:13:44Z

@timstaley thanks for your report. This is probably link to #3939 which has been fixed in the current astroids master branch for pandas 1.1.4. However the bug reappears with pandas 1.1.5.
It definitely needs investigation so that we can decide whether or not it is legitimate to raise the recursion limit.

Pierre-Sassoulas · 2020-12-12T20:09:15Z

So it seems that releasing a new version of astroid and pylint would fix the problem for all pandas < 1.1.5. As long as 2.6.1 is not released, the fix to the init_hook provided by @timstaley seems acceptable : #3836 (comment) We can probably wait for the fix for all version of pandas before releasing 2.6.1 ?

hippo91 · 2020-12-19T14:29:57Z

@Pierre-Sassoulas yes we can try to solve the bug for pandas 1.1.5 before releasing what i think could be a major pylint version (2.7).

Pierre-Sassoulas · 2021-03-07T12:49:48Z

This was solved in pylint 2.7 / astroid 2.5.

Pierre-Sassoulas added the Cannot reproduce 🤷 label Sep 15, 2020

Pierre-Sassoulas added Crash 💥 A bug that makes pylint crash Needs investigation 🔬 A bug or crash where it's not immediately obvious what is happenning and removed Cannot reproduce 🤷 labels Sep 16, 2020

atodorov added a commit to kiwitcms/Kiwi that referenced this issue Oct 23, 2020

linter: ignore setuptools dir

db79094

which was discovered to trigger a recursion loop: pylint-dev/pylint#3836 (comment)

atodorov added a commit to kiwitcms/Kiwi that referenced this issue Oct 23, 2020

linter: ignore setuptools dir

cd2d6c8

which was discovered to trigger a recursion loop: pylint-dev/pylint#3836 (comment)

chinandrew mentioned this issue Oct 27, 2020

Pylint/Astroid gives "RecursionError: maximum recursion depth exceeded" in CI tests cmu-delphi/covidcast-indicators#333

Closed

asankov pushed a commit to asankov/Kiwi that referenced this issue Dec 2, 2020

linter: ignore setuptools dir

0da87d5

which was discovered to trigger a recursion loop: pylint-dev/pylint#3836 (comment)

Pierre-Sassoulas added Bug 🪲 and removed Needs investigation 🔬 A bug or crash where it's not immediately obvious what is happenning labels Jan 1, 2021

Pierre-Sassoulas mentioned this issue Jan 1, 2021

Maximum recursion depth crash with pyreverse -S option #3602

Open

hippo91 mentioned this issue Jan 23, 2021

RecursionError: maximum recursion depth exceeded with Astroid > 2.2.5 pylint-dev/astroid#865

Closed

Pierre-Sassoulas mentioned this issue Feb 12, 2021

RecursionError: maximum recursion depth exceeded while calling a Python object #4081

Closed

Pierre-Sassoulas closed this as completed Mar 7, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Maximum recursion depth exceeded during analysis of two files #3836

Maximum recursion depth exceeded during analysis of two files #3836

psass-edfsf commented Sep 15, 2020

hippo91 commented Sep 15, 2020

Uh oh!

psass-edfsf commented Sep 16, 2020

Uh oh!

hippo91 commented Sep 19, 2020

Uh oh!

atodorov commented Oct 13, 2020 •

edited

Loading

Uh oh!

atodorov commented Oct 13, 2020

Uh oh!

atodorov commented Oct 23, 2020

Uh oh!

hippo91 commented Nov 26, 2020

Uh oh!

timstaley commented Dec 10, 2020

Uh oh!

timstaley commented Dec 10, 2020 •

edited

Loading

Uh oh!

Pierre-Sassoulas commented Dec 10, 2020

Uh oh!

AWhetter commented Dec 10, 2020

Uh oh!

atodorov commented Dec 10, 2020

Uh oh!

hippo91 commented Dec 12, 2020

Uh oh!

Pierre-Sassoulas commented Dec 12, 2020

Uh oh!

hippo91 commented Dec 19, 2020

Uh oh!

Pierre-Sassoulas commented Mar 7, 2021

Uh oh!

Uh oh!

Maximum recursion depth exceeded during analysis of two files #3836

Maximum recursion depth exceeded during analysis of two files #3836

Comments

psass-edfsf commented Sep 15, 2020

Steps to reproduce

Current behavior

Expected behavior

pylint --version output

hippo91 commented Sep 15, 2020

Uh oh!

psass-edfsf commented Sep 16, 2020

Uh oh!

hippo91 commented Sep 19, 2020

Uh oh!

atodorov commented Oct 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

atodorov commented Oct 13, 2020

Uh oh!

atodorov commented Oct 23, 2020

Uh oh!

hippo91 commented Nov 26, 2020

Uh oh!

timstaley commented Dec 10, 2020

Uh oh!

timstaley commented Dec 10, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Pierre-Sassoulas commented Dec 10, 2020

Uh oh!

AWhetter commented Dec 10, 2020

Uh oh!

atodorov commented Dec 10, 2020

Uh oh!

hippo91 commented Dec 12, 2020

Uh oh!

Pierre-Sassoulas commented Dec 12, 2020

Uh oh!

hippo91 commented Dec 19, 2020

Uh oh!

Pierre-Sassoulas commented Mar 7, 2021

Uh oh!

atodorov commented Oct 13, 2020 •

edited

Loading

timstaley commented Dec 10, 2020 •

edited

Loading