Pylint processes eat all memory

[gatopeich]

Issue Type: Bug

While working on a python 3.7 project with Pylint enabled, a lot of pylint processes are spawned in background. They never exit and eventually eat all my system's memory (>20GB out of 32)

Extension version: 2019.10.44104
VS Code version: Code 1.39.2 (6ab598523be7a800d7f3eb4d92d7ab9a66069390, 2019-10-15T15:33:40.634Z)
OS version: Linux x64 4.15.0-66-generic

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-4770 CPU @ 3.40GHz (8 x 3797)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on native_gpu_memory_buffers: disabled_software oop_rasterization: disabled_off protected_video_decode: unavailable_off rasterization: disabled_software skia_deferred_display_list: disabled_off skia_renderer: disabled_off surface_synchronization: enabled_on video_decode: unavailable_off viz_display_compositor: disabled_off webgl: enabled webgl2: enabled
Load (avg)	9, 11, 7
Memory (System)	31.31GB (20.20GB free)
Process Argv	--unity-launch
Screen Reader	no
VM	0%

[ericsnowcurrently]

@gatopeich, thanks for letting us know about this. In the "Python" output channel you should find the command that is run for pylint. What is it? Also, what happens when you run that same command in a normal terminal?

[sevdog]

I also have seen this behaviours. For what I have found seems that in some pylint processes remains active and do not terminate.

If you keep vscode opened enough time this lead to memory exaustion due to unterminated pylint processes.

I have found this issue on linux (Ubuntu 18.04 LTS), with some process explorer tool I dicovered more than 100 processes of pylint:

$ ps aux | grep pylint | wc -l
$ pgrep python | wc -l

By killing those processes I have solved temporary my memory issue, but this is not a definitive solution.

[sevdog]

Here is the output for ps aux | grep pylint on my pc:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
sevdog    3509  0.0  0.2 337236 46804 tty2     Sl+  11:04   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    3529  0.0  0.5 160888 93316 tty2     S+   11:04   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    3640  0.0  0.2 337236 46680 tty2     Sl+  11:04   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    3651  0.0  0.5 160868 93152 tty2     S+   11:04   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    6028  0.0  0.2 337236 46716 tty2     Sl+  11:06   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/filters.py
sevdog    6036  0.0  0.4 132880 65536 tty2     S+   11:06   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/filters.py
sevdog    7434  0.0  0.2 337236 46852 tty2     Sl+  11:07   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    7452  0.0  0.5 160836 93060 tty2     S+   11:07   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    7662  0.0  0.2 337236 46856 tty2     Sl+  11:07   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    7670  0.0  0.5 160880 93052 tty2     S+   11:07   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    8102  0.0  0.2 337236 46784 tty2     Sl+  11:07   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    8111  0.0  0.5 160864 93188 tty2     S+   11:07   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    8155  0.0  0.2 337236 46776 tty2     Sl+  11:07   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    8164  0.0  0.5 160868 93380 tty2     S+   11:07   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9064  0.0  0.2 337236 46632 tty2     Sl+  10:17   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/views.py
sevdog    9072  0.0  0.4 138360 71008 tty2     S+   10:17   0:01 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/views.py
sevdog    9253  0.0  0.2 337236 46724 tty2     Sl+  11:08   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9261  0.0  0.5 160896 93292 tty2     S+   11:08   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9316  0.0  0.2 345432 46852 tty2     Sl+  11:08   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9324  0.1  0.5 160892 93232 tty2     S+   11:08   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9689  0.0  0.2 345432 46816 tty2     Sl+  11:08   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog    9697  0.0  0.5 160872 93408 tty2     S+   11:08   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10097  0.0  0.2 337236 46744 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10106  0.0  0.5 160888 93160 tty2     S+   11:09   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10793  0.0  0.2 337236 46856 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10801  0.0  0.5 160872 93100 tty2     S+   11:09   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10820  0.0  0.2 337236 46872 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10828  0.0  0.5 160872 93164 tty2     S+   11:09   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   10998  0.0  0.2 337236 46764 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   11006  0.0  0.5 160876 93428 tty2     S+   11:09   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   11166  0.0  0.2 337236 46688 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   11174  0.0  0.5 160872 93292 tty2     S+   11:09   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   11221  0.0  0.2 337236 46720 tty2     Sl+  11:09   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   11229  0.0  0.5 160892 93264 tty2     S+   11:09   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   13436  0.0  0.2 337236 46724 tty2     Sl+  11:11   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   13444  0.0  0.5 160872 93436 tty2     S+   11:11   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   13584  0.0  0.2 337236 46768 tty2     Sl+  11:11   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   13592  0.0  0.5 160892 93536 tty2     S+   11:11   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   14136  0.0  0.2 337236 46672 tty2     Sl+  11:12   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   14144  0.0  0.5 160888 93368 tty2     S+   11:12   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   14271  0.0  0.2 337236 46860 tty2     Sl+  11:12   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   14282  0.0  0.5 160900 93296 tty2     S+   11:12   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   17269  0.0  0.2 337236 46756 tty2     Sl+  11:14   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/utils/filters.py
sevdog   17277  0.0  0.4 134824 67164 tty2     S+   11:14   0:01 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/utils/filters.py
sevdog   20063  0.0  0.2 337236 46752 tty2     Sl+  11:16   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/utils/filters.py
sevdog   20071  0.0  0.4 134820 67312 tty2     S+   11:16   0:01 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/utils/filters.py
sevdog   21531  0.0  0.2 337236 46736 tty2     Sl+  11:17   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   21539  0.0  0.5 160876 93432 tty2     S+   11:17   0:02 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   28040  0.0  0.2 337236 46860 tty2     Sl+  11:22   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/views.py
sevdog   28048  0.0  0.4 138336 70988 tty2     S+   11:22   0:01 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/views.py
sevdog   31698  0.0  0.2 337236 46684 tty2     Sl+  11:24   0:00 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py
sevdog   31706  0.1  0.5 160888 93212 tty2     S+   11:24   0:03 /dvp/myproject/.venv/bin/python -m pylint --rcfile=.pylintrc --msg-template='{line},{column},{category},{symbol}:{msg}' --reports=n --output-format=text /dvp/myproject/src/reim/tests/test_api.py

NOTE: while checking for pylint processes vscode was closed! All those processes have PPID=1.

[gatopeich]

Issue seems to be gone on vscode 1.41.1 with this extension version 2019.11.50794:

Name: Python
Id: ms-python.python
Description: Linting, Debugging (multi-threaded, remote), Intellisense, code formatting, refactoring, unit tests, snippets, and more.
Version: 2019.11.50794
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-python.python

[sevdog]

In my case this issue persists: I still got tens of pylint processes after about 20 minutes.

Version: 1.41.1
Commit: 26076a4de974ead31f97692a0d32f90d735645c0
Date: 2019-12-18T15:04:31.999Z
Electron: 6.1.5
Chrome: 76.0.3809.146
Node.js: 12.4.0
V8: 7.6.303.31-electron.0
OS: Linux x64 4.15.0-72-generic

[email protected]

Python 3.6.5

pylint 2.3.1
astroid 2.2.5
Python 3.6.5 (default, Apr  1 2018, 05:46:30) 
[GCC 7.3.0]

[ericsnowcurrently]

@gatopeich, sounds good. We'll close this issue.

@sevdog, yours appears to be a different problem then. Please open a separate issue and we'll look into it. Thanks!