Skip to content

Refactor message disabling and enabling #5596

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 28, 2021

Conversation

DanielNoord
Copy link
Collaborator

  • Add yourself to CONTRIBUTORS if you are a new contributor.
  • Write a good description on what the PR does.

Type of Changes

Type
🔨 Refactoring

Description

Closes #5587.

This is the main improvement I found for enabling and disabling messages. The problem was that we synced the config object after every message, instead of doing so after all messages have been set.
With this version we first find all messages to set, set them and then sync the config object.

For comparison:
Profiler command:
python -m cProfile -s cumtime pylint/__main__.py test.py --disable=all

Without change:

1    0.000    0.000    0.372    0.372 config_initialization.py:14(_config_initialization)

With change:

1    0.000    0.000    0.026    0.026 config_initialization.py:14(_config_initialization)

About a 10x improvement 😄

@DanielNoord DanielNoord added performance Maintenance Discussion or action around maintaining pylint or the dev workflow labels Dec 24, 2021
@coveralls
Copy link

coveralls commented Dec 24, 2021

Pull Request Test Coverage Report for Build 1628212835

  • 21 of 21 (100.0%) changed or added relevant lines in 1 file are covered.
  • No unchanged relevant lines lost coverage.
  • Overall coverage increased (+0.003%) to 93.711%

Totals Coverage Status
Change from base Build 1617209026: 0.003%
Covered Lines: 14319
Relevant Lines: 15280

💛 - Coveralls

Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, this is great thank you ! Small question to maybe trim a sorting on top of what was already done.

# sync configuration object
msgs = self._msgs_state
self.config.enable = [
self._message_symbol(mid) for mid, val in sorted(msgs.items()) if val
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering why we sort here. If it's in order to be able to test the class, we could sort in the test directly (where it won't affect our run time performance).

@Pierre-Sassoulas Pierre-Sassoulas added this to the 2.13.0 milestone Dec 27, 2021
@DanielNoord
Copy link
Collaborator Author

@Pierre-Sassoulas Could you add a suggestion and see if it works without sorting? (I'm on mobile)

@Pierre-Sassoulas
Copy link
Member

The change seems to work locally but I can't use the profiling command since we added pylint/typing.py There's a sys path mixup with the official typing module and our own : python -m cProfile -s cumtime pylint/__main__.py test.py --disable=all

File "/home/pierre/pylint/pylint/typing.py", line 6, in <module>
    from typing import NamedTuple, Optional, Union
ImportError: cannot import name 'NamedTuple' from partially initialized module 'typing' (most likely due to a circular import) (/home/pierre/pylint/pylint/typing.py)

Any idea how to solve this ?

@Pierre-Sassoulas
Copy link
Member

I made a change without checking that the performance got improved. (Theoretically it should be better, but...)

Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I found a way to test performance, and the change I made is actually a tad better:

git rebase origin/main -i --exec="python -m cProfile -s cumtime -m pylint --disable=all test.py|grep config_initialization"
Executing: python -m cProfile -s cumtime -m pylint --disable=all test.py|grep config_initialization
        1    0.000    0.000    0.163    0.163 config_initialization.py:14(_config_initialization)
        1    0.000    0.000    0.000    0.000 config_initialization.py:4(<module>)
Executing: python -m cProfile -s cumtime -m pylint --disable=all test.py|grep config_initialization
        1    0.000    0.000    0.014    0.014 config_initialization.py:14(_config_initialization)
        1    0.000    0.000    0.000    0.000 config_initialization.py:4(<module>)
Executing: python -m cProfile -s cumtime -m pylint --disable=all test.py|grep config_initialization
        1    0.000    0.000    0.012    0.012 config_initialization.py:14(_config_initialization)
        1    0.000    0.000    0.000    0.000 config_initialization.py:4(<module>)
Successfully rebased and updated refs/heads/performance.

Successfully rebased and updated refs/heads/performance.

I think we can merge, let me know whay you think @DanielNoord

Copy link
Collaborator Author

@DanielNoord DanielNoord left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@DanielNoord DanielNoord merged commit 39b52ac into pylint-dev:main Dec 28, 2021
@DanielNoord DanielNoord deleted the performance branch December 28, 2021 12:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Maintenance Discussion or action around maintaining pylint or the dev workflow performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimizing speed of the message store during starting and initialisation phase
3 participants