hooks: pylint: pre-commit failure

[casperdcl]

.pre-commit-config.yaml states that the pylint hook is defined locally, yet .pre-commit-hooks.yaml fails to define it. As a result, attempting to commit will result in an error:

~/gits/dvc (master)$ git commit
black....................................................................Passed
seed isort known_third_party.............................................Passed
isort....................................................................Passed
flake8...................................................................Passed
pylint...................................................................Failed
- hook id: pylint
- exit code: 1

Executable `pylint` not found

beautysh.............................................(no files to check)Skipped
DVC pre-commit...........................................................Passed
- hook id: dvc-pre-commit
- duration: 0.5s

Data and pipelines are up to date.

Defining the hook's language as system and asking devs to manually run pip install .[tests] goes against fundamental principles of pre-commit hooks. None of the other hooks require this.

In any case pylint itself defines .pre-commit-hooks.yaml so we should just use theirs.

[skshetry]

@casperdcl, as pre-commit runs on an isolated environment, we'd need to disable a lot of configuration to do this way.

Also, contributors should already install from .[tests]. Without that, how'll they be running tests?

None of the other hooks require this.

dvc-pre-commit is already a local run, for which you do need to have activated virtualenv beforehand and installed all reqs.

Is it too difficult to pip install .[tests]? If it's that inconvenience, we could just pop it off from the pre-commit hooks.

See: pre-commit/pre-commit-hooks#157

[casperdcl]

• setting correct configuration via the hook itself (e.g. commandline args) is the correct way
• I'd strongly recommend against pip install .[tests] especially for DVC where there are many tests which won't even run locally due to lack of dependencies/credentials. pre-commit => quick local tests, CI => full tests. On the rare occasion where an outside contributor is developing a big new feature where it's a pain to constantly wait for the full CI tests, it's usually easiest (and probably required) to write a new test... and just run that test manually locally when debugging. Certainly not the whole test suite.
  • I'd even more strongly recommend against breaking pre-commit install by suddenly insisting on pip install .[tests] as a new pre-requisite
• re: dvc-pre-commit; having a local run hook that is genuinely provided locally is acceptable - in this case, dvc is really provided by the local repo
• even with the hook's dependencies locally installed - pylint, pylint-pytest (not available via conda) and pylint-plugin-utils - the hook takes a long time to run, which is annoying.

For now I'm fine with popping off from the pre-commit hooks as you suggest; this seems far less painful than leaving it as-is.

[efiop]

re: dvc-pre-commit; having a local run hook that is genuinely provided locally is acceptable - in this case, dvc is really provided by the local repo

@casperdcl We still need to pip install all dvc dependencies for that, and pylint is a test dependency, so we can run system's pylint just fine.

Do I understand correctly that you guys are discussing removing pylint from pre-commit hooks? I'm strongly against that, it is helping to discover issues before creating a PR and is certainly faster than waiting for CI to error-out.

[skshetry]

@efiop, I'm still looking for answer for "Is it too difficult to pip install .[tests]?" from @casperdcl.
Ideally, I'd also love to have it isolated, but in practice, the benefits outweigh the cons.

If it's too hard to install our test dependencies, I could create a custom script that just throws a message to the user
complaining that pylint is not installed and does not break pre-commit runs. But, there should be a very strong argument to do so.

[shcheklein]

@skshetry @casperdcl why does pylint is different in this sense from black, isort and other "linters"? (just curious where does this problem come from). They should be the same to mind, have the same flow, conventions, etc.

[skshetry]

@shcheklein, pylint also does dynamic analysis for some checks, not just static analysis.

You can see some failures here: https://github.com/iterative/dvc/pull/4146/checks?check_run_id=827566035#step:5:124

When in isolated environment, not all dependencies are installed and pylint starts complaining about missing-import or no-member. We could either disable these, but as we have just started now, it's better to keep it for now, and we'll see.

Also, we are using a plugin pylint_pytest that does dynamic execution to get all the fixtures to cut down the noise (related to function arguments name being same as pytest fixtures).
And, also I wanted to enable pylint for all of our codebase (including tests), we have our own custom plugin to silence some issues that are thrown in tests.

So, TLDR: it does dynamic execution too, and is hence able to find more issues.