fix sys.path for local workers Fixes #421 #667
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
graingert
commented
Jun 3, 2021
| @@ -261,7 +262,8 @@ def setup(self): | |||
| remote_module = self.config.hook.pytest_xdist_getremotemodule() | |||
| self.channel = self.gateway.remote_exec(remote_module) | |||
| # change sys.path only for remote workers | |||
| change_sys_path = not self.gateway.spec.popen | |||
| # restore sys.path from a frozen copy for local workers | |||
| change_sys_path = _sys_path if self.gateway.spec.popen else None | |||
There was a problem hiding this comment.
we have to use a frozen copy of sys.path because --import-mode=prepend changes the sys.path
graingert
commented
Jun 3, 2021
graingert
commented
Jun 3, 2021
RonnyPfannschmidt
approved these changes
Jun 16, 2021
There was a problem hiding this comment.
this one is a nice solution for a gnarly quirk
i would propose to eventually put multiprocessing and/or introcuding a python -m execnet.knownn_worker backend to execnet to make this robust even in the face of remote execution
however thats not something i can touch within the next 2 years and i cant expect anyone else to touch it
RonnyPfannschmidt
approved these changes
Jun 16, 2021
stanislavlevin
added a commit
to stanislavlevin/mypy
that referenced
this pull request
Mar 3, 2022
Description:
`run_stubtest` creates temp directory and prepend `sys.path` with
relative path (dot `.`) wrongly assuming that the dot will be
resolved to absolute path on *every* import attempt.
But in Python dot(`.`) in sys.path is actually resolved by
PathFinder and cached in `sys.path_importer_cache` like:
```
sys.path_importer_cache['.']
FileFinder('/somepath/.')
```
later calls for `find_module` return None and import of
`test_module` fails.
This resulted in only the first test in stubtest's suite passed in
non-pytest-xdist environments.
This issue was hidden with bug or feature in pytest-xdist < 2.3.0:
pytest-dev/pytest-xdist#421
It was fixed in pytest-xdist 2.3.0:
pytest-dev/pytest-xdist#667
- sys.path for pytest-xdist < 2.3.0
`'.', 'project_path', ''`
- sys.path for pytest-xdist >= 2.3.0 or without xdist
`'.', 'project_path'`
Fix:
In Python for denoting cwd the empty path `''` can be used as a
special case, but for readability `sys.path` is prepended with
resolved absolute path of temp directory. Also it's essential to
restore back `sys.path` after a test to not break subsequent tests.
Fixes: python#11019
Signed-off-by: Stanislav Levin <[email protected]>
hauntsaninja
pushed a commit
to python/mypy
that referenced
this pull request
Mar 4, 2022
Fixes #11019 `run_stubtest` creates temp directory and prepend `sys.path` with relative path (dot `.`) wrongly assuming that the dot will be resolved to absolute path on *every* import attempt. But in Python dot(`.`) in sys.path is actually resolved by PathFinder and cached in `sys.path_importer_cache` like: ``` sys.path_importer_cache['.'] FileFinder('/somepath/.') ``` later calls for `find_module` return None and import of `test_module` fails. This resulted in only the first test in stubtest's suite passed in non-pytest-xdist environments. This issue was hidden with bug or feature in pytest-xdist < 2.3.0: pytest-dev/pytest-xdist#421 It was fixed in pytest-xdist 2.3.0: pytest-dev/pytest-xdist#667 - sys.path for pytest-xdist < 2.3.0 `'.', 'project_path', ''` - sys.path for pytest-xdist >= 2.3.0 or without xdist `'.', 'project_path'` In Python for denoting cwd the empty path `''` can be used as a special case, but for readability `sys.path` is prepended with resolved absolute path of temp directory. Also it's essential to restore back `sys.path` after a test to not break subsequent tests. Signed-off-by: Stanislav Levin <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#421
Thanks for submitting a PR, your contribution is really appreciated!
Here's a quick checklist that should be present in PRs:
Make sure to include reasonable tests for your change if necessary
We use towncrier for changelog management, so please add a news file into the
changelogfolder following these guidelines:Name it
$issue_id.$typefor example588.bugfix;If you don't have an issue_id change it to the PR id after creating it
Ensure type is one of
removal,feature,bugfix,vendor,docortrivialMake sure to use full sentences with correct case and punctuation, for example: