pytest.PytestWarning: open() missing required argument 'flags' (pos 2)

[blueyed]

force_writable_and_retry might call open with missing args:

% COLUMNS=80 p --lf --pdb --tb=short --showlocals
============================= test session starts ==============================
platform linux -- Python 3.7.3, pytest-4.6.1.dev186+gc9923a3a5, py-1.8.0, pluggy-0.12.0
hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('…/Vcs/pytest/.hypothesis/examples')
rootdir: …/Vcs/pytest, inifile: tox.ini, testpaths: testing
plugins: hypothesis-4.23.8
collected 975 items / 287 deselected / 688 selected
run-last-failure: rerun previous 688 failures (skipped 150 files)

testing/acceptance_test.py E
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
/usr/lib/python3.7/shutil.py:423: in _rmtree_safe_fd
    dirfd = os.open(entry.name, os.O_RDONLY, dir_fd=topfd)
E   PermissionError: [Errno 13] Permission denied: '.pytest_cache'

During handling of the above exception, another exception occurred:
src/_pytest/pathlib.py:68: in force_writable_and_retry
    function(str(p))
E   TypeError: open() missing required argument 'flags' (pos 2)

During handling of the above exception, another exception occurred:
/usr/lib/python3.7/shutil.py:491: in rmtree
    _rmtree_safe_fd(fd, path, onerror)
/usr/lib/python3.7/shutil.py:429: in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
/usr/lib/python3.7/shutil.py:425: in _rmtree_safe_fd
    onerror(os.open, fullname, sys.exc_info())
E   pytest.PytestWarning: (rm_rf) error removing /tmp/pytest-of-user/garbage-6af7e2cc-af3d-4612-8e51-ba8b1e27ff9f/test_cache_failure_warns0/.pytest_cache: open() missing required argument 'flags' (pos 2)
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>
[32] > …/Vcs/pytest/src/_pytest/pathlib.py(70)force_writable_and_retry()
-> warnings.warn(PytestWarning("(rm_rf) error removing {}: {}".format(p, e)))
   6 frames hidden (try 'help hidden_frames')
(Pdb++) l
 65             chmod_w(p)
 66             try:
 67                 # retry the function that failed
 68                 function(str(p))
 69             except Exception as e:
 70  ->             warnings.warn(PytestWarning("(rm_rf) error removing {}: {}".format(p, e)))
 71
 72         shutil.rmtree(str(path), onerror=force_writable_and_retry)
 73
 74
 75     def find_prefixed(root, prefix):
(Pdb++) p
PosixPath('/tmp/pytest-of-user/garbage-6af7e2cc-af3d-4612-8e51-ba8b1e27ff9f/test_cache_failure_warns0/.pytest_cache')
(Pdb++) function
<built-in function open>
(Pdb++) excinfo
(<class 'PermissionError'>, PermissionError(13, 'Permission denied'), <traceback object at 0x7fc214c2dd08>)

[blueyed]

I think it is quite a bad idea to chmod + retry in general - pytest should not mess with / change file permisisons.
(for reference: recently added in #5588)

[blueyed]

sudo ls -ld shows:

d-w------- 2 user user 40 Jul 18 18:24 /tmp/pytest-of-user/garbage-510ea6f8-9654-4901-8e7e-a77a3a330090/test_cache_failure_warns0/.pytest_cache/

[asottile]

I think it is quite a bad idea to chmod + retry in general - pytest should not mess with / change file permisisons.
(for reference: recently added in #5588)

I disagree, especially on windows

the following cases are fairly common:

• readonly files cannot be deleted on windows (git for example creates readonly files)
• files cannot be deleted from readonly directories on posix (go mod for example creates readonly directories)

[asottile]

here's (for example) what we use in pre-commit for the same issue: https://github.com/pre-commit/pre-commit/blob/95afd642f52b4d2a12ad605683c8f752e63530d0/pre_commit/util.py#L159-L172

[blueyed]

I disagree, especially on windows

Yes, I've created a PR already to fix it.. ;)

As for the pre-commit function, it might still fail - one of the issues is that os.open fails when it is not readable - so would need | stat.S_IRUSR there also likely.

Have you never seen the issue / bug?
(just happened to me again)

[asottile]

pre-commit only retries a very specific set of functions and error conditions -- os.open isn't one it retries: https://github.com/pre-commit/pre-commit/blob/95afd642f52b4d2a12ad605683c8f752e63530d0/pre_commit/util.py#L164-L165

[blueyed]

Yeah, just notices this also - it would not help in this case then altogether.

Also I am not sure if func should be used really for retrying - but that might be ok for the set of final options.