jupyter-core==4.8.1 causes PermissionError on Windows

[CHDev93]

I upgraded to the latest version of jupyter-core and find that I get the following error

PermissionError: [WinError 5] Access is denied: '<my_home_drive>'

This error seems to be due to the fact that pathlib.Path().resolve() does not work on windows with networkshares. Since my home drive is on a network share, I get a crash from

  File "<my_environment>\lib\site-packages\jupyter_core\paths.py", line 41, in get_home_dir
    homedir = str(Path(homedir).resolve())

Is it possible to patch this to avoid using resolve()?

Related comment from the same issue being discovered in pytest

[jasongrout]

Ironically, we switched to using Path.resolve to solve a different Windows issue: #222 (comment)

The discussion in that pytest issue certainly looks like Path.resolve causes issues. Do you happen to have instructions I could use on a Windows machine to try to replicate this? Also, what exact Python version are you using? It seems that the behavior has changed over the years with Python, and possibly moving back to realpath won't help in the future since its behavior seems to have changed as well, if I read the pytest discussion correctly.

[CHDev93]

Thanks for the quick response! I did do a little digging and realised just how subtle dealing with these windows paths across various python versions is.

To replicate this,

• make sure the environment variable $env:HOME is pointed to a network share
• create a python 3.6.6 virtualenv (very old I know) and activate it (did this in PowerShell myself but don't think it matters).
• install the latest jupyter and run jupyter notebook

Doing pip install -U jupyter-core==4.7.0 resolves the issue

[jasongrout]

Just curious, is it easy for you to check if it still works with jupyter-core 4.7.0 and Python 3.9 (or even 3.8)? My cursory understanding is that they changed realpath (see https://bugs.python.org/issue37993) to behave more like Path.resolve (which is what we were using before the PR mentioned above), so I'm not sure that even reverting to the code in 4.7.0 will help with later versions of Python.

[CHDev93]

Ran the tests and here are the results
python3.9 + jupyter-core==4.8.1: Works
python3.9 + jupyter-core==4.7.0: Works
python3.6 + jupyter-core==4.8.1: Fails
python3.6 + jupyter-core==4.7.0: Works

Rather surprised that it worked with python3.9 actually. Currently I'm a bit pinned down to python3.6 but this should change in the near-ish future.

[jasongrout]

That is very interesting, thanks for testing!

If you have time, can you test Python 3.7? I ask because Python 3.6 is almost EOL (Dec 2021), so if it is a 3.6-specific problem, and not a problem on Python 3.7, it may be easiest to document the issue, recommend those having this issue with 3.6 stay with jupyter_core 4.7.0, and recommend upgrading to Python 3.7.

[CHDev93]

Sorry for the late response on this! Looks like something changed in python 3.8 which avoids the error

python3.7 + jupyter-core==4.8.1: Fails
python3.7 + jupyter-core==4.7.0: Works
python3.8 + jupyter-core==4.8.1: Works
python3.8 + jupyter-core==4.7.0: Works

[jasongrout]

More data:

Python 3.7:

In [9]: import sys;sys.version
Out[9]: '3.7.10 | packaged by conda-forge | (default, Oct 13 2021, 20:21:52) [MSC v.1916 64 bit (AMD64)]'

In [10]: import os; from pathlib import Path

In [11]: s=r'z:\filelist.txt'

In [12]: Path(s).resolve()
Out[12]: WindowsPath('//Mac/Home/filelist.txt')

In [13]: os.path.realpath(s)
Out[13]: 'z:\\filelist.txt'

Python 3.8:

In [10]: import sys; sys.version
Out[10]: '3.8.12 | packaged by conda-forge | (default, Oct 12 2021, 21:19:05) [MSC v.1916 64 bit (AMD64)]'

In [11]: import os; from pathlib import Path

In [12]: s=r'z:\filelist.txt'

In [13]: Path(s).resolve()
Out[13]: WindowsPath('//Mac/Home/filelist.txt')

In [14]: os.path.realpath(s)
Out[14]: '\\\\Mac\\Home\\filelist.txt'

[jasongrout]

I'm trying to reproduce this problem on Windows. I tried your reproduction steps (within the miniconda prompt). Here, z: is a mapped network share (I think--I'm not too familiar with network shares on Windows).

conda create -y -n jcore36 python=3.6.6 jupyter_core=4.8.1 notebook
conda activate jcore36
set USERPROFILE=z:\deltmp

With this setup, it seems that paths.get_home_dir() works great, i.e., it resolves to the network share name.

In [1]: from jupyter_core import paths

In [2]: paths.get_home_dir()
Out[2]: '\\\\Mac\\Home\\deltmp'

On your computer, this similar setup gives that permission error?

[jasongrout]

One option is for us to revert the get_home_dir change in line 40 from a942444#diff-e185ceb5aea025405e24e7c43c570a827b66ff50b33e43a99f2d7c66ea192316R41, which was done for consistency with the actual fix in this PR over in line 93. However, this is subtle enough that I'd rather understand more precisely if (a) the change in line 93 is still good, and (b) exactly what the problem is with line 40.

[CHDev93]

On your computer, this similar setup gives that permission error?
When using the code with Path.resolve I do get a permission error. Here's the unabridged error

PermissionError                           Traceback (most recent call last)
<ipython-input-9-e4dbeb924fb7> in <module>
----> 1 str(Path(homedir).resolve())

<python_path>\lib\pathlib.py in resolve(self, strict)
   1137         if self._closed:
   1138             self._raise_closed()
-> 1139         s = self._flavour.resolve(self, strict=strict)
   1140         if s is None:
   1141             # No symlink resolution => for consistency, raise an error if

<python_path>\lib\pathlib.py in resolve(self, path, strict)
    191                 while True:
    192                     try:
--> 193                         s = self._ext_to_normal(_getfinalpathname(s))
    194                     except FileNotFoundError:
    195                         previous_s = s

PermissionError: [WinError 5] Access is denied: '<my_correctly_resolved_network_drive>'

Unfortunately I don't know much about windows network shares either. I appreciate that this is a very specific issue and it might not be worth rolling back. I have a workaround by downgrading jupyter-core and simultaneously planning to migrate to python 3.8

[jasongrout]

Given that Python 3.7 support ends in 10 months at this point (June 2023), I think either downgrading to jupyter-core 4.7 if you are on Python 3.7 and this issue is a problem, or migrating to Python 3.8+ is a great strategy.