presence of empty file "package....egg-info" causes pip to report "Requirement satisfied"

[mjdellwo]

• Pip version: 9.0.1
• Python version: 3.4.1
• Operating system: RHEL 6.8

Description:

The mere existence of an empty "-<...>.egg-info" file in a directory searched by PYTHONPATH environment setting is sufficient for pip to believe that the package is already installed, and report that the required package is already available. More relevant to my particular use case, the same problem naturally applies to a broken symbolic link.

This problem does NOT occur if an empty "-....egg" file is present. However, empty directory, or broken link, gives an actual exception, not just a "requirement met" message.

In my particular case, I need to manage multiple versions of packages in a 'library', and use symbolic links to a central PYTHONPATH for the 'current' blessed version. I tend to recompile / re-install items frequently as part of managing the service. This problem prevents recompiles unless I first clear out the broken links.

Managing multiple versions would be so much simpler if pip promoted an unpacked 'egg' directory containing everything, instead of an 'egg-info' directory next to the module directory. The egg-info preference makes it very hard to manage multiple versions. I've tried using the --egg flag to pip to accomplish this, but not only is it deprecated but it doesn't have the intended effect in every case (I still get egg-info rather than egg for some packages). Ah, but I digress.

Leaving out the --exists-action or --no-binary flags below makes no difference.

What I've run:

$ uname -a
Linux awsacdnva1004 2.6.32-642.4.2.el6.x86_64 #1 SMP Mon Aug 15 02:06:41 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
$ pip3.4 --version
pip 9.0.1 from /my/library/Python/3.4.1/lib/python3.4/site-packages (python 3.4)
$ ls -l /my/pypath/boto3-1.4.4-py3.4.egg-info
-rw-r--r--+ 1 mjdellwo users 0 Jan 31 16:09 /my/path/boto3-1.4.4-py3.4.egg-info
$ ls -l /my/pypath/boto3
ls: cannot access /my/pypath/boto3: No such file or directory
$ env PYTHONPATH=/my/pypath pip3.4 install --disable-pip-version-check --exists-action w
--no-cache-dir --no-binary :all:
--prefix=/my/library/boto3/1.4.4_64-3.4 boto3==1.4.4
Requirement already satisfied: boto3==1.4.4 in
$ rm /my/pypath/boto3-1.4.4-py3.4.egg-info
$ env PYTHONPATH=/my/pypath pip3.4 install --disable-pip-version-check --exists-action w
--no-cache-dir --no-binary :all:
--prefix=/my/library/boto3/1.4.4_64-3.4 boto3==1.4.4
Collecting boto3==1.4.4
Downloading boto3-1.4.4.tar.gz (92kB)
100% |████████████████████████████████| 102kB 4.4MB/s
Requirement already satisfied: botocore<1.6.0,>=1.5.0 in [...]
[... and other requirements fulfilled...]
Installing collected packages: boto3
Running setup.py install for boto3 ... done
Successfully installed boto3-1.4.4
$ ls -l /my/library/boto3/1.4.4_64-3.4/lib/python3.4/site-packages/
total 16
drwxrwxr-x+ 9 mjdellwo users 4096 Jan 31 16:39 boto3
drwxrwxr-x+ 2 mjdellwo users 4096 Jan 31 16:39 boto3-1.4.4-py3.4.egg-info
$ rm -rf /my/library/boto3/1.4.4_64-3.4
$ touch /my/pypath/boto3-1.4.4-py3.4.egg
[... run same pip install command, it succeeds again...]
$ rm -rf /my/library/boto3/1.4.4_64-3.4
$ rm /my/pypath/boto3-1.4.4-py3.4.egg
$ mkdir /my/pypath/boto3-1.4.4-py3.4.egg
[... running the command gives requirement already met followed by an exception:]
$ env PYTHONPATH=/my/pypath pip3.4 install --disable-pip-version-check --exists-action w --no-cache-dir --no-binary :all: --prefix=/my/library/boto3/1.4.4_64-3.4 boto3==1.4.4
Requirement already satisfied: boto3==1.4.4 in /my/pypath/boto3-1.4.4-py3.4.egg
Exception:
Traceback (most recent call last):
File "/my/library/Python/3.4.1/lib/python3.4/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/my/library/Python/3.4.1/lib/python3.4/site-packages/pip/commands/install.py", line 324, in run
requirement_set.prepare_files(finder)
File "/my/library/Python/3.4.1/lib/python3.4/site-packages/pip/req/req_set.py", line 380, in prepare_files
ignore_dependencies=self.ignore_dependencies))
File "/my/library/Python/3.4.1/lib/python3.4/site-packages/pip/req/req_set.py", line 666, in _prepare_file
check_dist_requires_python(dist)
File "/my/library/Python/3.4.1/lib/python3.4/site-packages/pip/utils/packaging.py", line 48, in check_dist_requires_python
feed_parser.feed(metadata)
File "/my/library/Python/3.4.1/lib/python3.4/email/feedparser.py", line 167, in feed
self._input.push(data)
File "/my/library/Python/3.4.1/lib/python3.4/email/feedparser.py", line 100, in push
data, self._partial = self._partial + data, ''
TypeError: Can't convert 'NoneType' object to str implicitly

[pfmoore]

I'm not sure what you are suggesting here. You're manually modifying the files in $PYTHONPATH, so it's really your responsibility to ensure that you do so in a way that pip (and all rest of the Python ecosystem) recognises. Pip supports the standard (PEP 376) dist-info format, and offers limited support for the older setuptools egg-info support, but we're not likely to do anything to extend how we support egg-info files as they are obsolete (the same goes, even more so, for the egg format).

[mjdellwo]

Fair enough. I guess I'm suggesting that your user community never behaves in the proscribed fashion and you have an edge case. :) But, I can see well enough how to work around it by clearing broken links in advance. The hard Exception in the last example seems like something you wouldn't ever want to occur in any case; I didn't fully try out the dist-info case, but am guessing the same problem would occur in the case of an empty *.dist-info directory or file. I did try a binary install in the presence of the empty egg-info file and had the same problem. I don't think the problem is necessarily with the package format, but definitely a consequence of my environment's complexity. [It's also the case in my very large and complex ecosystem that not all packages are available in the newer format. I'll deal.]

As you rightly point out, this is not likely to occur for most. I'll close this if you like. (or you can, I guess? )

[dstufft]

I'm going to go ahead and close this since this is just the way the on disk package db works currently and changing that would require a PEP.