(pypy/debian only) setuptools broken during an upgrade

[fungi]

On some systems, and so far I've been unable to determine what makes them "special" in this way, upgrading setuptools with pip at the same time as upgrading particular other packages causes their setup.py invocation to fail with "error: option --single-version-externally-managed not recognized". Upgrading them independently of setuptools (either before or after) does not exhibit this behavior.

I saw it briefly weeks ago, but the release of setuptools 2.2 seemed to cause it to cease. The most recent occurrences seem to possibly coincide with Saturday's release of setuptools 3.1, but that could simply be a red herring.

I have narrowed it down to a simple means of recreating this on the affected systems where we're seeing it currently: within a pypy virtualenv, use pip to upgrade setuptools and mccabe together on the same command line. Examples...

Working consistently on Ubuntu 12.04 VMs in one service provider: http://paste.openstack.org/show/73058/

Failing consistently on seemingly-identical Ubuntu 12.04 VMs in another service provider: http://paste.openstack.org/show/73060/

I discussed this with Alex Gaynor just now, and we agree that this feels like some sort of strangeness in pip which is getting triggered by hash ordering differences in different systems/interpreters rather than being a PyPy-specific bug. On the systems where this fails in a pypy virtualenv, it succeeds normally in a python2.7 or python3.3 virtualenv.

[dstufft]

<fungi>     dstufft: inspecting the virtualenv shows that the new setuptools installed after it throws that error
<fungi>     dstufft: foo/bin/pypy -c 'import setuptools;print(setuptools.__version__)' says 3.3 but list says setuptools (3.1)
<fungi>     dstufft: another point of interest, 'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described
<fungi>     dstufft: also, downgrading pip to 1.3.1 in the virtualenv before installing/upgrading setuptools and other packages seems to cause this problem not to happen

[fungi]

This started impacting systems on our remaining provider over the weekend, and seems coincidentally timed to the release of setuptools 3.3 (no idea whether that could be related to the new shift in behavior).

[dstufft]

/cc @jaraco

[jaraco]

It sounds like the issue is related to something about how the setuptools package is updated during the installation process. It sounds as if the issue is not related to a specific setuptools version, but rather the way in which the version changes. I would not be surprised if this issue is related to the way the 'sandbox' is created during builds, though I really don't have any insight to add yet.

I can say that it seems unlikely that the recent changes would have affected the way builds are done with respect to that command line argument.

[qwcode]

here's the story:

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use __pycache__, e.g. on debian), that results in an empty __pycache__ folder being written to the dist-info folder. you can see it in a fresh pypy virtualenv for pip and setuptools. i.e. the root of this is not just a setuptools issue. this is true for any wheel install under pypy (that uses __pycache__)

pip uses the dist-info/RECORD file to uninstall things. it doesn't know about this dist-info __pycache__ folder, so it's get left, along with the parent dist-info folder, during an uninstall (or an upgrade, that does an uninstall).

so right now, pip/pypy uninstalls of any wheels, are leaving empty shells of old metadata around, which sets up a failure scenario later due to metadata mismatches.

so back to the setuptools case:

for me, it's not just failing with: `pip install -U setuptools mccabe'

but also much worse, like this:

pip install -U setuptools
pip install mccabe

pip's internal use of setuptools fails with the duplicate metadata. if you remove the old metadata dir, everything is ok.

so, what's the fix? the RECORD file needs to have an entry for the dist-info __pycache__ folder.

and this looks worthy of a patch release.

[qwcode]

'foo/bin/pip install -U setuptools==3.1 mccabe' succeeds but ==3.2 or 3.3 breaks as described

I don't see that. breaks for 3.1 too.

[pfmoore]

when pip installs from wheel, it runs compileall.compile_dir. Under pypy (specifically pypy distributions that use pycache, e.g. on debian), that results in an empty pycache folder being written to the dist-info folder

Maybe I'm being dense but isn't this a bug in pypy? Why is it creating a __pycache__ directory in a directory that contains no Python files?

I'm not saying we don't need to work around it, but it's worth a pypy bug report too.

[Ivoz]

AFAIK the issue that is *debian pypy is running a Python 2 implementation, yet __pycache__ is really from python 3. Additionally, the API to detect this is in Python 3, but obviously not in Python 2. So pypy is importing a Python 3 feature into Python 2 and without the accompanying API to manage or detect the feature.

I'm guessing they just never guessed that importing __pycache__ into a python 2 implementation would manage to end up screwing up pip while it uninstalls things.

[dstufft]

I'm not sure the fact that it's a __pycache__ folder really matters, just that there's one being created in the dist-info directory which doesn't happen in CPython.

[arigo]

PyPy (version 2.7.x) doesn't contain any line of code with __pycache__ in it. Grepping both the source code and the official binary doesn't find it (just like it doesn't find it on CPython 2.7.x). But Debian patches PyPy and some of their patches does contain __pycache__. Based on this I think it's safe to say that, if anything, it's an issue that should be reported to the Debian maintainers.

[arigo]

More precisely, grep __pycache__ turns out quite a number of hits in pypy_2.2.1+dfsg-1.debian.tar.gz from the url https://packages.debian.org/source/jessie/pypy for example.

[alex]

Yes, it seems debian (and the PPA) include a set of patches which do cause
pycache to be used.

On Wed, Mar 19, 2014 at 7:40 AM, arigo [email protected] wrote:

More precisely, grep pycache turns out quite a number of hits in
pypy_2.2.1+dfsg-1.debian.tar.gz from the url
https://packages.debian.org/source/jessie/pypy for example.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/1632#issuecomment-38057950
.

"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: 125F 5C67 DFE9 4084

[dstufft]

Here's an upstream bug https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=742132

Probably we should work around this in pip too. It seems like we shouldn't fail because of an empty directory that we installed, even if it was created erroneously.

[pfmoore]

It's probably reasonable not to worry about changing RECORD, because we don't typically put directory names in there as far as I can see. Instead, let's just delete __pycache__ on the assumption that if the source file is gone, the cache can go too.

How about, if we delete files from a directory, then when we're done if the directory contains nothing but a __pycache__ subdirectory, we delete that? That's a pretty good general principle, __pycache__ directories are intended to be ignorable, after all...

[dstufft]

Well the question is, if someone has an empty directory in their dist-info already what will we do? I think ATM we'll install it and then fail in the same way that this fails. IOW this failure isn't specific to __pycache__, it just happened to be exposed by it.

[pfmoore]

We've got to the point where I should look at the code before guessing, but I think that we don't currently record directories in RECORD, only files. That's why I'm a bit reluctant to include __pycache__ as it might expose other issues (trying to calculate the hash of a directory, using unlink rather than rmdir etc). But I take your point.

[dstufft]

Actually, what's probably a better solution is to make it so we only deal with files. Right now we have this sort of bad state where when installing we'll happily install directories, but uninstall we only uninstall files. My assertion is that if we install something, then we should also uninstall it.

So I would say ideally we should only install files (creating any directories we need to) and we should only uninstall directories (removing any directories that we make empty, or empty sans .pyc and __pycache__ directories.

This should solve the problem that is the cause of this ticket, and make things work in a much saner fashion.

[qwcode]

the API to detect this is in Python 3,
but obviously not in Python 2
So pypy is importing a Python 3 feature into Python 2
and without the accompanying API to manage or detect the feature.

debian hacks the imp lib as well. I discovered this 2 years ago, when I was working another pip uninstall issue related to __pycache__ (#646)

[Ivoz]

Why debian, why

[qwcode]

for history's sake, here's the pypy thread from 2012, when it was discovered that the debian distro of pypy adds __pycache__.
https://mail.python.org/pipermail/pypy-dev/2012-July/010369.html

the debian maintainer chimes in after a few posts.

[alex]

Thanks so much for fixing this @qwcode -- can this issue be closed?

[qwcode]

yes, closing