Wheel support - Initial implementation, DO NOT MERGE

[pfmoore]

This is an initial implementation of wheel support for virtualenv.

If distlib is available (either installed, or as a wheel in virtualenv_support or any extra search dir) then it will be used to install whichever of pip, distribute and/or setuptools have been requested as part of the initial install.

At present, no wheels are supplied by default - as none of the above projects distribute their code in wheel format, the wheels must be built manually and this is too complex to implement within the current simple refresh-support-files.py script. (A more complex script could be written, but this has its own risks and issues).

Notes:

• As setuptools does not understand the new .dist-info format, there are issues using pip and setuptools installed from wheels (possible resolution - require use of distribute if wheels are in use?)
• For anything that cannot be installed from a wheel (which is everything if distlib isn't present) the old code for installing from source and/or easy_install/ez_setup is used. With a fully populated virtualenv_support, this old code will never be needed.
• There are still some questions and issues to resolve - see TODO comments in the code.
• Documentation and tests have yet to be updated.

[pfmoore]

One further note - distlib 0.1.0 (the current PyPI version) does not work here, as it has a bug in setting script #! lines. The development version (or a later release) is needed for scripts/wrappers like pip.exe to work.

[pnasrat]

We probably want to allow failures with 2.5 as we're planning to drop - can you add that change in this PR.

[pfmoore]

OK, but can I clarify what you want here - just remove 2.5 from .travis.yml, or do you want an explicit version check in the code?

[pnasrat]

I was meaning edit .travis.yml to set failures to be allowed on 2.5

http://about.travis-ci.org/docs/user/build-configuration/#Rows-That-are-Allowed-To-Fail

After we actually write the drop notes, etc we can drop 2.5 from the matrix.

[pfmoore]

OK, nice. I didn't know that was possible. Thanks.

[qwcode]

If distlib is available

as already mentioned, we could use distlib from pip, and just include pip.

require use of distribute if wheels are in use

or we vendorize pkg_resources into pip that supports dist-info.

At present, no wheels are supplied by default

but it is your intention to include them, right? and have some kind of process for updating them, even if it's just documented, and not scripted.

[pfmoore]

On 13 March 2013 01:10, Marcus Smith [email protected] wrote:

If distlib is available

as already mentioned, we could use distlib from pip, and just include
pip.

Yep, on the list.

require use of distribute if wheels are in use

or we vendorize pkg_resources into pip that supports dist-info.

Nope, that would not work. pip.exe calls pip-script.py which does "from
pkg_resources import load_entry_point". That's autogenerated by setuptools.
If the target environment doesn't have a pkg_resources in place that
understands that pip now has entry_points.txt in .dist-info, the entry
point lookup will fail and pip.exe won't run.

Long term solution is to transition off older tools that don't understand
.dist_info. Short term, we need to rely on distribute, I think...

At present, no wheels are supplied by default

but it is your intention to include them, right? and have some kind of
process for updating them, even if it's just documented, and not scripted.

Yes, I wasn't sure how acceptable just documenting the process would be.
But that's certainly an option, and I'll at least do that in the first
instance.
Paul

[qwcode]

pip.exe calls pip-script.py which does "from pkg_resources import load_entry_point".
understands that pip now has entry_points.txt in .dist-info

hmm, why not just "vendorize" pkg_resources as a top-level module then?
we gotta get out from under the yoke somehow.

[qwcode]

why not just "vendorize" pkg_resources as a top-level module then?

scratch that... that's only going to work circumstantially based on the order of sys.path being right.

[pfmoore]

why not just "vendorize" pkg_resources as a top-level module then?

scratch that... that's only going to work circumstantially based on the
order of sys.path being right.

Precisely. Better might be to just stop using console_scripts in pip's
setup.py. But getting the same effect otherwise won't be easy because tool
support needs to catch up. It'd be nice if python -m pip worked, though, as
that gives people an alternative option. But making a simple "pip" typed at
the command prompt work relies on exe wrappers on Windows (or a .py file
and PATHEXT containing .py when the Windows Python launcher is installed)
and a Python script with no .py extension on Unix.

Vinay's distlib wrappers don't need pkg_resources, so maybe switching pip's
build process to not rely on setuptools would work. But we'd have to
rewrite a lot of the convenience features setuptools adds to the build in
that case.

Daniel pointed out that the wheel PEP's intention regarding scripts is that
exe wrappers or executable bits get added at wheel install time, and aren't
present in the wheel itself. But current tools don't do that. And I think
thrashing out the details of that are probably best left to version 2 of
the wheel spec.

Paul.

[qwcode]

hmm, I fear making virtualenv cluttered with these new install features, if we can't make it work across the board right now.

[pfmoore]

hmm, I fear making virtualenv cluttered with these new install features,
if we can't make it work across the board right now

Yep, I have no intention of going down that route. Simply wheel installing
what we have works fine. Wheels for pip and distribute built with current
tools work just fine. If setuptools gets updated to support dist-info then
that will work as well, otherwise we have to make a judgement on how much
we want to be held back by tools that won't support current standards. (My
view is we just require distribute if we are installing pip from a wheel
and be done with it).

The only issue here is one for pip - does it want to have a runtime
dependency on an installed pkg_resources (not a vendorized copy)? If not,
then it has to move away from setuptools-form wrapper scripts. If it does
that, virtualenv can then allow the option to install pip without
distribute. But that would be a new feature for virtualenv.

My preference would be to simplify virtualenv by removing the existing
code to install via easy_install & co. But that means requiring a populated
virtualenv_support directory if you want to install pip and/or distribute,
and I've had no feedback on the acceptability of that. So I'm being
cautious for now and leaving the old behaviour as a fallback.

[qwcode]

Yep, I have no intention of going down that route.

well, you're seriously considering supporting 2 pathways.. one for Setuptools, and one for Distribute.
that's the clutter I'm talking about. I'm not saying don't do it, but until we have more than 7 tests for virtualenv, I'd be scared to touch it myself. I'm more concerned with simplicity and testing, than adding on top the ability to install pip/distribute from wheel at the moment.

simplify virtualenv by removing the existing code to install via easy_install & co.

I'm +1 on not supporting the isolated "virtualenv.py" use case, and the code simplification that could lead to.

[pfmoore]

I'm +1 on not supporting the isolated "virtualenv.py" use case, and the code simplification that could lead to.

Excellent. Some implications of that, just to be sure we're talking about the same thing:

1. virtualenv.py would only be supported when a virtualenv_support exists next to it with pip and distribute wheels (and setuptools wheels, presuming we can resolve the dist-info issue for setuptools)
2. virtualenv.py would never download files (no need, as we're assured that the packages we need are present in virtualenv_support)
3. We could remove the embedded ez_setup, distribute_setup and distribute_from_egg scripts as they are only needed for non-wheel installs.
4. We could easily add the ability for the user to include extra packages on virtualenv creation, just by dropping the relevant wheels into virtualenv_support. (I'd probably add an option to allow the user to choose at runtime)

[qwcode]

I'm talking about the simplification independent of wheel changes. i.e. doing that first.
_install_req is so gnarly right now. I'm hoping we could get rid of refreshing/using those bootstraps.

[pfmoore]

Ah, OK. I haven't even looked at that - two reasons:

1. It's not worth it if we're dropping the whole lot when we add wheel support.
2. The code isn't that gnarly. It mostly just hunts for a local file, and then runs the bootstrap, either passing the local file name (if found) or passing nothing (if no local file). The nastiness is all around making sure the bootstraps run right, logging output, and junk like that.

Basically support for no virtualenv_support is essentially free because we use the bootstraps at the moment. There's no value desupporting it unless we drop the bootstraps, and if we do that we need an alternative install process. In my view, that's wheels. I'm not going to write my own code to install from a sdist when the bootstraps do that now.

[qwcode]

my experience with it's gnarliness, came in trying to fix #327,and then I gave up out of fear. (btw, we really need to update the docs about how that doesn't actually work)

if you can get Setuptools/PJ to add dist-info, that would be great.... or better yet, get the 2 projects to remerge again.

[qwcode]

in large part, you're wanting to add this so windows virtualenv creation can be faster, and have it be a distribute-only option, but doesn't distribute have a bug on windows, that setuptools doesn't have, so that doesn't deflate this plan a bit at the moment?

https://bitbucket.org/tarek/distribute/issue/143/easy_install-opens-new-console-cant-read

[pfmoore]

in large part, you're wanting to add this so windows virtualenv creation
can be faster, and have it be a distribute-only option, but doesn't
distribute have a bug on windows, that setuptools doesn't have, so that
doesn't deflate this plan a bit at the moment?

https://bitbucket.org/tarek/distribute/issue/143/easy_install-opens-new-console-cant-read

Ah, I've seen that issue, but I never use easy_install so it doesn't affect
me. It's related to Windows UAC and elevation issues. I think I know how to
fix it, I'll pop over to the tracker entry and comment there.

[qwcode]

given the Distribute/Setuptools merge announcement, I would say shelve this until that's released, and then all this will be easier to consider.

[tomprince]

setuptools/distribute has now merged.

[pfmoore]

My current plan is to put the supplied pip wheel onto sys.path, then use pip.main to do the install. There are still some issues to iron out, though, as to install from a wheel, pip currently needs features of setuptools that don't seem to work when pip/setuptools are running from a wheel (i.e., a zipfile).

Closing this pull request - I will open a new one when the above plan actually works...