Remove pybundles

[ianb]

Pybundles are an experiment that I don't think is really going to go anywhere. I think they could be removed.

[hltbra]

I don't think so Ian. I need to investigate, but if I am not wrong, some folks in Globo.com (the company I work for) are willing to use them in our servers.

I need to investigate the reasons to continue this conversation. But the idea of have a file with all you need is very good (no internet dependent is nice).

[hltbra]

I have talked to some guys and the company does not need bundles.

[ianlewis]

I kind of wish bundles would go somewhere. I'd like to use them as part of a deployment process as the number of projects and various environments we have for customers make setting up and using a local pypi impossible.

[bradleyayers]

I too would like to see bundles developed further. oh, and i'm actually using the current implementation, so I strongly object to it being removed!

[pnasrat]

Probably should take the discussion to the mailing list. I think we need a better idea of the problem we are trying to solve, and what cases we need to think about (eg bundles with native, etc).

[hvdklauw]

Don't remove it please. It works fine as it is.

In our deployment script we check last revision of the requirements.txt file and then check if the {revision}.pybundle file exists. If it doesn't we create it on the server. Then we use the bundle to install the packages.

The upside of this is that if the requirements haven't changed we can deploy even if pypi or something is down, thus reducing our dependency and speeding up pip install too.

[ashwoods]

+1 for non-remove. We are using it right now in the same way as hvdklauw, and its a great alternative to having to run your own internal pypi server. Our deployment is dependent on github and pypi, and using pybundles has eliminated that dependence somewhat. Our CI generates the bundle anytime requirements change. Would hate to see it go.

[oinopion]

I'm using it in my projects to speed up hudson runs. It's not pleasant to wait 5+ minutes to clone Django git repo just to run 20 seconds test suite in fresh virtualenv.

[carljm]

I have never understood what advantage bundles have over a directory full of sdists. The latter seems to meet exactly the same use cases, and is more granular and much easier to inspect and modify. Decent tools for transferring files to servers etc are just as easily able to work with directories as with single files.

It's not entirely true that bundles "work fine as is" - #282 seems like a pretty egregious bug (though I wouldn't really know, as a non-bundle-user).

[ianb]

The download cache would seem to handle most testing cases just as well.

[oinopion]

Download cache didn't work well with git repos. Maybe something changed since, Ive got to re check.

[ianb]

Nothing with git repos, though I believe -e will do a git pull rather than clone when possible, if everything is configured appropriately. Otherwise using a locally-created tarball seems best.

[bradleyayers]

I use it in a distributed system to send packages around. Sometimes I need to make modifications to the code in dependencies and this makes it easy to bundle everything together. If bundles were removed I'd probably just end up having to reimplement it essentially in my application.

[carljm]

The download cache in general isn't as effective as one would hope, because a good chunk of pip's slowness actually comes from the PackageFinder and its scraping requests, not the download of the sdist itself. Download cache only helps with the latter. That's the advantage of bundles or directories of sdists (used with --find-links and --no-index); you can avoid HTTP package-finding entirely.

[oinopion]

@carljm @ianb how would one automate making directory of sdist? (I'm packaging noob)

[carljm]

@oinopion "pip install -d some/directory -r requirements.txt" will dump sdist tarballs for every package in requirements.txt into some/directory (and will not actually install them).

[hvdklauw]

And how would you install them then based on the requirements?

[carljm]

@hvdklauw "pip install --no-index --find-links file:///path/to/some/directory -r requirements.txt"

[oinopion]

Thanks, I'll try that.

[Helcium]

Is it possible to create a bundle with no need for registration or uploading it to a shared repo? I'd like to make a bundle of a project, but I am not allowed to share its source code. I just saw the solution of carljm, but it would be nice if it could create a bundle (which does not need to register or upload it or even make any download of it from the repo) and still install all the requirements from the repo (and in this case download them all).

[carljm]

@helcivm The solution I mentioned does not require you to register, share, upload or download any code; you can do it all locally with a machine that doesn't even have network access. You can run "python setup.py sdist" to create an sdist tarball of a project in a dist/ subdirectory, and then place that sdist tarball in some local directory and run pip install --no-index --find-links file:///path/to/that/directory projectname to install that project without reference to any external resources.

This kind of query would fit better on the mailing list (https://groups.google.com/group/python-virtualenv) than on a ticket.

[Helcium]

Thanks, carljm. I was googling for a solution, then I was redirected to this thread. I'll try what you've just said. Thanks again.

[sblask]

I use pip bundle in combination with Ansible. My setup.py contains dependency links to SVN and GIT repos. I need sudo rights for installing my package, but sudo doesn't play well with public key authentication. So I create a bundle with the user that has access to SSH and install it using sudo. Of course I could do this with with sdist, but I'd lose the automatic resolving of dependencies.

[dholth]

@sblask you should also try wheel, a binary package format documented as PEP 427. https://github.com/pypa/pip/tree/wheel

[sblask]

@dholth thanks for the hint. There are probably advantages over sdist, but I've got a hard time seeing them. Anyway, they don't seem to be a replacement for bundles, which can contain multiple distributions, not just one. However, to me, bundles' biggest advantage is that pip can create them automatically from requirements or through dependencies from one package, I don't really mind it's format. Pip could also just create a directory of sdists or wheels for all requirements or dependencies.

I don't know whether it has been like this forever, but bundles do not appear in the documentation. That might just be the reason that not that many people use it.

[dholth]

The pip wheel command also produces a directory full of wheels for all of a package's dependencies or from the requirements file.

The biggest advantage of the wheel format is that it's precompiled, and the installation side does not require setuptools.

[sblask]

Right, that sounds good. I think that would be worth being mentioned under http://www.pip-installer.org/en/wheel/usage.html#pip-wheel If I was looking for it and didn't see it, how about people who are not looking for it? They will never know.

The wheel documentation could also use some more information about the internals. I found it confusing that the PEP states that wheels don't include .pyc even though they are precompiled. Where do the savings come from?

[dholth]

wheel.rtfd.org has the details taken out so they can be put in PEP 427, good suggestion to put some back in.

The savings come from when you have C extensions. I think what wheel does is probably simpler & faster than what setup.py does even if you don't have C extensions; it would be a good idea to time it.

[pfmoore]

On Windows, it is definitely faster to use wheels than setup.py even just for pure Python. And in particular, for code that invokes 2to3 at install time, prebuilding wheels is a lot faster.

[msabramo]

I thought I'd mention that the pybundle concept (not the implementation) is use by some other tools such as terrarium and Twitter's pants/pex. These are focused on creating very reproducible and fast virtualenvs. They might be worth looking at for pip users and developers alike.