Upstream downstream patches for allowing to make a relocatable (standalone) Python installation

[FFY00]

Feature or enhancement

Proposal:

I am currently looking at https://github.com/indygreg/python-build-standalone/tree/main/cpython-unix

This would be the first step towards python.org possibly providing a binary build for Linux.

cc @jezdez @indygreg

Has this already been discussed elsewhere?

No response given

Links to previous discussion of this feature:

No response

[vstinner]

You should elaborate the issue a little bit :-) Which changes do you want to upstream? To do what?

[FFY00]

Yes, sorry! And sorry for not seeing this earlier, Github notifications are a mess. The goal would be to ultimately have a fully relocatable Python instalation (there's some discussion on GH-62509). This issue is specific is to track the work on understanding which downstream pacthes make sense to be upstreamed, and to upstream them.

https://github.com/indygreg/python-build-standalone/tree/main/cpython-unix is a good starting point.

My thinking here was to not re-do all the work, and base on downstream projects already doing this. Does that make sense?

[vstinner]

There are around 26 patches, it's scary. I suppose that we can decide on a case by case basis. But having an overall rationale for what is a "relocatable Python installation" would help.

[FFY00]

Well, I'd like to at some point being able to ship official binaries for Linux on python.org, that's my main motivation. Relocatable installs also make a lot of sense for applications like embedding, etc.

[vstinner]

Can you please describe what is a "relocatable Python installation"? What are the current issues that you're trying to solve? Why is it important for Python to support such installation?

[warsaw]

Can you please describe what is a "relocatable Python installation"? What are the current issues that you're trying to solve? Why is it important for Python to support such installation?

Maybe this should be a PEP?

[zanieb]

Hi!

As commented in the PEP 711 PyBI discussion, standalone Python distributions are being used today in a several tools and by many users. For example, Hatch, PyApp, Rye, and uv all use indygreg's standalone builds. I think this is demonstrative of some need.

@mitsuhiko has some documentation about the use of standalone builds in Rye:

The motivation for this is that it makes it easy to switch between Python versions, to have a common experience across different Rye users and to avoid odd bugs caused by changes in behavior.

Unlike many other Python versions you can install on your computer are non-portable which means that if you move them to a new location on your machine, or you copy it onto another computer (even with the same operating system) they will no longer run. This is undesirable for what Rye wants to do. For one we want the same experience for any of the Python developers, no matter which operating system they used. Secondly we want to enable self-contained Python builds later, which requires that the Python installation is portable.

@ofek uses standalone builds in PyApp to create bootstrapped standalone Python applications.

For use in uv (which I maintain), it is particularly important for us to be able to download optimized builds and place them in an arbitrary location on the user's machine. We need support for many Python versions. We don't want the user to need to install dependencies other than uv itself. We don't want behavior of Python to change depending on the versions of libraries on the user's machine. Compiling Python on the user's machine isn't compatible with those goals. We can't invoke distribution-specific package managers because they require root we don't want to require root to bootstrap Python.

I'm not an expert and I don't have context on why CPython builds use absolute paths today. I'm sure it's a complicated topic! I have some team members that can probably provide some more detail, but I figured I'd take a swing at an initial answer.

While writing this, I found some previous discussion here if you want more context. I also thought there was some relevant discussion in PEP 711.

[zooba]

I'm not an expert and I don't have context on why CPython builds use absolute paths today.

It's just platform convention. All the paths on Windows are relative by default, because that's how that platform works, but other platforms use absolute references by default.

Python's build system was also designed to integrate into an OS (I undid a lot of that for Windows back around 3.5), not to be a standalone app. Since that was basically the only distribution mechanism back in the 90's, it makes perfect sense, but it should certainly be open to updating.

[zanieb]

That's good to know, thanks for sharing that additional context. It makes sense that it's mostly historical and aligned with the distribution mechanisms of the time. I think it's fair to say that lately there is an increasing focus on portable distribution mechanisms for applications and it'd be great to see Python move towards supporting those use-cases. I'm curious if there are reasons this wouldn't be desirable? (Other than the obvious large amount of work necessary to get there)

[zooba]

The main reason that I find it undesirable is that portable (particularly "many-Linux" portable) distributions will put the build, maintenance, and release burden back on the volunteer team. Right now, it's either handled by paid or volunteer employees of the companies that produce their own distributions.

If we were to maintain our current policy (no binaries for security-fix only releases) but get users onto our own portable releases, many users would find themselves stuck when they need to switch to a different release mechanism to get trustworthy security fixes. It's possible that new companies would start distributing portable builds with fixes integrated (it's worth noting that Anaconda already does this), but I expect most Linux distros would stick with the builds that integrate into the rest of their OS.

Now, don't get me wrong, I think most Python users should avoid their distro's own build of Python (it's usually the system Python, meant for system apps, and the user's app shouldn't blindly adopt the system's one). But I don't think that most are ready to do it, and at best will end up relying on repackaged builds from upstream. So there's a pretty significant community dynamic here that is not obviously going to work out in users' favour.

All that said, having the capability to build a relocatable install would be great. I have my own set of build hacks to make it possible for my $WORK scenarios, so I'd love to drop those. Making relocatable releases is actually a totally separate question, but as it's the motivation for having the changes upstream, until somebody says they're going to make those releases, it looks like pretty pointless churn unfortunately.

[vstinner]

The cpython-unix/ directory of https://github.com/indygreg/python-build-standalone contains 37 patches. Which patches are the most important? I don't see which patch magically makes Python relocatable. Is it another script which makes it relocatable?

[zanieb]

That's a good question — I can try to answer it but I'll need to dig around quite a bit since I didn't write the project. Let me loop in @indygreg, if he has time.

[zooba]

This is the most critical (native) part:

https://github.com/indygreg/python-build-standalone/blob/ef71d13ba98706995ef056801fea1e88e172d7c6/cpython-unix/build-cpython.sh#L516-L566

Changes in the build script would presumably pass different linker options rather than patching the built binaries.

I believe most of the other changes are to clean up assumptions within the stdlib (primarily sysconfig I'd imagine) about the layout.