[MAJOR] Restructuring the repository, and backwards compatibility concerns

[pradyunsg]

Coming from pypa/get-virtualenv#1

This is an issue for discussing potential areas of concern when we rework bootstrap.pypa.io's deployment assumptions and, therefore, this repository's layout.

[pradyunsg]

I've discussed this on other channels (private chats, IRC, etc) with various folks (including @ewdurbin), summarizing those discussions:

• Should be easy enough to update our invoke task.
  • Would be a good chance to upgrade/switch to nox. :P
• We can't serve redirects from the current bootstrap.pypa.io/get-pip.py since curl does not follow redirects by default.
  • Do we want to serve the same file, or a file containing an additional warning?
• Can we compute digests on the server side, or do we want them to be a part of this repository?
  • The latter seems easier.

[pradyunsg]

@ewdurbin How would we handle the transition between "current" and "next-gen"?

Should I make a commit that adds the next-gen variants in a public/ folder, and then, wait till we switch the deployment over? In a follow up, I'll likely replace the existing scripts with helpful-error-printing-scripts, which we'll keep around for 2 weeks or so.

[ewdurbin]

So is the goal to eventually remove:

• /2.6/get-pip.py
• /3.2/get-pip.py
• /3.3/get-pip.py
• /3.4/get-pip.py

and replace them with

• /pip/2.6/get-pip.py
• /pip/3.2/get-pip.py
• /pip/3.3/get-pip.py
• /pip/3.4/get-pip.py

?

If so, yes the initial step would simply be to add the public directory, which would be updated to serve from bootstrap.pypa.io/pip, symlinking public/get-pip.py so that bootstrap.pypa.io/get-pip.py serves the latest and greatest.

Once that's deployed, you could drop get-pip.py from the root of the repo and adjust the other versions to do as you want.

Eventually when you're ready to fully deprecate the top level version directories, we can remove them from the configuration on the server and they can be removed from this repository as well.

[pradyunsg]

TBH, my goal is to make it so that we don't need to make a PR to psf-salt whenever pip drops support for a Python minor version. I figure that moving the scripts in this repository into a "clean" public directory that can just be dumped into the bootstrap.pypa.io folder, should make it straightforward for us to decouple that.

I'm "meh" on making changes to the server side URLs -- it would need transitioning users though, so if someone wants to do that, we should definitely discuss how we'd go about doing that. :)

[pradyunsg]

FWIW, I've filed #102 and if @ewdurbin is generally positive for making the change I described above (change the psf-salt to "just use" the public directory in this repository), then we can go ahead and:

• merge Add current scripts to public/ directory #102. :)
• make the changes in psf-salt, and let the server update.
• change the non-public scripts in this repository to error out with a message pointing to the new location.
• wait 4 weeks.
• drop the non-public scripts from this repository.

[pradyunsg]

TBH, my goal is to make it so that we don't need to make a PR to psf-salt whenever pip drops support for a Python minor version.

@ewdurbin nudge to ask if this is possible by moving this project's psf-salt to using public/, so that this project can remove the generated scripts in the root of this repo.

[ewdurbin]

I'm all for this approach. The only issue I'm seeing is that the structure of directory "composes" the various projects all together under the root:

ernestd@web0:/srv/bootstrap/www$ sudo tree -a
.
├── 2.6
│   └── get-pip.py -> /srv/bootstrap/pip/2.6/get-pip.py
├── 2.7
│   └── get-pip.py -> /srv/bootstrap/pip/2.7/get-pip.py
├── 3.2
│   └── get-pip.py -> /srv/bootstrap/pip/3.2/get-pip.py
├── 3.3
│   └── get-pip.py -> /srv/bootstrap/pip/3.3/get-pip.py
├── 3.4
│   └── get-pip.py -> /srv/bootstrap/pip/3.4/get-pip.py
├── 3.5
│   └── get-pip.py -> /srv/bootstrap/pip/3.5/get-pip.py
├── bootstrap-buildout.py -> /srv/bootstrap/buildout/bootstrap/bootstrap.py
├── ez_setup.py -> /srv/bootstrap/setuptools/ez_setup.py
├── get-pip.py -> /srv/bootstrap/pip/get-pip.py
├── virtualenv -> /srv/bootstrap/virtualenv/public
└── virtualenv.pyz -> /srv/bootstrap/virtualenv/public/virtualenv.pyz

Thus there would remain some breakage to URLs to make it work, we could leave the existing 2.6, 2.7,...3.4, 3.5 symlinks, or perhaps implement redirects (that wouldn't be followed by default necessarily)... so new (and existing) versioned would be accessible via https://bootstrap.pypa.io/pip/$VERSION/get-pip.py

[ewdurbin]

ah, missed your "error out" step, that works too :-D

[ewdurbin]

See python/psf-salt#202... I think that's all that's needed to get you moving (towards the error out versions)

[pradyunsg]

Hmm... That'll mean we're transitioning folks from https;//bootstrap.pypa.io/2.7/get-pip.py to https;//bootstrap.pypa.io/pip/2.7/get-pip.py?

I'm on board, let's do this! :)

[ewdurbin]

Correct, otherwise we'd have to compose the other projects into pip's public folder. 👎

I prefer what virtualenv has done. It has clearly let them move at their own pace (haven't heard a peep since that was set up), and propose we retain https://bootstrap.pypa.io/get-pip.py at least, similar to virtualenv.pyz

[pradyunsg]

Yea, let's keep the root get-pip.py; and move everything else to the nicer location.

[pradyunsg]

OH OH. If you could also update the psf-salt to source bootstrap.pypa.io/get-pip.py from the public/ folder (w/ a symlink?), that'd be perfect! I can just change over the scripts-to-be-removed-here in one go; to the error-out variants.

[ewdurbin]

OK, that's all going out now.

[pradyunsg]

Hurray! Thanks @ewdurbin! ^>^

[ewdurbin]

Current status of the bootstrap.pypa.io webroot:

ernestd@web0:/srv/bootstrap/www$ sudo tree -a
.
├── 2.6
│   └── get-pip.py -> /srv/bootstrap/pip/2.6/get-pip.py
├── 2.7
│   └── get-pip.py -> /srv/bootstrap/pip/2.7/get-pip.py
├── 3.2
│   └── get-pip.py -> /srv/bootstrap/pip/3.2/get-pip.py
├── 3.3
│   └── get-pip.py -> /srv/bootstrap/pip/3.3/get-pip.py
├── 3.4
│   └── get-pip.py -> /srv/bootstrap/pip/3.4/get-pip.py
├── 3.5
│   └── get-pip.py -> /srv/bootstrap/pip/3.5/get-pip.py
├── bootstrap-buildout.py -> /srv/bootstrap/buildout/bootstrap/bootstrap.py
├── ez_setup.py -> /srv/bootstrap/setuptools/ez_setup.py
├── get-pip.py -> /srv/bootstrap/pip/public/get-pip.py
├── pip -> /srv/bootstrap/pip/public
├── virtualenv -> /srv/bootstrap/virtualenv/public
└── virtualenv.pyz -> /srv/bootstrap/virtualenv/public/virtualenv.pyz

8 directories, 10 files

[pradyunsg]

Nice!

[pradyunsg]

Oh oh, can we also move from master to main on this repo? (we can track this separately too, if that helps)

[ewdurbin]

sure: python/psf-salt#203

[ewdurbin]

If you update the default branch to main, all will now work.

[pradyunsg]

Awesome! Done! :)

[pradyunsg]

Following #61 (comment), #104 does the "error out" part. In ~4 weeks, we'd need to (1) make MOVED_SCRIPTS empty, and (2) revert b44c524.

@pypa/pip-committers if you think we should wait longer, holler here!

[aharnly-amplify]

This change broke our production deploys (for legacy but still maintained systems that pull down pip on converge). I know you said "We don't have a good mechanism to make more gradual changes here", but … a good mechanism would be helpful.

[pradyunsg]

If it had already existed, sure. It didn't though, and as I said in the message, we probably won't need to do something like this anytime soon.

[pradyunsg]

I modified get-pip.py in this repository's root, 5a43acd since that file also has to be removed.

python/psf-salt#202 uses /public/get-pip.py from this repository for /get-pip.py anyway. :)

[pradyunsg]

wait 4 weeks

We've done 12.

drop the non-public scripts from this repository

I'mma do this today. :)

[solarkraft]

Did you guys drop Python 3.1 in the process? I'm told ERROR: This script does not work on Python 3.1 The minimum supported Python version is 3.7. Please use https://bootstrap.pypa.io/pip/3.1/get-pip.py instead., but there's nothing there.

[pradyunsg]

We've never had a get-pip.py for Python 3.1.

https://github.com/pypa/get-pip/commits/main/3.1 vs https://github.com/pypa/get-pip/commits/main/3.3

And I don't think we're going to be adding one for a version of Python that went EoL in 2014.

[solarkraft]

Thanks for clearing it up! Fair! I'm messing around with very old software. I created #149 in the hope of avoiding such a misunderstanding with other users in the future.