Using multiple PIP indexes on the same hostname with different credentials does not work

[roman-kouzmenko]

Description

I have a need to simultaneously access two PIP indexes from the same hostname (pkgs.dev.azure.com) but using different credentials.

When configuring it like this:

PIP_INDEX_URL=https://build:password1@pkgs.dev.azure.com/feed1
PIP_EXTRA_INDEX_URL=https://build:password2@pkgs.dev.azure.com/feed2

pip seems to try credentials for feed2 for both feed1 and feed2 failing my builds.

I've worked around this for now by setting the same credentials for both feeds.

Expected behavior

feed1 credentials are used with feed1 and feed2 credentials are used with feed2

pip version

21.1.3

Python version

3.9

OS

linux

How to Reproduce

1. Create two Azure feeds in different organizations, for example pkgs.dev.azure.com/org1/_packaging/org-feed/pypi/simple and pkgs.dev.azure.com/org2/_packaging/org-feed/pypi/simple
2. Upload package1 to feed1, package2 to feed2
3. Generate different personal access tokens PAT1 and PAT2 for the two feeds
4. Set environment variables ``IP_INDEX_URL=https://build:PAT1@pkgs.dev.azure.com/org1/_packaging/org-feed/pypi/simple` and PIP_EXTRA_INDEX_URL=https://build:PAT2@pkgs.dev.azure.com/org2/_packaging/org-feed/pypi/simple
5. Run pip install package1 package2

Output

pip interactively prompts for username breaking the build instead of installing the two packages.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pradyunsg]

I think this is expected behaviour. I suggest reaching out to Azure support, asking them to explore solutions for this on their end.

Our authentication mechanisms have the requirement/assumption baked in that each domain will only have a single username-password pair (or auth token) associated with it. A PR documenting this would be welcome.

21.1.3

Can you confirm that the behaviour is same with the current release of pip?

[roman-kouzmenko]

Just tested and confirm that the behaviour is the same on the current release. Let me also raise a PR to document this.

I will expore the Azure support suggestion too.

It would be nice if it worked as it is possible to pass different authentication to a same domain and it's natural to expect it to work.

It took me around an hour to understand why my pipeline was not working as this was all hidden behind something like this in Azure pipelines that automatically populates the PIP_INDEX_URL and PIP_EXTRA_INDEX_URL environment variables:

- task: PipAuthenticate@1
  displayName: 'Authenticate to feeds'
  inputs:
      pythonDownloadServiceConnections: feed1, feed2

[potiuk]

I believe it's something that shoud be fixed on the pip side as Azure's scheme is fully compliant with RFC7617 2.2.

Reusing Credentials

Given the absolute URI ([RFC3986], Section 4.3) of an authenticated
request, the authentication scope of that request is obtained by
removing all characters after the last slash ("/") character of the
path component ("hier_part"; see [RFC3986], Section 3). A client
SHOULD assume that resources identified by URIs with a prefix-match
of the authentication scope are also within the protection space
specified by the realm value of that authenticated request.

[pradyunsg]

In which case, a PR fixing this would be welcome. :)

[potiuk]

In which case, a PR fixing this would be welcome. :)

Actually I've planned to take a look at adding it :)

[roman-kouzmenko]

In which case, a PR fixing this would be welcome. :)

Actually I've planned to take a look at adding it :)

I looked a bit at the code, I think the fix should be simple by replacing netloc with the "root repository URL" when we cache credentials over here:

pip/src/pip/_internal/network/auth.py

Line 202 in ec8edbf

self.passwords[netloc] = (username, password)

What I am not clear about however is how does one define a "root repository URL" exactly.

PEP503 seems to define it to stop after simple.

However, in the Azure feed case, my index URL looks like pkgs.dev.azure.com/ORG_NAME/_packaging/FEED_NAME/pypi/simple, but i see pip install trying to resolve URLs (and fetching credentials for them) such as pkgs.dev.azure.com/ORG_NAME/_packaging/FEED_NAME_UUID/pypi/download.

Two issues with this:

1. The root seems to be at pypi which should be fine to implement.
2. The Azure feed name gets replaced by its UUID in subsequent calls. Would these be redirects or some HATEOAS features of how the repository API works?

[potiuk]

I looked a bit at the code, I think the fix should be simple by replacing netloc with the "root repository URL" when we cache credentials over here:

pip/src/pip/_internal/network/auth.py

Line 202 in ec8edbf

self.passwords[netloc] = (username, password)

Yep. looks like the right place.

What I am not clear about however is how does one define a "root repository URL" exactly.

PEP503 seems to define it to stop after simple.

I believe PEP503 does not define the "root" repository to have "simple". The "simple is just an example (this is how PyPI indesx define it) but it can be arbitrary path (also one including '/' - that's how I read it at least). However it does define that there must be a "project" as the last part of the path or how long the "root" URL is.

However, in the Azure feed case, my index URL looks like pkgs.dev.azure.com/ORG_NAME/_packaging/FEED_NAME/pypi/simple, but i see pip install trying to resolve URLs (and fetching credentials for them) such as pkgs.dev.azure.com/ORG_NAME/_packaging/FEED_NAME_UUID/pypi/download.

I think that should not really matter. From https://datatracker.ietf.org/doc/html/rfc7617#section-2.2 - the "security" context can also match the prefix - not the whole "URL" including the path (also if you look further - it does not decide what to do in case of overlapping contexts - it leaves it to the decision of the client).

So I think what should happen here, the authentication information should be kept in an an array with (prefix, authinfo) tuples, not a dictionary. and the check should not find the exact match, but it should find the longest match from all registered prefixes.