Hash algorithm transition plan

[mnm678]

dstufft

In the future if/when an attack against the current hash, we'll need a plan to transition to a new hash (which I think we need documented anyways, even with 2 hashes because the weakness could affect both of them).

[lukpueh]

This might be as simple as having the repository code changing the hash algorithm key in the HASHES dictionary in tuf metadata from e.g. "shaNOW-CONSIDER-WEAK" to "shaMOMENTARLY-STRONG", and to henceforth use that algorithm to hash metadata and/or target files?

If the server code does not yet support that algorithm, it needs to be updated.
If the client code does not yet support it, it should fail with an "unsupported hash algorithm error" and recommend to the user to update the client.

To me, two questions remain:

• Should old metadata be re-generated with the new hash algorithm?

  I lean towards yes. In the worst case a migration from shaNOW-CONSIDER-WEAK to shaMOMENTARLY-STRONG is as critical as a key rotation after a compromise, in which case we also recommend renewing all metadata. And we only need to renew non-root metadata there. So no offline key handling. Still, the client has to re-download things. But I think that’s fair.

• How does the client update itself, if it does not support the new hash algorithm?

  Simple answer: Out of band. However, to avoid that, support for new algorithms should be pushed to clients before migrating the metadata.

UPDATE on 28/12/19: Adding answers to my questions.

[lukpueh]

Btw and IIUC, Donald's comment only regarded the hash algorithms used to hash other metadata files and target files, do we also need to discuss keyid hash algorithm migration here?

There's already a half-lively discussion about the scope and algorithms for keyids, and the keyid_hash_algorithms feature, which is available in python-tuf but not mentioned in the tuf specification. See theupdateframework/python-tuf#848.

[trishankatdatadog]

@lukpueh I agree, this is not as straightforward as it sounds like to fix

[brainwane]

Here is a rough plan for transitioning hashes:

0. get consensus that we need to turn off support for an old hash algorithm, by rough consensus of PyPI maintainers, announced to https://discuss.python.org/c/packaging
1. email out an announcement to https://mail.python.org/mailman3/lists/pypi-announce.python.org/ and put it in various other announcement media (like conference talks, Twitter, documentation, etc.)
2. put a deprecation notice in Warehouse's API output for people uploading/downloading stuff having to do with the old hash
   2a. assuming we have a package preview feature https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Package_preview_feature_for_PyPI , use that to start warning uploaders in the browser UI
3. look at analytics and see how the curve is going
4. do another announcement
5. stop supporting the old algorithm and only support the new algorithm(s)

[mnm678]

As part of the transition plan, would it make sense to have some amount of time where both algorithms are supported? Maybe we can add a note about how clients can chose the hash algorithm they will use during that transition time (the most recent supported one)

[brainwane]

Right! I had been assuming that we'd STARTED the transition plan with both algorithms being supported.

Also, I'm remembering now:

This PEP does not prescribe how package managers, such as pip, should be adapted to install or update projects from PyPI with TUF metadata.

That's a job for the TUF docs, and then detailed specification regarding the upload tooling side is going to be in PEP 480. PEP 458 is just going to be about Warehouse.

So actually a fuller plan goes like:

0. Implement new algorithm in Warehouse, and have APIs include support for old + new.
1. Announce new feature. Advise official download clients (pip and bandersnatch, let's say) and official upload tooling (probably twine, setuptools, distutils) to implement support for new hashing algorithm.
2. Once key clients have support for new hash algorithm, start defaulting to the new one.
3. get consensus that we need to turn off support for an old hash algorithm, by rough consensus of PyPI maintainers, announced to https://discuss.python.org/c/packaging
4. email out an announcement to https://mail.python.org/mailman3/lists/pypi-announce.python.org/ and put it in various other announcement media (like conference talks, Twitter, documentation, etc.)
5. Put a deprecation notice in Warehouse's API output for people uploading/downloading stuff having to do with the old hash. Assuming we have a package preview feature https://wiki.python.org/psf/Fundable%20Packaging%20Improvements#Package_preview_feature_for_PyPI , use that to start warning uploaders in the browser UI
6. Look at analytics and see how the curve is going
7. Make another announcement
8. Remove Warehouse support for the old algorithm and only support the new algorithm(s).

[lukpueh]

#76 is a first stab of getting the gist of the discussion here into the pep.

Feel free to throw rocks at it. Above all, @brainwane, let me know if you think I should add more of the details you provided (i.e. which channels to use for communication, how to assess if the adoption of the new algorithm is sufficient, etc.).

My rationale for keeping it that brief was that client and uploader workflows are not actually in the scope of PEP 458.

[brainwane]

@lukpueh I think your reasoning is good and I'm fine with the draft you wrote. Thanks.

[lukpueh]

Thanks for the review, @brainwane.

[brainwane]

Since the related PR has been merged, can we now close this?

[mnm678]

Yes, resolved via python#1253