Merge pull request #10054 from pradyunsg/topic/caching

Author: pradyunsg

docs/html/cli/pip_install.rst

@@ -574,62 +574,14 @@ overridden by using ``--cert`` option or by using ``PIP_CERT``,
 Caching
 -------
 
-Starting with v6.0, pip provides an on-by-default cache which functions
-similarly to that of a web browser. While the cache is on by default and is
-designed do the right thing by default you can disable the cache and always
-access PyPI by utilizing the ``--no-cache-dir`` option.
-
-When making any HTTP request pip will first check its local cache to determine
-if it has a suitable response stored for that request which has not expired. If
-it does then it simply returns that response and doesn't make the request.
-
-If it has a response stored, but it has expired, then it will attempt to make a
-conditional request to refresh the cache which will either return an empty
-response telling pip to simply use the cached item (and refresh the expiration
-timer) or it will return a whole new response which pip can then store in the
-cache.
-
-While this cache attempts to minimize network activity, it does not prevent
-network access altogether. If you want a local install solution that
-circumvents accessing PyPI, see :ref:`Installing from local packages`.
-
-The default location for the cache directory depends on the operating system:
-
-Unix
-  :file:`~/.cache/pip` and it respects the ``XDG_CACHE_HOME`` directory.
-macOS
-  :file:`~/Library/Caches/pip`.
-Windows
-  :file:`<CSIDL_LOCAL_APPDATA>\\pip\\Cache`
-
-Run ``pip cache dir`` to show the cache directory and see :ref:`pip cache` to
-inspect and manage pip’s cache.
-
+This is now covered in :doc:`../topics/caching`
 
 .. _`Wheel cache`:
 
 Wheel Cache
 ^^^^^^^^^^^
 
-pip will read from the subdirectory ``wheels`` within the pip cache directory
-and use any packages found there. This is disabled via the same
-``--no-cache-dir`` option that disables the HTTP cache. The internal structure
-of that is not part of the pip API. As of 7.0, pip makes a subdirectory for
-each sdist that wheels are built from and places the resulting wheels inside.
-
-As of version 20.0, pip also caches wheels when building from an immutable Git
-reference (i.e. a commit hash).
-
-pip attempts to choose the best wheels from those built in preference to
-building a new wheel. Note that this means when a package has both optional
-C extensions and builds ``py`` tagged wheels when the C extension can't be built
-that pip will not attempt to build a better wheel for Pythons that would have
-supported it, once any generic wheel is built. To correct this, make sure that
-the wheels are built with Python specific tags - e.g. pp on PyPy.
-
-When no wheels are found for an sdist, pip will attempt to build a wheel
-automatically and insert it into the wheel cache.
-
+This is now covered in :doc:`../topics/caching`
 
 .. _`hash-checking mode`:
 

docs/html/topics/caching.md

@@ -0,0 +1,86 @@
+# Caching
+
+```{versionadded} 6.0
+```
+
+pip provides an on-by-default caching, designed to reduce the amount of time
+spent on duplicate downloads and builds.
+
+## What is cached
+
+### HTTP responses
+
+This cache functions like a web browser cache.
+
+When making any HTTP request, pip will first check its local cache to determine
+if it has a suitable response stored for that request which has not expired. If
+it does then it returns that response and doesn't re-download the content.
+
+If it has a response stored but it has expired, then it will attempt to make a
+conditional request to refresh the cache which will either return an empty
+response telling pip to simply use the cached item (and refresh the expiration
+timer) or it will return a whole new response which pip can then store in the
+cache.
+
+While this cache attempts to minimize network activity, it does not prevent
+network access altogether. If you want a local install solution that
+circumvents accessing PyPI, see {ref}`Installing from local packages`.
+
+### Locally built wheels
+
+pip attempts to use wheels from its local wheel cache whenever possible.
+
+This means that if there is a cached wheel for the same version of a specific
+package name, pip will use that wheel instead of rebuilding the project.
+
+When no wheels are found for a source distribution, pip will attempt to build a
+wheel using the package's build system. If the build is successful, this wheel
+is added to the cache and used in subsequent installs for the same package
+version.
+
+```{versionchanged} 20.0
+pip now caches wheels when building from an immutable Git reference
+(i.e. a commit hash).
+```
+
+## Avoiding caching
+
+pip tries to use its cache whenever possible, and it is designed do the right
+thing by default.
+
+In some cases, pip's caching behaviour can be undesirable. As an example, if you
+have package with optional C extensions, that generates a pure Python wheel
+when the C extension can’t be built, pip will use that cached wheel even when
+you later invoke it from an environment that could have built those optional C
+extensions. This is because pip is seeing a cached wheel for that matches the
+package being built, and pip assumes that the result of building a package from
+a package index is deterministic.
+
+The recommended approach for dealing with these situations is to directly
+install from a source distribution instead of letting pip auto-discover the
+package when it is trying to install. Installing directly from a source
+distribution will make pip build a wheel, regardless of whether there is a
+matching cached wheel. This usually means doing something like:
+
+```{pip-cli}
+$ pip download sampleproject==1.0.0 --no-binary :all:
+$ pip install sampleproject-1.0.0.tar.gz
+```
+
+It is also a good idea to remove the offending cached wheel using the
+{ref}`pip cache` command.
+
+## Cache management
+
+The {ref}`pip cache` command can be used to manage pip's cache.
+
+The exact filesystem structure of pip's cache is considered to be an
+implementation detail and may change between any two versions of pip.
+
+## Disabling caching
+
+pip's caching behaviour is disabled by passing the ``--no-cache-dir`` option.
+
+It is, however, recommended to **NOT** disable pip's caching. Doing so can
+significantly slow down pip (due to repeated operations and package builds)
+and result in significantly more network usage.

docs/html/topics/index.md

@@ -11,4 +11,5 @@ This section of the documentation is currently being fleshed out. See
 :maxdepth: 1
 
 authentication
+caching
 ```