PEP 689: Semi-stable C API tier

.github/CODEOWNERS

@@ -569,6 +569,7 @@ pep-0685.rst  @brettcannon
 pep-0686.rst  @methane
 pep-0687.rst  @encukou
 pep-0688.rst  @jellezijlstra
+pep-0689.rst  @encukou
 # ...
 # pep-0754.txt
 # ...

pep-0689.rst

@@ -0,0 +1,222 @@
+PEP: 689
+Title: Semi-stable C API tier
+Author: Petr Viktorin <[email protected]>
+Status: Draft
+Type: Standards Track
+Content-Type: text/x-rst
+Requires: 523
+Created: 22-Apr-2022
+Python-Version: 3.11
+
+
+Abstract
+========
+
+Some functions and types of the C-API are designated *semi-stable*,
+meaning that they will not change in patch (bugfix/security) releases,
+but may change between minor releases (e.g. between 3.11 and 3.12) without
+deprecation warnings.
+
+
+Motivation & Rationale
+======================
+
+The Python C-API is currently divided into `three tiers <https://devguide.python.org/c-api/>`__:
+
+- Limited API, with high compatibility expectations
+- Public API, which follows the :pep:`backwards compatibility policy
+  <387>`, and requires deprecation warnings before changes
+- Private (internal) API, which can change at any time.
+
+We need a tier between Public and Private to accommodate tools like
+advanced debuggers and JIT compilers, which need access to CPython
+internals but assume the C-API they use does not change in patch releases.
+
+
+Setting Stability Expectations
+------------------------------
+
+Currently, there are no guarantees for the internal API -- that is, anything
+that requires ``Py_BUILD_CORE`` or is named with a leading underscore.
+This API can change without warning at any time, and code that uses it
+is pinned to a specific build of Python.
+
+However, in practice, even the internal API usually happens to be stable
+in patch releases:
+
+- Some CPython core developers take this as an an unwritten rule.
+- Patch releases only contain bugfixes, which are unlikely to
+  change the API.
+
+Semi-stable API will make the stability expectations more explicit.
+
+It will also hopefully encourage existing users of the private API to
+reach out to python-dev, so we can expose, standardize and test an API
+for some of their use cases.
+
+
+Reserving underscores for Private API
+-------------------------------------
+
+:pep:`523` introduced functions for use by debuggers and JIT compilers,
+which are stable only across minor releases.
+The functions names have leading underscores to suggest their limited
+stability.
+
+However, leading underscores usually mark *fully private* API.
+CPython developers familiar with the “underscore means internal”
+convention are unlikely to check if underscored functions they are
+changing are documented and used outside CPython itself.
+
+This proposal brings us a bit closer to reserving underscores
+only for truly private, unstable, hands-off API.
+
+
+Warning about API that is changed often
+---------------------------------------
+
+The ``PyCode_New()`` family is an example of functions that are
+documented as unstable, and also often change in practice.
+
+Moving it to the semi-stable tier will make its status obvious even
+to people who don't read the docs carefully enough, and will make it
+hard to use accidentally.
+
+
+Changes during the Beta period
+------------------------------
+
+Since the API itself can change continuously up until Beta 1 (feature freeze)
+of a minor version, major users of this API are unlikely to test
+Alpha releases and provide feedback.
+It is very difficult to determine what needs to be exposed as semi-stable.
+
+Additions to the semi-stable tier will count as *stabilization*,
+and will be allowed up to Release Candidate 1.
+
+
+Specification
+=============
+
+Several functions and types (“APIs”) will be moved to a new *semi-stable* tier.
+
+They will be expected to stay stable across patch releases,
+but may change or be removed without warning in minor releases (3.x.0),
+including Alpha and Beta releases of 3.x.0.
+
+When they change significantly, code that uses them should no longer compile
+(e.g. arguments should be added/removed, or a function should be renamed,
+but the semantic meaning of an argument should not change).
+
+Their definitions will be moved to a new directory, ``Include/semistable/``,
+and will be included from ``Python.h``.
+
+From Python 3.12 on, these APIs will only be usable when the
+``Py_USING_SEMI_STABLE_API`` macro is defined.
+CPython will only define the macro for building CPython itself
+(``Py_BUILD_CORE``).
+
+To make transition to semi-stable API easier,
+in Python 3.11 the APIs will be available without ``Py_USING_SEMI_STABLE_API``
+defined. In this case, using them will generate a deprecation warning on
+compilers that support ``Py_DEPRECATED``.
+
+A similar deprecation period will be used when making more APIs semi-stable
+in the future:
+
+- When moving from public API, the deprecation period should follow Python's
+  backwards compatibility policy (currently, it should last at least
+  two releases).
+- When moving from public API that is documented as unstable,
+  the deprecation period can only last one release.
+- When moving from private API or adding new API, no deprecation period
+  is necessary.
+
+Leading underscores will be removed from the names of the moved APIs.
+The old underscored name of a renamed API will be available (as an alias
+using ``#define``) at least until that API changes.
+
+The semi-stable C-API tier and ``Py_USING_SEMI_STABLE_API`` will be documented,
+and documentation of each semi-stable API will be updated.
+
+
+Adjustments during Beta periods
+-------------------------------
+
+New APIs can be added to the semi-stable tier, and private APIs can be moved
+to it, up to the first release candidate of a new minor version.
+Consensus on the ``capi-sig`` or ``python-dev`` is needed in the Beta period.
+
+In the Beta period, no API may be moved to more private tier, e.g.
+what is public in Beta 1 must stay public until the final release.
+
+
+Initial semi-stable API
+-----------------------
+
+The following API will initially be semi-stable.
+The set may be adjusted for 3.11.
+
+Code object constructors:
+
+- ``PyCode_New()``
+- ``PyCode_NewWithPosOnlyArgs()``
+
+Frame evaluation API (PEP 523):
+
+- ``_PyFrameEvalFunction``
+- ``_PyInterpreterState_GetEvalFrameFunc()``
+- ``_PyInterpreterState_SetEvalFrameFunc()``
+- ``_PyEval_RequestCodeExtraIndex()``
+- ``_PyCode_GetExtra()``
+- ``_PyCode_SetExtra()``
+- ``struct _PyInterpreterFrame`` (as an incomplete, opaque struct)
+- ``_PyFrame_GetFrameObject``
+- ``PyEval_EvalFrameDefault``
+  (new function that calls ``_PyEval_EvalFrameDefault``, but takes
+  ``PyFrameObject`` rather than ``_PyInterpreterFrame``)
+
+(Leading underscores will be removed as mentioned above.)
+
+
+Backwards Compatibility
+=======================
+
+The C API backwards compatibility story will be made clearer.
+
+
+How to Teach This
+=================
+
+The changes affect advanced C programmers, who should consult the
+updated reference documentation, devguide and/or What's New document·.
+
+
+Reference Implementation
+========================
+
+https://github.com/python/cpython/issues/91744
+
+
+Rejected Ideas
+==============
+
+It might be good to add a similar tier in the Python (not C) API,
+e.g. for ``types.CodeType``.
+However, the opt-in mechanism would need to be different (if any).
+This is outside the scope of the PEP.
+
+
+Open Issues
+===========
+
+- “Semi-stable” is not a perfect name.
+
+- The exact set of exposed API may change.
+
+
+Copyright
+=========
+
+This document is placed in the public domain or under the
+CC0-1.0-Universal license, whichever is more permissive.