The examples of "c extern" in the "Doc/includes/newtypes" directory of Python 3.11 and later versions need to be optimized, the memory structure of the __dict__ in the C extern example under Doc/includes/newtypes is outdated.

[aoaoluming]

Documentation

For Python 3.11 and later, refer to:
faster-cpython/ideas#80 （implementation details in CPython source: _PyObject_InitializeDict)
https://www.youtube.com/watch?v=xKk7IXm0XO0
The memory structure of the Python object dict pointer (dict ptr) is optimized:

However, in CPython directory Doc/includes/newtypes, the C extern example that involves adding a __dict__, such as:

from custom2 import Custom
class ItemData2(Custom):                                      // c-extern dict-ptr is old-style memory structure
    def __init__(self):
        self.itemid = 0
        self.item = 0

still uses the old dict ptr structure (prior to Python 3.10).

This involves two issues:

1. The dict ptr structure uses the previous type (as stated above)
2. The dict ptr memory optimization in Python 3.11 and later does not work in the C extern example:

The memory optimization in _PyObject_InitializeDict->init_inline_values is not utilized:

static int
init_inline_values(PyObject *obj, PyTypeObject *tp)
{
    assert(tp->tp_flags & Py_TPFLAGS_HEAPTYPE);
    // assert(type->tp_dictoffset > 0);  -- TODO: Update this assert.
    assert(tp->tp_flags & Py_TPFLAGS_MANAGED_DICT);
    PyDictKeysObject *keys = CACHED_KEYS(tp);
    assert(keys != NULL);
    if (keys->dk_usable > 1) {
        keys->dk_usable--; // Key optimization for memory when creating many objects with __dict__
    }
}

When objects are created in large quantities, the memory per single object can differ by nearly 200 ~ 300bytes 。

detail

memory of ItemData2 = 392 bytes

typedef struct {
    PyObject_HEAD
} CustomObject;   # remove custom2.c CustomObject field: first, last, number

"The memory size allocated for new_values as PyDictValues is determined by arg size 。

arg size default value is 30(when the __dict__ contains fewer than 30 members):

When arg size is 30, the memory size of PyDictValues is 272 bytes:"

total size:

56(object ptr basic) + 64(dict ptr) + 0(cache key ptr, shared memory) + 272(values ptr) = 392 bytes

memory of ItemData = 104 bytes

when the __dict__ contains 2 memebers:

memory of values ptr = prefix_size + size * sizeof(PyObject*) = prefix_size + (dk_entries + dk_usable) * sizeof(PyObject*) 
= 32 + (2 + 0） * 8 = 48 bytes
56(object basic memory) + 0(dict ptr) + 0(cache key, shared) + 48(value ptr)= 104 bytes 

I try to fix it, like:

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CustomObject *self;

    // 直接调用object.tp_new， 注意不要把 args 直接传PyBaseObject_Type.tp_new，`__init__`带参数时会报错
    static PyObject* intern_tuple0 = NULL; 
    if (intern_tuple0 == NULL) 
        intern_tuple0 = PyTuple_New(0);
    self = PyBaseObject_Type.tp_new(type, intern_tuple0, NULL);

    //self = (CustomObject *) type->tp_alloc(type, 0);

[Wulian233]

translate comment:

static PyObject *
Custom_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
    CustomObject *self;

    // Call object.tp_new directly, but be careful not to pass args directly to PyBaseObject_Type.tp_new. If you pass args directly, you will get an error when `__init__` is called with parameters.
    static PyObject* intern_tuple0 = NULL; 
    if (intern_tuple0 == NULL) 
        intern_tuple0 = PyTuple_New(0);
    self = PyBaseObject_Type.tp_new(type, intern_tuple0, NULL);

    //self = (CustomObject *) type->tp_alloc(type, 0);