Check that new heap types cannot be created uninitialised: add Py_TPFLAGS_DISALLOW_INSTANTIATION type flag #88082
Comments
|
I'm creating this issue and marking it as a deferred blocker to make sure we don't forget to check this (adding tests) before the release. Check this message from Serhiy regarding heap typed concerns: |
pablogsal
added
deferred-blocker
3.10
|
Serhiy's comment for reference: Static type with tp_new = NULL does not have public constructor, but heap type inherits constructor from base class. As result, it allows to create instances without proper initialization, that can lead to crash. It was fixed for few standard heap types in bpo-23815, then reintroduced, then fixed again in bpo-42694. But it should be checked for every type without constructor. |
|
From Objects/typeobject.c: |
|
The explaining comment was added in f884b74. The code itself is much older. It was originally added in 6d6c1a3, then All types except static types which are direct descendants of object inherit tp_new from a base class. We need to check which static types had tp_new = NULL before they were converted to heap types and set it to NULL manually after type creation. |
I did a quick git grep. Results pasted into the list over at https://discuss.python.org/t/list-of-built-in-types-converted-to-heap-types/8403. |
|
It is incomplete. For example arrayiterator did have tp_new = NULL (in other words, PyArrayIter_Type.tp_new was not set to non-NULL value). In 3.9: >>> import array
>>> type(iter(array.array('I')))()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: cannot create 'arrayiterator' instancesIn 3.10: >>> import array
>>> type(iter(array.array('I')))()
<array.arrayiterator object at 0x7f02f88db5c0> |
Yes, I know. I'm working on completing it manually. Some of the static structs were only partially initialised (most stop at tp_members). The rest of the struct (including tp_new) would then be initialised to zero. |
|
Thank you for your work Erlend. |
|
Serhiy, do you think this should be a release blocker for the beta? |
Anytime! I've updated the list now. There may still be errors, but I think it's pretty accurate now. |
|
Here's a plaintext version:
|
|
Is there any way we can fix this in Objects/typeobject.c, or do we actually have to patch every single type manually? |
That would be probably a bad idea since typeobject.c is supposed to be generic. We would be inverting the responsibility concerns to the base. |
|
I afraid that changing the corresponding code in Objects/typeobject.c will break existing user code. For now, it is safer to patch every single type manually. Later we can design a general solution. |
|
I am marking this as a release blocker, giving that this is probably going to involve a big change. |
|
Is it worth it adding tests for this? I'm thinking a generic version of msg391910 (maybe Lib/test/test_uninitialised_heap_types.py) coupled with a dataset. |
|
I have had this workaround in _hashopenssl.c for a while. Can I get rid of the function with "{Py_tp_new, NULL}" or is there a more generic way to accomplish the same goal? /* {Py_tp_new, NULL} doesn't block __new__ */
static PyObject *
_disabled_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
PyErr_Format(PyExc_TypeError,
"cannot create '%.100s' instances", _PyType_Name(type));
return NULL;
} |
|
test_uninitialised_heap_types.py will need to import unrelited modules (some of which can be platform depending). And more modules will be added as more types be converted. I think it is better to add tests for different modules in corresponding module test files. They are pretty trivial, for example: def test_new_tcl_obj(self):
self.assertRaises(TypeError, _tkinter.Tcl_Obj) |
|
You're right, that would become messy. Complementing existing test suites is a better approach. |
|
Christian:
Unfortunately not. The workaround in 993e88c does the trick though. |
|
The problem is that the curses module is using a function that requires to be built in the core: #ifdef Py_BUILD_CORE
extern int _PyStructSequence_InitType(
PyTypeObject *type,
PyStructSequence_Desc *desc,
unsigned long tp_flags);
#endif |
|
The build errors were in the CI, but because Python doesn't complain if it cannot build a module, then we never catched this. I have opened #25768 to fix it to unblock the release, since the curses module is broken currently. |
|
|
Gotcha, will try to do that as soon as possible |
|
I have transformed https://discuss.python.org/t/list-of-built-in-types-converted-to-heap-types/8403 into a wiki. Tell me if you need some alterations of something is missing. |
|
Erlend added test.support.check_disallow_instantiation() function in bpo-43988 to write tests for immutable types. |
|
Please, check also the discusion happening here: https://mail.python.org/archives/list/[email protected]/thread/FHFI7QKWNHAVWVFTCHJGTYD3ZFVEUXDD/ |
|
FYI, bpo-44087 added the Py_TPFLAGS_DISALLOW_INSTANTIATION flag to sqlite3.Statement. (Side note: I find the issue title a little bit confusing.) |
|
In Python 3.11, 41 types are declared explicitly with the
|
|
Can we close this issue? Or is there a remaining task? |
3.9 is still affected; we should fix those types first. |
erlend-aasland
added
3.9
erlend-aasland
changed the title
erlend-aasland
changed the title
|
FYI: There are only two bug-fix releases left for 3.9. |
I'm against backporting the new type flag, but we can try to set explicitly tp_set to NULL *after* calling PyType_Ready(). |
|
Victor:
3.9 has now entered security-only phase, so I guess that ship has sailed. Suggesting to close this. |
erlend-aasland
added
the
pending
|
Right, I closed the issue. |
erlend-aasland
removed
the
pending
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: