From 13f6258e05cb834ecd71bfdbe4fb01acbe3841b3 Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Tue, 6 Jul 2021 15:09:53 -0400 Subject: [PATCH 01/14] bpo-43950: Add documentation for PEP-657 Co-authored-by: Pablo Galindo Co-authored-by: Batuhan Taskaya Co-authored-by: Ammar Askar --- Doc/c-api/code.rst | 8 +++ Doc/reference/datamodel.rst | 30 ++++++++++ Doc/whatsnew/3.11.rst | 59 +++++++++++++++++++ .../2021-07-06-15-27-11.bpo-43950.LhL2-q.rst | 6 ++ 4 files changed, 103 insertions(+) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 6e18a4225e8f43..7d1a1003871eb3 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -59,3 +59,11 @@ bound into a function. For efficiently iterating over the line numbers in a code object, use `the API described in PEP 626 `_. + +.. c:function:: int PyCode_Addr2Location(PyObject *co, int byte_offset, int *start_line, int *start_column, int *end_line, int *end_column) + + Sets the passed ``int`` pointers to the source code line and column numbers + for the instruction at ``byte_offset``. Sets the value to ``0`` when + information is not available for any particular element. + + Returns ``1`` if the function succeeds. diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index b6dae810d781b7..66a311d9bfcc72 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1015,6 +1015,36 @@ Internal types If a code object represents a function, the first item in :attr:`co_consts` is the documentation string of the function, or ``None`` if undefined. + .. method:: codeobject.co_positions() + + Returns an iterable over the source code positions of each bytecode + instruction in the code object. + + The iterator returns tuples containing the ``(start_line, end_line, + start_column, end_column)``. The *i-th* tuple corresponds to the + position of the source code that compiled to the *i-th* instruction. + Column information is 0-indexed utf-8 byte offsets on the given source + line. + + This positional information can be missing. A non-exhaustive lists of + cases where this may happen: + + - Running the interpreter with ``-X no_debug_ranges``. + - Loading a pyc file compiled while using ``-X no_debug_ranges``. + - Position tuples corresponding to artificial instructions. + - Line and column numbers that can't be represented due to + implementation specific limitations. + + When this occurs, some or all of the tuple elements can be + :const:`None`. + + .. versionadded:: 3.11 + + .. note:: + Storing positions may increase the disk usage of compiled Python + files and memory usage of the interpreter. This feature can be + disabled with the ``-X no_debug_ranges`` command line flag. + .. _frame-objects: Frame objects diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index bfadda1a881f2e..15dfe2fd046b51 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -70,6 +70,65 @@ Summary -- Release highlights New Features ============ +.. _whatsnew311-pep657: + +Enhanced error locations in tracebacks +-------------------------------------- + +When printing tracebacks, the interpreter will now point to the exact expression +that caused the error instead of just the line. For example: + +.. code-block:: python + + Traceback (most recent call last): + File "distance.py", line 11, in + print(manhattan_distance(p1, p2)) + ^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "distance.py", line 6, in manhattan_distance + return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y) + ^^^^^^^^^ + AttributeError: 'NoneType' object has no attribute 'x' + +Previous versions of the interpreter would point to just the line making it +ambiguous which object was ``None``. These enhanced errors can also be helpful +when dealing with deeply nested dictionary objects and multiple function calls: + +.. code-block:: python + + Traceback (most recent call last): + File "query.py", line 37, in + magic_arithmetic('foo') + ^^^^^^^^^^^^^^^^^^^^^^^ + File "query.py", line 18, in magic_arithmetic + return add_counts(x) / 25 + ^^^^^^^^^^^^^ + File "query.py", line 24, in add_counts + return 25 + query_user(user1) + query_user(user2) + ^^^^^^^^^^^^^^^^^ + File "query.py", line 32, in query_user + return 1 + query_count(db, response['a']['b']['c']['user'], retry=True) + ^^^^^^^^^^^^^^^^^^^^^^^ + TypeError: 'NoneType' object is not subscriptable + +See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya +and Ammar Askar in :issue:`43950`.) + +Column information for code +~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The information used by the enhanced traceback feature is made available as a +general API that can be used to correlate bytecode instructions with source +code. This information can be retrieved using: + +- The :meth:`codeobject.co_positions` method in Python. +- The :c:func:`PyCode_Addr2Location` function in the C-API. + +The ``-X no_debug_ranges`` option and the environment variable +:envvar:`PYTHONNODEBUGRANGES` can be used to disable this feature. + +See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya +and Ammar Askar in :issue:`43950`.) + Other Language Changes diff --git a/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst b/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst new file mode 100644 index 00000000000000..dde5399626b7ef --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2021-07-06-15-27-11.bpo-43950.LhL2-q.rst @@ -0,0 +1,6 @@ +Code objects can now provide the column information for instructions when +available. This is levaraged during traceback printing to show the +expressions responsible for errors. + +Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar as part of +:pep:`657`. From fcc9b9300b2714cf5cdc58ed356f82669281f66d Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Sat, 10 Jul 2021 16:18:32 -0400 Subject: [PATCH 02/14] Update for new subscript specialized traceback --- Doc/whatsnew/3.11.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 15dfe2fd046b51..f729ec16580b23 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -107,7 +107,7 @@ when dealing with deeply nested dictionary objects and multiple function calls: ^^^^^^^^^^^^^^^^^ File "query.py", line 32, in query_user return 1 + query_count(db, response['a']['b']['c']['user'], retry=True) - ^^^^^^^^^^^^^^^^^^^^^^^ + ~~~~~~~~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya From 577d0fa86ca04249109e0a8d5656f3bd2a614bba Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Mon, 12 Jul 2021 16:37:08 -0400 Subject: [PATCH 03/14] Add example of binop specialization --- Doc/whatsnew/3.11.rst | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index f729ec16580b23..b4d39d75e3c0d4 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -91,7 +91,7 @@ that caused the error instead of just the line. For example: Previous versions of the interpreter would point to just the line making it ambiguous which object was ``None``. These enhanced errors can also be helpful -when dealing with deeply nested dictionary objects and multiple function calls: +when dealing with deeply nested dictionary objects and multiple function calls, .. code-block:: python @@ -110,6 +110,16 @@ when dealing with deeply nested dictionary objects and multiple function calls: ~~~~~~~~~~~~~~~~~~^^^^^ TypeError: 'NoneType' object is not subscriptable +as well as complex arithmetic expressions: + +.. code-block:: python + + Traceback (most recent call last): + File "calculation.py", line 54, in + result = (x / y / z) * (a / b / c) + ~~~~~~^~~ + ZeroDivisionError: division by zero + See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in :issue:`43950`.) From 32e1a54789943d977fcbda4986d9137fc576ca2a Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Mon, 12 Jul 2021 19:41:30 -0400 Subject: [PATCH 04/14] Add return value for failure Co-authored-by: Pablo Galindo Salgado --- Doc/c-api/code.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/c-api/code.rst b/Doc/c-api/code.rst index 7d1a1003871eb3..2b0cdf43243405 100644 --- a/Doc/c-api/code.rst +++ b/Doc/c-api/code.rst @@ -66,4 +66,4 @@ bound into a function. for the instruction at ``byte_offset``. Sets the value to ``0`` when information is not available for any particular element. - Returns ``1`` if the function succeeds. + Returns ``1`` if the function succeeds and 0 otherwise. From 835d08622522bd1d33a643637d7275aa4aaa2852 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 00:43:50 +0100 Subject: [PATCH 05/14] Update Doc/whatsnew/3.11.rst --- Doc/whatsnew/3.11.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b4d39d75e3c0d4..0a1ba996e33f24 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -123,8 +123,8 @@ as well as complex arithmetic expressions: See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in :issue:`43950`.) -Column information for code -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +Column information for code objects +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The information used by the enhanced traceback feature is made available as a general API that can be used to correlate bytecode instructions with source From 27e5ba5a0ba1da0e0d325a6c7084a4f0d8748d72 Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Mon, 12 Jul 2021 19:49:48 -0400 Subject: [PATCH 06/14] Add note about memory usage in whatsnew --- Doc/whatsnew/3.11.rst | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 0a1ba996e33f24..b426a300b799b8 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -123,6 +123,12 @@ as well as complex arithmetic expressions: See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in :issue:`43950`.) +.. note:: + + This feature requires storing column positions in code objects which may + increase disk usage of compiled Python files or interpreter memory usage. + It can be disabled with the ``-X no_debug_ranges`` command line flag. + Column information for code objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 23d962d7c63613fa4da139e8ceb42eec27d5d2b8 Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Mon, 12 Jul 2021 19:54:36 -0400 Subject: [PATCH 07/14] Reword note with Pablo's suggestion Co-authored-by: Pablo Galindo Salgado --- Doc/whatsnew/3.11.rst | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index b426a300b799b8..f2bf0c5c6d5580 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -126,8 +126,9 @@ and Ammar Askar in :issue:`43950`.) .. note:: This feature requires storing column positions in code objects which may - increase disk usage of compiled Python files or interpreter memory usage. - It can be disabled with the ``-X no_debug_ranges`` command line flag. + result in a small increase of disk usage of compiled Python files or interpreter memory usage. + To avoid storing the extra information or to deactivate printing the extra traceback information, + the feature can be fully disabled with the ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` environment variable. Column information for code objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From f69904fd5af2393e1180a5f3b104c184b84e4514 Mon Sep 17 00:00:00 2001 From: Ammar Askar Date: Mon, 12 Jul 2021 19:57:44 -0400 Subject: [PATCH 08/14] Copy note to codeobject docs --- Doc/reference/datamodel.rst | 9 ++++++--- Doc/whatsnew/3.11.rst | 9 +++++---- 2 files changed, 11 insertions(+), 7 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 66a311d9bfcc72..53c6bfc83a5e87 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1041,9 +1041,12 @@ Internal types .. versionadded:: 3.11 .. note:: - Storing positions may increase the disk usage of compiled Python - files and memory usage of the interpreter. This feature can be - disabled with the ``-X no_debug_ranges`` command line flag. + This feature requires storing column positions in code objects which may + result in a small increase of disk usage of compiled Python files or + interpreter memory usage. To avoid storing the extra information and/or + deactivate printing the extra traceback information, the + ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` + environment variable can be used. .. _frame-objects: diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index f2bf0c5c6d5580..3c9e23b586a466 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -124,11 +124,12 @@ See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya and Ammar Askar in :issue:`43950`.) .. note:: - This feature requires storing column positions in code objects which may - result in a small increase of disk usage of compiled Python files or interpreter memory usage. - To avoid storing the extra information or to deactivate printing the extra traceback information, - the feature can be fully disabled with the ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` environment variable. + result in a small increase of disk usage of compiled Python files or + interpreter memory usage. To avoid storing the extra information and/or + deactivate printing the extra traceback information, the + ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` + environment variable can be used. Column information for code objects ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ From 637869003cf0e1b2e337a32d113471a0832d1a01 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:18:16 +0100 Subject: [PATCH 09/14] Update Doc/whatsnew/3.11.rst --- Doc/whatsnew/3.11.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index 3c9e23b586a466..abde2178169418 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -128,7 +128,7 @@ and Ammar Askar in :issue:`43950`.) result in a small increase of disk usage of compiled Python files or interpreter memory usage. To avoid storing the extra information and/or deactivate printing the extra traceback information, the - ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` + ``-X no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` environment variable can be used. Column information for code objects From 997bad4505d900787a86d68d8aa10811c7ca292b Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:18:21 +0100 Subject: [PATCH 10/14] Update Doc/reference/datamodel.rst --- Doc/reference/datamodel.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 53c6bfc83a5e87..1aaae8f7b0a0e9 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1045,7 +1045,7 @@ Internal types result in a small increase of disk usage of compiled Python files or interpreter memory usage. To avoid storing the extra information and/or deactivate printing the extra traceback information, the - ``-X no_debug_ranges`` command line flag or the ``PYTHONNODEBUGRANGES`` + ``-X no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` environment variable can be used. .. _frame-objects: From 2ba721d23e50d36399e4f74d12bf1809cf1e0ee9 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:20:31 +0100 Subject: [PATCH 11/14] Update Doc/reference/datamodel.rst --- Doc/reference/datamodel.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index 1aaae8f7b0a0e9..e4c8eef22a7f7f 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1045,7 +1045,7 @@ Internal types result in a small increase of disk usage of compiled Python files or interpreter memory usage. To avoid storing the extra information and/or deactivate printing the extra traceback information, the - ``-X no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` + :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` environment variable can be used. .. _frame-objects: From c65c283e10f8ef6ff2b69d2a94e6d0bb0340fd5c Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:20:55 +0100 Subject: [PATCH 12/14] Update Doc/reference/datamodel.rst --- Doc/reference/datamodel.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Doc/reference/datamodel.rst b/Doc/reference/datamodel.rst index e4c8eef22a7f7f..bb0b7e059f132d 100644 --- a/Doc/reference/datamodel.rst +++ b/Doc/reference/datamodel.rst @@ -1029,8 +1029,8 @@ Internal types This positional information can be missing. A non-exhaustive lists of cases where this may happen: - - Running the interpreter with ``-X no_debug_ranges``. - - Loading a pyc file compiled while using ``-X no_debug_ranges``. + - Running the interpreter with :option:`-X` ``no_debug_ranges``. + - Loading a pyc file compiled while using :option:`-X` ``no_debug_ranges``. - Position tuples corresponding to artificial instructions. - Line and column numbers that can't be represented due to implementation specific limitations. From 0ca45fcf6d675e9561a7ebee9f72a0f1145bb905 Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:21:25 +0100 Subject: [PATCH 13/14] Update Doc/whatsnew/3.11.rst --- Doc/whatsnew/3.11.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index abde2178169418..f719cf4ff628ce 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -128,7 +128,7 @@ and Ammar Askar in :issue:`43950`.) result in a small increase of disk usage of compiled Python files or interpreter memory usage. To avoid storing the extra information and/or deactivate printing the extra traceback information, the - ``-X no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` + :option:`-X` ``no_debug_ranges`` command line flag or the :envvar:`PYTHONNODEBUGRANGES` environment variable can be used. Column information for code objects From 95048a5efb1072f4f9724358831599fbe90b097f Mon Sep 17 00:00:00 2001 From: Pablo Galindo Salgado Date: Tue, 13 Jul 2021 01:21:47 +0100 Subject: [PATCH 14/14] Update Doc/whatsnew/3.11.rst --- Doc/whatsnew/3.11.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/Doc/whatsnew/3.11.rst b/Doc/whatsnew/3.11.rst index f719cf4ff628ce..57e9667c15776d 100644 --- a/Doc/whatsnew/3.11.rst +++ b/Doc/whatsnew/3.11.rst @@ -141,7 +141,7 @@ code. This information can be retrieved using: - The :meth:`codeobject.co_positions` method in Python. - The :c:func:`PyCode_Addr2Location` function in the C-API. -The ``-X no_debug_ranges`` option and the environment variable +The :option:`-X` ``no_debug_ranges`` option and the environment variable :envvar:`PYTHONNODEBUGRANGES` can be used to disable this feature. See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya