bpo-43950: Initial base implementation for PEP 657

[pablogsal]

⚠️ This is not the full feature and contains no optimizations for the time being, just the base implementation to avoid merge conflicts in the future ⚠️

All the internal structures, and data representation are subjected to change in future PRs as we add optimizations

This PR is part of PEP 657 and augments the compiler to emit ending
line numbers as well as starting and ending columns from the AST
into compiled code objects. This allows bytecodes to be correlated
to the exact source code ranges that generated them.

This information is made available through the following public APIs:

• The co_positions method on code objects.
• The C API function PyCode_Addr2Location.

Co-authored-by: Pablo Galindo [email protected]
Co-authored-by: Batuhan Taskaya [email protected]
Co-authored-by: Ammar Asker [email protected]

https://bugs.python.org/issue43950

[isidentical]

A couple doctest failures FYI: https://travis-ci.com/github/python/cpython/jobs/520228793

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @pablogsal for commit 0673116a6fa30547c1138fd3f96059412d99a979 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[markshannon]

Instead of adding three additional location fields to each instruction, and continually update the equivalent fields in the scope struct, why not change i_lineno to i_ast_node to hold the AST node with the matching location?

Doesn't PEP 657 require that the locations match those of the AST exactly?

[pablogsal]

Instead of adding three additional location fields to each instruction, and continually update the equivalent fields in the scope struct, why not change i_lineno to i_ast_node to hold the AST node with the matching location?

That would be basically the same because you are modifiying these fields in the compilation and assembler steps. For example:

...
b->b_instr[i].i_lineno = prev_lineno;
...

that means that if we use a i_ast_node pointer all instructions will share the same ast node, so changes to the lineno of one instruction will affect others, which is a big problem. So either we start adding a bunch of macros to get and set to handle the default case a NULL pointer for i_ast_node or we would end copiying the ast node location structure, which is almost exactly what we are doing currently.

Doesn't PEP 657 require that the locations match those of the AST exactly?

No, the PEP just states what is the source of the information, but it doesn't explain how is propagated. So we have some freedom to select from which nodes do we pick the information or to "fix it" if we ever need to.

[pablogsal]

@isidentical By the way, it seems that there are more places where we are setting lineno = -1 that we didn't update to also account for the new fields

[ammaraskar]

@terryjreedy Adding you as a reviewer as requested. You might not be too interested in this part, it's just exposing the lower level offset data but the tests might give you sort of an idea on how they can be used from Python. The follow up PRs will add the traceback.py changes that you might be more interested in.

[bedevere-bot]

🤖 New build scheduled with the buildbot fleet by @isidentical for commit ba64180d3f61ee75b2b624ae2055ec22a2fb27fc 🤖

If you want to schedule another build, you need to add the ":hammer: test-with-buildbots" label again.

[terryjreedy]

If I understand, the only officially visible change is the addition of code.co_positions, which returns an iterable of (start_line, end_line, start_column, end_column) tuples, where column numbers are 1-based. Suggested title after bpo tag: "Add code.co_positions (PEP 657)" or "Add PEP-657 code.co_positions".

I presume a doc change will come. I will review it for readability if pinged.

I read the tests and have two suggestions.

[ammaraskar]

Amended the commit message as per Terry's recommendation and rebased #26958 on top of the latest changes, tests for the traceback changes are still passing.