the f_lineno getter is broken

[xdegaye]

BPO	24565
Nosy	@abalkin, @blueyed, @xdegaye, @serhiy-storchaka
PRs	bpo-24565: f->f_lineno is now -1 when tracing is not set #6233 bpo-24565: f->f_lineno is now -1 when tracing is not set  #12419
Files	f_lineno.patch f_lineno_tests.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2015-07-04.20:36:38.056>
labels = ['interpreter-core', '3.8', 'type-bug', '3.7', '3.9']
title = 'the f_lineno getter is broken'
updated_at = <Date 2019-12-05.11:21:41.373>
user = 'https://github.com/xdegaye'

bugs.python.org fields:

activity = <Date 2019-12-05.11:21:41.373>
actor = 'blueyed'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2015-07-04.20:36:38.056>
creator = 'xdegaye'
dependencies = []
files = ['39865', '39866']
hgrepos = []
issue_num = 24565
keywords = ['patch']
message_count = 8.0
messages = ['246275', '246276', '314237', '314243', '314409', '314410', '339667', '357845']
nosy_count = 4.0
nosy_names = ['belopolsky', 'blueyed', 'xdegaye', 'serhiy.storchaka']
pr_nums = ['6233', '12419']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24565'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[xdegaye]

The last paragraph of Objects/lnotab_notes.txt explains that the f_lineno member of the PyFrameObject structure is needed to store the line number of the last "line" tracing event so that this value may be used as the line number of the "return" event instead of the (sometimes confusing) value computed from f_lasti. The f_lineno getter must then return the value of f->f_lineno (instead of the value computed from f->f_lasti) when tracing is set. The current implementation translates "tracing is set" as "the local f_trace trace function is not NULL", this is wrong for the following reasons:

• AFAIK it is not documented anywhere that Python users implementing a trace function must delete the local f_trace functions of all the frames when setting the
  global trace function to None via sys.settrace(None) (bpo-7238).

• Bdb.set_continue() in the bdb module of the std lib seems to know about this and attempts to do it, but fails to delete f_trace from the topmost frame, named botframe (sic) (bpo-16482, bpo-17697).

• It is not obvious how to delete the f_trace of all suspended generators when setting the global trace function to None (bpo-17277).

• _PyTraceback_Add() in Python/traceback.c sets frame->f_lineno, obviously forgetting that it is useless since its f_trace is NULL.

This patch changes the semantics of f_lineno by stating that f_lineno is invalid when its value is -1. When tracing, the f_lineno of all the frames on the stack is valid. The f_lineno of a suspended generator is always invalid.

[xdegaye]

Uploading the corresponding test cases.

[serhiy-storchaka]

Could you please convert your patches to a PR Xavier?

[xdegaye]

I will work on it shortly.

[xdegaye]

A trace function may also be set in extension modules by PyEval_SetTrace() and it may not use f->f_trace. This is another reason why f->f_trace cannot be used in PyFrame_GetLineNumber() to know when f->f_lineno is valid.

[xdegaye]

Added PR 6233.
One of the GitHub bots failed to link this issue with PR 6233. Maybe this is related to the fact that connections to bpo are currently failing intermitently with the message:

An error occurred during a connection to bugs.python.org. Peer’s certificate has an invalid signature. Error code: SEC_ERROR_BAD_SIGNATURE

[xdegaye]

Fixed a bug in the implementation of PR 12419 while running the coverage.py test suite: f_lineno must be valid upon 'call' trace events. The confusion stems from the reason why until now we have prevented line jumps from 'call' trace events, see below.

Summary:

• Full coverage.py tests OK (including the C tracer).
• Fixes issues bpo-7238, bpo-16482, bpo-17277 and bpo-17697.
• We could now remove the restriction that prevents jumps from a call trace event. The restriction was based on the fact that f->f_trace is NULL until after the first return from call_trampoline() and setting f_lineno from within this first call to call_trampoline() would have had the f_lineno getter still return the value returned by PyCode_Addr2Line(), very confusing for the user !
• A trace function may be set in an extension modules by PyEval_SetTrace() and the extension module may not use f->f_trace and still make jumps using f_lineno.
• Fixes _PyTraceback_Add() at

  cpython/Python/traceback.c

  Line 272 in 58721a9

  frame->f_lineno = lineno;

[blueyed]

This is likely covered by existing/linked issues already, but wanted to leave it here nonetheless:

Given t-pdb.py:

import sys


def main():
    sys.stdout.write("main...\n")
    assert 0


if __name__ == "__main__":
    main()

Without the fix from the PR:

% python3.8 -m pdb -c cont t-pdb.py
main...
Traceback (most recent call last):
  File "…/pyenv/3.8.0/lib/python3.8/pdb.py", line 1702, in main
    pdb._runscript(mainpyfile)
  File "…/pyenv/3.8.0/lib/python3.8/pdb.py", line 1571, in _runscript
    self.run(statement)
  File "…/pyenv/3.8.0/lib/python3.8/bdb.py", line 587, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "…/Vcs/cpython/t-pdb.py", line 1, in <module>
    import sys
  File "…/Vcs/cpython/t-pdb.py", line 6, in main
    assert 0
AssertionError
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> …/Vcs/cpython/t-pdb.py(6)main()
-> assert 0
(Pdb)

With the fix:

% /tmp/cpython-bisect/bin/python3.8 -m pdb -c cont t-pdb.py
main...
Traceback (most recent call last):
  File "/tmp/cpython-bisect/lib/python3.8/pdb.py", line 1703, in main
    pdb._runscript(mainpyfile)
  File "/tmp/cpython-bisect/lib/python3.8/pdb.py", line 1572, in _runscript
    self.run(statement)
  File "/tmp/cpython-bisect/lib/python3.8/bdb.py", line 583, in run
    exec(cmd, globals, locals)
  File "<string>", line 1, in <module>
  File "…/Vcs/cpython/t-pdb.py", line 10, in <module>
    main()
  File "…/Vcs/cpython/t-pdb.py", line 6, in main
    assert 0
AssertionError
Uncaught exception. Entering post mortem debugging
Running 'cont' or 'step' will restart the program
> …/Vcs/cpython/t-pdb.py(6)main()
-> assert 0
(Pdb)

As you can see the traceback in the fixed case contains main() correctly,
while it has import sys (the scripts first line) otherwise.

I can only repeat myself to ask for reviewing/merging #12419.

[gaogaotiantian]

It's been a while and the mechanisms were changed a lot in the past couple of versions. There's no usage of frame->f_trace in PyFrame_GetLineNumber anymore. The example given above also worked correctly for main. Close this for now, if someone found a repro, feel free to reopen this.