exec of list comprehension fails on NameError

[sdeibel]

BPO	13557
Nosy	@rhettinger, @terryjreedy, @amauryfa, @merwok, @voidspace, @Trundle, @florentx, @durban
Files	execlistcomp.py: Illustrates the bug exec-eval-clarification.diff: Attempt at clarifying the docs

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-07-08.21:53:49.751>
created_at = <Date 2011-12-08.20:25:54.033>
labels = ['easy', 'type-bug', 'docs']
title = 'exec of list comprehension fails on NameError'
updated_at = <Date 2012-07-08.21:53:49.749>
user = 'https://bugs.python.org/sdeibel'

bugs.python.org fields:

activity = <Date 2012-07-08.21:53:49.749>
actor = 'python-dev'
assignee = 'docs@python'
closed = True
closed_date = <Date 2012-07-08.21:53:49.751>
closer = 'python-dev'
components = ['Documentation']
creation = <Date 2011-12-08.20:25:54.033>
creator = 'sdeibel'
dependencies = []
files = ['23883', '23937']
hgrepos = []
issue_num = 13557
keywords = ['patch', 'easy']
message_count = 9.0
messages = ['149050', '149096', '149100', '149168', '149357', '154174', '154180', '164873', '165039']
nosy_count = 15.0
nosy_names = ['rhettinger', 'terry.reedy', 'sdeibel', 'amaury.forgeotdarc', 'eric.araujo', 'michael.foord', 'Trundle', 'jonathan.hartley', 'flox', 'daniel.urban', 'docs@python', 'westley.martinez', 'python-dev', 'mjs0', 'josmiley']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue13557'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[sdeibel]

Calling exec() on code that includes a list comprehension that references a defined local variable x fails incorrectly on "NameError: global name 'x' not defined".

[amauryfa]

This is expected and documented: http://docs.python.org/py3k/reference/executionmodel.html#interaction-with-dynamic-features
"Free variables are not resolved in the nearest enclosing namespace, but in the global namespace.", a free variable being a variable "used in a code block but not defined there".
And yes, a list comprehension defines a code block.

Try using exec(code, locals()). It's a good habit anyway to always pass a namespace to exec().

[sdeibel]

Ah, thanks, there it is... I thought this must be dealt with somewhere but couldn't find it. Maybe should add something to the 'exec' statement docs http://docs.python.org/py3k/library/functions.html#exec to reference this (from a usability-of-docs standpoint).

BTW, my code already sends the namespaces to exec (from current stack frame; it's part of a debugger) and probably would break other cases to alter this.

Well, anyway, sorry for the invalid bug report...

[merwok]

Would you like to make a doc patch?

[sdeibel]

Here's a patch to the docs that notes the issue and refers the reader to the relevant execution model docs page. I have also attempted to clarify the "interaction with dynamic features" section of the execution model page.

[terryjreedy]

Issues like this, about exec, have come up multiple times. I just closed bpo-14049 as a duplicate of this, and listed there some other issues. So I think that the doc for exec (and execfile in 2.7) could be better still. I would like to see something like the following added, whether instead of or in addition to the current proposal -- perhaps after "If provided, locals can be any mapping object." (quote from the 3.2 exec entry)

"At module level, globals and locals are the same dictionary. If one passes two separate objects as globals and locals, the code will be executed as if it were embedded in a class definition."

To me, this is friendlier and less intimidating than 'free variable' and the Execution model doc chapter. It summarizes the essential problem with such exec calls.

To illustrate, the following gives the same NameError. and for the same reason, as exec(code,{},{}), where code is the quoted unindented version with the class line deleted.

class x:
    x = 1
    def incx():
        return x+1
    print(incx())

[merwok]

Stefan: This fell off my radar, sorry I haven’t reviewed your patch yet.

Terry: +1

[florentx]

Issue bpo-11796 marked as duplicate of this one.

However the issue described in bpo-11796 does not involve exec/execfile.
It is about scopes for list comprehension like this one.

Another doc patch should probably be written to cover the case described in bpo-11796 too.

[python-dev]

New changeset ab22ffa6fb2e by Terry Jan Reedy in branch '2.7':
Issue bpo-13557: Clarify effect of giving two different namespaces to exec or
http://hg.python.org/cpython/rev/ab22ffa6fb2e

New changeset ea670d71a36d by Terry Jan Reedy in branch '3.2':
Issue bpo-13557: Clarify effect of giving two different namespaces to exec or
http://hg.python.org/cpython/rev/ea670d71a36d

New changeset b47ae7a9e685 by Terry Jan Reedy in branch 'default':
Merge 3.2 closes bpo-13557
http://hg.python.org/cpython/rev/b47ae7a9e685

[erlkonig]

Present in Python 3.10 but not in 3.12

#!/usr/bin/python3.10

import sys

code = '''
avoid = 'wrench'
for tool in [t for t in ('hammer', 'wrench', 'scythe')
             if True                     # w/o this line, t is undefined instead
             and t != avoid]:            # avoid is undefined
   print(tool)
'''

def main():
    exec(code)    # move this outside of main() and it works

sys.exit(0 == main())

#---eof

Output (in its default above):

Traceback (most recent call last):
  File "/home/erlkonig/tmp/./py3broken", line 17, in <module>
    sys.exit(0 == main())
  File "/home/erlkonig/tmp/./py3broken", line 15, in main
    exec(code)    # move this outside of main() and it works
  File "<string>", line 3, in <module>
  File "<string>", line 5, in <listcomp>
NameError: name 'avoid' is not defined

Expected output (seen if the exec() is outside of main(), for example):

hammer
scythe