Skip to content

Fix internal error on list/dict comprehension with walrus operator in global scope #9062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Aug 4, 2020
Merged

Fix internal error on list/dict comprehension with walrus operator in global scope #9062

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
e66891f
Add tests for dicts using assignment expression
dhood Jun 28, 2020
c0f633f
Duplicate walrus dict/list tests to global scope (passing)
dhood Jun 28, 2020
822813d
Duplicate walrus comprehension tests to global scope (now failing)
dhood Jun 28, 2020
ed9ba4f
Use global symbol table if 1st non-comprehension scope
dhood Jun 28, 2020
a68bd68
Confirm the symbol table retrieved is not None
dhood Jul 5, 2020
8585a4f
Add tests for assignment expression in class scope
dhood Jul 5, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion mypy/semanal.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4542,9 +4542,18 @@ def current_symbol_table(self, escape_comprehensions: bool = False) -> SymbolTab
if self.is_func_scope():
assert self.locals[-1] is not None
if escape_comprehensions:
assert len(self.locals) == len(self.is_comprehension_stack)
# Retrieve the symbol table from the enclosing non-comprehension scope.
for i, is_comprehension in enumerate(reversed(self.is_comprehension_stack)):
if not is_comprehension:
names = self.locals[-1 - i]
if i == len(self.locals) - 1: # The last iteration.
# The caller of the comprehension is in the global space.
names = self.globals
else:
names_candidate = self.locals[-1 - i]
assert names_candidate is not None, \
"Escaping comprehension from invalid scope"
names = names_candidate
break
else:
assert False, "Should have at least one non-comprehension scope"
Expand Down
56 changes: 56 additions & 0 deletions test-data/unit/check-python38.test
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,27 @@ if a := 2:
while b := "x":
reveal_type(b) # N: Revealed type is 'builtins.str'

l = [y2 := 1, y2 + 2, y2 + 3]
reveal_type(y2) # N: Revealed type is 'builtins.int'
reveal_type(l) # N: Revealed type is 'builtins.list[builtins.int*]'

filtered_data = [y3 for x in l if (y3 := a) is not None]
reveal_type(filtered_data) # N: Revealed type is 'builtins.list[builtins.int*]'
reveal_type(y3) # N: Revealed type is 'builtins.int'

d = {'a': (a2 := 1), 'b': a2 + 1, 'c': a2 + 2}
reveal_type(d) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(a2) # N: Revealed type is 'builtins.int'

d2 = {(prefix := 'key_') + 'a': (start_val := 1), prefix + 'b': start_val + 1, prefix + 'c': start_val + 2}
reveal_type(d2) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(prefix) # N: Revealed type is 'builtins.str'
reveal_type(start_val) # N: Revealed type is 'builtins.int'

filtered_dict = {k: new_v for k, v in [('a', 1), ('b', 2), ('c', 3)] if (new_v := v + 1) == 2}
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If reviewers are wondering why I didn't just use d.items() here instead of the list of tuples, it's because it's causing error: "Dict[str, int]" has no attribute "items".
I couldn't figure out why that was happening, I tried different flags and casting and whatnot but something in the test environment must be different... in a regular python script filtered_dict = {k: new_v for k, v in d.items() if (new_v := v + 1) == 2} can have its type revealed without such an error. Open to suggestions if you have ideas what's causing that!

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is almost certainly because the test fixture you're using happens not to define dict.items. What you're doing here is fine.

reveal_type(filtered_dict) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(new_v) # N: Revealed type is 'builtins.int'

def f(x: int = (c := 4)) -> int:
if a := 2:
reveal_type(a) # N: Revealed type is 'builtins.int'
Expand All @@ -218,6 +239,19 @@ def f(x: int = (c := 4)) -> int:
reveal_type(filtered_data) # N: Revealed type is 'builtins.list[builtins.int*]'
reveal_type(y3) # N: Revealed type is 'builtins.int'

d = {'a': (a2 := 1), 'b': a2 + 1, 'c': a2 + 2}
reveal_type(d) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(a2) # N: Revealed type is 'builtins.int'

d2 = {(prefix := 'key_') + 'a': (start_val := 1), prefix + 'b': start_val + 1, prefix + 'c': start_val + 2}
reveal_type(d2) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(prefix) # N: Revealed type is 'builtins.str'
reveal_type(start_val) # N: Revealed type is 'builtins.int'

filtered_dict = {k: new_v for k, v in [('a', 1), ('b', 2), ('c', 3)] if (new_v := v + 1) == 2}
reveal_type(filtered_dict) # N: Revealed type is 'builtins.dict[builtins.str*, builtins.int*]'
reveal_type(new_v) # N: Revealed type is 'builtins.int'

# https://www.python.org/dev/peps/pep-0572/#exceptional-cases
(y4 := 3)
reveal_type(y4) # N: Revealed type is 'builtins.int'
Expand Down Expand Up @@ -304,6 +338,28 @@ def check_narrow(x: Optional[int], s: List[int]) -> None:
if isinstance((y := x), int):
reveal_type(y) # N: Revealed type is 'builtins.int'

class AssignmentExpressionsClass:
x = (y := 1) + (z := 2)
reveal_type(z) # N: Revealed type is 'builtins.int'

l = [x2 := 1, 2, 3]
reveal_type(x2) # N: Revealed type is 'builtins.int'

def __init__(self) -> None:
reveal_type(self.z) # N: Revealed type is 'builtins.int'

l = [z2 := 1, z2 + 2, z2 + 3]
reveal_type(z2) # N: Revealed type is 'builtins.int'
reveal_type(l) # N: Revealed type is 'builtins.list[builtins.int*]'

filtered_data = [z3 for x in l if (z3 := 1) is not None]
reveal_type(filtered_data) # N: Revealed type is 'builtins.list[builtins.int*]'
reveal_type(z3) # N: Revealed type is 'builtins.int'

# Assignment expressions from inside the class should not escape the class scope.
reveal_type(x2) # E: Name 'x2' is not defined # N: Revealed type is 'Any'
reveal_type(z2) # E: Name 'z2' is not defined # N: Revealed type is 'Any'

[builtins fixtures/isinstancelist.pyi]

[case testWalrusPartialTypes]
Expand Down