gh-115528: Update language reference for PEP 646 #121181

mrahtz · 2024-06-30T11:45:35Z

There are two aims here:

Make *args: *Ts valid.
Make starred expression valid in the context of subscription, such that Generic[*a] and list[*a] are valid.

1 is straightforward, but 2 requires a bit more work:

The subscription rule relies on expression_list. But it's not as simple as changing expression_list, because it's also used by yield. For yield, although yield *a, b etc is valid, yield *a is not - the closest is yield *a,.

I also noticed that rules elsewhere that use this group of expression rules aren't reflective of the current grammar. Subscriptions, lists and sets all allow assignment expressions: Generic[x := 1], [x := 1] and {x := 1} all seem to be valid, at least in a REPL loop on Python 3.12.4.

Combining all this, this PR introduces a new rule flexible expression: assignment_expression | starred_expression, and changes expression_list, list_display and set_display to use this new rule. So all that fixes both starred expressions and assignment expressions valid in the contexts mentioned above.

(I've also made this whole group of expression rules less hopefully confusing by a) changing starred_expression to literally just be a single expression rather than potentially a list of expressions, and b) changed star_item to only be a starred expression.)

But for yield, things are different. Assignment expressions are not valid in the context of yield, and we need to express the stipulation that if the yield expression begins with a starred expression, there needs to be a comma. So I've introduced another new rule yield_list, that expresses this requirement.

Issue: PEP 646 grammar changes are not documented #115528

📚 Documentation preview 📚: https://cpython-previews--121181.org.readthedocs.build/

To recap: the objective is to make starred expressions valid in `subscription`, which is used for generics: `Generic[...]`, `list[...]`, etc. What _is_ gramatically valid in such contexts? Seemingly any of the following. (At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.) Generic[x] Generic[*x] Generic[*x, y] Generic[y, *x] Generic[x := 1] Generic[x := 1, y := 2] So introducting flexible_expression: expression | assignment_expression | starred_item end then switching `subscription` to use `flexible_expression` sorts that. But then we need to field `yield` - for which any of the following are apparently valid: yield x yield x, yield x, y yield *x, yield *x, *y Introducing a separate `yield_list` is the simplest way I've been figure out to do this - separating out the special case of `starred_item ,`.

mrahtz · 2024-06-30T12:20:37Z

@JelleZijlstra Sorry this revision took a while - had a very busy few weeks at work, and it took a while to figure this out 😅

JelleZijlstra

Thanks! It's quite tricky to get all these expression variants right.

JelleZijlstra · 2024-06-30T14:25:28Z

Doc/reference/expressions.rst

-   starred_list: `starred_item` ("," `starred_item`)* [","]
-   starred_expression: `expression` | (`starred_item` ",")* [`starred_item`]
-   starred_item: `assignment_expression` | "*" `or_expr`
+   starred_expression: ["*"] `or_expr`


or_expr is valid for what comes after *, but starred_expression now also gets used (via expression_list) in the rule for list_display, and there other expressions are valid, e.g. comparisons. I think this should be:

Suggested change

starred_expression: ["*"] `or_expr`

starred_expression: "*" `or_expr` | `expression`

I think expression is already included in expression_list through assignment_expression - though maybe it would be worth adding the extra | expression here to make it clear?

Ah you're right, I missed that assignment_expression includes expression.

But I think there's still something wrong here: yield a == b, c == d is valid, but a == b is a comparison, and the grammar for yield_list indicates that after the comma we can only have a starred_expression.

Doc/reference/expressions.rst

Co-authored-by: Jelle Zijlstra <[email protected]>

JelleZijlstra

Thanks, this looks right to me now but I'd like someone else to look over the grammar too, since it's been quite tricky to get right.

AlexWaygood

LGTM (though I'm also not an expert here)

miss-islington-app · 2024-09-26T19:35:20Z

Thanks @mrahtz for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

To recap: the objective is to make starred expressions valid in `subscription`, which is used for generics: `Generic[...]`, `list[...]`, etc. What _is_ gramatically valid in such contexts? Seemingly any of the following. (At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.) Generic[x] Generic[*x] Generic[*x, y] Generic[y, *x] Generic[x := 1] Generic[x := 1, y := 2] So introducting flexible_expression: expression | assignment_expression | starred_item end then switching `subscription` to use `flexible_expression` sorts that. But then we need to field `yield` - for which any of the following are apparently valid: yield x yield x, yield x, y yield *x, yield *x, *y Introducing a separate `yield_list` is the simplest way I've been figure out to do this - separating out the special case of `starred_item ,`. (cherry picked from commit 7d3497f) Co-authored-by: Matthew Rahtz <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]>

bedevere-app · 2024-09-26T19:35:32Z

GH-124632 is a backport of this pull request to the 3.13 branch.

bedevere-app · 2024-09-26T19:35:36Z

GH-124633 is a backport of this pull request to the 3.12 branch.

…124633) gh-115528: Update language reference for PEP 646 (GH-121181) To recap: the objective is to make starred expressions valid in `subscription`, which is used for generics: `Generic[...]`, `list[...]`, etc. What _is_ gramatically valid in such contexts? Seemingly any of the following. (At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.) Generic[x] Generic[*x] Generic[*x, y] Generic[y, *x] Generic[x := 1] Generic[x := 1, y := 2] So introducting flexible_expression: expression | assignment_expression | starred_item end then switching `subscription` to use `flexible_expression` sorts that. But then we need to field `yield` - for which any of the following are apparently valid: yield x yield x, yield x, y yield *x, yield *x, *y Introducing a separate `yield_list` is the simplest way I've been figure out to do this - separating out the special case of `starred_item ,`. (cherry picked from commit 7d3497f) Co-authored-by: Matthew Rahtz <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]>

…124632) gh-115528: Update language reference for PEP 646 (GH-121181) To recap: the objective is to make starred expressions valid in `subscription`, which is used for generics: `Generic[...]`, `list[...]`, etc. What _is_ gramatically valid in such contexts? Seemingly any of the following. (At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.) Generic[x] Generic[*x] Generic[*x, y] Generic[y, *x] Generic[x := 1] Generic[x := 1, y := 2] So introducting flexible_expression: expression | assignment_expression | starred_item end then switching `subscription` to use `flexible_expression` sorts that. But then we need to field `yield` - for which any of the following are apparently valid: yield x yield x, yield x, y yield *x, yield *x, *y Introducing a separate `yield_list` is the simplest way I've been figure out to do this - separating out the special case of `starred_item ,`. (cherry picked from commit 7d3497f) Co-authored-by: Matthew Rahtz <[email protected]> Co-authored-by: Jelle Zijlstra <[email protected]>

mrahtz added 5 commits June 30, 2024 12:01

Update language reference for PEP 646

a44de6d

Cleanups

3059bd9

Simplify

e7c8c25

Another fix

79953ce

bedevere-app bot mentioned this pull request Jun 30, 2024

PEP 646 grammar changes are not documented #115528

Closed

bedevere-app bot added docs Documentation in the Doc dir skip news labels Jun 30, 2024

Fix some whitespace changes

b8e2f02

AlexWaygood requested review from JelleZijlstra and AlexWaygood June 30, 2024 11:51

Fix order

6fed4d5

mrahtz marked this pull request as ready for review June 30, 2024 12:20

bedevere-app bot added the awaiting review label Jun 30, 2024

JelleZijlstra added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jun 30, 2024

JelleZijlstra reviewed Jun 30, 2024

View reviewed changes

mrahtz and others added 2 commits June 30, 2024 21:43

Simplify yield_list

e01e592

Co-authored-by: Jelle Zijlstra <[email protected]>

Add expression_list to yield_list

73ddba1

JelleZijlstra approved these changes Jul 3, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels Jul 3, 2024

AlexWaygood approved these changes Sep 26, 2024

View reviewed changes

JelleZijlstra merged commit 7d3497f into python:main Sep 26, 2024
27 checks passed

bedevere-app bot removed the awaiting merge label Sep 26, 2024

bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Sep 26, 2024

bedevere-app bot removed the needs backport to 3.12 only security fixes label Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

gh-115528: Update language reference for PEP 646 #121181

gh-115528: Update language reference for PEP 646 #121181

Uh oh!

mrahtz commented Jun 30, 2024 •

edited

Loading

Uh oh!

mrahtz commented Jun 30, 2024

Uh oh!

JelleZijlstra left a comment

Uh oh!

JelleZijlstra Jun 30, 2024

Uh oh!

mrahtz Jun 30, 2024

Uh oh!

JelleZijlstra Jun 30, 2024

Uh oh!

Uh oh!

JelleZijlstra left a comment

Uh oh!

AlexWaygood left a comment

Uh oh!

Uh oh!

miss-islington-app bot commented Sep 26, 2024

Uh oh!

bedevere-app bot commented Sep 26, 2024

Uh oh!

bedevere-app bot commented Sep 26, 2024

Uh oh!

Uh oh!

	starred_expression: ["*"] `or_expr`
	starred_expression: "*" `or_expr` \| `expression`

Uh oh!

gh-115528: Update language reference for PEP 646 #121181

gh-115528: Update language reference for PEP 646 #121181

Uh oh!

Conversation

mrahtz commented Jun 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mrahtz commented Jun 30, 2024

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

mrahtz Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

JelleZijlstra Jun 30, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

JelleZijlstra left a comment

Choose a reason for hiding this comment

Uh oh!

AlexWaygood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

miss-islington-app bot commented Sep 26, 2024

Uh oh!

bedevere-app bot commented Sep 26, 2024

Uh oh!

bedevere-app bot commented Sep 26, 2024

Uh oh!

Uh oh!

mrahtz commented Jun 30, 2024 •

edited

Loading