Skip to content

gh-115528: Update language reference for PEP 646 #121181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 26, 2024
Merged

Conversation

mrahtz
Copy link
Contributor

@mrahtz mrahtz commented Jun 30, 2024

There are two aims here:

  1. Make *args: *Ts valid.
  2. Make starred expression valid in the context of subscription, such that Generic[*a] and list[*a] are valid.

1 is straightforward, but 2 requires a bit more work:

The subscription rule relies on expression_list. But it's not as simple as changing expression_list, because it's also used by yield. For yield, although yield *a, b etc is valid, yield *a is not - the closest is yield *a,.

I also noticed that rules elsewhere that use this group of expression rules aren't reflective of the current grammar. Subscriptions, lists and sets all allow assignment expressions: Generic[x := 1], [x := 1] and {x := 1} all seem to be valid, at least in a REPL loop on Python 3.12.4.

Combining all this, this PR introduces a new rule flexible expression: assignment_expression | starred_expression, and changes expression_list, list_display and set_display to use this new rule. So all that fixes both starred expressions and assignment expressions valid in the contexts mentioned above.

(I've also made this whole group of expression rules less hopefully confusing by a) changing starred_expression to literally just be a single expression rather than potentially a list of expressions, and b) changed star_item to only be a starred expression.)

But for yield, things are different. Assignment expressions are not valid in the context of yield, and we need to express the stipulation that if the yield expression begins with a starred expression, there needs to be a comma. So I've introduced another new rule yield_list, that expresses this requirement.


📚 Documentation preview 📚: https://cpython-previews--121181.org.readthedocs.build/

mrahtz added 5 commits June 30, 2024 12:01
To recap: the objective is to make starred expressions valid in `subscription`,
which is used for generics: `Generic[...]`, `list[...]`, etc.

What _is_ gramatically valid in such contexts? Seemingly any of the following.
(At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.)

    Generic[x]
    Generic[*x]
    Generic[*x, y]
    Generic[y, *x]
    Generic[x := 1]
    Generic[x := 1, y := 2]

So introducting

    flexible_expression: expression | assignment_expression | starred_item

end then switching `subscription` to use `flexible_expression` sorts that.

But then we need to field `yield` - for which any of the following are
apparently valid:

    yield x
    yield x,
    yield x, y
    yield *x,
    yield *x, *y

Introducing a separate `yield_list` is the simplest way I've been figure out to
do this - separating out the special case of `starred_item ,`.
@bedevere-app bedevere-app bot added docs Documentation in the Doc dir skip news labels Jun 30, 2024
@mrahtz mrahtz marked this pull request as ready for review June 30, 2024 12:20
@mrahtz
Copy link
Contributor Author

mrahtz commented Jun 30, 2024

@JelleZijlstra Sorry this revision took a while - had a very busy few weeks at work, and it took a while to figure this out 😅

@JelleZijlstra JelleZijlstra added needs backport to 3.12 only security fixes needs backport to 3.13 bugs and security fixes labels Jun 30, 2024
Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It's quite tricky to get all these expression variants right.

starred_list: `starred_item` ("," `starred_item`)* [","]
starred_expression: `expression` | (`starred_item` ",")* [`starred_item`]
starred_item: `assignment_expression` | "*" `or_expr`
starred_expression: ["*"] `or_expr`
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or_expr is valid for what comes after *, but starred_expression now also gets used (via expression_list) in the rule for list_display, and there other expressions are valid, e.g. comparisons. I think this should be:

Suggested change
starred_expression: ["*"] `or_expr`
starred_expression: "*" `or_expr` | `expression`

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think expression is already included in expression_list through assignment_expression - though maybe it would be worth adding the extra | expression here to make it clear?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah you're right, I missed that assignment_expression includes expression.

But I think there's still something wrong here: yield a == b, c == d is valid, but a == b is a comparison, and the grammar for yield_list indicates that after the comma we can only have a starred_expression.

Copy link
Member

@JelleZijlstra JelleZijlstra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this looks right to me now but I'd like someone else to look over the grammar too, since it's been quite tricky to get right.

Copy link
Member

@AlexWaygood AlexWaygood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM (though I'm also not an expert here)

@JelleZijlstra JelleZijlstra merged commit 7d3497f into python:main Sep 26, 2024
27 checks passed
@miss-islington-app
Copy link

Thanks @mrahtz for the PR, and @JelleZijlstra for merging it 🌮🎉.. I'm working now to backport this PR to: 3.12, 3.13.
🐍🍒⛏🤖

miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 26, 2024
To recap: the objective is to make starred expressions valid in `subscription`,
which is used for generics: `Generic[...]`, `list[...]`, etc.

What _is_ gramatically valid in such contexts? Seemingly any of the following.
(At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.)

    Generic[x]
    Generic[*x]
    Generic[*x, y]
    Generic[y, *x]
    Generic[x := 1]
    Generic[x := 1, y := 2]

So introducting

    flexible_expression: expression | assignment_expression | starred_item

end then switching `subscription` to use `flexible_expression` sorts that.

But then we need to field `yield` - for which any of the following are
apparently valid:

    yield x
    yield x,
    yield x, y
    yield *x,
    yield *x, *y

Introducing a separate `yield_list` is the simplest way I've been figure out to
do this - separating out the special case of `starred_item ,`.

(cherry picked from commit 7d3497f)

Co-authored-by: Matthew Rahtz <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
miss-islington pushed a commit to miss-islington/cpython that referenced this pull request Sep 26, 2024
To recap: the objective is to make starred expressions valid in `subscription`,
which is used for generics: `Generic[...]`, `list[...]`, etc.

What _is_ gramatically valid in such contexts? Seemingly any of the following.
(At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.)

    Generic[x]
    Generic[*x]
    Generic[*x, y]
    Generic[y, *x]
    Generic[x := 1]
    Generic[x := 1, y := 2]

So introducting

    flexible_expression: expression | assignment_expression | starred_item

end then switching `subscription` to use `flexible_expression` sorts that.

But then we need to field `yield` - for which any of the following are
apparently valid:

    yield x
    yield x,
    yield x, y
    yield *x,
    yield *x, *y

Introducing a separate `yield_list` is the simplest way I've been figure out to
do this - separating out the special case of `starred_item ,`.

(cherry picked from commit 7d3497f)

Co-authored-by: Matthew Rahtz <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
@bedevere-app
Copy link

bedevere-app bot commented Sep 26, 2024

GH-124632 is a backport of this pull request to the 3.13 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.13 bugs and security fixes label Sep 26, 2024
@bedevere-app
Copy link

bedevere-app bot commented Sep 26, 2024

GH-124633 is a backport of this pull request to the 3.12 branch.

@bedevere-app bedevere-app bot removed the needs backport to 3.12 only security fixes label Sep 26, 2024
AlexWaygood pushed a commit that referenced this pull request Sep 26, 2024
…124633)

gh-115528: Update language reference for PEP 646 (GH-121181)

To recap: the objective is to make starred expressions valid in `subscription`,
which is used for generics: `Generic[...]`, `list[...]`, etc.

What _is_ gramatically valid in such contexts? Seemingly any of the following.
(At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.)

    Generic[x]
    Generic[*x]
    Generic[*x, y]
    Generic[y, *x]
    Generic[x := 1]
    Generic[x := 1, y := 2]

So introducting

    flexible_expression: expression | assignment_expression | starred_item

end then switching `subscription` to use `flexible_expression` sorts that.

But then we need to field `yield` - for which any of the following are
apparently valid:

    yield x
    yield x,
    yield x, y
    yield *x,
    yield *x, *y

Introducing a separate `yield_list` is the simplest way I've been figure out to
do this - separating out the special case of `starred_item ,`.

(cherry picked from commit 7d3497f)

Co-authored-by: Matthew Rahtz <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
Yhg1s pushed a commit that referenced this pull request Sep 26, 2024
…124632)

gh-115528: Update language reference for PEP 646 (GH-121181)

To recap: the objective is to make starred expressions valid in `subscription`,
which is used for generics: `Generic[...]`, `list[...]`, etc.

What _is_ gramatically valid in such contexts? Seemingly any of the following.
(At least, none of the following throw `SyntaxError` in a 3.12.3 REPL.)

    Generic[x]
    Generic[*x]
    Generic[*x, y]
    Generic[y, *x]
    Generic[x := 1]
    Generic[x := 1, y := 2]

So introducting

    flexible_expression: expression | assignment_expression | starred_item

end then switching `subscription` to use `flexible_expression` sorts that.

But then we need to field `yield` - for which any of the following are
apparently valid:

    yield x
    yield x,
    yield x, y
    yield *x,
    yield *x, *y

Introducing a separate `yield_list` is the simplest way I've been figure out to
do this - separating out the special case of `starred_item ,`.

(cherry picked from commit 7d3497f)

Co-authored-by: Matthew Rahtz <[email protected]>
Co-authored-by: Jelle Zijlstra <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Documentation in the Doc dir skip news
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants