Fix multiple cases where the parser produced token kinds that didn’t match the token kinds in the syntax node definitions #1436
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ahoppen
force-pushed
the
ahoppen/token-kind-mismatches
branch
2 times, most recently
from
7596ccb to
f028ced
Compare
|
@swift-ci Please test |
bnbarham
reviewed
Mar 24, 2023
Sources/SwiftParser/Types.swift
Outdated
| let (name, _) = self.parseDeclNameRef(flags) | ||
| mutating func parseTypeNameWithGenerics(allowKeywordAsName: Bool) -> (RawTokenSyntax, RawGenericArgumentClauseSyntax?) { | ||
| let name: RawTokenSyntax | ||
| if let identOrSelf = self.consume(if: .identifier, .keyword(.Self), .keyword(.Self)) { |
There was a problem hiding this comment.
Oh, one of them should’ve been lowercase self.
| @@ -551,7 +551,7 @@ public let EXPR_NODES: [Node] = [ | |||
| children: [ | |||
| Child( | |||
| name: "Identifier", | |||
| kind: .token(choices: [.token(tokenKind: "IdentifierToken")]) | |||
| kind: .token(choices: [.token(tokenKind: "IdentifierToken"), .keyword(text: "self"), .keyword(text: "Self"), .keyword(text: "init")]) | |||
There was a problem hiding this comment.
Editor placeholder is always <##> so ... seems weird we'd need this.
There was a problem hiding this comment.
You’re right, we don’t actually need it here.
ahoppen
force-pushed
the
ahoppen/token-kind-mismatches
branch
from
f028ced to
fa14be6
Compare
|
@swift-ci Please test |
|
@swift-ci Please test Windows |
bnbarham
approved these changes
Mar 27, 2023
…match the token kinds in the syntax node definitions The main problem here was that we were accepting `self` and `Self` in `expectIdentifier` but not handling that in the syntax tree. I went through every call of `expectIdentifier` and checked what the C++ parser does to maintain the behavior. Where we need to accept `self` or `Self` for compatibility reasons, it is now remapped to an identifier.
ahoppen
force-pushed
the
ahoppen/token-kind-mismatches
branch
from
fa14be6 to
aea4442
Compare
|
@swift-ci Please test |
|
@swift-ci Please test Windows |
ahoppen
added a commit
to ahoppen/swift-syntax
that referenced
this pull request
Mar 31, 2023
…matches Fix multiple cases where the parser produced token kinds that didn’t match the token kinds in the syntax node definitions
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The main problem here was that we were accepting
selfandSelfinexpectIdentifierbut not handling that in the syntax tree. I went through every call ofexpectIdentifierand checked what the C++ parser does to maintain the behavior. Where we need to acceptselforSelffor compatibility reasons, it is now remapped to an identifier.In other cases, the syntax node definitions just needed to be updated.
These problems were found by the verification added in #1339.