Make TokenSet faster

[matklad]

We have a TokenSet abstraction over here: https://github.com/matklad/libsyntax2/blob/c8cf1d8cdac48f48caf9505bd5dc20dd2b962317/src/parser/event_parser/parser.rs#L70-L92

It is a set of SyntaxKinds, which is used during parsing for error recovery: the idea is to define TokenSets for things like FIRST(item) and then, when an error occurs, skip tokens until a token from this set is encountered.

• move TokenSet to a separate file
• use [u64; 4] to represet a bit-set of tokens
• add macros to declare and union const TokenSet

[Plasticcaz]

Not sure if I'm biting off more than I can chew here, but I'd be willing to give this a go. I'm forking this and giving it my best shot!

[matklad]

Looks like most of it was already fixed: https://github.com/matklad/libsyntax2/blob/751562d2f77a7bc3eeeffbf8e53c9f3515fa9653/crates/libsyntax2/src/parser_api.rs#L7-L32.

It would still be nice to move this to a separate file though!

[Plasticcaz]

ah, okay. That should be fairly straight forward.

[Plasticcaz]

Should we close this issue, or is there something else to do?