Word converter: line comments in ANTLR are handled badly

[jskeet]

Sample input:

reference_type
    : class_type
    | interface_type
    | array_type
    // This is a line comment
    | delegate_type
    | 'dynamic'
    ;

Output in Word:

reference_type
	: class_type
	| interface_type
	| array_type         // This is a line comment
	| delegate_type
	| 'dynamic'
	;

The comment that was intended to be about delegate_type is instead associated with array_type.

Additionally, we can't handle comments before the first colon (as Nigel has discovered with his string interpolation PR).

We should probably check that we don't actually use line comments in any ANTLR bits, as otherwise we're already broken (rather than just "not supporting something that would be nice").

[RexJaeschke]

We now use line comments in quite a few places as a result of my merging the unsafe grammar into the main grammar. For example, in 15.2.2.1 General, we have:

class_modifier
    : 'new'
...
    | 'static'
    | unsafe_modifier   // unsafe code support
    ;

[jskeet]

Sorry, I should be clear - "end of line comments as a suffix" are fine; it's "comment that is a line on its own" that causes a problem.

[Nigel-Ecma]

TL;DR There is quite a bit of usage of such line comments. There are quite a few uses of such comments: grep -E '^\s*//[^/]' *.md produces 185 hits – this number of occurrences will be a little higher after the grammar extraction tool has been run to update grammar.md and more are copied there from other sections. Of these 164 are currently auto-generated ones in grammar.md where the grammar extraction tool prefixes each extract grammar chunk with a comment, e.g.: input_element : Whitespace | Comment | token ; // Source: §7.3.2 Line terminators New_Line : New_Line_Character | '\u000D\u000A' // carriage return, line feed ; We get line comments to introduce sections in the grammar (i.e. just as they’re used in above in grammar.md but hand written ones), and in folded comments, e.g.: fragment Escaped_Identifier : '@' Basic_Identifier // includes keywords and contextual keywords prefixed by '@', // see note below ; from lexical-structure.md.

…

On 17/11/2021, at 1:03 am, Jon Skeet ***@***.***> wrote: Sample input: reference_type : class_type | interface_type | array_type // This is a line comment | delegate_type | 'dynamic' ; Output in Word: reference_type : class_type | interface_type | array_type // This is a line comment | delegate_type | 'dynamic' ; The comment that was intended to be about delegate_type is instead associated with array_type. Additionally, we can't handle comments before the first colon (as Nigel has discovered with his string interpolation PR). We should probably check that we don't actually use line comments in any ANTLR bits, as otherwise we're already broken (rather than just "not supporting something that would be nice"). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#394>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSYVWYZA36H7SY7GXGKPIDUMJB7TANCNFSM5IECVGCA>. Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[jskeet]

Thanks for checking that, Nigel. It looks like line comments before a production are okay - it's only within a production that it's a problem.
I'll have a look to see how many we've got within productions - your regex finds various comments in C# code, of course.

[Nigel-Ecma]

Yeah, I thought about writing one which matched the comments and the open/close fences so it was a (little) more obvious which ones were ANTLR and which C# – but I haven’t had my first coffee of the day yet ;-)

…

On 17/11/2021, at 8:25 am, Jon Skeet ***@***.***> wrote: your regex finds various comments in C# code, of course

[jskeet]

As noted in #434, I think the best we can do before C# 6 is make sure we can find all the places where this happens, either via grep etc or via warnings in the converter, and manually fix them afterwards. I don't think there will be that many.