Resolve numerous ANTLR issues

[RexJaeschke]

This PR addresses the issues raised in Issue #230 as well as Issue #37, Proposal 3. Specifically, it

• Replaces all lexer rule alternatives of the form '<...>' with correct content
• Uses semantic predicates to resolve numerous issues, especially w.r.t identifier spelling
• Uses ranges

@BillWagner, to support the semantic predicates, from a validation pov, we'll need to get the following text into the grammar file that we extract/generate, immediately following the current grammar directive in tools/validate-grammar.sh:

echo "grammar CSGrammar;" > $OUTPUT_FILE

@lexer::members {
bool IsLetterCharacter()       { return true; }
bool IsCombiningCharacter()    { return true; }
bool IsDecimalDigitCharacter() { return true; }
bool IsConnectingCharacter()   { return true; }
bool IsFormattingCharacter()   { return true; }
bool IsNotAKeyword()           { return true; }
bool IsNotTrueOrFalse()        { return true; }
}

Note that these are simply place-holder methods to allow validation. Clearly, they are quite incomplete from a lexer pov.

For now, I'm made this a DRAFT PR pending pushback on the two final uses of semantic predicates I proposed in Issue 230.

[BillWagner]

@RexJaeschke I think the best way to proceed is for me to make a PR against this branch that adds the changes to the grammar validator. That way, they are in one place.

Does that make sense, or do you need those changes for other work?

[RexJaeschke]

@BillWagner. OK; go ahead and make the grammar extractor changes a separate PR.

[BillWagner]

@RexJaeschke

I just created #310 with the updates you mention. It's not working correctly, so we need to figure that out.

[RexJaeschke]

@BillWagner, while I don't have a copy of the grammar file you extracted, my guess is these messages are coming from the pseudo-extensions to the grammar from unsafe.md, where an ellipses (...) is used to begin each grammar fragment. If we can get @Nigel-Ecma to review/merge PR ##233, all the ... lines (and their parent rules) will go away, and the grammar left in unsafe.md will all be needed and correct as is and no longer be considered as an add-on. This probably will fix the "missing ;" as well, as that is in a rule that is completely removed by that PR.

And you should change this PR (261) from Draft, and merge it as well, as a final test.

[BillWagner]

@RexJaeschke #310 had a very obvious shell scripting mistake (obvious, at least, as soon as I saw it).

The results of the grammar validator on that PR is the same as this one now. Is that what you expected, or should we be expecting different diagnostics from ANTLR?

[RexJaeschke]

@BillWagner I removed Draft status from this PR and assigned you as a reviewer.

If you want to test further without Nigel having reviewed/merged PR 233 to integrate the unsafe grammar changes, you could omit the inclusion of unsafe.md in your script.

[BillWagner]

This is a big improvement @RexJaeschke

Let's merge it.

You can decide if you want to take #311 into this branch, or wait for those kinds of changes for #233