Compiler: Modernize the js lexer, now utf-8 aware

[hhugo]

In particular, it allows to recognize and emit utf8 identifiers.
This lexer now uses sedlex. The implementation was taken from flow and cleaned to remove unused features.

[hhugo]

@dbuenzli, fully fix #1034

[dbuenzli]

I'm not familiar enough with the code base to go through these changes to approve them and don't know how much non US-ASCII entry points there are in the wild JavaScript world but the fact that needless string munging is avoided is very nice.

[hhugo]

Minification and variable renaming is currently broken with this PR.
The optimisation rely of a free variable analysis, which rely on string equality of idents.

Idents are currently not normalized and even contain escape sequence verbatim.
We could either

• normalize ident
• disable the variable renaming for unicode ident (We would still need to resolve escape sequence)

[dbuenzli]

Did you check what regular JavaScript minifier do ? I suspect they don't do these things.

Note that there are technologies like xml that rely on balancing unicode identifiers and throw normalization out of the window and no-one seems to ever run in the problematic and puzzling cases :-)

[hhugo]

Did you check what regular JavaScript minifier do ? I suspect they don't do these things.

Note that there are technologies like xml that rely on balancing unicode identifiers and throw normalization out of the window and no-one seems to ever run in the problematic and puzzling cases :-)

I had a quick look at terser and flow. I didn't see any unicode normalization. However, they both decode escape sequence in ident.

[vouillon]

According to the ECMAScript specification, one should not normalize but just decode escape sequences (and check that this still results in a valid identifier).

[hhugo]

The last commit decodes escape sequence in ident

[dbuenzli]

Just had a look at the last commit. Don't claim I undersood everything of it but the tests look right :-)