[BUG] Diagnose use of C++1 alternative tokens

[JohelEGP]

The use of C++1 alternative tokens lead to invalid Cpp2 or generated C++1 depending on use.

Currently, they're treated as identifiers. So cppfront emits an error if used C++1-correctly (x = y and z;, 3 identifiers in a row is ill-formed), or emits bad C++1 code (x = and;).
-- #304.

---

Minimal reproducer (https://godbolt.org/z/4dT5hre58):

and : int = 0;

Commands:

cppfront x.cpp2
clang++17 -std=c++2b -I $CPPFRONT_INCLUDE_DIR x.cpp

Expected result: Diagnose and as reserved.

Actual result and error:

Generated C++1

//=== Cpp2 type declarations ====================================================


#include "cpp2util.h"


//=== Cpp2 type definitions and function declarations ===========================

#line 1 "x.cpp2"
extern int and;
//=== Cpp2 function definitions =================================================

#line 1 "x.cpp2"
int and {0}; 

x.cpp2:1:15: error: expected unqualified-id
extern int and;
              ^
x.cpp2:1:9: error: expected unqualified-id
int and {0}; 
        ^
2 errors generated.

---

Minimal reproducer (https://godbolt.org/z/WMzzjhGWM):

x : int = y and z;

Commands:

cppfront x.cpp2 

Expected result: Diagnose and as reserved, suggest &&.

Actual result and error:

x.cpp2...
x.cpp2(1,13): error: ill-formed initializer (at 'and')
x.cpp2(1,1): error: unexpected text at end of Cpp2 code section (at 'x')
x.cpp2(1,0): error: parse failed for section starting here

[JohelEGP]

For the first reproducer, I noticed that for GCC, the error message points to the Cpp2 source, so the marker is off.
Clang:

x.cpp2:1:15: error: expected unqualified-id
extern int and;
              ^
x.cpp2:1:9: error: expected unqualified-id
int and {0}; 
        ^
2 errors generated.

GCC:

x.cpp2:1:15: error: expected unqualified-id before ‘;’ token
    1 | and : int = 0;
      |               ^
x.cpp2:1:9: error: expected unqualified-id before ‘{’ token
    1 | and : int = 0;
      |         ^

[gregmarr]

It doesn't diagnose most uses of reserved words as variable names. For example, all of these are passed through to be diagnosed by the C++1 compiler: nullptr, enum, try, catch, explicit, template, signed, short, long, int.

int : int = 0;

//=== Cpp2 type definitions and function declarations ===========================

extern int int;
//=== Cpp2 function definitions =================================================

int int {0}; 

[gregmarr]

FYI, both of those Compiler Explorer links contain the Cpp2 code in a C++ environment, and there are a bunch of other files loaded in various panes.

[JohelEGP]

It'd be OK to just diagnose alternative tokens where their primary token would be valid. Like the second reproducer.

FYI, both of those Compiler Explorer links contain the Cpp2 code in a C++ environment, and there are a bunch of other files loaded in various panes.

Yes, those bring Cpp2 and C++ modules support.

[gregmarr]

It'd be OK to just diagnose alternative tokens where their primary token would be valid. Like the second reproducer.

That would be interesting. I wonder if it would be more work to just support them or to diagnose them. Have you seen anything from Herb that says that he's not planning to support them?

Yes, those bring Cpp2 and C++ modules support.

Okay, but the main file is still marked as C++ instead of CppFront, so I have to copy the contents, change the type to CppFront, and paste the text. I don't know if this is a CE bug with the links, or if something else is weird.

[JohelEGP]

Have you seen anything from Herb that says that he's not planning to support them?

Not quite. #152 (comment) mentions

Re and et al.: I considered this, also for other reasons, but I think the familiarity of && and || is too strong to break without stronger justification. And even if we used and, that still wouldn't be a full solution for this tension since it wouldn't resolve the a**b or a~~b cases.

And the design note linked below that, as #304 puts it:

https://github.com/hsutter/cppfront/wiki/Design-note%3A-Postfix-unary-operators-vs-binary-operators talks about "alternative tokens". It seems they were supported at some point before this repository was published.

Okay, but the main file is still marked as C++ instead of CppFront, so I have to copy the contents, change the type to CppFront, and paste the text. I don't know if this is a CE bug with the links, or if something else is weird.

CE's not bugged. CE's Cpp2-cppfront language option only compiles with cppfront. It has no option to compile the generated C++1. That's why I use C++ and a CMake project to compile both.

[gregmarr]

Not quite.

It would be good to hear from him on whether he thinks they should still be supported.

That's why I use C++ and a CMake project to compile both.

Interesting. BTW, that transpile to C++1 and compile the converted file in CE was one of the things that Herb called out at the end of his keynote as something he'd like someone to help him with. (That was before he knew that the transpile stage had been put up on CE during his talk.)

[JohelEGP]

The CMake module used was added to the wiki: #4 (comment). But unlike CE's support for Cpp2, I have to manually update cpp2util.h, and print the generated C++1 to the build output. See compiler-explorer/compiler-explorer#4074. Still, that wouldn't let me mix Cpp2 with C++ modules or CMake projects (a.k.a. support for multiple source files). What I want is to be able to output to a read-only editor too.

[hsutter]

Re Godbolt Compiler Explorer: Yes, I'd love for there to be a way in CE to compile Cpp2 using cppfront + a choice of C++ compiler as the second compiler (or for now even just one hardwired compiler).

Re alternative spellings for operators:

• Option 1: I could emit an error on them. The downside is that a few of them, such as and, could be a useful name for a local variable.
• Option 2: I could also just rename them. For example, I could automatically rename and to and_ (and similarly for the other 10) to make the names available as regular names in Cpp2 code.

I'm more inclined to Option 2... it's similar to how I already rename new to cpp2_new and this to (*this) so I can make those words mean what I want in Cpp2... from emit(token:

    if (n == "new") {
        printer.print_cpp2("cpp2_new", pos);
    }
    else if (n == "this") {
        printer.print_cpp2("(*this)", pos);
    }

A non-option is to support the alternative spellings with their current meaning, because they violate "don't have two ways to say the same thing"... 😁🦆🏃 (trying an emoji version of the old <gdr> -- but I'm also serious about this) After all, the alternative spellings in <iso646.h> exist for a real reason, but one that AFAIK has not been valid for a long time, namely: keyboards that don't have &, |, !, ^, or ~. Do those keyboards even exist anymore outside a museum? Even antiquated/obsolete US missile silo keyboards seem to have them, though hopefully the folks in those locations aren't doing a lot of programming there. IIRC, IBM systems using EBCDIC were the original motivators for these (as well as for digraphs and trigraphs, for keyboards that don't have { } [ ] # \... and note that trigraphs have already been removed, first from C++ and then from C).

[hsutter]

Option 2 FTW! Thanks.

[JohelEGP]

Thank you.

I see that cb8c1c5 actually appends _ to them. It's common practice to append _ to keywords when one wants an identifier that represents them in some way (i.e., see https://github.com/search?l=C%2B%2B&q=and_+not_&type=code). This could lead to unexpected results (shadowing, accidental use) and friction when adopting Cpp2. How about prefixing cpp2_ instead?