[BUG] Missing escape of `char` literal in interpolation

[JohelEGP]

Title: Missing escape of char literal in UFCS in interpolation.

Minimal reproducer (https://cpp2.godbolt.org/z/r1f73EP85):

#include <ranges>
main: () = {
  using namespace std::views;
  s := "".std::string();
  _ = "((_ = s.split('\n'), 0))$";
}

Commands:

cppfront main.cpp2
clang++18 -std=c++23 -stdlib=libc++ -lc++abi -pedantic-errors -Wall -Wextra -Wconversion -Werror=unused-result -Werror=unused-value -Werror=unused-parameter -I . main.cpp

Expected result: static_cast<void>(cpp2::to_string((static_cast<void>(CPP2_UFCS(split, std::move(s), '\n')), 0)));.

Actual result and error: static_cast<void>(cpp2::to_string((static_cast<void>(CPP2_UFCS(split, std::move(s), 'n')), 0)));.

Cpp2 lowered to Cpp1:

//=== Cpp2 type declarations ====================================================


#include "cpp2util.h"

#line 1 "/app/example.cpp2"


//=== Cpp2 type definitions and function declarations ===========================

#line 1 "/app/example.cpp2"
#include <ranges>
auto main() -> int;

//=== Cpp2 function definitions =================================================

#line 1 "/app/example.cpp2"

#line 2 "/app/example.cpp2"
auto main() -> int{
  using namespace std::views;
  auto s {CPP2_UFCS_0(std::string, "")}; 
  static_cast<void>(cpp2::to_string((static_cast<void>(CPP2_UFCS(split, std::move(s), 'n')), 0)));
}

Output:

Program returned: 0

[JohelEGP]

Hmm... Maybe I should be using '\\\n' or a interpolated raw string literal.

[gregmarr]

Looks like it's not disabling string quoting inside the capture. You get the same result with just this:

main: () = {
  _ = "('\n')$";
}

[JohelEGP]

Hmm... Maybe I should be using '\\\n' or a interpolated raw string literal.

Documentation might help to build an intuition on how this should work.
Looking at the grammar, I should even be able to use any expression in an interpolation.
That's because the string doesn't continue until the interpolation ends.
So I think just '\n' should work, without extra \s and without using a raw string literal.

[gregmarr]

Character literals such as \x10 and \u0010 don't parse properly either.

[hsutter]

It looks like this behavior was added in #464 and #481. The block of code that does it is here:

https://github.com/hsutter/cppfront/blame/main/source/lex.h#L430-L436

I'm not sure why we're removing those \ characters. @filipsajdak do you recall?

When I comment out that block, all regression tests pass unchanged, and the code reported in this issue is fixed.

[gregmarr]

It was to fix #461
Looks like something more involved is needed to fix that while not breaking character literals.

[hsutter]

Here's the code from #461, showing the state before #461 was fixed:

  std::cout << "\"\"" << '\n'; // Prints `""`.
  std::cout << $R"("")" << '\n'; // Prints `""`.
  std::cout << "\"(\"\")$\"" << '\n'; // Prints `""`.
  std::cout << "(\"\\\"\\\"\")$" << '\n'; // cppfront asserts

My question is, should those last two have to escape inside the captures? Because what's in the captures should be an expression, and \"\" and \"\\\"\\\"\" aren't valid expressions (you can't just write those in code).

In other words, the capture source expression is not part of the string itself (and therefore shouldn't be escaped) -- its evaluation is part of the string (after being passed through to_string).

So I would instead expect this behavior:

  std::cout << "\"\"" << '\n';            // prints "" (as above)
  std::cout << $R"("")" << '\n';          // prints "" (as above)
  std::cout << "\"(\"\")$\"" << '\n';     // error, \"\" isn't a valid expression
  std::cout << "\"("")$\"" << '\n';       // this should print ""
  std::cout << "(\"\\\"\\\"\")$" << '\n'; // error, \"\\\"\\\"\" isn't a valid expression
  std::cout << "("\\\"\\\"")$" << '\n';   // this should print \\\"\\\"
  std::cout << "("\"\"")$" << '\n';       // this should print \"\"

Before I make that change, does that make sense to this group?

[JohelEGP]

  std::cout << "("\\\"\\\"")$" << '\n';   // this should print \\\"\\\"
  std::cout << "("\"\"")$" << '\n';       // this should print \"\"

The comments look wrong to me.
I think they should be

  std::cout << "("\\\"\\\"")$" << '\n';   // this should print \"\"
  std::cout << "("\"\"")$" << '\n';       // this should print ""

If this is correct, it's the behavior I expect.

[hsutter]

Ah, I think you're right about those last two, thanks.

In particular, these two should be the same and print \"\" as you say:

std::cout << "("\\\"\\\"")$" << '\n';
std::cout << "" << "\\\"\\\"" << "" << '\n';

And these two should be the same and print "" as you say:

std::cout << "("\"\"")$" << '\n';
std::cout << "" << "\"\"" << "" << '\n';

Right?

Let me try a few other cases...

a := 40;
b := 2;
s: std::string = "xyzzy";

std::cout << "(a+b)$"; // should print 42
std::cout << "(s+"plugh")$"; // should print xyzzyplugh
std::cout << "("plugh")$"; // should print plugh
std::cout << "(s+"\"\n")$"; // should print xyzzy" plus a newline
std::cout << "("\"\n")$"; // should print " plus a newline
std::cout << "("")$"; // should print nothing

Do all these look correct now?

[JohelEGP]

Yes.

[gregmarr]

I'm a bit unsure about how unquoted double quotes should interact with interpolation captures.

std::cout << "(" + s + ")$";

Should this be interpreted as the string ( plus a string variable plus the string )$ and be printed as (xyzzy)$ or as a string containing a capture of a string iteral and be printed as + s +?

[hsutter]

OK, I'm just going to fix this immediate Issue.

However, I would still like to get to where " doesn't need to be escaped inside capture-expressions inside string literals. It currently does because in an example like "("")$" that currently lexes as a string literal "(" followed by a string literal of ")$" which gets an error because it doesn't have the matching ( (but even if it wasn't an error, it would be flagged as ungrammatical later since two string literals can't be adjacent).

Of course, you can see above that my wish is to lex "("")$" as a single string literal, but I now see that requires knowing that the second " is not intended to be the closing quote to terminate the string literal... which would require not only lookahead but (I fear possibly) speculative lexing which is a terrible idea. So for now I'm going to leave " inside string interpolations alone, they must still be escaped (but \ and such do not need to be escaped). But this may well be what tips me over the edge to making $ prefix, which would make lexing this simple, align with other languages, and address the steady trickle of requests to make prefix 😉 ... at the expense of making captures in functions require parentheses more often, but maybe it's fine to make captures stand out more (notably, capture-by-pointer such as var&$*++ would become $(var&)*++ which may be actually better and address that current slightly warty syntax).

[hsutter]

I'm a bit unsure about how unquoted double quotes should interact with interpolation captures.

Right you are. I think you just identified the same issue in a racing comment while I was writing the above. 👍