Expanding macro gives wrong result when trying to "specialize" metavariables

[andylizi]

Consider the following macro:

macro_rules! what_number {
    ($num:literal) => {
        what_number!(@inner $num)
    };
    (@inner 42) => {
        "the answer to everything"
    };
    (@inner $num:literal) => {
        stringify!($num)
    };
}

fn main() {
    let result = what_number!(42);
    println!("{result}");
}

The compiler expands what_number!(42) to "42", but RA shows "the answer to everything".

See here for an explanation of why "42" should be the correct result.

---

rust-analyzer version: rust-analyzer version: 0.0.0 (427061d 2022-06-19)

rustc version: rustc 1.63.0-nightly (ca122c7eb 2022-06-13)

[lowr]

rustc handles this by wrapping macro-expanded nodes in special Interpolated token, which we apparently don't have. Even rustc wants to get rid of it (see the comment on Interpolated), so I don't think we want to implement it as-is, and I'm not really sure what it'd take to implement this behavior without it.

[Veykril]

I imagine to solve this we'd want to add a special kind of delimiter to our syntax tree that signals interpolation (opaqueness), (since we have to re-parse intermediate macro expansions in our architecture). That would've been my immediate thought here at least.

What is the concept rustc wants to make use instead of Interpolated tokens though, is there an issue tracking that? 🤔

Somewhat related #11521 (the problem there is empty captures via the vis fragment type)

[lowr]

I imagine to solve this we'd want to add a special kind of delimiter to our syntax tree that signals interpolation (opaqueness)

Hmm yeah, this sounds reasonable and doable, I might go ahead and experiment with this approach.

What is the concept rustc wants to make use instead of Interpolated tokens though, is there an issue tracking that? 🤔

I'm really curious about this too, but there's none afaik. Looking at rust-lang/rust#96546 that added that comment:

When parser librarification was still in plans, one of the alternatives was to switch rustc_parse to the proc_macro-style token model. (link)

[Veykril]

From the looks of that (assuming I understood that right), they would like to store them as tokenstreams instead of nodes. So us adding a special delimiter for this seems like a decent idea then.

Also note, those delimiters cannot be empty tokens (we'd need to give them some weird character not used by rust) as empty tokens break rowan assumptions.

[lowr]

After some more digging, I ended up at #10623 - is it explaining how "special delimiter" nodes don't really work without much complication?

[Veykril]

Hmm right, none groupings are actually this ... To me having a special set of delimiters still seems like the proper way to handle this, given how we handle macro expansions opposed to rustc though ...