Incorrect truncated macro expansion

[jonas-schievink]

This code results in a wrong expansion of the macro, where the resulting enum only has the first specified variant:

macro_rules! enum_with_unknown {
    (
        enum $name:ident {
            $(
              $variant:ident $(,)*
            ),*
        }
    ) => {
        enum $name {
            $(
              $variant,
            )*
        }
    }
}

enum_with_unknown! {
    enum ControlOpcode {
        ConnectionUpdateReq,
        ChannelMapReq,
    }
}

Resulting code according to "Expand macro recursively":

enum ControlOpcode {
  ConnectionUpdateReq,
}

(EDIT: Reduced)

[lnicola]

Triage: this still happens, now with a "leftover tokens" warning.

@edwin0cheng you might want to take a look?

[edwin0cheng]

I know what's the problem now, and I think currently we can't fix it without using another macro parsing algorithm.

Here is a simpler example for the same problem :

macro_rules! foo {
    ($($i:ident $(,)*),*) => {};
}
foo!(a,b);

Our current mbe parser is greedy such that when it matches the middle comma by $(,)*, the outer seperator , could not be matched. Of course we could hack and fix this bug but I think an proper solution is rewritten it as NFA-based or Earley parser.

cc @matklad