Add a new grammar renderer

[ehuss]

This introduces a new grammar renderer. Instead of trying to write the grammar in markdown/html hybrid, this introduces a new syntax that is parsed by the mdbook-spec plugin. This grammar is then converted into markdown/html hybrid, and also to railroad diagrams.

There are a lot of changes here (and some can be split into separate PRs if desired). A general overview of what to see here:

• Grammar rules are now written inside a code block (instead of a blockquote). The syntax is pretty similar to the old syntax with various small changes. See the docs/grammar.md file for a complete description.
• There is now a summary chapter which shows the entire grammar all on one page.
• The grammar is parsed by mdbook-spec/src/grammar/parser.rs into an internal representation.
• The internal representation is converted to markdown in mdbook-spec/src/grammar/render_markdown.rs, and railroad diagrams in mdbook-spec/src/grammar/render_railroad.rs.
  • The railroad diagrams are generated using the railroad crate.
  • There is a toggle button the show/hide the railroad diagrams. It uses localstorage to keep that state sticky.
• The basic definitions and driver in the mdbook plugin is in mdbook-spec/src/grammar.rs. There are several pieces here:
  • The internal representation.
  • Code to load the grammar from the code blocks inside the chapters.
  • Some validation.
  • Code that will replace the code block with the rendered output.
  • Code for handling the summary chapter.
• All nonterminals are now linked to the rule definition.
• The text may now link to grammar rules by just putting them in brackets like [FunctionParameters]. Link definitions are automatically added to every page.
• Some rules were added or changed to accommodate the new renderer. I think all changes are put into separate commits to help with reviewing.
• Various misc fixes, see the individual commits.

I'd like to thank @lukaslueg for creating the railroad library which made this possible.

Closes #221
Closes #398
Closes #596
Closes #1513
Closes #1677

[lukaslueg]

railroad upstream here.

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

AFAICS there are cases where the grammar diverges from its graphical representation with respect to repeated elements. In the two examples below, the diagram only allows for for at least two consecutive Statement (... two consecutive Expression), while the grammar requires "at least one".

[ia0]

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

From the live demo I can see that *-repeated elements are in theory printed like this:

   .------->-------.
   |               |
->-+-+--[ foo ]--+-+--->-
     |           |
     '-----<-----'

What about doing it like this?

->-+------>------+->-
   |             |
   '-<-[ foo ]-<-'

This uses one less path and can be concatenated with a previous foo for +-repeat:

->-[ foo ]->-+------>------+->-
             |             |
             '-<-[ foo ]-<-'

The main problem is that foo is somehow to be read backwards, which may confuse people at first.

[lukaslueg]

The railroad codebase hasn't seen a lot of love with respect to graphical layout. Suggestions are welcome.

From the live demo I can see that *-repeated elements are in theory printed like this:

   .------->-------.
   |               |
->-+-+--[ foo ]--+-+--->-
     |           |
     '-----<-----'

What about doing it like this?

->-+------>------+->-
   |             |
   '-<-[ foo ]-<-'

This uses one less path and can be concatenated with a previous foo for +-repeat:

->-[ foo ]->-+------>------+->-
             |             |
             '-<-[ foo ]-<-'

The main problem is that foo is somehow to be read backwards, which may confuse people at first.

With respect to *-Elements, both examples (1 and 2) are technically valid. See the Zero or more table-constraints block in the create-table-stmt example, which demonstrates the second case.
Also see the One or more column-definitions-example, which should cover the +-Element case in example 3.

It's possible to implement dyn Node downstream to build more specialized primitives for certain situations. For instance, one might want to cook up a graphical representation for the "any character except ..."-case. Upstream might also provide them, if the need arises.

[ia0]

I see, so that's already supported and just a matter of generating the proper diagram downstream.

[lukaslueg]

AFAICS there are some problems left in 1214d68. I'm not sure on how to proceed here, since this PR is already quite large, and not just about eyecandy-diagrams. Ping me if you need my two cents.

[traviscross]

The conflicting directions one would be resolved by #1787 (comment). The UNICODE_ESCAPE one is actually the correct grammar. E.g.:

fn main() {
    let x: &str = "\u{a____________________________________}";
    println!("_{x}_");
}

Playground link

(On INNER_LINE_DOC, were you meaning to point out some problem by highlighting it?)

[lukaslueg]

The UNICODE_ESCAPE one is actually the correct grammar. E.g.:

👀 I was actually to lazy to check, sorry for the confusion. [The syntax is somewhat hilarious?!]

(On INNER_LINE_DOC, were you meaning to point out some problem by highlighting it?)

On INNER_LINE_DOC, I was highlighting the fact that reading direction - graphically indicated by the arrows - is correct (green), while in INNER_BLOCK_DOC reading direction gets inverted on char [CHAR]-branch (red); as a mental image: "two trains would collide head-on in the red sections".

[traviscross]

I've pushed up a set of commits. I had originally planned to merge this first and do these separately, but they're somewhat intertwined with fixing issues that we probably should fix here, so perhaps it's best to look at these now.