Add support for defining procedural macros

[DJMcNab]

Continues #688.

TODO:

• Support procedural (FnLike) macros in nameres
• Add nameres for Derive macros (and expand them, possibly in a separate PR)
  • (Separate PR) Build the first builtin derive (probably Copy as that should be simplest)
• Add nameres for Attribute macros
• (Separate PR) Parse macros 2.0 macros to allow docs for the Copy macro Support declarative macros 2.0 #2248
• (Separate PR) Experiment adding support for @dtolnay's watt, to demonstrate that we can support 'proper' procedural macros, or at least the easiest (pure) subset.

[DJMcNab]

@matklad I need some help with this - I like the approach of splitting MacroDef by FnLike(Builtin/Declarative/Procedural)/Attribute(Builtin/Procedural)/Derive(Builtin/Procedural), but I'm not sure how to work with FromSource for that. You said in #2248 (comment) that we shouldn't have a node in ra_syntax for MacroDefs which include procedural macros through FnDef, but FromSource for MacroDef requires some ast type which represents it. Should I just use ModuleItem?

My current work was in https://github.com/DJMcNab/rust-analyzer/tree/proc-nameres-saved - I was trying to rebase but ended up in a state where Github thought the PR head was upstream master so closed the PR.

[matklad]

which include procedural macros through FnDef

This shouldn't be the case: we shouldn't have proc-macro crates in the crate graph, we should only have compiled proc-macros themselves (probably directly in the crate graph). In other words, the definition of a proc macro is, in general, unknowable, because it lives in a completely separate compilation session. Proc macros should be another kind of Def, which hasn't source attributes, and can only be used to transform tokens.

…

On Sat, 23 Nov 2019 at 14:31, Daniel McNab ***@***.***> wrote: @matklad <https://github.com/matklad> I need some help with this - I like the approach of splitting MacroDef by FnLike(Builtin/Declarative/Procedural)/Attribute(Builtin/Procedural)/Derive(Builtin/Procedural), but I'm not sure how to work with FromSource for that. You said in #2248 (comment) <#2248 (comment)> that we shouldn't have a node in ra_syntax for MacroDefs which include procedural macros through FnDef, but FromSource for MacroDef requires some ast type which represents it. Should I just use ModuleItem? My current work was in https://github.com/DJMcNab/rust-analyzer/tree/proc-nameres-saved - I was trying to rebase but ended up in a state where Github thought the PR head was upstream master so closed the PR. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2315>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AANB3M373Z6GURYFBMMOCNLQVEICPANCNFSM4JPIVX5A> .

[DJMcNab]

But surely from an IDE point of view, the only way to define a proc macro is using a function with a #[proc_macro[...]] attribute?

[matklad]

But surely from an IDE point of view, the only way to define a proc macro is using a function with a #[proc_macro[...]] attribute?

This can't work, because IDE can't magically compile this function to a proc macro. The IDE should learn about proc-macros the same way compiler learns about them: by dynamically loading .sos:

--extern my-proc-macro=/home/matklad/tmp/hello/target/debug/deps/libmy-proc-macro.so

[matklad]

That is, in the CrateGraph we'll have someting like this

proc_macros: FxHashMap<String, PathBuf> // points to .so

[matklad]

If we go the WATT route, we should use not the SO, but the .wasm file. And we also should have a separate cargo command (?) which puts a bunch of .wasm files to ./target

[DJMcNab]

I'm not sure. I might have to throw in the towel on this - I'm not certain I know how I would implement this within your vision

[matklad]

Uhu, sorry for not really digging into this this time: I'd love to do some design work here, but my design-bandwidth is exhausted by the hir split at the moment.

I think the rough plan should work as follows:

• On the cargo side, we need some way to compile proc macros (both .so and .wasm compilation should work the same). I think this might be just running cargo check --message-format=json and looking for proc macro outputs.
• Then in out project_model crate, we should support loading proc-macros. That means fetching paths from the first step, "parsing" the .so/.wasm files to figure out the names of proc macros, and integrating them into the CrateGraph data structure.
• Then, we need some way to execute proc macros, by using an external process (if .so) or an embedded wasm runtime (if .wasm)
• Then, we need to integrate macros from crate graph into name resolution (this will need a new kind of def)
• Finally, we need integrate the two previous steps and run the proc-macros using our tt::TokenStream.

The most important bit here is that we never actually look at the source code of a proc_macro crate, we only work with compiled representation of proc macros.

[edwin0cheng]

I think the external process should accept an absolute path of a proc-macro : e.g. serde::Serialize, and a stream of tokens (e.g an array in json or whatever format) , and output a stream of expanded tokens. Using which methods to expand the macro (.so/.wasm) should be out of scope in RA point of view. (I can imagine the simplest solution: it could just generate a new project and fill in the tokens and use cargo-expand to expand it back to RA)