Filename errors from proc macros are hard to debug

[Michael-F-Bryan]

I have a project that uses procedural macros from the wit-bindgen-rust and wit-bindgen-wasmtime to generate my WebAssembly glue code.

The project layout looks somewhat like this:

• /
  • Cargo.toml (workspace root)
  • wit-files/ (git submodule)
    • rune
      • runtime-v1.wit
  • xtask/
    • Cargo.toml
    • src/
      • lib.rs
      • metadata.rs (uses wit_bindgen_wasmtime::export!() and friends)

At the moment I'm seeing errors where I can successfully build my xtask crate from either the xtask/ directory or the workspace root on my desktop. However, when I am running the same code in CI, it errors out with the following:

error: failed to read: wit-files/rune/runtime-v1.wit
Error:   --> /home/runner/work/proc-blocks/proc-blocks/xtask/src/metadata.rs:10:1
   |
10 | wit_bindgen_wasmtime::export!("wit-files/rune/runtime-v1.wit");
   | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

I've added a dummy ls step to verify that wit-files/rune/runtime-v1.wit is present at that location relative to the checkout folder, but I suspect CI is running cargo build from a different directory and therefore we get different results when we call Interface::parse_file() with the string that was passed to the macro.

What confuses me is that I've run cargo build fro different directories (e.g. the workspace root - /, the xtask/ directory, and /xtask/src/) and it all works as intended on my computer.

Do you have any troubleshooting suggestions or an explanation of what is going on?

$ rustc --version --verbose
rustc 1.60.0-nightly (bfe156467 2022-01-22)
binary: rustc
commit-hash: bfe15646761a75f0259e204cab071565eed2b1e5
commit-date: 2022-01-22
host: x86_64-unknown-linux-gnu
release: 1.60.0-nightly
LLVM version: 13.0.0

My Cargo.lock says I am currently using e9c7c0a from the wit-bindgen repo. My code is accessible via hotg-ai/proc-blocks@51ad84e and I was seeing the error on cargo xtask dist.

[alexcrichton]

If possible I'd recommend patching this crate to always do file I/O through absolute paths (appending to env::current_dir()) and seeing what error pops out to double-check that it's reading the file you think it is. We could do that by default as well if it proves useful

[Michael-F-Bryan]

I think rustc or cargo is doing something funny here. I ran the command from the workspace root once, then did a cargo clean and prepended the included path with ../, and the crate compiled successfully anyway.

This is my project tree:

$ tree
.
├── README.md
├── Cargo.toml
├── ...
├── wit-files
│   ├── README.md
│   └── rune
│       ├── rune-v1.wit
│       └── runtime-v1.wit
└── xtask
    ├── Cargo.toml
    └── src
        ├── bin
        │   └── xtask.rs
        ├── build.rs
        ├── lib.rs
        └── metadata.rs

The build runs fine if I specify my WIT files relative to the xtask/ directory.

$ cargo clean
$ rg wit_bindgen_wasmtime
xtask/src/metadata.rs
10:wit_bindgen_wasmtime::export!("../wit-files/rune/runtime-v1.wit");
11:wit_bindgen_wasmtime::import!("../wit-files/rune/rune-v1.wit");
$ cargo build --package xtask
   ...
   Compiling xtask v0.1.0 (/home/consulting/Documents/hotg-ai/proc-blocks/xtask)
    Finished dev [unoptimized + debuginfo] target(s) in 31.03s

But I can also specify a path relative to the workspace root and that seems to build successfully too? 🤷

$ cargo clean
$ rg wit_bindgen_wasmtime
xtask/src/metadata.rs
10:wit_bindgen_wasmtime::export!("wit-files/rune/runtime-v1.wit");
11:wit_bindgen_wasmtime::import!("wit-files/rune/rune-v1.wit");
$ cargo build --package xtask
   ...
   Compiling xtask v0.1.0 (/home/consulting/Documents/hotg-ai/proc-blocks/xtask)
    Finished dev [unoptimized + debuginfo] target(s) in 31.72s

Reading rust-lang/rust-analyzer#7057 (comment) from the rust-analyzer repo, I don't think procedural macros can make any assumptions about where they'll be run from. I know this was a problem with the include_dir crate, so instead of relying on the current directory being correct, I implemented environment variable interpolation so users could write include_dir!("$CARGO_MANIFEST_DIR/src/whatever/").

I'll do some more troubleshooting and let you know how I go.

[tailhook]

I have a similar setup. And I've noticed that cargo run and cargo build works fine. But cargo run chokes on Doc-tests on such crate.

It's probably possible that doc build runs in a different directory?

[Michael-F-Bryan]

It was mentioned on u.rl.o that doctests processing still happens in the package root.

Bjorn3: A proc macro runs from whichever directory cargo invokes rustc. In the past this used to.be the package root, but it was changed to the workspace root to make error messages relative to the workspace root. I don't think there i sany guarantee that this won't change again in the future.

VorfeedCanal: Note that doctests processing still uses package root and not workspace root.

[Michael-F-Bryan]

@alexcrichton how would you feel about using something similar to include_dir's environment variable interpolation in wit-bindgen's macros?

I'm not sure whether rustc or cargo make any guarantees around which environment variables will be accessible, but until Span::source_file() becomes stable that might be a robust solution.

[alexcrichton]

Thanks for #160, but I feel like that's a heavy-handed way to solve a somewhat more focused problem here. For example the proc-macro could read CARGO_MANIFEST_DIR itself and .join paths onto that to get a suitable anchor.

Ideally Cargo could be relied upon here and IIRC it should consistently pick a working directory nowadays. It might be good to narrow down what's going wrong at the Cargo level and report an issue upstream?

[Michael-F-Bryan]

I agree - using shellexpand for this is a very heavy-handed approach.

After looking back at which jobs failed, I think the issue was that the current directory changed some point between the 3 Rust versions I'm testing in CI - 1.57.0, stable, and nightly. When I added a .. (i.e. assuming we execute from the crate directory so we need to go up a level to find the *.wit files) one build would fail, but then if I removed the .. (i.e. executing from the workspace root) a different build would fail.

I'll see if I can modify #160 to always import files relative to $CARGO_MANIFEST_DIR.

[Michael-F-Bryan]

Superseded by #167.