"error[internal]: left behind trailing whitespace"

[davepacheco]

Sorry in advance if this is a dup! I saw #2896 and several others with this internal error but I can't tell if this is the same cause.

Describe the bug

rustfmt produced an internal error about whitespace and exited with a non-zero exit status (which can break tooling like IDEs -- or vim integration in my case).

To Reproduce

Here's the minimum input I found that reproduced this issue:

fn f() {
    async {
    
        // Hmm.
    }
}

Note that there's trailing whitespace in the first line of the async block. Here's the result:

$ rustfmt --check tmp.rs 
error[internal]: left behind trailing whitespace
 --> /Users/dap/Desktop/rustfmt-issue/tmp.rs:3:3:0
  |
3 |     
  | ^^^^
  |

warning: rustfmt has failed to format. See previous 1 errors.

This is inside an otherwise empty directory, so there's no rustfmt configuration file. I tried the following simplifications, but each of these caused the problem to go away:

• Removing the line with the comment
• Changing async to loop
• Dropping the async { and } line (moving the contents of the async block to the top level of the function)

Expected behavior

I expected the code to be reformatted like this:

fn f() {
    async {

        // Hmm.
    }
}

and for rustfmt to exit with status code 0.

Meta

• rustfmt version: rustfmt 1.4.25-stable (0f29ff6d 2020-11-11)
• From where did you install rustfmt?: rustup
• How do you run rustfmt: rustfmt --check tmp.rs

This sounds a little bit like what's described here in #2896 but I'm not sure if it's the same or not.

[calebcartwright]

Sorry in advance if this is a dup! I saw #2896 and several others with this internal error but I can't tell if this is the same cause.

No worries! The trailing whitespace error is basically a catchall, so these are one of the exceptions where a new issue is typically preferable unless there's an existing issue open with an identical reproduction snippet.

$ rustfmt --check tmp.rs
there's no rustfmt configuration file

This would be the problem, try running with rustfmt --check tmp.rs --edition 2018

The shipped versions of rustfmt, like rustc itself, still have the default edition set to 2015, and async and 2015 didn't exactly play well together. Also like rustc, rustfmt itself does not work with Cargo's manifest files, like cargo and cargo fmt do, so when running the rustfmt bin directly in 2018 edition scenarios you need to explicitly set the edition to 2018 via the cli opt or the rustfmt config file (rustfmt.toml) as rustfmt/rustc have no insight into the contents within a Cargo.toml file.

Editors/IDEs handle editions and formatting differently (some have some level of edition detection, some use the rustfmt lib while others run the bin, etc.), so our recommendation is for users to include a minimal rustfmt.toml file with the 2018 edition set to avoid any issues.

Going to go ahead and close, though please let us know if you have any follow up questions and feel free to reopen if you can reproduce with the 2018 edition set.

[davepacheco]

Thanks for the detailed explanation! I originally saw this with the 2018 edition, but when I started simplifying it I switched to running rustfmt by hand. I will go back to my original case, make sure I'm running with the right edition, and update with the simplest case I can come up with.

Thanks for the tip about putting the edition into rustfmt.toml -- I've been wondering about the right way to specify the edition for rustfmt on a per-crate basis, so that's helpful.

[davepacheco]

Okay, here's the problematic code that also breaks in the 2018 edition:

fn f() {
    async {
    
        /* Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin urna felis, molestie ex. */
    }
    .boxed()
}

This fails (again, empty directory and no rustfmt.toml):

$ rustfmt -V
rustfmt 1.4.25-stable (0f29ff6d 2020-11-11)
$ rustfmt --edition=2018 --check tmp.rs
error[internal]: left behind trailing whitespace
 --> /Users/dap/Desktop/rustfmt-issue/tmp.rs:3:3:0
  |
3 |     
  | ^^^^
  |

warning: rustfmt has failed to format. See previous 1 errors.

$ echo $?
1
$ 

I've tried simplifying further:

• dropping the async block (so its contents are at the function scope)
• removing any character from the comment
• removing the comment
• removing the .boxed()

All of these cause the problem to go away (rustfmt correctly just removes the trailing whitespace on line 3).

I haven't dug into the cause much. I seem to recall that @ahl found a case where rustfmt miscalculated the appropriate length of a line when wrapping C-style comments. Maybe that was this comment in #4079? I'm not sure if that ever landed on stable Rust -- if not, I wonder if that might contribute here, since the problem goes away if the comment is made any shorter.

[davepacheco]

@calebcartwright I don't seem to be able to reopen the issue. Can you?

[calebcartwright]

Okay, here's the problematic code that also breaks in the 2018 edition:

Here your comment is exceeding the max width (looks like it just north of the 100 default). Try running rustfmt tmp.rs --check --edition 2018 --config wrap_comments=true if you want to give rustfmt permission to wrap the comments for you to fit within the width limits, or split it yourself to fit within your desired width.

You can also run rustfmt with error_on_line_overflow enabled to get more detailed error info, including the specific width issue
rustfmt tmp.rs --check --edition 2018 --config error_on_line_overflow=true

So the real problem here is the width limit, which in turn results in the inability to format the block and emitting the original snippet that contains the trailing whitespace, finally producing the error you see.

One of the things I'd love to change one day is to minimize the amount of silent failing that happens within rustfmt, or at least make it easier for folks to opt-in to being informed in these cases.

[davepacheco]

That makes sense, but I still think there's a bug here. I muddied the question of the exit status by using --check in my examples above. The real problem for me is that even without --check, rustfmt returns non-zero status in this case (indicating not that the code needed to be reformatted, but that rustfmt itself failed):

$ rustfmt --edition 2018 tmp.rs
error[internal]: left behind trailing whitespace
 --> /Users/dap/Desktop/rustfmt-issue/tmp.rs:3:3:0
  |
3 |     
  | ^^^^
  |

warning: rustfmt has failed to format. See previous 1 errors.

$ echo $?
1

But if I change the C-style comment to a C++-style line comment, so that the input looks like this:

fn f() {
    async {
    
        // Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin urna felis, molestie ex at.
    }
    .boxed()
}

then rustfmt handles that fine, even though it still has a long comment line:

$ rustfmt --backup --edition 2018 tmp-line-comment.rs 
$ echo $?
0
$ diff tmp-line-comment.{bk,rs}
3c3
<     
---
> 

Similarly, if I go back to the original, but drop the .boxed() line, so that the input looks like this:

fn f() {
    async {
    
        /* Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin urna felis, molestie ex. */
    }
}

then it's happy to format that too:

$ rustfmt --backup --edition 2018 tmp-no-boxed.rs 
$ echo $?
0
$ diff tmp-no-boxed.{bk,rs}
3c3
<     
---
> 

It seems that there's something more going on than just a long comment with wrap_comments=false.

[calebcartwright]

I muddied the question of the exit status by using --check in my examples above.

No worries, I was also going to note that this thread has been a bit of a moving target. It's always great (and much appreciated!) to trim down the original use case into a minimal reproducible example, but in this instance a lot of important context was left out of the original report and we've been iteratively building it back out.

That's not a big deal, just has complicated things a bit because even the smallest looking changes have non-trivial impacts on the AST and formatting logic.

I've also been trying to balance providing recommendations for resolving the issue/moving past it, along with the what/why context because this is running into some of the well-known warts of rustfmt that don't have an easy path to resolution. However, will expand a bit below.

but drop the .boxed() line

This is a great example of a seemingly small change that has a big impact. When this is present we go from a single async block expression to having a chain, and the Style Guide that defines additional formatting rules that rustfmt has to comply with, and thus brings additional logic (and bugs) into the fold.

When the snippet is a chain (i.e. .boxed() call included) you're going to run into a long standing issue (#3863) that exists with chains where there are additional width checks that are applied to the individual chain items and the resulting lines in the entire formatted chain, and if any those checks fail then that results in a failure to format the entire chain. That in turn kicks off the cycle I described in previous comments above. Those granular width checks are chain-specific here and not run when the formatting is just the async block expression.

Now as you'd observed, there's a difference between the behavior with chains and the width-limit-offending line is a line comment vs. a block comment. That's because those granular chain width checks (which include a line-by-line scan against the final result) seemingly exclude/skip line style comments but not block style.

I do think there's likely a bug there, as if we want to exclude comments from width checks then that shouldn't vary on comment style. Will take a closer look in the near future to confirm there's not some block-style edge case I'm missing, but will open a new issue if confirmed to have a more narrow focus given the length of this thread and unaffiliated topics (e.g. edition).

Keep in mind though that the larger chain width issues would still be at play, for example if the width-offending line were a string lit or if your snippet included heavily indented blocks and/or long idents, and you'd see the exact same behavior if that offending line were anything other than a comment.

[davepacheco]

Thanks for that explanation, and sorry for the red herrings along the way. If I'm understanding right, I seem to be running into some combination of #3863 plus maybe a second bug related to block-style comments within chains. That makes sense.

[calebcartwright]

Thanks for that explanation, and sorry for the red herrings along the way.

No worries, and thanks for sticking with it!

If I'm understanding right, I seem to be running into some combination of #3863 plus maybe a second bug related to block-style comments within chains. That makes sense

👍 I've opened #4668 to address the latter so feel free to subscribe there for updates.

Side note, these are the types of cases that can result in rustfmt seeming like it "changes" the formatting after an upgrade. For example, if your original snippet didn't have the inner trailing whitespace then the trailing whitespace error would never have been surfaced, and you wouldn't have known that rustfmt actually wasn't formatting that chain at all.

So if your original input had some formatting issues those would have been masked because rustfmt was unable to format the chain. Often times the formatting issues are quite subtle so folks understandably assume that things are being formatted successfully and their snippet aligns with the formatting rules, but that's not actually the case. For illustrative purposes, here's an obviously misformatted extension of your snippet that rustfmt will silently leave in place, with the issue being much clearer:

fn f() {
    async {
let x  =   true  ;
        /* Lorem ipsum dolor sit amet, consectetur adipiscing elit. Proin urna felis, molestie ex. */
    }
    .boxed()
}

Once we fix the bug with the block-style comment exclusion, rustfmt will be able to format that chain, which will result in that snippet finally being formatted correctly. It's not that the formatting rules changed, it's just that there was a bug fixed within rustfmt and once folks upgrade to a fixed version of rustfmt it's actually able to format code that it wasn't touching before.

This also happens when things are fixed or improved upstream in rustc, with common examples being macro defs and calls.

[davepacheco]

Thanks for that!

[piaoger]

I got same error "left behind trailing whitespace" in both local and playground. Removing "//FLOAT" or adding simple line "//.." above it in default code block will fix the rustfmt error.

use serde::{Deserialize, Serialize};

#[derive(Debug, Serialize, Deserialize)]
pub struct A {
    /// default: 1.5
    size: f32, // FLOAT
}

impl Default for A {
    fn default() -> Self {
        Self {  
size:1.5, //FLOAT
        }
    }
}

[ytmimi]

@piaoger It's probably worth opening up a new issue for your particular trailing whitespace error. Your input snippet contains trailing whitespace after the Self { and the trailing line comment prevents formatting.