new: Don't add `#[cfg(test)] mod tests {}` by default

[cgwalters]

Problem

See discussion in https://rust-lang.zulipchat.com/#narrow/stream/122651-general/topic/why.20mod.20test.3F

Having unit tests be under a separate module has no benefit for simple
unit tests. I had originally thought that #[test] functions were
still compiled even in non-test builds, but that's not true;
it seems that #[test] is implicitly also #[cfg(test)] - which
makes sense.

A separate module is really only necessary when one scales up a
test suite (e.g. adding types like structs that are only used by tests,
helper functions). But even then, one can add #[cfg(test)]
to these extra helpers to start.

But at that scale, there are also other potentially better ways
to handle things, such as moving the tests to an entirely separate
file which works better with incremental compilation.

There's also the philosophy that one should focus on integration,
not unit tests:
https://matklad.github.io/2021/02/27/delete-cargo-integration-tests.html

In short, there's lots of ways to handle tests; I think the book
should try to cover all the tradeoffs and advantages of these.

Injecting #[cfg(test)] mod test {} into every cargo new Rust
is a strong statement that this is a standard, but I think it shouldn't
be. Instead again, at non-trivial scale users should be referencing
the docs.

Note the docs will need to be updated after this, because it
does reference this explicitly. The documentation even says:

For now, let’s ignore the top two lines and focus on the function to see how it works.

Which to me is a strong indication that this is actually not useful
for every person new to Rust to see by default.

Proposed Solution

Don't add #[cfg(test)] mod test { ... } in cargo new by default, and change the book.

Notes

No response

[cgwalters]

This issue was originally a PR with code to implement the change: #10530
If accepted, then we should be able to just reopen that PR.

[joshtriplett]

But even then, one can add #[cfg(test)] to these extra helpers to start.

That's true, but in addition to being less convenient, it's easy to forget to do, in which case you get warnings about unused functions/imports. Putting #[cfg(test)] on a single test module has the advantage of making it zero-friction to just add contents to the test module without worrying about that on each item.

[ehuss]

We discussed this proposal with the team, and I don't think we'll be making this change at this time. There are various tradeoffs for what the initial template contains, and I think the current approach has sufficient merits to keep around for now.

New users often get confused by unused warnings particularly around imports that are only used in tests. Putting imports within a module helps avoid that confusion. Once you start needing helper functions or imports, it can be awkward to then need to add #[cfg(test)] in lots of places.

I also think the current approach helps guide and inform new users on a way to help organize tests. Also, editors often have code folding which provides a convenient way to hide a large batch of tests.

There is merit to using mod tests; as a better way to organize tests, but adding a separate file to the initial template I think would be frustrating.

It's definitely worthy to examine the best onramp for new projects and new users. However, there are some larger questions, some of which are conflicting:

• What is the ultimate best practice for structuring tests?
• What is the best getting started project?
• What is the best for teaching a new user?

I think in the future this conversation might be better served if and when cargo has a built-in templating system similar to tools like cargo generate. I think the "getting started" experience could benefit if a user jumps directly into a template that is tailored to what they want to achieve. And if users want custom templates, such as something like just an empty file, it would accommodate that too.

Thanks for the suggestion, though! I think it is really good to think about these things.