thoughts on the src/main.rs and src/lib.rs pattern

[ashleygwilliams]

in yesterday's cargo team meeting we got into some interesting discussions around this PR:
rust-lang/cargo#5433

the gist of this PR is that cargo new --lib --bin should work.

in discussing this we got into a convo about the src/main.rs and src/lib.rs cohabiting pattern that is pretty prevalent in the rust ecosystem, but also decidedly baffling to many people. before we pave a road to that pattern, we want to make sure this pattern is deliberate and something that we want to continue.

speaking for myself, i am a user of the pattern, largely due to the restrictions of cargo test. i'm curious to hear others' opinions on this, particularly the libs team, and other people focused on API guidelines. looking forward to hearing ya'lls thoughts!

@rust-lang-nursery/libs @rust-lang-nursery/cargo

[nabijaczleweli]

As a user (or, should I say, proponent, even) of crates containing both a library and binaries, I'd be severely in favour of formalising that pattern as explicitly supported (since, as my ground-ear @Enet4 reports, it's
"not a written convention, AFAIUI"), as it's the only correct way of making a binary crate (and yields, IME, better code, as one is not prone to make bad domain-specific hacks since they're writing a library – an example of this pattern is checksums (and, well, all my exec crates), wherein the executable is a thin wrapper around the library, which, as an added benefit, is ipso facto usable as a library by other crates).

If the pattern is not mentioned within the guide (whichever one is official nowadays) and "decidedly baffling", it'd probably be a good idea to mention it there.

TL;DR: pattern good, formalise and promote pl0x.

[sfackler]

I definitely like the concept of a lib + bin crate, but I prefer the structure of src/lib.rs + src/bin/crate_name.rs. With the src/lib.rs + src/main.rs approach, you have overlapping crate roots which I'd imagine could get really confusing for someone if they want to add a submodule for their binary or pull parts of their library in as submodules rather than an extern crate.

[shepmaster]

And throwing in a third option, I really prefer to use a workspace for this case. Doing so allows me to have binary-specific dependencies (e.g. all the fancy command line parsing you want for these utility programs).

[carols10cents]

This convention is written down, though perhaps not in the most discoverable location... https://doc.rust-lang.org/stable/book/second-edition/ch12-03-improving-error-handling-and-modularity.html#separation-of-concerns-for-binary-projects

[shepmaster]

I also tend to look at these as a continuum as I'm coding along:

1. All code in my main.rs
2. Need to reuse it a little? Split into src/main.rs / src/lib.rs
3. Need to reuse it more? src/lib.rs / src/bin/a.rs / src/bin/b.rs
4. Want to make the command line nice? Workspaces

largely due to the restrictions of cargo test

Which restrictions are we talking about here? You can put tests in a main.rs and test the same kinds of things you would by moving functions / types / modules into a library. If my end goal is to produce a binary without the need for reusing the code as a library, I don't think I'd need to create the lib.rs.

[ashleygwilliams]

loving the convo so far ya'll, please keep it going!

@shepmaster

You can put tests in a main.rs

i really like my tests separate. i am not saying this is right or good, it's just the case and expectation for a lot of folks. as a result, everytime i write a binary i make a lib.rs that has basically everything and the binary is a weird shell, e.g. https://github.com/ashleygwilliams/wasm-pack/blob/master/src/main.rs. again, not saying i'm correct or amazing, just that this way of testing makes more sense to a lot of people, particularly those coming from ecosystems where this (tests in tests dir) is the norm.

[dwijnand]

Note that there are references to both src/lib.rs and src/main.rs in https://doc.rust-lang.org/cargo/guide/project-layout.html that might need adjusting with to the outcome of this discussion.

[BurntSushi]

I basically never use this pattern and probably wouldn't choose to recommend it on my own either. My reasons are basically what @shepmaster already stated: the dependencies for a library may be quite different than the dependencies for a command line application. A command line application typically requires an argv parser as an example of something that the library would generally not need. In practice, I've found the disparity quite a bit larger. Recently, a CLI program I wrote required an HTTP client where as the library doesn't. This one thing alone makes the dependency tree for the CLI program roughly an order of magnitude larger than the dependency tree for the library. This matters to me.

One exception here where I think this pattern might be useful is if there is a use case for others making use of your command line application directly via argv (whether because it's easier or you want to avoid process creation overhead or what not), but I haven't felt compelled to do that for any CLI program I've written. I admit, this sounds useful for tests, but my tests for CLI programs are always written to invoke the executable itself, because that is invariably how the end user uses the program.

To be clear, I generally agree with the high level goal of trying to make programs relatively thin by trying to push more code into libraries, and I do think workspaces are a good way to accomplish that. Workspaces "scale" too and have other benefits. That is, a src/main.rs and src/lib.rs encourages one binary and one library, but workspaces permit any number of libraries. The other benefit of workspaces, for better or worse, is compile times.

[Ixrec]

If cargo supported "lib-only deps" and/or "bin-only deps", would that significantly change the tradeoffs?

[withoutboats]

I don't have a strong opinion about a separate workspace vs just a binary in the same project, but I do prefer not having them rooted in the same directory. For this reason, I prefer to see src/bin/my_project.rs to src/main.rs if you also have a src/lib.rs. For small projects, usually the main.rs has no submodules, but sometimes you want to have a submodule for main.rs, and now it can be confusing which files are a part of which crate.

(I've also seen people who have two overlapping crates, where main and lib both have mod statements for the same modules. I think this should be strongly discouraged, as it makes the project structure very confusing!)

Overall, I'd be glad to see us formulate more guidelines around multi-crate projects, even beyond 1 lib and 1 binary, to recommendations for how to organize multiple binaries, how to organize a workspace with different packages in it, etc.

it's just the case and expectation for a lot of folks. as a result, everytime i write a binary i make a lib.rs that has basically everything and the binary is a weird shell

Seems like a solution to this particular problem would be to adjust cargo test to let it compile binaries as libraries.

[kornelski]

I love the pattern and use it for all my CLI applications.

• the app functionality is easier to test via library interface rather than all the way through stringly-typed executables.

• the separation helps me avoid writing spaghetti code in the main.rs file. I have to pause to think whether an extern crate belongs to bin or lib. It changes my frame of thinking from series of imperative commands to thinking about interfaces.

• it's lightweight and convenient. Crates.io doesn't support hidden/internal crates, so a "real" library crate needs to be a separate project published, versioned and documented separately. That may be great for big projects, but it's an undesirable overhead for projects that are mainly CLI.

So the way I see it projects that are primarily a library could be better served by having separate satellite CLI crates, but projects that are primarily CLI tools are best served by having a private low-maintenance library.

A bin+lib project can always be separated if it outgrows that setup, so I don't see harm in starting out that way.

[ashleygwilliams]

Seems like a solution to this particular problem would be to adjust cargo test to let it compile binaries as libraries.

i would really really like this.

[kornelski]

I've noticed required_features exists, so required_features = ["bin"] can be used to pull in bin-only dependencies (like argument parsing) optionally.

[kornelski]

Another argument in favor of combined bin+lib: it's surprising that doccomment tests don't work in bin targets.

[WiSaGaN]

I think the original idea is to make creating crates really easy, so that you can create separate crates for binaries and libraries except some really small binaries inside a crate. And we also have established pattern of multiple crates inside a single repo.

Like mentioned above, the dependencies can be quite different. And one of my use cases also needs build.rs. There is no way to specify different build.rs for bin and lib. Until cargo is really able to handle those, I wouldn't want to recommend the use of bin and lib in the same crate except for some really small binaries.

[nrxus]

I am unsure if this a lot to the conversation but it is possible to have test files separate without having a bin and a lib within the same project by specifying test modules in main.rs, and having those modules in separate files.

I am personally a fan of having my tests in a separate file that the code they are testing (even for unit tests) so I recently moved one of my tests in a bin project using the above strategy:

nrxus/safebus@f59570b

I am unsure how much I like this strategy when compared against moving api.rs into a lib.rs and having a more normal integration test by having a test directory but I am playing with it.

Con: I have to specify every top test module in main.rs
Pro: It fits better with my idea of what a "library" is. The code in api.rs is not meant to be used as a library, it is meant to be used as the executable.

[cfsamson]

I'm new to Rust but the exact pattern above has confused me a lot! Really, after borrowing and a few more concepts, it's the one that I have messed around with the most. Why have so many options here instead of just having one pattern unless you choose actively to change it in the .toml file.

This would be easier to grasp in my eyes:

cargo create --lib
--src
--src/lib.rs

cargo create --bin
--src
--src/lib.rs (code and tests, exactly the same as the --lib option)
--src/bin/main.rs (primary executable)

The files inside "bin" could use the same pattern of accessing the library through "extern crate myproject" (like it does now when you want to create a second executable) when you have submodules. If you have no submodules you just put the code in main and compile.

Does this make sense only to me?

[mre]

Your suggestion makes a lot of sense to me @cfsamson. I'm just a bit concerned about onboarding beginners though. Creating a lib.rs when calling cargo create --bin might be confusing.

[cfsamson]

Hmm, I guess after using it regularely for a while I see your point here. Also, the changes in 2018 edition will certainly make modules in general less confusing in the start.

When you're new, the logical way of thinking about it would be that in Rust you basically put all modules and divide your code up in a library as a default way of organizing a project. A library can either be a stand alone library "project", called a crate, or an "internal" library where you structure your code for your executable - and you call your own internal library like any other crate. If you're only writing some short code you'll just delete or ignore the lib.rs file and code in bin/main.rs as you do now.

Problem for me in the start was that you could not necessarily build understanding of A onto understanding of B to get to C. I.e. what happens when you add a lib.rs to a --bin project, why do you need to reorganize and suddenly use "extern crate" and add a lib.rs file if you want two executables in the same --bin project. And what happened with the submodules you used in your main.rs when you did that change? Why is there no testing setup by default in --bin project? Are there other changes than file/folder structure in --bin vs --lib that I haven't gotten to yet (I guess few start to learn by getting a full understanding of the configuration options in the .toml file)?

IMHO those situations create a bit of unneccesary uncertainty and headache in the start.

[prasanna]

Looks like this issue is still open - it has been more than a year with no new comment. As a rust-lang n00b, I am in favor of allowing both --bin and --lib. Reading the comments in this issue, it seems the community is widely in support of this as well, though people aren't converging to one solution.

As far as I understood from this thread, there are two main contenders for this:

1. lib.rs and main.rs at top-level
2. lib.rs. at top-level and main.rs inside bin

As I said, I'm still a n00b, but option 2 above, as explained by @cfsamson, makes sense to me.

@mre to me, it is not confusing at all. In order to add tests in tests folder, I needed to create a lib.rs anyway. It took me a while to figure that out. If it worked with defaults, it would make the experience better for beginners.

[mre]

There's a third option: separate workspaces for lib and bin (as proposed by @shepmaster above).
This has some advantages:

• clear separation of concerns
• separate dependency trees allow for smaller crates
• faster compile times (see @BurntSushi's comment above)

At the moment, I'd lean towards that idea.

That said, I personally work very iteratively, similar to @shepmaster:
In the beginning all my code is in main.rs, which is great while prototyping. Once I'm happy with the design, I move things into a separate src/lib.rs. Only if I end up building something that is worth being used as a standalone library, I restructure the project into separate workspaces.
This requires some discipline, but it feels quite natural and it isn't overwhelming for beginners.

From what I can tell, Rubyists seem to stick to the convention of having a separate lib and bin folder, but I'm not sure if that's enforced by the language. I'd love to hear how that works in practice. Also, does any other language besides Ruby have a similar concept?

No matter how we decide, it's important to communicate this clearly in the book or provide more guidance here.

[prasanna]

To me, the idea of writing tests at the bottom of the same file feels weird (as a n00b to rust). I wanted tests to be in a separate folder. But I had created the project as a bin, so I couldn't extern or use the production modules from tests.

That's the main use-case. If you are saying the canonical way is to have test and production code in the same file, then fine. Documentation should probably reflect that as the "right" way to do it (as opposed to one of the ways to do it).

[prasanna]

Also, the "lib" folder, by conventions in Java (old school) or C/C++ would be where the libraries you depend on show up. Not the libraries you write.

[BurntSushi]

f you are saying the canonical way is to have test and production code in the same file, then fine. Documentation should probably reflect that as the "right" way to do it (as opposed to one of the ways to do it).

It does: https://doc.rust-lang.org/book/ch11-03-test-organization.html

Note that this is for unit tests. Cargo also supports integration style tests, which do typically wind up in a separate directory. The link above outlines that as well.

[prasanna]

Yep. That document talks about using src/lib.rs. However, if you do the same thing with src/main.rs instead (which is created by the default settings for init), then it didn't work. I had to also create a src/lib.rs and then it worked.

[BurntSushi]

It works just fine.

Cargo.toml:

$ cat Cargo.toml
[package]
name = "maintest"
version = "0.1.0"
authors = ["Andrew Gallant <[email protected]>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

src/main.rs:

$ cat src/main.rs
fn main() {
    println!("Hello, world!");
}

#[cfg(test)]
mod tests {
    #[test]
    fn itworks() {
        assert_eq!(4, 2);
    }
}

And running tests does what you expect:

$ cargo test
   Compiling maintest v0.1.0 (/tmp/maintest)
    Finished test [unoptimized + debuginfo] target(s) in 0.22s
     Running target/debug/deps/maintest-75067320f863f02f

running 1 test
test tests::itworks ... FAILED

failures:

---- tests::itworks stdout ----
thread 'tests::itworks' panicked at 'assertion failed: `(left == right)`
  left: `4`,
 right: `2`', src/main.rs:9:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    tests::itworks

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--bin maintest'

[prasanna]

Sorry, my comment wasn't clear. I was talking about integration test not working.

cargo new ekko
cd ekko
cat <<EOT >> src/ekko.rs
pub fn ekko(a: i32) -> i32 {
    a
}
EOT
mkdir tests
cat <<EOT >> tests/pretend_integration_test_of_ekko.rs
extern crate ekko;
#[test]
fn it_ekkos() {
  assert_eq!(2, ekko::ekko::ekko(2));
}
EOT
cargo test

will give you an error error[E0463]: can't find crate for ekko

However, if you do this:

echo "pub mod ekko;" >  src/lib.rs
cargo test

then it works.

I know in this example, the test isn't really an integration test, but I don't think that should matter. Does it?

[epage]

Another aspect for separate bin/lib is when both are meant as user-accessible targets, semver guarantees for both can be dramatically different and is the primary reason I end up splitting them.