Reduce dependency tree

[bheisler]

Hey, thanks for the work on mdbook!

Would it be possible to try to prune mdBook's dependency tree?

My documentation build in Travis needs to download and compile ~300 crates to install mdbook on every build. This takes a long time (~10 minutes for mdbook). I then install mdbook-linkcheck, which depends on mdbook and so it has to recompile the whole tree again. The long compile times also make it more painful to install and use mdbook normally, or to update from older versions.

[ehuss]

Binary releases are made available at https://github.com/rust-lang-nursery/mdBook/releases, which is what I use to reduce CI time (example). That may not help with mdbook-linkcheck, though. There is also a number of Cargo features which you can turn off to reduce the compile time (almost twice as fast with all of them off).

There are a few deps that could maybe be trimmed, but they will unlikely make a noticeable impact. The slower dependencies (syn, serde, regex, clap) can't really be removed.

You can also try to take advantage of CI caching. sccache is one option. You can also just cache ~/.cargo/bin. I've also been experimenting with the unstable -Z install-upgrade cargo option to make this easier (such as here).

[bheisler]

Ah, I hadn't known about the cargo features; that will help.

I currently force-install on every build to make sure that it has the latest version, so caching doesn't really help there. I'll keep an eye on the install-upgrade option; hopefully it's stabilized soon.

[Lokathor]

@bheisler until then try something like this

cache:
  cargo

before_script:
  - test -x $HOME/.cargo/bin/cargo-install-update || cargo install cargo-update
  - test -x $HOME/.cargo/bin/mdbook || cargo install mdbook
  - cargo install-update -a

[Michael-F-Bryan]

I use mdbook for several documenting several closed-source projects and use a similar technique to @Lokathor. The only difference being I'll check the --version and do a --force install otherwise.

Another optimisation you can use is cargo install --debug mdbook to skip optimisations.

That may not help with mdbook-linkcheck, though.

Thanks for pointing that out @ehuss. There's some work I'd like to do soon, so I'll see if I can upload some release binaries for mdbook-linkcheck with the next release.

[Michael-F-Bryan]

Something that may be useful in diagnosing the cause of slow compilations is cargo build -Ztimings. It generates a really useful HTML file that visualises the compilation process.

I had a quick look around c9dae17...

This is the timings report: timings.zip

The dependency graph:

Some notes:

• regex is a hefty dependency (15s for regex and 15s for regex_syntax)
• syn is compiled twice, once for 1.0.5 and again for v0.15.44 (both about 30s)
• handlebars pulls in pest and pest's dependency tree adds a good 15s
• toml_query depends on old versions of syn and quote, plus it pulls in hefty dependencies like failure_derive and darling

[ehuss]

Yea, I've looked at this in the past, there are a large number of duplicated dependencies. Getting rid of toml_query seems doable, it is fairly expensive. I also noticed iron had some outdated dependencies, and hadn't been updated in a long time. If anyone knows of a lightweight web server that is actively maintained, I think it might be good to look at that. It might be hard to get rid of regex.

I've also wondered, for plugins (like mdbook-linkcheck), would it make sense to have a separate crate that just provides the bare minimum of what they need? Or make more aggressive use of features to reduce the size?

[Michael-F-Bryan]

Getting rid of toml_query seems doable, it is fairly expensive

I'm working on a minimal replacement for toml_query. The API is pretty straightforward so it shouldn't be too difficult.

I've also wondered, for plugins (like mdbook-linkcheck), would it make sense to have a separate crate that just provides the bare minimum of what they need?

The mdbook-linkcheck program has a --standalone flag so it can be used outside of a mdbook plugin context (e.g. mdbook-linkcheck path/to/my/book), so under the hood it'll require everything related to MDBook::load().

We could always split the HTML renderer out into its own executable like mdbook-linkcheck, hidden behind a feature flag. That'd remove a lot of unnecessary code around templating and HTML rendering for 3rd party plugins.

Historically speaking, the HTML renderer has always been a pretty messy piece of code and really tightly coupled with the core of mdbook, so that wouldn't necessarily be a bad thing... just a lot of work.