x.py downloads much more than necessary

[AaronKutch]

One problem I am having is that running ./x.py test --stage 0 src/libcore is updating several submodules that I assume are not needed at all for just testing src/libcore. For example, it should not need any documentation related submodules, rustfmt, rls, and a bunch of other stuff.

Another problem is that some of these submodules are huge with src/llvm-project taking almost 2 Gigabytes (edit: I estimated wrong, I was reading git object percentage and the actual was 555 MiB) of my precious internet usage. I expect llvm-project to be quite large, but is it trying to download the entire git history? I have to run clones with --depth 1 to make the downloads bearable.

[Mark-Simulacrum]

We don't have the resources to make patches that optimize for internet usage at this point and largely, x.py doesn't have the knowledge to know whether it needs or doesn't need a submodule. Most of the submodules are small, anyway, aside from llvm.

Since we don't have the resources to change things here I'm going to close this issue. If there are non-invasive patches that can be optionally enabled then we may be able to accept them via PR.

[AaronKutch]

So, is the git history needed or can I submit some patch to fix that?

[Mark-Simulacrum]

On high bandwidth/unmetered networks my understanding is that limiting git history results in slower performance (both for the client and the server); that means that shallow operations are generally non-ideal for us.

[Mark-Simulacrum]

To be clear -- I think such an option would definitely need to be opt-in, but if it's somewhat non-invasive I'd be willing to consider it.

[AaronKutch]

thank you