cargo publish "memory allocation of .. bytes failed" with large, ignored test assets

[FauxFaux]

Problem

cargo publish fails with:

memory allocation of 1099511627782 bytes failed
zsh: abort (core dumped) cargo publish --dry-run

...if you have large files in your build directory, even if they are excluded.

I expect this to complete in all cases, i.e. a crate can be published regardless of how much memory I have (within reason!). But, I especially expect it to pass in this case, where the file causing the error is ignored, and will not be published.

Steps

1. Have a test data directory which is ignored.

mkdir -p tests/generated
echo '*' > tests/generated/.gitignore
g add -f tests/generated/.gitignore

2. Have a build.rs which creates a large file in this test data directory.

use std::io::{Seek, SeekFrom, Write};
use std::fs;

fn main() {
    let mut f = fs::File::create("tests/generated/large.txt").unwrap();
    f.seek(SeekFrom::Start(1024 * 1024 * 1024 * 1024)).unwrap();
    f.write_all(b"hello").unwrap();
}

% du -h tests/generated/large.txt    
4.0K	tests/generated/large.txt

% du --apparent-size -h tests/generated/large.txt
1.1T	tests/generated/large.txt

3. Run cargo package or cargo publish.

% cargo +stable publish --dry-run --allow-dirty
    Updating crates.io index
warning: manifest has no description, license, license-file, documentation, homepage or repository.
See https://doc.rust-lang.org/cargo/reference/manifest.html#package-metadata for more info.
   Packaging dozen v0.1.0 (foo/dozen)
   Verifying dozen v0.1.0 (foo/dozen)
   Compiling dozen v0.1.0 (foo/dozen/target/package/dozen-0.1.0)
    Finished dev [unoptimized + debuginfo] target(s) in 0.53s
memory allocation of 1099511627782 bytes failed
zsh: abort (core dumped)  cargo +stable publish --dry-run --allow-dirty

Possible Solution(s)

I believe this was introduced by 78a60bc , which intentionally hashes all of build.rs's output, to detect changes it has made. I believe that change is maybe correct in this case; if build.rs changed the test data, then maybe the tests will perform differently, so it's a different test?

The allocation is failing here, which you can see with gdb, but not with logging or backtraces:

cargo/src/cargo/ops/cargo_package.rs

Lines 693 to 694 in 759431f

	let contents = fs::read(entry.path())?;
	let hash = util::hex::hash_u64(&contents);

i.e. read the entire file into memory, then hash the contents. Chunked hashing should not require unbounded memory?

Maybe:

diff --git a/src/cargo/ops/cargo_package.rs b/src/cargo/ops/cargo_package.rs
index a1b9a5f6..804ddb26 100644
--- a/src/cargo/ops/cargo_package.rs
+++ b/src/cargo/ops/cargo_package.rs
@@ -690,6 +690,7 @@ fn hash_all(path: &Path) -> CargoResult<HashMap<PathBuf, u64>> {
             let entry = entry?;
             let file_type = entry.file_type();
             if file_type.is_file() {
+                debug!("hashing {:?}", entry.path());
                 let contents = fs::read(entry.path())?;
                 let hash = util::hex::hash_u64(&contents);
                 result.insert(entry.path().to_path_buf(), hash);

I saw this in a real project, https://github.com/FauxFaux/ext4-rs , which extracts some sparse (i.e. tiny but apparently large) disc images from a tiny tar file during its tests. It publishes fine on e.g. 1.34.2, but not on stable.

Notes

Output of cargo version:

% cargo +stable --version                      
cargo 1.38.0 (23ef9a4ef 2019-08-20)

I'm on amd64 Ubuntu with ~30gb of memory available. If you have over 1TB of virtual memory available, then the above testcase might pass (lucky you).

[alexcrichton]

Thanks for the detailed bug report!

I think yeah the right solution here is to just avoid reading everything in-memory and hashing it externally. It'll still probably take forever to read and hash a 1TB file though so processing exclude rules in a reasonable way would also be reasonable here!

[hbina]

Hi, since this have been marked easy I thought I could help with the issue. I have read through the description but I fail to come up with a solution without knowing some facts about the application.

Does cargo have any information on what files are being ignored by git or not? I find this unlikely, as there is no reason that it should. Additionally, any arbitrary fixes that checks the size of the file will backfire some time in the future.

So any ideas?

[ehuss]

@hbina Thanks for taking a look! I'm not sure I understand the question about git. Cargo does know which files are ignored. However, this code path is used to explicitly check if any file is modified (ignored or not).

The solution is to stop using fs::read inside hash_all, and instead use something that streams the file into the hasher. We already have this implemented for Sha256. We need it implemented for the hasher used here. If possible, it would be great to change the Sha256 implemented to be generic for the hasher type, and reuse the code.

[hbina]

I see, I will take a look again after school today.

Edit: Having skimmed through it a bit. It seems like SipHasher does not implement Writer so it can't be used like a stream. Do I just replace it with the Hasher in your linked? Except use Algorithm::MD5, which I assume is cheaper

But this seems like it will break other code.

Edit: Nevermind...