Implement the 64-bit variant of xxHash.

[ghost]

Matches C in speed, but it looks like SipHash was not the only bottleneck for #11783.

Tests include the official data and the relevant ones from SipHash.

[rust-highfive]

Warning

• These commits modify unsafe code. Please review it carefully!

[thestinger]

For security, I'd say it could replace SipHash wholesale, as per this post.

That post doesn't go into any in-depth cryptanalysis. Security against DoS attacks requires a lot more than just good results on statistical tests and avalanche effect resistance.

[ghost]

Oh well. At least it's fast :)

[thestinger]

Can this replace the FNV implementation that's currently used in rustc?

[ghost]

@thestinger: If used with large enough chunks (>32b), it can beat anything. With small chunks, it degrades the same way SipHash does, but I think that's inevitable with a decent hash.

[pczarn]

nit: I think the convention is XxState

[pczarn]

This implementation has quite a bit of unsafe code. SipHash's doesn't have any.

Have you checked the C implementation's speed? I think it should be faster than FNV for 8-byte or larger chunks when inlined. I've measured my implementation:

test xxhash::rust::chunks_8_xxh64   ... bench:     76874 ns/iter (+/- 603) = 852 MB/s
test xxhash::rust::chunks_15_xxh64  ... bench:     56387 ns/iter (+/- 681) = 1162 MB/s
test xxhash::rust::chunks_32_xxh64  ... bench:     26776 ns/iter (+/- 121) = 2447 MB/s
test xxhash::rust::chunks_128_xxh64 ... bench:     14792 ns/iter (+/- 263) = 4430 MB/s
test xxhash::rust::chunks_256_xxh64 ... bench:     12603 ns/iter (+/- 354) = 5200 MB/s

test xxhash::c::chunks_8_xxh64      ... bench:     62287 ns/iter (+/- 441) = 1052 MB/s
test xxhash::c::chunks_15_xxh64     ... bench:     43615 ns/iter (+/- 459) = 1502 MB/s
test xxhash::c::chunks_64_xxh64     ... bench:     11531 ns/iter (+/- 109) = 5683 MB/s
test xxhash::c::chunks_128_xxh64    ... bench:      9555 ns/iter (+/- 98) = 6858 MB/s
test xxhash::c::chunks_256_xxh64    ... bench:      8516 ns/iter (+/- 57) = 7695 MB/s

What makes you think a hash can't be fast? Xxhash has two parts at its core.

• the inner loop that consumes 32 bytes at a time
• the finalizer

For 8 byte chunks, it does only the following:

let total_len = 8;
let mut h64 = self.seed + PRIME5 + total_len;

let mut k1: u64 = source * PRIME2;
k1 = rotl64(k1, 31);
k1 *= PRIME1;
h64 ^= k1;
h64 = rotl64(h64, 27) * PRIME1 + PRIME4;

h64 ^= h64 >> 33;
h64 *= PRIME2;
h64 ^= h64 >> 29;
h64 *= PRIME3;
h64 ^= h64 >> 32;
return h64;

However, the finalizer alone is responsible for avalanche properties and can be used with chunks that have exactly 8 bytes:

let mut hash = source;
hash ^= hash >> 33;
hash *= PRIME2;
hash ^= hash >> 29;
hash *= PRIME3;
hash ^= hash >> 32;
return hash;

[gereeter]

Where did this seed come from?

[ghost]

It's a large prime. This should be #[cfg(test)] I think, but I wanted to be consistent with SipHash which has new_with_keys(0, 0) here.

For real uses, the seeds should be randomized, which is done with RandomSipHasher in libstd.

[ghost]

@pczarn: It's not the hash that's slow, but Rust's handling of them. Hash is inefficient because it needs to be stable (i.e. a.hash() == a.hash(), regardless of where it was computed), and that forces slow paths way too often. I've opened this topic.

[arthurprs]

I expressed this concern a couple of times already. I think we're taking security too far in the hashmap implementations. The majority of programming languages are using a per process seed (either for sip hash or others). Rust has a seed per hash map AND a slow overly-secure hash as the default.

[vks]

a slow overly-secure hash as the default

This is a better default than an insecure and fast hash. If you care about performance and not about security, you can opt in. If you don't care about performance, you should use the secure version.

[arthurprs]

Did you guys see http://discuss.rust-lang.org/t/unstable-hash-architecture/578 ? It's something worth discussing.

[alexcrichton]

Unfortunately deleting the source repository causes bors to get stuck, @Jurily would you mind reopening? Thanks!