Support consistent hashing in cache.NewMemcachedClient

[rfratto]

This PR adds support for consistent hashing of memcached servers by server DNS name.
Support for this is behind a boolean configuration variable in the client config, available in the flags as -memcached.consistent-hash

The use of jump hash enables consistent hashing when adding or removing a memcached server only requires 1/N keys being moved. A downside to jump hash in particular is that the order of DNS names used for the servers must be predictable to maximize efficiency. A good example of a predictable DNS name is memcached-[pod index].cortex.svc.cluster.local, which is what Kubernetes would create when running memcached as a StatefulSet on a headless service.

/cc @tomwilkie

[tomwilkie]

I have a question about the sort of the hostnames returns from DNS. Are the numbers zero padded? Does the sort handled going from 10 to 11 servers correctly?

Otherwise LGTM

[rfratto]

@tomwilkie The numbers aren't zero-padded; the sort won't handle it correctly and a lot of keys will start being moved after scaling past 10 servers.

If that's a problem that needs to be addressed, most flexible solution would be to abandon the jump hash and use a different consistent hashing algorithm that doesn't depend on the order of the DNS names. Switching over to the rendezvous hash would handle arbitrarily ordered servers and allow to scale past 10 servers cleanly (i.e., only 1/N keys moved).

[bboreham]

FWIW I would look at something like https://github.com/facebook/mcrouter to scale memcached - this allows multiple replicas of the same data and back-filling data that is in one replica but not another.

[tomwilkie]

@rfratto could we use a natural sort on the domain names instead? https://godoc.org/bitbucket.org/zombiezen/cardcpx/natsort

@bboreham Yeah I checked that out before we started this work - I don't think mcrouter actually helps the case we're looking at here; it suffers from the same problem that increasing the size of the pool resulting in a complete invalidation of the cache. And it doesn't do service discovery... If it was in go I'd have said add these features there, but this seems easier.

[rfratto]

@tomwilkie Ah, that’s cool, I didn’t know about natural sorts. That will fix our problem here.

[bboreham]

And [mcrouter] doesn't do service discovery

There are 3rd-party (4th-party?) projects that do that, e.g. https://github.com/mlaccetti/mcrouter-kubernetes-provisioner, https://github.com/ianlewis/memcached-operator

[rfratto]

@gouthamve @tomwilkie PTAL, the natural sort works as expected with StatefulSet DNS names.

[tomwilkie]

Can you find out what the result of an SRV lookup looks like on k8s, and make sure the natsort does the right thing? A unit test would be great.

[rfratto]

@tomwilkie PTAL. SRV lookup for a headless service backed by a stateful set in k8s looks like this:

;; QUESTION SECTION:
;memcached.cortex.svc.cluster.local. IN SRV

;; ANSWER SECTION:
memcached.cortex.svc.cluster.local. 30 IN SRV   10 25 0 memcached-0.memcached.cortex.svc.cluster.local.
memcached.cortex.svc.cluster.local. 30 IN SRV   10 25 0 memcached-1.memcached.cortex.svc.cluster.local.
memcached.cortex.svc.cluster.local. 30 IN SRV   10 25 0 memcached-2.memcached.cortex.svc.cluster.local.