[Experimental] Memberlist singleton and support for multiple codecs

[pstibrany]

This PR changes experimental memberlist implementation of KV store in following ways:

• memberlist.Client is now only a small wrapper around memberlist.KV. KV is a top-level cortex component (but cannot be run on its own), and is a singleton. KV is only initialized if any component actually wants to use memberlist as its KV store.
• Instead of using custom binary protocol, memberlist KV store now exchanges protobuf messages defined in kv.proto
• memberlist.KV now supports multiple codecs. For now, this is only used in tests. Codecs must be registered during KV initialization.

These changes allow Cortex in single-binary mode to use memberlist as KV store. Previously, such attempts would fail with panic, since all components (distributor/ingester/querier) tried to create a memberlist instance, and it failed with metrics registration. Even if it didn't fail there, it wouldn't work because each instance would try to open same port for listening.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[jtlisi]

LGTM besides some small nits

[pracucci]

Good job @pstibrany! I left few minor comments.

Would you be up to add a integration test with single-binary Cortex and gossip to (hopefully) freeze the current implementation from future regressions?

[pstibrany]

Would you be up to add a integration test with single-binary Cortex and gossip to (hopefully) freeze the current implementation from future regressions?

Added integration test that verifies that Cortex in single-binary mode works using memberlist setup.

[pracucci]

LGTM

I've found out this test is quite slow due to the table-manager stop. Look at the timestamps (I've enabled info log level):

cortex-1: level=info ts=2020-02-12T15:50:19.5615697Z caller=cortex.go:281 msg=stopping module=table-manager
cortex-1: level=info ts=2020-02-12T15:50:36.8263139Z caller=cortex.go:281 msg=stopping module=runtime-config

This is caused by:

cortex/pkg/chunk/table_manager.go

Line 180 in 58ef607

time.Sleep(time.Duration(rand.Int63n(int64(m.cfg.DynamoDBPollInterval))))

Would you mind fixing it in a separate PR, please? I should be just a matter of changing the table manager initial time.Sleep() into a select which returns if m.done is closed.