Clean up gossip and remove `experimental` flag from gossiped ring method in docs and config

[mattmendick]

Gossiped Ring works well, and so we should remove the experimental flag from the docs and config comments. In addition, we should do a few polishing steps:

• investigate error messages about being unable to ping, but tcp working. They seem harmless. Document their harmlessness or silence them if possible.
• Randomize node names used in gossip. This fixes problems with stateful sets reusing node names. "Problems" are that new nodes with old names are ignored for short period of time, which produces log messages, but they will eventually (after 30s in default config) connect.
• Periodic reconnect to cluster -- may help in situations after networking problems, or in misconfigured rollout.

[mattmendick]

cc @pstibrany

[pstibrany]

One more thing:

• KV is passed to memberlist.Create as a delegate before it is fully constructed, which can lead to race conditions

[pstibrany]

investigate error messages about being unable to ping, but tcp working. They seem harmless. Document their harmlessness or silence them if possible.

Error message says: Was able to connect to %s but other probes failed, network may be misconfigured. "Was able to connect to" refers to direct TCP connection, while "other probes" are direct (in our case via TCP connection) or indirect probes (via other memberlist nodes).

From reading the code, during the probeNode memberlist tries to:

1. send direct message (which uses TCP in our setup) to the node. If that fails within the ProbeTimeout (defaults to 500 ms) then
2. send indirect probes via other nodes + fallback ping via TCP connection to the node

If no ping response comes from other nodes in ProbeInterval (defaults to 1s, but can be higher depending on node health score, but timeout starts running at the beginning of probeNode, so it's effectively ProbeInterval - ProbeTimeout), and the only response comes from node itself via TCP connection, memberlist logs the above error message.

I cannot reproduce it now, although I've seen it previously even when running integration tests. Note that we have recently updated memberlist library (indirectly, through updating Thanos), so some details may have changed. Let's see if we see it again, or if people complain.