Ingester to wait for some time before joining gossip based ring

[codesome]

This issue is particularly of interest when the ingester names remain the same across rollout/restarts, which is going to be the thing when using WAL with statefulsets.

In my tests I have found out that as the gossip ring takes some time to propagate its changes across all pods, if an ingester terminates and starts quickly enough, it picks can pick up the old LEAVING state from the ring and cause issues (just stuck).

One possible (hacky) solution can be to wait before contacting the ring for it to settle (which is what I have in place in my tests for now).

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[codesome]

Keepalive? Or does the handling of LEAVING state in #1921 is all that we need?
/cc @pstibrany

[pstibrany]

Yes, I think #1921 helps here. Also #2127 should help – on shutdown, memberlist KV now waits a little to push out all messages.

We however still have different problem related to reusing same names. I'll make a new issue for it.

[pstibrany]

We however still have different problem related to reusing same names. I'll make a new issue for it.

Done: #2130

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.