Faster rolling upgrades

[tomwilkie]

Currently rolling upgrades involve flushing all in memory chunks, which takes ages - and as we get more users, will get worse.

Idea would be to have new ingesters come up in a "joining" state, which is then "claimed" by one of the shutting down ingesters, all in-memory content streamed to it, then the leaving ingesters tokens are claimed by the joining ingester.

Sequence would look like this:

Running/exiting ingester:

1. Is signalled to exit.
2. Enters "Leaving" state.
3. Waits a fixed amount of time for an ingester to appear in "Joining" state.
   • if no ingester found, continues to flush all its state to chunk store
4. Does RPC to Joining ingester, streaming all its chunks.
5. When streaming RPC is done, waits for other ingester to claim its tokens, then removes itself from the ring and exits.

New ingester:

1. Joins ring on startup.
2. Enters "Joining" state - but does not add any tokens to the ring.
3. Waits a fixed amount of time to be contacted by a "Leaving" ingester .
   • after timeout, add tokens to ring and enters normal state.
4. When it receives a streaming RPC from ingester, "locks" itself to that ingester.
   • puts all chunks it received in memory
5. When all chunks are received, claims leaving ingesters tokens.

The timeouts exist so that new ingesters added for the sake of scaling don't wait for ever. I would envisage they would be quite small.

If the RPC at step 4 fails, both ingesters go back to step 3.

WDYT?

[jml]

Idea would be to have new ingesters come up in a "joining" state, which is then "claimed" by one of the shutting down ingesters, all in-memory content streamed to it, then the leaving ingesters tokens are claimed by the joining ingester.

This sounds like the way the do it in the Dynamo paper?

[juliusv]

Sounds good to me. More complexity, but probably worth it to have more agility and chances to fix bad ingester flushing code than currently.

[tomwilkie]

This sounds like the way the do it in the Dynamo paper?

Yes, its a little like that. They do it by splitting ranges and transferring the split ranges (when adding new peers) and doing the opposite when removing peers.

[jml]

Got an estimate of how much faster this will be?

Does RPC to Joining ingester, streaming all its chunks.

If a joining ingester appears while this is happening, then tough luck?

When streaming RPC is done, waits for other ingester to claim its tokens, then removes itself from the ring and exits.

What happens if the other ingester fails to claim tokens?

"locks" itself to that ingester

What's the mechanism for this? Just some in-memory state?

If the RPC at step 4 fails, both ingesters go back to step 3.

Am not super familiar w/ streaming RPC mechanics—got a link?

Would the Joining ingester discard all its received chunks on failure? Is there any sort of timeout for this?

Are there circumstances where Joiner thinks everything has gone OK but Leaver thinks something went wrong? & vice versa.

[Joiner] claims leaving ingesters tokens

Can the new ingester atomically do that, or does it need to coordinate with leaver? Am concerned about a battle over tokens. Looking at https://github.com/weaveworks/cortex/blob/master/ring/ingester_lifecycle.go, though, things are OK, since unregistering means stopping the loop that would overwrite tokens.

[tomwilkie]

Got an estimate of how much faster this will be?

Lets say it takes ~10mins to flush an ingester, when it works. Theres about 200k chunks per ingester in dev. So we can achieve about 300 chunks/s, or 300KB/s

I'd be surprised it we can't transfer at at least ~10MB/s, which would get this done in 20s.

If a joining ingester appears while this is happening, then tough luck?

They will timeout waiting for an ingester to "claim" them, as continue as them do today (add tokens to the ring and start serving).

What happens if the other ingester fails to claim tokens?

The leaving ingester should timeout, and go back to looking for a new ingester to stream to.

What's the mechanism for this? Just some in-memory state?

Yeah, just some in memory state.

Am not super familiar w/ streaming RPC mechanics—got a link?

http://www.grpc.io/docs/tutorials/basic/go.html#client-side-streaming-rpc

Would the Joining ingester discard all its received chunks on failure? Is there any sort of timeout for this?

Yes, thats the idea. Probably want a timeout too.

Are there circumstances where Joiner thinks everything has gone OK but Leaver thinks something went wrong? & vice versa.

As we're running this all through consul, I guess a failure there? Also, during a partition, both sides would reset back to (2), timeout and then proceed as they do today.

Can the new ingester atomically do that, or does it need to coordinate with leaver?

It should be fine, as its done in a CAS in consul. So it there is a race, only one side will win.

[jml]

If a joining ingester appears while this is happening, then tough luck?

They will timeout waiting for an ingester to "claim" them, as continue as them do today (add tokens to the ring and start serving).

I guess there's a potential optimization for a Leaver to notice that a Joiner has arrived while flushing-on-exit, and switch instead to streaming to the Joiner. Even as I type this it sounds way too complicated.

Would the Joining ingester discard all its received chunks on failure? Is there any sort of timeout for this?

Yes, thats the idea.

In many (most?) circumstances, the retry is going to be between the same pairs of nodes. Seems like some sort of resume functionality would be useful. Not for the first pass though.