Document the ingester hand-over process

docs/architecture.md

@@ -67,6 +67,8 @@ Ingesters are semi-stateful in that they always retain the last 12 hours worth o
 
 As *semi*-stateful processes, ingesters are *not* designed to be long-term data stores. In Cortex, that role is played by the [chunk store](#chunk-store).
 
+A [hand-over process](ingester-handover.md) manages the state when ingesters are added, removed or replaced.
+
 #### Write de-amplification
 
 Ingesters store the last 12 hours worth of samples in order to perform **write de-amplification**, i.e. batching and compressing samples for the same series and flushing them out to the [chunk store](#chunk-store). Under normal operations, there should be *many* orders of magnitude fewer queries per second (QPS) worth of writes to the chunk store than to the ingesters.

docs/arguments.md

@@ -1,5 +1,7 @@
 # Cortex Arguments Explained
 
+Duration arguments should be specified with a unit like `5s` or `3h`. Valid time units are "ms", "s", "m", "h".
+
 ## Querier
 
 - `-querier.max-concurrent`
@@ -155,6 +157,14 @@ It also talks to a KVStore and has it's own copies of the same flags used by the
 
 ## Ingester
 
+- `-ingester.join-after`
+
+   How long to wait in PENDING state during the [hand-over process](ingester-handover.md). (default 0s)
+
+- `-ingester.ingester.max-transfer-retries`
+
+   How many times a LEAVING ingester tries to find a PENDING ingester during the [hand-over process](ingester-handover.md). Each attempt takes a second or so. (default 10)
+
 - `-ingester.normalise-tokens`
 
    Write out "normalised" tokens to the ring.  Normalised tokens consume less memory to encode and decode; as the ring is unmarshalled regularly, this significantly reduces memory usage of anything that watches the ring.

docs/ingester-handover.md

@@ -0,0 +1,45 @@
+# Ingester Hand-over
+
+The [ingester](architecture.md#ingester) holds several hours of sample
+data in memory. When we want to shut down an ingester, either for
+software version update or to drain a node for maintenance, this data
+must not be discarded.
+
+Each ingester goes through different states in its lifecycle. When
+working normally, the state is `ACTIVE`.
+
+On start-up, an ingester first goes into state `PENDING`. After a
+[short time](arguments.md#ingester), if nothing happens, it adds
+itself to the ring and goes into state ACTIVE.
+
+A running ingester is notified to shut down by Unix signal
+`SIGINT`. On receipt of this signal it goes into state `LEAVING` and
+looks for an ingester in state `PENDING`. If it finds one, that
+ingester goes into state `JOINING` and the leaver transfers all its
+in-memory data over to the joiner. On successful transfer the leaver
+removes itself from the ring and exits and the joiner changes to
+`ACTIVE`, taking over ownership of the leaver's
+[ring tokens](architecture.md#hashing).
+
+If a leaving ingester does not find a pending ingester after [several
+attempts](arguments.md#ingester), it will flush all of its chunks to
+the backing database, then remove itself from the ring and exit. This
+may take tens of minutes to complete.
+
+During hand-over, neither the leaving nor joining ingesters will
+accept new samples. Distributors are aware of this, and "spill" the
+samples to the next ingester in the ring. This creates a set of extra
+"spilled" chunks which will idle out and flush after hand-over is
+complete. The sudden increase in flush queue can be alarming!
+
+The following metrics can be used to observe this process:
+
+ - `cortex_member_ring_tokens_owned` - how many tokens each ingester thinks it owns
+ - `cortex_ring_tokens_owned` - how many tokens each ingester is seen to own by other components
+ - `cortex_ring_member_ownership_percent` same as `cortex_ring_tokens_owned` but expressed as a percentage
+ - `cortex_ring_members` - how many ingesters can be seen in each state, by other components
+ - `cortex_ingester_sent_chunks` - number of chunks sent by leaving ingester
+ - `cortex_ingester_received_chunks` - number of chunks received by joining ingester
+
+You can see the current state of the ring via http browser request to
+`/ring` on a distributor.