Removed ingesters blocks transfer support #2996
Conversation
| Similarly to the blocks storage, when Cortex is running the chunks storage with WAL enabled, it requires ingesters to run with a persistent disk where the WAL is stored (eg. a StatefulSet when deployed on Kubernetes). | ||
|
|
||
| During a rolling update, the leaving ingester closes the WAL, synchronize the data to disk (`fsync`) and releases the disk resources. | ||
| The new ingester, which is expected to reuse the same disk of the leaving one, will replay the WAL on startup in order to load back in memory the time series that have not been flushed to the storage yet. |
There was a problem hiding this comment.
It actually loads all data into memory, even flushed samples. WAL doesn't preserve "flushed" flag, so data will be possibly flushed twice.
There was a problem hiding this comment.
Right. I'm not sure we should mention the "data will be possibly flushed twice" here, given this page is focused on rolling updates. Maybe it's something we should improve in the "Ingesters with WAL" page. WDYT?
In the meanwhile, I've rephrased it inhttps://github.com//pull/2996/commits/caa7dd453b918cd0674cbc52d999490e3ceebe0b.
There was a problem hiding this comment.
Nit: In that change you have introduce "checkpoint" term, with no explanation. I think we can be bit more vague and just say that ingester reloads data from WAL.
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
pracucci
force-pushed
the
remove-blocks-transfer-support
branch
from
7f2a9c1 to
37dade1
Compare
Signed-off-by: Marco Pracucci <[email protected]>
Signed-off-by: Marco Pracucci <[email protected]>
What this PR does:
As discussed in #2966, this PR removes the blocks transfer support from ingesters.
/cc @codesome
Which issue(s) this PR fixes:
Fixes #2966 and #1828.
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]