Add compactor auto-forget from ring on unhealthy #6563
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Ben Ye <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
yeya24
reviewed
Jan 28, 2025
pkg/ring/lifecycler.go
Outdated
| type LifecyclerDelegate interface { | ||
| // OnRingInstanceHeartbeat is called while the instance is updating its heartbeat | ||
| // in the ring. | ||
| OnRingInstanceHeartbeat(lifecycler *Lifecycler, ctx context.Context) |
There was a problem hiding this comment.
I'd love to see that the same function parameter to be reused. The current interface is a bit weird and we are not using the lifecycler at all in the compactor implementation
yeya24
reviewed
Jan 28, 2025
pkg/compactor/lifecycle.go
Outdated
| "github.com/cortexproject/cortex/pkg/ring" | ||
| ) | ||
|
|
||
| func (c *Compactor) OnRingInstanceHeartbeat(_ *ring.Lifecycler, ctx context.Context) { |
There was a problem hiding this comment.
I would do similar to the basicLifecycler.
Create the generic delegate for the ring and make compactor use it.
…LifecyclerDelegate Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
yeya24
reviewed
Jan 30, 2025
pkg/ring/lifecycler.go
Outdated
| instanceDesc.Zone = i.Zone | ||
| instanceDesc.RegisteredTimestamp = i.getRegisteredAt().Unix() | ||
| ringDesc.Ingesters[i.ID] = instanceDesc | ||
| i.delegate.OnRingInstanceHeartbeat(i, ringDesc) |
There was a problem hiding this comment.
We should move this line outside the else block. We should call OnRingInstanceHeartbeat no matter the instance exists in the ring or not. Same behavior as UpdateInstance in the basic lifecycler
pkg/compactor/compactor.go
Outdated
| f.BoolVar(&cfg.SkipBlocksWithOutOfOrderChunksEnabled, "compactor.skip-blocks-with-out-of-order-chunks-enabled", false, "When enabled, mark blocks containing index with out-of-order chunks for no compact instead of halting the compaction.") | ||
| f.IntVar(&cfg.BlockFilesConcurrency, "compactor.block-files-concurrency", 10, "Number of goroutines to use when fetching/uploading block files from object storage.") | ||
| f.IntVar(&cfg.BlocksFetchConcurrency, "compactor.blocks-fetch-concurrency", 3, "Number of goroutines to use when fetching blocks from object storage when compacting.") | ||
| f.DurationVar(&cfg.AutoForgetDelay, "compactor.auto-forget-delay", 15*time.Minute, "Time since last heartbeat before compactor will be removed from ring. 0 to disable") |
There was a problem hiding this comment.
In ruler, alertmanager and sg, we dont have a config for this. We usually just do 2x heartbeat timeout.
I am fine adding a new config, but lets add the default as 2*heartbeat_timeout.
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
|
Test failure seems related. |
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
Signed-off-by: Essam Eldaly <[email protected]>
yeya24
approved these changes
Feb 13, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
auto_forget_delayfor ring to auto forget unhealthy compactors.OnRingInstanceHeartbeatWhich issue(s) this PR fixes:
Fixes #6533
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]