Fix TestAlertmanager_StateReplicationWithSharding. #4078
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The test had two issues: 1. The most common issue was that replicating the silence sometimes took a small amount of time, as it happens asynchronously. Replacing Equal with Eventually when checking the replication metrics is enough to make the tests reliable. 2. The second issue was more rare: When a particular shard does not have any users assigned to it, _and_ the test randomly picks _that_ shard to write the silence to, the test would panic (in rand.Intn). Due to the alertmanagersMtx being held, the test actually just hung up while trying to shutdown the alertmanagers. Signed-off-by: Steve Simpson <[email protected]>
gotjosh
reviewed
Apr 14, 2021
Signed-off-by: Steve Simpson <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The test had two issues:
The most common issue was that replicating the silence sometimes
took a small amount of time, as it happens asynchronously. Replacing
Equal with Eventually when checking the replication metrics is enough
to make the tests reliable.
The second issue was more rare: When a particular shard does not have
any users assigned to it, and the test randomly picks that shard
to write the silence to, the test would panic (in rand.Intn). Due to
the alertmanagersMtx being held, the test actually just hung up while
trying to shutdown the alertmanagers.
Signed-off-by: Steve Simpson [email protected]