[alerts] add alert when autoscaler adds nodes rapidly #10016
Conversation
| summary: "Autoscaler is adding new nodes rapidly", | ||
| description: 'Autoscaler in cluster {{ $labels.cluster }} is rapidly adding new nodes.', | ||
| }, | ||
| expr: '((sum(cluster_autoscaler_nodes_count) by (cluster)) - (sum(cluster_autoscaler_nodes_count offset 10m) by (cluster))) > 10', |
There was a problem hiding this comment.
This would have fired a lot recently, are all valid? Seems like a lot.
There was a problem hiding this comment.
Hmmm.
We do seem to scale them up quite aggressively, not sure if this is normal or not:
grafana link
I can tweak alert to be less aggressive, so to catch only really large spikes like the one May 11th. wdyt?
There was a problem hiding this comment.
Yes, I think that would be a good skateboard. I'm just trying to avoid sending unnecessary pages knowing that we might have to tweak this again in the future.
15 may be a good number., reduces to 2 incidents.
| labels: { | ||
| severity: 'critical', | ||
| }, | ||
| 'for': '1m', |
There was a problem hiding this comment.
Why is duration needed in this condition?
@ArthurSens wdyt?
There was a problem hiding this comment.
technically not needed.
There was a problem hiding this comment.
Cool, let's see if we can remove? You can check to see if it passes validation.
|
/lgtm |
roboquat
added
deployed: workspace
Description
Related Issue(s)
Fixes #9946
How to test
Test alert expression in grafana
Release Notes
Documentation