Fix Azure client requests stuck issues on http.StatusTooManyRequests

[feiskyer]

In go-autorest SDK https://github.com/Azure/go-autorest/blob/master/autorest/sender.go#L242,
if ARM returns http.StatusTooManyRequests, the sender doesn't increase the retry attempt count,
hence the Azure clients will keep retrying forever until it gets a status code other than 429.

// don't count a 429 against the number of attempts
// so that we continue to retry until it succeeds
if resp == nil || resp.StatusCode != http.StatusTooManyRequests {
    attempt++
}

So this PR explicitly remove http.StatusTooManyRequests from autorest.StatusCodesForRetry. To reduce the API throttling issues, it also adds caches for vmss instances.

Fix #2124.

/cc @andyzhangx @nilo19

[k8s-ci-robot]

@feiskyer: GitHub didn't allow me to request PR reviews from the following users: nilo19.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

In go-autorest SDK https://github.com/Azure/go-autorest/blob/master/autorest/sender.go#L242,
if ARM returns http.StatusTooManyRequests, the sender doesn't increase the retry attempt count,
hence the Azure clients will keep retrying forever until it gets a status code other than 429.

  			// don't count a 429 against the number of attempts
  			// so that we continue to retry until it succeeds
  			if resp == nil || resp.StatusCode != http.StatusTooManyRequests {
  				attempt++
  			}

So this PR explicitly remove http.StatusTooManyRequests from autorest.StatusCodesForRetry. To reduce the API throttling issues, it also adds caches for vmss instances.

Fix #2124.

/cc @andyzhangx @nilo19

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[andyzhangx]

@feiskyer In azure cloud provider, we would still retry after hit 409, since it's exponential backoff, is it safe in azure cloud provider code?

		// HTTP 4xx (except 412) or 5xx suggests we should retry.
		if 399 < resp.StatusCode && resp.StatusCode < 600 {
			return true
		}
``

[feiskyer]

@feiskyer In azure cloud provider, we would still retry after hit 409, since it's exponential backoff, is it safe in azure cloud provider code?

Yep, it's safe from SDK's client side. CA doesn't need backoff retry since it would run again later. The issue is from go-autorest internal implementations, we should also add some fixes in the Kubernetes cloud provider.

By the way, the error code here is 429, not 409.

[andyzhangx]

/lgtm

[feiskyer]

Rebased to solve the conflicts

[feiskyer]

@andyzhangx @losipiuk Could you help to approve the changes?

[andyzhangx]

/lgtm

[losipiuk]

/lgtm
/approve
Thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: losipiuk

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cluster-autoscaler/cloudprovider/azure/OWNERS [losipiuk]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment