🌱 add preWatch options for leader-elections runnables to reduce failover time when leader changes #2005
Conversation
k8s-ci-robot
added
cncf-cla: yes
|
Hi @nicktming. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: nicktming The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
size/M
|
Hey @mengqiy @varshaprasad96, what do you think? thanks for your suggestions. |
|
Hey @mengqiy @varshaprasad96 , what do you think? |
| @@ -316,6 +319,12 @@ type Runnable interface { | |||
| Start(context.Context) error | |||
| } | |||
|
|
|||
| // runnable with preWatch resources | |||
| type RunnableWithPreWatch interface { | |||
There was a problem hiding this comment.
I don't really understand why this would be needed. We already have the LeaderElectionRunnable interace to start thigns before the leader lock was acquired:
controller-runtime/pkg/manager/manager.go
Line 330 in 8ad090e
Moreover, we start the cache before we start anything else:
controller-runtime/pkg/manager/internal.go
Line 466 in 8ad090e
We don't star the Source itself which might lead to us not getting an informer which then might mean we don't actually start a watch. But changing that is probably a breaking change, as it will lead to increased memory consumption on non-leader instances.
There was a problem hiding this comment.
@alvaroaleman thanks for the discussion。
it has the obvious problem that it will lead to increased memory consumption on non-leader instances, so i provider one option for users who need this feature. users can enable this feature when it is helpful for them.
in our env(big clusters) and when leader changes, it takes too long time to list resources from apiserver, so we want to save failover time by memory consumption.
k8s-ci-robot
added
the
needs-rebase
|
@nicktming: PR needs rebase. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale |
k8s-ci-robot
added
the
lifecycle/stale
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten |
k8s-ci-robot
added
lifecycle/rotten
|
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages PRs according to the following rules:
You can:
Please send feedback to sig-contributor-experience at kubernetes/community. /close |
|
@k8s-triage-robot: Closed this PR. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
What does this do, and why do we need it?
As we know leader-elections runnables only have one working instance, when the leader instance changes, the new leader should watch all resources;
The problem is that watching resources takes long time in the big clusters, and this will have a affect for components who cares about the failover time。
This PR is used to provide a preWatch option for leader-election runnables to preWatch kubernetes resources (it does not handle resource events) for instances which does not acquire leader, when the instance become leader, it starts to process events;