Provide a better solution than standard kubectl drain

[Kun483]

What would you like to be added (User Story)?

As a developer, I would like to be able filter out certain pods (certain pods won't be deleted) during the node draining process.

Detailed Description

I used Portworx as a distributed storage for container storage interface (CSI) drivers. When I upgraded my cluster, it caused “repaving” — replacing old nodes in the cluster one by one with new nodes that have the new desired state in place. When worker machines were repaving, CAPI continuously kill px-<cluster-name> pods, which caused CSI plugin never getting re-registered and the node hanging forever. Therefore, cluster upgrades got stuck.

In this case, if we can have a field in machine spec to filter out the pods that we don't want CAPI to delete during the node draining process, then pods like px-<cluster-name> can be re-registered and repaving will be done successfully. As discussion, we may need to not delete pods that have such toleration.

  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists

var (
       unreachableToleration = corev1.Toleration{
		Key:      nodeUnreachableKey,
		Effect:   corev1.TaintEffectNoSchedule,
		Operator: corev1.TolerationOpExists,
	}
)

drainer := []kubedrain.PodFilter{
	SkipFuncGenerator(m.Spec.NodeDrainPodFilters),
},

func skipUnreachableTolerationPods(pod corev1.Pod) kubedrain.PodDeleteStatus {
	if pod.Spec.Tolerations == nil {
		return kubedrain.MakePodDeleteStatusOkay()
	}
	if HasTolerations(&pod, &unreachableToleration) {
		return kubedrain.MakePodDeleteStatusSkip()
	}
	return kubedrain.MakePodDeleteStatusOkay()
}

With helper function

func HasTolerations(pod *corev1.Pod, toleration *corev1.Toleration) bool {
	for _, t := range pod.Spec.Tolerations {
		if t.MatchToleration(toleration) {
			return true
		}
	}
	return false
}

We may add a field called NodeDrainPodFilters in MachineSpec. We can also add this field in KubeadmControlPlaneTemplateMachineTemplate struct

NodeDrainPodFilters *metav1.LabelSelector json:"nodeDrainPodFilters,omitempty"

Anything else you would like to add?

No response

Label(s) to be applied

/kind feature

[sbueringer]

Q: Shouldn't pods like this not be deployed via a DaemonSet?

[kreeuwijk]

Q: Shouldn't pods like this not be deployed via a DaemonSet?

A: That's not how Portworx does it I'm afraid. We have no control over that. Portworx is deployed through an operator and there is no daemonset for the main portworx worker pods. Instead the operator creates these pods itself on nodes it deems that need to participate in the Portworx storagecluster. The portworx-api pods are deployed as a daemonset by the operator though.

[sbueringer]

So that means that portworx clusters are in general not compatible with kubectl drain? (because we are just doing the same)

[sadysnaat]

I fear that's true @sbueringer, portworx will fail regular drain.

@kreeuwijk is there any guide around node drain options, maybe portworx has some advanced options.

But regardless of how portworx works. @sbueringer will it make sense to have pod filters functionality added to cluster api as it is part of kubectl drain cli

[kreeuwijk]

No no, Portworx is perfectly fine with normal node drains. It's just the CAPI-based node drain that is a problem, as capi-controller will keep killing any and all non-daemonset workloads that get started on the drained node.

Portworx expects to be able to restart the portworx worker pod on a drained node. That pod has tolerations to allow it to come back up on an already drained node. It also just connects to the "real" systemd portworx service that runs on the host OS, so loss of the worker pod does not automatically mean loss of the storage service. However, when capi-controller keeps killing the portworx worker pod, it causes the CSI registration for that node to go into a bad state, which then causes PVC attachment issues for specific workloads, most notable Kubevirt VMs that are trying to live-migrate away to a different node.

Hence, we need to be able to inform the CAPI node drainer to purposely ignore pods with the name: portworx label, so that the pod is left alone and can restart on the node a few moments after the node drain.

Portworx is not going to change how their product works for this, in their eyes the problem is with CAPI, not with them.

[sbueringer]

Wondering if we should just skip evicting Pods that have that toleration

Heads up. I'm planning to refactor the whole drain logic in the next few weeks. Basically it's just a bad idea to use the drain helper in a controller

[kreeuwijk]

Wondering if we should just skip evicting Pods that have that toleration

That's not a bad idea. For reference, these are the tolerations on the pod in question:

  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/disk-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/memory-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/pid-pressure
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists
  - effect: NoSchedule
    key: node.kubernetes.io/network-unavailable
    operator: Exists

[sbueringer]

Yeah that should be the toleration for the taint that is set by cordon

  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists

The reasoning would be something like: Why should we drain/evict Pods that tolerate running on cordon'ed Nodes and thus can and will be simply re-scheduled on the same Node.

This would also solve other problems if folks are adding this toleration to their Pods and it wouldn't require any configuration / API change.

[sadysnaat]

Thanks @kreeuwijk @sbueringer

Yeah that would be neat idea.

[Kun483]

Based on the discussion, I made a new change in issue description to introduce the AdditionalFilters to filter out pods that have such toleration.

  - effect: NoSchedule
    key: node.kubernetes.io/unschedulable
    operator: Exists