CAPI waiting forever for the volume to be detached

[furkatgofurov7]

What steps did you take and what happened:

We have a use case where we are running a daemon set that mounts a volume. When draining the node, CAPI does not touch the daemon sets. Due to #4945 CAPI waits for the volumes to be detached and that is not an issue normally since all pods are deleted when draining. But since volumes are attached to the daemon set, they are never unmounted because the pod keeps running, which results in CAPI waiting forever for the volumes to be detached.

What did you expect to happen:
Draining DS and detaching the volume successfully without deadlock

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api version: v0.4..4
• Minikube/KIND version: NA
• Kubernetes version: (use kubectl version): 1.23.3
• OS (e.g. from /etc/os-release): SLES 15SP2

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

[furkatgofurov7]

/cc @sbueringer @fabriziopandini @vincepri

[sbueringer]

I think this happens because we use ignore all daemonsets: https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/machine/machine_controller.go#L532

I don't think we can just change the hard-coded value to false, maybe we should make it configurable.

[enxebre]

This is intrinsic to DaemonSets, even if the property was false, drain would just be blocked but it won't graceful terminate the pod since it'd be automatically recreated by the DaemonSet.
To fully remove the Pod from that Node you could leverage labels/NodeSelectors so this might come in handy #6255 for this scenario.

[maelk]

What about making it configurable whether we wait for the volumes to be all detached or not. Would it be problematic to delete a node where a daemonset still has a volume attached ? Once the node is deleted, the pod will be deleted and the pvc too. Would this be too risky ?

[sbueringer]

I think this probably depends on the infrastructure provider. I saw some weird issues in cases like this with OpenStack. But this might be due to that specific OpenStack.

In the OpenStack case the CSI pod running on that node would be responsible to unmount etc. (I don't remember the details). Not sure how safe this is in general and how graceful this will be handled by CSI and the infrastructure if a node/server is just deleted with attached volumes.

[lentzi90]

I think this old issue kubernetes/kubernetes#54368 (comment) could be relevant here. It is about StatefulSets, but a DaemonSet with volumes is coming quite close.

From a safety perspective I guess the best thing would be to shutdown the node but then we are already in infrastructure provider territory.

[furkatgofurov7]

The draining of Daemonsets had been discussed in kubernetes/kubernetes#75482 upstream k8s repo issue for a long time apparently but it has always been postponed (first been milestoned to v1.16) up until now.
On the other hand, there is #6158 which is similar to what we have been discussing here.

[enxebre]

You can use .NodeDrainTimeout to bypass draining after a timeout, the machine controller will proceed to delete the Node even though it has volumes attached. That would have the pertinent implications in each particular platform.

As for what cluster autoscaler does, we could consider doing something similar i.e taint deleting Nodes and evict DS pods manually. I'd question how strong is the use case as to support this opinionatedly vs enabling a consumer to do it as in #6285 (comment)

To get more context can you please elaborate your use case for running a DS with a volume?
Also wil

[maelk]

We have the node drain timeout, but CAPI is still blocking on the volumes even after the timeout has expired. Maybe that could be modified so that we don't wait for the volumes to be detached after the timeout has expired ?

Regarding the use-case, we don't have much info. It is a customer using those and they did not give us any specific information.

Extending the timeout to the volumes would be acceptable solution for us.

[enxebre]

We have the node drain timeout, but CAPI is still blocking on the volumes even after the timeout has expired.

@maelk By looking a the code I'd be surprised if that's the case, can you validate/confirm share logs?
https://github.com/kubernetes-sigs/cluster-api/blob/main/internal/controllers/machine/machine_controller.go#L309-L353

[fabriziopandini]

/milestone v1.2

[enxebre]

/priority awaiting-more-evidence
to hear from #6285 (comment)