clusterapi provider for cluster autoscaler should expose labels and taints when scaling from zero

[elmiko]

User Story

As a user I would like to ensure that the cluster autoscaler will use the labels and taints associated with my cluster api scalable resources (MachineDeployment, MachineSet, etc) when predicting the node shape in scale from zero scenarios so that i can be sure that pod scheduling will occur as expected.

Detailed Description

Currently, the clusterapi provider for the cluster autoscaler does not have a convenient way to expose the labels and taints that will be applied to nodes created from the scalable resources. Part of this is due to the fact that the cluster api community is still coordinating about how these labels and taints will be propagated to the nodes that are created.

Anything else you would like to add:

This will require some coordination with the cluster api community to solve completely, there are a couple pieces of prior art around the labels that should be noted:

• Support syncing a set of labels from MachineDeployment/MachineSet/Machine to Nodes #493
• 📖 Label Sync Between Machine and underlying Kubernetes Nodes #7296
• 📖 Label Sync Between MachineDeployment and underlying Kubernetes Nodes #6255

/kind feature

[fabriziopandini]

/triage accepted
Let's try to nail down this before the work on label propagation starts

[sbueringer]

I'm not sure I understand what the requirement for Cluster API is. This sounds a bit like it's something for the autoscaler "I would like to ensure that the cluster autoscaler will use ". But I assume I'm misunderstanding it :)

What does this mean:

convenient way to expose the labels and annotations that will be applied to nodes created from the scalable resources

Does the autoscaler want to set labels on nodes via Cluster API resources?

Or is it about mapping the labels Cluster API will apply to nodes based on Cluster API resources to the labels which are found on the nodes by the autoscaler?

[elmiko]

Or is it about mapping the labels Cluster API will apply to nodes based on Cluster API resources to the labels which are found on the nodes by the autoscaler?

it's about this, we need to agree on a way to expose those labels and annotations (although i think we already agreed on a method for the labels).

essentially, when the autoscaler is attempting a prediction for a node group with zero replicas, it needs a way to understand what the labels and annotations will be present on the nodes created from the node group. when there are replicas >= 1, then the autoscaler can inspect an actual node for this information. it uses this information to know whether the pending pods will actually be scheduled on any node that it can create.

[elmiko]

@cnmcavoy has proposed a novel solution in kubernetes/autoscaler#5382

[elmiko]

updated this to talk about labels and taints, i don't think we need the annotations from the nodes.

[sbueringer]

Thx, I think now I've got it.

I think the situation is the following:

• today:
  • with the CABPK provider labels & taints on Nodes can be added via KubeadmConfig(Template)
  • there is no standard across bootstrap providers
• after https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220927-label-sync-between-machine-and-nodes.md is implemented (✨ Add Node managed labels support #7173) additionally certain labels from MD/MS are synced to nodes (here's the list: https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220927-label-sync-between-machine-and-nodes.md#label-domains--prefixes)

P.S. also answered on kubernetes/autoscaler#5382

[k8s-triage-robot]

This issue has not been updated in over 1 year, and should be re-triaged.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted