node-role.kubernetes.io/ in labels leads to errors for k8s>=1.16

[scottyhq]

What happened?
Under eksctl nodeGroups config a label with node-role.kubernetes.io as described here #582 leads to new nodegroup creation hanging on [ℹ] waiting for at least 1 node(s) to become ready in

    labels:
      node-role.kubernetes.io/user: user

What you expected to happen?
An error message saying node-role.kubernetes.io/ is not a permitted label for kubernetes versions>=1.16 as described here kubernetes/kubernetes#85073

How to reproduce it?
See config above

Anything else we need to know?
The use case for this was to easily see node roles in the output of kubectl get nodes A workaround is just to use a generic label:

    labels:
      role: user

The use a flag kubectl get nodes -L role

I suspect there may be other open issues that this will fix (#2124). This was really annoying to debug since there weren't really any informative error messages or warnings, just nodes visable in ec2 console but not via kubectl. Adding some logic to check for disallowed labels would be nice.

Versions
Please paste in the output of these commands:

$ eksctl version 1.19
$ kubectl version 1.16

[TBBle]

I think this bug has been mis-resolved, or we've lost an important feature. The intention from kubernetes/kubernetes#84912 (comment) (referenced from kubernetes/kubernetes#85073) is that kubelet should not be self-selecting or changing it's role, as that would allow arbitrary kubelets to access and subvert privileged or restricted workloads.

See also the use-case description in the original proposal, which lets a user add arbitrary labels to the 'kubelet may not set these' list using the node-restriction.kubernetes.io/ prefix (which has a similar restriction and cannot be self-applied by kubelets).

However, the use-case for eksctl's config is the eksctl user selecting the role for the nodegroup, which is described as the desired usecase in the linked issue.

The workaround for this change is that I have to use a locally-defined taint/label to controlling nodegroup allocation for pods, and I cannot add that to the node-restriction.kubernetes.io/ label, so I do not have the security that the node-role has now gained. So I have to choose between secure node-roles driven, or adding a label to every new node manually whenever one is created by the Cluster Autoscaler. Clearly the latter defeats the point of the autoscaler, so I'm stuck with the former.

I think the correct solution (and this may need support from the EKS API or backend?) is that the label is applied not via kubelet --node-labels (the failure case) but by some other system, presumably a Node Admission plugin or mutating webhook.

For comparison, kubeadm applies the node-role.kubernetes.io/master label and taint as a separate step immediately after node creation.

I haven't tested with a 1.16 cluster, but do ManagedNodeGroups also fail to set the node-role.kubernetes.io/ label?

[michaelbeaumont]

@cPu1 @martina-if What's the advantage of using --node-labels at all? They're also only effective at registration so can't be used for something like eksctl apply or #830

[TBBle]

The major issue is that we need to apply the labels in a way that works with auto-scaled nodes, which themselves (deliberately) do not have permission to apply these labels, and eksctl isn't involved so cannot apply them as a post-process like kubeadm can.

I suspect that, perhaps for non-kubelet-labels, we'd need something running in the cluster to apply labels based on... AWS Tags, perhaps? That would also make those labels more-visible (and perhaps editable) in the AWS console as Tags rather than buried in the user-data blob. That seems like something the CloudProvider should support (since it creates the Node object in the first place), but perhaps it could be knocked up as a third-party controller or admission webhook in the meantime.

metal3-io/metal3-dev-env#205 (comment) has just hit upon the same problem, and the idea of pushing it up to a controller too.

[TBBle]

I've raised a feature request on the AWS Cloud Provider to support labelling nodes from AWS Tags as they are created. That's a long-turnaround solution though.