Pods stuck in ContainerCreating after cluster autoscaling (AWS CNI race condition) #247
Description
Description
See:
- Race condition between CNI plugin install and aws-k8s-agent startup aws/amazon-vpc-cni-k8s#282
- ENI warming is delayed for at least for 1 minute, probably caused by #480 aws/amazon-vpc-cni-k8s#525
- Kubelet will start scheduling pods before amazon-vpc-cni-k8s daemon set is fully functional causing workloads to error with "failed to assign an IP address to container" aws/amazon-vpc-cni-k8s#330
To reproduce
- Run cluster with two t3.mediums and cluster autoscaling enabled
- Deploy iris and let run to completion
- Deploy pipelines/iris
This will trigger cluster autoscaling, and once the new node has joined the cluster, the pending Spark job will get scheduled but will be stuck in ContainerCreating