[New Provisioner] New provisioner for k8s

[Michaelvll]

It supports all the functionality of the original k8s support, and also supports custom image (adopted from #2599)

Special thanks to @landscapepainter for the excellent contribution to the custom image support PR #2599, which formulates this PR.
Co-authored-by: @landscapepainter

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
  • pytest tests/test_smoke.py --kubernetes -k "not TestStorageWithCredentials"
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[romilbhardwaj]

@Michaelvll I've pushed some changes to:

1. Removed the python3 and python3-pip installation during setup. This was unnecessary since we install conda anyway.
2. Added a simple kubernetes_handler under FailoverCloudErrorHandlerV2 to surface any Kubernetes provisioning errors to the suer.

Also ran some quick benchmarks to test provisioning speed compared to current master (Ray autoscaler based provisioning) using the default SkyPilot image:

#==== New provisioner ====
# Single node
time sky launch -y -c test -- echo hi # 1:11.45 total
# Multi node
time sky launch -y -c test --num-nodes 2 -- echo hi # 1:21.14 total


#==== Old Ray provisioner ====
# Single node
time sky launch -y -c test -- echo hi # 1:08.87 total
# Multi node
time sky launch -y -c test --num-nodes 2 -- echo hi # 1:40.66 total

I guess some slowdown is expected because of custom image support, which runs additional setup steps (running apt update etc), which negates the speed ups from using the new provisioner. Overall, provisioning time is roughly the same for single node, and we see some improvement for multi-node.

Running smoke tests now.

[romilbhardwaj]

Smoke tests (both Kubernetes and general) pass! LGTM.

[romilbhardwaj]

Updated docker images to 1) use the same base image as SkyPilot official docker image continuumio/miniconda3:23.3.1-0, 2) Remove unnecessary dependencies.

Tested with simple sky launch, running k8s smoke tests now.

Also tested backward compatibility (i.e., new image also works with the old provisioner).

Update - smoke tests pass, I have updated the image at us-central1-docker.pkg.dev/skypilot-375900/skypilotk8s/skypilot[-gpu]:latest.