[Core] Add option to avoid uploading credentials and custom labels for GCP

[Michaelvll]

This is to avoid uploading credentials of the current cloud to be launched, as we can use the service account assigned to the instance to access the buckets and cloud APIs.

Fixes #2820

To enable this feature, a user should specify the following config in the ~/.sky/config.yaml, and this only supports AWS and GCP for now.

aws:
  remote_identity: SERVICE_ACCOUNT

gcp:
  remote_identity: SERVICE_ACCOUNT

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
  • sky launch --cloud gcp --cpus 2 -i 0 echo hi
  • sky launch --cloud gcp --cpus 2 -i 0 echo hi with the following config.yaml

    gcp:
      remote_identity: SERVICE_ACCOUNT
      instance_tags:
        owner: zhwu
  • sky launch --cloud aws --cpus 2 -i 0 echo hi with the following config.yaml

    aws:
      remote_identity: SERVICE_ACCOUNT
      instance_tags:
        owner: zhwu
  • sky launch --cloud gcp --cpus 2 -i 0 --num-nodes 2 echo hi
• All smoke tests: pytest tests/test_smoke.py --gcp
• All smoke tests: pytest tests/test_smoke.py --aws
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
  • pytest tests/test_smoke.py::test_file_mounts --aws with two private buckets (1 aws, 1 gcp) enabled in using_file_mounts.yaml.
  • pytest tests/test_smoke.py::test_file_mounts --gcp with two private buckets (1 aws, 1 gcp) enabled in using_file_mounts.yaml.
• Backward compatibility tests: bash tests/backward_comaptibility_tests.sh

[concretevitamin]

Thanks @Michaelvll, did a pass and some questions.

[concretevitamin]

If it's always a singleton, any reason to make the arg a list?

[Michaelvll]

It was for the generality of the function, as in the future it might be able to allow the remote instance to not have multiple clouds' credential, e.g., if the image has the cloud credential set already. I am ok to make it singleton first, but I guess it does not matter that much to have an argument as a list. : )

[concretevitamin]

Actually a question: Do we want users to toggle this setting vs. always defaulting to use a service account for clouds that support it?

[Michaelvll]

Actually a question: Do we want users to toggle this setting vs. always defaulting to use a service account for clouds that support it?

I think it might be better to ask the user to specify the config explicitly to turn this on. Reasons:

1. If SERVICE_ACCOUNTS is turned on by default, any managed spot jobs/service replica launched on the clouds different from the controller will not be able to access the resources (e.g., the buckets) on the cloud as the controller. (This might be possible to solve by having some special handling for the controller in our code to still upload those credentials to the controllers)
2. The service accounts can have different permissions than the user account, it might be quite suprising if users can access a bucket locally while the remote instance fails to access it.
3. Considering this feature is requested by some of our users, it might be easier and more convenient for most of the others to have the same permission on the remote instances created.

Wdyt?

[concretevitamin]

For (2) & (3): Is it true that our main branch currently uses service accounts for AWS/GCP remote nodes? IIRC service accounts take precedence over the credential files present on those nodes in identity discovery.

If this is true and the permission issue hasn't caused any troubles, maybe it's fine. #2820 mentioned some permission mismatch which I don't fully understand.

For (1): I think there are two cases

• single-cloud users
• cloud X controller (spot or serve) to launch clusters on cloud Y

Since former is more common, we probably should minimize their config overhead and default to the no-upload behavior if we can make it work. Once the user wants the latter behavior, they should be able to set this flag and have the (maybe an existing) controller use the upload behavior for new jobs.

---

Relatedly we need to test the AWS_PROFILE=xx sky launch case in #2820.

[concretevitamin]

Thanks, some minor nits.

[concretevitamin]

Indeed the boundary is subjective. Here I think we want to have a minimal cognitive load for readers that browse and learn about the interface. If that reader is a new cloud implementor or someone who wants to know "what does it take to implement a Cloud", I think it's best to leave out such one-liner utility funcs.

[concretevitamin]

LGTM, thanks @Michaelvll!

[concretevitamin]

(In the future we can put utils in this init module too)