Skip to content

[Storage]Mount cached mode #4369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 82 commits into from
Apr 2, 2025
Merged

[Storage]Mount cached mode #4369

merged 82 commits into from
Apr 2, 2025

Conversation

landscapepainter
Copy link
Collaborator

@landscapepainter landscapepainter commented Nov 15, 2024 •
edited by SeungjinYang
Loading

Tested (run the relevant ones):

  • Code formatting: bash format.sh
  • Check if mountpoints remain mounted after restarting by running sky stop and sky start:
    • pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop --aws
    • pytest tests/test_smoke.py::test_gcp_storage_mounts_with_stop --gcp
    • Manually running sky stop and sky start with comp_test.yaml to see if all the MOUNT_CACHED remains storage remains mounted.
  • Set of comprehensive manual tests with comp_test.yaml:
    • launch tests:
      • sky launch comp_test.yaml --cloud aws -y
      • sky launch comp_test.yaml --cloud gcp -y
    • sky launch two instances mounting same backend storage. Create new file on mountpoint on instance 1, and check if it appears shortly in instance 2.
    • managed spot jobs tests:
      • sky jobs launch comp_test.yaml --use-spot --cloud aws -y
      • sky jobs launch comp_test.yaml --use-spot --cloud gcp -y
    • launch on docker containers:
      • sky launch comp_test.yaml --cloud aws --image-id docker:continuumio/miniconda3:latest -y
      • sky launch comp_test.yaml --cloud gcp --image-id docker:continuumio/miniconda3:latest -y

comp_test.yaml:

file_mounts:
  #### S3 tests ####
  /s3-sky-managed-copy:
    name: s3-sky-managed-copy
    source: ~/sky_logs
    store: s3
    mode: COPY

  /s3-sky-managed-mount:
    name: s3-sky-managed-mount
    source: ~/sky_logs
    store: s3
    mode: MOUNT

  /s3-sky-managed-mount-cached:
    name: s3-sky-managed-mount-cached
    source: ~/sky_logs
    store: s3
    mode: MOUNT_CACHED

  /s3-external-private-by-user:
    source: s3://s3-external-private-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-by-user:
    source: s3://s3-external-public-by-user-dy
    mode: MOUNT_CACHED

  /s3-external-public-not-by-user:
    source: s3://digitalcorpora
    mode: MOUNT_CACHED

  #### GCS tests ####
  /gcs-sky-managed-copy:
    name: gcs-sky-managed-copy
    source: ~/sky_logs
    store: gcs
    mode: COPY

  /gcs-sky-managed-mount:
    name: gcs-sky-managed-mount
    source: ~/sky_logs
    store: gcs
    mode: MOUNT

  /gcs-sky-managed-mount-cached:
    name: gcs-sky-managed-mount-cached
    source: ~/sky_logs
    store: gcs
    mode: MOUNT_CACHED

  /gcs-external-private-by-user:
    source: gs://gcs-external-private-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-by-user:
    source: gs://gcs-external-public-by-user-dy
    mode: MOUNT_CACHED

  /gcs-external-public-not-by-user:
    source: gs://gcp-public-data-sentinel-2
    mode: MOUNT_CACHED


workdir: ~/yaml

setup: |
  echo hi

run: |
  # Show workdir
  ls -l .

  # Show private/public storage copy/mount
  ls -l /s3-sky-managed-copy
  ls -l /s3-sky-managed-mount
  ls -l /s3-sky-managed-mount-cached
  ls -l /s3-external-private-by-user
  ls -l /s3-external-public-by-user
  ls -l /s3-external-public-not-by-user
  ls -l /gcs-sky-managed-copy
  ls -l /gcs-sky-managed-mount
  ls -l /gcs-sky-managed-mount-cached
  ls -l /gcs-external-private-by-user
  ls -l /gcs-external-public-by-user
  ls -l /gcs-external-public-not-by-user

  # Write files on a mounted storage with access
  date > /s3-sky-managed-mount/hellotest.txt
  date > /s3-sky-managed-mount-cached/hellotest.txt
  date > /s3-external-private-by-user/hellotest.txt
  date > /s3-external-public-by-user/hellotest.txt
  date > /gcs-sky-managed-mount/hellotest.txt
  date > /gcs-sky-managed-mount-cached/hellotest.txt
  date > /gcs-external-private-by-user/hellotest.txt
  date > /gcs-external-public-by-user/hellotest.txt

  # Confirm file was written
  cat /s3-sky-managed-mount/hellotest.txt
  cat /s3-sky-managed-mount-cached/hellotest.txt
  cat /s3-external-private-by-user/hellotest.txt
  cat /s3-external-public-by-user/hellotest.txt
  cat /gcs-sky-managed-mount/hellotest.txt
  cat /gcs-sky-managed-mount-cached/hellotest.txt
  cat /gcs-external-private-by-user/hellotest.txt
  cat /gcs-external-public-by-user/hellotest.txt

Sheth and others added 30 commits February 16, 2024 23:04
initial commit
7dfcba1
@shethhriday29
Merge branch 'skypilot-org:master' into master
212df29
newline
a08c034
comments
b85cbf9
run linter
4fbfe17
reminder for down
6fc77e1
tentatively done with example
d6cb993
formatting
2d5aceb
yapf
4e1954a
@shethhriday29
Merge branch 'skypilot-org:master' into master
eb22f62
@romilbhardwaj
[Storage] Storage mounting tool permissions fix (skypilot-org#3215)
27a8905 
* fix permissions

* fix permissions
@Michaelvll @concretevitamin
[LLM] Example for Serving Gemma (skypilot-org#3207)
41a63df 
* Add serve for gemma and fix mixtral dependency

* Add hf token

* fix model len

* Add comment

* Serve your private gemma

* fix serve yaml

* readme

* Remove chat completion due to the wrong template

* add readme

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* address comments

* Update README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Update llm/gemma/README.md

Co-authored-by: Zongheng Yang <[email protected]>

* Change to it

* Add chat API

* use HF_TOKEN env

* typo

---------

Co-authored-by: Zongheng Yang <[email protected]>
@Michaelvll
[LLM] Add logo for Gemma (skypilot-org#3220)
2b17e91
@Michaelvll
Minor fixes for release 0.5.0 (skypilot-org#3212)
b326d12 
* when removing cudo credential, sky check fails

* remove tips

* minor hint fix

* fix cluster version for k8s

* fix typo
@Michaelvll
[Docker] Add retry for docker pull due to daemon not ready (skypilot-…
6d77872 
…org#3218)

* Add retry for docker pull due to daemon not ready

* longer wait time

* longer wait time

* retry earlier

* add retry for retries as well

* longer wait time

* change wait time

* format

* Add comment

* Fix

* Fix indent for azure docker config

* Fix docker login config

* Fix comments

* More robust docker login config

* Add retry for docker check

* minor fix

* Add additional test for stop and start with docker

* Fix cancelled
added comments
cb695d5
@shethhriday29
Merge branch 'skypilot-org:master' into master
94221eb
quick fix
888f1a8
finished pip issues
4877669
fix
7a208dd
@romilbhardwaj
fix storage error message, add example link to docs
152e36a
@romilbhardwaj
Merge branch 'master' of https://github.com/skypilot-org/skypilot int…
cf556ed 
…o hriday/unsloth

# Conflicts:
#	README.md
#	docs/source/index.rst
@shethhriday29
Merge branch 'skypilot-org:master' into master
8213f02
@shethhriday29
Merge branch 'skypilot-org:master' into master
60a530d
@shethhriday29
Merge branch 'skypilot-org:master' into master
8df52d6
@shethhriday29
Merge branch 'skypilot-org:master' into master
31291f4
prototype for gcs
c868494
removed non git files
953b4a6
rclone support for aws
e63e943
minor formatting fix
a4ddd48
cg505
cg505 previously approved these changes Apr 1, 2025
View reviewed changes
@cg505 cg505 dismissed their stale review April 1, 2025 22:10

did not mean to approve

cg505
cg505 reviewed Apr 2, 2025
View reviewed changes
cg505
cg505 approved these changes Apr 2, 2025
View reviewed changes
Copy link
Collaborator

@cg505 cg505 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can add this comment ^ otherwise looks good.
Let's run all the test_mount_and_storage smoke tests once more before merging.

@cg505
Copy link
Collaborator

cg505 commented Apr 2, 2025

Maybe also re-run the manual tests in the PR description.

@SeungjinYang
Copy link
Collaborator

/smoke-test -k test_aws_storage_mounts_with_stop

@SeungjinYang
Copy link
Collaborator

/smoke-test --gcp -k test_gcp_storage_mounts_with_stop

@SeungjinYang
Copy link
Collaborator

/smoke-test -k test_docker_storage_mounts --gcp

@SeungjinYang
Copy link
Collaborator

/smoke-test -k test_docker_storage_mounts --aws

@SeungjinYang
Copy link
Collaborator

redid the following tests:

  • Check if mountpoints remain mounted after restarting by running sky stop and sky start:
    • pytest tests/test_smoke.py::test_aws_storage_mounts_with_stop --aws
    • pytest tests/test_smoke.py::test_gcp_storage_mounts_with_stop --gcp
    • Manually running sky stop and sky start with comp_test.yaml to see if all the MOUNT_CACHED remains storage remains mounted.
  • Set of comprehensive manual tests with comp_test.yaml:
    • launch tests:
      • sky launch comp_test.yaml --cloud aws -y
      • sky launch comp_test.yaml --cloud gcp -y
    • sky launch two instances mounting same backend storage. Create new file on mountpoint on instance 1, and check if it appears shortly in instance 2.
    • managed spot jobs tests:
      • sky jobs launch comp_test.yaml --use-spot --cloud aws -y
      • sky jobs launch comp_test.yaml --use-spot --cloud gcp -y
    • launch on docker containers:
      • sky launch comp_test.yaml --cloud aws --image-id docker:continuumio/miniconda3:latest -y
      • sky launch comp_test.yaml --cloud gcp --image-id docker:continuumio/miniconda3:latest -y

@SeungjinYang
Copy link
Collaborator

/smoke-test --azure -k test_azure_storage_mounts_with_stop

@SeungjinYang
Copy link
Collaborator

/smoke-test -k test_file_mounts

@SeungjinYang
Copy link
Collaborator

/smoke-test -k test_using_file_mounts_with_env_vars

@SeungjinYang
Copy link
Collaborator

/smoke-test --kubernetes -k test_kubernetes_storage_mounts

@SeungjinYang
Copy link
Collaborator

/smoke-test --kubernetes -k test_kubernetes_context_switch

@SeungjinYang SeungjinYang merged commit 515e5df into skypilot-org:master Apr 2, 2025
21 checks passed
@SeungjinYang SeungjinYang mentioned this pull request Apr 2, 2025
Merged
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@SeungjinYang SeungjinYang SeungjinYang left review comments

@cg505 cg505 cg505 approved these changes

@romilbhardwaj romilbhardwaj Awaiting requested review from romilbhardwaj

@KeplerC KeplerC Awaiting requested review from KeplerC

Assignees

@SeungjinYang SeungjinYang

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@landscapepainter @romilbhardwaj @SeungjinYang @cg505 @shethhriday29 @Michaelvll