[k8s] Improve /dev/fuse access on k8s

[aylei]

close #4108

This PR introduces a privileged kubernetes DaemonSet to proxy fuse mount/unmount operations, so that we get rid of the additional privileges and capabilities of SkyPilot Pods. For elaboration, refer to https://github.com/skypilot-org/skypilot/blob/improve-k8s-fuse/addons/fuse-proxy/README.md

• This PR also close [Storage] Unmount failed on kubernetes #4970 by not doing mount/unmount in the sky container
• close [K8S] wget may be unavailable when using custom image #5148 by adding wget in k8s start script

Benchmark:

command: fio --name=64kseqwrites --rw=write --direct=1 --bs=1k --numjobs=1 --iodepth=8 --size=100M --group_reporting

Client	Storage	Bandwidth
GKE with fuse-proxy	S3	10.9MiB/s
GKE with fuse-proxy	GCS	9245KiB/s
GKE with smarter-device-plugin	S3	N/A (Error occurred)
GKE with smarter-device-plugin	GCS	8304KiB/s
GCP VM	S3	13.7MiB/s
GCP VM	GCS	12.4MiB/s

There is a performance degradation compared to plain VM (both using 2c resources), need to figure why in the future. But compared to existing solution (smarter-device-plugin), there is performance regression.

Tested (run the relevant ones):

• Code formatting: install pre-commit (auto-check on commit) or bash format.sh
• Any manual or new tests for this PR (please specify below)
  • GCS fuse mount/unmount
  • S3 fuse mount/unmount
• All smoke tests: /smoke-test (CI) or pytest tests/test_smoke.py (local)
• /smoke-test --kubernetes -k test_docker_storage_mounts
• /smoke-test --kubernetes -k test_kubernetes_storage_mounts
• /smoke-test --aws -k test_docker_storage_mounts verify there is no regression for non-k8s fuse mount
• Backward compatibility: /quicktest-core (CI) or pytest tests/smoke_tests/test_backward_compat.py (local)
• Manually tested GKE autoscaling from 0 with GPU request and GCS mount

Future TODOs

• Maintain goofys fork under skypilot org: Maintain goofys fork under skypilot-org #5189
• Figure out the performance gap of FUSE between Kubernetes and VM deployment: FUSE performance is worse on Kubernetes compared to VM deployment #5190

[aylei]

/smoke-test --kubernetes -k test_kubernetes_storage_mounts

[aylei]

/smoke-test --kubernetes -k test_kubernetes_storage_mounts

[aylei]

/smoke-test --aws -k test_docker_storage_mounts

https://buildkite.com/skypilot-1/smoke-tests/builds/631 Verify there is no regression for non-k8s fuse mount

[SeungjinYang]

Go code LGTM

[romilbhardwaj]

This is amazing work @aylei! Sending some quick comments, will give it a go in a bit. Mostly looks good to me

[romilbhardwaj]

@aylei have we stress tested our new fuse solution in some way?

I'm trying to run fio --name=64kseqwrites --rw=write --direct=1 --bs=1k --numjobs=1 --iodepth=8 --size=1M --group_reporting by sshing into a cluster created with this YAML:

num_nodes: 2

file_mounts:
  /mydata:
    name: romiltestbucket9
    store: s3
    source: ~/tmp-workdir

run: |
  ls /mydata
  sudo apt install fio -y 

gets stuck on:

64kseqwrites: (g=0): rw=write, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=psync, iodepth=8
fio-3.25
Starting 1 process
64kseqwrites: Laying out IO file (1 file / 1MiB)

To be fair, our current smarter-devices-fuse based solution also fails with a transport endpoint failure. However, it works when running on cloud VMs.

[aylei]

@romilbhardwaj Thanks for testing it out! I will take a look

[aylei]

/smoke-test --aws -k test_docker_storage_mounts
https://buildkite.com/skypilot-1/smoke-tests/builds/664

[aylei]

gets stuck on:

64kseqwrites: (g=0): rw=write, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=psync, iodepth=8

@romilbhardwaj It turns out there will be an error when running fio, and if the environment does not have syslog enabled, goofys will crash

2025/04/11 09:20:07.153474 fuse.ERROR *fuseops.FallocateOp error: function not implemented

After the latest commit: 5b6fc58, this problem is addressed:

fio --name=64kseqwrites --rw=write --direct=1 --bs=1k --numjobs=1 --iodepth=8 --size=1M --group_reporting
64kseqwrites: (g=0): rw=write, bs=(R) 1024B-1024B, (W) 1024B-1024B, (T) 1024B-1024B, ioengine=psync, iodepth=8
fio-3.25
Starting 1 process
64kseqwrites: Laying out IO file (1 file / 1MiB)

64kseqwrites: (groupid=0, jobs=1): err= 0: pid=17212: Fri Apr 11 09:20:07 2025
  write: IOPS=8827, BW=8828KiB/s (9039kB/s)(1024KiB/116msec); 0 zone resets
    clat (usec): min=66, max=919, avg=111.66, stdev=57.73
     lat (usec): min=66, max=919, avg=111.78, stdev=57.78
    clat percentiles (usec):
     |  1.00th=[   71],  5.00th=[   77], 10.00th=[   79], 20.00th=[   84],
     | 30.00th=[   88], 40.00th=[   91], 50.00th=[   94], 60.00th=[   99],
     | 70.00th=[  106], 80.00th=[  122], 90.00th=[  165], 95.00th=[  208],
     | 99.00th=[  338], 99.50th=[  412], 99.90th=[  676], 99.95th=[  922],
     | 99.99th=[  922]
  lat (usec)   : 100=61.33%, 250=35.35%, 500=3.12%, 750=0.10%, 1000=0.10%
  cpu          : usr=0.00%, sys=18.26%, ctx=2049, majf=0, minf=10
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,1024,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
  WRITE: bw=8828KiB/s (9039kB/s), 8828KiB/s-8828KiB/s (9039kB/s-9039kB/s), io=1024KiB (1049kB), run=116-116msec

[aylei]

/smoke-test --kubernetes -k test_kubernetes_storage_mounts
/smoke-test --kubernetes -k test_docker_storage_mounts

https://buildkite.com/skypilot-1/smoke-tests/builds/665

[aylei]

@romilbhardwaj I also update the stress test result in the PR description, ready for another round of review!

[romilbhardwaj]

Amazing work @aylei! Super excited to get this in! LGTM.

[SeungjinYang]

Nice! This is an exciting improvement

[aylei]

/smoke-test

https://buildkite.com/skypilot-1/smoke-tests/builds/679

[aylei]

Given the smoke test & stress test result, merging into master and track follow-ups in separate issues