[Roadmap] SkyPilot Roadmap Q1 2025

[Michaelvll]

Here is the development roadmap for 2025 Q1. Contributions and feedback are welcome! Please feel free to join our Slack.

This doc will be updated as we make progress.

Deployment for Teams

• Client-Server rearchitect [Core] SkyPilot Client-Server Architecture #4660
• Deployment of fault-tolerant SkyPilot API server for teams [API Server] Add helm release workflows #4737
• Multiple Kubernetes clusters support [Docs] Multiple k8s support #4586
• Performance improvement of client server request scheduling [Client/Server] server hung on memory starvation #4732
• Observability for clusters/jobs/services
• Robustness and UX improvement of the API server

Managed Jobs

• Support 2000+ in-parallel jobs for managed jobs [jobs] revamp scheduling for managed jobs #4485
• Avoid requirements of cloud credentials for k8s use case [jobs] two-hop file_mounts upload #4708
• Multi-tenant managed jobs and services
  • Consolidate jobs/serve controller Consolidate jobs controller across multiple users #4686
  • Job/Service owner
• UX improvement of large-scale batch inference. Proposoal: add sky batch and sky jobs batch CLIs #4735
• Documents for authentication on controller
• Integration with AI applications
  • Large-scale image search: https://blog.skypilot.co/large-scale-vector-database/
  • more blogs to come

Serving LLMs

• High-availability of the serve controller on Kubernetes. Prototype: Introduce High Availability Service Controller #4564
• Low-latency load balancing for LLM serving [WIP][Serve] Enable launching multiple external LB on controller. #4362
• Low-cost and highly available serving with spot instances [Policy] Add SpotHedge. #4628
• Observability improvement
  • Additional port for metrics [Serve] Enable multiple ports in SkyServe replicas #4356
  • Setting up metrics/dashboard

Clouds

• Robustness maintanence of existing clouds
  • H100/H200 GPUs
  • AMD GPUs and ARM CPUs [Core] Support ARM CPU on the cloud #4793 [Image] Support custom ARM image for AWS, GCP and Azure #4863
• High-performance AI networking setup on clouds (helps wanted)
  • Docs for the high-performance networking
• Automatic catalog fetching for more clouds
• Open port support for Lambda Cloud
• Cloud coverage
  • Digital Ocean [DigitalOcean] droplet integration #3832
  • Vast AI Preliminary Vast AI support #4365
  • Nebius Cloud Add Nebius Cloud #4573

Developer Productivity

• Buildkite integration for smoke tests Support buildkite CICD and restructure smoke tests #4396
• Weekly tests weekly smoke tests trigger on Github action #4755
• Cost reduction for the smoke tests

Ecosystem Projects

• AI workloads using SkyPilot
• Integration with other frameworks
  • Airflow (Example)