We are excited to announce SkyPilot 0.6, which brings a host of new features and improvements to SkyPilot. This release is aimed at making your AI workloads more efficient and cost-effective by allowing them to run anywhere.
In this post, we will highlight some of the key features and improvements in SkyPilot 0.6.
Managed Jobs #
We’ve introduced a new Managed Jobs API to queue jobs, automatically recover from failures (e.g., spot instance preemption, GPU failures) and run pipelines.
Launch jobs and pipelines with a simple command:
sky jobs queue shows the details of the jobs, including the status, logs, and more.
To learn more, refer to the Jobs API documentation.
Mix Spot and On-demand Instances for Serving #
Spot instances are great for cost savings, but they can be preempted leading to service downtime. SkyServe now supports mixing on-demand instances to ensure high availability while also using spot instances to deliver cost savings.
For example, to autoscale a service with a minimum of 2 replicas and a maximum of 3 replicas, and ensuring that one of the replicas is run on on-demand instances, you can use base_ondemand_fallback_replicas:
Similarly, to dynamically fall back to on-demand replicas if spot instances get preempted, use dynamic_ondemand_fallback:
Read the docs for more details on mixing spot and on-demand instances.
SkyServe on Kubernetes #
SkyServe now integrates with Kubernetes. Both the SkyServe controller and service replicas can be run on Kubernetes and managed by SkyPilot.
Launch your SkyServe service with:
SkyServe can also burst out of the Kubernetes cluster to the cloud for additional capacity when needed. You get one common endpoint to run all your queries, while SkyServe manages the underlying infrastructure (Kubernetes and the clouds) for you.
Example Llama-2 service running on Kubernetes and GCP spot instances:
Paperspace support #
SkyPilot now supports a total of 14 clouds, including Paperspace.
To get started, follow the installation guide.
Other features and improvements #
SkyPilot 0.6 also includes several other features and improvements, including:
- New LLM Recipes: Llama-3, Qwen, Ollama, DBRX, Unsloth, Cog.
- Proxy support in SkyServe for improved performance and security.
sky show-gpusshows realtime GPU availability on your Kubernetes cluster.$ sky show-gpus --cloud kubernetes GPU QTY_PER_NODE TOTAL_GPUS TOTAL_FREE_GPUS L4 1, 2, 3, 4 8 6 H100 1, 2 4 2- Support for autoscaling Kubernetes clusters.
- Use service accounts for authentication with Kubernetes clusters.
- Improved GPU isolation on Kubernetes.
- Support for Python 3.11.
… and many more optimizations and features! Check out the release notes for a detailed list.
Summary #
SkyPilot 0.6 makes the deployment and management of AI workloads efficient and cost-effective on any infra. With the new Managed Jobs API, SkyServe on Kubernetes, and support for Paperspace, you can now run your AI workloads anywhere with ease. We hope you enjoy this release and stay tuned for more!
To receive latest updates, please star and watch the project’s GitHub repo, follow @skypilot_org, or join the SkyPilot community Slack.