Serving Kimi K3 on Your Own GPUs with SkyPilot

Today, on July 27, Moonshot AI open-sourced the weights for Kimi K3. It’s the biggest open model we’ve seen so far, the first to approach the 3-trillion-parameter mark. On most benchmarks it sits right behind the top closed models.

The weights are there, but can you serve it yourself? K3 has 2.8 trillion parameters, and even in its native 4-bit (MXFP4) format the weights run about 1.4 TB. That rules out a single GPU and pretty much any GPU node you have sitting around. The weights alone just barely fit on one top-end 8-GPU node, but that leaves nothing for the 1M-token KV cache or the headroom real throughput needs.

To serve it in practice you need many GPUs across several nodes, usually across whatever clouds and clusters you can scrape capacity from.

We built SkyPilot Endpoints to simplify setting up production-grade serving on your GPUs. This post covers what K3 is, how it stacks up against the current frontier models, and how to serve it on your own fleet with a single YAML.

The K3 architecture #

K3 is a Mixture-of-Experts model: 2.8T parameters total, with 16 of 896 experts plus 2 shared experts firing on each token, which works out to about 104B active parameters. It takes text, images, and video in, carries a 1M-token context, and writes text out. It’s a reasoning model, with the effort level configurable across low, high, and max (the default).

Under the hood it uses Kimi Delta Attention, a linear/hybrid attention scheme, with roughly three linear-attention layers for every full-attention layer. That ratio is what keeps a million-token context affordable. Moonshot also trained it quantization-aware in MXFP4/MXFP8, so the released weights are meant to survive low-bit inference instead of falling apart the way a naive post-training quant would.

How it compares #

On the public benchmarks, K3 lands a notch behind Claude Fable 5, ahead of GPT-5.6 Sol and Claude Opus 4.8, and first among open-weight models. A few numbers from independent evals:

AA-Briefcase is Artificial Analysis’s agentic-knowledge-work benchmark; Elo scores as reported at launch.

The picture changes depending on the benchmark. K3 is currently the #1 WebDev model on LMArena’s Frontend Code Arena at Elo 1678 (human votes, ahead of Fable 5), scores 57 on Artificial Analysis’s Intelligence Index (top 3 of the ~186 models they track), and shows the shape we’ve come to expect from recent Chinese models: strong at multi-turn agentic coding, weaker one-shot.

Two things to keep in mind before you read too much into any of this. The launch-day numbers are mostly Moonshot’s own, so treat them as directional until the technical report and more third-party runs are in. Many hands-on reports seem to be split: some people can’t tell K3 from Fable on real work, while others find it slower and flakier on tool calls.

Serving Kimi K3 on your GPUs with SkyPilot #

Open weight models are only useful if you can run them, and 2.8T parameters means multi-node inference: tensor, data, and expert parallelism spread across a lot of large GPUs, usually pulled from more than one cloud or cluster.

That’s what SkyPilot Endpoints does. You describe the endpoint in one YAML, and we stand up the whole serving stack behind it: the engine, autoscaler, gateway, cache-aware routing, and metrics, across as many of your Kubernetes clusters as you want, all under one URL. We treat those clusters as a single pool: replicas land wherever there’s capacity, and if a cluster goes down we bring them back on a healthy one without the URL changing.

SkyPilot YAML for Kimi K3

Here's a SkyPilot Endpoints YAML to run Kimi K3 on your GPUs:

Launch with one command:

That one file stands up the full stack and hands you a public or private URL. From there you can tune the knobs you need: raise max_replicas to autoscale on KV-cache utilization or queue depth, mount a shared model cache with volumes to cut cold starts, or lock the endpoint down with network.visibility and auth.api_key.

You also get one dashboard for the whole fleet: per-replica health and placement, serving metrics (TTFT, TPOT, throughput, KV-cache utilization), live logs, and an in-browser playground to sanity-check the model.

To use the Claude Code harness with your self-hosted Kimi K3 endpoint, run the following commands:

Try it out #

K3 is the strongest open model you can self-host right now, and the main thing between you and running it is scale. SkyPilot takes care of that part. Run sky endpoints up and you have a frontier-class model running on your own GPUs behind one URL.

Serving it yourself also means you run the weights exactly as Moonshot shipped them. Because K3 is trained quantization-aware in MXFP4, you get their intended precision, not whatever quant a hosted API happens to serve behind the scenes to squeeze the model onto cheaper hardware and push more requests through the same GPUs.

SkyPilot Endpoints is in early access. Request access here to serve K3 on your own fleet.

To receive latest updates, please follow @skypilot_org, star and watch the project’s GitHub repo, or join the SkyPilot community Slack.