Gang scheduling in Kubernetes: native, all-or-nothing scheduling for AI training

How it evolved over three releases

The mechanics are straightforward. A pod group declares a minimum count. In the original design, the scheduler held pods at a permit gate until it had found placements for the whole group; if only part of the group fitted within the timeout, all of them went back to the queue and their reserved resources were released¹. Later releases made the pod group the unit that is queued and scheduled²³.

Why v1.37 matters

Beta means the core APIs are one step from general availability, in the project's own words³. Three changes stand out for AI platforms:

• Workload-aware preemption. A high-priority training job can preempt lower-priority work as a group, rather than evicting pods one by one in ways that don't free a usable block of GPUs.
• CompositePodGroup. Multi-level groups with topology constraints support structures commonly managed by JobSet and LeaderWorkerSet, which covers many distributed training and multi-host inference patterns³.
• Elastic minimum counts. Controllers can change a group's minimum size on the fly, so jobs can shrink or grow without interrupting pods already scheduled³.

Alpha API versions have changed with breaking changes between releases³. If you adopted early, budget time for migration.

Where Kueue fits

Gang scheduling decides how one job's pods are placed. Kueue answers a different question: which jobs may run at all, given team quotas and shared capacity. A job becomes a Kueue workload, is evaluated against its local and cluster queues, resource flavours and optional cohort, and is resumed only when admitted⁴.

On a shared GPU cluster, I use both layers: Kueue for fairness and quotas between teams, and native gang scheduling so that admitted jobs land together. The same blog post is a useful guide to answering the most common question on such platforms: why is my job still pending?⁴

How I would plan a GPU cluster now

1. Separate admission from placement. Quotas and priorities in a queueing layer; all-or-nothing placement in the scheduler.
2. Model topology. Training performance depends on how close workers are. Use topology-aware placement where your hardware has meaningful racks or interconnect domains.
3. Use DRA for GPUs, as in Kubernetes DRA for GPUs, now that pod groups can share resource claims.
4. Set preemption policy deliberately. Decide which work can be interrupted, and make sure training jobs checkpoint so preemption doesn't waste hours.
5. Measure idle GPU time. Partially scheduled jobs and fragmentation show up as expensive idle hardware, which the cost practices in Kubernetes FinOps and FOCUS for AI spend can make visible.
6. Pilot on beta in non-production before replacing an existing custom scheduler, and keep an eye on the path to GA.

How I can help

I design Kubernetes platforms for AI training and inference, from scheduling and quotas to GPU allocation and cost visibility. See my AI architecture work, or analyse your challenge and bring the result to a call.