Adhityaa Chandrasekar
111 lines
5.5 KiB
Markdown
111 lines
5.5 KiB
Markdown
---
|
|
title: "Introducing Suspended Jobs"
|
|
date: 2021-04-12
|
|
slug: introducing-suspended-jobs
|
|
layout: blog
|
|
---
|
|
|
|
**Author:** Adhityaa Chandrasekar (Google)
|
|
|
|
[Jobs](/docs/concepts/workloads/controllers/job/) are a crucial part of
|
|
Kubernetes' API. While other kinds of workloads such as [Deployments](/docs/concepts/workloads/controllers/deployment/),
|
|
[ReplicaSets](/docs/concepts/workloads/controllers/replicaset/),
|
|
[StatefulSets](/docs/concepts/workloads/controllers/statefulset/), and
|
|
[DaemonSets](/docs/concepts/workloads/controllers/daemonset/)
|
|
solve use-cases that require Pods to run forever, Jobs are useful when Pods need
|
|
to run to completion. Commonly used in parallel batch processing, Jobs can be
|
|
used in a variety of applications ranging from video rendering and database
|
|
maintenance to sending bulk emails and scientific computing.
|
|
|
|
While the amount of parallelism and the conditions for Job completion are
|
|
configurable, the Kubernetes API lacked the ability to suspend and resume Jobs.
|
|
This is often desired when cluster resources are limited and a higher priority
|
|
Job needs to execute in the place of another Job. Deleting the lower priority
|
|
Job is a poor workaround as Pod completion history and other metrics associated
|
|
with the Job will be lost.
|
|
|
|
With the recent Kubernetes 1.21 release, you will be able to suspend a Job by
|
|
updating its spec. The feature is currently in **alpha** and requires you to
|
|
enable the `SuspendJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/)
|
|
on the [API server](/docs/reference/command-line-tools-reference/kube-apiserver/)
|
|
and the [controller manager](/docs/reference/command-line-tools-reference/kube-controller-manager/)
|
|
in order to use it.
|
|
|
|
## API changes
|
|
|
|
We introduced a new boolean field `suspend` into the `.spec` of Jobs. Let's say
|
|
I create the following Job:
|
|
|
|
```yaml
|
|
apiVersion: batch/v1
|
|
kind: Job
|
|
metadata:
|
|
name: my-job
|
|
spec:
|
|
suspend: true
|
|
parallelism: 2
|
|
completions: 10
|
|
template:
|
|
spec:
|
|
containers:
|
|
- name: my-container
|
|
image: busybox
|
|
command: ["sleep", "5"]
|
|
restartPolicy: Never
|
|
```
|
|
|
|
Jobs are not suspended by default, so I'm explicitly setting the `suspend` field
|
|
to _true_ in the `.spec` of the above Job manifest. In the above example, the
|
|
Job controller will refrain from creating Pods until I'm ready to start the Job,
|
|
which I can do by updating `suspend` to false.
|
|
|
|
As another example, consider a Job that was created with the `suspend` field
|
|
omitted. The Job controller will happily create Pods to work towards Job
|
|
completion. However, before the Job completes, if I explicitly set the field to
|
|
true with a Job update, the Job controller will terminate all active Pods that
|
|
are running and will wait indefinitely for the flag to be flipped back to false.
|
|
Typically, Pod termination is done by sending a SIGTERM signal to all container
|
|
processes in the Pod; the [graceful termination period](/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination)
|
|
defined in the Pod spec will be honoured. Pods terminated this way will not be
|
|
counted as failures by the Job controller.
|
|
|
|
It is important to understand that succeeded and failed Pods from the past will
|
|
continue to exist after you suspend a Job. That is, that they will count towards
|
|
Job completion once you resume it. You can verify this by looking at Job's
|
|
status before and after suspension.
|
|
|
|
Read the [documentation](/docs/concepts/workloads/controllers/job#suspending-a-job)
|
|
for a full overview of this new feature.
|
|
|
|
## Where is this useful?
|
|
|
|
Let's say I'm the operator of a large cluster. I have many users submitting Jobs
|
|
to the cluster, but not all Jobs are created equal — some Jobs are more
|
|
important than others. Cluster resources aren't infinite either, so all users
|
|
must share resources. If all Jobs were created in the suspended state and placed
|
|
in a pending queue, I can achieve priority-based Job scheduling by resuming Jobs
|
|
in the right order.
|
|
|
|
As another motivational use-case, consider a cloud provider where compute
|
|
resources are cheaper at night than in the morning. If I have a long-running Job
|
|
that takes multiple days to complete, being able to suspend the Job in the
|
|
morning and then resume it in the evening every day can reduce costs.
|
|
|
|
Since this field is a part of the Job spec, [CronJobs](/docs/concepts/workloads/controllers/cron-jobs/)
|
|
automatically get this feature for free too.
|
|
|
|
## References and next steps
|
|
|
|
If you're interested in a deeper dive into the rationale behind this feature and
|
|
the decisions we have taken, consider reading the [enhancement proposal](https://github.com/kubernetes/enhancements/tree/master/keps/sig-apps/2232-suspend-jobs).
|
|
There's more detail on suspending and resuming jobs in the documentation for [Job](/docs/concepts/workloads/controllers/job#suspending-a-job).
|
|
|
|
As previously mentioned, this feature is currently in alpha and is available
|
|
only if you explicitly opt-in through the `SuspendJob` [feature gate](/docs/reference/command-line-tools-reference/feature-gates/).
|
|
If this is a feature you're interested in, please consider testing suspended
|
|
Jobs in your cluster and providing feedback. You can discuss this enhancement [on GitHub](https://github.com/kubernetes/enhancements/issues/2232).
|
|
The SIG Apps community also [meets regularly](https://github.com/kubernetes/community/tree/master/sig-apps#meetings)
|
|
and can be reached through [Slack or the mailing list](https://github.com/kubernetes/community/tree/master/sig-apps#contact).
|
|
Barring any unexpected changes to the API, we intend to graduate the feature to
|
|
beta in Kubernetes 1.22, so that the feature becomes available by default.
|