Add a user document for the scheduling framework (#14388)

2019-06-10 18:38:13 -07:00
parent c22345dc2f
commit 5532ab32c1
2 changed files with 296 additions and 0 deletions
@@ -0,0 +1,296 @@
 ---
 reviewers:
 - ahg-g
 title: Scheduling Framework
 content_template: templates/concept
 weight: 70
 ---
 {{% capture overview %}}
 {{< feature-state for_k8s_version="1.15" state="alpha" >}}
 The scheduling framework is a new plugable architecture for Kubernetes Scheduler
 that makes scheduler customizations easy. It adds a new set of "plugin" APIs to
 the existing scheduler. Plugins are compiled into the scheduler. The APIs
 allow most scheduling features to be implemented as plugins, while keeping the
 scheduling "core" simple and maintainable. Refer to the [design proposal of the
 scheduling framework][kep] for more technical information on the design of the
 framework.
 [kep]: (https://github.com/kubernetes/enhancements/blob/master/keps/sig-scheduling/20180409-scheduling-framework.md)
 {{% /capture %}}
 {{% capture body %}}
 # Framework workflow
 The Scheduling Framework defines a few extension points. Scheduler plugins
 register to be invoked at one or more extension points. Some of these plugins
 can change the scheduling decisions and some are informational only.
 Each attempt to schedule one Pod is split into two phases, the **scheduling
 cycle** and the **binding cycle**.
 ## Scheduling Cycle & Binding Cycle
 The scheduling cycle selects a node for the Pod, and the binding cycle applies
 that decision to the cluster. Together, a scheduling cycle and binding cycle are
 referred to as a "scheduling context".
 Scheduling cycles are run serially, while binding cycles may run concurrently.
 A scheduling or binding cycle can be aborted if the Pod is determined to
 be unschedulable or if there is an internal error. The Pod will be returned to
 the queue and retried.
 ## Extension points
 The following picture shows the scheduling context of a Pod and the extension
 points that the scheduling framework exposes. In this picture "Filter" is
 equivalent to "Predicate" and "Scoring" is equivalent to "Priority function".
 One plugin may register at multiple extension points to perform more complex or
 stateful tasks.
 {{< figure src="/images/docs/scheduling-framework.png" title="scheduling framework extension points" >}}
 ### Queue sort
 These plugins are used to sort Pods in the scheduling queue. A queue sort plugin
 essentially will provide a "less(Pod1, Pod2)" function. Only one queue sort
 plugin may be enabled at a time.
 ### Pre-filter
 These plugins are used to pre-process info about the Pod, or to check certain
 conditions that the cluster or the Pod must meet. If a pre-filter plugin returns
 an error, the scheduling cycle is aborted.
 ### Filter
 These plugins are used to filter out nodes that cannot run the Pod. For each
 node, the scheduler will call filter plugins in their configured order. If any
 filter plugin marks the node as infeasible, the remaining plugins will not be
 called for that node. Nodes may be evaluated concurrently.
 ### Post-filter
 This is an informational extension point. Plugins will be called with a list of
 nodes that passed the filtering phase. A plugin may use this data to update
 internal state or to generate logs/metrics.
 **Note:** Plugins wishing to perform "pre-scoring" work should use the
 post-filter extension point.
 ### Scoring
 These plugins are used to rank nodes that have passed the filtering phase. The
 scheduler will call each scoring plugin for each node. There will be a well
 defined range of integers representing the minimum and maximum scores. After the
 [normalize scoring](#normalize-scoring) phase, the scheduler will combine node
 scores from all plugins according to the configured plugin weights.
 ### Normalize scoring
 These plugins are used to modify scores before the scheduler computes a final
 ranking of Nodes. A plugin that registers for this extension point will be
 called with the [scoring](#scoring) results from the same plugin. This is called
 once per plugin per scheduling cycle.
 For example, suppose a plugin `BlinkingLightScorer` ranks Nodes based on how
 many blinking lights they have.
 ```go
 func ScoreNode(_ *v1.pod, n *v1.Node) (int, error) {
   return getBlinkingLightCount(n)
 }
 ```
 However, the maximum count of blinking lights may be small compared to
 `NodeScoreMax`. To fix this, `BlinkingLightScorer` should also register for this
 extension point.
 ```go
 func NormalizeScores(scores map[string]int) {
   highest := 0
   for _, score := range scores {
      highest = max(highest, score)
   }
   for node, score := range scores {
      scores[node] = score*NodeScoreMax/highest
   }
 }
 ```
 If any normalize-scoring plugin returns an error, the scheduling cycle is
 aborted.
 **Note:** Plugins wishing to perform "pre-reserve" work should use the
 normalize-scoring extension point.
 ### Reserve
 This is an informational extension point. Plugins which maintain runtime state
 (aka "stateful plugins") should use this extension point to be notified by the
 scheduler when resources on a node are being reserved for a given Pod. This
 happens before the scheduler actually binds the Pod to the Node, and it exists
 to prevent race conditions while the scheduler waits for the bind to succeed.
 This is the last step in a scheduling cycle. Once a Pod is in the reserved
 state, it will either trigger [Un-reserve](#un-reserve) plugins (on failure) or
 [Post-bind](#post-bind) plugins (on success) at the end of the binding cycle.
 *Note: This concept used to be referred to as "assume".*
 ### Permit
 These plugins are used to prevent or delay the binding of a Pod. A permit plugin
 can do one of three things.
 1.  **approve** \
    Once all permit plugins approve a Pod, it is sent for binding.
 1.  **deny** \
    If any permit plugin denies a Pod, it is returned to the scheduling queue.
    This will trigger [Un-reserve](#un-reserve) plugins.
 1.  **wait** (with a timeout) \
    If a permit plugin returns "wait", then the Pod is kept in the permit phase
    until a [plugin approves it](#frameworkhandle). If a timeout occurs, **wait**
    becomes **deny** and the Pod is returned to the scheduling queue, triggering
    [un-reserve](#un-reserve) plugins.
 **Approving a Pod binding**
 While any plugin can access the list of "waiting" Pods from the cache and
 approve them (see [`FrameworkHandle`](#frameworkhandle)) we expect only the permit
 plugins to approve binding of reserved Pods that are in "waiting" state. Once a
 Pod is approved, it is sent to the pre-bind phase.
 ### Pre-bind
 These plugins are used to perform any work required before a Pod is bound. For
 example, a pre-bind plugin may provision a network volume and mount it on the
 target node before allowing the Pod to run there.
 If any pre-bind plugin returns an error, the Pod is [rejected](#un-reserve) and
 returned to the scheduling queue.
 ### Bind
 These plugins are used to bind a Pod to a Node. Bind plugins will not be called
 until all pre-bind plugins have completed. Each bind plugin is called in the
 configured order. A bind plugin may choose whether or not to handle the given
 Pod. If a bind plugin chooses to handle a Pod, **the remaining bind plugins are
 skipped**.
 ### Post-bind
 This is an informational extension point. Post-bind plugins are called after a
 Pod is successfully bound. This is the end of a binding cycle, and can be used
 to clean up associated resources.
 ### Unreserve
 This is an informational extension point. If a Pod was reserved and then
 rejected in a later phase, then unreserve plugins will be notified. Unreserve
 plugins should clean up state associated with the reserved Pod.
 Plugins that use this extension point usually should also use
 [Reserve](#reserve).
 ## Plugin API
 There are two steps to the plugin API. First, plugins must register and get
 configured, then they use the extension point interfaces. Extension point
 interfaces have the following form.
 ```go
 type Plugin interface {
   Name() string
 }
 type QueueSortPlugin interface {
   Plugin
   Less(*v1.pod, *v1.pod) bool
 }
 type PreFilterPlugin interface {
   Plugin
   PreFilter(PluginContext, *v1.pod) error
 }
 // ...
 ```
 # Plugin Configuration
 Plugins can be enabled in the scheduler configuration. Also, default plugins can
 be disabled in the configuration. In 1.15, there are no default plugins for the
 scheduling framework.
 The scheduler configuration can include configuration for plugins as well. Such
 configurations are passed to the plugins at the time the scheduler initializes
 them. The configuration is an arbitrary value. The receiving plugin should
 decode and process the configuration.
 The following example shows a scheduler configuration that enables some
 plugins at `reserve` and `preBind` extension points and disables a plugin. It
 also provides a configuration to plugin `foo`. 
 ```yaml
 apiVersion: kubescheduler.config.k8s.io/v1alpha1
 kind: KubeSchedulerConfiguration
 ...
 plugins:
  reserve:
    enabled:
    - name: foo
    - name: bar
    disabled:
    - name: baz
  preBind:
    enabled:
    - name: foo
    disabled:
    - name: baz
 pluginConfig:
 - name: foo
  args: >
    Arbitrary set of args to plugin foo
 ```
 When an extension point is omitted from the configuration default plugins for
 that extension points are used. When an extension point exists and `enabled` is
 provided, the `enabled` plugins are called in addition to default plugins.
 Default plugins are called first and then the additional enabled plugins are
 called in the same order specified in the configuration. If a different order of
 calling default plugins is desired, default plugins must be `disabled` and
 `enabled` in the desired order.
 Assuming there is a default plugin called `foo` at `reserve` and we are adding
 pluing `bar` that we want to be invoked before `foo`, we should disable `foo`
 and enable `bar` and `foo` in order. The following example shows the
 configuration that achieves this:
 ```yaml
 apiVersion: kubescheduler.config.k8s.io/v1alpha1
 kind: KubeSchedulerConfiguration
 ...
 plugins:
  reserve:
    enabled:
    - name: bar
    - name: foo
    disabled:
    - name: foo
 ```
 {{% /capture %}}