From 800a602e36cd6de1363724ceafb37105fe3b38da Mon Sep 17 00:00:00 2001 From: Mike Spreitzer Date: Wed, 22 Jul 2020 19:54:12 -0400 Subject: [PATCH 1/2] Document new API Priority and Fairness metrics Also brush up the descriptions of some of the older metrics. --- .../cluster-administration/flow-control.md | 113 +++++++++++++----- 1 file changed, 85 insertions(+), 28 deletions(-) diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md index 26fc1194df..f6cafecb52 100644 --- a/content/en/docs/concepts/cluster-administration/flow-control.md +++ b/content/en/docs/concepts/cluster-administration/flow-control.md @@ -308,10 +308,12 @@ exports additional metrics. Monitoring these can help you determine whether your configuration is inappropriately throttling important traffic, or find poorly-behaved workloads that may be harming system health. -* `apiserver_flowcontrol_rejected_requests_total` counts requests that - were rejected, grouped by the name of the assigned priority level, - the name of the assigned FlowSchema, and the reason for rejection. - The reason will be one of the following: +* `apiserver_flowcontrol_rejected_requests_total` is a vector of + counters (cumulative since server start) of requests that were + rejected, broken down by the labels `flowSchema` (indicating the one + that matched the request), `priorityLevel` (indicating the one to + which ithe request was assigned), and `reason`. The `reason` label + will be have one of the following values: * `queue-full`, indicating that too many requests were already queued, * `concurrency-limit`, indicating that the @@ -320,23 +322,73 @@ poorly-behaved workloads that may be harming system health. * `time-out`, indicating that the request was still in the queue when its queuing time limit expired. -* `apiserver_flowcontrol_dispatched_requests_total` counts requests - that began executing, grouped by the name of the assigned priority - level and the name of the assigned FlowSchema. +* `apiserver_flowcontrol_dispatched_requests_total` is a vector of + counters (cumulative since server start) of requests that began + executing, broken down by the labels `flowSchema` (indicating the + one that matched the request) and `priorityLevel` (indicating the + one to which ithe request was assigned). -* `apiserver_flowcontrol_current_inqueue_requests` gives the - instantaneous total number of queued (not executing) requests, - grouped by priority level and FlowSchema. +* `apiserver_current_inqueue_requests` is a vector gauges of recent + high water marks of the number of queued requests, grouped by a + label named `request_kind` whose value is `mutating` or `readOnly`. + These high water marks describe the largest number seen in the one + second window most recently completed. These complement the older + `apiserver_current_inflight_requests` gauge vector that holds the + last window's high water mark of number of requests actively being + served. -* `apiserver_flowcontrol_current_executing_requests` gives the instantaneous - total number of executing requests, grouped by priority level and FlowSchema. +* `apiserver_flowcontrol_read_vs_write_request_count_samples` is a + vector of histograms of observations of the then-current number of + requests, broken down by the labels `phase` (which takes on the + values `waiting` and `executing`) and `request_kind` (which takes on + the values `mutating` and `readOnly`). The observations ae made + periodically at a high rate. -* `apiserver_flowcontrol_request_queue_length_after_enqueue` gives a - histogram of queue lengths for the queues, grouped by priority level - and FlowSchema, as sampled by the enqueued requests. Each request - that gets queued contributes one sample to its histogram, reporting - the length of the queue just after the request was added. Note that - this produces different statistics than an unbiased survey would. +* `apiserver_flowcontrol_read_vs_write_request_count_watermarks` is a + vector of histograms of high or low water marks of the number of + requests broken down by the labels `phase` (which takes on the + values `waiting` and `executing`) and `request_kind` (which takes on + the values `mutating` and `readOnly`); the label `mark` takes on + values `high` and `low`. The water marks are accumulated over + windows bounded by the times when an observation was added to + `apiserver_flowcontrol_read_vs_write_request_count_samples`. These + water marks show the range of values that occurred between samples. + +* `apiserver_flowcontrol_current_inqueue_requests` is a vector of + gauges holding the instantaneous number of queued (not executing) + requests, broken down by the labels `priorityLevel` and + `flowSchema`. + +* `apiserver_flowcontrol_current_executing_requests` is a vector of + gauges holding the instantaneous number of executing (not waiting in + a queue) requests, broken down by the labels `priorityLevel` and + `flowSchema`. + +* `apiserver_flowcontrol_priority_level_request_count_samples` is a + vector of histograms of observations of the then-current number of + requests broken down by the labels `phase` (which takes on the + values `waiting` and `executing`) and `priorityLevel`. Each + histogram gets observations taken periodically, up through the last + activity of the relevant sort. The observations are made at a high + rate. + +* `apiserver_flowcontrol_priority_level_request_count_watermarks` is a + vector of histograms of high or low water marks of the number of + requests broken down by the labels `phase` (which takes on the + values `waiting` and `executing`) and `priorityLevel`; the label + `mark` takes on values `high` and `low`. The water marks are + accumulated over windows bounded by the times when an observation + was added to + `apiserver_flowcontrol_priority_level_request_count_samples`. These + water marks show the range of values that occurred between samples. + +* `apiserver_flowcontrol_request_queue_length_after_enqueue` is a + vector of histograms of queue lengths for the queues, broken down by + the labels `priorityLevel` and `flowSchema`, as sampled by the + enqueued requests. Each request that gets queued contributes one + sample to its histogram, reporting the length of the queue just + after the request was added. Note that this produces different + statistics than an unbiased survey would. {{< note >}} An outlier value in a histogram here means it is likely that a single flow (i.e., requests by one user or for one namespace, depending on @@ -346,14 +398,17 @@ poorly-behaved workloads that may be harming system health. to increase that PriorityLevelConfiguration's concurrency shares. {{< /note >}} -* `apiserver_flowcontrol_request_concurrency_limit` gives the computed - concurrency limit (based on the API server's total concurrency limit and PriorityLevelConfigurations' - concurrency shares) for each PriorityLevelConfiguration. +* `apiserver_flowcontrol_request_concurrency_limit` is a vector of + gauges hoding the computed concurrency limit (based on the API + server's total concurrency limit and PriorityLevelConfigurations' + concurrency shares), broken down by the label `priorityLevel`. -* `apiserver_flowcontrol_request_wait_duration_seconds` gives a histogram of how - long requests spent queued, grouped by the FlowSchema that matched the - request, the PriorityLevel to which it was assigned, and whether or not the - request successfully executed. +* `apiserver_flowcontrol_request_wait_duration_seconds` is a vector of + histograms of how long requests spent queued, broken down by the + labels `flowSchema` (indicating which one matched the request), + `priorityLevel` (indicating the one to which ithe request was + assigned), and `execute` (indicating whether the request started + executing). {{< note >}} Since each FlowSchema always assigns requests to a single PriorityLevelConfiguration, you can add the histograms for all the @@ -361,9 +416,11 @@ poorly-behaved workloads that may be harming system health. requests assigned to that priority level. {{< /note >}} -* `apiserver_flowcontrol_request_execution_seconds` gives a histogram of how - long requests took to actually execute, grouped by the FlowSchema that matched the - request and the PriorityLevel to which it was assigned. +* `apiserver_flowcontrol_request_execution_seconds` is a vector of + histograms of how long requests took to actually execute, broken + down by the labels `flowSchema` (indicating which one matched the + request), `priorityLevel` (indicating the one to which the request + was assigned). From 8250aa2df07485576b92a0d86a26f571b86818a5 Mon Sep 17 00:00:00 2001 From: Mike Spreitzer Date: Fri, 31 Jul 2020 15:26:47 -0400 Subject: [PATCH 2/2] Tweak APF metric doc in response to review --- .../cluster-administration/flow-control.md | 71 +++++++++---------- 1 file changed, 35 insertions(+), 36 deletions(-) diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md index f6cafecb52..fb0a72a60e 100644 --- a/content/en/docs/concepts/cluster-administration/flow-control.md +++ b/content/en/docs/concepts/cluster-administration/flow-control.md @@ -308,12 +308,12 @@ exports additional metrics. Monitoring these can help you determine whether your configuration is inappropriately throttling important traffic, or find poorly-behaved workloads that may be harming system health. -* `apiserver_flowcontrol_rejected_requests_total` is a vector of - counters (cumulative since server start) of requests that were - rejected, broken down by the labels `flowSchema` (indicating the one - that matched the request), `priorityLevel` (indicating the one to - which ithe request was assigned), and `reason`. The `reason` label - will be have one of the following values: +* `apiserver_flowcontrol_rejected_requests_total` is a counter vector + (cumulative since server start) of requests that were rejected, + broken down by the labels `flowSchema` (indicating the one that + matched the request), `priorityLevel` (indicating the one to which + the request was assigned), and `reason`. The `reason` label will be + have one of the following values: * `queue-full`, indicating that too many requests were already queued, * `concurrency-limit`, indicating that the @@ -322,13 +322,13 @@ poorly-behaved workloads that may be harming system health. * `time-out`, indicating that the request was still in the queue when its queuing time limit expired. -* `apiserver_flowcontrol_dispatched_requests_total` is a vector of - counters (cumulative since server start) of requests that began +* `apiserver_flowcontrol_dispatched_requests_total` is a counter + vector (cumulative since server start) of requests that began executing, broken down by the labels `flowSchema` (indicating the one that matched the request) and `priorityLevel` (indicating the - one to which ithe request was assigned). + one to which the request was assigned). -* `apiserver_current_inqueue_requests` is a vector gauges of recent +* `apiserver_current_inqueue_requests` is a gauge vector of recent high water marks of the number of queued requests, grouped by a label named `request_kind` whose value is `mutating` or `readOnly`. These high water marks describe the largest number seen in the one @@ -338,14 +338,14 @@ poorly-behaved workloads that may be harming system health. served. * `apiserver_flowcontrol_read_vs_write_request_count_samples` is a - vector of histograms of observations of the then-current number of + histogram vector of observations of the then-current number of requests, broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on - the values `mutating` and `readOnly`). The observations ae made + the values `mutating` and `readOnly`). The observations are made periodically at a high rate. * `apiserver_flowcontrol_read_vs_write_request_count_watermarks` is a - vector of histograms of high or low water marks of the number of + histogram vector of high or low water marks of the number of requests broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `request_kind` (which takes on the values `mutating` and `readOnly`); the label `mark` takes on @@ -354,18 +354,17 @@ poorly-behaved workloads that may be harming system health. `apiserver_flowcontrol_read_vs_write_request_count_samples`. These water marks show the range of values that occurred between samples. -* `apiserver_flowcontrol_current_inqueue_requests` is a vector of - gauges holding the instantaneous number of queued (not executing) - requests, broken down by the labels `priorityLevel` and - `flowSchema`. +* `apiserver_flowcontrol_current_inqueue_requests` is a gauge vector + holding the instantaneous number of queued (not executing) requests, + broken down by the labels `priorityLevel` and `flowSchema`. -* `apiserver_flowcontrol_current_executing_requests` is a vector of - gauges holding the instantaneous number of executing (not waiting in - a queue) requests, broken down by the labels `priorityLevel` and +* `apiserver_flowcontrol_current_executing_requests` is a gauge vector + holding the instantaneous number of executing (not waiting in a + queue) requests, broken down by the labels `priorityLevel` and `flowSchema`. * `apiserver_flowcontrol_priority_level_request_count_samples` is a - vector of histograms of observations of the then-current number of + histogram vector of observations of the then-current number of requests broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `priorityLevel`. Each histogram gets observations taken periodically, up through the last @@ -373,7 +372,7 @@ poorly-behaved workloads that may be harming system health. rate. * `apiserver_flowcontrol_priority_level_request_count_watermarks` is a - vector of histograms of high or low water marks of the number of + histogram vector of high or low water marks of the number of requests broken down by the labels `phase` (which takes on the values `waiting` and `executing`) and `priorityLevel`; the label `mark` takes on values `high` and `low`. The water marks are @@ -383,7 +382,7 @@ poorly-behaved workloads that may be harming system health. water marks show the range of values that occurred between samples. * `apiserver_flowcontrol_request_queue_length_after_enqueue` is a - vector of histograms of queue lengths for the queues, broken down by + histogram vector of queue lengths for the queues, broken down by the labels `priorityLevel` and `flowSchema`, as sampled by the enqueued requests. Each request that gets queued contributes one sample to its histogram, reporting the length of the queue just @@ -398,15 +397,15 @@ poorly-behaved workloads that may be harming system health. to increase that PriorityLevelConfiguration's concurrency shares. {{< /note >}} -* `apiserver_flowcontrol_request_concurrency_limit` is a vector of - gauges hoding the computed concurrency limit (based on the API - server's total concurrency limit and PriorityLevelConfigurations' - concurrency shares), broken down by the label `priorityLevel`. +* `apiserver_flowcontrol_request_concurrency_limit` is a gauge vector + hoding the computed concurrency limit (based on the API server's + total concurrency limit and PriorityLevelConfigurations' concurrency + shares), broken down by the label `priorityLevel`. -* `apiserver_flowcontrol_request_wait_duration_seconds` is a vector of - histograms of how long requests spent queued, broken down by the - labels `flowSchema` (indicating which one matched the request), - `priorityLevel` (indicating the one to which ithe request was +* `apiserver_flowcontrol_request_wait_duration_seconds` is a histogram + vector of how long requests spent queued, broken down by the labels + `flowSchema` (indicating which one matched the request), + `priorityLevel` (indicating the one to which the request was assigned), and `execute` (indicating whether the request started executing). {{< note >}} @@ -416,11 +415,11 @@ poorly-behaved workloads that may be harming system health. requests assigned to that priority level. {{< /note >}} -* `apiserver_flowcontrol_request_execution_seconds` is a vector of - histograms of how long requests took to actually execute, broken - down by the labels `flowSchema` (indicating which one matched the - request), `priorityLevel` (indicating the one to which the request - was assigned). +* `apiserver_flowcontrol_request_execution_seconds` is a histogram + vector of how long requests took to actually execute, broken down by + the labels `flowSchema` (indicating which one matched the request) + and `priorityLevel` (indicating the one to which the request was + assigned).