From dc432198e500304a50c0a099f5d1bd08b2d3329b Mon Sep 17 00:00:00 2001
From: Mike Spreitzer <mspreitz@us.ibm.com>
Date: Wed, 22 Jul 2020 17:39:33 -0400
Subject: [PATCH] Document APF debug URL paths

Documented the debug paths introduced by #90967
---
 .../cluster-administration/flow-control.md    | 55 +++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/content/en/docs/concepts/cluster-administration/flow-control.md b/content/en/docs/concepts/cluster-administration/flow-control.md
index 26fc1194df..8a8e631742 100644
--- a/content/en/docs/concepts/cluster-administration/flow-control.md
+++ b/content/en/docs/concepts/cluster-administration/flow-control.md
@@ -303,6 +303,9 @@ to get a mapping of UIDs to names for both FlowSchemas and
 PriorityLevelConfigurations.
 
 ## Observability
+
+### Metrics
+
 When you enable the API Priority and Fairness feature, the kube-apiserver
 exports additional metrics. Monitoring these can help you determine whether your
 configuration is inappropriately throttling important traffic, or find
@@ -365,9 +368,61 @@ poorly-behaved workloads that may be harming system health.
   long requests took to actually execute, grouped by the FlowSchema that matched the
   request and the PriorityLevel to which it was assigned.
 
+### Debug endpoints
 
+When you enable the API Priority and Fairness feature, the kube-apiserver serves the following additional paths at its HTTP[S] ports.
 
+- `/debug/api_priority_and_fairness/dump_priority_levels` - a listing of all the priority levels and the current state of each.  You can fetch like this:
+  ```shell
+  kubectl get --raw /debug/api_priority_and_fairness/dump_priority_levels
+  ```
+  The output is similar to this:
+  ```
+  PriorityLevelName, ActiveQueues, IsIdle, IsQuiescing, WaitingRequests, ExecutingRequests,
+  workload-low,      0,            true,   false,       0,               0,
+  global-default,    0,            true,   false,       0,               0,
+  exempt,            <none>,       <none>, <none>,      <none>,          <none>,
+  catch-all,         0,            true,   false,       0,               0,
+  system,            0,            true,   false,       0,               0,
+  leader-election,   0,            true,   false,       0,               0,
+  workload-high,     0,            true,   false,       0,               0,
+  ```
 
+- `/debug/api_priority_and_fairness/dump_queues` - a listing of all the queues and their current state.  You can fetch like this:
+  ```shell
+  kubectl get --raw /debug/api_priority_and_fairness/dump_queues
+  ```
+  The output is similar to this:
+  ```
+  PriorityLevelName, Index,  PendingRequests, ExecutingRequests, VirtualStart,
+  workload-high,     0,      0,               0,                 0.0000,
+  workload-high,     1,      0,               0,                 0.0000,
+  workload-high,     2,      0,               0,                 0.0000,
+  ...
+  leader-election,   14,     0,               0,                 0.0000,
+  leader-election,   15,     0,               0,                 0.0000,
+  ```
+
+- `/debug/api_priority_and_fairness/dump_requests` - a listing of all the requests that are currently waiting in a queue.  You can fetch like this:
+  ```shell
+  kubectl get --raw /debug/api_priority_and_fairness/dump_requests
+  ```
+  The output is similar to this:
+  ```
+  PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime,
+  exempt,            <none>,         <none>,     <none>,              <none>,          <none>,
+  ```
+
+  You can get a more detailed listing with a command like this:
+  ```shell
+  kubectl get --raw '/debug/api_priority_and_fairness/dump_requests?includeRequestDetails=1'
+  ```
+  The output is similar to this:
+  ```
+  PriorityLevelName, FlowSchemaName, QueueIndex, RequestIndexInQueue, FlowDistingsher, ArriveTime, UserName, Verb,   APIPath, Namespace, Name,   APIVersion, Resource, SubResource,
+  exempt,            <none>,         <none>,     <none>,              <none>,          <none>,     <none>,   <none>, <none>,  <none>,    <none>, <none>,     <none>,   <none>,
+  ```
+  
 ## {{% heading "whatsnext" %}}