From e638ab5ee8618cec6796ebdbcb1b96fafc835531 Mon Sep 17 00:00:00 2001 From: Kermit Alexander Date: Wed, 30 Mar 2022 01:40:17 +0000 Subject: [PATCH] Reword resource constraint section. --- .../custom-resource-definitions.md | 119 ++++++++++++++++-- 1 file changed, 107 insertions(+), 12 deletions(-) diff --git a/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md b/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md index af6886ec8e..0576a2c057 100644 --- a/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md +++ b/content/en/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions.md @@ -1051,20 +1051,115 @@ A portion of the schema is correlatable if all `array` parent schemas are of typ #### Resource Constraints -CEL expressions have the potential to consume unacceptable amounts of API server resources. We constrain the resource utilization in following ways: -- Validation of CEL expression's "cost" when a CEL expression is written to a field in a CRD (at CRD creation/update time) -- Runtime cost budget during CEL evaluation - - CEL validation might fail due to runtime cost budget exceed with error message `validation failed due to running out of cost budget, no further validation rules will be run` - - CEL validation might fail due to cost limit exceed per expression with message `operation cancelled: actual cost limit exceeded: no further validation rules will be run due to call cost exceeds limit for rule:{$rule}` -- Go context cancellation to bound CEL expression evaluation to the request lifetime +Resource consumption of validation rules is checked at CustomResourceDefinition creation and update time. If a rule is estimated to be prohibitively expensive to execute, it will result in a validation error. A similar +system is used at runtime that observes the actions the interpreter takes. If the interpreter executes +too many instructions, execution of the rule will be halted, and an error will result. +Each CustomResourceDefinition is also allowed a certain amount of resources to finish executing all of +its validation rules. If the sum total of its rules are estimated at creation time to go over that limit, +then a validation error will also occur. -Guidelines for working with estimated limits: -- Adding MaxItems, MaxProperties and MaxLength limits on all data accessed by CEL rules is the best practice. -- O(n) - For simple rules, it is possible to iterate across a single map/list/string without exceeding the limit, but adding limits on all data accessed by CEL rules is the best practice -- O(n^2)+ the product of the max lengths usually needs to be <1,000,000. E.g. 1000 for 2 levels of nesting, 100 for 3 levels of nesting -- O(n^3) - should generally be avoided +In general, both systems will allow rules that do not need to iterate; these rules will +always take the same amount of time regardless of how large their input is. `self.foo == 1` will be allowed. +But if `foo` is a string and we instead have `self.foo.contains("someString")`, our rule will take +longer to execute depending on how long `foo` is. Another example would be if `foo` was an array, and we +had a rule `self.foo.all(x, x > 5)`. The cost system will always assume the worst-case scenario if +a limit on the length of `foo` is not given, and this will happen for anything that can be iterated +over (lists, maps, etc.). -// TODO: edit info for cost estimation +Because of this, it is considered best practice to put a limit via `maxItems`, `maxProperties`, and +`maxLength` for anything that will be processed in a validation rule in order to prevent validation errors during cost estimation. For example, given this schema with one rule: + +```yaml +openAPIV3Schema: + type: object + properties: + foo: + type: array + items: + type: string + x-kubernetes-validations: + - rule: "self.all(x, x.contains('a string'))" +``` + +The cost system will not allow this rule. Using `self.all` means calling `contains` on every string in `foo`, +which in turn will check the given string to see if it contains `'a string'`. Without limits, this is a very +expensive rule: + +``` + spec.validation.openAPIV3Schema.properties[spec].properties[foo].x-kubernetes-validations[0].rule: Forbidden: + CEL rule exceeded budget by more than 100x (try simplifying the rule, or adding maxItems, maxProperties, and + maxLength where arrays, maps, and strings are used) +``` + +Without limits being set, the estimated cost of this rule will exceed the per-rule cost limit. But if we +add limits in the appropriate places, the rule will be allowed: + +```yaml +openAPIV3Schema: + type: object + properties: + foo: + type: array + maxItems: 25 + items: + type: string + maxLength: 10 + x-kubernetes-validations: + - rule: "self.all(x, x.contains('a string'))" +``` + +The cost estimation system takes into account how many times the rule will be executed in addition to the +estimated cost of the rule itself. For instance, the following rule will have the same estimated cost as the +previous example (despite the rule now being defined on the individual array items): + +```yaml +openAPIV3Schema: + type: object + properties: + foo: + type: array + maxItems: 25 + items: + type: string + x-kubernetes-validations: + - rule: "self.contains('a string'))" + maxLength: 10 +``` + +If a list inside of a list has a validation rule that uses `self.all`, that is significantly more expensive +than a non-nested list with the same rule. A rule that would have been allowed on a non-nested list might need lower limits set on both nested lists in order to be allowed. For example, even without having limits set, +the following rule is allowed: + +```yaml +openAPIV3Schema: + type: object + properties: + foo: + type: array + items: + type: integer + x-kubernetes-validations: + - rule: "self.all(x, x == 5)" +``` + +But the same rule on the following schema (with a nested array added) produces a validation error: + +```yaml +openAPIV3Schema: + type: object + properties: + foo: + type: array + items: + type: array + items: + type: integer + x-kubernetes-validations: + - rule: "self.all(x, x == 5)" +``` + +This is because each item of `foo` is itself an array, and each subarray in turn calls `self.all`. Avoid nested +lists and maps if possible where validation rules are used. ### Defaulting