whentimeslows/website
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add configuration reference for API server encryption

Browse Source
This commit is contained in:
Qiming Teng
2022-02-10 11:27:52 +08:00
parent 6b6c7a2df8
commit 3e58334b07
3 changed files with 385 additions and 83 deletions
Download Patch File Download Diff File Expand all files Collapse all files
content/en/docs/reference/_index.md
+1
View File
@@ -75,6 +75,7 @@ operator to use or manage a cluster.
* [kube-apiserver configuration (v1)](/docs/reference/config-api/apiserver-config.v1/) * [kube-apiserver configuration (v1)](/docs/reference/config-api/apiserver-config.v1/)
* [kube-apiserver encryption (v1)](/docs/reference/config-api/apiserver-encryption.v1/)
* [kubelet configuration (v1alpha1)](/docs/reference/config-api/kubelet-config.v1alpha1/) and * [kubelet configuration (v1alpha1)](/docs/reference/config-api/kubelet-config.v1alpha1/) and
[kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/) [kubelet configuration (v1beta1)](/docs/reference/config-api/kubelet-config.v1beta1/)
* [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) and * [kube-scheduler configuration (v1beta2)](/docs/reference/config-api/kube-scheduler-config.v1beta2/) and
content/en/docs/reference/config-api/apiserver-encryption.v1.md
+279
View File
@@ -0,0 +1,279 @@
---
title: kube-apiserver Encryption Configuration (v1)
content_type: tool-reference
package: apiserver.config.k8s.io/v1
auto_generated: true
---
<p>Package v1 is the v1 version of the API.</p>
## Resource Types
- [EncryptionConfiguration](#apiserver-config-k8s-io-v1-EncryptionConfiguration)
## `EncryptionConfiguration` {#apiserver-config-k8s-io-v1-EncryptionConfiguration}
<p>EncryptionConfiguration stores the complete configuration for encryption providers.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>apiVersion</code><br/>string</td><td><code>apiserver.config.k8s.io/v1</code></td></tr>
<tr><td><code>kind</code><br/>string</td><td><code>EncryptionConfiguration</code></td></tr>
<tr><td><code>resources</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-ResourceConfiguration"><code>[]ResourceConfiguration</code></a>
</td>
<td>
<p>resources is a list containing resources, and their corresponding encryption providers.</p>
</td>
</tr>
</tbody>
</table>
## `AESConfiguration` {#apiserver-config-k8s-io-v1-AESConfiguration}
**Appears in:**
- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration)
<p>AESConfiguration contains the API configuration for an AES transformer.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>keys</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-Key"><code>[]Key</code></a>
</td>
<td>
<p>keys is a list of keys to be used for creating the AES transformer.
Each key has to be 32 bytes long for AES-CBC and 16, 24 or 32 bytes for AES-GCM.</p>
</td>
</tr>
</tbody>
</table>
## `IdentityConfiguration` {#apiserver-config-k8s-io-v1-IdentityConfiguration}
**Appears in:**
- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration)
<p>IdentityConfiguration is an empty struct to allow identity transformer in provider configuration.</p>
## `KMSConfiguration` {#apiserver-config-k8s-io-v1-KMSConfiguration}
**Appears in:**
- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration)
<p>KMSConfiguration contains the name, cache size and path to configuration file for a KMS based envelope transformer.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>name</code> <B>[Required]</B><br/>
<code>string</code>
</td>
<td>
<p>name is the name of the KMS plugin to be used.</p>
</td>
</tr>
<tr><td><code>cachesize</code><br/>
<code>int32</code>
</td>
<td>
<p>cachesize is the maximum number of secrets which are cached in memory. The default value is 1000.
Set to a negative value to disable caching.</p>
</td>
</tr>
<tr><td><code>endpoint</code> <B>[Required]</B><br/>
<code>string</code>
</td>
<td>
<p>endpoint is the gRPC server listening address, for example &quot;unix:///var/run/kms-provider.sock&quot;.</p>
</td>
</tr>
<tr><td><code>timeout</code><br/>
<a href="https://pkg.go.dev/k8s.io/apimachinery/pkg/apis/meta/v1#Duration"><code>meta/v1.Duration</code></a>
</td>
<td>
<p>timeout for gRPC calls to kms-plugin (ex. 5s). The default is 3 seconds.</p>
</td>
</tr>
</tbody>
</table>
## `Key` {#apiserver-config-k8s-io-v1-Key}
**Appears in:**
- [AESConfiguration](#apiserver-config-k8s-io-v1-AESConfiguration)
- [SecretboxConfiguration](#apiserver-config-k8s-io-v1-SecretboxConfiguration)
<p>Key contains name and secret of the provided key for a transformer.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>name</code> <B>[Required]</B><br/>
<code>string</code>
</td>
<td>
<p>name is the name of the key to be used while storing data to disk.</p>
</td>
</tr>
<tr><td><code>secret</code> <B>[Required]</B><br/>
<code>string</code>
</td>
<td>
<p>secret is the actual key, encoded in base64.</p>
</td>
</tr>
</tbody>
</table>
## `ProviderConfiguration` {#apiserver-config-k8s-io-v1-ProviderConfiguration}
**Appears in:**
- [ResourceConfiguration](#apiserver-config-k8s-io-v1-ResourceConfiguration)
<p>ProviderConfiguration stores the provided configuration for an encryption provider.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>aesgcm</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-AESConfiguration"><code>AESConfiguration</code></a>
</td>
<td>
<p>aesgcm is the configuration for the AES-GCM transformer.</p>
</td>
</tr>
<tr><td><code>aescbc</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-AESConfiguration"><code>AESConfiguration</code></a>
</td>
<td>
<p>aescbc is the configuration for the AES-CBC transformer.</p>
</td>
</tr>
<tr><td><code>secretbox</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-SecretboxConfiguration"><code>SecretboxConfiguration</code></a>
</td>
<td>
<p>secretbox is the configuration for the Secretbox based transformer.</p>
</td>
</tr>
<tr><td><code>identity</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-IdentityConfiguration"><code>IdentityConfiguration</code></a>
</td>
<td>
<p>identity is the (empty) configuration for the identity transformer.</p>
</td>
</tr>
<tr><td><code>kms</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-KMSConfiguration"><code>KMSConfiguration</code></a>
</td>
<td>
<p>kms contains the name, cache size and path to configuration file for a KMS based envelope transformer.</p>
</td>
</tr>
</tbody>
</table>
## `ResourceConfiguration` {#apiserver-config-k8s-io-v1-ResourceConfiguration}
**Appears in:**
- [EncryptionConfiguration](#apiserver-config-k8s-io-v1-EncryptionConfiguration)
<p>ResourceConfiguration stores per resource configuration.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>resources</code> <B>[Required]</B><br/>
<code>[]string</code>
</td>
<td>
<p>resources is a list of kubernetes resources which have to be encrypted.</p>
</td>
</tr>
<tr><td><code>providers</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-ProviderConfiguration"><code>[]ProviderConfiguration</code></a>
</td>
<td>
<p>providers is a list of transformers to be used for reading and writing the resources to disk.
eg: aesgcm, aescbc, secretbox, identity.</p>
</td>
</tr>
</tbody>
</table>
## `SecretboxConfiguration` {#apiserver-config-k8s-io-v1-SecretboxConfiguration}
**Appears in:**
- [ProviderConfiguration](#apiserver-config-k8s-io-v1-ProviderConfiguration)
<p>SecretboxConfiguration contains the API configuration for an Secretbox transformer.</p>
<table class="table">
<thead><tr><th width="30%">Field</th><th>Description</th></tr></thead>
<tbody>
<tr><td><code>keys</code> <B>[Required]</B><br/>
<a href="#apiserver-config-k8s-io-v1-Key"><code>[]Key</code></a>
</td>
<td>
<p>keys is a list of keys to be used for creating the Secretbox transformer.
Each key has to be 32 bytes long.</p>
</td>
</tr>
</tbody>
</table>
content/en/docs/tasks/administer-cluster/encrypt-data.md
+105 -83
View File
@@ -1,7 +1,7 @@
--- ---
title: Encrypting Secret Data at Rest
reviewers: reviewers:
- smarterclayton - smarterclayton
title: Encrypting Secret Data at Rest
content_type: task content_type: task
min-kubernetes-server-version: 1.13 min-kubernetes-server-version: 1.13
--- ---
@@ -9,27 +9,26 @@ min-kubernetes-server-version: 1.13
<!-- overview --> <!-- overview -->
This page shows how to enable and configure encryption of secret data at rest. This page shows how to enable and configure encryption of secret data at rest.
## {{% heading "prerequisites" %}} ## {{% heading "prerequisites" %}}
* {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}} * {{< include "task-tutorial-prereqs.md" >}} {{< version-check >}}
* etcd v3.0 or later is required * etcd v3.0 or later is required
<!-- steps --> <!-- steps -->
## Configuration and determining whether encryption at rest is already enabled ## Configuration and determining whether encryption at rest is already enabled
The `kube-apiserver` process accepts an argument `--encryption-provider-config` The `kube-apiserver` process accepts an argument `--encryption-provider-config`
that controls how API data is encrypted in etcd. An example configuration that controls how API data is encrypted in etcd.
is provided below. The configuration is provided as an API named
[`EncryptionConfiguration`](/docs/reference/config-api/apiserver-encryption.v1/).
An example configuration is provided below.
{{< caution >}} {{< caution >}}
**IMPORTANT:** For multi-master configurations (with two or more control plane nodes) the encryption configuration file must be the same! **IMPORTANT:** For high-availability configurations (with two or more control plane nodes), the
Otherwise, the kube-apiserver can't decrypt data stored inside the key-value store. encryption configuration file must be the same! Otherwise, the `kube-apiserver` component cannot
decrypt data stored in the etcd.
{{< /caution >}} {{< /caution >}}
## Understanding the encryption at rest configuration. ## Understanding the encryption at rest configuration.
@@ -39,39 +38,44 @@ apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration kind: EncryptionConfiguration
resources: resources:
- resources: - resources:
- secrets - secrets
providers: providers:
- identity: {} - identity: {}
- aesgcm: - aesgcm:
keys: keys:
- name: key1 - name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ== secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2 - name: key2
secret: dGhpcyBpcyBwYXNzd29yZA== secret: dGhpcyBpcyBwYXNzd29yZA==
- aescbc: - aescbc:
keys: keys:
- name: key1 - name: key1
secret: c2VjcmV0IGlzIHNlY3VyZQ== secret: c2VjcmV0IGlzIHNlY3VyZQ==
- name: key2 - name: key2
secret: dGhpcyBpcyBwYXNzd29yZA== secret: dGhpcyBpcyBwYXNzd29yZA==
- secretbox: - secretbox:
keys: keys:
- name: key1 - name: key1
secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY= secret: YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXoxMjM0NTY=
``` ```
Each `resources` array item is a separate config and contains a complete configuration. The Each `resources` array item is a separate config and contains a complete configuration. The
`resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`) `resources.resources` field is an array of Kubernetes resource names (`resource` or `resource.group`)
that should be encrypted. The `providers` array is an ordered list of the possible encryption that should be encrypted. The `providers` array is an ordered list of the possible encryption
providers. Only one provider type may be specified per entry (`identity` or `aescbc` may be provided, but not both in the same item). providers.
The first provider in the list is used to encrypt resources going into storage. When reading Only one provider type may be specified per entry (`identity` or `aescbc` may be provided,
resources from storage each provider that matches the stored data attempts to decrypt the data in but not both in the same item).
order. If no provider can read the stored data due to a mismatch in format or secret key, an error The first provider in the list is used to encrypt resources written into the storage. When reading
resources from storage, each provider that matches the stored data attempts in order to decrypt the
data. If no provider can read the stored data due to a mismatch in format or secret key, an error
is returned which prevents clients from accessing that resource. is returned which prevents clients from accessing that resource.
For more detailed information about the `EncryptionConfiguration` struct, please refer to the
[encryption configuration API](/docs/reference/config-api/apiserver-encryption.v1/).
{{< caution >}} {{< caution >}}
**IMPORTANT:** If any resource is not readable via the encryption config (because keys were changed), If any resource is not readable via the encryption config (because keys were changed),
the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to the only recourse is to delete that key from the underlying etcd directly. Calls that attempt to
read that resource will fail until it is deleted or a valid decryption key is provided. read that resource will fail until it is deleted or a valid decryption key is provided.
{{< /caution >}} {{< /caution >}}
@@ -90,15 +94,24 @@ Name | Encryption | Strength | Speed | Key Length | Other Considerations
Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider Each provider supports multiple keys - the keys are tried in order for decryption, and if the provider
is the first provider, the first key is used for encryption. is the first provider, the first key is used for encryption.
__Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption.
Please use `kms` provider for additional security.__ By default, the `identity` provider is used to protect secrets in etcd, which
provides no encryption. `EncryptionConfiguration` was introduced to encrypt secrets locally, with a locally managed key.
Encrypting secrets with a locally managed key protects against an etcd compromise, but it fails to protect against a host compromise. {{< caution >}}
Since the encryption keys are stored on the host in the EncryptionConfig YAML file, a skilled attacker can access that file and Storing the raw encryption key in the EncryptionConfig only moderately improves your security
extract the encryption keys. posture, compared to no encryption. Please use `kms` provider for additional security.
{{< /caution >}}
Envelope encryption creates dependence on a separate key, not stored in Kubernetes. In this case, an attacker would need to compromise etcd, the kubeapi-server, and the third-party KMS provider to retrieve the plaintext values, providing a higher level of security than locally-stored encryption keys. By default, the `identity` provider is used to protect Secrets in etcd, which provides no
encryption. `EncryptionConfiguration` was introduced to encrypt Secrets locally, with a locally
managed key.
Encrypting Secrets with a locally managed key protects against an etcd compromise, but it fails to
protect against a host compromise. Since the encryption keys are stored on the host in the
EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption
keys.
Envelope encryption creates dependence on a separate key, not stored in Kubernetes. In this case,
an attacker would need to compromise etcd, the `kubeapi-server`, and the third-party KMS provider to
retrieve the plaintext values, providing a higher level of security than locally stored encryption keys.
## Encrypting your data ## Encrypting your data
@@ -109,113 +122,122 @@ apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration kind: EncryptionConfiguration
resources: resources:
- resources: - resources:
- secrets - secrets
providers: providers:
- aescbc: - aescbc:
keys: keys:
- name: key1 - name: key1
secret: <BASE 64 ENCODED SECRET> secret: <BASE 64 ENCODED SECRET>
- identity: {} - identity: {}
``` ```
To create a new secret perform the following steps: To create a new Secret, perform the following steps:
1. Generate a 32 byte random key and base64 encode it. If you're on Linux or macOS, run the following command: 1. Generate a 32-byte random key and base64 encode it. If you're on Linux or macOS, run the following command:
```shell ```shell
head -c 32 /dev/urandom | base64 head -c 32 /dev/urandom | base64
``` ```
2. Place that value in the secret field. 1. Place that value in the `secret` field of the `EncryptionConfiguration` struct.
3. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to the location of the config file. 1. Set the `--encryption-provider-config` flag on the `kube-apiserver` to point to
4. Restart your API server. the location of the config file.
1. Restart your API server.
{{< caution >}} {{< caution >}}
Your config file contains keys that can decrypt content in etcd, so you must properly restrict permissions on your masters so only the user who runs the kube-apiserver can read it. Your config file contains keys that can decrypt the contents in etcd, so you must properly restrict
permissions on your control-plane nodes so only the user who runs the `kube-apiserver` can read it.
{{< /caution >}} {{< /caution >}}
## Verifying that data is encrypted ## Verifying that data is encrypted
Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or Data is encrypted when written to etcd. After restarting your `kube-apiserver`, any newly created or
updated secret should be encrypted when stored. To check, you can use the `etcdctl` command line updated Secret should be encrypted when stored. To check this, you can use the `etcdctl` command line
program to retrieve the contents of your secret. program to retrieve the contents of your Secret.
1. Create a new secret called `secret1` in the `default` namespace: 1. Create a new Secret called `secret1` in the `default` namespace:
```shell ```shell
kubectl create secret generic secret1 -n default --from-literal=mykey=mydata kubectl create secret generic secret1 -n default --from-literal=mykey=mydata
``` ```
2. Using the etcdctl commandline, read that secret out of etcd: 1. Using the `etcdctl` command line, read that Secret out of etcd:
`ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C` `ETCDCTL_API=3 etcdctl get /registry/secrets/default/secret1 [...] | hexdump -C`
where `[...]` must be the additional arguments for connecting to the etcd server. where `[...]` must be the additional arguments for connecting to the etcd server.
3. Verify the stored secret is prefixed with `k8s:enc:aescbc:v1:` which indicates the `aescbc` provider has encrypted the resulting data. 1. Verify the stored Secret is prefixed with `k8s:enc:aescbc:v1:` which indicates
the `aescbc` provider has encrypted the resulting data.
4. Verify the secret is correctly decrypted when retrieved via the API: 1. Verify the Secret is correctly decrypted when retrieved via the API:
```shell ```shell
kubectl describe secret secret1 -n default kubectl describe secret secret1 -n default
``` ```
should match `mykey: bXlkYXRh`, mydata is encoded, check [decoding a secret](/docs/tasks/configmap-secret/managing-secret-using-kubectl/#decoding-secret) to The output should contain `mykey: bXlkYXRh`, with contents of `mydata` encoded, check
completely decode the secret. [decoding a Secret](/docs/tasks/configmap-secret/managing-secret-using-kubectl/#decoding-secret)
to completely decode the Secret.
## Ensure all Secrets are encrypted
## Ensure all secrets are encrypted Since Secrets are encrypted on write, performing an update on a Secret will encrypt that content.
Since secrets are encrypted on write, performing an update on a secret will encrypt that content.
```shell ```shell
kubectl get secrets --all-namespaces -o json | kubectl replace -f - kubectl get secrets --all-namespaces -o json | kubectl replace -f -
``` ```
The command above reads all secrets and then updates them to apply server side encryption. The command above reads all Secrets and then updates them to apply server side encryption.
{{< note >}} {{< note >}}
If an error occurs due to a conflicting write, retry the command. If an error occurs due to a conflicting write, retry the command.
For larger clusters, you may wish to subdivide the secrets by namespace or script an update. For larger clusters, you may wish to subdivide the secrets by namespace or script an update.
{{< /note >}} {{< /note >}}
## Rotating a decryption key ## Rotating a decryption key
Changing the secret without incurring downtime requires a multi step operation, especially in Changing a Secret without incurring downtime requires a multi-step operation, especially in
the presence of a highly available deployment where multiple `kube-apiserver` processes are running. the presence of a highly-available deployment where multiple `kube-apiserver` processes are running.
1. Generate a new key and add it as the second key entry for the current provider on all servers 1. Generate a new key and add it as the second key entry for the current provider on all servers
2. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key 1. Restart all `kube-apiserver` processes to ensure each server can decrypt using the new key
3. Make the new key the first entry in the `keys` array so that it is used for encryption in the config 1. Make the new key the first entry in the `keys` array so that it is used for encryption in the config
4. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key 1. Restart all `kube-apiserver` processes to ensure each server now encrypts using the new key
5. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all existing secrets with the new key 1. Run `kubectl get secrets --all-namespaces -o json | kubectl replace -f -` to encrypt all
6. Remove the old decryption key from the config after you back up etcd with the new key in use and update all secrets existing Secrets with the new key
1. Remove the old decryption key from the config after you have backed up etcd with the new key in use
With a single `kube-apiserver`, step 2 may be skipped. and updated all Secrets
When running a single `kube-apiserver` instance, step 2 may be skipped.
## Decrypting all data ## Decrypting all data
To disable encryption at rest place the `identity` provider as the first entry in the config: To disable encryption at rest, place the `identity` provider as the first entry in the config
and restart all `kube-apiserver` processes.
```yaml ```yaml
apiVersion: apiserver.config.k8s.io/v1 apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration kind: EncryptionConfiguration
resources: resources:
- resources: - resources:
- secrets - secrets
providers: providers:
- identity: {} - identity: {}
- aescbc: - aescbc:
keys: keys:
- name: key1 - name: key1
secret: <BASE 64 ENCODED SECRET> secret: <BASE 64 ENCODED SECRET>
``` ```
and restart all `kube-apiserver` processes. Then run: Then run the following command to force decrypt
all Secrets:
```shell ```shell
kubectl get secrets --all-namespaces -o json | kubectl replace -f - kubectl get secrets --all-namespaces -o json | kubectl replace -f -
``` ```
to force all secrets to be decrypted.
## {{% heading "whatsnext" %}}
* Learn more about the [EncryptionConfiguration configuration API (v1)](/docs/reference/config-api/apiserver-encryption.v1/).