Administering Cluters section; Edit This Page funcitonality

This commit is contained in:
John Mulhausen
2016-02-11 02:15:49 -08:00
parent 5a960ec8e7
commit f782b2e0fb
439 changed files with 156 additions and 86841 deletions
+7 -22
View File
@@ -1,18 +1,16 @@
---
layout: docwithnav
title: "Kubernetes Large Cluster"
title: "Using Large Clusters"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# Kubernetes Large Cluster
## Support
At v1.0, Kubernetes supports clusters up to 100 nodes with 30 pods per node and 1-2 containers per pod.
## Table of Contents
{% include pagetoc.html %}
## Setup
A cluster is a set of nodes (physical or virtual machines) running Kubernetes agents, managed by a "master" (the cluster-level control plane).
@@ -70,17 +68,4 @@ To avoid running into cluster addon resource issues, when creating a cluster wit
* [FluentD with ElasticSearch Plugin](http://releases.k8s.io/release-1.1/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml)
* [FluentD with GCP Plugin](http://releases.k8s.io/release-1.1/cluster/saltbase/salt/fluentd-gcp/fluentd-gcp.yaml)
For directions on how to detect if addon containers are hitting resource limits, see the [Troubleshooting section of Compute Resources](../user-guide/compute-resources.html#troubleshooting).
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/admin/cluster-large.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
For directions on how to detect if addon containers are hitting resource limits, see the [Troubleshooting section of Compute Resources](../user-guide/compute-resources.html#troubleshooting).
+6 -45
View File
@@ -1,37 +1,7 @@
---
layout: docwithnav
title: "High Availability Kubernetes Clusters"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# High Availability Kubernetes Clusters
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [High Availability Kubernetes Clusters](#high-availability-kubernetes-clusters)
- [Introduction](#introduction)
- [Overview](#overview)
- [Initial set-up](#initial-set-up)
- [Reliable nodes](#reliable-nodes)
- [Establishing a redundant, reliable data storage layer](#establishing-a-redundant-reliable-data-storage-layer)
- [Clustering etcd](#clustering-etcd)
- [Validating your cluster](#validating-your-cluster)
- [Even more reliable storage](#even-more-reliable-storage)
- [Replicated API Servers](#replicated-api-servers)
- [Installing configuration files](#installing-configuration-files)
- [Starting the API Server](#starting-the-api-server)
- [Load balancing](#load-balancing)
- [Master elected components](#master-elected-components)
- [Installing configuration files](#installing-configuration-files)
- [Running the podmaster](#running-the-podmaster)
- [Conclusion](#conclusion)
- [Vagrant up!](#vagrant-up)
<!-- END MUNGE: GENERATED_TOC -->
## Introduction
@@ -43,6 +13,10 @@ or try [Google Container Engine](https://cloud.google.com/container-engine/) for
Also, at this time high availability support for Kubernetes is not continuously tested in our end-to-end (e2e) testing. We will
be working to add this continuous testing, but for now the single-node master installations are more heavily tested.
## Table of Contents
{% include pagetoc.html %}
## Overview
Setting up a truly reliable, highly available distributed system requires a number of steps, it is akin to
@@ -264,17 +238,4 @@ set the `--apiserver` flag to your replicated endpoint.
We indeed have an initial proof of concept tester for this, which is available [here](https://releases.k8s.io/release-1.1/examples/high-availability).
It implements the major concepts (with a few minor reductions for simplicity), of the podmaster HA implementation alongside a quick smoke test using k8petstore.
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/admin/high-availability.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
It implements the major concepts (with a few minor reductions for simplicity), of the podmaster HA implementation alongside a quick smoke test using k8petstore.
+6 -21
View File
@@ -1,17 +1,15 @@
---
layout: docwithnav
title: "Kubernetes Cluster Admin Guide"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# Kubernetes Cluster Admin Guide
The cluster admin guide is for anyone creating or administering a Kubernetes cluster.
It assumes some familiarity with concepts in the [User Guide](../user-guide/README.html).
## Table of Contents
{% include pagetoc.html %}
## Planning a cluster
There are many different examples of how to setup a kubernetes cluster. Many of them are listed in this
@@ -80,17 +78,4 @@ project](salt.html).
* **Authorization** [authorization](authorization.html)
* **Admission Controllers** [admission_controllers](admission-controllers.html)
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/admin/introduction.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
* **Admission Controllers** [admission_controllers](admission-controllers.html)
+2 -21
View File
@@ -1,13 +1,7 @@
---
layout: docwithnav
title: "Considerations for running multiple Kubernetes clusters"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# Considerations for running multiple Kubernetes clusters
You may want to set up multiple Kubernetes clusters, both to
have clusters in different regions to be nearer to your users, and to tolerate failures and/or invasive maintenance.
@@ -67,17 +61,4 @@ you may need even more clusters. Kubernetes v1.0 currently supports clusters up
When you have multiple clusters, you would typically create services with the same config in each cluster and put each of those
service instances behind a load balancer (AWS Elastic Load Balancer, GCE Forwarding Rule or HTTP Load Balancer) spanning all of them, so that
failures of a single cluster are not visible to end users.
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/admin/multi-cluster.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
failures of a single cluster are not visible to end users.
+2 -21
View File
@@ -1,13 +1,7 @@
---
layout: docwithnav
title: "Kubernetes Namespaces"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
## Kubernetes Namespaces
Kubernetes _[namespaces](../../../docs/admin/namespaces.html)_ help different projects, teams, or customers to share a Kubernetes cluster.
@@ -286,17 +280,4 @@ cattle-zz7ea 1/1 Running 0 12s
At this point, it should be clear that the resources users create in one namespace are hidden from the other namespace.
As the policy support in Kubernetes evolves, we will extend this scenario to show how you can provide different
authorization rules for each namespace.
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/admin/namespaces/README.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
authorization rules for each namespace.
+8 -7
View File
@@ -3,18 +3,19 @@ title: "Running Kubernetes locally via Docker"
section: guides
---
## Table of Contents
{% include pagetoc.html %}
### Overview
## Overview
The following instructions show you how to set up a simple, single node Kubernetes cluster using Docker.
Here's a diagram of what the final result will look like:
![Kubernetes Single Node on Docker](k8s-singlenode-docker.png)
### Prerequisites
![Kubernetes Single Node on Docker](/{{ page.version }}/docs/getting-started-guides/k8s-singlenode-docker.png)
## Table of Contents
{% include pagetoc.html %}
## Prerequisites
1. You need to have docker installed on one machine.
2. Your kernel should support memory and swap accounting. Ensure that the
+4 -37
View File
@@ -1,30 +1,10 @@
---
layout: docwithnav
title: "Getting started locally"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
## Table of Contents
<!-- END MUNGE: UNVERSIONED_WARNING -->
Getting started locally
-----------------------
**Table of Contents**
- [Requirements](#requirements)
- [Linux](#linux)
- [Docker](#docker)
- [etcd](#etcd)
- [go](#go)
- [Starting the cluster](#starting-the-cluster)
- [Running a container](#running-a-container)
- [Running a user defined pod](#running-a-user-defined-pod)
- [Troubleshooting](#troubleshooting)
- [I cannot reach service IPs on the network.](#i-cannot-reach-service-ips-on-the-network)
- [I cannot create a replication controller with replica size greater than 1! What gives?](#i-cannot-create-a-replication-controller-with-replica-size-greater-than-1--what-gives)
- [I changed Kubernetes code, how do I run it?](#i-changed-kubernetes-code-how-do-i-run-it)
- [kubectl claims to start a container but `get pods` and `docker ps` don't show it.](#kubectl-claims-to-start-a-container-but-get-pods-and-docker-ps-dont-show-it)
- [The pods fail to connect to the services by host names](#the-pods-fail-to-connect-to-the-services-by-host-names)
{% include pagetoc.html %}
### Requirements
@@ -145,17 +125,4 @@ One or more of the KUbernetes daemons might've crashed. Tail the logs of each in
#### The pods fail to connect to the services by host names
The local-up-cluster.sh script doesn't start a DNS service. Similar situation can be found [here](http://issue.k8s.io/6667). You can start a manually. Related documents can be found [here](https://releases.k8s.io/release-1.1/cluster/addons/dns#how-do-i-configure-it)
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/getting-started-guides/locally.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
The local-up-cluster.sh script doesn't start a DNS service. Similar situation can be found [here](http://issue.k8s.io/6667). You can start a manually. Related documents can be found [here](https://releases.k8s.io/release-1.1/cluster/addons/dns#how-do-i-configure-it)
+22 -22
View File
@@ -1,11 +1,11 @@
---
title: "Getting started from Scratch"
title: "Creating a Custom Cluster From Scratch"
section: guides
---
This guide is for people who want to craft a custom Kubernetes cluster. If you
can find an existing Getting Started Guide that meets your needs on [this
list](README.html), then we recommend using it, as you will be able to benefit
list](/{{ page.version }}/docs/getting-started-guides/README/), then we recommend using it, as you will be able to benefit
from the experience of others. However, if you have specific IaaS, networking,
configuration management, or operating system requirements not met by any of
those guides, then this guide will provide an outline of the steps you need to
@@ -25,10 +25,10 @@ steps that existing cluster setup scripts are making.
1. You should be familiar with using Kubernetes already. We suggest you set
up a temporary cluster by following one of the other Getting Started Guides.
This will help you become familiar with the CLI ([kubectl](../user-guide/kubectl/kubectl.html)) and concepts ([pods](../user-guide/pods.html), [services](../user-guide/services.html), etc.) first.
This will help you become familiar with the CLI ([kubectl](/{{ page.version }}/docs/user-guide/kubectl/kubectl)) and concepts ([pods](/{{ page.version }}/docs/user-guide/pods), [services](/{{ page.version }}/docs/user-guide/services), etc.) first.
1. You should have `kubectl` installed on your desktop. This will happen as a side
effect of completing one of the other Getting Started Guides. If not, follow the instructions
[here](../user-guide/prereqs.html).
[here](/{{ page.version }}/docs/user-guide/prereqs).
### Cloud Provider
@@ -55,7 +55,7 @@ on how flags are set on various components.
### Network
Kubernetes has a distinctive [networking model](../admin/networking.html).
Kubernetes has a distinctive [networking model](/{{ page.version }}/docs/admin/networking).
Kubernetes allocates an IP address to each pod. When creating a cluster, you
need to allocate a block of IPs for Kubernetes to use as Pod IPs. The simplest
@@ -65,7 +65,7 @@ another pod using the IP of the second pod. This connectivity can be
accomplished in two ways:
- Configure network to route Pod IPs
- Harder to setup from scratch.
- Google Compute Engine ([GCE](gce.html)) and [AWS](aws.html) guides use this approach.
- Google Compute Engine ([GCE](gce)) and [AWS](aws) guides use this approach.
- Need to make the Pod IPs routable by programming routers, switches, etc.
- Can be configured external to Kubernetes, or can implement in the "Routes" interface of a Cloud Provider module.
- Generally highest performance.
@@ -97,7 +97,7 @@ You need to select an address range for the Pod IPs.
using `10.10.0.0/24` through `10.10.255.0/24`, respectively.
- Need to make these routable or connect with overlay.
Kubernetes also allocates an IP to each [service](../user-guide/services.html). However,
Kubernetes also allocates an IP to each [service](/{{ page.version }}/docs/user-guide/services). However,
service IPs do not necessarily need to be routable. The kube-proxy takes care
of translating Service IPs to Pod IPs before traffic leaves the node. You do
need to Allocate a block of IPs for services. Call this
@@ -139,7 +139,7 @@ You will need binaries for:
A Kubernetes binary release includes all the Kubernetes binaries as well as the supported release of etcd.
You can use a Kubernetes binary release (recommended) or build your Kubernetes binaries following the instructions in the
[Developer Documentation](../devel/README.html). Only using a binary release is covered in this guide.
[Developer Documentation](/{{ page.version }}/docs/devel/README). Only using a binary release is covered in this guide.
Download the [latest binary release](https://github.com/kubernetes/kubernetes/releases/latest) and unzip it.
Then locate `./kubernetes/server/kubernetes-server-linux-amd64.tar.gz` and unzip *that*.
@@ -229,10 +229,10 @@ The admin user (and any users) need:
Your tokens and passwords need to be stored in a file for the apiserver
to read. This guide uses `/var/lib/kube-apiserver/known_tokens.csv`.
The format for this file is described in the [authentication documentation](../admin/authentication.html).
The format for this file is described in the [authentication documentation](/{{ page.version }}/docs/admin/authentication).
For distributing credentials to clients, the convention in Kubernetes is to put the credentials
into a [kubeconfig file](../user-guide/kubeconfig-file.html).
into a [kubeconfig file](/{{ page.version }}/docs/user-guide/kubeconfig-file).
The kubeconfig file for the administrator can be created as follows:
- If you have already used Kubernetes with a non-custom cluster (for example, used a Getting Started
@@ -346,7 +346,7 @@ The minimum version required is [v0.5.6](https://github.com/coreos/rkt/releases/
[systemd](http://www.freedesktop.org/wiki/Software/systemd/) is required on your node to run rkt. The
minimum version required to match rkt v0.5.6 is
[systemd 215](http://lists.freedesktop.org/archives/systemd-devel/2014-July/020903.html).
[systemd 215](http://lists.freedesktop.org/archives/systemd-devel/2014-July/020903).
[rkt metadata service](https://github.com/coreos/rkt/blob/master/Documentation/networking.md) is also required
for rkt networking support. You can start rkt metadata service by using command like
@@ -371,7 +371,7 @@ Arguments to consider:
- `--docker-root=`
- `--root-dir=`
- `--configure-cbr0=` (described above)
- `--register-node` (described in [Node](../admin/node.html) documentation.)
- `--register-node` (described in [Node](/{{ page.version }}/docs/admin/node) documentation.)
### kube-proxy
@@ -392,7 +392,7 @@ Each node needs to be allocated its own CIDR range for pod networking.
Call this `NODE_X_POD_CIDR`.
A bridge called `cbr0` needs to be created on each node. The bridge is explained
further in the [networking documentation](../admin/networking.html). The bridge itself
further in the [networking documentation](/{{ page.version }}/docs/admin/networking). The bridge itself
needs an address from `$NODE_X_POD_CIDR` - by convention the first IP. Call
this `NODE_X_BRIDGE_ADDR`. For example, if `NODE_X_POD_CIDR` is `10.0.0.0/16`,
then `NODE_X_BRIDGE_ADDR` is `10.0.0.1/16`. NOTE: this retains the `/16` suffix
@@ -425,7 +425,7 @@ iptables -w -t nat -A POSTROUTING -o eth0 -j MASQUERADE \! -d ${CLUSTER_SUBNET}
This will rewrite the source address from
the PodIP to the Node IP for traffic bound outside the cluster, and kernel
[connection tracking](http://www.iptables.info/en/connection-state.html)
[connection tracking](http://www.iptables.info/en/connection-state)
will ensure that responses destined to the node still reach
the pod.
@@ -446,7 +446,7 @@ traffic to the internet, but have no problem with them inside your GCE Project.
The previous steps all involved "conventional" system administration techniques for setting up
machines. You may want to use a Configuration Management system to automate the node configuration
process. There are examples of [Saltstack](../admin/salt.html), Ansible, Juju, and CoreOS Cloud Config in the
process. There are examples of [Saltstack](/{{ page.version }}/docs/admin/salt), Ansible, Juju, and CoreOS Cloud Config in the
various Getting Started Guides.
## Bootstrapping the Cluster
@@ -466,7 +466,7 @@ You will need to run one or more instances of etcd.
- Alternative: run 3 or 5 etcd instances.
- Log can be written to non-durable storage because storage is replicated.
- run a single apiserver which connects to one of the etc nodes.
See [cluster-troubleshooting](../admin/cluster-troubleshooting.html) for more discussion on factors affecting cluster
See [cluster-troubleshooting](/{{ page.version }}/docs/admin/cluster-troubleshooting) for more discussion on factors affecting cluster
availability.
To run an etcd instance:
@@ -576,7 +576,7 @@ Here are some apiserver flags you may need to set:
- `--tls-cert-file=/srv/kubernetes/server.cert`
- `--tls-private-key-file=/srv/kubernetes/server.key`
- `--admission-control=$RECOMMENDED_LIST`
- See [admission controllers](../admin/admission-controllers.html) for recommended arguments.
- See [admission controllers](/{{ page.version }}/docs/admin/admission-controllers) for recommended arguments.
- `--allow-privileged=true`, only if you trust your cluster user to run pods as root.
If you are following the firewall-only security approach, then use these arguments:
@@ -743,7 +743,7 @@ Flags to consider using with controller manager:
- `--allocate-node-cidrs=`
- *TODO*: explain when you want controller to do this and when you want to do it another way.
- `--cloud-provider=` and `--cloud-config` as described in apiserver section.
- `--service-account-private-key-file=/srv/kubernetes/server.key`, used by the [service account](../user-guide/service-accounts.html) feature.
- `--service-account-private-key-file=/srv/kubernetes/server.key`, used by the [service account](/{{ page.version }}/docs/user-guide/service-accounts) feature.
- `--master=127.0.0.1:8080`
#### Starting and Verifying Apiserver, Scheduler, and Controller Manager
@@ -801,12 +801,12 @@ Otherwise, you will need to manually create node objects.
### Inspect pods and services
Try to run through the "Inspect your cluster" section in one of the other Getting Started Guides, such as [GCE](gce.html#inspect-your-cluster).
Try to run through the "Inspect your cluster" section in one of the other Getting Started Guides, such as [GCE](/{{ page.version }}/docs/getting-started-guides/gce.html#inspect-your-cluster).
You should see some services. You should also see "mirror pods" for the apiserver, scheduler and controller-manager, plus any add-ons you started.
### Try Examples
At this point you should be able to run through one of the basic examples, such as the [nginx example](../../examples/simple-nginx.html).
At this point you should be able to run through one of the basic examples, such as the [nginx example](/{{ page.version }}/examples/simple-nginx).
### Running the Conformance Test
@@ -819,5 +819,5 @@ pinging or SSH-ing from one node to another.
### Getting Help
If you run into trouble, please see the section on [troubleshooting](gce.html#troubleshooting), post to the
[google-containers group](https://groups.google.com/forum/#!forum/google-containers), or come ask questions on [Slack](../troubleshooting.html#slack).
If you run into trouble, please see the section on [troubleshooting](/{{ page.version }}/docs/getting-started-guides/gce#troubleshooting), post to the
[google-containers group](https://groups.google.com/forum/#!forum/google-containers), or come ask questions on [Slack](/{{ page.version }}/docs/troubleshooting#slack).
+5 -42
View File
@@ -1,35 +1,11 @@
---
layout: docwithnav
title: "User Guide to Accessing the Cluster"
title: "Accessing Clusters"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
## Table of Contents
<!-- END MUNGE: UNVERSIONED_WARNING -->
# User Guide to Accessing the Cluster
**Table of Contents**
<!-- BEGIN MUNGE: GENERATED_TOC -->
- [User Guide to Accessing the Cluster](#user-guide-to-accessing-the-cluster)
- [Accessing the cluster API](#accessing-the-cluster-api)
- [Accessing for the first time with kubectl](#accessing-for-the-first-time-with-kubectl)
- [Directly accessing the REST API](#directly-accessing-the-rest-api)
- [Using kubectl proxy](#using-kubectl-proxy)
- [Without kubectl proxy](#without-kubectl-proxy)
- [Programmatic access to the API](#programmatic-access-to-the-api)
- [Accessing the API from a Pod](#accessing-the-api-from-a-pod)
- [Accessing services running on the cluster](#accessing-services-running-on-the-cluster)
- [Ways to connect](#ways-to-connect)
- [Discovering builtin services](#discovering-builtin-services)
- [Manually constructing apiserver proxy URLs](#manually-constructing-apiserver-proxy-urls)
- [Examples](#examples)
- [Using web browsers to access services running on the cluster](#using-web-browsers-to-access-services-running-on-the-cluster)
- [Requesting redirects](#requesting-redirects)
- [So Many Proxies](#so-many-proxies)
<!-- END MUNGE: GENERATED_TOC -->
{% include pagetoc.html %}
## Accessing the cluster API
@@ -303,17 +279,4 @@ There are several different proxies you may encounter when using Kubernetes:
Kubernetes users will typically not need to worry about anything other than the first two types. The cluster admin
will typically ensure that the latter types are setup correctly.
<!-- BEGIN MUNGE: IS_VERSIONED -->
<!-- TAG IS_VERSIONED -->
<!-- END MUNGE: IS_VERSIONED -->
<!-- BEGIN MUNGE: GENERATED_ANALYTICS -->
[![Analytics](https://kubernetes-site.appspot.com/UA-36037335-10/GitHub/docs/user-guide/accessing-the-cluster.md?pixel)]()
<!-- END MUNGE: GENERATED_ANALYTICS -->
will typically ensure that the latter types are setup correctly.
+2 -9
View File
@@ -1,14 +1,7 @@
---
layout: docwithnav
title: "kubeconfig files"
title: "Authenticating Across Clusters with kubeconfig"
section: guides
---
<!-- BEGIN MUNGE: UNVERSIONED_WARNING -->
<!-- END MUNGE: UNVERSIONED_WARNING -->
# kubeconfig files
Authentication in kubernetes can differ for different individuals.
- A running kubelet might have one way of authenticating (i.e. certificates).