committed by
Kubernetes Prow Robot
parent
0268ed0c18
commit
b05129acc3
@@ -21,41 +21,41 @@ Author: Daniel Imberman (Bloomberg LP)
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
作者: Daniel Imberman (Bloomberg LP)
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
## Introduction
|
||||
|
||||
|
||||
|
||||
|
||||
As part of Bloomberg's continued commitment to developing the Kubernetes ecosystem, we are excited to announce the Kubernetes Airflow Operator; a mechanism for Apache Airflow, a popular workflow orchestration framework to natively launch arbitrary Kubernetes Pods using the Kubernetes API.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
## 介绍
|
||||
|
||||
|
||||
|
||||
|
||||
作为Bloomberg [继续致力于开发Kubernetes生态系统]的一部分(https://www.techatbloomberg.com/blog/bloomberg-awarded-first-cncf-end-user-award-contributions-kubernetes/),我们很高兴能够宣布Kubernetes Airflow Operator的发布; [Apache Airflow](https://airflow.apache.org/)的机制,一种流行的工作流程编排框架,使用Kubernetes API可以在本机启动任意的Kubernetes Pod。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
## What Is Airflow?
|
||||
|
||||
|
||||
|
||||
|
||||
Apache Airflow is one realization of the DevOps philosophy of "Configuration As Code." Airflow allows users to launch multi-step pipelines using a simple Python object DAG (Directed Acyclic Graph). You can define dependencies, programmatically construct complex workflows, and monitor scheduled jobs in an easy to read UI.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -63,109 +63,109 @@ Apache Airflow is one realization of the DevOps philosophy of "Configuration As
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
## 什么是Airflow?
|
||||
|
||||
|
||||
|
||||
|
||||
Apache Airflow是DevOps“Configuration As Code”理念的一种实现。 Airflow允许用户使用简单的Python对象DAG(有向无环图)启动多步骤流水线。 您可以在易于阅读的UI中定义依赖关系,以编程方式构建复杂的工作流,并监视调度的作业。
|
||||
|
||||
|
||||
|
||||
|
||||
<img src =“/ images / blog / 2018-05-25-Airflow-Kubernetes-Operator / 2018-05-25-airflow_dags.png”width =“85%”alt =“Airflow DAGs”/>
|
||||
|
||||
<img src =“/ images / blog / 2018-05-25-Airflow-Kubernetes-Operator / 2018-05-25-airflow.png”width =“85%”alt =“Airflow UI”/>
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
## Why Airflow on Kubernetes?
|
||||
|
||||
|
||||
|
||||
|
||||
Since its inception, Airflow's greatest strength has been its flexibility. Airflow offers a wide range of integrations for services ranging from Spark and HBase, to services on various cloud providers. Airflow also offers easy extensibility through its plug-in framework. However, one limitation of the project is that Airflow users are confined to the frameworks and clients that exist on the Airflow worker at the moment of execution. A single organization can have varied Airflow workflows ranging from data science pipelines to application deployments. This difference in use-case creates issues in dependency management as both teams might use vastly different libraries for their workflows.
|
||||
|
||||
|
||||
|
||||
|
||||
To address this issue, we've utilized Kubernetes to allow users to launch arbitrary Kubernetes pods and configurations. Airflow users can now have full power over their run-time environments, resources, and secrets, basically turning Airflow into an "any job you want" workflow orchestrator.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
## 为什么在Kubernetes上使用Airflow?
|
||||
|
||||
|
||||
|
||||
|
||||
自成立以来,Airflow的最大优势在于其灵活性。 Airflow提供广泛的服务集成,包括Spark和HBase,以及各种云提供商的服务。 Airflow还通过其插件框架提供轻松的可扩展性。但是,该项目的一个限制是Airflow用户仅限于执行时Airflow站点上存在的框架和客户端。单个组织可以拥有各种Airflow工作流程,范围从数据科学流到应用程序部署。用例中的这种差异会在依赖关系管理中产生问题,因为两个团队可能会在其工作流程使用截然不同的库。
|
||||
|
||||
|
||||
|
||||
|
||||
为了解决这个问题,我们使Kubernetes允许用户启动任意Kubernetes pod和配置。 Airflow用户现在可以在其运行时环境,资源和机密上拥有全部权限,基本上将Airflow转变为“您想要的任何工作”工作流程协调器。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
## The Kubernetes Operator
|
||||
|
||||
|
||||
|
||||
|
||||
Before we move any further, we should clarify that an Operator in Airflow is a task definition. When a user creates a DAG, they would use an operator like the "SparkSubmitOperator" or the "PythonOperator" to submit/monitor a Spark job or a Python function respectively. Airflow comes with built-in operators for frameworks like Apache Spark, BigQuery, Hive, and EMR. It also offers a Plugins entrypoint that allows DevOps engineers to develop their own connectors.
|
||||
|
||||
|
||||
|
||||
|
||||
Airflow users are always looking for ways to make deployments and ETL pipelines simpler to manage. Any opportunity to decouple pipeline steps, while increasing monitoring, can reduce future outages and fire-fights. The following is a list of benefits provided by the Airflow Kubernetes Operator:
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
## Kubernetes运营商
|
||||
|
||||
|
||||
|
||||
|
||||
在进一步讨论之前,我们应该澄清Airflow中的[Operator](https://airflow.apache.org/concepts.html#operators)是一个任务定义。 当用户创建DAG时,他们将使用像“SparkSubmitOperator”或“PythonOperator”这样的operator分别提交/监视Spark作业或Python函数。 Airflow附带了Apache Spark,BigQuery,Hive和EMR等框架的内置运算符。 它还提供了一个插件入口点,允许DevOps工程师开发自己的连接器。
|
||||
|
||||
|
||||
|
||||
|
||||
Airflow用户一直在寻找更易于管理部署和ETL流的方法。 在增加监控的同时,任何解耦流程的机会都可以减少未来的停机等问题。 以下是Airflow Kubernetes Operator提供的好处:
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
* Increased flexibility for deployments:
|
||||
* Increased flexibility for deployments:
|
||||
|
||||
Airflow's plugin API has always offered a significant boon to engineers wishing to test new functionalities within their DAGs. On the downside, whenever a developer wanted to create a new operator, they had to develop an entirely new plugin. Now, any task that can be run within a Docker container is accessible through the exact same operator, with no extra Airflow code to maintain.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
* 提高部署灵活性:
|
||||
|
||||
Airflow的插件API一直为希望在其DAG中测试新功能的工程师提供了重要的福利。 不利的一面是,每当开发人员想要创建一个新的operator时,他们就必须开发一个全新的插件。 现在,任何可以在Docker容器中运行的任务都可以通过完全相同的运算符访问,而无需维护额外的Airflow代码。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
* Flexibility of configurations and dependencies:
|
||||
|
||||
For operators that are run within static Airflow workers, dependency management can become quite difficult. If a developer wants to run one task that requires SciPy and another that requires NumPy, the developer would have to either maintain both dependencies within all Airflow workers or offload the task to an external machine (which can cause bugs if that external machine changes in an untracked manner). Custom Docker images allow users to ensure that the tasks environment, configuration, and dependencies are completely idempotent.
|
||||
For operators that are run within static Airflow workers, dependency management can become quite difficult. If a developer wants to run one task that requires SciPy and another that requires NumPy, the developer would have to either maintain both dependencies within all Airflow workers or offload the task to an external machine (which can cause bugs if that external machine changes in an untracked manner). Custom Docker images allow users to ensure that the tasks environment, configuration, and dependencies are completely idempotent.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
* 配置和依赖的灵活性:
|
||||
|
||||
对于在静态Airflow工作程序中运行的operator,依赖关系管理可能变得非常困难。 如果开发人员想要运行一个需要[SciPy](https://www.scipy.org) 的任务和另一个需要[NumPy](http://www.numpy.org) 的任务,开发人员必须维护所有Airflow节点中的依赖关系或将任务卸载到其他计算机(如果外部计算机以未跟踪的方式更改,则可能导致错误)。 自定义Docker镜像允许用户确保任务环境,配置和依赖关系完全是幂等的。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
@@ -175,71 +175,71 @@ Handling sensitive data is a core responsibility of any DevOps engineer. At ever
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
* 使用kubernetes Secret以增加安全性:
|
||||
|
||||
处理敏感数据是任何开发工程师的核心职责。 Airflow用户总有机会在严格条款的基础上隔离任何API密钥,数据库密码和登录凭据。 使用Kubernetes运算符,用户可以利用Kubernetes Vault技术存储所有敏感数据。 这意味着Airflow工作人员将永远无法访问此信息,并且可以容易地请求仅使用他们需要的密码信息构建pod。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
# Architecture
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
The Kubernetes Operator uses the Kubernetes Python Client to generate a request that is processed by the APIServer (1). Kubernetes will then launch your pod with whatever specs you've defined (2). Images will be loaded with all the necessary environment variables, secrets and dependencies, enacting a single command. Once the job is launched, the operator only needs to monitor the health of track logs (3). Users will have the choice of gathering logs locally to the scheduler or to any distributed logging service currently in their Kubernetes cluster.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
#架构
|
||||
|
||||
|
||||
|
||||
|
||||
<img src =“/ images / blog / 2018-05-25-Airflow-Kubernetes-Operator / 2018-05-25-airflow-architecture.png”width =“85%”alt =“Airflow Architecture”/>
|
||||
|
||||
|
||||
|
||||
|
||||
Kubernetes Operator使用[Kubernetes Python客户端](https://github.com/kubernetes-client/Python)生成由APIServer处理的请求(1)。 然后,Kubernetes将使用您定义的需求启动您的pod(2)。映像文件中将加载环境变量,Secret和依赖项,执行单个命令。 一旦启动作业,operator只需要监视跟踪日志的状况(3)。 用户可以选择将日志本地收集到调度程序或当前位于其Kubernetes集群中的任何分布式日志记录服务。
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
# Using the Kubernetes Operator
|
||||
|
||||
|
||||
|
||||
|
||||
## A Basic Example
|
||||
|
||||
|
||||
|
||||
|
||||
The following DAG is probably the simplest example we could write to show how the Kubernetes Operator works. This DAG creates two pods on Kubernetes: a Linux distro with Python and a base Ubuntu distro without it. The Python pod will run the Python request correctly, while the one without Python will report a failure to the user. If the Operator is working correctly, the passing-task pod should complete, while the failing-task pod returns a failure to the Airflow webserver.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
#使用Kubernetes Operator
|
||||
|
||||
|
||||
|
||||
|
||||
##一个基本的例子
|
||||
|
||||
|
||||
|
||||
|
||||
以下DAG可能是我们可以编写的最简单的示例,以显示Kubernetes Operator的工作原理。 这个DAG在Kubernetes上创建了两个pod:一个带有Python的Linux发行版和一个没有它的基本Ubuntu发行版。 Python pod将正确运行Python请求,而没有Python的那个将向用户报告失败。 如果Operator正常工作,则应该完成“passing-task”pod,而“falling-task”pod则向Airflow网络服务器返回失败。
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
```Python
|
||||
|
||||
@@ -272,7 +272,7 @@ default_args = {
|
||||
|
||||
}
|
||||
|
||||
|
||||
|
||||
|
||||
dag = DAG(
|
||||
|
||||
@@ -329,29 +329,29 @@ failing.set_upstream(start)
|
||||
|
||||
## But how does this relate to my workflow?
|
||||
|
||||
|
||||
|
||||
|
||||
While this example only uses basic images, the magic of Docker is that this same DAG will work for any image/command pairing you want. The following is a recommended CI/CD pipeline to run production-ready code on an Airflow DAG.
|
||||
|
||||
|
||||
|
||||
|
||||
### 1: PR in github
|
||||
|
||||
Use Travis or Jenkins to run unit and integration tests, bribe your favorite team-mate into PR'ing your code, and merge to the master branch to trigger an automated CI build.
|
||||
|
||||
|
||||
|
||||
|
||||
### 2: CI/CD via Jenkins -> Docker Image
|
||||
|
||||
|
||||
|
||||
|
||||
Generate your Docker images and bump release version within your Jenkins build.
|
||||
|
||||
|
||||
|
||||
|
||||
### 3: Airflow launches task
|
||||
|
||||
|
||||
|
||||
|
||||
Finally, update your DAGs to reflect the new release version and you should be ready to go!
|
||||
|
||||
@@ -359,33 +359,33 @@ Finally, update your DAGs to reflect the new release version and you should be r
|
||||
|
||||
##但这与我的工作流程有什么关系?
|
||||
|
||||
|
||||
|
||||
|
||||
虽然这个例子只使用基本映像,但Docker的神奇之处在于,这个相同的DAG可以用于您想要的任何图像/命令配对。 以下是推荐的CI / CD管道,用于在Airflow DAG上运行生产就绪代码。
|
||||
|
||||
|
||||
|
||||
|
||||
### 1:github中的PR
|
||||
|
||||
使用Travis或Jenkins运行单元和集成测试,请您的朋友PR您的代码,并合并到主分支以触发自动CI构建。
|
||||
|
||||
|
||||
|
||||
|
||||
### 2:CI / CD构建Jenkins - > Docker Image
|
||||
|
||||
|
||||
|
||||
|
||||
[在Jenkins构建中生成Docker镜像和缓冲版本](https://getintodevops.com/blog/building-your-first-Docker-image-with-jenkins-2-guide-for-developers)。
|
||||
|
||||
|
||||
|
||||
|
||||
### 3:Airflow启动任务
|
||||
|
||||
|
||||
|
||||
|
||||
最后,更新您的DAG以反映新版本,您应该准备好了!
|
||||
|
||||
|
||||
|
||||
|
||||
```Python
|
||||
|
||||
@@ -415,61 +415,61 @@ production_task = KubernetesPodOperator(namespace='default',
|
||||
|
||||
# Launching a test deployment
|
||||
|
||||
|
||||
|
||||
|
||||
Since the Kubernetes Operator is not yet released, we haven't released an official helm chart or operator (however both are currently in progress). However, we are including instructions for a basic deployment below and are actively looking for foolhardy beta testers to try this new feature. To try this system out please follow these steps:
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 1: Set your kubeconfig to point to a kubernetes cluster
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 2: Clone the Airflow Repo:
|
||||
|
||||
|
||||
|
||||
|
||||
Run git clone https://github.com/apache/incubator-airflow.git to clone the official Airflow repo.
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 3: Run
|
||||
|
||||
|
||||
|
||||
|
||||
To run this basic deployment, we are co-opting the integration testing script that we currently use for the Kubernetes Executor (which will be explained in the next article of this series). To launch this deployment, run these three commands:
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
#启动测试部署
|
||||
|
||||
|
||||
|
||||
|
||||
由于Kubernetes运营商尚未发布,我们尚未发布官方[helm](https://helm.sh/) 图表或operator(但两者目前都在进行中)。 但是,我们在下面列出了基本部署的说明,并且正在积极寻找测试人员来尝试这一新功能。 要试用此系统,请按以下步骤操作:
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤1:将kubeconfig设置为指向kubernetes集群
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤2:clone Airflow 仓库:
|
||||
|
||||
|
||||
|
||||
|
||||
运行git clone https:// github.com / apache / incubator-airflow.git来clone官方Airflow仓库。
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤3:运行
|
||||
|
||||
|
||||
|
||||
|
||||
为了运行这个基本Deployment,我们正在选择我们目前用于Kubernetes Executor的集成测试脚本(将在本系列的下一篇文章中对此进行解释)。 要启动此部署,请运行以下三个命令:
|
||||
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
@@ -485,77 +485,77 @@ sed -ie "s/KubernetesExecutor/LocalExecutor/g" scripts/ci/kubernetes/kube/config
|
||||
|
||||
Before we move on, let's discuss what these commands are doing:
|
||||
|
||||
|
||||
|
||||
|
||||
### sed -ie "s/KubernetesExecutor/LocalExecutor/g" scripts/ci/kubernetes/kube/configmaps.yaml
|
||||
|
||||
|
||||
|
||||
|
||||
The Kubernetes Executor is another Airflow feature that allows for dynamic allocation of tasks as idempotent pods. The reason we are switching this to the LocalExecutor is simply to introduce one feature at a time. You are more then welcome to skip this step if you would like to try the Kubernetes Executor, however we will go into more detail in a future article.
|
||||
|
||||
|
||||
|
||||
|
||||
### ./scripts/ci/kubernetes/Docker/build.sh
|
||||
|
||||
|
||||
|
||||
|
||||
This script will tar the Airflow master source code build a Docker container based on the Airflow distribution
|
||||
|
||||
|
||||
|
||||
|
||||
### ./scripts/ci/kubernetes/kube/deploy.sh
|
||||
|
||||
|
||||
|
||||
|
||||
Finally, we create a full Airflow deployment on your cluster. This includes Airflow configs, a postgres backend, the webserver + scheduler, and all necessary services between. One thing to note is that the role binding supplied is a cluster-admin, so if you do not have that level of permission on the cluster, you can modify this at scripts/ci/kubernetes/kube/airflow.yaml
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 4: Log into your webserver
|
||||
|
||||
|
||||
|
||||
|
||||
Now that your Airflow instance is running let's take a look at the UI! The UI lives in port 8080 of the Airflow pod, so simply run
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
在我们继续之前,让我们讨论这些命令正在做什么:
|
||||
|
||||
|
||||
|
||||
|
||||
### sed -ie“s / KubernetesExecutor / LocalExecutor / g”scripts / ci / kubernetes / kube / configmaps.yaml
|
||||
|
||||
|
||||
|
||||
|
||||
Kubernetes Executor是另一种Airflow功能,允许动态分配任务已解决幂等pod的问题。我们将其切换到LocalExecutor的原因只是一次引入一个功能。如果您想尝试Kubernetes Executor,欢迎您跳过此步骤,但我们将在以后的文章中详细介绍。
|
||||
|
||||
|
||||
|
||||
|
||||
### ./scripts/ci/kubernetes/Docker/build.sh
|
||||
|
||||
|
||||
|
||||
|
||||
此脚本将对Airflow主分支代码进行打包,以根据Airflow的发行文件构建Docker容器
|
||||
|
||||
|
||||
|
||||
|
||||
### ./scripts/ci/kubernetes/kube/deploy.sh
|
||||
|
||||
|
||||
|
||||
|
||||
最后,我们在您的群集上创建完整的Airflow部署。这包括Airflow配置,postgres后端,webserver +调度程序以及之间的所有必要服务。需要注意的一点是,提供的角色绑定是集群管理员,因此如果您没有该集群的权限级别,可以在scripts / ci / kubernetes / kube / airflow.yaml中进行修改。
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤4:登录您的网络服务器
|
||||
|
||||
|
||||
|
||||
|
||||
现在您的Airflow实例正在运行,让我们来看看UI!用户界面位于Airflow pod的8080端口,因此只需运行即可
|
||||
|
||||
|
||||
|
||||
|
||||
```
|
||||
|
||||
@@ -569,59 +569,59 @@ kubectl port-forward $WEB 8080:8080
|
||||
|
||||
Now the Airflow UI will exist on http://localhost:8080. To log in simply enter airflow/airflow and you should have full access to the Airflow web UI.
|
||||
|
||||
|
||||
|
||||
|
||||
## Step 5: Upload a test document
|
||||
|
||||
|
||||
|
||||
|
||||
To modify/add your own DAGs, you can use kubectl cp to upload local files into the DAG folder of the Airflow scheduler. Airflow will then read the new DAG and automatically upload it to its system. The following command will upload any local file into the correct directory:
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
现在,Airflow UI将存在于http://localhost:8080上。 要登录,只需输入airflow /airflow,您就可以完全访问Airflow Web UI。
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤5:上传测试文档
|
||||
|
||||
|
||||
|
||||
|
||||
要修改/添加自己的DAG,可以使用kubectl cp将本地文件上传到Airflow调度程序的DAG文件夹中。 然后,Airflow将读取新的DAG并自动将其上传到其系统。 以下命令将任何本地文件上载到正确的目录中:
|
||||
|
||||
|
||||
|
||||
|
||||
kubectl cp <local file> <namespace>/<pod>:/root/airflow/dags -c scheduler
|
||||
|
||||
|
||||
|
||||
|
||||
<!--
|
||||
|
||||
## Step 6: Enjoy!
|
||||
|
||||
|
||||
|
||||
|
||||
# So when will I be able to use this?
|
||||
|
||||
|
||||
|
||||
|
||||
While this feature is still in the early stages, we hope to see it released for wide release in the next few months.
|
||||
|
||||
|
||||
|
||||
|
||||
# Get Involved
|
||||
|
||||
|
||||
|
||||
|
||||
This feature is just the beginning of multiple major efforts to improves Apache Airflow integration into Kubernetes. The Kubernetes Operator has been merged into the 1.10 release branch of Airflow (the executor in experimental mode), along with a fully k8s native scheduler called the Kubernetes Executor (article to come). These features are still in a stage where early adopters/contributers can have a huge influence on the future of these features.
|
||||
|
||||
|
||||
|
||||
|
||||
For those interested in joining these efforts, I'd recommend checkint out these steps:
|
||||
|
||||
|
||||
|
||||
|
||||
* Join the airflow-dev mailing list at dev@airflow.apache.org.
|
||||
|
||||
@@ -631,37 +631,37 @@ For those interested in joining these efforts, I'd recommend checkint out these
|
||||
|
||||
* Reach us on slack at #sig-big-data on kubernetes.slack.com
|
||||
|
||||
|
||||
|
||||
|
||||
Special thanks to the Apache Airflow and Kubernetes communities, particularly Grant Nicholas, Ben Goldberg, Anirudh Ramanathan, Fokko Dreisprong, and Bolke de Bruin, for your awesome help on these features as well as our future efforts.
|
||||
|
||||
-->
|
||||
|
||||
|
||||
|
||||
|
||||
##步骤6:使用它!
|
||||
|
||||
|
||||
|
||||
|
||||
#那么我什么时候可以使用它?
|
||||
|
||||
|
||||
|
||||
|
||||
虽然此功能仍处于早期阶段,但我们希望在未来几个月内发布该功能以进行广泛发布。
|
||||
|
||||
|
||||
|
||||
|
||||
#参与其中
|
||||
|
||||
|
||||
|
||||
|
||||
此功能只是将Apache Airflow集成到Kubernetes中的多项主要工作的开始。 Kubernetes Operator已合并到[Airflow的1.10发布分支](https://github.com/apache/incubator-airflow/tree/v1-10-test)(实验模式中的执行模块),以及完整的k8s本地调度程序称为Kubernetes Executor(即将发布文章)。这些功能仍处于早期采用者/贡献者可能对这些功能的未来产生巨大影响的阶段。
|
||||
|
||||
|
||||
|
||||
|
||||
对于有兴趣加入这些工作的人,我建议按照以下步骤:
|
||||
|
||||
|
||||
|
||||
|
||||
*加入airflow-dev邮件列表dev@airflow.apache.org。
|
||||
|
||||
@@ -671,6 +671,6 @@ Special thanks to the Apache Airflow and Kubernetes communities, particularly Gr
|
||||
|
||||
*在kubernetes.slack.com上的#sig-big-data找到我们。
|
||||
|
||||
|
||||
|
||||
|
||||
特别感谢Apache Airflow和Kubernetes社区,特别是Grant Nicholas,Ben Goldberg,Anirudh Ramanathan,Fokko Dreisprong和Bolke de Bruin,感谢您对这些功能的巨大帮助以及我们未来的努力。
|
||||
|
||||
@@ -176,21 +176,21 @@ Here comes an example:
|
||||
Port: http 3080/TCP
|
||||
Endpoints: 10.244.0.235:8080,10.244.1.237:8080
|
||||
Session Affinity: None
|
||||
|
||||
|
||||
# ip addr
|
||||
...
|
||||
73: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
|
||||
link/ether 1a:ce:f5:5f:c1:4d brd ff:ff:ff:ff:ff:ff
|
||||
inet 10.102.128.4/32 scope global kube-ipvs0
|
||||
valid_lft forever preferred_lft forever
|
||||
|
||||
|
||||
# ipvsadm -ln
|
||||
IP Virtual Server version 1.2.1 (size=4096)
|
||||
Prot LocalAddress:Port Scheduler Flags
|
||||
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
||||
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
||||
TCP 10.102.128.4:3080 rr
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0 0
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0 0
|
||||
|
||||
-->
|
||||
|
||||
@@ -204,21 +204,21 @@ Here comes an example:
|
||||
Port: http 3080/TCP
|
||||
Endpoints: 10.244.0.235:8080,10.244.1.237:8080
|
||||
Session Affinity: None
|
||||
|
||||
|
||||
# ip addr
|
||||
...
|
||||
73: kube-ipvs0: <BROADCAST,NOARP> mtu 1500 qdisc noop state DOWN qlen 1000
|
||||
link/ether 1a:ce:f5:5f:c1:4d brd ff:ff:ff:ff:ff:ff
|
||||
inet 10.102.128.4/32 scope global kube-ipvs0
|
||||
valid_lft forever preferred_lft forever
|
||||
|
||||
|
||||
# ipvsadm -ln
|
||||
IP Virtual Server version 1.2.1 (size=4096)
|
||||
Prot LocalAddress:Port Scheduler Flags
|
||||
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
||||
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
|
||||
TCP 10.102.128.4:3080 rr
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0 0
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0 0
|
||||
|
||||
<!--
|
||||
|
||||
@@ -241,8 +241,8 @@ There are three proxy modes in IPVS: NAT (masq), IPIP and DR. Only NAT mode supp
|
||||
IPVS 中有三种代理模式:NAT(masq),IPIP 和 DR。 只有 NAT 模式支持端口映射。 Kube-proxy 利用 NAT 模式进行端口映射。 以下示例显示 IPVS 服务端口3080到Pod端口8080的映射。
|
||||
|
||||
TCP 10.102.128.4:3080 rr
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0
|
||||
-> 10.244.0.235:8080 Masq 1 0 0
|
||||
-> 10.244.1.237:8080 Masq 1 0
|
||||
|
||||
<!--
|
||||
|
||||
@@ -262,7 +262,7 @@ IPVS 支持客户端 IP 会话关联(持久连接)。 当服务指定会话
|
||||
IP: 10.102.128.4
|
||||
Port: http 3080/TCP
|
||||
Session Affinity: ClientIP
|
||||
|
||||
|
||||
# ipvsadm -ln
|
||||
IP Virtual Server version 1.2.1 (size=4096)
|
||||
Prot LocalAddress:Port Scheduler Flags
|
||||
@@ -301,32 +301,32 @@ IPVS proxier 在上述场景中利用 iptables。 具体来说,ipvs proxier
|
||||
|
||||
<!--
|
||||
|
||||
set name members usage
|
||||
set name members usage
|
||||
KUBE-CLUSTER-IP All Service IP + port masquerade for cases that masquerade-all=true or clusterCIDR specified
|
||||
KUBE-LOOP-BACK All Service IP + port + IP masquerade for resolving hairpin issue
|
||||
KUBE-EXTERNAL-IP Service External IP + port masquerade for packets to external IPs
|
||||
KUBE-LOOP-BACK All Service IP + port + IP masquerade for resolving hairpin issue
|
||||
KUBE-EXTERNAL-IP Service External IP + port masquerade for packets to external IPs
|
||||
KUBE-LOAD-BALANCER Load Balancer ingress IP + port masquerade for packets to Load Balancer type service
|
||||
KUBE-LOAD-BALANCER-LOCAL Load Balancer ingress IP + port with externalTrafficPolicy=local accept packets to Load Balancer with externalTrafficPolicy=local
|
||||
KUBE-LOAD-BALANCER-FW Load Balancer ingress IP + port with loadBalancerSourceRanges Drop packets for Load Balancer type Service with loadBalancerSourceRanges specified
|
||||
KUBE-LOAD-BALANCER-SOURCE-CIDR Load Balancer ingress IP + port + source CIDR accept packets for Load Balancer type Service with loadBalancerSourceRanges specified
|
||||
KUBE-NODE-PORT-TCP NodePort type Service TCP port masquerade for packets to NodePort(TCP)
|
||||
KUBE-NODE-PORT-TCP NodePort type Service TCP port masquerade for packets to NodePort(TCP)
|
||||
KUBE-NODE-PORT-LOCAL-TCP NodePort type Service TCP port with externalTrafficPolicy=local accept packets to NodePort Service with externalTrafficPolicy=local
|
||||
KUBE-NODE-PORT-UDP NodePort type Service UDP port masquerade for packets to NodePort(UDP)
|
||||
KUBE-NODE-PORT-UDP NodePort type Service UDP port masquerade for packets to NodePort(UDP)
|
||||
KUBE-NODE-PORT-LOCAL-UDP NodePort type service UDP port with externalTrafficPolicy=local accept packets to NodePort Service with externalTrafficPolicy=local
|
||||
|
||||
-->
|
||||
|
||||
设置名称 成员 用法
|
||||
设置名称 成员 用法
|
||||
KUBE-CLUSTER-IP 所有服务 IP + 端口 masquerade-all=true 或 clusterCIDR 指定的情况下进行伪装
|
||||
KUBE-LOOP-BACK 所有服务 IP +端口+ IP 解决数据包欺骗问题
|
||||
KUBE-EXTERNAL-IP 服务外部 IP +端口 将数据包伪装成外部 IP
|
||||
KUBE-LOAD-BALANCER 负载均衡器入口 IP +端口 将数据包伪装成 Load Balancer 类型的服务
|
||||
KUBE-LOOP-BACK 所有服务 IP +端口+ IP 解决数据包欺骗问题
|
||||
KUBE-EXTERNAL-IP 服务外部 IP +端口 将数据包伪装成外部 IP
|
||||
KUBE-LOAD-BALANCER 负载均衡器入口 IP +端口 将数据包伪装成 Load Balancer 类型的服务
|
||||
KUBE-LOAD-BALANCER-LOCAL 负载均衡器入口 IP +端口 以及 externalTrafficPolicy=local 接受数据包到 Load Balancer externalTrafficPolicy=local
|
||||
KUBE-LOAD-BALANCER-FW 负载均衡器入口 IP +端口 以及 loadBalancerSourceRanges 使用指定的 loadBalancerSourceRanges 丢弃 Load Balancer类型Service的数据包
|
||||
KUBE-LOAD-BALANCER-SOURCE-CIDR 负载均衡器入口 IP +端口 + 源 CIDR 接受 Load Balancer 类型 Service 的数据包,并指定loadBalancerSourceRanges
|
||||
KUBE-NODE-PORT-TCP NodePort 类型服务 TCP 将数据包伪装成 NodePort(TCP)
|
||||
KUBE-NODE-PORT-TCP NodePort 类型服务 TCP 将数据包伪装成 NodePort(TCP)
|
||||
KUBE-NODE-PORT-LOCAL-TCP NodePort 类型服务 TCP 端口,带有 externalTrafficPolicy=local 接受数据包到 NodePort 服务 使用 externalTrafficPolicy=local
|
||||
KUBE-NODE-PORT-UDP NodePort 类型服务 UDP 端口 将数据包伪装成 NodePort(UDP)
|
||||
KUBE-NODE-PORT-UDP NodePort 类型服务 UDP 端口 将数据包伪装成 NodePort(UDP)
|
||||
KUBE-NODE-PORT-LOCAL-UDP NodePort 类型服务 UDP 端口 使用 externalTrafficPolicy=local 接受数据包到NodePort服务 使用 externalTrafficPolicy=local
|
||||
|
||||
<!--
|
||||
|
||||
@@ -4,7 +4,7 @@ title: '新贡献者工作坊上海站'
|
||||
date: 2018-12-05
|
||||
---
|
||||
|
||||
<!--
|
||||
<!--
|
||||
---
|
||||
layout: blog
|
||||
title: 'New Contributor Workshop Shanghai'
|
||||
@@ -12,103 +12,103 @@ date: 2018-12-05
|
||||
---
|
||||
-->
|
||||
|
||||
<!--
|
||||
<!--
|
||||
**Authors**: Josh Berkus (Red Hat), Yang Li (The Plant), Puja Abbassi (Giant Swarm), XiangPeng Zhao (ZTE)
|
||||
-->
|
||||
|
||||
**作者**: Josh Berkus (红帽), Yang Li (The Plant), Puja Abbassi (Giant Swarm), XiangPeng Zhao (中兴通讯)
|
||||
|
||||
<!--
|
||||
<!--
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/attendees.png" caption="KubeCon Shanghai New Contributor Summit attendees. Photo by Jerry Zhang" >}}
|
||||
-->
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/attendees.png" caption="KubeCon 上海站新贡献者峰会与会者,摄影:Jerry Zhang" >}}
|
||||
|
||||
<!--
|
||||
<!--
|
||||
We recently completed our first New Contributor Summit in China, at the first KubeCon in China. It was very exciting to see all of the Chinese and Asian developers (plus a few folks from around the world) interested in becoming contributors. Over the course of a long day, they learned how, why, and where to contribute to Kubernetes, created pull requests, attended a panel of current contributors, and got their CLAs signed.
|
||||
-->
|
||||
|
||||
最近,在中国的首次 KubeCon 上,我们完成了在中国的首次新贡献者峰会。看到所有中国和亚洲的开发者(以及来自世界各地的一些人)有兴趣成为贡献者,这令人非常兴奋。在长达一天的课程中,他们了解了如何、为什么以及在何处为 Kubernetes 作出贡献,创建了 PR,参加了贡献者圆桌讨论,并签署了他们的 CLA。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
This was our second New Contributor Workshop (NCW), building on the one created and led by SIG Contributor Experience members in Copenhagen. Because of the audience, it was held in both Chinese and English, taking advantage of the superb simultaneous interpretation services the CNCF sponsored. Likewise, the NCW team included both English and Chinese-speaking members of the community: Yang Li, XiangPeng Zhao, Puja Abbassi, Noah Abrahams, Tim Pepper, Zach Corleissen, Sen Lu, and Josh Berkus. In addition to presenting and helping students, the bilingual members of the team translated all of the slides into Chinese. Fifty-one students attended.
|
||||
-->
|
||||
|
||||
这是我们的第二届新贡献者工作坊(NCW),它由前一次贡献者体验 SIG 成员创建和领导的哥本哈根研讨会延伸而来。根据受众情况,本次活动采用了中英文两种语言,充分利用了 CNCF 赞助的一流的同声传译服务。同样,NCW 团队由社区成员组成,既有说英语的,也有说汉语的:Yang Li、XiangPeng Zhao、Puja Abbassi、Noah Abrahams、Tim Pepper、Zach Corleissen、Sen Lu 和 Josh Berkus。除了演讲和帮助学员外,团队的双语成员还将所有幻灯片翻译成了中文。共有五十一名学员参加。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/noahabrahams.png" caption="Noah Abrahams explains Kubernetes communications channels. Photo by Jerry Zhang" >}}
|
||||
-->
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/noahabrahams.png" caption="Noah Abrahams 讲解 Kubernetes 沟通渠道。摄影:Jerry Zhang" >}}
|
||||
|
||||
<!--
|
||||
<!--
|
||||
The NCW takes participants through the stages of contributing to Kubernetes, starting from deciding where to contribute, followed by an introduction to the SIG system and our repository structure. We also have "guest speakers" from Docs and Test Infrastructure who cover contributing in those areas. We finally wind up with some hands-on exercises in filing issues and creating and approving PRs.
|
||||
-->
|
||||
|
||||
NCW 让参与者完成了为 Kubernetes 作出贡献的各个阶段,从决定在哪里作出贡献开始,接着介绍了 SIG 系统和我们的代码仓库结构。我们还有来自文档和测试基础设施领域的「客座讲者」,他们负责讲解有关的贡献。最后,我们在创建 issue、提交并批准 PR 的实践练习后,结束了工作坊。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
Those hands-on exercises use a repository known as [the contributor playground](https://github.com/kubernetes-sigs/contributor-playground), created by SIG Contributor Experience as a place for new contributors to try out performing various actions on a Kubernetes repo. It has modified Prow and Tide automation, uses Owners files like in the real repositories. This lets students learn how the mechanics of contributing to our repositories work without disrupting normal development.
|
||||
-->
|
||||
|
||||
这些实践练习使用一个名为[贡献者游乐场](https://github.com/kubernetes-sigs/contributor-playground)的代码仓库,由贡献者体验 SIG 创建,让新贡献者尝试在一个 Kubernetes 仓库中执行各种操作。它修改了 Prow 和 Tide 自动化,使用与真实代码仓库类似的 Owners 文件。这可以让学员了解为我们的仓库做出贡献的有关机制,同时又不妨碍正常的开发流程。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/yangli.png" caption="Yang Li talks about getting your PRs reviewed. Photo by Josh Berkus" >}}
|
||||
-->
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/yangli.png" caption="Yang Li 讲到如何让你的 PR 通过评审。摄影:Josh Berkus" >}}
|
||||
|
||||
<!--
|
||||
<!--
|
||||
Both the "Great Firewall" and the language barrier prevent contributing Kubernetes from China from being straightforward. What's more, because open source business models are not mature in China, the time for employees work on open source projects is limited.
|
||||
-->
|
||||
|
||||
「防火长城」和语言障碍都使得在中国为 Kubernetes 作出贡献变得困难。而且,中国的开源商业模式并不成熟,员工在开源项目上工作的时间有限。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
Chinese engineers are eager to participate in the development of Kubernetes, but many of them don't know where to start since Kubernetes is such a large project. With this workshop, we hope to help those who want to contribute, whether they wish to fix some bugs they encountered, improve or localize documentation, or they need to work with Kubernetes at their work. We are glad to see more and more Chinese contributors joining the community in the past few years, and we hope to see more of them in the future.
|
||||
-->
|
||||
|
||||
中国工程师渴望参与 Kubernetes 的研发,但他们中的许多人不知道从何处开始,因为 Kubernetes 是一个如此庞大的项目。通过本次工作坊,我们希望帮助那些想要参与贡献的人,不论他们希望修复他们遇到的一些错误、改进或本地化文档,或者他们需要在工作中用到 Kubernetes。我们很高兴看到越来越多的中国贡献者在过去几年里加入社区,我们也希望将来可以看到更多。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
"I have been participating in the Kubernetes community for about three years," said XiangPeng Zhao. "In the community, I notice that more and more Chinese developers are showing their interest in contributing to Kubernetes. However, it's not easy to start contributing to such a project. I tried my best to help those who I met in the community, but I think there might still be some new contributors leaving the community due to not knowing where to get help when in trouble. Fortunately, the community initiated NCW at KubeCon Copenhagen and held a second one at KubeCon Shanghai. I was so excited to be invited by Josh Berkus to help organize this workshop. During the workshop, I met community friends in person, mentored attendees in the exercises, and so on. All of this was a memorable experience for me. I also learned a lot as a contributor who already has years of contributing experience. I wish I had attended such a workshop when I started contributing to Kubernetes years ago."
|
||||
-->
|
||||
|
||||
「我已经参与了 Kubernetes 社区大约三年」,XiangPeng Zhao 说,「在社区,我注意到越来越多的中国开发者表现出对 Kubernetes 贡献的兴趣。但是,开始为这样一个项目做贡献并不容易。我尽力帮助那些我在社区遇到的人,但是,我认为可能仍有一些新的贡献者离开社区,因为他们在遇到麻烦时不知道从哪里获得帮助。幸运的是,社区在 KubeCon 哥本哈根站发起了 NCW,并在 KubeCon 上海站举办了第二届。我很高兴受到 Josh Berkus 的邀请,帮助组织这个工作坊。在工作坊期间,我当面见到了社区里的朋友,在练习中指导了与会者,等等。所有这些对我来说都是难忘的经历。作为有着多年贡献者经验的我,也学习到了很多。我希望几年前我开始为 Kubernetes 做贡献时参加过这样的工作坊」。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/panel.png" caption="Panel of contributors. Photo by Jerry Zhang" >}}
|
||||
-->
|
||||
|
||||
{{< figure src="/images/blog/2018-12-05-new-contributor-shanghai/panel.png" caption="贡献者圆桌讨论。摄影:Jerry Zhang" >}}
|
||||
|
||||
<!--
|
||||
<!--
|
||||
The workshop ended with a panel of current contributors, featuring Lucas Käldström, Janet Kuo, Da Ma, Pengfei Ni, Zefeng Wang, and Chao Xu. The panel aimed to give both new and current contributors a look behind the scenes on the day-to-day of some of the most active contributors and maintainers, both from China and around the world. Panelists talked about where to begin your contributor's journey, but also how to interact with reviewers and maintainers. They further touched upon the main issues of contributing from China and gave attendees an outlook into exciting features they can look forward to in upcoming releases of Kubernetes.
|
||||
-->
|
||||
|
||||
工作坊以现有贡献者圆桌讨论结束,嘉宾包括 Lucas Käldström、Janet Kuo、Da Ma、Pengfei Ni、Zefeng Wang 和 Chao Xu。这场圆桌讨论旨在让新的和现有的贡献者了解一些最活跃的贡献者和维护者的幕后日常工作,不论他们来自中国还是世界各地。嘉宾们讨论了从哪里开始贡献者的旅程,以及如何与评审者和维护者进行互动。他们进一步探讨了在中国参与贡献的主要问题,并向与会者预告了在 Kubernetes 的未来版本中可以期待的令人兴奋的功能。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
After the workshop, XiangPeng Zhao chatted with some attendees on WeChat and Twitter about their experiences. They were very glad to have attended the NCW and had some suggestions on improving the workshop. One attendee, Mohammad, said, "I had a great time at the workshop and learned a lot about the entire process of k8s for a contributor." Another attendee, Jie Jia, said, "The workshop was wonderful. It systematically explained how to contribute to Kubernetes. The attendee could understand the process even if s/he knew nothing about that before. For those who were already contributors, they could also learn something new. Furthermore, I could make new friends from inside or outside of China in the workshop. It was awesome!"
|
||||
-->
|
||||
|
||||
工作坊结束后,XiangPeng Zhao 和一些与会者就他们的经历在微信和 Twitter 上进行了交谈。他们很高兴参加了 NCW,并就改进工作坊提出了一些建议。一位名叫 Mohammad 的与会者说:「我在工作坊上玩得很开心,学习了参与 k8s 贡献的整个过程。」另一位与会者 Jie Jia 说:「工作坊非常精彩。它系统地解释了如何为 Kubernetes 做出贡献。即使参与者之前对此一无所知,他(她)也可以理解这个过程。对于那些已经是贡献者的人,他们也可以学习到新东西。此外,我还可以在工作坊上结识来自国内外的新朋友。真是棒极了!」
|
||||
|
||||
<!--
|
||||
<!--
|
||||
SIG Contributor Experience will continue to run New Contributor Workshops at each upcoming KubeCon, including Seattle, Barcelona, and the return to Shanghai in June 2019. If you failed to get into one this year, register for one at a future KubeCon. And, when you meet an NCW attendee, make sure to welcome them to the community.
|
||||
-->
|
||||
|
||||
贡献者体验 SIG 将继续在未来的 KubeCon 上举办新贡献者工作坊,包括西雅图站、巴塞罗那站,然后在 2019 年六月回到上海。如果你今年未能参加,请在未来的 KubeCon 上注册。并且,如果你遇到工作坊的与会者,请务必欢迎他们加入社区。
|
||||
|
||||
<!--
|
||||
<!--
|
||||
Links:
|
||||
-->
|
||||
|
||||
链接:
|
||||
|
||||
<!--
|
||||
<!--
|
||||
* English versions of the slides: [PDF](https://gist.github.com/jberkus/889be25c234b01761ce44eccff816380#file-kubernetes-shanghai-english-pdf) or [Google Docs with speaker notes](https://docs.google.com/presentation/d/1l5f_iAFsKg50LFq3N80KbZKUIEL_tyCaUoWPzSxColo/edit?usp=sharing)
|
||||
* Chinese version of the slides: [PDF](https://gist.github.com/jberkus/889be25c234b01761ce44eccff816380#file-kubernetes-shanghai-cihinese-pdf)
|
||||
* [Contributor playground](https://github.com/kubernetes-sigs/contributor-playground)
|
||||
|
||||
Reference in New Issue
Block a user