Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

YARN-7736. Fix itemization in YARN federation document 

Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
This commit is contained in:
Sen Zhao 2018-03-05 11:22:09 -08:00 committed by Akira Ajisaka
parent 1054b48c27
commit 245751ffdc
No known key found for this signature in database
GPG Key ID: C1EDBB9CA400FD50
1 changed files with 2 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/Federation.md
View File

@ -29,6 +29,7 @@ This architecture can provide very tight enforcement of scheduling invariants wi
Federation is designed as a “layer” atop of existing YARN codebase, with limited changes in the core YARN mechanisms. Federation is designed as a “layer” atop of existing YARN codebase, with limited changes in the core YARN mechanisms.
Assumptions: Assumptions:
* We assume reasonably good connectivity across sub-clusters (e.g., we are not looking to federate across DC yet, though future investigations of this are not excluded). * We assume reasonably good connectivity across sub-clusters (e.g., we are not looking to federate across DC yet, though future investigations of this are not excluded).
* We rely on HDFS federation (or equivalently scalable DFS solutions) to take care of scalability of the store side. * We rely on HDFS federation (or equivalently scalable DFS solutions) to take care of scalability of the store side.
@ -71,6 +72,7 @@ to minimize overhead on the scheduling infrastructure (more in section on scalab
![Architecture of the AMRMProxy interceptor chain | width=800](./images/amrmproxy_architecture.png) ![Architecture of the AMRMProxy interceptor chain | width=800](./images/amrmproxy_architecture.png)
*Role of AMRMProxy* *Role of AMRMProxy*
1. Protect the sub-cluster YARN RMs from misbehaving AMs. The AMRMProxy can prevent DDOS attacks by throttling/killing AMs that are asking too many resources. 1. Protect the sub-cluster YARN RMs from misbehaving AMs. The AMRMProxy can prevent DDOS attacks by throttling/killing AMs that are asking too many resources.
2. Mask the multiple YARN RMs in the cluster, and can transparently allow the AM to span across sub-clusters. All container allocations are done by the YARN RM framework that consists of the AMRMProxy fronting the home and other sub-cluster RMs. 2. Mask the multiple YARN RMs in the cluster, and can transparently allow the AM to span across sub-clusters. All container allocations are done by the YARN RM framework that consists of the AMRMProxy fronting the home and other sub-cluster RMs.
3. Intercepts all the requests, thus it can enforce application quotas, which would not be enforceable by sub-cluster RM (as each only see a fraction of the AM requests). 3. Intercepts all the requests, thus it can enforce application quotas, which would not be enforceable by sub-cluster RM (as each only see a fraction of the AM requests).