From a8848aebe7d22cb3a6f319771bd46a9b50dad8af Mon Sep 17 00:00:00 2001 From: Thomas White Date: Wed, 3 Apr 2013 18:03:21 +0000 Subject: [PATCH] Merge -r 1464129:1464130 from trunk to branch-2. Fixes: YARN-381. Improve fair scheduler docs. Contributed by Sandy Ryza. git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1464131 13f79535-47bb-0310-9956-ffa450edef68 --- .../src/site/apt/ClusterSetup.apt.vm | 2 +- hadoop-yarn-project/CHANGES.txt | 2 ++ .../src/site/apt/FairScheduler.apt.vm | 30 ++++++++++++++----- 3 files changed, 26 insertions(+), 8 deletions(-) diff --git a/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm b/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm index da69b303b37..7ac6755b5a4 100644 --- a/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm +++ b/hadoop-common-project/hadoop-common/src/site/apt/ClusterSetup.apt.vm @@ -249,7 +249,7 @@ Hadoop MapReduce Next Generation - Cluster Setup *-------------------------+-------------------------+------------------------+ | <<>> | | | | | <<>> Scheduler class. | | -| | | <<>> (recommended) or <<>> | +| | | <<>> (recommended), <<>> (also recommended), or <<>> | *-------------------------+-------------------------+------------------------+ | <<>> | | | | | Minimum limit of memory to allocate to each container request at the <<>>. | | diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 73b46c59475..6b4ec7852d3 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -57,6 +57,8 @@ Release 2.0.5-beta - UNRELEASED YARN-447. Move ApplicationComparator in CapacityScheduler to use comparator in ApplicationId. (Nemon Lou via vinodkv) + YARN-381. Improve fair scheduler docs. (Sandy Ryza via tomwhite) + OPTIMIZATIONS BUG FIXES diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm index 2d12699bf73..c7a8229d36f 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm @@ -85,8 +85,9 @@ Hadoop MapReduce Next Generation - Fair Scheduler cause too much intermediate data to be created or too much context-switching. Limiting the apps does not cause any subsequently submitted apps to fail, only to wait in the scheduler's queue until some of the user's earlier apps - finish. apps to run from each user/queue are chosen in order of priority and - then submit time, as in the default FIFO scheduler in Hadoop. + finish. Apps to run from each user/queue are chosen in the same fair sharing + manner, but can alternatively be configured to be chosen in order of submit + time, as in the default FIFO scheduler in Hadoop. Certain add-ons are not yet supported which existed in the original (MR1) Fair Scheduler. Among them, is the use of a custom policies governing @@ -142,7 +143,9 @@ Hadoop MapReduce Next Generation - Fair Scheduler * <<>> * Whether to assign shares to individual apps based on their size, rather than - providing an equal share to all apps regardless of size. Defaults to false. + providing an equal share to all apps regardless of size. When set to true, + apps are weighted by the natural logarithm of one plus the app's total + requested memory, divided by the natural logarithm of 2. Defaults to false. * <<>> @@ -180,16 +183,29 @@ Allocation file format * <>, which represent queues. Each may contain the following properties: - * minResources: minimum amount of aggregate memory + * minResources: minimum MB of aggregate memory the queue expects. If a queue + demands resources, and its current allocation is below its configured minimum, + it will be assigned available resources before any queue that is not in this + situation. If multiple queues are in this situation, resources go to the + queue with the smallest ratio between allocation and minimum. Note that it is + possible that a queue that is below its minimum may not immediately get up to + its minimum when it submits an application, because already-running jobs may + be using those resources. - * maxResources: maximum amount of aggregate memory + * maxResources: maximum MB of aggregate memory a queue is allowed. A queue + will never be assigned a container that would put it over this limit. * maxRunningApps: limit the number of apps from the queue to run at once - * weight: to share the cluster non-proportionally with other queues + * weight: to share the cluster non-proportionally with other queues. Weights + default to 1, and a queue with weight 2 should receive approximately twice + as many resources as a queue with the default weight. * schedulingMode: either "fifo" or "fair" depending on the in-queue scheduling - policy desired + policy desired. Defaults to "fair". If "fifo", apps with earlier submit + times are given preference for containers, but apps submitted later may + run concurrently if there is leftover space on the cluster after satisfying + the earlier app's requests. * aclSubmitApps: a list of users that can submit apps to the queue. A (default) value of "*" means that any users can submit apps. A queue inherits the ACL of