From 6686dab93e8f8a8b1989174f2f8da14206e0dabe Mon Sep 17 00:00:00 2001 From: Sanford Ryza Date: Mon, 26 Aug 2013 19:59:12 +0000 Subject: [PATCH] YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via Sandy Ryza) git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1517669 13f79535-47bb-0310-9956-ffa450edef68 --- hadoop-yarn-project/CHANGES.txt | 3 ++ .../src/site/apt/FairScheduler.apt.vm | 29 ++++++++++++------- 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index c4069888867..5f4efcc502a 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -33,6 +33,9 @@ Release 2.1.1-beta - UNRELEASED YARN-1074. Cleaned up YARN CLI application list to only display running applications by default. (Xuan Gong via vinodkv) + YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via + Sandy Ryza) + OPTIMIZATIONS BUG FIXES diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm index a5adb4e838c..37b63e5962c 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm @@ -47,7 +47,7 @@ Hadoop MapReduce Next Generation - Fair Scheduler The scheduler organizes apps further into "queues", and shares resources fairly between these queues. By default, all users share a single queue, - called “default”. If an app specifically lists a queue in a container resource + named “default”. If an app specifically lists a queue in a container resource request, the request is submitted to that queue. It is also possible to assign queues based on the user name included with the request through configuration. Within each queue, a scheduling policy is used to share @@ -85,7 +85,7 @@ Hadoop MapReduce Next Generation - Fair Scheduler their parents in the fair scheduler configuration file. A queue's name starts with the names of its parents, with periods as - separators. So a queue named "queue1" under the root named, would be referred + separators. So a queue named "queue1" under the root queue, would be referred to as "root.queue1", and a queue named "queue2" under a queue named "parent1" would be referred to as "root.parent1.queue2". When referring to queues, the root part of the name is optional, so queue1 could be referred to as just @@ -118,22 +118,23 @@ Hadoop MapReduce Next Generation - Fair Scheduler Customizing the Fair Scheduler typically involves altering two files. First, scheduler-wide options can be set by adding configuration properties in the - fair-scheduler.xml file in your existing configuration directory. Second, in + yarn-site.xml file in your existing configuration directory. Second, in most cases users will want to create a manifest file listing which queues exist and their respective weights and capacities. The location of this file - is flexible - but it must be declared in fair-scheduler.xml. + is flexible - but it must be declared in yarn-site.xml. * <<>> * Path to allocation file. An allocation file is an XML manifest describing queues and their properties, in addition to certain policy defaults. This file must be in XML format as described in the next section. + Defaults to fair-scheduler.xml in configuration directory. * <<>> * Whether to use the username associated with the allocation as the default queue name, in the event that a queue name is not specified. If this is set - to "false" or unset, all jobs have a shared default queue, called "default". + to "false" or unset, all jobs have a shared default queue, named "default". Defaults to true. * <<>> @@ -178,14 +179,15 @@ Hadoop MapReduce Next Generation - Fair Scheduler Allocation file format - The allocation file must be in XML format. The format contains four types of + The allocation file must be in XML format. The format contains five types of elements: * <>, which represent queues. Each may contain the following properties: * minResources: minimum resources the queue is entitled to, in the form - "X mb, Y vcores". If a queue's minimum share is not satisfied, it will be + "X mb, Y vcores". For the single-resource fairness policy, the vcores + value is ignored. If a queue's minimum share is not satisfied, it will be offered available resources before any other queue under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied if its memory usage is below its minimum memory @@ -199,7 +201,8 @@ Allocation file format may be using those resources. * maxResources: maximum resources a queue is allowed, in the form - "X mb, Y vcores". A queue will never be assigned a container that would + "X mb, Y vcores". For the single-resource fairness policy, the vcores + value is ignored. A queue will never be assigned a container that would put its aggregate usage over this limit. * maxRunningApps: limit the number of apps from the queue to run at once @@ -234,19 +237,23 @@ Allocation file format its fair share before it will try to preempt containers to take resources from other queues. + * <>, which sets the default scheduling + policy for queues; overriden by the schedulingPolicy element in each queue + if specified. Defaults to "fair". + An example allocation file is given here: --- - 10000 mb - 90000 mb + 10000 mb,0vcores + 90000 mb,0vcores 50 2.0 fair - 5000 mb + 5000 mb,0vcores