YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via Sandy Ryza)

git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1517669 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
Sanford Ryza 2013-08-26 19:59:12 +00:00
parent b45b1394b6
commit 6686dab93e
2 changed files with 21 additions and 11 deletions

View File

@ -33,6 +33,9 @@ Release 2.1.1-beta - UNRELEASED
YARN-1074. Cleaned up YARN CLI application list to only display running YARN-1074. Cleaned up YARN CLI application list to only display running
applications by default. (Xuan Gong via vinodkv) applications by default. (Xuan Gong via vinodkv)
YARN-1093. Corrections to Fair Scheduler documentation (Wing Yew Poon via
Sandy Ryza)
OPTIMIZATIONS OPTIMIZATIONS
BUG FIXES BUG FIXES

View File

@ -47,7 +47,7 @@ Hadoop MapReduce Next Generation - Fair Scheduler
The scheduler organizes apps further into "queues", and shares resources The scheduler organizes apps further into "queues", and shares resources
fairly between these queues. By default, all users share a single queue, fairly between these queues. By default, all users share a single queue,
called “default”. If an app specifically lists a queue in a container resource named “default”. If an app specifically lists a queue in a container resource
request, the request is submitted to that queue. It is also possible to assign request, the request is submitted to that queue. It is also possible to assign
queues based on the user name included with the request through queues based on the user name included with the request through
configuration. Within each queue, a scheduling policy is used to share configuration. Within each queue, a scheduling policy is used to share
@ -85,7 +85,7 @@ Hadoop MapReduce Next Generation - Fair Scheduler
their parents in the fair scheduler configuration file. their parents in the fair scheduler configuration file.
A queue's name starts with the names of its parents, with periods as A queue's name starts with the names of its parents, with periods as
separators. So a queue named "queue1" under the root named, would be referred separators. So a queue named "queue1" under the root queue, would be referred
to as "root.queue1", and a queue named "queue2" under a queue named "parent1" to as "root.queue1", and a queue named "queue2" under a queue named "parent1"
would be referred to as "root.parent1.queue2". When referring to queues, the would be referred to as "root.parent1.queue2". When referring to queues, the
root part of the name is optional, so queue1 could be referred to as just root part of the name is optional, so queue1 could be referred to as just
@ -118,22 +118,23 @@ Hadoop MapReduce Next Generation - Fair Scheduler
Customizing the Fair Scheduler typically involves altering two files. First, Customizing the Fair Scheduler typically involves altering two files. First,
scheduler-wide options can be set by adding configuration properties in the scheduler-wide options can be set by adding configuration properties in the
fair-scheduler.xml file in your existing configuration directory. Second, in yarn-site.xml file in your existing configuration directory. Second, in
most cases users will want to create a manifest file listing which queues most cases users will want to create a manifest file listing which queues
exist and their respective weights and capacities. The location of this file exist and their respective weights and capacities. The location of this file
is flexible - but it must be declared in fair-scheduler.xml. is flexible - but it must be declared in yarn-site.xml.
* <<<yarn.scheduler.fair.allocation.file>>> * <<<yarn.scheduler.fair.allocation.file>>>
* Path to allocation file. An allocation file is an XML manifest describing * Path to allocation file. An allocation file is an XML manifest describing
queues and their properties, in addition to certain policy defaults. This file queues and their properties, in addition to certain policy defaults. This file
must be in XML format as described in the next section. must be in XML format as described in the next section.
Defaults to fair-scheduler.xml in configuration directory.
* <<<yarn.scheduler.fair.user-as-default-queue>>> * <<<yarn.scheduler.fair.user-as-default-queue>>>
* Whether to use the username associated with the allocation as the default * Whether to use the username associated with the allocation as the default
queue name, in the event that a queue name is not specified. If this is set queue name, in the event that a queue name is not specified. If this is set
to "false" or unset, all jobs have a shared default queue, called "default". to "false" or unset, all jobs have a shared default queue, named "default".
Defaults to true. Defaults to true.
* <<<yarn.scheduler.fair.preemption>>> * <<<yarn.scheduler.fair.preemption>>>
@ -178,14 +179,15 @@ Hadoop MapReduce Next Generation - Fair Scheduler
Allocation file format Allocation file format
The allocation file must be in XML format. The format contains four types of The allocation file must be in XML format. The format contains five types of
elements: elements:
* <<Queue elements>>, which represent queues. Each may contain the following * <<Queue elements>>, which represent queues. Each may contain the following
properties: properties:
* minResources: minimum resources the queue is entitled to, in the form * minResources: minimum resources the queue is entitled to, in the form
"X mb, Y vcores". If a queue's minimum share is not satisfied, it will be "X mb, Y vcores". For the single-resource fairness policy, the vcores
value is ignored. If a queue's minimum share is not satisfied, it will be
offered available resources before any other queue under the same parent. offered available resources before any other queue under the same parent.
Under the single-resource fairness policy, a queue Under the single-resource fairness policy, a queue
is considered unsatisfied if its memory usage is below its minimum memory is considered unsatisfied if its memory usage is below its minimum memory
@ -199,7 +201,8 @@ Allocation file format
may be using those resources. may be using those resources.
* maxResources: maximum resources a queue is allowed, in the form * maxResources: maximum resources a queue is allowed, in the form
"X mb, Y vcores". A queue will never be assigned a container that would "X mb, Y vcores". For the single-resource fairness policy, the vcores
value is ignored. A queue will never be assigned a container that would
put its aggregate usage over this limit. put its aggregate usage over this limit.
* maxRunningApps: limit the number of apps from the queue to run at once * maxRunningApps: limit the number of apps from the queue to run at once
@ -234,19 +237,23 @@ Allocation file format
its fair share before it will try to preempt containers to take resources from its fair share before it will try to preempt containers to take resources from
other queues. other queues.
* <<A defaultQueueSchedulingPolicy element>>, which sets the default scheduling
policy for queues; overriden by the schedulingPolicy element in each queue
if specified. Defaults to "fair".
An example allocation file is given here: An example allocation file is given here:
--- ---
<?xml version="1.0"?> <?xml version="1.0"?>
<allocations> <allocations>
<queue name="sample_queue"> <queue name="sample_queue">
<minResources>10000 mb</minResources> <minResources>10000 mb,0vcores</minResources>
<maxResources>90000 mb</maxResources> <maxResources>90000 mb,0vcores</maxResources>
<maxRunningApps>50</maxRunningApps> <maxRunningApps>50</maxRunningApps>
<weight>2.0</weight> <weight>2.0</weight>
<schedulingPolicy>fair</schedulingPolicy> <schedulingPolicy>fair</schedulingPolicy>
<queue name="sample_sub_queue"> <queue name="sample_sub_queue">
<minResources>5000 mb</minResources> <minResources>5000 mb,0vcores</minResources>
</queue> </queue>
</queue> </queue>
<user name="sample_user"> <user name="sample_user">