From 1104c7d16685bd348df0510ea30ed30ca810ddd9 Mon Sep 17 00:00:00 2001 From: Alejandro Abdelnur Date: Wed, 12 Jun 2013 19:24:23 +0000 Subject: [PATCH] YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu) git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1492388 13f79535-47bb-0310-9956-ffa450edef68 --- hadoop-yarn-project/CHANGES.txt | 2 + .../src/site/apt/FairScheduler.apt.vm | 72 ++++++++++--------- 2 files changed, 42 insertions(+), 32 deletions(-) diff --git a/hadoop-yarn-project/CHANGES.txt b/hadoop-yarn-project/CHANGES.txt index 6f822ec2631..29a2bba3332 100644 --- a/hadoop-yarn-project/CHANGES.txt +++ b/hadoop-yarn-project/CHANGES.txt @@ -318,6 +318,8 @@ Release 2.1.0-beta - UNRELEASED YARN-600. Hook up cgroups CPU settings to the number of virtual cores allocated. (sandyr via tucu) + YARN-648. FS: Add documentation for pluggable policy. (kkambatl via tucu) + OPTIMIZATIONS YARN-512. Log aggregation root directory check is more expensive than it diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm index 371d7c244a2..96c55f21329 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/FairScheduler.apt.vm @@ -45,31 +45,16 @@ Hadoop MapReduce Next Generation - Fair Scheduler work with app priorities - the priorities are used as weights to determine the fraction of total resources that each app should get. - The scheduler organizes apps further into "queues", and shares resources - fairly between these queues. By default, all users share a single queue, - called “default”. If an app specifically lists a queue in a container - resource request, the request is submitted to that queue. It is also - possible to assign queues based on the user name included with the request - through configuration. Within each queue, a scheduling policy is used to share + The scheduler organizes apps further into "queues", and shares resources + fairly between these queues. By default, all users share a single queue, + called “default”. If an app specifically lists a queue in a container resource + request, the request is submitted to that queue. It is also possible to assign + queues based on the user name included with the request through + configuration. Within each queue, a scheduling policy is used to share resources between the running apps. The default is memory-based fair sharing, but FIFO and multi-resource with Dominant Resource Fairness can also be - configured. Queues can be configured with weights to share the cluster non-evenly. - - The fair scheduler supports hierarchical queues. All queues descend from a - queue named "root". Available resources are distributed among the children - of the root queue in the typical fair scheduling fashion. Then, the children - distribute the resources assigned to them to their children in the same - fashion. Applications may only be scheduled on leaf queues. Queues can be - specified as children of other queues by placing them as sub-elements of - their parents in the fair scheduler configuration file. - - A queue's name starts with the names of its parents, with periods as - separators. So a queue named "queue1" under the root named, would be - referred to as "root.queue1", and a queue named "queue2" under a queue - named "parent1" would be referred to as "root.parent1.queue2". When - referring to queues, the root part of the name is optional, so queue1 could - be referred to as just "queue1", and a queue2 could be referred to as just - "parent1.queue2". + configured. Queues can be arranged in a hierarchy to divide resources and + configured with weights to share the cluster in specific proportions. In addition to providing fair sharing, the Fair Scheduler allows assigning guaranteed minimum shares to queues, which is useful for ensuring that @@ -87,9 +72,31 @@ Hadoop MapReduce Next Generation - Fair Scheduler cause too much intermediate data to be created or too much context-switching. Limiting the apps does not cause any subsequently submitted apps to fail, only to wait in the scheduler's queue until some of the user's earlier apps - finish. Apps to run from each user/queue are chosen in the same fair sharing - manner, but can alternatively be configured to be chosen in order of submit - time, as in the default FIFO scheduler in Hadoop. + finish. + +* {Hierarchical queues with pluggable policies} + + The fair scheduler supports hierarchical queues. All queues descend from a + queue named "root". Available resources are distributed among the children + of the root queue in the typical fair scheduling fashion. Then, the children + distribute the resources assigned to them to their children in the same + fashion. Applications may only be scheduled on leaf queues. Queues can be + specified as children of other queues by placing them as sub-elements of + their parents in the fair scheduler configuration file. + + A queue's name starts with the names of its parents, with periods as + separators. So a queue named "queue1" under the root named, would be referred + to as "root.queue1", and a queue named "queue2" under a queue named "parent1" + would be referred to as "root.parent1.queue2". When referring to queues, the + root part of the name is optional, so queue1 could be referred to as just + "queue1", and a queue2 could be referred to as just "parent1.queue2". + + Additionally, the fair scheduler allows setting a different custom policy for + each queue to allow sharing the queue's resources in any which way the user + wants. A custom policy can be built by extending + <<>>. + FifoPolicy, FairSharePolicy (default), and DominantResourceFairnessPolicy are + built-in and can be readily used. Certain add-ons are not yet supported which existed in the original (MR1) Fair Scheduler. Among them, is the use of a custom policies governing @@ -201,11 +208,12 @@ Allocation file format default to 1, and a queue with weight 2 should receive approximately twice as many resources as a queue with the default weight. - * schedulingMode: either "fifo" or "fair" depending on the in-queue scheduling - policy desired. Defaults to "fair". If "fifo", apps with earlier submit - times are given preference for containers, but apps submitted later may - run concurrently if there is leftover space on the cluster after satisfying - the earlier app's requests. + * schedulingPolicy: to set the scheduling policy of any queue. The allowed + values are "fifo"/"fair"/"drf" or any class that extends + <<>>. + Defaults to "fair". If "fifo", apps with earlier submit times are given preference + for containers, but apps submitted later may run concurrently if there is + leftover space on the cluster after satisfying the earlier app's requests. * aclSubmitApps: a list of users that can submit apps to the queue. A (default) value of "*" means that any users can submit apps. A queue inherits the ACL of @@ -236,7 +244,7 @@ Allocation file format 90000 mb 50 2.0 - fair + fair 5000 mb