Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda)

Browse Source 
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e
(cherry picked from commit 6b88cd1901a65b977fed759e322dcd75a2cd33b6)
This commit is contained in:
Wangda Tan 2017-09-07 10:07:35 -07:00 committed by Daniel Templeton
parent 4bf29910a8
commit 335e3a6334
2 changed files with 80 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
hadoop-project/src/site/site.xml
View File

@ -144,6 +144,7 @@
<item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/> <item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/>
<item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/> <item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/>
<item name="Shared Cache" href="hadoop-yarn/hadoop-yarn-site/SharedCache.html"/> <item name="Shared Cache" href="hadoop-yarn/hadoop-yarn-site/SharedCache.html"/>
<item name="Resource Profiles" href="hadoop-yarn/hadoop-yarn-site/ResourceProfiles.html"/>
</menu> </menu>
<menu name="YARN REST APIs" inherit="top"> <menu name="YARN REST APIs" inherit="top">

79
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md Normal file
View File

@ -0,0 +1,79 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
Hadoop: YARN Resource Types
===========================
Overview
--------
Resource types support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resource­types. This solution also helps the users to submit jobs with ease to specify the resources they need.
Resource model of YARN
-----------------------
Resource Manager will load a new configuration file named `resource-types.xml` to determine the set of resource ­types for which scheduling is enabled. Sample XML will look like below.
```xml
<configuration>
<property>
<name>yarn.resource-types</name>
<value>resource1, resource2</value>
</property>
<property>
<name>yarn.resource-types.resource1.units</name>
<value>G</value>
</property>
</configuration>
```
Similarly, a new configuration file `node­-resources.xml` will also be loaded by Node Manager where the resource capabilities of a node can be specified.
```xml
<configuration>
<property>
<name>yarn.nodemanager.resource-type.resource1</name>
<value>5G</value>
</property>
<property>
<name>yarn.nodemanager.resource-type.resource2</name>
<value>2m</value>
</property>
</configuration>
```
Node Manager will use these custom resource types and will register it's capability to Resource Manager.
Configurations
-------------
Please note that, `resource-types.xml` and `node­-resources.xml` file also need to be placed in conf directory if new resources are to be added to YARN.
*In `resource-types.xml`*
| Configuration Property | Value | Description |
|:---- |:---- |:---- |
| `yarn.resource-types` | resource1 | Custom resource |
| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type |
*In `node­-resources.xml`*
| Configuration Property | Value | Description |
|:---- |:---- |:---- |
| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named 'resource1'. |