YARN-7369. Improve the resource types docs
This commit is contained in:
Daniel Templeton
parent
5b55a74bad
commit
c2f479c248
|
|
@ -128,6 +128,7 @@
|
||||||
<item name="Fair Scheduler" href="hadoop-yarn/hadoop-yarn-site/FairScheduler.html"/>
|
<item name="Fair Scheduler" href="hadoop-yarn/hadoop-yarn-site/FairScheduler.html"/>
|
||||||
<item name="ResourceManager Restart" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html"/>
|
<item name="ResourceManager Restart" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html"/>
|
||||||
<item name="ResourceManager HA" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html"/>
|
<item name="ResourceManager HA" href="hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html"/>
|
||||||
|
<item name="Resource Model" href="hadoop-yarn/hadoop-yarn-site/ResourceModel.html"/>
|
||||||
<item name="Node Labels" href="hadoop-yarn/hadoop-yarn-site/NodeLabel.html"/>
|
<item name="Node Labels" href="hadoop-yarn/hadoop-yarn-site/NodeLabel.html"/>
|
||||||
<item name="Web Application Proxy" href="hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html"/>
|
<item name="Web Application Proxy" href="hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html"/>
|
||||||
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html"/>
|
<item name="Timeline Server" href="hadoop-yarn/hadoop-yarn-site/TimelineServer.html"/>
|
||||||
|
|
@ -144,7 +145,6 @@
|
||||||
<item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/>
|
<item name="Opportunistic Containers" href="hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html"/>
|
||||||
<item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/>
|
<item name="YARN Federation" href="hadoop-yarn/hadoop-yarn-site/Federation.html"/>
|
||||||
<item name="Shared Cache" href="hadoop-yarn/hadoop-yarn-site/SharedCache.html"/>
|
<item name="Shared Cache" href="hadoop-yarn/hadoop-yarn-site/SharedCache.html"/>
|
||||||
<item name="Resource Profiles" href="hadoop-yarn/hadoop-yarn-site/ResourceProfiles.html"/>
|
|
||||||
</menu>
|
</menu>
|
||||||
|
|
||||||
<menu name="YARN REST APIs" inherit="top">
|
<menu name="YARN REST APIs" inherit="top">
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,217 @@
|
||||||
|
<!--
|
||||||
|
Licensed to the Apache Software Foundation (ASF) under one or more
|
||||||
|
contributor license agreements. See the NOTICE file distributed with
|
||||||
|
this work for additional information regarding copyright ownership.
|
||||||
|
The ASF licenses this file to You under the Apache License, Version 2.0
|
||||||
|
(the "License"); you may not use this file except in compliance with
|
||||||
|
the License. You may obtain a copy of the License at
|
||||||
|
|
||||||
|
http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
|
||||||
|
Unless required by applicable law or agreed to in writing, software
|
||||||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
||||||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||||
|
See the License for the specific language governing permissions and
|
||||||
|
limitations under the License.
|
||||||
|
-->
|
||||||
|
|
||||||
|
Hadoop: YARN Resource Configuration
|
||||||
|
===================================
|
||||||
|
|
||||||
|
Overview
|
||||||
|
--------
|
||||||
|
YARN supports an extensible resource model. By default YARN tracks CPU and
|
||||||
|
memory for all nodes, applications, and queues, but the resource definition
|
||||||
|
can be extended to include arbitrary "countable" resources. A countable
|
||||||
|
resource is a resource that is consumed while a container is running, but is
|
||||||
|
released afterwards. CPU and memory are both countable resources. Other examples
|
||||||
|
include GPU resources and software licenses.
|
||||||
|
|
||||||
|
Configuration
|
||||||
|
-------------
|
||||||
|
|
||||||
|
The following configuration properties are supported. See below for details.
|
||||||
|
|
||||||
|
`resource-types.xml`
|
||||||
|
|
||||||
|
| Configuration Property | Value | Description |
|
||||||
|
|:---- |:---- |:---- |
|
||||||
|
| `yarn.resource-types` | Comma-separated list of additional resources. May not include `memory`, `memory-mb`, or `vcores` |
|
||||||
|
| `yarn.resource-types.<resource>.units` | Default unit for the specified resource type |
|
||||||
|
| `yarn.resource-types.<resource>.minimum` | The minimum request for the specified resource type |
|
||||||
|
| `yarn.resource-types.<resource>.maximum` | The maximum request for the specified resource type |
|
||||||
|
|
||||||
|
`node-resources.xml`
|
||||||
|
|
||||||
|
| Configuration Property | Value | Description |
|
||||||
|
|:---- |:---- |:---- |
|
||||||
|
| `yarn.nodemanager.resource-type.<resource>` | The count of the specified resource available from the node manager |
|
||||||
|
|
||||||
|
Please note that the `resource-types.xml` and `node-resources.xml` files
|
||||||
|
also need to be placed in the same configuration directory as `yarn-site.xml` if
|
||||||
|
they are used. Alternatively, the properties may be placed into the
|
||||||
|
`yarn-site.xml` file instead.
|
||||||
|
|
||||||
|
YARN Resource Model
|
||||||
|
-------------------
|
||||||
|
|
||||||
|
### Resource Manager
|
||||||
|
The resource manager is the final arbiter of what resources in the cluster are
|
||||||
|
tracked. The resource manager loads its resource definition from XML
|
||||||
|
configuration files. For example, to define a new resource in addition to
|
||||||
|
CPU and memory, the following property should be configured:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<configuration>
|
||||||
|
<property>
|
||||||
|
<name>yarn.resource-types</name>
|
||||||
|
<value>resource1,resource2</value>
|
||||||
|
<description>
|
||||||
|
The resources to be used for scheduling. Use resource-types.xml
|
||||||
|
to specify details about the individual resource types.
|
||||||
|
</description>
|
||||||
|
</property>
|
||||||
|
</configuration>
|
||||||
|
```
|
||||||
|
|
||||||
|
A valid resource name must begin with a letter and contain only letters, numbers,
|
||||||
|
and any of: '.', '_', or '-'. A valid resource name may also be optionally
|
||||||
|
preceded by a name space followed by a slash. A valid name space consists of
|
||||||
|
period-separated groups of letters, numbers, and dashes. For example, the
|
||||||
|
following are valid resource names:
|
||||||
|
|
||||||
|
* myresource
|
||||||
|
* my_resource
|
||||||
|
* My-Resource01
|
||||||
|
* com.acme/myresource
|
||||||
|
|
||||||
|
The following are examples of invalid resource names:
|
||||||
|
|
||||||
|
* 10myresource
|
||||||
|
* my resource
|
||||||
|
* com/acme/myresource
|
||||||
|
* $NS/myresource
|
||||||
|
* -none-/myresource
|
||||||
|
|
||||||
|
For each new resource type defined an optional unit property can be added to
|
||||||
|
set the default unit for the resource type. Valid values are:
|
||||||
|
|
||||||
|
|Unit Name | Meaning |
|
||||||
|
|:---- |:---- |
|
||||||
|
| p | pico |
|
||||||
|
| n | nano |
|
||||||
|
| u | micro |
|
||||||
|
| m | milli |
|
||||||
|
| | default, i.e. no unit |
|
||||||
|
| k | kilo |
|
||||||
|
| M | mega |
|
||||||
|
| G | giga |
|
||||||
|
| T | tera |
|
||||||
|
| P | peta |
|
||||||
|
| Ki | binary kilo, i.e. 1024 |
|
||||||
|
| Mi | binary mega, i.e. 1024^2 |
|
||||||
|
| Gi | binary giga, i.e. 1024^3 |
|
||||||
|
| Ti | binary tera, i.e. 1024^4 |
|
||||||
|
| Pi | binary peta, i.e. 1024^5 |
|
||||||
|
|
||||||
|
The property must be named `yarn.resource-types.<resource>.units`. Each defined
|
||||||
|
resource may also have optional minimum and maximum properties. The properties
|
||||||
|
must be named `yarn.resource-types.<resource>.minimum` and
|
||||||
|
`yarn.resource-types.<resource>.maximum`.
|
||||||
|
|
||||||
|
The `yarn.resource-types` property and any unit, mimimum, or maximum properties
|
||||||
|
may be defined in either the usual `yarn-site.xml` file or in a file named
|
||||||
|
`resource-types.xml`. For example, the following could appear in either file:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<configuration>
|
||||||
|
<property>
|
||||||
|
<name>yarn.resource-types</name>
|
||||||
|
<value>resource1, resource2</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>yarn.resource-types.resource1.units</name>
|
||||||
|
<value>G</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>yarn.resource-types.resource2.minimum</name>
|
||||||
|
<value>1</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>yarn.resource-types.resource2.maximum</name>
|
||||||
|
<value>1024</value>
|
||||||
|
</property>
|
||||||
|
</configuration>
|
||||||
|
```
|
||||||
|
|
||||||
|
### Node Manager
|
||||||
|
|
||||||
|
Each node manager independently defines the resources that are available from
|
||||||
|
that node. The resource definition is done through setting a property for each
|
||||||
|
available resource. The property must be named
|
||||||
|
`yarn.nodemanager.resource-type.<resource>` and may be placed in the usual
|
||||||
|
`yarn-site.xml` file or in a file named `noderesources.xml`. The value of the
|
||||||
|
property should be the amount of that resource offered by the node. For
|
||||||
|
example:
|
||||||
|
|
||||||
|
```xml
|
||||||
|
<configuration>
|
||||||
|
<property>
|
||||||
|
<name>yarn.nodemanager.resource-type.resource1</name>
|
||||||
|
<value>5G</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
<property>
|
||||||
|
<name>yarn.nodemanager.resource-type.resource2</name>
|
||||||
|
<value>2m</value>
|
||||||
|
</property>
|
||||||
|
|
||||||
|
</configuration>
|
||||||
|
```
|
||||||
|
|
||||||
|
Note that the units used for these resources need not match the definition
|
||||||
|
held by the resource manager. If the units do not match, the resource
|
||||||
|
manager will automatically do a conversion.
|
||||||
|
|
||||||
|
### Using Resources With MapReduce
|
||||||
|
|
||||||
|
MapReduce requests three different kinds of containers from YARN: the
|
||||||
|
application master container, map containers, and reduce containers. For each
|
||||||
|
container type, there is a corresponding set of properties that can be used to
|
||||||
|
set the resources requested.
|
||||||
|
|
||||||
|
The properties for setting resource requests in MapReduce are:
|
||||||
|
|
||||||
|
| Property | Description |
|
||||||
|
|:---- |:---- |
|
||||||
|
| `yarn.app.mapreduce.am.resource.mb` | Sets the memory requested for the application master container to the value in MB. No longer preferred. Use `yarn.app.mapreduce.am.resource.memory-mb` instead. Defaults to 1536. |
|
||||||
|
| `yarn.app.mapreduce.am.resource.memory` | Sets the memory requested for the application master container to the value in MB. No longer preferred. Use `yarn.app.mapreduce.am.resource.memory-mb` instead. Defaults to 1536. |
|
||||||
|
| `yarn.app.mapreduce.am.resource.memory-mb` | Sets the memory requested for the application master container to the value in MB. Defaults to 1536. |
|
||||||
|
| `yarn.app.mapreduce.am.resource.cpu-vcores` | Sets the CPU requested for the application master container to the value. No longer preferred. Use `yarn.app.mapreduce.am.resource.vcores` instead. Defaults to 1. |
|
||||||
|
| `yarn.app.mapreduce.am.resource.vcores` | Sets the CPU requested for the application master container to the value. Defaults to 1. |
|
||||||
|
| `yarn.app.mapreduce.am.resource.<resource>` | Sets the quantity requested of `<resource>` for the application master container to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. |
|
||||||
|
| `mapreduce.map.memory.mb` | Sets the memory requested for the all map task containers to the value in MB. No longer preferred. Use `mapreduce.map.resource.memory-mb` instead. Defaults to 1024. |
|
||||||
|
| `mapreduce.map.resource.memory` | Sets the memory requested for the all map task containers to the value in MB. No longer preferred. Use `mapreduce.map.resource.memory-mb` instead. Defaults to 1024. |
|
||||||
|
| `mapreduce.map.resource.memory-mb` | Sets the memory requested for the all map task containers to the value in MB. Defaults to 1024. |
|
||||||
|
| `mapreduce.map.cpu.vcores` | Sets the CPU requested for the all map task containers to the value. No longer preferred. Use `mapreduce.map.resource.vcores` instead. Defaults to 1. |
|
||||||
|
| `mapreduce.map.resource.vcores` | Sets the CPU requested for the all map task containers to the value. Defaults to 1. |
|
||||||
|
| `mapreduce.map.resource.<resource>` | Sets the quantity requested of `<resource>` for the all map task containers to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. |
|
||||||
|
| `mapreduce.reduce.memory.mb` | Sets the memory requested for the all reduce task containers to the value in MB. No longer preferred. Use `mapreduce.reduce.resource.memory-mb` instead. Defaults to 1024. |
|
||||||
|
| `mapreduce.reduce.resource.memory` | Sets the memory requested for the all reduce task containers to the value in MB. No longer preferred. Use `mapreduce.reduce.resource.memory-mb` instead. Defaults to 1024. |
|
||||||
|
| `mapreduce.reduce.resource.memory-mb` | Sets the memory requested for the all reduce task containers to the value in MB. Defaults to 1024. |
|
||||||
|
| `mapreduce.reduce.cpu.vcores` | Sets the CPU requested for the all reduce task containers to the value. No longer preferred. Use `mapreduce.reduce.resource.vcores` instead. Defaults to 1. |
|
||||||
|
| `mapreduce.reduce.resource.vcores` | Sets the CPU requested for the all reduce task containers to the value. Defaults to 1. |
|
||||||
|
| `mapreduce.reduce.resource.<resource>` | Sets the quantity requested of `<resource>` for the all reduce task containers to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. |
|
||||||
|
|
||||||
|
Note that these resource requests may be modified by YARN to meet the configured
|
||||||
|
minimum and maximum resource values or to be a multiple of the configured
|
||||||
|
increment. See the `yarn.scheduler.maximum-allocation-mb`,
|
||||||
|
`yarn.scheduler.minimum-allocation-mb`,
|
||||||
|
`yarn.scheduler.increment-allocation-mb`,
|
||||||
|
`yarn.scheduler.maximum-allocation-vcores`,
|
||||||
|
`yarn.scheduler.minimum-allocation-vcores`, and
|
||||||
|
`yarn.scheduler.increment-allocation-vcores` properties in the YARN scheduler
|
||||||
|
configuration.
|
||||||
|
|
@ -1,79 +0,0 @@
|
||||||
<!--
|
|
||||||
Licensed to the Apache Software Foundation (ASF) under one or more
|
|
||||||
contributor license agreements. See the NOTICE file distributed with
|
|
||||||
this work for additional information regarding copyright ownership.
|
|
||||||
The ASF licenses this file to You under the Apache License, Version 2.0
|
|
||||||
(the "License"); you may not use this file except in compliance with
|
|
||||||
the License. You may obtain a copy of the License at
|
|
||||||
|
|
||||||
http://www.apache.org/licenses/LICENSE-2.0
|
|
||||||
|
|
||||||
Unless required by applicable law or agreed to in writing, software
|
|
||||||
distributed under the License is distributed on an "AS IS" BASIS,
|
|
||||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
||||||
See the License for the specific language governing permissions and
|
|
||||||
limitations under the License.
|
|
||||||
-->
|
|
||||||
|
|
||||||
Hadoop: YARN Resource Types
|
|
||||||
===========================
|
|
||||||
|
|
||||||
Overview
|
|
||||||
--------
|
|
||||||
Resource types support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resourcetypes. This solution also helps the users to submit jobs with ease to specify the resources they need.
|
|
||||||
|
|
||||||
Resource model of YARN
|
|
||||||
-----------------------
|
|
||||||
Resource Manager will load a new configuration file named `resource-types.xml` to determine the set of resource types for which scheduling is enabled. Sample XML will look like below.
|
|
||||||
|
|
||||||
```xml
|
|
||||||
<configuration>
|
|
||||||
<property>
|
|
||||||
<name>yarn.resource-types</name>
|
|
||||||
<value>resource1, resource2</value>
|
|
||||||
</property>
|
|
||||||
|
|
||||||
<property>
|
|
||||||
<name>yarn.resource-types.resource1.units</name>
|
|
||||||
<value>G</value>
|
|
||||||
</property>
|
|
||||||
</configuration>
|
|
||||||
```
|
|
||||||
|
|
||||||
Similarly, a new configuration file `node-resources.xml` will also be loaded by Node Manager where the resource capabilities of a node can be specified.
|
|
||||||
|
|
||||||
```xml
|
|
||||||
<configuration>
|
|
||||||
<property>
|
|
||||||
<name>yarn.nodemanager.resource-type.resource1</name>
|
|
||||||
<value>5G</value>
|
|
||||||
</property>
|
|
||||||
|
|
||||||
<property>
|
|
||||||
<name>yarn.nodemanager.resource-type.resource2</name>
|
|
||||||
<value>2m</value>
|
|
||||||
</property>
|
|
||||||
|
|
||||||
</configuration>
|
|
||||||
```
|
|
||||||
|
|
||||||
Node Manager will use these custom resource types and will register it's capability to Resource Manager.
|
|
||||||
|
|
||||||
Configurations
|
|
||||||
-------------
|
|
||||||
|
|
||||||
Please note that, `resource-types.xml` and `node-resources.xml` file also need to be placed in conf directory if new resources are to be added to YARN.
|
|
||||||
|
|
||||||
*In `resource-types.xml`*
|
|
||||||
|
|
||||||
| Configuration Property | Value | Description |
|
|
||||||
|:---- |:---- |:---- |
|
|
||||||
| `yarn.resource-types` | resource1 | Custom resource |
|
|
||||||
| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type |
|
|
||||||
|
|
||||||
*In `node-resources.xml`*
|
|
||||||
|
|
||||||
| Configuration Property | Value | Description |
|
|
||||||
|:---- |:---- |:---- |
|
|
||||||
| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named 'resource1'. |
|
|
||||||
|
|
||||||
Loading…
Reference in New Issue