diff --git a/hadoop-project/src/site/site.xml b/hadoop-project/src/site/site.xml index 2aa1da7f531..35aa8220418 100644 --- a/hadoop-project/src/site/site.xml +++ b/hadoop-project/src/site/site.xml @@ -128,6 +128,7 @@ + @@ -144,7 +145,6 @@ - diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceModel.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceModel.md new file mode 100644 index 00000000000..ce968cef100 --- /dev/null +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceModel.md @@ -0,0 +1,217 @@ + + +Hadoop: YARN Resource Configuration +=================================== + +Overview +-------- +YARN supports an extensible resource model. By default YARN tracks CPU and +memory for all nodes, applications, and queues, but the resource definition +can be extended to include arbitrary "countable" resources. A countable +resource is a resource that is consumed while a container is running, but is +released afterwards. CPU and memory are both countable resources. Other examples +include GPU resources and software licenses. + +Configuration +------------- + +The following configuration properties are supported. See below for details. + +`resource-types.xml` + +| Configuration Property | Value | Description | +|:---- |:---- |:---- | +| `yarn.resource-types` | Comma-separated list of additional resources. May not include `memory`, `memory-mb`, or `vcores` | +| `yarn.resource-types..units` | Default unit for the specified resource type | +| `yarn.resource-types..minimum` | The minimum request for the specified resource type | +| `yarn.resource-types..maximum` | The maximum request for the specified resource type | + +`node­-resources.xml` + +| Configuration Property | Value | Description | +|:---- |:---- |:---- | +| `yarn.nodemanager.resource-type.` | The count of the specified resource available from the node manager | + +Please note that the `resource-types.xml` and `node­-resources.xml` files +also need to be placed in the same configuration directory as `yarn-site.xml` if +they are used. Alternatively, the properties may be placed into the +`yarn-site.xml` file instead. + +YARN Resource Model +------------------- + +### Resource Manager +The resource manager is the final arbiter of what resources in the cluster are +tracked. The resource manager loads its resource definition from XML +configuration files. For example, to define a new resource in addition to +CPU and memory, the following property should be configured: + +```xml + + + yarn.resource-types + resource1,resource2 + + The resources to be used for scheduling. Use resource-types.xml + to specify details about the individual resource types. + + + +``` + +A valid resource name must begin with a letter and contain only letters, numbers, +and any of: '.', '_', or '-'. A valid resource name may also be optionally +preceded by a name space followed by a slash. A valid name space consists of +period-separated groups of letters, numbers, and dashes. For example, the +following are valid resource names: + +* myresource +* my_resource +* My-Resource01 +* com.acme/myresource + +The following are examples of invalid resource names: + +* 10myresource +* my resource +* com/acme/myresource +* $NS/myresource +* -none-/myresource + +For each new resource type defined an optional unit property can be added to +set the default unit for the resource type. Valid values are: + +|Unit Name | Meaning | +|:---- |:---- | +| p | pico | +| n | nano | +| u | micro | +| m | milli | +| | default, i.e. no unit | +| k | kilo | +| M | mega | +| G | giga | +| T | tera | +| P | peta | +| Ki | binary kilo, i.e. 1024 | +| Mi | binary mega, i.e. 1024^2 | +| Gi | binary giga, i.e. 1024^3 | +| Ti | binary tera, i.e. 1024^4 | +| Pi | binary peta, i.e. 1024^5 | + +The property must be named `yarn.resource-types..units`. Each defined +resource may also have optional minimum and maximum properties. The properties +must be named `yarn.resource-types..minimum` and +`yarn.resource-types..maximum`. + +The `yarn.resource-types` property and any unit, mimimum, or maximum properties +may be defined in either the usual `yarn-site.xml` file or in a file named +`resource-types.xml`. For example, the following could appear in either file: + +```xml + + + yarn.resource-types + resource1, resource2 + + + + yarn.resource-types.resource1.units + G + + + + yarn.resource-types.resource2.minimum + 1 + + + + yarn.resource-types.resource2.maximum + 1024 + + +``` + +### Node Manager + +Each node manager independently defines the resources that are available from +that node. The resource definition is done through setting a property for each +available resource. The property must be named +`yarn.nodemanager.resource-type.` and may be placed in the usual +`yarn-site.xml` file or in a file named `node­resources.xml`. The value of the +property should be the amount of that resource offered by the node. For +example: + +```xml + + + yarn.nodemanager.resource-type.resource1 + 5G + + + + yarn.nodemanager.resource-type.resource2 + 2m + + + +``` + +Note that the units used for these resources need not match the definition +held by the resource manager. If the units do not match, the resource +manager will automatically do a conversion. + +### Using Resources With MapReduce + +MapReduce requests three different kinds of containers from YARN: the +application master container, map containers, and reduce containers. For each +container type, there is a corresponding set of properties that can be used to +set the resources requested. + +The properties for setting resource requests in MapReduce are: + +| Property | Description | +|:---- |:---- | +| `yarn.app.mapreduce.am.resource.mb` | Sets the memory requested for the application master container to the value in MB. No longer preferred. Use `yarn.app.mapreduce.am.resource.memory-mb` instead. Defaults to 1536. | +| `yarn.app.mapreduce.am.resource.memory` | Sets the memory requested for the application master container to the value in MB. No longer preferred. Use `yarn.app.mapreduce.am.resource.memory-mb` instead. Defaults to 1536. | +| `yarn.app.mapreduce.am.resource.memory-mb` | Sets the memory requested for the application master container to the value in MB. Defaults to 1536. | +| `yarn.app.mapreduce.am.resource.cpu-vcores` | Sets the CPU requested for the application master container to the value. No longer preferred. Use `yarn.app.mapreduce.am.resource.vcores` instead. Defaults to 1. | +| `yarn.app.mapreduce.am.resource.vcores` | Sets the CPU requested for the application master container to the value. Defaults to 1. | +| `yarn.app.mapreduce.am.resource.` | Sets the quantity requested of `` for the application master container to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. | +| `mapreduce.map.memory.mb` | Sets the memory requested for the all map task containers to the value in MB. No longer preferred. Use `mapreduce.map.resource.memory-mb` instead. Defaults to 1024. | +| `mapreduce.map.resource.memory` | Sets the memory requested for the all map task containers to the value in MB. No longer preferred. Use `mapreduce.map.resource.memory-mb` instead. Defaults to 1024. | +| `mapreduce.map.resource.memory-mb` | Sets the memory requested for the all map task containers to the value in MB. Defaults to 1024. | +| `mapreduce.map.cpu.vcores` | Sets the CPU requested for the all map task containers to the value. No longer preferred. Use `mapreduce.map.resource.vcores` instead. Defaults to 1. | +| `mapreduce.map.resource.vcores` | Sets the CPU requested for the all map task containers to the value. Defaults to 1. | +| `mapreduce.map.resource.` | Sets the quantity requested of `` for the all map task containers to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. | +| `mapreduce.reduce.memory.mb` | Sets the memory requested for the all reduce task containers to the value in MB. No longer preferred. Use `mapreduce.reduce.resource.memory-mb` instead. Defaults to 1024. | +| `mapreduce.reduce.resource.memory` | Sets the memory requested for the all reduce task containers to the value in MB. No longer preferred. Use `mapreduce.reduce.resource.memory-mb` instead. Defaults to 1024. | +| `mapreduce.reduce.resource.memory-mb` | Sets the memory requested for the all reduce task containers to the value in MB. Defaults to 1024. | +| `mapreduce.reduce.cpu.vcores` | Sets the CPU requested for the all reduce task containers to the value. No longer preferred. Use `mapreduce.reduce.resource.vcores` instead. Defaults to 1. | +| `mapreduce.reduce.resource.vcores` | Sets the CPU requested for the all reduce task containers to the value. Defaults to 1. | +| `mapreduce.reduce.resource.` | Sets the quantity requested of `` for the all reduce task containers to the value. If no unit is specified, the default unit for the resource is assumed. See the section on units above. | + +Note that these resource requests may be modified by YARN to meet the configured +minimum and maximum resource values or to be a multiple of the configured +increment. See the `yarn.scheduler.maximum-allocation-mb`, +`yarn.scheduler.minimum-allocation-mb`, +`yarn.scheduler.increment-allocation-mb`, +`yarn.scheduler.maximum-allocation-vcores`, +`yarn.scheduler.minimum-allocation-vcores`, and +`yarn.scheduler.increment-allocation-vcores` properties in the YARN scheduler +configuration. diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md deleted file mode 100644 index e7b38e13399..00000000000 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceProfiles.md +++ /dev/null @@ -1,79 +0,0 @@ - - -Hadoop: YARN Resource Types -=========================== - -Overview --------- -Resource types support in YARN helps to extend the YARN resource model to a more flexible model which makes it easier to add new countable resource­types. This solution also helps the users to submit jobs with ease to specify the resources they need. - -Resource model of YARN ------------------------ -Resource Manager will load a new configuration file named `resource-types.xml` to determine the set of resource ­types for which scheduling is enabled. Sample XML will look like below. - -```xml - - - yarn.resource-types - resource1, resource2 - - - - yarn.resource-types.resource1.units - G - - -``` - -Similarly, a new configuration file `node­-resources.xml` will also be loaded by Node Manager where the resource capabilities of a node can be specified. - -```xml - - - yarn.nodemanager.resource-type.resource1 - 5G - - - - yarn.nodemanager.resource-type.resource2 - 2m - - - -``` - -Node Manager will use these custom resource types and will register it's capability to Resource Manager. - -Configurations -------------- - -Please note that, `resource-types.xml` and `node­-resources.xml` file also need to be placed in conf directory if new resources are to be added to YARN. - -*In `resource-types.xml`* - -| Configuration Property | Value | Description | -|:---- |:---- |:---- | -| `yarn.resource-types` | resource1 | Custom resource | -| `yarn.resource-types.resource1.units` | G | Default unit for resource1 type | - -*In `node­-resources.xml`* - -| Configuration Property | Value | Description | -|:---- |:---- |:---- | -| `yarn.nodemanager.resource-type.resource1` | 5G | Resource capability for resource named 'resource1'. | -