YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
(Contributed by Miklos Szegedi via Daniel Templeton)
This commit is contained in:
Daniel Templeton
parent
98912950b6
commit
47b145b9b4
|
|
@ -134,7 +134,7 @@
|
|||
|
||||
<property>
|
||||
<description>
|
||||
This configures the HTTP endpoint for Yarn Daemons.The following
|
||||
This configures the HTTP endpoint for YARN Daemons.The following
|
||||
values are supported:
|
||||
- HTTP_ONLY : Service is provided only on http
|
||||
- HTTPS_ONLY : Service is provided only on https
|
||||
|
|
@ -1063,14 +1063,14 @@
|
|||
DeletionService will delete the application's localized file directory
|
||||
and log directory.
|
||||
|
||||
To diagnose Yarn application problems, set this property's value large
|
||||
To diagnose YARN application problems, set this property's value large
|
||||
enough (for example, to 600 = 10 minutes) to permit examination of these
|
||||
directories. After changing the property's value, you must restart the
|
||||
nodemanager in order for it to have an effect.
|
||||
|
||||
The roots of Yarn applications' work directories is configurable with
|
||||
The roots of YARN applications' work directories is configurable with
|
||||
the yarn.nodemanager.local-dirs property (see below), and the roots
|
||||
of the Yarn applications' log directories is configurable with the
|
||||
of the YARN applications' log directories is configurable with the
|
||||
yarn.nodemanager.log-dirs property (see also below).
|
||||
</description>
|
||||
<name>yarn.nodemanager.delete.debug-delay-sec</name>
|
||||
|
|
@ -1510,28 +1510,45 @@
|
|||
<property>
|
||||
<description>The cgroups hierarchy under which to place YARN proccesses (cannot contain commas).
|
||||
If yarn.nodemanager.linux-container-executor.cgroups.mount is false
|
||||
(that is, if cgroups have been pre-configured) and the Yarn user has write
|
||||
(that is, if cgroups have been pre-configured) and the YARN user has write
|
||||
access to the parent directory, then the directory will be created.
|
||||
If the directory already exists, the administrator has to give Yarn
|
||||
If the directory already exists, the administrator has to give YARN
|
||||
write permissions to it recursively.
|
||||
Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description>
|
||||
This property only applies when the LCE resources handler is set to
|
||||
CgroupsLCEResourcesHandler.</description>
|
||||
<name>yarn.nodemanager.linux-container-executor.cgroups.hierarchy</name>
|
||||
<value>/hadoop-yarn</value>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<description>Whether the LCE should attempt to mount cgroups if not found.
|
||||
Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler.</description>
|
||||
This property only applies when the LCE resources handler is set to
|
||||
CgroupsLCEResourcesHandler.
|
||||
</description>
|
||||
<name>yarn.nodemanager.linux-container-executor.cgroups.mount</name>
|
||||
<value>false</value>
|
||||
</property>
|
||||
|
||||
<property>
|
||||
<description>Where the LCE should attempt to mount cgroups if not found. Common locations
|
||||
include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux
|
||||
distribution in use. This path must exist before the NodeManager is launched.
|
||||
Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and
|
||||
yarn.nodemanager.linux-container-executor.cgroups.mount is true.</description>
|
||||
<description>This property sets the path from which YARN will read the
|
||||
CGroups configuration. YARN has built-in functionality to discover the
|
||||
system CGroup mount paths, so use this property only if YARN's automatic
|
||||
mount path discovery does not work.
|
||||
|
||||
The path specified by this property must exist before the NodeManager is
|
||||
launched.
|
||||
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to true,
|
||||
YARN will first try to mount the CGroups at the specified path before
|
||||
reading them.
|
||||
If yarn.nodemanager.linux-container-executor.cgroups.mount is set to
|
||||
false, YARN will read the CGroups at the specified path.
|
||||
If this property is empty, YARN tries to detect the CGroups location.
|
||||
|
||||
Please refer to NodeManagerCgroups.html in the documentation for further
|
||||
details.
|
||||
This property only applies when the LCE resources handler is set to
|
||||
CgroupsLCEResourcesHandler.
|
||||
</description>
|
||||
<name>yarn.nodemanager.linux-container-executor.cgroups.mount-path</name>
|
||||
</property>
|
||||
|
||||
|
|
|
|||
|
|
@ -23,6 +23,9 @@ package org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resourc
|
|||
import org.apache.hadoop.classification.InterfaceAudience;
|
||||
import org.apache.hadoop.classification.InterfaceStability;
|
||||
|
||||
import java.util.HashSet;
|
||||
import java.util.Set;
|
||||
|
||||
/**
|
||||
* Provides CGroups functionality. Implementations are expected to be
|
||||
* thread-safe
|
||||
|
|
@ -54,6 +57,18 @@ public interface CGroupsHandler {
|
|||
String getName() {
|
||||
return name;
|
||||
}
|
||||
|
||||
/**
|
||||
* Get the list of valid cgroup names.
|
||||
* @return The set of cgroup name strings
|
||||
*/
|
||||
public static Set<String> getValidCGroups() {
|
||||
HashSet<String> validCgroups = new HashSet<>();
|
||||
for (CGroupController controller : CGroupController.values()) {
|
||||
validCgroups.add(controller.getName());
|
||||
}
|
||||
return validCgroups;
|
||||
}
|
||||
}
|
||||
|
||||
String CGROUP_FILE_TASKS = "tasks";
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ class CGroupsHandlerImpl implements CGroupsHandler {
|
|||
* @param mtab mount file location
|
||||
* @throws ResourceHandlerException if initialization failed
|
||||
*/
|
||||
public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
|
||||
CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
|
||||
privilegedOperationExecutor, String mtab)
|
||||
throws ResourceHandlerException {
|
||||
this.cGroupPrefix = conf.get(YarnConfiguration.
|
||||
|
|
@ -115,7 +115,7 @@ class CGroupsHandlerImpl implements CGroupsHandler {
|
|||
* PrivilegedContainerOperations
|
||||
* @throws ResourceHandlerException if initialization failed
|
||||
*/
|
||||
public CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
|
||||
CGroupsHandlerImpl(Configuration conf, PrivilegedOperationExecutor
|
||||
privilegedOperationExecutor) throws ResourceHandlerException {
|
||||
this(conf, privilegedOperationExecutor, MTAB_FILE);
|
||||
}
|
||||
|
|
@ -142,11 +142,18 @@ class CGroupsHandlerImpl implements CGroupsHandler {
|
|||
// the same hierarchy will be mounted at each mount point with the same
|
||||
// subsystem set.
|
||||
|
||||
Map<String, Set<String>> newMtab;
|
||||
Map<String, Set<String>> newMtab = null;
|
||||
Map<CGroupController, String> cPaths;
|
||||
try {
|
||||
// parse mtab
|
||||
newMtab = parseMtab(mtabFile);
|
||||
if (this.cGroupMountPath != null && !this.enableCGroupMount) {
|
||||
newMtab = ResourceHandlerModule.
|
||||
parseConfiguredCGroupPath(this.cGroupMountPath);
|
||||
}
|
||||
|
||||
if (newMtab == null) {
|
||||
// parse mtab
|
||||
newMtab = parseMtab(mtabFile);
|
||||
}
|
||||
|
||||
// find cgroup controller paths
|
||||
cPaths = initializeControllerPathsFromMtab(newMtab);
|
||||
|
|
@ -203,10 +210,8 @@ class CGroupsHandlerImpl implements CGroupsHandler {
|
|||
throws IOException {
|
||||
Map<String, Set<String>> ret = new HashMap<>();
|
||||
BufferedReader in = null;
|
||||
HashSet<String> validCgroups = new HashSet<>();
|
||||
for (CGroupController controller : CGroupController.values()) {
|
||||
validCgroups.add(controller.getName());
|
||||
}
|
||||
Set<String> validCgroups =
|
||||
CGroupsHandler.CGroupController.getValidCGroups();
|
||||
|
||||
try {
|
||||
FileInputStream fis = new FileInputStream(new File(mtab));
|
||||
|
|
@ -487,7 +492,8 @@ class CGroupsHandlerImpl implements CGroupsHandler {
|
|||
try (BufferedReader inl =
|
||||
new BufferedReader(new InputStreamReader(new FileInputStream(cgf
|
||||
+ "/tasks"), "UTF-8"))) {
|
||||
if ((str = inl.readLine()) != null) {
|
||||
str = inl.readLine();
|
||||
if (str != null) {
|
||||
LOG.debug("First line in cgroup tasks file: " + cgf + " " + str);
|
||||
}
|
||||
} catch (IOException e) {
|
||||
|
|
|
|||
|
|
@ -31,6 +31,13 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileg
|
|||
import org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler;
|
||||
import org.apache.hadoop.yarn.server.nodemanager.util.DefaultLCEResourcesHandler;
|
||||
|
||||
import java.io.File;
|
||||
import java.io.IOException;
|
||||
import java.util.Set;
|
||||
import java.util.HashSet;
|
||||
import java.util.Map;
|
||||
import java.util.HashMap;
|
||||
import java.util.Arrays;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
|
||||
|
|
@ -113,8 +120,8 @@ public class ResourceHandlerModule {
|
|||
}
|
||||
|
||||
private static TrafficControlBandwidthHandlerImpl
|
||||
getTrafficControlBandwidthHandler(Configuration conf)
|
||||
throws ResourceHandlerException {
|
||||
getTrafficControlBandwidthHandler(Configuration conf)
|
||||
throws ResourceHandlerException {
|
||||
if (conf.getBoolean(YarnConfiguration.NM_NETWORK_RESOURCE_ENABLED,
|
||||
YarnConfiguration.DEFAULT_NM_NETWORK_RESOURCE_ENABLED)) {
|
||||
if (trafficControlBandwidthHandler == null) {
|
||||
|
|
@ -137,8 +144,8 @@ public class ResourceHandlerModule {
|
|||
}
|
||||
|
||||
public static OutboundBandwidthResourceHandler
|
||||
getOutboundBandwidthResourceHandler(Configuration conf)
|
||||
throws ResourceHandlerException {
|
||||
getOutboundBandwidthResourceHandler(Configuration conf)
|
||||
throws ResourceHandlerException {
|
||||
return getTrafficControlBandwidthHandler(conf);
|
||||
}
|
||||
|
||||
|
|
@ -176,7 +183,7 @@ public class ResourceHandlerModule {
|
|||
}
|
||||
|
||||
private static CGroupsMemoryResourceHandlerImpl
|
||||
getCgroupsMemoryResourceHandler(
|
||||
getCgroupsMemoryResourceHandler(
|
||||
Configuration conf) throws ResourceHandlerException {
|
||||
if (cGroupsMemoryResourceHandler == null) {
|
||||
synchronized (MemoryResourceHandler.class) {
|
||||
|
|
@ -229,4 +236,45 @@ public class ResourceHandlerModule {
|
|||
static void nullifyResourceHandlerChain() throws ResourceHandlerException {
|
||||
resourceHandlerChain = null;
|
||||
}
|
||||
|
||||
/**
|
||||
* If a cgroup mount directory is specified, it returns cgroup directories
|
||||
* with valid names.
|
||||
* The requirement is that each hierarchy has to be named with the comma
|
||||
* separated names of subsystems supported.
|
||||
* For example: /sys/fs/cgroup/cpu,cpuacct
|
||||
* @param cgroupMountPath Root cgroup mount path (/sys/fs/cgroup in the
|
||||
* example above)
|
||||
* @return A path to cgroup subsystem set mapping in the same format as
|
||||
* {@link CGroupsHandlerImpl#parseMtab(String)}
|
||||
* @throws IOException if the specified directory cannot be listed
|
||||
*/
|
||||
public static Map<String, Set<String>> parseConfiguredCGroupPath(
|
||||
String cgroupMountPath) throws IOException {
|
||||
File cgroupDir = new File(cgroupMountPath);
|
||||
File[] list = cgroupDir.listFiles();
|
||||
if (list == null) {
|
||||
throw new IOException("Empty cgroup mount directory specified: " +
|
||||
cgroupMountPath);
|
||||
}
|
||||
|
||||
Map<String, Set<String>> pathSubsystemMappings = new HashMap<>();
|
||||
Set<String> validCGroups =
|
||||
CGroupsHandler.CGroupController.getValidCGroups();
|
||||
for (File candidate: list) {
|
||||
Set<String> cgroupList =
|
||||
new HashSet<>(Arrays.asList(candidate.getName().split(",")));
|
||||
// Collect the valid subsystem names
|
||||
cgroupList.retainAll(validCGroups);
|
||||
if (!cgroupList.isEmpty()) {
|
||||
if (candidate.isDirectory() && candidate.canWrite()) {
|
||||
pathSubsystemMappings.put(candidate.getAbsolutePath(), cgroupList);
|
||||
} else {
|
||||
LOG.warn("The following cgroup is not a directory or it is not"
|
||||
+ " writable" + candidate.getAbsolutePath());
|
||||
}
|
||||
}
|
||||
}
|
||||
return pathSubsystemMappings;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -27,6 +27,7 @@ import java.io.InputStreamReader;
|
|||
import java.io.OutputStreamWriter;
|
||||
import java.io.PrintWriter;
|
||||
import java.io.Writer;
|
||||
import java.util.Arrays;
|
||||
import java.util.ArrayList;
|
||||
import java.util.Collections;
|
||||
import java.util.HashMap;
|
||||
|
|
@ -39,7 +40,6 @@ import java.util.regex.Pattern;
|
|||
|
||||
import com.google.common.annotations.VisibleForTesting;
|
||||
|
||||
import com.google.common.collect.Sets;
|
||||
import org.apache.commons.logging.Log;
|
||||
import org.apache.commons.logging.LogFactory;
|
||||
import org.apache.hadoop.conf.Configuration;
|
||||
|
|
@ -51,6 +51,8 @@ import org.apache.hadoop.yarn.conf.YarnConfiguration;
|
|||
import org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor;
|
||||
import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.privileged.PrivilegedOperation;
|
||||
import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsCpuResourceHandlerImpl;
|
||||
import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.CGroupsHandler;
|
||||
import org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.ResourceHandlerModule;
|
||||
import org.apache.hadoop.yarn.util.Clock;
|
||||
import org.apache.hadoop.yarn.util.ResourceCalculatorPlugin;
|
||||
import org.apache.hadoop.yarn.util.SystemClock;
|
||||
|
|
@ -87,11 +89,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
|
||||
private long deleteCgroupTimeout;
|
||||
private long deleteCgroupDelay;
|
||||
// package private for testing purposes
|
||||
@VisibleForTesting
|
||||
Clock clock;
|
||||
|
||||
private float yarnProcessors;
|
||||
int nodeVCores;
|
||||
private int nodeVCores;
|
||||
|
||||
public CgroupsLCEResourcesHandler() {
|
||||
this.controllerPaths = new HashMap<String, String>();
|
||||
|
|
@ -132,8 +134,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
this.strictResourceUsageMode =
|
||||
conf
|
||||
.getBoolean(
|
||||
YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
|
||||
YarnConfiguration.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
|
||||
YarnConfiguration
|
||||
.NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE,
|
||||
YarnConfiguration
|
||||
.DEFAULT_NM_LINUX_CONTAINER_CGROUPS_STRICT_RESOURCE_USAGE);
|
||||
|
||||
int len = cgroupPrefix.length();
|
||||
if (cgroupPrefix.charAt(len - 1) == '/') {
|
||||
|
|
@ -169,8 +173,10 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
if (systemProcessors != (int) yarnProcessors) {
|
||||
LOG.info("YARN containers restricted to " + yarnProcessors + " cores");
|
||||
int[] limits = getOverallLimits(yarnProcessors);
|
||||
updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US, String.valueOf(limits[0]));
|
||||
updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US, String.valueOf(limits[1]));
|
||||
updateCgroup(CONTROLLER_CPU, "", CPU_PERIOD_US,
|
||||
String.valueOf(limits[0]));
|
||||
updateCgroup(CONTROLLER_CPU, "", CPU_QUOTA_US,
|
||||
String.valueOf(limits[1]));
|
||||
} else if (CGroupsCpuResourceHandlerImpl.cpuLimitsExist(
|
||||
pathForCgroup(CONTROLLER_CPU, ""))) {
|
||||
LOG.info("Removing CPU constraints for YARN containers.");
|
||||
|
|
@ -178,8 +184,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
}
|
||||
}
|
||||
|
||||
int[] getOverallLimits(float yarnProcessors) {
|
||||
return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessors);
|
||||
int[] getOverallLimits(float yarnProcessorsArg) {
|
||||
return CGroupsCpuResourceHandlerImpl.getOverallLimits(yarnProcessorsArg);
|
||||
}
|
||||
|
||||
|
||||
|
|
@ -204,7 +210,7 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
LOG.debug("createCgroup: " + path);
|
||||
}
|
||||
|
||||
if (! new File(path).mkdir()) {
|
||||
if (!new File(path).mkdir()) {
|
||||
throw new IOException("Failed to create cgroup at " + path);
|
||||
}
|
||||
}
|
||||
|
|
@ -251,7 +257,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
try (BufferedReader inl =
|
||||
new BufferedReader(new InputStreamReader(new FileInputStream(cgf
|
||||
+ "/tasks"), "UTF-8"))) {
|
||||
if ((str = inl.readLine()) != null) {
|
||||
str = inl.readLine();
|
||||
if (str != null) {
|
||||
LOG.debug("First line in cgroup tasks file: " + cgf + " " + str);
|
||||
}
|
||||
} catch (IOException e) {
|
||||
|
|
@ -337,9 +344,9 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
(containerVCores * yarnProcessors) / (float) nodeVCores;
|
||||
int[] limits = getOverallLimits(containerCPU);
|
||||
updateCgroup(CONTROLLER_CPU, containerName, CPU_PERIOD_US,
|
||||
String.valueOf(limits[0]));
|
||||
String.valueOf(limits[0]));
|
||||
updateCgroup(CONTROLLER_CPU, containerName, CPU_QUOTA_US,
|
||||
String.valueOf(limits[1]));
|
||||
String.valueOf(limits[1]));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -400,6 +407,8 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
private Map<String, Set<String>> parseMtab() throws IOException {
|
||||
Map<String, Set<String>> ret = new HashMap<String, Set<String>>();
|
||||
BufferedReader in = null;
|
||||
Set<String> validCgroups =
|
||||
CGroupsHandler.CGroupController.getValidCGroups();
|
||||
|
||||
try {
|
||||
FileInputStream fis = new FileInputStream(new File(getMtabFileName()));
|
||||
|
|
@ -415,8 +424,11 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
String options = m.group(3);
|
||||
|
||||
if (type.equals(CGROUPS_FSTYPE)) {
|
||||
HashSet<String> value = Sets.newHashSet(options.split(","));
|
||||
ret.put(path, value);
|
||||
Set<String> cgroupList =
|
||||
new HashSet<>(Arrays.asList(options.split(",")));
|
||||
// Collect the valid subsystem names
|
||||
cgroupList.retainAll(validCgroups);
|
||||
ret.put(path, cgroupList);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -448,7 +460,16 @@ public class CgroupsLCEResourcesHandler implements LCEResourcesHandler {
|
|||
|
||||
private void initializeControllerPaths() throws IOException {
|
||||
String controllerPath;
|
||||
Map<String, Set<String>> parsedMtab = parseMtab();
|
||||
Map<String, Set<String>> parsedMtab = null;
|
||||
|
||||
if (this.cgroupMountPath != null && !this.cgroupMount) {
|
||||
parsedMtab = ResourceHandlerModule.
|
||||
parseConfiguredCGroupPath(this.cgroupMountPath);
|
||||
}
|
||||
|
||||
if (parsedMtab == null) {
|
||||
parsedMtab = parseMtab();
|
||||
}
|
||||
|
||||
// CPU
|
||||
|
||||
|
|
|
|||
|
|
@ -573,4 +573,29 @@ public class TestCGroupsHandlerImpl {
|
|||
new File(new File(newMountPoint, "cpu"), this.hierarchy);
|
||||
assertTrue("Yarn cgroup should exist", hierarchyFile.exists());
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@Test
|
||||
public void testManualCgroupSetting() throws ResourceHandlerException {
|
||||
YarnConfiguration conf = new YarnConfiguration();
|
||||
conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH, tmpPath);
|
||||
conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_HIERARCHY,
|
||||
"/hadoop-yarn");
|
||||
File cpu = new File(new File(tmpPath, "cpuacct,cpu"), "/hadoop-yarn");
|
||||
|
||||
try {
|
||||
Assert.assertTrue("temp dir should be created", cpu.mkdirs());
|
||||
|
||||
CGroupsHandlerImpl cGroupsHandler = new CGroupsHandlerImpl(conf, null);
|
||||
cGroupsHandler.initializeCGroupController(
|
||||
CGroupsHandler.CGroupController.CPU);
|
||||
|
||||
Assert.assertEquals("CPU CGRoup path was not set", cpu.getAbsolutePath(),
|
||||
new File(cGroupsHandler.getPathForCGroup(
|
||||
CGroupsHandler.CGroupController.CPU, "")).getAbsolutePath());
|
||||
|
||||
} finally {
|
||||
FileUtils.deleteQuietly(cpu);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -41,6 +41,8 @@ import java.util.Scanner;
|
|||
import java.util.Set;
|
||||
import java.util.concurrent.CountDownLatch;
|
||||
|
||||
import static org.mockito.Mockito.when;
|
||||
|
||||
@Deprecated
|
||||
public class TestCgroupsLCEResourcesHandler {
|
||||
private static File cgroupDir = null;
|
||||
|
|
@ -388,4 +390,33 @@ public class TestCgroupsLCEResourcesHandler {
|
|||
FileUtils.deleteQuietly(memory);
|
||||
}
|
||||
}
|
||||
|
||||
@Test
|
||||
public void testManualCgroupSetting() throws IOException {
|
||||
CgroupsLCEResourcesHandler handler = new CgroupsLCEResourcesHandler();
|
||||
YarnConfiguration conf = new YarnConfiguration();
|
||||
conf.set(YarnConfiguration.NM_LINUX_CONTAINER_CGROUPS_MOUNT_PATH,
|
||||
cgroupDir.getAbsolutePath());
|
||||
handler.setConf(conf);
|
||||
File cpu = new File(new File(cgroupDir, "cpuacct,cpu"), "/hadoop-yarn");
|
||||
|
||||
try {
|
||||
Assert.assertTrue("temp dir should be created", cpu.mkdirs());
|
||||
|
||||
final int numProcessors = 4;
|
||||
ResourceCalculatorPlugin plugin =
|
||||
Mockito.mock(ResourceCalculatorPlugin.class);
|
||||
Mockito.doReturn(numProcessors).when(plugin).getNumProcessors();
|
||||
Mockito.doReturn(numProcessors).when(plugin).getNumCores();
|
||||
when(plugin.getNumProcessors()).thenReturn(8);
|
||||
handler.init(null, plugin);
|
||||
|
||||
Assert.assertEquals("CPU CGRoup path was not set", cpu.getParent(),
|
||||
handler.getControllerPaths().get("cpu"));
|
||||
|
||||
} finally {
|
||||
FileUtils.deleteQuietly(cpu);
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@
|
|||
-->
|
||||
|
||||
|
||||
Graceful Decommission of Yarn Nodes
|
||||
Graceful Decommission of YARN Nodes
|
||||
===============
|
||||
|
||||
* [Overview](#overview)
|
||||
|
|
@ -29,19 +29,19 @@ Graceful Decommission of Yarn Nodes
|
|||
Overview
|
||||
--------
|
||||
|
||||
Yarn is scalable very easily: any new NodeManager could join to the configured ResourceManager and start to execute jobs. But to achieve full elasticity we need a decommissioning process which helps to remove existing nodes and down-scale the cluster.
|
||||
YARN is scalable very easily: any new NodeManager could join to the configured ResourceManager and start to execute jobs. But to achieve full elasticity we need a decommissioning process which helps to remove existing nodes and down-scale the cluster.
|
||||
|
||||
Yarn Nodes could be decommissioned NORMAL or GRACEFUL.
|
||||
YARN Nodes could be decommissioned NORMAL or GRACEFUL.
|
||||
|
||||
Normal Decommission of Yarn Nodes means an immediate shutdown.
|
||||
Normal Decommission of YARN Nodes means an immediate shutdown.
|
||||
|
||||
Graceful Decommission of Yarn Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED.
|
||||
Graceful Decommission of YARN Nodes is the mechanism to decommission NMs while minimize the impact to running applications. Once a node is in DECOMMISSIONING state, RM won't schedule new containers on it and will wait for running containers and applications to complete (or until decommissioning timeout exceeded) before transition the node into DECOMMISSIONED.
|
||||
|
||||
## Quick start
|
||||
|
||||
To do a normal decommissioning:
|
||||
|
||||
1. Start a Yarn cluster (with NodeManageres and ResourceManager)
|
||||
1. Start a YARN cluster (with NodeManageres and ResourceManager)
|
||||
2. Start a yarn job (for example with `yarn jar...` )
|
||||
3. Add `yarn.resourcemanager.nodes.exclude-path` property to your `yarn-site.xml` (Note: you don't need to restart the ResourceManager)
|
||||
4. Create a text file (the location is defined in the previous step) with one line which contains the name of a selected NodeManager
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ Using CGroups with YARN
|
|||
|
||||
<!-- MACRO{toc|fromDepth=0|toDepth=3} -->
|
||||
|
||||
CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to be limited in their resource usage. A good example of this is CPU usage. Without CGroups, it becomes hard to limit container CPU usage. Currently, CGroups is only used for limiting CPU usage.
|
||||
CGroups is a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour. CGroups is a Linux kernel feature and was merged into kernel version 2.6.24. From a YARN perspective, this allows containers to be limited in their resource usage. A good example of this is CPU usage. Without CGroups, it becomes hard to limit container CPU usage.
|
||||
|
||||
CGroups Configuration
|
||||
---------------------
|
||||
|
|
@ -30,9 +30,9 @@ The following settings are related to setting up CGroups. These need to be set i
|
|||
|:---- |:---- |
|
||||
| `yarn.nodemanager.container-executor.class` | This should be set to "org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor". CGroups is a Linux kernel feature and is exposed via the LinuxContainerExecutor. |
|
||||
| `yarn.nodemanager.linux-container-executor.resources-handler.class` | This should be set to "org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler". Using the LinuxContainerExecutor doesn't force you to use CGroups. If you wish to use CGroups, the resource-handler-class must be set to CGroupsLCEResourceHandler. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured) and the Yarn user has write access to the parent directory, then the directory will be created. If the directory already exists, the administrator has to give Yarn write permissions to it recursively. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.hierarchy` | The cgroups hierarchy under which to place YARN proccesses(cannot contain commas). If yarn.nodemanager.linux-container-executor.cgroups.mount is false (that is, if cgroups have been pre-configured) and the YARN user has write access to the parent directory, then the directory will be created. If the directory already exists, the administrator has to give YARN write permissions to it recursively. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.mount` | Whether the LCE should attempt to mount cgroups if not found - can be true or false. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Where the LCE should attempt to mount cgroups if not found. Common locations include /sys/fs/cgroup and /cgroup; the default location can vary depending on the Linux distribution in use. This path must exist before the NodeManager is launched. Only used when the LCE resources handler is set to the CgroupsLCEResourcesHandler, and yarn.nodemanager.linux-container-executor.cgroups.mount is true. A point to note here is that the container-executor binary will try to mount the path specified + "/" + the subsystem. In our case, since we are trying to limit CPU the binary tries to mount the path specified + "/cpu" and that's the path it expects to exist. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.mount-path` | Optional. Where CGroups are located. LCE will try to mount them here, if `yarn.nodemanager.linux-container-executor.cgroups.mount` is true. LCE will try to use CGroups from this location, if `yarn.nodemanager.linux-container-executor.cgroups.mount` is false. If specified, this path and its subdirectories (CGroup hierarchies) must exist and they should be readable and writable by YARN before the NodeManager is launched. See CGroups mount options below for details. |
|
||||
| `yarn.nodemanager.linux-container-executor.group` | The Unix group of the NodeManager. It should match the setting in "container-executor.cfg". This configuration is required for validating the secure access of the container-executor binary. |
|
||||
|
||||
The following settings are related to limiting resource usage of YARN containers:
|
||||
|
|
@ -42,6 +42,17 @@ The following settings are related to limiting resource usage of YARN containers
|
|||
| `yarn.nodemanager.resource.percentage-physical-cpu-limit` | This setting lets you limit the cpu usage of all YARN containers. It sets a hard upper limit on the cumulative CPU usage of the containers. For example, if set to 60, the combined CPU usage of all YARN containers will not exceed 60%. |
|
||||
| `yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage` | CGroups allows cpu usage limits to be hard or soft. When this setting is true, containers cannot use more CPU usage than allocated even if spare CPU is available. This ensures that containers can only use CPU that they were allocated. When set to false, containers can use spare CPU if available. It should be noted that irrespective of whether set to true or false, at no time can the combined CPU usage of all containers exceed the value specified in "yarn.nodemanager.resource.percentage-physical-cpu-limit". |
|
||||
|
||||
CGroups mount options
|
||||
---------------------
|
||||
|
||||
YARN uses CGroups through a directory structure mounted into the file system by the kernel. There are three options to attach to CGroups.
|
||||
|
||||
| Option | Description |
|
||||
|:---- |:---- |
|
||||
| Discover CGroups mounted already | This should be used on newer systems like RHEL7 or Ubuntu16 or if the administrator mounts CGroups before YARN starts. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to false and leave other settings set to their defaults. YARN will locate the mount points in `/proc/mounts`. Common locations include `/sys/fs/cgroup` and `/cgroup`. The default location can vary depending on the Linux distribution in use.|
|
||||
| CGroups mounted by YARN | If the system does not have CGroups mounted or it is mounted to an inaccessible location then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path` to an empty directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to true. A point to note here is that the container-executor binary will try to create and mount each subsystem as a subdirectory under this path. If `cpu` is already mounted somewhere with `cpuacct`, then the directory `cpu,cpuacct` will be created for the hierarchy.|
|
||||
| CGroups mounted already or linked but not in `/proc/mounts` | If cgroups is accessible through lxcfs or simulated by another filesystem, then point `yarn.nodemanager.linux-container-executor.cgroups.mount-path` to your CGroups root directory. Set `yarn.nodemanager.linux-container-executor.cgroups.mount` to false. YARN tries to use this path first, before any CGroup mount point discovery. The path should have a subdirectory for each CGroup hierarchy named by the comma separated CGroup subsystems supported like `<path>/cpu,cpuacct`. Valid subsystem names are `cpu, cpuacct, cpuset, memory, net_cls, blkio, freezer, devices`.|
|
||||
|
||||
CGroups and security
|
||||
--------------------
|
||||
|
||||
|
|
|
|||
|
|
@ -56,7 +56,7 @@ Following are the important interfaces:
|
|||
|
||||
* Under very rare circumstances, programmer may want to directly use the 3 protocols to implement an application. However, note that *such behaviors are no longer encouraged for general use cases*.
|
||||
|
||||
Writing a Simple Yarn Application
|
||||
Writing a Simple YARN Application
|
||||
---------------------------------
|
||||
|
||||
### Writing a simple Client
|
||||
|
|
@ -574,4 +574,4 @@ Useful Links
|
|||
Sample Code
|
||||
-----------
|
||||
|
||||
Yarn distributed shell: in `hadoop-yarn-applications-distributedshell` project after you set up your development environment.
|
||||
YARN distributed shell: in `hadoop-yarn-applications-distributedshell` project after you set up your development environment.
|
||||
|
|
|
|||
|
|
@ -84,7 +84,7 @@ container ID.
|
|||
|
||||
## The binding problem
|
||||
Hadoop YARN allows applications to run on the Hadoop cluster. Some of these are
|
||||
batch jobs or queries that can managed via Yarn’s existing API using its
|
||||
batch jobs or queries that can managed via YARN’s existing API using its
|
||||
application ID. In addition YARN can deploy ong-lived services instances such a
|
||||
pool of Apache Tomcat web servers or an Apache HBase cluster. YARN will deploy
|
||||
them across the cluster depending on the individual each component requirements
|
||||
|
|
@ -121,7 +121,7 @@ services accessible from within the Hadoop cluster
|
|||
/services/yarn
|
||||
/services/oozie
|
||||
|
||||
Yarn-deployed services belonging to individual users.
|
||||
YARN-deployed services belonging to individual users.
|
||||
|
||||
/users/joe/org-apache-hbase/demo1
|
||||
/users/joe/org-apache-hbase/demo1/components/regionserver1
|
||||
|
|
@ -148,7 +148,7 @@ their application master, to which they heartbeat regularly.
|
|||
|
||||
## Unsupported Registration use cases:
|
||||
|
||||
1. A short-lived Yarn application is registered automatically in the registry,
|
||||
1. A short-lived YARN application is registered automatically in the registry,
|
||||
including all its containers. and unregistered when the job terminates.
|
||||
Short-lived applications with many containers will place excessive load on a
|
||||
registry. All YARN applications will be given the option of registering, but it
|
||||
|
|
@ -259,7 +259,7 @@ service since it supports many of the properties, We pick a part of the ZK
|
|||
namespace to be the root of the service registry ( default: `yarnRegistry`).
|
||||
|
||||
On top this base implementation we build our registry service API and the
|
||||
naming conventions that Yarn will use for its services. The registry will be
|
||||
naming conventions that YARN will use for its services. The registry will be
|
||||
accessed by the registry API, not directly via ZK - ZK is just an
|
||||
implementation choice (although unlikely to change in the future).
|
||||
|
||||
|
|
@ -297,7 +297,7 @@ them.
|
|||
6. Core services will be registered using the following convention:
|
||||
`/services/{servicename}` e.g. `/services/hdfs`.
|
||||
|
||||
7. Yarn services SHOULD be registered using the following convention:
|
||||
7. YARN services SHOULD be registered using the following convention:
|
||||
|
||||
/users/{username}/{serviceclass}/{instancename}
|
||||
|
||||
|
|
@ -823,8 +823,8 @@ The `RegistryPathStatus` class summarizes the contents of a node in the registry
|
|||
## Security
|
||||
|
||||
The registry will allow a service instance can only be registered under the
|
||||
path where it has permissions. Yarn will create directories with appropriate
|
||||
permissions for users where Yarn deployed services can be registered by a user.
|
||||
path where it has permissions. YARN will create directories with appropriate
|
||||
permissions for users where YARN deployed services can be registered by a user.
|
||||
of the user account of the service instance. The admin will also create
|
||||
directories (such as `/services`) with appropriate permissions (where core Hadoop
|
||||
services can register themselves.
|
||||
|
|
|
|||
Loading…
Reference in New Issue