1964 lines
133 KiB
HTML
1964 lines
133 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<!--
|
|
| Generated by Apache Maven Doxia at 2023-02-14
|
|
| Rendered using Apache Maven Stylus Skin 1.5
|
|
-->
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<title>Apache Hadoop 3.4.0-SNAPSHOT – Hadoop: Capacity Scheduler</title>
|
|
<style type="text/css" media="all">
|
|
@import url("./css/maven-base.css");
|
|
@import url("./css/maven-theme.css");
|
|
@import url("./css/site.css");
|
|
</style>
|
|
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
|
|
<meta name="Date-Revision-yyyymmdd" content="20230214" />
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
</head>
|
|
<body class="composite">
|
|
<div id="banner">
|
|
<a href="http://hadoop.apache.org/" id="bannerLeft">
|
|
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
|
|
</a>
|
|
<a href="http://www.apache.org/" id="bannerRight">
|
|
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
|
|
</a>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="breadcrumbs">
|
|
|
|
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
|
|
|
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
|
|
|
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
|
|
|
|
| Last Published: 2023-02-14
|
|
| Version: 3.4.0-SNAPSHOT
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="leftColumn">
|
|
<div id="navcolumn">
|
|
|
|
<h5>General</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Common</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/index.html">Hadoop KMS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AsyncProfilerServlet.html">Async Profiler</a>
|
|
</li>
|
|
</ul>
|
|
<h5>HDFS</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN Service</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Hadoop Compatible File Systems</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure/index.html">Azure Blob Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-huaweicloud/cloud-storage/index.html">Huaweicloud OBS</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Auth</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Examples.html">Examples</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Tools</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-distcp/DistCp.html">DistCp</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-federation-balance/HDFSFederationBalance.html">HDFS Federation Balance</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-gridmix/GridMix.html">GridMix</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-rumen/Rumen.html">Rumen</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Reference</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../api/index.html">Java API docs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Configuration</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/kms-default.html">kms-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
|
|
</li>
|
|
</ul>
|
|
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
|
|
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
|
|
</a>
|
|
|
|
</div>
|
|
</div>
|
|
<div id="bodyColumn">
|
|
<div id="contentBox">
|
|
<!---
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License. See accompanying LICENSE file.
|
|
-->
|
|
<h1>Hadoop: Capacity Scheduler</h1>
|
|
<ul>
|
|
<li><a href="#Purpose">Purpose</a></li>
|
|
<li><a href="#Overview">Overview</a></li>
|
|
<li><a href="#Features">Features</a></li>
|
|
<li><a href="#Configuration">Configuration</a>
|
|
<ul>
|
|
<li><a href="#Setting_up_ResourceManager_to_use_CapacityScheduler">Setting up ResourceManager to use CapacityScheduler</a></li>
|
|
<li><a href="#Setting_up_queues">Setting up queues</a></li>
|
|
<li><a href="#Queue_Properties">Queue Properties</a></li>
|
|
<li><a href="#JSON-based_queue_mapping_configuration">JSON-based queue mapping configuration</a>
|
|
<ul>
|
|
<li><a href="#Syntax">Syntax</a></li>
|
|
<li><a href="#How_to_enable_JSON-based_queue_mapping">How to enable JSON-based queue mapping</a></li>
|
|
<li><a href="#Differences_between_legacy_and_flexible_queue_auto-creation_modes">Differences between legacy and flexible queue auto-creation modes</a></li>
|
|
<li><a href="#Rules">Rules</a></li>
|
|
<li><a href="#Policies">Policies</a></li>
|
|
<li><a href="#Variables">Variables</a></li>
|
|
<li><a href="#Converting_the_old_mapping_rule_format_to_the_new_one">Converting the old mapping rule format to the new one</a></li>
|
|
<li><a href="#Example">Example</a></li></ul></li>
|
|
<li><a href="#Setup_for_application_priority.">Setup for application priority.</a></li>
|
|
<li><a href="#Capacity_Scheduler_container_preemption">Capacity Scheduler container preemption</a></li>
|
|
<li><a href="#Reservation_Properties">Reservation Properties</a></li>
|
|
<li><a href="#Configuring_ReservationSystem_with_CapacityScheduler">Configuring ReservationSystem with CapacityScheduler</a></li>
|
|
<li><a href="#Dynamic_Auto-Creation_and_Management_of_Leaf_Queues">Dynamic Auto-Creation and Management of Leaf Queues</a></li>
|
|
<li><a href="#Other_Properties">Other Properties</a></li>
|
|
<li><a href="#Reviewing_the_configuration_of_the_CapacityScheduler">Reviewing the configuration of the CapacityScheduler</a></li></ul></li>
|
|
<li><a href="#Changing_Queue_Configuration">Changing Queue Configuration</a>
|
|
<ul>
|
|
<li><a href="#Changing_queue_configuration_via_file">Changing queue configuration via file</a>
|
|
<ul>
|
|
<li><a href="#Deleting_queue_via_file">Deleting queue via file</a></li></ul></li>
|
|
<li><a href="#Enabling_periodic_configuration_refresh">Enabling periodic configuration refresh</a></li>
|
|
<li><a href="#Changing_queue_configuration_via_API">Changing queue configuration via API</a></li></ul></li>
|
|
<li><a href="#Updating_a_Container_.28Experimental_-_API_may_change_in_the_future.29">Updating a Container (Experimental - API may change in the future)</a></li>
|
|
<li><a href="#Activities">Activities</a>
|
|
<ul>
|
|
<li><a href="#Scheduler_Activities">Scheduler Activities</a></li>
|
|
<li><a href="#Application_Activities">Application Activities</a></li>
|
|
<li><a href="#Configuration">Configuration</a></li>
|
|
<li><a href="#Web_UI">Web UI</a></li></ul></li></ul>
|
|
<section>
|
|
<h2><a name="Purpose"></a>Purpose</h2>
|
|
<p>This document describes the <code>CapacityScheduler</code>, a pluggable scheduler for Hadoop which allows for multiple-tenants to securely share a large cluster such that their applications are allocated resources in a timely manner under constraints of allocated capacities.</p></section><section>
|
|
<h2><a name="Overview"></a>Overview</h2>
|
|
<p>The <code>CapacityScheduler</code> is designed to run Hadoop applications as a shared, multi-tenant cluster in an operator-friendly manner while maximizing the throughput and the utilization of the cluster.</p>
|
|
<p>Traditionally each organization has it own private set of compute resources that have sufficient capacity to meet the organization’s SLA under peak or near-peak conditions. This generally leads to poor average utilization and overhead of managing multiple independent clusters, one per each organization. Sharing clusters between organizations is a cost-effective manner of running large Hadoop installations since this allows them to reap benefits of economies of scale without creating private clusters. However, organizations are concerned about sharing a cluster because they are worried about others using the resources that are critical for their SLAs.</p>
|
|
<p>The <code>CapacityScheduler</code> is designed to allow sharing a large cluster while giving each organization capacity guarantees. The central idea is that the available resources in the Hadoop cluster are shared among multiple organizations who collectively fund the cluster based on their computing needs. There is an added benefit that an organization can access any excess capacity not being used by others. This provides elasticity for the organizations in a cost-effective manner.</p>
|
|
<p>Sharing clusters across organizations necessitates strong support for multi-tenancy since each organization must be guaranteed capacity and safe-guards to ensure the shared cluster is impervious to single rogue application or user or sets thereof. The <code>CapacityScheduler</code> provides a stringent set of limits to ensure that a single application or user or queue cannot consume disproportionate amount of resources in the cluster. Also, the <code>CapacityScheduler</code> provides limits on initialized and pending applications from a single user and queue to ensure fairness and stability of the cluster.</p>
|
|
<p>The primary abstraction provided by the <code>CapacityScheduler</code> is the concept of <i>queues</i>. These queues are typically setup by administrators to reflect the economics of the shared cluster.</p>
|
|
<p>To provide further control and predictability on sharing of resources, the <code>CapacityScheduler</code> supports <i>hierarchical queues</i> to ensure resources are shared among the sub-queues of an organization before other queues are allowed to use free resources, thereby providing <i>affinity</i> for sharing free resources among applications of a given organization.</p></section><section>
|
|
<h2><a name="Features"></a>Features</h2>
|
|
<p>The <code>CapacityScheduler</code> supports the following features:</p>
|
|
<ul>
|
|
|
|
<li>
|
|
|
|
<p><b>Hierarchical Queues</b> - Hierarchy of queues is supported to ensure resources are shared among the sub-queues of an organization before other queues are allowed to use free resources, thereby providing more control and predictability.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Capacity Guarantees</b> - Queues are allocated a fraction of the capacity of the grid in the sense that a certain capacity of resources will be at their disposal. All applications submitted to a queue will have access to the capacity allocated to the queue. Administrators can configure soft limits and optional hard limits on the capacity allocated to each queue.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Security</b> - Each queue has strict ACLs which controls which users can submit applications to individual queues. Also, there are safe-guards to ensure that users cannot view and/or modify applications from other users. Also, per-queue and system administrator roles are supported.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Elasticity</b> - Free resources can be allocated to any queue beyond its capacity. When there is demand for these resources from queues running below capacity at a future point in time, as tasks scheduled on these resources complete, they will be assigned to applications on queues running below the capacity (preemption is also supported). This ensures that resources are available in a predictable and elastic manner to queues, thus preventing artificial silos of resources in the cluster which helps utilization.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Multi-tenancy</b> - Comprehensive set of limits are provided to prevent a single application, user and queue from monopolizing resources of the queue or the cluster as a whole to ensure that the cluster isn’t overwhelmed.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Operability</b></p>
|
|
<ul>
|
|
|
|
<li>
|
|
|
|
<p>Runtime Configuration - The queue definitions and properties such as capacity, ACLs can be changed, at runtime, by administrators in a secure manner to minimize disruption to users. Also, a console is provided for users and administrators to view current allocation of resources to various queues in the system. Administrators can <i>add additional queues</i> at runtime, but queues cannot be <i>deleted</i> at runtime unless the queue is STOPPED and has no pending/running apps.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>Drain applications - Administrators can <i>stop</i> queues at runtime to ensure that while existing applications run to completion, no new applications can be submitted. If a queue is in <code>STOPPED</code> state, new applications cannot be submitted to <i>itself</i> or <i>any of its child queues</i>. Existing applications continue to completion, thus the queue can be <i>drained</i> gracefully. Administrators can also <i>start</i> the stopped queues.</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Resource-based Scheduling</b> - Support for resource-intensive applications, where-in a application can optionally specify higher resource-requirements than the default, thereby accommodating applications with differing resource requirements. Currently, <i>memory</i> is the resource requirement supported.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Queue Mapping Interface based on Default or User Defined Placement Rules</b> - This feature allows users to map a job to a specific queue based on some default placement rule. For instance based on user & group, or application name. User can also define their own placement rule.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Priority Scheduling</b> - This feature allows applications to be submitted and scheduled with different priorities. Higher integer value indicates higher priority for an application. Currently Application priority is supported only for FIFO ordering policy.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Absolute Resource Configuration</b> - Administrators could specify absolute resources to a queue instead of providing percentage based values. This provides better control for admins to configure required amount of resources for a given queue.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Dynamic Auto-Creation and Management of Leaf Queues</b> - This feature supports auto-creation of <b>leaf queues</b> in conjunction with <b>queue-mapping</b> which currently supports <b>user-group</b> based queue mappings for application placement to a queue. The scheduler also supports capacity management for these queues based on a policy configured on the parent queue.</p>
|
|
</li>
|
|
</ul></section><section>
|
|
<h2><a name="Configuration"></a>Configuration</h2><section>
|
|
<h3><a name="Setting_up_ResourceManager_to_use_CapacityScheduler"></a>Setting up <code>ResourceManager</code> to use <code>CapacityScheduler</code></h3>
|
|
<p>To configure the <code>ResourceManager</code> to use the <code>CapacityScheduler</code>, set the following property in the <b>conf/yarn-site.xml</b>:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Value </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.scheduler.class</code> </td>
|
|
<td align="left"> <code>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</code> </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Setting_up_queues"></a>Setting up queues</h3>
|
|
<p><code>etc/hadoop/capacity-scheduler.xml</code> is the configuration file for the <code>CapacityScheduler</code>.</p>
|
|
<p>The <code>CapacityScheduler</code> has a predefined queue called <i>root</i>. All queues in the system are children of the root queue.</p>
|
|
<p>Further queues can be setup by configuring <code>yarn.scheduler.capacity.root.queues</code> with a list of comma-separated child queues.</p>
|
|
<p>The configuration for <code>CapacityScheduler</code> uses a concept called <i>queue path</i> to configure the hierarchy of queues. The <i>queue path</i> is the full path of the queue’s hierarchy, starting at <i>root</i>, with . (dot) as the delimiter.</p>
|
|
<p>A given queue’s children can be defined with the configuration knob: <code>yarn.scheduler.capacity.<queue-path>.queues</code>. Children do not inherit properties directly from the parent unless otherwise noted.</p>
|
|
<p>Here is an example with three top-level child-queues <code>a</code>, <code>b</code> and <code>c</code> and some sub-queues for <code>a</code> and <code>b</code>:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre><property>
|
|
<name>yarn.scheduler.capacity.root.queues</name>
|
|
<value>a,b,c</value>
|
|
<description>The queues at the this level (root is the root queue).
|
|
</description>
|
|
</property>
|
|
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.a.queues</name>
|
|
<value>a1,a2</value>
|
|
<description>The queues at the this level (root is the root queue).
|
|
</description>
|
|
</property>
|
|
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.b.queues</name>
|
|
<value>b1,b2,b3</value>
|
|
<description>The queues at the this level (root is the root queue).
|
|
</description>
|
|
</property>
|
|
</pre></div></div>
|
|
</section><section>
|
|
<h3><a name="Queue_Properties"></a>Queue Properties</h3>
|
|
<ul>
|
|
|
|
<li>Resource Allocation</li>
|
|
</ul>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.capacity</code> </td>
|
|
<td align="left"> Queue <i>capacity</i> in percentage (%) as a float (e.g. 12.5), weight as a float with the postfix <i>w</i> (e.g. 2.0w) or as absolute resource queue minimum capacity. When using percentage values the sum of capacities for all queues, at each level, must be equal to 100. If absolute resource is configured, sum of absolute resources of child queues could be less than its parent absolute resource capacity. Applications in the queue may consume more resources than the queue’s capacity if there are free resources, providing elasticity. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.maximum-capacity</code> </td>
|
|
<td align="left"> Maximum queue capacity in percentage (%) as a float (when the <i>capacity</i> property is defined with either percentages or weights) or as absolute resource queue maximum capacity. This limits the <i>elasticity</i> for applications in the queue. 1) Value is between 0 and 100. 2) Admin needs to make sure absolute maximum capacity >= absolute capacity for each queue. Also, setting this value to -1 sets maximum capacity to 100%. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.minimum-user-limit-percent</code> / <code>yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent</code> </td>
|
|
<td align="left"> Each queue enforces a limit on the percentage of resources allocated to a user at any given time, if there is demand for resources. The user limit can vary between a minimum and maximum value. The former (the minimum value) is set to this property value and the latter (the maximum value) depends on the number of users who have submitted applications. For e.g., suppose the value of this property is 25. If two users have submitted applications to a queue, no single user can use more than 50% of the queue resources. If a third user submits an application, no single user can use more than 33% of the queue resources. With 4 or more users, no user can use more than 25% of the queues resources. A value of 100 implies no user limits are imposed. The default is 100. Value is specified as an integer. This can be set for all queues with <code>yarn.scheduler.capacity.minimum-user-limit-percent</code> and can also be overridden on a per queue basis by setting <code>yarn.scheduler.capacity.<queue-path>.minimum-user-limit-percent</code>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.user-limit-factor</code> / <code>yarn.scheduler.capacity.<queue-path>.user-limit-factor</code> </td>
|
|
<td align="left"> User limit factor provides a way to control the max amount of resources that a single user can consume. It is the multiple of the queue’s capacity. By default this is set to 1 which ensures that a single user can never take more than the queue’s configured capacity irrespective of how idle the cluster is. Increasing it means a single user can use more than the minimum capacity of the cluster, while decreasing it results in lower maximum resources. Setting this to -1 will disable the feature. Value is specified as a float. Note: using the flexible auto queue creation (yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2) with weights will automatically set this property to -1, as the dynamic queues will be created with the hardcoded weight of 1 and in idle cluster scenarios they should be able to use more resources than calculated. This can be set for all queues with <code>yarn.scheduler.capacity.user-limit-factor</code> and can also be overridden on a per queue basis by setting <code>yarn.scheduler.capacity.<queue-path>.user-limit-factor</code>. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.maximum-allocation-mb</code> </td>
|
|
<td align="left"> The per queue maximum limit of memory to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration <code>yarn.scheduler.maximum-allocation-mb</code>. This value must be smaller than or equal to the cluster maximum. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.maximum-allocation-vcores</code> </td>
|
|
<td align="left"> The per queue maximum limit of virtual cores to allocate to each container request at the Resource Manager. This setting overrides the cluster configuration <code>yarn.scheduler.maximum-allocation-vcores</code>. This value must be smaller than or equal to the cluster maximum. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.user-settings.<user-name>.weight</code> </td>
|
|
<td align="left"> This floating point value is used when calculating the user limit resource values for users in a queue. This value will weight each user more or less than the other users in the queue. For example, if user A should receive 50% more resources in a queue than users B and C, this property will be set to 1.5 for user A. Users B and C will default to 1.0. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Configuring Resource Allocation</li>
|
|
</ul>
|
|
<p><code>CapacityScheduler</code> supports three different resource allocation configuration modes: percentage values (<i>relative mode</i>), weights and absolute resources.</p>
|
|
<p>Relative mode provides a way to describe queue’s resources as a fraction of its parent’s resources. For example if <i>capacity</i> is set as 50.0 for the queue <code>root.users</code>, users queue has 50% of root’s resources set as minimum capacity.</p>
|
|
<p>In weight mode the resources are divided based on how the queue’s weight relates to the sum of configured weights under the same parent. For example if there are three queues under a parent with weights <i>3w</i>, <i>2w</i>, <i>5w</i>, the sum is 10, so the calculated minimum <i>capacity</i> will be 30%, 20% and 50% respectively. The benefit of using this mode is flexibility. When using percentages every time a new queue gets added the percentage values need to be manually recalculated, as the sum under a parent must to be 100%, but with weights this is performed automatically. Using the previous example when a new queue gets added under the same parent as the previous three with weight <i>10w</i> the new sum will be 20, so the new calculated <i>capacities</i> will be: 15%, 10%, 25%, 50%. Note: <code>yarn.scheduler.capacity.<queue-path>.max-capacity</code> must be configured with percentages, as there is no weight mode for <i>maximum-capacity</i>.</p>
|
|
<p>To use absolute resources mode both <code>yarn.scheduler.capacity.<queue-path>.capacity</code> and <code>yarn.scheduler.capacity.<queue-path>.max-capacity</code> should have absolute resource values like <code>[memory=10240,vcores=12]</code>. This configuration indicates 10GB Memory and 12 VCores.</p>
|
|
<p>It is possible to mix weights and percentages in a queue structure, but child queues under one parent must use the same <i>capacity</i> mode.</p>
|
|
<ul>
|
|
|
|
<li>Running and Pending Application Limits</li>
|
|
</ul>
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to control the running and pending applications:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.maximum-applications</code> / <code>yarn.scheduler.capacity.<queue-path>.maximum-applications</code> </td>
|
|
<td align="left"> Maximum number of applications in the system which can be concurrently active both running and pending. Limits on each queue are directly proportional to their queue capacities and user limits. This is a hard limit and any applications submitted when this limit is reached will be rejected. Default is 10000. This can be set for all queues with <code>yarn.scheduler.capacity.maximum-applications</code> and can also be overridden on a per queue basis by setting <code>yarn.scheduler.capacity.<queue-path>.maximum-applications</code>. When this property is not set for a specific queue path, the maximum application number is calculated by taking all configured node labels into consideration, and choosing the highest possible value. Integer value expected. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.maximum-am-resource-percent</code> / <code>yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent</code> </td>
|
|
<td align="left"> Maximum percent of resources in the cluster which can be used to run application masters - controls number of concurrent active applications. Limits on each queue are directly proportional to their queue capacities and user limits. Specified as a float - ie 0.5 = 50%. Default is 10%. This can be set for all queues with <code>yarn.scheduler.capacity.maximum-am-resource-percent</code> and can also be overridden on a per queue basis by setting <code>yarn.scheduler.capacity.<queue-path>.maximum-am-resource-percent</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.max-parallel-apps</code> / <code>yarn.scheduler.capacity.<queue-path>.max-parallel-apps</code> </td>
|
|
<td align="left"> Maximum number of applications that can run at the same time. Unlike to <code>maximum-applications</code>, application submissions are <i>not</i> rejected when this limit is reached. Instead they stay in <code>ACCEPTED</code> state until they are eligible to run. This can be set for all queues with <code>yarn.scheduler.capacity.max-parallel-apps</code> and can also be overridden on a per queue basis by setting <code>yarn.scheduler.capacity.<queue-path>.max-parallel-apps</code>. Integer value is expected. By default, there is no limit. The maximum parallel application limit is an inherited property in the queue hierarchy, meaning that the lowest value will be selected as the enforced limit in every branch of the hierarchy. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>You can also limit the number of parallel applications on a per user basis.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.user.max-parallel-apps</code> </td>
|
|
<td align="left"> Maximum number of applications that can run at the same time for all users. Default value is unlimited. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.user.<username>.max-parallel-apps</code> </td>
|
|
<td align="left"> Maximum number of applications that can run at the same for a specific user. This overrides the global setting. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The evaluation of these limits happens in the following order:</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>
|
|
|
|
<p><code>maximum-applications</code> check - if the limit is exceeded, the submission is rejected immediately.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><code>max-parallel-apps</code> check - the submission is accepted, but the application will not transition to <code>RUNNING</code> state. It stays in <code>ACCEPTED</code> until the queue / user limits are satisfied.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><code>maximum-am-resource-percent</code> check - if there are too many Application Masters running, the application stays in <code>ACCEPTED</code> state until there is enough room for it.</p>
|
|
</li>
|
|
</ol>
|
|
<ul>
|
|
|
|
<li>Queue Administration & Permissions</li>
|
|
</ul>
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to the administer the queues:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.state</code> </td>
|
|
<td align="left"> The <i>state</i> of the queue. Can be one of <code>RUNNING</code> or <code>STOPPED</code>. If a queue is in <code>STOPPED</code> state, new applications cannot be submitted to <i>itself</i> or <i>any of its child queues</i>. Thus, if the <i>root</i> queue is <code>STOPPED</code> no applications can be submitted to the entire cluster. Existing applications continue to completion, thus the queue can be <i>drained</i> gracefully. Value is specified as Enumeration. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue-path>.acl_submit_applications</code> </td>
|
|
<td align="left"> The <i>ACL</i> which controls who can <i>submit</i> applications to the given queue. If the given user/group has necessary ACLs on the given queue or <i>one of the parent queues in the hierarchy</i> they can submit applications. <i>ACLs</i> for this property <i>are</i> inherited from the parent queue if not specified. If a tilde (~) is prepended to a user name in this list, the real user’s ACLs will allow the proxied user to submit to the queue. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue-path>.acl_administer_queue</code> </td>
|
|
<td align="left"> The <i>ACL</i> which controls who can <i>administer</i> applications on the given queue. If the given user/group has necessary ACLs on the given queue or <i>one of the parent queues in the hierarchy</i> they can administer applications. <i>ACLs</i> for this property <i>are</i> inherited from the parent queue if not specified. If a tilde (~) is prepended to a user name in this list, the real user’s ACLs will allow the proxied user to administer apps the queue. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p><b>Note:</b> An <i>ACL</i> is of the form <i>user1</i>,<i>user2</i> <i>space</i> <i>group1</i>,<i>group2</i>. The special value of * implies <i>anyone</i>. The special value of <i>space</i> implies <i>no one</i>. The default is * for the root queue if not specified.</p>
|
|
<ul>
|
|
|
|
<li>Queue lifetime for applications
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to lifetime of an application:</p></li>
|
|
</ul>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.maximum-application-lifetime</code> </td>
|
|
<td align="left"> Maximum lifetime (in seconds) of an application which is submitted to a queue. Any value less than or equal to zero will be considered as disabled. The default is -1. If positive value is configured then any application submitted to this queue will be killed after it exceeds the configured lifetime. User can also specify lifetime per application in application submission context. However, user lifetime will be overridden if it exceeds queue maximum lifetime. It is point-in-time configuration. Note: This feature can be set at any level in the queue hierarchy. Child queues will inherit their parent’s value unless overridden at the child level. A value of 0 means no max lifetime and will override a parent’s max lifetime. If this property is not set or is set to a negative number, then this queue’s max lifetime value will be inherited from it’s parent.</td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue-path>.default-application-lifetime</code> </td>
|
|
<td align="left"> Default lifetime (in seconds) of an application which is submitted to a queue. Any value less than or equal to zero will be considered as disabled. If the user has not submitted application with lifetime value then this value will be taken. It is point-in-time configuration. This feature can be set at any level in the queue hierarchy. Child queues will inherit their parent’s value unless overridden at the child level. If set to less than or equal to 0, the queue’s max value must also be unlimited. Default lifetime can’t exceed maximum lifetime. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Queue Mapping based on User or Group, Application Name or user defined placement rules</li>
|
|
</ul>
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to configure the queue mapping based on user or group, user & group, or application name. User can also define their own placement rule:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.queue-mappings</code> </td>
|
|
<td align="left"> This configuration specifies the mapping of user or group to a specific queue. You can map a single user or a list of users to queues. Syntax: <code>[u or g]:[name]:[queue_name][,next_mapping]*</code>. Here, <i>u or g</i> indicates whether the mapping is for a user or group. The value is <i>u</i> for user and <i>g</i> for group. <i>name</i> indicates the user name or group name. To specify the user who has submitted the application, %user can be used. <i>queue_name</i> indicates the queue name for which the application has to be mapped. To specify queue name same as user name, <i>%user</i> can be used. To specify queue name same as the name of the primary group for which the user belongs to, <i>%primary_group</i> can be used. Secondary group can be referenced as <i>%secondary_group</i> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.queue-placement-rules.app-name</code> </td>
|
|
<td align="left"> This configuration specifies the mapping of application_name to a specific queue. You can map a single application or a list of applications to queues. Syntax: <code>[app_name]:[queue_name][,next_mapping]*</code>. Here, <i>app_name</i> indicates the application name you want to do the mapping. <i>queue_name</i> indicates the queue name for which the application has to be mapped. To specify the current application’s name as the app_name, %application can be used.</td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.queue-mappings-override.enable</code> </td>
|
|
<td align="left"> This function is used to specify whether the user specified queues can be overridden. This is a Boolean value and the default value is <i>false</i>. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Example:</p>
|
|
<p>Below example covers single mapping separately. In case of multiple mappings with comma separated values, evaluation would be from left to right, and the first valid mapping will be used. Below example order has been documented based on actual order of execution at runtime in case of multiple mappings.</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre> <property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:%user:%primary_group.%user</value>
|
|
<description>Maps users to queue with the same name as user but
|
|
parent queue name should be same as primary group of the user</description>
|
|
</property>
|
|
...
|
|
 <property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:%user:%secondary_group.%user</value>
|
|
<description>Maps users to queue with the same name as user but
|
|
parent queue name should be same as any secondary group of the user</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:%user:%user</value>
|
|
<description>Maps users to queues with the same name as user</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:user2:%primary_group</value>
|
|
<description>user2 is mapped to queue name same as primary group</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:user3:%secondary_group</value>
|
|
<description>user3 is mapped to queue name same as secondary group</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:user1:queue1</value>
|
|
<description>user1 is mapped to queue1</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>g:group1:queue2</value>
|
|
<description>group1 is mapped to queue2</description>
|
|
</property>
|
|
...
|
|
<property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:user1:queue1,u:user2:queue2</value>
|
|
<description>Here, <user1> is mapped to <queue1>, <user2> is mapped to <queue2> respectively</description>
|
|
</property>
|
|
|
|
<property>
|
|
<name>yarn.scheduler.queue-placement-rules.app-name</name>
|
|
<value>appName1:queue1,%application:%application</value>
|
|
<description>
|
|
Here, <appName1> is mapped to <queue1>, maps applications to queues with
|
|
the same name as application respectively. The mappings will be
|
|
evaluated from left to right, and the first valid mapping will be used.
|
|
</description>
|
|
</property>
|
|
</pre></div></div>
|
|
</section><section>
|
|
<h3><a name="JSON-based_queue_mapping_configuration"></a>JSON-based queue mapping configuration</h3>
|
|
<p>In order to make the queue mapping feature more versatile, a new format and evaluation engine has been added to Capacity Scheduler. The new engine is fully backwards compatible with the old one and adds several new features. Note that it can also parse the old format, but the new features are only available if you specify the mappings in JSON.</p><section>
|
|
<h4><a name="Syntax"></a>Syntax</h4>
|
|
<p>Based on the current JSON schema, users can define mapping rules the following way:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>{
|
|
"rules": [
|
|
{
|
|
"type": "...",
|
|
"matches": "...",
|
|
"policy": "...",
|
|
"parentQueue": "...",
|
|
"customPlacement": "...",
|
|
"fallbackResult":"...",
|
|
"create": true/false,
|
|
"value": "...",
|
|
"customPlacement": "..."
|
|
},
|
|
{
|
|
... next rule ...
|
|
}
|
|
]
|
|
}
|
|
</pre></div></div>
|
|
|
|
<p>Rules are evaluated from top to bottom. Compared to the legacy mapping rule evaluator, it can be adjusted more flexibly what happens when the evaluation stops and a given rule does not match.</p></section><section>
|
|
<h4><a name="How_to_enable_JSON-based_queue_mapping"></a>How to enable JSON-based queue mapping</h4>
|
|
<p>The following properties control how the new placement engine expects rules.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Setting </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.mapping-rule-format</code> </td>
|
|
<td align="left"> Allowed values are <code>legacy</code> or <code>json</code>. If it is not set, then the engine assumes that the old format might be in use so it also checks the value of <code>yarn.scheduler.capacity.queue-mappings</code>. Therefore, this must be set to <code>json</code> and cannot be left empty. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.mapping-rule-json</code> </td>
|
|
<td align="left"> The value of this property should contain the entire chain of rules inline. This is the preferred way of configuring Capacity Scheduler if you use the Mutation API, ie. modify configuration real-time via the REST interface. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.mapping-rule-json-file</code> </td>
|
|
<td align="left"> Defines an absolute path to a JSON file which contains the rules. For example, <code>/opt/hadoop/config/mapping-rules.json</code>. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The property <code>yarn.scheduler.capacity.mapping-rule-json</code> takes precedence over <code>yarn.scheduler.capacity.mapping-rule-json-file</code>. If the format is set to <code>json</code> but you don’t define either of these, then you’ll get a warning but the initialization of Capacity Scheduler will not fail.</p></section><section>
|
|
<h4><a name="Differences_between_legacy_and_flexible_queue_auto-creation_modes"></a>Differences between legacy and flexible queue auto-creation modes</h4>
|
|
<p>To use the flexible Queue Auto-Creation under a parent the queue capacities must be configured with weights. The flexible mode gives the user much more freedom to automatically create new leaf queues or entire queue hierarchies based on mapping rules. “Legacy” mode refers to either percentage-based configuration or where capacities are defined with absolute resources.</p>
|
|
<p>In flexible Queue Auto-Creation mode, every parent queue can have dynamically created parent or leaf queues (if the <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.enabled</code> property is set to true), even if it already has static child queues. This also means that certain settings influence the outcome of the queue placement depending on how the scheduler is configured.</p>
|
|
<p>When the mode is relevant, the document explains how certain settings or flags affect the overall logic.</p></section><section>
|
|
<h4><a name="Rules"></a>Rules</h4>
|
|
<p>Each mapping rule can have the following settings:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Setting </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>type</code> </td>
|
|
<td align="left"> Possible values: <code>user</code>, <code>group</code>, <code>application</code>. It tells the engine what the current rule should be matched against. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>matches</code> </td>
|
|
<td align="left"> The string to match, or an asterisk “*” which means “all”. For example, if the type is <code>user</code> and this string is “hadoop” then the rule will only be evaluated if the submitter user is “hadoop”. The “*” does not work with groups. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>policy</code> </td>
|
|
<td align="left"> Selects a list of pre-defined policies which defines where the application should be placed. This will be explained later in the “Policies” section. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>parentQueue</code> </td>
|
|
<td align="left"> In case of <code>user</code>, <code>primaryGroup</code>, <code>primaryGroupUser</code>, <code>secondaryGroup</code>, <code>secondaryGroupUser</code> policies, this tells the engine where the matching queue should be looked for. For example, if the policy is <code>primaryGroup</code>, parent is <code>root.groups</code> and the submitter’s group is “admins”, then the resulting queue will be “root.groups.admin” </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>fallbackResult</code> </td>
|
|
<td align="left"> If the target queue does not exist or it cannot be created, it defines a fallback action. Valid values are <code>skip</code>, <code>reject</code> and <code>placeDefault</code>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>create</code> </td>
|
|
<td align="left"> If set to “false”, then the queue will not be created if it does not exist. This flag works differently in flexible and in legacy mode (see below). </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>value</code> </td>
|
|
<td align="left"> If the policy is <code>setDefaultQueue</code>, then the default queue will change to this setting from “root.default”. Otherwise ignored. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>customPlacement</code> </td>
|
|
<td align="left"> Only works with <code>custom</code> placement policy. The value of this field will be evaluated directly by the engine, which means that various placeholders such as <code>%application</code> or <code>%primary_group</code> will be replaced with their respective values. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p><code>type</code> is the equivalent of the first column in the old format. It is either “g” or “u” and there is a separate property for application mappings. <code>matches</code> is the second column. The only difference is that <code>%user</code> means to match all users, but it’s not expressive enough. So in the new format, it’s been changed to <code>*</code>. The <code>fallbackResult</code> setting is checked what to do when the target queue cannot be created or does not exist. The three settings work the following way:</p>
|
|
<ul>
|
|
|
|
<li>
|
|
|
|
<p><code>skip</code>: ignore the current rule and proceed to the next. This is how Fair Scheduler evaluates placement rules.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><code>placeDefault</code>: place the application to the default queue <code>root.default</code> (unless it’s overridden to something else). This is how Capacity Scheduler works with the old mapping rules.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><code>reject</code>: rejects the submission.</p>
|
|
</li>
|
|
</ul>
|
|
<p>The <code>create</code> flag is affected by the mode:</p>
|
|
<ul>
|
|
|
|
<li>
|
|
|
|
<p><b>Legacy</b> mode: applies to all parent queues that have the <code>yarn.scheduler.capacity.<queue-path>.auto-create-child-queue.enabled</code> set to true.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p><b>Flexible</b> mode: applies to all parent queues that have the <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.enabled</code> set to true.</p>
|
|
</li>
|
|
</ul></section><section>
|
|
<h4><a name="Policies"></a>Policies</h4>
|
|
<p>There are a number of pre-defined placement policies which are similar to those in Fair Scheduler. Many of them can be expressed as a “custom” placement policy as you will see soon, but in many cases, it’s safer and more straightforward to use them directly.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Policy </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>specified</code> </td>
|
|
<td align="left"> Places the application to the queue that was defined during submission. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>reject</code> </td>
|
|
<td align="left"> Rejects the submission. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>defaultQueue</code> </td>
|
|
<td align="left"> Places the application into the default queue <code>root.default</code> or to its overwritten value set by <code>setDefaultQueue</code>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>user</code> </td>
|
|
<td align="left"> Places the application into a queue which matches the username of the submitter. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>applicationName</code> </td>
|
|
<td align="left"> Places the application into a queue which matches the name of the application. Important: it is case-sensitive, white spaces are not removed. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>primaryGroup</code> </td>
|
|
<td align="left"> Places the application into a queue which matches the primary group of the submitter. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>primaryGroupUser</code> </td>
|
|
<td align="left"> Places the application into the queue hierarchy <code>root.[parentQueue].<primaryGroup>.<userName></code>. Note that <code>parentQueue</code> is optional. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>secondaryGroup</code> </td>
|
|
<td align="left"> Places the application into a queue which matches the secondary group of the submitter. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>secondaryGroupUser</code> </td>
|
|
<td align="left"> Places the application into the queue hierarchy <code>root.[parentQueue].<secondaryGroup>.<userName></code>. Note that <code>parentQueue</code> is optional. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>setDefaultQueue</code> </td>
|
|
<td align="left"> Changes the default queue from <code>root.default</code>. The change is permanent in a sense that it is not restored in the next rule. You can change the default queue at any point and as many times as necessary. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>custom</code> </td>
|
|
<td align="left"> Enables the user to use custom placement strings. See explanation below. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Notes:</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>
|
|
|
|
<p>The <code>setDefaultQueue</code> rule only changes the default queue. If you want to restore the default queue back to <code>root.default</code>, then it has to be added to the rule chain again.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>The nested rules <code>primaryGroupUser</code> and <code>secondaryGroupUser</code> also work differently in legacy and flexible mode:</p>
|
|
<ul>
|
|
|
|
<li><b>Legacy</b> mode: they expect the parent queues to exist, ie. they cannot be created automatically. More specifically: when you use <code>primaryGroupUser</code>, it will result in a queue path like <code>root.<primaryGroup>.<userName></code> and <code>root.<primaryGroup></code> must exist. It can be a managed parent in order to have <code>userName</code> leaf created automatically, but the parent still has to be created by hand.</li>
|
|
<li><b>Flexible</b> mode: as long as the parent allows dynamic queues to be created, there are no limitations. The requested queues will be created.</li>
|
|
</ul>
|
|
</li>
|
|
<li>
|
|
|
|
<p>The <code>custom</code> placement policy can describe other policies with the appropriate variable placeholders (see below). For example, <code>primaryGroupUser</code> with the parent queue <code>root.groups</code> can be expressed as <code>root.groups.%primary_group.%user</code>. The primary reason for the rules to exist is that its easier to understand for user who have background in configuring Fair Scheduler and it is more natural to configure the mapping rules this way. It is also more robust because it’s less likely that the user makes a mistake. The “Variables” section describes what variables are available if you intend to use the <code>custom</code> policy.</p>
|
|
</li>
|
|
</ol></section><section>
|
|
<h4><a name="Variables"></a>Variables</h4>
|
|
<p>Internally, the tool populates certain variables with appropriate values. These can be used if <code>custom</code> mapping policy is selected. Note that the engine does only minimal verification when it comes to replacing them - therefore it is your responsibility to provide the correct string.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Variable </th>
|
|
<th align="left"> Meaning </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>%application</code> </td>
|
|
<td align="left"> The name of the submitted application. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>%user</code> </td>
|
|
<td align="left"> The user who submitted the application. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>%primary_group</code> </td>
|
|
<td align="left"> Primary group of the submitter. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>%secondary_group</code> </td>
|
|
<td align="left"> Secondary (supplementary) group of the submitter. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>%default</code> </td>
|
|
<td align="left"> The default queue of the scheduler. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>%specified</code> </td>
|
|
<td align="left"> Contains the queue what the submitter defined. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Example: let’s say we submit a MapReduce application to a queue <code>root.users.mrjobs</code>. In this case, the value of <code>%specified</code> will be set to <code>root.users.mrjobs</code>.</p>
|
|
<p>As explained in the “Policies” section, quite a few policies can be achieved with <code>custom</code>. So, instead of using the <code>specified</code> policy, you can use <code>custom</code> with setting the <code>customPlacement</code> field to <code>%specified</code>. However, you have much greater control over it, because you can also append or prepend an extra string to these variables. So the following setting is possible: <code>%specified.%user.largejobs</code>. Keep in mind that the string must be resolved to a valid queue path in order to have a proper placement.</p></section><section>
|
|
<h4><a name="Converting_the_old_mapping_rule_format_to_the_new_one"></a>Converting the old mapping rule format to the new one</h4>
|
|
<p>In this table, you can see how to rewrite the old, colon-separated rules to the new format.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Old mapping rule </th>
|
|
<th align="left"> JSON-based mapping rule </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>u:username:root.user.queue</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "username",<br />"policy": "custom",<br />"customPlacement": "root.user.queue",<br />"fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>u:%user:%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "user",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>u:%user:root.parent.%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br />"policy": "user",<br /> "parentQueue": "root.parent",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>u:%user:%primary_group</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "primaryGroup",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>u:%user:%primary_group.%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "primaryGroupUser",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>u:%user:root.groups.%primary_group.%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "primaryGroupUser", <br />"parentQueue": "root.groups",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>u:%user:%secondary_group</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "secondaryGroup",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>u:%user:%secondary_group.%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "secondaryGroupUser",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>u:%user:root.groups.%secondary_group.%user</code> </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "secondaryGroupUser",<br /> "parentQueue": "root.groups",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>g:hadoop:root.groups.hadoop</code> </td>
|
|
<td align="left"> <code>{ "type": "group",<br />"matches": "hadoop",<br /> "policy": "custom",<br /> "customPlacement": "root.groups.hadoop",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>%application:%application</code> (application mapping) </td>
|
|
<td align="left"> <code>{ "type": "user",<br />"matches": "*",<br /> "policy": "applicationName",<br /> "fallbackResult":"placeDefault" }</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>hive_query:root.query.hive</code> (application mapping) </td>
|
|
<td align="left"> <code>{ "type": "application",<br />"matches": "hive_query",<br /> "policy": "custom",<br />"customPlacement": "root.query.hive",<br />"fallbackResult":"placeDefault" }</code> </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>It’s worth noting that <code>%application:%application</code> requires a <code>user</code> type matcher. It is because internally, the “*” is interpreted only for users. If you set the <code>type</code> to <code>application</code>, then the “*” means to match an application which is named “*”.</p></section><section>
|
|
<h4><a name="Example"></a>Example</h4>
|
|
<p>We have a cluster which is shared among developers, QA engineers and test developers.</p>
|
|
<p>We’d like to achieve the following placement logic:</p>
|
|
<ol style="list-style-type: decimal">
|
|
|
|
<li>
|
|
|
|
<p>If the user belongs to the <code>devs</code> primary group, it should be placed to <code>root.users.devs</code>. This is reserved for developers.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If the user belongs to the <code>qa</code> primary group, then the application should go to <code>root.users.lowpriogroups.<primaryGroup></code>. These queues have lower capacities and are intended for testers.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If the user belongs to the <code>qa-dev</code> primary group, then the application should go to <code>root.users.highpriogroups.<primaryGroup></code>. These queues have higher capacities and are intended for test developers.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>Put the application into the queue which matches the user name.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If there is no such queue, take the queue from the application submission context, but the queue should not be created if it does not exist and the parent is managed.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If none of the above matches, then the application should be placed to <code>root.default</code>.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If the default placement fails for whatever reason, we change the default queue to <code>root.users.default</code>.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>Try a placement to the default queue again.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>If that fails, reject the submission altogether.</p>
|
|
</li>
|
|
</ol>
|
|
<p>This means a chain of 9 rules:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>{
|
|
"rules":[
|
|
{
|
|
"type": "group",
|
|
"matches": "devs",
|
|
"policy": "custom",
|
|
"customPlacement": "root.users.devs",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "group",
|
|
"matches": "qa",
|
|
"policy": "primaryGroup",
|
|
"parentQueue": "root.users.lowpriogroups",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "group",
|
|
"matches": "qa-dev",
|
|
"policy": "primaryGroup",
|
|
"parentQueue": "root.users.highpriogroups",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "user",
|
|
"matches": "*",
|
|
"policy": "user",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "user",
|
|
"matches": "*",
|
|
"policy": "specified",
|
|
"create": false,
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "user",
|
|
"matches": "*",
|
|
"policy": "defaultQueue",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type": "user",
|
|
"matches": "*",
|
|
"policy": "setDefaultQueue",
|
|
"value": "root.users.default",
|
|
"fallbackResult": "skip"
|
|
},
|
|
{
|
|
"type": "user",
|
|
"matches": "*",
|
|
"policy": "defaultQueue",
|
|
"fallbackResult":"skip"
|
|
},
|
|
{
|
|
"type":"user",
|
|
"matches":"*",
|
|
"policy":"reject"
|
|
}
|
|
]
|
|
}
|
|
</pre></div></div>
|
|
|
|
<p>Note: it’s actually possible to set the <code>fallbackResult</code> to <code>reject</code> on the 8th rule, so you don’t need the final <code>reject</code>. But using <code>reject</code> on its own has its merits: since the <code>type</code> and <code>matches</code> fields are mandatory, you can reject submissions from certain groups, applications or users.</p></section></section><section>
|
|
<h3><a name="Setup_for_application_priority."></a>Setup for application priority.</h3>
|
|
<p>Application priority works only along with FIFO ordering policy. Default ordering policy is FIFO.</p>
|
|
<p>Default priority for an application can be at cluster level and queue level.</p>
|
|
<ul>
|
|
|
|
<li>Cluster-level priority : Any application submitted with a priority greater than the cluster-max priority will have its priority reset to the cluster-max priority. $HADOOP_HOME/etc/hadoop/yarn-site.xml is the configuration file for cluster-max priority.</li>
|
|
</ul>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.cluster.max-application-priority</code> </td>
|
|
<td align="left"> Defines maximum application priority in a cluster. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Leaf Queue-level priority : Each leaf queue provides default priority by the administrator. The queue’s default priority will be used for any application submitted without a specified priority. $HADOOP_HOME/etc/hadoop/capacity-scheduler.xml is the configuration file for queue-level priority.</li>
|
|
</ul>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<leaf-queue-path>.default-application-priority</code> </td>
|
|
<td align="left"> Defines default application priority in a leaf queue. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p><b>Note:</b> Priority of an application will not be changed when application is moved to different queue.</p></section><section>
|
|
<h3><a name="Capacity_Scheduler_container_preemption"></a>Capacity Scheduler container preemption</h3>
|
|
<p>The <code>CapacityScheduler</code> supports preemption of container from the queues whose resource usage is more than their guaranteed capacity. The following configuration parameters need to be enabled in yarn-site.xml for supporting preemption of application containers.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.scheduler.monitor.enable</code> </td>
|
|
<td align="left"> Enable a set of periodic monitors (specified in yarn.resourcemanager.scheduler.monitor.policies) that affect the scheduler. Default value is false. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.scheduler.monitor.policies</code> </td>
|
|
<td align="left"> The list of SchedulingEditPolicy classes that interact with the scheduler. Configured policies need to be compatible with the scheduler. Default value is <code>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</code> which is compatible with <code>CapacityScheduler</code> </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The following configuration parameters can be configured in yarn-site.xml to control the preemption of containers when <code>ProportionalCapacityPreemptionPolicy</code> class is configured for <code>yarn.resourcemanager.scheduler.monitor.policies</code></p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.observe_only</code> </td>
|
|
<td align="left"> If true, run the policy but do not affect the cluster with preemption and kill events. Default value is false </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.monitoring_interval</code> </td>
|
|
<td align="left"> Time in milliseconds between invocations of this ProportionalCapacityPreemptionPolicy policy. Default value is 3000 </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.max_wait_before_kill</code> </td>
|
|
<td align="left"> Time in milliseconds between requesting a preemption from an application and killing the container. Default value is 15000 </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.total_preemption_per_round</code> </td>
|
|
<td align="left"> Maximum percentage of resources preempted in a single round. By controlling this value one can throttle the pace at which containers are reclaimed from the cluster. After computing the total desired preemption, the policy scales it back within this limit. Default value is <code>0.1</code> </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.max_ignored_over_capacity</code> </td>
|
|
<td align="left"> Maximum amount of resources above the target capacity ignored for preemption. This defines a deadzone around the target capacity that helps prevent thrashing and oscillations around the computed target balance. High values would slow the time to capacity and (absent natural.completions) it might prevent convergence to guaranteed capacity. Default value is <code>0.1</code> </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.preemption.natural_termination_factor</code> </td>
|
|
<td align="left"> Given a computed preemption target, account for containers naturally expiring and preempt only this percentage of the delta. This determines the rate of geometric convergence into the deadzone (<code>MAX_IGNORED_OVER_CAPACITY</code>). For example, a termination factor of 0.5 will reclaim almost 95% of resources within 5 * #<code>WAIT_TIME_BEFORE_KILL</code>, even absent natural termination. Default value is <code>0.2</code> </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The <code>CapacityScheduler</code> supports the following configurations in capacity-scheduler.xml to control the preemption of application containers submitted to a queue.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.disable_preemption</code> </td>
|
|
<td align="left"> This configuration can be set to <code>true</code> to selectively disable preemption of application containers submitted to a given queue. This property applies only when system wide preemption is enabled by configuring <code>yarn.resourcemanager.scheduler.monitor.enable</code> to <i>true</i> and <code>yarn.resourcemanager.scheduler.monitor.policies</code> to <i>ProportionalCapacityPreemptionPolicy</i>. If this property is not set for a queue, then the property value is inherited from the queue’s parent. Default value is false.</td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.intra-queue-preemption.disable_preemption</code> </td>
|
|
<td align="left"> This configuration can be set to <i>true</i> to selectively disable intra-queue preemption of application containers submitted to a given queue. This property applies only when system wide preemption is enabled by configuring <code>yarn.resourcemanager.scheduler.monitor.enable</code> to <i>true</i>, <code>yarn.resourcemanager.scheduler.monitor.policies</code> to <i>ProportionalCapacityPreemptionPolicy</i>, and <code>yarn.resourcemanager.monitor.capacity.preemption.intra-queue-preemption.enabled</code> to <i>true</i>. If this property is not set for a queue, then the property value is inherited from the queue’s parent. Default value is <i>false</i>.</td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Reservation_Properties"></a>Reservation Properties</h3>
|
|
<ul>
|
|
|
|
<li>Reservation Administration & Permissions</li>
|
|
</ul>
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to control the creation, deletion, update, and listing of reservations. Note that any user can update, delete, or list their own reservations. If reservation ACLs are enabled but not defined, everyone will have access. In the examples below, <queue> is the queue name. For example, to set the reservation ACL to administer reservations on the default queue, use the property <code>yarn.scheduler.capacity.root.default.acl_administer_reservations</code></p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue>.acl_administer_reservations</code> </td>
|
|
<td align="left"> The ACL which controls who can <i>administer</i> reservations to the given queue. If the given user/group has necessary ACLs on the given queue or they can submit, delete, update and list all reservations. ACLs for this property <i>are not</i> inherited from the parent queue if not specified. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue>.acl_list_reservations</code> </td>
|
|
<td align="left"> The ACL which controls who can <i>list</i> reservations to the given queue. If the given user/group has necessary ACLs on the given queue they can list all applications. ACLs for this property <i>are not</i> inherited from the parent queue if not specified. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.root.<queue>.acl_submit_reservations</code> </td>
|
|
<td align="left"> The ACL which controls who can <i>submit</i> reservations to the given queue. If the given user/group has necessary ACLs on the given queue they can submit reservations. ACLs for this property <i>are not</i> inherited from the parent queue if not specified. </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Configuring_ReservationSystem_with_CapacityScheduler"></a>Configuring <code>ReservationSystem</code> with <code>CapacityScheduler</code></h3>
|
|
<p>The <code>CapacityScheduler</code> supports the <b>ReservationSystem</b> which allows users to reserve resources ahead of time. The application can request the reserved resources at runtime by specifying the <code>reservationId</code> during submission. The following configuration parameters can be configured in yarn-site.xml for <code>ReservationSystem</code>.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.reservation-system.enable</code> </td>
|
|
<td align="left"> <i>Mandatory</i> parameter: to enable the <code>ReservationSystem</code> in the <b>ResourceManager</b>. Boolean value expected. The default value is <i>false</i>, i.e. <code>ReservationSystem</code> is not enabled by default. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.reservation-system.class</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name of the <code>ReservationSystem</code>. The default value is picked based on the configured Scheduler, i.e. if <code>CapacityScheduler</code> is configured, then it is <code>CapacityReservationSystem</code>. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.reservation-system.plan.follower</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name of the <code>PlanFollower</code> that runs on a timer, and synchronizes the <code>CapacityScheduler</code> with the <code>Plan</code> and viceversa. The default value is picked based on the configured Scheduler, i.e. if <code>CapacityScheduler</code> is configured, then it is <code>CapacitySchedulerPlanFollower</code>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.reservation-system.planfollower.time-step</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the frequency in milliseconds of the <code>PlanFollower</code> timer. Long value expected. The default value is <i>1000</i>. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The <code>ReservationSystem</code> is integrated with the <code>CapacityScheduler</code> queue hierachy and can be configured for any <b>LeafQueue</b> currently. The <code>CapacityScheduler</code> supports the following parameters to tune the <code>ReservationSystem</code>:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservable</code> </td>
|
|
<td align="left"> <i>Mandatory</i> parameter: indicates to the <code>ReservationSystem</code> that the queue’s resources is available for users to reserve. Boolean value expected. The default value is <i>false</i>, i.e. reservations are not enabled in <i>LeafQueues</i> by default. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-agent</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name that will be used to determine the implementation of the <code>ReservationAgent</code> which will attempt to place the user’s reservation request in the <code>Plan</code>. The default value is <i>org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy</i>. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-move-on-expiry</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter to specify to the <code>ReservationSystem</code> whether the applications should be moved or killed to the parent reservable queue (configured above) when the associated reservation expires. Boolean value expected. The default value is <i>true</i> indicating that the application will be moved to the reservable queue. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.show-reservations-as-queues</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter to show or hide the reservation queues in the Scheduler UI. Boolean value expected. The default value is <i>false</i>, i.e. reservation queues will be hidden. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-policy</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name that will be used to determine the implementation of the <code>SharingPolicy</code> which will validate if the new reservation doesn’t violate any invariants.. The default value is <i>org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy</i>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-window</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter representing the time in milliseconds for which the <code>SharingPolicy</code> will validate if the constraints in the Plan are satisfied. Long value expected. The default value is one day. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.instantaneous-max-capacity</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: maximum capacity at any time in percentage (%) as a float that the <code>SharingPolicy</code> allows a single user to reserve. The default value is 1, i.e. 100%. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.average-capacity</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the average allowed capacity which will aggregated over the <i>ReservationWindow</i> in percentage (%) as a float that the <code>SharingPolicy</code> allows a single user to reserve. The default value is 1, i.e. 100%. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-planner</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name that will be used to determine the implementation of the <i>Planner</i> which will be invoked if the <code>Plan</code> capacity fall below (due to scheduled maintenance or node failures) the user reserved resources. The default value is <i>org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner</i> which scans the <code>Plan</code> and greedily removes reservations in reversed order of acceptance (LIFO) till the reserved resources are within the <code>Plan</code> capacity </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.reservation-enforcement-window</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter representing the time in milliseconds for which the <code>Planner</code> will validate if the constraints in the Plan are satisfied. Long value expected. The default value is one hour. </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Dynamic_Auto-Creation_and_Management_of_Leaf_Queues"></a>Dynamic Auto-Creation and Management of Leaf Queues</h3>
|
|
<p>The <code>CapacityScheduler</code> supports two types of queue auto-creation modes: legacy and flexible. Legacy mode allows the creation of <b>leaf queues</b> under parent queues which have been configured to use this feature. Flexible mode allows the creation of both <b>parent queues</b> and <b>leaf queues</b>. Note: The created queues will be and can only be configured with weights as <i>capacity</i>.</p>
|
|
<ul>
|
|
|
|
<li>Setup for dynamic auto-created leaf queues through queue mapping</li>
|
|
</ul>
|
|
<p><b>user-group queue mapping(s)</b> listed in <code>yarn.scheduler.capacity.queue-mappings</code> need to specify an additional parent queue parameter to identify which parent queue the auto-created leaf queues need to be created under. Refer above <code>Queue Mapping based on User or Group</code> section for more details. Please note that such parent queues also need to enable auto-creation of child queues as mentioned in <code>Parent queue configuration for dynamic leaf queue creation and management</code> section below</p>
|
|
<p>Example:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre> <property>
|
|
<name>yarn.scheduler.capacity.queue-mappings</name>
|
|
<value>u:user1:queue1,g:group1:queue2,u:user2:%primary_group,u:%user:parent1.%user</value>
|
|
<description>
|
|
Here, u:%user:parent1.%user mapping allows any <user> other than user1,
|
|
user2 to be mapped to its own user specific leaf queue which
|
|
will be auto-created under <parent1>.
|
|
</description>
|
|
</property>
|
|
</pre></div></div>
|
|
|
|
<ul>
|
|
|
|
<li>Parent queue configuration for <b>legacy</b> dynamic leaf queue auto-creation and management</li>
|
|
</ul>
|
|
<p>The <code>Dynamic Queue Auto-Creation and Management</code> feature is integrated with the <code>CapacityScheduler</code> queue hierarchy and can be configured for a <b>ParentQueue</b> currently to auto-create leaf queues. Such parent queues do not support other pre-configured queues to co-exist along with auto-created queues. The <code>CapacityScheduler</code> supports the following parameters to enable auto-creation of queues</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-create-child-queue.enabled</code> </td>
|
|
<td align="left"> <i>Mandatory</i> parameter: Indicates to the <code>CapacityScheduler</code> that auto leaf queue creation needs to be enabled for the specified parent queue. Boolean value expected. The default value is <i>false</i>, i.e. auto leaf queue creation is not enabled in <i>ParentQueue</i> by default. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-create-child-queue.management-policy</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: the class name that will be used to determine the implementation of the <code>AutoCreatedQueueManagementPolicy</code> which will manage leaf queues and their capacities dynamically under this parent queue. The default value is <i>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.queuemanagement.GuaranteedOrZeroCapacityOverTimePolicy</i>. Users or groups might submit applications to the auto-created leaf queues for a limited time and stop using them. Hence there could be more number of leaf queues auto-created under the parent queue than its guaranteed capacity. The current policy implementation allots either configured or zero capacity on a <b>best-effort</b> basis based on availability of capacity on the parent queue and the application submission order across leaf queues. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Configuring <b>legacy</b> <code>Auto-Created Leaf Queues</code> with <code>CapacityScheduler</code></li>
|
|
</ul>
|
|
<p>The parent queue which has been enabled for auto leaf queue creation,supports the configuration of template parameters for automatic configuration of the auto-created leaf queues. The auto-created queues support all of the leaf queue configuration parameters except for <b>Absolute Resource</b> configurations.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.leaf-queue-template.capacity</code> </td>
|
|
<td align="left"> <i>Mandatory</i> parameter: Specifies the minimum guaranteed capacity for the auto-created leaf queues. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.leaf-queue-template.maximum-capacity</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: Specifies the maximum capacity for the auto-created leaf queues. This value must be smaller than or equal to the cluster maximum. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.leaf-queue-template.<leaf-queue-property></code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: For other queue parameters that can be configured on auto-created leaf queues like maximum-capacity, user-limit-factor, maximum-am-resource-percent … - Refer <b>Queue Properties</b> section </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Example 1:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre> <property>
|
|
<name>yarn.scheduler.capacity.root.parent1.auto-create-child-queue.enabled</name>
|
|
<value>true</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.capacity</name>
|
|
<value>5</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.maximum-capacity</name>
|
|
<value>100</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.user-limit-factor</name>
|
|
<value>3.0</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.ordering-policy</name>
|
|
<value>fair</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.GPU.capacity</name>
|
|
<value>50</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.accessible-node-labels</name>
|
|
<value>GPU,SSD</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.accessible-node-labels</name>
|
|
<value>GPU</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent1.leaf-queue-template.accessible-node-labels.GPU.capacity</name>
|
|
<value>5</value>
|
|
</property>
|
|
</pre></div></div>
|
|
|
|
<p>Example 2:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre> <property>
|
|
<name>yarn.scheduler.capacity.root.parent2.auto-create-child-queue.enabled</name>
|
|
<value>true</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent2.leaf-queue-template.capacity</name>
|
|
<value>[memory=1024,vcores=1]</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent2.leaf-queue-template.maximum-capacity</name>
|
|
<value>[memory=10240,vcores=10]</value>
|
|
</property>
|
|
</pre></div></div>
|
|
|
|
<ul>
|
|
|
|
<li>Scheduling Edit Policy configuration for auto-created queue management</li>
|
|
</ul>
|
|
<p>Admins need to specify an additional <code>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueManagementDynamicEditPolicy</code> scheduling edit policy to the list of current scheduling edit policies as a comma separated string in <code>yarn.resourcemanager.scheduler.monitor.policies</code> configuration. For more details, refer <code>Capacity Scheduler container preemption</code> section above</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval</code> </td>
|
|
<td align="left"> Time in milliseconds between invocations of this QueueManagementDynamicEditPolicy policy. Default value is 1500 </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Parent queue configuration for <b>flexible</b> dynamic leaf queue auto-creation and management</li>
|
|
</ul>
|
|
<p>The <code>Flexible Dynamic Queue Auto-Creation and Management</code> feature allows a <b>ParentQueue</b> to auto-create both parent and leaf queues. Such parent queues can have other pre-configured queues co-existing with the auto-created queues. The auto-created queues will have weights as <i>capacity</i> so the pre-configured queues under the parent must be configured the same way. The <code>CapacityScheduler</code> supports the following parameters to enable auto-creation of queues</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.enabled</code> </td>
|
|
<td align="left"> <i>Mandatory</i> parameter: Indicates to the <code>CapacityScheduler</code> that flexible auto queue creation needs to be enabled for the specified parent queue. Boolean value expected. The default value is <i>false</i>, i.e. auto leaf queue creation is not enabled in <i>ParentQueue</i> by default. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.max-queues</code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: Limits the number of dynamic queues created under a parent queue. Integer value expected. The default value is <i>1000</i>. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Configuring <b>flexible</b> <code>Auto-Created Leaf Queues</code> with <code>CapacityScheduler</code></li>
|
|
</ul>
|
|
<p>The parent queue which has the flexible auto queue creation enabled supports the configuration of dynamically created leaf and parent queues through template parameters. The auto-created queues support all of the leaf queue configuration parameters except for <b>Absolute Resource</b> configurations.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.template.<queue-property></code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: Specifies a queue property (like capacity, maximum-capacity, user-limit-factor, maximum-am-resource-percent … - Refer <b>Queue Properties</b> section) inherited by the auto-created <b>parent</b> and <b>leaf</b> queues. Dynamic Queue ACLs set here can be overwritten by the parent-template for dynamic parent queues and with the leaf-template for dynamic leaf queues. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.leaf-template.<queue-property></code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: Specifies a queue property inherited by auto-created <b>leaf</b> queues. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.parent-template.<queue-property></code> </td>
|
|
<td align="left"> <i>Optional</i> parameter: Specifies a queue property inherited by auto-created <b>parent</b> queues. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Using the following example configuration snippet will instruct the <code>CapacityScheduler</code> to: * enable the flexible auto queue creation for root.parent * create all of the dynamic queues <b>two levels below</b> <code>root.parent</code> (for example <code>root.parent.parent-auto.leaf-auto</code>) with 80% as the maximum capacity, because of the wildcard queue path (root.parent.*) * create the dynamic parent queues <b>directly</b> under <code>root.parent</code> with weight 2 * add the GPU label to every leaf queue created <b>directly</b> under <code>root.parent</code> * set the GPU label related weight of every queue <b>directly</b> under <code>root.parent</code></p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre> <property>
|
|
<name>yarn.scheduler.capacity.root.parent.auto-queue-creation-v2.enabled</name>
|
|
<value>true</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent.*.auto-queue-creation-v2.template.maximum-capacity</name>
|
|
<value>80</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent.auto-queue-creation-v2.parent-template.capacity</name>
|
|
<value>2w</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent.auto-queue-creation-v2.leaf-template.accessible-node-labels</name>
|
|
<value>GPU</value>
|
|
</property>
|
|
<property>
|
|
<name>yarn.scheduler.capacity.root.parent.auto-queue-creation-v2.leaf-template.accessible-node-labels.GPU.capacity</name>
|
|
<value>5w</value>
|
|
</property>
|
|
</pre></div></div>
|
|
</section><section>
|
|
<h3><a name="Other_Properties"></a>Other Properties</h3>
|
|
<ul>
|
|
|
|
<li>Resource Calculator</li>
|
|
</ul>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.resource-calculator</code> </td>
|
|
<td align="left"> The ResourceCalculator implementation to be used to compare Resources in the scheduler. The default i.e. org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator only uses Memory while DominantResourceCalculator uses Dominant-resource to compare multi-dimensional resources such as Memory, CPU etc. A Java ResourceCalculator class name is expected. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<ul>
|
|
|
|
<li>Data Locality</li>
|
|
</ul>
|
|
<p>Capacity Scheduler leverages <code>Delay Scheduling</code> to honor task locality constraints. There are 3 levels of locality constraint: node-local, rack-local and off-switch. The scheduler counts the number of missed opportunities when the locality cannot be satisfied, and waits this count to reach a threshold before relaxing the locality constraint to next level. The threshold can be configured in following properties:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.node-locality-delay</code> </td>
|
|
<td align="left"> Number of missed scheduling opportunities after which the CapacityScheduler attempts to schedule rack-local containers. Typically, this should be set to number of nodes in the cluster. By default is setting approximately number of nodes in one rack which is 40. Positive integer value is expected. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.rack-locality-additional-delay</code> </td>
|
|
<td align="left"> Number of additional missed scheduling opportunities over the node-locality-delay ones, after which the CapacityScheduler attempts to schedule off-switch containers. By default this value is set to -1, in this case, the number of missed opportunities for assigning off-switch containers is calculated based on the formula <code>L * C / N</code>, where <code>L</code> is number of locations (nodes or racks) specified in the resource request, <code>C</code> is the number of requested containers, and <code>N</code> is the size of the cluster. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Note, this feature should be disabled if YARN is deployed separately with the file system, as locality is meaningless. This can be done by setting <code>yarn.scheduler.capacity.node-locality-delay</code> to <code>-1</code>, in this case, request’s locality constraint is ignored.</p>
|
|
<ul>
|
|
|
|
<li>Container Allocation per NodeManager Heartbeat</li>
|
|
</ul>
|
|
<p>The <code>CapacityScheduler</code> supports the following parameters to control how many containers can be allocated in each NodeManager heartbeat. These parameters are refreshable via <i>yarn rmadmin -refreshQueues</i>.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.per-node-heartbeat.multiple-assignments-enabled</code> </td>
|
|
<td align="left"> Whether to allow multiple container assignments in one NodeManager heartbeat. Defaults to true. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.capacity.per-node-heartbeat.maximum-container-assignments</code> </td>
|
|
<td align="left"> If <code>multiple-assignments-enabled</code> is true, the maximum amount of containers that can be assigned in one NodeManager heartbeat. Default value is 100, which limits the maximum number of container assignments per heartbeat to 100. Set this value to -1 will disable this limit. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.capacity.per-node-heartbeat.maximum-offswitch-assignments</code> </td>
|
|
<td align="left"> If <code>multiple-assignments-enabled</code> is true, the maximum amount of off-switch containers that can be assigned in one NodeManager heartbeat. Defaults to 1, which represents only one off-switch allocation allowed in one heartbeat. </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Reviewing_the_configuration_of_the_CapacityScheduler"></a>Reviewing the configuration of the CapacityScheduler</h3>
|
|
<p>Once the installation and configuration is completed, you can review it after starting the YARN cluster from the web-ui.</p>
|
|
<ul>
|
|
|
|
<li>
|
|
|
|
<p>Start the YARN cluster in the normal manner.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>Open the <code>ResourceManager</code> web UI.</p>
|
|
</li>
|
|
<li>
|
|
|
|
<p>The <i>/scheduler</i> web-page should show the resource usages of individual queues.</p>
|
|
</li>
|
|
</ul></section></section><section>
|
|
<h2><a name="Changing_Queue_Configuration"></a>Changing Queue Configuration</h2>
|
|
<p>Changing queue/scheduler properties and adding/removing queues can be done in two ways, via file or via API. This behavior can be changed via <code>yarn.scheduler.configuration.store.class</code> in yarn-site.xml. Possible values are <i>file</i>, which allows modifying properties via file; <i>memory</i>, which allows modifying properties via API, but does not persist changes across restart; <i>leveldb</i>, which allows modifying properties via API and stores changes in leveldb backing store; and <i>zk</i>, which allows modifying properties via API and stores changes in zookeeper backing store. The default value is <i>file</i>.</p><section>
|
|
<h3><a name="Changing_queue_configuration_via_file"></a>Changing queue configuration via file</h3>
|
|
<p>To edit by file, you need to edit <b>conf/capacity-scheduler.xml</b> and run <i>yarn rmadmin -refreshQueues</i>.</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>$ vi $HADOOP_CONF_DIR/capacity-scheduler.xml
|
|
$ $HADOOP_YARN_HOME/bin/yarn rmadmin -refreshQueues
|
|
</pre></div></div>
|
|
<section>
|
|
<h4><a name="Deleting_queue_via_file"></a>Deleting queue via file</h4>
|
|
<p>Step 1: Stop the queue</p>
|
|
<p>Before deleting a leaf queue, the leaf queue should not have any running/pending apps and has to BE STOPPED by changing <code>yarn.scheduler.capacity.<queue-path>.state</code>. See the [Queue Administration & Permissions](CapacityScheduler.html#Queue Properties) section. Before deleting a parent queue, all its child queues should not have any running/pending apps and have to BE STOPPED. The parent queue also needs to be STOPPED</p>
|
|
<p>Step 2: Delete the queue</p>
|
|
<p>Remove the queue configurations from the file and run refresh as described above</p></section></section><section>
|
|
<h3><a name="Enabling_periodic_configuration_refresh"></a>Enabling periodic configuration refresh</h3>
|
|
<p>Enabling queue configuration periodic refresh allows reloading and applying the configuration by editing the <i>conf/capacity-scheduler.xml</i> without the necessicity of calling yarn rmadmin -refreshQueues.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.scheduler.monitor.enable</code> </td>
|
|
<td align="left"> Enabling monitoring is necessary for the periodic refresh. Default value is false. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.scheduler.monitor.policies</code> </td>
|
|
<td align="left"> This is a configuration property that holds a list of classes. Adding more classes means more monitor tasks will be launched, Add <code>org.apache.hadoop.yarn.server.resourcemanager.capacity.QueueConfigurationAutoRefreshPolicy</code> to the policies list to enable the periodic refresh. Default value of this property is <code>org.apache.hadoop.yarn.server.resourcemanager.monitor.capacity.ProportionalCapacityPreemptionPolicy</code>, it means the preemption feature is enabled by default. If the ProportionalCapacityPreemptionPolicy class is removed from the list, it disables the preemption feature. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.queue.auto.refresh.monitoring-interval</code> </td>
|
|
<td align="left"> Adjusting the auto-refresh monitoring interval is possible with this configuration property. The value is in milliseconds. The default value is 5000 (5 seconds). </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Changing_queue_configuration_via_API"></a>Changing queue configuration via API</h3>
|
|
<p>Editing by API uses a backing store for the scheduler configuration. To enable this, the following parameters can be configured in yarn-site.xml.</p>
|
|
<p><b>Note:</b> This feature is in alpha phase and is subject to change.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.configuration.store.class</code> </td>
|
|
<td align="left"> The type of backing store to use, as described <a href="CapacityScheduler.html#Changing_Queue_Configuration">above</a>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.configuration.mutation.acl-policy.class</code> </td>
|
|
<td align="left"> An ACL policy can be configured to restrict which users can modify which queues. Default value is <i>org.apache.hadoop.yarn.server.resourcemanager.scheduler.DefaultConfigurationMutationACLPolicy</i>, which only allows YARN admins to make any configuration modifications. Another value is <i>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.conf.QueueAdminConfigurationMutationACLPolicy</i>, which only allows queue modifications if the caller is an admin of the queue. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.configuration.store.max-logs</code> </td>
|
|
<td align="left"> Configuration changes are audit logged in the backing store, if using leveldb or zookeeper. This configuration controls the maximum number of audit logs to store, dropping the oldest logs when exceeded. Default is 1000. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.configuration.leveldb-store.path</code> </td>
|
|
<td align="left"> The storage path of the configuration store when using leveldb. Default value is <i>${hadoop.tmp.dir}/yarn/system/confstore</i>. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.scheduler.configuration.leveldb-store.compaction-interval-secs</code> </td>
|
|
<td align="left"> The interval for compacting the configuration store in seconds, when using leveldb. Default value is 86400, or one day. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.scheduler.configuration.zk-store.parent-path</code> </td>
|
|
<td align="left"> The zookeeper root node path for configuration store related information, when using zookeeper. Default value is <i>/confstore</i>. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p><b>Note:</b> When enabling scheduler configuration mutations via <code>yarn.scheduler.configuration.store.class</code>, <i>yarn rmadmin -refreshQueues</i> will be disabled, i.e. it will no longer be possible to update configuration via file.</p>
|
|
<p>See the <a href="ResourceManagerRest.html#Scheduler_Configuration_Mutation_API">YARN Resource Manager REST API</a> for examples on how to change scheduler configuration via REST, and <a href="YarnCommands.html#schedulerconf">YARN Commands Reference</a> for examples on how to change scheduler configuration via command line.</p></section></section><section>
|
|
<h2><a name="Updating_a_Container_.28Experimental_-_API_may_change_in_the_future.29"></a>Updating a Container (Experimental - API may change in the future)</h2>
|
|
<p>Once an Application Master has received a Container from the Resource Manager, it may request the Resource Manager to update certain attributes of the container.</p>
|
|
<p>Currently only two types of container updates are supported:</p>
|
|
<ul>
|
|
|
|
<li><b>Resource Update</b> : Where the AM can request the RM to update the resource size of the container. For eg: Change the container from a 2GB, 2 vcore container to a 4GB, 2 vcore container.</li>
|
|
<li><b>ExecutionType Update</b> : Where the AM can request the RM to update the ExecutionType of the container. For eg: Change the execution type from <i>GUARANTEED</i> to <i>OPPORTUNISTIC</i> or vice versa.</li>
|
|
</ul>
|
|
<p>This is facilitated by the AM populating the <b>updated_containers</b> field, which is a list of type <b>UpdateContainerRequestProto</b>, in <b>AllocateRequestProto.</b> The AM can make multiple container update requests in the same allocate call.</p>
|
|
<p>The schema of the <b>UpdateContainerRequestProto</b> is as follows:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>message UpdateContainerRequestProto {
|
|
required int32 container_version = 1;
|
|
required ContainerIdProto container_id = 2;
|
|
required ContainerUpdateTypeProto update_type = 3;
|
|
optional ResourceProto capability = 4;
|
|
optional ExecutionTypeProto execution_type = 5;
|
|
}
|
|
</pre></div></div>
|
|
|
|
<p>The <b>ContainerUpdateTypeProto</b> is an enum:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>enum ContainerUpdateTypeProto {
|
|
INCREASE_RESOURCE = 0;
|
|
DECREASE_RESOURCE = 1;
|
|
PROMOTE_EXECUTION_TYPE = 2;
|
|
DEMOTE_EXECUTION_TYPE = 3;
|
|
}
|
|
</pre></div></div>
|
|
|
|
<p>As constrained by the above enum, the scheduler currently supports changing either the resource update OR executionType of a container in one update request.</p>
|
|
<p>The AM must also provide the latest <b>ContainerProto</b> it received from the RM. This is the container which the RM will attempt to update.</p>
|
|
<p>If the RM is able to update the requested container, the updated container will be returned, in the <b>updated_containers</b> list field of type <b>UpdatedContainerProto</b> in the <b>AllocateResponseProto</b> return value of either the same allocate call or in one of the subsequent calls.</p>
|
|
<p>The schema of the <b>UpdatedContainerProto</b> is as follows:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>message UpdatedContainerProto {
|
|
required ContainerUpdateTypeProto update_type = 1;
|
|
required ContainerProto container = 2;
|
|
}
|
|
</pre></div></div>
|
|
|
|
<p>It specifies the type of container update that was performed on the Container and the updated Container object which container an updated token.</p>
|
|
<p>The container token can then be used by the AM to ask the corresponding NM to either start the container, if the container has not already been started or update the container using the updated token.</p>
|
|
<p>The <b>DECREASE_RESOURCE</b> and <b>DEMOTE_EXECUTION_TYPE</b> container updates are automatic - the AM does not explicitly have to ask the NM to decrease the resources of the container. The other update types require the AM to explicitly ask the NM to update the container.</p>
|
|
<p>If the <b>yarn.resourcemanager.auto-update.containers</b> configuration parameter is set to <b>true</b> (false by default), The RM will ensure that all container updates are automatic.</p></section><section>
|
|
<h2><a name="Activities"></a>Activities</h2>
|
|
<p>Scheduling activities are activity messages used for debugging on some critical scheduling path, they can be recorded and exposed via RESTful API with minor impact on the scheduler performance. Currently, there are two types of activities supported: <b>scheduler activities</b> and <b>application activities</b>.</p><section>
|
|
<h3><a name="Scheduler_Activities"></a>Scheduler Activities</h3>
|
|
<p>Scheduler activities include useful scheduling info in a scheduling cycle, which illustrate how the scheduler allocates a container. Scheduler activities REST API (<code>http://rm-http-address:port/ws/v1/cluster/scheduler/activities</code>) provides a way to enable recording scheduler activities and fetch them from cache. To eliminate the performance impact, scheduler automatically disables recording activities at the end of a scheduling cycle, you can query the RESTful API again to get the latest scheduler activities.</p>
|
|
<p>See the <a href="ResourceManagerRest.html#Scheduler_Activities_API">YARN Resource Manager REST API</a> for query parameters, output structure and examples about scheduler activities.</p></section><section>
|
|
<h3><a name="Application_Activities"></a>Application Activities</h3>
|
|
<p>Application activities include useful scheduling info for a specified application, which illustrate how the requirements are satisfied or just skipped. Application activities REST API (<code>http://rm-http-address:port/ws/v1/cluster/scheduler/app-activities/{appid}</code>) provides a way to enable recording application activities for a specified application within a few seconds or fetch historical application activities from cache, available actions which include “refresh” and “get” can be specified by the “actions” parameter:</p>
|
|
<ul>
|
|
|
|
<li>Query with parameter “actions=refresh” will enable recording application activities for the specified application for a certain time (defaults to 3 seconds) and get a simple response like: {“appActivities”:{“applicationId”:“application_1562308866454_0001”,“diagnostic”:“Successfully received action: refresh”,“timestamp”:1562308869253,“dateTime”:“Fri Jul 05 14:41:09 CST 2019”}}.</li>
|
|
<li>Query with parameter “actions=get” will not enable recording but directly get historical application activities from cache.</li>
|
|
<li>If no actions parameter is specified, default actions are “refresh,get”, which means both “refresh” and “get” will be performed.</li>
|
|
</ul>
|
|
<p>See the <a href="ResourceManagerRest.html#Application_Activities_API">YARN Resource Manager REST API</a> for query parameters, output structure and examples about application activities.</p></section><section>
|
|
<h3><a name="Configuration"></a>Configuration</h3>
|
|
<p>The CapacityScheduler supports the following parameters to control the cache size and the expiration of scheduler/application activities.</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.activities-manager.cleanup-interval-ms</code> </td>
|
|
<td align="left"> The cleanup interval for activities in milliseconds. Defaults to 5000. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.activities-manager.scheduler-activities.ttl-ms</code> </td>
|
|
<td align="left"> Time to live for scheduler activities in milliseconds. Defaults to 600000. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.activities-manager.app-activities.ttl-ms</code> </td>
|
|
<td align="left"> Time to live for application activities in milliseconds. Defaults to 600000. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>yarn.resourcemanager.activities-manager.app-activities.max-queue-length</code> </td>
|
|
<td align="left"> Max queue length for app activities. Defaults to 100. </td></tr>
|
|
</tbody>
|
|
</table></section><section>
|
|
<h3><a name="Web_UI"></a>Web UI</h3>
|
|
<p>Activities info is available in the application attempt page on RM Web UI, where outstanding requests are aggregated and displayed. Simply click the refresh button to get the latest activities info.</p></section></section>
|
|
</div>
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
<div id="footer">
|
|
<div class="xright">
|
|
© 2008-2023
|
|
Apache Software Foundation
|
|
|
|
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
|
|
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|