758 lines
48 KiB
HTML
758 lines
48 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<!--
|
|
| Generated by Apache Maven Doxia at 2023-03-27
|
|
| Rendered using Apache Maven Stylus Skin 1.5
|
|
-->
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<title>Apache Hadoop 3.4.0-SNAPSHOT – Placement Constraints</title>
|
|
<style type="text/css" media="all">
|
|
@import url("./css/maven-base.css");
|
|
@import url("./css/maven-theme.css");
|
|
@import url("./css/site.css");
|
|
</style>
|
|
<link rel="stylesheet" href="./css/print.css" type="text/css" media="print" />
|
|
<meta name="Date-Revision-yyyymmdd" content="20230327" />
|
|
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
|
|
</head>
|
|
<body class="composite">
|
|
<div id="banner">
|
|
<a href="http://hadoop.apache.org/" id="bannerLeft">
|
|
<img src="http://hadoop.apache.org/images/hadoop-logo.jpg" alt="" />
|
|
</a>
|
|
<a href="http://www.apache.org/" id="bannerRight">
|
|
<img src="http://www.apache.org/images/asf_logo_wide.png" alt="" />
|
|
</a>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="breadcrumbs">
|
|
|
|
<div class="xright"> <a href="http://wiki.apache.org/hadoop" class="externalLink">Wiki</a>
|
|
|
|
|
<a href="https://gitbox.apache.org/repos/asf/hadoop.git" class="externalLink">git</a>
|
|
|
|
|
<a href="http://hadoop.apache.org/" class="externalLink">Apache Hadoop</a>
|
|
|
|
| Last Published: 2023-03-27
|
|
| Version: 3.4.0-SNAPSHOT
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
<div id="leftColumn">
|
|
<div id="navcolumn">
|
|
|
|
<h5>General</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SingleCluster.html">Single Node Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ClusterSetup.html">Cluster Setup</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CommandsManual.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FileSystemShell.html">FileSystem Shell</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Compatibility.html">Compatibility Specification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DownstreamDev.html">Downstream Developer's Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AdminCompatibilityGuide.html">Admin Compatibility Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/InterfaceClassification.html">Interface Classification</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/filesystem/index.html">FileSystem Specification</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Common</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CLIMiniCluster.html">CLI Mini Cluster</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/FairCallQueue.html">Fair Call Queue</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/NativeLibraries.html">Native Libraries</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Superusers.html">Proxy User</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/RackAwareness.html">Rack Awareness</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/SecureMode.html">Secure Mode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/ServiceLevelAuth.html">Service Level Authorization</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/HttpAuthentication.html">HTTP Authentication</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/CredentialProviderAPI.html">Credential Provider API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/index.html">Hadoop KMS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Tracing.html">Tracing</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellGuide.html">Unix Shell Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/registry/index.html">Registry</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/AsyncProfilerServlet.html">Async Profiler</a>
|
|
</li>
|
|
</ul>
|
|
<h5>HDFS</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDesign.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html">User Guide</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithQJM.html">NameNode HA With QJM</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSHighAvailabilityWithNFS.html">NameNode HA With NFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html">Observer NameNode</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/Federation.html">Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFs.html">ViewFs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ViewFsOverloadScheme.html">ViewFsOverloadScheme</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html">Snapshots</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsEditsViewer.html">Edits Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsImageViewer.html">Image Viewer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsPermissionsGuide.html">Permissions and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html">Quotas and HDFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/LibHdfs.html">libhdfs (C API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/WebHDFS.html">WebHDFS (REST API)</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/index.html">HttpFS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ShortCircuitLocalReads.html">Short Circuit Local Reads</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/CentralizedCacheManagement.html">Centralized Cache Management</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsNfsGateway.html">NFS Gateway</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html">Rolling Upgrade</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ExtendedAttributes.html">Extended Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html">Transparent Encryption</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsMultihoming.html">Multihoming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/ArchivalStorage.html">Storage Policies</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/MemoryStorage.html">Memory Storage Support</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/SLGUserGuide.html">Synthetic Load Generator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSErasureCoding.html">Erasure Coding</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HDFSDiskbalancer.html">Disk Balancer</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsUpgradeDomain.html">Upgrade Domain</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsDataNodeAdminGuide.html">DataNode Admin</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html">Router Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/HdfsProvidedStorage.html">Provided Storage</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html">Tutorial</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html">Compatibility with 1.x</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/EncryptedShuffle.html">Encrypted Shuffle</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/PluggableShuffleAndPluggableSort.html">Pluggable Shuffle/Sort</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/DistributedCacheDeploy.html">Distributed Cache Deploy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/SharedCacheSupport.html">Support for YARN Shared Cache</a>
|
|
</li>
|
|
</ul>
|
|
<h5>MapReduce REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapredAppMasterRest.html">MR Application Master</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-hs/HistoryServerRest.html">MR History Server</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YARN.html">Architecture</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnCommands.html">Commands Reference</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html">Capacity Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/FairScheduler.html">Fair Scheduler</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRestart.html">ResourceManager Restart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerHA.html">ResourceManager HA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceModel.html">Resource Model</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeLabel.html">Node Labels</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeAttributes.html">Node Attributes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebApplicationProxy.html">Web Application Proxy</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html">Timeline Service V.2</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WritingYarnApplications.html">Writing YARN Applications</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnApplicationSecurity.html">YARN Application Security</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManager.html">NodeManager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/DockerContainers.html">Running Applications in Docker Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/RuncContainers.html">Running Applications in runC Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerCgroups.html">Using CGroups</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SecureContainer.html">Secure Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ReservationSystem.html">Reservation System</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/GracefulDecommission.html">Graceful Decommission</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/OpportunisticContainers.html">Opportunistic Containers</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/Federation.html">YARN Federation</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/SharedCache.html">Shared Cache</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingGpus.html">Using GPU</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/UsingFPGA.html">Using FPGA</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/PlacementConstraints.html">Placement Constraints</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/YarnUI2.html">YARN UI2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN REST APIs</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html">Introduction</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html">Resource Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/NodeManagerRest.html">Node Manager</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServer.html#Timeline_Server_REST_API_v1">Timeline Server</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/TimelineServiceV2.html#Timeline_Service_v.2_REST_API">Timeline Service V.2</a>
|
|
</li>
|
|
</ul>
|
|
<h5>YARN Service</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Overview.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/QuickStart.html">QuickStart</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/Concepts.html">Concepts</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/YarnServiceAPI.html">Yarn Service API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/ServiceDiscovery.html">Service Discovery</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-site/yarn-service/SystemServices.html">System Services</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Hadoop Compatible File Systems</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-aliyun/tools/hadoop-aliyun/index.html">Aliyun OSS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-aws/tools/hadoop-aws/index.html">Amazon S3</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure/index.html">Azure Blob Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-azure-datalake/index.html">Azure Data Lake Storage</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-cos/cloud-storage/index.html">Tencent COS</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-huaweicloud/cloud-storage/index.html">Huaweicloud OBS</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Auth</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/index.html">Overview</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Examples.html">Examples</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/Configuration.html">Configuration</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-auth/BuildingIt.html">Building</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Tools</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-streaming/HadoopStreaming.html">Hadoop Streaming</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archives/HadoopArchives.html">Hadoop Archives</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-archive-logs/HadoopArchiveLogs.html">Hadoop Archive Logs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-distcp/DistCp.html">DistCp</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-federation-balance/HDFSFederationBalance.html">HDFS Federation Balance</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-gridmix/GridMix.html">GridMix</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-rumen/Rumen.html">Rumen</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-resourceestimator/ResourceEstimator.html">Resource Estimator Service</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-sls/SchedulerLoadSimulator.html">Scheduler Load Simulator</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Benchmarking.html">Hadoop Benchmarking</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-dynamometer/Dynamometer.html">Dynamometer</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Reference</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/release/">Changelog and Release Notes</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../api/index.html">Java API docs</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/UnixShellAPI.html">Unix Shell API</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/Metrics.html">Metrics</a>
|
|
</li>
|
|
</ul>
|
|
<h5>Configuration</h5>
|
|
<ul>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/core-default.xml">core-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs/hdfs-default.xml">hdfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-hdfs-rbf/hdfs-rbf-default.xml">hdfs-rbf-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml">mapred-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-yarn/hadoop-yarn-common/yarn-default.xml">yarn-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-kms/kms-default.html">kms-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-hdfs-httpfs/httpfs-default.html">httpfs-default.xml</a>
|
|
</li>
|
|
<li class="none">
|
|
<a href="../../hadoop-project-dist/hadoop-common/DeprecatedProperties.html">Deprecated Properties</a>
|
|
</li>
|
|
</ul>
|
|
<a href="http://maven.apache.org/" title="Built by Maven" class="poweredBy">
|
|
<img alt="Built by Maven" src="./images/logos/maven-feather.png"/>
|
|
</a>
|
|
|
|
</div>
|
|
</div>
|
|
<div id="bodyColumn">
|
|
<div id="contentBox">
|
|
<!---
|
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|
you may not use this file except in compliance with the License.
|
|
You may obtain a copy of the License at
|
|
|
|
http://www.apache.org/licenses/LICENSE-2.0
|
|
|
|
Unless required by applicable law or agreed to in writing, software
|
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
See the License for the specific language governing permissions and
|
|
limitations under the License. See accompanying LICENSE file.
|
|
-->
|
|
<h1>Placement Constraints</h1>
|
|
<ul>
|
|
<li><a href="#Overview">Overview</a></li>
|
|
<li><a href="#Quick_Guide">Quick Guide</a>
|
|
<ul>
|
|
<li><a href="#Enabling_placement_constraints">Enabling placement constraints</a></li>
|
|
<li><a href="#Experimenting_with_placement_constraints_using_distributed_shell">Experimenting with placement constraints using distributed shell</a></li></ul></li>
|
|
<li><a href="#Defining_Placement_Constraints">Defining Placement Constraints</a>
|
|
<ul>
|
|
<li><a href="#Allocation_tags">Allocation tags</a>
|
|
<ul>
|
|
<li><a href="#Allocation_tags_namespace">Allocation tags namespace</a></li>
|
|
<li><a href="#Differences_between_node_labels.2C_node_attributes_and_allocation_tags">Differences between node labels, node attributes and allocation tags</a></li></ul></li>
|
|
<li><a href="#Placement_constraints_API">Placement constraints API</a></li>
|
|
<li><a href="#Specifying_constraints_in_applications">Specifying constraints in applications</a></li></ul></li></ul>
|
|
<section>
|
|
<h2><a name="Overview"></a>Overview</h2>
|
|
<p>YARN allows applications to specify placement constraints in the form of data locality (preference to specific nodes or racks) or (non-overlapping) node labels. This document focuses on more expressive placement constraints in YARN. Such constraints can be crucial for the performance and resilience of applications, especially those that include long-running containers, such as services, machine-learning and streaming workloads.</p>
|
|
<p>For example, it may be beneficial to co-locate the allocations of a job on the same rack (<i>affinity</i> constraints) to reduce network costs, spread allocations across machines (<i>anti-affinity</i> constraints) to minimize resource interference, or allow up to a specific number of allocations in a node group (<i>cardinality</i> constraints) to strike a balance between the two. Placement decisions also affect resilience. For example, allocations placed within the same cluster upgrade domain would go offline simultaneously.</p>
|
|
<p>The applications can specify constraints without requiring knowledge of the underlying topology of the cluster (e.g., one does not need to specify the specific node or rack where their containers should be placed with constraints) or the other applications deployed. Currently, all constraints are <b>hard</b>, that is, if a constraint for a container cannot be satisfied due to the current cluster condition or conflicting constraints, the container request will remain pending or get rejected.</p>
|
|
<p>Note that in this document we use the notion of “allocation” to refer to a unit of resources (e.g., CPU and memory) that gets allocated in a node. In the current implementation of YARN, an allocation corresponds to a single container. However, in case an application uses an allocation to spawn more than one containers, an allocation could correspond to multiple containers.</p></section><section>
|
|
<h2><a name="Quick_Guide"></a>Quick Guide</h2>
|
|
<p>We first describe how to enable scheduling with placement constraints and then provide examples of how to experiment with this feature using the distributed shell, an application that allows to run a given shell command on a set of containers.</p><section>
|
|
<h3><a name="Enabling_placement_constraints"></a>Enabling placement constraints</h3>
|
|
<p>To enable placement constraints, the following property has to be set to <code>placement-processor</code> or <code>scheduler</code> in <b>conf/yarn-site.xml</b>:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Property </th>
|
|
<th align="left"> Description </th>
|
|
<th align="left"> Default value </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>yarn.resourcemanager.placement-constraints.handler</code> </td>
|
|
<td align="left"> Specify which handler will be used to process PlacementConstraints. Acceptable values are: <code>placement-processor</code>, <code>scheduler</code>, and <code>disabled</code>. </td>
|
|
<td align="left"> <code>disabled</code> </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>We now give more details about each of the three placement constraint handlers:</p>
|
|
<ul>
|
|
|
|
<li><code>placement-processor</code>: Using this handler, the placement of containers with constraints is determined as a pre-processing step before the capacity or the fair scheduler is called. Once the placement is decided, the capacity/fair scheduler is invoked to perform the actual allocation. The advantage of this handler is that it supports all constraint types (affinity, anti-affinity, cardinality). Moreover, it considers multiple containers at a time, which allows to satisfy more constraints than a container-at-a-time approach can achieve. As it sits outside the main scheduler, it can be used by both the capacity and fair schedulers. Note that at the moment it does not account for task priorities within an application, given that such priorities might be conflicting with the placement constraints.</li>
|
|
<li><code>scheduler</code>: Using this handler, containers with constraints will be placed by the main scheduler (as of now, only the capacity scheduler supports SchedulingRequests). It currently supports anti-affinity constraints (no affinity or cardinality). The advantage of this handler, when compared to the <code>placement-processor</code>, is that it follows the same ordering rules for queues (sorted by utilization, priority), apps (sorted by FIFO/fairness/priority) and tasks within the same app (priority) that are enforced by the existing main scheduler.</li>
|
|
<li><code>disabled</code>: Using this handler, if a SchedulingRequest is asked by an application, the corresponding allocate call will be rejected.</li>
|
|
</ul>
|
|
<p>The <code>placement-processor</code> handler supports a wider range of constraints and can allow more containers to be placed, especially when applications have demanding constraints or the cluster is highly-utilized (due to considering multiple containers at a time). However, if respecting task priority within an application is important for the user and the capacity scheduler is used, then the <code>scheduler</code> handler should be used instead.</p></section><section>
|
|
<h3><a name="Experimenting_with_placement_constraints_using_distributed_shell"></a>Experimenting with placement constraints using distributed shell</h3>
|
|
<p>Users can experiment with placement constraints by using the distributed shell application through the following command:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>$ yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.4.0-SNAPSHOT.jar -shell_command sleep -shell_args 10 -placement_spec PlacementSpec
|
|
</pre></div></div>
|
|
|
|
<p>where <b>PlacementSpec</b> is of the form:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>PlacementSpec => "" | PlacementExpr;PlacementSpec
|
|
PlacementExpr => SourceTag,ConstraintExpr
|
|
SourceTag => String(NumContainers)
|
|
ConstraintExpr => SingleConstraint | CompositeConstraint
|
|
SingleConstraint => "IN",Scope,TargetTag | "NOTIN",Scope,TargetTag | "CARDINALITY",Scope,TargetTag,MinCard,MaxCard | NodeAttributeConstraintExpr
|
|
NodeAttributeConstraintExpr => NodeAttributeName=Value, NodeAttributeName!=Value
|
|
CompositeConstraint => AND(ConstraintList) | OR(ConstraintList)
|
|
ConstraintList => Constraint | Constraint:ConstraintList
|
|
NumContainers => int
|
|
Scope => "NODE" | "RACK"
|
|
TargetTag => String
|
|
MinCard => int
|
|
MaxCard => int
|
|
</pre></div></div>
|
|
|
|
<p>Note:</p>
|
|
<ul>
|
|
|
|
<li>When the <code>-placement_spec</code> argument is specified (except NodeAttributeConstraintExpr) in the distributed shell command, the <code>-num-containers</code> argument should not be used. In case <code>-num-containers</code> argument is used in conjunction with <code>-placement-spec</code>, the former is ignored. This is because in PlacementSpec, we determine the number of containers per tag, making the <code>-num-containers</code> redundant and possibly conflicting. Moreover, if <code>-placement_spec</code> is used, all containers will be requested with GUARANTEED execution type.</li>
|
|
<li>When the <code>NodeAttributeConstraintExpr</code> is specified, <code>SourceTag(NumContainers)</code> is optional and the value of <code>-num-containers</code> will be considered for the number of containers to request.</li>
|
|
</ul>
|
|
<p>An example of PlacementSpec is the following:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>zk(3),NOTIN,NODE,zk:hbase(5),IN,RACK,zk:spark(7),CARDINALITY,NODE,hbase,1,3
|
|
</pre></div></div>
|
|
|
|
<p>The above encodes three constraints:</p>
|
|
<ul>
|
|
|
|
<li>place 3 containers with tag “zk” (standing for ZooKeeper) with node anti-affinity to each other, i.e., do not place more than one container per node (notice that in this first constraint, the SourceTag and the TargetTag of the constraint coincide);</li>
|
|
<li>place 5 containers with tag “hbase” with affinity to a rack on which containers with tag “zk” are running (i.e., an “hbase” container should not be placed at a rack where an “zk” container is running, given that “zk” is the TargetTag of the second constraint);</li>
|
|
<li>place 7 containers with tag “spark” in nodes that have at least one, but no more than three, containers with tag “hbase”.</li>
|
|
</ul>
|
|
<p>Another example below demonstrates a composite form of constraint:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>zk(5),AND(IN,RACK,hbase:NOTIN,NODE,zk)
|
|
</pre></div></div>
|
|
|
|
<p>The above constraint uses the conjunction operator <code>AND</code> to combine two constraints. The AND constraint is satisfied when both its children constraints are satisfied. The specific PlacementSpec requests to place 5 “zk” containers in a rack where at least one “hbase” container is running, and on a node that no “zk” container is running. Similarly, an <code>OR</code> operator can be used to define a constraint that is satisfied when at least one of its children constraints is satisfied. Note that in case “zk” and “hbase” are containers belonging to different applications (which is most probably the case in real use cases), the allocation tags in the PlacementSpec should include namespaces, as we describe below (see <a href="#Allocation_tags_namespace">Allocation tags namespace</a>).</p></section></section><section>
|
|
<h2><a name="Defining_Placement_Constraints"></a>Defining Placement Constraints</h2><section>
|
|
<h3><a name="Allocation_tags"></a>Allocation tags</h3>
|
|
<p>Allocation tags are string tags that an application can associate with (groups of) its containers. Tags are used to identify components of applications. For example, an HBase Master allocation can be tagged with “hbase-m”, and Region Servers with “hbase-rs”. Other examples are “latency-critical” to refer to the more general demands of the allocation, or “app_0041” to denote the job ID. Allocation tags play a key role in constraints, as they allow to refer to multiple allocations that share a common tag.</p>
|
|
<p>Note that instead of using the <code>ResourceRequest</code> object to define allocation tags, we use the new <code>SchedulingRequest</code> object. This has many similarities with the <code>ResourceRequest</code>, but better separates the sizing of the requested allocations (number and size of allocations, priority, execution type, etc.), and the constraints dictating how these allocations should be placed (resource name, relaxed locality). Applications can still use <code>ResourceRequest</code> objects, but in order to define allocation tags and constraints, they need to use the <code>SchedulingRequest</code> object. Within a single <code>AllocateRequest</code>, an application should use either the <code>ResourceRequest</code> or the <code>SchedulingRequest</code> objects, but not both of them.</p><section>
|
|
<h4><a name="Allocation_tags_namespace"></a>Allocation tags namespace</h4>
|
|
<p>Allocation tags might refer to containers of the same or different applications, and are used to express intra- or inter-application constraints, respectively. We use allocation tag namespaces in order to specify the scope of applications that an allocation tag can refer to. By coupling an allocation tag with a namespace, we can restrict whether the tag targets containers that belong to the same application, to a certain group of applications, or to any application in the cluster.</p>
|
|
<p>We currently support the following namespaces:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Namespace </th>
|
|
<th align="left"> Syntax </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> SELF </td>
|
|
<td align="left"> <code>self/${allocationTag}</code> </td>
|
|
<td align="left"> The allocation tag refers to containers of the current application (to which the constraint will be applied). This is the default namespace. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> NOT_SELF </td>
|
|
<td align="left"> <code>not-self/${allocationTag}</code> </td>
|
|
<td align="left"> The allocation tag refers only to containers that do not belong to the current application. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> ALL </td>
|
|
<td align="left"> <code>all/${allocationTag}</code> </td>
|
|
<td align="left"> The allocation tag refers to containers of any application. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> APP_ID </td>
|
|
<td align="left"> <code>app-id/${applicationID}/${allocationTag}</code> </td>
|
|
<td align="left"> The allocation tag refers to containers of the application with the specified application ID. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> APP_TAG </td>
|
|
<td align="left"> <code>app-tag/application_tag_name/${allocationTag}</code> </td>
|
|
<td align="left"> The allocation tag refers to containers of applications that are tagged with the specified application tag. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>To attach an allocation tag namespace <code>ns</code> to a target tag <code>targetTag</code>, we use the syntax <code>ns/allocationTag</code> in the PlacementSpec. Note that the default namespace is <code>SELF</code>, which is used for <b>intra-app</b> constraints. The remaining namespace tags are used to specify <b>inter-app</b> constraints. When the namespace is not specified next to a tag, <code>SELF</code> is assumed.</p>
|
|
<p>The example constraints used above could be extended with namespaces as follows:</p>
|
|
|
|
<div class="source">
|
|
<div class="source">
|
|
<pre>zk(3),NOTIN,NODE,not-self/zk:hbase(5),IN,RACK,all/zk:spark(7),CARDINALITY,NODE,app-id/appID_0023/hbase,1,3
|
|
</pre></div></div>
|
|
|
|
<p>The semantics of these constraints are the following:</p>
|
|
<ul>
|
|
|
|
<li>place 3 containers with tag “zk” (standing for ZooKeeper) to nodes that do not have “zk” containers from other applications running;</li>
|
|
<li>place 5 containers with tag “hbase” with affinity to a rack on which containers with tag “zk” (from any application, be it the same or a different one) are running;</li>
|
|
<li>place 7 containers with tag “spark” in nodes that have at least one, but no more than three, containers with tag “hbase” belonging to application with ID <code>appID_0023</code>.</li>
|
|
</ul></section><section>
|
|
<h4><a name="Differences_between_node_labels.2C_node_attributes_and_allocation_tags"></a>Differences between node labels, node attributes and allocation tags</h4>
|
|
<p>The difference between allocation tags and node labels or <a href="NodeAttributes.html">node attributes</a>, is that allocation tags are attached to allocations and not to nodes. When an allocation gets allocated to a node by the scheduler, the set of tags of that allocation are automatically added to the node for the duration of the allocation. Hence, a node inherits the tags of the allocations that are currently allocated to the node. Likewise, a rack inherits the tags of its nodes. Moreover, similar to node labels and unlike node attributes, allocation tags have no value attached to them. As we show below, our constraints can refer to allocation tags, as well as node labels and node attributes.</p></section></section><section>
|
|
<h3><a name="Placement_constraints_API"></a>Placement constraints API</h3>
|
|
<p>Applications can use the public API in the <code>PlacementConstraints</code> to construct placement constraint. Before describing the methods for building constraints, we describe the methods of the <code>PlacementTargets</code> class that are used to construct the target expressions that will then be used in constraints:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Method </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>allocationTag(String... allocationTags)</code> </td>
|
|
<td align="left"> Constructs a target expression on an allocation tag. It is satisfied if there are allocations with one of the given tags. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>allocationTagWithNamespace(String namespace, String... allocationTags)</code> </td>
|
|
<td align="left"> Similar to <code>allocationTag(String...)</code>, but allows to specify a namespace for the given allocation tags. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>nodePartition(String... nodePartitions)</code> </td>
|
|
<td align="left"> Constructs a target expression on a node partition. It is satisfied for nodes that belong to one of the <code>nodePartitions</code>. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>nodeAttribute(String attributeKey, String... attributeValues)</code> </td>
|
|
<td align="left"> Constructs a target expression on a node attribute. It is satisfied if the specified node attribute has one of the specified values. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>Note that the <code>nodeAttribute</code> method above is not yet functional, as it requires the ongoing node attributes feature.</p>
|
|
<p>The methods of the <code>PlacementConstraints</code> class for building constraints are the following:</p>
|
|
<table border="0" class="bodyTable">
|
|
<thead>
|
|
|
|
<tr class="a">
|
|
<th align="left"> Method </th>
|
|
<th align="left"> Description </th></tr>
|
|
</thead><tbody>
|
|
|
|
<tr class="b">
|
|
<td align="left"> <code>targetIn(String scope, TargetExpression... targetExpressions)</code> </td>
|
|
<td align="left"> Creates a constraint that requires allocations to be placed on nodes that satisfy all target expressions within the given scope (e.g., node or rack). For example, <code>targetIn(RACK, allocationTag("hbase-m"))</code>, allows allocations on nodes that belong to a rack that has at least one allocation with tag “hbase-m”. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>targetNotIn(String scope, TargetExpression... targetExpressions)</code> </td>
|
|
<td align="left"> Creates a constraint that requires allocations to be placed on nodes that belong to a scope (e.g., node or rack) that does not satisfy any of the target expressions. </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>cardinality(String scope, int minCardinality, int maxCardinality, String... allocationTags)</code> </td>
|
|
<td align="left"> Creates a constraint that restricts the number of allocations within a given scope (e.g., node or rack). For example, {@code cardinality(NODE, 3, 10, “zk”)} is satisfied on nodes where there are no less than 3 allocations with tag “zk” and no more than 10. </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>minCardinality(String scope, int minCardinality, String... allocationTags)</code> </td>
|
|
<td align="left"> Similar to <code>cardinality(String, int, int, String...)</code>, but determines only the minimum cardinality (the maximum cardinality is unbound). </td></tr>
|
|
<tr class="b">
|
|
<td align="left"> <code>maxCardinality(String scope, int maxCardinality, String... allocationTags)</code> </td>
|
|
<td align="left"> Similar to <code>cardinality(String, int, int, String...)</code>, but determines only the maximum cardinality (the minimum cardinality is 0). </td></tr>
|
|
<tr class="a">
|
|
<td align="left"> <code>targetCardinality(String scope, int minCardinality, int maxCardinality, String... allocationTags)</code> </td>
|
|
<td align="left"> This constraint generalizes the cardinality and target constraints. Consider a set of nodes N that belongs to the scope specified in the constraint. If the target expressions are satisfied at least minCardinality times and at most maxCardinality times in the node set N, then the constraint is satisfied. For example, <code>targetCardinality(RACK, 2, 10, allocationTag("zk"))</code>, requires an allocation to be placed within a rack that has at least 2 and at most 10 other allocations with tag “zk”. </td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p>The <code>PlacementConstraints</code> class also includes method for building compound constraints (AND/OR expressions with multiple constraints). Adding support for compound constraints is work in progress.</p></section><section>
|
|
<h3><a name="Specifying_constraints_in_applications"></a>Specifying constraints in applications</h3>
|
|
<p>Applications have to specify the containers for which each constraint will be enabled. To this end, applications can provide a mapping from a set of allocation tags (source tags) to a placement constraint. For example, an entry of this mapping could be “hbase”->constraint1, which means that constraint1 will be applied when scheduling each allocation with tag “hbase”.</p>
|
|
<p>When using the <code>placement-processor</code> handler (see <a href="#Enabling_placement_constraints">Enabling placement constraints</a>), this constraint mapping is specified within the <code>RegisterApplicationMasterRequest</code>.</p>
|
|
<p>When using the <code>scheduler</code> handler, the constraints can also be added at each <code>SchedulingRequest</code> object. Each such constraint is valid for the tag of that scheduling request. In case constraints are specified both at the <code>RegisterApplicationMasterRequest</code> and the scheduling requests, the latter override the former.</p></section></section>
|
|
</div>
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
<div id="footer">
|
|
<div class="xright">
|
|
© 2008-2023
|
|
Apache Software Foundation
|
|
|
|
- <a href="http://maven.apache.org/privacy-policy.html">Privacy Policy</a>.
|
|
Apache Maven, Maven, Apache, the Apache feather logo, and the Apache Maven project logos are trademarks of The Apache Software Foundation.
|
|
</div>
|
|
<div class="clear">
|
|
<hr/>
|
|
</div>
|
|
</div>
|
|
</body>
|
|
</html>
|