<li><ahref="#Items_for_Future_Work">Items for Future Work</a></li></ul>
<section>
<h2><aname="Purpose"></a>Purpose</h2>
<p>This document introduces the notion of <b>opportunistic</b> container execution, and discusses how opportunistic containers are allocated and executed.</p></section><section>
<h2><aname="Quick_Guide"></a>Quick Guide</h2>
<p>We start by providing a brief overview of opportunistic containers, including how a user can enable this feature and run a sample job using such containers.</p><section>
<h3><aname="Main_Goal"></a>Main Goal</h3>
<p>Unlike existing YARN containers that are scheduled in a node only if there are unallocated resources, opportunistic containers can be dispatched to an NM, even if their execution at that node cannot start immediately. In such a case, opportunistic containers will be queued at that NM until resources become available. The main goal of opportunistic container execution is to improve cluster resource utilization, and therefore increase task throughput. Resource utilization and task throughput improvements are more pronounced for workloads that include relatively short tasks (in the order of seconds).</p></section><section>
<tdalign="left"> Determines the max number of opportunistic containers that can be queued at an NM. </td>
<tdalign="left"><code>0</code></td></tr>
</tbody>
</table>
<p>The first parameter above has to be set to <code>true</code>. The second one has to be set to a positive value to allow queuing of opportunistic containers at the NM. A value of <code>10</code> can be used to start experimenting with opportunistic containers. The optimal value depends on the jobs characteristics, the cluster configuration and the target utilization.</p>
<p>By default, allocation of opportunistic containers is performed centrally through the RM. However, a user can choose to enable distributed allocation of opportunistic containers, which can further improve allocation latency for short tasks. Distributed scheduling can be enabling by setting to <code>true</code> the following parameter (note that non-opportunistic containers will continue being scheduled through the RM):</p>
<p>In order to submit jobs to a cluster that has AMRMProxy turned on, one must create a separate set of configs for the client from which jobs will be submitted. In these, the <b>conf/yarn-site.xml</b> should have the following additional configurations:</p>
<tdalign="left"> Redirects jobs to the Node Manager’s AMRMProxy port.</td></tr>
</tbody>
</table></section><section>
<h3><aname="Running_a_Sample_Job"></a>Running a Sample Job</h3><section>
<h4><aname="MapReduce_PI"></a>MapReduce PI</h4>
<p>The following command can be used to run a sample pi map-reduce job, executing 40% of mappers using opportunistic containers:</p>
<divclass="source">
<divclass="source">
<pre>$ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.4.0-SNAPSHOT.jar pi -Dmapreduce.job.num-opportunistic-maps-percent="40" 50 100
</pre></div></div>
<p>By changing the value of <code>mapreduce.job.num-opportunistic-maps-percent</code> in the above command, we can specify the percentage of mappers that can be executed through opportunistic containers.</p></section><section>
<p>Another sample job is the distributed shell, it allows us to run a given shell command on a set of containers. The following command can be used to run <code>sleep 10</code> command in 10 opportunistic containers:</p>
<p>By change the value of <code>container_type</code> to <code>OPPORTUNISTIC</code> or <code>GUARANTEED</code> in the above command, we can specify the tasks to be running in opportunistic or guaranteed containers. The default type is <code>GUARANTEED</code>. By adding flag <code>-promote_opportunistic_after_start</code> to the above command, application master will attempt to promote all opportunistic containers to guaranteed once they are started. By adding flag ‘-enforce_execution_type’ to the above command, scheduler will honor execution type of the containers.</p></section></section><section>
<h3><aname="Opportunistic_Containers_in_Web_UI"></a>Opportunistic Containers in Web UI</h3>
<p>When opportunistic container allocation is enabled, the following new columns can be observed in the Nodes page of the Web UI (<code>rm-address:8088/cluster/nodes</code>):</p>
<ul>
<li>Running Containers (O): number of running opportunistic containers on each node;</li>
<li>Mem Used (O): Total memory used by opportunistic containers on each node;</li>
<li>VCores Used (O): Total CPU virtual cores used by opportunistic containers on each node;</li>
<li>Queued Containers: Number of containers queued at each node.</li>
</ul>
<p>When clicking on a specific container running on a node, the execution type of the container is also shown.</p>
<p>In the rest of the document, we provide an in-depth description of opportunistic containers, including details about their allocation and execution.</p></section></section><section>
<h2><aname="Overview"></a>Overview</h2>
<p>The existing schedulers in YARN (Fair and Capacity Scheduler) allocate containers to a node only if there are unallocated resources at that node at the moment of scheduling the containers. This <b>guaranteed</b> type of execution has the advantage that once the AM dispatches a container to a node, the container execution will start immediately, since it is guaranteed that there will be available resources. Moreover, unless fairness or capacity constraints are violated, containers are guaranteed to run to completion without being preempted.</p>
<p>Although this design offers a more predictable task execution, it has two main drawbacks that can lead to suboptimal cluster resource utilization:</p>
<ul>
<li><b>Feedback delays.</b> When a container finishes its execution at a node, the RM gets notified that there are available resources through the next NM-RM heartbeat, then the RM schedules a new container at that node, the AM gets notified through the next AM-RM heartbeat, and finally the AM launches the new container at the node. These delays result in idle node resources, which in turn lead to lower resource utilization, especially when workloads involve tasks whose duration is relatively short.</li>
<li><b>Allocated vs. utilized resources.</b> The RM allocates containers based on the <i>allocated</i> resources at each node, which might be significantly higher than the actually <i>utilized</i> resources (e.g., think of a container for which 4GB memory have been allocated, but only 2GB are being utilized). This lowers effective resource utilization, and can be avoided if the RM takes into account the utilized resources during scheduling. However, this has to be done in a way that allows resources to be reclaimed in case the utilized resources of a running container increase.</li>
</ul>
<p>To mitigate the above problems, in addition to the existing containers (which we term <b>guaranteed</b> containers hereafter), we introduce the notion of <b>opportunistic</b> containers. An opportunistic container can be dispatched to an NM, even if there are no available (unallocated) resources for it at the moment of scheduling. In such a case, the opportunistic container will be queued at the NM, waiting for resources to become available for its execution to start. The opportunistic containers are of lower priority than the guaranteed ones, which means that they can be preempted for guaranteed containers to start their execution. Therefore, they can be used to improve cluster resource utilization without impacting the execution of existing guaranteed containers.</p>
<p>An additional advantage of opportunistic containers is that they introduce a notion of <b>execution priority at the NMs</b>. For instance, a lower priority job that does not require strict execution guarantees can use opportunistic containers or a mix of container execution types for its tasks.</p>
<p>We have introduced two ways of allocating opportunistic containers: a <b>centralized</b> and a <b>distributed</b> one. In the centralized scheduling, opportunistic containers are allocated through the YARN RM, whereas in the distributed one, through local schedulers that reside at each NM. Centralized allocation allows for higher quality placement decisions and for implementing more involved sharing policies across applications (e.g., fairness). On the other hand, distributed scheduling can offer faster container allocation, which is useful for short tasks, as it avoids the round-trip to the RM. In both cases, the scheduling of guaranteed containers remains intact and happens through the YARN RM (using the existing Fair or Capacity Scheduler).</p>
<p>Note that in the current implementation, we are allocating containers based on allocated (and not utilized) resources. Therefore, we tackle the “feedback delays” problem mentioned above, but not the “allocated vs. utilized resources” one. There is ongoing work (<code>YARN-1011</code>) that employs opportunistic containers to address the latter problem too.</p>
<p>Below, we describe in more detail the <ahref="#Container_Execution_Types">container execution types</a>, as well as the <ahref="#Execution_of_Opportunistic_Containers">execution</a> (including the container queuing at the NMs) and <ahref="#Allocation_of_Opportunistic_Containers">allocation</a> of opportunistic containers. Then we discuss how to fine-tune opportunistic containers through some <ahref="#Advanced_Configuration">advanced configuration parameters</a>. Finally, we discuss open items for <ahref="#Items_for_Future_Work">future work</a>.</p></section><section>
<p>We introduce the following two types of containers:</p>
<ul>
<li><b>Guaranteed containers</b> correspond to the existing YARN containers. They are allocated by the Fair or Capacity Scheduler, and once dispatched to a node, it is guaranteed that there are available resources for their execution to start immediately. Moreover, these containers run to completion (as long as there are no failures). They can be preempted only in case the scheduler’s queue to which they belong, violates fairness or capacity constraints.</li>
<li><b>Opportunistic containers</b> are not guaranteed to have resources for their execution to start when they get dispatched to a node. Instead, they might be queued at the NM until resources become available. In case a guaranteed container arrives at a node and there are no resources available for it, one or more opportunistic containers will be preempted to execute the guaranteed one.</li>
</ul>
<p>When an AM submits its resource requests to the RM, it specifies the type for each container (default is guaranteed), determining the way the container will be <ahref="#Allocation_of_Opportunistic_Containers">allocated</a>. Subsequently, when the container is launched by the AM at an NM, its type determines how it will be <ahref="#Execution_of_Opportunistic_Containers">executed</a> by the NM.</p></section><section>
<h2><aname="Execution_of_Opportunistic_Containers"></a>Execution of Opportunistic Containers</h2>
<p>When a container arrives at an NM, its execution is determined by the available resources at the NM and the container type. Guaranteed containers start their execution immediately, and if needed, the NM will kill running opportunistic containers to ensure there are sufficient resources for the guaranteed ones to start. On the other hand, opportunistic containers can be queued at the NM, if there are no resources available to start their execution when they arrive at the NM. To enable this, we extended the NM by allowing queuing of containers at each node. The NM monitors the local resources, and when there are sufficient resources available, it starts the execution of the opportunistic container that is at the head of the queue.</p>
<p>In particular, when a container arrives at an NM, localization is performed (i.e., all required resources are downloaded), and then the container moves to a <code>SCHEDULED</code> state, in which the container is queued, waiting for its execution to begin:</p>
<ul>
<li>If there are available resources, the execution of the container starts immediately, irrespective of its execution type.</li>
<li>If there are no available resources:
<ul>
<li>If the container is guaranteed, we kill as many running opportunistic containers as required for the guaranteed container to be executed, and then start its execution.</li>
<li>If the container is opportunistic, it remains at the queue until resources become available.</li>
</ul>
</li>
<li>When a container (guaranteed or opportunistic) finishes its execution and resources get freed up, we examine the queued containers and if there are available resources we start their execution. We pick containers from the queue in a FIFO order.</li>
</ul>
<p>In the <ahref="#Items_for_Future_Work">future work items</a> below, we discuss different ways of prioritizing task execution (queue reordering) and of killing opportunistic containers to make space for guaranteed ones.</p></section><section>
<h2><aname="Allocation_of_Opportunistic_Containers"></a>Allocation of Opportunistic Containers</h2>
<p>As mentioned above, we provide both a centralized and a distributed way of allocating opportunistic containers, which we describe below.</p><section>
<p>We have introduced a new service at the RM, namely the <code>OpportunisticContainerAllocatorAMService</code>, which extends the <code>ApplicationMasterService</code>. When the centralized opportunistic allocation is enabled, the resource requests from the AMs are served at the RM side by the <code>OpportunisticContainerAllocatorAMService</code>, which splits them into two sets of resource requests:</p>
<ul>
<li>The guaranteed set is forwarded to the existing <code>ApplicationMasterService</code> and is subsequently handled by the Fair or Capacity Scheduler.</li>
<li>The opportunistic set is handled by the new <code>OpportunisticContainerAllocator</code>, which performs the scheduling of opportunistic containers to nodes.</li>
</ul>
<p>The <code>OpportunisticContainerAllocator</code> maintains a list with the <ahref="#Determining_Nodes_for_Allocation">least loaded nodes</a> of the cluster at each moment, and assigns containers to them in a round-robin fashion. Note that in the current implementation, we purposely do not take into account node locality constraints. Since an opportunistic container (unlike the guaranteed ones) might wait at the queue of an NM before its execution starts, it is more important to allocate it at a node that is less loaded (i.e., where queuing delay will be smaller) rather than respect its locality constraints. Moreover, we do not take into account sharing (fairness/capacity) constraints for opportunistic containers at the moment. Support for both locality and sharing constraints can be added in the future if required.</p></section><section>
<p>In order to enable distributed scheduling of opportunistic containers, we have introduced a new service at each NM, called <code>AMRMProxyService</code>. The <code>AMRMProxyService</code> implements the <code>ApplicationMasterService</code> protocol, and acts as a proxy between the AMs running at that node and the RM. When the <code>AMRMProxyService</code> is enabled (through a parameter), we force all AMs running at a particular node to communicate with the <code>AMRMProxyService</code> of the same node, instead of going directly to the RM. Moreover, to ensure that the AMs will not talk directly with the RM, when a new AM gets initialized, we replace its <code>AMRMToken</code> with a token signed by the <code>AMRMProxyService</code>.</p>
<p>A chain of interceptors can be registered with the <code>AMRMProxyService</code>. One of these interceptors is the <code>DistributedScheduler</code> that is responsible for allocating opportunistic containers in a distributed way, without needing to contact the RM. This modular design makes the <code>AMRMProxyService</code> instrumental in other scenarios too, such as YARN federation (<code>YARN-2915</code>) or throttling down misbehaving AMs, which can be enabled simply by adding additional interceptors at the interceptor chain.</p>
<p>When distributed opportunistic scheduling is enabled, each AM sends its resource requests to the <code>AMRMProxyService</code> running at the same node. The <code>AMRMProxyService</code> splits the resource requests into two sets:</p>
<ul>
<li>The guaranteed set is forwarded to the RM. In this case the <code>AMRMProxyService</code> simply acts as a proxy between the AM and the RM, and the container allocation remains intact (using the Fair or Capacity Scheduler).</li>
<li>The opportunistic set is not forwarded to the RM. Instead, it is handled by the <code>DistributedScheduler</code> that is running locally at the node. In particular, the <code>DistributedScheduler</code> maintains a list with the least loaded nodes in the cluster, and allocates containers to them in a round-robin fashion. The RM informs the <code>DistributedScheduler</code> about the least loaded nodes at regular intervals through the NM-RM heartbeats.</li>
</ul>
<p>The above procedure is similar to the one performed by the <code>OpportunisticContainerAllocatorAMService</code> in the case of centralized opportunistic scheduling described above. The main difference is that in the distributed case, the splitting of requests into guaranteed and opportunistic happens locally at the node, and only the guaranteed requests are forwarded to the RM, while the opportunistic ones are handled without contacting the RM.</p></section><section>
<h3><aname="Determining_Nodes_for_Allocation"></a>Determining Nodes for Allocation</h3>
<p>Each NM informs the RM periodically through the NM-RM heartbeats about the number of running guaranteed and opportunistic containers, as well as the number of queued opportunistic containers. The RM gathers this information from all nodes and determines the least loaded ones.</p>
<p>In the case of centralized allocation of opportunistic containers, this information is immediately available, since the allocation happens centrally. In the case of distributed scheduling, the list with the least loaded nodes is propagated to all NMs (and thus becomes available to the <code>DistributedSchedulers</code>) through the heartbeat responses from the RM to the NMs. The number of least loaded nodes sent to the NMs is configurable.</p>
<p>At the moment, we take into account only the number of queued opportunistic containers at each node in order to estimate the time an opportunistic container would have to wait if sent to that node and, thus, determine the least loaded nodes. If the AM provided us with information about the estimated task durations, we could take them into account in order to have better estimates of the queue waiting times.</p></section><section>
<p>Occasionally poor placement choices for opportunistic containers may be made (due to stale queue length estimates), which can lead to load imbalance between nodes. The problem is more pronounced under high cluster load, and also in the case of distributed scheduling (multiple <code>DistributedSchedulers</code> may place containers at the same NM, since they do not coordinate with each other). To deal with this load imbalance between the NM queues, we perform load shedding to dynamically re-balance the load between NMs. In particular, while aggregating at the RM the queue time estimates published by each NM, we construct a distribution and find a targeted maximal value for the length of the NM queues (based on the mean and standard deviation of the distribution). Then the RM disseminates this value to the various NMs through the heartbeat responses. Subsequently, using this information, an NM on a node whose queue length is above the threshold discards opportunistic containers to meet this maximal value. This forces the associated individual AMs to reschedule those containers elsewhere.</p></section></section><section>
<p>The main properties for enabling opportunistic container allocation and choosing between centralized and distributed allocation were described in the <ahref="#Quick_Guide">quick guide</a> in the beginning of this document. Here we present more advanced configuration. Note that using default values for those parameters should be sufficient in most cases. All parameters below have to be defined in the <b>conf/yarn-site.xml</b> file.</p>
<p>To determine the number of <ahref="#Determining_Nodes_for_Allocation">least loaded nodes</a> that will be used when scheduling opportunistic containers and how often this list will be refreshed, we use the following parameters:</p>
<tdalign="left"> Number of least loaded nodes to be used by the Opportunistic Container allocator for dispatching containers during container allocation. A higher value can improve load balance in large clusters. </td>
<tdalign="left"> Frequency for computing least loaded nodes. </td>
<tdalign="left"><code>1000</code></td></tr>
</tbody>
</table>
<p>As discussed in the <ahref="#Rebalancing_Node_Load">node load rebalancing</a> section above, at regular intervals, the RM gathers all NM queue lengths and computes their mean value (<code>avg</code>) and standard deviation (<code>stdev</code>), as well as the value <code>avg + k*stdev</code> (where <code>k</code> a float). This value gets propagated through the NM-RM heartbeats to all NMs, who should respect that value by dequeuing containers (if required), as long as their current queue length is between a <code>queue_min_length</code> and a <code>queue_max_length</code> value (these values are used to avoid dequeuing tasks from very short queues and to aggressively dequeue tasks from long queues, respectively). The parameters <code>k</code>, <code>queue_min_length</code> and <code>queue_max_length</code> can be specified as follows:</p>
<tdalign="left"> The number of threads that are used at each NM for serving the interceptors register to the <code>AMRMProxyService</code> by different jobs. </td>
<tdalign="left"><code>3</code></td></tr>
</tbody>
</table></section><section>
<h2><aname="Items_for_Future_Work"></a>Items for Future Work</h2>
<p>Here we describe multiple ways in which we can extend/enhance the allocation and execution of opportunistic containers. We also provide the JIRAs that track each item.</p>
<ul>
<li><b>Resource overcommitment</b> (<code>YARN-1011</code>). As already discussed, in order to further improve the cluster resource utilization, we can schedule containers not based on the allocated resources but on the actually utilized ones. When over-committing resources, there is the risk of running out of resources in case we have an increase in the utilized resources of the already running containers. Therefore, opportunistic execution should be used for containers whose allocation goes beyond the capacity of a node. This way, we can choose opportunistic containers to kill for reclaiming resources.</li>
<li><b>NM Queue reordering</b> (<code>YARN-5886</code>). Instead of executing queued containers in a FIFO order, we can employ reordering strategies that dynamically determine which opportunistic container will be executed next. For example, we can prioritize containers that are expected to be short-running or which belong to applications that are close to completion.</li>
<li><b>Out of order killing at NMs</b> (<code>YARN-5887</code>). As described above, when we need to free up resources for a guaranteed container to start its execution, we kill opportunistic containers in reverse order of arrival (first the most recently started ones). This might not always be the right decision. For example, we might want to minimize the number of containers killed or to refrain from killing containers of jobs that are very close to completion.</li>
<li><b>Container pausing</b> (<code>YARN-5292</code>): At the moment we kill opportunistic containers to make room for guaranteed in case of resource contention. In busy clusters this can lower the effective cluster utilization: whenever we kill a running opportunistic container, it has to be restarted, and thus we lose work. To this end, we can instead pause running opportunistic containers. Note that this will require support from the container executor (e.g., the container technology used) and from the application.</li>
<li><b>Container promotion</b> (<code>YARN-5085</code>). There are cases where changing the execution type of a container during its execution can be beneficial. For instance, an application might submit a container as opportunistic, and when its execution starts, it can request its promotion to a guaranteed container to avoid it getting killed.</li>
