<ahref="http://maven.apache.org/"title="Built by Maven"class="poweredBy">
<imgalt="Built by Maven"src="../images/logos/maven-feather.png"/>
</a>
</div>
</div>
<divid="bodyColumn">
<divid="contentBox">
<!---
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<h1>Using the Hadoop Service Registry</h1>
<p>The Hadoop service registry can be used in a number of ways :-</p>
<olstyle="list-style-type: decimal">
<li>To register dynamic YARN-deployed applications with entries that match the lifespan of the YARN application. Service Records can be set to be deleted on the completion of the YARN application, the application attempt, or an individual container.</li>
<li>To look up static or dynamic applications and the mechanisms to communicate with them. Those mechanisms can include: HTTP(S) URLs, Zookeeper paths, hostnames and ports and even paths in a Hadoop filesystem to configuration data.</li>
<li>On a secure cluster, to verify that a service binding has been published by a specific user, or a system account. This can be done simply by looking at the path under which an entry has been placed.</li>
<li>To register static applications. These will remain in the registry until deleted. They can be updated as required.</li>
</ol>
<p>A user of the registry may be both a publisher of entries —Service Records— and a consumer of other services located via their service records. Different parts of a distributed application may also use it for different purposes. As an example, the Application Master of a YARN application can publish bindings for use by its worker containers. The code running in the containers which can then look up the bindings to communicate with that manager even if it was restarted on different nodes in the cluster. Client applications can look up external service endpoints to interact with the AM via a public API.</p>
<p>The registry cannot be used:-</p>
<ul>
<li>To subscribe to service records or registry paths and listen for changes.</li>
<li>To directly share arbitrary data from a server for their clients. Such data must be published by some other means, a means which the registry entry can publish.</li>
<li>To share secrets between processes. The registry is world readable.</li>
<h3><aname="Short-lived_YARN_Application_Masters_registering_their_public_service_endpoints."></a>Short-lived YARN Application Masters registering their public service endpoints.</h3>
<olstyle="list-style-type: decimal">
<li>A YARN application is deployed. In a secure cluster, it is given the kerberos token to write to the registry.</li>
<li>When launched, it creates a service record at a known path</li>
<li>This record MAY have application attempt persistence policy of and an ID of the application attempt
<p>This means that the record will be deleted when the application attempt completes, even if a new attempt is created. Every Application attempt will have to re-register the endpoint —which may be needed to locate the service anyway.</p></li>
<li>Alternatively, the record MAY have the persistence policy of “application”:
<p>This means that the record will persist even between application attempts, albeit with out of date endpoint information.</p></li>
<li>Client applications look up the service by way of the path.</li>
</ol>
<p>The choice of path is an application specific one. For services with a YARN application name guaranteed to be unique, we recommend a convention of:</p>
<p>The latter makes mapping a YARN application listing entry to a service record trivial.</p>
<p>Client applications may locate the service</p>
<ul>
<li>By enumerating all instances of a service class and selecting one by specific critera.</li>
<li>From a supplied service class and instance name</li>
<li>If listed by application ID, from the service class and application ID.</li>
</ul>
<p>After locating a service record, the client can enumerate the <code>external</code> bindings and locate the entry with the desired API.</p></section><section>
<h3><aname="YARN_Containers_registering_their_public_service_endpoints"></a>YARN Containers registering their public service endpoints</h3>
<p>Here all containers in a YARN application are publishing service endpoints for public consumption.</p>
<olstyle="list-style-type: decimal">
<li>The deployed containers are passed the base path under which they should register themselves.</li>
<li>Long-lived containers must be passed an <code>id:password</code> pair which gives them the right to update these entries without the kerberos credentials of the user. This allows the containers to update their entries even after the user tokens granting the AM write access to a registry path expire.</li>
<li>The containers instantiate a registry operations instance with the <code>id:password</code> pair.</li>
<li>They then a register service record on a path consisting of:
<p>This record should have the container persistence policy an ID of the container</p>
<divclass="source">
<divclass="source">
<pre>yarn:persistence = "container"
yarn:id = containerId
</pre></div></div>
<p>When the container is terminated, the entry will be automatically deleted.</p></li>
<li>
<p>The exported service endpoints of this container-deployed service should be listed in the <code>external</code> endpoint list of the service record.</p>
</li>
<li>Clients can enumerate all containers exported by a YARN application by listing the entries under <code>${base-path}</code>.</li>
<p>Services which are generally fixed in a cluster, but which need to publish binding and configuration information may be published in the registry. Example: an Apache Oozie service. Services external to the cluster to which deployed applications may also be published. Example: An Amazon Dynamo instance.</p>
<p>These services can be registered under paths which belong to the users running the service, such as <code>/users/oozie</code> or <code>/users/hbase</code>. Client applications would use this path. While this can authenticate the validity of the service record, it does rely on the client applications knowing the username a service is deployed on, or being configured with the full path.</p>
<p>The alternative is for the services to be deployed under a static services path, under <code>/services</code>. For example, <code>/services/oozie</code> could contain the registration of the Oozie service. As the permissions for this path are restricted to pre-configured system accounts, the presence of a service registration on this path on a secure cluster, confirms that it was registered by the cluster administration tools.</p>
<olstyle="list-style-type: decimal">
<li>The service is deployed by some management tool, or directly by the cluster operator.</li>
<li>The deployed application can register itself under its own user name if given the binding information for the registry.</li>
<li>If the application is to be registered under <code>/services</code> and it has been deployed by one of the system user accounts —it may register itself directly.</li>
<li>If the application does not have the permissions to do so, the cluster administration tools must register the service instead.</li>
<li>Client applications may locate a service by resolving its well known/configured path.</li>
<li>If a service is stopped, the administration tools may delete the entry, or retain the entry but delete all it service endpoints. This is a proposed convention to indicate “the service is known but not currently reachable”.</li>
<li>When a service is restarted, its binding information may be updated, or its entire registry entry recreated.</li>
</ol></section><section>
<h3><aname="YARN_containers_locating_their_Application_Master"></a>YARN containers locating their Application Master</h3>
<p>Here YARN containers register with their AM to receive work, usually by some heartbeat mechanism where they report in regularly. If the AM is configured for containers to outlive the application attempt, when an AM fails the containers keep running. These containers will need to bind to any restarted AM. They may also wish to conclude that if an AM does not restart, that they should eventually time out and terminate themselves. Such a policy helps the application react to network partitions.</p>
<olstyle="list-style-type: decimal">
<li>The YARN AM publishes its service endpoints such as the FQDN and socket port needed for IPC communications, or an HTTP/HTTPS URL needed for a REST channel. These are published in the <code>internal</code> endpoint list, with the <code>api</code> field set to a URL of the specific API the containers use.</li>
<li>The YARN containers are launched with the path to the service record (somehow) passed to them. Environment variables or command line parameters are two viable mechanisms. Shared secrets should also be passed that way: command line parameters are visible in the unix <code>ps</code> command. More secure is saving shared secrets to the cluster filesystem, passing down the path to the containers. The URI to such as path MAY be one of the registered internal endpoints of the application.</li>
<li>The YARN containers look up the service registry to identify the communications binding.</li>
<li>If the registered service entry cannot be found, the container MAY do one of: exit. spin with some (jittered) retry period, polling for the entry, until the entry reappears. This implies that the AM has been found.</li>
<li>If the service entry is found, the client should attempt to communicate with the AM on its channel. Shared authentication details may be used to validate the client with the server and vice versa.</li>
<li>The client report in to the AM until the connections start failing to connect or authenticate, or when a long lived connection is broken and cannot be restarted.</li>
<li>A this point the client may revert to step (3). Again, some backoff policy with some jitter helps stop a newly-restarted AM being overloaded. Containers may also with to have some timeout after which they conclude that the AM is not coming back and exit.</li>
<li>We recommend that alongside the functional commands that an AM may issue to a client, a “terminate” command can be issued to a container. This allows the system to handle the specific situation of the YARN Node Manager terminating while spawned containers keep running.</li>
</ol></section><section>
<h3><aname="YARN_Applications_and_containers_publishing_their_management_and_metrics_bindings"></a>YARN Applications and containers publishing their management and metrics bindings</h3>
<p>Management ports and bindings are simply others endpoint to publish. These should be published as <i>internal</i> endpoints, as they are not intended for public consumption.</p></section><section>
<h3><aname="Client_application_enumerating_services_by_endpoint_APIs"></a>Client application enumerating services by endpoint APIs</h3>
<p>A client application wishes to locate all services implementing a specific API, such as <code>"classpath://org.apache.hbase"</code></p>
<olstyle="list-style-type: decimal">
<li>The client starts from a path in the registry</li>
<li>The client calls <code>registryOperations.list(path)</code> to list all nodes directly under that path, getting a relative list of child nodes.</li>
<li>the client enumerates the child record statuses by calling <code>stat()</code> on each child.</li>
<li>For all status entries, if the size of the entry is greater than the value of <code>ServiceRecordHeader.getLength()</code>, it MAY contain a service record.</li>
<li>The contents can be retrieved using the <code>resolve()</code> operation. If successful, it does contain a service record —so the client can enumerate the <code>external</code> endpoints and locate the one with the desired API.</li>
<li>The <code>children</code> field of each <code>RegistryPathStatus</code> status entry should be examined. If it is >= 0, the enumeration should be performed recursively on the path of that entry.</li>
<li>The operation ultimately completes with a list of all entries.</li>
<li>One of the enumerated endpoints may be selected and used as the binding information for a service</li>
</ol>
<p>This algorithm describes a depth first search of the registry tree. Variations are of course possible, including breadth-first search, or immediately halting the search as soon as a single entry point. There is also the option of parallel searches of different subtrees —this may reduce search time, albeit at the price of a higher client load on the registry infrastructure.</p>
<p>A utility class <code>RegistryUtils</code> provides static utility methods for common registry operations,in particular, <code>RegistryUtils.listServiceRecords(registryOperations, path)</code> performs the listing and collection of all immediate child record entries of a specified path.</p>
<p>Client applications are left with the problem of “what to do when the endpoint is not valid”, specifically, when a service is not running —what should be done?</p>
<p>Some transports assume that the outage is transient, and that spinning retries against the original binding is the correct strategy. This is the default policy of the Hadoop IPC client.</p>
<p>Other transports fail fast, immediately reporting the failure via an exception or other mechanism. This is directly visible to the client —but does allow the client to rescan the registry and rebind to the application.</p>
<p>Finally, some application have been designed for dynamic failover from the outset: their published binding information is actually a zookeeper path. Apache HBase and Apache Accumulo are examples of this. The registry is used for the initial lookup of the binding, after which the clients are inherently resilient to failure.</p></section></section>
