From ea5200d922fae31749ed3985f5d03f1f6c275751 Mon Sep 17 00:00:00 2001 From: Thomas White Date: Fri, 4 Jun 2010 16:34:18 +0000 Subject: [PATCH] HADOOP-6738. Move cluster_setup.xml, hod_scheduler, commands_manual from MapReduce to Common. git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@951480 13f79535-47bb-0310-9956-ffa450edef68 --- CHANGES.txt | 3 + .../content/xdocs/cluster_setup.xml | 536 ++++-- .../content/xdocs/commands_manual.xml | 772 +++++++++ .../content/xdocs/hod_scheduler.xml | 1445 +++++++++++++++++ .../content/xdocs/single_node_setup.xml | 2 +- .../src/documentation/content/xdocs/site.xml | 11 + 6 files changed, 2633 insertions(+), 136 deletions(-) create mode 100644 src/docs/src/documentation/content/xdocs/commands_manual.xml create mode 100644 src/docs/src/documentation/content/xdocs/hod_scheduler.xml diff --git a/CHANGES.txt b/CHANGES.txt index aed97ec79fe..4ef678179f6 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -932,6 +932,9 @@ Release 0.21.0 - Unreleased HADOOP-6585. Add FileStatus#isDirectory and isFile. (Eli Collins via tomwhite) + HADOOP-6738. Move cluster_setup.xml from MapReduce to Common. + (Tom White via tomwhite) + OPTIMIZATIONS HADOOP-5595. NameNode does not need to run a replicator to choose a diff --git a/src/docs/src/documentation/content/xdocs/cluster_setup.xml b/src/docs/src/documentation/content/xdocs/cluster_setup.xml index 2fbc8cacbbc..7c52eebfc0c 100644 --- a/src/docs/src/documentation/content/xdocs/cluster_setup.xml +++ b/src/docs/src/documentation/content/xdocs/cluster_setup.xml @@ -33,20 +33,20 @@ Hadoop clusters ranging from a few nodes to extremely large clusters with thousands of nodes.

- To play with Hadoop, you may first want to install Hadoop on a single machine (see Single Node Setup). + To play with Hadoop, you may first want to install Hadoop on a single machine (see Hadoop Quick Start).

- Prerequisites + Pre-requisites
  1. - Make sure all required software + Make sure all requisite software is installed on all nodes in your cluster.
  2. - Download the Hadoop software. + Get the Hadoop software.
@@ -81,21 +81,23 @@
  1. Read-only default configuration - - src/common/common-default.xml, - src/hdfs/hdfs-default.xml and - src/mapred/mapred-default.xml. + src/core/core-default.xml, + src/hdfs/hdfs-default.xml, + src/mapred/mapred-default.xml and + conf/mapred-queues.xml.template.
  2. Site-specific configuration - - conf/core-site.xml, - conf/hdfs-site.xml and - conf/mapred-site.xml. + conf/core-site.xml, + conf/hdfs-site.xml, + conf/mapred-site.xml and + conf/mapred-queues.xml.

To learn more about how the Hadoop framework is controlled by these - configuration files see - Class Configuration.

+ configuration files, look + here.

Additionally, you can control the Hadoop scripts found in the bin/ directory of the distribution, by setting site-specific @@ -163,9 +165,8 @@ Configuring the Hadoop Daemons

This section deals with important parameters to be specified in the - following: -
- conf/core-site.xml:

+ following:

+

conf/core-site.xml:

@@ -180,7 +181,7 @@
-


conf/hdfs-site.xml:

+

conf/hdfs-site.xml:

@@ -212,7 +213,7 @@
-


conf/mapred-site.xml:

+

conf/mapred-site.xml:

@@ -221,12 +222,12 @@ - + - + - + - + - - + + - - - - - -
Notes
mapred.job.trackermapreduce.jobtracker.address Host or IP and port of JobTracker. host:port pair.
mapred.system.dirmapreduce.jobtracker.system.dir Path on the HDFS where where the Map/Reduce framework stores system files e.g. /hadoop/mapred/system/. @@ -237,7 +238,7 @@
mapred.local.dirmapreduce.cluster.local.dir Comma-separated list of paths on the local filesystem where temporary Map/Reduce data is written. @@ -264,7 +265,7 @@
mapred.hosts/mapred.hosts.excludemapreduce.jobtracker.hosts.filename/mapreduce.jobtracker.hosts.exclude.filename List of permitted/excluded TaskTrackers. If necessary, use these files to control the list of allowable @@ -272,82 +273,331 @@
mapred.queue.namesComma separated list of queues to which jobs can be submitted.mapreduce.cluster.job-authorization-enabledBoolean, specifying whether job ACLs are supported for + authorizing view and modification of a job - The Map/Reduce system always supports atleast one queue - with the name as default. Hence, this parameter's - value should always contain the string default. - Some job schedulers supported in Hadoop, like the - Capacity Scheduler, - support multiple queues. If such a scheduler is - being used, the list of configured queue names must be - specified here. Once queues are defined, users can submit - jobs to a queue using the property name - mapred.job.queue.name in the job configuration. - There could be a separate - configuration file for configuring properties of these - queues that is managed by the scheduler. - Refer to the documentation of the scheduler for information on - the same. + If true, job ACLs would be checked while viewing or + modifying a job. More details are available at + Job Authorization.
mapred.acls.enabledSpecifies whether ACLs are supported for controlling job - submission and administration - If true, ACLs would be checked while submitting - and administering jobs. ACLs can be specified using the - configuration parameters of the form - mapred.queue.queue-name.acl-name, defined below. -
- -


conf/mapred-queue-acls.xml

- - - - - - - - - - - - - - - - - -
ParameterValueNotes
mapred.queue.queue-name.acl-submit-jobList of users and groups that can submit jobs to the - specified queue-name. - The list of users and groups are both comma separated - list of names. The two lists are separated by a blank. - Example: user1,user2 group1,group2. - If you wish to define only a list of groups, provide - a blank at the beginning of the value. -
mapred.queue.queue-name.acl-administer-jobList of users and groups that can change the priority - or kill jobs that have been submitted to the - specified queue-name. - The list of users and groups are both comma separated - list of names. The two lists are separated by a blank. - Example: user1,user2 group1,group2. - If you wish to define only a list of groups, provide - a blank at the beginning of the value. Note that an - owner of a job can always change the priority or kill - his/her own job, irrespective of the ACLs. -
- + +

Typically all the above parameters are marked as final to ensure that they cannot be overriden by user-applications.

+

conf/mapred-queues.xml + :

+

This file is used to configure the queues in the Map/Reduce + system. Queues are abstract entities in the JobTracker that can be + used to manage collections of jobs. They provide a way for + administrators to organize jobs in specific ways and to enforce + certain policies on such collections, thus providing varying + levels of administrative control and management functions on jobs. +

+

One can imagine the following sample scenarios:

+
    +
  • Jobs submitted by a particular group of users can all be + submitted to one queue.
    • +
  • Long running jobs in an organization can be submitted to a + queue.
    • +
  • Short running jobs can be submitted to a queue and the number + of jobs that can run concurrently can be restricted.
    • +
+

The usage of queues is closely tied to the scheduler configured + at the JobTracker via mapreduce.jobtracker.taskscheduler. + The degree of support of queues depends on the scheduler used. Some + schedulers support a single queue, while others support more complex + configurations. Schedulers also implement the policies that apply + to jobs in a queue. Some schedulers, such as the Fairshare scheduler, + implement their own mechanisms for collections of jobs and do not rely + on queues provided by the framework. The administrators are + encouraged to refer to the documentation of the scheduler they are + interested in for determining the level of support for queues.

+

The Map/Reduce framework supports some basic operations on queues + such as job submission to a specific queue, access control for queues, + queue states, viewing configured queues and their properties + and refresh of queue properties. In order to fully implement some of + these operations, the framework takes the help of the configured + scheduler.

+

The following types of queue configurations are possible:

+
    +
  • Single queue: The default configuration in Map/Reduce comprises + of a single queue, as supported by the default scheduler. All jobs + are submitted to this default queue which maintains jobs in a priority + based FIFO order.
    • +
  • Multiple single level queues: Multiple queues are defined, and + jobs can be submitted to any of these queues. Different policies + can be applied to these queues by schedulers that support this + configuration to provide a better level of support. For example, + the capacity scheduler + provides ways of configuring different + capacity and fairness guarantees on these queues.
    • +
  • Hierarchical queues: Hierarchical queues are a configuration in + which queues can contain other queues within them recursively. The + queues that contain other queues are referred to as + container queues. Queues that do not contain other queues are + referred as leaf or job queues. Jobs can only be submitted to leaf + queues. Hierarchical queues can potentially offer a higher level + of control to administrators, as schedulers can now build a + hierarchy of policies where policies applicable to a container + queue can provide context for policies applicable to queues it + contains. It also opens up possibilities for delegating queue + administration where administration of queues in a container queue + can be turned over to a different set of administrators, within + the context provided by the container queue. For example, the + capacity scheduler + uses hierarchical queues to partition capacity of a cluster + among container queues, and allowing queues they contain to divide + that capacity in more ways.
    • +
+ +

Most of the configuration of the queues can be refreshed/reloaded + without restarting the Map/Reduce sub-system by editing this + configuration file as described in the section on + reloading queue + configuration. + Not all configuration properties can be reloaded of course, + as will description of each property below explain.

+ +

The format of conf/mapred-queues.xml is different from the other + configuration files, supporting nested configuration + elements to support hierarchical queues. The format is as follows: +

+ + + <queues aclsEnabled="$aclsEnabled"> + <queue> + <name>$queue-name</name> + <state>$state</state> + <queue> + <name>$child-queue1</name> + <properties> + <property key="$key" value="$value"/> + ... + </properties> + <queue> + <name>$grand-child-queue1</name> + ... + </queue> + </queue> + <queue> + <name>$child-queue2</name> + ... + </queue> + ... + ... + ... + <queue> + <name>$leaf-queue</name> + <acl-submit-job>$acls</acl-submit-job> + <acl-administer-jobs>$acls</acl-administer-jobs> + <properties> + <property key="$key" value="$value"/> + ... + </properties> + </queue> + </queue> + </queues> + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Tag/AttributeValue + Refresh-able? + Notes
queuesRoot element of the configuration file.Not-applicableAll the queues are nested inside this root element of the + file. There can be only one root queues element in the file.
aclsEnabledBoolean attribute to the + <queues> tag + specifying whether ACLs are supported for controlling job + submission and administration for all the queues + configured. + YesIf false, ACLs are ignored for all the + configured queues.

+ If true, the user and group details of the user + are checked against the configured ACLs of the corresponding + job-queue while submitting and administering jobs. ACLs can be + specified for each queue using the queue-specific tags + "acl-$acl_name", defined below. ACLs are checked only against + the job-queues, i.e. the leaf-level queues; ACLs configured + for the rest of the queues in the hierarchy are ignored. +
queueA child element of the + <queues> tag or another + <queue>. Denotes a queue + in the system. + Not applicableQueues can be hierarchical and so this element can contain + children of this same type.
nameChild element of a + <queue> specifying the + name of the queue.NoName of the queue cannot contain the character ":" + which is reserved as the queue-name delimiter when addressing a + queue in a hierarchy.
stateChild element of a + <queue> specifying the + state of the queue. + YesEach queue has a corresponding state. A queue in + 'running' state can accept new jobs, while a queue in + 'stopped' state will stop accepting any new jobs. State + is defined and respected by the framework only for the + leaf-level queues and is ignored for all other queues. +

+ The state of the queue can be viewed from the command line using + 'bin/mapred queue' command and also on the the Web + UI.

+ Administrators can stop and start queues at runtime using the + feature of reloading + queue configuration. If a queue is stopped at runtime, it + will complete all the existing running jobs and will stop + accepting any new jobs. +
acl-submit-jobChild element of a + <queue> specifying the + list of users and groups that can submit jobs to the specified + queue.Yes + Applicable only to leaf-queues.

+ The list of users and groups are both comma separated + list of names. The two lists are separated by a blank. + Example: user1,user2 group1,group2. + If you wish to define only a list of groups, provide + a blank at the beginning of the value. +

+
acl-administer-jobChild element of a + <queue> specifying the + list of users and groups that can change the priority of a job + or kill a job that has been submitted to the specified queue. + Yes + Applicable only to leaf-queues.

+ The list of users and groups are both comma separated + list of names. The two lists are separated by a blank. + Example: user1,user2 group1,group2. + If you wish to define only a list of groups, provide + a blank at the beginning of the value. Note that an + owner of a job can always change the priority or kill + his/her own job, irrespective of the ACLs. +
propertiesChild element of a + <queue> specifying the + scheduler specific properties.Not applicableThe scheduler specific properties are the children of this + element specified as a group of <property> tags described + below. The JobTracker completely ignores these properties. These + can be used as per-queue properties needed by the scheduler + being configured. Please look at the scheduler specific + documentation as to how these properties are used by that + particular scheduler. +
propertyChild element of + <properties> for a + specific queue.Not applicableA single scheduler specific queue-property. Ignored by + the JobTracker and used by the scheduler that is configured.
keyAttribute of a + <property> for a + specific queue.Scheduler-specificThe name of a single scheduler specific queue-property.
valueAttribute of a + <property> for a + specific queue.Scheduler-specificThe value of a single scheduler specific queue-property. + The value can be anything that is left for the proper + interpretation by the scheduler that is configured.
+ +

Once the queues are configured properly and the Map/Reduce + system is up and running, from the command line one can + get the list + of queues and + obtain + information specific to each queue. This information is also + available from the web UI. On the web UI, queue information can be + seen by going to queueinfo.jsp, linked to from the queues table-cell + in the cluster-summary table. The queueinfo.jsp prints the hierarchy + of queues as well as the specific information for each queue. +

+ +

Users can submit jobs only to a + leaf-level queue by specifying the fully-qualified queue-name for + the property name mapreduce.job.queuename in the job + configuration. The character ':' is the queue-name delimiter and so, + for e.g., if one wants to submit to a configured job-queue 'Queue-C' + which is one of the sub-queues of 'Queue-B' which in-turn is a + sub-queue of 'Queue-A', then the job configuration should contain + property mapreduce.job.queuename set to the + <value>Queue-A:Queue-B:Queue-C</value>

+
Real-World Cluster Configurations @@ -383,7 +633,7 @@ conf/mapred-site.xml - mapred.reduce.parallel.copies + mapreduce.reduce.shuffle.parallelcopies 20 Higher number of parallel copies run by reduces to fetch @@ -392,7 +642,7 @@ conf/mapred-site.xml - mapred.map.child.java.opts + mapreduce.map.java.opts -Xmx512M Larger heap-size for child jvms of maps. @@ -400,7 +650,7 @@ conf/mapred-site.xml - mapred.reduce.child.java.opts + mapreduce.reduce.java.opts -Xmx512M Larger heap-size for child jvms of reduces. @@ -417,13 +667,13 @@ conf/core-site.xml - io.sort.factor + mapreduce.task.io.sort.factor 100 More streams merged at once while sorting files. conf/core-site.xml - io.sort.mb + mapreduce.task.io.sort.mb 200 Higher memory-limit while sorting data. @@ -448,7 +698,7 @@ conf/mapred-site.xml - mapred.job.tracker.handler.count + mapreduce.jobtracker.handler.count 60 More JobTracker server threads to handle RPCs from large @@ -457,13 +707,13 @@ conf/mapred-site.xml - mapred.reduce.parallel.copies + mapreduce.reduce.shuffle.parallelcopies 50 conf/mapred-site.xml - tasktracker.http.threads + mapreduce.tasktracker.http.threads 50 More worker threads for the TaskTracker's http server. The @@ -473,7 +723,7 @@ conf/mapred-site.xml - mapred.map.child.java.opts + mapreduce.map.java.opts -Xmx512M Larger heap-size for child jvms of maps. @@ -481,7 +731,7 @@ conf/mapred-site.xml - mapred.reduce.child.java.opts + mapreduce.reduce.java.opts -Xmx1024M Larger heap-size for child jvms of reduces. @@ -500,11 +750,11 @@ or equal to the -Xmx passed to JavaVM, else the VM might not start.

-

Note: mapred.child.java.opts are used only for +

Note: mapred.{map|reduce}.child.java.opts are used only for configuring the launched child tasks from task tracker. Configuring - the memory options for daemons is documented under + the memory options for daemons is documented in - Configuring the Environment of the Hadoop Daemons.

+ cluster_setup.html

The memory available to some parts of the framework is also configurable. In map and reduce tasks, performance may be influenced @@ -558,7 +808,7 @@ - +
NameTypeDescription
mapred.tasktracker.taskmemorymanager.monitoring-interval
mapreduce.tasktracker.taskmemorymanager.monitoringinterval long The time interval, in milliseconds, between which the TT checks for any memory violation. The default value is 5000 msec @@ -668,10 +918,11 @@ the tasks. For maximum security, this task controller sets up restricted permissions and user/group ownership of local files and directories used by the tasks such as the - job jar files, intermediate files and task log files. Currently - permissions on distributed cache files are opened up to be - accessible by all users. In future, it is expected that stricter - file permissions are set for these files too. + job jar files, intermediate files, task log files and distributed + cache files. Particularly note that, because of this, except the + job owner and tasktracker, no other user can access any of the + local files/directories including those localized as part of the + distributed cache.
@@ -684,7 +935,7 @@ PropertyValueNotes - mapred.task.tracker.task-controller + mapreduce.tasktracker.taskcontroller Fully qualified class name of the task controller class Currently there are two implementations of task controller in the Hadoop system, DefaultTaskController and LinuxTaskController. @@ -715,21 +966,35 @@

The executable must have specific permissions as follows. The executable should have 6050 or --Sr-s--- permissions - user-owned by root(super-user) and group-owned by a group - of which only the TaskTracker's user is the sole group member. + user-owned by root(super-user) and group-owned by a special group + of which the TaskTracker's user is the group member and no job + submitter is. If any job submitter belongs to this special group, + security will be compromised. This special group name should be + specified for the configuration property + "mapreduce.tasktracker.group" in both mapred-site.xml and + task-controller.cfg. For example, let's say that the TaskTracker is run as user mapred who is part of the groups users and - mapredGroup any of them being the primary group. + specialGroup any of them being the primary group. Let also be that users has both mapred and - another user X as its members, while mapredGroup - has only mapred as its member. Going by the above + another user (job submitter) X as its members, and X does + not belong to specialGroup. Going by the above description, the setuid/setgid executable should be set 6050 or --Sr-s--- with user-owner as mapred and - group-owner as mapredGroup which has - only mapred as its member(and not users which has + group-owner as specialGroup which has + mapred as its member(and not users which has X also as its member besides mapred).

+ +

+ The LinuxTaskController requires that paths including and leading up + to the directories specified in + mapreduce.cluster.local.dir and hadoop.log.dir to + be set 755 permissions. +

+
+ task-controller.cfg

The executable requires a configuration file called taskcontroller.cfg to be present in the configuration directory passed to the ant target @@ -747,8 +1012,8 @@

- - + @@ -760,14 +1025,16 @@ permissions on the log files so that they can be written to by the user's tasks and read by the TaskTracker for serving on the web UI. + + + +
NameDescription
mapred.local.dirPath to mapred local directories. Should be same as the value + mapreduce.cluster.local.dirPath to mapreduce.cluster.local.directories. Should be same as the value which was provided to key in mapred-site.xml. This is required to validate paths passed to the setuid executable in order to prevent arbitrary paths being passed to it.
mapreduce.tasktracker.groupGroup to which the TaskTracker belongs. The group owner of the + taskcontroller binary should be this group. Should be same as + the value with which the TaskTracker is configured. This + configuration is required for validating the secure access of the + task-controller binary.
- -

- The LinuxTaskController requires that paths including and leading up to - the directories specified in - mapred.local.dir and hadoop.log.dir to be set 755 - permissions. -

+
@@ -800,7 +1067,7 @@ monitoring script in mapred-site.xml.

- + - + - + - + @@ -857,17 +1124,17 @@ History Logging

The job history files are stored in central location - hadoop.job.history.location which can be on DFS also, + mapreduce.jobtracker.jobhistory.location which can be on DFS also, whose default value is ${HADOOP_LOG_DIR}/history. The history web UI is accessible from job tracker web UI.

The history files are also logged to user specified directory - hadoop.job.history.user.location + mapreduce.job.userhistorylocation which defaults to job output directory. The files are stored in "_logs/history/" in the specified directory. Hence, by default - they will be in "mapred.output.dir/_logs/history/". User can stop + they will be in "mapreduce.output.fileoutputformat.outputdir/_logs/history/". User can stop logging by giving the value none for - hadoop.job.history.user.location

+ mapreduce.job.userhistorylocation

User can view the history logs summary in specified directory using the following command
@@ -880,7 +1147,6 @@ $ bin/hadoop job -history all output-dir

-

Once all the necessary configuration is complete, distribute the files to the HADOOP_CONF_DIR directory on all the machines, @@ -891,9 +1157,9 @@

Map/Reduce

The job tracker restart can recover running jobs if - mapred.jobtracker.restart.recover is set true and + mapreduce.jobtracker.restart.recover is set true and JobHistory logging is enabled. Also - mapred.jobtracker.job.history.block.size value should be + mapreduce.jobtracker.jobhistory.block.size value should be set to an optimal value to dump job history to disk as soon as possible, the typical value is 3145728(3MB).

@@ -951,7 +1217,7 @@ and starts the TaskTracker daemon on all the listed slaves.

- +
Hadoop Shutdown diff --git a/src/docs/src/documentation/content/xdocs/commands_manual.xml b/src/docs/src/documentation/content/xdocs/commands_manual.xml new file mode 100644 index 00000000000..76e4df64d08 --- /dev/null +++ b/src/docs/src/documentation/content/xdocs/commands_manual.xml @@ -0,0 +1,772 @@ + + + + + +
+ Hadoop Commands Guide +
+ + +
+ Overview +

+ All Hadoop commands are invoked by the bin/hadoop script. Running the Hadoop + script without any arguments prints the description for all commands. +

+

+ Usage: hadoop [--config confdir] [COMMAND] [GENERIC_OPTIONS] [COMMAND_OPTIONS] +

+

+ Hadoop has an option parsing framework that employs parsing generic options as well as running classes. +

+
NameDescription
mapred.healthChecker.script.path
mapreduce.tasktracker.healthchecker.script.path Absolute path to the script which is periodically run by the TaskTracker to determine if the node is healthy or not. The file should be executable by the TaskTracker. @@ -809,18 +1076,18 @@ is not started.
mapred.healthChecker.intervalmapreduce.tasktracker.healthchecker.interval Frequency at which the node health script is run, in milliseconds
mapred.healthChecker.script.timeoutmapreduce.tasktracker.healthchecker.script.timeout Time after which the node health script will be killed by the TaskTracker if unresponsive. The node is marked unhealthy. if node health script times out.
mapred.healthChecker.script.argsmapreduce.tasktracker.healthchecker.script.args Extra arguments that can be passed to the node health script when launched. These should be comma separated list of arguments.
+ + + + + + + + + + + + + + +
COMMAND_OPTION Description
--config confdirOverwrites the default Configuration directory. Default is ${HADOOP_HOME}/conf.
GENERIC_OPTIONSThe common set of options supported by multiple commands.
COMMAND
COMMAND_OPTIONS		Various commands with their options are described in the following sections. The commands + have been grouped into User Commands + and Administration Commands.
+
+ Generic Options +

+ The following options are supported by dfsadmin, + fs, fsck and + job. + Applications should implement + Tool to support + + GenericOptions. +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
GENERIC_OPTION Description
-conf <configuration file>Specify an application configuration file.
-D <property=value>Use value for given property.
-fs <local|namenode:port>Specify a namenode.
-jt <local|jobtracker:port>Specify a job tracker. Applies only to job.
-files <comma separated list of files>Specify comma separated files to be copied to the map reduce cluster. + Applies only to job.
-libjars <comma seperated list of jars>Specify comma separated jar files to include in the classpath. + Applies only to job.
-archives <comma separated list of archives>Specify comma separated archives to be unarchived on the compute machines. + Applies only to job.
+
+ + +
+ User Commands +

Commands useful for users of a Hadoop cluster.

+
+ archive +

+ Creates a Hadoop archive. More information see the Hadoop Archives Guide. +

+

+ Usage: hadoop archive -archiveName NAME <src>* <dest> +

+ + + + + + + + + + + + + + +
COMMAND_OPTION Description
-archiveName NAMEName of the archive to be created.
srcFilesystem pathnames which work as usual with regular expressions.
destDestination directory which would contain the archive.
+
+ +
+ distcp +

+ Copy file or directories recursively. More information can be found at DistCp Guide. +

+

+ Usage: hadoop distcp <srcurl> <desturl> +

+ + + + + + + + + + + +
COMMAND_OPTION Description
srcurlSource Url
desturlDestination Url
+
+ +
+ fs +

+ Runs a generic filesystem user client. +

+

+ Usage: hadoop fs [GENERIC_OPTIONS] + [COMMAND_OPTIONS] +

+

+ The various COMMAND_OPTIONS can be found at + File System Shell Guide. +

+
+ +
+ fsck +

+ Runs a HDFS filesystem checking utility. See Fsck for more info. +

+

Usage: hadoop fsck [GENERIC_OPTIONS] + <path> [-move | -delete | -openforwrite] [-files [-blocks + [-locations | -racks]]]

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
<path>Start checking from this path.
-moveMove corrupted files to /lost+found
-deleteDelete corrupted files.
-openforwritePrint out files opened for write.
-filesPrint out files being checked.
-blocksPrint out block report.
-locationsPrint out locations for every block.
-racksPrint out network topology for data-node locations.
+
+ +
+ jar +

+ Runs a jar file. Users can bundle their Map Reduce code in a jar file and execute it using this command. +

+

+ Usage: hadoop jar <jar> [mainClass] args... +

+

+ The streaming jobs are run via this command. For examples, see + Hadoop Streaming. +

+

+ The WordCount example is also run using jar command. For examples, see the + MapReduce Tutorial. +

+
+ +
+ job +

+ Command to interact with Map Reduce Jobs. +

+

+ Usage: hadoop job [GENERIC_OPTIONS] + [-submit <job-file>] | [-status <job-id>] | + [-counter <job-id> <group-name> <counter-name>] | [-kill <job-id>] | + [-events <job-id> <from-event-#> <#-of-events>] | [-history [all] <historyFile>] | + [-list [all]] | [-kill-task <task-id>] | [-fail-task <task-id>] | + [-set-priority <job-id> <priority>] +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-submit <job-file>Submits the job.
-status <job-id>Prints the map and reduce completion percentage and all job counters.
-counter <job-id> <group-name> <counter-name>Prints the counter value.
-kill <job-id>Kills the job.
-events <job-id> <from-event-#> <#-of-events>Prints the events' details received by jobtracker for the given range.
-history [all] <historyFile>-history <historyFile> prints job details, failed and killed tip details. More details + about the job such as successful tasks and task attempts made for each task can be viewed by + specifying the [all] option.
-list [all]-list all displays all jobs. -list displays only jobs which are yet to complete.
-kill-task <task-id>Kills the task. Killed tasks are NOT counted against failed attempts.
-fail-task <task-id>Fails the task. Failed tasks are counted against failed attempts.
-set-priority <job-id> <priority>Changes the priority of the job. + Allowed priority values are VERY_HIGH, HIGH, NORMAL, LOW, VERY_LOW
+
+ +
+ pipes +

+ Runs a pipes job. +

+

+ Usage: hadoop pipes [-conf <path>] [-jobconf <key=value>, <key=value>, ...] + [-input <path>] [-output <path>] [-jar <jar file>] [-inputformat <class>] + [-map <class>] [-partitioner <class>] [-reduce <class>] [-writer <class>] + [-program <executable>] [-reduces <num>] +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-conf <path>Configuration for job
-jobconf <key=value>, <key=value>, ...Add/override configuration for job
-input <path>Input directory
-output <path>Output directory
-jar <jar file>Jar filename
-inputformat <class>InputFormat class
-map <class>Java Map class
-partitioner <class>Java Partitioner
-reduce <class>Java Reduce class
-writer <class>Java RecordWriter
-program <executable>Executable URI
-reduces <num>Number of reduces
+
+
+ queue +

+ command to interact and view Job Queue information +

+

+ Usage : hadoop queue [-list] | [-info <job-queue-name> [-showJobs]] | [-showacls] +

+ + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-list Gets list of Job Queues configured in the system. Along with scheduling information + associated with the job queues. +
-info <job-queue-name> [-showJobs] + Displays the job queue information and associated scheduling information of particular + job queue. If -showJobs options is present a list of jobs submitted to the particular job + queue is displayed. +
-showaclsDisplays the queue name and associated queue operations allowed for the current user. + The list consists of only those queues to which the user has access. +
+
+
+ version +

+ Prints the version. +

+

+ Usage: hadoop version +

+
+
+ CLASSNAME +

+ Hadoop script can be used to invoke any class. +

+

+ Runs the class named CLASSNAME. +

+ +

+ Usage: hadoop CLASSNAME +

+ +
+
+
+ Administration Commands +

Commands useful for administrators of a Hadoop cluster.

+
+ balancer +

+ Runs a cluster balancing utility. An administrator can simply press Ctrl-C to stop the + rebalancing process. For more details see + Rebalancer. +

+

+ Usage: hadoop balancer [-threshold <threshold>] +

+ + + + + + + +
COMMAND_OPTION Description
-threshold <threshold>Percentage of disk capacity. This overwrites the default threshold.
+
+ +
+ daemonlog +

+ Get/Set the log level for each daemon. +

+

+ Usage: hadoop daemonlog -getlevel <host:port> <name>
+ Usage: hadoop daemonlog -setlevel <host:port> <name> <level> +

+ + + + + + + + + + + +
COMMAND_OPTION Description
-getlevel <host:port> <name>Prints the log level of the daemon running at <host:port>. + This command internally connects to http://<host:port>/logLevel?log=<name>
-setlevel <host:port> <name> <level>Sets the log level of the daemon running at <host:port>. + This command internally connects to http://<host:port>/logLevel?log=<name>
+
+ +
+ datanode +

+ Runs a HDFS datanode. +

+

+ Usage: hadoop datanode [-rollback] +

+ + + + + + + +
COMMAND_OPTION Description
-rollbackRollsback the datanode to the previous version. This should be used after stopping the datanode + and distributing the old Hadoop version.
+
+ +
+ dfsadmin +

+ Runs a HDFS dfsadmin client. +

+

+ Usage: hadoop dfsadmin [GENERIC_OPTIONS] [-report] [-safemode enter | leave | get | wait] [-refreshNodes] + [-finalizeUpgrade] [-upgradeProgress status | details | force] [-metasave filename] + [-setQuota <quota> <dirname>...<dirname>] [-clrQuota <dirname>...<dirname>] + [-restoreFailedStorage true|false|check] + [-help [cmd]] +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-reportReports basic filesystem information and statistics.
-safemode enter | leave | get | waitSafe mode maintenance command. + Safe mode is a Namenode state in which it
+ 1. does not accept changes to the name space (read-only)
+ 2. does not replicate or delete blocks.
+ Safe mode is entered automatically at Namenode startup, and + leaves safe mode automatically when the configured minimum + percentage of blocks satisfies the minimum replication + condition. Safe mode can also be entered manually, but then + it can only be turned off manually as well.
-refreshNodesRe-read the hosts and exclude files to update the set + of Datanodes that are allowed to connect to the Namenode + and those that should be decommissioned or recommissioned.
-finalizeUpgradeFinalize upgrade of HDFS. + Datanodes delete their previous version working directories, + followed by Namenode doing the same. + This completes the upgrade process.
-printTopologyPrint a tree of the rack/datanode topology of the + cluster as seen by the NameNode.
-upgradeProgress status | details | forceRequest current distributed upgrade status, + a detailed status or force the upgrade to proceed.
-metasave filenameSave Namenode's primary data structures + to <filename> in the directory specified by hadoop.log.dir property. + <filename> will contain one line for each of the following
+ 1. Datanodes heart beating with Namenode
+ 2. Blocks waiting to be replicated
+ 3. Blocks currrently being replicated
+ 4. Blocks waiting to be deleted
-setQuota <quota> <dirname>...<dirname>Set the quota <quota> for each directory <dirname>. + The directory quota is a long integer that puts a hard limit on the number of names in the directory tree.
+ Best effort for the directory, with faults reported if
+ 1. N is not a positive integer, or
+ 2. user is not an administrator, or
+ 3. the directory does not exist or is a file, or
+ 4. the directory would immediately exceed the new quota.
-clrQuota <dirname>...<dirname>Clear the quota for each directory <dirname>.
+ Best effort for the directory. with fault reported if
+ 1. the directory does not exist or is a file, or
+ 2. user is not an administrator.
+ It does not fault if the directory has no quota.
-restoreFailedStorage true | false | checkThis option will turn on/off automatic attempt to restore failed storage replicas. + If a failed storage becomes available again the system will attempt to restore + edits and/or fsimage during checkpoint. 'check' option will return current setting.
-help [cmd] Displays help for the given command or all commands if none + is specified.
+
+
+ mradmin +

Runs MR admin client

+

Usage: hadoop mradmin [ + GENERIC_OPTIONS + ] [-refreshServiceAcl] [-refreshQueues] [-refreshNodes] [-help [cmd]]

+ + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-refreshServiceAcl Reload the service-level authorization policies. Jobtracker + will reload the authorization policy file.
-refreshQueues

Reload the queues' configuration at the JobTracker. + Most of the configuration of the queues can be refreshed/reloaded + without restarting the Map/Reduce sub-system. Administrators + typically own the + + conf/mapred-queues.xml + file, can edit it while the JobTracker is still running, and can do + a reload by running this command.

+

It should be noted that while trying to refresh queues' + configuration, one cannot change the hierarchy of queues itself. + This means no operation that involves a change in either the + hierarchy structure itself or the queues' names will be allowed. + Only selected properties of queues can be changed during refresh. + For example, new queues cannot be added dynamically, neither can an + existing queue be deleted.

+

If during a reload of queue configuration, + a syntactic or semantic error in made during the editing of the + configuration file, the refresh command fails with an exception that + is printed on the standard output of this command, thus informing the + requester with any helpful messages of what has gone wrong during + the edit/reload. Importantly, the existing queue configuration is + untouched and the system is left in a consistent state. +

+

As described in the + + conf/mapred-queues.xml section, the + + <properties> tag in the queue configuration file can + also be used to specify per-queue properties needed by the scheduler. + When the framework's queue configuration is reloaded using this + command, this scheduler specific configuration will also be reloaded + , provided the scheduler being configured supports this reload. + Please see the documentation of the particular scheduler in use.

+
-refreshNodes Refresh the hosts information at the jobtracker.
-help [cmd]Displays help for the given command or all commands if none + is specified.
+
+
+ jobtracker +

+ Runs the MapReduce job Tracker node. +

+

+ Usage: hadoop jobtracker [-dumpConfiguration] +

+ + + + + + + + +
COMMAND_OPTION Description
-dumpConfiguration Dumps the configuration used by the JobTracker alongwith queue + configuration in JSON format into Standard output used by the + jobtracker and exits.
+ +
+ +
+ namenode +

+ Runs the namenode. For more information about upgrade, rollback and finalize see + Upgrade and Rollback. +

+

+ Usage: hadoop namenode [-format] | [-upgrade] | [-rollback] | [-finalize] | [-importCheckpoint] | [-checkpoint] | [-backup] +

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
COMMAND_OPTION Description
-regularStart namenode in standard, active role rather than as backup or checkpoint node. This is the default role.
-checkpointStart namenode in checkpoint role, creating periodic checkpoints of the active namenode metadata.
-backupStart namenode in backup role, maintaining an up-to-date in-memory copy of the namespace and creating periodic checkpoints.
-formatFormats the namenode. It starts the namenode, formats it and then shut it down.
-upgradeNamenode should be started with upgrade option after the distribution of new Hadoop version.
-rollbackRollsback the namenode to the previous version. This should be used after stopping the cluster + and distributing the old Hadoop version.
-finalizeFinalize will remove the previous state of the files system. Recent upgrade will become permanent. + Rollback option will not be available anymore. After finalization it shuts the namenode down.
-importCheckpointLoads image from a checkpoint directory and saves it into the current one. Checkpoint directory + is read from property fs.checkpoint.dir + (see Import Checkpoint). +
-checkpointEnables checkpointing + (see Checkpoint Node).
-backupEnables checkpointing and maintains an in-memory, up-to-date copy of the file system namespace + (see Backup Node).
+
+ +
+ secondarynamenode + + The Secondary NameNode has been deprecated. Instead, consider using the + Checkpoint Node or + Backup Node. + +

+ Runs the HDFS secondary + namenode. See Secondary NameNode + for more info. +

+

+ Usage: hadoop secondarynamenode [-checkpoint [force]] | [-geteditsize] +

+ + + + + + + + + + + +
COMMAND_OPTION Description
-checkpoint [force]Checkpoints the Secondary namenode if EditLog size >= fs.checkpoint.size. + If -force is used, checkpoint irrespective of EditLog size.
-geteditsizePrints the EditLog size.
+
+ +
+ tasktracker +

+ Runs a MapReduce task Tracker node. +

+

+ Usage: hadoop tasktracker +

+
+ +
+ + + + + + diff --git a/src/docs/src/documentation/content/xdocs/hod_scheduler.xml b/src/docs/src/documentation/content/xdocs/hod_scheduler.xml new file mode 100644 index 00000000000..db0e94bf2d6 --- /dev/null +++ b/src/docs/src/documentation/content/xdocs/hod_scheduler.xml @@ -0,0 +1,1445 @@ + + + + + +
+ + HOD Scheduler + +
+ + + + + +
+Introduction +

Hadoop On Demand (HOD) is a system for provisioning and managing independent Hadoop MapReduce and +Hadoop Distributed File System (HDFS) instances on a shared cluster of nodes. HOD is a tool that makes it easy +for administrators and users to quickly setup and use Hadoop. HOD is also a very useful tool for Hadoop developers +and testers who need to share a physical cluster for testing their own Hadoop versions.

+ +

HOD uses the Torque resource manager to do node allocation. On the allocated nodes, it can start Hadoop +MapReduce and HDFS daemons. It automatically generates the appropriate configuration files (hadoop-site.xml) +for the Hadoop daemons and client. HOD also has the capability to distribute Hadoop to the nodes in the virtual +cluster that it allocates. HOD supports Hadoop from version 0.15 onwards.

+
+ +
+ HOD Users +

This section shows users how to get started using HOD, reviews various HOD features and command line options, + and provides detailed troubleshooting help.

+ +
+ Getting Started +

In this section, we shall see a step-by-step introduction on how to use HOD for the most basic operations. Before + following these steps, it is assumed that HOD and its dependent hardware and software components are setup and + configured correctly. This is a step that is generally performed by system administrators of the cluster.

+ +

The HOD user interface is a command line utility called hod. It is driven by a configuration file, + that is typically setup for users by system administrators. Users can override this configuration when using + the hod, which is described later in this documentation. The configuration file can be specified in + two ways when using hod, as described below:

+
    +
  • Specify it on command line, using the -c option. Such as + hod <operation> <required-args> -c path-to-the-configuration-file [other-options]
    • +
  • Set up an environment variable HOD_CONF_DIR where hod will be run. + This should be pointed to a directory on the local file system, containing a file called hodrc. + Note that this is analogous to the HADOOP_CONF_DIR and hadoop-site.xml file for Hadoop. + If no configuration file is specified on the command line, hod shall look for the HOD_CONF_DIR + environment variable and a hodrc file under that.
    • +
+

In examples listed below, we shall not explicitly point to the configuration option, assuming it is correctly specified.

+ +
A Typical HOD Session +

A typical session of HOD will involve at least three steps: allocate, run hadoop jobs, deallocate. In order to do this, + perform the following steps.

+ +

Create a Cluster Directory

+ +

The cluster directory is a directory on the local file system where hod will generate the + Hadoop configuration, hadoop-site.xml, corresponding to the cluster it allocates. Pass this directory to the + hod operations as stated below. If the cluster directory passed doesn't already exist, HOD will automatically + try to create it and use it. Once a cluster is allocated, a user can utilize it to run Hadoop jobs by specifying the cluster + directory as the Hadoop --config option.

+ +

Operation allocate

+ +

The allocate operation is used to allocate a set of nodes and install and provision Hadoop on them. + It has the following syntax. Note that it requires a cluster_dir ( -d, --hod.clusterdir) and the number of nodes + (-n, --hod.nodecount) needed to be allocated:

+ + $ hod allocate -d cluster_dir -n number_of_nodes [OPTIONS] + +

If the command completes successfully, then cluster_dir/hadoop-site.xml will be generated and + will contain information about the allocated cluster. It will also print out the information about the Hadoop web UIs.

+ +

An example run of this command produces the following output. Note in this example that ~/hod-clusters/test + is the cluster directory, and we are allocating 5 nodes:

+ + +$ hod allocate -d ~/hod-clusters/test -n 5 +INFO - HDFS UI on http://foo1.bar.com:53422 +INFO - Mapred UI on http://foo2.bar.com:55380 + +

Running Hadoop jobs using the allocated cluster

+ +

Now, one can run Hadoop jobs using the allocated cluster in the usual manner. This assumes variables like JAVA_HOME + and path to the Hadoop installation are set up correctly.:

+ + $ hadoop --config cluster_dir hadoop_command hadoop_command_args +

or

+ + +$ export HADOOP_CONF_DIR=cluster_dir +$ hadoop hadoop_command hadoop_command_args + +

Continuing our example, the following command will run a wordcount example on the allocated cluster:

+ $ hadoop --config ~/hod-clusters/test jar /path/to/hadoop/hadoop-examples.jar wordcount /path/to/input /path/to/output + +

or

+ + +$ export HADOOP_CONF_DIR=~/hod-clusters/test +$ hadoop jar /path/to/hadoop/hadoop-examples.jar wordcount /path/to/input /path/to/output + +

Operation deallocate

+

The deallocate operation is used to release an allocated cluster. When finished with a cluster, deallocate must be + run so that the nodes become free for others to use. The deallocate operation has the following syntax. Note that it + requires the cluster_dir (-d, --hod.clusterdir) argument:

+ $ hod deallocate -d cluster_dir + +

Continuing our example, the following command will deallocate the cluster:

+ $ hod deallocate -d ~/hod-clusters/test + +

As can be seen, HOD allows the users to allocate a cluster, and use it flexibly for running Hadoop jobs. For example, users + can run multiple jobs in parallel on the same cluster, by running hadoop from multiple shells pointing to the same configuration.

+
+ +
Running Hadoop Scripts Using HOD +

The HOD script operation combines the operations of allocating, using and deallocating a cluster into a single operation. + This is very useful for users who want to run a script of hadoop jobs and let HOD handle the cleanup automatically once the script completes. + In order to run hadoop scripts using hod, do the following:

+ +

Create a script file

+ +

This will be a regular shell script that will typically contain hadoop commands, such as:

+ + $ hadoop jar jar_file options + +

However, the user can add any valid commands as part of the script. HOD will execute this script setting HADOOP_CONF_DIR + automatically to point to the allocated cluster. So users do not need to worry about this. The users however need to specify a cluster directory + just like when using the allocate operation.

+

Running the script

+

The syntax for the script operation as is as follows. Note that it requires a cluster directory ( -d, --hod.clusterdir), number of + nodes (-n, --hod.nodecount) and a script file (-s, --hod.script):

+ + $ hod script -d cluster_directory -n number_of_nodes -s script_file +

Note that HOD will deallocate the cluster as soon as the script completes, and this means that the script must not complete until the + hadoop jobs themselves are completed. Users must take care of this while writing the script.

+
+
+
+ HOD Features +
Provisioning and Managing Hadoop Clusters +

The primary feature of HOD is to provision Hadoop MapReduce and HDFS clusters. This is described above in the Getting Started section. + Also, as long as nodes are available, and organizational policies allow, a user can use HOD to allocate multiple MapReduce clusters simultaneously. + The user would need to specify different paths for the cluster_dir parameter mentioned above for each cluster he/she allocates. + HOD provides the list and the info operations to enable managing multiple clusters.

+ +

Operation list

+ +

The list operation lists all the clusters allocated so far by a user. The cluster directory where the hadoop-site.xml is stored for the cluster, + and its status vis-a-vis connectivity with the JobTracker and/or HDFS is shown. The list operation has the following syntax:

+ + $ hod list + +

Operation info

+

The info operation shows information about a given cluster. The information shown includes the Torque job id, and locations of the important + daemons like the HOD Ringmaster process, and the Hadoop JobTracker and NameNode daemons. The info operation has the following syntax. + Note that it requires a cluster directory (-d, --hod.clusterdir):

+ + $ hod info -d cluster_dir + +

The cluster_dir should be a valid cluster directory specified in an earlier allocate operation.

+
+ +
Using a Tarball to Distribute Hadoop +

When provisioning Hadoop, HOD can use either a pre-installed Hadoop on the cluster nodes or distribute and install a Hadoop tarball as part + of the provisioning operation. If the tarball option is being used, there is no need to have a pre-installed Hadoop on the cluster nodes, nor a need + to use a pre-installed one. This is especially useful in a development / QE environment where individual developers may have different versions of + Hadoop to test on a shared cluster.

+ +

In order to use a pre-installed Hadoop, you must specify, in the hodrc, the pkgs option in the gridservice-hdfs + and gridservice-mapred sections. This must point to the path where Hadoop is installed on all nodes of the cluster.

+ +

The syntax for specifying tarball is as follows:

+ + $ hod allocate -d cluster_dir -n number_of_nodes -t hadoop_tarball_location + +

For example, the following command allocates Hadoop provided by the tarball ~/share/hadoop.tar.gz:

+ $ hod allocate -d ~/hadoop-cluster -n 10 -t ~/share/hadoop.tar.gz + +

Similarly, when using hod script, the syntax is as follows:

+ $ hod script -d cluster_directory -s script_file -n number_of_nodes -t hadoop_tarball_location + +

The hadoop_tarball specified in the syntax above should point to a path on a shared file system that is accessible from all the compute nodes. + Currently, HOD only supports NFS mounted file systems.

+

Note:

+
    +
  • For better distribution performance it is recommended that the Hadoop tarball contain only the libraries and binaries, and not the source or documentation.
    • + +
  • When you want to run jobs against a cluster allocated using the tarball, you must use a compatible version of hadoop to submit your jobs. + The best would be to untar and use the version that is present in the tarball itself.
    • +
  • You need to make sure that there are no Hadoop configuration files, hadoop-env.sh and hadoop-site.xml, present in the conf directory of the + tarred distribution. The presence of these files with incorrect values could make the cluster allocation to fail.
    • +
+
+ +
Using an External HDFS +

In typical Hadoop clusters provisioned by HOD, HDFS is already set up statically (without using HOD). This allows data to persist in HDFS after + the HOD provisioned clusters is deallocated. To use a statically configured HDFS, your hodrc must point to an external HDFS. Specifically, set the + following options to the correct values in the section gridservice-hdfs of the hodrc:

+ + +external = true +host = Hostname of the HDFS NameNode +fs_port = Port number of the HDFS NameNode +info_port = Port number of the HDFS NameNode web UI + + +

Note: You can also enable this option from command line. That is, to use a static HDFS, you will need to say:
+

+ $ hod allocate -d cluster_dir -n number_of_nodes --gridservice-hdfs.external + +

HOD can be used to provision an HDFS cluster as well as a MapReduce cluster, if required. To do so, set the following option in the section + gridservice-hdfs of the hodrc:

+ external = false +
+ +
Options for Configuring Hadoop +

HOD provides a very convenient mechanism to configure both the Hadoop daemons that it provisions and also the hadoop-site.xml that + it generates on the client side. This is done by specifying Hadoop configuration parameters in either the HOD configuration file, or from the + command line when allocating clusters.

+ +

Configuring Hadoop Daemons

+ +

For configuring the Hadoop daemons, you can do the following:

+ +

For MapReduce, specify the options as a comma separated list of key-value pairs to the server-params option in the + gridservice-mapred section. Likewise for a dynamically provisioned HDFS cluster, specify the options in the + server-params option in the gridservice-hdfs section. If these parameters should be marked as + final, then include these in the final-server-params option of the appropriate section.

+

For example:

+ +server-params = mapred.reduce.parallel.copies=20,io.sort.factor=100,io.sort.mb=128,io.file.buffer.size=131072 +final-server-params = mapred.child.java.opts=-Xmx512m,dfs.block.size=134217728,fs.inmemory.size.mb=128 + +

In order to provide the options from command line, you can use the following syntax:

+

For configuring the MapReduce daemons use:

+ + $ hod allocate -d cluster_dir -n number_of_nodes -Mmapred.reduce.parallel.copies=20 -Mio.sort.factor=100 + +

In the example above, the mapred.reduce.parallel.copies parameter and the io.sort.factor + parameter will be appended to the other server-params or if they already exist in server-params, + will override them. In order to specify these are final parameters, you can use:

+ + $ hod allocate -d cluster_dir -n number_of_nodes -Fmapred.reduce.parallel.copies=20 -Fio.sort.factor=100 + +

However, note that final parameters cannot be overwritten from command line. They can only be appended if not already specified.

+ +

Similar options exist for configuring dynamically provisioned HDFS daemons. For doing so, replace -M with -H and -F with -S.

+ +

Configuring Hadoop Job Submission (Client) Programs

+ +

As mentioned above, if the allocation operation completes successfully then cluster_dir/hadoop-site.xml will be generated + and will contain information about the allocated cluster's JobTracker and NameNode. This configuration is used when submitting jobs to the cluster. + HOD provides an option to include additional Hadoop configuration parameters into this file. The syntax for doing so is as follows:

+ + $ hod allocate -d cluster_dir -n number_of_nodes -Cmapred.userlog.limit.kb=200 -Cmapred.child.java.opts=-Xmx512m + +

In this example, the mapred.userlog.limit.kb and mapred.child.java.opts options will be included into + the hadoop-site.xml that is generated by HOD.

+
+ +
Viewing Hadoop Web-UIs +

The HOD allocation operation prints the JobTracker and NameNode web UI URLs. For example:

+ + +$ hod allocate -d ~/hadoop-cluster -n 10 -c ~/hod-conf-dir/hodrc +INFO - HDFS UI on http://host242.foo.com:55391 +INFO - Mapred UI on http://host521.foo.com:54874 + + +

The same information is also available via the info operation described above.

+
+ +
Collecting and Viewing Hadoop Logs +

To get the Hadoop logs of the daemons running on one of the allocated nodes:

+
    +
  • Log into the node of interest. If you want to look at the logs of the JobTracker or NameNode, then you can find the node running these by + using the list and info operations mentioned above.
    • +
  • Get the process information of the daemon of interest (for example, ps ux | grep TaskTracker)
    • +
  • In the process information, search for the value of the variable -Dhadoop.log.dir. Typically this will be a decendent directory + of the hodring.temp-dir value from the hod configuration file.
    • +
  • Change to the hadoop.log.dir directory to view daemon and user logs.
    • +
+

HOD also provides a mechanism to collect logs when a cluster is being deallocated and persist them into a file system, or an externally + configured HDFS. By doing so, these logs can be viewed after the jobs are completed and the nodes are released. In order to do so, configure + the log-destination-uri to a URI as follows:

+ +log-destination-uri = hdfs://host123:45678/user/hod/logs +log-destination-uri = file://path/to/store/log/files + +

Under the root directory specified above in the path, HOD will create a path user_name/torque_jobid and store gzipped log files for each + node that was part of the job.

+

Note that to store the files to HDFS, you may need to configure the hodring.pkgs option with the Hadoop version that + matches the HDFS mentioned. If not, HOD will try to use the Hadoop version that it is using to provision the Hadoop cluster itself.

+
+ +
Auto-deallocation of Idle Clusters +

HOD automatically deallocates clusters that are not running Hadoop jobs for a given period of time. Each HOD allocation includes a + monitoring facility that constantly checks for running Hadoop jobs. If it detects no running Hadoop jobs for a given period, it will automatically + deallocate its own cluster and thus free up nodes which are not being used effectively.

+ +

Note: While the cluster is deallocated, the cluster directory is not cleaned up automatically. The user must + deallocate this cluster through the regular deallocate operation to clean this up.

+
+
Specifying Additional Job Attributes +

HOD allows the user to specify a wallclock time and a name (or title) for a Torque job.

+

The wallclock time is the estimated amount of time for which the Torque job will be valid. After this time has expired, Torque will + automatically delete the job and free up the nodes. Specifying the wallclock time can also help the job scheduler to better schedule + jobs, and help improve utilization of cluster resources.

+

To specify the wallclock time, use the following syntax:

+ +$ hod allocate -d cluster_dir -n number_of_nodes -l time_in_seconds +

The name or title of a Torque job helps in user friendly identification of the job. The string specified here will show up in all information + where Torque job attributes are displayed, including the qstat command.

+

To specify the name or title, use the following syntax:

+$ hod allocate -d cluster_dir -n number_of_nodes -N name_of_job + +

Note: Due to restriction in the underlying Torque resource manager, names which do not start with an alphabet character + or contain a 'space' will cause the job to fail. The failure message points to the problem being in the specified job name.

+
+ +
Capturing HOD Exit Codes in Torque +

HOD exit codes are captured in the Torque exit_status field. This will help users and system administrators to distinguish successful + runs from unsuccessful runs of HOD. The exit codes are 0 if allocation succeeded and all hadoop jobs ran on the allocated cluster correctly. + They are non-zero if allocation failed or some of the hadoop jobs failed on the allocated cluster. The exit codes that are possible are + mentioned in the table below. Note: Hadoop job status is captured only if the version of Hadoop used is 16 or above.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Exit Code Meaning
6 Ringmaster failure
7 HDFS failure
8 Job tracker failure
10 Cluster dead
12 Cluster already allocated
13 HDFS dead
14 Mapred dead
16 All MapReduce jobs that ran on the cluster failed. Refer to hadoop logs for more details.
17 Some of the MapReduce jobs that ran on the cluster failed. Refer to hadoop logs for more details.
+
+
+ Command Line +

HOD command line has the following general syntax:

+ hod <operation> [ARGS] [OPTIONS] + +

Allowed operations are 'allocate', 'deallocate', 'info', 'list', 'script' and 'help'. For help with a particular operation do:

+ hod help <operation> + +

To have a look at possible options do:

+ hod help options + +
    + +
  • allocate
    + Usage : hod allocate -d cluster_dir -n number_of_nodes [OPTIONS]
    + Allocates a cluster on the given number of cluster nodes, and store the allocation information in cluster_dir for use with subsequent + hadoop commands. Note that the cluster_dir must exist before running the command.
    • + +
  • list
    + Usage : hod list [OPTIONS]
    + Lists the clusters allocated by this user. Information provided includes the Torque job id corresponding to the cluster, the cluster + directory where the allocation information is stored, and whether the MapReduce daemon is still active or not.
    • + +
  • info
    + Usage : hod info -d cluster_dir [OPTIONS]
    + Lists information about the cluster whose allocation information is stored in the specified cluster directory.
    • + +
  • deallocate
    + Usage : hod deallocate -d cluster_dir [OPTIONS]
    + Deallocates the cluster whose allocation information is stored in the specified cluster directory.
    • + +
  • script
    + Usage : hod script -s script_file -d cluster_directory -n number_of_nodes [OPTIONS]
    + Runs a hadoop script using HODscript operation. Provisions Hadoop on a given number of nodes, executes the given + script from the submitting node, and deallocates the cluster when the script completes.
    • + +
  • help
    + Usage : hod help [operation | 'options']
    + When no argument is specified, hod help gives the usage and basic options, and is equivalent to + hod --help (See below). When 'options' is given as argument, hod displays only the basic options + that hod takes. When an operation is specified, it displays the usage and description corresponding to that particular + operation. For e.g, to know about allocate operation, one can do a hod help allocate
    • +
+ + +

Besides the operations, HOD can take the following command line options.

+ +
    + +
  • --help
    + Prints out the help message to see the usage and basic options.
    • + +
  • --verbose-help
    + All configuration options provided in the hodrc file can be passed on the command line, using the syntax + --section_name.option_name[=value]. When provided this way, the value provided on command line + overrides the option provided in hodrc. The verbose-help command lists all the available options in the hodrc file. + This is also a nice way to see the meaning of the configuration options.
    "
    • +
+ +

See Options Configuring HOD for a description of most important hod configuration options. + For basic options do hod help options and for all options possible in hod configuration do hod --verbose-help. + See HOD Configuration for a description of all options.

+ + +
+ +
Options Configuring HOD +

As described above, HOD is configured using a configuration file that is usually set up by system administrators. + This is a INI style configuration file that is divided into sections, and options inside each section. Each section relates + to one of the HOD processes: client, ringmaster, hodring, mapreduce or hdfs. The options inside a section comprise + of an option name and value.

+ +

Users can override the configuration defined in the default configuration in two ways:

+
    +
  • Users can supply their own configuration file to HOD in each of the commands, using the -c option
    • +
  • Users can supply specific configuration options to HOD/ Options provided on command line override + the values provided in the configuration file being used.
    • +
+

This section describes some of the most commonly used configuration options. These commonly used options are + provided with a short option for convenience of specification. All other options can be specified using + a long option that is also described below.

+ +
    + +
  • -c config_file
    + Provides the configuration file to use. Can be used with all other options of HOD. Alternatively, the + HOD_CONF_DIR environment variable can be defined to specify a directory that contains a file + named hodrc, alleviating the need to specify the configuration file in each HOD command.
    • + +
  • -d cluster_dir
    + This is required for most of the hod operations. As described under Create a Cluster Directory, + the cluster directory is a directory on the local file system where hod will generate the Hadoop configuration, + hadoop-site.xml, corresponding to the cluster it allocates. Pass it to the hod operations as an argument + to -d or --hod.clusterdir. If it doesn't already exist, HOD will automatically try to create it and use it. Once a cluster is allocated, a + user can utilize it to run Hadoop jobs by specifying the clusterdirectory as the Hadoop --config option.
    • + +
  • -n number_of_nodes
    + This is required for the hod 'allocation' operation and for script operation. This denotes the number of nodes to be allocated.
    • + +
  • -s script-file
    + Required when using script operation, specifies the script file to execute.
    • + +
  • -b 1|2|3|4
    + Enables the given debug level. Can be used with all other options of HOD. 4 is most verbose.
    • + +
  • -t hadoop_tarball
    + Provisions Hadoop from the given tar.gz file. This option is only applicable to the allocate operation. For better + distribution performance it is strongly recommended that the Hadoop tarball is created after removing the source + or documentation.
    • + +
  • -N job-name
    + The Name to give to the resource manager job that HOD uses underneath. For e.g. in the case of Torque, this translates to + the qsub -N option, and can be seen as the job name using the qstat command.
    • + +
  • -l wall-clock-time
    + The amount of time for which the user expects to have work on the allocated cluster. This is passed to the resource manager + underneath HOD, and can be used in more efficient scheduling and utilization of the cluster. Note that in the case of Torque, + the cluster is automatically deallocated after this time expires.
    • + +
  • -j java-home
    + Path to be set to the JAVA_HOME environment variable. This is used in the script operation. HOD sets the + JAVA_HOME environment variable tot his value and launches the user script in that.
    • + +
  • -A account-string
    + Accounting information to pass to underlying resource manager.
    • + +
  • -Q queue-name
    + Name of the queue in the underlying resource manager to which the job must be submitted.
    • + +
  • -Mkey1=value1 -Mkey2=value2
    + Provides configuration parameters for the provisioned MapReduce daemons (JobTracker and TaskTrackers). A + hadoop-site.xml is generated with these values on the cluster nodes.
    + Note: Values which have the following characters: space, comma, equal-to, semi-colon need to be + escaped with a '\' character, and need to be enclosed within quotes. You can escape a '\' with a '\' too.
    • + +
  • -Hkey1=value1 -Hkey2=value2
    + Provides configuration parameters for the provisioned HDFS daemons (NameNode and DataNodes). A hadoop-site.xml + is generated with these values on the cluster nodes
    + Note: Values which have the following characters: space, comma, equal-to, semi-colon need to be + escaped with a '\' character, and need to be enclosed within quotes. You can escape a '\' with a '\' too.
    • + +
  • -Ckey1=value1 -Ckey2=value2
    + Provides configuration parameters for the client from where jobs can be submitted. A hadoop-site.xml is generated + with these values on the submit node.
    + Note: Values which have the following characters: space, comma, equal-to, semi-colon need to be + escaped with a '\' character, and need to be enclosed within quotes. You can escape a '\' with a '\' too.
    • + +
  • --section-name.option-name=value
    + This is the method to provide options using the long format. For e.g. you could say --hod.script-wait-time=20
    • +
+ +
+
+ + +
+ Troubleshooting +

The following section identifies some of the most likely error conditions users can run into when using HOD and ways to trouble-shoot them

+ +
HOD Hangs During Allocation + +

Possible Cause: One of the HOD or Hadoop components have failed to come up. In such a case, the + hod command will return after a few minutes (typically 2-3 minutes) with an error code of either 7 or 8 + as defined in the Error Codes section. Refer to that section for further details.

+

Possible Cause: A large allocation is fired with a tarball. Sometimes due to load in the network, or on + the allocated nodes, the tarball distribution might be significantly slow and take a couple of minutes to come back. + Wait for completion. Also check that the tarball does not have the Hadoop sources or documentation.

+

Possible Cause: A Torque related problem. If the cause is Torque related, the hod + command will not return for more than 5 minutes. Running hod in debug mode may show the + qstat command being executed repeatedly. Executing the qstat command from + a separate shell may show that the job is in the Q (Queued) state. This usually indicates a + problem with Torque. Possible causes could include some nodes being down, or new nodes added that Torque + is not aware of. Generally, system administator help is needed to resolve this problem.

+
+ +
HOD Hangs During Deallocation + +

Possible Cause: A Torque related problem, usually load on the Torque server, or the allocation is very large. + Generally, waiting for the command to complete is the only option.

+
+ +
HOD Fails With an Error Code and Error Message + +

If the exit code of the hod command is not 0, then refer to the following table + of error exit codes to determine why the code may have occurred and how to debug the situation.

+

Error Codes

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Error CodeMeaningPossible Causes and Remedial Actions
1 Configuration error Incorrect configuration values specified in hodrc, or other errors related to HOD configuration. + The error messages in this case must be sufficient to debug and fix the problem.
2 Invalid operation Do hod help for the list of valid operations.
3 Invalid operation arguments Do hod help operation for listing the usage of a particular operation.
4 Scheduler failure 1. Requested more resources than available. Run checknodes cluster_name to see if enough nodes are available.
+ 2. Requested resources exceed resource manager limits.
+ 3. Torque is misconfigured, the path to Torque binaries is misconfigured, or other Torque problems. Contact system administrator.
5 Job execution failure 1. Torque Job was deleted from outside. Execute the Torque qstat command to see if you have any jobs in the + R (Running) state. If none exist, try re-executing HOD.
+ 2. Torque problems such as the server momentarily going down, or becoming unresponsive. Contact system administrator.
+ 3. The system administrator might have configured account verification, and an invalid account is specified. Contact system administrator.
6 Ringmaster failure HOD prints the message "Cluster could not be allocated because of the following errors on the ringmaster host <hostname>". + The actual error message may indicate one of the following:
+ 1. Invalid configuration on the node running the ringmaster, specified by the hostname in the error message.
+ 2. Invalid configuration in the ringmaster section,
+ 3. Invalid pkgs option in gridservice-mapred or gridservice-hdfs section,
+ 4. An invalid hadoop tarball, or a tarball which has bundled an invalid configuration file in the conf directory,
+ 5. Mismatched version in Hadoop between the MapReduce and an external HDFS.
+ The Torque qstat command will most likely show a job in the C (Completed) state.
+ One can login to the ringmaster host as given by HOD failure message and debug the problem with the help of the error message. + If the error message doesn't give complete information, ringmaster logs should help finding out the root cause of the problem. + Refer to the section Locating Ringmaster Logs below for more information.
7 HDFS failure When HOD fails to allocate due to HDFS failures (or Job tracker failures, error code 8, see below), it prints a failure message + "Hodring at <hostname> failed with following errors:" and then gives the actual error message, which may indicate one of the following:
+ 1. Problem in starting Hadoop clusters. Usually the actual cause in the error message will indicate the problem on the hostname mentioned. + Also, review the Hadoop related configuration in the HOD configuration files. Look at the Hadoop logs using information specified in + Collecting and Viewing Hadoop Logs section above.
+ 2. Invalid configuration on the node running the hodring, specified by the hostname in the error message
+ 3. Invalid configuration in the hodring section of hodrc. ssh to the hostname specified in the + error message and grep for ERROR or CRITICAL in hodring logs. Refer to the section + Locating Hodring Logs below for more information.
+ 4. Invalid tarball specified which is not packaged correctly.
+ 5. Cannot communicate with an externally configured HDFS.
+ When such HDFS or Job tracker failure occurs, one can login into the host with hostname mentioned in HOD failure message and debug the problem. + While fixing the problem, one should also review other log messages in the ringmaster log to see which other machines also might have had problems + bringing up the jobtracker/namenode, apart from the hostname that is reported in the failure message. This possibility of other machines also having problems + occurs because HOD continues to try and launch hadoop daemons on multiple machines one after another depending upon the value of the configuration + variable ringmaster.max-master-failures. + See Locating Ringmaster Logs for more information.
8 Job tracker failure Similar to the causes in DFS failure case.
10 Cluster dead 1. Cluster was auto-deallocated because it was idle for a long time.
+ 2. Cluster was auto-deallocated because the wallclock time specified by the system administrator or user was exceeded.
+ 3. Cannot communicate with the JobTracker and HDFS NameNode which were successfully allocated. Deallocate the cluster, and allocate again.
12 Cluster already allocated The cluster directory specified has been used in a previous allocate operation and is not yet deallocated. + Specify a different directory, or deallocate the previous allocation first.
13 HDFS dead Cannot communicate with the HDFS NameNode. HDFS NameNode went down.
14 Mapred dead 1. Cluster was auto-deallocated because it was idle for a long time.
+ 2. Cluster was auto-deallocated because the wallclock time specified by the system administrator or user was exceeded.
+ 3. Cannot communicate with the MapReduce JobTracker. JobTracker node went down.
+
15 Cluster not allocated An operation which requires an allocated cluster is given a cluster directory with no state information.
Any non-zero exit code HOD script error If the hod script option was used, it is likely that the exit code is from the script. Unfortunately, this could clash with the + exit codes of the hod command itself. In order to help users differentiate these two, hod writes the script's exit code to a file + called script.exitcode in the cluster directory, if the script returned an exit code. You can cat this file to determine the script's + exit code. If it does not exist, then it is a hod command exit code.
+
+
Hadoop DFSClient Warns with a + NotReplicatedYetException +

Sometimes, when you try to upload a file to the HDFS immediately after + allocating a HOD cluster, DFSClient warns with a NotReplicatedYetException. It + usually shows a message something like -

+ + +WARN hdfs.DFSClient: NotReplicatedYetException sleeping <filename> retries left 3 +08/01/25 16:31:40 INFO hdfs.DFSClient: org.apache.hadoop.ipc.RemoteException: java.io.IOException: +File <filename> could only be replicated to 0 nodes, instead of 1 + +

This scenario arises when you try to upload a file + to the HDFS while the DataNodes are still in the process of contacting the + NameNode. This can be resolved by waiting for some time before uploading a new + file to the HDFS, so that enough DataNodes start and contact the NameNode.

+
+ +
Hadoop Jobs Not Running on a Successfully Allocated Cluster + +

This scenario generally occurs when a cluster is allocated, and is left inactive for sometime, and then hadoop jobs + are attempted to be run on them. Then Hadoop jobs fail with the following exception:

+ + 08/01/25 16:31:40 INFO ipc.Client: Retrying connect to server: foo.bar.com/1.1.1.1:53567. Already tried 1 time(s). + +

Possible Cause: No Hadoop jobs were run for a significant portion of time. Thus the cluster would have got + deallocated as described in the section Auto-deallocation of Idle Clusters. Deallocate the cluster and allocate it again.

+

Possible Cause: The wallclock limit specified by the Torque administrator or the -l option + defined in the section Specifying Additional Job Attributes was exceeded since allocation time. Thus the cluster + would have got released. Deallocate the cluster and allocate it again.

+

Possible Cause: There is a version mismatch between the version of the hadoop being used in provisioning + (typically via the tarball option) and the external HDFS. Ensure compatible versions are being used.

+

Possible Cause: There is a version mismatch between the version of the hadoop client being used to submit + jobs and the hadoop used in provisioning (typically via the tarball option). Ensure compatible versions are being used.

+

Possible Cause: You used one of the options for specifying Hadoop configuration -M or -H, + which had special characters like space or comma that were not escaped correctly. Refer to the section + Options Configuring HOD for checking how to specify such options correctly.

+
+
My Hadoop Job Got Killed +

Possible Cause: The wallclock limit specified by the Torque administrator or the -l + option defined in the section Specifying Additional Job Attributes was exceeded since allocation time. + Thus the cluster would have got released. Deallocate the cluster and allocate it again, this time with a larger wallclock time.

+

Possible Cause: Problems with the JobTracker node. Refer to the section in Collecting and Viewing Hadoop Logs to get more information.

+
+
Hadoop Job Fails with Message: 'Job tracker still initializing' +

Possible Cause: The hadoop job was being run as part of the HOD script command, and it started before the JobTracker could come up fully. + Allocate the cluster using a large value for the configuration option --hod.script-wait-time. + Typically a value of 120 should work, though it is typically unnecessary to be that large.

+
+
The Exit Codes For HOD Are Not Getting Into Torque +

Possible Cause: Version 0.16 of hadoop is required for this functionality to work. + The version of Hadoop used does not match. Use the required version of Hadoop.

+

Possible Cause: The deallocation was done without using the hod + command; for e.g. directly using qdel. When the cluster is deallocated in this manner, + the HOD processes are terminated using signals. This results in the exit code to be based on the + signal number, rather than the exit code of the program.

+
+
The Hadoop Logs are Not Uploaded to HDFS +

Possible Cause: There is a version mismatch between the version of the hadoop being used for uploading the logs + and the external HDFS. Ensure that the correct version is specified in the hodring.pkgs option.

+
+
Locating Ringmaster Logs +

To locate the ringmaster logs, follow these steps:

+
    +
  • Execute hod in the debug mode using the -b option. This will print the Torque job id for the current run.
    • +
  • Execute qstat -f torque_job_id and look up the value of the exec_host parameter in the output. + The first host in this list is the ringmaster node.
    • +
  • Login to this node.
    • +
  • The ringmaster log location is specified by the ringmaster.log-dir option in the hodrc. The name of the log file will be + username.torque_job_id/ringmaster-main.log.
    • +
  • If you don't get enough information, you may want to set the ringmaster debug level to 4. This can be done by passing + --ringmaster.debug 4 to the hod command line.
    • +
+
+
Locating Hodring Logs +

To locate hodring logs, follow the steps below:

+
    +
  • Execute hod in the debug mode using the -b option. This will print the Torque job id for the current run.
    • +
  • Execute qstat -f torque_job_id and look up the value of the exec_host parameter in the output. + All nodes in this list should have a hodring on them.
    • +
  • Login to any of these nodes.
    • +
  • The hodring log location is specified by the hodring.log-dir option in the hodrc. The name of the log file will be + username.torque_job_id/hodring-main.log.
    • +
  • If you don't get enough information, you may want to set the hodring debug level to 4. This can be done by passing + --hodring.debug 4 to the hod command line.
    • +
+
+
+
+ + + + + +
+ HOD Administrators +

This section show administrators how to install, configure and run HOD.

+
+Getting Started + +

The basic system architecture of HOD includes these components:

+
    +
  • A Resource manager, possibly together with a scheduler (see Prerequisites)
    • +
  • Various HOD components
    • +
  • Hadoop MapReduce and HDFS daemons
    • +
+ +

+HOD provisions and maintains Hadoop MapReduce and, optionally, HDFS instances +through interaction with the above components on a given cluster of nodes. A cluster of +nodes can be thought of as comprising two sets of nodes:

+
    +
  • Submit nodes: Users use the HOD client on these nodes to allocate clusters, and then +use the Hadoop client to submit Hadoop jobs.
    • +
  • Compute nodes: Using the resource manager, HOD components are run on these nodes to +provision the Hadoop daemons. After that Hadoop jobs run on them.
    • +
+ +

+Here is a brief description of the sequence of operations in allocating a cluster and +running jobs on them. +

+ +
    +
  • The user uses the HOD client on the Submit node to allocate a desired number of +cluster nodes and to provision Hadoop on them.
    • +
  • The HOD client uses a resource manager interface (qsub, in Torque) to submit a HOD +process, called the RingMaster, as a Resource Manager job, to request the user's desired number +of nodes. This job is submitted to the central server of the resource manager (pbs_server, in Torque).
    • +
  • On the compute nodes, the resource manager slave daemons (pbs_moms in Torque) accept +and run jobs that they are assigned by the central server (pbs_server in Torque). The RingMaster +process is started on one of the compute nodes (mother superior, in Torque).
    • +
  • The RingMaster then uses another resource manager interface (pbsdsh, in Torque) to run +the second HOD component, HodRing, as distributed tasks on each of the compute +nodes allocated.
    • +
  • The HodRings, after initializing, communicate with the RingMaster to get Hadoop commands, +and run them accordingly. Once the Hadoop commands are started, they register with the RingMaster, +giving information about the daemons.
    • +
  • All the configuration files needed for Hadoop instances are generated by HOD itself, +some obtained from options given by user in its own configuration file.
    • +
  • The HOD client keeps communicating with the RingMaster to find out the location of the +JobTracker and HDFS daemons.
    • +
+ +
+ +
+Prerequisites +

To use HOD, your system should include the following components.

+ +
    + +
  • Operating System: HOD is currently tested on RHEL4.
    • + +
  • Nodes: HOD requires a minimum of three nodes configured through a resource manager.
    • + +
  • Software: The following components must be installed on ALL nodes before using HOD: +
    • + +
  • Software (optional): The following components are optional and can be installed to obtain better +functionality from HOD: +
      +
    • Twisted Python: This can be + used for improving the scalability of HOD. If this module is detected to be + installed, HOD uses it, else it falls back to default modules.
      • +
    • Hadoop: HOD can automatically + distribute Hadoop to all nodes in the cluster. However, it can also use a + pre-installed version of Hadoop, if it is available on all nodes in the cluster. + HOD currently supports Hadoop 0.15 and above.
      • +
    • + +
+ +

Note: HOD configuration requires the location of installs of these +components to be the same on all nodes in the cluster. It will also +make the configuration simpler to have the same location on the submit +nodes. +

+
+ +
+Resource Manager +

Currently HOD works with the Torque resource manager, which it uses for its node + allocation and job submission. Torque is an open source resource manager from + Cluster Resources, a community effort + based on the PBS project. It provides control over batch jobs and distributed compute nodes. Torque is + freely available for download from here. +

+ +

All documentation related to torque can be seen under + the section TORQUE Resource Manager here. You can + get wiki documentation from here. + Users may wish to subscribe to TORQUE’s mailing list or view the archive for questions, + comments here. +

+ +

To use HOD with Torque:

+
    +
  • Install Torque components: pbs_server on one node (head node), pbs_mom on all + compute nodes, and PBS client tools on all compute nodes and submit + nodes. Perform at least a basic configuration so that the Torque system is up and + running, that is, pbs_server knows which machines to talk to. Look here + for basic configuration. + + For advanced configuration, see here
    • +
  • Create a queue for submitting jobs on the pbs_server. The name of the queue is the + same as the HOD configuration parameter, resource-manager.queue. The HOD client uses this queue to + submit the RingMaster process as a Torque job.
    • +
  • Specify a cluster name as a property for all nodes in the cluster. + This can be done by using the qmgr command. For example: + qmgr -c "set node node properties=cluster-name". The name of the cluster is the same as + the HOD configuration parameter, hod.cluster.
    • +
  • Make sure that jobs can be submitted to the nodes. This can be done by + using the qsub command. For example: + echo "sleep 30" | qsub -l nodes=3
    • +
+ +
+ +
+Installing HOD + +

Once the resource manager is set up, you can obtain and +install HOD.

+
    +
  • If you are getting HOD from the Hadoop tarball, it is available under the + 'contrib' section of Hadoop, under the root directory 'hod'.
    • +
  • If you are building from source, you can run ant tar from the Hadoop root + directory to generate the Hadoop tarball, and then get HOD from there, + as described above.
    • +
  • Distribute the files under this directory to all the nodes in the + cluster. Note that the location where the files are copied should be + the same on all the nodes.
    • +
  • Note that compiling hadoop would build HOD with appropriate permissions + set on all the required script files in HOD.
    • +
+
+ +
+Configuring HOD + +

You can configure HOD once it is installed. The minimal configuration needed +to run HOD is described below. More advanced configuration options are discussed +in the HOD Configuration.

+
+ Minimal Configuration +

To get started using HOD, the following minimal configuration is + required:

+
    +
  • On the node from where you want to run HOD, edit the file hodrc + located in the <install dir>/conf directory. This file + contains the minimal set of values required to run hod.
    • +
  • +

    Specify values suitable to your environment for the following + variables defined in the configuration file. Note that some of these + variables are defined at more than one place in the file.

    + +
      +
    • ${JAVA_HOME}: Location of Java for Hadoop. Hadoop supports Sun JDK + 1.6.x and above.
      • +
    • ${CLUSTER_NAME}: Name of the cluster which is specified in the + 'node property' as mentioned in resource manager configuration.
      • +
    • ${HADOOP_HOME}: Location of Hadoop installation on the compute and + submit nodes.
      • +
    • ${RM_QUEUE}: Queue configured for submitting jobs in the resource + manager configuration.
      • +
    • ${RM_HOME}: Location of the resource manager installation on the + compute and submit nodes.
      • +
    +
    • + +
  • +

    The following environment variables may need to be set depending on + your environment. These variables must be defined where you run the + HOD client and must also be specified in the HOD configuration file as the + value of the key resource_manager.env-vars. Multiple variables can be + specified as a comma separated list of key=value pairs.

    + +
      +
    • HOD_PYTHON_HOME: If you install python to a non-default location + of the compute nodes, or submit nodes, then this variable must be + defined to point to the python executable in the non-standard + location.
      • +
    +
    • +
+
+ +
+ Advanced Configuration +

You can review and modify other configuration options to suit + your specific needs. See HOD Configuration for more information.

+
+
+ +
+ Running HOD +

You can run HOD once it is configured. Refer to HOD Users for more information.

+
+ +
+ Supporting Tools and Utilities +

This section describes supporting tools and utilities that can be used to + manage HOD deployments.

+ +
+ logcondense.py - Manage Log Files +

As mentioned under + Collecting and Viewing Hadoop Logs, + HOD can be configured to upload + Hadoop logs to a statically configured HDFS. Over time, the number of logs uploaded + to HDFS could increase. logcondense.py is a tool that helps + administrators to remove log files uploaded to HDFS.

+
+ Running logcondense.py +

logcondense.py is available under hod_install_location/support folder. You can either + run it using python, for example, python logcondense.py, or give execute permissions + to the file, and directly run it as logcondense.py. logcondense.py needs to be + run by a user who has sufficient permissions to remove files from locations where log + files are uploaded in the HDFS, if permissions are enabled. For example as mentioned under + hodring options, the logs could + be configured to come under the user's home directory in HDFS. In that case, the user + running logcondense.py should have super user privileges to remove the files from under + all user home directories.

+
+
+ Command Line Options for logcondense.py +

The following command line options are supported for logcondense.py.

+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Short OptionLong optionMeaningExample
-p--packageComplete path to the hadoop script. The version of hadoop must be the same as the + one running HDFS./usr/bin/hadoop
-d--daysDelete log files older than the specified number of days7
-c--configPath to the Hadoop configuration directory, under which hadoop-site.xml resides. + The hadoop-site.xml must point to the HDFS NameNode from where logs are to be removed./home/foo/hadoop/conf
-l--logsA HDFS path, this must be the same HDFS path as specified for the log-destination-uri, + as mentioned under hodring options, + without the hdfs:// URI string/user
-n--dynamicdfsIf true, this will indicate that the logcondense.py script should delete HDFS logs + in addition to MapReduce logs. Otherwise, it only deletes MapReduce logs, which is also the + default if this option is not specified. This option is useful if + dynamic HDFS installations + are being provisioned by HOD, and the static HDFS installation is being used only to collect + logs - a scenario that may be common in test clusters.false
-r--retain-master-logsIf true, this will keep the JobTracker logs of job in hod-logs inside HDFS and it + will delete only the TaskTracker logs. Also, this will keep the Namenode logs along with + JobTracker logs and will only delete the Datanode logs if 'dynamicdfs' options is set + to true. Otherwise, it will delete the complete job directory from hod-logs inside + HDFS. By default it is set to false.false
+

So, for example, to delete all log files older than 7 days using a hadoop-site.xml stored in + ~/hadoop-conf, using the hadoop installation under ~/hadoop-0.17.0, you could say:

+

python logcondense.py -p ~/hadoop-0.17.0/bin/hadoop -d 7 -c ~/hadoop-conf -l /user

+
+
+
+ checklimits.sh - Monitor Resource Limits +

checklimits.sh is a HOD tool specific to the Torque/Maui environment + (Maui Cluster Scheduler is an open source job + scheduler for clusters and supercomputers, from clusterresources). The + checklimits.sh script + updates the torque comment field when newly submitted job(s) violate or + exceed + over user limits set up in Maui scheduler. It uses qstat, does one pass + over the torque job-list to determine queued or unfinished jobs, runs Maui + tool checkjob on each job to see if user limits are violated and then + runs torque's qalter utility to update job attribute 'comment'. Currently + it updates the comment as User-limits exceeded. Requested:([0-9]*) + Used:([0-9]*) MaxLimit:([0-9]*) for those jobs that violate limits. + This comment field is then used by HOD to behave accordingly depending on + the type of violation.

+
+ Running checklimits.sh +

checklimits.sh is available under the hod_install_location/support + folder. This shell script can be run directly as sh + checklimits.sh or as ./checklimits.sh after enabling + execute permissions. Torque and Maui binaries should be available + on the machine where the tool is run and should be in the path + of the shell script process. To update the + comment field of jobs from different users, this tool must be run with + torque administrative privileges. This tool must be run repeatedly + after specific intervals of time to frequently update jobs violating + constraints, for example via cron. Please note that the resource manager + and scheduler commands used in this script can be expensive and so + it is better not to run this inside a tight loop without sleeping.

+
+
+ +
+ verify-account Script +

Production systems use accounting packages to charge users for using + shared compute resources. HOD supports a parameter + resource_manager.pbs-account to allow users to identify the + account under which they would like to submit jobs. It may be necessary + to verify that this account is a valid one configured in an accounting + system. The hod-install-dir/bin/verify-account script + provides a mechanism to plug-in a custom script that can do this + verification.

+ +
+ Integrating the verify-account script with HOD +

HOD runs the verify-account script passing in the + resource_manager.pbs-account value as argument to the script, + before allocating a cluster. Sites can write a script that verify this + account against their accounting systems. Returning a non-zero exit + code from this script will cause HOD to fail allocation. Also, in + case of an error, HOD will print the output of script to the user. + Any descriptive error message can be passed to the user from the + script in this manner.

+

The default script that comes with the HOD installation does not + do any validation, and returns a zero exit code.

+

If the verify-account script is not found, then HOD will treat + that verification is disabled, and continue allocation as is.

+
+
+
+
+ + + + +
+ HOD Configuration +

This section discusses how to work with the HOD configuration options.

+ +
+ Getting Started + +

Configuration options can be specified in two ways: as a configuration file + in the INI format and as command line options to the HOD shell, + specified in the format --section.option[=value]. If the same option is + specified in both places, the value specified on the command line + overrides the value in the configuration file.

+ +

To get a simple description of all configuration options use:

+ $ hod --verbose-help + +
+ +
+ Configuation Options +

HOD organizes configuration options into these sections:

+ +
    +
  • common: Options that appear in more than one section. Options defined in a section are used by the + process for which that section applies. Common options have the same meaning, but can have different values in each section.
    • +
  • hod: Options for the HOD client
    • +
  • resource_manager: Options for specifying which resource manager to use, and other parameters for using that resource manager
    • +
  • ringmaster: Options for the RingMaster process,
    • +
  • hodring: Options for the HodRing processes
    • +
  • gridservice-mapred: Options for the MapReduce daemons
    • +
  • gridservice-hdfs: Options for the HDFS daemons.
    • +
+ +
+ common options +
    +
  • temp-dir: Temporary directory for usage by the HOD processes. Make + sure that the users who will run hod have rights to create + directories under the directory specified here. If you + wish to make this directory vary across allocations, + you can make use of the environmental variables which will + be made available by the resource manager to the HOD + processes. For example, in a Torque setup, having + --ringmaster.temp-dir=/tmp/hod-temp-dir.$PBS_JOBID would + let ringmaster use different temp-dir for each + allocation; Torque expands this variable before starting + the ringmaster.
    • + +
  • debug: Numeric value from 1-4. 4 produces the most log information, + and 1 the least.
    • + +
  • log-dir: Directory where log files are stored. By default, this is + <install-location>/logs/. The restrictions and notes for the + temp-dir variable apply here too. +
    • + +
  • xrs-port-range: Range of ports, among which an available port shall + be picked for use to run an XML-RPC server.
    • + +
  • http-port-range: Range of ports, among which an available port shall + be picked for use to run an HTTP server.
    • + +
  • java-home: Location of Java to be used by Hadoop.
    • +
  • syslog-address: Address to which a syslog daemon is bound to. The format + of the value is host:port. If configured, HOD log messages + will be logged to syslog using this value.
    • + +
+
+ +
+ hod options + +
    +
  • cluster: Descriptive name given to the cluster. For Torque, this is specified as a 'Node property' for every node in the cluster. + HOD uses this value to compute the number of available nodes.
    • + +
  • client-params: Comma-separated list of hadoop config parameters specified as key-value pairs. + These will be used to generate a hadoop-site.xml on the submit node that should be used for running MapReduce jobs.
    • + +
  • job-feasibility-attr: Regular expression string that specifies whether and how to check job feasibility - resource + manager or scheduler limits. The current implementation corresponds to the torque job attribute 'comment' and by default is disabled. + When set, HOD uses it to decide what type of limit violation is triggered and either deallocates the cluster or stays in queued state + according as the request is beyond maximum limits or the cumulative usage has crossed maximum limits. The torque comment attribute may be updated + periodically by an external mechanism. For example, comment attribute can be updated by running + checklimits.sh script in hod/support directory, + and then setting job-feasibility-attr equal to the value TORQUE_USER_LIMITS_COMMENT_FIELD, "User-limits exceeded. Requested:([0-9]*) + Used:([0-9]*) MaxLimit:([0-9]*)", will make HOD behave accordingly.
    • +
+
+ +
+ resource_manager options + +
    +
  • queue: Name of the queue configured in the resource manager to which + jobs are to be submitted.
    • + +
  • batch-home: Install directory to which 'bin' is appended and under + which the executables of the resource manager can be + found.
    • + +
  • env-vars: Comma-separated list of key-value pairs, + expressed as key=value, which would be passed to the jobs + launched on the compute nodes. + For example, if the python installation is + in a non-standard location, one can set the environment + variable 'HOD_PYTHON_HOME' to the path to the python + executable. The HOD processes launched on the compute nodes + can then use this variable.
    • +
  • options: Comma-separated list of key-value pairs, + expressed as + <option>:<sub-option>=<value>. When + passing to the job submission program, these are expanded + as -<option> <sub-option>=<value>. These + are generally used for specifying additional resource + contraints for scheduling. For instance, with a Torque + setup, one can specify + --resource_manager.options='l:arch=x86_64' for + constraining the nodes being allocated to a particular + architecture; this option will be passed to Torque's qsub + command as "-l arch=x86_64".
    • +
+
+ +
+ ringmaster options + +
    +
  • work-dirs: Comma-separated list of paths that will serve + as the root for directories that HOD generates and passes + to Hadoop for use to store DFS and MapReduce data. For + example, + this is where DFS data blocks will be stored. Typically, + as many paths are specified as there are disks available + to ensure all disks are being utilized. The restrictions + and notes for the temp-dir variable apply here too.
    • +
  • max-master-failures: Number of times a hadoop master + daemon can fail to launch, beyond which HOD will fail + the cluster allocation altogether. In HOD clusters, + sometimes there might be a single or few "bad" nodes due + to issues like missing java, missing or incorrect version + of Hadoop etc. When this configuration variable is set + to a positive integer, the RingMaster returns an error + to the client only when the number of times a hadoop + master (JobTracker or NameNode) fails to start on these + bad nodes because of above issues, exceeds the specified + value. If the number is not exceeded, the next HodRing + which requests for a command to launch is given the same + hadoop master again. This way, HOD tries its best for a + successful allocation even in the presence of a few bad + nodes in the cluster. +
    • +
  • workers_per_ring: Number of workers per service per HodRing. + By default this is set to 1. If this configuration + variable is set to a value 'n', the HodRing will run + 'n' instances of the workers (TaskTrackers or DataNodes) + on each node acting as a slave. This can be used to run + multiple workers per HodRing, so that the total number of + workers in a HOD cluster is not limited by the total + number of nodes requested during allocation. However, note + that this will mean each worker should be configured to use + only a proportional fraction of the capacity of the + resources on the node. In general, this feature is only + useful for testing and simulation purposes, and not for + production use.
    • +
+
+ +
+ gridservice-hdfs options + +
    +
  • external: If false, indicates that a HDFS cluster must be + bought up by the HOD system, on the nodes which it + allocates via the allocate command. Note that in that case, + when the cluster is de-allocated, it will bring down the + HDFS cluster, and all the data will be lost. + If true, it will try and connect to an externally configured + HDFS system. + Typically, because input for jobs are placed into HDFS + before jobs are run, and also the output from jobs in HDFS + is required to be persistent, an internal HDFS cluster is + of little value in a production system. However, it allows + for quick testing.
    • + +
  • host: Hostname of the externally configured NameNode, if any
    • + +
  • fs_port: Port to which NameNode RPC server is bound.
    • + +
  • info_port: Port to which the NameNode web UI server is bound.
    • + +
  • pkgs: Installation directory, under which bin/hadoop executable is + located. This can be used to use a pre-installed version of + Hadoop on the cluster.
    • + +
  • server-params: Comma-separated list of hadoop config parameters + specified key-value pairs. These will be used to + generate a hadoop-site.xml that will be used by the + NameNode and DataNodes.
    • + +
  • final-server-params: Same as above, except they will be marked final.
    • +
+
+ +
+ gridservice-mapred options + +
    +
  • external: If false, indicates that a MapReduce cluster must be + bought up by the HOD system on the nodes which it allocates + via the allocate command. + If true, if will try and connect to an externally + configured MapReduce system.
    • + +
  • host: Hostname of the externally configured JobTracker, if any
    • + +
  • tracker_port: Port to which the JobTracker RPC server is bound
    • + +
  • info_port: Port to which the JobTracker web UI server is bound.
    • + +
  • pkgs: Installation directory, under which bin/hadoop executable is + located
    • + +
  • server-params: Comma-separated list of hadoop config parameters + specified key-value pairs. These will be used to + generate a hadoop-site.xml that will be used by the + JobTracker and TaskTrackers
    • + +
  • final-server-params: Same as above, except they will be marked final.
    • +
+
+ +
+ hodring options + +
    +
  • mapred-system-dir-root: Directory in the DFS under which HOD will + generate sub-directory names and pass the full path + as the value of the 'mapred.system.dir' configuration + parameter to Hadoop daemons. The format of the full + path will be value-of-this-option/userid/mapredsystem/cluster-id. + Note that the directory specified here should be such + that all users can create directories under this, if + permissions are enabled in HDFS. Setting the value of + this option to /user will make HOD use the user's + home directory to generate the mapred.system.dir value.
    • + +
  • log-destination-uri: URL describing a path in an external, static DFS or the + cluster node's local file system where HOD will upload + Hadoop logs when a cluster is deallocated. To specify a + DFS path, use the format 'hdfs://path'. To specify a + cluster node's local file path, use the format 'file://path'. + + When clusters are deallocated by HOD, the hadoop logs will + be deleted as part of HOD's cleanup process. To ensure these + logs persist, you can use this configuration option. + + The format of the path is + value-of-this-option/userid/hod-logs/cluster-id + + Note that the directory you specify here must be such that all + users can create sub-directories under this. Setting this value + to hdfs://user will make the logs come in the user's home directory + in DFS.
    • + +
  • pkgs: Installation directory, under which bin/hadoop executable is located. This will + be used by HOD to upload logs if a HDFS URL is specified in log-destination-uri + option. Note that this is useful if the users are using a tarball whose version + may differ from the external, static HDFS version.
    • + +
  • hadoop-port-range: Range of ports, among which an available port shall + be picked for use to run a Hadoop Service, like JobTracker or TaskTracker.
    • + + +
+
+
+
+ + + + diff --git a/src/docs/src/documentation/content/xdocs/single_node_setup.xml b/src/docs/src/documentation/content/xdocs/single_node_setup.xml index edfa6d162be..9cc0f7bd316 100644 --- a/src/docs/src/documentation/content/xdocs/single_node_setup.xml +++ b/src/docs/src/documentation/content/xdocs/single_node_setup.xml @@ -97,7 +97,7 @@ -
+
Download

diff --git a/src/docs/src/documentation/content/xdocs/site.xml b/src/docs/src/documentation/content/xdocs/site.xml index fe384f3c10c..1004832499f 100644 --- a/src/docs/src/documentation/content/xdocs/site.xml +++ b/src/docs/src/documentation/content/xdocs/site.xml @@ -39,10 +39,12 @@ See http://forrest.apache.org/docs/linking.html for more info. + + @@ -69,6 +71,15 @@ See http://forrest.apache.org/docs/linking.html for more info. + + + + + + + + +