Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw)

This commit is contained in:
Allen Wittenauer 2015-01-15 07:48:55 -08:00
parent 533e551eb4
commit ce0117636a
6 changed files with 560 additions and 287 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -265,6 +265,8 @@ Trunk (Unreleased)
HDFS-7407. Minor typo in privileged pid/out/log names (aw) HDFS-7407. Minor typo in privileged pid/out/log names (aw)
HDFS-7581. HDFS documentation needs updating post-shell rewrite (aw)
Release 2.7.0 - UNRELEASED Release 2.7.0 - UNRELEASED
INCOMPATIBLE CHANGES INCOMPATIBLE CHANGES

20
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/Federation.apt.vm
View File

@ -64,7 +64,7 @@ HDFS Federation
In order to scale the name service horizontally, federation uses multiple In order to scale the name service horizontally, federation uses multiple
independent Namenodes/namespaces. The Namenodes are federated, that is, the independent Namenodes/namespaces. The Namenodes are federated, that is, the
Namenodes are independent and don’t require coordination with each other. Namenodes are independent and do not require coordination with each other.
The datanodes are used as common storage for blocks by all the Namenodes. The datanodes are used as common storage for blocks by all the Namenodes.
Each datanode registers with all the Namenodes in the cluster. Datanodes Each datanode registers with all the Namenodes in the cluster. Datanodes
send periodic heartbeats and block reports and handles commands from the send periodic heartbeats and block reports and handles commands from the
@ -202,7 +202,7 @@ HDFS Federation
<<Step 1>>: Format a namenode using the following command: <<Step 1>>: Format a namenode using the following command:
---- ----
> $HADOOP_PREFIX_HOME/bin/hdfs namenode -format [-clusterId <cluster_id>] [hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format [-clusterId <cluster_id>]
---- ----
Choose a unique cluster_id, which will not conflict other clusters in Choose a unique cluster_id, which will not conflict other clusters in
your environment. If it is not provided, then a unique ClusterID is your environment. If it is not provided, then a unique ClusterID is
@ -211,7 +211,7 @@ HDFS Federation
<<Step 2>>: Format additional namenode using the following command: <<Step 2>>: Format additional namenode using the following command:
---- ----
> $HADOOP_PREFIX_HOME/bin/hdfs namenode -format -clusterId <cluster_id> [hdfs]$ $HADOOP_PREFIX/bin/hdfs namenode -format -clusterId <cluster_id>
---- ----
Note that the cluster_id in step 2 must be same as that of the Note that the cluster_id in step 2 must be same as that of the
cluster_id in step 1. If they are different, the additional Namenodes cluster_id in step 1. If they are different, the additional Namenodes
@ -224,7 +224,7 @@ HDFS Federation
During upgrade you can provide a ClusterID as follows: During upgrade you can provide a ClusterID as follows:
---- ----
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID> [hdfs]$ $HADOOP_PREFIX/bin/hdfs start namenode --config $HADOOP_CONF_DIR -upgrade -clusterId <cluster_ID>
---- ----
If ClusterID is not provided, it is auto generated. If ClusterID is not provided, it is auto generated.
@ -248,7 +248,7 @@ HDFS Federation
the following command: the following command:
---- ----
> $HADOOP_PREFIX_HOME/bin/hdfs dfadmin -refreshNameNode <datanode_host_name>:<datanode_rpc_port> [hdfs]$ $HADOOP_PREFIX/bin/hdfs dfadmin -refreshNameNode <datanode_host_name>:<datanode_rpc_port>
---- ----
* The above command must be run against all the datanodes in the cluster. * The above command must be run against all the datanodes in the cluster.
@ -260,13 +260,13 @@ HDFS Federation
To start the cluster run the following command: To start the cluster run the following command:
---- ----
> $HADOOP_PREFIX_HOME/bin/start-dfs.sh [hdfs]$ $HADOOP_PREFIX/sbin/start-dfs.sh
---- ----
To stop the cluster run the following command: To stop the cluster run the following command:
---- ----
> $HADOOP_PREFIX_HOME/bin/stop-dfs.sh [hdfs]$ $HADOOP_PREFIX/sbin/stop-dfs.sh
---- ----
These commands can be run from any node where the HDFS configuration is These commands can be run from any node where the HDFS configuration is
@ -282,7 +282,7 @@ HDFS Federation
balance the cluster. Balancer can be run using the command: balance the cluster. Balancer can be run using the command:
---- ----
"$HADOOP_PREFIX"/bin/hadoop-daemon.sh --config $HADOOP_CONF_DIR --script "$bin"/hdfs start balancer [-policy <policy>] [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start balancer [-policy <policy>]
---- ----
Policy could be: Policy could be:
@ -307,13 +307,13 @@ HDFS Federation
following command: following command:
---- ----
"$HADOOP_PREFIX"/bin/distributed-exclude.sh <exclude_file> [hdfs]$ $HADOOP_PREFIX/sbin/distributed-exclude.sh <exclude_file>
---- ----
<<Step 2>>: Refresh all the Namenodes to pick up the new exclude file. <<Step 2>>: Refresh all the Namenodes to pick up the new exclude file.
---- ----
"$HADOOP_PREFIX"/bin/refresh-namenodes.sh [hdfs]$ $HADOOP_PREFIX/sbin/refresh-namenodes.sh
---- ----
The above command uses HDFS configuration to determine the Namenodes The above command uses HDFS configuration to determine the Namenodes

457
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm
View File

@ -18,7 +18,7 @@
HDFS Commands Guide HDFS Commands Guide
%{toc|section=1|fromDepth=2|toDepth=4} %{toc|section=1|fromDepth=2|toDepth=3}
* Overview * Overview
@ -26,39 +26,37 @@ HDFS Commands Guide
hdfs script without any arguments prints the description for all hdfs script without any arguments prints the description for all
commands. commands.
Usage: <<<hdfs [--config confdir] [--loglevel loglevel] [COMMAND] Usage: <<<hdfs [SHELL_OPTIONS] COMMAND [GENERIC_OPTIONS] [COMMAND_OPTIONS]>>>
[GENERIC_OPTIONS] [COMMAND_OPTIONS]>>>
Hadoop has an option parsing framework that employs parsing generic options Hadoop has an option parsing framework that employs parsing generic options as
as well as running classes. well as running classes.
*-----------------------+---------------+ *---------------+--------------+
|| COMMAND_OPTION || Description || COMMAND_OPTIONS || Description |
*-----------------------+---------------+ *-------------------------+-------------+
| <<<--config confdir>>>| Overwrites the default Configuration directory. | SHELL_OPTIONS | The common set of shell options. These are documented on the {{{../../hadoop-project-dist/hadoop-common/CommandsManual.html#Shell Options}Commands Manual}} page.
| | Default is <<<${HADOOP_HOME}/conf>>>. *-------------------------+----+
*-----------------------+---------------+ | GENERIC_OPTIONS | The common set of options supported by multiple commands. See the Hadoop {{{../../hadoop-project-dist/hadoop-common/CommandsManual.html#Generic Options}Commands Manual}} for more information.
| <<<--loglevel loglevel>>>| Overwrites the log level. Valid log levels are *------------------+---------------+
| | FATAL, ERROR, WARN, INFO, DEBUG, and TRACE. | COMMAND COMMAND_OPTIONS | Various commands with their options are described
| | Default is INFO. | | in the following sections. The commands have been
*-----------------------+---------------+ | | grouped into {{User Commands}} and
| GENERIC_OPTIONS | The common set of options supported by multiple | | {{Administration Commands}}.
| | commands. Full list is *-------------------------+--------------+
| | {{{../hadoop-common/CommandsManual.html#Generic_Options}here}}.
*-----------------------+---------------+
| COMMAND_OPTIONS | Various commands with their options are described in
| | the following sections. The commands have been
| | grouped into {{{User Commands}}} and
| | {{{Administration Commands}}}.
*-----------------------+---------------+
* User Commands * {User Commands}
Commands useful for users of a hadoop cluster. Commands useful for users of a hadoop cluster.
** <<<classpath>>>
Usage: <<<hdfs classpath>>>
Prints the class path needed to get the Hadoop jar and the required libraries
** <<<dfs>>> ** <<<dfs>>>
Usage: <<<hdfs dfs [GENERIC_OPTIONS] [COMMAND_OPTIONS]>>> Usage: <<<hdfs dfs [COMMAND [COMMAND_OPTIONS]]>>>
Run a filesystem command on the file system supported in Hadoop. Run a filesystem command on the file system supported in Hadoop.
The various COMMAND_OPTIONS can be found at The various COMMAND_OPTIONS can be found at
@ -66,43 +64,46 @@ HDFS Commands Guide
** <<<fetchdt>>> ** <<<fetchdt>>>
Gets Delegation Token from a NameNode. Usage: <<<hdfs fetchdt [--webservice <namenode_http_addr>] <path> >>>
See {{{./HdfsUserGuide.html#fetchdt}fetchdt}} for more info.
Usage: <<<hdfs fetchdt [GENERIC_OPTIONS]
[--webservice <namenode_http_addr>] <path> >>>
*------------------------------+---------------------------------------------+ *------------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description || COMMAND_OPTION || Description
*------------------------------+---------------------------------------------+ *------------------------------+---------------------------------------------+
| <fileName> | File name to store the token into.
*------------------------------+---------------------------------------------+
| --webservice <https_address> | use http protocol instead of RPC | --webservice <https_address> | use http protocol instead of RPC
*------------------------------+---------------------------------------------+ *------------------------------+---------------------------------------------+
| <fileName> | File name to store the token into.
*------------------------------+---------------------------------------------+
Gets Delegation Token from a NameNode.
See {{{./HdfsUserGuide.html#fetchdt}fetchdt}} for more info.
** <<<fsck>>> ** <<<fsck>>>
Runs a HDFS filesystem checking utility. Usage:
See {{{./HdfsUserGuide.html#fsck}fsck}} for more info.
Usage: <<<hdfs fsck [GENERIC_OPTIONS] <path> ---
hdfs fsck <path>
[-list-corruptfileblocks | [-list-corruptfileblocks |
[-move | -delete | -openforwrite] [-move | -delete | -openforwrite]
[-files [-blocks [-locations | -racks]]] [-files [-blocks [-locations | -racks]]]
[-includeSnapshots] [-showprogress]>>> [-includeSnapshots] [-showprogress]
---
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description || COMMAND_OPTION || Description
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| <path> | Start checking from this path. | <path> | Start checking from this path.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -move | Move corrupted files to /lost+found.
*------------------------+---------------------------------------------+
| -delete | Delete corrupted files. | -delete | Delete corrupted files.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -files | Print out files being checked. | -files | Print out files being checked.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -openforwrite | Print out files opened for write. | -files -blocks | Print out the block report
*------------------------+---------------------------------------------+
| -files -blocks -locations | Print out locations for every block.
*------------------------+---------------------------------------------+
| -files -blocks -racks | Print out network topology for data-node locations.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| | Include snapshot data if the given path | | Include snapshot data if the given path
| -includeSnapshots | indicates a snapshottable directory or | -includeSnapshots | indicates a snapshottable directory or
@ -111,52 +112,259 @@ HDFS Commands Guide
| -list-corruptfileblocks| Print out list of missing blocks and | -list-corruptfileblocks| Print out list of missing blocks and
| | files they belong to. | | files they belong to.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -blocks | Print out block report. | -move | Move corrupted files to /lost+found.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -locations | Print out locations for every block. | -openforwrite | Print out files opened for write.
*------------------------+---------------------------------------------+
| -racks | Print out network topology for data-node locations.
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
| -showprogress | Print out dots for progress in output. Default is OFF | -showprogress | Print out dots for progress in output. Default is OFF
| | (no progress). | | (no progress).
*------------------------+---------------------------------------------+ *------------------------+---------------------------------------------+
Runs the HDFS filesystem checking utility.
See {{{./HdfsUserGuide.html#fsck}fsck}} for more info.
** <<<getconf>>>
Usage:
---
hdfs getconf -namenodes
hdfs getconf -secondaryNameNodes
hdfs getconf -backupNodes
hdfs getconf -includeFile
hdfs getconf -excludeFile
hdfs getconf -nnRpcAddresses
hdfs getconf -confKey [key]
---
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -namenodes | gets list of namenodes in the cluster.
*------------------------+---------------------------------------------+
| -secondaryNameNodes | gets list of secondary namenodes in the cluster.
*------------------------+---------------------------------------------+
| -backupNodes | gets list of backup nodes in the cluster.
*------------------------+---------------------------------------------+
| -includeFile | gets the include file path that defines the datanodes that can join the cluster.
*------------------------+---------------------------------------------+
| -excludeFile | gets the exclude file path that defines the datanodes that need to decommissioned.
*------------------------+---------------------------------------------+
| -nnRpcAddresses | gets the namenode rpc addresses
*------------------------+---------------------------------------------+
| -confKey [key] | gets a specific key from the configuration
*------------------------+---------------------------------------------+
Gets configuration information from the configuration directory, post-processing.
** <<<groups>>>
Usage: <<<hdfs groups [username ...]>>>
Returns the group information given one or more usernames.
** <<<lsSnapshottableDir>>>
Usage: <<<hdfs lsSnapshottableDir [-help]>>>
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -help | print help
*------------------------+---------------------------------------------+
Get the list of snapshottable directories. When this is run as a super user,
it returns all snapshottable directories. Otherwise it returns those directories
that are owned by the current user.
** <<<jmxget>>>
Usage: <<<hdfs jmxget [-localVM ConnectorURL | -port port | -server mbeanserver | -service service]>>>
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -help | print help
*------------------------+---------------------------------------------+
| -localVM ConnectorURL | connect to the VM on the same machine
*------------------------+---------------------------------------------+
| -port <mbean server port> | specify mbean server port, if missing
| | it will try to connect to MBean Server in
| | the same VM
*------------------------+---------------------------------------------+
| -service | specify jmx service, either DataNode or NameNode, the default
*------------------------+---------------------------------------------+
Dump JMX information from a service.
** <<<oev>>>
Usage: <<<hdfs oev [OPTIONS] -i INPUT_FILE -o OUTPUT_FILE>>>
*** Required command line arguments:
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
|-i,--inputFile <arg> | edits file to process, xml (case
| insensitive) extension means XML format,
| any other filename means binary format
*------------------------+---------------------------------------------+
| -o,--outputFile <arg> | Name of output file. If the specified
| file exists, it will be overwritten,
| format of the file is determined
| by -p option
*------------------------+---------------------------------------------+
*** Optional command line arguments:
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -f,--fix-txids | Renumber the transaction IDs in the input,
| so that there are no gaps or invalid transaction IDs.
*------------------------+---------------------------------------------+
| -h,--help | Display usage information and exit
*------------------------+---------------------------------------------+
| -r,--recover | When reading binary edit logs, use recovery
| mode. This will give you the chance to skip
| corrupt parts of the edit log.
*------------------------+---------------------------------------------+
| -p,--processor <arg> | Select which type of processor to apply
| against image file, currently supported
| processors are: binary (native binary format
| that Hadoop uses), xml (default, XML
| format), stats (prints statistics about
| edits file)
*------------------------+---------------------------------------------+
| -v,--verbose | More verbose output, prints the input and
| output filenames, for processors that write
| to a file, also output to screen. On large
| image files this will dramatically increase
| processing time (default is false).
*------------------------+---------------------------------------------+
Hadoop offline edits viewer.
** <<<oiv>>>
Usage: <<<hdfs oiv [OPTIONS] -i INPUT_FILE>>>
*** Required command line arguments:
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
|-i,--inputFile <arg> | edits file to process, xml (case
| insensitive) extension means XML format,
| any other filename means binary format
*------------------------+---------------------------------------------+
*** Optional command line arguments:
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -h,--help | Display usage information and exit
*------------------------+---------------------------------------------+
| -o,--outputFile <arg> | Name of output file. If the specified
| file exists, it will be overwritten,
| format of the file is determined
| by -p option
*------------------------+---------------------------------------------+
| -p,--processor <arg> | Select which type of processor to apply
| against image file, currently supported
| processors are: binary (native binary format
| that Hadoop uses), xml (default, XML
| format), stats (prints statistics about
| edits file)
*------------------------+---------------------------------------------+
Hadoop Offline Image Viewer for newer image files.
** <<<oiv_legacy>>>
Usage: <<<hdfs oiv_legacy [OPTIONS] -i INPUT_FILE -o OUTPUT_FILE>>>
*------------------------+---------------------------------------------+
|| COMMAND_OPTION || Description
*------------------------+---------------------------------------------+
| -h,--help | Display usage information and exit
*------------------------+---------------------------------------------+
|-i,--inputFile <arg> | edits file to process, xml (case
| insensitive) extension means XML format,
| any other filename means binary format
*------------------------+---------------------------------------------+
| -o,--outputFile <arg> | Name of output file. If the specified
| file exists, it will be overwritten,
| format of the file is determined
| by -p option
*------------------------+---------------------------------------------+
Hadoop offline image viewer for older versions of Hadoop.
** <<<snapshotDiff>>>
Usage: <<<hdfs snapshotDiff <path> <fromSnapshot> <toSnapshot> >>>
Determine the difference between HDFS snapshots. See the
{{{./HdfsSnapshots.html#Get_Snapshots_Difference_Report}HDFS Snapshot Documentation}} for more information.
** <<<version>>> ** <<<version>>>
Prints the version.
Usage: <<<hdfs version>>> Usage: <<<hdfs version>>>
Prints the version.
* Administration Commands * Administration Commands
Commands useful for administrators of a hadoop cluster. Commands useful for administrators of a hadoop cluster.
** <<<balancer>>> ** <<<balancer>>>
Runs a cluster balancing utility. An administrator can simply press Ctrl-C
to stop the rebalancing process. See
{{{./HdfsUserGuide.html#Balancer}Balancer}} for more details.
Usage: <<<hdfs balancer [-threshold <threshold>] [-policy <policy>]>>> Usage: <<<hdfs balancer [-threshold <threshold>] [-policy <policy>]>>>
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
|| COMMAND_OPTION | Description || COMMAND_OPTION | Description
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
| -threshold <threshold> | Percentage of disk capacity. This overwrites the
| | default threshold.
*------------------------+----------------------------------------------------+
| -policy <policy> | <<<datanode>>> (default): Cluster is balanced if | -policy <policy> | <<<datanode>>> (default): Cluster is balanced if
| | each datanode is balanced. \ | | each datanode is balanced. \
| | <<<blockpool>>>: Cluster is balanced if each block | | <<<blockpool>>>: Cluster is balanced if each block
| | pool in each datanode is balanced. | | pool in each datanode is balanced.
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
| -threshold <threshold> | Percentage of disk capacity. This overwrites the
| | default threshold.
*------------------------+----------------------------------------------------+
Runs a cluster balancing utility. An administrator can simply press Ctrl-C
to stop the rebalancing process. See
{{{./HdfsUserGuide.html#Balancer}Balancer}} for more details.
Note that the <<<blockpool>>> policy is more strict than the <<<datanode>>> Note that the <<<blockpool>>> policy is more strict than the <<<datanode>>>
policy. policy.
** <<<datanode>>> ** <<<cacheadmin>>>
Runs a HDFS datanode. Usage: <<<hdfs cacheadmin -addDirective -path <path> -pool <pool-name> [-force] [-replication <replication>] [-ttl <time-to-live>]>>>
See the {{{./CentralizedCacheManagement.html#cacheadmin_command-line_interface}HDFS Cache Administration Documentation}} for more information.
** <<<crypto>>>
Usage:
---
hdfs crypto -createZone -keyName <keyName> -path <path>
hdfs crypto -help <command-name>
hdfs crypto -listZones
---
See the {{{./TransparentEncryption.html#crypto_command-line_interface}HDFS Transparent Encryption Documentation}} for more information.
** <<<datanode>>>
Usage: <<<hdfs datanode [-regular | -rollback | -rollingupgrace rollback]>>> Usage: <<<hdfs datanode [-regular | -rollback | -rollingupgrace rollback]>>>
@ -172,12 +380,14 @@ HDFS Commands Guide
| -rollingupgrade rollback | Rollback a rolling upgrade operation. | -rollingupgrade rollback | Rollback a rolling upgrade operation.
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
Runs a HDFS datanode.
** <<<dfsadmin>>> ** <<<dfsadmin>>>
Runs a HDFS dfsadmin client. Usage:
+------------------------------------------+ ------------------------------------------
Usage: hdfs dfsadmin [GENERIC_OPTIONS] hdfs dfsadmin [GENERIC_OPTIONS]
[-report [-live] [-dead] [-decommissioning]] [-report [-live] [-dead] [-decommissioning]]
[-safemode enter | leave | get | wait] [-safemode enter | leave | get | wait]
[-saveNamespace] [-saveNamespace]
@ -210,7 +420,7 @@ HDFS Commands Guide
[-getDatanodeInfo <datanode_host:ipc_port>] [-getDatanodeInfo <datanode_host:ipc_port>]
[-triggerBlockReport [-incremental] <datanode_host:ipc_port>] [-triggerBlockReport [-incremental] <datanode_host:ipc_port>]
[-help [cmd]] [-help [cmd]]
+------------------------------------------+ ------------------------------------------
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
|| COMMAND_OPTION || Description || COMMAND_OPTION || Description
@ -323,11 +533,11 @@ HDFS Commands Guide
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
| -allowSnapshot \<snapshotDir\> | Allowing snapshots of a directory to be | -allowSnapshot \<snapshotDir\> | Allowing snapshots of a directory to be
| created. If the operation completes successfully, the | created. If the operation completes successfully, the
| directory becomes snapshottable. | directory becomes snapshottable. See the {{{./HdfsSnapshots.html}HDFS Snapshot Documentation}} for more information.
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
| -disallowSnapshot \<snapshotDir\> | Disallowing snapshots of a directory to | -disallowSnapshot \<snapshotDir\> | Disallowing snapshots of a directory to
| be created. All snapshots of the directory must be deleted | be created. All snapshots of the directory must be deleted
| before disallowing snapshots. | before disallowing snapshots. See the {{{./HdfsSnapshots.html}HDFS Snapshot Documentation}} for more information.
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
| -fetchImage \<local directory\> | Downloads the most recent fsimage from the | -fetchImage \<local directory\> | Downloads the most recent fsimage from the
| NameNode and saves it in the specified local directory. | NameNode and saves it in the specified local directory.
@ -351,30 +561,68 @@ HDFS Commands Guide
| is specified. | is specified.
*-----------------+-----------------------------------------------------------+ *-----------------+-----------------------------------------------------------+
** <<<mover>>> Runs a HDFS dfsadmin client.
Runs the data migration utility. ** <<<haadmin>>>
See {{{./ArchivalStorage.html#Mover_-_A_New_Data_Migration_Tool}Mover}} for more details.
Usage:
---
hdfs haadmin -checkHealth <serviceId>
hdfs haadmin -failover [--forcefence] [--forceactive] <serviceId> <serviceId>
hdfs haadmin -getServiceState <serviceId>
hdfs haadmin -help <command>
hdfs haadmin -transitionToActive <serviceId> [--forceactive]
hdfs haadmin -transitionToStandby <serviceId>
---
*--------------------+--------------------------------------------------------+
|| COMMAND_OPTION || Description
*--------------------+--------------------------------------------------------+
| -checkHealth | check the health of the given NameNode
*--------------------+--------------------------------------------------------+
| -failover | initiate a failover between two NameNodes
*--------------------+--------------------------------------------------------+
| -getServiceState | determine whether the given NameNode is Active or Standby
*--------------------+--------------------------------------------------------+
| -transitionToActive | transition the state of the given NameNode to Active (Warning: No fencing is done)
*--------------------+--------------------------------------------------------+
| -transitionToStandby | transition the state of the given NameNode to Standby (Warning: No fencing is done)
*--------------------+--------------------------------------------------------+
See {{{./HDFSHighAvailabilityWithNFS.html#Administrative_commands}HDFS HA with NFS}} or
{{{./HDFSHighAvailabilityWithQJM.html#Administrative_commands}HDFS HA with QJM}} for more
information on this command.
** <<<journalnode>>>
Usage: <<<hdfs journalnode>>>
This comamnd starts a journalnode for use with {{{./HDFSHighAvailabilityWithQJM.html#Administrative_commands}HDFS HA with QJM}}.
** <<<mover>>>
Usage: <<<hdfs mover [-p <files/dirs> | -f <local file name>]>>> Usage: <<<hdfs mover [-p <files/dirs> | -f <local file name>]>>>
*--------------------+--------------------------------------------------------+ *--------------------+--------------------------------------------------------+
|| COMMAND_OPTION || Description || COMMAND_OPTION || Description
*--------------------+--------------------------------------------------------+ *--------------------+--------------------------------------------------------+
| -p \<files/dirs\> | Specify a space separated list of HDFS files/dirs to migrate.
*--------------------+--------------------------------------------------------+
| -f \<local file\> | Specify a local file containing a list of HDFS files/dirs to migrate. | -f \<local file\> | Specify a local file containing a list of HDFS files/dirs to migrate.
*--------------------+--------------------------------------------------------+ *--------------------+--------------------------------------------------------+
| -p \<files/dirs\> | Specify a space separated list of HDFS files/dirs to migrate.
*--------------------+--------------------------------------------------------+
Runs the data migration utility.
See {{{./ArchivalStorage.html#Mover_-_A_New_Data_Migration_Tool}Mover}} for more details.
Note that, when both -p and -f options are omitted, the default path is the root directory. Note that, when both -p and -f options are omitted, the default path is the root directory.
** <<<namenode>>> ** <<<namenode>>>
Runs the namenode. More info about the upgrade, rollback and finalize is at Usage:
{{{./HdfsUserGuide.html#Upgrade_and_Rollback}Upgrade Rollback}}.
+------------------------------------------+ ------------------------------------------
Usage: hdfs namenode [-backup] | hdfs namenode [-backup] |
[-checkpoint] | [-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] | [-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReserved<k-v pairs>] ] | [-upgrade [-clusterid cid] [-renameReserved<k-v pairs>] ] |
@ -387,7 +635,7 @@ HDFS Commands Guide
[-bootstrapStandby] | [-bootstrapStandby] |
[-recover [-force] ] | [-recover [-force] ] |
[-metadataVersion ] [-metadataVersion ]
+------------------------------------------+ ------------------------------------------
*--------------------+--------------------------------------------------------+ *--------------------+--------------------------------------------------------+
|| COMMAND_OPTION || Description || COMMAND_OPTION || Description
@ -443,11 +691,23 @@ HDFS Commands Guide
| metadata versions of the software and the image. | metadata versions of the software and the image.
*--------------------+--------------------------------------------------------+ *--------------------+--------------------------------------------------------+
** <<<secondarynamenode>>> Runs the namenode. More info about the upgrade, rollback and finalize is at
{{{./HdfsUserGuide.html#Upgrade_and_Rollback}Upgrade Rollback}}.
Runs the HDFS secondary namenode.
See {{{./HdfsUserGuide.html#Secondary_NameNode}Secondary Namenode}} ** <<<nfs3>>>
for more info.
Usage: <<<hdfs nfs3>>>
This comamnd starts the NFS3 gateway for use with the {{{./HdfsNfsGateway.html#Start_and_stop_NFS_gateway_service}HDFS NFS3 Service}}.
** <<<portmap>>>
Usage: <<<hdfs portmap>>>
This comamnd starts the RPC portmap for use with the {{{./HdfsNfsGateway.html#Start_and_stop_NFS_gateway_service}HDFS NFS3 Service}}.
** <<<secondarynamenode>>>
Usage: <<<hdfs secondarynamenode [-checkpoint [force]] | [-format] | Usage: <<<hdfs secondarynamenode [-checkpoint [force]] | [-format] |
[-geteditsize]>>> [-geteditsize]>>>
@ -465,6 +725,33 @@ HDFS Commands Guide
| the NameNode. | the NameNode.
*----------------------+------------------------------------------------------+ *----------------------+------------------------------------------------------+
Runs the HDFS secondary namenode.
See {{{./HdfsUserGuide.html#Secondary_NameNode}Secondary Namenode}}
for more info.
** <<<storagepolicies>>>
Usage: <<<hdfs storagepolicies>>>
Lists out all storage policies. See the {{{./ArchivalStorage.html}HDFS Storage Policy Documentation}} for more information.
** <<<zkfc>>>
Usage: <<<hdfs zkfc [-formatZK [-force] [-nonInteractive]]>>>
*----------------------+------------------------------------------------------+
|| COMMAND_OPTION || Description
*----------------------+------------------------------------------------------+
| -formatZK | Format the Zookeeper instance
*----------------------+------------------------------------------------------+
| -h | Display help
*----------------------+------------------------------------------------------+
This comamnd starts a Zookeeper Failover Controller process for use with {{{./HDFSHighAvailabilityWithQJM.html#Administrative_commands}HDFS HA with QJM}}.
* Debug Commands * Debug Commands
Useful commands to help administrators debug HDFS issues, like validating Useful commands to help administrators debug HDFS issues, like validating
@ -472,30 +759,25 @@ HDFS Commands Guide
** <<<verify>>> ** <<<verify>>>
Verify HDFS metadata and block files. If a block file is specified, we
will verify that the checksums in the metadata file match the block
file.
Usage: <<<hdfs dfs verify [-meta <metadata-file>] [-block <block-file>]>>> Usage: <<<hdfs dfs verify [-meta <metadata-file>] [-block <block-file>]>>>
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
|| COMMAND_OPTION | Description || COMMAND_OPTION | Description
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
| -meta <metadata-file> | Absolute path for the metadata file on the local file
| | system of the data node.
*------------------------+----------------------------------------------------+
| -block <block-file> | Optional parameter to specify the absolute path for | -block <block-file> | Optional parameter to specify the absolute path for
| | the block file on the local file system of the data | | the block file on the local file system of the data
| | node. | | node.
*------------------------+----------------------------------------------------+
| -meta <metadata-file> | Absolute path for the metadata file on the local file
| | system of the data node.
*------------------------+----------------------------------------------------+ *------------------------+----------------------------------------------------+
Verify HDFS metadata and block files. If a block file is specified, we
will verify that the checksums in the metadata file match the block
file.
** <<<recoverLease>>> ** <<<recoverLease>>>
Recover the lease on the specified path. The path must reside on an
HDFS filesystem. The default number of retries is 1.
Usage: <<<hdfs dfs recoverLease [-path <path>] [-retries <num-retries>]>>> Usage: <<<hdfs dfs recoverLease [-path <path>] [-retries <num-retries>]>>>
*-------------------------------+--------------------------------------------+ *-------------------------------+--------------------------------------------+
@ -507,3 +789,6 @@ HDFS Commands Guide
| | recoverLease. The default number of retries | | recoverLease. The default number of retries
| | is 1. | | is 1.
*-------------------------------+---------------------------------------------+ *-------------------------------+---------------------------------------------+
Recover the lease on the specified path. The path must reside on an
HDFS filesystem. The default number of retries is 1.

6
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSHighAvailabilityWithNFS.apt.vm
View File

@ -585,7 +585,7 @@ Usage: DFSHAAdmin [-ns <nameserviceId>]
from one of the NameNode hosts. from one of the NameNode hosts.
---- ----
$ hdfs zkfc -formatZK [hdfs]$ $HADOOP_PREFIX/bin/zkfc -formatZK
---- ----
This will create a znode in ZooKeeper inside of which the automatic failover This will create a znode in ZooKeeper inside of which the automatic failover
@ -605,7 +605,7 @@ $ hdfs zkfc -formatZK
can start the daemon by running: can start the daemon by running:
---- ----
$ hadoop-daemon.sh start zkfc [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start zkfc
---- ----
** Securing access to ZooKeeper ** Securing access to ZooKeeper
@ -646,7 +646,7 @@ digest:hdfs-zkfcs:mypassword
a command like the following: a command like the following:
---- ----
$ java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword [hdfs]$ java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword
output: hdfs-zkfcs:mypassword->hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs= output: hdfs-zkfcs:mypassword->hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs=
---- ----

6
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSHighAvailabilityWithQJM.apt.vm
View File

@ -623,7 +623,7 @@ Usage: DFSHAAdmin [-ns <nameserviceId>]
from one of the NameNode hosts. from one of the NameNode hosts.
---- ----
$ hdfs zkfc -formatZK [hdfs]$ $HADOOP_PREFIX/bin/hdfs zkfc -formatZK
---- ----
This will create a znode in ZooKeeper inside of which the automatic failover This will create a znode in ZooKeeper inside of which the automatic failover
@ -643,7 +643,7 @@ $ hdfs zkfc -formatZK
can start the daemon by running: can start the daemon by running:
---- ----
$ hadoop-daemon.sh start zkfc [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start zkfc
---- ----
** Securing access to ZooKeeper ** Securing access to ZooKeeper
@ -684,7 +684,7 @@ digest:hdfs-zkfcs:mypassword
a command like the following: a command like the following:
---- ----
$ java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword [hdfs]$ java -cp $ZK_HOME/lib/*:$ZK_HOME/zookeeper-3.4.2.jar org.apache.zookeeper.server.auth.DigestAuthenticationProvider hdfs-zkfcs:mypassword
output: hdfs-zkfcs:mypassword->hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs= output: hdfs-zkfcs:mypassword->hdfs-zkfcs:P/OQvnYyU/nF/mGYvB/xurX8dYs=
---- ----

36
hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
View File

@ -199,23 +199,17 @@ HDFS NFS Gateway
{{{https://bugzilla.redhat.com/show_bug.cgi?id=731542}rpcbind bug}}. More detailed discussions can {{{https://bugzilla.redhat.com/show_bug.cgi?id=731542}rpcbind bug}}. More detailed discussions can
be found in {{{https://issues.apache.org/jira/browse/HDFS-4763}HDFS-4763}}. be found in {{{https://issues.apache.org/jira/browse/HDFS-4763}HDFS-4763}}.
[[1]] Stop nfs/rpcbind/portmap services provided by the platform (commands can be different on various Unix platforms): [[1]] Stop nfsv3 and rpcbind/portmap services provided by the platform (commands can be different on various Unix platforms):
------------------------- -------------------------
service nfs stop [root]> service nfs stop
[root]> service rpcbind stop
service rpcbind stop
------------------------- -------------------------
[[2]] Start Hadoop's portmap (needs root privileges):
[[2]] Start package included portmap (needs root privileges):
------------------------- -------------------------
hdfs portmap [root]> $HADOOP_PREFIX/bin/hdfs --daemon start portmap
OR
hadoop-daemon.sh start portmap
------------------------- -------------------------
[[3]] Start mountd and nfsd. [[3]] Start mountd and nfsd.
@ -226,22 +220,14 @@ HDFS NFS Gateway
as long as the user has read access to the Kerberos keytab defined in "nfs.keytab.file". as long as the user has read access to the Kerberos keytab defined in "nfs.keytab.file".
------------------------- -------------------------
hdfs nfs3 [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon start nfs3
OR
hadoop-daemon.sh start nfs3
------------------------- -------------------------
Note, if the hadoop-daemon.sh script starts the NFS gateway, its log can be found in the hadoop log folder.
[[4]] Stop NFS gateway services. [[4]] Stop NFS gateway services.
------------------------- -------------------------
hadoop-daemon.sh stop nfs3 [hdfs]$ $HADOOP_PREFIX/bin/hdfs --daemon stop nfs3
[root]> $HADOOP_PREFIX/bin/hdfs --daemon stop portmap
hadoop-daemon.sh stop portmap
------------------------- -------------------------
Optionally, you can forgo running the Hadoop-provided portmap daemon and Optionally, you can forgo running the Hadoop-provided portmap daemon and
@ -263,7 +249,7 @@ HDFS NFS Gateway
[[1]] Execute the following command to verify if all the services are up and running: [[1]] Execute the following command to verify if all the services are up and running:
------------------------- -------------------------
rpcinfo -p $nfs_server_ip [root]> rpcinfo -p $nfs_server_ip
------------------------- -------------------------
You should see output similar to the following: You should see output similar to the following:
@ -293,7 +279,7 @@ HDFS NFS Gateway
[[2]] Verify if the HDFS namespace is exported and can be mounted. [[2]] Verify if the HDFS namespace is exported and can be mounted.
------------------------- -------------------------
showmount -e $nfs_server_ip [root]> showmount -e $nfs_server_ip
------------------------- -------------------------
You should see output similar to the following: You should see output similar to the following:
@ -319,7 +305,7 @@ HDFS NFS Gateway
The users can mount the HDFS namespace as shown below: The users can mount the HDFS namespace as shown below:
------------------------------------------------------------------- -------------------------------------------------------------------
mount -t nfs -o vers=3,proto=tcp,nolock,noacl $server:/ $mount_point [root]>mount -t nfs -o vers=3,proto=tcp,nolock,noacl $server:/ $mount_point
------------------------------------------------------------------- -------------------------------------------------------------------
Then the users can access HDFS as part of the local file system except that, Then the users can access HDFS as part of the local file system except that,