From 8095c612a39af51680df37c312f52f6afd43fc63 Mon Sep 17 00:00:00 2001 From: Masatake Iwasaki Date: Thu, 3 Mar 2016 18:57:23 +0900 Subject: [PATCH] HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) (cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383) --- hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt | 3 +++ .../hadoop-distcp/src/site/markdown/DistCp.md.vm | 13 +++++++------ 2 files changed, 10 insertions(+), 6 deletions(-) diff --git a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt index 759bb133f38..ad9a1b70f09 100644 --- a/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt +++ b/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt @@ -1963,6 +1963,9 @@ Release 2.7.3 - UNRELEASED HDFS-8791. block ID-based DN storage layout can be very slow for datanode on ext4 (Chris Trezzo via kihwal) + HDFS-9048. DistCp documentation is out-of-dated + (Daisuke Kobayashi via iwasakims) + OPTIMIZATIONS BUG FIXES diff --git a/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm b/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm index 31f0444b0d3..0faa9759f4e 100644 --- a/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm +++ b/hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm @@ -412,12 +412,13 @@ $H3 Map sizing $H3 Copying Between Versions of HDFS - For copying between two different versions of Hadoop, one will usually use - HftpFileSystem. This is a read-only FileSystem, so DistCp must be run on the - destination cluster (more specifically, on NodeManagers that can write to the - destination cluster). Each source is specified as - `hftp:///` (the default `dfs.http.address` is - `:50070`). + For copying between two different major versions of Hadoop (e.g. between 1.X + and 2.X), one will usually use WebHdfsFileSystem. Unlike the previous + HftpFileSystem, as webhdfs is available for both read and write operations, + DistCp can be run on both source and destination cluster. + Remote cluster is specified as `webhdfs://:`. + When copying between same major versions of Hadoop cluster (e.g. between 2.X + and 2.X), use hdfs protocol for better performance. $H3 MapReduce and other side-effects