Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

HDFS-9860. Backport HDFS-9638 to branch-2.7. Contributed by Wei-Chiu Chuang.

This commit is contained in:
Akira Ajisaka 2016-03-17 02:59:51 +09:00
parent b8e73cff8f
commit 6902789a92
4 changed files with 9 additions and 5 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
View File

@ -44,6 +44,8 @@ Release 2.7.3 - UNRELEASED
HDFS-9048. DistCp documentation is out-of-dated
(Daisuke Kobayashi via iwasakims)
HDFS-9860. Backport HDFS-9638 to branch-2.7. (Wei-Chiu Chuang via aajisaka)
OPTIMIZATIONS
HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via

2
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptionSwitch.java
View File

@ -82,7 +82,7 @@ public enum DistCpOptionSwitch {
*/
SSL_CONF(DistCpConstants.CONF_LABEL_SSL_CONF,
new Option("mapredSslConf", true, "Configuration for ssl config file" +
", to use with hftps://")),
", to use with hftps://. Must be in the classpath.")),
/**
* Max number of maps to use during copy. DistCp will split work

7
hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
View File

@ -216,7 +216,7 @@ Command Line Options
Flag | Description | Notes
----------------- | ------------------------------------ | --------
`-p[rbugpcax]` | Preserve r: replication number b: block size u: user g: group p: permission c: checksum-type a: ACL x: XAttr | Modification times are not preserved. Also, when `-update` is specified, status updates will **not** be synchronized unless the file sizes also differ (i.e. unless the file is re-created). If -pa is specified, DistCp preserves the permissions also because ACLs are a super-set of permissions.
`-p[rbugpcaxt]` | Preserve r: replication number b: block size u: user g: group p: permission c: checksum-type a: ACL x: XAttr t: timestamp | When `-update` is specified, status updates will **not** be synchronized unless the file sizes also differ (i.e. unless the file is re-created). If -pa is specified, DistCp preserves the permissions also because ACLs are a super-set of permissions.
`-i` | Ignore failures | As explained in the Appendix, this option will keep more accurate statistics about the copy than the default case. It also preserves logs from failed copies, which can be valuable for debugging. Finally, a failing map will not cause the job to fail before all splits are attempted.
`-log <logdir>` | Write logs to \<logdir\> | DistCp keeps logs of each file it attempts to copy as map output. If a map fails, the log output will not be retained if it is re-executed.
`-m <num_maps>` | Maximum number of simultaneous copies | Specify the number of maps to copy data. Note that more maps may not necessarily improve throughput.
@ -231,6 +231,8 @@ Flag | Description | Notes
`-atomic {-tmp <tmp_dir>}` | Specify atomic commit, with optional tmp directory. | `-atomic` instructs DistCp to copy the source data to a temporary target location, and then move the temporary target to the final-location atomically. Data will either be available at final target in a complete and consistent form, or not at all. Optionally, `-tmp` may be used to specify the location of the tmp-target. If not specified, a default is chosen. **Note:** tmp_dir must be on the final target cluster.
`-mapredSslConf <ssl_conf_file>` | Specify SSL Config file, to be used with HSFTP source | When using the hsftp protocol with a source, the security- related properties may be specified in a config-file and passed to DistCp. \<ssl_conf_file\> needs to be in the classpath.
`-async` | Run DistCp asynchronously. Quits as soon as the Hadoop Job is launched. | The Hadoop Job-id is logged, for tracking.
`-diff` | Use snapshot diff report to identify the difference between source and target. |
`-skipcrccheck` | Whether to skip CRC checks between source and target paths. |
Architecture of DistCp
----------------------
@ -439,8 +441,7 @@ $H3 SSL Configurations for HSFTP sources
* `ssl.client.truststore.password`: (Optional) Password for the trust-store
file.
The following is an example of the contents of the contents of a SSL
Configuration file:
The following is an example SSL configuration file:
<configuration>
<property>

3
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
View File

@ -459,7 +459,7 @@ public class TestOptionsParser {
Assert.assertFalse(options.shouldPreserve(FileAttribute.XATTR));
options = OptionsParser.parse(new String[] {
"-pbrgupcax",
"-pbrgupcaxt",
"-f",
"hdfs://localhost:8020/source/first",
"hdfs://localhost:8020/target/"});
@ -471,6 +471,7 @@ public class TestOptionsParser {
Assert.assertTrue(options.shouldPreserve(FileAttribute.CHECKSUMTYPE));
Assert.assertTrue(options.shouldPreserve(FileAttribute.ACL));
Assert.assertTrue(options.shouldPreserve(FileAttribute.XATTR));
Assert.assertTrue(options.shouldPreserve(FileAttribute.TIMES));
options = OptionsParser.parse(new String[] {
"-pc",