Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,817 Commits 326 Branches 371 Tags
Commit Graph

142 Commits

Author SHA1 Message Date
Masatake Iwasaki
50e30bd30a HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

Conflicts:
	hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

(cherry picked from commit 7985d9b1ced4371f4cdc48ea74fbf120eab50309)

Conflicts:
	hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
 2019-04-12 12:35:32 +09:00
Kai Xie
a49cb4465e
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-02-27 20:09:27 +00:00
Kai Xie
6d3e7a8570
HADOOP-16049. DistCp result has data and checksum mismatch when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-01-27 16:56:57 +00:00
Wei-Chiu Chuang
f2739f3f59 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
(cherry picked from commit a7dd244a4995cbf20feb054530fc5913ccf384ac)

	modified:   hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyCommitter.java
(cherry picked from commit 84726679f0441bfa0788ad5a5ac2bfa4353afedb)

(cherry picked from commit 8360d15abc29e9ed5f7a2117ab6ac882248e8458)
 2018-10-24 09:39:31 -07:00
Junping Du
2b48854cfd HADOOP-15385. Many tests are failing in hadoop-distcp project in branch-2. Contributed by Jason Lowe. 2018-04-25 10:11:41 +08:00
Surendra Singh Lilhore
8d69b7f2d4 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-13 10:29:19 +05:30
Akira Ajisaka
82fc80a987
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca409f5e57e540849a80e6448bb4924cc6)
 2017-11-02 18:33:25 +09:00
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Steve Loughran
062d9c9bab HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc847560d53e7955ced9ce7ce2773c805793)
 2017-10-20 22:27:59 +01:00
Xiaoyu Yao
b669aca518 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-11 11:40:48 -07:00
Yongjun Zhang
dd552a97b7 HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-24 19:05:40 -07:00
Yongjun Zhang
49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang
97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li." 
This reverts commit c411adac32cda42433b15979253248336487d977.
 2017-05-24 18:54:51 -07:00
Yongjun Zhang
39474ed50d Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)" 
This reverts commit fe185e2c3a07d4e6a82ed4fd255c98ec8b561ad6.
 2017-05-24 18:54:50 -07:00
Sunil G
ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Yongjun Zhang
fe185e2c3a HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-19 21:11:38 -07:00
Yongjun Zhang
c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Chris Douglas
72aa0a7ab3 HADOOP-13626. Remove distcp dependency on FileStatus serialization 
(cherry picked from commit a1a0281e12ea96476e75b076f76d5b5eb5254eea)
 2017-04-05 15:47:46 -07:00
Yongjun Zhang
16ca70553c HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 
(cherry picked from commit d235dcdf0b6f36162280e32c7c0342b7970c9b7b)
 2017-02-27 20:46:17 -08:00
Steve Loughran
b2704702f9 HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fcca5127c9908156971eeec468371f47)
 2017-01-19 11:25:20 +00:00
Chris Nauroth
9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e130230457fc897ed7a2a09e14d078be90a)
 2017-01-05 15:05:01 -08:00
Mingliang Liu
9af7d1e298 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:19:32 -08:00
Mingliang Liu
4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b6443715b7a741296c1bd520a1840f9a7c)
 2016-11-15 11:36:08 -08:00
Masatake Iwasaki
7e918637de HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82a4510f16df49238d57c9f78ac28ae7)
 2016-10-27 15:54:03 +09:00
Yongjun Zhang
ce72314657 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 
(cherry picked from commit 0f0c15f7a5ea33ced781978bea971f3750883f41)
 2016-10-25 12:29:28 -07:00
Yongjun Zhang
7bc170ba26 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 
(cherry picked from commit 8650cc84f20e7d8c32dcdcd91c94372d476e2276)

Conflicts
	hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
 2016-10-25 11:32:26 -07:00
Allen Wittenauer
a5a56c3564 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 
(cherry picked from commit 422c73a8657d8699920f7db13d4be200e16c4272)
 2016-10-17 16:35:32 -07:00
Jing Zhao
ad69baf6a9 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d079838f532a13ca237300386d1b3bc1b178)
 2016-10-13 13:26:02 -07:00
Brahma Reddy Battula
5737d04c5f HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970bea8213660bb6c550104783069153236)
 2016-09-27 10:47:07 +05:30
Steve Loughran
247cdbb86a HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:32 +01:00
Chris Nauroth
e19f910245 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb5139769eb55893971b43b9c23da9513a784)
 2016-09-19 15:16:56 -07:00
Andrew Wang
38b1eafdbc HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8dbdc1841f225a34346d0392fa76a47)
 2016-07-11 18:06:35 -07:00
Yongjun Zhang
8a121dc3b8 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3af913e6dcd09626953c64d43168efbd)
 2016-06-28 23:29:48 -07:00
Jing Zhao
856bc4e28b HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 2016-06-22 15:15:47 -07:00
Yongjun Zhang
82a88c7e67 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee72a27382a26bf450bb8b16784aeebbb)
 2016-05-26 23:32:46 -07:00
Steve Loughran
e29d7fb585 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:47 +01:00
Jing Zhao
dd7f5a3b88 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015c962eac1a35fa5df39356e8b84731c)
 2016-05-17 15:53:39 -07:00
Steve Loughran
dfc062a092 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:01:50 +01:00
Chris Nauroth
bb41547fb4 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d59e69e5dd64802fa9025dad1b315be5)
 2016-05-16 11:53:25 -07:00
Jing Zhao
d43d8a1bcd HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu. 
(cherry picked from commit af942585a108d70e0946f6dd4c465a54d068eabf)
 2016-05-04 10:24:06 -07:00
Yongjun Zhang
7955b6f66e HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd1216dfac78d05b438828e8503108d963)
 2016-04-26 16:27:59 -07:00
Akira Ajisaka
58642d2244 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d92e74be7089083555e85da0e73f7fda)
 2016-04-25 22:34:18 +09:00
Yongjun Zhang
f98f4151ac HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0ceaa843aa83146b6bea19e031c8dc3296)
 2016-04-21 12:23:42 -07:00
Jing Zhao
864baf23aa HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328b00a42ec8b952ad08cd7a80207c7f2)
 2016-04-14 10:35:43 -07:00
Masatake Iwasaki
8095c612a3 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383)
 2016-03-03 18:57:41 +09:00
Yongjun Zhang
b144e0f15d HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a8d6a18b4b7365c49d09635d24e579a)
 2016-02-19 10:29:27 -08:00
Yongjun Zhang
2fb711bde9 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6246ddc66218eb01009c44b0236eaaa)
 2016-01-29 12:13:03 -08:00
Yongjun Zhang
ee012c1f6f HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb707801071db3cc22bfcd14f87be443a)
 2016-01-15 10:10:28 -08:00
Colin Patrick Mccabe
2baeaa20a3 HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8315582c4ff2951144b096c23a64e753f397572d)
 2016-01-13 16:33:02 -08:00
Xiaoyu Yao
521f60c094 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e134555010ec28da296bcfef4ba2613a5c6c)
 2016-01-11 17:12:43 -08:00