Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,029 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

119 Commits

Author SHA1 Message Date
Steve Loughran 77401bd5fc HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 2017-01-19 11:24:58 +00:00
Junping Du 7706a63fb4 Preparing for 2.8.1 development after cutting off 2.8.0 branch. 2017-01-10 18:03:51 -08:00
Chris Nauroth 15e89634a5 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e1302)
(cherry picked from commit 9c61858ee7)
 2017-01-05 15:05:12 -08:00
Mingliang Liu ed9dcdae57 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:24:42 -08:00
Masatake Iwasaki 6d9b6fac86 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82)
 2016-10-27 15:55:35 +09:00
Jing Zhao d3c4465147 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d07983)
(cherry picked from commit ad69baf6a9)
 2016-10-13 13:28:05 -07:00
Brahma Reddy Battula 686cbf45d4 HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970be)
 2016-09-27 10:49:04 +05:30
Steve Loughran a2ce4d0503 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:54 +01:00
Chris Nauroth 911ae15f69 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb51397)
(cherry picked from commit e19f910245)
 2016-09-19 15:17:50 -07:00
Andrew Wang 4e7e48cdc1 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8)
(cherry picked from commit 38b1eafdbc)
 2016-07-11 18:06:45 -07:00
Yongjun Zhang 4a57c03f30 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3a)
 2016-06-28 23:34:44 -07:00
Jing Zhao d17265f467 HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 
(cherry picked from commit 856bc4e28b)
 2016-06-22 15:22:54 -07:00
Yongjun Zhang 3fc4d4a369 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee7)
 2016-05-26 23:34:32 -07:00
Steve Loughran 5930e813c5 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-21 19:10:16 +01:00
Steve Loughran ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c571.
 2016-05-20 14:00:26 +01:00
Steve Loughran edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Jing Zhao 8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015)
(cherry picked from commit dd7f5a3b88)
 2016-05-17 15:55:31 -07:00
Steve Loughran b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth 28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d5)
(cherry picked from commit bb41547fb4)
 2016-05-16 11:53:33 -07:00
Yongjun Zhang 7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd12)
(cherry picked from commit 7955b6f66e)
 2016-04-26 16:30:54 -07:00
Jing Zhao b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328)
(cherry picked from commit 864baf23aa)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka 6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d9)
(cherry picked from commit 58642d2244)
 2016-04-25 22:34:37 +09:00
Yongjun Zhang a1beb5fe1e HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0cea)
 2016-04-21 12:29:49 -07:00
Masatake Iwasaki 55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Yongjun Zhang 5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a)
 2016-02-19 10:29:46 -08:00
Yongjun Zhang f028e3d7bc HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6)
 2016-01-29 12:13:18 -08:00
Yongjun Zhang 4c8131b1bf HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-15 10:12:15 -08:00
Xiaoyu Yao a9cbfd717c HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e13455)
 2016-01-11 17:16:43 -08:00
Zhe Zhang 87d614c70b HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
 2016-01-11 09:57:01 -08:00
Akira Ajisaka c5c663c05d HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 
(cherry picked from commit f94d892704)
 2015-11-13 21:53:23 +09:00
Kihwal Lee dae9f0ef53 Addendum to MAPREDUCE-6451 
(cherry picked from commit b24fe06483)
 2015-10-30 16:06:50 -05:00
Kihwal Lee 52819fe4ef MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 
(cherry picked from commit 2868ca0328)
 2015-10-30 14:57:57 -05:00
Haohui Mai ee04add696 HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:29 -07:00
Haohui Mai f6b0fcdc75 Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu." 
This reverts commit a412edf5c4.
 2015-10-10 03:25:16 -07:00
Haohui Mai a412edf5c4 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:13 -07:00
Harsh J 03e370e306 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit 66dad854c0)
 2015-09-28 13:14:20 +05:30
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab)
 2015-09-22 12:54:44 +09:00
Steve Loughran 988749e6ea HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:06 +01:00
Yongjun Zhang b085c5ef6f HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 
(cherry picked from commit 0bc15cb6e6)
 2015-08-20 08:10:34 -07:00
Robert Kanter 3f79ee7494 HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) 
(cherry picked from commit aaafa0b2ee)
 2015-06-30 16:50:42 -07:00
Colin Patrick Mccabe 3dec58dd78 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 
(cherry picked from commit 7dba7005b7)
 2015-05-26 12:40:41 -07:00
Jing Zhao 5caea4cd46 HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. 
(cherry picked from commit 0790275f05)
 2015-05-18 13:26:09 -07:00
Jonathan Eagles 9ed6b77da1 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) 
(cherry picked from commit 2d7363b273)

Conflicts:
	pom.xml
 2015-05-01 19:04:41 -07:00
Ravi Prakash 122262a1fb HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak) 2015-04-24 13:39:48 -07:00
Ravi Prakash aa34aa5caa HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak) 2015-04-21 16:50:40 -07:00
Jing Zhao d4dd97eabd HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. 
(cherry picked from commit 4c097e473b)
 2015-04-15 12:39:09 -07:00
Colin Patrick Mccabe 386b90a700 HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) 
(cherry picked from commit 932730df7d)
 2015-04-03 14:08:53 -07:00
Haohui Mai 9e114ee607 HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao. 2015-04-01 16:51:09 -07:00
Tsuyoshi Ozawa 24b60a7d1a HADOOP-11760. Fix typo of javadoc in DistCp. Contributed by Brahma Reddy Battula. 
(cherry picked from commit e074952bd6)
 2015-03-27 23:16:37 +09:00
Yongjun Zhang be0d228ed1 HADOOP-11724. DistCp throws NPE when the target directory is root. (Lei Eddy Xu via Yongjun Zhang) 
(cherry-picked from commit 65ee02d9ed50e026c2d37ec80babd4176075ff84)
 2015-03-25 15:50:28 -07:00