Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,338 Commits 326 Branches 371 Tags
Commit Graph

121 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
396de0fd8f Preparing for 2.8.3 development 2017-06-07 15:38:20 -07:00
Junping Du
dfac95a3a2 Preparing for 2.8.1 development 2017-04-11 11:21:49 -07:00
Steve Loughran
77401bd5fc HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 2017-01-19 11:24:58 +00:00
Junping Du
7706a63fb4 Preparing for 2.8.1 development after cutting off 2.8.0 branch. 2017-01-10 18:03:51 -08:00
Chris Nauroth
15e89634a5 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e130230457fc897ed7a2a09e14d078be90a)
(cherry picked from commit 9c61858ee754b167a90533375e85ca5bde3cdce1)
 2017-01-05 15:05:12 -08:00
Mingliang Liu
ed9dcdae57 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:24:42 -08:00
Masatake Iwasaki
6d9b6fac86 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82a4510f16df49238d57c9f78ac28ae7)
 2016-10-27 15:55:35 +09:00
Jing Zhao
d3c4465147 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d079838f532a13ca237300386d1b3bc1b178)
(cherry picked from commit ad69baf6a9139c0af81e9f72e41c1e3aeb119ebc)
 2016-10-13 13:28:05 -07:00
Brahma Reddy Battula
686cbf45d4 HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970bea8213660bb6c550104783069153236)
 2016-09-27 10:49:04 +05:30
Steve Loughran
a2ce4d0503 HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:54 +01:00
Chris Nauroth
911ae15f69 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb5139769eb55893971b43b9c23da9513a784)
(cherry picked from commit e19f910245cd42c6caa88fea5930f446ab618c94)
 2016-09-19 15:17:50 -07:00
Andrew Wang
4e7e48cdc1 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8dbdc1841f225a34346d0392fa76a47)
(cherry picked from commit 38b1eafdbcbd91bd806b842729b84d84f3600524)
 2016-07-11 18:06:45 -07:00
Yongjun Zhang
4a57c03f30 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3af913e6dcd09626953c64d43168efbd)
 2016-06-28 23:34:44 -07:00
Jing Zhao
d17265f467 HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 
(cherry picked from commit 856bc4e28b356f31d3c48eed2c352fb9010232d7)
 2016-06-22 15:22:54 -07:00
Yongjun Zhang
3fc4d4a369 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee72a27382a26bf450bb8b16784aeebbb)
 2016-05-26 23:34:32 -07:00
Steve Loughran
5930e813c5 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-21 19:10:16 +01:00
Steve Loughran
ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c57123ce4cba109a41a7dc21a45da7d27c.
 2016-05-20 14:00:26 +01:00
Steve Loughran
edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Jing Zhao
8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015c962eac1a35fa5df39356e8b84731c)
(cherry picked from commit dd7f5a3b88cc37945bfc96ca91ed5f39d44ee394)
 2016-05-17 15:55:31 -07:00
Steve Loughran
b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth
28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d59e69e5dd64802fa9025dad1b315be5)
(cherry picked from commit bb41547fb42b9dd2b80ec2a7286db6d35fd251b5)
 2016-05-16 11:53:33 -07:00
Yongjun Zhang
7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd1216dfac78d05b438828e8503108d963)
(cherry picked from commit 7955b6f66ee670a57ccdeb7b6a9e6af25c0d67c7)
 2016-04-26 16:30:54 -07:00
Jing Zhao
b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328b00a42ec8b952ad08cd7a80207c7f2)
(cherry picked from commit 864baf23aa3df201f81b721852221c6475388295)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka
6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d92e74be7089083555e85da0e73f7fda)
(cherry picked from commit 58642d2244c3062672cc8a8b8a7cdbcd8929cd14)
 2016-04-25 22:34:37 +09:00
Yongjun Zhang
a1beb5fe1e HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0ceaa843aa83146b6bea19e031c8dc3296)
 2016-04-21 12:29:49 -07:00
Masatake Iwasaki
55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Yongjun Zhang
5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a8d6a18b4b7365c49d09635d24e579a)
 2016-02-19 10:29:46 -08:00
Yongjun Zhang
f028e3d7bc HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6246ddc66218eb01009c44b0236eaaa)
 2016-01-29 12:13:18 -08:00
Yongjun Zhang
4c8131b1bf HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb707801071db3cc22bfcd14f87be443a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-15 10:12:15 -08:00
Xiaoyu Yao
a9cbfd717c HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e134555010ec28da296bcfef4ba2613a5c6c)
 2016-01-11 17:16:43 -08:00
Zhe Zhang
87d614c70b HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
 2016-01-11 09:57:01 -08:00
Akira Ajisaka
c5c663c05d HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 
(cherry picked from commit f94d89270464ea8e0d19e26e425835cd6a5dc5de)
 2015-11-13 21:53:23 +09:00
Kihwal Lee
dae9f0ef53 Addendum to MAPREDUCE-6451 
(cherry picked from commit b24fe0648348d325d14931f80cee8a170fb3358a)
 2015-10-30 16:06:50 -05:00
Kihwal Lee
52819fe4ef MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 
(cherry picked from commit 2868ca0328d908056745223fb38d9a90fd2811ba)
 2015-10-30 14:57:57 -05:00
Haohui Mai
ee04add696 HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:29 -07:00
Haohui Mai
f6b0fcdc75 Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu." 
This reverts commit a412edf5c4838e007217652e1cb6a24e73b7e735.
 2015-10-10 03:25:16 -07:00
Haohui Mai
a412edf5c4 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:13 -07:00
Harsh J
03e370e306 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit 66dad854c0aea8c137017fcf198b165cc1bd8bdd)
 2015-09-28 13:14:20 +05:30
Tsuyoshi Ozawa
52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab0fae3839c9cc5d552aa0304444f956)
 2015-09-22 12:54:44 +09:00
Steve Loughran
988749e6ea HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:06 +01:00
Yongjun Zhang
b085c5ef6f HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 
(cherry picked from commit 0bc15cb6e60dc60885234e01dec1c7cb4557a926)
 2015-08-20 08:10:34 -07:00
Robert Kanter
3f79ee7494 HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) 
(cherry picked from commit aaafa0b2ee64f6cfb7fdc717500e1c483b9df8cc)
 2015-06-30 16:50:42 -07:00
Colin Patrick Mccabe
3dec58dd78 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 
(cherry picked from commit 7dba7005b79994106321b0f86bc8f4ea51a3c185)
 2015-05-26 12:40:41 -07:00
Jing Zhao
5caea4cd46 HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. 
(cherry picked from commit 0790275f058b0cf41780ad337c9150a1e8ebebc6)
 2015-05-18 13:26:09 -07:00
Jonathan Eagles
9ed6b77da1 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) 
(cherry picked from commit 2d7363b27360e36fdd62546c0f9d0b1d78133f29)

Conflicts:
	pom.xml
 2015-05-01 19:04:41 -07:00
Ravi Prakash
122262a1fb HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak) 2015-04-24 13:39:48 -07:00
Ravi Prakash
aa34aa5caa HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak) 2015-04-21 16:50:40 -07:00
Jing Zhao
d4dd97eabd HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. 
(cherry picked from commit 4c097e473bb1f18d1510deb61bae2bcb8c156f18)
 2015-04-15 12:39:09 -07:00
Colin Patrick Mccabe
386b90a700 HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) 
(cherry picked from commit 932730df7d62077f7356464ad27f69469965d77a)
 2015-04-03 14:08:53 -07:00
Haohui Mai
9e114ee607 HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao. 2015-04-01 16:51:09 -07:00