Commit Graph

186 Commits

Author SHA1 Message Date
Akira Ajisaka afd844059c HADOOP-16331. Fix ASF License check in pom.xml
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
Akira Ajisaka 9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Andrew Olson c15b3bca86
HADOOP-16294: Enable access to input options by DistCp subclasses.
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
2019-05-16 16:11:12 +02:00
Giovanni Matteo Fumarola 7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Masatake Iwasaki bbdbc7a9a1 HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 2019-04-12 11:52:18 +09:00
Siyao Meng ce4bafdf44
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
Contributed by Siyao Meng
2019-03-26 18:42:54 +00:00
Andrew Olson faba3591d3
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
Author:    Andrew Olson
2019-03-22 10:35:30 +00:00
Ranith Sardar 546c5d70ef
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 2019-02-07 21:48:07 +00:00
Andrew Olson de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
Akira Ajisaka 1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Giovanni Matteo Fumarola fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
Kai Xie 188bebbe7e HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
Contributed by Kai Xie.
2019-01-08 11:57:57 +00:00
Akira Ajisaka 7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
This reverts commit d7152332b3.
2019-01-08 14:51:30 +09:00
Steve Loughran d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
Arpit Agarwal 914b0cf15f HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 2018-11-15 13:58:13 -08:00
Ted Yu e2cecb681e HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-19 13:21:06 -07:00
Steve Loughran e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus.
Contributed by Ted Yu.
2018-10-12 09:59:19 +01:00
Sunil G 58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Surendra Singh Lilhore 96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Akira Ajisaka 3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Steve Loughran ca8b80bf59
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan.
Contributed by Steve Loughran.
2018-07-10 10:43:59 +01:00
Akira Ajisaka 2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma. 2018-06-28 14:37:22 +09:00
Xiao Chen 7c9cdad6d0 HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo. 2018-04-10 21:31:48 -07:00
Steve Loughran 1976e0066e HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
Chris Douglas 45cccadd2e HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang 2018-03-13 11:08:11 -07:00
Steve Loughran 7ef4d942dd HADOOP-15273.distcp can't handle remote stores with different checksum algorithms.
Contributed by Steve Loughran.
2018-03-08 11:24:06 +00:00
Steve Loughran 3bd6b1fd85 HADOOP-15292. Distcp's use of pread is slowing it down.
Contributed by Virajith Jalaparti.
2018-03-08 11:15:46 +00:00
fang zhenyi 4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
Wangda Tan 60f9e60b3b Preparing for 3.2.0 development
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
Anu Engineer 4304fcd5bd HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 2018-02-06 13:43:45 -08:00
Arpit Agarwal d4e13a4647 HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh. 2018-02-02 11:37:51 -08:00
Surendra Singh Lilhore 00129c5314 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 2017-12-12 00:28:02 +05:30
Akira Ajisaka cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 2017-11-02 18:32:24 +09:00
Steve Loughran f36cbc8475 HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
Contributed by Andras Bokor.
2017-10-20 22:27:04 +01:00
ChenSammi e0b3c644e1 HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen 2017-09-14 09:15:29 +08:00
Xiaoyu Yao 63720ef574 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-05 23:34:55 -07:00
Andrew Wang 0d419c984f Preparing for 3.1.0 development 2017-09-01 11:53:48 -07:00
Andrew Wang f29a0fc288 HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou. 2017-08-25 14:14:23 -07:00
Andrew Wang dd7916d3cd HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas. 2017-08-17 09:35:36 -07:00
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Wei-Chiu Chuang 44350fdf49 HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default). Contributed by Bharat Viswanadham. 2017-07-20 18:23:13 -07:00
Andrew Wang af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Jason Lowe dd65eea74b HADOOP-8143. Change distcp to have -pb on by default. Contributed by Mithun Radhakrishnan 2017-06-20 09:53:47 -05:00
Andrew Wang 16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Sunil G b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Yongjun Zhang b4adc8392c HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-18 15:35:22 -07:00
Mingliang Liu 26172a94d6 HADOOP-14267. Make DistCpOptions immutable. Contributed by Mingliang Liu 2017-03-31 20:04:26 -07:00
Yongjun Zhang bf3fb585aa HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-03-30 17:38:56 -07:00
Yongjun Zhang 144f1cf765 Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen."
This reverts commit 064c8b25ec.
2017-03-30 17:38:18 -07:00
Yongjun Zhang 064c8b25ec HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen. 2017-03-30 17:01:15 -07:00