c1a2b29c0f
HADOOP-16158. DistCp to support checksum validation when copy blocks in parallel ( #919 )
...
* DistCp to support checksum validation when copy blocks in parallel
* address review comments
* add checksums comparison test for combine mode
(cherry picked from commit c765584eb2b3c14d4132
2019-08-18 18:55:59 -07:00
befafe9408
HADOOP-16440. Distcp can not preserve timestamp with -delete option. Contributed by ludun.
2019-07-20 13:43:38 +05:30
2f6f42bc7b
HDFS-12564. Add the documents of swebhdfs configurations on the client side. Contributed by Takanobu Asanuma.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 98d2065643
2019-06-20 20:18:00 -07:00
8098ddaf40
HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:39:49 +09:00
f8bd5deec1
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:28:21 +09:00
fd90af892c
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
...
(cherry picked from commit bbdbc7a9a1
2019-04-12 12:03:11 +09:00
49d54633e0
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 23:15:18 +00:00
352ebc6ed9
Revert "HADOOP-15281. Distcp to add no-rename copy option."
...
Revert "HADOOP-15281. Distcp to add no-rename copy option. Contributed by Andrew Olson."
This reverts commit d2765ffc2e
2019-02-07 20:18:32 +00:00
d2765ffc2e
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
(cherry picked from commit de804e53b9
2019-02-07 10:09:55 +00:00
fee33057f0
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
...
Contributed by Kai Xie.
(cherry picked from commit 188bebbe7e
2019-01-08 13:35:17 +00:00
31ea2f7806
Preparing for 3.1.3 development
...
Change-Id: I3c3d3ee47dc4fef239127b4452ff14676fa26e3d
2019-01-07 10:04:58 -08:00
3e57adee0a
HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia.
...
(cherry picked from commit 914b0cf15f
2018-11-15 13:58:38 -08:00
84726679f0
HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681ea7dd244a49
2018-10-19 13:22:31 -07:00
5583711419
Preparing for 3.1.2 release
...
Change-Id: If2793e2ed2b5b349a9e1f98f78df43f309dcfcbd
2018-07-31 13:08:55 -07:00
d54241e9c9
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan.
...
Contributed by Steve Loughran.
(cherry picked from commit ca8b80bf59
2018-07-10 10:50:40 +01:00
805e33b62c
HDFS-13056. Expose file-level composite CRCs in HDFS which are comparable across different instances/layouts. Contributed by Dennis Huo.
2018-04-16 11:58:10 -07:00
25bf1d3ac6
Updated branch-3.1 to track 3.1.1
...
Change-Id: Idd55c79921b8acacc8a8a3258450e476f3abb706
2018-04-09 10:58:50 -07:00
0bd99621c5
Preparing for 3.1.0 development
2018-03-19 15:16:54 -07:00
1976e0066e
HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
...
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
45cccadd2e
HDFS-12780. Fix spelling mistake in DistCpUtils.java. Contributed by Jianfei Jiang
2018-03-13 11:08:11 -07:00
7ef4d942dd
HADOOP-15273.distcp can't handle remote stores with different checksum algorithms.
...
Contributed by Steve Loughran.
2018-03-08 11:24:06 +00:00
3bd6b1fd85
HADOOP-15292. Distcp's use of pread is slowing it down.
...
Contributed by Virajith Jalaparti.
2018-03-08 11:15:46 +00:00
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
4304fcd5bd
HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen.
2018-02-06 13:43:45 -08:00
d4e13a4647
HADOOP-15198. Correct the spelling in CopyFilter.java. Contributed by Mukul Kumar Singh.
2018-02-02 11:37:51 -08:00
00129c5314
HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani.
2017-12-12 00:28:02 +05:30
cc3f3eca40
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi.
2017-11-02 18:32:24 +09:00
f36cbc8475
HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
...
Contributed by Andras Bokor.
2017-10-20 22:27:04 +01:00
e0b3c644e1
HDFS-12414. Ensure to use CLI command to enable/disable erasure coding policy. Contributed by Sammi Chen
2017-09-14 09:15:29 +08:00
63720ef574
HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin.
2017-09-05 23:34:55 -07:00
0d419c984f
Preparing for 3.1.0 development
2017-09-01 11:53:48 -07:00
f29a0fc288
HDFS-12303. Change default EC cell size to 1MB for better performance. Contributed by Wei Zhou.
2017-08-25 14:14:23 -07:00
dd7916d3cd
HDFS-12250. Reduce usage of FsPermissionExtension in unit tests. Contributed by Chris Douglas.
2017-08-17 09:35:36 -07:00
1a1bf6b7d0
HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer.
2017-08-02 12:25:05 -06:00
44350fdf49
HADOOP-14557. Document HADOOP-8143 (Change distcp to have -pb on by default). Contributed by Bharat Viswanadham.
2017-07-20 18:23:13 -07:00
af2773f609
Updating version for 3.0.0-beta1 development
2017-06-29 17:57:40 -07:00
dd65eea74b
HADOOP-8143. Change distcp to have -pb on by default. Contributed by Mithun Radhakrishnan
2017-06-20 09:53:47 -05:00
16ad896d5c
Update maven version for 3.0.0-alpha4 development
2017-05-26 14:09:44 -07:00
b6f66b0da1
YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang.
2017-05-22 14:10:06 +05:30
b4adc8392c
HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)
2017-05-18 15:35:22 -07:00
26172a94d6
HADOOP-14267. Make DistCpOptions immutable. Contributed by Mingliang Liu
2017-03-31 20:04:26 -07:00
bf3fb585aa
HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li.
2017-03-30 17:38:56 -07:00
144f1cf765
Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen."
...
This reverts commit 064c8b25ec
2017-03-30 17:38:18 -07:00
064c8b25ec
HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen.
2017-03-30 17:01:15 -07:00
8c591b8d19
HDFS-10974. Document replication factor for EC files. Contributed by Yiqun Lin.
2017-03-30 11:16:05 -07:00
0e6f8e4bc6
HDFS-10971. Distcp should not copy replication factor if source file is erasure coded. Contributed by Manoj Govindassamy.
2017-03-28 22:14:03 -07:00
d235dcdf0b
HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang)
2017-02-27 20:42:13 -08:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
ed33ce11dd
HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu
...
(cherry picked from commit 77401bd5fc
2017-01-19 11:25:40 +00:00
KAI XIE
Ayush Saxena
Takanobu Asanuma
Akira Ajisaka
Masatake Iwasaki
Eric E Payne
Andrew Olson
Wangda Tan
Arpit Agarwal
Ted Yu
Steve Loughran
Xiao Chen
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
Chris Douglas
fang zhenyi
Anu Engineer
Surendra Singh Lilhore
ChenSammi
Xiaoyu Yao
Andrew Wang
Sean Mackrory
Wei-Chiu Chuang
Jason Lowe
Sunil G
Yongjun Zhang
Mingliang Liu