Junping Du
2b48854cfd
HADOOP-15385. Many tests are failing in hadoop-distcp project in branch-2. Contributed by Jason Lowe.
2018-04-25 10:11:41 +08:00
Sammi Chen
1d78825bf1
HADOOP-14999. AliyunOSS: provide one asynchronous multi-part based uploading mechanism. Contributed by Genmao Yu.
2018-04-10 14:24:31 +08:00
Steve Loughran
e5351608ca
HADOOP-14651. Update okhttp version to 2.7.5.
...
Contributed by Ray Chiang and John Zhuge.
2018-04-04 15:16:20 +01:00
Chris Douglas
cc0a791794
HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao
...
(cherry picked from commit 081c350188
)
(cherry picked from commit ac16e8f4d3
)
2018-03-28 12:04:20 -07:00
Sammi Chen
2285afb32e
HADOOP-15262. AliyunOSS: move files under a directory in parallel when rename a directory. Contributed by Jinhu Wu.
...
(cherry picked from commit d67a5e2dec
)
2018-03-19 15:09:23 +08:00
Chris Douglas
340cd5f1b1
HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
...
(cherry picked from commit 45d1b0fdcc
)
(cherry picked from commit 09940b1eb3
)
(cherry picked from commit a03c8ea61f
)
2018-03-12 20:08:46 -07:00
Steve Loughran
c245050538
HADOOP-15090. Add ADL troubleshooting doc.
...
Contributed by Steve Loughran; backport to branch-2 by Masatake Iwasaki.
2018-03-05 12:46:19 +00:00
Gergely Novák
278676508f
MAPREDUCE-7023. TestHadoopArchiveLogs.testCheckFilesAndSeedApps fails on rerun
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e99a1e7767
)
2018-03-02 16:22:20 -08:00
Steve Loughran
fb0f2cc2fb
Revert "HADOOP-15090. Add ADL troubleshooting doc."
...
This reverts commit 6f944452f7
.
2018-02-28 21:19:03 +00:00
Chris Douglas
762125b864
Backport HADOOP-13514 (surefire upgrade) to branch-2
2018-02-26 16:32:06 -08:00
Xuan Gong
a0b55392d4
MAPREDUCE-7027: HadoopArchiveLogs shouldn't delete the original logs if the HAR creation fails. Contributed by Gergely Novák
...
(cherry picked from commit 68ce193efc
)
2018-02-23 14:40:51 -08:00
Steve Loughran
07c7df4b26
HADOOP-13972. ADLS to support per-store configuration.
...
Contributed by Sharad Sonker.
(cherry picked from commit 050f5287b7
)
2018-02-15 16:29:44 +00:00
Steve Loughran
6f944452f7
HADOOP-15090. Add ADL troubleshooting doc.
...
Contributed by Steve Loughran.
(cherry picked from commit 58a2120e8a
)
2018-02-15 14:26:36 +00:00
Sammi Chen
2816bd1f43
HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)
...
(cherry picked from commit 9195a6e302
)
(cherry picked from commit 91184299c5
)
2018-01-30 16:25:02 +08:00
Steve Loughran
312c571694
HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common
...
Contributed by Genmao Yu
2018-01-27 20:31:21 -08:00
Jason Lowe
4d7bbee0a1
YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang
2018-01-25 17:56:36 -06:00
Jason Lowe
fd09f72151
Revert "HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)"
...
This reverts commit 896dc7c780
.
2018-01-17 09:47:43 -06:00
Sammi Chen
896dc7c780
HADOOP-15027. AliyunOSS: Support multi-thread pre-read to improve sequential read from Hadoop to Aliyun OSS performance. (Contributed by Jinhu Wu)
...
(cherry picked from commit 9195a6e302
)
(cherry picked from commit 55142849db
)
(cherry picked from commit 082a707bae
)
2018-01-17 16:36:03 +08:00
Steve Loughran
d2ceef0f20
HADOOP-15086. NativeAzureFileSystem file rename is not atomic.
...
Contributed by Thomas Marquardt
(Backported to branch-2 via /HADOOP-15156)
(cherry picked from commit a35267b47a
)
2018-01-08 15:13:50 +00:00
Sammi Chen
1fee447bf2
HADOOP-15111. Aliyun OSS: backport HADOOP-14993 to branch-2. Contributed by Genmao Yu.
2017-12-15 14:28:39 +08:00
Sammi Chen
27807e4ccb
HADOOP-15104. Aliyun OSS: change the default value of max error retry. Contributed by Jinhu Wu.
...
(cherry picked from commit ce04340ec7
)
2017-12-15 13:53:48 +08:00
Surendra Singh Lilhore
8d69b7f2d4
HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani.
2017-12-13 10:29:19 +05:30
Sammi Chen
94390fcd52
HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server.
...
(cherry picked from commit 6e24fe2998
)
2017-12-08 22:02:00 +08:00
Xiao Chen
c9d48352c2
HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge.
2017-12-07 21:20:33 -08:00
Steve Loughran
5c37a0b841
HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
...
Contributed by Rajesh Balamohan
(cherry picked from commit 0ea182d0fa
)
2017-11-28 11:57:51 +00:00
Kai Zheng
30ab9b6aef
HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2. Contributed by Sammi Chen
...
The consolidated commits in this backport are as follows:
HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
2017-11-13 09:29:33 +08:00
Subru Krishnan
5991c218a0
HADOOP-15026. Rebase ResourceEstimator start/stop scripts for branch-2. (Rui Li via Subru).
2017-11-09 00:32:34 -08:00
Subru Krishnan
46a740a82e
HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru).
...
(cherry picked from commit f2df6b8983
)
2017-11-08 18:09:02 -08:00
Arun Suresh
0200fbac5b
HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh)
...
(cherry picked from commit 53c0fb7efe
)
2017-11-02 17:18:03 -07:00
Akira Ajisaka
82fc80a987
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi.
...
(cherry picked from commit cc3f3eca40
)
2017-11-02 18:33:25 +09:00
Arun Suresh
71a55bebfb
Preparing for 2.10.0 development
2017-10-31 12:51:14 -07:00
Chris Douglas
ee89241282
HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li
...
(cherry picked from commit 139cc752d2
)
(cherry picked from commit 77ad73501c
)
2017-10-27 22:01:41 -07:00
Steve Loughran
3fbcf76324
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure.
...
Contributed by Steve Loughran.
(backport JIRA HADOOP-14870)
2017-10-27 12:55:12 +01:00
Subru Krishnan
9897538a96
HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru).
...
(cherry picked from commit 625039ef20
)
2017-10-26 19:14:24 -07:00
Akira Ajisaka
13f1a7e0ed
Revert "HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru)."
...
This reverts commit da7f989d86
.
2017-10-26 15:33:53 +09:00
Subru Krishnan
da7f989d86
HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru).
...
(cherry picked from commit 625039ef20
)
2017-10-25 17:05:35 -07:00
Steve Loughran
062d9c9bab
HADOOP-14942. DistCp#cleanup() should check whether jobFS is null.
...
Contributed by Andras Bokor.
(cherry picked from commit f36cbc8475
)
2017-10-20 22:27:59 +01:00
Wangda Tan
d894f910d0
YARN-6608. Backport all SLS improvements from trunk to branch-2. (Carlo Curino via wangda)
2017-10-17 19:45:09 -07:00
Steve Loughran
8902178d3e
HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
...
Contributed by Santhosh G Nayak
Cherry-picked from commit 9fcc3a1fc8
2017-10-16 16:06:41 +01:00
Steve Loughran
ccfd1a4a56
HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
...
Contributed by Varada Hemeswari.
Cherry-picked from commit ref 3de574413c
2017-10-11 18:09:13 +01:00
Steve Loughran
34040e8d73
HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
...
Contributed by Kannapiran Srinivasan.
2017-10-07 11:20:18 +01:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
2017-10-05 16:03:13 +01:00
Steve Loughran
28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests.
...
Contributed by Steve Loughran
2017-10-03 20:06:45 +01:00
Steve Loughran
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
...
Contributed by Varada Hemeswari
2017-10-02 14:28:59 +01:00
Steve Loughran
a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A.
...
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
(cherry-picked from/based on commit 621b43e254
)
2017-09-29 11:29:22 +01:00
Steve Loughran
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Contributed by Sivaguru Sankaridurg
2017-09-29 11:16:31 +01:00
Steve Loughran
27a41848ac
HADOOP-14890 Move up to AWS SDK 1.11.199.
...
Contributed by Steve Loughran.
2017-09-26 20:11:10 +01:00
Jason Lowe
2c10abaddd
MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari
...
(cherry picked from commit b133dc5700
)
2017-09-22 15:30:53 -05:00
Steve Loughran
12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory.
...
Contributed by Esfandiar Manii.
2017-09-19 11:02:10 +01:00
Xiaoyu Yao
b669aca518
HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin.
2017-09-11 11:40:48 -07:00