Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,165 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

681 Commits

Author SHA1 Message Date
Surendra Singh Lilhore 4f0ae1b108 HDFS-12833. Distcp : Update the usage of delete option for dependency with update and overwrite option. Contributed by usharani. 
(cherry picked from commit 8d69b7f2d4)
 2017-12-13 10:50:01 +05:30
Sammi Chen 4a064dd644 HADOOP-15024. AliyunOSS: Support user agent configuration and include that & Hadoop version information to oss server. 
(cherry picked from commit 94390fcd52)
 2017-12-08 22:37:17 +08:00
Sammi Chen b756beb679 HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2. Contributed by Sammi Chen. 
The consolidated commits in this backport are as follows:
        HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
        HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
        HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
        HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
        HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
        HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
        HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
        HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
        HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
        HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
        HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
        HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
        HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
        HADOOP-13624.  Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
        HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
        HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
        HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou

 (cherry picked from commit 30ab9b6aef)
 2017-11-27 11:35:17 +08:00
Sammi Chen 12901cdcee Revert "HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches. Contributed by Sammi Chen." 
This reverts commit 32a88442d0.
 2017-11-27 10:46:15 +08:00
Sammi Chen 32a88442d0 HADOOP-14964. AliyunOSS: backport Aliyun OSS module to branch-2 and 2.8+ branches. Contributed by Sammi Chen. 2017-11-24 14:06:01 +08:00
Subru Krishnan 26110ec335 HADOOP-15026. Rebase ResourceEstimator start/stop scripts for branch-2. (Rui Li via Subru). 
(cherry picked from commit 5991c218a0)
 2017-11-09 00:33:23 -08:00
Subru Krishnan d45672a144 HADOOP-15025. Ensure singleton for ResourceEstimatorService. (Rui Li via Subru). 
(cherry picked from commit f2df6b8983)
(cherry picked from commit 46a740a82e)
 2017-11-08 18:11:32 -08:00
Arun Suresh bb6a1aa5fa Preparing for 2.9.1 development 2017-11-02 22:49:38 -07:00
Arun Suresh e44ff3f60a HADOOP-15013. Fix ResourceEstimator findbugs issues. (asuresh) 
(cherry picked from commit 53c0fb7efe)
 2017-11-02 17:21:57 -07:00
Akira Ajisaka 05b7b3d126
MAPREDUCE-6999. Fix typo onf in DynamicInputChunk.java. Contributed by fang zhenyi. 
(cherry picked from commit cc3f3eca40)
(cherry picked from commit 82fc80a987)
 2017-11-02 18:34:46 +09:00
Chris Douglas ee89241282 HADOOP-14991. Add missing figures to Resource Estimator tool. Contributed by Rui Li 
(cherry picked from commit 139cc752d2)
(cherry picked from commit 77ad73501c)
 2017-10-27 22:01:41 -07:00
Steve Loughran 3fbcf76324 HADOOP-14553. Add (parallelized) integration tests to hadoop-azure. 
Contributed by Steve Loughran.

(backport JIRA HADOOP-14870)
 2017-10-27 12:55:12 +01:00
Subru Krishnan 9897538a96 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20)
 2017-10-26 19:14:24 -07:00
Akira Ajisaka 13f1a7e0ed
Revert "HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru)." 
This reverts commit da7f989d86.
 2017-10-26 15:33:53 +09:00
Subru Krishnan da7f989d86 HADOOP-14840. Tool to estimate resource requirements of an application pipeline based on prior executions. (Rui Li via Subru). 
(cherry picked from commit 625039ef20)
 2017-10-25 17:05:35 -07:00
Steve Loughran 062d9c9bab HADOOP-14942. DistCp#cleanup() should check whether jobFS is null. 
Contributed by Andras Bokor.

(cherry picked from commit f36cbc8475)
 2017-10-20 22:27:59 +01:00
Wangda Tan d894f910d0 YARN-6608. Backport all SLS improvements from trunk to branch-2. (Carlo Curino via wangda) 2017-10-17 19:45:09 -07:00
Steve Loughran 8902178d3e HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak

Cherry-picked from commit 9fcc3a1fc8
 2017-10-16 16:06:41 +01:00
Steve Loughran ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c
 2017-10-11 18:09:13 +01:00
Steve Loughran 34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-07 11:20:18 +01:00
Steve Loughran fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
 2017-10-05 16:03:13 +01:00
Steve Loughran 28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. 
Contributed by Steve Loughran
 2017-10-03 20:06:45 +01:00
Steve Loughran 1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-10-02 14:28:59 +01:00
Steve Loughran a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.

(cherry-picked from/based on commit 621b43e254)
 2017-09-29 11:29:22 +01:00
Steve Loughran a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-09-29 11:16:31 +01:00
Steve Loughran 27a41848ac
HADOOP-14890 Move up to AWS SDK 1.11.199. 
Contributed by Steve Loughran.
 2017-09-26 20:11:10 +01:00
Jason Lowe 2c10abaddd MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b133dc5700)
 2017-09-22 15:30:53 -05:00
Steve Loughran 12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory. 
Contributed by Esfandiar Manii.
 2017-09-19 11:02:10 +01:00
Xiaoyu Yao b669aca518 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-11 11:40:48 -07:00
Steve Loughran 87af3f4991
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov
 2017-09-11 14:15:58 +01:00
John Zhuge 1421196d20 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 
(cherry picked from commit a4661850c1)
 2017-09-08 14:15:51 -07:00
Steve Loughran 6e20ae954c
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.

(cherry picked from commit 83449abb40)
 2017-09-07 19:58:40 +01:00
John Zhuge 4c34c03b27 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-06 01:03:18 -07:00
Steve Loughran f382f66255
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS. 
Contributed by Sivaguru Sankaridurg
 2017-09-05 14:18:43 +01:00
John Zhuge 2442a8d716 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria. 
(cherry picked from commit 7ecc6dbed6)
 2017-08-31 21:13:50 -07:00
Yufei Gu 626fa7c8e1 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-30 23:08:02 -07:00
Steve Loughran 25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses. 
Contributed by Sivaguru Sankaridurg

(cherry picked from commit 021974f4cb)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2017-08-30 22:40:50 +01:00
Arun Suresh f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 8410d862d3)
 2017-08-20 08:31:17 -07:00
Steve Loughran c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted. 
Contributed by Thomas Marquardt

(cherry picked from commit c6b4e656b7)
 2017-08-18 14:38:04 +01:00
Steve Loughran d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded. 
Contributed by Thomas Marquardt.
 2017-08-16 13:49:47 +01:00
John Zhuge 0efc590e6d HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 
(cherry picked from commit 7769e96149)
 2017-08-13 00:27:33 -07:00
Steve Loughran 96befde480
HADOOP-14183. Remove service loader config file for wasb fs. 
Contributed by Esfandiar Manii.

(cherry picked from commit 54356b1e83)
 2017-08-10 16:49:51 +01:00
Steve Loughran bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing. 
Contributed by Esfandiar Manii.
 2017-08-10 16:49:19 +01:00
Steve Loughran 7f7ab0302f
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek. 
Contributed by Thomas Marquardt

(cherry picked from commit d91b7a8451)
 2017-08-06 20:30:40 +01:00
Steve Loughran ced547d5f0
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran
 2017-08-04 11:07:34 +01:00
John Zhuge ba94c68e0b HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge. 
(cherry picked from commit f735ad1b67)
 2017-07-28 09:51:32 -07:00
Jitendra Pandey 554c3cd21d HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:35:40 -07:00
Jitendra Pandey ab95086ed1 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-25 08:31:35 -07:00
Jitendra Pandey 23b920cd7a HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 11:37:36 -07:00
Jitendra Pandey b9f2effaf6 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 16:25:05 -07:00