Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,158 Commits 326 Branches 371 Tags
Commit Graph

605 Commits

Author SHA1 Message Date
Yongjun Zhang
dd552a97b7 HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-24 19:05:40 -07:00
Yongjun Zhang
49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang
97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li." 
This reverts commit c411adac32cda42433b15979253248336487d977.
 2017-05-24 18:54:51 -07:00
Yongjun Zhang
39474ed50d Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)" 
This reverts commit fe185e2c3a07d4e6a82ed4fd255c98ec8b561ad6.
 2017-05-24 18:54:50 -07:00
Sunil G
ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Yongjun Zhang
fe185e2c3a HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-19 21:11:38 -07:00
Steve Loughran
2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. 
Contributed by Steve Loughran.
 2017-05-18 15:38:46 +01:00
Mingliang Liu
f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit b415c6fe743242acf1d1d3eb7ea7091d90d2c0d4)
 2017-05-16 11:28:12 -07:00
Andrew Wang
18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu
586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak 
(cherry picked from commit eed731496f63537350206274af09d2118b116a17)
 2017-05-10 15:05:54 -07:00
Akira Ajisaka
753954f7be
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong. 
(cherry picked from commit c60164fb60d3d8afb0fb8410936ccc231c4593f9)
 2017-05-09 05:39:02 -05:00
Mingliang Liu
616413ecb2 HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran 
(cherry picked from commit 83dded556dc179fcff078451fb80533065e116f0)
 2017-05-03 11:23:15 -07:00
Mingliang Liu
e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak 
(cherry picked from commit 8b5f2c372e70999f3ee0a0bd685a494e06bc3652)
 2017-04-26 13:51:58 -07:00
Mingliang Liu
5ceb61cb43 HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu 
(cherry picked from commit e4c32bccde3147f51263cb58cdb480453b1cb3e9)
 2017-04-25 20:55:56 -07:00
Steve Loughran
d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus. 
Contributed by Steve Moist.
 2017-04-25 17:42:30 +01:00
Mingliang Liu
3b7bb7b94b HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:07:57 -07:00
Steve Loughran
1ae6be46b4
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran
 2017-04-19 10:19:43 +01:00
Yongjun Zhang
c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Wangda Tan
5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 
(cherry picked from commit eac6b4c35c50e555c2f1b5f913bb2c4d839f1ff4)
 2017-04-13 13:16:31 -07:00
Mingliang Liu
03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit 0cab57223e3f54be17a5f27cefdb6d1da1b073e5)
 2017-04-12 16:09:20 -07:00
Mingliang Liu
508389ee54 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
 2017-04-12 14:30:54 -07:00
Chris Nauroth
1bfb382296 HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu. 2017-04-12 10:07:00 -07:00
John Zhuge
ff2870310e HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge. 
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-04-07 15:01:29 -07:00
Chris Douglas
72aa0a7ab3 HADOOP-13626. Remove distcp dependency on FileStatus serialization 
(cherry picked from commit a1a0281e12ea96476e75b076f76d5b5eb5254eea)
 2017-04-05 15:47:46 -07:00
Mingliang Liu
8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak 
(cherry picked from commit 56ab02eed9b61e1c80605104dfc4c87fc6abac96)
 2017-04-04 11:16:48 -07:00
Mingliang Liu
ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak 
(cherry picked from commit 27a44b60c1a55cabbf1873bca62db4e321c901ca)
 2017-04-04 11:04:42 -07:00
Mingliang Liu
8c21b2a259 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 
(cherry picked from commit 5faa949b782be48ef400d2eb1695f420455de764)
 2017-04-03 11:07:56 -07:00
Akira Ajisaka
bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0d053eeb3002f348d34cf3aac5b43333af8ff10a)
 2017-03-30 18:13:01 +09:00
John Zhuge
ef43a62482 HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 
(cherry picked from commit 253e3e78abb9ebe0e8103d25538ff07875b33c84)
 2017-03-27 23:07:08 -07:00
John Zhuge
100b8ce05b HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge. 
(cherry picked from commit d1b7439b48caa18d64a94be1ad5e4927ce573ab8)
 2017-03-24 13:41:09 -07:00
Chris Douglas
62b8665726 HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge 
(cherry picked from commit c55195588f5853b16d34cb6389cc6f84acc3edea)
 2017-03-23 12:17:42 -07:00
Chris Douglas
05148e33ce HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria 
(cherry picked from commit 7d4884a57bcaf2ab1ad120c436f9c7e6739dbc2a)
 2017-03-23 12:13:33 -07:00
Mingliang Liu
b24b416db5 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 
(cherry picked from commit 2e30aa72e01de7b5774fcb312406a393221e0908)
 2017-03-23 11:37:14 -07:00
Steve Loughran
20878d052c
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(). 
Contributed by Yuanbo Liu
 2017-03-23 12:52:22 +00:00
John Zhuge
84e2c11f96 HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge. 2017-03-21 08:23:30 -07:00
Steve Loughran
b4481fef41
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9ddacd5fdb740ac1bf9bfeb0ffbbd6c0e)
 2017-03-21 13:14:43 +00:00
Mingliang Liu
22930b09a0 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak 
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e240597f821968e14893eabfea39815de207)
 2017-03-20 16:37:57 -07:00
Arpit Agarwal
90e3283a0a HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:30 -07:00
Mingliang Liu
ed0d426a88 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 15:00:59 -07:00
John Zhuge
0da850a472 HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge. 
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
(cherry picked from commit 5a40bafdaeec693e613aa02e79dbaaccfdab6f60)
 2017-03-13 08:36:46 -07:00
Mingliang Liu
0d9030b732 HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu 
(cherry picked from commit 881ec4d97bd1db4582027aec3a4204156a4eda17)
 2017-03-10 13:22:24 -08:00
Steve Loughran
851c0df5d3 HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran 
(cherry picked from commit e60c6543d57611039b0438d5dcb4cb19ee239bb6)
 2017-03-10 13:22:02 -08:00
John Zhuge
f298b992f4 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge. 
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded46dcb1ac1ea4c1ada4949c50bc89afb)
 2017-03-09 19:53:23 -08:00
Mingliang Liu
acf20c8318 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane 
(cherry picked from commit 924def78544a64449785f305cb6984c3559aea4d)
 2017-03-09 15:16:28 -08:00
Mingliang Liu
a146866802 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane 
(cherry picked from commit f4329990250bed62efdebe3ce2bc740092cf9573)
 2017-03-09 15:16:28 -08:00
Lei Xu
ed4388f903 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei) 
(cherry picked from commit 71c23c9fc94cfdf58de80effbc3f51c0925d0cfe)
 2017-03-09 15:16:28 -08:00
Lei Xu
d3a6124ffa HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) 
(cherry picked from commit ccf2d662443fc169835d66b715441529658b245c)
 2017-03-09 15:16:28 -08:00
Lei Xu
01624f0c1b HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 
(cherry picked from commit e015b563197a475e354bf84fd27e7bbcc67e00a4)
 2017-03-09 15:16:28 -08:00
Mingliang Liu
881b63499e HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane 
(cherry picked from commit ef34bf2bb92a4e8def6617b185ae72db81450de8)
 2017-03-09 15:16:28 -08:00
Mingliang Liu
7fd0556b2b HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane 
(cherry picked from commit 4113ec5fa5ca049ebaba039b1faf3911c6a34f7b)
 2017-03-09 15:16:28 -08:00