Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,052 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

666 Commits

Author SHA1 Message Date
Naganarasimha ac058c75a4 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:41:13 +05:30
Mingliang Liu c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu 
(cherry picked from commit 325163f23f)
 2017-06-09 13:07:35 -07:00
Mingliang Liu de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan 
(cherry picked from commit df35e4cc3b)
 2017-06-07 13:55:11 -07:00
Mingliang Liu e3a72c2b5c HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu 
(cherry picked from commit 536f057158)
 2017-06-06 11:13:42 -07:00
Mingliang Liu bacfd1baae HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu 
(cherry picked from commit 6b5285bbcb)
 2017-06-06 11:07:42 -07:00
Christopher Douglas 95c2978640 YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. 
Contributed by Carlo Curino. Backport by Ye Zhou.

(cherry picked from commit e07f3c87cd)
 2017-06-05 16:13:51 -07:00
Mingliang Liu 53e90a8064 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan 
(cherry picked from commit 5fd9742c83)
 2017-06-05 16:01:27 -07:00
Mingliang Liu 867026d94e HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 
(cherry picked from commit 6aeda55bb8)
 2017-06-05 11:31:13 -07:00
Mingliang Liu 0b78db35fd HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu 
(cherry picked from commit ece33208b8)
 2017-06-01 11:52:37 -07:00
John Zhuge f72f194e24 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 
(cherry picked from commit 6c6a7a5962)
 2017-05-28 22:29:03 -07:00
Mingliang Liu 9d7e67ccf9 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari 
(cherry picked from commit 89bb8bfe58)
 2017-05-26 17:59:14 -07:00
Yongjun Zhang dd552a97b7 HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-24 19:05:40 -07:00
Yongjun Zhang 49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang 97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li." 
This reverts commit c411adac32.
 2017-05-24 18:54:51 -07:00
Yongjun Zhang 39474ed50d Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)" 
This reverts commit fe185e2c3a.
 2017-05-24 18:54:50 -07:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Yongjun Zhang fe185e2c3a HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-19 21:11:38 -07:00
Steve Loughran 2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. 
Contributed by Steve Loughran.
 2017-05-18 15:38:46 +01:00
Mingliang Liu f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit b415c6fe74)
 2017-05-16 11:28:12 -07:00
Andrew Wang 18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu 586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak 
(cherry picked from commit eed731496f)
 2017-05-10 15:05:54 -07:00
Akira Ajisaka 753954f7be
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong. 
(cherry picked from commit c60164fb60)
 2017-05-09 05:39:02 -05:00
Mingliang Liu 616413ecb2 HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran 
(cherry picked from commit 83dded556d)
 2017-05-03 11:23:15 -07:00
Mingliang Liu e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak 
(cherry picked from commit 8b5f2c372e)
 2017-04-26 13:51:58 -07:00
Mingliang Liu 5ceb61cb43 HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu 
(cherry picked from commit e4c32bccde)
 2017-04-25 20:55:56 -07:00
Steve Loughran d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus. 
Contributed by Steve Moist.
 2017-04-25 17:42:30 +01:00
Mingliang Liu 3b7bb7b94b HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:07:57 -07:00
Steve Loughran 1ae6be46b4
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran
 2017-04-19 10:19:43 +01:00
Yongjun Zhang c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Wangda Tan 5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 
(cherry picked from commit eac6b4c35c)
 2017-04-13 13:16:31 -07:00
Mingliang Liu 03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit 0cab57223e)
 2017-04-12 16:09:20 -07:00
Mingliang Liu 508389ee54 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
 2017-04-12 14:30:54 -07:00
Chris Nauroth 1bfb382296 HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu. 2017-04-12 10:07:00 -07:00
John Zhuge ff2870310e HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge. 
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a20)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-04-07 15:01:29 -07:00
Chris Douglas 72aa0a7ab3 HADOOP-13626. Remove distcp dependency on FileStatus serialization 
(cherry picked from commit a1a0281e12)
 2017-04-05 15:47:46 -07:00
Mingliang Liu 8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak 
(cherry picked from commit 56ab02eed9)
 2017-04-04 11:16:48 -07:00
Mingliang Liu ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak 
(cherry picked from commit 27a44b60c1)
 2017-04-04 11:04:42 -07:00
Mingliang Liu 8c21b2a259 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 
(cherry picked from commit 5faa949b78)
 2017-04-03 11:07:56 -07:00
Akira Ajisaka bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0d053eeb30)
 2017-03-30 18:13:01 +09:00
John Zhuge ef43a62482 HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 
(cherry picked from commit 253e3e78ab)
 2017-03-27 23:07:08 -07:00
John Zhuge 100b8ce05b HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge. 
(cherry picked from commit d1b7439b48)
 2017-03-24 13:41:09 -07:00
Chris Douglas 62b8665726 HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge 
(cherry picked from commit c55195588f)
 2017-03-23 12:17:42 -07:00
Chris Douglas 05148e33ce HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria 
(cherry picked from commit 7d4884a57bcaf2ab1ad120c436f9c7e6739dbc2a)
 2017-03-23 12:13:33 -07:00
Mingliang Liu b24b416db5 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 
(cherry picked from commit 2e30aa72e0)
 2017-03-23 11:37:14 -07:00
Steve Loughran 20878d052c
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(). 
Contributed by Yuanbo Liu
 2017-03-23 12:52:22 +00:00
John Zhuge 84e2c11f96 HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge. 2017-03-21 08:23:30 -07:00
Steve Loughran b4481fef41
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9dd)
 2017-03-21 13:14:43 +00:00
Mingliang Liu 22930b09a0 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak 
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e24059)
 2017-03-20 16:37:57 -07:00
Arpit Agarwal 90e3283a0a HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:30 -07:00
Mingliang Liu ed0d426a88 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 15:00:59 -07:00
John Zhuge 0da850a472 HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge. 
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
(cherry picked from commit 5a40bafdae)
 2017-03-13 08:36:46 -07:00
Mingliang Liu 0d9030b732 HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu 
(cherry picked from commit 881ec4d97b)
 2017-03-10 13:22:24 -08:00
Steve Loughran 851c0df5d3 HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran 
(cherry picked from commit e60c6543d5)
 2017-03-10 13:22:02 -08:00
John Zhuge f298b992f4 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge. 
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded4)
 2017-03-09 19:53:23 -08:00
Mingliang Liu acf20c8318 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane 
(cherry picked from commit 924def7854)
 2017-03-09 15:16:28 -08:00
Mingliang Liu a146866802 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane 
(cherry picked from commit f432999025)
 2017-03-09 15:16:28 -08:00
Lei Xu ed4388f903 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei) 
(cherry picked from commit 71c23c9fc9)
 2017-03-09 15:16:28 -08:00
Lei Xu d3a6124ffa HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) 
(cherry picked from commit ccf2d66244)
 2017-03-09 15:16:28 -08:00
Lei Xu 01624f0c1b HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 
(cherry picked from commit e015b56319)
 2017-03-09 15:16:28 -08:00
Mingliang Liu 881b63499e HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane 
(cherry picked from commit ef34bf2bb9)
 2017-03-09 15:16:28 -08:00
Mingliang Liu 7fd0556b2b HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane 
(cherry picked from commit 4113ec5fa5)
 2017-03-09 15:16:28 -08:00
Chris Douglas edf149b979 HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane 2017-03-09 15:16:28 -08:00
John Zhuge f1fd20be82 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran" 
This reverts commit 53a12fa721.

(cherry picked from commit 98142d2f72)

Change-Id: Ia5fa901fa97391cd46d41f59cd05f7a19c7c7532
 2017-03-08 11:46:27 -08:00
Mingliang Liu d0e26075b8 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak 
(cherry picked from commit 28daaf0eb2)
 2017-03-07 14:57:41 -08:00
Akira Ajisaka d2d0872f4d
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 
(cherry picked from commit f597f4c43e)
 2017-03-07 15:16:18 +09:00
Mingliang Liu ff050866bf HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 
(cherry picked from commit 686823529b)
 2017-03-06 17:35:06 -08:00
Mingliang Liu 8323e651a1 Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth" 
This reverts commit b26870c58e.
 2017-03-06 17:18:36 -08:00
Mingliang Liu 2d059b5a40 HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA 
(cherry picked from commit c571cda5c7)
 2017-03-06 17:01:01 -08:00
Mingliang Liu b26870c58e HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:05 -08:00
Steve Loughran 51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran
 2017-03-02 21:41:49 +00:00
Yongjun Zhang 16ca70553c HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 
(cherry picked from commit d235dcdf0b)
 2017-02-27 20:46:17 -08:00
Steve Loughran 29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.
 2017-02-25 15:31:26 +00:00
Steve Loughran c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 
(cherry picked from commit ff87ca8441)
 2017-02-24 10:40:58 +00:00
Wangda Tan 650ff95e00 YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 
(cherry picked from commit 0a55bd841e)
 2017-02-23 17:22:25 -08:00
Mingliang Liu b0f8de184a HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari. 
(cherry picked from commit 82607fce39)
 2017-02-23 14:11:08 -08:00
Mingliang Liu db955f0844 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 
(cherry picked from commit 0013090fb4)
 2017-02-22 14:38:57 -08:00
Steve Loughran 171b18693a HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:41:24 +00:00
Steve Loughran 6843659689 HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 2017-02-21 18:29:30 +00:00
Steve Loughran 274c02d2fa HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:00 +00:00
Steve Loughran 8a88e8ea57 HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1d)
 2017-02-18 18:17:54 +00:00
Mingliang Liu b23e93d465 HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 
(cherry picked from commit 658702efff)
 2017-02-16 16:59:31 -08:00
Mingliang Liu 29e1007172 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 
(cherry picked from commit bdad8b7b97)
 2017-02-16 16:42:56 -08:00
Akira Ajisaka 25ac54a52e HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252)
 2017-02-15 01:46:59 +09:00
Lei Xu 6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Arpit Agarwal c88ec54588 HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus. 2017-02-10 10:09:39 -08:00
Akira Ajisaka b204a1eb5f HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows. 
(cherry picked from commit 98c35bb1a4)
 2017-01-21 17:35:57 +09:00
Steve Loughran b2704702f9 HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fc)
 2017-01-19 11:25:20 +00:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804)
 2017-01-09 08:42:38 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Chris Nauroth 9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e1302)
 2017-01-05 15:05:01 -08:00
Akira Ajisaka 3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe938)
 2017-01-04 14:15:00 +09:00
Mingliang Liu ec646b1a9d HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki 
(cherry picked from commit c6a5b689db)
 2016-12-29 16:24:38 -08:00
Mingliang Liu edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 
(cherry picked from commit e92a77099b)
 2016-12-22 20:33:51 -08:00
Akira Ajisaka 522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
 2016-12-23 04:06:53 +09:00
Mingliang Liu 867389247f HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade 
(cherry picked from commit f92913c35b)
 2016-12-15 12:38:31 -08:00
Mingliang Liu 2dbd86bcf4 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:06:39 -08:00
Mingliang Liu 89eaa94c59 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth 
(cherry picked from commit 15dd1f3381)
 2016-12-05 12:06:35 -08:00
Mingliang Liu c37e814fae HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:48:38 -08:00
Mingliang Liu a4e0539636 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:39:23 -08:00