Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,687 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

716 Commits

Author SHA1 Message Date
Naganarasimha ac058c75a4 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:41:13 +05:30
Mingliang Liu c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu 
(cherry picked from commit 325163f23f)
 2017-06-09 13:07:35 -07:00
Mingliang Liu de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan 
(cherry picked from commit df35e4cc3b)
 2017-06-07 13:55:11 -07:00
Mingliang Liu e3a72c2b5c HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu 
(cherry picked from commit 536f057158)
 2017-06-06 11:13:42 -07:00
Mingliang Liu bacfd1baae HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu 
(cherry picked from commit 6b5285bbcb)
 2017-06-06 11:07:42 -07:00
Christopher Douglas 95c2978640 YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors. 
Contributed by Carlo Curino. Backport by Ye Zhou.

(cherry picked from commit e07f3c87cd)
 2017-06-05 16:13:51 -07:00
Mingliang Liu 53e90a8064 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan 
(cherry picked from commit 5fd9742c83)
 2017-06-05 16:01:27 -07:00
Mingliang Liu 867026d94e HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 
(cherry picked from commit 6aeda55bb8)
 2017-06-05 11:31:13 -07:00
Mingliang Liu 0b78db35fd HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu 
(cherry picked from commit ece33208b8)
 2017-06-01 11:52:37 -07:00
John Zhuge f72f194e24 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 
(cherry picked from commit 6c6a7a5962)
 2017-05-28 22:29:03 -07:00
Mingliang Liu 9d7e67ccf9 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari 
(cherry picked from commit 89bb8bfe58)
 2017-05-26 17:59:14 -07:00
Yongjun Zhang dd552a97b7 HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-24 19:05:40 -07:00
Yongjun Zhang 49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang 97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li." 
This reverts commit c411adac32.
 2017-05-24 18:54:51 -07:00
Yongjun Zhang 39474ed50d Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)" 
This reverts commit fe185e2c3a.
 2017-05-24 18:54:50 -07:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Yongjun Zhang fe185e2c3a HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-19 21:11:38 -07:00
Steve Loughran 2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. 
Contributed by Steve Loughran.
 2017-05-18 15:38:46 +01:00
Mingliang Liu f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit b415c6fe74)
 2017-05-16 11:28:12 -07:00
Andrew Wang 18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu 586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak 
(cherry picked from commit eed731496f)
 2017-05-10 15:05:54 -07:00
Akira Ajisaka 753954f7be
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong. 
(cherry picked from commit c60164fb60)
 2017-05-09 05:39:02 -05:00
Mingliang Liu 616413ecb2 HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran 
(cherry picked from commit 83dded556d)
 2017-05-03 11:23:15 -07:00
Mingliang Liu e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak 
(cherry picked from commit 8b5f2c372e)
 2017-04-26 13:51:58 -07:00
Mingliang Liu 5ceb61cb43 HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu 
(cherry picked from commit e4c32bccde)
 2017-04-25 20:55:56 -07:00
Steve Loughran d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus. 
Contributed by Steve Moist.
 2017-04-25 17:42:30 +01:00
Mingliang Liu 3b7bb7b94b HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:07:57 -07:00
Steve Loughran 1ae6be46b4
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran
 2017-04-19 10:19:43 +01:00
Yongjun Zhang c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Wangda Tan 5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 
(cherry picked from commit eac6b4c35c)
 2017-04-13 13:16:31 -07:00
Mingliang Liu 03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg 
(cherry picked from commit 0cab57223e)
 2017-04-12 16:09:20 -07:00
Mingliang Liu 508389ee54 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
 2017-04-12 14:30:54 -07:00
Chris Nauroth 1bfb382296 HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu. 2017-04-12 10:07:00 -07:00
John Zhuge ff2870310e HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge. 
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a20)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
 2017-04-07 15:01:29 -07:00
Chris Douglas 72aa0a7ab3 HADOOP-13626. Remove distcp dependency on FileStatus serialization 
(cherry picked from commit a1a0281e12)
 2017-04-05 15:47:46 -07:00
Mingliang Liu 8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak 
(cherry picked from commit 56ab02eed9)
 2017-04-04 11:16:48 -07:00
Mingliang Liu ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak 
(cherry picked from commit 27a44b60c1)
 2017-04-04 11:04:42 -07:00
Mingliang Liu 8c21b2a259 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 
(cherry picked from commit 5faa949b78)
 2017-04-03 11:07:56 -07:00
Akira Ajisaka bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0d053eeb30)
 2017-03-30 18:13:01 +09:00
John Zhuge ef43a62482 HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge. 
(cherry picked from commit 253e3e78ab)
 2017-03-27 23:07:08 -07:00
John Zhuge 100b8ce05b HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge. 
(cherry picked from commit d1b7439b48)
 2017-03-24 13:41:09 -07:00
Chris Douglas 62b8665726 HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge 
(cherry picked from commit c55195588f)
 2017-03-23 12:17:42 -07:00
Chris Douglas 05148e33ce HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria 
(cherry picked from commit 7d4884a57bcaf2ab1ad120c436f9c7e6739dbc2a)
 2017-03-23 12:13:33 -07:00
Mingliang Liu b24b416db5 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 
(cherry picked from commit 2e30aa72e0)
 2017-03-23 11:37:14 -07:00
Steve Loughran 20878d052c
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(). 
Contributed by Yuanbo Liu
 2017-03-23 12:52:22 +00:00
John Zhuge 84e2c11f96 HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge. 2017-03-21 08:23:30 -07:00
Steve Loughran b4481fef41
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9dd)
 2017-03-21 13:14:43 +00:00
Mingliang Liu 22930b09a0 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak 
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e24059)
 2017-03-20 16:37:57 -07:00
Arpit Agarwal 90e3283a0a HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:30 -07:00
Mingliang Liu ed0d426a88 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 15:00:59 -07:00
John Zhuge 0da850a472 HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge. 
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
(cherry picked from commit 5a40bafdae)
 2017-03-13 08:36:46 -07:00
Mingliang Liu 0d9030b732 HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu 
(cherry picked from commit 881ec4d97b)
 2017-03-10 13:22:24 -08:00
Steve Loughran 851c0df5d3 HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran 
(cherry picked from commit e60c6543d5)
 2017-03-10 13:22:02 -08:00
John Zhuge f298b992f4 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge. 
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded4)
 2017-03-09 19:53:23 -08:00
Mingliang Liu acf20c8318 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane 
(cherry picked from commit 924def7854)
 2017-03-09 15:16:28 -08:00
Mingliang Liu a146866802 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane 
(cherry picked from commit f432999025)
 2017-03-09 15:16:28 -08:00
Lei Xu ed4388f903 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei) 
(cherry picked from commit 71c23c9fc9)
 2017-03-09 15:16:28 -08:00
Lei Xu d3a6124ffa HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei) 
(cherry picked from commit ccf2d66244)
 2017-03-09 15:16:28 -08:00
Lei Xu 01624f0c1b HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei) 
(cherry picked from commit e015b56319)
 2017-03-09 15:16:28 -08:00
Mingliang Liu 881b63499e HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane 
(cherry picked from commit ef34bf2bb9)
 2017-03-09 15:16:28 -08:00
Mingliang Liu 7fd0556b2b HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane 
(cherry picked from commit 4113ec5fa5)
 2017-03-09 15:16:28 -08:00
Chris Douglas edf149b979 HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane 2017-03-09 15:16:28 -08:00
John Zhuge f1fd20be82 Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran" 
This reverts commit 53a12fa721.

(cherry picked from commit 98142d2f72)

Change-Id: Ia5fa901fa97391cd46d41f59cd05f7a19c7c7532
 2017-03-08 11:46:27 -08:00
Mingliang Liu d0e26075b8 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak 
(cherry picked from commit 28daaf0eb2)
 2017-03-07 14:57:41 -08:00
Akira Ajisaka d2d0872f4d
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 
(cherry picked from commit f597f4c43e)
 2017-03-07 15:16:18 +09:00
Mingliang Liu ff050866bf HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 
(cherry picked from commit 686823529b)
 2017-03-06 17:35:06 -08:00
Mingliang Liu 8323e651a1 Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth" 
This reverts commit b26870c58e.
 2017-03-06 17:18:36 -08:00
Mingliang Liu 2d059b5a40 HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA 
(cherry picked from commit c571cda5c7)
 2017-03-06 17:01:01 -08:00
Mingliang Liu b26870c58e HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:05 -08:00
Steve Loughran 51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran
 2017-03-02 21:41:49 +00:00
Yongjun Zhang 16ca70553c HADOOP-14127. Add log4j configuration to enable logging in hadoop-distcp's tests. (Xiao Chen via Yongjun Zhang) 
(cherry picked from commit d235dcdf0b)
 2017-02-27 20:46:17 -08:00
Steve Loughran 29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.
 2017-02-25 15:31:26 +00:00
Steve Loughran c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 
(cherry picked from commit ff87ca8441)
 2017-02-24 10:40:58 +00:00
Wangda Tan 650ff95e00 YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 
(cherry picked from commit 0a55bd841e)
 2017-02-23 17:22:25 -08:00
Mingliang Liu b0f8de184a HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari. 
(cherry picked from commit 82607fce39)
 2017-02-23 14:11:08 -08:00
Mingliang Liu db955f0844 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 
(cherry picked from commit 0013090fb4)
 2017-02-22 14:38:57 -08:00
Steve Loughran 171b18693a HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:41:24 +00:00
Steve Loughran 6843659689 HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 2017-02-21 18:29:30 +00:00
Steve Loughran 274c02d2fa HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:00 +00:00
Steve Loughran 8a88e8ea57 HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1d)
 2017-02-18 18:17:54 +00:00
Mingliang Liu b23e93d465 HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 
(cherry picked from commit 658702efff)
 2017-02-16 16:59:31 -08:00
Mingliang Liu 29e1007172 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 
(cherry picked from commit bdad8b7b97)
 2017-02-16 16:42:56 -08:00
Akira Ajisaka 25ac54a52e HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252)
 2017-02-15 01:46:59 +09:00
Lei Xu 6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Arpit Agarwal c88ec54588 HADOOP-14055. SwiftRestClient includes pass length in exception if auth fails. Contributed by Marcell Hegedus. 2017-02-10 10:09:39 -08:00
Akira Ajisaka b204a1eb5f HADOOP-13858. TestGridmixMemoryEmulation and TestResourceUsageEmulators fail on the environment other than Linux or Windows. 
(cherry picked from commit 98c35bb1a4)
 2017-01-21 17:35:57 +09:00
Steve Loughran b2704702f9 HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 
(cherry picked from commit 77401bd5fc)
 2017-01-19 11:25:20 +00:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804)
 2017-01-09 08:42:38 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Chris Nauroth 9c61858ee7 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 4c8f9e1302)
 2017-01-05 15:05:01 -08:00
Akira Ajisaka 3e2d26a97f HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang. 
(cherry picked from commit 01d31fe938)
 2017-01-04 14:15:00 +09:00
Mingliang Liu ec646b1a9d HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki 
(cherry picked from commit c6a5b689db)
 2016-12-29 16:24:38 -08:00
Mingliang Liu edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth 
(cherry picked from commit e92a77099b)
 2016-12-22 20:33:51 -08:00
Akira Ajisaka 522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
 2016-12-23 04:06:53 +09:00
Mingliang Liu 867389247f HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade 
(cherry picked from commit f92913c35b)
 2016-12-15 12:38:31 -08:00
Mingliang Liu 2dbd86bcf4 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:06:39 -08:00
Mingliang Liu 89eaa94c59 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth 
(cherry picked from commit 15dd1f3381)
 2016-12-05 12:06:35 -08:00
Mingliang Liu c37e814fae HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:48:38 -08:00
Mingliang Liu a4e0539636 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:39:23 -08:00
Mingliang Liu 39b74b6f9e HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99f)
 2016-11-30 13:04:44 -08:00
Mingliang Liu 1b831d8e01 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8a)
 2016-11-28 16:35:18 -08:00
Steve Loughran 9701ab48e8 HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:34:11 +00:00
Steve Loughran f3fff5c2e1 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:04 +00:00
Mingliang Liu cae4a8c05d HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:05:58 -08:00
Mingliang Liu 9af7d1e298 HADOOP-13655. document object store use with fs shell and distcp. Contributed by Steve Loughran 2016-11-22 13:19:32 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Mingliang Liu d064d5dbb2 HADOOP-13687. Provide a unified dependency artifact that transitively includes the cloud storage modules shipped with Hadoop. Contributed by Chris Nauroth 
(cherry picked from commit 89354f0475)
 2016-11-10 08:59:07 -08:00
Jason Lowe 8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00
Akira Ajisaka 3b279fe426 HADOOP-13798. TestHadoopArchives times out. 
(cherry picked from commit b970446b2c)
 2016-11-07 19:54:50 +09:00
Steve Loughran c4ccafdaff HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:52:49 +00:00
Masatake Iwasaki 7e918637de HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran. 
(cherry picked from commit 0bdd263d82)
 2016-10-27 15:54:03 +09:00
Chris Nauroth 082d69ee66 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
 2016-10-26 08:57:29 -07:00
Chris Nauroth 67e01f7218 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
 2016-10-26 08:27:33 -07:00
Akira Ajisaka e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a667)
 2016-10-26 15:34:40 +09:00
Yongjun Zhang ce72314657 HDFS-11040. Add documentation for HDFS-9820 distcp improvement. Contributed by Yongjun Zhang. 
(cherry picked from commit 0f0c15f7a5)
 2016-10-25 12:29:28 -07:00
Yongjun Zhang 7bc170ba26 HDFS-9820. Improve distcp to support efficient restore to an earlier snapshot. Contributed by Yongjun Zhang. 
(cherry picked from commit 8650cc84f2)

Conflicts
	hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpOptions.java
	hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestOptionsParser.java
 2016-10-25 11:32:26 -07:00
Chris Nauroth 05d772e297 HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
 2016-10-25 09:03:11 -07:00
Chris Nauroth 5c2f67bdae HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e94030)
 2016-10-24 21:54:16 -07:00
Chris Nauroth 5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
 2016-10-24 21:22:46 -07:00
Steve Loughran 385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Steve Loughran f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb.
 2016-10-20 22:48:26 +01:00
Anu Engineer 42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran c640ba5fc9 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:30 +01:00
Steve Loughran bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Allen Wittenauer a5a56c3564 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 
(cherry picked from commit 422c73a865)
 2016-10-17 16:35:32 -07:00
Andrew Wang 4ed7cf3b36 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 
(cherry picked from commit 987ee51141)
 2016-10-17 13:32:39 -07:00
Jing Zhao ad69baf6a9 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d07983)
 2016-10-13 13:26:02 -07:00
Chris Nauroth 2de2d0df1d HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
 2016-10-07 11:41:31 -07:00
Steve Loughran a6bb21eec4 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:47:14 +01:00
Steve Loughran 6ed335a018 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:15 +01:00
Chris Nauroth 213f0ad7f0 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
 2016-10-04 10:37:04 -07:00
Steve Loughran 896df3f55a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:00:34 +01:00
Chris Nauroth 1f1e47e411 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
 2016-09-28 16:21:00 -07:00
Brahma Reddy Battula 5737d04c5f HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970be)
 2016-09-27 10:47:07 +05:30
Steve Loughran 247cdbb86a HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:32 +01:00
Chris Nauroth e19f910245 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb51397)
 2016-09-19 15:16:56 -07:00
Steve Loughran 3b9268cfba HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:44:45 +01:00
Steve Loughran 6a016faf59 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:34 +01:00
Steve Loughran 5f8b6f0313 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:24:15 +01:00
Chris Nauroth e28930a38b HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
 2016-09-06 09:40:41 -07:00
Arun Suresh b279f42d79 YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 
(cherry picked from commit d6d9cff21b)
 2016-08-30 16:33:31 -07:00
Chris Nauroth cda68de9b9 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:19:20 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Chris Nauroth 804a40720b HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 
(cherry picked from commit ae4db25443)
 2016-08-18 10:45:03 -07:00
Junping Du d464483bf7 YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 2016-08-18 07:23:29 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Chris Nauroth 986162f977 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
 2016-08-17 14:55:07 -07:00
Chris Nauroth 555a57adf2 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
 2016-08-16 17:09:28 -07:00
Chris Nauroth 287688d6ca HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka. 
(cherry picked from commit 2ed58c40e5)
 2016-08-08 12:28:46 -07:00