551 Commits

Author SHA1 Message Date
Junping Du
ab72dbb809 Set branch version to 2.8.2 before preparing RC. 2017-08-31 18:08:16 -07:00
Naganarasimha
582bdfd423 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:47:25 +05:30
Mingliang Liu
7eee4fed89 HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
(cherry picked from commit 325163f23f727e82379d4a385b73aa3a04a510f6)
2017-06-09 13:07:55 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.)
396de0fd8f Preparing for 2.8.3 development 2017-06-07 15:38:20 -07:00
Christopher Douglas
d013f4134b YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
Contributed by Carlo Curino. Backport by Ye Zhou.

(cherry picked from commit e07f3c87cd3542760efcf713f909375b69eec00f)
2017-06-05 16:19:55 -07:00
John Zhuge
f8a3de64f8 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
(cherry picked from commit 6c6a7a59622ba7c1e4faa5534f4479de0cd84993)
2017-05-28 22:29:49 -07:00
Xuan
cef5ef2377 YARN-4612. Fix rumen and scheduler load simulator handle killed tasks
properly. Contributed by Ming Ma.

(cherry picked from commit 4efdf3a979c361348612f817a3253be6d0de58f7)
2017-05-21 12:51:16 -07:00
Akira Ajisaka
8007d024b8
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong.
(cherry picked from commit c60164fb60d3d8afb0fb8410936ccc231c4593f9)
(cherry picked from commit 753954f7beb3a1ca4a60cc3f59c809aed3839dd2)
2017-05-09 05:39:26 -05:00
Mingliang Liu
c3c9a91900 HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu
(cherry picked from commit e4c32bccde3147f51263cb58cdb480453b1cb3e9)
(cherry picked from commit e7a08cc6dc1f729d04dea9239f0a8c90c8be5598)
2017-04-25 20:55:42 -07:00
Steve Loughran
952113ebfd
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
Contributed by Steve Loughran

(cherry picked from commit 1ae6be46b4a02870f9471b05f8d39101b8885830)
2017-04-19 15:37:27 +01:00
Junping Du
dfac95a3a2 Preparing for 2.8.1 development 2017-04-11 11:21:49 -07:00
John Zhuge
fa8e411d47 HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge.
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
2017-04-07 15:05:09 -07:00
Mingliang Liu
e47c95ffbf HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 2017-04-03 11:15:03 -07:00
Akira Ajisaka
6e005ad326
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
(cherry picked from commit 0d053eeb3002f348d34cf3aac5b43333af8ff10a)
(cherry picked from commit bb187e145a851e734b00a31d9df2efe31a8ac214)
2017-03-30 18:13:55 +09:00
John Zhuge
c543c0549d HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge.
(cherry picked from commit 253e3e78abb9ebe0e8103d25538ff07875b33c84)
2017-03-27 23:08:03 -07:00
John Zhuge
d370d8ddb8 HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge.
(cherry picked from commit d1b7439b48caa18d64a94be1ad5e4927ce573ab8)
2017-03-24 13:41:34 -07:00
Chris Douglas
5e794c74e5 HADOOP-14197. Fix ADLS doc for credential provider. Contributed by John Zhuge
(cherry picked from commit c55195588f5853b16d34cb6389cc6f84acc3edea)
(cherry picked from commit 62b866572674e47f812be3d0e762bd5d488bfbdb)
2017-03-23 12:17:52 -07:00
Chris Douglas
88252e85ad HADOOP-14196. Azure Data Lake doc is missing required config entry. Contributed by Atul Sikaria
(cherry picked from commit 7d4884a57bcaf2ab1ad120c436f9c7e6739dbc2a)
(cherry picked from commit 2cea8293ef519ead772564750bd5e7f8b9d33da5)
2017-03-23 12:13:42 -07:00
Mingliang Liu
45956131b9 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu
(cherry picked from commit 2e30aa72e01de7b5774fcb312406a393221e0908)
2017-03-23 11:42:23 -07:00
John Zhuge
3a8b120c78 HADOOP-14205. No FileSystem for scheme: adl. Contributed by John Zhuge.
(cherry picked from commit 84e2c11f969825e52b6bdb4696f48e4567875d41)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
2017-03-21 08:37:18 -07:00
Steve Loughran
595501c9dd
HADOOP-14204 S3A multipart commit failing,
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran
2017-03-21 13:13:16 +00:00
John Zhuge
72fc7e0520 HADOOP-14173. Remove unused AdlConfKeys#ADL_EVENTS_TRACKING_SOURCE. Contributed by John Zhuge.
Change-Id: I5dc6f885816b8834f718874542dfa373458b0333
(cherry picked from commit 5a40bafdaeec693e613aa02e79dbaaccfdab6f60)
2017-03-13 09:15:09 -07:00
Mingliang Liu
b4d768123b HADOOP-14153. ADL module has messed doc structure. Contributed by Mingliang Liu
(cherry picked from commit 881ec4d97bd1db4582027aec3a4204156a4eda17)
2017-03-10 13:23:17 -08:00
Steve Loughran
1f92a9dbae HADOOP-14113. Review ADL Docs. Contributed by Steve Loughran
(cherry picked from commit e60c6543d57611039b0438d5dcb4cb19ee239bb6)
2017-03-10 13:23:11 -08:00
John Zhuge
843fa685a6 HADOOP-14123. Remove misplaced ADL service provider config file for FileSystem. Contributed by John Zhuge.
Change-Id: Ic956e2eb8189625916442eaffdc69163d32f730e
(cherry picked from commit c5ee7fded46dcb1ac1ea4c1ada4949c50bc89afb)
2017-03-09 19:54:19 -08:00
Mingliang Liu
518705b8e3 HADOOP-14017. User friendly name for ADLS user and group. Contributed by Vishwajeet Dusane
(cherry picked from commit 924def78544a64449785f305cb6984c3559aea4d)
(cherry picked from commit acf20c8318270eaa8fdf6259ec7b07344e3bd7e5)
2017-03-09 15:21:33 -08:00
Mingliang Liu
f0df481c76 HADOOP-14049. Honour AclBit flag associated to file/folder permission for Azure datalake account. Contributed by Vishwajeet Dusane
(cherry picked from commit f4329990250bed62efdebe3ce2bc740092cf9573)
(cherry picked from commit a14686680228d13386ccde7272e3b5bdabe7c792)
2017-03-09 15:21:32 -08:00
Lei Xu
674b9cdab1 HADOOP-13929. ADLS connector should not check in contract-test-options.xml. (John Zhuge via lei)
(cherry picked from commit 71c23c9fc94cfdf58de80effbc3f51c0925d0cfe)
(cherry picked from commit ed4388f903d34a2c09845d200e1e717397b217e1)
2017-03-09 15:21:32 -08:00
Lei Xu
87abe01ab1 HADOOP-13962. Update ADLS SDK to 2.1.4. (John Zhuge via lei)
(cherry picked from commit ccf2d662443fc169835d66b715441529658b245c)
(cherry picked from commit d3a6124ffaa76f0aeaad1a3c58e005a65f507a10)
2017-03-09 15:21:32 -08:00
Lei Xu
5eaec560d9 HADOOP-13956. Read ADLS credentials from Credential Provider. (John Zhuge via lei)
(cherry picked from commit e015b563197a475e354bf84fd27e7bbcc67e00a4)
(cherry picked from commit 01624f0c1b7eba9fe4b5778e1da9d38545af3722)
2017-03-09 15:21:32 -08:00
Mingliang Liu
8a51e4f89d HADOOP-13900. Remove snapshot version of SDK dependency from Azure Data Lake Store File System. Contributed by Vishwajeet Dusane
(cherry picked from commit ef34bf2bb92a4e8def6617b185ae72db81450de8)
(cherry picked from commit 881b63499e10b808b3c58ca34e8d4ad5a6634ed5)
2017-03-09 15:21:31 -08:00
Mingliang Liu
f1671e9519 HADOOP-13257. Improve Azure Data Lake contract tests. Contributed by Vishwajeet Dusane
(cherry picked from commit 4113ec5fa5ca049ebaba039b1faf3911c6a34f7b)
(cherry picked from commit 7fd0556b2bcc4eb18c5301a580646a2d7d502b50)
2017-03-09 15:21:31 -08:00
Chris Douglas
b4b4ca9199 HADOOP-13037. Refactor Azure Data Lake Store as an independent FileSystem. Contributed by Vishwajeet Dusane
(cherry picked from commit edf149b9790a96563fe7bba289a040542c8ab8f2)
2017-03-09 15:21:21 -08:00
John Zhuge
cdfa86cadd Revert "HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran"
This reverts commit 53a12fa721bb431f7d481aac7d245c93efb56153.

(cherry picked from commit 98142d2f722e82d57b0e2bae6276f7c17fd99598)

Change-Id: I3a7613a7f4f0e9c80bb6d3c40a53742f807fc689
2017-03-08 11:47:36 -08:00
Akira Ajisaka
20fa7664d1
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri.
(cherry picked from commit f597f4c43e0a6e2304b9bcaf727d6d8d15a365f9)
(cherry picked from commit d2d0872f4d109b09cdc3d8e9b163544e56469621)
2017-03-07 15:17:42 +09:00
Mingliang Liu
ff706261be HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
(cherry picked from commit c571cda5c7d929477961dfff4176d7de4944d874)
2017-03-06 17:01:26 -08:00
Steve Loughran
2acc4d913e
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38bf40373d2855163fa6444821b0f994c)
2017-03-02 21:43:05 +00:00
Steve Loughran
05ab09ca87
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
Contributed by Steve Loughran.
2017-02-25 16:10:15 +00:00
Steve Loughran
ff87ca8441
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 2017-02-24 10:35:21 +00:00
Mingliang Liu
b1ec790662 HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
(cherry picked from commit 82607fce39151fc6ba5bced738088e2bc176dc77)
2017-02-23 14:11:34 -08:00
Steve Loughran
bb4b2f641c HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
2017-02-21 18:29:46 +00:00
Steve Loughran
8a05ea4aba HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:12 +00:00
Steve Loughran
b1c1f05b1d HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 2017-02-18 18:16:19 +00:00
Mingliang Liu
3b4f840ccb HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
(cherry picked from commit bdad8b7b97d7f48119f016d68f32982d680c8796)
2017-02-16 16:43:22 -08:00
Akira Ajisaka
4718f65545 HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
(cherry picked from commit b9f8491252f5a23a91a1d695d748556a0fd803ae)
(cherry picked from commit 25ac54a52eff0d5b8f4bee83b6f4471277a3ac61)
2017-02-15 01:47:21 +09:00
Steve Loughran
77401bd5fc HADOOP-13496. Include file lengths in Mismatch in length error for distcp. Contributed by Ted Yu 2017-01-19 11:24:58 +00:00
Steve Loughran
e32cd52481 HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:15:01 +00:00
Junping Du
7706a63fb4 Preparing for 2.8.1 development after cutting off 2.8.0 branch. 2017-01-10 18:03:51 -08:00
Chris Nauroth
15e89634a5 HDFS-9483. Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 4c8f9e130230457fc897ed7a2a09e14d078be90a)
(cherry picked from commit 9c61858ee754b167a90533375e85ca5bde3cdce1)
2017-01-05 15:05:12 -08:00
Akira Ajisaka
58a1753373 HADOOP-12733. Remove references to obsolete io.seqfile configuration variables. Contributed by Ray Chiang.
(cherry picked from commit 01d31fe9389ccdc153d7f4bf6574bf8e509867c1)
(cherry picked from commit 3e2d26a97f83461d491180bbc6aa0ba926c0681b)
2017-01-04 14:20:04 +09:00