725 Commits

Author SHA1 Message Date
Mingliang Liu
ba76897392 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
(cherry picked from commit 990aa34de23c625163745ebc338483065d955bbe)
2017-06-28 14:20:44 -07:00
Steve Loughran
44d43a8848
HADOOP-14190. Add more on S3 regions to the s3a documentation.
Contributed by Steve Loughran

(cherry picked from commit ee243e5289212aa2912d191035802ea023367e19)
2017-06-28 10:26:05 +01:00
Mingliang Liu
ce57458ec5 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
(cherry picked from commit 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)
2017-06-27 17:32:56 -07:00
Mingliang Liu
1b1065a163 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
(cherry picked from commit 686a634f01e454537d1a9d531330e60a60233ca4)
2017-06-27 16:49:48 -07:00
Mingliang Liu
ea402c364f HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
(cherry picked from commit bc4dfe9c9cb31f39e8f1532d5c5837faf2e92bde)
2017-06-27 13:20:03 -07:00
Steve Loughran
3a87591df8 HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu.
(cherry picked from commit 48f4a229a93b65df9e99df2505719dd3538f1188)
2017-06-26 15:22:44 -07:00
Akira Ajisaka
239b382153
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he.
(cherry picked from commit a594f96fb1155d94fe08ed8f85dca789a9a4c790)
2017-06-26 16:41:44 +09:00
Steve Loughran
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
Contributed by Thomas.

(cherry picked from commit c6bd73c6c5760c3a52203e6a51628587ceec6896)
2017-06-23 21:40:05 +01:00
Akira Ajisaka
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:21:12 +09:00
Naganarasimha
ac058c75a4 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:41:13 +05:30
Mingliang Liu
c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
(cherry picked from commit 325163f23f727e82379d4a385b73aa3a04a510f6)
2017-06-09 13:07:35 -07:00
Mingliang Liu
de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan
(cherry picked from commit df35e4cc3b0c67bcaa3d809c6cb5a0a2d82986af)
2017-06-07 13:55:11 -07:00
Mingliang Liu
e3a72c2b5c HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
(cherry picked from commit 536f057158c445a57049f6c392869ae2f0be4f24)
2017-06-06 11:13:42 -07:00
Mingliang Liu
bacfd1baae HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu
(cherry picked from commit 6b5285bbcb439944ba6c4701571ffbb00258d5a1)
2017-06-06 11:07:42 -07:00
Christopher Douglas
95c2978640 YARN-1471. Preserve scheduler typeinfo in simulator to work with resource monitors.
Contributed by Carlo Curino. Backport by Ye Zhou.

(cherry picked from commit e07f3c87cd3542760efcf713f909375b69eec00f)
2017-06-05 16:13:51 -07:00
Mingliang Liu
53e90a8064 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan
(cherry picked from commit 5fd9742c83fbeae96bf0913bdcdf77fafbf15b2f)
2017-06-05 16:01:27 -07:00
Mingliang Liu
867026d94e HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu
(cherry picked from commit 6aeda55bb8f741d9dafd41f6dfbf1a88acdd4003)
2017-06-05 11:31:13 -07:00
Mingliang Liu
0b78db35fd HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu
(cherry picked from commit ece33208b89a6a06c1b362d10fab31343fe5df13)
2017-06-01 11:52:37 -07:00
John Zhuge
f72f194e24 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
(cherry picked from commit 6c6a7a59622ba7c1e4faa5534f4479de0cd84993)
2017-05-28 22:29:03 -07:00
Mingliang Liu
9d7e67ccf9 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari
(cherry picked from commit 89bb8bfe582ba85566cede321b233bb642f1c675)
2017-05-26 17:59:14 -07:00
Yongjun Zhang
dd552a97b7 HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-24 19:05:40 -07:00
Yongjun Zhang
49dc547252 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-05-24 18:54:51 -07:00
Yongjun Zhang
97317b1aaa Revert "HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li."
This reverts commit c411adac32cda42433b15979253248336487d977.
2017-05-24 18:54:51 -07:00
Yongjun Zhang
39474ed50d Revert "HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang)"
This reverts commit fe185e2c3a07d4e6a82ed4fd255c98ec8b561ad6.
2017-05-24 18:54:50 -07:00
Sunil G
ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Yongjun Zhang
fe185e2c3a HADOOP-14407. DistCp - Introduce a configurable copy buffer size. (Omkar Aradhya K S via Yongjun Zhang) 2017-05-19 21:11:38 -07:00
Steve Loughran
2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
Contributed by Steve Loughran.
2017-05-18 15:38:46 +01:00
Mingliang Liu
f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg
(cherry picked from commit b415c6fe743242acf1d1d3eb7ea7091d90d2c0d4)
2017-05-16 11:28:12 -07:00
Andrew Wang
18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu
586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak
(cherry picked from commit eed731496f63537350206274af09d2118b116a17)
2017-05-10 15:05:54 -07:00
Akira Ajisaka
753954f7be
HADOOP-14374. License error in GridmixTestUtils.java. Contributed by lixinglong.
(cherry picked from commit c60164fb60d3d8afb0fb8410936ccc231c4593f9)
2017-05-09 05:39:02 -05:00
Mingliang Liu
616413ecb2 HDFS-11739. Fix regression in tests caused by YARN-679. Contributed by Steve Loughran
(cherry picked from commit 83dded556dc179fcff078451fb80533065e116f0)
2017-05-03 11:23:15 -07:00
Mingliang Liu
e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak
(cherry picked from commit 8b5f2c372e70999f3ee0a0bd685a494e06bc3652)
2017-04-26 13:51:58 -07:00
Mingliang Liu
5ceb61cb43 HADOOP-14349. Rename ADLS CONTRACT_ENABLE_KEY. Contributed by Mingliang Liu
(cherry picked from commit e4c32bccde3147f51263cb58cdb480453b1cb3e9)
2017-04-25 20:55:56 -07:00
Steve Loughran
d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
Contributed by Steve Moist.
2017-04-25 17:42:30 +01:00
Mingliang Liu
3b7bb7b94b HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:07:57 -07:00
Steve Loughran
1ae6be46b4
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
Contributed by Steve Loughran
2017-04-19 10:19:43 +01:00
Yongjun Zhang
c411adac32 HADOOP-11794. Enable distcp to copy blocks in parallel. Contributed by Yongjun Zhang, Wei-Chiu Chuang, Xiao Chen, Rosie Li. 2017-04-14 10:14:02 -07:00
Wangda Tan
5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda)
(cherry picked from commit eac6b4c35c50e555c2f1b5f913bb2c4d839f1ff4)
2017-04-13 13:16:31 -07:00
Mingliang Liu
03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg
(cherry picked from commit 0cab57223e3f54be17a5f27cefdb6d1da1b073e5)
2017-04-12 16:09:20 -07:00
Mingliang Liu
508389ee54 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
2017-04-12 14:30:54 -07:00
Chris Nauroth
1bfb382296 HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu. 2017-04-12 10:07:00 -07:00
John Zhuge
ff2870310e HADOOP-14174. Set default ADLS access token provider type to ClientCredential. Contributed by John Zhuge.
Signed-off-by: John Zhuge <jzhuge@apache.org>
(cherry picked from commit 56e81f2a204d5e65d29222df1b2ba4d892f9c0d5)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
	hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/conf/TestCommonConfigurationFields.java
2017-04-07 15:01:29 -07:00
Chris Douglas
72aa0a7ab3 HADOOP-13626. Remove distcp dependency on FileStatus serialization
(cherry picked from commit a1a0281e12ea96476e75b076f76d5b5eb5254eea)
2017-04-05 15:47:46 -07:00
Mingliang Liu
8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak
(cherry picked from commit 56ab02eed9b61e1c80605104dfc4c87fc6abac96)
2017-04-04 11:16:48 -07:00
Mingliang Liu
ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak
(cherry picked from commit 27a44b60c1a55cabbf1873bca62db4e321c901ca)
2017-04-04 11:04:42 -07:00
Mingliang Liu
8c21b2a259 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka
(cherry picked from commit 5faa949b782be48ef400d2eb1695f420455de764)
2017-04-03 11:07:56 -07:00
Akira Ajisaka
bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
(cherry picked from commit 0d053eeb3002f348d34cf3aac5b43333af8ff10a)
2017-03-30 18:13:01 +09:00
John Zhuge
ef43a62482 HADOOP-14038. Rename ADLS credential properties. Contributed by John Zhuge.
(cherry picked from commit 253e3e78abb9ebe0e8103d25538ff07875b33c84)
2017-03-27 23:07:08 -07:00
John Zhuge
100b8ce05b HADOOP-14230. TestAdlFileSystemContractLive fails to clean up. Contributed by John Zhuge.
(cherry picked from commit d1b7439b48caa18d64a94be1ad5e4927ce573ab8)
2017-03-24 13:41:09 -07:00