Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,783 Commits 326 Branches 371 Tags
Commit Graph

663 Commits

Author SHA1 Message Date
Steve Loughran
ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c4d5554213d02bd0ad343ba82cf82aa
 2017-10-11 18:09:13 +01:00
Steve Loughran
34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-07 11:20:18 +01:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
 2017-10-05 16:03:13 +01:00
Steve Loughran
28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. 
Contributed by Steve Loughran
 2017-10-03 20:06:45 +01:00
Steve Loughran
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-10-02 14:28:59 +01:00
Steve Loughran
a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.

(cherry-picked from/based on commit 621b43e254afaff708cd6fc4698b29628f6abc33)
 2017-09-29 11:29:22 +01:00
Steve Loughran
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-09-29 11:16:31 +01:00
Steve Loughran
27a41848ac
HADOOP-14890 Move up to AWS SDK 1.11.199. 
Contributed by Steve Loughran.
 2017-09-26 20:11:10 +01:00
Jason Lowe
2c10abaddd MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b133dc5700b66318ac6d80e463b049723c511f37)
 2017-09-22 15:30:53 -05:00
Steve Loughran
12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory. 
Contributed by Esfandiar Manii.
 2017-09-19 11:02:10 +01:00
Xiaoyu Yao
b669aca518 HADOOP-14839. DistCp log output should contain copied and deleted files and directories. Contributed by Yiqun Lin. 2017-09-11 11:40:48 -07:00
Steve Loughran
87af3f4991
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov
 2017-09-11 14:15:58 +01:00
John Zhuge
1421196d20 HADOOP-14844. Remove requirement to specify TenantGuid for MSI Token Provider. Contributed by Atul Sikaria. 
(cherry picked from commit a4661850c1e0794baf493a468191e12681d68ab4)
 2017-09-08 14:15:51 -07:00
Steve Loughran
6e20ae954c
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.

(cherry picked from commit 83449abb40633a36b22de538cf82fca240af067e)
 2017-09-07 19:58:40 +01:00
John Zhuge
4c34c03b27 HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 2017-09-06 01:03:18 -07:00
Steve Loughran
f382f66255
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS. 
Contributed by Sivaguru Sankaridurg
 2017-09-05 14:18:43 +01:00
John Zhuge
2442a8d716 HADOOP-14824. Update ADLS SDK to 2.2.2 for MSI fix. Contributed by Atul Sikaria. 
(cherry picked from commit 7ecc6dbed62c80397f71949bee41dcd03065755c)
 2017-08-31 21:13:50 -07:00
Yufei Gu
626fa7c8e1 YARN-7077. TestAMSimulator and TestNMSimulator fail. (Contributed by Akira Ajisaka via Yufei Gu) 2017-08-30 23:08:02 -07:00
Steve Loughran
25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses. 
Contributed by Sivaguru Sankaridurg

(cherry picked from commit 021974f4cb1eb8cf530dabc95039fed46e66a43d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2017-08-30 22:40:50 +01:00
Arun Suresh
f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 8410d862d3a72740f461ef91dddb5325955e1ca5)
 2017-08-20 08:31:17 -07:00
Steve Loughran
c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted. 
Contributed by Thomas Marquardt

(cherry picked from commit c6b4e656b76b68cc1d0dbcc15a5aa5ea23335b7b)
 2017-08-18 14:38:04 +01:00
Steve Loughran
d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded. 
Contributed by Thomas Marquardt.
 2017-08-16 13:49:47 +01:00
John Zhuge
0efc590e6d HADOOP-14627. Support MSI and DeviceCode token provider in ADLS. Contributed by Atul Sikaria. 
(cherry picked from commit 7769e9614956283a86eda9e4e69aaa592c0ca960)
 2017-08-13 00:27:33 -07:00
Steve Loughran
96befde480
HADOOP-14183. Remove service loader config file for wasb fs. 
Contributed by Esfandiar Manii.

(cherry picked from commit 54356b1e8366a23fff1bb45601efffc743306efc)
 2017-08-10 16:49:51 +01:00
Steve Loughran
bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing. 
Contributed by Esfandiar Manii.
 2017-08-10 16:49:19 +01:00
Steve Loughran
7f7ab0302f
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek. 
Contributed by Thomas Marquardt

(cherry picked from commit d91b7a8451489f97bdde928cea774764155cfe03)
 2017-08-06 20:30:40 +01:00
Steve Loughran
ced547d5f0
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran
 2017-08-04 11:07:34 +01:00
John Zhuge
ba94c68e0b HADOOP-14678. AdlFilesystem#initialize swallows exception when getting user name. Contributed by John Zhuge. 
(cherry picked from commit f735ad1b67ed82d9b11b1afd7ae39035a6aed18b)
 2017-07-28 09:51:32 -07:00
Jitendra Pandey
554c3cd21d HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:35:40 -07:00
Jitendra Pandey
ab95086ed1 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-25 08:31:35 -07:00
Jitendra Pandey
23b920cd7a HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 11:37:36 -07:00
Jitendra Pandey
b9f2effaf6 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 16:25:05 -07:00
Steve Loughran
df69b13b47
HADOOP-14658. branch-2 compilation is broken in hadoop-azure 
Contributed by Sunil G.

Signed-off-by: Steve Loughran <stevel@apache.org>
 2017-07-13 18:25:38 +01:00
Steve Loughran
3dfd107070
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb. 
Contributed by Varada Hemeswari
(fix for java 7 compilation)
 2017-07-12 11:07:22 +01:00
Steve Loughran
1e69e52603
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb. 
Contributed by Varada Hemeswari
 2017-07-12 10:35:59 +01:00
Steve Loughran
41e83b2ca2
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs. 
Contributed by Thomas Marquardt

(cherry picked from commit d670c3a4da7dd80dccf6c6308603bb3bb013b3b0)
 2017-07-11 21:45:24 +01:00
Jitendra Pandey
d94b30cb03 HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-10 17:30:31 -07:00
Mingliang Liu
2d69952925 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran 
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
(cherry picked from commit 72993b33b704991f2a0bf743f31b164e58a2dabc)
 2017-06-29 17:08:51 -07:00
Mingliang Liu
ba76897392 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran 
(cherry picked from commit 990aa34de23c625163745ebc338483065d955bbe)
 2017-06-28 14:20:44 -07:00
Steve Loughran
44d43a8848
HADOOP-14190. Add more on S3 regions to the s3a documentation. 
Contributed by Steve Loughran

(cherry picked from commit ee243e5289212aa2912d191035802ea023367e19)
 2017-06-28 10:26:05 +01:00
Mingliang Liu
ce57458ec5 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas 
(cherry picked from commit 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)
 2017-06-27 17:32:56 -07:00
Mingliang Liu
1b1065a163 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran 
(cherry picked from commit 686a634f01e454537d1a9d531330e60a60233ca4)
 2017-06-27 16:49:48 -07:00
Mingliang Liu
ea402c364f HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu 
(cherry picked from commit bc4dfe9c9cb31f39e8f1532d5c5837faf2e92bde)
 2017-06-27 13:20:03 -07:00
Steve Loughran
3a87591df8 HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu. 
(cherry picked from commit 48f4a229a93b65df9e99df2505719dd3538f1188)
 2017-06-26 15:22:44 -07:00
Akira Ajisaka
239b382153
HADOOP-14549. Use GenericTestUtils.setLogLevel when available in hadoop-tools. Contributed by wenxin he. 
(cherry picked from commit a594f96fb1155d94fe08ed8f85dca789a9a4c790)
 2017-06-26 16:41:44 +09:00
Steve Loughran
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations. 
Contributed by Thomas.

(cherry picked from commit c6bd73c6c5760c3a52203e6a51628587ceec6896)
 2017-06-23 21:40:05 +01:00
Akira Ajisaka
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:21:12 +09:00
Naganarasimha
ac058c75a4 YARN-6719. Fix findbugs warnings in SLSCapacityScheduler.java. Contributed by Akira Ajisaka. 2017-06-19 12:41:13 +05:30
Mingliang Liu
c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu 
(cherry picked from commit 325163f23f727e82379d4a385b73aa3a04a510f6)
 2017-06-09 13:07:35 -07:00
Mingliang Liu
de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan 
(cherry picked from commit df35e4cc3b0c67bcaa3d809c6cb5a0a2d82986af)
 2017-06-07 13:55:11 -07:00