121 Commits

Author SHA1 Message Date
Xiao Chen
52adaa43db HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge.
(cherry picked from commit c9d48352c232ac9e031559cb99a18ff249d2b6c1)
2017-12-29 13:57:55 -08:00
Arun Suresh
bb6a1aa5fa Preparing for 2.9.1 development 2017-11-02 22:49:38 -07:00
Steve Loughran
3fbcf76324 HADOOP-14553. Add (parallelized) integration tests to hadoop-azure.
Contributed by Steve Loughran.

(backport JIRA HADOOP-14870)
2017-10-27 12:55:12 +01:00
Steve Loughran
8902178d3e HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
Contributed by Santhosh G Nayak

Cherry-picked from commit 9fcc3a1fc8cab873034f5c308ceb2d5671a954e8
2017-10-16 16:06:41 +01:00
Steve Loughran
ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c4d5554213d02bd0ad343ba82cf82aa
2017-10-11 18:09:13 +01:00
Steve Loughran
34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
Contributed by Kannapiran Srinivasan.
2017-10-07 11:20:18 +01:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
2017-10-05 16:03:13 +01:00
Steve Loughran
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
Contributed by Varada Hemeswari
2017-10-02 14:28:59 +01:00
Steve Loughran
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
Contributed by Sivaguru Sankaridurg
2017-09-29 11:16:31 +01:00
Steve Loughran
12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory.
Contributed by Esfandiar Manii.
2017-09-19 11:02:10 +01:00
Steve Loughran
87af3f4991
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
Contributed by Georgi Chalakov
2017-09-11 14:15:58 +01:00
Steve Loughran
f382f66255
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS.
Contributed by Sivaguru Sankaridurg
2017-09-05 14:18:43 +01:00
Steve Loughran
25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses.
Contributed by Sivaguru Sankaridurg

(cherry picked from commit 021974f4cb1eb8cf530dabc95039fed46e66a43d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
2017-08-30 22:40:50 +01:00
Steve Loughran
c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted.
Contributed by Thomas Marquardt

(cherry picked from commit c6b4e656b76b68cc1d0dbcc15a5aa5ea23335b7b)
2017-08-18 14:38:04 +01:00
Steve Loughran
d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded.
Contributed by Thomas Marquardt.
2017-08-16 13:49:47 +01:00
Steve Loughran
96befde480
HADOOP-14183. Remove service loader config file for wasb fs.
Contributed by Esfandiar Manii.

(cherry picked from commit 54356b1e8366a23fff1bb45601efffc743306efc)
2017-08-10 16:49:51 +01:00
Steve Loughran
bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing.
Contributed by Esfandiar Manii.
2017-08-10 16:49:19 +01:00
Steve Loughran
7f7ab0302f
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek.
Contributed by Thomas Marquardt

(cherry picked from commit d91b7a8451489f97bdde928cea774764155cfe03)
2017-08-06 20:30:40 +01:00
Jitendra Pandey
554c3cd21d HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:35:40 -07:00
Jitendra Pandey
ab95086ed1 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-25 08:31:35 -07:00
Jitendra Pandey
23b920cd7a HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 11:37:36 -07:00
Jitendra Pandey
b9f2effaf6 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 16:25:05 -07:00
Steve Loughran
df69b13b47
HADOOP-14658. branch-2 compilation is broken in hadoop-azure
Contributed by Sunil G.

Signed-off-by: Steve Loughran <stevel@apache.org>
2017-07-13 18:25:38 +01:00
Steve Loughran
3dfd107070
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
Contributed by Varada Hemeswari
(fix for java 7 compilation)
2017-07-12 11:07:22 +01:00
Steve Loughran
1e69e52603
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
Contributed by Varada Hemeswari
2017-07-12 10:35:59 +01:00
Steve Loughran
41e83b2ca2
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
Contributed by Thomas Marquardt

(cherry picked from commit d670c3a4da7dd80dccf6c6308603bb3bb013b3b0)
2017-07-11 21:45:24 +01:00
Jitendra Pandey
d94b30cb03 HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-10 17:30:31 -07:00
Mingliang Liu
ba76897392 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
(cherry picked from commit 990aa34de23c625163745ebc338483065d955bbe)
2017-06-28 14:20:44 -07:00
Mingliang Liu
ce57458ec5 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
(cherry picked from commit 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)
2017-06-27 17:32:56 -07:00
Mingliang Liu
1b1065a163 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
(cherry picked from commit 686a634f01e454537d1a9d531330e60a60233ca4)
2017-06-27 16:49:48 -07:00
Steve Loughran
3a87591df8 HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu.
(cherry picked from commit 48f4a229a93b65df9e99df2505719dd3538f1188)
2017-06-26 15:22:44 -07:00
Steve Loughran
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
Contributed by Thomas.

(cherry picked from commit c6bd73c6c5760c3a52203e6a51628587ceec6896)
2017-06-23 21:40:05 +01:00
Akira Ajisaka
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:21:12 +09:00
Mingliang Liu
c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
(cherry picked from commit 325163f23f727e82379d4a385b73aa3a04a510f6)
2017-06-09 13:07:35 -07:00
Mingliang Liu
de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan
(cherry picked from commit df35e4cc3b0c67bcaa3d809c6cb5a0a2d82986af)
2017-06-07 13:55:11 -07:00
Mingliang Liu
e3a72c2b5c HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
(cherry picked from commit 536f057158c445a57049f6c392869ae2f0be4f24)
2017-06-06 11:13:42 -07:00
Mingliang Liu
bacfd1baae HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu
(cherry picked from commit 6b5285bbcb439944ba6c4701571ffbb00258d5a1)
2017-06-06 11:07:42 -07:00
Mingliang Liu
53e90a8064 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan
(cherry picked from commit 5fd9742c83fbeae96bf0913bdcdf77fafbf15b2f)
2017-06-05 16:01:27 -07:00
Mingliang Liu
0b78db35fd HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu
(cherry picked from commit ece33208b89a6a06c1b362d10fab31343fe5df13)
2017-06-01 11:52:37 -07:00
Mingliang Liu
9d7e67ccf9 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari
(cherry picked from commit 89bb8bfe582ba85566cede321b233bb642f1c675)
2017-05-26 17:59:14 -07:00
Mingliang Liu
f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg
(cherry picked from commit b415c6fe743242acf1d1d3eb7ea7091d90d2c0d4)
2017-05-16 11:28:12 -07:00
Andrew Wang
18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu
586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak
(cherry picked from commit eed731496f63537350206274af09d2118b116a17)
2017-05-10 15:05:54 -07:00
Mingliang Liu
e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak
(cherry picked from commit 8b5f2c372e70999f3ee0a0bd685a494e06bc3652)
2017-04-26 13:51:58 -07:00
Mingliang Liu
03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg
(cherry picked from commit 0cab57223e3f54be17a5f27cefdb6d1da1b073e5)
2017-04-12 16:09:20 -07:00
Mingliang Liu
8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak
(cherry picked from commit 56ab02eed9b61e1c80605104dfc4c87fc6abac96)
2017-04-04 11:16:48 -07:00
Mingliang Liu
ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak
(cherry picked from commit 27a44b60c1a55cabbf1873bca62db4e321c901ca)
2017-04-04 11:04:42 -07:00
Mingliang Liu
22930b09a0 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e240597f821968e14893eabfea39815de207)
2017-03-20 16:37:57 -07:00
Mingliang Liu
d0e26075b8 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
(cherry picked from commit 28daaf0eb206d723d2baf0f9d91e43d98bb2fd26)
2017-03-07 14:57:41 -08:00
Mingliang Liu
ff050866bf HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
(cherry picked from commit 686823529be09bea2a6cecb3503ef722017475bc)
2017-03-06 17:35:06 -08:00