103 Commits

Author SHA1 Message Date
Jitendra Pandey
554c3cd21d HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:35:40 -07:00
Jitendra Pandey
ab95086ed1 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-25 08:31:35 -07:00
Jitendra Pandey
23b920cd7a HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 11:37:36 -07:00
Jitendra Pandey
b9f2effaf6 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 16:25:05 -07:00
Steve Loughran
df69b13b47
HADOOP-14658. branch-2 compilation is broken in hadoop-azure
Contributed by Sunil G.

Signed-off-by: Steve Loughran <stevel@apache.org>
2017-07-13 18:25:38 +01:00
Steve Loughran
3dfd107070
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
Contributed by Varada Hemeswari
(fix for java 7 compilation)
2017-07-12 11:07:22 +01:00
Steve Loughran
1e69e52603
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
Contributed by Varada Hemeswari
2017-07-12 10:35:59 +01:00
Steve Loughran
41e83b2ca2
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
Contributed by Thomas Marquardt

(cherry picked from commit d670c3a4da7dd80dccf6c6308603bb3bb013b3b0)
2017-07-11 21:45:24 +01:00
Jitendra Pandey
d94b30cb03 HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-10 17:30:31 -07:00
Mingliang Liu
ba76897392 HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
(cherry picked from commit 990aa34de23c625163745ebc338483065d955bbe)
2017-06-28 14:20:44 -07:00
Mingliang Liu
ce57458ec5 HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
(cherry picked from commit 7e031c2c18b8812ec9f843ed3b4abe9e6d12bb28)
2017-06-27 17:32:56 -07:00
Mingliang Liu
1b1065a163 HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
(cherry picked from commit 686a634f01e454537d1a9d531330e60a60233ca4)
2017-06-27 16:49:48 -07:00
Steve Loughran
3a87591df8 HADOOP-14461 Azure: handle failure gracefully in case of missing account access key. Contributed by Mingliang Liu.
(cherry picked from commit 48f4a229a93b65df9e99df2505719dd3538f1188)
2017-06-26 15:22:44 -07:00
Steve Loughran
093eb0d361
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
Contributed by Thomas.

(cherry picked from commit c6bd73c6c5760c3a52203e6a51628587ceec6896)
2017-06-23 21:40:05 +01:00
Akira Ajisaka
8a77cf1ee4
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools. 2017-06-20 13:21:12 +09:00
Mingliang Liu
c953e0ce8b HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
(cherry picked from commit 325163f23f727e82379d4a385b73aa3a04a510f6)
2017-06-09 13:07:35 -07:00
Mingliang Liu
de4e4ec774 HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan
(cherry picked from commit df35e4cc3b0c67bcaa3d809c6cb5a0a2d82986af)
2017-06-07 13:55:11 -07:00
Mingliang Liu
e3a72c2b5c HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
(cherry picked from commit 536f057158c445a57049f6c392869ae2f0be4f24)
2017-06-06 11:13:42 -07:00
Mingliang Liu
bacfd1baae HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu
(cherry picked from commit 6b5285bbcb439944ba6c4701571ffbb00258d5a1)
2017-06-06 11:07:42 -07:00
Mingliang Liu
53e90a8064 HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan
(cherry picked from commit 5fd9742c83fbeae96bf0913bdcdf77fafbf15b2f)
2017-06-05 16:01:27 -07:00
Mingliang Liu
0b78db35fd HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu
(cherry picked from commit ece33208b89a6a06c1b362d10fab31343fe5df13)
2017-06-01 11:52:37 -07:00
Mingliang Liu
9d7e67ccf9 HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari
(cherry picked from commit 89bb8bfe582ba85566cede321b233bb642f1c675)
2017-05-26 17:59:14 -07:00
Mingliang Liu
f3015425d0 HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg
(cherry picked from commit b415c6fe743242acf1d1d3eb7ea7091d90d2c0d4)
2017-05-16 11:28:12 -07:00
Andrew Wang
18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Mingliang Liu
586126956b HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak
(cherry picked from commit eed731496f63537350206274af09d2118b116a17)
2017-05-10 15:05:54 -07:00
Mingliang Liu
e84588eb03 HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak
(cherry picked from commit 8b5f2c372e70999f3ee0a0bd685a494e06bc3652)
2017-04-26 13:51:58 -07:00
Mingliang Liu
03aa142e35 HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg
(cherry picked from commit 0cab57223e3f54be17a5f27cefdb6d1da1b073e5)
2017-04-12 16:09:20 -07:00
Mingliang Liu
8ee3e7e4b1 HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak
(cherry picked from commit 56ab02eed9b61e1c80605104dfc4c87fc6abac96)
2017-04-04 11:16:48 -07:00
Mingliang Liu
ce8cdc109b HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak
(cherry picked from commit 27a44b60c1a55cabbf1873bca62db4e321c901ca)
2017-04-04 11:04:42 -07:00
Mingliang Liu
22930b09a0 HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
(cherry picked from commit 8e15e240597f821968e14893eabfea39815de207)
2017-03-20 16:37:57 -07:00
Mingliang Liu
d0e26075b8 HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
(cherry picked from commit 28daaf0eb206d723d2baf0f9d91e43d98bb2fd26)
2017-03-07 14:57:41 -08:00
Mingliang Liu
ff050866bf HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
(cherry picked from commit 686823529be09bea2a6cecb3503ef722017475bc)
2017-03-06 17:35:06 -08:00
Mingliang Liu
8323e651a1 Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
This reverts commit b26870c58ec6f88c6a3722c5fc856bbbd60c5550.
2017-03-06 17:18:36 -08:00
Mingliang Liu
2d059b5a40 HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
(cherry picked from commit c571cda5c7d929477961dfff4176d7de4944d874)
2017-03-06 17:01:01 -08:00
Mingliang Liu
b26870c58e HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth 2017-03-03 17:09:05 -08:00
Mingliang Liu
b0f8de184a HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
(cherry picked from commit 82607fce39151fc6ba5bced738088e2bc176dc77)
2017-02-23 14:11:08 -08:00
Mingliang Liu
ec646b1a9d HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
(cherry picked from commit c6a5b689db573046819f0e9193cb042e6c7298f4)
2016-12-29 16:24:38 -08:00
Mingliang Liu
edac82737a HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
(cherry picked from commit e92a77099b91620cee84513cc879089907468075)
2016-12-22 20:33:51 -08:00
Mingliang Liu
867389247f HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
(cherry picked from commit f92913c35bfda0d565606f9fb9a301ddd4105fd8)
2016-12-15 12:38:31 -08:00
Mingliang Liu
89eaa94c59 HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
(cherry picked from commit 15dd1f3381069c5fdc6690e3ab1907a133ba14bf)
2016-12-05 12:06:35 -08:00
Mingliang Liu
4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
(cherry picked from commit 5af572b6443715b7a741296c1bd520a1840f9a7c)
2016-11-15 11:36:08 -08:00
Chris Nauroth
804a40720b HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss.
(cherry picked from commit ae4db2544346370404826d5b55b2678f5f92fe1f)
2016-08-18 10:45:03 -07:00
Chris Nauroth
287688d6ca HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka.
(cherry picked from commit 2ed58c40e5dcbf5c5303c00e85096085b1055f85)
2016-08-08 12:28:46 -07:00
Chris Nauroth
8ec3d702a3 HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth.
(cherry picked from commit 4ef1bc0bdb6b2c907431492e4c8395035fa96c84)
2016-08-02 22:32:37 -07:00
Chris Nauroth
59eec8248a HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg.
(cherry picked from commit b43de80031d1272e8a08ea5bd31027efe45e9d70)

Conflicts:
	hadoop-project/pom.xml
2016-07-27 15:52:07 -07:00
Steve Loughran
e29d7fb585 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:47 +01:00
Chris Nauroth
26a23eff82 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran.
(cherry picked from commit 843ee8d59d8bacbca0d87ccf0790772e39d16138)
2016-04-08 13:37:46 -07:00
Chris Nauroth
2e60387401 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade.
(cherry picked from commit c50aad0f854b74ede9668e35db314b0a93be81b2)
2016-03-04 22:57:56 -08:00
Chris Nauroth
bce33b5179 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty.
(cherry picked from commit f98dff329b1f94c9f53022baf0209fc1a7aaf7c2)
2016-03-01 13:18:57 -08:00
Chris Nauroth
2e761dabd8 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay.
(cherry picked from commit 7634d404b750eafa135a37fa275325d0398255fb)
2016-02-28 11:23:02 -08:00