Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,968 Commits 326 Branches 371 Tags
Commit Graph

199 Commits

Author SHA1 Message Date
Thomas Marquardt
e52f4a1ad2 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. 
Contributed by Sean Mackrory.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
36b2bab888 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. 
Contributed by Sneha Varma.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
a192d85a5f HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation 
for storage account with no namespace feature.

Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
8bb713e31a Fixing findbugs and license issues related to: 
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
305e750e82 HADOOP-15703. ABFS - Implement client-side throttling. 
Contributed by Sneha Varma and Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
12c621eda4 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
3bd089ddca HADOOP-15663. ABFS: Simplify configuration. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
92f3f9cf09 HADOOP-15692. ABFS: extensible support for custom oauth. 
Contributed by Junhua Gu and Rajeev Bansal.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
b59741f66b HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. 
Contributed by Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
08d3d50275 HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. 
Contributed by Sean Mackrory.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
e727c87500 HADOOP-15661. ABFS: Add support for ACL. 
Contributed by Junhua Gu and Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
9d47230c8f HADOOP-15660. ABFS: Add support for OAuth 
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
92e37a0b73 HADOOP-15669. ABFS: Improve HTTPS Performance. 
Contributed by Vishwajeet Dusane.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
e320b64367 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2019-04-24 21:46:25 +00:00
Thomas Marquardt
5da4bb8892 HADOOP-15659. Code changes for bug fix and new tests. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
b537d8c442 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. 
Contributed by Steve Loughran and Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
37426c2da8 HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
1b64b0f924 HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit. 
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
 2019-04-24 21:46:25 +00:00
Wanqiang Ji
313608e459
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
(cherry picked from commit 875435dc7d6a7cb7925a25ca3347a7ebaabec595)
(cherry picked from commit 6f0896e08742a792049137ed19700135f8594a42)
 2019-04-24 11:11:51 +01:00
Steve Loughran
4e342f603f
HADOOP-15547/ WASB: improve listStatus performance. 
Contributed by Thomas Marquardt.
 2018-09-04 16:32:22 +01:00
Chris Douglas
4789b8e9c9 HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 
(cherry picked from commit f34744603ee93e082e7ba148df1400af5ac7c30c)
(cherry picked from commit 7e655b642d0ed5b85e0e8c212e004d067318ed4c)
(cherry picked from commit d1dcc39222b6d1d8ba10f38a3f2fb69e4d6548b3)
 2018-06-17 23:14:12 -07:00
Steve Loughran
0b5d0c3740 HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks. 
Contributed by Esfandiar Manii.

(cherry picked from commit d901be679554eb6b323f3bc6e8de267d85dd2e06)
(cherry picked from commit baac7c2b285454d71d0371505fb7a3403a548176)
(cherry picked from commit df338f2e1a19ed9c0c5c13f7d4aad08f9836de9f)
 2018-06-11 15:34:54 -07:00
Steve Loughran
e2d32bd9c5 HADOOP-15478. WASB: hflush() and hsync() regression. 
Contributed by Thomas Marquardt.

(cherry picked from commit ba842847c94d31d3f737226d954c566b5d88656b)
 2018-05-21 11:13:57 +01:00
Steve Loughran
67468651b1 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication. 
Contributed by Thomas Marquardt

(cherry picked from commit 5b11b9fd413470e134ecdc7c50468f8c7b39fa50)
 2018-05-10 10:33:03 -07:00
Chris Douglas
cc0a791794 HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao 
(cherry picked from commit 081c3501885c543bb1f159929d456d1ba2e3650c)
(cherry picked from commit ac16e8f4d3957c64bc2afddb9e58bd2f511f6c6e)
 2018-03-28 12:04:20 -07:00
Chris Douglas
762125b864 Backport HADOOP-13514 (surefire upgrade) to branch-2 2018-02-26 16:32:06 -08:00
Steve Loughran
d2ceef0f20 HADOOP-15086. NativeAzureFileSystem file rename is not atomic. 
Contributed by Thomas Marquardt
(Backported to branch-2 via /HADOOP-15156)

(cherry picked from commit a35267b47abf62d2ab32bc6860d2080284323042)
 2018-01-08 15:13:50 +00:00
Xiao Chen
c9d48352c2 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-12-07 21:20:33 -08:00
Steve Loughran
5c37a0b841 HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0. 
Contributed by Rajesh Balamohan

(cherry picked from commit 0ea182d0faa35c726dcb37249d48786bfc8ca04c)
 2017-11-28 11:57:51 +00:00
Arun Suresh
71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Steve Loughran
3fbcf76324 HADOOP-14553. Add (parallelized) integration tests to hadoop-azure. 
Contributed by Steve Loughran.

(backport JIRA HADOOP-14870)
 2017-10-27 12:55:12 +01:00
Steve Loughran
8902178d3e HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled. 
Contributed by Santhosh G Nayak

Cherry-picked from commit 9fcc3a1fc8cab873034f5c308ceb2d5671a954e8
 2017-10-16 16:06:41 +01:00
Steve Loughran
ccfd1a4a56 HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB. 
Contributed by Varada Hemeswari.

Cherry-picked from commit ref 3de574413c4d5554213d02bd0ad343ba82cf82aa
 2017-10-11 18:09:13 +01:00
Steve Loughran
34040e8d73 HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB 
Contributed by Kannapiran Srinivasan.
 2017-10-07 11:20:18 +01:00
Steve Loughran
fa6b8feb9e
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Final fixup of TestNativeAzureFileSystemAuthorization in branch-2 to match trunk's changes
Contributed by Sivaguru Sankaridurg
 2017-10-05 16:03:13 +01:00
Steve Loughran
1eecf8a97c
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB 
Contributed by Varada Hemeswari
 2017-10-02 14:28:59 +01:00
Steve Loughran
a641bcec01
HADOOP-14845. Azure wasb: getFileStatus not making any auth check. 
Contributed by Sivaguru Sankaridurg
 2017-09-29 11:16:31 +01:00
Steve Loughran
12a026badb
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory. 
Contributed by Esfandiar Manii.
 2017-09-19 11:02:10 +01:00
Steve Loughran
87af3f4991
HADOOP-14520. WASB: Block compaction for Azure Block Blobs. 
Contributed by Georgi Chalakov
 2017-09-11 14:15:58 +01:00
Steve Loughran
f382f66255
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS. 
Contributed by Sivaguru Sankaridurg
 2017-09-05 14:18:43 +01:00
Steve Loughran
25e9c29608
HADOOP-14802. Add support for using container saskeys for all accesses. 
Contributed by Sivaguru Sankaridurg

(cherry picked from commit 021974f4cb1eb8cf530dabc95039fed46e66a43d)

Conflicts:
	hadoop-common-project/hadoop-common/src/main/resources/core-default.xml
 2017-08-30 22:40:50 +01:00
Steve Loughran
c42e694ee5
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted. 
Contributed by Thomas Marquardt

(cherry picked from commit c6b4e656b76b68cc1d0dbcc15a5aa5ea23335b7b)
 2017-08-18 14:38:04 +01:00
Steve Loughran
d5680c08ed
HADOOP-14660. wasb: improve throughput by 34% when account limit exceeded. 
Contributed by Thomas Marquardt.
 2017-08-16 13:49:47 +01:00
Steve Loughran
96befde480
HADOOP-14183. Remove service loader config file for wasb fs. 
Contributed by Esfandiar Manii.

(cherry picked from commit 54356b1e8366a23fff1bb45601efffc743306efc)
 2017-08-10 16:49:51 +01:00
Steve Loughran
bbf8c4bacb
HADOOP-14715. TestWasbRemoteCallHelper failing. 
Contributed by Esfandiar Manii.
 2017-08-10 16:49:19 +01:00
Steve Loughran
7f7ab0302f
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek. 
Contributed by Thomas Marquardt

(cherry picked from commit d91b7a8451489f97bdde928cea774764155cfe03)
 2017-08-06 20:30:40 +01:00
Jitendra Pandey
554c3cd21d HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt. 2017-07-25 16:35:40 -07:00
Jitendra Pandey
ab95086ed1 HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov. 2017-07-25 08:31:35 -07:00
Jitendra Pandey
23b920cd7a HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg. 2017-07-19 11:37:36 -07:00
Jitendra Pandey
b9f2effaf6 HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak. 2017-07-17 16:25:05 -07:00