5944d28130
HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing.
...
Contributed by Bilahari T H.
2020-02-06 18:48:00 +00:00
55f2421580
HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug.
...
Contributed by Sneha Vijayarajan.
2020-02-06 18:41:06 +00:00
978c487672
HADOOP-16826. ABFS: update abfs.md to include config keys for identity transformation
...
Contributed by Karthick Narendran
2020-01-23 20:35:57 -08:00
f206b736f0
HADOOP-16346. Stabilize S3A OpenSSL support.
...
Introduces `openssl` as an option for `fs.s3a.ssl.channel.mode`.
The new option is documented and marked as experimental.
For details on how to use this, consult the peformance document
in the s3a documentation.
This patch is the successor to HADOOP-16050 "S3A SSL connections
should use OpenSSL" -which was reverted because of
incompatibilities between the wildfly OpenSSL client and the AWS
HTTPS servers (HADOOP-16347). With the Wildfly release moved up
to 1.0.7.Final (HADOOP-16405) everything should now work.
Related issues:
* HADOOP-15669. ABFS: Improve HTTPS Performance
* HADOOP-16050: S3A SSL connections should use OpenSSL
* HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8
* HADOOP-16405: Upgrade Wildfly Openssl version to 1.0.7.Final
Contributed by Sahil Takiar
Change-Id: I80a4bc5051519f186b7383b2c1cea140be42444e
2020-01-21 16:37:51 +00:00
6a859d33aa
HADOOP-16785. followup to abfs close() fix.
...
Adds one extra test to the ABFS close logic, to explicitly
verify that the close sequence of FilterOutputStream is
not going to fail.
This is just a due-diligence patch, but it helps ensure
that no regressions creep in in future.
Contributed by Steve Loughran.
Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48
2020-01-20 16:23:41 +00:00
c36f09deb9
HADOOP-16005. NativeAzureFileSystem does not support setXAttr.
...
Contributed by Clemens Wolff.
2020-01-14 17:28:37 -08:00
17aa8f6764
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.
...
This hardens the wasb and abfs output streams' resilience to being invoked
in/after close().
wasb:
Explicity raise IOEs on operations invoked after close,
rather than implicitly raise NPEs.
This ensures that invocations which catch and swallow IOEs will perform as
expected.
abfs:
When rethrowing an IOException in the close() call, explicitly wrap it
with a new instance of the same subclass.
This is needed to handle failures in try-with-resources clauses, where
any exception in closed() is added as a suppressed exception to the one
thrown in the try {} clause
*and you cannot attach the same exception to itself*
Contributed by Steve Loughran.
Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b
2020-01-08 11:46:54 +00:00
d1f5976c00
HADOOP-16699. Add verbose TRACE logging to ABFS.
...
Contributed by Sneha Vijayarajan,
Change-Id: Ic616a10406e6e9f11616c9cc05d8630ebbedaf65
2020-01-07 18:05:47 +00:00
f777cd398f
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor.
2019-12-20 13:10:26 +09:00
82ad9b549f
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.
...
Contributed by Sneha Vijayarajan.
2019-11-27 15:10:21 -08:00
9e69628f55
HADOOP-16455. ABFS: Implement FileSystem.access() method.
...
Contributed by Bilahari T H.
2019-11-27 15:56:38 +00:00
de38045021
HADOOP-16687. ABFS: Fix testcase added for HADOOP-16138 for namespace enabled account. ( #1701 )
2019-11-21 11:24:12 +09:00
b033c681e4
HADOOP-16612. Track Azure Blob File System client-perceived latency
...
Contributed by Jeetesh Mangwani.
This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.
The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.
2019-11-19 09:00:24 -08:00
96c4520f89
HADOOP-16710. Testing_azure.md documentation is misleading.
...
Contributed by Andras Bokor.
Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
2019-11-17 17:04:29 +00:00
9a8edb0aed
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI
2019-10-16 13:14:15 -07:00
1a77a15fe4
HADOOP-16587. Make ABFS AAD endpoints configurable.
...
Contributed by Bilahari T H.
This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.
Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
2019-10-07 13:07:46 +01:00
770adc5d4a
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists
2019-10-01 17:38:11 -07:00
c0edc848a8
HADOOP-16548 : Disable Flush() over config
2019-09-28 20:39:42 -07:00
e346e3638c
HADOOP-15691 Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
Change-Id: I80bfebe94f4a8bdad8f3ac055495735b824968f5
2019-09-25 12:16:41 +01:00
aa664d7259
HADOOP-16138. hadoop fs mkdir / of nonexistent abfs container raises NPE ( #1302 ). Contributed by Gabor Bota.
...
Change-Id: I2f637865c871e400b95fe7ddaa24bf99fa192023
2019-09-23 13:29:01 +02:00
55ce454ce4
HADOOP-16371: Option to disable GCM for SSL connections when running on Java 8.
...
Contributed by Sahil Takiar.
This moves the SSLSocketFactoryEx class from hadoop-azure into hadoop-common
as the DelegatingSSLSocketFactory and binds the S3A connector to it so that
it can avoid using those HTTPS algorithms which are underperformant on Java 8.
Change-Id: Ie9e6ac24deac1aa05e136e08899620efa7d22abd
2019-09-17 11:32:03 +01:00
43a91f820a
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
...
Contributed by Da Zhou
Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
2019-08-09 12:38:13 +01:00
5840df86d7
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
...
Contributed by Bilahari T H
Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
2019-08-08 19:08:48 +01:00
0b45293abb
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
...
Contributed by: Arun Singh
2019-07-19 20:21:28 -07:00
450c070a8f
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations.
...
Contributed by Robert Levas.
This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.
Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
2019-06-19 17:43:14 +01:00
1da09bd9d5
HADOOP-16376. ABFS: Override access() to no-op.
...
Contributed by Da Zhou.
Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
2019-06-16 19:20:46 +01:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
ff27e8eabd
HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata.
...
Author: Da Zhou
2019-05-10 13:57:30 -07:00
5f1a34649f
Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
...
This reverts commit 7c2d7c05a9
2019-05-10 13:55:56 -07:00
7c2d7c05a9
HADOOP-16251. ABFS: add FSMainOperationsBaseTest
...
Author: Da Zhou
2019-05-10 12:46:03 -07:00
3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
...
Author: Da Zhou
2019-05-08 17:20:46 +01:00
1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
...
Contributed by Da Zhou.
2019-04-29 13:27:28 +01:00
4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19
2019-04-15 10:46:10 +09:00
8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
2019-04-03 19:35:02 +09:00
1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.
...
Contributed by Vishwajeet Dusane.
2019-03-18 13:18:08 +00:00
ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console.
...
Follow-on to HADOOP-15851.
Author: Denes Gerencser <dgerencser@cloudera.com>
2019-03-09 18:45:17 +00:00
bb8ad096e7
HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
2019-03-09 09:55:56 -08:00
e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
2019-03-08 13:53:08 +00:00
05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens.
...
Contributed by Da Zhou.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:02:34 +00:00
65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store.
...
Contributed by Steve Loughran.
2019-02-28 14:22:32 +00:00
3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
2019-02-26 15:37:24 +00:00
a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
2019-02-21 14:08:46 +00:00
aa3ad36605
HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki.
2019-02-20 22:00:57 +09:00
1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-07 21:58:21 +00:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
2019-02-05 14:39:45 +00:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
02eb91856e
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali.
...
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
2019-01-29 16:50:14 -08:00
00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser.
...
Contributed by Da Zhou.
2019-01-22 16:40:42 +00:00
30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
...
Contributed by Da Zhou.
2019-01-14 19:32:56 +00:00
dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
...
Contributed by Da Zhou.
(cherry picked from commit 8b5fbe7a12
2019-01-11 11:13:41 +00:00
852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
...
Contributed by Da Zhou.
2019-01-10 12:07:09 +00:00
7211269142
HADOOP-15662. Better exception handling of DNS errors.
...
Contributed by Da Zhou.
2019-01-10 12:02:58 +00:00
e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
...
Contributed by Da Zhou.
2019-01-10 11:58:39 +00:00
13f0ee21f2
HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali.
2019-01-02 09:33:09 -07:00
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
...
Contributed by Da Zhou.
2018-12-17 11:15:20 +00:00
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500.
...
Contributed by Da Zhou.
(cherry picked from commit d09dbcc8fb
2018-12-17 11:10:22 +00:00
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
...
Contributed by Da Zhou.
2018-12-17 11:04:40 +00:00
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
...
Contributed by Da Zhou.
2018-12-04 13:39:10 +00:00
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
...
Contributed by Da Zhou.
2018-11-30 10:12:41 +00:00
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename.
...
Contributed by Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit 1a3a4960d9
2018-11-27 18:13:07 +00:00
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2.
...
Contributed by Junhua Gu and Da Zhou.
2018-11-23 14:17:36 +00:00
a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
...
Contributed by Da Zhou.
2018-11-13 21:46:18 +00:00
9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console.
...
Author: Da Zhou <da.zhou@microsoft.com>
2018-11-09 11:06:23 +00:00
724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
...
Contributed by Junhua Gu.
(cherry picked from commit 66715005f9
2018-11-08 17:21:40 +00:00
e374584479
HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI
...
(Contributed by Da Zhou via Daniel Templeton)
Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
2018-10-24 14:18:35 -07:00
ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console.
...
Contributed by Vishwajeet Dusane.
2018-10-15 22:53:55 +01:00
bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth.
...
Contributed by Da Zhou.
2018-10-09 20:02:12 +01:00
273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call.
...
Contributed by Da Zhou
2018-10-05 11:29:03 +01:00
7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS.
...
Contributed by Da Zhou.
2018-10-03 12:52:53 +01:00
e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld.
...
Contributed by Abhishek Modi.
2018-10-02 18:57:20 +01:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore.
...
Contributed by Da Zhou
2018-10-02 11:42:52 +01:00
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization
...
Contributed by Yuan Gao.
2018-10-02 11:37:28 +01:00
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies.
...
Contributed by Da Zhou.
2018-09-25 20:58:32 +01:00
d5da9928c9
HADOOP-15723. ABFS: Ranger Support.
...
Contributed by Yuan Gao.
2018-09-25 19:13:10 +01:00
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read.
...
Contributed by Sneha Varma.
2018-09-21 11:06:24 +01:00
a5692c2da5
HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable.
...
Contributed by Steve Loughran.
2018-09-20 17:36:18 +01:00
8e831ba458
HADOOP-15773. Fixing checkstyle and other issues raised by Yetus.
2018-09-19 16:56:33 -06:00
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped.
...
Contributed by Steve Loughran
2018-09-19 13:57:39 +01:00
df2166a643
HADOOP-15719. Fail-fast when using OAuth over http.
...
Contributed by Da Zhou.
2018-09-18 12:20:52 +01:00
51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution.
...
Contributed by Da Zhou.
2018-09-18 12:09:25 +01:00
524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min.
...
Contributed by Da Zhou
2018-09-18 11:48:46 +01:00
1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
...
Contributed by Steve Loughran.
2018-09-18 10:56:56 +01:00
b4c23043d3
HADOOP-15757. ABFS: remove dependency on common-codec Base64.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
26211019c8
HADOOP-15753. ABFS: support path "abfs://mycluster/file/path"
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
e5593cbd83
HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts.
...
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
13c70e9ba3
HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept.
...
Contributed by Sneha Varma.
2018-09-17 19:54:01 +00:00
6801b30733
HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation
...
for storage account with no namespace feature.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
347a52a866
Fixing findbugs and license issues related to:
...
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
97f06b3fc7
HADOOP-15703. ABFS - Implement client-side throttling.
...
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
4410eacba7
HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
81dc4a995c
HADOOP-15663. ABFS: Simplify configuration.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
df57c6c3b1
HADOOP-15692. ABFS: extensible support for custom oauth.
...
Contributed by Junhua Gu and Rajeev Bansal.
2018-09-17 19:54:01 +00:00
dd2b22fa31
HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle.
...
Contributed by Thomas Marquardt.
2018-09-17 19:54:01 +00:00
6b6f8cc2be
HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice.
...
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
9c1e4e8139
HADOOP-15661. ABFS: Add support for ACL.
...
Contributed by Junhua Gu and Da Zhou.
2018-09-17 19:54:01 +00:00
9149b9703e
HADOOP-15660. ABFS: Add support for OAuth
...
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
2018-09-17 19:54:01 +00:00
d6a4f39bd5
HADOOP-15669. ABFS: Improve HTTPS Performance.
...
Contributed by Vishwajeet Dusane.
2018-09-17 19:54:01 +00:00
cc5cc60c41
Fixing issue due to commit 2b2399d6 after rebase onto trunk.
2018-09-17 19:54:01 +00:00
b54b0c1b67
HADOOP-15659. Code changes for bug fix and new tests.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
ce03a93f78
HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
...
Contributed by Steve Loughran and Da Zhou.
2018-09-17 19:54:01 +00:00
a271fd0eca
HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies.
...
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
f044deedbb
HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
...
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2018-09-17 19:54:01 +00:00
1eef0acfe5
HADOOP-15677. WASB: Add support for StreamCapabilities.
...
Contributed by Thomas Marquardt.
2018-09-10 14:24:41 +01:00
e8d138ca7c
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out.
...
Contributed by Andras Bokor.
2018-08-30 14:36:00 +01:00
45d9568aaa
HADOOP-15547/ WASB: improve listStatus performance.
...
Contributed by Thomas Marquardt.
(cherry picked from commit 749fff577e
2018-07-19 12:31:19 -07:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
7a3c6e9c3c
HADOOP-15550. Avoid static initialization of ObjectMappers
2018-06-25 15:36:45 -07:00
f34744603e
HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii
2018-06-17 23:12:18 -07:00
d901be6795
HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks.
...
Contributed by Esfandiar Manii.
2018-06-06 18:28:14 +01:00
ba842847c9
HADOOP-15478. WASB: hflush() and hsync() regression.
...
Contributed by Thomas Marquardt.
2018-05-21 11:12:34 +01:00
ba051b0686
HADOOP-15354. hadoop-aliyun & hadoop-azure modules to mark hadoop-common as provided
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-10 18:38:08 +09:00
5b11b9fd41
HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication.
...
Contributed by Thomas Marquardt
2018-05-07 11:54:08 +01:00
1312f9ae4c
HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.
2018-04-18 16:04:00 +08:00
081c350188
HADOOP-15320. Remove customized getFileBlockLocations for hadoop-azure and hadoop-azure-datalake. Contributed by Shanyu Zhao
2018-03-28 11:58:59 -07:00
5e013d50d1
HADOOP-15305. Replace FileUtils.writeStringToFile(File, String) with (File, String, Charset) to fix deprecation warnings.
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-03-15 16:12:07 +09:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
52babbb4a0
HADOOP-15086. NativeAzureFileSystem file rename is not atomic.
...
Contributed by Thomas Marquardt
2017-12-22 11:39:55 +00:00
0ea182d0fa
HADOOP-15042. Azure PageBlobInputStream.skip() can return negative value when numberOfPagesRemaining is 0.
...
Contributed by Rajesh Balamohan
2017-11-28 11:52:59 +00:00
6903cf096e
HADOOP-13514. Upgrade maven surefire plugin to 2.20.1
...
Signed-off-by: Allen Wittenauer <aw@apache.org>
2017-11-19 12:39:37 -08:00
bf6a660232
HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge.
2017-11-09 10:16:12 -08:00
9fcc3a1fc8
HADOOP-14935. Azure: POSIX permissions are taking effect in access() method even when authorization is enabled.
...
Contributed by Santhosh G Nayak
2017-10-16 16:01:47 +01:00
3de574413c
HADOOP-14913. Sticky bit implementation for rename() operation in Azure WASB.
...
Contributed by Varada Hemeswari.
2017-10-11 18:06:43 +01:00
572cdb5463
HADOOP-14899. Restrict Access to setPermission operation when authorization is enabled in WASB
...
Contributed by Kannapiran Srinivasan.
2017-10-06 17:43:38 +01:00
9288206cb3
HADOOP-14845. Azure wasb: getFileStatus not making any auth check.
...
Contributed by Sivaguru Sankaridurg
2017-10-05 15:05:23 +01:00
a530e7ab3b
HADOOP-14768. Honoring sticky bit during Deletion when authorization is enabled in WASB
...
Contributed by Varada Hemeswari
2017-09-28 19:52:56 +01:00
2d2d97fa7d
HADOOP-14553. Add (parallelized) integration tests to hadoop-azure
...
Contributed by Steve Loughran
2017-09-15 17:03:01 +01:00
13eda50003
HADOOP-14520. WASB: Block compaction for Azure Block Blobs.
...
Contributed by Georgi Chalakov
2017-09-07 18:35:03 +01:00
792eff9ea7
HADOOP-14820 Wasb mkdirs security checks inconsistent with HDFS.
...
Contributed by Sivaguru Sankaridurg
2017-09-05 14:16:57 +01:00
0d419c984f
Preparing for 3.1.0 development
2017-09-01 11:53:48 -07:00
9374f38820
HADOOP-14583. wasb throws an exception if you try to create a file and there's no parent directory
...
Contributed by Esfandiar Manii and Thomas Marquardt.
2017-08-29 19:03:44 +01:00
021974f4cb
HADOOP-14802. Add support for using container saskeys for all accesses.
...
Contributed by Sivaguru Sankaridurg
2017-08-29 19:02:43 +01:00
c6b4e656b7
HADOOP-14769. WASB: delete recursive should not fail if a file is deleted.
...
Contributed by Thomas Marquardt
2017-08-18 14:13:40 +01:00
645a8f2a4d
HADOOP-14726. Mark FileStatus::isDir as final
2017-08-14 21:57:20 -07:00
54356b1e83
HADOOP-14183. Remove service loader config file for wasb fs.
...
Contributed by Esfandiar Manii.
2017-08-10 16:46:33 +01:00
f4e1aa0508
HADOOP-14715. TestWasbRemoteCallHelper failing.
...
Contributed by Esfandiar Manii.
2017-08-08 23:37:47 +01:00
d91b7a8451
HADOOP-14722. Azure: BlockBlobInputStream position incorrect after seek.
...
Contributed by Thomas Marquardt
2017-08-06 20:19:23 +01:00
778d4edd9a
HADOOP-14660 wasb: improve throughput by 34% when account limit exceeded.
...
Contributed by Thomas Marquardt.
2017-08-01 21:36:28 +01:00
a92bf39e23
HADOOP-14680. Azure: IndexOutOfBoundsException in BlockBlobInputStream. Contributed by Thomas Marquardt.
2017-07-25 16:26:48 -07:00
f2921e51f0
HADOOP-14518. Customize User-Agent header sent in HTTP/HTTPS requests by WASB. Contributed by Georgi Chalakov.
2017-07-24 23:01:01 -07:00
2843c688bc
HADOOP-14642. wasb: add support for caching Authorization and SASKeys. Contributed by Sivaguru Sankaridurg.
2017-07-19 00:13:06 -07:00
b0e78ae085
HADOOP-14640. Azure: Support affinity for service running on localhost and reuse SPNEGO hadoop.auth cookie for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak.
2017-07-17 02:27:55 -07:00
7d272ea124
HADOOP-14581. Restrict setOwner to list of user when security is enabled in wasb.
...
Contributed by Varada Hemeswari
(cherry picked from commit 1e69e52603
2017-07-12 10:37:39 +01:00
d670c3a4da
HADOOP-14535 wasb: implement high-performance random access and seek of block blobs.
...
Contributed by Thomas Marquardt
2017-07-11 21:34:27 +01:00
38996fdcf0
HADOOP-14443. Azure: Support retry and client side failover for authorization, SASKey and delegation token generation. Contributed by Santhosh G Nayak
2017-06-30 16:53:48 -07:00
af2773f609
Updating version for 3.0.0-beta1 development
2017-06-29 17:57:40 -07:00
b08cc97396
HADOOP-14601. Azure: Reuse ObjectMapper. Contributed by Mingliang Liu
2017-06-29 09:22:00 -07:00
990aa34de2
HADOOP-14609. NPE in AzureNativeFileSystemStore.checkContainer() if StorageException lacks an error code. Contributed by Steve Loughran
2017-06-28 14:18:59 -07:00
7e031c2c18
HADOOP-14546. Azure: Concurrent I/O does not work when secure.mode is enabled. Contributed by Thomas
2017-06-27 17:32:07 -07:00
686a634f01
HADOOP-14573. regression: Azure tests which capture logs failing with move to SLF4J. Contributed by Steve Loughran
2017-06-27 16:48:47 -07:00
48f4a229a9
HADOOP-14461 Azure: handle failure gracefully in case of missing account access key.
...
Contributed by Mingliang Liu.
2017-06-26 13:39:47 +01:00
c6bd73c6c5
HADOOP-14547. [WASB] the configured retry policy is not used for all storage operations.
...
Contributed by Thomas.
2017-06-23 21:38:41 +01:00
3369540653
HADOOP-14296. Move logging APIs over to slf4j in hadoop-tools.
2017-06-20 13:18:26 +09:00
325163f23f
HADOOP-14512. WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename. Contributed by Duo Xu
2017-06-09 13:07:11 -07:00
df35e4cc3b
HADOOP-14500. Azure: TestFileSystemOperationExceptionHandling{,MultiThreaded} fails. Contributed by Rajesh Balamohan
2017-06-07 13:54:52 -07:00
536f057158
HADOOP-14491. Azure has messed doc structure. Contributed by Mingliang Liu
2017-06-06 11:09:28 -07:00
6b5285bbcb
HADOOP-14472. Azure: TestReadAndSeekPageBlobAfterWrite fails intermittently. Contributed by Mingliang Liu
2017-06-06 11:06:49 -07:00
5fd9742c83
HADOOP-14478. Optimize NativeAzureFsInputStream for positional reads. Contributed by Rajesh Balamohan
2017-06-05 15:56:43 -07:00
ece33208b8
HADOOP-14460. Azure: update doc for live and contract tests. Contributed by Mingliang Liu
2017-06-01 11:52:11 -07:00
89bb8bfe58
HADOOP-14442. Owner support for ranger-wasb integration. Contributed by Varada Hemeswari
2017-05-26 17:54:00 -07:00
16ad896d5c
Update maven version for 3.0.0-alpha4 development
2017-05-26 14:09:44 -07:00
6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu.
2017-05-25 15:36:44 +09:00
b415c6fe74
HADOOP-14416. Path starting with 'wasb:///' not resolved correctly while authorizing with WASB-Ranger. Contributed by Sivaguru Sankaridurg
2017-05-16 11:22:32 -07:00
0d5c8ed8e0
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor.
2017-05-11 16:37:32 -05:00
eed731496f
HADOOP-14361. Azure: NativeAzureFileSystem.getDelegationToken() call fails sometimes when invoked concurrently. Contributed by Santhosh G Nayak
2017-05-10 15:05:11 -07:00
2ba9903932
HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang.
2017-05-10 05:57:12 -05:00
54fd0e44b7
HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy.
2017-05-08 21:59:49 -07:00
8b5f2c372e
HADOOP-14351. Azure: RemoteWasbAuthorizerImpl and RemoteSASKeyGeneratorImpl should not use Kerberos interactive user cache. Contributed by Santhosh G Nayak
2017-04-26 13:47:18 -07:00
0cab57223e
HADOOP-14274. Azure: Simplify Ranger-WASB policy model. Contributed by Sivaguru Sankaridurg
2017-04-12 16:07:10 -07:00
56ab02eed9
HADOOP-14273. Azure: NativeAzureFileSystem should respect config for kerberosSupportEnabled flag. Contributed by Santhosh G Nayak
2017-04-04 11:10:28 -07:00
27a44b60c1
HADOOP-14272. Azure: WasbRemoteCallHelper should use String equals for comparison. Contributed by Santhosh G Nayak
2017-04-04 11:03:59 -07:00
6eba79232f
HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang
2017-04-03 20:13:14 -07:00
8e15e24059
HADOOP-13945. Azure: Add Kerberos and Delegation token support to WASB client. Contributed by Santhosh G Nayak
...
Change-Id: I39fb0a3b0491bf2160571366939a0502a0045429
2017-03-20 16:31:24 -07:00
28daaf0eb2
HADOOP-14150. Implement getHomeDirectory() method in NativeAzureFileSystem. Contributed by Santhosh G Nayak
2017-03-07 14:55:52 -08:00
686823529b
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-06 17:16:36 -08:00
52d7d5aa1a
Revert "HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth"
...
This reverts commit 6b7cd62b8c
2017-03-06 17:10:11 -08:00
c571cda5c7
HADOOP-14048. REDO operation of WASB#AtomicRename should create placeholder blob for destination folder. Contributed by NITIN VERMA
2017-03-06 17:00:13 -08:00
6b7cd62b8c
HADOOP-13930. Azure: Add Authorization support to WASB. Contributed by Sivaguru Sankaridurg and Dushyanth
2017-03-03 17:09:12 -08:00
82607fce39
HADOOP-14091. AbstractFileSystem implementaion for 'wasbs' scheme. Contributed Varada Hemeswari.
2017-02-23 13:48:44 -08:00
5d8b80ea9b
Preparing for 3.0.0-alpha3 development
2017-01-19 15:50:07 -08:00
c6a5b689db
HADOOP-13942. Build failure due to errors of javadoc build in hadoop-azure. Contributed by Kai Sasaki
2016-12-29 16:24:06 -08:00
e92a77099b
HADOOP-13863. Azure: Add a new SAS key mode for WASB. Contributed by Dushyanth
2016-12-22 20:15:18 -08:00
f92913c35b
HADOOP-13831. Correct check for error code to detect Azure Storage Throttling and provide retries. Contributed by Gaurav Kanade
2016-12-15 12:35:08 -08:00
15dd1f3381
HADOOP-13675. Bug in return value for delete() calls in WASB. Contributed by Dushyanth
2016-12-05 12:04:07 -08:00
c0b1a44f6c
HADOOP-13660. Upgrade commons-configuration version. Contributed by Sean Mackrory.
2016-11-17 22:48:35 -06:00
5af572b644
HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
2016-11-15 10:57:00 -08:00
20c4d8efab
HADOOP-13787. Azure testGlobStatusThrowsExceptionForUnreadableDir fails. Contributed by John Zhuge.
2016-11-03 11:16:07 -07:00
5877f20f9c
HADOOP-10075. Update jetty dependency to version 9 (rkanter)
2016-10-27 16:09:00 -07:00
2cc841f16e
HADOOP-13678 Update jackson from 1.9.13 to 2.x in hadoop-tools. Contributed by Akira Ajisaka.
2016-10-06 16:31:00 +01:00
8b57be108f
HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth.
...
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
(cherry picked from commit a65e159c59
2016-08-29 15:30:49 -07:00
ae4db25443
HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss.
2016-08-18 10:44:36 -07:00
2ed58c40e5
HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka.
2016-08-08 12:28:40 -07:00
4ef1bc0bdb
HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth.
2016-08-02 22:32:21 -07:00
b43de80031
HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg.
2016-07-27 15:50:38 -07:00
54fe17a607
HADOOP-9427. Use JUnit assumptions to skip platform-specific tests. Contributed by Gergely Novák.
2016-07-27 19:41:09 +09:00
bilaharith
Sneha Vijayarajan
Karthick Narendran
Sahil Takiar
Steve Loughran
Clemens Wolff
Sneha Vijayarajan
Akira Ajisaka
Jeetesh Mangwani
Andras Bokor
Da Zhou
bilaharith
Gabor Bota
Da Zhou
Arun Singh
Robert Levas
DadanielZ
Aaron Fabbri
Vishwajeet Dusane
Denes Gerencser
Shweta Yakkali
Steve Loughran
Masatake Iwasaki
Shweta Yakkali
Sean Mackrory
Junhua Gu
Daniel Templeton
Sunil G
Thomas Marquardt
Todd Lipcon
Chris Douglas
Steve Loughran
Yiqun Lin
fang zhenyi
Wangda Tan
John Zhuge
Andrew Wang
Jitendra Pandey
Mingliang Liu
Wei-Chiu Chuang
Xiao Chen
Robert Kanter
Xiaoyu Yao
Chris Nauroth