He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development
2020-10-16 14:52:41 +08:00
Steve Loughran
42c71a5790
HADOOP-15691. Add PathCapabilities to FileSystem and FileContext.
...
Contributed by Steve Loughran.
This complements the StreamCapabilities Interface by allowing applications to probe for a specific path on a specific instance of a FileSystem client
to offer a specific capability.
This is intended to allow applications to determine
* Whether a method is implemented before calling it and dealing with UnsupportedOperationException.
* Whether a specific feature is believed to be available in the remote store.
As well as a common set of capabilities defined in CommonPathCapabilities,
file systems are free to add their own capabilities, prefixed with
fs. + schema + .
The plan is to identify and document more capabilities -and for file systems which add new features, for a declaration of the availability of the feature to always be available.
Note
* The remote store is not expected to be checked for the feature;
It is more a check of client API and the client's configuration/knowledge
of the state of the remote system.
* Permissions are not checked.
2020-08-19 17:15:06 +01:00
Steve Loughran
429d5db3d9
HADOOP-16785. followup to abfs close() fix.
...
Adds one extra test to the ABFS close logic, to explicitly
verify that the close sequence of FilterOutputStream is
not going to fail.
This is just a due-diligence patch, but it helps ensure
that no regressions creep in in future.
Contributed by Steve Loughran.
Change-Id: Ifd33a8c322d32513411405b15f50a1aebcfa6e48
2020-01-20 16:26:33 +00:00
Steve Loughran
e21cb8f96e
HADOOP-16785. Improve wasb and abfs resilience on double close() calls.
...
This hardens the wasb and abfs output streams' resilience to being invoked
in/after close().
wasb:
Explicity raise IOEs on operations invoked after close,
rather than implicitly raise NPEs.
This ensures that invocations which catch and swallow IOEs will perform as
expected.
abfs:
When rethrowing an IOException in the close() call, explicitly wrap it
with a new instance of the same subclass.
This is needed to handle failures in try-with-resources clauses, where
any exception in closed() is added as a suppressed exception to the one
thrown in the try {} clause
*and you cannot attach the same exception to itself*
Contributed by Steve Loughran.
Change-Id: Ic44b494ff5da332b47d6c198ceb67b965d34dd1b
2020-01-08 12:04:11 +00:00
Akira Ajisaka
7201384e1b
HADOOP-16771. Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0. Contributed by Andras Bokor.
...
(cherry picked from commit f777cd398f
)
2019-12-20 13:12:02 +09:00
Sneha Vijayarajan
aa9cd0a2d6
HADOOP-16660. ABFS: Make RetryCount in ExponentialRetryPolicy Configurable.
...
Contributed by Sneha Vijayarajan.
2019-12-08 21:32:13 -08:00
bilaharith
c225efe237
HADOOP-16455. ABFS: Implement FileSystem.access() method.
...
Contributed by Bilahari T H.
2019-12-08 21:32:02 -08:00
Jeetesh Mangwani
b1e748f45b
HADOOP-16612. Track Azure Blob File System client-perceived latency
...
Contributed by Jeetesh Mangwani.
This add the ability to track the end-to-end performance of ADLS Gen 2 REST APIs by measuring latency in the Hadoop ABFS driver.
The latency information is sent back to the ADLS Gen 2 REST API endpoints in the subsequent requests.
2019-12-08 21:31:51 -08:00
bilaharith
ffeb6d8ece
HADOOP-16587. Make ABFS AAD endpoints configurable.
...
Contributed by Bilahari T H.
This also addresses HADOOP-16498: AzureADAuthenticator cannot authenticate
in China.
Change-Id: I2441dd48b50b59b912b0242f7f5a4418cf94a87c
2019-12-08 21:31:39 -08:00
Sneha Vijayarajan
8b2c7e0c4d
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists
2019-12-08 21:31:24 -08:00
Sneha Vijayarajan
546db6428e
HADOOP-16548 : Disable Flush() over config
2019-12-08 21:31:08 -08:00
Andras Bokor
89e95370a4
HADOOP-16710. Testing_azure.md documentation is misleading.
...
Contributed by Andras Bokor.
Change-Id: Icf07a53145936953629c7dace2e9648b7b21588d
2019-11-17 17:06:10 +00:00
Da Zhou
fe96407451
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI
...
(cherry picked from commit 9a8edb0aed
)
2019-10-16 14:27:11 -07:00
Rohith Sharma K S
7d5bb2ebb7
Preparing for 3.2.2-SNAPSHOT development.
2019-09-07 08:52:08 +05:30
bilaharith
3b3c0c4b87
HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC.
...
Contributed by Bilahari T H
Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
2019-08-27 19:08:38 +00:00
Robert Levas
ce23e971b4
HADOOP-16340. ABFS driver continues to retry on IOException responses from REST operations.
...
Contributed by Robert Levas.
This makes the HttpException constructor protected rather than public, so it is possible
to implement custom subclasses of this exception -exceptions which will not be retried.
Change-Id: Ie8aaa23a707233c2db35948784908b6778ff3a8f
2019-08-27 19:08:29 +00:00
Da Zhou
a6d50a9054
HADOOP-16376. ABFS: Override access() to no-op.
...
Contributed by Da Zhou.
Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
2019-08-27 19:04:16 +00:00
Da Zhou
dd636127e9
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
...
Author: Da Zhou
2019-08-27 19:01:21 +00:00
Da Zhou
006ae258b3
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens.
...
Contributed by Da Zhou.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-08-27 19:01:21 +00:00
Da Zhou
330e450397
HADOOP-16315. ABFS: transform full UPN for named user in AclStatus
...
Contributed by Da Zhou
Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
2019-08-12 09:41:52 +08:00
Arun Singh
5f2d07af1b
HADOOP-16404. ABFS default blocksize change(256MB from 512MB)
...
Contributed by: Arun Singh
2019-07-19 20:34:28 -07:00
Masatake Iwasaki
b6718c754a
HADOOP-16401. ABFS: port Azure doc to 3.2 branch.
...
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
2019-07-10 17:16:43 +09:00
DadanielZ
9c8e40fbdb
HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata.
...
Author: Da Zhou
(cherry picked from commit ff27e8eabd
)
2019-06-07 18:09:38 +01:00
Da Zhou
bf0bb2470f
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
...
Contributed by Da Zhou.
(cherry picked from commit 1cef194a28
)
2019-06-07 18:09:38 +01:00
Vishwajeet Dusane
907a016142
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.
...
Contributed by Vishwajeet Dusane.
(cherry picked from commit 1edf1914ac
)
2019-06-07 18:09:37 +01:00
Shweta Yakkali
6b115966bc
HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
(cherry picked from commit bb8ad096e7
)
2019-06-07 18:09:37 +01:00
Shweta Yakkali
57c6060c3a
HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.).
...
Contributed by Shweta Yakkali.
(cherry picked from commit 13f0ee21f2
)
Change-Id: Ibd010d2e6adc15f53a9c5357482e57313bf84d2e
2019-06-07 18:09:37 +01:00
Da Zhou
3593b66693
HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI
...
(Contributed by Da Zhou via Daniel Templeton)
Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
(cherry picked from commit e374584479
)
2019-06-07 18:09:37 +01:00
Denes Gerencser
ede5cbd707
HADOOP-16174. Disable wildfly logs to the console.
...
Follow-on to HADOOP-15851.
Author: Denes Gerencser <dgerencser@cloudera.com>
(cherry picked from commit ddede7ae6f
)
2019-06-07 18:09:37 +01:00
Steve Loughran
96489069b0
HADOOP-15851. Disable wildfly logs to the console.
...
Contributed by Vishwajeet Dusane.
(cherry picked from commit ef9dc6c44c
)
2019-06-07 18:09:37 +01:00
Steve Loughran
baa8670256
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth.
...
Contributed by Da Zhou.
(cherry picked from commit bd50fa956b
)
2019-06-07 18:09:37 +01:00
Takanobu Asanuma
a9a3450560
HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:34:16 +09:00
Akira Ajisaka
855dc997d6
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:27:33 +09:00
Akira Ajisaka
df5d8f05d9
HADOOP-16227. Upgrade checkstyle to 8.19
...
(cherry picked from commit 4b4fef2f0e
)
2019-04-15 10:47:02 +09:00
Akira Ajisaka
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
...
(cherry picked from commit 8b6deebb1d
)
2019-04-03 19:36:17 +09:00
Da Zhou
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ad
)
2019-03-08 13:53:44 +00:00
Da Zhou
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
(cherry picked from commit 3988e75ca3
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:47:58 +00:00
Da Zhou
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
(cherry picked from commit 1f1655028e
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:44:46 +00:00
Da Zhou
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS.
...
Contributed by Shweta Yakkali.
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e
)
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 10:39:37 +00:00
Steve Loughran
685a41f449
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
2019-02-21 13:39:37 +00:00
Masatake Iwasaki
dc9c3ce30b
HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki.
...
(cherry picked from commit aa3ad36605
)
2019-02-20 22:17:18 +09:00
Da Zhou
84ce0f1bfa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
(cherry picked from commit ba9efe06fa
)
2019-02-05 14:41:15 +00:00
Da Zhou
29de303e0a
HADOOP-16048. ABFS: Fix Date format parser.
...
Contributed by Da Zhou.
(cherry picked from commit 00ad9e23e8
)
2019-01-22 16:41:33 +00:00
Da Zhou
1d4390e16b
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
...
Contributed by Da Zhou.
(cherry picked from commit 30863c5ae3
)
2019-01-14 19:45:30 +00:00
Da Zhou
8b5fbe7a12
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
...
Contributed by Da Zhou.
2019-01-11 11:12:39 +00:00
Da Zhou
9cb6000c8a
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
...
Contributed by Da Zhou.
(cherry picked from commit 852701f793
)
2019-01-10 12:08:27 +00:00
Da Zhou
6c2500d7ca
HADOOP-15662. Better exception handling of DNS errors.
...
Contributed by Da Zhou.
(cherry picked from commit 7211269142
)
2019-01-10 12:03:48 +00:00
Da Zhou
f7de630e85
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
...
Contributed by Da Zhou.
(cherry picked from commit e8d1900369
)
2019-01-10 11:59:29 +00:00
Da Zhou
f122ae7279
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
...
Contributed by Da Zhou.
(cherry picked from commit 346c0c8aff
)
2018-12-17 11:18:12 +00:00
Da Zhou
d09dbcc8fb
HADOOP-15972 ABFS: reduce list page size to to 500.
...
Contributed by Da Zhou.
2018-12-17 11:08:17 +00:00