Contributed by Thomas Marquardt
DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.
See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.
TESTS: A new test was added to validate the fix. All tests are passing:
$mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 231, Failures: 0, Errors: 0, Skipped: 4
Tests run: 588, Failures: 0, Errors: 0, Skipped: 12
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
Tests run: 407, Failures: 0, Errors: 0, Skipped: 34
Tests run: 151, Failures: 0, Errors: 0, Skipped: 19
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
Hadoop 16890. Change in expiry calculation for MSI token provider. Contributed by Bilahari T H
(cherry picked from commit 0b931f36ec)
HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H
(cherry picked from commit 5944d28130)
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
This reverts commit e4fca6aae4.
Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."
This reverts commit 2b635125fb.
(cherry picked from commit 1f425271a7)
The consolidated commits in this backport are as follows:
HADOOP-14787. AliyunOSS: Implement the `createNonRecursive` operator.
HADOOP-14649. Update aliyun-sdk-oss version to 2.8.1. (Genmao Yu via rchiang)
HADOOP-14194. Aliyun OSS should not use empty endpoint as default. Contributed by Genmao Yu
HADOOP-14466. Remove useless document from TestAliyunOSSFileSystemContract.java. Contributed by Chen Liang.
HADOOP-14458. Add missing imports to TestAliyunOSSFileSystemContract.java. Contributed by Mingliang Liu.
HADOOP-14192. AliyunOSS FileSystem contract test should implement getTestBaseDir(). Contributed by Mingliang Liu
HADOOP-14072. AliyunOSS: Failed to read from stream when seek beyond the download size. Contributed by Genmao Yu
HADOOP-13769. AliyunOSS: update oss sdk version. Contributed by Genmao Yu
HADOOP-14069. AliyunOSS: listStatus returns wrong file info. Contributed by Fei Hui
HADOOP-13768. AliyunOSS: handle the failure in the batch delete operation `deleteDirs`. Contributed by Genmao Yu
HADOOP-14065. AliyunOSS: oss directory filestatus should use meta time. Contributed by Fei Hui
HADOOP-14045. Aliyun OSS documentation missing from website. Contributed by Yiqun Lin.
HADOOP-13723. AliyunOSSInputStream#read() should update read bytes stat correctly. Contributed by Mingliang Liu
HADOOP-13624. Rename TestAliyunOSSContractDispCp. Contributed by Genmao Yu
HADOOP-13591. Unit test failure in TestOSSContractGetFileStatus and TestOSSContractRootDir. Contributed by Genmao Yu
HADOOP-13481. User documents for Aliyun OSS FileSystem. Contributed by Genmao Yu.
HADOOP-12756. Incorporate Aliyun OSS file system implementation. Contributed by Mingfei Shi and Lin Zhou
Akira Ajisaka
Masatake Iwasaki
Thomas Marquardt
bilaharith
Jonathan Hung
John Zhuge
Siyao Meng
Wei-Chiu Chuang
Ayush Saxena
Duo Zhang
stack
Yi Sheng
Takanobu Asanuma
Sneha Vijayarajan
Steve Loughran
Sunil G
Ayush Saxena
Sammi Chen
Arpit Agarwal
weiy
Rohith Sharma K S
Chris Douglas
Brahma Reddy Battula
Ray Chiang
Kihwal Lee
Kai Zheng
Arun Suresh
Subru Krishnan