Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,799 Commits 326 Branches 371 Tags
Commit Graph

772 Commits

Author SHA1 Message Date
Da Zhou
06e6a677f3 HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
6f200a61b9 HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9cf1fb6e95 HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d91aa594283e447963fb2e407d4c6af5)
 2019-04-24 21:46:25 +00:00
Da Zhou
afa5705ff7 HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9e1d39bbae HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
9f220c41c4 HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-04-24 21:46:25 +00:00
Junhua Gu
d37098196d HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9e8f4f25faa352a06d142b75a029f0e)
 2019-04-24 21:46:25 +00:00
Daniel Templeton
8330f182dd HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI 
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
 2019-04-24 21:46:25 +00:00
Steve Loughran
885c726d38 HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
73bbada59a HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00
Steve Loughran
1b08e8d537 HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
e3033a5ba8 HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.
 2019-04-24 21:46:25 +00:00
Steve Loughran
d5e4c40760 HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00
Steve Loughran
0c5074a228 HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2019-04-24 21:46:25 +00:00
Steve Loughran
ee0fa0d15c HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
bde00d56df HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2019-04-24 21:46:25 +00:00
Steve Loughran
4f3cc1d417 HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2019-04-24 21:46:25 +00:00
Steve Loughran
6c7c8b83f7 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2019-04-24 21:46:25 +00:00
Sean Mackrory
920b77c8d7 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2019-04-24 21:46:25 +00:00
Steve Loughran
10e149d882 HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2019-04-24 21:46:25 +00:00
Steve Loughran
1efa1b73a9 HADOOP-15719. Fail-fast when using OAuth over http. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
8acab6c6aa HADOOP-15714. Tune abfs/wasb parallel and sequential test execution. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
26ce4e9301 HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min. 
Contributed by Da Zhou
 2019-04-24 21:46:25 +00:00
Steve Loughran
94d0de2eab HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch. 
Contributed by Steve Loughran.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
3599cd3dc9 HADOOP-15757. ABFS: remove dependency on common-codec Base64. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
263094f962 HADOOP-15753. ABFS: support path "abfs://mycluster/file/path" 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
e52f4a1ad2 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts. 
Contributed by Sean Mackrory.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
36b2bab888 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept. 
Contributed by Sneha Varma.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
a192d85a5f HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation 
for storage account with no namespace feature.

Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
8bb713e31a Fixing findbugs and license issues related to: 
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
305e750e82 HADOOP-15703. ABFS - Implement client-side throttling. 
Contributed by Sneha Varma and Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
12c621eda4 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
3bd089ddca HADOOP-15663. ABFS: Simplify configuration. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
92f3f9cf09 HADOOP-15692. ABFS: extensible support for custom oauth. 
Contributed by Junhua Gu and Rajeev Bansal.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
b59741f66b HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle. 
Contributed by Thomas Marquardt.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
08d3d50275 HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice. 
Contributed by Sean Mackrory.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
e727c87500 HADOOP-15661. ABFS: Add support for ACL. 
Contributed by Junhua Gu and Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
9d47230c8f HADOOP-15660. ABFS: Add support for OAuth 
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
92e37a0b73 HADOOP-15669. ABFS: Improve HTTPS Performance. 
Contributed by Vishwajeet Dusane.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
e320b64367 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2019-04-24 21:46:25 +00:00
Thomas Marquardt
5da4bb8892 HADOOP-15659. Code changes for bug fix and new tests. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Thomas Marquardt
b537d8c442 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests. 
Contributed by Steve Loughran and Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
37426c2da8 HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Steve Loughran
1b64b0f924 HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit. 
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
 2019-04-24 21:46:25 +00:00
Wanqiang Ji
313608e459
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
(cherry picked from commit 875435dc7d6a7cb7925a25ca3347a7ebaabec595)
(cherry picked from commit 6f0896e08742a792049137ed19700135f8594a42)
 2019-04-24 11:11:51 +01:00
Masatake Iwasaki
50e30bd30a HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 
(cherry picked from commit bbdbc7a9a158f36955c2253acb0edb14219ccb04)

Conflicts:
	hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm

(cherry picked from commit 7985d9b1ced4371f4cdc48ea74fbf120eab50309)

Conflicts:
	hadoop-tools/hadoop-distcp/src/site/markdown/DistCp.md.vm
 2019-04-12 12:35:32 +09:00
Weiwei Yang
5f3abefab3 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:15:50 +08:00
Steve Loughran
9b8044d00b
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589) 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
 2019-03-13 15:33:58 +00:00
Kai Xie
a49cb4465e
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-02-27 20:09:27 +00:00
Kai Xie
6d3e7a8570
HADOOP-16049. DistCp result has data and checksum mismatch when blocks per chunk > 0. 
Contributed by Kai Xie.
 2019-01-27 16:56:57 +00:00