Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,400 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

825 Commits

Author SHA1 Message Date
Jim Brennan f31c122ec9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit 4d323cd47c)
 2021-02-26 21:51:47 +00:00
Kengo Seki 5f969ced67
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a)
 2021-02-20 19:16:30 +09:00
yzhangal bac234b0a8
HADOOP-17338 Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2692)) 
Backported from commit ID 3d2193cd64

Contributed by Yongjun Zhang
 2021-02-09 12:04:14 +00:00
yzhangal ceea43115b
HADOOP-16767 Handle non-IO exceptions in reopen() (#2685) 
Contributed by Sergei Poganshev.

Catches Exception instead of IOException in closeStream() 
and so handle exceptions such as SdkClientException by 
aborting the wrapped stream. This will increase resilience
to failures, as any which occuring during stream closure
will be caught. Furthermore, because the
underlying HTTP connection is aborted, rather than closed,
it will not be recycled to cause problems on subsequent
operations.
 2021-02-08 16:03:43 +00:00
Chen Liang ac82334041 HADOOP-17336. Backport HADOOP-16005 NativeAzureFileSystem does not support setXAttr and HADOOP-16785. Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix to branch-2.10. Contributed by Sally Zuo. 2020-11-03 12:00:57 -08:00
Masatake Iwasaki f4e0c14fe9 Preparing for 2.10.2 development 2020-09-13 14:33:36 +09:00
Steve Loughran a75f2915f9 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: Ief23c39256b194abcc445231b5f7a018185ea9f6
 2020-09-03 18:29:30 +01:00
Masatake Iwasaki a9f82a11df
HADOOP-17202. Fix findbugs warnings in hadoop-tools on branch-2.10. (#2214) 2020-08-18 13:58:35 +09:00
Akira Ajisaka 2dffc1d8cb HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 
(cherry picked from commit 2ba9903932)
 2020-08-11 14:19:35 +09:00
Sneha Vijayarajan a6b0892539
HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan. 2020-06-26 02:12:22 +00:00
Thomas Marquardt 0d4f9c7789
HADOOP-17089: WASB: Update azure-storage-java SDK 
Contributed by Thomas Marquardt

DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.

See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.

TESTS: A new test was added to validate the fix. All tests are passing:

$mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 231, Failures: 0, Errors: 0, Skipped: 4
Tests run: 588, Failures: 0, Errors: 0, Skipped: 12
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
Tests run: 407, Failures: 0, Errors: 0, Skipped: 34
Tests run: 151, Failures: 0, Errors: 0, Skipped: 19
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
 2020-06-26 02:10:58 +00:00
Ravi Prakash 2096113d06 HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed by Ravi Prakash. 
(cherry picked from commit 94ca52ae9e)
 2020-04-20 12:53:09 +09:00
Masatake Iwasaki df7cc1edb0 HADOOP-16739. Fix native build failure of hadoop-pipes on CentOS 8. 2020-04-20 12:50:44 +09:00
bilaharith 8f60971138 HADOOP-16933. Backport HADOOP-16890 and HADOOP-16825 to branch-2 
Hadoop 16890. Change in expiry calculation for MSI token provider. Contributed by Bilahari T H
(cherry picked from commit 0b931f36ec)

HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H
(cherry picked from commit 5944d28130)
 2020-04-16 11:31:51 -07:00
Sneha Vijayarajan 3bc2e26cd7 HADOOP-16778. ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2. 
Contributed by Sneha Vijayarajan.
 2020-04-16 11:27:46 -07:00
Bilahari T H 17a0bf8d78 HADOOP-16734. Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2. 
Contributed by Bilahari T H.
 2020-04-16 11:25:27 -07:00
Jonathan Hung 2c321c3c30
HADOOP-14918. Remove the Local Dynamo DB test option (branch-2.10) (#1864). Contributed by Jonathan Hung and Gabor Bota. 2020-03-23 16:22:26 +01:00
Weiwei Yang 5d46b06d1f HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71e)
 2020-03-08 22:03:32 -07:00
Akira Ajisaka 0c0be3b9f0
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf40)
 2020-01-21 18:05:46 +09:00
Mingliang Liu a8044c75ad
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:56:46 -08:00
Jonathan Hung d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Mingliang Liu cdb0cafff5
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:56:02 -08:00
Da Zhou 7f71473888
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aed)
 2019-10-16 22:04:36 -07:00
Bilahari T H 6cb9ee32e1
HADOOP-16652. Backport of HADOOP-16587: Make ABFS AAD endpoints configurable 2019-10-16 15:39:39 -07:00
Jonathan Hung 108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Sneha Vijayarajan 27d7d09fa4
HADOOP-16630 : Backport of Hadoop-16548 : Disable Flush() over config 2019-10-09 16:50:06 -07:00
Sneha Vijayarajan 2ccb1683ba
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists 2019-10-09 16:23:25 -07:00
Jonathan Hung 493b36279b YARN-9806. TestNMSimulator#testNMSimulator fails in branch-2 2019-09-03 11:06:51 -07:00
Da Zhou 6cb74629ec HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-28 22:42:28 +00:00
bilaharith 1376f2ed3d HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-28 22:42:28 +00:00
Arun Singh 8044415ea1 HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-08-28 22:42:28 +00:00
Da Zhou 42816bf698 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-28 22:42:28 +00:00
Aaron Fabbri 67fe5398fd Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
 2019-08-28 22:42:28 +00:00
DadanielZ 605c812749 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou 50953c586b HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou e5bb8498df HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-08-28 22:42:28 +00:00
Takanobu Asanuma e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Weiwei Yang 11fb89bd68 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb82)
 2019-05-14 14:09:12 -07:00
Yuan Gao 18f66a05cf
HADOOP-16205 Backport ABFS driver from trunk to branch 2.0: Fix build and test failures. 
Contributed by Yuan Gao.
 2019-04-25 18:36:14 +01:00
Vishwajeet Dusane 662a75d65b HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.
 2019-04-24 21:46:25 +00:00
Da Zhou 024a694684 HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-04-24 21:46:25 +00:00
Da Zhou c2d7429691 HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou 80c8b77cec HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-04-24 21:46:25 +00:00
Shweta Yakkali 0bbbb9c4b9 HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali. 
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
 2019-04-24 21:46:25 +00:00
Da Zhou f799c4644d HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou 00b990d0b6 HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou 34dbac8347 HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.

(cherry picked from commit 8b5fbe7a12)
 2019-04-24 21:46:25 +00:00
Da Zhou a0b99f42e5 HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou 4a1761067d HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00