Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,493 Commits 326 Branches 371 Tags
Commit Graph

828 Commits

Author SHA1 Message Date
Akira Ajisaka
7d88a54d40 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83beb4f68e1ff18907589b062a0467e8)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml

(cherry picked from commit 94ca965e21067f1d957d21f19ae8e6abca9da971)
 2021-12-10 01:47:31 +09:00
Akira Ajisaka
eda934025b HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 3854b59bd1b06e9740cf33efc7844b66eec30ba4)
 2021-12-10 01:27:38 +09:00
Akira Ajisaka
f207d9e620
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin. Contributed by Ahmed Hussein. 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
 2021-03-29 09:44:28 +09:00
Jim Brennan
f31c122ec9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit 4d323cd47c0787c1bb7592ed56acd4aced3e11e7)
 2021-02-26 21:51:47 +00:00
Kengo Seki
5f969ced67
HADOOP-16764. Rewrite Python example codes using Python3 (#1762) 
(cherry picked from commit fd7de2b82a6b9d600b43909a0770a6b0afea899a)
 2021-02-20 19:16:30 +09:00
yzhangal
bac234b0a8
HADOOP-17338 Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc (#2692)) 
Backported from commit ID 3d2193cd648

Contributed by Yongjun Zhang
 2021-02-09 12:04:14 +00:00
yzhangal
ceea43115b
HADOOP-16767 Handle non-IO exceptions in reopen() (#2685) 
Contributed by Sergei Poganshev.

Catches Exception instead of IOException in closeStream() 
and so handle exceptions such as SdkClientException by 
aborting the wrapped stream. This will increase resilience
to failures, as any which occuring during stream closure
will be caught. Furthermore, because the
underlying HTTP connection is aborted, rather than closed,
it will not be recycled to cause problems on subsequent
operations.
 2021-02-08 16:03:43 +00:00
Chen Liang
ac82334041 HADOOP-17336. Backport HADOOP-16005 NativeAzureFileSystem does not support setXAttr and HADOOP-16785. Improve wasb and abfs resilience on double close() calls. followup to abfs close() fix to branch-2.10. Contributed by Sally Zuo. 2020-11-03 12:00:57 -08:00
Masatake Iwasaki
f4e0c14fe9 Preparing for 2.10.2 development 2020-09-13 14:33:36 +09:00
Steve Loughran
a75f2915f9 HADOOP-17199. S3A Directory Marker HADOOP-13230 backport 
This backports the listing-side changes of HADOOP-13230.

With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.

It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"

The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.

Read the documentation!

Change-Id: Ief23c39256b194abcc445231b5f7a018185ea9f6
 2020-09-03 18:29:30 +01:00
Masatake Iwasaki
a9f82a11df
HADOOP-17202. Fix findbugs warnings in hadoop-tools on branch-2.10. (#2214) 2020-08-18 13:58:35 +09:00
Akira Ajisaka
2dffc1d8cb HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 
(cherry picked from commit 2ba9903932e3c99afb0e6abb7cd3c5d29a554bd9)
 2020-08-11 14:19:35 +09:00
Sneha Vijayarajan
a6b0892539
HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan. 2020-06-26 02:12:22 +00:00
Thomas Marquardt
0d4f9c7789
HADOOP-17089: WASB: Update azure-storage-java SDK 
Contributed by Thomas Marquardt

DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.

See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.

TESTS: A new test was added to validate the fix. All tests are passing:

$mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 231, Failures: 0, Errors: 0, Skipped: 4
Tests run: 588, Failures: 0, Errors: 0, Skipped: 12
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
Tests run: 407, Failures: 0, Errors: 0, Skipped: 34
Tests run: 151, Failures: 0, Errors: 0, Skipped: 19
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
 2020-06-26 02:10:58 +00:00
Ravi Prakash
2096113d06 HADOOP-14597. Native compilation broken with OpenSSL-1.1.0. Contributed by Ravi Prakash. 
(cherry picked from commit 94ca52ae9ec0ae04854d726bf2ac1bc457b96a9c)
 2020-04-20 12:53:09 +09:00
Masatake Iwasaki
df7cc1edb0 HADOOP-16739. Fix native build failure of hadoop-pipes on CentOS 8. 2020-04-20 12:50:44 +09:00
bilaharith
8f60971138 HADOOP-16933. Backport HADOOP-16890 and HADOOP-16825 to branch-2 
Hadoop 16890. Change in expiry calculation for MSI token provider. Contributed by Bilahari T H
(cherry picked from commit 0b931f36ec83dc72729a9e84a0d313f471061c64)

HADOOP-16825: ITestAzureBlobFileSystemCheckAccess failing. Contributed by Bilahari T H
(cherry picked from commit 5944d28130925fe1452f545e96b5e44f064bc69e)
 2020-04-16 11:31:51 -07:00
Sneha Vijayarajan
3bc2e26cd7 HADOOP-16778. ABFS: Backport HADOOP-16660 ABFS: Make RetryCount in ExponentialRetryPolicy Configurable to Branch-2. 
Contributed by Sneha Vijayarajan.
 2020-04-16 11:27:46 -07:00
Bilahari T H
17a0bf8d78 HADOOP-16734. Backport HADOOP-16455- "ABFS: Implement FileSystem.access() method" to branch-2. 
Contributed by Bilahari T H.
 2020-04-16 11:25:27 -07:00
Jonathan Hung
2c321c3c30
HADOOP-14918. Remove the Local Dynamo DB test option (branch-2.10) (#1864). Contributed by Jonathan Hung and Gabor Bota. 2020-03-23 16:22:26 +01:00
Weiwei Yang
5d46b06d1f HADOOP-16840. AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket. Contributed by wujinhu. 
(cherry picked from commit 6dfe00c71eb3721e9be3fc42349a81c4b013ada1)
 2020-03-08 22:03:32 -07:00
Akira Ajisaka
0c0be3b9f0
HADOOP-16808. Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin. Contributed by Xieming Li. 
(cherry picked from commit f6d20daf404fab28b596171172afa4558facb504)
 2020-01-21 18:05:46 +09:00
Mingliang Liu
a8044c75ad
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml (#1753) 
Contributed by Mingliang Liu
 2019-12-11 11:56:46 -08:00
Jonathan Hung
d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Mingliang Liu
cdb0cafff5
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu 
This closes #1733
 2019-12-05 17:56:02 -08:00
Da Zhou
7f71473888
HADOOP-16640. WASB: Override getCanonicalServiceName() to return URI 
(cherry picked from commit 9a8edb0aeddd7787b2654f6e2a8465c325e048a2)
 2019-10-16 22:04:36 -07:00
Bilahari T H
6cb9ee32e1
HADOOP-16652. Backport of HADOOP-16587: Make ABFS AAD endpoints configurable 2019-10-16 15:39:39 -07:00
Jonathan Hung
108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Sneha Vijayarajan
27d7d09fa4
HADOOP-16630 : Backport of Hadoop-16548 : Disable Flush() over config 2019-10-09 16:50:06 -07:00
Sneha Vijayarajan
2ccb1683ba
HADOOP-16578 : Avoid FileSystem API calls when FileSystem already exists 2019-10-09 16:23:25 -07:00
Jonathan Hung
493b36279b YARN-9806. TestNMSimulator#testNMSimulator fails in branch-2 2019-09-03 11:06:51 -07:00
Da Zhou
6cb74629ec HADOOP-16315. ABFS: transform full UPN for named user in AclStatus 
Contributed by Da Zhou

Change-Id: Ibc78322415fcbeff89c06c8586c53f5695550290
 2019-08-28 22:42:28 +00:00
bilaharith
1376f2ed3d HADOOP-16479. ABFS FileStatus.getModificationTime returns localized time instead of UTC. 
Contributed by Bilahari T H

Change-Id: I532055baaadfd7c324710e4b25f60cdf0378bdc0
 2019-08-28 22:42:28 +00:00
Arun Singh
8044415ea1 HADOOP-16404. ABFS default blocksize change(256MB from 512MB) 
Contributed by: Arun Singh
 2019-08-28 22:42:28 +00:00
Da Zhou
42816bf698 HADOOP-16376. ABFS: Override access() to no-op. 
Contributed by Da Zhou.

Change-Id: Ia0024bba32250189a87eb6247808b2473c331ed0
 2019-08-28 22:42:28 +00:00
Aaron Fabbri
67fe5398fd Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest" 
This reverts commit 7c2d7c05a9a9cc981674f97cc3825e917a17b1f7.

Git Commit name and email were incorrect. Will re-commit.
 2019-08-28 22:42:28 +00:00
DadanielZ
605c812749 HADOOP-16251. ABFS: add FSMainOperationsBaseTest 
Author: Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou
50953c586b HADOOP-16269. ABFS: add listFileStatus with StartFrom. 
Author:    Da Zhou
 2019-08-28 22:42:28 +00:00
Da Zhou
e5bb8498df HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream. 
Contributed by Da Zhou.
 2019-08-28 22:42:28 +00:00
Takanobu Asanuma
e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka
cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Weiwei Yang
11fb89bd68 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 
(cherry picked from commit 2d8282bb8248e6984878626c4cdc7148aa2e7202)
 2019-05-14 14:09:12 -07:00
Yuan Gao
18f66a05cf
HADOOP-16205 Backport ABFS driver from trunk to branch 2.0: Fix build and test failures. 
Contributed by Yuan Gao.
 2019-04-25 18:36:14 +01:00
Vishwajeet Dusane
662a75d65b HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file. 
Contributed by Vishwajeet Dusane.
 2019-04-24 21:46:25 +00:00
Da Zhou
024a694684 HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
 2019-04-24 21:46:25 +00:00
Da Zhou
c2d7429691 HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
80c8b77cec HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.
 2019-04-24 21:46:25 +00:00
Shweta Yakkali
0bbbb9c4b9 HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali. 
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
 2019-04-24 21:46:25 +00:00
Da Zhou
f799c4644d HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00
Da Zhou
00b990d0b6 HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.
 2019-04-24 21:46:25 +00:00