Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,445 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

15445 Commits

Author SHA1 Message Date
Akira Ajisaka b87e080433
HADOOP-17204. Fix typo in Hadoop KMS document. Contributed by Xieming Li. 
(cherry picked from commit 141c62584b)
 2020-08-12 16:10:11 +09:00
Akira Ajisaka 2dffc1d8cb HADOOP-14400. Fix warnings from spotbugs in hadoop-tools. Contributed by Weiwei Yang. 
(cherry picked from commit 2ba9903932)
 2020-08-11 14:19:35 +09:00
Akira Ajisaka 7dedf344de MAPREDUCE-6881. Fix warnings from Spotbugs in hadoop-mapreduce. Contributed by Weiwei Yang. 
(cherry picked from commit 3ed3062fe3)
 2020-08-11 11:19:01 +09:00
Eric E Payne 21788f9fd4 YARN-8459. Improve Capacity Scheduler logs to debug invalid states. Contributed by Wangda Tan and Jim Brennan. 2020-08-10 20:52:44 +00:00
Jonathan Hung 865828ae63 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:45:04 -07:00
Eric Badger 120cd9eae6 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan. 2020-08-06 19:40:33 +00:00
Eric Badger 9bf554deae YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 647be0c0f6)
 2020-08-05 23:20:52 +00:00
sguggilam 4eb1f81876
HADOOP-17164 UGI loginUserFromKeytab doesn't set the last login time (#2194) 
Contributed by Sandeep Guggilam.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Steve Loughran <stevel@apache.org>
 2020-08-05 09:04:01 -07:00
Mingliang Liu 8d5821b9c3
HDFS-15499. Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusion (#2188) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2020-08-04 23:27:46 -07:00
Akira Ajisaka cd596a0fdb
HADOOP-17185. Set MAVEN_OPTS in Jenkinsfile (#2195) 2020-08-05 15:22:09 +09:00
Akira Ajisaka 39c118c802
HADOOP-14056. Update maven-javadoc-plugin to 2.10.4. 
(cherry picked from commit fad766e1d2)
 2020-08-05 12:04:09 +09:00
Mingliang Liu 0d4ddd9066
HADOOP-17184. Add --mvn-custom-repos parameter to yetus calls (#2193) 
(cherry picked from commit 49996f67f7)
 2020-08-05 10:53:21 +09:00
Eric Badger c84ae8f3d9 YARN-10363. TestRMAdminCLI.testHelp is failing in branch-2.10. Contributed by 
Bilwa S T.

(cherry picked from commit 87816e8a51)
 2020-07-31 22:40:41 +00:00
Eric E Payne 898ee03181 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 20:02:22 +00:00
S O'Donnell a1fe3eed9b HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-07-30 15:48:15 +01:00
Jonathan Hung 50e68e67b6 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:45:14 -07:00
Eric Badger 2fae1c5b05 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
 2020-07-24 23:08:31 +00:00
Akira Ajisaka 8cd8b4152d YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 
(cherry picked from commit 1a18d5e514)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
 2020-07-21 22:02:41 +09:00
Chen Liang 3044a13080 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e0)
(cherry picked from commit 370efc6d95)
(cherry picked from commit 1e33696bc8)
 2020-07-20 15:26:44 -07:00
Akira Ajisaka e53186954d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65a)
(cherry picked from commit 0d949d375e)
(cherry picked from commit 635067d03d)
 2020-07-20 13:39:00 -07:00
He Xiaoqiao 685b81f447 HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-17 00:17:17 +08:00
Erik Krogen f36d524a10 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:33:42 -07:00
Eric Badger a4b419cdf5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.

(cherry picked from commit 09f1547697)
 2020-07-14 18:33:56 +00:00
Eric E Payne 7190507aa2 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b75)
 2020-07-13 21:51:32 +00:00
He Xiaoqiao b2c44db79e Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell." 
This reverts commit 3ebd783dc7.
 2020-07-13 23:57:11 +08:00
He Xiaoqiao 3ebd783dc7 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:45:09 +08:00
Yiqun Lin c63068be5f HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin. 
(cherry picked from commit 2e43c28e01)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2020-07-10 15:16:51 +09:00
Chris Douglas d50f5b7df4 HADOOP-14354. SysInfoWindows is not thread safe. Contributed by Inigo Goiri 
(cherry picked from commit 689fb2fc66)
 2020-07-10 14:37:05 +09:00
Akira Ajisaka 3969327077 HADOOP-12940. Fix warnings from Spotbugs in hadoop-common. 
Conflicts:
	hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java
 2020-07-10 13:38:08 +09:00
Masatake Iwasaki 7257b17793 HADOOP-17120. Fix failure of docker image creation due to pip2 install error. (#2130) 
(cherry picked from commit 53cfe60904)
 2020-07-09 03:43:22 +00:00
Akira Ajisaka 9ac6c63c0d HADOOP-17090. Increase precommit job timeout from 5 hours to 20 hours. (#2111). Contributed by Akira Ajisaka. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-07-01 15:17:52 +05:30
Eric E Payne 76fa956d3b YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit e6794f2fc4)
 2020-06-29 19:55:18 +00:00
Masatake Iwasaki d755af71c9 HADOOP-17094. vulnerabilities reported in jackson and jackson-databind in branch-2.10. Contributed by Ahmed Hussein. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-06-29 08:57:49 +09:00
Akira Ajisaka fb694e87a6
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac33)
 2020-06-28 17:23:27 +09:00
Sneha Vijayarajan a6b0892539
HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan. 2020-06-26 02:12:22 +00:00
Thomas Marquardt 0d4f9c7789
HADOOP-17089: WASB: Update azure-storage-java SDK 
Contributed by Thomas Marquardt

DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.

See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.

TESTS: A new test was added to validate the fix. All tests are passing:

$mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 231, Failures: 0, Errors: 0, Skipped: 4
Tests run: 588, Failures: 0, Errors: 0, Skipped: 12
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
Tests run: 407, Failures: 0, Errors: 0, Skipped: 34
Tests run: 151, Failures: 0, Errors: 0, Skipped: 19
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
 2020-06-26 02:10:58 +00:00
Eric Yang e81002bcb1
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2020-06-15 11:15:03 +09:00
Kihwal Lee 17109758dd HDFS-12453. TestDataNodeHotSwapVolumes fails in trunk Jenkins runs. Contributed by Jim Brennan and Lei Xu. 2020-06-12 16:19:36 -05:00
Eric Badger a11c606910 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan

(cherry picked from commit 7363931942)
 2020-06-12 19:21:45 +00:00
Tao Yang a91d4d612f YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Jim Brennan. 2020-06-12 10:56:24 +08:00
Mingliang Liu 8e01bd317b
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-10 09:03:57 -07:00
Eric E Payne af324e3153 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db302)
(cherry picked from commit 034d458511)
(cherry picked from commit 2e4892061a)
 2020-06-09 22:16:16 +00:00
Toshihiro Suzuki d1f4c8f10f
HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2054) 
Contributed by Toshihiro Suzuki.
 2020-06-09 21:44:27 +01:00
Kihwal Lee 14ff6171a5 HADOOP-16255. Add ChecksumFs.rename(path, path, boolean). Contributed by Jungtaek Lim. 2020-06-08 17:31:16 -05:00
Eric Badger 6eb7c28293 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 568f185bf8.
 2020-06-08 20:03:02 +00:00
Mingliang Liu dcc6d63828
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:34:01 -07:00
Inigo Goiri b411fcee86 HDFS-14054. TestLeaseRecovery2: testHardLeaseRecoveryAfterNameNodeRestart2 and testHardLeaseRecoveryWithRenameAfterNameNodeRestart are flaky. Contributed by Zsolt Venczel. 
(cherry picked from commit 993c2140cc)
 2020-06-03 14:16:38 -07:00
Akira Ajisaka 75516620c8
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53b)
 2020-06-04 06:08:28 +09:00
Kihwal Lee 54c7d23e58 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c)
 2020-06-02 09:52:06 -05:00
Dhiraj e49573631c
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:08:32 -07:00