Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,432 Commits 326 Branches 371 Tags
Commit Graph

15432 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Eric E Payne
898ee03181 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 20:02:22 +00:00
S O'Donnell
a1fe3eed9b HDFS-15313. Ensure inodes in active filesytem are not deleted during snapshot delete. Contributed by Shashikant Banerjee. 2020-07-30 15:48:15 +01:00
Jonathan Hung
50e68e67b6 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:45:14 -07:00
Eric Badger
2fae1c5b05 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef0f0ad4210e4027f53877760fa606a5)
 2020-07-24 23:08:31 +00:00
Akira Ajisaka
8cd8b4152d YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 
(cherry picked from commit 1a18d5e514d13aa3a88e9b6089394a27296d6bc3)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
 2020-07-21 22:02:41 +09:00
Chen Liang
3044a13080 HDFS-15404. ShellCommandFencer should expose info about source. Contributed by Chen Liang. 
(cherry picked from commit 3833c616e087518196bcb77ac2479c66a0b188d8)
(cherry picked from commit 370efc6d95fde8065dd5ba742446d0e9e570a5e9)
(cherry picked from commit 1e33696bc8fa74fbb00248c5de485e951a692814)
 2020-07-20 15:26:44 -07:00
Akira Ajisaka
e53186954d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
(cherry picked from commit 0d949d375e3e7602978f094f7ef7952783ab8421)
(cherry picked from commit 635067d03de05252be5a364c8839af557dc75c13)
 2020-07-20 13:39:00 -07:00
He Xiaoqiao
685b81f447 HDFS-14498. LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-17 00:17:17 +08:00
Erik Krogen
f36d524a10 HADOOP-17127. Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime. Contributed by Jim Brennan. 2020-07-15 08:33:42 -07:00
Eric Badger
a4b419cdf5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.

(cherry picked from commit 09f1547697d0aa51380a0351df6d77f54af074a0)
 2020-07-14 18:33:56 +00:00
Eric E Payne
7190507aa2 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b7543d412f4fafe631b7ace289662d28c)
 2020-07-13 21:51:32 +00:00
He Xiaoqiao
b2c44db79e Revert "HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell." 
This reverts commit 3ebd783dc7423bc2db405b930b758da86507bb54.
 2020-07-13 23:57:11 +08:00
He Xiaoqiao
3ebd783dc7 HDFS-14498 LeaseManager can loop forever on the file for which create has failed. Contributed by Stephen O'Donnell. 2020-07-13 14:45:09 +08:00
Yiqun Lin
c63068be5f HDFS-11696. Fix warnings from Spotbugs in hadoop-hdfs. Contributed by Yiqun Lin. 
(cherry picked from commit 2e43c28e01fe006210e71aab179527669f6412ed)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2020-07-10 15:16:51 +09:00
Chris Douglas
d50f5b7df4 HADOOP-14354. SysInfoWindows is not thread safe. Contributed by Inigo Goiri 
(cherry picked from commit 689fb2fc661f233367ecbfa26795ff4c01408f3e)
 2020-07-10 14:37:05 +09:00
Akira Ajisaka
3969327077 HADOOP-12940. Fix warnings from Spotbugs in hadoop-common. 
Conflicts:
	hadoop-common-project/hadoop-common/dev-support/findbugsExcludeFile.xml
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/erasurecode/ECSchema.java
 2020-07-10 13:38:08 +09:00
Masatake Iwasaki
7257b17793 HADOOP-17120. Fix failure of docker image creation due to pip2 install error. (#2130) 
(cherry picked from commit 53cfe6090419649dde18e3bf0d6ff757f76d716f)
 2020-07-09 03:43:22 +00:00
Akira Ajisaka
9ac6c63c0d HADOOP-17090. Increase precommit job timeout from 5 hours to 20 hours. (#2111). Contributed by Akira Ajisaka. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2020-07-01 15:17:52 +05:30
Eric E Payne
76fa956d3b YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit e6794f2fc4d763459ce13ffa8db4c064bcb076dc)
 2020-06-29 19:55:18 +00:00
Masatake Iwasaki
d755af71c9 HADOOP-17094. vulnerabilities reported in jackson and jackson-databind in branch-2.10. Contributed by Ahmed Hussein. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-06-29 08:57:49 +09:00
Akira Ajisaka
fb694e87a6
HDFS-15421. IBR leak causes standby NN to be stuck in safe mode. 
(cherry picked from commit c71ce7ac3370e220995bad0ae8b59d962c8d30a7)
 2020-06-28 17:23:27 +09:00
Sneha Vijayarajan
a6b0892539
HADOOP-16845: Disable ITestAbfsClient.testContinuationTokenHavingEqualSign due to ADLS Gen2 service bug. Contributed by Sneha Vijayarajan. 2020-06-26 02:12:22 +00:00
Thomas Marquardt
0d4f9c7789
HADOOP-17089: WASB: Update azure-storage-java SDK 
Contributed by Thomas Marquardt

DETAILS: WASB depends on the Azure Storage Java SDK. There is a concurrency
bug in the Azure Storage Java SDK that can cause the results of a list blobs
operation to appear empty. This causes the Filesystem listStatus and similar
APIs to return empty results. This has been seen in Spark work loads when jobs
use more than one executor core.

See Azure/azure-storage-java#546 for details on the bug in the Azure Storage SDK.

TESTS: A new test was added to validate the fix. All tests are passing:

$mvn -T 1C -Dparallel-tests=wasb -Dscale -DtestsThreadCount=8 clean verify
Tests run: 231, Failures: 0, Errors: 0, Skipped: 4
Tests run: 588, Failures: 0, Errors: 0, Skipped: 12
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0

$mvn -T 1C -Dparallel-tests=abfs -Dscale -DtestsThreadCount=8 clean verify
Tests run: 37, Failures: 0, Errors: 0, Skipped: 0
Tests run: 407, Failures: 0, Errors: 0, Skipped: 34
Tests run: 151, Failures: 0, Errors: 0, Skipped: 19
Tests run: 206, Failures: 0, Errors: 0, Skipped: 24
 2020-06-26 02:10:58 +00:00
Eric Yang
e81002bcb1
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 81d8a887b0406380e469c76ed2e41022a6372dd7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java
 2020-06-15 11:15:03 +09:00
Kihwal Lee
17109758dd HDFS-12453. TestDataNodeHotSwapVolumes fails in trunk Jenkins runs. Contributed by Jim Brennan and Lei Xu. 2020-06-12 16:19:36 -05:00
Eric Badger
a11c606910 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan

(cherry picked from commit 73639319422b8443df943cf818b122c1ee2afcf6)
 2020-06-12 19:21:45 +00:00
Tao Yang
a91d4d612f YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Jim Brennan. 2020-06-12 10:56:24 +08:00
Mingliang Liu
8e01bd317b
HADOOP-17059. ArrayIndexOfboundsException in ViewFileSystem#listStatus. Contributed by hemanthboyina 2020-06-10 09:03:57 -07:00
Eric E Payne
af324e3153 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db3022705635580c4d2f8b0abde109f954f)
(cherry picked from commit 034d458511692341636f0d2ef0574b7516c01ed6)
(cherry picked from commit 2e4892061a2ff1ae99ef5aacffd7b229dc3dac1b)
 2020-06-09 22:16:16 +00:00
Toshihiro Suzuki
d1f4c8f10f
HDFS-15386. ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directories (#2054) 
Contributed by Toshihiro Suzuki.
 2020-06-09 21:44:27 +01:00
Kihwal Lee
14ff6171a5 HADOOP-16255. Add ChecksumFs.rename(path, path, boolean). Contributed by Jungtaek Lim. 2020-06-08 17:31:16 -05:00
Eric Badger
6eb7c28293 Revert "MAPREDUCE-7277. IndexCache totalMemoryUsed differs from cache contents. Contributed by Jon Eagles (jeagles)." 
This reverts commit 568f185bf8c27887d3f4c2559b301275d4bfa486.
 2020-06-08 20:03:02 +00:00
Mingliang Liu
dcc6d63828
HADOOP-17047. TODO comment exist in trunk while related issue HADOOP-6223 is already fixed. Contributed by Rungroj Maipradit 2020-06-08 11:34:01 -07:00
Inigo Goiri
b411fcee86 HDFS-14054. TestLeaseRecovery2: testHardLeaseRecoveryAfterNameNodeRestart2 and testHardLeaseRecoveryWithRenameAfterNameNodeRestart are flaky. Contributed by Zsolt Venczel. 
(cherry picked from commit 993c2140ccfb4c6dfd09c0c686b85a4b58cb39a2)
 2020-06-03 14:16:38 -07:00
Akira Ajisaka
75516620c8
HADOOP-17062. Fix shelldocs path in Jenkinsfile (#2049) 
(cherry picked from commit 704409d53bf7ebf717a3c2e988ede80f623bbad3)
 2020-06-04 06:08:28 +09:00
Kihwal Lee
54c7d23e58 HDFS-14931. hdfs crypto commands limit column width. Contributed by Eric Badger 
(cherry picked from commit 9ef6ed9c1c83b9752e772ece7a716a33045752bf)
 2020-06-02 09:52:06 -05:00
Dhiraj
e49573631c
HADOOP-17052. NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort (#2036) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-06-01 11:08:32 -07:00
Jonathan Hung
b9a0f99966 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-06-01 10:48:41 -07:00
cnauroth
73ae3cee85 HDFS-9376. TestSeveralNameNodes fails occasionally. Contributed by Masatake Iwasaki. 
(cherry picked from commit 84a81477912644290173518d566b586305b85bf7)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2020-05-20 08:22:47 +09:00
Chen Liang
d0504cf74d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:10:05 -07:00
Masatake Iwasaki
e2c7c8d96f HADOOP-17045. Configuration javadoc describes supporting environment variables, but the feature is not available. 2020-05-18 11:11:46 +09:00
Yiqun Lin
e6374f031a HDFS-15264. Backport Datanode detection to branch-2.10. Contributed by Lisheng Sun. 2020-05-17 11:59:10 +08:00
Inigo Goiri
b5eb08a0d9 HDFS-15038. TestFsck testFsckListCorruptSnapshotFiles is failing in trunk. Contributed by hemanthboyina. 2020-05-14 11:26:31 -07:00
Virajith Jalaparti
cf12c1349d Revert "HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov" 
This reverts commit e6b7112c8837d918614fe3571d1fe077cc061aa1.
 2020-05-13 08:18:43 -07:00
Chris Douglas
e6b7112c88 HADOOP-12077. Provide a multi-URI replication Inode for ViewFs. Contributed by Gera Shegalov 
(cherry picked from commit 6d3d01d7ad5bff0e057761c288a2e0c9263b9ed3)
 2020-05-13 07:44:50 -07:00
Andrew Wang
ae2bc55bfc HADOOP-8299. ViewFileSystem link slash mount point crashes with IndexOutOfBoundsException. Contributed by Manoj Govindassamy. 
(cherry picked from commit 22ff0eff4d58ac0beda7a5a3ae0e5d108da14f7f)
 2020-05-13 06:56:15 -07:00
Akira Ajisaka
14bd270ca0
Revert "SPNEGO TLS verification" 
This reverts commit 28715b584ab25dedc600cc2d5d22866865026bf7.
 2020-05-13 17:17:07 +09:00
Virajith Jalaparti
284a1e58d7 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 20:41:19 -07:00
Jonathan Hung
b8c88f6968 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
(cherry picked from commit a7ea55e0156299ec8b80af1f3e681a3a7a31a3b4)
(cherry picked from commit b3e9aff5f7bcafea8b82f9b07719ff53d3ab2f12)
 2020-05-12 10:53:37 -07:00
Eric E Payne
d0b0fb6e04 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 2020-05-11 17:07:35 +00:00