Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,904 Commits 326 Branches 371 Tags
Commit Graph

5235 Commits

Author SHA1 Message Date
Szilard Nemeth
ac30549b06 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-31 14:55:43 +02:00
9uapaw
3f14a9b21f YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212) 
Co-authored-by: Andras Gyori <gandras@cloudera.com>
(cherry picked from commit b4a524722a2466ede935aefd7b24a1e08fdbf076)
(cherry picked from commit f001595b47ae9f60cc2ca13676d9daf3baff9963)
(cherry picked from commit 1dc343726c0910141cc156b5e3f58d248a8a67c5)
 2021-07-28 17:15:21 +02:00
Akira Ajisaka
76e92af4c4
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3b025e4c0c844e8bfe8d9007af2be3b)
 2021-05-25 18:29:58 +09:00
lujiefsi
e0b84c45d1
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 15:44:00 +09:00
Peter Bacsko
4cd1596fe7 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-05-11 15:18:16 +02:00
Akira Ajisaka
badb6b3d0d YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc137239682e6189d1663b6b868757938)
 2021-05-07 00:26:46 +09:00
Eric Badger
c7ed492800 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 23:04:07 +00:00
Eric Badger
2af2c03560 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 18:01:41 +00:00
Eric Badger
9c3acb3018 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 
(cherry picked from commit bf0d058b698b15d1c51927b8fd8cf678ab95661a)
 2021-04-19 20:18:20 +00:00
Eric Badger
c70ac91d3f YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:41:28 +00:00
Akira Ajisaka
b26eac6e8a
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:39 +09:00
Akira Ajisaka
7b7ec5687e
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml

(cherry picked from commit 0b278502e25e7a5d52bc9c2873058413f77dd7fa)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml
 2021-03-27 00:40:24 +09:00
Jim Brennan
34e507cb8c YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:57:23 +00:00
Eric Payne
2439e07708 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7af122163789a731ced05f740562e45)
 2021-03-15 19:55:13 +00:00
Jim Brennan
a6c2fced53 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:47:43 +00:00
Jim Brennan
4d323cd47c MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 21:39:04 +00:00
Eric Badger
5862543c89 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec45ded55e8b7fdc29f8caf85547cca3)
 2021-02-26 19:36:55 +00:00
Jonathan Hung
eb6c08e423 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963ded94adf6f447ff53f2ba66b99120ca)
(cherry picked from commit 6863a5bb8ace591de3374102920bba916dbebfda)
 2021-02-25 15:35:01 -08:00
Jim Brennan
0a45b5460d [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit 3795f66364e39d9e7bafacd54d16bb60ab1d2188)
 2021-02-25 20:09:27 +00:00
Eric Badger
ea73b6ad5c YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3edb9076d3aac7775de8712a7685dc135)
 2021-02-22 19:15:51 +00:00
Eric Badger
9327039fb8 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b2bdf36589da1174d4d8ee29acc57f6)
 2021-02-19 23:44:28 +00:00
Jim Brennan
1b5de609ec [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit e6f5dbbe7f556e3525293814ad2239e4f04d88ad)
 2021-02-16 17:20:52 +00:00
Jim Brennan
d8d8546678 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 
(cherry picked from commit 2117ab6b7125bfcbdae907015001a7599e2f0c4c)
 2021-02-11 21:47:41 +00:00
Eric Badger
d5b8b2b50d YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba3f82802e2ebf39d44fe6c30037583c)
 2021-01-14 00:01:32 +00:00
Eric Payne
a4dea9de14 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit a093bd859da0d1f2834f873d4dbe2bbd2f00f2d2)
 2021-01-13 19:54:14 +00:00
Szilard Nemeth
770a56315e YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:51:06 +01:00
Eric Badger
52791d2005 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)
 2020-12-21 23:32:18 +00:00
Andrea Scarpino
c8178a9caa
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:18:12 +09:00
Eric Payne
73cf0bdba1 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit 1184284bafacbc2cbb31050e829886e2e3d8aeea)
 2020-12-02 17:27:32 +00:00
kevinzhao1661
a60aacc0c8
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:19:03 +09:00
Akira Ajisaka
4638ed94db
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7dfcfe76f928d06fcb29ca0cc4d778737)
 2020-11-24 15:23:34 +09:00
Ahmed Hussein
de9f778750 YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1fb2c92d3b7d2b2183e532b7637d93b9)
 2020-11-16 21:30:50 +00:00
Peter Bacsko
a86c91bc06 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-11-16 11:54:19 +01:00
Eric E Payne
c7b1bdba63 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bddeeb98e63f09882420505e3cfaedad1)
 2020-11-05 22:33:58 +00:00
Eric E Payne
052b9799c0 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 31154fdde5d81d0bf4593087507b926b73543a68)
 2020-11-02 17:59:58 +00:00
Jonathan Hung
42fab7897a YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
(cherry picked from commit f95c0824b01175590fe98e2fba1e5988694a52da)
(cherry picked from commit d0104e72c5fc83b2ede80bda4d178cfeee90539e)
 2020-10-28 10:42:02 -07:00
Eric Badger
50eba43c7f YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:01:50 +00:00
Jim Brennan
7bda43fbd5 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 
(cherry picked from commit e1c6804ace57ce21ba6acd1e81d73019f939e2b2)
 2020-10-08 21:13:39 +00:00
Jim Brennan
3a360b4bf1 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:33:16 +00:00
Jim Brennan
c36435c033 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8d1f546d90e92d076f32cb5681ce944)
 2020-10-06 18:38:23 +00:00
Adam Antal
b5e6ae97f4 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:53:00 +02:00
Jim Brennan
c83a3177dc YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 1efb54bd524d6961511712b43d3883c9e32a2b2a)
 2020-09-15 16:37:19 +00:00
Eric E Payne
1a8d9ddd14 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 17:07:09 +00:00
bibinchundatt
daff39a601 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80ad7864b332b60d5a01585fdf00848ee)
 2020-09-08 21:47:00 +00:00
Eric Badger
c3dbbd66b9 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
(cherry picked from commit 01ada576f3ecb551d99f7a61746e7563daf4ef09)
 2020-09-08 17:31:04 +00:00
Sunil G
2f7e584c21 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 7369fc31b456e2b90893212f499a30755fab5cc9.
 2020-08-20 19:13:03 +05:30
Sunil G
7369fc31b4 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f4421c162a505ba5e5b329e4be199878a7)
 2020-08-19 12:02:40 +05:30
Kihwal Lee
0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
 2020-08-13 19:02:03 -05:00
Jonathan Hung
c18a38bd07 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 17d18a2a3a6a2014257e65dd4874c54a84bb4f9e)
 2020-08-07 17:44:26 -07:00
Eric Badger
647be0c0f6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 23:14:27 +00:00