Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,588 Commits 326 Branches 371 Tags
Commit Graph

5497 Commits

Author SHA1 Message Date
lujiefsi
bc97dd0d26
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790e5ad7d8e21fc3949cdd0f74d496b1b)
 2021-05-17 14:05:26 +09:00
Akira Ajisaka
a5bb211a2b YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 7cb7b8adc137239682e6189d1663b6b868757938)
 2021-05-07 00:25:43 +09:00
Eric Badger
951ce9b371 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6ac566a60336c0a28951f09ecda39f24)
 2021-04-23 22:55:25 +00:00
Eric Badger
b36810ee3f YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 
(cherry picked from commit 6cb90005a7d0651474883ac4e1b6961ef74fe513)
 2021-04-20 17:54:20 +00:00
Eric Badger
bf0d058b69 YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko. 2021-04-19 20:06:36 +00:00
Eric Badger
899cef53bd YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor. 
Contributed by Jim Brennan.
 2021-04-08 18:33:42 +00:00
Akira Ajisaka
d93f2c8cb8
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9f674e734e7f4c917ee2d51c77e527d)
 2021-03-30 17:48:20 +09:00
Takanobu Asanuma
9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfeee0035dd042357038b963843169d44c)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
Jim Brennan
0fcb2f28ce YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 174f3a96b10a0ab0fd8aed1b0f904ca5f0c3f268)
 2021-03-23 18:37:32 +00:00
Eric Badger
8bfa4cc6d8 YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 
(cherry picked from commit 49f89f1d3de66f3bb4db5952e8873432ba62f71a)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClusterMetrics.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCSAllocateCustomResource.java
 2021-03-17 18:50:00 +00:00
Eric Payne
5617bfa0d4 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7af122163789a731ced05f740562e45)
 2021-03-15 19:38:33 +00:00
Jim Brennan
4a98b518d6 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 
(cherry picked from commit 5f067cf0f304df673d709bbc6faa9a635651c404)
 2021-03-12 17:38:18 +00:00
Akira Ajisaka
0b278502e2
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753) 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 23b343aed17119b905dfa61ab1ce32ab554bfb57)

 Conflicts:
	dev-support/docker/Dockerfile
	hadoop-project/pom.xml

(cherry picked from commit de2904f123a7e3a56e4f8572f24b2e75ea606e80)

 Conflicts:
	dev-support/docker/Dockerfile_aarch64
	hadoop-cloud-storage-project/hadoop-cos/pom.xml
	hadoop-common-project/hadoop-registry/pom.xml
	hadoop-project/pom.xml
 2021-03-11 15:37:56 +09:00
Eric Badger
f30d1092bc YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim 
Brennan.
 2021-03-08 20:20:52 +00:00
Peter Bacsko
0b8bfc50c6 YARN-10672. All testcases in TestReservations are flaky. Contributed by Szilard Nemeth. 2021-03-08 14:32:13 +01:00
Peter Bacsko
0d032d7b68 YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu. 2021-03-08 14:22:14 +01:00
Ahmed Hussein
e4c638eb21 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit e04bcb3a061f7d89fb1353cd5ef6f550c049f36c)
 2021-02-26 20:38:13 +00:00
Eric Badger
49375c0b53 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec45ded55e8b7fdc29f8caf85547cca3)
 2021-02-26 19:17:24 +00:00
Jonathan Hung
6863a5bb8a YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963ded94adf6f447ff53f2ba66b99120ca)
 2021-02-25 15:26:57 -08:00
Jim Brennan
3795f66364 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 20:07:30 +00:00
Eric Badger
4ef5ed382b YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3edb9076d3aac7775de8712a7685dc135)
 2021-02-22 19:07:51 +00:00
Eric Badger
2c25f80a9c YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b2bdf36589da1174d4d8ee29acc57f6)
 2021-02-19 23:37:13 +00:00
Jim Brennan
e6f5dbbe7f [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 17:19:15 +00:00
Jim Brennan
2117ab6b71 RN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 2021-02-11 21:32:07 +00:00
Jim Brennan
6cc0eb3e30 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af4368cd4fb1d630f84d0d0c5cc11b224f)
 2021-02-05 17:52:16 +00:00
Eric Badger
d79f705a30 YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba3f82802e2ebf39d44fe6c30037583c)
 2021-01-13 23:53:16 +00:00
Eric Payne
a093bd859d YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 19:44:26 +00:00
He Xiaoqiao
a17793534c
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:46 +08:00
Szilard Nemeth
59795ec3d6 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:49:58 +01:00
Eric Badger
264dd67018 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)
 2020-12-21 23:22:43 +00:00
Andrea Scarpino
66e8f1ae09
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:17:48 +09:00
Eric Payne
1184284baf YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-02 17:22:49 +00:00
kevinzhao1661
7e89bd5e42
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:18:41 +09:00
Akira Ajisaka
752c890f9d
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7dfcfe76f928d06fcb29ca0cc4d778737)
 2020-11-24 15:23:13 +09:00
Ahmed Hussein
c107e75424 YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1fb2c92d3b7d2b2183e532b7637d93b9)
 2020-11-16 21:21:58 +00:00
Peter Bacsko
c5ae78b793 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-11-16 11:48:50 +01:00
Eric E Payne
d6a55caa9a YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bddeeb98e63f09882420505e3cfaedad1)
 2020-11-05 22:23:24 +00:00
Eric E Payne
31154fdde5 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:33:57 +00:00
Jim Brennan
63888afdd0 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 20:17:51 +00:00
Jonathan Hung
d0104e72c5 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
(cherry picked from commit f95c0824b01175590fe98e2fba1e5988694a52da)
 2020-10-28 10:38:58 -07:00
Eric Badger
4c61136616 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 18:51:53 +00:00
He Xiaoqiao
3274fd139d
Preparing for 3.2.3 development 2020-10-16 14:52:41 +08:00
Akira Ajisaka
a2c1fb7c8c
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 2020-10-16 01:04:45 +09:00
Jim Brennan
e1c6804ace YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 2020-10-08 21:09:30 +00:00
Jim Brennan
4ef9cf9d71 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:01:38 +00:00
Jim Brennan
ecf91638a8 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:36:51 +00:00
Adam Antal
b7420eb4b0 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:39:14 +02:00
Eric E Payne
947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
Jim Brennan
1efb54bd52 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-15 16:27:08 +00:00
Eric E Payne
5b14af6d09 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 16:46:28 +00:00