Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,484 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4006 Commits

Author SHA1 Message Date
Eric Payne 9ee5265fb3 YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:48:06 +00:00
Akira Ajisaka 7d88a54d40 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml

(cherry picked from commit 94ca965e21)
 2021-12-10 01:47:31 +09:00
Akira Ajisaka eda934025b HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 3854b59bd1)
 2021-12-10 01:27:38 +09:00
Ashutosh Gupta 0b9ae87ee7 YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5a950b8900)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/src/test/java/org/apache/hadoop/yarn/server/timeline/TestRollingLevelDBTimelineStore.java
 2021-12-06 20:22:35 +09:00
Sunil G 29f81c6121 YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 215f2052fc)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSPreemptionThread.java
 2021-11-29 14:35:47 +09:00
Shubham Gupta 484cac36fd YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313) 
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
(cherry picked from commit e3cd627069)
 2021-11-29 14:21:56 +09:00
Eric Yang 0d88d832cd YARN-7266. Fixed deadlock in Timeline Server thread initialization. 
Contributed by Prabhu Joseph

(cherry picked from commit 7b42e0e32a)
 2021-11-29 12:43:05 +09:00
Ahmed Hussein de120b16ad YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-22 17:02:38 +00:00
Weiwei Yang 5f2047d491 YARN-8222. Fix potential NPE when gets RMApp from RM context. Contributed by Tao Yang. 
(cherry picked from commit 251f528814)
 2021-10-12 17:43:43 +00:00
Weiwei Yang bdd396b26d YARN-8546. Resource leak caused by a reserved container being released more than once under async scheduling. Contributed by Tao Yang. 
(cherry picked from commit 5be9f4a5d0)
 2021-10-08 16:08:45 +00:00
Weiwei Yang dc03afc7df YARN-8127. Resource leak when async scheduling is enabled. Contributed by Tao Yang. 
(cherry picked from commit 7eb783e263)
 2021-10-04 20:16:40 +00:00
Eric Badger 008bd8afc3 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-23 17:12:45 +00:00
Szilard Nemeth b196130c29
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 
(cherry picked from commit 79a46599f7)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/policy/TestFairOrderingPolicy.java

(cherry picked from commit 7aea2e1b5c)
 2021-09-01 13:16:30 +09:00
Abhishek Modi b79cd11fcd YARN-9744. RollingLevelDBTimelineStore.getEntityByTime fails with NPE. Contributed by Prabhu Joseph. 
(cherry picked from commit b4097b96a3)
 2021-08-18 17:54:10 -05:00
zhuqi-lucas 34acf9d4c8 YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:35:45 +08:00
Jim Brennan 577ed175f9 YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 15:21:02 +00:00
Artem Smotrakov 9241470dfd HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)

 Conflicts:
	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 15:06:37 +09:00
Jim Brennan f7bcc58e0f YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-29 14:22:39 +00:00
Jim Brennan 51be95e234 YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 
(cherry picked from commit 7c7d02edbd)
 2021-06-25 20:58:50 +00:00
Ahmed Hussein b58be3a12d
HADOOP-17769. Upgrade JUnit to 4.13.2. fixes TestBlockRecovery (#3131). Contributed by Ahmed Hussein. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-23 20:05:31 +05:30
Hideyuki Furue a07293f819
Fix container-executor 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9c7b8cf54e)
(cherry picked from commit f5568e15d0)
 2021-06-06 09:14:18 +09:00
Akira Ajisaka 76eea30bdd
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja. 
(cherry picked from commit 1cbcde04f3)
 2021-05-25 18:30:21 +09:00
lujiefsi 13a2e751e0
YARN-10555. Missing access check before getAppAttempts (#2608) 
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit d92a25b790)
 2021-05-17 19:47:03 +09:00
Eric Badger 7b3a6e96d9 YARN-10479. Can't remove all node labels after add node label without 
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy

(cherry picked from commit 6857a05d6a)
 2021-04-23 23:10:50 +00:00
bibinchundatt 94c1cf2ff2 YARN-9594. Fix missing break statement in ContainerScheduler#handle. Contributed by lujie. 
(cherry picked from commit 6d80b9bc3f)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/scheduler/ContainerScheduler.java

(cherry picked from commit 4a39165b41)
 2021-04-23 16:48:04 +00:00
Eric Badger eb176bb2af YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2021-04-20 00:31:08 +00:00
Ahmed Hussein 55645d78e1
YARN-10733. TimelineService Hbase tests failing with timeouts (#2899) 2021-04-14 13:44:38 -05:00
Akira Ajisaka d822bc3def
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 
(cherry picked from commit 8ad77a26c9)
 2021-03-30 17:49:01 +09:00
Eric Badger 616256b400 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.
 2021-03-29 18:11:58 +00:00
Akira Ajisaka f207d9e620
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin. Contributed by Ahmed Hussein. 
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
 2021-03-29 09:44:28 +09:00
Jim Brennan 33c4d4570d YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 
(cherry picked from commit 34e507cb8c)
 2021-03-23 19:05:57 +00:00
Wangda Tan c6033eeeef YARN-1151. Ability to configure auxiliary services from HDFS-based JAR files. (Xuan Gong via wangda) 
Change-Id: Ied37ff11e507fc86847753ba79486652c8fadfe9
(cherry picked from commit c962371430)
 2021-03-16 11:03:56 -07:00
Eric Payne d53ca0b887 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 
(cherry picked from commit aa4c17b9d7)
 2021-03-15 20:12:58 +00:00
Jim Brennan f31c122ec9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit 4d323cd47c)
 2021-02-26 21:51:47 +00:00
Jonathan Hung 1d76a8e73f YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963d)
(cherry picked from commit 6863a5bb8a)
(cherry picked from commit eb6c08e423)
 2021-02-25 15:47:36 -08:00
Jim Brennan 4ed7b80b19 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 20:30:42 +00:00
Jim Brennan 7363e6eee1 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit e6f5dbbe7f)
 2021-02-16 17:30:08 +00:00
Jim Brennan d0562d6cd0 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 2021-02-11 22:45:08 +00:00
Eric Badger 7b4034cd88 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
(cherry picked from commit 958e8c0e25)
 2021-02-08 20:04:56 +00:00
Eric Badger ea3c3d9022 YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba)
 2021-01-15 19:53:02 +00:00
Eric Badger b80ff6ea57 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b)
 2021-01-15 19:18:58 +00:00
Eric Badger f239da08af YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
 2020-12-21 23:42:18 +00:00
Zhankun Tang 406957f5e8 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 
(cherry picked from commit fff4fbc957)
 2020-12-10 18:05:30 +00:00
kevinzhao1661 2fe36b0038
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398)
 2020-11-30 11:19:24 +09:00
Jim Brennan 6a58e6293f YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt. 2020-11-17 17:36:22 +00:00
Jonathan Hung 6f436a6776 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:45:34 -07:00
Eric Badger c4b42fa1ae YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:29:04 +00:00
Jim Brennan 4bb0124c69 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 2020-10-12 15:41:33 +00:00
Eric Badger 3335fcfb05 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by 
Ahmed Hussein.
 2020-10-09 20:41:58 +00:00
Adam Antal 1c0fe2eb20 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-07 16:46:22 +02:00