Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,773 Commits 326 Branches 371 Tags
Commit Graph

5208 Commits

Author SHA1 Message Date
Andrea Scarpino
c8178a9caa
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b173472d9b71141142b31e5824357fb)
 2020-12-04 00:18:12 +09:00
Eric Payne
73cf0bdba1 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit 1184284bafacbc2cbb31050e829886e2e3d8aeea)
 2020-12-02 17:27:32 +00:00
kevinzhao1661
a60aacc0c8
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:19:03 +09:00
Akira Ajisaka
4638ed94db
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7dfcfe76f928d06fcb29ca0cc4d778737)
 2020-11-24 15:23:34 +09:00
Ahmed Hussein
de9f778750 YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1fb2c92d3b7d2b2183e532b7637d93b9)
 2020-11-16 21:30:50 +00:00
Peter Bacsko
a86c91bc06 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-11-16 11:54:19 +01:00
Eric E Payne
c7b1bdba63 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bddeeb98e63f09882420505e3cfaedad1)
 2020-11-05 22:33:58 +00:00
Eric E Payne
052b9799c0 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 31154fdde5d81d0bf4593087507b926b73543a68)
 2020-11-02 17:59:58 +00:00
Jonathan Hung
42fab7897a YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743f54f48cc2f31b4e5c8b6d4e5a5cfb8)
(cherry picked from commit f95c0824b01175590fe98e2fba1e5988694a52da)
(cherry picked from commit d0104e72c5fc83b2ede80bda4d178cfeee90539e)
 2020-10-28 10:42:02 -07:00
Eric Badger
50eba43c7f YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:01:50 +00:00
Jim Brennan
7bda43fbd5 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 
(cherry picked from commit e1c6804ace57ce21ba6acd1e81d73019f939e2b2)
 2020-10-08 21:13:39 +00:00
Jim Brennan
3a360b4bf1 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32bafdd3065e3c2f243d84ef79209838e9)
 2020-10-08 19:33:16 +00:00
Jim Brennan
c36435c033 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8d1f546d90e92d076f32cb5681ce944)
 2020-10-06 18:38:23 +00:00
Adam Antal
b5e6ae97f4 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760dffeeffb9cc739f734c0a91b81a0c9d0)
 2020-10-05 10:53:00 +02:00
Jim Brennan
c83a3177dc YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 1efb54bd524d6961511712b43d3883c9e32a2b2a)
 2020-09-15 16:37:19 +00:00
Eric E Payne
1a8d9ddd14 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed1721467aef7f2cd025d713273b12a6ca)
 2020-09-11 17:07:09 +00:00
bibinchundatt
daff39a601 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80ad7864b332b60d5a01585fdf00848ee)
 2020-09-08 21:47:00 +00:00
Eric Badger
c3dbbd66b9 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d20a17b522a0686ca3fd2d97e7e6838)
(cherry picked from commit 01ada576f3ecb551d99f7a61746e7563daf4ef09)
 2020-09-08 17:31:04 +00:00
Sunil G
2f7e584c21 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 7369fc31b456e2b90893212f499a30755fab5cc9.
 2020-08-20 19:13:03 +05:30
Sunil G
7369fc31b4 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f4421c162a505ba5e5b329e4be199878a7)
 2020-08-19 12:02:40 +05:30
Kihwal Lee
0c35e44184 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
 2020-08-13 19:02:03 -05:00
Jonathan Hung
c18a38bd07 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 17d18a2a3a6a2014257e65dd4874c54a84bb4f9e)
 2020-08-07 17:44:26 -07:00
Eric Badger
647be0c0f6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.
 2020-08-05 23:14:27 +00:00
Eric Badger
87816e8a51 YARN-10363. TestRMAdminCLI.testHelp is failing in branch-2.10. Contributed by 
Bilwa S T.
 2020-07-31 22:39:39 +00:00
Eric E Payne
8d8366166e YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166df48a47267dbc81d124ab78267e039)
(cherry picked from commit 863689ff9a34d9712541a06757c34499b6005f12)
 2020-07-30 17:27:35 +00:00
Jonathan Hung
b0edec2918 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 
(cherry picked from commit ffb920de2a60558a9863afb31b566b8c688a8263)
 2020-07-28 13:44:40 -07:00
Eric Badger
2255bcff87 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef0f0ad4210e4027f53877760fa606a5)
 2020-07-24 23:02:19 +00:00
Ayush Saxena
4592af898b HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 19:05:13 +05:30
Akira Ajisaka
635067d03d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
(cherry picked from commit 0d949d375e3e7602978f094f7ef7952783ab8421)
 2020-07-20 13:37:08 -07:00
Ahmed Hussein
b7b9bd32db HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71427a8a7476eaef64187a5643596552)
 2020-07-15 12:19:42 -05:00
Ahmed Hussein
34a6cbb5bb HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f9ef910654574f69591fcdc621523af)
(cherry picked from commit 43a865dc07437e0f767a902c4cfe6ca0472bf57f)
 2020-07-15 11:01:34 -05:00
Eric Badger
4cf5c282d0 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.

(cherry picked from commit 09f1547697d0aa51380a0351df6d77f54af074a0)
 2020-07-14 18:27:23 +00:00
Eric E Payne
ae4e6d1677 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b7543d412f4fafe631b7ace289662d28c)
 2020-07-13 21:37:37 +00:00
Masatake Iwasaki
4fa8055aa4 YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1. 2020-07-09 08:19:15 +09:00
Eric E Payne
b03e6fc44a YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit e6794f2fc4d763459ce13ffa8db4c064bcb076dc)
 2020-06-29 19:37:25 +00:00
Eric Badger
b669735124 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan

(cherry picked from commit 73639319422b8443df943cf818b122c1ee2afcf6)
 2020-06-12 19:09:37 +00:00
Szilard Nemeth
59b20a1ebf YARN-10295. CapacityScheduler NPE can cause apps to get stuck without resources. Contributed by Benjamin Teke 2020-06-10 18:15:08 +02:00
Szilard Nemeth
0c209f3f7d YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:02:05 +02:00
Eric E Payne
2e4892061a YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db3022705635580c4d2f8b0abde109f954f)
(cherry picked from commit 034d458511692341636f0d2ef0574b7516c01ed6)
 2020-06-09 21:34:20 +00:00
Jonathan Hung
aa19cb20ea YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb665761e997bcfc1dc00908f70b069fa2)
(cherry picked from commit 7a323a45aad07eed532d684d6dbe8436ba39c31c)
(cherry picked from commit a80595a6deb3124a3d6d99057e9d5298cd7237d8)
 2020-05-29 10:56:31 -07:00
Jonathan Hung
b3e9aff5f7 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
(cherry picked from commit a7ea55e0156299ec8b80af1f3e681a3a7a31a3b4)
 2020-05-12 10:53:23 -07:00
Szilard Nemeth
7e6c5e5ad2 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 
(cherry picked from commit 52e9ee39a12ce91b3a545603dcf1103518ad2920)
 2020-05-07 18:37:02 +00:00
Ahmed Hussein
4e29738f4c YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb40199aec0890b868b8d6bf2d7fe90abbf)
 2020-05-06 12:36:30 -05:00
Ahmed Hussein
e25ac17e2b YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d51fe781da97f07ac4dca8dd49c98e2)
 2020-05-04 11:11:14 -05:00
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Jonathan Hung
6dc1f7e154 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit 49ae9b2137dddaff1b579586e593a24da21a9e7d)
(cherry picked from commit d1af4e0fae842d46c29562444ebb16905f2caf0d)
 2020-04-17 10:36:26 -07:00
Jonathan Hung
6271a2852e YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
(cherry picked from commit e3a52804b03d646f15048c078f8c5292d5cbecfa)
(cherry picked from commit 54599b177c46ed511e096909bed0c4f17bca1fe0)
 2020-04-09 10:44:39 -07:00
Akira Ajisaka
d8033bfa96
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 
(cherry picked from commit e53d472bb07fc25d6cc5a13af1bd5ffdbf671164)
 2020-04-09 01:50:01 +09:00
Jonathan Hung
9c6dd8c83a YARN-10200. Add number of containers to RMAppManager summary 
(cherry picked from commit 2de0572cdc1c6fdbfaab108b169b2d5b0c077e86)
(cherry picked from commit 5d3fb0ebe9d3f3395320b82a76194ba6fad01e00)
 2020-03-25 10:33:18 -07:00
Eric E Payne
80e394e81b YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1723147430fc426161326d46698507f)
 2020-03-10 15:25:10 +00:00