Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,390 Commits 326 Branches 371 Tags
Commit Graph

4166 Commits

Author SHA1 Message Date
Jim Brennan
f31c122ec9 MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 
(cherry picked from commit 4d323cd47c0787c1bb7592ed56acd4aced3e11e7)
 2021-02-26 21:51:47 +00:00
Jonathan Hung
1d76a8e73f YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
(cherry picked from commit be6e99963ded94adf6f447ff53f2ba66b99120ca)
(cherry picked from commit 6863a5bb8ace591de3374102920bba916dbebfda)
(cherry picked from commit eb6c08e423dd06bf37ff44665ffb98c97e26ad08)
 2021-02-25 15:47:36 -08:00
Jim Brennan
4ed7b80b19 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 20:30:42 +00:00
Jim Brennan
7363e6eee1 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit e6f5dbbe7f556e3525293814ad2239e4f04d88ad)
 2021-02-16 17:30:08 +00:00
Jim Brennan
d0562d6cd0 YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) Contributed by Masatake Iwasaki 2021-02-11 22:45:08 +00:00
Eric Badger
7b4034cd88 YARN-6977. Node information is not provided for non am containers in RM logs. (Suma Shivaprasad via wangda) 
Change-Id: I0c44d09a560446dee2ba68c2b9ae69fce0ec1d3e
(cherry picked from commit 8a42e922fad613f3cf1cc6cb0f3fa72546a9cc56)
(cherry picked from commit 958e8c0e257216c82f68fee726e5280a919da94a)
 2021-02-08 20:04:56 +00:00
Eric Badger
ea3c3d9022 YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba3f82802e2ebf39d44fe6c30037583c)
 2021-01-15 19:53:02 +00:00
Eric Badger
b80ff6ea57 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b73c0f52a7d5af2495355f4a0799344)
 2021-01-15 19:18:58 +00:00
Eric Badger
f239da08af YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230fa7adbb4a041c9a33bea2ef5a36fd3)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeResourceMonitorImpl.java
 2020-12-21 23:42:18 +00:00
Zhankun Tang
406957f5e8 YARN-9785. Fix DominantResourceCalculator when one resource is zero. Contributed by Bibin A Chundatt, Sunil Govindan, Bilwa S T. 
(cherry picked from commit fff4fbc9576d393a57489f3cd40770ec882f25dc)
 2020-12-10 18:05:30 +00:00
kevinzhao1661
2fe36b0038
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398949acfd0b6406abdaeb17456a25c37)
 2020-11-30 11:19:24 +09:00
Jim Brennan
6a58e6293f YARN-8558. NM recovery level db not cleaned up properly on container finish. Contributed by Bibin A Chundatt. 2020-11-17 17:36:22 +00:00
Jonathan Hung
6f436a6776 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:45:34 -07:00
Eric Badger
c4b42fa1ae YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.
 2020-10-16 19:29:04 +00:00
Jim Brennan
4bb0124c69 YARN-9667. Container-executor.c duplicates messages to stdout. Contributed by Peter Bacsko 2020-10-12 15:41:33 +00:00
Eric Badger
3335fcfb05 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by 
Ahmed Hussein.
 2020-10-09 20:41:58 +00:00
Adam Antal
1c0fe2eb20 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-07 16:46:22 +02:00
Jim Brennan
0bf270d2ed YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit ecf91638a8d1f546d90e92d076f32cb5681ce944)
 2020-10-06 18:46:08 +00:00
Wangda Tan
4885ea61fe Updated timeline reader to use AuthenticationFilter 
Change-Id: I961771589180c1eb377d36c37a79aa23754effbf
(cherry picked from commit 837338788eb903d0e8bbb1230694782a707891be)
(cherry picked from commit ea18e70a74e811ffa48c7e18e68510dd37dda63d)
 2020-09-14 21:48:43 +09:00
Masatake Iwasaki
f4e0c14fe9 Preparing for 2.10.2 development 2020-09-13 14:33:36 +09:00
Wangda Tan
9a895a90d9 YARN-8210. AMRMClient logging on every heartbeat to track updation of AM RM token causes too many log lines to be generated in AM logs. (Suma Shivaprasad via wangda) 
Change-Id: I70edd6e301fd5e78d479e1882aedc9332a0827aa
(cherry picked from commit b1833d9ba2c078582161da45ac392dd5c361dcdf)
(cherry picked from commit f9add5fdd9add4a5e6154de8ba6d9a17d123ac70)
 2020-09-10 14:40:05 -07:00
Eric E Payne
e5bd8d2840 YARN-10177: Backport YARN-7307 to branch-2.10 Allow client/AM update supported resource types via YARN APIs 2020-09-04 18:23:08 +00:00
Masatake Iwasaki
ecf88b972f
YARN-10358. Fix findbugs warnings in hadoop-yarn-project on branch-2.10. (#2164) 2020-08-31 13:52:29 +09:00
Kihwal Lee
283def8bc5 MAPREDUCE-7069. Add ability to specify user environment variables individually. Contributed by Jim Brennan 
(cherry picked from commit 4571351cccf6d4977469d3d623cf045b06a5f5f0)
 2020-08-13 18:54:31 -05:00
Eric E Payne
21788f9fd4 YARN-8459. Improve Capacity Scheduler logs to debug invalid states. Contributed by Wangda Tan and Jim Brennan. 2020-08-10 20:52:44 +00:00
Jonathan Hung
865828ae63 YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:45:04 -07:00
Eric Badger
120cd9eae6 YARN-7677. Docker image cannot set HADOOP_CONF_DIR. Contributed by Jim Brennan. 2020-08-06 19:40:33 +00:00
Eric Badger
9bf554deae YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 647be0c0f69887afd7ad63aa1dfa6373a4b65ae6)
 2020-08-05 23:20:52 +00:00
Eric Badger
c84ae8f3d9 YARN-10363. TestRMAdminCLI.testHelp is failing in branch-2.10. Contributed by 
Bilwa S T.

(cherry picked from commit 87816e8a51e3f30916972aae23ac4b0fed7fbfff)
 2020-07-31 22:40:41 +00:00
Eric E Payne
898ee03181 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 20:02:22 +00:00
Jonathan Hung
50e68e67b6 YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:45:14 -07:00
Eric Badger
2fae1c5b05 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef0f0ad4210e4027f53877760fa606a5)
 2020-07-24 23:08:31 +00:00
Akira Ajisaka
8cd8b4152d YARN-6515. Fix warnings from Spotbugs in hadoop-yarn-server-nodemanager. Contributed by Naganarasimha G R. 
(cherry picked from commit 1a18d5e514d13aa3a88e9b6089394a27296d6bc3)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/ContainerLocalizer.java
 2020-07-21 22:02:41 +09:00
Akira Ajisaka
e53186954d HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65af5faf5117978f4f2cf6a24d17a2e76)
(cherry picked from commit 0d949d375e3e7602978f094f7ef7952783ab8421)
(cherry picked from commit 635067d03de05252be5a364c8839af557dc75c13)
 2020-07-20 13:39:00 -07:00
Eric Badger
a4b419cdf5 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.

(cherry picked from commit 09f1547697d0aa51380a0351df6d77f54af074a0)
 2020-07-14 18:33:56 +00:00
Eric E Payne
7190507aa2 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b7543d412f4fafe631b7ace289662d28c)
 2020-07-13 21:51:32 +00:00
Eric E Payne
76fa956d3b YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit e6794f2fc4d763459ce13ffa8db4c064bcb076dc)
 2020-06-29 19:55:18 +00:00
Eric Badger
a11c606910 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan

(cherry picked from commit 73639319422b8443df943cf818b122c1ee2afcf6)
 2020-06-12 19:21:45 +00:00
Tao Yang
a91d4d612f YARN-8011. TestOpportunisticContainerAllocatorAMService#testContainerPromoteAndDemoteBeforeContainerStart fails intermittently. Contributed by Jim Brennan. 2020-06-12 10:56:24 +08:00
Eric E Payne
af324e3153 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db3022705635580c4d2f8b0abde109f954f)
(cherry picked from commit 034d458511692341636f0d2ef0574b7516c01ed6)
(cherry picked from commit 2e4892061a2ff1ae99ef5aacffd7b229dc3dac1b)
 2020-06-09 22:16:16 +00:00
Jonathan Hung
b9a0f99966 YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 2020-06-01 10:48:41 -07:00
Jonathan Hung
b8c88f6968 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c1226ec23841b04dd478e8b97f31abbeba)
(cherry picked from commit 564d3211f27c35bf3143a4bd1b3f8eeac2c6b01f)
(cherry picked from commit a7ea55e0156299ec8b80af1f3e681a3a7a31a3b4)
(cherry picked from commit b3e9aff5f7bcafea8b82f9b07719ff53d3ab2f12)
 2020-05-12 10:53:37 -07:00
Eric E Payne
d0b0fb6e04 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 2020-05-11 17:07:35 +00:00
Ahmed Hussein
0f0707fb0d YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-05-06 12:48:12 -05:00
Ahmed Hussein
712da57658 YARN-10255. fix intermittent failure TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic in branch-2.10 (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-04-30 17:50:55 -05:00
Jonathan Hung
27ad054696 YARN-8193. YARN RM hangs abruptly (stops allocating resources) when running successive applications. (Zian Chen via wangda) 2020-04-30 12:16:15 -07:00
Miklos Szegedi
71063434a6 YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian. 
(cherry picked from commit 925fdf761a513130e23c10575c7328c8681cff1d)
(cherry picked from commit 21948b048e9b91bb4ec646b7220b7f15deaf2203)
 2020-04-27 14:14:42 -07:00
Wangda Tan
34804679e3 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda) 
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669f197d43387d67dc00040d14cd00948)
 2020-04-19 12:51:56 +09:00
Jonathan Hung
ebce5c74e6 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit cd6c10de442fc3a53c9ed5521ac1d944a6ac95c6)
(cherry picked from commit 2c79865b951d0fdea7f576ce31e310b4074ecedd)
 2020-04-17 10:35:39 -07:00
Eric Yang
7945ed40f0 HADOOP-16361. Fixed TestSecureLogins#testValidKerberosName on branch-2. 
Contributed by Jim Brennan via eyang
 2020-04-16 12:12:21 -04:00