Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,611 Commits 326 Branches 371 Tags
Commit Graph

1924 Commits

Author SHA1 Message Date
Sunil G
d9b00cdd6b YARN-8404. Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down. Contributed by Rohith Sharma K S 
(cherry picked from commit 6307962b932e0ee69ba61f5796388c175d79195a)
 2018-06-13 16:10:24 +05:30
Weiwei Yang
ef105abb70 YARN-8394. Improve data locality documentation for Capacity Scheduler. Contributed by Weiwei Yang. 
(Cherry picked from commit 29024a62038c297f11e8992601f2522ffffc7da7)
 2018-06-13 13:57:35 +08:00
Rohith Sharma K S
96e7b7e8ae YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace06cfd2b3193a14cd455297f8f989617)
 2018-06-12 17:25:22 +05:30
Robert Kanter
d5e6d0d5f4 YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter) 
(cherry picked from commit 0cd145a44390bc1a01113dce4be4e629637c3e8a)
 2018-06-04 15:42:46 -07:00
Yongjun Zhang
f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Sunil G
521ada1a11 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne. 
(cherry picked from commit 7c343669baf660df3b70d58987d6e68aec54d6fa)
 2018-05-28 16:34:29 +05:30
Wangda Tan
a5a9c8cf0f YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda) 
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
(cherry picked from commit 6b96a73bb0f0ad1c877a062b19091e3e15a33ec4)
 2018-05-22 10:34:15 -05:00
Eric E Payne
1b6c662546 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdeef62945b592d5761666ad026b629c0b)
 2018-05-21 20:31:39 +00:00
Weiwei Yang
4e46dc764f YARN-7003. DRAINING state of queues is not recovered after RM restart. Contributed by Tao Yang. 2018-05-11 11:04:50 +08:00
bibinchundatt
c198550cc7 YARN-8201. Skip stacktrace of few exception from ClientRMService. Contributed by Bilwa S T. 
(cherry picked from commit cc0310a5266c8b8351f338f5fc8087a203c68cac)
 2018-05-10 09:32:45 +05:30
Weiwei Yang
a0b7abf278 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f676237290d18ddbbc9aec369267692)
 2018-05-07 13:59:50 +08:00
Rohith Sharma K S
317deaafc4 YARN-8217. RmAuthenticationFilterInitializer and TimelineAuthenticationFilterInitializer should use Configuration.getPropsWithPrefix instead of iterator. Contributed by Suma Shivaprasad. 
(cherry picked from commit ee2ce923a922bfc3e89ad6f0f6a25e776fe91ffb)
 2018-05-03 18:21:11 +05:30
Weiwei Yang
c2ed611885 YARN-8222. Fix potential NPE when gets RMApp from RM context. Contributed by Tao Yang. 
(Cherry picked from commit 251f528814c4a4647cac0af6effb9a73135db180)
 2018-05-02 18:07:34 +08:00
Weiwei Yang
f12c78120e YARN-8212. Pending backlog for async allocation threads should be configurable. Contributed by Tao Yang. 
(Cherry picked from commit 2d319e37937c1e20c6a7dc4477ef88defd1f8464)
 2018-05-01 10:19:08 +08:00
Rohith Sharma K S
8d3e6f99c1 YARN-8221. RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 
(cherry picked from commit ef3ecc308dbea41c6a88bd4d16739c7bbc10cdda)
 2018-04-27 22:59:48 +05:30
Sunil G
f1f3cbb4df YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218db59cab0b03bbba427e5e9ef5b3003c)
 2018-04-27 10:43:24 +05:30
Sunil G
62921db4b9 YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S. 
(cherry picked from commit 1634de0fc1430d86b7688d16259a81462fba482f)
 2018-04-27 10:28:41 +05:30
Wangda Tan
baa003035d YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda) 
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
(cherry picked from commit 3043a93d461fd8b9ccc2ff4b8d17e5430ed77615)
 2018-04-24 17:49:32 -07:00
Robert Kanter
101d9005dd HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter) 
(cherry picked from commit 7ab08a9c37a76edbe02d556fcfb2e637f45afc21)
 2018-04-23 15:53:53 -07:00
Jason Lowe
4bff2df137 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie 
(cherry picked from commit 766544c0b008da9e78bcea6285b2c478653df75a)
 2018-04-20 13:26:29 -05:00
Sunil G
070bee57a3 YARN-6827. [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7d06806dfdeb3252ac0defe23e8c468eabfa8b5e)
 2018-04-20 00:07:12 +05:30
Inigo Goiri
688933e6c7 YARN-8165. Incorrect queue name logging in AbstractContainerAllocator. Contributed by Weiwei Yan. 
(cherry picked from commit dd5e18c4aecba56f140c3cc11affc2cb5e61c79d)
 2018-04-17 15:29:39 +05:30
Lei Xu
3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Eric E Payne
43d2ee9de6 YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe 
(cherry picked from commit 18844599aef42f79d2af4500aa2eee472dda95cb)
 2018-04-12 18:10:42 +00:00
Eric E Payne
081ea1ec39 YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 
(cherry picked from commit 6bb128dfb893cf0e4aa2d3ecc65440668a1fc8d7)
 2018-04-12 16:22:48 +00:00
Vrushali C
ec258b8ed1 YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 
(cherry picked from commit 345e7624d58a058a1bad666bd1e5ce4b346a9056)
 2018-04-11 09:49:25 +05:30
Wangda Tan
48778eee52 YARN-8068. Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ (Sunil G via wangda) 
Change-Id: I7910bd1064a1b4dbbe2084080c060822ea6f3b48
(cherry picked from commit 9eef19b2ad78b8464da252d0e23c08675898b9d8)
 2018-03-26 11:33:33 -07:00
Wangda Tan
a465026207 YARN-8062. yarn rmadmin -getGroups returns group from which the user has been removed. (Sunil G via wangda) 
Change-Id: I80ed63846502bf7751b890b6c6c6a7c0679e2b4a
(cherry picked from commit 5d381570f83022b411a8740d58486a7f68ab2af6)
 2018-03-26 11:33:24 -07:00
Weiwei Yang
7c286fe804 YARN-7636. Re-reservation count may overflow when cluster resource exhausted for a long time. contributed by Tao Yang. 2018-03-16 19:11:54 +08:00
Sunil G
09bb378c10 YARN-7947. Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps. Contributed by Eric Payne. 
(cherry picked from commit bdd2a184d78379d99c802a43ebec7d2cef0bbaf7)
 2018-02-21 15:06:28 +05:30
Jason Lowe
85c611ad7d YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 2018-02-19 14:38:49 -06:00
Eric Payne
55ded70716 Revert "YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue" 
This reverts commit 36c9dda07f9d0ac14f0c979c7d519536fc9b3d28.
 2018-02-14 14:38:31 -06:00
Eric Payne
36c9dda07f YARN-7813: Capacity Scheduler Intra-queue Preemption should be configurable for each queue 2018-02-14 08:58:28 -06:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Sunil G
0463ede09d YARN-7790. Improve Capacity Scheduler Async Scheduling to better handle node failures. Contributed by Wangda Tan. 
(cherry picked from commit e9c72d04beddfe0252d2e81123a9fe66bdf04078)
 2018-01-29 20:46:32 +05:30
Jinjiang Ling
1d7e9b9ad8
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3400d0c535aeb151c3f283cc41111b15d66990e5)
 2018-01-30 00:12:40 +09:00
Jason Lowe
217f408aaa YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 
(cherry picked from commit ff8378eb1b960c72d18a984c7e5d145b407ca11a)
 2018-01-25 17:51:51 -06:00
Sunil G
88bacad50f YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 
(cherry picked from commit 82cc6f6968d2897ae006c5ef29eba74023f3483e)
 2018-01-25 14:33:42 +05:30
Rohith Sharma K S
2a01652809 YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 
(cherry picked from commit d09058b2fd18803d12f0835fdf78aef5e0b99c90)
 2018-01-16 08:42:07 +05:30
Sunil G
f270818ed3 YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 
(cherry picked from commit 128d773a2315fa6baaa3a52b13c53c77e741b69c)
 2018-01-12 15:42:06 +05:30
Haibo Chen
7b32d6811c YARN-7665. Allow FS scheduler state dump to be turned on/off separately from FS debug log. (Wilfred Spiegelenburg via Haibo Chen) 
(cherry picked from commit 8ee7080e5da15d8841d5f7bbf72ca033905c9751)
 2018-01-09 16:53:10 -08:00
Jason Lowe
effce9b117 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 
(cherry picked from commit 90c86d6e090d668414681fcb5eb02add5d419b33)
 2018-01-09 10:52:21 -06:00
Jason Lowe
d2c2d8c913 YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 
(cherry picked from commit ebff4decfe86571d203859928e28e6855862737b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/RMAppImpl.java
 2018-01-09 09:57:45 -06:00
bibinchundatt
c57e71b542 YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 
(cherry picked from commit b26e30ab1655a8cdf369862a5512db5c2a88ec46)
 2018-01-09 20:01:42 +05:30
Rohith Sharma K S
281350674c YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 2018-01-08 15:52:03 +05:30
Sunil G
a79abbc03e YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 2018-01-05 14:50:17 +05:30
Rohith Sharma K S
2c67985bd2 YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 
(cherry picked from commit c9bf813c9a6c018d14f2bef49ba086ec0e60c761)
 2018-01-03 18:21:47 +05:30
Eric Yang
62c3801af0 YARN-7466. addendum patch for failing unit test. (Contributed by Chandni Singh) 2017-12-19 19:56:40 -05:00
Rohith Sharma K S
53033c69f3 YARN-7190. Ensure only NM classpath in 2.x gets TSv2 related hbase jars, not the user classpath. Contributed by Varun Saxena. 
(cherry picked from commit 09d996fdd429a85822a06ab87f0e2322d0d7ca68)
 2017-12-15 22:17:43 +05:30
Wangda Tan
41796ea60a YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I67e086d6d35c3166ec0f6305490ea6527bd6d799
 2017-12-08 21:43:28 -08:00