Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,754 Commits 326 Branches 371 Tags
Commit Graph

1510 Commits

Author SHA1 Message Date
Karthik Kambatla
ca7a6a7365 YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha) 
(cherry picked from commit 718ad9f6ee93d4145f2bb19b7582ce4e1174feaf)
 2017-02-22 15:52:44 -08:00
Daniel Templeton
93219b6454 YARN-6125. The application attempt's diagnostic message should have a maximum size 
(Contributed by Andras Piros via Daniel Templeton)

(cherry picked from commit c7a36e613053ec8b111146004b887c2f13535469)
 2017-02-17 14:38:06 -08:00
Karthik Kambatla
5575328bb6 YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha) 
(cherry picked from commit dbbfcf74ab44d7bfdc805b63affd0defc57182b8)
 2017-02-17 14:07:47 -08:00
Daniel Templeton
10fd5627bb YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with 
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)

(cherry picked from commit 585168cc1bdb87398d4f388f3eeccd1c71114982)
 2017-02-17 13:14:37 -08:00
Karthik Kambatla
afc8124ff6 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha) 
(cherry picked from commit a77f432449aad67da31bd8bf8644b71def741bde)
 2017-02-16 15:03:35 -08:00
Rohith Sharma K S
f07b52dc5a YARN-6200. Reverting since the same functionality achieved by YARN-1623. 
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit a37b3694ead2ab4e8753e68213f31868b68376b8.
 2017-02-16 17:58:38 +05:30
Karthik Kambatla
9fecd5063f YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha) 
(cherry picked from commit 74dd14225059322825f706120aa57cf673820daf)
 2017-02-16 00:03:26 -08:00
Karthik Kambatla
fefac1276a YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha) 
(cherry picked from commit 11be3f70e029c2324b167563168c8a254d234aef)
 2017-02-15 23:51:51 -08:00
Karthik Kambatla
46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha) 
(cherry picked from commit 6c25dbcdc0517a825b92fb16444aa1d3761e160c)
 2017-02-15 23:23:18 -08:00
Varun Saxena
9cab25a63d YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit b7613e0f406fb2b9bd5b1b3c79658e801f63c587)
 2017-02-15 14:58:39 +05:30
Karthik Kambatla
58acb676a2 YARN-6061. Addendum. Remove extraneous change. 
(cherry picked from commit 353a9b2d9165a221491395edbadf8acc3a39990b)
 2017-02-14 15:20:11 -08:00
Karthik Kambatla
37921b3fef YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 
(cherry picked from commit 652679aa8ad6f9e61b8ed8e2b04b3e0332025e94)
 2017-02-14 13:40:06 -08:00
Karthik Kambatla
30ff5bff1a YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 
(cherry picked from commit 646c6d6509f515b1373288869fb92807fa2ddc9b)
 2017-02-13 11:27:02 -08:00
Karthik Kambatla
e259a818ad YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 
(cherry picked from commit 08f93978f3ec724b24a93d7ef538f158da75802f)
 2017-02-09 14:45:21 -08:00
Karthik Kambatla
41361ec39e YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:12:21 -08:00
Daniel Templeton
be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 
(cherry picked from commit 3e06475307e30377092547dfdebe6c2c6ac6e78f)
 2017-01-31 16:26:41 -08:00
Daniel Templeton
ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 
(cherry picked from commit 87852b6ef4b9d973b7b3999974d41c8860fb1495)
 2017-01-30 21:46:38 -08:00
Karthik Kambatla
c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 
(cherry picked from commit 11e44bdda74b3357714f4b11a66f703d64c0f14f)
 2017-01-30 13:39:35 -08:00
Karthik Kambatla
1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 
(cherry picked from commit 4d3f73acc0a5cabc748132889dbe670bea178a3f)
 2017-01-26 14:11:50 -08:00
Daniel Templeton
c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9f1c1c1444cc00750b03468cbf40fb9)
 2017-01-26 14:08:46 -08:00
Daniel Templeton
443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 
(cherry picked from commit b3befc021b0e2d63d1a3710ea450797d1129f1f5)
 2017-01-26 14:07:46 -08:00
Karthik Kambatla
73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 
(cherry picked from commit abedb8a9d86b4593a37fd3d2313fbcb057c7846a)
 2017-01-25 12:19:27 -08:00
Karthik Kambatla
71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 
(cherry picked from commit e224c9623493d6c4c2f3ff731fd3c72c0f448b19)
 2017-01-25 12:19:20 -08:00
Karthik Kambatla
0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Wangda Tan
1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Rohith Sharma K S
26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 
(cherry picked from commit 0101267d9d801eab4cb3b4df289c402ecb591685)
 2017-01-24 11:01:52 +05:30
Jason Lowe
94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Sunil G
f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 
(cherry picked from commit 3fa0d540dfca579f3c2840a959b748a7528b02ed)
 2017-01-23 12:24:39 +05:30
Wangda Tan
8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 
(cherry picked from commit 72054a817dfb43e93916d7036eba19cf2f49cea2)
 2017-01-18 13:35:30 -08:00
Sunil G
f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 
(cherry picked from commit d3170f9eba9bc5c38b5fa50c24e37ca2bd5636c2)
 2017-01-13 18:30:31 +05:30
Naganarasimha
4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 
(cherry picked from commit a6b06f71797ad1ed9edbcef279bcf7d9e569f955)
 2017-01-12 06:42:18 +05:30
Jason Lowe
cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 
(cherry picked from commit 357eab95668dbc419239857ac5ce763d76fd40e7)
 2017-01-11 17:15:14 +00:00
Junping Du
ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b55213929e5b86f2abb0ed84a21719b5)
 2017-01-10 18:22:45 -08:00
Wangda Tan
c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha
ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Sunil G
3190a4ba44 YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 
(cherry picked from commit 9594c35dcb655add1991d8fd15897b40c4ad6205)
 2017-01-10 14:10:25 +05:30
Junping Du
b600577095 YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 
(cherry picked from commit 945db55f2e6521d33d4f90bbb09179b0feba5e7a)
 2017-01-09 18:15:21 -08:00
Wangda Tan
447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804a869723ae36605a3c2d2b3eae3941e)
 2017-01-09 08:42:38 -08:00
Junping Du
f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Jian He
ba5d482366 YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 
(cherry picked from commit 020316458dfe6059b700f8d93a9791e4cb817b3f)
 2017-01-06 18:27:36 -08:00
Arun Suresh
81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 
(cherry picked from commit ac1e5d4f77e3b9df8dcacb0b1f72eecc27931eb8)
 2017-01-06 11:25:49 -08:00
Arun Suresh
2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f3895163046cca09ff5e983df301d22)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh
3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit b0aace21b1ef3436ba9d516186208fee9a9ceef2)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
 2017-01-06 11:15:09 -08:00
Arun Suresh
44774eb21c YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh) 
(cherry picked from commit 005850b28feb2f7bb8c2844d11e3f9d21b45d754)
(cherry picked from commit cbff10b4147f98a89b393519b17e16385294af07)
 2017-01-06 11:15:09 -08:00
Arun Suresh
368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac7d12aee343f6ab2980b3357fc618b6)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh
047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 283fa33febe043bd7b4fa87546be26c9c5a8f8b5)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
 2017-01-06 11:15:09 -08:00
Arun Suresh
7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d401a2f49d0f9ca556343f9ba871e979)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Arun Suresh
28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 
(cherry picked from commit aa3cab1eb29c56368d15882d7260a994e615e8d8)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
 2017-01-06 11:15:09 -08:00
Arun Suresh
f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 10be45986cdf86a89055065b752959bd6369d54f)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
 2017-01-06 11:15:09 -08:00
Sunil G
65e7ae5dcf YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2017-01-06 21:30:52 +05:30