Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,755 Commits 326 Branches 371 Tags
Commit Graph

2242 Commits

Author SHA1 Message Date
Karthik Kambatla
37edbd35f1 YARN-6194. Cluster capacity in SchedulingPolicy is updated only on allocation file reload. (Yufei Gu via kasha) 
(cherry picked from commit b10e962224a8ae1c6031a05322b0cc5e564bd078)
 2017-02-22 15:59:11 -08:00
Karthik Kambatla
ca7a6a7365 YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha) 
(cherry picked from commit 718ad9f6ee93d4145f2bb19b7582ce4e1174feaf)
 2017-02-22 15:52:44 -08:00
Daniel Templeton
93219b6454 YARN-6125. The application attempt's diagnostic message should have a maximum size 
(Contributed by Andras Piros via Daniel Templeton)

(cherry picked from commit c7a36e613053ec8b111146004b887c2f13535469)
 2017-02-17 14:38:06 -08:00
Karthik Kambatla
5575328bb6 YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha) 
(cherry picked from commit dbbfcf74ab44d7bfdc805b63affd0defc57182b8)
 2017-02-17 14:07:47 -08:00
Daniel Templeton
10fd5627bb YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with 
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)

(cherry picked from commit 585168cc1bdb87398d4f388f3eeccd1c71114982)
 2017-02-17 13:14:37 -08:00
Karthik Kambatla
afc8124ff6 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha) 
(cherry picked from commit a77f432449aad67da31bd8bf8644b71def741bde)
 2017-02-16 15:03:35 -08:00
Rohith Sharma K S
f07b52dc5a YARN-6200. Reverting since the same functionality achieved by YARN-1623. 
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit a37b3694ead2ab4e8753e68213f31868b68376b8.
 2017-02-16 17:58:38 +05:30
Karthik Kambatla
9fecd5063f YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha) 
(cherry picked from commit 74dd14225059322825f706120aa57cf673820daf)
 2017-02-16 00:03:26 -08:00
Karthik Kambatla
fefac1276a YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha) 
(cherry picked from commit 11be3f70e029c2324b167563168c8a254d234aef)
 2017-02-15 23:51:51 -08:00
Karthik Kambatla
46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha) 
(cherry picked from commit 6c25dbcdc0517a825b92fb16444aa1d3761e160c)
 2017-02-15 23:23:18 -08:00
Junping Du
2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong. 
(cherry picked from commit ce2d5bfa5f84e7e563980796549b56ef1e4bbf1e)
 2017-02-15 09:08:54 -08:00
Varun Saxena
9cab25a63d YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit b7613e0f406fb2b9bd5b1b3c79658e801f63c587)
 2017-02-15 14:58:39 +05:30
Karthik Kambatla
58acb676a2 YARN-6061. Addendum. Remove extraneous change. 
(cherry picked from commit 353a9b2d9165a221491395edbadf8acc3a39990b)
 2017-02-14 15:20:11 -08:00
Karthik Kambatla
37921b3fef YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 
(cherry picked from commit 652679aa8ad6f9e61b8ed8e2b04b3e0332025e94)
 2017-02-14 13:40:06 -08:00
Karthik Kambatla
30ff5bff1a YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 
(cherry picked from commit 646c6d6509f515b1373288869fb92807fa2ddc9b)
 2017-02-13 11:27:02 -08:00
Junping Du
205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong. 
(cherry picked from commit 464ff479ceec76609dca3539081de6b503b17325)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
 2017-02-13 06:16:04 -08:00
Karthik Kambatla
e259a818ad YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 
(cherry picked from commit 08f93978f3ec724b24a93d7ef538f158da75802f)
 2017-02-09 14:45:21 -08:00
Karthik Kambatla
41361ec39e YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:12:21 -08:00
Junping Du
ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 
(cherry picked from commit 9dbfab1284015af6983b4a55b79c9e65bc0194e5)
 2017-02-07 02:14:09 -08:00
Junping Du
c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton
be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 
(cherry picked from commit 3e06475307e30377092547dfdebe6c2c6ac6e78f)
 2017-01-31 16:26:41 -08:00
Daniel Templeton
ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 
(cherry picked from commit 87852b6ef4b9d973b7b3999974d41c8860fb1495)
 2017-01-30 21:46:38 -08:00
Karthik Kambatla
c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 
(cherry picked from commit 11e44bdda74b3357714f4b11a66f703d64c0f14f)
 2017-01-30 13:39:35 -08:00
Jason Lowe
4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Karthik Kambatla
1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 
(cherry picked from commit 4d3f73acc0a5cabc748132889dbe670bea178a3f)
 2017-01-26 14:11:50 -08:00
Daniel Templeton
c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9f1c1c1444cc00750b03468cbf40fb9)
 2017-01-26 14:08:46 -08:00
Daniel Templeton
443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 
(cherry picked from commit b3befc021b0e2d63d1a3710ea450797d1129f1f5)
 2017-01-26 14:07:46 -08:00
Jason Lowe
b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed05284ad897d04f4575ab0a4ee7a88b.
 2017-01-26 14:42:22 +00:00
Jason Lowe
45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1950cbcfcd1969461a8a910efca0767)
 2017-01-25 21:54:17 +00:00
Karthik Kambatla
73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 
(cherry picked from commit abedb8a9d86b4593a37fd3d2313fbcb057c7846a)
 2017-01-25 12:19:27 -08:00
Karthik Kambatla
71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 
(cherry picked from commit e224c9623493d6c4c2f3ff731fd3c72c0f448b19)
 2017-01-25 12:19:20 -08:00
Karthik Kambatla
0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du
8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Wangda Tan
1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Jason Lowe
1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Rohith Sharma K S
26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 
(cherry picked from commit 0101267d9d801eab4cb3b4df289c402ecb591685)
 2017-01-24 11:01:52 +05:30
Jason Lowe
94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f59378fd19a2c65db8019393d7c05a2)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Sunil G
f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 
(cherry picked from commit 3fa0d540dfca579f3c2840a959b748a7528b02ed)
 2017-01-23 12:24:39 +05:30
Chris Nauroth
5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer. 
(cherry picked from commit b913677365ad77ca7daa5741c04c14df1a0313cd)
 2017-01-19 20:48:27 +05:30
Chris Nauroth
198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer. 
(cherry picked from commit ef501b1a0b4c34a2cc43eb082d1c2364684cd7f1)
 2017-01-19 20:48:27 +05:30
Wangda Tan
8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 
(cherry picked from commit 72054a817dfb43e93916d7036eba19cf2f49cea2)
 2017-01-18 13:35:30 -08:00
Subru Krishnan
fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit 4d1f3d9020b8a8bf1d2a81e4d6ad20418ed9bcc2)
 2017-01-17 14:51:27 -08:00
Junping Du
861e275646 YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong. 
(cherry picked from commit cf695577aa6d9715b77ab7309ecd792ef226c439)
 2017-01-16 16:41:16 -08:00
Sunil G
f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 
(cherry picked from commit d3170f9eba9bc5c38b5fa50c24e37ca2bd5636c2)
 2017-01-13 18:30:31 +05:30
Naganarasimha
4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 
(cherry picked from commit a6b06f71797ad1ed9edbcef279bcf7d9e569f955)
 2017-01-12 06:42:18 +05:30
Daniel Templeton
718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
 2017-01-11 15:36:18 -08:00
Jason Lowe
cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 
(cherry picked from commit 357eab95668dbc419239857ac5ce763d76fd40e7)
 2017-01-11 17:15:14 +00:00
Junping Du
ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b55213929e5b86f2abb0ed84a21719b5)
 2017-01-10 18:22:45 -08:00
Wangda Tan
c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha
a7cdd39234 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash. 
(cherry picked from commit 4c431a694059e40e78365b02a1497a6c7e479a70)
 2017-01-10 15:55:46 +05:30