Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,321 Commits 326 Branches 371 Tags
Commit Graph

2385 Commits

Author SHA1 Message Date
Wangda Tan
cbb5f601f1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda) 
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
(cherry picked from commit c583ab02c730be0a63d974039a78f2dc67dc2db6)
 2017-07-13 10:44:07 -05:00
Xuan
edc0aa95f1 YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:45:51 -07:00
Wangda Tan
f2d440b3b3 YARN-5889. Improve and refactor user-limit calculation in Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit 5fb723bb77722d41df6959eee23e1b0cfeb5584e)
 2017-07-12 10:06:02 -05:00
Eric Payne
e6cdf770ca Revert "YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G)" 
This reverts commit eda4ac07c1835031aca7e27cc673f1c5913813bb.
Commit eda4ac07c1835031aca7e27cc673f1c5913813bb was a separate patch from trunk rather than a cherry-pick. I will cherryp-ick dependencies and then cherry-pick the trunk commit for YARN-2113.
 2017-07-12 08:56:40 -05:00
Yufei Gu
20a2770d72 YARN-6764. Simplify the logic in FairScheduler#attemptScheduling. Contributed by Yufei Gu. 
(cherry picked from commit 3de47ab5ea5cb75805a03010dc72e11b0cf6c173)
 2017-07-09 16:11:26 -07:00
Naganarasimha
b08ef44d9e YARN-6428. Queue AM limit is not honored in CS always. Contributed by Bibin A Chundatt. 
(cherry picked from commit 626d730bfc4c1f6ea54e8a246a0a676cdb74937f)
 2017-07-09 17:27:50 +05:30
Daniel Templeton
fcef28e53b YARN-6410. FSContext.scheduler should be final (Contributed by Yeliang Cang via Daniel Templeton) 
(cherry picked from commit 9d278804425f0118b590d302985408e0794b88bb)
 2017-07-09 19:38:37 +09:00
Daniel Templeton
a6d194d720 YARN-6746. SchedulerUtils.checkResourceRequestMatchingNodePartition() is dead code 
(Contributed by Deepti Sawhney via Daniel Templeton)

(cherry picked from commit c5a0c38d8e1752031d1dd8bdd4a7a3894cf775cb)
 2017-07-09 19:31:20 +09:00
Daniel Templeton
edcfae8258 YARN-6752. Display reserved resources in web UI per application 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit 06159858868a9cbeefc31bc6be5ae48a59cc8a6c)
 2017-07-09 19:19:07 +09:00
Eric Payne
eda4ac07c1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G) 2017-07-06 10:43:00 -05:00
Jason Lowe
53c2441c10 YARN-6708. Nodemanager container crash after ext3 folder limit. Contributed by Bibin A Chundatt 
(cherry picked from commit 7576a688ea84aed7206321b1f03594e43a5f216e)
 2017-07-06 09:44:02 -05:00
Naganarasimha
87c81cfad3 YARN-6681. Eliminate double-copy of child queues in canAssignToThisQueue. Contributed by Daryn Sharp. 
(cherry picked from commit fa1aaee87b0141a0255b5f8e5fd8e8f49d7efe86)
 2017-07-01 12:32:01 +05:30
Daniel Templeton
93c5421a43 YARN-6751. Display reserved resources in web UI per queue 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit ec975197799417a1d5727dedc395fe6c15c30eb2)
 2017-06-29 16:57:23 -07:00
Naganarasimha
a5ae7c0cae YARN-6467. CSQueueMetrics needs to update the current metrics for default partition only. Contributed by Manikandan R. 2017-06-29 00:09:30 +05:30
Jason Lowe
44b3c6ffaa YARN-6738. LevelDBCacheTimelineStore should reuse ObjectMapper instances. Contributed by Zoltan Haindrich 
(cherry picked from commit 63ce1593c5b78eb172773e7498d9c321debe81e8)
 2017-06-27 17:13:49 -05:00
Naganarasimha
92e266c2b6 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-24 23:14:51 +05:30
Yufei Gu
4c59b446e2 YARN-5876. TestResourceTrackerService#testGracefulDecommissionWithApp fails intermittently on trunk. (Robert Kanter via Yufei Gu) 
(cherry picked from commit 0b77262890d76b0a3a35fa64befc8a406bc70b27)
 2017-06-23 13:30:20 -07:00
Arun Suresh
66f780e6ab YARN-6127. Add support for work preserving NM restart when AMRMProxy is enabled. (Botong Huang via asuresh). 2017-06-22 15:07:38 -07:00
Akira Ajisaka
35f6c7c9c7
YARN-6713. Fix dead link in the Javadoc of FairSchedulerEventLog.java. Contributed by Weiwei Yang. 
(cherry picked from commit 099dfe92a9743bcb7e27506f8877b8b23b0ff95a)
 2017-06-20 17:23:06 +09:00
Naganarasimha
578318473c YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp 
(cherry picked from commit ee89ac84e68d3e181b75c63f74a0444f9d28146f)
 2017-06-19 22:45:21 +05:30
bibinchundatt
3f9acf65fb YARN-6703. RM startup failure with old state store due to version mismatch. Contributed by Varun Saxena 
(cherry picked from commit d64c842743da0d9d91c46985a9fd7350ea14b204)
 2017-06-12 18:32:23 +05:30
Eric Payne
33605e0896 YARN-6585. RM fails to start when upgrading from 2.7 for clusters with node labels. Contributed by Sunil G. 
(cherry picked from commit 5578af860335ae44c9780082508c3dcf726f60fc)
 2017-06-09 16:18:16 -05:00
Naganarasimha
85d561a5b3 YARN-6681. Improve performance of AssignmentInformation datastructures. Contributed by Daryn sharp. 2017-06-09 11:46:51 +05:30
Daniel Templeton
c5b28c03a1 YARN-6679. Reduce Resource instance overhead via non-PBImpl 
(Contributed by Daryn Sharp via Daniel Templeton)
 2017-06-08 10:41:32 -07:00
Daniel Templeton
0bcbf2a09b YARN-6208. Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application 
(Contributed by Akira Ajisaka via Daniel Templeton)

(cherry picked from commit 73ecb19312879d54e1cbe80199fe950d81c81104)
 2017-06-02 08:52:27 -07:00
Daniel Templeton
10b7a5fa98 YARN-6246. Identifying starved apps does not need the scheduler writelock 
(Contributed by Karthik Kambatla via Daniel Templeton)

(cherry picked from commit d5b71e4175c13679d451710be150fc461a661263)
 2017-05-31 15:50:11 -07:00
Nathan Roberts
c112bf6838 YARN-6649. RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception. Contributed by Jon Eagles. 
(cherry picked from commit 4369690ce63566131aee28696bf2683a3cb20205)
 2017-05-31 12:35:50 -05:00
Jonathan Eagles
f5f12b576e YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles) 
(cherry picked from commit cbfed0e82f57e96b8d5309e0613057963840554f)
 2017-05-31 10:19:51 -05:00
Varun Vasudev
1eecde3355 YARN-6366. Refactor the NodeManager DeletionService to support additional DeletionTask types. Contributed by Shane Kumpf. 
(cherry picked from commit 547f18cb96aeda55cc19b38be2be4d631b3a5f4f)
 2017-05-31 16:16:33 +05:30
Jason Lowe
3c0eb9c415 YARN-6641. Non-public resource localization on a bad disk causes subsequent containers failure. Contributed by Kuhu Shukla 
(cherry picked from commit aea42930bbb9566ea6988f684dbd72a72a2bdadf)
 2017-05-26 09:39:22 -05:00
Yufei Gu
fc6cb4b2dd YARN-6582. FSAppAttempt demand can be updated atomically in updateDemand(). (Karthik Kambatla via Yufei Gu) 
(cherry picked from commit 87590090c887829e874a7132be9cf8de061437d6)
 2017-05-25 14:25:49 -07:00
Jason Lowe
4c44ff69df YARN-6643. TestRMFailover fails rarely due to port conflict. Contributed by Robert Kanter 
(cherry picked from commit 3fd6a2da4e537423d1462238e10cc9e1f698d1c2)
 2017-05-25 16:09:08 -05:00
Varun Vasudev
1a6c532301 Addendum patch to fix Docker sanitization. 
(cherry picked from commit 2ff2a1f50e8c7c0f33676b010b256d6c8daf912d)
 2017-05-25 14:54:08 +05:30
Akira Ajisaka
ca1c0cbc62
YARN-6141. ppc64le on Linux doesn't trigger __linux get_executable codepath. Contributed by Sonia Garudi and Ayappan. 
(cherry picked from commit bc28da65fb1c67904aa3cefd7273cb7423521014)
 2017-05-25 17:07:50 +09:00
Subru Krishnan
d1724ef919 YARN-6587. Refactor of ResourceManager#startWebApp in a Util class. (Giovanni Matteo Fumarola via Subru). 2017-05-24 13:40:49 -07:00
Jason Lowe
96a97e7eb2 YARN-6615. AmIpFilter drops query parameters on redirect. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 8bf1949c0efed700781eb47cf18f9f88443ed506)
 2017-05-24 11:24:00 -05:00
Sunil G
ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Wangda Tan
9a8fdfa385 YARN-6493. Print requested node partition in assignContainer logs. (Jonathan Hung via wangda) 
Change-Id: Ib20326daac10acad363befb0ae0998441fa5b32d
(cherry picked from commit 8e0f83e49a8987cf45a72c8a9bb8587b86e4c0ed)
 2017-05-22 14:43:44 -07:00
Robert Kanter
2719cc0388 YARN-6602. Impersonation does not work if standby RM is contacted first (rkanter) 
(cherry picked from commit 9855225a790e66835c79816712f432ffa8c599b4)
 2017-05-19 15:43:43 -07:00
Yufei Gu
50e263a316 YARN-6249. TestFairSchedulerPreemption fails inconsistently. (Tao Jie via Yufei Gu) 
(cherry picked from commit 66bba8c0244e5b17d496e2fa3999991705c11bc7)
 2017-05-19 15:22:14 -07:00
Eric Payne
46c54bf200 YARN-6618. TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than once. Contributed by Jason Lowe. 
(cherry picked from commit 19482e0d0990feb719678d71971aa849095982fd)
 2017-05-19 11:29:02 -05:00
Varun Vasudev
cd1a04e3c6 Sanitize arguments before launching Docker containers. 2017-05-18 11:46:03 +05:30
Varun Vasudev
603a0a316c Validate docker image name before launching container. 2017-05-18 11:31:12 +05:30
Chris Douglas
3d2afb209c YARN-6577. Remove unused ContainerLocalization classes. Contributed by ZhangBing Lin 
(cherry picked from commit b23fcc86c670aa896151a2bd8878154d7bb45d13)
 2017-05-17 20:40:20 -07:00
Rohith Sharma K S
379e5bd26f HADOOP-14412. HostsFileReader#getHostDetails is very expensive on large clusters. Contributed by Jason Lowe. 2017-05-17 08:27:50 +05:30
Jian He
82751ff0cb YARN-6306. NMClient API change for container upgrade. Contributed by Arun Suresh 2017-05-16 10:49:12 -07:00
Jason Lowe
feb7e92121 YARN-6603. NPE in RMAppsBlock. Contributed by Jason Lowe 
(cherry picked from commit 489f85933c508bc26de607b921e56e23b979fce8)
 2017-05-16 09:28:42 -05:00
Jonathan Eagles
d370edc6aa YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles) 
(cherry picked from commit c48f2976a3de60b95c4a5ada4f0131c4cdde177a)
 2017-05-15 10:32:47 -05:00
Haibo Chen
6284cc4663 YARN-6580. Incorrect logger for FairSharePolicy. (Vrushali C via Haibo Chen) 
(cherry picked from commit 6c35001b9f93fd85a02c3465e87bfd1612f4cce9)
 2017-05-12 13:07:17 -07:00
Daniel Templeton
717f03ecf7 YARN-6380. FSAppAttempt keeps redundant copy of the queue 
(cherry picked from commit 90cb5b4635b7a0849912458afad052f569131a59)
 2017-05-11 13:48:46 -07:00