Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,678 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1699 Commits

Author SHA1 Message Date
Daniel Templeton 7d3cea232d Revert "YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator" 
This reverts commit ce0cdc50d9.
 2017-08-11 20:52:07 -07:00
Daniel Templeton ce0cdc50d9 YARN-6882. AllocationFileLoaderService.reloadAllocations() should use the diamond operator 
(Contributed by Larry Lo via Daniel Templeton)

(cherry picked from commit 0996acde6c)
 2017-08-11 14:51:42 -07:00
Daniel Templeton 11e3673748 YARN-6967. Limit application attempt's diagnostic message size thoroughly 
(Contributed by Chengbing Liu via Daniel Templeton)

(cherry picked from commit 65364defb4)
 2017-08-11 14:31:11 -07:00
Daniel Templeton 2566e5fcf7 YARN-6884. AllocationFileLoaderService.loadQueue() has an if without braces 
(Contributed by weiyuan via Daniel Templeton)

(cherry picked from commit c7680d4cc4)
 2017-08-11 14:23:17 -07:00
Subru Krishnan e9ec1e8f87 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-08-08 14:01:22 -07:00
Daniel Templeton c5b3d80a4b YARN-6879. TestLeafQueue.testDRFUserLimits() has commented out code 
(Contributed by Angela Wang via Daniel Templeton)

(cherry picked from commit e0c24145d2)
 2017-08-08 13:36:49 -07:00
Wangda Tan e18415a350 YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
(cherry picked from commit adb84f34db)
 2017-08-07 11:34:45 -07:00
Yufei Gu b9549e108e YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-08-04 17:23:44 -07:00
Carlo Curino 7654b52167 YARN-6634. [API] Refactor ResourceManager WebServices to make API explicit (Contributed by Giovanni Matteo Fumarola via curino). 
(Edited cherry-pick of 547cb53e8a).
 2017-08-04 17:04:20 -07:00
Arun Suresh 4cfd248a72 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 
(cherry picked from commit 077fcf6a96)
 2017-08-04 16:50:21 -07:00
Arun Suresh 043b7d133e YARN-6776. Refactor ApplicaitonMasterService to move actual processing logic to a separate class. (asuresh) 
(cherry picked from commit 5496a34c0c)
 2017-08-04 16:43:35 -07:00
Arun Suresh 2d62af6545 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 
(cherry picked from commit aaf106fde3)
 2017-08-04 16:19:44 -07:00
Arun Suresh 88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 
(cherry picked from commit 35dc782923)
 2017-08-03 21:18:43 -07:00
Yufei Gu b6729a7be5 YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-03 16:56:15 -07:00
Yufei Gu ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00
Sunil G 82ec700241 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 
(cherry picked from commit f64cfeaf61)
 2017-08-03 19:28:16 +05:30
Eric Payne 223229b99c YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 
(cherry picked from commit 8ce8672b6b)
 2017-08-02 12:58:07 -05:00
Jian He 6ee0fe70c7 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:59 -07:00
Jian He 4006cc266b YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 10:00:04 -07:00
Yufei Gu e29d1c75ea YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit 9902be72cb)
 2017-07-27 23:21:30 -07:00
Varun Saxena 05bbbcbdbe Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit 11ece0bda1)
 2017-07-28 00:33:41 +05:30
Varun Saxena 0b342a644a YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit f66fd11e51)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMRestart.java
 2017-07-28 00:33:37 +05:30
Sunil G 08d5f3acf6 Addendum patch for YARN-6102. 2017-07-26 18:40:52 +05:30
Daniel Templeton 00f4a80e8f YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit f81a4efb8c)
 2017-07-25 13:02:00 -07:00
Akira Ajisaka 22f237c5d9
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 
(cherry picked from commit cca51e916b)
 2017-07-25 15:25:36 +09:00
Daniel Templeton 9b6e787230 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 10583625c1)
 2017-07-24 13:45:09 -07:00
Daniel Templeton eb1751670a YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)

(cherry picked from commit bb30bd3771)
 2017-07-24 12:15:35 -07:00
Sunil G a657472b42 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 20:57:56 +05:30
Wangda Tan 5070c9ba39 YARN-6775. CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls. (Nathan Roberts via wangda) 
Change-Id: Iaf9bb7e5ed3aa1300abdccf87ae6fcbddbd25e3e
 2017-07-19 15:25:21 -07:00
Rohith Sharma K S f806e8627b YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 
(cherry picked from commit 845c4e52bd)
 2017-07-19 11:16:17 +05:30
Yufei Gu d04bb465d2 YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit daaf530fce)
 2017-07-18 16:40:13 -07:00
Sunil G 487b0a3d7f Addendum patch for YARN-5731 
(cherry picked from commit 0b7afc060c)
 2017-07-18 13:46:43 +05:30
Sunil G 9b1abb448d YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. Contributed by Tao Yang. 2017-07-17 13:22:04 +05:30
Sunil G d462e48336 YARN-6280. Introduce deselect query param to skip ResourceRequest from getApp/getApps REST API. Contributed by Lantao Jin. 2017-07-17 12:25:41 +05:30
Jason Lowe 9eb93c6f3f YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 
(cherry picked from commit a5ae5ac50e)
 2017-07-14 15:03:48 -05:00
bibinchundatt af80d10e54 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 
(cherry picked from commit 75c0220b44)
 2017-07-14 14:05:38 +05:30
Yufei Gu b2a34fe8d6 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 
(cherry picked from commit 4a574e9a84)
 2017-07-13 23:11:27 -07:00
Sunil G fd6935709f YARN-5731. Preemption calculation is not accurate when reserved containers are present in queue. Contributed by Wangda Tan. 2017-07-14 08:45:17 +05:30
Sunil G f5cdee6bfd YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 228ddaa31d)
 2017-07-14 08:08:32 +05:30
Sunil G d83e8712f6 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne. 
(cherry picked from commit ca13b224b2)
 2017-07-13 14:50:28 -05:00
Wangda Tan cbb5f601f1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda) 
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
(cherry picked from commit c583ab02c7)
 2017-07-13 10:44:07 -05:00
Xuan edc0aa95f1 YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:45:51 -07:00
Wangda Tan f2d440b3b3 YARN-5889. Improve and refactor user-limit calculation in Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit 5fb723bb77)
 2017-07-12 10:06:02 -05:00
Eric Payne e6cdf770ca Revert "YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G)" 
This reverts commit eda4ac07c1.
Commit eda4ac07c1 was a separate patch from trunk rather than a cherry-pick. I will cherryp-ick dependencies and then cherry-pick the trunk commit for YARN-2113.
 2017-07-12 08:56:40 -05:00
Yufei Gu 20a2770d72 YARN-6764. Simplify the logic in FairScheduler#attemptScheduling. Contributed by Yufei Gu. 
(cherry picked from commit 3de47ab5ea)
 2017-07-09 16:11:26 -07:00
Naganarasimha b08ef44d9e YARN-6428. Queue AM limit is not honored in CS always. Contributed by Bibin A Chundatt. 
(cherry picked from commit 626d730bfc)
 2017-07-09 17:27:50 +05:30
Daniel Templeton fcef28e53b YARN-6410. FSContext.scheduler should be final (Contributed by Yeliang Cang via Daniel Templeton) 
(cherry picked from commit 9d27880442)
 2017-07-09 19:38:37 +09:00
Daniel Templeton a6d194d720 YARN-6746. SchedulerUtils.checkResourceRequestMatchingNodePartition() is dead code 
(Contributed by Deepti Sawhney via Daniel Templeton)

(cherry picked from commit c5a0c38d8e)
 2017-07-09 19:31:20 +09:00
Daniel Templeton edcfae8258 YARN-6752. Display reserved resources in web UI per application 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit 0615985886)
 2017-07-09 19:19:07 +09:00
Eric Payne eda4ac07c1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G) 2017-07-06 10:43:00 -05:00
Naganarasimha 87c81cfad3 YARN-6681. Eliminate double-copy of child queues in canAssignToThisQueue. Contributed by Daryn Sharp. 
(cherry picked from commit fa1aaee87b)
 2017-07-01 12:32:01 +05:30
Daniel Templeton 93c5421a43 YARN-6751. Display reserved resources in web UI per queue 
(Contributed by Abdullah Yousufi via Daniel Templeton)

(cherry picked from commit ec97519779)
 2017-06-29 16:57:23 -07:00
Naganarasimha a5ae7c0cae YARN-6467. CSQueueMetrics needs to update the current metrics for default partition only. Contributed by Manikandan R. 2017-06-29 00:09:30 +05:30
Naganarasimha 92e266c2b6 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-24 23:14:51 +05:30
Yufei Gu 4c59b446e2 YARN-5876. TestResourceTrackerService#testGracefulDecommissionWithApp fails intermittently on trunk. (Robert Kanter via Yufei Gu) 
(cherry picked from commit 0b77262890)
 2017-06-23 13:30:20 -07:00
Akira Ajisaka 35f6c7c9c7
YARN-6713. Fix dead link in the Javadoc of FairSchedulerEventLog.java. Contributed by Weiwei Yang. 
(cherry picked from commit 099dfe92a9)
 2017-06-20 17:23:06 +09:00
Naganarasimha 578318473c YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp 
(cherry picked from commit ee89ac84e6)
 2017-06-19 22:45:21 +05:30
bibinchundatt 3f9acf65fb YARN-6703. RM startup failure with old state store due to version mismatch. Contributed by Varun Saxena 
(cherry picked from commit d64c842743)
 2017-06-12 18:32:23 +05:30
Eric Payne 33605e0896 YARN-6585. RM fails to start when upgrading from 2.7 for clusters with node labels. Contributed by Sunil G. 
(cherry picked from commit 5578af8603)
 2017-06-09 16:18:16 -05:00
Naganarasimha 85d561a5b3 YARN-6681. Improve performance of AssignmentInformation datastructures. Contributed by Daryn sharp. 2017-06-09 11:46:51 +05:30
Daniel Templeton 10b7a5fa98 YARN-6246. Identifying starved apps does not need the scheduler writelock 
(Contributed by Karthik Kambatla via Daniel Templeton)

(cherry picked from commit d5b71e4175)
 2017-05-31 15:50:11 -07:00
Jonathan Eagles f5f12b576e YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles) 
(cherry picked from commit cbfed0e82f)
 2017-05-31 10:19:51 -05:00
Yufei Gu fc6cb4b2dd YARN-6582. FSAppAttempt demand can be updated atomically in updateDemand(). (Karthik Kambatla via Yufei Gu) 
(cherry picked from commit 87590090c8)
 2017-05-25 14:25:49 -07:00
Jason Lowe 4c44ff69df YARN-6643. TestRMFailover fails rarely due to port conflict. Contributed by Robert Kanter 
(cherry picked from commit 3fd6a2da4e)
 2017-05-25 16:09:08 -05:00
Subru Krishnan d1724ef919 YARN-6587. Refactor of ResourceManager#startWebApp in a Util class. (Giovanni Matteo Fumarola via Subru). 2017-05-24 13:40:49 -07:00
Wangda Tan 9a8fdfa385 YARN-6493. Print requested node partition in assignContainer logs. (Jonathan Hung via wangda) 
Change-Id: Ib20326daac10acad363befb0ae0998441fa5b32d
(cherry picked from commit 8e0f83e49a)
 2017-05-22 14:43:44 -07:00
Yufei Gu 50e263a316 YARN-6249. TestFairSchedulerPreemption fails inconsistently. (Tao Jie via Yufei Gu) 
(cherry picked from commit 66bba8c024)
 2017-05-19 15:22:14 -07:00
Rohith Sharma K S 379e5bd26f HADOOP-14412. HostsFileReader#getHostDetails is very expensive on large clusters. Contributed by Jason Lowe. 2017-05-17 08:27:50 +05:30
Jason Lowe feb7e92121 YARN-6603. NPE in RMAppsBlock. Contributed by Jason Lowe 
(cherry picked from commit 489f85933c)
 2017-05-16 09:28:42 -05:00
Haibo Chen 6284cc4663 YARN-6580. Incorrect logger for FairSharePolicy. (Vrushali C via Haibo Chen) 
(cherry picked from commit 6c35001b9f)
 2017-05-12 13:07:17 -07:00
Daniel Templeton 717f03ecf7 YARN-6380. FSAppAttempt keeps redundant copy of the queue 
(cherry picked from commit 90cb5b4635)
 2017-05-11 13:48:46 -07:00
Min Shen 9310d24b08 YARN-5543. ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread. Contributed by Min Shen. 
(cherry picked from commit 2ada100da7)
 2017-05-11 13:29:02 -07:00
Brahma Reddy Battula 3766e54893 HADOOP-14410. Correct spelling of 'beginning' and variants. Contributed By Dongtao Zhang 
(cherry picked from commit 4aae2d40a3)
 2017-05-11 23:38:18 +05:30
Daniel Templeton df3d7415c0 YARN-6571. Fix JavaDoc issues in SchedulingPolicy 
(Contributed by Weiwei Yang via Daniel Templeton)

(cherry picked from commit 18d54b5ae2f76513548c453e05cdd6d87777cd06)
 2017-05-10 14:37:36 -07:00
Jason Lowe 3230dd1ab6 YARN-6552. Increase YARN test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 
(cherry picked from commit 6099deebcb)
 2017-05-10 08:48:47 -05:00
Karthik Kambatla 3403ed3148 YARN-3742. YARN RM will shut down if ZKClient creation times out. (Daniel Templeton via kasha) 
(cherry picked from commit 166be0ee95)
 2017-05-09 14:44:39 -07:00
Daniel Templeton b337a1da31 YARN-6481. Yarn top shows negative container number in FS 
(Contributed by Tao Jie via Daniel Templeton)

(cherry picked from commit 9f0aea0ee2)
 2017-05-02 13:15:18 -07:00
Naganarasimha 5e28311341 YARN-6519. Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager. Contributed by Weiwei Yang. 2017-05-03 00:36:33 +05:30
Subru Krishnan 6d53224afc YARN-5331. Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru). 
(cherry picked from commit 6bf42e48ef)
 2017-05-01 18:49:14 -07:00
Subru Krishnan 2030f408e0 YARN-6542. Fix the logger in TestAlignedPlanner and TestGreedyReservationAgent. 
(cherry picked from commit e514fc432a)
 2017-05-01 18:31:08 -07:00
Subru Krishnan 0eae1c6368 YARN-4359. Update LowCost agents logic to take advantage of YARN-4358. (Jonathan Yaniv and Ishai Menache via Subru). 
(cherry picked from commit a3a615eeab)
 2017-05-01 18:30:58 -07:00
Daniel Templeton a8512d5aeb YARN-2962. ZKRMStateStore: Limit the number of znodes under a znode (Contributed by Varun Sexena via Daniel Templeton) 
(cherry picked from commit 2e52789edf)
 2017-04-28 13:39:11 -07:00
Wangda Tan 503dd650bf YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via wangda) 
(cherry picked from commit 2ba21d6376)
 2017-04-24 13:18:47 -07:00
Eric Payne 421e51cb9c YARN-5617. AMs only intended to run one attempt can be run more than once? Contributed by Jason Lowe. 
(cherry picked from commit 52adf71914)
 2017-04-24 12:08:35 -05:00
Sunil G a9495b206b YARN-6164. Expose Queue Configurations per Node Label through YARN client api. Contributed by Benson Qiu. 2017-04-24 12:16:18 +05:30
Jason Lowe f5a9287bb1 YARN-6501. FSSchedulerNode.java fails to compile with JDK7. Contributed by John Zhuge 2017-04-20 09:52:44 -05:00
Junping Du 9289f4527d YARN-6304. Skip rm.transitionToActive call to RM if RM is already active. Contributed by Rohith Sharma K S. 
(cherry picked from commit 6ed9d36224)
 2017-04-17 15:52:02 -07:00
Karthik Kambatla 6f5912c106 YARN-6432. FairScheduler: Reserve preempted resources for corresponding applications. (Miklos Szegedi via kasha) 2017-04-17 14:34:28 -07:00
Wangda Tan 6ade9e6ce6 YARN-6406. Remove SchedulerRequestKeys when no more pending ResourceRequest. (Arun Suresh via wangda) 
(cherry picked from commit 87e2ef8c98)
 2017-04-15 21:34:29 -07:00
Jason Lowe 1699b20526 YARN-6480. Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart. Contributed by Eric Badger 
(cherry picked from commit 4168805502)
 2017-04-14 17:17:46 -05:00
Eric Payne 157a9f3104 YARN-5994. TestCapacityScheduler.testAMLimitUsage fails intermittently. Contributed by Eric Badger 
(cherry-picked from commit a41f8dd58e)
 2017-04-14 11:04:18 -05:00
Wangda Tan 5756256280 YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 
(cherry picked from commit eac6b4c35c)
 2017-04-13 13:16:31 -07:00
Wangda Tan 408d23477f YARN-6040. Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes. (wangda) 2017-04-13 11:11:57 -07:00
Wangda Tan 2f08c86b7a YARN-6439. Fix ReservationSystem creation of default ReservationQueue. (Carlo Curino via wangda) 
(cherry picked from commit 4d4ad0ebb7)
 2017-04-11 14:59:34 -07:00
Naganarasimha b04c09163b YARN-6463. Correct spelling mistake in FileSystemRMStateStore. Contributed by Yeliang Cang. 
(cherry picked from commit 7d873c465b)
 2017-04-11 22:46:09 +05:30
Jason Lowe 0cb22a0f15 YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. Contributed by Benson Qiu 
(cherry picked from commit 0e065f2ede)
 2017-04-11 08:45:16 -05:00
Konstantinos Karanasos 9b3ba25980 YARN-6344. Add parameter for rack locality delay in CapacityScheduler. (kkaranasos) 
(cherry picked from commit 3587b46164514e6751d7cebb09c8ab5e988853b4)
 2017-04-10 17:22:29 -07:00
Karthik Kambatla e0b143e51a Revert "YARN-6298. Metric preemptCall is not used in new preemption. (Yufei Gu via kasha)" 
This reverts commit dcfce5fcdd.
 2017-04-07 17:37:31 -07:00
Karthik Kambatla dcfce5fcdd YARN-6298. Metric preemptCall is not used in new preemption. (Yufei Gu via kasha) 
(cherry picked from commit 2aa8967809)
 2017-04-07 17:36:39 -07:00
Daniel Templeton 59ad5cd68d YARN-6381. FSAppAttempt has several variables that should be final 
(Contributed by Ameet Zaveri via Daniel Templeton)

(cherry picked from commit 984df31a31c25ffc8541467dc41913f62f69e1df)
 2017-04-05 16:44:35 -07:00