Commit Graph

864 Commits

Author SHA1 Message Date
Wangda Tan 9bf1e74171 YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. (Naganarasimha G R via wangda)
(cherry picked from commit 6f606214e7)
2015-10-26 16:38:39 -07:00
Wangda Tan baa2998e93 YARN-4285. Display resource usage as percentage of queue and cluster in the RM UI (Varun Vasudev via wangda)
(cherry picked from commit 3cc73773eb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-10-26 13:12:56 -07:00
Jason Lowe 7d66e2e666 YARN-4284. condition for AM blacklisting is too narrow. Contributed by Sangjin Lee
(cherry picked from commit 33a03af3c3)
2015-10-26 19:54:16 +00:00
Arun Suresh f0580dcac0 YARN-3738. Add support for recovery of reserved apps running under dynamic queues (subru via asuresh)
(cherry picked from commit ab8eb8770c)
2015-10-24 22:54:45 -07:00
Jason Lowe 235e84bf3d YARN-4041. Slow delegation token renewal can severely prolong RM recovery. Contributed by Sunil G
(cherry picked from commit d3a34a4f38)
2015-10-23 20:57:53 +00:00
Ming Ma 4bb7e68eb6 YARN-2913. Fair scheduler should have ability to set MaxResourceDefault for each queue. (Siqi Li via mingma)
(cherry picked from commit 934d96a334)
2015-10-23 08:37:46 -07:00
Zhihai Xu 9fe7ccc61c YARN-4256. YARN fair scheduler vcores with decimal values. Contributed by Jun Gong
(cherry picked from commit 960201b79b)
2015-10-22 12:31:28 -07:00
Anubhav Dhoot c44401f362 YARN-3739. Add reservation system recovery to RM recovery process. Contributed by Subru Krishnan.
(cherry picked from commit 2798723a54)
2015-10-22 06:52:41 -07:00
Arun Suresh f2009dc89c YARN-3985. Make ReservationSystem persist state using RMStateStore reservation APIs. (adhoot via asuresh)
(cherry picked from commit 506d1b1dbc)
2015-10-20 16:48:58 -07:00
Arun Suresh acc0e718d6 YARN-4270. Limit application resource reservation on nodes for non-node/rack specific requests (asuresh)
(cherry picked from commit 7e2837f830)
2015-10-19 20:02:10 -07:00
Jian He 1ba73169c7 YARN-4170. AM need to be notified with priority in AllocateResponse. Contributed by Sunil G
(cherry picked from commit f9da5cdb2b)
2015-10-16 15:26:46 -07:00
Wangda Tan 204ee6a40f YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API. (Naganarasimha G R via wangda)
(cherry picked from commit 4337b263aa)
2015-10-16 15:10:08 -07:00
Jian He 2862057f11 YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart. Contributed by Varun Saxena
(cherry picked from commit cf23f2c2b5)
2015-10-15 17:13:09 -07:00
Jian He ee1ce393f5 YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. Contributed by Meng Ding
(cherry picked from commit 9849c8b386)
2015-10-12 11:52:04 -07:00
Zhihai Xu 9988b57e71 YARN-4201. AMBlacklist does not work for minicluster. Contributed by Jun Gong.
(cherry picked from commit 049c6e8dc0)
2015-10-12 00:16:40 -07:00
Devaraj K 28edc7b129 YARN-3964. Support NodeLabelsProvider at Resource Manager side.
Contributed by Dian Fu.

(cherry picked from commit db93047881)
2015-10-11 11:24:05 +05:30
Wangda Tan 37642abb83 YARN-4140. RM container allocation delayed incase of app submitted to Nodelabel partition. (Bibin A Chundatt via wangda)
(cherry picked from commit def374e666)
2015-10-09 16:40:10 -07:00
Jason Lowe 0139517811 YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S
(cherry picked from commit a0bca2b5ad)
2015-10-09 14:19:41 +00:00
Rohith Sharma K S 21609e8e9a YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned for a user. (Anubhav Dhoot via rohithsharmaks)
(cherry picked from commit 8f195387a4)
2015-10-09 10:11:53 +05:30
Rohith Sharma K S 053fcf4c0d YARN-4209. RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks)
(cherry picked from commit 9156fc60c6)
2015-10-07 09:37:14 +05:30
Wangda Tan 5453a63612 YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request. (Naganarasimha G R via wangda)
(cherry picked from commit 29a582ada0)
2015-10-06 11:59:10 -07:00
Xuan 53bddc410f YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma
(cherry picked from commit 8f08532bde)
2015-10-02 18:53:04 -07:00
Anubhav Dhoot 22f2501476 YARN-4180. AMLauncher does not retry on failures when talking to NM. (adhoot)
(cherry picked from commit 9735afe967)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
2015-09-28 16:54:06 -07:00
Jason Lowe 600022ae69 YARN-4141. Runtime Application Priority change should not throw exception for applications at finishing states. Contributed by Sunil G
(cherry picked from commit 9f53a95ff6)
2015-09-28 22:56:07 +00:00
Anubhav Dhoot 4d8b99423e YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot)
(cherry picked from commit fb2e525c07)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-09-28 09:39:52 -07:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
2015-09-24 10:30:04 -07:00
Jian He fa0a554ae0 YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding
(cherry picked from commit 5f5a968d65)
2015-09-24 10:23:46 -07:00
Jian He c19e8ffdc7 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan)
(cherry picked from commit 83a18add10)
2015-09-24 10:23:25 -07:00
Rohith Sharma K S fbb1d923f4 YARN-4044. Running applications information changes such as movequeue is not published to TimeLine server. (Sunil G via rohithsharmaks) 2015-09-24 12:17:55 +05:30
Arun Suresh b70c7b0e51 YARN-3920. FairScheduler container reservation on a node should be configurable to limit it to large containers (adhoot via asuresh)
(cherry picked from commit 94dec5a916)
2015-09-18 14:04:03 -07:00
Wangda Tan 4a657e9326 YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. (Junping Du via wangda)
(cherry picked from commit 9bc913a35c)
2015-09-18 10:05:04 -07:00
Rohith Sharma K S d94c092ad1 YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state.(Nijel S F via rohithsharmaks)
(cherry picked from commit 723c31d45b)
2015-09-18 08:46:36 +05:30
Jian He 5d2f85021f YARN-4034. Render cluster Max Priority in scheduler metrics in RM web UI. Contributed by Rohith Sharma K S
(cherry picked from commit 6c6e734f0b)
2015-09-17 14:56:32 +08:00
Jian He 9eda3ce3fa YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. Contributed by Naganarasimha G R
(cherry picked from commit 452079af8b)
2015-09-16 15:01:23 +08:00
Wangda Tan c31fe79979 YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda)
(cherry picked from commit ae5308fe1d)
2015-09-15 11:41:57 -07:00
Junping Du df714e25aa YARN-313. Add Admin API for supporting node resource configuration in command line. (Contributed by Inigo Goiri, Kenji Kikushima and Junping Du)
(cherry picked from commit 73e3a49eb0)
2015-09-15 07:58:47 -07:00
Jian He eacc18677a YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan
(cherry picked from commit 5468baa80a)
2015-09-15 19:02:17 +08:00
Jian He b705d9d0ac YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt
(cherry picked from commit e1b1d7e4ae)
2015-09-14 14:15:04 +08:00
Karthik Kambatla 1f9f219062 YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. (Zhihai Xu via kasha)
(cherry picked from commit 332b520a48)
2015-09-13 18:08:03 -07:00
Karthik Kambatla 11e2fa151c YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha)
(cherry picked from commit 81df7b586a)
2015-09-13 17:06:04 -07:00
Robert Kanter 43f051a09a YARN-4145. Make RMHATestBase abstract so its not run when running all tests under that namespace (adhoot via rkanter)
(cherry picked from commit ea4bb2749f)
2015-09-11 11:46:35 -07:00
Wangda Tan 9f97b86816 YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda)
(cherry picked from commit bcc85e3bab)
2015-09-04 15:14:34 -07:00
Jason Lowe 6a5220cab3 YARN-4105. Capacity Scheduler headroom for DRF is wrong. Contributed by Chang Li
(cherry picked from commit 6eaca2e363)
2015-09-04 15:31:37 +00:00
Varun Vasudev 64c2e8011e YARN-4103. RM WebServices missing scheme for appattempts logLinks. Contributed by Jonathan Eagles.
(cherry picked from commit 1b2e2ca9104dfc05e9aa7f93954ae2e00b3e62c4)
2015-09-04 14:32:07 +05:30
Varun Vasudev a53054c853 YARN-3970. Add REST api support for Application Priority. Contributed by Naganarasimha G R.
(cherry picked from commit 223a306da925ad031bc398f2ea5e56f0e6521edf)
2015-09-03 16:40:42 +05:30
Jian He cd82fa2f83 YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager gets connection issue. Contributed by Xuan Gong
(cherry picked from commit 09c64ba1ba)
2015-09-02 17:46:04 -07:00
Rohith Sharma K S 6e41aa64a5 YARN-3893. Both RM in active state when Admin#transitionToActive failure from refeshAll() (Bibin A Chundatt via rohithsharmaks)
(cherry picked from commit 7d6687fe76)
2015-09-02 15:27:33 +05:30
Varun Vasudev 855e0f8b00 YARN-4082. Container shouldn't be killed when node's label updated. Contributed by Wangda Tan.
(cherry picked from commit bf669b6d9f)
2015-09-01 14:20:54 +05:30
Junping Du 792b9c0eaf YARN-1556. NPE getting application report with a null appId. Contributed by Weiwei Yang.
(cherry picked from commit beb65c9465)
2015-08-28 05:58:45 -07:00
Jian He 306c535395 YARN-3250. Support admin cli interface in for Application Priority. Contributed by Rohith Sharma K S
(cherry picked from commit a9c8ea71aa)
2015-08-27 13:27:34 -07:00
Jian He 387076894f YARN-4014. Support user cli interface in for Application Priority. Contributed by Rohith Sharma K S
(cherry picked from commit 57c7ae1aff)
2015-08-24 20:37:18 -07:00
Rohith Sharma K S 6dc732f2f7 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks)
(cherry picked from commit feaf034994)
2015-08-24 11:30:46 +05:30
Xuan 205b98c059 YARN-221. NM should provide a way for AM to tell it not to aggregate
logs. Contributed by Ming Ma

(cherry picked from commit 37e1c3d82a)
2015-08-22 16:26:39 -07:00
Rohith Sharma K S 27ced8253b YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface
(cherry picked from commit 22de7c1dca)
2015-08-21 10:56:45 +05:30
Zhihai Xu 8f2eb99d71 YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao.
(cherry picked from commit 3a76a010b8)
2015-08-18 10:53:16 -07:00
Jian He 65d22b3686 YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan
(cherry picked from commit e5003be907)
2015-08-12 15:09:46 -07:00
rohithsharmaks c5b20e0db6 YARN-4023. Publish Application Priority to TimelineServer. (Sunil G via rohithsharmaks)
(cherry picked from commit 1c12adb71f)
2015-08-12 14:47:23 +05:30
Xuan 2ebdf5bfce YARN-3999. RM hangs on draing events. Contributed by Jian He
(cherry picked from commit 3ae716fa69)
2015-08-11 18:26:00 -07:00
Jian He df97e3265b YARN-3887. Support changing Application priority during runtime. Contributed by Sunil G
(cherry picked from commit fa1d84ae27)
2015-08-10 20:52:18 -07:00
Wangda Tan 380d401da5 YARN-3873. PendingApplications in LeafQueue should also use OrderingPolicy. (Sunil G via wangda)
(cherry picked from commit cf9d3c9256)
2015-08-10 14:58:19 -07:00
Rohith Sharma K S a0da1ec010 YARN-3948. Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks)
(cherry picked from commit b6265d39c5)
2015-08-07 10:47:08 +05:30
Carlo Curino 8e23a7997f YARN-3974. Refactor the reservation system test cases to use parameterized base test. (subru via curino)
(cherry picked from commit 8572a5a14b)
2015-08-02 02:06:06 -07:00
Arun Suresh 84ea7cd408 YARN-3961. Expose pending, running and reserved containers of a queue in REST api and yarn top (adhoot via asuresh)
(cherry picked from commit 154c9d2e42)
2015-08-05 23:15:33 -07:00
rohithsharmaks dc76c4b035 YARN-3992. TestApplicationPriority.testApplicationPriorityAllocation fails intermittently. (Contributed by Sunil G)
(cherry picked from commit df9e7280db)
2015-08-06 10:49:37 +05:30
Jian He 1466772827 YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan
(cherry picked from commit ba2313d614)
2015-08-05 13:51:08 -07:00
Arun Suresh 707b96fa58 YARN-3736. Add RMStateStore apis to store and load accepted reservations for failover (adhoot via asuresh)
(cherry picked from commit f271d37735)
2015-08-05 12:58:37 -07:00
Xuan 8126fde884 YARN-3543. ApplicationReport should be able to tell whether the
Application is AM managed or not. Contributed by Rohith Sharma K S

(cherry picked from commit 0306d902f5)
2015-08-03 15:47:21 -07:00
Jonathan Eagles 899df5bce0 YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles)
(cherry picked from commit 3cd02b9522)
2015-08-03 10:40:04 -05:00
Jason Lowe c31e3ba921 YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt
(cherry picked from commit 32e490b6c0)
2015-07-31 17:38:49 +00:00
Zhihai Xu 0e2019fa30 YARN-433. When RM is catching up with node updates then it should not expire acquired containers. Contributed by Xuan Gong
(cherry picked from commit ab80e27703)
2015-07-30 22:03:04 -07:00
Wangda Tan 53a0653826 YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (Bibin A Chundatt via wangda)
(cherry picked from commit 91b42e7d6e)
2015-07-30 10:01:01 -07:00
ccurino 26ea045814 YARN-3656. LowCost: A Cost-Based Placement Agent for YARN Reservations. (Jonathan Yaniv and Ishai Menache via curino)
(cherry picked from commit 156f24ead0)
2015-07-25 07:47:11 -07:00
Jian He 3233284e87 YARN-3026. Move application-specific container allocation logic from LeafQueue to FiCaSchedulerApp. Contributed by Wangda Tan
(cherry picked from commit 83fe34ac08)
2015-07-24 14:01:36 -07:00
Karthik Kambatla d2ee02940a YARN-3957. FairScheduler NPE In FairSchedulerQueueInfo causing scheduler page to return 500. (Anubhav Dhoot via kasha)
(cherry picked from commit d19d187753)
2015-07-24 11:49:46 -07:00
carlo curino 6a71a0a230 YARN-3969. Allow jobs to be submitted to reservation that is active but does not have any allocations. (subru via curino)
(cherry picked from commit 0fcb4a8cf2)
2015-07-23 19:36:54 -07:00
Robert Kanter ba3b40ae79 YARN-3900. Protobuf layout of yarn_security_token causes errors in other protos that include it (adhoot via rkanter)
(cherry picked from commit 1d3026e7b3)
2015-07-23 14:47:28 -07:00
Wangda Tan 6baebcaedc YARN-3941. Proportional Preemption policy should try to avoid sending duplicate PREEMPT_CONTAINER event to scheduler. (Sunil G via wangda)
(cherry picked from commit 3bba180051)
2015-07-23 10:08:30 -07:00
Wangda Tan 3576b0ac5e YARN-3932. SchedulerApplicationAttempt#getResourceUsageReport and UserInfo should based on total-used-resources. (Bibin A Chundatt via wangda)
(cherry picked from commit 76ec26de80)
2015-07-22 11:54:38 -07:00
Wangda Tan 3dd113fa7c YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda)
(cherry picked from commit c39ca541f4)
2015-07-21 09:57:53 -07:00
Arun Suresh 8018041b49 YARN-3535. Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED (rohithsharma and peng.zhang via asuresh)
(cherry picked from commit 9b272ccae7)
2015-07-17 04:33:04 -07:00
Wangda Tan a7de3cde13 YARN-3885. ProportionalCapacityPreemptionPolicy doesn't preempt if queue is more than 2 level. (Ajith S via wangda)
(cherry picked from commit 3540d5fe4b)
2015-07-16 16:14:07 -07:00
Arun Suresh 5f58be7dd4 YARN-3453. Ensure preemption logic in FairScheduler uses DominantResourceCalculator in DRF queues to prevent unnecessary thrashing. (asuresh)
(cherry picked from commit ac94ba3e18)
2015-07-14 00:25:57 -07:00
Wangda Tan 87360be089 YARN-3894. RM startup should fail for wrong CS xml NodeLabel capacity configuration. (Bibin A Chundatt via wangda)
(cherry picked from commit 5ed1fead6b)
2015-07-12 21:53:08 -07:00
Wangda Tan c36090fd3f YARN-3849. Too much of preemption activity causing continuos killing of containers across queues. (Sunil G via wangda)
(cherry picked from commit 1df39c1efc)
2015-07-11 10:27:58 -07:00
Zhijie Shen 37a93c2d78 YARN-3116. RM notifies NM whether a container is an AM container or normal task container. Contributed by Giovanni Matteo Fumarola.
(cherry picked from commit 1ea36299a4)
2015-07-10 19:02:13 -07:00
Ming Ma b169889f01 YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma)
(cherry picked from commit 08244264c0)
2015-07-10 08:34:01 -07:00
carlo curino 753d87066f YARN-3800. Reduce storage footprint for ReservationAllocation. Contributed by Anubhav Dhoot.
(cherry picked from commit 0e602fa3a1)
2015-07-09 16:52:47 -07:00
Wangda Tan d61dd10b50 YARN-3508. Prevent processing preemption events on the main RM dispatcher. (Varun Saxena via wangda)
(cherry picked from commit 0e4b06690f)
2015-07-01 17:32:57 -07:00
Devaraj K c343250657 YARN-3830. AbstractYarnScheduler.createReleaseCache may try to clean a
null attempt. Contributed by nijel.

(cherry picked from commit 80a68d6056)
2015-07-01 19:05:27 +05:30
Xuan 1a8d162bc4 YARN-2871. TestRMRestart#testRMRestartGetApplicationList sometime fails
in trunk. Contributed by zhihai xu

(cherry picked from commit fe6c1bd73a)
2015-06-26 19:45:00 -07:00
Xuan 86b75ac544 YARN-3802. Two RMNodes for the same NodeId are used in RM sometimes
after NM is reconnected. Contributed by zhihai xu

(cherry picked from commit 5b5bb8dcdc)
2015-06-18 14:38:52 -07:00
Xuan 16d2412a25 YARN-3804. Both RM are on standBy state when kerberos user not in yarn.admin.acl. Contributed by Varun Saxena
(cherry picked from commit a826d432f9)
2015-06-17 16:24:50 -07:00
Devaraj K 3863342ca9 YARN-3794. TestRMEmbeddedElector fails because of ambiguous LOG reference.
Contributed by Chengbing Liu.

(cherry picked from commit d8dcfa98e3)
2015-06-12 13:45:05 +05:30
Xuan 8a6c7d9973 YARN-3785. Support for Resource as an argument during submitApp call in
MockRM test class. Contributed by Sunil G

(cherry picked from commit 5583f88bf7)
2015-06-10 21:41:58 -07:00
Jian He a24ead8c6d YARN-2716. Refactor ZKRMStateStore retry code with Apache Curator. Contributed by Karthik Kambatla
(cherry picked from commit 960b8f19ca)
2015-06-08 14:51:25 -07:00
Karthik Kambatla 322e7d0d5e YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation and container reservation. (Zhihai Xu via kasha)
(cherry picked from commit bd69ea408f)
2015-06-07 11:38:14 -07:00
Xuan 4d385b48f0 YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong
(cherry picked from commit 3e000a919f)
2015-06-05 12:49:51 -07:00
Karthik Kambatla 429754de09 YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha)
(cherry picked from commit 75885852cc)
2015-06-05 09:42:12 -07:00
Jian He 6325e4b7dd YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. Contributed by Wangda Tan
(cherry picked from commit 6ad4e59cfc)
2015-06-04 10:53:24 -07:00
Wangda Tan 78d626fa89 YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda)
(cherry picked from commit ebd797c48f)
2015-06-04 10:23:33 -07:00