Commit Graph

1199 Commits

Author SHA1 Message Date
Wangda Tan 3efbde5378 YARN-4315. NaN in Queue percentage for cluster apps page. (Bibin A Chundatt via wangda)
(cherry picked from commit 561abb9fee)
2015-12-29 13:29:19 -08:00
Jian He c7cc9d6bac YARN-4417. Make RM and Timeline-server REST APIs more consistent. Contributed by Wangda Tan
(cherry picked from commit d0a22bae9b)
2015-12-28 15:54:14 -08:00
Karthik Kambatla 2f67645475 YARN-4156. TestAMRestart#testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler. (Anubhav Dhoot via kasha)
(cherry picked from commit 0af492b4bd)
2015-12-23 17:52:54 -08:00
rohithsharmaks 08ddb536eb YARN-4109. Exception on RM scheduler page loading with labels. (Mohammad Shahid Khan via rohithsharmaks)
(cherry picked from commit 8c180a13c8)
2015-12-23 09:16:58 +05:30
Arun Suresh 31893468b1 YARN-4477. FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling. (Tao Jie via asuresh)
(cherry picked from commit e88422df45)
2015-12-21 22:41:49 -08:00
Wangda Tan f16daa59a4 YARN-4454. NM to nodelabel mapping going wrong after RM restart. (Bibin A Chundatt via wangda)
(cherry picked from commit bc038b382c)
2015-12-21 11:30:37 -08:00
Jian He 1147b086d9 YARN-4164. Changed updateApplicationPriority API to return the updated application priority. Contributed by Rohith Sharma K S 2015-12-18 14:14:31 -08:00
Junping Du 6d2914a697 YARN-3226. UI changes for decommissioning node. Contributed by Sunil G.
(cherry picked from commit 1de56b0448)
2015-12-17 15:21:42 -08:00
Jason Lowe fad2c5b862 YARN-4461. Redundant nodeLocalityDelay log in LeafQueue. Contributed by Eric Payne
(cherry picked from commit 91828fef6b)
2015-12-16 23:23:19 +00:00
Wangda Tan 5060fd1840 YARN-4416. Deadlock due to synchronised get Methods in AbstractCSQueue. (Naganarasimha G R via wangda)
(cherry picked from commit 9b856d9787)
2015-12-16 13:29:08 -08:00
Wangda Tan 2f5435acbd YARN-4225. Add preemption status to yarn queue -status for capacity scheduler. (Eric Payne via wangda)
(cherry picked from commit 7faa406f27)
2015-12-16 13:29:02 -08:00
Wangda Tan 9f4a6ec663 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda)
(cherry picked from commit 79c41b1d83)
2015-12-16 13:28:56 -08:00
Junping Du 1a2ef845b5 YARN-4452. NPE when submit Unmanaged application. Contributed by Naganarasimha G R.
(cherry picked from commit 50bd067e1d)
2015-12-16 10:58:40 -08:00
Zhihai Xu be50df687b YARN-4440. FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time. Contributed by Lin Yiqun
(cherry picked from commit 2aaed10327)
2015-12-15 00:21:13 -08:00
Jian He 34e86d8d47 YARN-4403. (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period. Contributed by Junping Du 2015-12-14 13:52:04 -08:00
Wangda Tan c915a658c8 YARN-4418. AM Resource Limit per partition can be updated to ResourceUsage as well. (Sunil G via wangda)
(cherry picked from commit 07b0fb996a)
2015-12-14 11:25:00 -08:00
Wangda Tan a1a723fdff YARN-3946. Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message. (Naganarasimha G R via wangda)
(cherry picked from commit 6cb0af3c39)
2015-12-14 10:54:20 -08:00
Arun Suresh 0bf328984a YARN-4358 addendum patch to fix javadoc error
(cherry picked from commit 7fb212e5e6)
2015-12-12 22:25:59 -08:00
rohithsharmaks 81f8a2b008 YARN-4421. Remove dead code in RmAppImpl.RMAppRecoveredTransition. (Daniel Templeton via rohithsharmaks)
(cherry picked from commit a5e2e1ecb0)
2015-12-09 11:33:49 +05:30
Wangda Tan 7013f9d6cd YARN-4424. Fix deadlock in RMAppImpl. (Jian he via wangda)
(cherry picked from commit 7e4715186d)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-12-08 14:30:01 -08:00
Chris Douglas bfe4796802 YARN-4248. Followup patch adding asf-licence exclusions for json test files
(cherry picked from commit 9f50e13d5d)
2015-12-08 12:08:35 -08:00
= 4ac1564418 YARN-4248. REST API for submit/update/delete Reservations. (curino)
(cherry picked from commit c25a635459)
2015-12-07 13:38:38 -08:00
Jonathan Eagles 7c604c3d06 YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page (Eric Payne via jeagles)
(cherry picked from commit 4ff973f96a)
2015-12-07 15:05:15 -06:00
Xuan 7decba83ef YARN-4392. ApplicationCreatedEvent event time resets after RM
restart/failover. Contributed by Naganarasimha G R and Xuan Gong

(cherry picked from commit 4546c7582b)
2015-12-07 12:26:09 -08:00
Steve Loughran af14458de7 HADOOP-12321. Make JvmPauseMonitor an AbstractService. (Sunil G via Stevel) [includes HDFS-8947 MAPREDUCE-6462 and YARN-4072] 2015-12-06 17:42:56 +00:00
Arun Suresh 78a07e99dd YARN-4358. Reservation System: Improve relationship between SharingPolicy and ReservationAgent. (Carlo Curino via asuresh)
(cherry picked from commit 742632e346)
2015-12-05 21:30:48 -08:00
Jian He be404e1680 YARN-4405. Support node label store in non-appendable file system. Contributed by Wangda Tan
(cherry picked from commit 755dda8dd8)
2015-12-03 17:46:09 -08:00
Wangda Tan 51e448d295 YARN-4292. ResourceUtilization should be a part of NodeInfo REST API. (Sunil G via wangda)
(cherry picked from commit a2c3bfc8c1)
2015-12-03 14:30:25 -08:00
Jian He 23fe1abc02 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999). Contributed by Mohammad Shahid Khan and Varun Saxena
(cherry picked from commit 9f77ccad73)
2015-12-03 12:49:22 -08:00
Jian He 48cbde9083 YARN-4398. Remove unnecessary synchronization in RMStateStore. Contributed by Ning Ding 2015-12-02 11:08:00 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Tsuyoshi Ozawa 9e458c3e9e YARN-4387. Fix typo in FairScheduler log message. Contributed by Xin Wang.
(cherry picked from commit 28dfe721b8)
2015-11-24 22:10:31 +09:00
Karthik Kambatla 49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha)
(cherry picked from commit 52948bb20b)
2015-11-24 13:52:29 +05:30
Jian He b339a4b8d6 YARN-4349. Support CallerContext in YARN. Contributed by Wangda Tan
(cherry picked from commit 8676a118a1)
2015-11-23 17:20:24 -08:00
Jason Lowe 9d303edbfb YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations. Contributed by Varun Vasudev
(cherry picked from commit d36b6e045f)
2015-11-23 20:31:50 +00:00
Arun Suresh cbbdbe3cdd YARN-3454. Add efficient merge operation to RLESparseResourceAllocation (Carlo Curino via asuresh)
(cherry picked from commit da1016365a)
2015-11-21 10:01:10 -08:00
Wangda Tan cdca460ad4 YARN-3769. Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler. (Eric Payne via wangda)
(cherry picked from commit 2346fa3141)
2015-11-20 15:56:17 -08:00
Jason Lowe f3bca8bc1a YARN-4374. RM capacity scheduler UI rounds user limit factor. Contributed by Chang Li
(cherry picked from commit 060cdcbe5d)
2015-11-20 23:13:27 +00:00
Arun Suresh d8e60080e9 YARN-4184. Remove update reservation state api from state store as its not used by ReservationSystem (Sean Po via asuresh)
(cherry picked from commit 6a61928fb7)
2015-11-17 15:51:59 -08:00
Jian He 2cf084ac76 Revert "YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan"
This reverts commit a4ff03b200.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-11-16 20:19:31 -08:00
Wangda Tan 13f8c7093b YARN-4347. Resource manager fails with Null pointer exception. (Jian He via wangda)
(cherry picked from commit 7f55a18071)
2015-11-12 11:24:04 -08:00
Wangda Tan 0c1fb15b29 YARN-4287. Capacity Scheduler: Rack Locality improvement (Nathan Roberts via wangda)
(cherry picked from commit 796638d9bc)
2015-11-12 11:12:11 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) bb5dd7f772 YARN-4183. Reverting the patch to fix behaviour change.
Revert "YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)"

This reverts commit 42bc525acb.
2015-11-11 10:42:52 -08:00
Jian He a4ff03b200 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan
(cherry picked from commit 8fbea531d7)
2015-11-09 11:05:12 -08:00
Jian He 2b197bbffd YARN-4127. RM fail with noAuth error if switched from failover to non-failover. Contributed by Varun Saxena
(cherry picked from commit e5b1733e04)
2015-10-29 15:43:26 -07:00
Jonathan Eagles 42bc525acb YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)
(cherry picked from commit c293c58954)
2015-10-29 16:42:49 -05:00
Arun Suresh 8943abfb08 YARN-4310. FairScheduler: Log skipping reservation messages at DEBUG level (asuresh)
(cherry picked from commit 58d1df585c)
2015-10-29 13:43:43 -07:00
Rohith Sharma K S e3fcb711d6 YARN-4130. Duplicate declaration of ApplicationId in RMAppManager#submitApplication method. (Kai Sasaki via rohithsharmaks)
(cherry picked from commit 656c8f9527)
2015-10-29 12:25:08 +05:30
Wangda Tan 1499af38eb YARN-3216. Max-AM-Resource-Percentage should respect node labels. (Sunil G via wangda)
(cherry picked from commit 56e4f6237a)
2015-10-26 16:45:20 -07:00
Wangda Tan 9bf1e74171 YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. (Naganarasimha G R via wangda)
(cherry picked from commit 6f606214e7)
2015-10-26 16:38:39 -07:00
Wangda Tan baa2998e93 YARN-4285. Display resource usage as percentage of queue and cluster in the RM UI (Varun Vasudev via wangda)
(cherry picked from commit 3cc73773eb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-10-26 13:12:56 -07:00
Jason Lowe 7d66e2e666 YARN-4284. condition for AM blacklisting is too narrow. Contributed by Sangjin Lee
(cherry picked from commit 33a03af3c3)
2015-10-26 19:54:16 +00:00
Rohith Sharma K S 33455fa415 YARN-2729. Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup. (Naganarasimha G R via rohithsharmaks)
(cherry picked from commit 5acdde4744)
2015-10-26 15:45:22 +05:30
Arun Suresh f0580dcac0 YARN-3738. Add support for recovery of reserved apps running under dynamic queues (subru via asuresh)
(cherry picked from commit ab8eb8770c)
2015-10-24 22:54:45 -07:00
Akira Ajisaka 356e922632 YARN-4294. [JDK8] Fix javadoc errors caused by wrong reference and illegal tag. (aajisaka)
(cherry picked from commit 7781fe1b9e)
2015-10-24 11:56:37 +09:00
Jason Lowe 235e84bf3d YARN-4041. Slow delegation token renewal can severely prolong RM recovery. Contributed by Sunil G
(cherry picked from commit d3a34a4f38)
2015-10-23 20:57:53 +00:00
Jonathan Eagles 6db7bfbcfd YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles)
(cherry picked from commit f8adeb712d)
2015-10-23 10:43:59 -05:00
Ming Ma 4bb7e68eb6 YARN-2913. Fair scheduler should have ability to set MaxResourceDefault for each queue. (Siqi Li via mingma)
(cherry picked from commit 934d96a334)
2015-10-23 08:37:46 -07:00
Junping Du d5145b58f6 YARN-4243. Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit. Contributed by Xuan Gong.
(cherry picked from commit 0fce5f9a49)
2015-10-22 13:42:12 -07:00
Zhihai Xu 9fe7ccc61c YARN-4256. YARN fair scheduler vcores with decimal values. Contributed by Jun Gong
(cherry picked from commit 960201b79b)
2015-10-22 12:31:28 -07:00
Anubhav Dhoot c44401f362 YARN-3739. Add reservation system recovery to RM recovery process. Contributed by Subru Krishnan.
(cherry picked from commit 2798723a54)
2015-10-22 06:52:41 -07:00
Arun Suresh f2009dc89c YARN-3985. Make ReservationSystem persist state using RMStateStore reservation APIs. (adhoot via asuresh)
(cherry picked from commit 506d1b1dbc)
2015-10-20 16:48:58 -07:00
Arun Suresh acc0e718d6 YARN-4270. Limit application resource reservation on nodes for non-node/rack specific requests (asuresh)
(cherry picked from commit 7e2837f830)
2015-10-19 20:02:10 -07:00
Jian He 1ba73169c7 YARN-4170. AM need to be notified with priority in AllocateResponse. Contributed by Sunil G
(cherry picked from commit f9da5cdb2b)
2015-10-16 15:26:46 -07:00
Wangda Tan 204ee6a40f YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API. (Naganarasimha G R via wangda)
(cherry picked from commit 4337b263aa)
2015-10-16 15:10:08 -07:00
Jian He 2862057f11 YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart. Contributed by Varun Saxena
(cherry picked from commit cf23f2c2b5)
2015-10-15 17:13:09 -07:00
rohithsharmaks 526be695c0 YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy Battula via rohithsharmaks)
(cherry picked from commit d6c8bad869)
2015-10-14 16:13:26 +05:30
Jian He ee1ce393f5 YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. Contributed by Meng Ding
(cherry picked from commit 9849c8b386)
2015-10-12 11:52:04 -07:00
Zhihai Xu 9988b57e71 YARN-4201. AMBlacklist does not work for minicluster. Contributed by Jun Gong.
(cherry picked from commit 049c6e8dc0)
2015-10-12 00:16:40 -07:00
Devaraj K 28edc7b129 YARN-3964. Support NodeLabelsProvider at Resource Manager side.
Contributed by Dian Fu.

(cherry picked from commit db93047881)
2015-10-11 11:24:05 +05:30
Wangda Tan 37642abb83 YARN-4140. RM container allocation delayed incase of app submitted to Nodelabel partition. (Bibin A Chundatt via wangda)
(cherry picked from commit def374e666)
2015-10-09 16:40:10 -07:00
Karthik Kambatla 2d8f9e3fd9 MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha)
(cherry picked from commit 4aa9b3e75c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
2015-10-09 07:54:47 -07:00
Jason Lowe 0139517811 YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S
(cherry picked from commit a0bca2b5ad)
2015-10-09 14:19:41 +00:00
Rohith Sharma K S 21609e8e9a YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned for a user. (Anubhav Dhoot via rohithsharmaks)
(cherry picked from commit 8f195387a4)
2015-10-09 10:11:53 +05:30
Rohith Sharma K S 99126714a3 YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks)
(cherry picked from commit 3793cbe4c3)
2015-10-07 10:20:36 +05:30
Rohith Sharma K S 053fcf4c0d YARN-4209. RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks)
(cherry picked from commit 9156fc60c6)
2015-10-07 09:37:14 +05:30
Wangda Tan 5453a63612 YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request. (Naganarasimha G R via wangda)
(cherry picked from commit 29a582ada0)
2015-10-06 11:59:10 -07:00
Harsh J 21b4ba48ce HADOOP-12458. Retries is typoed to spell Retires in parts of hadoop-yarn and hadoop-common. Contributed by Neelesh Srinivas Salian.
(cherry picked from commit c918f7be5e)
2015-10-03 18:39:22 +05:30
Xuan 53bddc410f YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma
(cherry picked from commit 8f08532bde)
2015-10-02 18:53:04 -07:00
Karthik Kambatla 64a7617eb9 YARN-4066. Large number of queues choke fair scheduler. (Johan Gustavsson via kasha)
(cherry picked from commit a0b5a0a419)
2015-09-29 08:13:25 -07:00
Anubhav Dhoot 22f2501476 YARN-4180. AMLauncher does not retry on failures when talking to NM. (adhoot)
(cherry picked from commit 9735afe967)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
2015-09-28 16:54:06 -07:00
Jason Lowe 600022ae69 YARN-4141. Runtime Application Priority change should not throw exception for applications at finishing states. Contributed by Sunil G
(cherry picked from commit 9f53a95ff6)
2015-09-28 22:56:07 +00:00
Anubhav Dhoot 4d8b99423e YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot)
(cherry picked from commit fb2e525c07)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-09-28 09:39:52 -07:00
Jian He ea2b6ee9e9 YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan
(cherry picked from commit b3f6b641dc)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-09-24 10:30:49 -07:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
2015-09-24 10:30:04 -07:00
Jian He fa0a554ae0 YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding
(cherry picked from commit 5f5a968d65)
2015-09-24 10:23:46 -07:00
Jian He c19e8ffdc7 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan)
(cherry picked from commit 83a18add10)
2015-09-24 10:23:25 -07:00
Rohith Sharma K S fbb1d923f4 YARN-4044. Running applications information changes such as movequeue is not published to TimeLine server. (Sunil G via rohithsharmaks) 2015-09-24 12:17:55 +05:30
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang.
(cherry picked from commit dfd807afab)
2015-09-22 12:54:44 +09:00
Rohith Sharma K S 52f343598e YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks)
(cherry picked from commit c9cb6a5960)
2015-09-21 10:06:39 +05:30
Arun Suresh b70c7b0e51 YARN-3920. FairScheduler container reservation on a node should be configurable to limit it to large containers (adhoot via asuresh)
(cherry picked from commit 94dec5a916)
2015-09-18 14:04:03 -07:00
Wangda Tan 4a657e9326 YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. (Junping Du via wangda)
(cherry picked from commit 9bc913a35c)
2015-09-18 10:05:04 -07:00
Rohith Sharma K S d94c092ad1 YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state.(Nijel S F via rohithsharmaks)
(cherry picked from commit 723c31d45b)
2015-09-18 08:46:36 +05:30
Jian He 5d2f85021f YARN-4034. Render cluster Max Priority in scheduler metrics in RM web UI. Contributed by Rohith Sharma K S
(cherry picked from commit 6c6e734f0b)
2015-09-17 14:56:32 +08:00
Jian He 9eda3ce3fa YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. Contributed by Naganarasimha G R
(cherry picked from commit 452079af8b)
2015-09-16 15:01:23 +08:00
Wangda Tan c31fe79979 YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda)
(cherry picked from commit ae5308fe1d)
2015-09-15 11:41:57 -07:00
Junping Du df714e25aa YARN-313. Add Admin API for supporting node resource configuration in command line. (Contributed by Inigo Goiri, Kenji Kikushima and Junping Du)
(cherry picked from commit 73e3a49eb0)
2015-09-15 07:58:47 -07:00
Jian He eacc18677a YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan
(cherry picked from commit 5468baa80a)
2015-09-15 19:02:17 +08:00
Jian He b705d9d0ac YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt
(cherry picked from commit e1b1d7e4ae)
2015-09-14 14:15:04 +08:00
Karthik Kambatla 1f9f219062 YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. (Zhihai Xu via kasha)
(cherry picked from commit 332b520a48)
2015-09-13 18:08:03 -07:00