Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,977 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1357 Commits

Author SHA1 Message Date
Sangjin Lee 432a2367ce YARN-4722. AsyncDispatcher logs redundant event queue sizes (Jason Lowe via sjlee) 
(cherry picked from commit 553b591ba0)
 2016-02-24 09:30:37 -08:00
Junping Du e3ce0ffdc3 YARN-3223. Resource update during NM graceful decommission. Contributed by Brook Zhou. 
(cherry picked from commit 9ed17f181d)
 2016-02-23 03:35:47 -08:00
Tsuyoshi Ozawa 4ee55d0322 YARN-4648. Move preemption related tests from TestFairScheduler to TestFairSchedulerPreemption. Contributed by Kai Sasaki. 
(cherry picked from commit 0e12114c9c)
 2016-02-23 19:50:40 +09:00
Junping Du 7402448387 YARN-4386. refreshNodesGracefully() should send recommission event to active RMNodes only. Contributed by Kuhu Shukla. 
(cherry picked from commit 3fab88540f)
 2016-02-22 07:38:40 -08:00
Sangjin Lee 23b5c71729 YARN-4690. Skip object allocation in FSAppAttempt#getResourceUsage when possible (Ming Ma via sjlee) 
(cherry picked from commit 7de70680fe)
 2016-02-17 21:05:10 -08:00
Karthik Kambatla 33c1988595 YARN-4689. FairScheduler: Cleanup preemptContainer to be more readable. (Kai Sasaki via kasha) 
(cherry picked from commit 2ab4c476ed)
 2016-02-17 18:28:08 -08:00
Arun Suresh 6a238e4a6b YARN-2575. Create separate ACLs for Reservation create/update/delete/list ops (Sean Po via asuresh) 
(cherry picked from commit 23f937e3b7)
 2016-02-11 10:49:11 -08:00
Varun Vasudev 3a5b39e931 YARN-4655. Log uncaught exceptions/errors in various thread pools in YARN. Contributed by Sidharta Seethana. 
(cherry picked from commit fa00d3e205)
 2016-02-11 12:09:29 +05:30
Jian He 358cfcf9ea YARN-4138. Roll back container resource allocation after resource increase token expires. Contributed by Meng Ding 2016-02-11 10:09:04 +08:00
= fc9eb4577c YARN-4420. Add REST API for List Reservations. (Sean Po via curino) 2016-02-10 14:26:38 -08:00
Arun Suresh 17f83c8803 YARN-4360. Improve GreedyReservationAgent to support "early" allocations, and performance improvements (curino via asuresh) 
(cherry picked from commit 5cf5c41a89)
 2016-02-10 09:13:04 -08:00
Devaraj K 329df98f49 YARN-4667. RM Admin CLI for refreshNodesResources throws NPE when nothing 
is configured. Contributed by Naganarasimha G R.

(cherry picked from commit 565af873d5)
 2016-02-08 15:03:11 +05:30
Varun Vasudev be04535f1a YARN-4669. Fix logging statements in resource manager's Application class. Contributed by Sidharta Seethana. 
(cherry picked from commit a4a30c789146c4e824fcc6ebcf10405711120eaa)
 2016-02-04 13:51:08 +05:30
Varun Vasudev 99966b1912 YARN-4307. Display blacklisted nodes for AM container in the RM web UI. Contributed by Naganarasimha G R. 
(cherry picked from commit 308d63f382)
 2016-02-04 13:33:38 +05:30
Varun Vasudev e211ab777e YARN-4625. Make ApplicationSubmissionContext and ApplicationSubmissionContextInfo more consistent. Contributed by Xuan Gong. 
(cherry picked from commit 1adb64e09b)
 2016-02-03 16:28:31 +05:30
Wangda Tan 9591363ad8 YARN-4340. Add list API to reservation system. (Sean Po via wangda) 
(cherry picked from commit 9875325d5c)
 2016-02-02 10:19:19 +08:00
Jason Lowe 36aae8050e YARN-3102. Decommisioned Nodes not listed in Web UI. Contributed by Kuhu Shukla 
(cherry picked from commit ed55950164)
 2016-02-01 23:16:18 +00:00
Rohith Sharma K S 676f215595 YARN-4615. Fix random test failure in TestAbstractYarnScheduler#testResourceRequestRecoveryToTheRightAppAttempt. (Sunil G via rohithsharmaks) 
(cherry picked from commit 2673cbaf55)
 2016-02-01 10:45:05 +05:30
Jason Lowe 016c685135 YARN-4428. Redirect RM page to AHS page when AHS turned on and RM page is not available. Contributed by Chang Li 
(cherry picked from commit 772ea7b41b)
 2016-01-29 21:54:56 +00:00
Jian He 885c76d85f YARN-4617. LeafQueue#pendingOrderingPolicy should always use fixed ordering policy instead of using same as active applications ordering policy. Contributed by Rohith Sharma K S 2016-01-29 12:22:44 -08:00
Devaraj K 123269c07f YARN-4411. RMAppAttemptImpl#createApplicationAttemptReport throws 
IllegalArgumentException. Contributed by Bibin A Chundatt and yarntime.

(cherry picked from commit a277bdc9ed)
 2016-01-29 13:52:47 +05:30
Jian He 36f0e73882 YARN-4519. Potential deadlock of CapacityScheduler between decrease container and assign containers. Contributed by Meng Ding 2016-01-28 14:52:00 -08:00
Rohith Sharma K S 6947a5d4be YARN-4633. Fix random test failure in TestRMRestart#testRMRestartAfterPreemption. (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit ef343be82b)
 2016-01-28 21:55:12 +05:30
Karthik Kambatla c25154576b YARN-4462. FairScheduler: Disallow preemption from a queue. (Tao Jie via kasha) 
(cherry picked from commit fb238d7e5d)
 2016-01-27 12:29:25 -08:00
Rohith Sharma K S 0b8767e01a YARN-4573. Fix test failure in TestRMAppTransitions#testAppRunningKill and testAppKilledKilled. (Takashi Ohnishi via rohithsharmaks) 
(cherry picked from commit c01bee0108)
 2016-01-27 08:24:37 +05:30
rohithsharmaks 7e66b2f22c YARN-4613. Fix test failure in TestClientRMService#testGetClusterNodes. (Takashi Ohnishi via rohithsharmaks) 
(cherry picked from commit 10dc2c0493)
 2016-01-24 23:37:11 +05:30
rohithsharmaks 280c9b8401 YARN-4614. Fix random failure in TestApplicationPriority#testApplicationPriorityAllocationWithChangeInPriority. (Sunil G via rohithsharmaks) 
(cherry picked from commit 99829eb221)
 2016-01-23 07:58:16 +05:30
rohithsharmaks 493275b27b YARN-4497. RM might fail to restart when recovering apps whose attempts are missing. (Jun Gong via rohithsharmaks) 
(cherry picked from commit d6258b33a7)
 2016-01-22 20:29:10 +05:30
Akira Ajisaka ddc22bbcb1 YARN-4605. Spelling mistake in the help message of "yarn applicationattempt" command. Contributed by Weiwei Yang. 
(cherry picked from commit 8f58f742ae)
 2016-01-22 19:43:37 +09:00
Rohith Sharma K S 844a8e3771 YARN-4584. RM startup failure when AM attempts greater than max-attempts. (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit e30668106d)
 2016-01-22 10:20:09 +05:30
Jason Lowe d051200a62 YARN-4610. Reservations continue looking for one app causes other apps to starve. Contributed by Jason Lowe 
(cherry picked from commit 468a53b22f)
 2016-01-21 18:32:47 +00:00
Karthik Kambatla 5c2f495fcc YARN-4603. FairScheduler should mention user requested queuename in error message when failed in queue ACL check. (Tao Jie via kasha) 
(cherry picked from commit 4992398aee)
 2016-01-21 17:43:49 +01:00
Wangda Tan 9512866740 YARN-4557. Fix improper Queues sorting in PartitionedQueueComparator when accessible-node-labels=*. (Naganarasimha G R via wangda) 
(cherry picked from commit c9fe0371a8ab40c0d3437fadd32788b320788a7a)
 2016-01-21 11:16:58 +08:00
Xuan be3322792f YARN-4559. Make leader elector and zk store share the same curator 
client. Contributed by Jian He

(cherry picked from commit 890a2ebd1a)
 2016-01-20 14:49:04 -08:00
Jian He bb2a9a78c0 YARN-4565. Fix a bug that leads to AM resource limit not hornored when sizeBasedWeight enabled for FairOrderingPolicy. Contributed by Wangda Tan 2016-01-18 21:05:03 -08:00
Wangda Tan 704b37d0e1 YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit a44ce3f14f)
 2016-01-19 09:31:13 +08:00
Wangda Tan 1625281ccb Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe5728563.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 150f5ae034)
 2016-01-19 09:31:07 +08:00
Jian He 6db022fc48 YARN-4596. SystemMetricPublisher should not swallow error messages from TimelineClient#putEntities. Contributed by Li Lu 2016-01-18 16:59:04 -08:00
Karthik Kambatla ef59521094 YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha) 
(cherry picked from commit d40859fab1)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
 2016-01-18 10:59:34 +01:00
Wangda Tan 552f15739c YARN-4304. AM max resource configuration per partition to be displayed/updated correctly in UI and in various partition related metrics. (Sunil G via wangda) 
(cherry picked from commit b08ecf5c75)
 2016-01-18 11:16:04 +08:00
Wangda Tan 6dab78c71a YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85e)
 2016-01-18 11:15:59 +08:00
Wangda Tan c1193b46d4 YARN-4538. QueueMetrics pending cores and memory metrics wrong. (Bibin A Chundatt via wangda) 
(cherry picked from commit 9523648d57)
 2016-01-18 11:15:53 +08:00
rohithsharmaks d3c08cde68 YARN-4389. Allow application to enable or disable am blacklisting. (Sunil G via rohithsharmaks) 2016-01-15 22:46:21 +05:30
Karthik Kambatla f0923819c3 YARN-3446. FairScheduler headroom calculation should exclude nodes in the blacklist. (Zhihai Xu via kasha) 
(cherry picked from commit 9d04f26d4c)
 2016-01-14 08:33:52 -08:00
Karthik Kambatla a1ff0b4145 YARN-4551. Address the duplication between StatusUpdateWhenHealthy and StatusUpdateWhenUnhealthy transitions. (Sunil G via kasha) 
(cherry picked from commit 321072ba81)
 2016-01-13 12:09:58 -08:00
Wangda Tan c0b97865e8 YARN-4571. Make app id/name available to the yarn authorizer provider for better auditing. (Jian He via wangda) 
(cherry picked from commit c0537bcd2c)
 2016-01-13 13:19:01 +08:00
Akira Ajisaka 93b7ef8aec YARN-4567. javadoc failing on java 8. Contributed by Steve Loughran. This closes #67. 
(cherry picked from commit da1e3e3c57)
 2016-01-12 15:13:38 +09:00
Wangda Tan 15e174fe79 YARN-4582. Label-related invalid resource request exception should be able to properly handled by application. (Bibin A Chundatt via wangda) 
(cherry picked from commit 9e792da014)
 2016-01-12 12:54:09 +08:00
Jian He c4aed35200 Missing file for YARN-4580 2016-01-11 17:05:11 -08:00
Jian He 591e7fb221 YARN-4537. Pull out priority comparison from fifocomparator and use compound comparator for FifoOrdering policy. Contributed by Rohith Sharma K S 
(cherry picked from commit b8942be888)
 2016-01-11 16:45:14 -08:00
Jian He d4ff8fd5d8 YARN-4479. Change CS LeafQueue pendingOrderingPolicy to hornor recovered apps. Contributed by Rohith Sharma K S 
(cherry picked from commit 109e528ef5)
 2016-01-08 15:51:33 -08:00
Xuan 2cbbf76c3d YARN-4438. Implement RM leader election with curator. Contributed by Jian He 
(cherry picked from commit 89022f8d4b)
 2016-01-07 14:34:14 -08:00
Junping Du 1cc001db4c YARN-4546. ResourceManager crash due to scheduling opportunity overflow. Contributed by Jason Lowe. 
(cherry picked from commit c1462a67ff)
 2016-01-06 05:51:28 -08:00
rohithsharmaks 0492552f30 YARN-4535. Fix checkstyle error in CapacityScheduler.java (Naganarasimha G R via rohithsharmaks) 
(cherry picked from commit 6da6d87872)
 2016-01-05 12:11:48 +05:30
Wangda Tan 05fa852d75 YARN-4524. Cleanup AppSchedulingInfo. (Karthik Kambatla via wangda) 2015-12-30 15:36:55 -08:00
Wangda Tan 9a7d5f69ec YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) 
(cherry picked from commit 8310b2e9ff)
 2015-12-30 15:31:11 -08:00
Junping Du 401d1e6e8a YARN-1382. Remove unusableRMNodesConcurrentSet (never used) in NodeListManager to get rid of memory leak. Contributed by Rohith Sharma K S. 
(cherry picked from commit 223ce323bb)
 2015-12-30 07:53:08 -08:00
Jian He 3661d4907a YARN-3480. Remove attempts that are beyond max-attempt limit from state store. Contributed by Jun Gong 
(cherry picked from commit 5273413411)
 2015-12-29 15:59:42 -08:00
Wangda Tan 3efbde5378 YARN-4315. NaN in Queue percentage for cluster apps page. (Bibin A Chundatt via wangda) 
(cherry picked from commit 561abb9fee)
 2015-12-29 13:29:19 -08:00
Jian He c7cc9d6bac YARN-4417. Make RM and Timeline-server REST APIs more consistent. Contributed by Wangda Tan 
(cherry picked from commit d0a22bae9b)
 2015-12-28 15:54:14 -08:00
Karthik Kambatla 2f67645475 YARN-4156. TestAMRestart#testAMBlacklistPreventsRestartOnSameNode assumes CapacityScheduler. (Anubhav Dhoot via kasha) 
(cherry picked from commit 0af492b4bd)
 2015-12-23 17:52:54 -08:00
rohithsharmaks 08ddb536eb YARN-4109. Exception on RM scheduler page loading with labels. (Mohammad Shahid Khan via rohithsharmaks) 
(cherry picked from commit 8c180a13c8)
 2015-12-23 09:16:58 +05:30
Arun Suresh 31893468b1 YARN-4477. FairScheduler: Handle condition which can result in an infinite loop in attemptScheduling. (Tao Jie via asuresh) 
(cherry picked from commit e88422df45)
 2015-12-21 22:41:49 -08:00
Wangda Tan f16daa59a4 YARN-4454. NM to nodelabel mapping going wrong after RM restart. (Bibin A Chundatt via wangda) 
(cherry picked from commit bc038b382c)
 2015-12-21 11:30:37 -08:00
Jian He 1147b086d9 YARN-4164. Changed updateApplicationPriority API to return the updated application priority. Contributed by Rohith Sharma K S 2015-12-18 14:14:31 -08:00
Junping Du 6d2914a697 YARN-3226. UI changes for decommissioning node. Contributed by Sunil G. 
(cherry picked from commit 1de56b0448)
 2015-12-17 15:21:42 -08:00
Jason Lowe fad2c5b862 YARN-4461. Redundant nodeLocalityDelay log in LeafQueue. Contributed by Eric Payne 
(cherry picked from commit 91828fef6b)
 2015-12-16 23:23:19 +00:00
Wangda Tan 5060fd1840 YARN-4416. Deadlock due to synchronised get Methods in AbstractCSQueue. (Naganarasimha G R via wangda) 
(cherry picked from commit 9b856d9787)
 2015-12-16 13:29:08 -08:00
Wangda Tan 2f5435acbd YARN-4225. Add preemption status to yarn queue -status for capacity scheduler. (Eric Payne via wangda) 
(cherry picked from commit 7faa406f27)
 2015-12-16 13:29:02 -08:00
Wangda Tan 9f4a6ec663 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 
(cherry picked from commit 79c41b1d83)
 2015-12-16 13:28:56 -08:00
Junping Du 1a2ef845b5 YARN-4452. NPE when submit Unmanaged application. Contributed by Naganarasimha G R. 
(cherry picked from commit 50bd067e1d)
 2015-12-16 10:58:40 -08:00
Zhihai Xu be50df687b YARN-4440. FSAppAttempt#getAllowedLocalityLevelByTime should init the lastScheduler time. Contributed by Lin Yiqun 
(cherry picked from commit 2aaed10327)
 2015-12-15 00:21:13 -08:00
Jian He 34e86d8d47 YARN-4403. (AM/NM/Container)LivelinessMonitor should use monotonic time when calculating period. Contributed by Junping Du 2015-12-14 13:52:04 -08:00
Wangda Tan c915a658c8 YARN-4418. AM Resource Limit per partition can be updated to ResourceUsage as well. (Sunil G via wangda) 
(cherry picked from commit 07b0fb996a)
 2015-12-14 11:25:00 -08:00
Wangda Tan a1a723fdff YARN-3946. Update exact reason as to why a submitted app is in ACCEPTED state to app's diagnostic message. (Naganarasimha G R via wangda) 
(cherry picked from commit 6cb0af3c39)
 2015-12-14 10:54:20 -08:00
Arun Suresh 0bf328984a YARN-4358 addendum patch to fix javadoc error 
(cherry picked from commit 7fb212e5e6)
 2015-12-12 22:25:59 -08:00
rohithsharmaks 81f8a2b008 YARN-4421. Remove dead code in RmAppImpl.RMAppRecoveredTransition. (Daniel Templeton via rohithsharmaks) 
(cherry picked from commit a5e2e1ecb0)
 2015-12-09 11:33:49 +05:30
Wangda Tan 7013f9d6cd YARN-4424. Fix deadlock in RMAppImpl. (Jian he via wangda) 
(cherry picked from commit 7e4715186d)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-12-08 14:30:01 -08:00
Chris Douglas bfe4796802 YARN-4248. Followup patch adding asf-licence exclusions for json test files 
(cherry picked from commit 9f50e13d5d)
 2015-12-08 12:08:35 -08:00
= 4ac1564418 YARN-4248. REST API for submit/update/delete Reservations. (curino) 
(cherry picked from commit c25a635459)
 2015-12-07 13:38:38 -08:00
Jonathan Eagles 7c604c3d06 YARN-4422. Generic AHS sometimes doesn't show started, node, or logs on App page (Eric Payne via jeagles) 
(cherry picked from commit 4ff973f96a)
 2015-12-07 15:05:15 -06:00
Xuan 7decba83ef YARN-4392. ApplicationCreatedEvent event time resets after RM 
restart/failover. Contributed by Naganarasimha G R and Xuan Gong

(cherry picked from commit 4546c7582b)
 2015-12-07 12:26:09 -08:00
Steve Loughran af14458de7 HADOOP-12321. Make JvmPauseMonitor an AbstractService. (Sunil G via Stevel) [includes HDFS-8947 MAPREDUCE-6462 and YARN-4072] 2015-12-06 17:42:56 +00:00
Arun Suresh 78a07e99dd YARN-4358. Reservation System: Improve relationship between SharingPolicy and ReservationAgent. (Carlo Curino via asuresh) 
(cherry picked from commit 742632e346)
 2015-12-05 21:30:48 -08:00
Jian He be404e1680 YARN-4405. Support node label store in non-appendable file system. Contributed by Wangda Tan 
(cherry picked from commit 755dda8dd8)
 2015-12-03 17:46:09 -08:00
Wangda Tan 51e448d295 YARN-4292. ResourceUtilization should be a part of NodeInfo REST API. (Sunil G via wangda) 
(cherry picked from commit a2c3bfc8c1)
 2015-12-03 14:30:25 -08:00
Jian He 23fe1abc02 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999). Contributed by Mohammad Shahid Khan and Varun Saxena 
(cherry picked from commit 9f77ccad73)
 2015-12-03 12:49:22 -08:00
Jian He 48cbde9083 YARN-4398. Remove unnecessary synchronization in RMStateStore. Contributed by Ning Ding 2015-12-02 11:08:00 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Tsuyoshi Ozawa 9e458c3e9e YARN-4387. Fix typo in FairScheduler log message. Contributed by Xin Wang. 
(cherry picked from commit 28dfe721b8)
 2015-11-24 22:10:31 +09:00
Karthik Kambatla 49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 
(cherry picked from commit 52948bb20b)
 2015-11-24 13:52:29 +05:30
Jian He b339a4b8d6 YARN-4349. Support CallerContext in YARN. Contributed by Wangda Tan 
(cherry picked from commit 8676a118a1)
 2015-11-23 17:20:24 -08:00
Jason Lowe 9d303edbfb YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations. Contributed by Varun Vasudev 
(cherry picked from commit d36b6e045f)
 2015-11-23 20:31:50 +00:00
Arun Suresh cbbdbe3cdd YARN-3454. Add efficient merge operation to RLESparseResourceAllocation (Carlo Curino via asuresh) 
(cherry picked from commit da1016365a)
 2015-11-21 10:01:10 -08:00
Wangda Tan cdca460ad4 YARN-3769. Consider user limit when calculating total pending resource for preemption policy in Capacity Scheduler. (Eric Payne via wangda) 
(cherry picked from commit 2346fa3141)
 2015-11-20 15:56:17 -08:00
Jason Lowe f3bca8bc1a YARN-4374. RM capacity scheduler UI rounds user limit factor. Contributed by Chang Li 
(cherry picked from commit 060cdcbe5d)
 2015-11-20 23:13:27 +00:00
Arun Suresh d8e60080e9 YARN-4184. Remove update reservation state api from state store as its not used by ReservationSystem (Sean Po via asuresh) 
(cherry picked from commit 6a61928fb7)
 2015-11-17 15:51:59 -08:00
Jian He 2cf084ac76 Revert "YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan" 
This reverts commit a4ff03b200.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-11-16 20:19:31 -08:00
Wangda Tan 13f8c7093b YARN-4347. Resource manager fails with Null pointer exception. (Jian He via wangda) 
(cherry picked from commit 7f55a18071)
 2015-11-12 11:24:04 -08:00
Wangda Tan 0c1fb15b29 YARN-4287. Capacity Scheduler: Rack Locality improvement (Nathan Roberts via wangda) 
(cherry picked from commit 796638d9bc)
 2015-11-12 11:12:11 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) bb5dd7f772 YARN-4183. Reverting the patch to fix behaviour change. 
Revert "YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles)"

This reverts commit 42bc525acb.
 2015-11-11 10:42:52 -08:00
Jian He a4ff03b200 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan 
(cherry picked from commit 8fbea531d7)
 2015-11-09 11:05:12 -08:00
Jian He 2b197bbffd YARN-4127. RM fail with noAuth error if switched from failover to non-failover. Contributed by Varun Saxena 
(cherry picked from commit e5b1733e04)
 2015-10-29 15:43:26 -07:00
Jonathan Eagles 42bc525acb YARN-4183. Enabling generic application history forces every job to get a timeline service delegation token (jeagles) 
(cherry picked from commit c293c58954)
 2015-10-29 16:42:49 -05:00
Arun Suresh 8943abfb08 YARN-4310. FairScheduler: Log skipping reservation messages at DEBUG level (asuresh) 
(cherry picked from commit 58d1df585c)
 2015-10-29 13:43:43 -07:00
Rohith Sharma K S e3fcb711d6 YARN-4130. Duplicate declaration of ApplicationId in RMAppManager#submitApplication method. (Kai Sasaki via rohithsharmaks) 
(cherry picked from commit 656c8f9527)
 2015-10-29 12:25:08 +05:30
Wangda Tan 1499af38eb YARN-3216. Max-AM-Resource-Percentage should respect node labels. (Sunil G via wangda) 
(cherry picked from commit 56e4f6237a)
 2015-10-26 16:45:20 -07:00
Wangda Tan 9bf1e74171 YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. (Naganarasimha G R via wangda) 
(cherry picked from commit 6f606214e7)
 2015-10-26 16:38:39 -07:00
Wangda Tan baa2998e93 YARN-4285. Display resource usage as percentage of queue and cluster in the RM UI (Varun Vasudev via wangda) 
(cherry picked from commit 3cc73773eb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-10-26 13:12:56 -07:00
Jason Lowe 7d66e2e666 YARN-4284. condition for AM blacklisting is too narrow. Contributed by Sangjin Lee 
(cherry picked from commit 33a03af3c3)
 2015-10-26 19:54:16 +00:00
Rohith Sharma K S 33455fa415 YARN-2729. Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup. (Naganarasimha G R via rohithsharmaks) 
(cherry picked from commit 5acdde4744)
 2015-10-26 15:45:22 +05:30
Arun Suresh f0580dcac0 YARN-3738. Add support for recovery of reserved apps running under dynamic queues (subru via asuresh) 
(cherry picked from commit ab8eb8770c)
 2015-10-24 22:54:45 -07:00
Akira Ajisaka 356e922632 YARN-4294. [JDK8] Fix javadoc errors caused by wrong reference and illegal tag. (aajisaka) 
(cherry picked from commit 7781fe1b9e)
 2015-10-24 11:56:37 +09:00
Jason Lowe 235e84bf3d YARN-4041. Slow delegation token renewal can severely prolong RM recovery. Contributed by Sunil G 
(cherry picked from commit d3a34a4f38)
 2015-10-23 20:57:53 +00:00
Jonathan Eagles 6db7bfbcfd YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles) 
(cherry picked from commit f8adeb712d)
 2015-10-23 10:43:59 -05:00
Ming Ma 4bb7e68eb6 YARN-2913. Fair scheduler should have ability to set MaxResourceDefault for each queue. (Siqi Li via mingma) 
(cherry picked from commit 934d96a334)
 2015-10-23 08:37:46 -07:00
Junping Du d5145b58f6 YARN-4243. Add retry on establishing Zookeeper conenction in EmbeddedElectorService#serviceInit. Contributed by Xuan Gong. 
(cherry picked from commit 0fce5f9a49)
 2015-10-22 13:42:12 -07:00
Zhihai Xu 9fe7ccc61c YARN-4256. YARN fair scheduler vcores with decimal values. Contributed by Jun Gong 
(cherry picked from commit 960201b79b)
 2015-10-22 12:31:28 -07:00
Anubhav Dhoot c44401f362 YARN-3739. Add reservation system recovery to RM recovery process. Contributed by Subru Krishnan. 
(cherry picked from commit 2798723a54)
 2015-10-22 06:52:41 -07:00
Arun Suresh f2009dc89c YARN-3985. Make ReservationSystem persist state using RMStateStore reservation APIs. (adhoot via asuresh) 
(cherry picked from commit 506d1b1dbc)
 2015-10-20 16:48:58 -07:00
Arun Suresh acc0e718d6 YARN-4270. Limit application resource reservation on nodes for non-node/rack specific requests (asuresh) 
(cherry picked from commit 7e2837f830)
 2015-10-19 20:02:10 -07:00
Jian He 1ba73169c7 YARN-4170. AM need to be notified with priority in AllocateResponse. Contributed by Sunil G 
(cherry picked from commit f9da5cdb2b)
 2015-10-16 15:26:46 -07:00
Wangda Tan 204ee6a40f YARN-4162. CapacityScheduler: Add resource usage by partition and queue capacity by partition to REST API. (Naganarasimha G R via wangda) 
(cherry picked from commit 4337b263aa)
 2015-10-16 15:10:08 -07:00
Jian He 2862057f11 YARN-4000. RM crashes with NPE if leaf queue becomes parent queue during restart. Contributed by Varun Saxena 
(cherry picked from commit cf23f2c2b5)
 2015-10-15 17:13:09 -07:00
rohithsharmaks 526be695c0 YARN-4250. NPE in AppSchedulingInfo#isRequestLabelChanged. (Brahma Reddy Battula via rohithsharmaks) 
(cherry picked from commit d6c8bad869)
 2015-10-14 16:13:26 +05:30
Jian He ee1ce393f5 YARN-4230. RM crashes with NPE when increasing container resource if there is no headroom left. Contributed by Meng Ding 
(cherry picked from commit 9849c8b386)
 2015-10-12 11:52:04 -07:00
Zhihai Xu 9988b57e71 YARN-4201. AMBlacklist does not work for minicluster. Contributed by Jun Gong. 
(cherry picked from commit 049c6e8dc0)
 2015-10-12 00:16:40 -07:00
Devaraj K 28edc7b129 YARN-3964. Support NodeLabelsProvider at Resource Manager side. 
Contributed by Dian Fu.

(cherry picked from commit db93047881)
 2015-10-11 11:24:05 +05:30
Wangda Tan 37642abb83 YARN-4140. RM container allocation delayed incase of app submitted to Nodelabel partition. (Bibin A Chundatt via wangda) 
(cherry picked from commit def374e666)
 2015-10-09 16:40:10 -07:00
Karthik Kambatla 2d8f9e3fd9 MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha) 
(cherry picked from commit 4aa9b3e75c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2015-10-09 07:54:47 -07:00
Jason Lowe 0139517811 YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S 
(cherry picked from commit a0bca2b5ad)
 2015-10-09 14:19:41 +00:00
Rohith Sharma K S 21609e8e9a YARN-4235. FairScheduler PrimaryGroup does not handle empty groups returned for a user. (Anubhav Dhoot via rohithsharmaks) 
(cherry picked from commit 8f195387a4)
 2015-10-09 10:11:53 +05:30
Rohith Sharma K S 99126714a3 YARN-4228. FileSystemRMStateStore use IOUtils#close instead of fs#close. (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit 3793cbe4c3)
 2015-10-07 10:20:36 +05:30
Rohith Sharma K S 053fcf4c0d YARN-4209. RMStateStore FENCED state doesn’t work due to updateFencedState called by stateMachine.doTransition. (Zhihai Xu via rohithsharmaks) 
(cherry picked from commit 9156fc60c6)
 2015-10-07 09:37:14 +05:30
Wangda Tan 5453a63612 YARN-4215. RMNodeLabels Manager Need to verify and replace node labels for the only modified Node Label Mappings in the request. (Naganarasimha G R via wangda) 
(cherry picked from commit 29a582ada0)
 2015-10-06 11:59:10 -07:00
Harsh J 21b4ba48ce HADOOP-12458. Retries is typoed to spell Retires in parts of hadoop-yarn and hadoop-common. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit c918f7be5e)
 2015-10-03 18:39:22 +05:30
Xuan 53bddc410f YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma 
(cherry picked from commit 8f08532bde)
 2015-10-02 18:53:04 -07:00
Karthik Kambatla 64a7617eb9 YARN-4066. Large number of queues choke fair scheduler. (Johan Gustavsson via kasha) 
(cherry picked from commit a0b5a0a419)
 2015-09-29 08:13:25 -07:00
Anubhav Dhoot 22f2501476 YARN-4180. AMLauncher does not retry on failures when talking to NM. (adhoot) 
(cherry picked from commit 9735afe967)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
 2015-09-28 16:54:06 -07:00
Jason Lowe 600022ae69 YARN-4141. Runtime Application Priority change should not throw exception for applications at finishing states. Contributed by Sunil G 
(cherry picked from commit 9f53a95ff6)
 2015-09-28 22:56:07 +00:00
Anubhav Dhoot 4d8b99423e YARN-4204. ConcurrentModificationException in FairSchedulerQueueInfo. (adhoot) 
(cherry picked from commit fb2e525c07)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-09-28 09:39:52 -07:00
Jian He ea2b6ee9e9 YARN-4171. Fix findbugs warnings in YARN-1197 branch. Contributed by Wangda Tan 
(cherry picked from commit b3f6b641dc)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-09-24 10:30:49 -07:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
 2015-09-24 10:30:04 -07:00
Jian He fa0a554ae0 YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding 
(cherry picked from commit 5f5a968d65)
 2015-09-24 10:23:46 -07:00
Jian He c19e8ffdc7 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan) 
(cherry picked from commit 83a18add10)
 2015-09-24 10:23:25 -07:00
Rohith Sharma K S fbb1d923f4 YARN-4044. Running applications information changes such as movequeue is not published to TimeLine server. (Sunil G via rohithsharmaks) 2015-09-24 12:17:55 +05:30
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab)
 2015-09-22 12:54:44 +09:00
Rohith Sharma K S 52f343598e YARN-4167. NPE on RMActiveServices#serviceStop when store is null. (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit c9cb6a5960)
 2015-09-21 10:06:39 +05:30
Arun Suresh b70c7b0e51 YARN-3920. FairScheduler container reservation on a node should be configurable to limit it to large containers (adhoot via asuresh) 
(cherry picked from commit 94dec5a916)
 2015-09-18 14:04:03 -07:00
Wangda Tan 4a657e9326 YARN-3212. RMNode State Transition Update with DECOMMISSIONING state. (Junping Du via wangda) 
(cherry picked from commit 9bc913a35c)
 2015-09-18 10:05:04 -07:00
Rohith Sharma K S d94c092ad1 YARN-4135. Improve the assertion message in MockRM while failing after waiting for the state.(Nijel S F via rohithsharmaks) 
(cherry picked from commit 723c31d45b)
 2015-09-18 08:46:36 +05:30
Jian He 5d2f85021f YARN-4034. Render cluster Max Priority in scheduler metrics in RM web UI. Contributed by Rohith Sharma K S 
(cherry picked from commit 6c6e734f0b)
 2015-09-17 14:56:32 +08:00
Jian He 9eda3ce3fa YARN-4078. Add getPendingResourceRequestForAttempt in YarnScheduler interface. Contributed by Naganarasimha G R 
(cherry picked from commit 452079af8b)
 2015-09-16 15:01:23 +08:00
Wangda Tan c31fe79979 YARN-3717. Expose app/am/queue's node-label-expression to RM web UI / CLI / REST-API. (Naganarasimha G R via wangda) 
(cherry picked from commit ae5308fe1d)
 2015-09-15 11:41:57 -07:00
Junping Du df714e25aa YARN-313. Add Admin API for supporting node resource configuration in command line. (Contributed by Inigo Goiri, Kenji Kikushima and Junping Du) 
(cherry picked from commit 73e3a49eb0)
 2015-09-15 07:58:47 -07:00
Jian He eacc18677a YARN-3635. Refactored current queue mapping implementation in CapacityScheduler to use a generic PlacementManager framework. Contributed by Wangda Tan 
(cherry picked from commit 5468baa80a)
 2015-09-15 19:02:17 +08:00
Jian He b705d9d0ac YARN-4126. RM should not issue delegation tokens in unsecure mode. Contributed by Bibin A Chundatt 
(cherry picked from commit e1b1d7e4ae)
 2015-09-14 14:15:04 +08:00
Karthik Kambatla 1f9f219062 YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. (Zhihai Xu via kasha) 
(cherry picked from commit 332b520a48)
 2015-09-13 18:08:03 -07:00
Karthik Kambatla 11e2fa151c YARN-2005. Blacklisting support for scheduling AMs. (Anubhav Dhoot via kasha) 
(cherry picked from commit 81df7b586a)
 2015-09-13 17:06:04 -07:00
Steve Loughran 988749e6ea HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:06 +01:00
Robert Kanter 43f051a09a YARN-4145. Make RMHATestBase abstract so its not run when running all tests under that namespace (adhoot via rkanter) 
(cherry picked from commit ea4bb2749f)
 2015-09-11 11:46:35 -07:00
Jian He f23be93dd1 YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil 
(cherry picked from commit 6f72f1e600)
 2015-09-08 11:24:07 +08:00
Xuan a0b7ef15d0 YARN-4087. Followup fixes after YARN-2019 regarding RM behavior when 
state-store error occurs. Contributed by Jian He

(cherry picked from commit 9b78e6e33d)
 2015-09-07 17:46:48 -07:00
Wangda Tan 9f97b86816 YARN-4024. YARN RM should avoid unnecessary resolving IP when NMs doing heartbeat. (Hong Zhiguo via wangda) 
(cherry picked from commit bcc85e3bab)
 2015-09-04 15:14:34 -07:00
Jason Lowe 6a5220cab3 YARN-4105. Capacity Scheduler headroom for DRF is wrong. Contributed by Chang Li 
(cherry picked from commit 6eaca2e363)
 2015-09-04 15:31:37 +00:00
Varun Vasudev 64c2e8011e YARN-4103. RM WebServices missing scheme for appattempts logLinks. Contributed by Jonathan Eagles. 
(cherry picked from commit 1b2e2ca9104dfc05e9aa7f93954ae2e00b3e62c4)
 2015-09-04 14:32:07 +05:30
Varun Vasudev a53054c853 YARN-3970. Add REST api support for Application Priority. Contributed by Naganarasimha G R. 
(cherry picked from commit 223a306da925ad031bc398f2ea5e56f0e6521edf)
 2015-09-03 16:40:42 +05:30
Jian He cd82fa2f83 YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager gets connection issue. Contributed by Xuan Gong 
(cherry picked from commit 09c64ba1ba)
 2015-09-02 17:46:04 -07:00
Rohith Sharma K S 6e41aa64a5 YARN-3893. Both RM in active state when Admin#transitionToActive failure from refeshAll() (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit 7d6687fe76)
 2015-09-02 15:27:33 +05:30
Varun Vasudev 855e0f8b00 YARN-4082. Container shouldn't be killed when node's label updated. Contributed by Wangda Tan. 
(cherry picked from commit bf669b6d9f)
 2015-09-01 14:20:54 +05:30
Jian He 48f5161cd5 YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. Contributed by Xuan Gong 
(cherry picked from commit a3fd2ccc86)
 2015-08-31 17:34:49 -07:00
Junping Du 792b9c0eaf YARN-1556. NPE getting application report with a null appId. Contributed by Weiwei Yang. 
(cherry picked from commit beb65c9465)
 2015-08-28 05:58:45 -07:00
Jian He 306c535395 YARN-3250. Support admin cli interface in for Application Priority. Contributed by Rohith Sharma K S 
(cherry picked from commit a9c8ea71aa)
 2015-08-27 13:27:34 -07:00
Jian He 387076894f YARN-4014. Support user cli interface in for Application Priority. Contributed by Rohith Sharma K S 
(cherry picked from commit 57c7ae1aff)
 2015-08-24 20:37:18 -07:00
Rohith Sharma K S 6dc732f2f7 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 
(cherry picked from commit feaf034994)
 2015-08-24 11:30:46 +05:30
Xuan 205b98c059 YARN-221. NM should provide a way for AM to tell it not to aggregate 
logs. Contributed by Ming Ma

(cherry picked from commit 37e1c3d82a)
 2015-08-22 16:26:39 -07:00
Rohith Sharma K S 27ced8253b YARN-3986. getTransferredContainers in AbstractYarnScheduler should be present in YarnScheduler interface 
(cherry picked from commit 22de7c1dca)
 2015-08-21 10:56:45 +05:30
Xuan fc6b176e70 YARN-4028. AppBlock page key update and diagnostics value null on 
recovery. Contributed by Bibin A Chundatt

(cherry picked from commit 22dc5fc209)
 2015-08-18 22:54:01 -07:00
Zhihai Xu 8f2eb99d71 YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. 
(cherry picked from commit 3a76a010b8)
 2015-08-18 10:53:16 -07:00
Jian He b2e869f2a8 YARN-3987. Send AM container completed msg to NM once AM finishes. Contributed by sandflee 
(cherry picked from commit 0a030546e2)
 2015-08-13 16:23:30 -07:00
Jian He 703fa1b141 YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe 
(cherry picked from commit 7a445fcfab)
 2015-08-13 16:13:28 -07:00
Jian He 65d22b3686 YARN-4026. Refactored ContainerAllocator to accept a list of priorites rather than a single priority. Contributed by Wangda Tan 
(cherry picked from commit e5003be907)
 2015-08-12 15:09:46 -07:00
rohithsharmaks c5b20e0db6 YARN-4023. Publish Application Priority to TimelineServer. (Sunil G via rohithsharmaks) 
(cherry picked from commit 1c12adb71f)
 2015-08-12 14:47:23 +05:30
Xuan 2ebdf5bfce YARN-3999. RM hangs on draing events. Contributed by Jian He 
(cherry picked from commit 3ae716fa69)
 2015-08-11 18:26:00 -07:00
Jian He df97e3265b YARN-3887. Support changing Application priority during runtime. Contributed by Sunil G 
(cherry picked from commit fa1d84ae27)
 2015-08-10 20:52:18 -07:00
Wangda Tan 380d401da5 YARN-3873. PendingApplications in LeafQueue should also use OrderingPolicy. (Sunil G via wangda) 
(cherry picked from commit cf9d3c9256)
 2015-08-10 14:58:19 -07:00
Wangda Tan cdb61b5fb2 YARN-3966. Fix excessive loggings in CapacityScheduler. (Jian He via wangda) 
(cherry picked from commit 4bc42d76e7)
 2015-08-07 09:53:42 -07:00
Rohith Sharma K S a0da1ec010 YARN-3948. Display Application Priority in RM Web UI.(Sunil G via rohithsharmaks) 
(cherry picked from commit b6265d39c5)
 2015-08-07 10:47:08 +05:30
Carlo Curino 8e23a7997f YARN-3974. Refactor the reservation system test cases to use parameterized base test. (subru via curino) 
(cherry picked from commit 8572a5a14b)
 2015-08-02 02:06:06 -07:00
Junping Du 7af531d2e3 YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter. 
(cherry picked from commit cfee02b3bd)
 2015-08-06 06:50:40 -07:00
Arun Suresh 84ea7cd408 YARN-3961. Expose pending, running and reserved containers of a queue in REST api and yarn top (adhoot via asuresh) 
(cherry picked from commit 154c9d2e42)
 2015-08-05 23:15:33 -07:00
rohithsharmaks dc76c4b035 YARN-3992. TestApplicationPriority.testApplicationPriorityAllocation fails intermittently. (Contributed by Sunil G) 
(cherry picked from commit df9e7280db)
 2015-08-06 10:49:37 +05:30
Jian He 1466772827 YARN-3983. Refactored CapacityScheduleri#FiCaSchedulerApp to easier extend container allocation logic. Contributed by Wangda Tan 
(cherry picked from commit ba2313d614)
 2015-08-05 13:51:08 -07:00
Arun Suresh 707b96fa58 YARN-3736. Add RMStateStore apis to store and load accepted reservations for failover (adhoot via asuresh) 
(cherry picked from commit f271d37735)
 2015-08-05 12:58:37 -07:00
Xuan 8126fde884 YARN-3543. ApplicationReport should be able to tell whether the 
Application is AM managed or not. Contributed by Rohith Sharma K S

(cherry picked from commit 0306d902f5)
 2015-08-03 15:47:21 -07:00
Jonathan Eagles 899df5bce0 YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) 
(cherry picked from commit 3cd02b9522)
 2015-08-03 10:40:04 -05:00
Jason Lowe c31e3ba921 YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt 
(cherry picked from commit 32e490b6c0)
 2015-07-31 17:38:49 +00:00
Zhihai Xu 0e2019fa30 YARN-433. When RM is catching up with node updates then it should not expire acquired containers. Contributed by Xuan Gong 
(cherry picked from commit ab80e27703)
 2015-07-30 22:03:04 -07:00
Wangda Tan 53a0653826 YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (Bibin A Chundatt via wangda) 
(cherry picked from commit 91b42e7d6e)
 2015-07-30 10:01:01 -07:00
Karthik Kambatla 8659315062 YARN-2768. Avoid cloning Resource in FSAppAttempt#updateDemand. (Hong Zhiguo via kasha) 
(cherry picked from commit 5205a330b3)
 2015-07-29 09:42:57 -07:00