Commit Graph

1357 Commits

Author SHA1 Message Date
Junping Du ea4cab2330 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev.
(cherry picked from commit d0162f2040)
2016-06-19 17:45:57 -07:00
Karthik Kambatla 25be8105cd YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha)
(cherry picked from commit 20f2799938)
2016-06-17 22:25:04 -07:00
Karthik Kambatla e9c1155233 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit. Addendum to fix javac warning. (Arun Suresh via kasha)
(cherry picked from commit fbbe0bb627)
2016-06-17 22:13:18 -07:00
Wangda Tan 863bfa4d6a YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:12:00 -07:00
Rohith Sharma K S c540b9bd7b YARN-4989. TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently. Contributed by Ajith S.
(cherry picked from commit 28b66ae919)
2016-06-13 11:23:10 +05:30
Arun Suresh 5985221b46 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh)
(cherry picked from commit 5279af7cd4)
2016-06-10 22:34:33 -07:00
Rohith Sharma K S efdfa68911 YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe.
(cherry picked from commit e0f4620cc7)
2016-06-11 10:31:00 +05:30
Wangda Tan a4094ddb5a YARN-5208. Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled. (Rohith Sharma K S via wangda)
(cherry picked from commit 244506f9c8)
2016-06-10 09:37:35 -07:00
Wangda Tan aef6e455b1 YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-06-09 10:08:31 -07:00
Arun Suresh d3655f4123 YARN-4525. Fix bug in RLESparseResourceAllocation.getRangeOverlapping(). (Ishai Menache and Carlo Curino via asuresh)
(cherry picked from commit 3a154f75ed)
2016-06-06 21:18:53 -07:00
Arun Suresh f0a869b52a YARN-5185. StageAllocaterGreedyRLE: Fix NPE in corner case. (Carlo Curino via asuresh)
(cherry picked from commit 7a9b7372a1)
2016-06-06 21:07:24 -07:00
Ming Ma 74e2b5efa2 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma)
(cherry picked from commit 4a1cedc010)
2016-06-06 14:34:47 -07:00
Arun Suresh 8ebd89dcfb YARN-5165. Fix NoOvercommitPolicy to take advantage of RLE representation of plan. (Carlo Curino via asuresh)
(cherry picked from commit db54670e83)
2016-06-03 14:50:51 -07:00
Vinod Kumar Vavilapalli 029888871f YARN-5098. Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time. Contributed by Jian He.
Made one minor edit for branch-2 patch.

(cherry picked from commit f10ebc67f5)
2016-06-03 13:11:17 -07:00
Jian He a022776c2e YARN-1815. Work preserving recovery of Unmanged AMs. Contributed by Subru Krishnan
(cherry picked from commit 097baaaeba)
2016-06-03 10:49:54 -07:00
Arun Suresh 384a84828f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh)
(cherry picked from commit dc26601d8f)
2016-06-02 09:14:30 -07:00
Varun Vasudev 4f36c3d214 YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:55:39 +05:30
Arun Suresh 0b1472fbc0 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh)
(cherry picked from commit aa975bc781)
2016-05-27 14:07:15 -07:00
Rohith Sharma K S ef3e0c6fd6 YARN-5005. TestRMWebServices#testDumpingSchedulerLogs fails randomly. Contributed by Bibin A Chundatt.
(cherry picked from commit 0a544f8a3e)
2016-05-27 10:48:47 +05:30
Karthik Kambatla 59335b4d7a YARN-5035. FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on. (kasha)
(cherry picked from commit 04ded558b0)
2016-05-26 15:09:47 -07:00
Arun Suresh 44cbf5b7f5 YARN-5162. Fix Exceptions thrown during in registerAM call when Distributed Scheduling is Enabled (Hitesh Sharma via asuresh)
(cherry picked from commit 5b41b288d0)
2016-05-26 14:58:15 -07:00
Karthik Kambatla 999cdd91b7 YARN-4866. FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy. (Yufei Gu via kasha)
(cherry picked from commit 4f513a4a8e)
2016-05-25 22:13:42 -07:00
Carlo Curino c656977961 YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino)
(cherry picked from commit 013532a95e)
2016-05-25 16:58:19 -07:00
Rohith Sharma K S b9567b89b9 YARN-5024. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers random failure. Contributed by Bibin A Chundatt
(cherry picked from commit 28bd63e92b)
2016-05-25 10:24:59 +05:30
Naganarasimha 02323603e2 YARN-5114. Add additional tests in TestRMWebServicesApps and rectify testInvalidAppAttempts failure in 2.8. Contributed by Bibin A Chundatt
(cherry picked from commit edd716e99c)
2016-05-25 06:15:50 +08:00
Karthik Kambatla 108db38788 YARN-4878. Expose scheduling policy and max running apps over JMX for Yarn queues. (Yufei Gu via kasha)
(cherry picked from commit f979d779e1)
2016-05-24 10:54:33 -07:00
Naganarasimha 155f25061e YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt
(cherry picked from commit b4078bd17b)
2016-05-24 08:14:53 +08:00
Karthik Kambatla f33f0b697b YARN-4979. FSAppAttempt demand calculation considers demands at multiple locality levels different. (Zhihai Xu via kasha)
(cherry picked from commit 6d043aa4cf)
2016-05-23 14:30:05 -07:00
Jason Lowe 7602070457 YARN-5055. max apps per user can be larger than max per queue. Contributed by Eric Badger
(cherry picked from commit ac954486c5)
2016-05-23 15:56:00 +00:00
Junping Du 2703ec6871 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 2016-05-19 14:10:15 -07:00
Jian He 2d846fdcbd YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong
(cherry picked from commit feb90ffcca)
2016-05-19 13:02:05 -07:00
Arun Suresh 15808717f3 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 1597630681)
2016-05-18 23:20:54 -07:00
Arun Suresh a555a320e8 YARN-5090. Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster. (asuresh)
(cherry picked from commit 8a9ecb7584)
2016-05-18 23:20:36 -07:00
Arun Suresh f31602893e YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh)
(cherry picked from commit ccc93e7812)
2016-05-18 23:20:08 -07:00
Arun Suresh bbcea3abf3 YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh)
(cherry picked from commit f45bc5a83e)
2016-05-18 23:12:19 -07:00
Arun Suresh 69a9ce3494 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh)
(cherry picked from commit f0ac18d001)
2016-05-18 23:09:11 -07:00
Arun Suresh 1b4600abcf YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
(cherry picked from commit 341888a0aa)
2016-05-18 22:35:55 -07:00
Arun Suresh d80d24aabf YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
(cherry picked from commit c282a08f38)
2016-05-18 22:08:55 -07:00
Jian He 264c06a438 YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 15:02:14 -07:00
Eric Payne 42b2a34ed8 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger.
(cherry picked from commit 1217c8f6b4)
2016-05-16 20:38:10 +00:00
Rohith Sharma K S a37b3694ea YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-16 15:25:15 +05:30
Jason Lowe bb95a5e18f YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger
(cherry picked from commit 013000fbc2)
2016-05-12 20:30:10 +00:00
Karthik Kambatla f96b2b6fac YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha)
(cherry picked from commit 4b4e4c6ba8)
2016-05-11 17:36:42 -07:00
Junping Du 9dc3683d87 YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong.
(cherry picked from commit 39f2bac38b)
2016-05-11 09:33:16 -07:00
Naganarasimha 3732a1e985 YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 2750fb900f)
2016-05-08 23:02:07 +05:30
Yongjun Zhang 5172d0e7b1 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang)
(cherry picked from commit 47c41e7ac7)
2016-05-06 22:38:56 -07:00
Jason Lowe 3895058a67 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena
(cherry picked from commit b2ed6ae731)
2016-05-06 23:00:25 +00:00
Wangda Tan b68e6b1d6d getApplicationReport call may raise NPE for removed queues. (Jian He via wangda)
(cherry picked from commit 23248f63aa)
2016-05-06 15:32:15 -07:00
Jian He a6b24c62ab YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan
(cherry picked from commit bb62e05925)
2016-05-05 12:56:55 -07:00
Jason Lowe ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit d0da13229c)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Rohith Sharma K S 5aad4070b2 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt
(cherry picked from commit 75e0450593)
2016-05-04 10:26:25 +05:30
Jason Lowe baac4e7db1 YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts
(cherry picked from commit ed54f5f1ff)
2016-05-03 22:16:17 +00:00
Jian He a9707dceaf YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:28:03 -07:00
Karthik Kambatla 864ecb4434 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha)
(cherry picked from commit 185c3d4de1)
2016-04-27 09:43:42 -07:00
Jian He 9d3ddb0b4d YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode
(cherry picked from commit 4beff01354)
2016-04-26 21:02:04 -07:00
Karthik Kambatla a5edb45b18 YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled
(cherry picked from commit 4b1dcbbe0c)
2016-04-26 05:10:29 -07:00
Wangda Tan 45ff579bfa YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda)
(cherry picked from commit 7cb3a3da96)
2016-04-22 11:41:08 -07:00
Eric Payne a7f903b2ba YARN-4556. TestFifoScheduler.testResourceOverCommit fails. Contributed by Akihiro Suda
(cherry picked from commit 3dce486d88)
2016-04-21 21:27:10 +00:00
Li Lu 054fa104c5 YARN-4968. A couple of AM retry unit tests need to wait SchedulerApplicationAttempt stopped. (Wangda Tan via gtcarrera9)
(cherry picked from commit 7c6339f66a)
2016-04-21 13:27:47 -07:00
Karthik Kambatla 75cf238354 YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha)
(cherry picked from commit 170c4fd4cd)
2016-04-20 23:58:30 -07:00
Wangda Tan 83a5cdc400 YARN-4890. Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate. (Sunil G via wangda)
(cherry picked from commit 33fd95a99c)
2016-04-20 17:38:22 -07:00
Wangda Tan 41cafeb5a1 YARN-4934. Reserved Resource for QueueMetrics needs to be handled correctly in few cases. (Sunil G via wangda)
(cherry picked from commit fdc46bfb37)
2016-04-16 22:50:00 -07:00
Jason Lowe cd148cb347 YARN-4940. yarn node -list -all failed if RM start with decommissioned node. Contributed by sandflee
(cherry picked from commit 69f3d428d5)
2016-04-15 20:38:04 +00:00
Jason Lowe f1a370ce8b Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
This reverts commit 814ceeb489.
2016-04-11 15:56:29 +00:00
Akira Ajisaka 8cf6630fc6 YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit 1ff27f9d12)
2016-04-11 14:55:37 +09:00
Karthik Kambatla 94a88ae87b YARN-4927. TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler. (Bibin A Chundatt via kasha)
(cherry picked from commit ff95fd547b)
2016-04-09 10:31:29 -07:00
Wangda Tan 12ccdd6540 YARN-3215. Respect labels in CapacityScheduler when computing headroom. (Naganarasimha G R via wangda)
(cherry picked from commit ec06957941)
2016-04-08 15:34:24 -07:00
Jian He 77a75de319 YARN-4740. AM may not receive the container complete msg when it restarts. Contributed by Jun Gong 2016-04-08 11:21:07 -07:00
Jian He 42bc565630 YARN-4769. Add support for CSRF header in the dump capacity scheduler logs and kill app buttons in RM web UI. Contributed by Varun Vasudev 2016-04-06 16:14:13 -07:00
Wangda Tan 11e796b5cd YARN-4699. Scheduler UI and REST o/p is not in sync when -replaceLabelsOnNode is used to change label of a node. (Sunil G via wangda)
(cherry picked from commit 21eb428448)
2016-04-05 16:25:55 -07:00
Junping Du eeff2e35f8 YARN-4893. Fix some intermittent test failures in TestRMAdminService. Contributed by Brahma Reddy Battula.
(cherry picked from commit 6be28bcc46)
2016-04-05 07:05:06 -07:00
Jason Lowe 814ceeb489 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit 1cbcd4a491)
2016-04-05 13:41:18 +00:00
Rohith Sharma K S 13a4e25f26 YARN-4609. RM Nodes list page takes too much time to load. Contributed by Bibin A Chundatt
(cherry picked from commit 776b549e2a)
2016-04-05 14:53:24 +05:30
Rohith Sharma K S eec23580b4 YARN-4880. Running TestZKRMStateStorePerf with real zookeeper cluster throws NPE. Contributed by Sunil G
(cherry picked from commit 552237d4a3)
2016-04-05 14:37:31 +05:30
naganarasimha 3772602848 YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 5092c94195)
2016-04-04 18:08:18 +05:30
Rohith Sharma K S c8271cd117 YARN-4607. Pagination support for AppAttempt page TotalOutstandingResource Requests table. Contributed by Bibin A Chundatt
(cherry picked from commit 1e6f92977d)
2016-04-04 08:13:03 +05:30
Allen Wittenauer 92a3dbe44f YARN-4850. test-fair-scheduler.xml isn't valid xml (Yufei Gu via aw)
(cherry picked from commit b1394d6307)
2016-04-01 16:57:31 -07:00
Robert Kanter 633f612d67 YARN-4639. Remove dead code in TestDelegationTokenRenewer added in YARN-3055 (templedf via rkanter)
(cherry picked from commit 7a021471c3)
2016-03-31 15:47:44 -07:00
Wangda Tan d36d9d676d YARN-4634. Scheduler UI/Metrics need to consider cases like non-queue label mappings. (Sunil G via wangda)
(cherry picked from commit 12b11e2e68)
2016-03-31 14:35:59 -07:00
Jian He f1f441b80f YARN-4822. Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates. Contributed by Wangda Tan 2016-03-30 12:46:36 -07:00
Wangda Tan 6856a7183a YARN-4865. Track Reserved resources in ResourceUsage and QueueCapacities. (Sunil G via wangda)
(cherry picked from commit fc055a3cbe)
2016-03-29 17:10:17 -07:00
Jian He c7d843af3b YARN-998. Keep NM resource updated through dynamic resource config for RM/NM restart. Contributed by Junping Du 2016-03-28 11:13:02 -07:00
Karthik Kambatla 4212f2e2bf YARN-4805. Don't go through all schedulers in ParameterizedTestBase. (kasha)
(cherry picked from commit 49ff54c860)
2016-03-26 21:45:29 -07:00
Junping Du c722262c75 YARN-4820. ResourceManager web redirects in HA mode drops query parameters. Contributed by Varun Vasudev.
(cherry picked from commit 19b645c938)
2016-03-23 19:35:14 -07:00
Junping Du 66257613b4 YARN-4785. inconsistent value type of the type field for LeafQueueInfo in response of RM REST API.
(cherry picked from commit ca8106d2dd)
2016-03-17 09:25:36 -07:00
Karthik Kambatla bbe9bb078c YARN-4812. TestFairScheduler#testContinuousScheduling fails intermittently. (kasha)
(cherry picked from commit f84af8bd58)
2016-03-17 05:54:40 -07:00
Wangda Tan 484976fa2b YARN-4108. CapacityScheduler: Improve preemption to only kill containers that would satisfy the incoming request. (Wangda Tan)
(cherry picked from commit 7e8c9beb41)
(cherry picked from commit ae14e5d07f)
2016-03-16 17:03:35 -07:00
Karthik Kambatla ab03266831 YARN-4560. Make scheduler error checking message more user friendly. (Ray Chiang via kasha)
(cherry picked from commit 3ef5500783)
2016-03-15 23:47:19 -07:00
Karthik Kambatla b4c8693096 YARN-4719. Add a helper library to maintain node state and allows common queries. (kasha)
(cherry picked from commit 20d389ce61)
2016-03-14 14:22:21 -07:00
Wangda Tan f7b38a7fb8 YARN-4465. SchedulerUtils#validateRequest for Label check should happen only when nodelabel enabled. (Bibin A Chundatt via wangda)
(cherry picked from commit 0233d4e0ee)
2016-03-08 14:28:26 -08:00
Jian He fb139b0c40 YARN-4764. Application submission fails when submitted queue is not available in scheduler xml. Contributed by Bibin A Chundatt
(cherry picked from commit 3c33158d1c)
2016-03-08 13:12:33 -08:00
Varun Vasudev e9a0ffc7f1 YARN-4737. Add CSRF filter support in YARN. Contributed by Jonathan Maron.
(cherry picked from commit 43416187c07afb35e3267f94d0a41d8d3cfb5735)
2016-03-07 15:23:36 +05:30
Zhihai Xu 7ac7ca48b7 YARN-4761. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler. Contributed by Sangjin Lee
(cherry picked from commit e1ccc9622b)
2016-03-06 19:49:47 -08:00
Rohith Sharma K S 1415e6190a YARN-4763. RMApps Page crashes with NPE. (Bibin A Chundatt via rohithsharmaks)
(cherry picked from commit 77e2c3e8c7365b2aca00b6169829f87c63e4b460)
2016-03-05 13:07:20 +05:30
Jian He 023c2d2e56 YARN-4671. There is no need to acquire CS lock when completing a container. Contributed by Meng Ding 2016-03-01 13:14:51 -08:00
Karthik Kambatla 84172b047b YARN-4704. TestResourceManager#testResourceAllocation() fails when using FairScheduler. (Yufei Gu via kasha)
(cherry picked from commit 9dafaaaf0d)
2016-02-29 16:10:26 -08:00
Haohui Mai c5db4ab0b4 HADOOP-12813. Migrate TestRPC and related codes to rebase on ProtobufRpcEngine. Contributed by Kai Zheng. 2016-02-29 14:10:18 -08:00
Karthik Kambatla f3b37d8020 YARN-4718. Rename variables in SchedulerNode to reduce ambiguity post YARN-1011. (Inigo Goiri via kasha)
(cherry picked from commit f9692770a5)
2016-02-28 10:01:48 -08:00
Jason Lowe 0bd7ba4ea8 YARN-4723. NodesListManager$UnknownNodeId ClassCastException. Contributed by Kuhu Shukla
(cherry picked from commit 6b0f813e89)
2016-02-26 20:25:56 +00:00
Karthik Kambatla 6a75c5af09 YARN-4729. SchedulerApplicationAttempt#getTotalRequiredResources can throw an NPE. (kasha)
(cherry picked from commit c684f2b007)
2016-02-24 18:34:21 -08:00