Commit Graph

2962 Commits

Author SHA1 Message Date
Sunil G 46b7054fa7 YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 2017-08-07 13:15:46 +05:30
Junping Du f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh 35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Subru Krishnan 0542e6f86e YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 2017-08-03 14:18:03 -07:00
Yufei Gu c617fe02b3 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:33:37 -07:00
Haibo Chen 293c74a81b YARN-6673 Add cpu cgroup configurations for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:56:51 -07:00
Haibo Chen c5d256c760 YARN-6674 Add memory cgroup settings for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:52:35 -07:00
Sunil G f64cfeaf61 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2017-08-03 19:27:10 +05:30
Eric Payne 8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu 45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne 48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino)
(cherry picked from commit b6240b92ab)
2017-08-01 17:28:28 -07:00
Subru Krishnan 86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru).
(cherry picked from commit 40453879ec)
2017-08-01 17:28:28 -07:00
Subru Krishnan 8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase.
(cherry picked from commit 69e3ed2680)
2017-08-01 17:28:27 -07:00
Subru Krishnan 712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase.
(cherry picked from commit f1eff212c5)
2017-08-01 17:28:27 -07:00
Subru Krishnan 3e1dc7ece0 YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin.
(cherry picked from commit cd9db822f0)
2017-08-01 17:28:27 -07:00
Subru Krishnan 825108d08a YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not.
(cherry picked from commit 8820693cd7)
2017-08-01 17:28:27 -07:00
Subru Krishnan 43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 52daa6d971)
2017-08-01 17:28:26 -07:00
Subru Krishnan 4cfec943b1 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit f8e5de5969)
2017-08-01 17:28:26 -07:00
Subru Krishnan 70b1a757f1 YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru).
(cherry picked from commit 8c988d235e)
2017-08-01 17:28:26 -07:00
Subru Krishnan bed1832c93 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru).
(cherry picked from commit 2399eb8200)
2017-08-01 17:28:25 -07:00
Subru Krishnan 42949a6e01 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru).
(cherry picked from commit e750907d0a)
2017-08-01 17:28:25 -07:00
Subru Krishnan f317e0cb48 Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase.
(cherry picked from commit bd9c7b1bec)
2017-08-01 17:28:25 -07:00
Subru Krishnan 859aa1f9d6 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru).
(cherry picked from commit 73bb2102ce)
2017-08-01 17:28:24 -07:00
Subru Krishnan 7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 67846a5519)
2017-08-01 17:28:24 -07:00
Subru Krishnan 80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 4846069061)
2017-08-01 17:28:24 -07:00
Carlo Curino 5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino).
(cherry picked from commit be99c1fe2e)
2017-08-01 17:28:24 -07:00
Carlo Curino c4bbdfeb66 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino)
(cherry picked from commit 121d55bd29)
2017-08-01 17:28:23 -07:00
Subru Krishnan e1da8f0667 YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 326a2e6bde)
2017-08-01 17:28:23 -07:00
Carlo Curino 1c64e1709b YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino).
(cherry picked from commit ce419881c3)
2017-08-01 17:28:23 -07:00
Subru Krishnan 86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru)
(cherry picked from commit 51aeb2ce0c)
2017-08-01 17:28:23 -07:00
Carlo Curino 8623644f45 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino)
(cherry picked from commit 5c486961cd)
2017-08-01 17:28:22 -07:00
Subru Krishnan 193ec456d6 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 04f111394b)
2017-08-01 17:28:22 -07:00
Subru Krishnan 98b45b0ed3 YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru).
(cherry picked from commit 66500f4fa6)
2017-08-01 17:28:22 -07:00
Subru Krishnan 91803305e5 YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru).
(cherry picked from commit db26bade05)
2017-08-01 17:28:22 -07:00
Subru Krishnan 78032e51d9 YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP.
(cherry picked from commit e0f0e7afed)
2017-08-01 17:28:22 -07:00
Subru Krishnan 4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru).
(cherry picked from commit 575137f41c)
2017-08-01 17:28:21 -07:00
Subru Krishnan 083a05bc14 YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru).
(cherry picked from commit d7672ce2bd)
2017-08-01 17:28:21 -07:00
Subru Krishnan 93d9fdeca6 YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru).
(cherry picked from commit 20893682ec)
2017-08-01 17:28:21 -07:00
Subru Krishnan 1dadd0b45a YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru).
(cherry picked from commit 11c5336522)
2017-08-01 17:28:21 -07:00
Subru Krishnan 0662996b6a YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru).
(cherry picked from commit 1298127bda)
2017-08-01 17:28:21 -07:00
Subru Krishnan 81472778d7 YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru).
(cherry picked from commit f8208fe0b5)
2017-08-01 17:28:20 -07:00
Jian He 2797507d51 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan
(cherry picked from commit 9ca2aba9cc)
2017-08-01 17:28:20 -07:00
Subru Krishnan 1882bc10fb YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit ac1ba2a304)
2017-08-01 17:28:20 -07:00
Jian He 94a24567d6 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan
(cherry picked from commit 8573c286e2)
2017-08-01 17:28:20 -07:00
Jian He bdfad4523f YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan
(cherry picked from commit 3307564a5f)
2017-08-01 17:28:20 -07:00
Subru Krishnan cfafd173bd YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru)
(cherry picked from commit bd44182e70)
2017-08-01 17:28:19 -07:00
Jian He 5c84382397 YARN-3672. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan
(cherry picked from commit 5e976cd2b9)
2017-08-01 17:28:19 -07:00
Subru Krishnan 52558df620 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru)
(cherry picked from commit e4f928cf9c)
2017-08-01 17:28:19 -07:00
Subru Krishnan 0733088c29 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru)
(cherry picked from commit b747d59f41)
2017-08-01 17:28:19 -07:00
Subru Krishnan e0c3a44396 YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru).
(cherry picked from commit 268389d021)
2017-08-01 17:28:19 -07:00
Subru Krishnan f1a508bdef YARN-3664. Federation PolicyStore internal APIs
(cherry picked from commit 2f377263a1)
2017-08-01 17:28:18 -07:00
Subru Krishnan 047a16bf15 YARN-5307. Federation Application State Store internal APIs
(cherry picked from commit 1af5292042)
2017-08-01 17:28:18 -07:00
Subru Krishnan d19b677301 YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui.
(cherry picked from commit d3dc461a93)
2017-08-01 17:28:18 -07:00
Subru Krishnan dcf8c52d80 YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui.
(cherry picked from commit 20d1d2be91)
2017-08-01 17:28:18 -07:00
Subru Krishnan 3e22896f86 YARN-3662. Federation Membership State Store internal APIs.
(cherry picked from commit 5424d0899c)
2017-08-01 17:28:17 -07:00
Jian He 91f120f743 YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 09:56:33 -07:00
Akira Ajisaka b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Akira Ajisaka a4aa1cb405
Revert "YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang."
This reverts commit 1a78c0ff01.
2017-08-01 12:15:03 +09:00
Akira Ajisaka 1a78c0ff01
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-01 10:53:32 +09:00
Akira Ajisaka f8bed5e9a7
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 2017-07-31 11:09:13 +09:00
Arun Suresh 890e14c02a YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 2017-07-28 16:32:43 -07:00
Subru Krishnan bcde66bed1 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-07-28 15:44:36 -07:00
Yufei Gu 9ea01fd956 YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-07-28 09:52:53 -07:00
Yufei Gu 9902be72cb YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 2017-07-27 23:19:39 -07:00
Akira Ajisaka 38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena 11ece0bda1 Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-28 00:10:45 +05:30
Varun Saxena f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Daniel Templeton f81a4efb8c YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 2017-07-25 13:00:31 -07:00
Jian He ac9489f7fc YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-25 09:57:14 -07:00
Akira Ajisaka cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Akira Ajisaka 218b1b33ff
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 2017-07-25 15:12:08 +09:00
Daniel Templeton 10583625c1 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue.
(Contributed by Yufei Gu via Daniel Templeton)
2017-07-24 13:44:00 -07:00
Daniel Templeton bb30bd3771 YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting
(Contributed by Yeliang Cang via Daniel Templeton)
2017-07-24 12:13:50 -07:00
Sunil G e315328428 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 11:39:03 +05:30
Brahma Reddy Battula 2054324d47 Revert "YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi"
This reverts commit 4a771d9010.
2017-07-23 12:56:18 +08:00
Jian He 4a771d9010 YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-21 11:31:33 -07:00
Jason Lowe c8df3668ec YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 2017-07-20 11:03:04 -05:00
Arun Suresh 077fcf6a96 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 2017-07-19 12:26:40 -07:00
Rohith Sharma K S 845c4e52bd YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 2017-07-19 11:10:52 +05:30
Yufei Gu daaf530fce YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 2017-07-18 16:38:07 -07:00
Ray Chiang f5f14a2ad6 YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 2017-07-18 12:35:08 -07:00
Sunil G 0b7afc060c Addendum patch for YARN-5731 2017-07-18 11:49:09 +05:30
Arun Suresh 5b007921cd YARN-6706. Refactor ContainerScheduler to make oversubscription change easier. (Haibo Chen via asuresh) 2017-07-17 14:11:14 -07:00
Yufei Gu 9e0cde1469 YARN-6625. yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment. (Yufei Gu) 2017-07-14 14:10:45 -07:00
Jason Lowe a5ae5ac50e YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 2017-07-14 14:56:00 -05:00
bibinchundatt 75c0220b44 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 2017-07-14 13:53:39 +05:30
Yufei Gu 4a574e9a84 YARN-6769. Make schedulables without demand less needy in FairSharePolicy#compare. (Yunfan Zhou via Yufei Gu) 2017-07-13 23:10:10 -07:00
Sunil G 228ddaa31d YARN-6792. Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo. Contributed by Giovanni Matteo Fumarola. 2017-07-14 08:07:05 +05:30
Jason Lowe ebc048cc05 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:44:47 -05:00
Jason Lowe 0ffca5d347 Revert "YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe"
This reverts commit f76f5c0919.
2017-07-13 17:42:38 -05:00
Jason Lowe f76f5c0919 YARN-6805. NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code. Contributed by Jason Lowe 2017-07-13 17:38:17 -05:00
Jason Lowe 5f1ee72b0e YARN-6654. RollingLevelDBTimelineStore backwards incompatible after fst upgrade. Contributed by Jonathan Eagles 2017-07-13 17:27:40 -05:00
Wangda Tan 945c0958bb YARN-6775. CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls. (Nathan Roberts via wangda)
Change-Id: I84ccd54200ccbaae23018ef320028e42b4c3509a
2017-07-13 10:30:15 -07:00
Sunil G cf0d0844d6 YARN-5731. Preemption calculation is not accurate when reserved containers are present in queue. Contributed by Wangda Tan. 2017-07-13 16:48:29 +05:30
Xuan 9845bea59c YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:40:45 -07:00
Sunil G 34f113df5c YARN-6714. IllegalStateException while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. Contributed by Tao Yang. 2017-07-11 14:52:44 +05:30
Arun Suresh 5496a34c0c YARN-6776. Refactor ApplicaitonMasterService to move actual processing logic to a separate class. (asuresh) 2017-07-10 14:34:58 -07:00
Steve Loughran 09653ea098
HADOOP-14634. Remove jline from main Hadoop pom.xml.
Contributed by Ray Chiang.
2017-07-10 11:53:13 +01:00
Yufei Gu 3de47ab5ea YARN-6764. Simplify the logic in FairScheduler#attemptScheduling. Contributed by Yufei Gu. 2017-07-09 16:09:12 -07:00
Naganarasimha 626d730bfc YARN-6428. Queue AM limit is not honored in CS always. Contributed by Bibin A Chundatt. 2017-07-09 17:26:09 +05:30
Daniel Templeton 9d27880442 YARN-6410. FSContext.scheduler should be final (Contributed by Yeliang Cang via Daniel Templeton) 2017-07-09 19:34:35 +09:00
Daniel Templeton c5a0c38d8e YARN-6746. SchedulerUtils.checkResourceRequestMatchingNodePartition() is dead code
(Contributed by Deepti Sawhney via Daniel Templeton)
2017-07-09 19:27:32 +09:00
Daniel Templeton 0615985886 YARN-6752. Display reserved resources in web UI per application
(Contributed by Abdullah Yousufi via Daniel Templeton)
2017-07-09 19:02:31 +09:00
Jason Lowe 7576a688ea YARN-6708. Nodemanager container crash after ext3 folder limit. Contributed by Bibin A Chundatt 2017-07-06 09:40:09 -05:00
Naganarasimha fa1aaee87b YARN-6681. Eliminate double-copy of child queues in canAssignToThisQueue. Contributed by Daryn Sharp. 2017-07-01 12:29:39 +05:30
Andrew Wang af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Daniel Templeton ec97519779 YARN-6751. Display reserved resources in web UI per queue
(Contributed by Abdullah Yousufi via Daniel Templeton)
2017-06-29 16:53:50 -07:00
Sunil G c1edca101c YARN-6280. Introduce deselect query param to skip ResourceRequest from getApp/getApps REST API. Contributed by Lantao Jin. 2017-06-28 15:40:58 -07:00
Jason Lowe 63ce1593c5 YARN-6738. LevelDBCacheTimelineStore should reuse ObjectMapper instances. Contributed by Zoltan Haindrich 2017-06-27 17:12:42 -05:00
Yufei Gu 0b77262890 YARN-5876. TestResourceTrackerService#testGracefulDecommissionWithApp fails intermittently on trunk. (Robert Kanter via Yufei Gu) 2017-06-23 13:26:03 -07:00
Sunil G ca13b224b2 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne. 2017-06-22 23:50:57 -07:00
Naganarasimha 740204b292 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-23 07:52:41 +05:30
Arun Suresh 49aa60e50d YARN-6127. Add support for work preserving NM restart when AMRMProxy is enabled. (Botong Huang via asuresh). 2017-06-22 11:27:13 -07:00
Akira Ajisaka 099dfe92a9
YARN-6713. Fix dead link in the Javadoc of FairSchedulerEventLog.java. Contributed by Weiwei Yang. 2017-06-20 17:20:27 +09:00
Naganarasimha ee89ac84e6 YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp 2017-06-19 22:42:45 +05:30
Naganarasimha e9c2aa1bc3 YARN-6467. CSQueueMetrics needs to update the current metrics for default partition only. Contributed by Manikandan R. 2017-06-19 21:31:40 +05:30
bibinchundatt d64c842743 YARN-6703. RM startup failure with old state store due to version mismatch. Contributed by Varun Saxena 2017-06-12 18:07:43 +05:30
Eric Payne 5578af8603 YARN-6585. RM fails to start when upgrading from 2.7 for clusters with node labels. Contributed by Sunil G. 2017-06-09 16:16:34 -05:00
Naganarasimha 99634d10ed YARN-6682. Improve performance of AssignmentInformation datastructures. Contributed by Daryn Sharp. 2017-06-09 11:10:13 +05:30
Daniel Templeton a062374c39 YARN-6679. Reduce Resource instance overhead via non-PBImpl
(Contributed by Daryn Sharp via Daniel Templeton)
2017-06-08 10:08:08 -07:00
Carlo Curino 547cb53e8a YARN-6634. [API] Refactor ResourceManager WebServices to make API explicit. (Giovanni Matteo Fumarola via curino) 2017-06-07 13:41:06 -07:00
Varun Saxena 0b7bff706e YARN-6604. Allow metric TTL for Application table to be specified through cmd (Haibo Chen via Varun Saxena) 2017-06-07 21:51:07 +05:30
Jian He 7311015ace YARN-6683. Invalid event: COLLECTOR_UPDATE at KILLED. Contributed by Rohith Sharma K S 2017-06-05 13:16:57 -07:00
Daniel Templeton 73ecb19312 YARN-6208. Improve the log when FinishAppEvent sent to the NodeManager which didn't run the application
(Contributed by Akira Ajisaka via Daniel Templeton)
2017-06-02 08:50:19 -07:00
vrushali d95c82cb79 YARN-6316 Provide help information and documentation for TimelineSchemaCreator (Contributed by Haibo Chen via Vrushali C) 2017-06-01 18:30:23 -07:00
Daniel Templeton d5b71e4175 YARN-6246. Identifying starved apps does not need the scheduler writelock
(Contributed by Karthik Kambatla via Daniel Templeton)
2017-05-31 15:48:04 -07:00
Nathan Roberts 4369690ce6 YARN-6649. RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception. Contributed by Jon Eagles. 2017-05-31 11:32:32 -05:00
Jonathan Eagles cbfed0e82f YARN-6497. Method length of ResourceManager#serviceInit() is too long (Gergely Novák via jeagles) 2017-05-31 10:18:42 -05:00
Varun Vasudev 547f18cb96 YARN-6366. Refactor the NodeManager DeletionService to support additional DeletionTask types. Contributed by Shane Kumpf. 2017-05-31 16:15:35 +05:30
Andrew Wang 16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Daniel Templeton d81372dfad YARN-6646. Modifier 'static' is redundant for inner enums
(Contributed by ZhangBing Lin via Daniel Templeton)
2017-05-26 12:05:48 -07:00
Jason Lowe aea42930bb YARN-6641. Non-public resource localization on a bad disk causes subsequent containers failure. Contributed by Kuhu Shukla 2017-05-26 09:37:56 -05:00
Haibo Chen 47474fffac YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen) 2017-05-25 21:15:27 -07:00
Yufei Gu 87590090c8 YARN-6582. FSAppAttempt demand can be updated atomically in updateDemand(). (Karthik Kambatla via Yufei Gu) 2017-05-25 14:22:13 -07:00
Jason Lowe 3fd6a2da4e YARN-6643. TestRMFailover fails rarely due to port conflict. Contributed by Robert Kanter 2017-05-25 16:07:52 -05:00
Varun Vasudev 1a56a3db59 Addendum patch to fix Docker sanitization. 2017-05-25 14:53:57 +05:30
Akira Ajisaka bc28da65fb
YARN-6141. ppc64le on Linux doesn't trigger __linux get_executable codepath. Contributed by Sonia Garudi and Ayappan. 2017-05-25 17:06:26 +09:00
Jason Lowe 8bf1949c0e YARN-6615. AmIpFilter drops query parameters on redirect. Contributed by Wilfred Spiegelenburg 2017-05-24 11:22:42 -05:00
Wangda Tan 8e0f83e49a YARN-6493. Print requested node partition in assignContainer logs. (Jonathan Hung via wangda)
Change-Id: Ib20326daac10acad363befb0ae0998441fa5b32d
2017-05-22 14:28:55 -07:00
Wangda Tan c583ab02c7 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda)
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
2017-05-22 14:26:13 -07:00
Sunil G b6f66b0da1 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-22 14:10:06 +05:30
Jian He e135c0b207 YARN-6627. Use deployed webapp folder to launch new YARN UI. Contributed by Sunil G 2017-05-19 20:14:03 -07:00
Robert Kanter 9855225a79 YARN-6602. Impersonation does not work if standby RM is contacted first (rkanter) 2017-05-19 15:43:22 -07:00
Yufei Gu 66bba8c024 YARN-6249. TestFairSchedulerPreemption fails inconsistently. (Tao Jie via Yufei Gu) 2017-05-19 14:53:27 -07:00
Eric Payne 19482e0d09 YARN-6618. TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than once. Contributed by Jason Lowe. 2017-05-19 11:24:54 -05:00
Varun Vasudev b46cd315f7 Sanitize arguments before launching Docker containers. 2017-05-18 10:37:16 +05:30
Varun Vasudev e120ee865a Validate docker image name before launching container. 2017-05-18 10:29:34 +05:30
Chris Douglas b23fcc86c6 YARN-6577. Remove unused ContainerLocalization classes. Contributed by ZhangBing Lin 2017-05-17 20:40:06 -07:00
Rohith Sharma K S d87a63a901 HADOOP-14412. HostsFileReader#getHostDetails is very expensive on large clusters. Contributed by Jason Lowe. 2017-05-17 08:27:45 +05:30
Robert Kanter 18c494a00c YARN-6447. Provide container sandbox policies for groups (gphillips via rkanter) 2017-05-16 18:02:39 -07:00
Jian He 8236130b2c YARN-6306. NMClient API change for container upgrade. Contributed by Arun Suresh 2017-05-16 10:48:46 -07:00
Jason Lowe 489f85933c YARN-6603. NPE in RMAppsBlock. Contributed by Jason Lowe 2017-05-16 09:26:44 -05:00
Jonathan Eagles c48f2976a3 YARN-6598. History server getApplicationReport NPE when fetching report for pre-2.8 job (Jason Lowe via jeagles) 2017-05-15 10:32:01 -05:00
Carlo Curino 6600abbb5c YARN-6587. Refactor of ResourceManager#startWebApp in a Util class. (Giovanni Matteo Fumarola via curino). 2017-05-12 17:48:55 -07:00
Haibo Chen 6c35001b9f YARN-6580. Incorrect logger for FairSharePolicy. (Vrushali C via Haibo Chen) 2017-05-12 13:05:40 -07:00
Daniel Templeton 90cb5b4635 YARN-6380. FSAppAttempt keeps redundant copy of the queue 2017-05-11 13:47:02 -07:00
Min Shen 2ada100da7 YARN-5543. ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread. Contributed by Min Shen. 2017-05-11 13:25:31 -07:00
Brahma Reddy Battula 4aae2d40a3 HADOOP-14410. Correct spelling of 'beginning' and variants. Contributed By Dongtao Zhang 2017-05-11 23:33:45 +05:30
Carlo Curino 5cb6e3e082 YARN-6473. Create ReservationInvariantChecker to validate ReservationSystem + Scheduler operations. 2017-05-10 14:49:25 -07:00
Daniel Templeton e7654c4a1f YARN-6571. Fix JavaDoc issues in SchedulingPolicy
(Contributed by Weiwei Yang via Daniel Templeton)
2017-05-10 14:34:13 -07:00
Daniel Templeton 74a61438ca YARN-6475. Fix some long function checkstyle issues
(Contributed by Soumabrata Chakraborty via Daniel Templeton)
2017-05-10 10:46:50 -07:00
Jason Lowe 6099deebcb YARN-6552. Increase YARN test timeouts from 1 second to 10 seconds. Contributed by Eric Badger 2017-05-10 08:47:48 -05:00
Haibo Chen ab2bb93a2a YARN-6561. Update exception message during timeline collector aux service initialization. (Vrushali C via Haibo Chen) 2017-05-09 21:37:30 -07:00
Haibo Chen 461ee44d28 YARN-6435. [ATSv2] Can't retrieve more than 1000 versions of metrics in time series. (Rohith Sharma K S via Haibo Chen) 2017-05-09 21:12:57 -07:00
Karthik Kambatla 166be0ee95 YARN-3742. YARN RM will shut down if ZKClient creation times out. (Daniel Templeton via kasha) 2017-05-09 14:44:16 -07:00
Daniel Templeton a2f680493f YARN-5301. NM mount cpu cgroups failed on some systems
(Contributed by Miklos Szegedi via Daniel Templeton)
2017-05-09 12:05:46 -07:00
Vrushali Channapattan 7dd258d8f4 YARN-6563 ConcurrentModificationException in TimelineCollectorManager while stopping RM (Contributed by Haibo Chen via Vrushali C) 2017-05-09 11:18:12 -07:00
Andrew Wang 543aac9f28 HADOOP-14386. Rewind trunk from Guava 21.0 back to Guava 11.0.2. 2017-05-09 09:22:53 -07:00
Subru Krishnan 749e5c09b9 YARN-6281. Cleanup when AMRMProxy fails to initialize a new interceptor chain. (Botong Huang via Subru)
(cherry picked from commit 57a9afbd45b7ef8e6021cc58f96bd0074bf1389d)
2017-05-08 16:55:47 -07:00
Subru Krishnan cd9ff27ffc YARN-6234. Support multiple attempts on the node when AMRMProxy is enabled. (Giovanni Matteo Fumarola via Subru). 2017-05-08 16:41:30 -07:00
Jason Lowe 424887ecb7 YARN-3839. Quit throwing NMNotYetReadyException. Contributed by Manikandan R 2017-05-08 17:14:37 -05:00
Subru Krishnan d6eed5acca YARN-6565. Fix memory leak and finish app trigger in AMRMProxy. (Botong Huang via Subru). 2017-05-05 16:27:49 -07:00
Chris Douglas e4f34ecb04 HADOOP-14390. Correct spelling of 'succeed' and variants. Contributed by Dongtao Zhang 2017-05-05 12:10:50 -07:00
Vrushali Channapattan 54e2b9e876 YARN-6375 App level aggregation should not consider metric values reported in the previous aggregation cycle (Varun Saxena via Vrushali C) 2017-05-04 15:25:56 -07:00
Sidharta S fd5cb2c946 YARN-6374. Improve test coverage and add utility classes for common Docker operations. Contributed by Shane Kumpf 2017-05-03 14:49:30 -07:00
Daniel Templeton 9f0aea0ee2 YARN-6481. Yarn top shows negative container number in FS
(Contributed by Tao Jie via Daniel Templeton)
2017-05-02 13:06:47 -07:00
Subru Krishnan 6bf42e48ef YARN-5331. Extend RLESparseResourceAllocation with period for supporting recurring reservations in YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru). 2017-05-01 18:48:36 -07:00
Subru Krishnan e514fc432a YARN-6542. Fix the logger in TestAlignedPlanner and TestGreedyReservationAgent. 2017-05-01 18:10:07 -07:00
Subru Krishnan a3a615eeab YARN-4359. Update LowCost agents logic to take advantage of YARN-4358. (Jonathan Yaniv and Ishai Menache via Subru). 2017-05-01 16:01:07 -07:00
Naganarasimha 30fc580196 YARN-6519. Fix warnings from Spotbugs in hadoop-yarn-server-resourcemanager. Contributed by Weiwei Yang. 2017-05-01 20:15:27 +05:30
Naganarasimha 0f1af3178e YARN-6518. Fix warnings from Spotbugs in hadoop-yarn-server-timelineservice. Contributed by Weiwei Yang. 2017-05-01 16:49:30 +05:30
Daniel Templeton 2e52789edf YARN-2962. ZKRMStateStore: Limit the number of znodes under a znode (Contributed by Varun Sexena via Daniel Templeton) 2017-04-28 13:30:28 -07:00
Robert Kanter 68e45f554b YARN-6472. Improve Java sandbox regex (gphillips via rkanter) 2017-04-28 11:01:50 -07:00
Robert Kanter 371b6467dc YARN-5894. fixed license warning caused by de.ruedigermoeller:fst:jar:2.24 (haibochen via rkanter) 2017-04-27 17:49:38 -07:00
Varun Saxena 793bbf216d YARN-6455. Enhance the timelinewriter.flush() race condition fix (Haibo Chen via Varun Saxena) 2017-04-27 15:01:58 +05:30
Akira Ajisaka 8a99eba96d
Revert "YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor."
This reverts commit 34ab8e73d4.
2017-04-26 15:34:23 +09:00
Wangda Tan 2ba21d6376 YARN-6392. Add submit time to Application Summary log. (Zhihai Xu via wangda) 2017-04-24 13:17:35 -07:00
Haibo Chen 8ac50e1322 YARN-6500. Do not mount inaccessible cgroups directories in CgroupsLCEResourcesHandler. (Miklos Szegedi via Haibo Chen) 2017-04-24 11:37:52 -07:00
Eric Payne 52adf71914 YARN-5617. AMs only intended to run one attempt can be run more than once? Contributed by Jason Lowe. 2017-04-24 12:03:28 -05:00
Daniel Templeton 46940d92e2 YARN-6302. Fail the node if Linux Container Executor is not configured properly
(Contributed by Miklos Szegedi via Daniel Templeton)
2017-04-19 12:23:49 -07:00
Daniel Templeton dd43b895c2 YARN-6202. Configuration item Dispatcher.DISPATCHER_EXIT_ON_ERROR_KEY is disregarded
(Contributed by Yufei Gu via Daniel Templeton)
2017-04-19 11:44:55 -07:00
Daniel Templeton 41ac190dba YARN-6438. Code can be improved in ContainersMonitorImpl.java
(Contributed by Miklos Szegedi via Daniel Templeton)
2017-04-19 10:53:08 -07:00
Sunil G 9ff25408bc YARN-6164. Expose Queue Configurations per Node Label through YARN client api. Contributed by Benson Qiu. 2017-04-19 20:39:09 +05:30
Chris Douglas af8e9842d2 YARN-6451. Add RM monitor validating metrics invariants. Contributed by Carlo Curino 2017-04-18 10:29:44 -07:00
Junping Du 6ed9d36224 YARN-6304. Skip rm.transitionToActive call to RM if RM is already active. Contributed by Rohith Sharma K S. 2017-04-17 15:51:10 -07:00
Jason Lowe 4168805502 YARN-6480. Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart. Contributed by Eric Badger 2017-04-14 17:15:48 -05:00
Karthik Kambatla 8a1d7480f7 YARN-6433. Only accessible cgroup mount directories should be selected for a controller. (Miklos Szegedi via kasha) 2017-04-14 15:07:14 -07:00
Eric Payne a41f8dd58e YARN-5994. TestCapacityScheduler.testAMLimitUsage fails intermittently. Contributed by Eric Badger 2017-04-14 10:53:09 -05:00
Karthik Kambatla c3375175d6 YARN-6432. FairScheduler: Reserve preempted resources for corresponding applications. (Miklos Szegedi via kasha) 2017-04-12 14:21:20 -07:00
Eric Payne a16ab2be91 YARN-6450. TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest? Contributed by Jason Lowe. 2017-04-12 10:20:08 -05:00
Wangda Tan 4d4ad0ebb7 YARN-6439. Fix ReservationSystem creation of default ReservationQueue. (Carlo Curino via wangda) 2017-04-11 14:56:18 -07:00
Naganarasimha 7d873c465b YARN-6463. Correct spelling mistake in FileSystemRMStateStore. Contributed by Yeliang Cang. 2017-04-11 22:42:08 +05:30
Jason Lowe 0e065f2ede YARN-6195. Export UsedCapacity and AbsoluteUsedCapacity to JMX. Contributed by Benson Qiu 2017-04-11 08:44:18 -05:00
Konstantinos Karanasos 7999318af1 YARN-6344. Add parameter for rack locality delay in CapacityScheduler. (kkaranasos) 2017-04-10 15:34:44 -07:00
Daniel Templeton e9ac61cc0e YARN-6372. Add default value for NM disk validator (Contributed by Yufei Gu via Daniel Templeton) 2017-04-10 14:56:42 -07:00
Karthik Kambatla 2aa8967809 YARN-6298. Metric preemptCall is not used in new preemption. (Yufei Gu via kasha) 2017-04-07 17:31:59 -07:00
Robert Kanter 63f7322522 YARN-6368. Decommissioning an NM results in a -1 exit code (miklos.szegedi@cloudera.com via rkanter) 2017-04-07 14:28:07 -07:00
Ming Ma 0116c3c957 YARN-5797. Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches. (Chris Trezzo via mingma) 2017-04-06 16:54:43 -07:00
Jason Lowe 1b081ca27e YARN-6288. Exceptions during aggregated log writes are mishandled. Contributed by Akira Ajisaka 2017-04-06 16:24:36 -05:00
Rohith Sharma K S 1a9439e299 YARN-6424. TimelineCollector is not stopped when an app finishes in RM. Contributed by Varun Saxena. 2017-04-06 10:15:22 +05:30
Daniel Templeton a2c57bb70d YARN-6381. FSAppAttempt has several variables that should be final
(Contributed by Ameet Zaveri via Daniel Templeton)
2017-04-05 16:07:24 -07:00
Andrew Wang 539ef5aa2e HFDS-11596. hadoop-hdfs-client jar is in the wrong directory in release tarball. Contributed by Yuanbo Liu. 2017-04-05 16:04:09 -07:00
Karthik Kambatla b4c4f36594 YARN-6448. Continuous scheduling thread crashes while sorting nodes. (Yufei Gu via kasha) 2017-04-05 15:43:14 -07:00
Varun Saxena 7d963c477a YARN-6437. TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently (Jason Lowe via Varun Saxena) 2017-04-06 01:20:41 +05:30
Wangda Tan 87e2ef8c98 YARN-6406. Remove SchedulerRequestKeys when no more pending ResourceRequest. (Arun Suresh via wangda) 2017-04-05 11:28:30 -07:00
Jason Lowe e8071aa249 YARN-6403. Invalid local resource request can raise NPE and make NM exit. Contributed by Tao Yang 2017-04-05 12:30:45 -05:00
Akira Ajisaka 34ab8e73d4
YARN-5007. Remove deprecated constructors of MiniYARNCluster and MiniMRYarnCluster. Contributed by Andras Bokor. 2017-04-05 17:33:07 +09:00
Varun Saxena 19b89c4c7b YARN-6420. RM startup failure due to wrong order in nodelabel editlog (Bibin A Chundatt via Varun Saxena) 2017-04-05 09:38:35 +05:30
Ming Ma 2d5c09b848 YARN-6004. Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines. (Chris Trezzo via mingma) 2017-04-04 17:56:21 -07:00
Karthik Kambatla 9cc04b4708 YARN-6436. TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low. (Eric Badger via kasha) 2017-04-04 17:08:33 -07:00
Wangda Tan 3fdae0a2b6 YARN-6109. Add an ability to convert ChildQueue to ParentQueue. (Xuan Gong via wangda) 2017-04-04 14:39:01 -07:00
Chris Douglas 6eba79232f HADOOP-14271. Correct spelling of 'occurred' and variants. Contributed by Yeliang Cang 2017-04-03 20:13:14 -07:00
Varun Saxena a4b5aa8493 YARN-6377. NMTimelinePublisher#serviceStop does not stop timeline clients (Haibo Chen via Varun Saxena) 2017-04-02 04:54:12 +05:30
Varun Saxena b8838578ce YARN-6414. ATSv2 HBase related tests fail due to guava version upgrade (Haibo Chen via Varun Saxena) 2017-04-02 04:37:34 +05:30
Eric Payne 318bfb01bc YARN-6354. LeveldbRMStateStore can parse invalid keys when recovering reservations. Contributed by Jason Lowe 2017-03-31 12:30:35 -05:00
Jason Lowe 4d1fac5df2 YARN-6411. Clean up the overwrite of createDispatcher() in subclass of MockRM. Contributed by Yufei Gu 2017-03-31 10:05:34 -05:00
Varun Saxena b58777a9c9 YARN-6376. Exceptions caused by synchronous putEntities requests can be swallowed (Haibo Chen via Varun Saxena) 2017-03-31 02:17:20 +05:30
Daniel Templeton 640ba1d23f YARN-5685. RM configuration allows all failover methods to disabled when automatic failover is enabled 2017-03-29 12:38:11 -07:00
Akira Ajisaka 82fb9ce8df
YARN-6329. Remove unnecessary TODO comment from AppLogAggregatorImpl.java. Contributed by victor bertschinger. 2017-03-29 10:36:24 +09:00
Varun Saxena 063b513b1c YARN-6357. Implement putEntitiesAsync API in TimelineCollector (Haibo Chen via Varun Saxena) 2017-03-29 03:48:03 +05:30
Varun Saxena 01aca54a22 YARN-5368. Memory leak in timeline server (Jonathan Eagles via Varun Saxena) 2017-03-29 01:53:20 +05:30
Jason Lowe fdf8f8ebca YARN-6359. TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition. Contributed by Robert Kanter 2017-03-28 09:33:26 -05:00
Robert Kanter 9bae6720cb YARN-6050. AMs can't be scheduled on racks or nodes (rkanter) 2017-03-27 17:23:13 -07:00
Wangda Tan cd014d57aa YARN-6339. Improve performance for createAndGetApplicationReport. (Yunjiong Zhao via wangda) 2017-03-27 13:29:09 -07:00
Akira Ajisaka b63ae854f9
YARN-6379. Remove unused argument in ClientRMService. Contributed by Kai Sasaki. 2017-03-27 11:23:41 +09:00
Sunil G cffea251be YARN-5956. Refactor ClientRMService for unify error handling across apis. Contributed by Kai Sasaki. 2017-03-25 22:26:19 +05:30
Tsuyoshi Ozawa 84ddedc0b2 HADOOP-10101. Update guava dependency to the latest version. (ozawa) 2017-03-25 10:06:52 +09:00
Ravi Prakash d4f73e7e27 HDFS-11574. Spelling mistakes in the Java source. Contributed by Hu Xiaodong. 2017-03-24 09:38:17 -07:00
Varun Saxena f462e1ff68 YARN-5934. Fix TestTimelineWebServices.testPrimaryFilterNumericString (Akira Ajisaka via Varun Saxena) 2017-03-22 04:11:53 +05:30
Daniel Templeton 0a05c5c598 YARN-6284. hasAlreadyRun should be final in ResourceManager.StandByTransitionRunnable
(Contributed by Laura Adams via Daniel Templeton)
2017-03-21 14:15:40 -07:00
Varun Saxena 35034653d0 YARN-6353. Clean up OrderingPolicy javadoc (Daniel Templeton via Varun Saxena) 2017-03-20 23:29:09 +05:30
Jason Lowe 4a8e304502 YARN-6217. TestLocalCacheDirectoryManager test timeout is too aggressive. Contributed by Miklos Szegedi 2017-03-17 14:06:35 -05:00
Jason Lowe 7114baddb6 YARN-4051. ContainerKillEvent lost when container is still recovering and application finishes. Contributed by sandflee 2017-03-16 09:30:10 -05:00
Sunil G 2d0e24eb01 YARN-6328. Fix a spelling mistake in CapacityScheduler. Contributed by Jin Yibo. This closes #202 2017-03-15 14:48:05 +05:30
Ray Chiang 4c66a8d19b YARN-6331. Fix flakiness in TestFairScheduler#testDumpState. (Yufei Gu via rchiang) 2017-03-14 15:09:47 -07:00
Naganarasimha 0a3aa40fe7 YARN-6327. Removing queues from CapacitySchedulerQueueManager and ParentQueue should be done with iterator. Contributed by Jonathan Hung. 2017-03-15 01:22:25 +05:30
Junping Du 34424e98a6 YARN-6314. Potential infinite redirection on YARN log redirection web service. Contributed by Xuan Gong.
(cherry picked from commit 5a9dda796f0e73060ada794ad5752cc6a237ab2e)
2017-03-14 02:58:07 -07:00
Ray Chiang 4db9cc70d0 YARN-6042. Dump scheduler and queue state information into FairScheduler DEBUG log. (Yufei Gu via rchiang) 2017-03-10 16:13:31 -08:00
Varun Saxena 229c7c9f89 YARN-6069. CORS support in timeline v2 (Rohith Sharma K S via Varun Saxena) 2017-03-11 04:09:01 +05:30
Jason Lowe 9649c27864 YARN-6321. TestResources test timeouts are too aggressive. Contributed by Eric Badger 2017-03-10 13:06:54 -06:00
Karthik Kambatla a96afae125 YARN-6264. AM not launched when a single vcore is available on the cluster. (Yufei Gu via kasha) 2017-03-09 23:11:54 -08:00
Daniel Templeton 846a0cd678 YARN-1047. Expose # of pre-emptions as a queue counter (Contributed by Karthik Kambatla via Daniel Templeton) 2017-03-09 17:51:47 -08:00
Sidharta S e96a0b8c92 YARN-5669. Add support for docker pull command (Contribtued by luhuichun) 2017-03-09 16:22:19 -08:00
Daniel Templeton 822a74f2ae YARN-6300. NULL_UPDATE_REQUESTS is redundant in TestFairScheduler (Contributed by Yuanbo Liu via Daniel Templeton) 2017-03-09 12:14:33 -08:00
Jason Lowe d7762a5511 YARN-6165. Intra-queue preemption occurs even when preemption is turned off for a specific queue. Contributed by Eric Payne 2017-03-08 16:46:09 -06:00
Robert Kanter 287ba4ffa6 YARN-6297. TestAppLogAggregatorImp.verifyFilesUploaded() should check # of filed uploaded with that of files expected (haibochen via rkanter) 2017-03-08 10:45:33 -08:00
Sunil G 1eb8186703 YARN-6207. Move application across queues should handle delayed event processing. Contributed by Bibin A Chundatt. 2017-03-08 12:04:30 +05:30
Daniel Templeton e0c239cdbd YARN-6287. RMCriticalThreadUncaughtExceptionHandler.rmContext should be final (Contributed by Corey Barker via Daniel Templeton) 2017-03-07 11:58:48 -08:00
Sunil G c1386a90d7 YARN-6248. user is not removed from UsersManager’s when app is killed with pending container requests. Contributed by Eric Payne. 2017-03-04 13:46:35 +05:30
Junping Du 8db7a8c3ae YARN-6271. yarn rmadin -getGroups returns information from standby RM. Contributed by Jian He. 2017-03-03 14:39:31 -08:00
Ray Chiang 2148b83993 YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via rchiang) 2017-03-03 13:04:51 -08:00
Jason Lowe e58fc76030 YARN-6263. NMTokenSecretManagerInRM.createAndGetNMToken is not thread safe. Contributed by Haibo Chen 2017-03-03 08:23:08 -06:00
Robert Kanter 6f6dfe0202 YARN-5280. Allow YARN containers to run with Java Security Manager (gphillips via rkanter) 2017-03-01 14:53:47 -08:00
Jian He 235203dffd YARN-6153. KeepContainer does not work when AM retry window is set. Contributed by kyungwan nam 2017-02-28 13:23:36 -08:00
Xuan e0bb867c3f YARN-6189: Improve application status log message when RM restarted when
app is in NEW state. Contributed by Junping Du
2017-02-28 11:04:56 -08:00
Wangda Tan eac6b4c35c YARN-6216. Unify Container Resizing code paths with Container Updates making it scheduler agnostic. (Arun Suresh via wangda) 2017-02-28 10:35:50 -08:00
Yiqun Lin 0f35443bf2 YARN-6239. Fix javadoc warnings in YARN that caused by deprecated FileSystem APIs. Contributed by Yiqun Lin. 2017-02-28 18:51:56 +08:00
Karthik Kambatla f187d63816 YARN-6231. FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness. (kasha) 2017-02-27 22:10:57 -08:00
Naganarasimha 5f5b031d1f YARN-5703. ReservationAgents are not correctly configured. Contributed by Manikandan R. 2017-02-27 20:38:29 +05:30
Karthik Kambatla fbfe86deea YARN-6172. FSLeafQueue demand update needs to be atomic. (Miklos Szegedi via kasha) 2017-02-26 20:36:33 -08:00
Karthik Kambatla 815d53506f YARN-6215. FairScheduler preemption and update should not run concurrently. (Tao Jie via kasha) 2017-02-26 20:16:36 -08:00
Xuan 53d372a255 YARN-6228: EntityGroupFSTimelineStore should allow configurable cache
stores. Contributed by Li Lu
2017-02-24 10:59:35 -08:00
Karthik Kambatla 694e680d20 YARN-6222. TestFairScheduler.testReservationMetrics is flaky. (Yufei Gu via kasha) 2017-02-23 15:21:52 -08:00
Sunil G a207aa9930 YARN-6211. Synchronization improvement for moveApplicationAcrossQueues and updateApplicationPriority. Contributed by Bibin A Chundatt. 2017-02-23 14:19:07 +05:30
Karthik Kambatla b10e962224 YARN-6194. Cluster capacity in SchedulingPolicy is updated only on allocation file reload. (Yufei Gu via kasha) 2017-02-22 15:58:49 -08:00
Karthik Kambatla 718ad9f6ee YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha) 2017-02-22 15:46:07 -08:00
Wangda Tan 1a6ca75f38 YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda) 2017-02-22 11:17:09 -08:00
Eric Yang 172b23af33 HADOOP-14077. Add ability to access jmx via proxy. Contributed by Yuanbo Liu. 2017-02-18 18:34:13 -08:00
Karthik Kambatla dbbfcf74ab YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha) 2017-02-17 14:07:31 -08:00
Daniel Templeton c7a36e6130 YARN-6125. The application attempt's diagnostic message should have a maximum size
(Contributed by Andras Piros via Daniel Templeton)
2017-02-17 13:40:58 -08:00
Daniel Templeton 9a92837786 YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)
2017-02-17 13:13:46 -08:00
Karthik Kambatla a77f432449 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha) 2017-02-16 14:54:58 -08:00
Sangjin Lee 4fa1afdb88 YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R. 2017-02-16 11:41:04 -08:00
Rohith Sharma K S e63a7814d2 YARN-6200. addendum to fix compilation error caused by reverting YARN-5068. 2017-02-16 18:19:57 +05:30
Rohith Sharma K S a393e84c6f YARN-6200. Reverting since the same functionality achieved by YARN-1623.
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit b7ac85259c.
2017-02-16 18:00:25 +05:30
Karthik Kambatla 74dd142250 YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha) 2017-02-16 00:03:09 -08:00
Karthik Kambatla 11be3f70e0 YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha) 2017-02-15 23:51:22 -08:00
Karthik Kambatla 6c25dbcdc0 YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha) 2017-02-15 23:16:12 -08:00
Junping Du ce2d5bfa5f YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong. 2017-02-15 09:05:14 -08:00
Varun Saxena b7613e0f40 YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena) 2017-02-15 14:48:17 +05:30
Karthik Kambatla 353a9b2d91 YARN-6061. Addendum. Remove extraneous change. 2017-02-14 15:19:52 -08:00
Karthik Kambatla 652679aa8a YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 2017-02-14 13:39:41 -08:00
Arun Suresh aaf106fde3 YARN-5966. AMRMClient changes to support ExecutionType update. (asuresh) 2017-02-14 06:09:10 -08:00
Karthik Kambatla 646c6d6509 YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 2017-02-13 11:26:30 -08:00
Junping Du 464ff479ce YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong. 2017-02-13 06:12:54 -08:00
Varun Saxena 649deb72fb YARN-6170. TimelineReaderServer should wait to join with HttpServer2 (Sangjin Lee via Varun Saxena) 2017-02-11 19:21:45 +05:30
Karthik Kambatla 08f93978f3 YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 2017-02-09 14:45:04 -08:00
Karthik Kambatla 9b85053583 YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:09:32 -08:00
Wangda Tan 5fb723bb77 YARN-5889. Improve and refactor user-limit calculation in Capacity Scheduler. (Sunil G via wangda) 2017-02-09 10:23:50 -08:00
Junping Du 9dbfab1284 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 2017-02-07 02:13:14 -08:00
Junping Du 327c9980aa YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:41:18 -08:00
Daniel Templeton 3e06475307 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 2017-01-31 16:25:33 -08:00
Daniel Templeton 87852b6ef4 YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 2017-01-30 21:44:48 -08:00
Karthik Kambatla 11e44bdda7 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 2017-01-30 13:35:23 -08:00
Sunil G 165f07f51a YARN-6123. [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed. Contributed by Wangda Tan. 2017-01-27 18:37:51 +05:30
Daniel Templeton f85b74ccf9 YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
(Contributed by Yufei Gu via Daniel Templeton)
2017-01-26 10:31:09 -08:00
Jason Lowe 9e19f758c1 YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-25 21:41:43 +00:00
Karthik Kambatla abedb8a9d8 YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 2017-01-25 12:17:28 -08:00
Junping Du 8528d85a68 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:26:53 -08:00
Jason Lowe a33ce45e35 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:17:36 +00:00
Rohith Sharma K S 0101267d9d YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 2017-01-24 10:59:55 +05:30
Wangda Tan ce832059db YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-23 10:52:14 -08:00
Jason Lowe 69fa81679f YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 2017-01-23 17:12:51 +00:00
Sunil G 3fa0d540df YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 2017-01-23 12:19:35 +05:30
Sangjin Lee b01514f65b YARN-5928. Move ATSv2 HBase backend code into a new module that is only dependent at runtime by yarn servers. Contributed by Haibo Chen. 2017-01-19 20:52:55 -08:00
Andrew Wang 5d8b80ea9b Preparing for 3.0.0-alpha3 development 2017-01-19 15:50:07 -08:00
Wangda Tan 72054a817d YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 2017-01-18 13:31:33 -08:00
Karthik Kambatla e224c96234 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 2017-01-17 17:01:31 -08:00
Subru Krishnan 4d1f3d9020 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 2017-01-17 14:48:03 -08:00
Daniel Templeton 695f110f93 YARN-6071. Fix incompatible API change on AM-RM protocol due to YARN-3866 (trunk only)
(Contributed by Wangda Tan via Daniel Templeton)
2017-01-17 12:55:47 -08:00
Junping Du cf695577aa YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong. 2017-01-16 16:20:24 -08:00
Sunil G d3170f9eba YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 2017-01-13 18:22:29 +05:30
Naganarasimha a6b06f7179 YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 2017-01-12 06:40:41 +05:30
Daniel Templeton e6f13fe5d1 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 2017-01-11 16:00:01 -08:00
Daniel Templeton 7979939428 YARN-5554. MoveApplicationAcrossQueues does not check user permission on the target queue
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
2017-01-11 14:34:10 -08:00
Jason Lowe 357eab9566 YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 2017-01-11 17:13:37 +00:00
Junping Du 4db119b7b5 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 2017-01-10 18:20:27 -08:00
Naganarasimha 4c431a6940 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash. 2017-01-10 15:54:16 +05:30
Sunil G 9594c35dcb YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 2017-01-10 14:08:01 +05:30
Junping Du 945db55f2e YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 2017-01-09 18:14:46 -08:00
Wangda Tan 287d3d6804 YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 2017-01-09 08:40:39 -08:00
Varun Saxena f59e36b4ce YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena) 2017-01-09 11:17:19 +05:30
Daniel Templeton ac16400e1f YARN-6022. Revert changes of AbstractResourceRequest (Contributed by Wangda Tan via Daniel Templeton) 2017-01-07 08:24:30 -08:00
Naganarasimha a59df15757 YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-07 08:11:58 +05:30
Jian He 020316458d YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 2017-01-06 18:19:09 -08:00
Arun Suresh 2977bc6a14 YARN-6040. Introduce api independent PendingAsk to replace usage of ResourceRequest within Scheduler classes. (Wangda Tan via asuresh) 2017-01-06 09:59:57 -08:00
Wangda Tan 0a55bd841e YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 2017-01-05 10:31:05 -08:00
Rohith Sharma K S e49e0a6e37 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S. 2017-01-04 12:09:29 +05:30
Junping Du 8fadd69047 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong. 2017-01-03 15:03:38 -08:00
Wangda Tan f69a107aec YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)
(cherry picked from commit e0f2379312)
2017-01-03 14:53:36 -08:00
Wangda Tan ab1faa4ba8 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" to add correct JIRA number
This reverts commit e0f2379312.
2017-01-03 14:53:13 -08:00
Wangda Tan e0f2379312 Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 2017-01-03 14:46:41 -08:00
Daniel Templeton 165d01a73e YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton) 2016-12-30 08:41:52 -08:00
Robert Kanter f216276d21 YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter) 2016-12-28 15:21:52 -08:00
Sunil G 9ca54f4810 YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2016-12-28 22:48:19 +05:30
Varun Vasudev 972da46cb4 YARN-5719. Enforce a C standard for native container-executor. Contributed by Chris Douglas. 2016-12-28 14:59:57 +05:30
Sunil G f6715b26b6 YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max. Contributed by Wangda Tan. 2016-12-28 12:33:02 +05:30
Wangda Tan 0840b4329b YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda) 2016-12-27 21:18:33 -08:00