Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
16,500 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3782 Commits

Author SHA1 Message Date
Subru Krishnan c61f2c4198 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 2017-08-07 16:58:29 -07:00
Wangda Tan adb84f34db YARN-4161. Capacity Scheduler : Assign single or multiple containers per heart beat driven by configuration. (Wei Yan via wangda) 
Change-Id: Ic441ae4e0bf72e7232411eb54243ec143d5fd0d3
 2017-08-07 11:32:21 -07:00
Akira Ajisaka b0fbf17965
YARN-6957. Moving logging APIs over to slf4j in hadoop-yarn-server-sharedcachemanager. Contributed by Yeliang Cang. 2017-08-07 19:25:40 +09:00
Akira Ajisaka 839e077faf
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-07 18:56:00 +09:00
Sunil G 46b7054fa7 YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang. 2017-08-07 13:15:46 +05:30
Junping Du f44b349b81 YARN-6811. [ATS1.5] All history logs should be kept under its own User Directory. Contributed by Rohith Sharma K S. 2017-08-04 16:03:56 -07:00
Arun Suresh 35dc782923 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh) 2017-08-03 21:15:40 -07:00
Subru Krishnan 0542e6f86e YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 2017-08-03 14:18:03 -07:00
Yufei Gu c617fe02b3 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 11:33:37 -07:00
Haibo Chen 293c74a81b YARN-6673 Add cpu cgroup configurations for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:56:51 -07:00
Haibo Chen c5d256c760 YARN-6674 Add memory cgroup settings for opportunistic containers. (Miklos Szegedi via Haibo Chen) 2017-08-03 09:52:35 -07:00
Sunil G f64cfeaf61 YARN-6678. Handle IllegalStateException in Async Scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2017-08-03 19:27:10 +05:30
Sean Mackrory 1a1bf6b7d0 HADOOP-13595. Rework hadoop_usage to be broken up by clients/daemons/etc. Contributed by Allen Wittenauer. 2017-08-02 12:25:05 -06:00
Eric Payne 8ce8672b6b YARN-5349. TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently. Contributed by Jason Lowe. 2017-08-02 12:53:22 -05:00
Yufei Gu 45535f8afa YARN-6895. [FairScheduler] Preemption reservation may cause regular reservation leaks. (Miklos Szegedi via Yufei Gu) 2017-08-02 09:25:19 -07:00
Eric Payne 48899134d2 YARN-6846. Nodemanager can fail to fully delete application local directories when applications are killed. Contributed by Jason Lowe. 2017-08-02 10:59:33 -05:00
Jian He f9139ac8f6 YARN-6872. [Addendum patch] Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-02 00:03:25 -07:00
Carlo Curino 9625a030de YARN-6853. Add MySql Scripts for FederationStateStore. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit 874ddbf0b5)
 2017-08-01 17:28:28 -07:00
Carlo Curino acda6b96a4 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92ab)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 86f05c3312 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec)
 2017-08-01 17:28:28 -07:00
Subru Krishnan 8737c2a8e9 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed2680)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 712e97d4cf Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 3e1dc7ece0 YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0)
 2017-08-01 17:28:27 -07:00
Subru Krishnan 825108d08a YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd7)
 2017-08-01 17:28:27 -07:00
Subru Krishnan f427e4201f YARN-6807. Adding required missing configs to Federation configuration guide based on e2e testing. (Tanuj Nayak via Subru). 
(cherry picked from commit b4ac9d1b63)
 2017-08-01 17:28:26 -07:00
Subru Krishnan 43a97174fe YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971)
 2017-08-01 17:28:26 -07:00
Subru Krishnan 4cfec943b1 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de5969)
 2017-08-01 17:28:26 -07:00
Arun Suresh 433ee44b58 YARN-6691. Update YARN daemon startup/shutdown scripts to include Router service. (Giovanni Matteo Fumarola via asuresh). 
(cherry picked from commit 790cba7ab9)
 2017-08-01 17:28:26 -07:00
Subru Krishnan 70b1a757f1 YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235e)
 2017-08-01 17:28:26 -07:00
Subru Krishnan bed1832c93 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200)
 2017-08-01 17:28:25 -07:00
Subru Krishnan d6c29b2f5b YARN-6484. Documenting the YARN Federation feature. (Carlo Curino via Subru). 
(cherry picked from commit 749ca890e6)
 2017-08-01 17:28:25 -07:00
Subru Krishnan 42949a6e01 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a)
 2017-08-01 17:28:25 -07:00
Subru Krishnan f317e0cb48 Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec)
 2017-08-01 17:28:25 -07:00
Subru Krishnan 859aa1f9d6 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce)
 2017-08-01 17:28:24 -07:00
Subru Krishnan 7444406d6d YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519)
 2017-08-01 17:28:24 -07:00
Subru Krishnan 80e1904000 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061)
 2017-08-01 17:28:24 -07:00
Carlo Curino 5272af8c7e YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2e)
 2017-08-01 17:28:24 -07:00
Carlo Curino c4bbdfeb66 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29)
 2017-08-01 17:28:23 -07:00
Subru Krishnan e1da8f0667 YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde)
 2017-08-01 17:28:23 -07:00
Carlo Curino 1c64e1709b YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c3)
 2017-08-01 17:28:23 -07:00
Subru Krishnan 86b2bec56e YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c)
 2017-08-01 17:28:23 -07:00
Carlo Curino 8623644f45 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 193ec456d6 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 98b45b0ed3 YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 91803305e5 YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade05)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 78032e51d9 YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed)
 2017-08-01 17:28:22 -07:00
Subru Krishnan 4128c9522d YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 083a05bc14 YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bd)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 93d9fdeca6 YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682ec)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 1dadd0b45a YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 0662996b6a YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bda)
 2017-08-01 17:28:21 -07:00
Subru Krishnan 81472778d7 YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b5)
 2017-08-01 17:28:20 -07:00
Jian He 2797507d51 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc)
 2017-08-01 17:28:20 -07:00
Subru Krishnan 1882bc10fb YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a304)
 2017-08-01 17:28:20 -07:00
Jian He 94a24567d6 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e2)
 2017-08-01 17:28:20 -07:00
Jian He bdfad4523f YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f)
 2017-08-01 17:28:20 -07:00
Subru Krishnan cfafd173bd YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70)
 2017-08-01 17:28:19 -07:00
Jian He 5c84382397 YARN-3672. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b9)
 2017-08-01 17:28:19 -07:00
Subru Krishnan 52558df620 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9c)
 2017-08-01 17:28:19 -07:00
Subru Krishnan 0733088c29 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41)
 2017-08-01 17:28:19 -07:00
Subru Krishnan e0c3a44396 YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d021)
 2017-08-01 17:28:19 -07:00
Subru Krishnan f1a508bdef YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a1)
 2017-08-01 17:28:18 -07:00
Subru Krishnan 047a16bf15 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042)
 2017-08-01 17:28:18 -07:00
Subru Krishnan d19b677301 YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a93)
 2017-08-01 17:28:18 -07:00
Subru Krishnan dcf8c52d80 YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91)
 2017-08-01 17:28:18 -07:00
Subru Krishnan 3e22896f86 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899c)
 2017-08-01 17:28:17 -07:00
Subru Krishnan dd75ffb26b YARN-5300. Exclude generated federation protobuf sources from YARN Javadoc/findbugs build 
(cherry picked from commit 9d23570172)
 2017-08-01 17:28:17 -07:00
Jian He 91f120f743 YARN-6872. Ensure apps could run given NodeLabels are disabled post RM switchover/restart. Contributed by Sunil G 2017-08-01 09:56:33 -07:00
Akira Ajisaka b38a1eea8e
HADOOP-14245. Use Mockito.when instead of Mockito.stub. Contributed by Andras Bokor. 2017-08-01 15:15:43 +09:00
Akira Ajisaka a4aa1cb405
Revert "YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang." 
This reverts commit 1a78c0ff01.
 2017-08-01 12:15:03 +09:00
Akira Ajisaka 1a78c0ff01
YARN-6873. Moving logging APIs over to slf4j in hadoop-yarn-server-applicationhistoryservice. Contributed by Yeliang Cang. 2017-08-01 10:53:32 +09:00
Akira Ajisaka f8bed5e9a7
YARN-5728. TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout. 2017-07-31 11:09:13 +09:00
Arun Suresh 890e14c02a YARN-6870. Fix floating point inaccuracies in resource availability check in AllocationBasedResourceUtilizationTracker. (Brook Zhou via asuresh) 2017-07-28 16:32:43 -07:00
Subru Krishnan bcde66bed1 YARN-6897. Refactoring RMWebServices by moving some util methods to RMWebAppUtil. (Giovanni Matteo Fumarola via Subru). 2017-07-28 15:44:36 -07:00
Yufei Gu 9ea01fd956 YARN-6802. Add max AM resource and AM resource usage to leaf queue view in FairScheduler WebUI. (YunFan Zhou via Yufei Gu) 2017-07-28 09:52:53 -07:00
Yufei Gu 9902be72cb YARN-6864. FSPreemptionThread cleanup for readability. (Daniel Templeton via Yufei Gu) 2017-07-27 23:19:39 -07:00
Akira Ajisaka 38c6fa5c7a
HADOOP-11875. [JDK9] Adding a second copy of Hamlet without _ as a one-character identifier. 2017-07-28 09:10:52 +09:00
Varun Saxena 11ece0bda1 Addendum for YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-28 00:10:45 +05:30
Varun Saxena f66fd11e51 YARN-5548. Use MockRMMemoryStateStore to reduce test failures (Bibin A Chundatt via Varun Saxena) 2017-07-27 23:14:50 +05:30
Daniel Templeton f81a4efb8c YARN-6307. Refactor FairShareComparator#compare (Contributed by Yufei Gu via Daniel Templeton) 2017-07-25 13:00:31 -07:00
Jian He ac9489f7fc YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-25 09:57:14 -07:00
Akira Ajisaka cca51e916b
HADOOP-14681. Remove MockitoMaker class. Contributed by Andras Bokor. 2017-07-25 15:24:56 +09:00
Akira Ajisaka 218b1b33ff
YARN-6150. TestContainerManagerSecurity tests for Yarn Server are flakey. Contributed by Daniel Sturman and Ray Chiang. 2017-07-25 15:12:08 +09:00
Daniel Templeton 10583625c1 YARN-6845. Variable scheduler of FSLeafQueue duplicates the one of its parent FSQueue. 
(Contributed by Yufei Gu via Daniel Templeton)
 2017-07-24 13:44:00 -07:00
Daniel Templeton bb30bd3771 YARN-6779. DominantResourceFairnessPolicy.DominantResourceFairnessComparator.calculateShares() should be @VisibleForTesting 
(Contributed by Yeliang Cang via Daniel Templeton)
 2017-07-24 12:13:50 -07:00
Daniel Templeton 4c40cd451c YARN-6844. AMRMClientImpl.checkNodeLabelExpression() has wrong error message 
(Contributed by Manikandan R via Daniel Templeton)
 2017-07-24 12:03:02 -07:00
Jason Lowe 24853bf32a YARN-6768. Improve performance of yarn api record toString and fromString. Contributed by Jonathan Eagles 2017-07-24 10:19:20 -05:00
Sunil G e315328428 YARN-6102. RMActiveService context to be updated with new RMContext on failover. Contributed by Rohith Sharma K S. 2017-07-24 11:39:03 +05:30
Brahma Reddy Battula 2054324d47 Revert "YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi" 
This reverts commit 4a771d9010.
 2017-07-23 12:56:18 +08:00
Jian He 4a771d9010 YARN-6804. [YARN core changes] Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-07-21 11:31:33 -07:00
Jason Lowe c8df3668ec YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling 2017-07-20 11:03:04 -05:00
Arun Suresh 077fcf6a96 YARN-6777. Support for ApplicationMasterService processing chain of interceptors. (asuresh) 2017-07-19 12:26:40 -07:00
Rohith Sharma K S 845c4e52bd YARN-6819. Application report fails if app rejected due to nodesize. Contributed by Bibin A Chundatt. 2017-07-19 11:10:52 +05:30
Yufei Gu daaf530fce YARN-6778. In ResourceWeights, weights and setWeights() should be final. (Daniel Templeton via Yufei Gu) 2017-07-18 16:38:07 -07:00
Ray Chiang f5f14a2ad6 YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang) 2017-07-18 12:35:08 -07:00
Sunil G 0b7afc060c Addendum patch for YARN-5731 2017-07-18 11:49:09 +05:30
Arun Suresh 5b007921cd YARN-6706. Refactor ContainerScheduler to make oversubscription change easier. (Haibo Chen via asuresh) 2017-07-17 14:11:14 -07:00
Yufei Gu 9e0cde1469 YARN-6625. yarn application -list returns a tracking URL for AM that doesn't work in secured and HA environment. (Yufei Gu) 2017-07-14 14:10:45 -07:00
Jason Lowe a5ae5ac50e YARN-3260. AM attempt fail to register before RM processes launch event. Contributed by Bibin A Chundatt 2017-07-14 14:56:00 -05:00
bibinchundatt 75c0220b44 YARN-6759. Fix TestRMRestart.testRMRestartWaitForPreviousAMToFinish failure. Contributed by Naganarasimha G R 2017-07-14 13:53:39 +05:30