Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,717 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

13717 Commits

Author SHA1 Message Date
Subru Krishnan 9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce)
(cherry picked from commit 859aa1f9d6)
 2017-09-21 16:47:43 -07:00
Subru Krishnan 7f00f938f2 YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519)
(cherry picked from commit 7444406d6d)
 2017-09-21 16:47:14 -07:00
Subru Krishnan dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061)
(cherry picked from commit 80e1904000)
 2017-09-21 16:43:08 -07:00
Carlo Curino 3c5dfa0fe5 YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2e)
(cherry picked from commit 5272af8c7e)
 2017-09-21 16:33:31 -07:00
Carlo Curino c38ac05bb4 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29)
(cherry picked from commit c4bbdfeb66)
 2017-09-21 16:25:11 -07:00
Subru Krishnan b9dcf9283e YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde)
(cherry picked from commit e1da8f0667)
 2017-09-21 16:25:10 -07:00
Carlo Curino b40bdafe31 YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c3)
(cherry picked from commit 1c64e1709b)
 2017-09-21 16:25:10 -07:00
Subru Krishnan 61c07e4f3f YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c)
(cherry picked from commit 86b2bec56e)
 2017-09-21 16:25:09 -07:00
Carlo Curino 6191fac914 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd)
(cherry picked from commit 8623644f45)
 2017-09-21 16:25:08 -07:00
Subru Krishnan 2dca88b924 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b)
(cherry picked from commit 193ec456d6)
 2017-09-21 16:25:02 -07:00
Subru Krishnan ce9110ab5c YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6)
(cherry picked from commit 98b45b0ed3)
 2017-09-21 16:23:47 -07:00
Subru Krishnan 002a77dcce YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade05)
(cherry picked from commit 91803305e5)
 2017-09-21 16:23:46 -07:00
Subru Krishnan e933a1790c YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed)
(cherry picked from commit 78032e51d9)
 2017-09-21 16:23:45 -07:00
Subru Krishnan d32ffa9e5e YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c)
(cherry picked from commit 4128c9522d)
 2017-09-21 16:23:45 -07:00
Subru Krishnan d87d2b546e YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bd)
(cherry picked from commit 083a05bc14)
 2017-09-21 16:23:44 -07:00
Subru Krishnan b3a1ab711c YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682ec)
(cherry picked from commit 93d9fdeca6)
 2017-09-21 16:23:43 -07:00
Subru Krishnan 5c5ab53f3d YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522)
(cherry picked from commit 1dadd0b45a)
 2017-09-21 16:23:42 -07:00
Subru Krishnan f792f7b068 YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bda)
(cherry picked from commit 0662996b6a)
 2017-09-21 16:23:42 -07:00
Subru Krishnan 23c42408ba YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b5)
(cherry picked from commit 81472778d7)
 2017-09-21 16:23:41 -07:00
Jian He aac8755125 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc)
(cherry picked from commit 2797507d51)
 2017-09-21 16:23:40 -07:00
Subru Krishnan 8409fef0d9 YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a304)
(cherry picked from commit 1882bc10fb)
 2017-09-21 16:23:39 -07:00
Jian He ea2ecdb796 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e2)
(cherry picked from commit 94a24567d6)
 2017-09-21 16:23:33 -07:00
Jian He 615c912b61 YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f)
(cherry picked from commit bdfad4523f)
 2017-09-21 16:13:06 -07:00
Subru Krishnan 32a8618f39 YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70)
(cherry picked from commit cfafd173bd)
 2017-09-21 16:13:06 -07:00
Jian He fb3f9b48fb YARN-3674. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b9)
(cherry picked from commit 5c84382397)
 2017-09-21 16:12:34 -07:00
Subru Krishnan 1889dfe409 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9c)
(cherry picked from commit 52558df620)
 2017-09-21 16:00:40 -07:00
Subru Krishnan cf15768f45 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41)
(cherry picked from commit 0733088c29)
 2017-09-21 16:00:39 -07:00
Subru Krishnan 4c3d45a1cf YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d021)
(cherry picked from commit e0c3a44396)
 2017-09-21 16:00:38 -07:00
Subru Krishnan d0dddb4ebd YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a1)
(cherry picked from commit f1a508bdef)
 2017-09-21 16:00:38 -07:00
Subru Krishnan 07834ad678 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042)
(cherry picked from commit 047a16bf15)
 2017-09-21 16:00:37 -07:00
Subru Krishnan 31d92378ba YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a93)
(cherry picked from commit d19b677301)
 2017-09-21 16:00:36 -07:00
Subru Krishnan 8b6a2afa6e YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91)
(cherry picked from commit dcf8c52d80)
 2017-09-21 16:00:35 -07:00
Subru Krishnan a939d2fdb8 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899c)
(cherry picked from commit 3e22896f86)
 2017-09-21 16:00:35 -07:00
Subru Krishnan 194524bdf8 YARN-5300. Exclude generated federation protobuf sources from YARN Javadoc/findbugs build 
(cherry picked from commit 9d23570172)
(cherry picked from commit dd75ffb26b)
 2017-09-21 16:00:34 -07:00
Jason Lowe fc6d9612c7 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 
(cherry picked from commit bfd1a72ba8)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2017-09-21 17:52:11 -05:00
Junping Du e67c8347c4 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:12:57 -07:00
Yufei Gu c3c8b0ffab YARN-7045. Remove FSLeafQueue#addAppSchedulable. (Contributed by Sen Zhao via Yufei Gu) 
(cherry picked from commit a92ef030a2)
 2017-09-21 12:40:37 -07:00
Arpit Agarwal 18e5f2068b HDFS-12496. Make QuorumJournalManager timeout properties configurable. Contributed by Ajay Kumar. 2017-09-21 08:53:27 -07:00
Jason Lowe cf2615961a MAPREDUCE-6964. BaileyBorweinPlouffe should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b9db0afa1e)
 2017-09-21 09:39:46 -05:00
Kihwal Lee 9640bfb808 HDFS-12371. BlockVerificationFailures and BlocksVerified show up as 0 in Datanode JMX. Contributed by Hanisha Koneru. 
(cherry picked from commit 6bf921a5c3)
 2017-09-21 08:48:04 -05:00
Jason Lowe ae42bf73c0 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger 
(cherry picked from commit 10d7493587)
 2017-09-21 08:39:31 -05:00
Akira Ajisaka 3527327f34
HDFS-12526. FSDirectory should use Time.monotonicNow for durations. Contributed by Bharat Viswanadham. 
(cherry picked from commit 50849ec9eb)
 2017-09-21 19:28:49 +09:00
Sangjin Lee a79422f717 YARN-6771. Use classloader inside configuration class to make new 
classes. Contributed by Jongyoul Lee.

(cherry picked from commit 8b336632ac)
 2017-09-20 22:46:10 -07:00
Akira Ajisaka 7d747df52d
Revert "MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergery Novák." 
This reverts commit e61baf9496.
 2017-09-21 11:16:48 +09:00
Ming Ma 3bb23f4be9 HDFS-11035. Better documentation for maintenace mode and upgrade domain. 
(cherry picked from commit ce943eb17a)
 2017-09-20 09:42:22 -07:00
Ming Ma 7dd662eafd HDFS-12473. Change hosts JSON file format. 2017-09-20 09:08:41 -07:00
Akira Ajisaka 6581f2dea3
HADOOP-7308. Remove unused TaskLogAppender configurations from log4j.properties. Contributed by Todd Lipcon and J.Andreina. 
(cherry picked from commit 7e58b2478c)
 2017-09-20 21:08:54 +09:00
Junping Du bdd8433d4c YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh. 
(cherry picked from commit 647b7527a9)
 2017-09-19 18:36:25 -07:00
Yufei Gu 64878193d5 YARN-6499. Remove the doc about Schedulable#redistributeShare(). (Contributed by Chetna Chaudhari via Yufei Gu) 
(cherry picked from commit a9019e1fb7)
 2017-09-19 18:29:24 -07:00
Brahma Reddy Battula eddc581860 HDFS-11799. Introduce a config to allow setting up write pipeline with fewer nodes than replication factor. Contributed by Brahma Reddy Battula 
(cherry picked from commit fda1221c55)
 2017-09-19 11:31:22 +05:30