Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,731 Commits 326 Branches 371 Tags
Commit Graph

13731 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Carlo Curino
bfd967d338 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
(cherry picked from commit acda6b96a4e92e432bd1d97fa14004a11e70387e)
 2017-09-21 17:13:28 -07:00
Subru Krishnan
049f7c84a1 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec860819b080ccc82454480a436c0adc)
(cherry picked from commit 86f05c33125af3b435d6793551680ffa35047aa7)
 2017-09-21 16:55:43 -07:00
Subru Krishnan
66980c00f6 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed26809e12dd62cb66d258ef51c66db3be0a)
(cherry picked from commit 8737c2a8e96799cf3fda730d25a01f6ea2f12e9b)
 2017-09-21 16:55:42 -07:00
Subru Krishnan
58e2458c2e Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5f13c62e8fc45b0af794b5bbeb577da)
(cherry picked from commit 712e97d4cfab15bec4cf4b11cc067b8f85c8bec8)
 2017-09-21 16:55:41 -07:00
Subru Krishnan
fdbd214d4a YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0c1efc52005b1c069d52910d88038d9)
(cherry picked from commit 3e1dc7ece0303f4d06816b8c3c1f2fae14db8eff)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
055138c5c4 YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd769065eed83193b673ec5d919d69500)
(cherry picked from commit 825108d08a9b49d7c19915624f9167101528cb95)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
d3fb1711a4 YARN-6807. Adding required missing configs to Federation configuration guide based on e2e testing. (Tanuj Nayak via Subru). 
(cherry picked from commit b4ac9d1b63dd4031eee8b17d2462087721050b9a)
(cherry picked from commit f427e4201f969dd4d1e086c24a48a247d10cdc2f)
 2017-09-21 16:55:39 -07:00
Subru Krishnan
169037cbb4 YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
(cherry picked from commit 43a97174fe49aa0c25d03b8a970a46d4bebf1aa8)
 2017-09-21 16:55:33 -07:00
Subru Krishnan
8ed4daddc4 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de59697cb78686f0e605dc7e93628b5f3297)
(cherry picked from commit 4cfec943b177e2123a935e70d39776521883c2bc)
 2017-09-21 16:55:01 -07:00
Subru Krishnan
35a38330ef YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235eaf0972783985b1ab24680d029aea79)
(cherry picked from commit 70b1a757f13b01a9192ea5fb0820ba7babfd974e)
 2017-09-21 16:49:55 -07:00
Subru Krishnan
7ede8c1a53 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200609246cb623c74450ca4a2032063cc)
(cherry picked from commit bed1832c934fe4ba44efdcdc49fce06457dc3d4f)
 2017-09-21 16:49:54 -07:00
Subru Krishnan
7dd6caff9e YARN-6484. Documenting the YARN Federation feature. (Carlo Curino via Subru). 
(cherry picked from commit 749ca890e6e62671b4df69d020396ec93edadf16)
(cherry picked from commit d6c29b2f5b953a0b747887cfc9cf4290f0402b1e)
 2017-09-21 16:49:54 -07:00
Subru Krishnan
b9fdc24e43 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a7e2fb4b33a7c876eaa4d9122a1deea)
(cherry picked from commit 42949a6e0108ed1d92a34a5ebb547691bbf16929)
 2017-09-21 16:49:48 -07:00
Subru Krishnan
e4832bef6d Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec3293fe338553c0d5c3612e6176fb26)
(cherry picked from commit f317e0cb48c16e072aa259a0a7ed71a05274bb80)
 2017-09-21 16:47:50 -07:00
Subru Krishnan
9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
(cherry picked from commit 859aa1f9d621d07693825e610bdc0149f7a2770a)
 2017-09-21 16:47:43 -07:00
Subru Krishnan
7f00f938f2 YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
(cherry picked from commit 7444406d6d48846d06b04e82587885bb8632a16c)
 2017-09-21 16:47:14 -07:00
Subru Krishnan
dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
(cherry picked from commit 80e1904000923819c98ae68456cb0452c255d5f3)
 2017-09-21 16:43:08 -07:00
Carlo Curino
3c5dfa0fe5 YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
(cherry picked from commit 5272af8c7eab76d779c621eb0208bf29ffa25613)
 2017-09-21 16:33:31 -07:00
Carlo Curino
c38ac05bb4 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29ac44289afad9a15e8737397ede298b)
(cherry picked from commit c4bbdfeb666f1f3f2fd4c9437f10b233ebedc2c5)
 2017-09-21 16:25:11 -07:00
Subru Krishnan
b9dcf9283e YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde1cf266ecc7d3b513cdaac6abcebbe4)
(cherry picked from commit e1da8f0667589dd660e6fcd776cc87f1b8ef6db9)
 2017-09-21 16:25:10 -07:00
Carlo Curino
b40bdafe31 YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c32b178c48c3a01b5a15e4e3a3e750f5)
(cherry picked from commit 1c64e1709b627846f29dc2fe6d637f074de8b403)
 2017-09-21 16:25:10 -07:00
Subru Krishnan
61c07e4f3f YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
(cherry picked from commit 86b2bec56e28a2d1ece53ab5a452860fd0444268)
 2017-09-21 16:25:09 -07:00
Carlo Curino
6191fac914 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd3a175b122ef86275c99b72964f2c50)
(cherry picked from commit 8623644f4599f51d34ba79c4c1453b3997205d8f)
 2017-09-21 16:25:08 -07:00
Subru Krishnan
2dca88b924 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b08de7d16a86f8025221f6f5e0dbcc5)
(cherry picked from commit 193ec456d624a99c0d1aafb53bdd64fc29edef71)
 2017-09-21 16:25:02 -07:00
Subru Krishnan
ce9110ab5c YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6155d29435d7c92fd6d68079c4cab86)
(cherry picked from commit 98b45b0ed34a060e0a529069cd15676d91600dff)
 2017-09-21 16:23:47 -07:00
Subru Krishnan
002a77dcce YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade059ff9b38da0aa160f56653769bd0143)
(cherry picked from commit 91803305e526a4c05e95834b8a104c9b335cecb6)
 2017-09-21 16:23:46 -07:00
Subru Krishnan
e933a1790c YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed51aac4ac9093ac362f6ce64601884d)
(cherry picked from commit 78032e51d90a42a3600ad942eecf1c3f787759d7)
 2017-09-21 16:23:45 -07:00
Subru Krishnan
d32ffa9e5e YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
(cherry picked from commit 4128c9522dcdc16bb3527f74a48ed1242458a165)
 2017-09-21 16:23:45 -07:00
Subru Krishnan
d87d2b546e YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bddb40fbaa77d3f6fec8c99f5589177f)
(cherry picked from commit 083a05bc1420d6d6aa30d98ab48c2ed8ead5b810)
 2017-09-21 16:23:44 -07:00
Subru Krishnan
b3a1ab711c YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682eced98dfba55d88edd63296993087c85)
(cherry picked from commit 93d9fdeca65a97434936ec9c2e25c362ee016783)
 2017-09-21 16:23:43 -07:00
Subru Krishnan
5c5ab53f3d YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522d3504598fb94eee288d54df73418c6)
(cherry picked from commit 1dadd0b45a6a605da72eb304808edb49fc66da45)
 2017-09-21 16:23:42 -07:00
Subru Krishnan
f792f7b068 YARN-5324. Stateless Federation router policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 1298127bdad60a961441dc7e25cdc2553c119f33)
(cherry picked from commit 0662996b6af19deece21b95b961d9362accc5159)
 2017-09-21 16:23:42 -07:00
Subru Krishnan
23c42408ba YARN-5323. Policies APIs for Federation Router and AMRMProxy policies. (Carlo Curino via Subru). 
(cherry picked from commit f8208fe0b536f29aa65af71d20c3b3e3765679fd)
(cherry picked from commit 81472778d7ef013ea6b2714bc734bc6fc3ca32fa)
 2017-09-21 16:23:41 -07:00
Jian He
aac8755125 YARN-5601. Make the RM epoch base value configurable. Contributed by Subru Krishnan 
(cherry picked from commit 9ca2aba9cc65090162b3517b194b5e655ee4a157)
(cherry picked from commit 2797507d51566ab3b8328e5fb1d0beb9fbce5bae)
 2017-09-21 16:23:40 -07:00
Subru Krishnan
8409fef0d9 YARN-5612. Return SubClusterId in FederationStateStoreFacade#addApplicationHomeSubCluster for Router Failover. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ac1ba2a30427263f4a2eed2018515b2f759fa591)
(cherry picked from commit 1882bc10fbb8937e947d1de82170945c31574a3f)
 2017-09-21 16:23:39 -07:00
Jian He
ea2ecdb796 YARN-3671. Integrate Federation services with ResourceManager. Contributed by Subru Krishnan 
(cherry picked from commit 8573c286e27623155c715a632d56a68f23523c72)
(cherry picked from commit 94a24567d65469091c8b521987dc2003d0327159)
 2017-09-21 16:23:33 -07:00
Jian He
615c912b61 YARN-3673. Create a FailoverProxy for Federation services. Contributed by Subru Krishnan 
(cherry picked from commit 3307564a5f8c8abc5fe84efcd05ee0f7dfdd921c)
(cherry picked from commit bdfad4523f1a5a776e20773c371ce99d0c538ac1)
 2017-09-21 16:13:06 -07:00
Subru Krishnan
32a8618f39 YARN-5467. InputValidator for the FederationStateStore internal APIs. (Giovanni Matteo Fumarola via Subru) 
(cherry picked from commit bd44182e70c273ad3371a6f9b458fe0f8a7a6abc)
(cherry picked from commit cfafd173bd854d59effb9256cd4e7b79f6beab5b)
 2017-09-21 16:13:06 -07:00
Jian He
fb3f9b48fb YARN-3674. Create Facade for Federation State and Policy Store. Contributed by Subru Krishnan 
(cherry picked from commit 5e976cd2b90ccf1bccb6039edf14140677804c4e)
(cherry picked from commit 5c843823972d030555e25729ae7b4b8fbafb59fd)
 2017-09-21 16:12:34 -07:00
Subru Krishnan
1889dfe409 YARN-5519. Add SubClusterId in AddApplicationHomeSubClusterResponse for Router Failover. (Ellen Hui via Subru) 
(cherry picked from commit e4f928cf9cb2a0d57131e4b7960f47a8dac7940c)
(cherry picked from commit 52558df6204954456fc27cf762e5495513928cbe)
 2017-09-21 16:00:40 -07:00
Subru Krishnan
cf15768f45 YARN-5407. In-memory based implementation of the FederationApplicationStateStore/FederationPolicyStateStore. (Ellen Hui via Subru) 
(cherry picked from commit b747d59f41f08dabe4f3a486a2dbd4bed5723867)
(cherry picked from commit 0733088c296eb29ef10399cae4ec6069a233c72e)
 2017-09-21 16:00:39 -07:00
Subru Krishnan
4c3d45a1cf YARN-5408. Compose Federation membership/application/policy APIs into an uber FederationStateStore API. (Ellen Hui via Subru). 
(cherry picked from commit 268389d0211a757439289bea84d502bed6f6ad52)
(cherry picked from commit e0c3a44396f075c6f402b000a6b286e1d9178eb8)
 2017-09-21 16:00:38 -07:00
Subru Krishnan
d0dddb4ebd YARN-3664. Federation PolicyStore internal APIs 
(cherry picked from commit 2f377263a17ec36547625dff23cc65c9aa89c58e)
(cherry picked from commit f1a508bdef5db7455588d586e898781fb47a0bf1)
 2017-09-21 16:00:38 -07:00
Subru Krishnan
07834ad678 YARN-5307. Federation Application State Store internal APIs 
(cherry picked from commit 1af5292042a336a5789b72f0fb6d5f4636d64d49)
(cherry picked from commit 047a16bf1540acc009b441640f2f5841dfed3997)
 2017-09-21 16:00:37 -07:00
Subru Krishnan
31d92378ba YARN-5390. Federation Subcluster Resolver. Contributed by Ellen Hui. 
(cherry picked from commit d3dc461a935c2af4ec3f0312ff0c26918c408467)
(cherry picked from commit d19b6773012d78cb20b92e98e13546013b6622c2)
 2017-09-21 16:00:36 -07:00
Subru Krishnan
8b6a2afa6e YARN-5406. In-memory based implementation of the FederationMembershipStateStore. Contributed by Ellen Hui. 
(cherry picked from commit 20d1d2be91bf841e0fee32522ee7b560a6f2598f)
(cherry picked from commit dcf8c52d80118edce70e310da4fc05045b4cf175)
 2017-09-21 16:00:35 -07:00
Subru Krishnan
a939d2fdb8 YARN-3662. Federation Membership State Store internal APIs. 
(cherry picked from commit 5424d0899cd68cb728f60adcefd2a95877cddbca)
(cherry picked from commit 3e22896f865ca05fd2944b24e159253fe0da034d)
 2017-09-21 16:00:35 -07:00
Subru Krishnan
194524bdf8 YARN-5300. Exclude generated federation protobuf sources from YARN Javadoc/findbugs build 
(cherry picked from commit 9d23570172ae7984f894406dfc6ad066648c61f4)
(cherry picked from commit dd75ffb26b36eaa1a18d33dacbd9ee645dcd35e4)
 2017-09-21 16:00:34 -07:00
Jason Lowe
fc6d9612c7 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger. 
(cherry picked from commit bfd1a72ba8fbb06da73fede2a85e0b544d6ab43f)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2017-09-21 17:52:11 -05:00
Junping Du
e67c8347c4 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:12:57 -07:00