Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,053 Commits 326 Branches 371 Tags
Commit Graph

14053 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Akira Ajisaka
c0dacda546
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao. 
(cherry picked from commit e928ee583c5a1367e24eab34057f8d8496891452)
 2017-09-26 02:05:35 +09:00
Akira Ajisaka
6004fb4676
HADOOP-14892. MetricsSystemImpl should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 
(cherry picked from commit 3a10367a17b99e2c3035ec54048f94f334d93227)
 2017-09-26 01:48:24 +09:00
Anu Engineer
de5744c903 HDFS-12531. Fix conflict in the javadoc of UnderReplicatedBlocks.java in branch-2. Contributed by Bharat Viswanadham. 2017-09-23 11:16:50 -07:00
Anu Engineer
d0c0bd573b Revert "YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi." 
This reverts commit de1d747644727ed7c469570bad56400741fb034e.
 2017-09-23 11:12:16 -07:00
Junping Du
de1d747644 YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi. 
(cherry picked from commit 4002bf0a9e7e3619a3bdcff071f0369cbf2873ad)
 2017-09-22 13:59:03 -07:00
Jason Lowe
2c10abaddd MAPREDUCE-6967. gridmix/SleepReducer should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit b133dc5700b66318ac6d80e463b049723c511f37)
 2017-09-22 15:30:53 -05:00
Arpit Agarwal
09290659e7 HDFS-12530. Processor argument in Offline Image Viewer should be case insensitive. Contributed by Hanisha Koneru. 2017-09-22 09:48:10 -07:00
Jason Lowe
b94acb6efb MAPREDUCE-6965. QuasiMonteCarlo should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari 
(cherry picked from commit 908d8e914ef55aca69cb8e725e62bda29d1073cc)
 2017-09-22 09:37:09 -05:00
Akira Ajisaka
d80e2f2eae
MAPREDUCE-6966. DistSum should use Time.monotonicNow for measuring durations. Contributed by Chetna Chaudhari. 
(cherry picked from commit c71d13796039815350d3c73c51f9fe48531af0fa)
 2017-09-22 15:10:19 +09:00
Akira Ajisaka
9d4de9c556
MAPREDUCE-6947. Moving logging APIs over to slf4j in hadoop-mapreduce-examples. Contributed by Gergely Novák. 
(cherry picked from commit 53be075241f1ba92bfe47e89c2dfc3f0664e2578)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-examples/src/main/java/org/apache/hadoop/examples/terasort/TeraOutputFormat.java
 2017-09-22 13:29:17 +09:00
Carlo Curino
7cd9018b13 [YARN FEDERATION BACKPORT] Fixing more Java 1.7 compilation issues 2017-09-21 18:19:36 -07:00
Carlo Curino
88b32edb8f YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit cc8893edc0b7960e958723c81062986c12f06ade)
 2017-09-21 18:09:44 -07:00
Arun Suresh
2aacb9d3fb YARN-6923. Metrics for Federation Router. (Giovanni Matteo Fumarola via asuresh) 
(cherry picked from commit ae8fb13b312b30de50d65b5450b565d50d690e9e)
 2017-09-21 18:09:37 -07:00
Arun Suresh
ac090b38ad YARN-5603. Metrics for Federation StateStore. (Ellen Hui via asuresh) 
(cherry picked from commit 75abc9a8e2cf1c7d2c574ede720df59421512be3)
 2017-09-21 18:09:30 -07:00
Subru Krishnan
261f769d79 YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 
(cherry picked from commit de462da04e167a04b89ecf0f40d464cf39dc6549)
 2017-09-21 18:09:23 -07:00
Subru Krishnan
9ad067efe9 YARN-6996. Change javax.cache library implementation from JSR107 to Apache Geronimo. (Ray Chiang via Subru). 
(cherry picked from commit 18f3603bce37e0e07c9075811b1179afc2c227eb)
 2017-09-21 18:09:16 -07:00
Carlo Curino
8220b19af7 YARN-6896. Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution). (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit cc59b5fb26ccf58dffcd8850fa12ec65250f127d)
 2017-09-21 18:09:07 -07:00
Subru Krishnan
a1ee4ad77f YARN-6970. Add PoolInitializationException as retriable exception in FederationFacade. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ad2a3506626728a6be47af0db3ca60610a568734)
 2017-09-21 18:08:30 -07:00
Carlo Curino
d11be2dcac [YARN FEDERATION BACKPORT] Fix compilation issues due to: hadoop-router/pom.xml versions and Java 1.7 2017-09-21 17:56:46 -07:00
Allen Wittenauer
baa2bec843 YARN-2280. Resource manager web service fields are not accessible (Krisztian Horvath via aw) 
(cherry picked from commit a5cf985bf501fd032124d121dcae80538db9e380)
 2017-09-21 17:52:25 -07:00
Carlo Curino
d3afff7603 YARN-6853. Add MySql Scripts for FederationStateStore. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit 874ddbf0b5b1d34aca70ee7fc303cbffdde67236)
(cherry picked from commit 9625a030dee1f567f3b91d74acccb8b15fe25428)
 2017-09-21 17:15:36 -07:00
Subru Krishnan
190b79af8c YARN-6902. Update Microsoft JDBC Driver for SQL Server version in License.txt. (Botong Huang via Subru). 
(cherry picked from commit c581e9438444966345613cb6e0585482936a783a)
(cherry picked from commit 894ff83dd6e773645ab7f0ce7dfbbaa3453df26d)
 2017-09-21 17:15:31 -07:00
Carlo Curino
bfd967d338 YARN-5412. Create a proxy chain for ResourceManager REST API in the Router. (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit b6240b92abf453affc5fd64e1eedf2d29842aa75)
(cherry picked from commit acda6b96a4e92e432bd1d97fa14004a11e70387e)
 2017-09-21 17:13:28 -07:00
Subru Krishnan
049f7c84a1 YARN-6866. Minor clean-up and fixes in anticipation of YARN-2915 merge with trunk. (Botong Huang via Subru). 
(cherry picked from commit 40453879ec860819b080ccc82454480a436c0adc)
(cherry picked from commit 86f05c33125af3b435d6793551680ffa35047aa7)
 2017-09-21 16:55:43 -07:00
Subru Krishnan
66980c00f6 Bumping up hadoop-yarn-server-router module to 3.0.0-beta1 post rebase. 
(cherry picked from commit 69e3ed26809e12dd62cb66d258ef51c66db3be0a)
(cherry picked from commit 8737c2a8e96799cf3fda730d25a01f6ea2f12e9b)
 2017-09-21 16:55:42 -07:00
Subru Krishnan
58e2458c2e Bumping up yarn-server-router (new) module pom to beta1 and fixing imports post rebase. 
(cherry picked from commit f1eff212c5f13c62e8fc45b0af794b5bbeb577da)
(cherry picked from commit 712e97d4cfab15bec4cf4b11cc067b8f85c8bec8)
 2017-09-21 16:55:41 -07:00
Subru Krishnan
fdbd214d4a YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0c1efc52005b1c069d52910d88038d9)
(cherry picked from commit 3e1dc7ece0303f4d06816b8c3c1f2fae14db8eff)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
055138c5c4 YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd769065eed83193b673ec5d919d69500)
(cherry picked from commit 825108d08a9b49d7c19915624f9167101528cb95)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
d3fb1711a4 YARN-6807. Adding required missing configs to Federation configuration guide based on e2e testing. (Tanuj Nayak via Subru). 
(cherry picked from commit b4ac9d1b63dd4031eee8b17d2462087721050b9a)
(cherry picked from commit f427e4201f969dd4d1e086c24a48a247d10cdc2f)
 2017-09-21 16:55:39 -07:00
Subru Krishnan
169037cbb4 YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
(cherry picked from commit 43a97174fe49aa0c25d03b8a970a46d4bebf1aa8)
 2017-09-21 16:55:33 -07:00
Subru Krishnan
8ed4daddc4 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de59697cb78686f0e605dc7e93628b5f3297)
(cherry picked from commit 4cfec943b177e2123a935e70d39776521883c2bc)
 2017-09-21 16:55:01 -07:00
Subru Krishnan
35a38330ef YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235eaf0972783985b1ab24680d029aea79)
(cherry picked from commit 70b1a757f13b01a9192ea5fb0820ba7babfd974e)
 2017-09-21 16:49:55 -07:00
Subru Krishnan
7ede8c1a53 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru). 
(cherry picked from commit 2399eb8200609246cb623c74450ca4a2032063cc)
(cherry picked from commit bed1832c934fe4ba44efdcdc49fce06457dc3d4f)
 2017-09-21 16:49:54 -07:00
Subru Krishnan
7dd6caff9e YARN-6484. Documenting the YARN Federation feature. (Carlo Curino via Subru). 
(cherry picked from commit 749ca890e6e62671b4df69d020396ec93edadf16)
(cherry picked from commit d6c29b2f5b953a0b747887cfc9cf4290f0402b1e)
 2017-09-21 16:49:54 -07:00
Subru Krishnan
b9fdc24e43 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a7e2fb4b33a7c876eaa4d9122a1deea)
(cherry picked from commit 42949a6e0108ed1d92a34a5ebb547691bbf16929)
 2017-09-21 16:49:48 -07:00
Subru Krishnan
e4832bef6d Bumping up hadoop-yarn-server-router module to 3.0.0-alpha4 post rebase. 
(cherry picked from commit bd9c7b1bec3293fe338553c0d5c3612e6176fb26)
(cherry picked from commit f317e0cb48c16e072aa259a0a7ed71a05274bb80)
 2017-09-21 16:47:50 -07:00
Subru Krishnan
9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
(cherry picked from commit 859aa1f9d621d07693825e610bdc0149f7a2770a)
 2017-09-21 16:47:43 -07:00
Subru Krishnan
7f00f938f2 YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
(cherry picked from commit 7444406d6d48846d06b04e82587885bb8632a16c)
 2017-09-21 16:47:14 -07:00
Subru Krishnan
dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
(cherry picked from commit 80e1904000923819c98ae68456cb0452c255d5f3)
 2017-09-21 16:43:08 -07:00
Carlo Curino
3c5dfa0fe5 YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
(cherry picked from commit 5272af8c7eab76d779c621eb0208bf29ffa25613)
 2017-09-21 16:33:31 -07:00
Carlo Curino
c38ac05bb4 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29ac44289afad9a15e8737397ede298b)
(cherry picked from commit c4bbdfeb666f1f3f2fd4c9437f10b233ebedc2c5)
 2017-09-21 16:25:11 -07:00
Subru Krishnan
b9dcf9283e YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde1cf266ecc7d3b513cdaac6abcebbe4)
(cherry picked from commit e1da8f0667589dd660e6fcd776cc87f1b8ef6db9)
 2017-09-21 16:25:10 -07:00
Carlo Curino
b40bdafe31 YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c32b178c48c3a01b5a15e4e3a3e750f5)
(cherry picked from commit 1c64e1709b627846f29dc2fe6d637f074de8b403)
 2017-09-21 16:25:10 -07:00
Subru Krishnan
61c07e4f3f YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
(cherry picked from commit 86b2bec56e28a2d1ece53ab5a452860fd0444268)
 2017-09-21 16:25:09 -07:00
Carlo Curino
6191fac914 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd3a175b122ef86275c99b72964f2c50)
(cherry picked from commit 8623644f4599f51d34ba79c4c1453b3997205d8f)
 2017-09-21 16:25:08 -07:00
Subru Krishnan
2dca88b924 YARN-5410. Bootstrap Router server module. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 04f111394b08de7d16a86f8025221f6f5e0dbcc5)
(cherry picked from commit 193ec456d624a99c0d1aafb53bdd64fc29edef71)
 2017-09-21 16:25:02 -07:00
Subru Krishnan
ce9110ab5c YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6155d29435d7c92fd6d68079c4cab86)
(cherry picked from commit 98b45b0ed34a060e0a529069cd15676d91600dff)
 2017-09-21 16:23:47 -07:00
Subru Krishnan
002a77dcce YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade059ff9b38da0aa160f56653769bd0143)
(cherry picked from commit 91803305e526a4c05e95834b8a104c9b335cecb6)
 2017-09-21 16:23:46 -07:00
Subru Krishnan
e933a1790c YARN-5905. Update the RM webapp host that is reported as part of Federation membership to current primary RM's IP. 
(cherry picked from commit e0f0e7afed51aac4ac9093ac362f6ce64601884d)
(cherry picked from commit 78032e51d90a42a3600ad942eecf1c3f787759d7)
 2017-09-21 16:23:45 -07:00
Subru Krishnan
d32ffa9e5e YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
(cherry picked from commit 4128c9522dcdc16bb3527f74a48ed1242458a165)
 2017-09-21 16:23:45 -07:00