Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
14,269 Commits 326 Branches 371 Tags
Commit Graph

278 Commits

Author SHA1 Message Date
Jason Lowe
76d4d5c759 YARN-7945. Java Doc error in UnmanagedAMPoolManager for branch-2. Contributed by Botong Huang 
(cherry picked from commit 7a6aa7bf99d667e277ef24eba0ca1bde648aa01e)
 2018-02-22 11:01:27 -06:00
Subru Krishnan
2b77729fdf YARN-7630. Fix AMRMToken rollover handling in AMRMProxy. Contributed by Botong Huang. 
(cherry picked from commit 17ba74be29193ac15474f73baaaf4e647a95078b)
(cherry picked from commit 8d3d7fa1c5fda4ec394b2f64baf1008e7b0d9524)
 2017-12-14 14:11:07 -08:00
Weiwei Yang
98eb4b2c6b YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 10:37:22 +08:00
Subru Krishnan
39d0fdf1b2 YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 
(cherry picked from commit 670e8d4ec7e71fc3b054cd3b2826f869b649a788)
(cherry picked from commit 850bd0ed7cebdddf982c8a586ee4d3ad833cfc57)
 2017-12-08 15:46:13 -08:00
Arun Suresh
cc0c966a4e YARN-7587. Skip dispatching opportunistic containers to nodes whose queue is already full. (Weiwei Yang via asuresh) 
(cherry picked from commit 37ca4169508c3003dbe9044fefd37eb8cd8c0503)
 2017-12-03 22:25:06 -08:00
Subru Krishnan
a6db6c9855 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 
(cherry picked from commit ed3109136100a21d971484f242d80f2a7e7d337d)
 2017-11-20 14:22:59 -08:00
Subru Krishnan
8bcd1d62c8 YARN-7476. Fix miscellaneous issues in ATSv2 after merge to branch-2. (Varun Saxena via Subru). 
(cherry picked from commit 0282efab147203e7323450fa18704af46bc524ad)
 2017-11-12 09:25:12 -08:00
Subru Krishnan
8746a65e78 YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 
(cherry picked from commit 0542e6f86e6734f8ceea00960bebe719827136db)
 2017-11-03 11:15:48 -07:00
Carlo Curino
fa86fdc3cc YARN-7339. LocalityMulticastAMRMProxyPolicy should handle cancel request properly. (Botong Huang via curino) 
Edited cherry-pick from 1c5c2b5dde6f2cffc587ca8f79a18828e1b1faf9
 2017-10-25 13:56:35 -07:00
Varun Saxena
56e0873e22 Fix javac, javadoc and checkstyle issues before branch-2 merge. 2017-10-17 23:36:41 +05:30
Rohith Sharma K S
5d9b4bd3e8 YARN-7041. Nodemanager NPE running jobs with security off. Contributed by Varun Saxena. 
(cherry picked from commit e976bf10bb0d42e4d971358554ea736ddcdafd19)
 2017-10-17 23:36:39 +05:30
Rohith Sharma K S
5a7c272e0a YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-10-17 23:36:39 +05:30
Jian He
50dd1372bf YARN-6638. [ATSv2 Security] Timeline reader side changes for loading auth filters and principals. Contributed by Varun Saxena 
(cherry picked from commit 7f06c46065936b88e9864c526bfcda3a89f3e257)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests/src/test/java/org/apache/hadoop/yarn/server/timelineservice/reader/AbstractTimelineReaderHBaseTestBase.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice/src/main/java/org/apache/hadoop/yarn/server/timelineservice/reader/TimelineReaderServer.java
 2017-10-17 23:36:39 +05:30
Jian He
9577900a91 YARN-5647. [ATSv2 Security] Collector side changes for loading auth filters and principals. Contributed by Varun Saxena 
(cherry picked from commit 79dae624793164cd4692c86992a511310145858a)
 2017-10-17 23:36:39 +05:30
Varun Saxena
c3db98aa10 YARN-5743. [Atsv2] Publish queue name and RMAppMetrics to ATS (Rohith Sharma K S via Varun Saxena) 2017-10-17 23:36:34 +05:30
Sangjin Lee
853a5da041 YARN-5699. Retrospect yarn entity fields which are publishing in events info fields. Contributed by Rohith Sharma K S. 
(cherry picked from commit 787d5942003a5441ba455a79fa08da28f60d5777)
 2017-10-17 23:36:34 +05:30
Sangjin Lee
4675f22e6a YARN-5638. Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery. Contributed by Li Lu. 
(cherry picked from commit 0c1863144649ea265da65ce25158707cc3a3fb4a)
 2017-10-17 23:36:34 +05:30
Sangjin Lee
e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Carlo Curino
b9426c0bf4 YARN-7317. Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy. (contributed by Botong Huang via curino) 
(cherry picked from commit 13fcfb3d46ee7a0d606b4bb221d1cd66ef2a5a7c)
 2017-10-12 10:42:36 -07:00
Arun Suresh
cb91bb3c96 YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh). 
(cherry picked from commit b733348dde18a242e6c9074c512116a8baf1d281)
 2017-10-05 15:22:20 -07:00
Subru Krishnan
d6da014f67 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru). 
(cherry picked from commit c61f2c419830e40bb47fb2b1fe1f7d6109ed29a9)
 2017-09-28 13:11:19 -07:00
Wangda Tan
179d8dc83c YARN-7118. AHS REST API can return NullPointerException. (Billie Rinaldi via wangda) 
Change-Id: I16ca93385ab80e3680d4d173b8d1b79929ea7174
 2017-09-25 21:41:00 -07:00
Anu Engineer
d0c0bd573b Revert "YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi." 
This reverts commit de1d747644727ed7c469570bad56400741fb034e.
 2017-09-23 11:12:16 -07:00
Junping Du
de1d747644 YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi. 
(cherry picked from commit 4002bf0a9e7e3619a3bdcff071f0369cbf2873ad)
 2017-09-22 13:59:03 -07:00
Carlo Curino
88b32edb8f YARN-7010. Federation: routing REST invocations transparently to multiple RMs (part 2 - getApps). (Contributed by Giovanni Matteo Fumarola via curino) 
(cherry picked from commit cc8893edc0b7960e958723c81062986c12f06ade)
 2017-09-21 18:09:44 -07:00
Arun Suresh
ac090b38ad YARN-5603. Metrics for Federation StateStore. (Ellen Hui via asuresh) 
(cherry picked from commit 75abc9a8e2cf1c7d2c574ede720df59421512be3)
 2017-09-21 18:09:30 -07:00
Subru Krishnan
261f769d79 YARN-6900. ZooKeeper based implementation of the FederationStateStore. (Íñigo Goiri via Subru). 
(cherry picked from commit de462da04e167a04b89ecf0f40d464cf39dc6549)
 2017-09-21 18:09:23 -07:00
Subru Krishnan
a1ee4ad77f YARN-6970. Add PoolInitializationException as retriable exception in FederationFacade. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit ad2a3506626728a6be47af0db3ca60610a568734)
 2017-09-21 18:08:30 -07:00
Carlo Curino
d11be2dcac [YARN FEDERATION BACKPORT] Fix compilation issues due to: hadoop-router/pom.xml versions and Java 1.7 2017-09-21 17:56:46 -07:00
Subru Krishnan
fdbd214d4a YARN-6821. Move FederationStateStore SQL DDL files from test resource to sbin. 
(cherry picked from commit cd9db822f0c1efc52005b1c069d52910d88038d9)
(cherry picked from commit 3e1dc7ece0303f4d06816b8c3c1f2fae14db8eff)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
055138c5c4 YARN-6815. FederationStateStoreFacade return behavior should be consistent irrespective of whether caching is enabled or not. 
(cherry picked from commit 8820693cd769065eed83193b673ec5d919d69500)
(cherry picked from commit 825108d08a9b49d7c19915624f9167101528cb95)
 2017-09-21 16:55:40 -07:00
Subru Krishnan
169037cbb4 YARN-3659. Federation: routing client invocations transparently to multiple RMs. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 52daa6d971ae408d121b3737ea8c0575e7e8516d)
(cherry picked from commit 43a97174fe49aa0c25d03b8a970a46d4bebf1aa8)
 2017-09-21 16:55:33 -07:00
Subru Krishnan
8ed4daddc4 YARN-6724. Add ability to blacklist sub-clusters when invoking Routing policies. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit f8e5de59697cb78686f0e605dc7e93628b5f3297)
(cherry picked from commit 4cfec943b177e2123a935e70d39776521883c2bc)
 2017-09-21 16:55:01 -07:00
Subru Krishnan
35a38330ef YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru). 
(cherry picked from commit 8c988d235eaf0972783985b1ab24680d029aea79)
(cherry picked from commit 70b1a757f13b01a9192ea5fb0820ba7babfd974e)
 2017-09-21 16:49:55 -07:00
Subru Krishnan
b9fdc24e43 YARN-6666. Fix unit test failure in TestRouterClientRMService. (Botong Huang via Subru). 
(cherry picked from commit e750907d0a7e2fb4b33a7c876eaa4d9122a1deea)
(cherry picked from commit 42949a6e0108ed1d92a34a5ebb547691bbf16929)
 2017-09-21 16:49:48 -07:00
Subru Krishnan
9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru). 
(cherry picked from commit 73bb2102ce4b82b3a3bed91319f7c8f067ddc3e8)
(cherry picked from commit 859aa1f9d621d07693825e610bdc0149f7a2770a)
 2017-09-21 16:47:43 -07:00
Subru Krishnan
7f00f938f2 YARN-5413. Create a proxy chain for ResourceManager Admin API in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 67846a5519b5905c2d925cf4c602f715b653e72c)
(cherry picked from commit 7444406d6d48846d06b04e82587885bb8632a16c)
 2017-09-21 16:47:14 -07:00
Subru Krishnan
dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 4846069061b6baa06da3b524b9e36567dd368388)
(cherry picked from commit 80e1904000923819c98ae68456cb0452c255d5f3)
 2017-09-21 16:43:08 -07:00
Carlo Curino
3c5dfa0fe5 YARN-3663. Federation State and Policy Store (DBMS implementation). (Giovanni Matteo Fumarola via curino). 
(cherry picked from commit be99c1fe2eb150fabd69902118d65941f82971f6)
(cherry picked from commit 5272af8c7eab76d779c621eb0208bf29ffa25613)
 2017-09-21 16:33:31 -07:00
Carlo Curino
c38ac05bb4 YARN-6203: Occasional test failure in TestWeightedRandomRouterPolicy (curino) 
(cherry picked from commit 121d55bd29ac44289afad9a15e8737397ede298b)
(cherry picked from commit c4bbdfeb666f1f3f2fd4c9437f10b233ebedc2c5)
 2017-09-21 16:25:11 -07:00
Subru Krishnan
b9dcf9283e YARN-5602. Utils for Federation State and Policy Store. (Giovanni Matteo Fumarola via Subru). 
(cherry picked from commit 326a2e6bde1cf266ecc7d3b513cdaac6abcebbe4)
(cherry picked from commit e1da8f0667589dd660e6fcd776cc87f1b8ef6db9)
 2017-09-21 16:25:10 -07:00
Carlo Curino
b40bdafe31 YARN-6370. Properly handle rack requests for non-active subclusters in LocalityMulticastAMRMProxyPolicy. (Contributed by Botong Huang via curino). 
(cherry picked from commit ce419881c32b178c48c3a01b5a15e4e3a3e750f5)
(cherry picked from commit 1c64e1709b627846f29dc2fe6d637f074de8b403)
 2017-09-21 16:25:10 -07:00
Subru Krishnan
61c07e4f3f YARN-6247. Share a single instance of SubClusterResolver instead of instantiating one per AM. (Botong Huang via Subru) 
(cherry picked from commit 51aeb2ce0c599176aca9466a939c3ad55df30036)
(cherry picked from commit 86b2bec56e28a2d1ece53ab5a452860fd0444268)
 2017-09-21 16:25:09 -07:00
Carlo Curino
6191fac914 YARN-6190. Validation and synchronization fixes in LocalityMulticastAMRMProxyPolicy. (Botong Huang via curino) 
(cherry picked from commit 5c486961cd3a175b122ef86275c99b72964f2c50)
(cherry picked from commit 8623644f4599f51d34ba79c4c1453b3997205d8f)
 2017-09-21 16:25:08 -07:00
Subru Krishnan
ce9110ab5c YARN-6093. Minor bugs with AMRMtoken renewal and state store availability when using FederationRMFailoverProxyProvider during RM failover. (Botong Huang via Subru). 
(cherry picked from commit 66500f4fa6155d29435d7c92fd6d68079c4cab86)
(cherry picked from commit 98b45b0ed34a060e0a529069cd15676d91600dff)
 2017-09-21 16:23:47 -07:00
Subru Krishnan
002a77dcce YARN-5872. Add AlwayReject policies for router and amrmproxy. (Carlo Curino via Subru). 
(cherry picked from commit db26bade059ff9b38da0aa160f56653769bd0143)
(cherry picked from commit 91803305e526a4c05e95834b8a104c9b335cecb6)
 2017-09-21 16:23:46 -07:00
Subru Krishnan
d32ffa9e5e YARN-5676. Add a HashBasedRouterPolicy, and small policies and test refactoring. (Carlo Curino via Subru). 
(cherry picked from commit 575137f41c27eb72d05d923337f3030a35403e8f)
(cherry picked from commit 4128c9522dcdc16bb3527f74a48ed1242458a165)
 2017-09-21 16:23:45 -07:00
Subru Krishnan
d87d2b546e YARN-5634. Simplify initialization/use of RouterPolicy via a RouterPolicyFacade. (Carlo Curino via Subru). 
(cherry picked from commit d7672ce2bddb40fbaa77d3f6fec8c99f5589177f)
(cherry picked from commit 083a05bc1420d6d6aa30d98ab48c2ed8ead5b810)
 2017-09-21 16:23:44 -07:00
Subru Krishnan
b3a1ab711c YARN-5391. PolicyManager to tie together Router/AMRM Federation policies. (Carlo Curino via Subru). 
(cherry picked from commit 20893682eced98dfba55d88edd63296993087c85)
(cherry picked from commit 93d9fdeca65a97434936ec9c2e25c362ee016783)
 2017-09-21 16:23:43 -07:00
Subru Krishnan
5c5ab53f3d YARN-5325. Stateless ARMRMProxy policies implementation. (Carlo Curino via Subru). 
(cherry picked from commit 11c5336522d3504598fb94eee288d54df73418c6)
(cherry picked from commit 1dadd0b45a6a605da72eb304808edb49fc66da45)
 2017-09-21 16:23:42 -07:00