Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,863 Commits 326 Branches 371 Tags
Commit Graph

778 Commits

Author SHA1 Message Date
Sangjin Lee
6466ead9e0 Preparing for 2.6.3 development 2015-10-21 10:58:06 -07:00
Tsuyoshi Ozawa
b898f8014f YARN-3798. ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED. (ozawa and Varun Saxena) 2015-10-21 23:08:02 +09:00
Jason Lowe
ac865de725 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 
(cherry picked from commit feaf0349949e831ce3f25814c1bbff52f17bfe8f)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 16:39:46 +00:00
Jason Lowe
528b809d2d YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith 
(cherry picked from commit a64dd3d24bfcb9af21eb63869924f6482b147fd3)
 2015-10-08 16:33:34 +00:00
Jason Lowe
1484ebb602 YARN-3802. Two RMNodes for the same NodeId are used in RM sometimes 
after NM is reconnected. Contributed by zhihai xu
(cherry picked from commit 5b5bb8dcdc888ba1ebc7e4eba0fa0e7e79edda9a)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 16:01:20 +00:00
Jason Lowe
4770f190b8 YARN-3780. Should use equals when compare Resource in 
RMNodeImpl#ReconnectNodeTransition. Contributed by zhihai xu.
(cherry picked from commit c7ee6c151c5771043a6de3b8a951cea13f59dd7b)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 15:37:08 +00:00
Xuan
1828ba00be YARN-4087. Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs. Contributed by Jian He 
(cherry picked from commit 9f7fcb54e798cf4fda1ea7972dd96491976e1857)
 2015-09-25 16:43:06 -07:00
Xuan
d27f09c936 YARN-2019. Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore. Contributed by Jian He. 
(cherry picked from commit db57d91ac91e895bcb9a23fa50af0b2fbcb1db5a)
 2015-09-25 16:30:49 -07:00
Jian He
c09bb46579 YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager gets connection issue. Contributed by Xuan Gong 
(cherry picked from commit 214fd1408c21f596d1d15217c11b58b34561aab7)
 2015-09-25 16:26:18 -07:00
Jian He
cc30002bc8 YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. Contributed by Xuan Gong 
(cherry picked from commit 6b3b487d3f4883a6e849c71886da52c4c4d9f0bf)
 2015-09-25 16:25:13 -07:00
Sangjin Lee
4cb7dbaead Preparing for 2.6.2 development: mvn versions:set -DnewVersion=2.6.2 2015-09-25 15:51:13 -07:00
Zhijie Shen
d57c3f0a26 YARN-3544. Got back AM logs link on the RM web UI for a completed app. Contributed by Xuan Gong. 
(cherry picked from commit 21bf2cdcb77f69abc906e6cd401a8fb221f250e9)
(cherry picked from commit c9ee316045b83b18cb068aa4de739a1f4b50f02a)
 2015-09-15 17:30:15 -07:00
Xuan
7af5d6b4ba YARN-3248. Display count of nodes blacklisted by apps in the web UI. 
Contributed by Varun Vasudev

(cherry picked from commit 4728bdfa15809db4b8b235faa286c65de4a48cf6)
(cherry picked from commit e26b6e55e96b763063dfbd39977096367eafc1e3)
 2015-09-15 17:30:06 -07:00
Jian He
e914220ab9 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 
(cherry picked from commit 4e886eb9cbd2dcb128bbfd17309c734083093a4c)

(cherry picked from commit 3f0c9e5fe36d201de021d989b23ebaeb2d9a027b)
 2015-09-14 12:54:01 -07:00
Zhijie Shen
f4154bdee8 YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong. 
(cherry picked from commit 85f6d67fa78511f255fcfa810afc9a156a7b483b)
(cherry picked from commit 426535007bcdc67331f7a37b5d69cc20b37c26e0)
 2015-09-11 11:45:29 -07:00
Vinod Kumar Vavilapalli
3462a00dd2 Preparing for release 2.6.1: mvn versions:set -DnewVersion=2.6.1 2015-09-09 15:29:57 -07:00
Jian He
2b526ba757 YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe 
(cherry picked from commit 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

(cherry picked from commit 703fa1b141a98449746bd6fb3b144e74d964d1f5)
 2015-09-08 22:57:35 -07:00
Xuan
d59bf81e08 YARN-3999. RM hangs on draing events. Contributed by Jian He 
(cherry picked from commit 3ae716fa696b87e849dae40225dc59fb5ed114cb)
(cherry picked from commit 2ebdf5bfcee9ede80681a5266df225885d830883)
 2015-09-08 22:57:35 -07:00
Jonathan Eagles
6ed2486c7e YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) 
(cherry picked from commit 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
(cherry picked from commit 899df5bce03ea4f94487e48c1d38bd30ae10c26f)
 2015-09-08 22:57:34 -07:00
Jian He
92742b4402 YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R 
(cherry picked from commit 258623ff8bb1a1057ae3501d4f20982d5a59ea34)

(cherry picked from commit 1d1e7682c9cad6a2f819b390ca3368dfa29c7097)
 2015-09-08 22:57:28 -07:00
Jian He
2336264900 YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan 
(cherry picked from commit f489a4ec969f3727d03c8e85d51af1018fc0b2a1)

(cherry picked from commit d817fbb34d6e34991c6e512c20d71387750a98f4)
 2015-09-06 14:15:33 -07:00
Jian He
ee2b6bc248 YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan 
(cherry picked from commit 18a594257e052e8f10a03e5594e6cc6901dc56be)

(cherry picked from commit 1be2d64ddddaa6322909073cfaf7f2f2eb46e18d)
 2015-09-06 11:54:40 -07:00
Jian He
637e7f9e39 YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan 
(cherry picked from commit c1957fef29b07fea70938e971b30532a1e131fd0)

(cherry picked from commit 3ddafaa7c854dcf21ecc790c276927e7c869e62c)
 2015-09-05 21:07:51 -07:00
Jian He
4c94f07140 YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan 
(cherry picked from commit 21d80b3dd90a8e33e51701887c8d9369ed4ab17d)

(cherry picked from commit c0b1311a93614becc4a255af48fb7b697d491b80)
 2015-09-05 20:54:20 -07:00
Jian He
d9281fbbab YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 86358221fc85a7743052a0b4c1647353508bf308)

(cherry picked from commit cabf97ae4f2dad53c7b9e3d10a67876b16d94074)
 2015-09-05 20:54:20 -07:00
Jian He
b0ad553841 YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan 
(cherry picked from commit 6f9fe76918bbc79109653edc6cde85df05148ba3)

(cherry picked from commit 61b4116b4b3c0eec8f514f079debd88bc757b28e)
 2015-09-05 20:54:19 -07:00
Jian He
419e18cb37 YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 
(cherry picked from commit dd57c2047bfd21910acc38c98153eedf1db75169)

(cherry picked from commit c61e8a7bfa7236e354f859a889083fab3d7ca9eb)
 2015-09-05 20:54:19 -07:00
Jian He
88f022da24 YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 
(cherry picked from commit fdf042dfffa4d2474e3cac86cfb8fe9ee4648beb)

(cherry picked from commit 411836b74c6c02c0b5aebbbce29c209d93db1de2)
 2015-09-05 20:54:18 -07:00
Wangda Tan
85d92721a4 YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda) 
(cherry picked from commit ebd797c48fe236b404cf3a125ac9d1f7714e291e)
(cherry picked from commit 78d626fa892415023827e35ad549636e2a83275d)
 2015-09-03 17:43:01 -07:00
Jian He
f1b35ffd4c YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch 
(cherry picked from commit c53420f58364b11fbda1dace7679d45534533382)

(cherry picked from commit 4931600030e13d9332d9a0e588487cb8684c667d)
 2015-09-03 17:40:24 -07:00
Jason Lowe
ca7fe71000 YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt 
(cherry picked from commit 32e490b6c035487e99df30ce80366446fe09bd6c)

(cherry picked from commit c31e3ba92132f232bd56b257f3854ffe430fbab9)
(cherry picked from commit 07d31d4c0808a169f4770187d655f38aa105255c)
 2015-09-03 14:40:20 -07:00
Xuan
7b1a71a7ad YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang 
(cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371)
(cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b)
(cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)
 2015-09-03 11:54:23 -07:00
Wangda Tan
e081593042 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9)
(cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43)
(cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)
 2015-09-02 11:28:22 -07:00
Wangda Tan
61f2ddb125 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70)
(cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c)
(cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)
 2015-09-02 11:14:35 -07:00
Vinod Kumar Vavilapalli
752e3da738 YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca)
(cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d)
(cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)
 2015-09-01 21:31:00 -07:00
Wangda Tan
005d865494 YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda) 
(cherry picked from commit 6bc7710ec7f2592c4c87dd940fbe5827ef81fe72)
(cherry picked from commit 8e142d27cbddfa1a1c83c5f8752bd14ac0a13612)
(cherry picked from commit 4d43be3c01b1bc0deb31a9081fca5395d0eb4e0d)
 2015-09-01 17:10:42 -07:00
Jian He
7ffdf7d105 YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan Gong 
(cherry picked from commit 95bfd087dc89e57a93340604cc8b96042fa1a05a)

(cherry picked from commit a5f3fb4dc14503bf7c454a48cf954fb0d6710de2)
(cherry picked from commit 27a2f0acb84202cc082090eef7eea57f6e42f9bb)
 2015-09-01 15:12:53 -07:00
Tsuyoshi Ozawa
81417f7572 YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. Contributed by Ryu Kobayashi. 
(cherry picked from commit 1b672096121fef775572b517d4f5721997abbac6)
(cherry picked from commit 6660c2f83b855535217582326746dc76d53fdf61)
(cherry picked from commit 6ea859e435e7cd6bc342f67e1551ccb86fbd976f)
 2015-09-01 14:37:21 -07:00
Wangda Tan
8b5bdac98e YARN-3230. Clarify application states on the web UI. (Jian He via wangda) 
(cherry picked from commit ce5bf927c3d9f212798de1bf8706e5e9def235a1)
(cherry picked from commit a1963968d2a9589fcefaab0d63feeb68c07f4d06)
(cherry picked from commit 591e261ccf1fb5dd25e87665c8d5c0341ff6fb24)
 2015-09-01 14:34:07 -07:00
Karthik Kambatla
5a6755cc0f YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client. (Zhihai Xu via kasha) 
(cherry picked from commit 8d88691d162f87f95c9ed7e0a569ef08e8385d4f)
(cherry picked from commit 0d62e948877e5d50f1b6fbe735a94ac6da5ff472)
(cherry picked from commit 4a5b0e708d42fbff571229a43d1762d1767e2db5)
 2015-09-01 14:06:34 -07:00
Karthik Kambatla
dbc5bab9fd YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi Li via kasha) 
(cherry picked from commit 22426a1c9f4bd616558089b6862fd34ab42d19a7)
(cherry picked from commit 721d7b574126c4070322f70ec5b49a7b8558a4c7)
(cherry picked from commit 5dfa25f22a989222e8b3d1013117b3350a48b2c5)
 2015-09-01 13:54:04 -07:00
Jian He
db92b09e03 YARN-3222. Fixed NPE on RMNodeImpl#ReconnectNodeTransition when a node is reconnected with a different port. Contributed by Rohith Sharmaks 
(cherry picked from commit b2f1ec312ee431aef762cfb49cb29cd6f4661e86)

(cherry picked from commit 888a44563819ba910dc3cc10d10ee0fb8f05db61)
(cherry picked from commit b78f87825bd593e30b2f2ea76f37c7a4fd673ab2)
 2015-09-01 13:39:35 -07:00
Zhijie Shen
28160a0bd6 YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K. 
(cherry picked from commit d5855c0e46404cfc1b5a63e59015e68ba668f0ea)
(cherry picked from commit fd75b8c9cadd069673afc80a0fc5661d779897bd)
(cherry picked from commit a62891971380e5f8e4a645ed36bd88aa6fe0e47a)
 2015-08-31 17:38:51 -07:00
Jian He
a703952d39 YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong 
(cherry picked from commit 0af6a99a3fcfa4b47d3bcba5e5cc5fe7b312a152)

(cherry picked from commit 61466809552f96a83aa19446d4d59cecd0d2cad5)
(cherry picked from commit ab654746fbad2da12b24b13425dc9bf17c46b50c)
 2015-08-31 17:17:47 -07:00
Tsuyoshi Ozawa
03f9ac2de7 YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks. 
(cherry picked from commit ddc5be48fc35868abf7f59088f747c636e76a42a)
(cherry picked from commit c116743bdda2b1792bf872020a5e2b14d772ac60)
(cherry picked from commit 3c9d26ae14625de3e9437c07eceda0d05f1985b2)
 2015-08-30 20:43:14 -07:00
Jian He
e7fc071906 YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla 
(cherry picked from commit 1454efe5d4fe4214ec5ef9142d55dbeca7dab953)

(cherry picked from commit ca0349b87ab1b2d0d2b9dc93de7806d26713165c)
(cherry picked from commit 2f6be218fa41fd0f39633ec5ed0df6e0fa0f54b6)
 2015-08-30 20:42:06 -07:00
Jian He
8cdcb792a1 YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks 
(cherry picked from commit 0d89859b51157078cc504ac81dc8aa75ce6b1782)

(cherry picked from commit 8b398a66ca3728f47363fc8b2fcf7e556e6bbf5a)
(cherry picked from commit 37f1789f419fa741c2fbae9855549d405e856bff)
 2015-08-30 20:40:09 -07:00
Jian He
6dfdcf094d YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 808cba3821d5bc4267f69d14220757f01cd55715)

(cherry picked from commit 9180d11b3bbb2a49127d5d25f53b38c5113bf7ea)
 2015-08-30 20:34:14 -07:00
Jason Lowe
570d52e53c YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He 
(cherry picked from commit 0402bada1989258ecbfdc437cb339322a1f55a97)

(cherry picked from commit 173664d70f0ed3b1852b6703d32e796778fb1c78)
(cherry picked from commit 04e71db1ce9572ae0641234a02b7db5d174668fd)
 2015-08-30 20:06:07 -07:00
Karthik Kambatla
6e954bc25c YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha) 
(cherry picked from commit a2e07a54561a57a83b943628ebbc53ed5ba52718)
(cherry picked from commit 1986ea8dd223267ced3e3aef69980b46e2fef740)
(cherry picked from commit 2b827a18d7b4eb41dc0095ea7277239273e7e396)
 2015-08-30 13:44:52 -07:00