Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,937 Commits 326 Branches 371 Tags
Commit Graph

793 Commits

Author SHA1 Message Date
Rohith Sharma K S
58a6142c14 YARN-3849. Too much of preemption activity causing continuos killing of containers across queues. (Sunil G via wangda) 2016-01-11 12:02:38 +05:30
Anubhav Dhoot
6b2abb7515 YARN-4180. AMLauncher does not retry on failures when talking to NM. (adhoot) 
(cherry picked from commit 9735afe967a660f356e953348cb6c34417f41055)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java

(cherry picked from commit 22f2501476d987afb7bc19080a7a0db94ea72be6)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java

(cherry picked from commit 7c9a368b45b0e38173521a94ab32dee8a2984bf8)
 2016-01-07 15:08:05 -08:00
Junping Du
74027c24c8 Addendum patch to fix build after porting YARN-4546. 2016-01-06 06:11:00 -08:00
Junping Du
c0ffe25a65 YARN-4546. ResourceManager crash due to scheduling opportunity overflow. Contributed by Jason Lowe. 
(cherry picked from commit c1462a67ff7bb632df50e1c52de971cced56c6a3)
(cherry picked from commit 1cc001db4c3767072b5d065d161bc5c6d1c480d4)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerApplicationAttempt.java
 2016-01-06 05:56:16 -08:00
Sangjin Lee
51a2e6304a Preparing for 2.6.4 development 2016-01-05 15:40:31 -08:00
Karthik Kambatla
62e032a4da YARN-3697. FairScheduler: ContinuousSchedulingThread can fail to shutdown. (Zhihai Xu via kasha) 
(cherry picked from commit 332b520a480994b7bd56c135f7941aad30b05e9c)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
 2016-01-05 00:30:37 -08:00
Rohith Sharma K S
7af87093ad YARN-3893. Both RM in active state when Admin#transitionToActive failure from refeshAll() (Bibin A Chundatt via rohithsharmaks) 
(cherry picked from commit 7d6687fe76f6152a577ff2298c358dd30fce41fb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2016-01-04 11:44:28 +05:30
Junping Du
0f9dd48842 YARN-4452. NPE when submit Unmanaged application. Contributed by Naganarasimha G R. 
(cherry picked from commit 50bd067e1d63d4c80dc1e7bf4024bfaf42cf4416)
(cherry picked from commit 1a2ef845b54166194b133e524ad9533cc259aed2)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/metrics/SystemMetricsPublisher.java
 2015-12-17 05:53:56 -08:00
Arun Suresh
c53d45a687 YARN-3535. Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED (rohithsharma and peng.zhang via asuresh) 
(cherry picked from commit 9b272ccae78918e7d756d84920a9322187d61eed)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/event/SchedulerEventType.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestAbstractYarnScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java
 2015-12-14 23:50:55 -08:00
Zhihai Xu
70289432f7 YARN-3857: Memory leak in ResourceManager with SIMPLE mode. Contributed by mujunchao. 
(cherry picked from commit 3a76a010b85176f2bcb85ed6f74c25dcb8acfe4d)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-12-14 22:17:23 -08:00
Karthik Kambatla
843dac5353 YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha) 
(cherry picked from commit 2abec14ec6e7d6d8b7e59239ed2596a15adc8475)
 2015-12-09 10:53:26 -08:00
Wangda Tan
5b063d6b7f YARN-4424. Fix deadlock in RMAppImpl. (Jian he via wangda) 
(cherry picked from commit 7e4715186d31ac889fba26d453feedcebb11fc70)

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 7013f9d6cda88e72a839b1c55757615b55101beb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2015-12-08 14:34:53 -08:00
Tsuyoshi Ozawa
b345ffd7df YARN-4348. ZKRMStateStore.syncInternal shouldn't wait for sync completion for avoiding blocking ZK's event thread. (ozawa) 
(cherry picked from commit 0460b8a8a3de232f236f49ef6769d38cda62cc28)
 2015-12-08 13:41:17 +09:00
Jason Lowe
5f05e5e5ba YARN-4344. NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations. Contributed by Varun Vasudev 2015-11-23 20:27:00 +00:00
Tsuyoshi Ozawa
5a00b23106 YARN-4312. TestSubmitApplicationWithRMHA fails on branch-2.7 and branch-2.6 as some of the test cases time out. Contributed by Varun Saxena. 
(cherry picked from commit 66364419118c64c9d1c623f808f027ac45688759)
 2015-11-05 10:40:23 -08:00
Sangjin Lee
6466ead9e0 Preparing for 2.6.3 development 2015-10-21 10:58:06 -07:00
Tsuyoshi Ozawa
b898f8014f YARN-3798. ZKRMStateStore shouldn't create new session without occurrance of SESSIONEXPIED. (ozawa and Varun Saxena) 2015-10-21 23:08:02 +09:00
Jason Lowe
ac865de725 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 
(cherry picked from commit feaf0349949e831ce3f25814c1bbff52f17bfe8f)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 16:39:46 +00:00
Jason Lowe
528b809d2d YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith 
(cherry picked from commit a64dd3d24bfcb9af21eb63869924f6482b147fd3)
 2015-10-08 16:33:34 +00:00
Jason Lowe
1484ebb602 YARN-3802. Two RMNodes for the same NodeId are used in RM sometimes 
after NM is reconnected. Contributed by zhihai xu
(cherry picked from commit 5b5bb8dcdc888ba1ebc7e4eba0fa0e7e79edda9a)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 16:01:20 +00:00
Jason Lowe
4770f190b8 YARN-3780. Should use equals when compare Resource in 
RMNodeImpl#ReconnectNodeTransition. Contributed by zhihai xu.
(cherry picked from commit c7ee6c151c5771043a6de3b8a951cea13f59dd7b)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 15:37:08 +00:00
Xuan
1828ba00be YARN-4087. Followup fixes after YARN-2019 regarding RM behavior when state-store error occurs. Contributed by Jian He 
(cherry picked from commit 9f7fcb54e798cf4fda1ea7972dd96491976e1857)
 2015-09-25 16:43:06 -07:00
Xuan
d27f09c936 YARN-2019. Retrospect on decision of making RM crashed if any exception throw in ZKRMStateStore. Contributed by Jian He. 
(cherry picked from commit db57d91ac91e895bcb9a23fa50af0b2fbcb1db5a)
 2015-09-25 16:30:49 -07:00
Jian He
c09bb46579 YARN-4101. RM should print alert messages if Zookeeper and Resourcemanager gets connection issue. Contributed by Xuan Gong 
(cherry picked from commit 214fd1408c21f596d1d15217c11b58b34561aab7)
 2015-09-25 16:26:18 -07:00
Jian He
cc30002bc8 YARN-4092. Fixed UI redirection to print useful messages when both RMs are in standby mode. Contributed by Xuan Gong 
(cherry picked from commit 6b3b487d3f4883a6e849c71886da52c4c4d9f0bf)
 2015-09-25 16:25:13 -07:00
Sangjin Lee
4cb7dbaead Preparing for 2.6.2 development: mvn versions:set -DnewVersion=2.6.2 2015-09-25 15:51:13 -07:00
Zhijie Shen
d57c3f0a26 YARN-3544. Got back AM logs link on the RM web UI for a completed app. Contributed by Xuan Gong. 
(cherry picked from commit 21bf2cdcb77f69abc906e6cd401a8fb221f250e9)
(cherry picked from commit c9ee316045b83b18cb068aa4de739a1f4b50f02a)
 2015-09-15 17:30:15 -07:00
Xuan
7af5d6b4ba YARN-3248. Display count of nodes blacklisted by apps in the web UI. 
Contributed by Varun Vasudev

(cherry picked from commit 4728bdfa15809db4b8b235faa286c65de4a48cf6)
(cherry picked from commit e26b6e55e96b763063dfbd39977096367eafc1e3)
 2015-09-15 17:30:06 -07:00
Jian He
e914220ab9 YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong 
(cherry picked from commit 4e886eb9cbd2dcb128bbfd17309c734083093a4c)

(cherry picked from commit 3f0c9e5fe36d201de021d989b23ebaeb2d9a027b)
 2015-09-14 12:54:01 -07:00
Zhijie Shen
f4154bdee8 YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong. 
(cherry picked from commit 85f6d67fa78511f255fcfa810afc9a156a7b483b)
(cherry picked from commit 426535007bcdc67331f7a37b5d69cc20b37c26e0)
 2015-09-11 11:45:29 -07:00
Vinod Kumar Vavilapalli
3462a00dd2 Preparing for release 2.6.1: mvn versions:set -DnewVersion=2.6.1 2015-09-09 15:29:57 -07:00
Jian He
2b526ba757 YARN-4047. ClientRMService getApplications has high scheduler lock contention. Contributed by Jason Lowe 
(cherry picked from commit 7a445fcfabcf9c6aae219051f65d3f6cb8feb87c)

(cherry picked from commit 703fa1b141a98449746bd6fb3b144e74d964d1f5)
 2015-09-08 22:57:35 -07:00
Xuan
d59bf81e08 YARN-3999. RM hangs on draing events. Contributed by Jian He 
(cherry picked from commit 3ae716fa696b87e849dae40225dc59fb5ed114cb)
(cherry picked from commit 2ebdf5bfcee9ede80681a5266df225885d830883)
 2015-09-08 22:57:35 -07:00
Jonathan Eagles
6ed2486c7e YARN-3978. Configurably turn off the saving of container info in Generic AHS (Eric Payne via jeagles) 
(cherry picked from commit 3cd02b95224e9d43fd63a4ef9ac5c44f113f710d)
(cherry picked from commit 899df5bce03ea4f94487e48c1d38bd30ae10c26f)
 2015-09-08 22:57:34 -07:00
Jian He
92742b4402 YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R 
(cherry picked from commit 258623ff8bb1a1057ae3501d4f20982d5a59ea34)

(cherry picked from commit 1d1e7682c9cad6a2f819b390ca3368dfa29c7097)
 2015-09-08 22:57:28 -07:00
Jian He
2336264900 YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan 
(cherry picked from commit f489a4ec969f3727d03c8e85d51af1018fc0b2a1)

(cherry picked from commit d817fbb34d6e34991c6e512c20d71387750a98f4)
 2015-09-06 14:15:33 -07:00
Jian He
ee2b6bc248 YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan 
(cherry picked from commit 18a594257e052e8f10a03e5594e6cc6901dc56be)

(cherry picked from commit 1be2d64ddddaa6322909073cfaf7f2f2eb46e18d)
 2015-09-06 11:54:40 -07:00
Jian He
637e7f9e39 YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan 
(cherry picked from commit c1957fef29b07fea70938e971b30532a1e131fd0)

(cherry picked from commit 3ddafaa7c854dcf21ecc790c276927e7c869e62c)
 2015-09-05 21:07:51 -07:00
Jian He
4c94f07140 YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan 
(cherry picked from commit 21d80b3dd90a8e33e51701887c8d9369ed4ab17d)

(cherry picked from commit c0b1311a93614becc4a255af48fb7b697d491b80)
 2015-09-05 20:54:20 -07:00
Jian He
d9281fbbab YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan 
(cherry picked from commit 86358221fc85a7743052a0b4c1647353508bf308)

(cherry picked from commit cabf97ae4f2dad53c7b9e3d10a67876b16d94074)
 2015-09-05 20:54:20 -07:00
Jian He
b0ad553841 YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan 
(cherry picked from commit 6f9fe76918bbc79109653edc6cde85df05148ba3)

(cherry picked from commit 61b4116b4b3c0eec8f514f079debd88bc757b28e)
 2015-09-05 20:54:19 -07:00
Jian He
419e18cb37 YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena 
(cherry picked from commit dd57c2047bfd21910acc38c98153eedf1db75169)

(cherry picked from commit c61e8a7bfa7236e354f859a889083fab3d7ca9eb)
 2015-09-05 20:54:19 -07:00
Jian He
88f022da24 YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan 
(cherry picked from commit fdf042dfffa4d2474e3cac86cfb8fe9ee4648beb)

(cherry picked from commit 411836b74c6c02c0b5aebbbce29c209d93db1de2)
 2015-09-05 20:54:18 -07:00
Wangda Tan
85d92721a4 YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda) 
(cherry picked from commit ebd797c48fe236b404cf3a125ac9d1f7714e291e)
(cherry picked from commit 78d626fa892415023827e35ad549636e2a83275d)
 2015-09-03 17:43:01 -07:00
Jian He
f1b35ffd4c YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch 
(cherry picked from commit c53420f58364b11fbda1dace7679d45534533382)

(cherry picked from commit 4931600030e13d9332d9a0e588487cb8684c667d)
 2015-09-03 17:40:24 -07:00
Jason Lowe
ca7fe71000 YARN-3990. AsyncDispatcher may overloaded with RMAppNodeUpdateEvent when Node is connected/disconnected. Contributed by Bibin A Chundatt 
(cherry picked from commit 32e490b6c035487e99df30ce80366446fe09bd6c)

(cherry picked from commit c31e3ba92132f232bd56b257f3854ffe430fbab9)
(cherry picked from commit 07d31d4c0808a169f4770187d655f38aa105255c)
 2015-09-03 14:40:20 -07:00
Xuan
7b1a71a7ad YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang 
(cherry picked from commit b0ad644083a0dfae3a39159ac88b6fc09d846371)
(cherry picked from commit 802676e1be350785d8c0ad35f6676eeb85b2467b)
(cherry picked from commit 2cadeb9e017c6a75db16e1f23b2accda04f12298)
 2015-09-03 11:54:23 -07:00
Wangda Tan
e081593042 YARN-3487. CapacityScheduler scheduler lock obtained unnecessarily when calling getQueue (Jason Lowe via wangda) 
(cherry picked from commit f47a5763acd55cb0b3f16152c7f8df06ec0e09a9)
(cherry picked from commit 3316cd4357ff6ccc4c76584813092adb1c2b4d43)
(cherry picked from commit 24d45ee9544abcfcf9e611ab835ec2f824333670)
 2015-09-02 11:28:22 -07:00
Wangda Tan
61f2ddb125 YARN-3493. RM fails to come up with error "Failed to load/recover state" when mem settings are changed. (Jian He via wangda) 
(cherry picked from commit f65eeb412d140a3808bcf99344a9f3a965918f70)
(cherry picked from commit e7cbecddc3e7ca5386c71aa4deb67f133611415c)
(cherry picked from commit 9d47d5aa5bffe427c4a77260f7ccc039d446e1fd)
 2015-09-02 11:14:35 -07:00
Vinod Kumar Vavilapalli
752e3da738 YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp. 
(cherry picked from commit 9c5911294e0ba71aefe4763731b0e780cde9d0ca)
(cherry picked from commit 1ff3fd33ed6f2ac09c774cc42b0107c5dbd9c19d)
(cherry picked from commit 82c722aae86669325672dd10840447434f15e7fd)
 2015-09-01 21:31:00 -07:00