Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,678 Commits 326 Branches 371 Tags
Commit Graph

722 Commits

Author SHA1 Message Date
Wangda Tan
881084fe5c YARN-3251. Fixed a deadlock in CapacityScheduler when computing absoluteMaxAvailableCapacity in LeafQueue (Craig Welch via wangda) 2015-02-26 17:05:25 -08:00
Vinod Kumar Vavilapalli
56020955fd YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user kill request races with ApplicationMaster finish. Contributed by Jian He. 
(cherry picked from commit 3651fe1b089851b38be351c00a9899817166bf3e)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-11-13 10:40:09 -08:00
Karthik Kambatla
b579c3d405 YARN-2635. TestRM, TestRMRestart, TestClientToAMTokens should run with both CS and FS. (Wei Yan and kasha via kasha) 
(cherry picked from commit 80d11eb68e60f88e16d7d41edecbddfc935a6b10)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java
 2014-11-13 10:36:13 -08:00
Arun C. Murthy
60584c732f Preparing to release hadoop-2.6.0: Set version in branch-2.6 to 2.6.1-SNAPSHOT. 2014-11-09 19:21:11 -08:00
Vinod Kumar Vavilapalli
6a9534e9cf YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery consistent with the (somewhat incorrect) behaviour in the non-recovery case. Contributed by Jian He. 
Fixed a minor import issue in the test during cherry-pick from trunk.

(cherry picked from commit e76faebc9589654e83c8244ef9aff88391e56b80)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestWorkPreservingRMRestart.java
 2014-11-09 19:09:48 -08:00
Arun C. Murthy
c2cc879f01 YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 
(cherry picked from commit 43cd07b408c6613d2c9aa89203cfa3110d830538)
 2014-11-09 15:04:55 -08:00
Zhijie Shen
33da0f8ecf YARN-2505. Supported get/add/remove/change labels in RM REST API. Contributed by Craig Welch. 
(cherry picked from commit 9a4e0d343e9e891c10ef6682e7b2231a59e69ade)
 2014-11-07 20:40:56 -08:00
Vinod Kumar Vavilapalli
6ff765bea2 YARN-2826. Fixed user-groups mappings' refresh bug caused by YARN-2826. Contributed by Wangda Tan. 
(cherry picked from commit df36edf751202db00d8f43103d7120ec56d70a04)
 2014-11-07 19:45:23 -08:00
Vinod Kumar Vavilapalli
6593aaf117 YARN-2753. Fixed a bunch of bugs in the NodeLabelsManager classes. Contributed by Zhihai xu. 
(cherry picked from commit 4cfd5bc7c18bb9a828f573b5c4d2b13fa28e732a)
 2014-11-07 14:17:23 -08:00
Vinod Kumar Vavilapalli
71a18a5303 YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan. 
(cherry picked from commit 2ac1be7dec4aef001e3162e364249933b2c4a6c4)
 2014-11-07 10:45:01 -08:00
Xuan
09955ea2c3 YARN-2810. TestRMProxyUsersConf fails on Windows VMs. Contributed by Varun Vasudev 
(cherry picked from commit 1e97f2f09464e871773188f642f3a01b744c580f)
(cherry picked from commit bf795418686e1559db4c37c0b107bb5c08bbf525)
 2014-11-07 09:46:53 -08:00
Vinod Kumar Vavilapalli
9f7396be55 YARN-2823. Fixed ResourceManager app-attempt state machine to inform schedulers about previous finished attempts of a running appliation to avoid expectation mismatch w.r.t transferred containers. Contributed by Jian He. 
(cherry picked from commit a5657182a7accebe08cd86e46b4cdeb163d4d1f2)
 2014-11-07 09:30:31 -08:00
Vinod Kumar Vavilapalli
21ef5afafa YARN-2744. Fixed CapacityScheduler to validate node-labels correctly against queues. Contributed by Wangda Tan. 
(cherry picked from commit a3839a9fbfb8eec396b9bf85472d25e0ffc3aab2)
 2014-11-06 17:29:39 -08:00
Jian He
e29e864c51 YARN-2579. Fixed a deadlock issue when EmbeddedElectorService and FatalEventDispatcher try to transition RM to StandBy at the same time. Contributed by Rohith Sharmaks 
(cherry picked from commit 395275af8622c780b9071c243422b0780e096202)
 2014-11-05 17:03:26 -08:00
Vinod Kumar Vavilapalli
812ddc3991 YARN-2805. Fixed ResourceManager to load HA configs correctly before kerberos login. Contributed by Wangda Tan. 
(cherry picked from commit 834e931d8efe4d806347b266e7e62929ce05389b)
 2014-11-05 15:32:49 -08:00
Zhijie Shen
a1764e4d33 YARN-2767. Added a test case to verify that http static user cannot kill or submit apps in the secure mode. Contributed by Varun Vasudev. 
(cherry picked from commit 7a4c92a9d55fcecef066053ac30dff0fcd4ec90c)
 2014-11-05 11:04:38 -08:00
Karthik Kambatla
36993e39d0 YARN-2010. Handle app-recovery failures gracefully. (Jian He and Karthik Kambatla via kasha) 
(cherry picked from commit b2cd2698028118b6384904732dbf94942f644732)
 2014-11-04 17:49:56 -08:00
Vinod Kumar Vavilapalli
f2ef8c7b48 YARN-2795. Fixed ResourceManager to not crash loading node-label data from HDFS in secure mode. Contributed by Wangda Tan. 
(cherry picked from commit ec6cbece8e7772868ce8ad996135d3136bd32245)
 2014-11-03 13:46:05 -08:00
Zhijie Shen
3e41828639 YARN-2785. Fixed intermittent TestContainerResourceUsage failure. Contributed by Varun Vasudev. 
(cherry picked from commit 27715ec63bd77f1d31ee922b7daba85071da54ca)
 2014-11-02 15:24:07 -08:00
Vinod Kumar Vavilapalli
a3f032031d YARN-2779. Fixed ResourceManager to not require delegation tokens for communicating with Timeline Service. Contributed by Zhijie Shen. 
(cherry picked from commit d1828d94435eca21761b0ba8458f9de2f125d012)
 2014-10-30 23:17:50 -07:00
Vinod Kumar Vavilapalli
85fcdaf8d8 YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan. 
(cherry picked from commit e0233c16ebd06bb3aabeb523cd17259008e243ef)
 2014-10-30 23:03:02 -07:00
Jian He
e181498a67 YARN-2503. Added node lablels in web UI. Contributed by Wangda Tan 
(cherry picked from commit d5e0a09721a5156fa2ee51ac1c32fbfd9905b8fb)
(cherry picked from commit 720de7eb4ca310e8d800938fbb56fd4de06fc6a5)
 2014-10-28 18:02:13 -07:00
Vinod Kumar Vavilapalli
0af8cc9ca2 YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the sake of localization and log-aggregation for long-running services. Contributed by Jian He. 
(cherry picked from commit a16d022ca4313a41425c8e97841c841a2d6f2f54)
 2014-10-27 15:51:20 -07:00
Xuan
55c5612caf YARN-2726. CapacityScheduler should explicitly log when an accessible label has no capacity. Contributed by Wangda Tan 2014-10-27 10:30:09 -07:00
Vinod Kumar Vavilapalli
259db7a726 YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken identifiers to be tampered and thus causing app submission failures in secure mode. Contributed by Jian He. 
(cherry picked from commit 018664550507981297fd9f91e29408e6b7801ea9)
 2014-10-26 11:29:06 -07:00
Jian He
728cc2394c YARN-1915. Fixed a race condition that client could use the ClientToAMToken to contact with AM before AM actually receives the ClientToAMTokenMasterKey. Contributed by Jason Lowe 
(cherry picked from commit 5864dd99a419b8a0fb973a4984c0f1d1e02ccf16)
 2014-10-24 22:49:19 -07:00
Zhijie Shen
09e422302f YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 
(cherry picked from commit 0f3b6900be1a3b2e4624f31f84656f4a32dadce9)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-23 22:18:27 -07:00
Zhijie Shen
b1ca12bc0b YARN-2721. Suppress NodeExist exception thrown by ZKRMStateStore when it retries creating znode. Contributed by Jian He. 
(cherry picked from commit 7e3b5e6f5cb4945b4fab27e8a83d04280df50e17)
 2014-10-21 21:53:27 -07:00
Vinod Kumar Vavilapalli
a1c5d4c51f YARN-2715. Fixed ResourceManager to respect common configurations for proxy users/groups beyond just the YARN level config. Contributed by Zhijie Shen. 
(cherry picked from commit c0e034336c85296be6f549d88d137fb2b2b79a15)
 2014-10-21 20:11:14 -07:00
Vinod Kumar Vavilapalli
88b7e5ca65 YARN-2504. Enhanced RM Admin CLI to support management of node-labels. Contribyted by Wangda Tan. 
(cherry picked from commit 82567664988b673f1b819a42a4baf31cb0dcb331)
 2014-10-18 12:09:02 -07:00
Vinod Kumar Vavilapalli
24b0529a9f YARN-2676. Enhanced Timeline auth-filter to support proxy users. Contributed by Zhijie Shen. 
(cherry picked from commit 39063cd36f96e351e4a6bf0bc2b6185711d4b059)
 2014-10-17 22:04:14 -07:00
Vinod Kumar Vavilapalli
ce42210651 YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing test-failures: added a dummy in-memory labels-manager. Contributed by Wangda Tan. 
(cherry picked from commit e9c66e8fd2ccb658db2848e1ab911f1502de4de5)
 2014-10-17 18:27:34 -07:00
Jian He
4d03915be9 YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit c3de2412eb7633ff16c67e71e73bbe27a982d984)
 2014-10-17 16:36:45 -07:00
Jian He
b7504f0394 YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks 
(cherry picked from commit a6aa6e42cacdbfcc1c2b7c19e7239204fe9ff654)
 2014-10-17 10:55:29 -07:00
Vinod Kumar Vavilapalli
778fbcd5d3 YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan. 
(cherry picked from commit abae63caf9c53b404f2f2db7d482555484eaeaf8)
 2014-10-17 08:59:49 -07:00
Vinod Kumar Vavilapalli
8a09fdccb3 YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan. 
(cherry picked from commit b3056c266a628a65cf7ceb61b55ab3bd0a09baf2)
 2014-10-15 18:49:06 -07:00
Vinod Kumar Vavilapalli
fe80fd5e88 YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan. 
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.

(cherry picked from commit f2ea555ac6c06a3f2f6559731f48711fff05d3f1)
 2014-10-15 18:35:34 -07:00
Jian He
bd45dba807 YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit f93d2ea27e7554272845fb11d97b3564704834b3)
 2014-10-15 15:31:57 -07:00
Zhijie Shen
eb055eadd7 YARN-2656. Made RM web services authentication filter support proxy user. Contributed by Varun Vasudev and Zhijie Shen. 
(cherry picked from commit 1220bb72d452521c6f09cebe1dd77341054ee9dd)
 2014-10-14 21:58:12 -07:00
Zhijie Shen
e71fa82ee5 HADOOP-11181. Generalized o.a.h.s.t.d.DelegationTokenManager to handle all sub-classes of AbstractDelegationTokenIdentifier. Contributed by Zhijie Shen. 
(cherry picked from commit cdce88376a60918dfe2f3bcd82a7666d74992a19)
 2014-10-14 11:39:06 -07:00
Jian He
29496c9be2 YARN-2308. Changed CapacityScheduler to explicitly throw exception if the queue 
to which the apps were submitted is changed across RM restart. Contributed by Craig Welch & Chang Li
(cherry picked from commit f9680d9a160ee527c8f2c1494584abf1a1f70f82)
 2014-10-13 14:16:57 -07:00
Zhijie Shen
6da637ec12 YARN-2651. Spun off LogRollingInterval from LogAggregationContext. Contributed by Xuan Gong. 
(cherry picked from commit 4aed2d8e91c7dccc78fbaffc409d3076c3316289)
 2014-10-13 10:56:22 -07:00
Vinod Kumar Vavilapalli
bb6c79f76c YARN-2494. Added NodeLabels Manager internal API and implementation. Contributed by Wangda Tan. 
(cherry picked from commit db7f1653198b950e89567c06898d64f6b930a0ee)
 2014-10-10 11:46:10 -07:00
Jian He
7ec65c8617 YARN-2649. Fixed TestAMRMRPCNodeUpdates test failure. Contributed by Ming Ma 
(cherry picked from commit e16e25ab1beac89c8d8be4e9f2a7fbefe81d35f3)
 2014-10-08 11:03:28 -07:00
Jian He
5ff984f33f YARN-1857. CapacityScheduler headroom doesn't account for other AM's running. Contributed by Chen He and Craig Welch 
(cherry picked from commit 30d56fdbb40d06c4e267d6c314c8c767a7adc6a3)
 2014-10-07 13:48:46 -07:00
Jian He
efc482d1f7 YARN-2644. Fixed CapacityScheduler to return up-to-date headroom when AM allocates. Contributed by Craig Welch 
(cherry picked from commit 519e5a7dd2bd540105434ec3c8939b68f6c024f8)
 2014-10-06 15:52:05 -07:00
Jian He
73a1b77223 YARN-2615. Changed ClientToAMTokenIdentifier/RM(Timeline)DelegationTokenIdentifier to use protobuf as payload. Contributed by Junping Du 
(cherry picked from commit ea26cc0b4ac02b8af686dfda80f540e5aa70c358)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/security/TestClientToAMTokens.java
 2014-10-06 10:52:20 -07:00
Chris Douglas
0b2dedc42d YARN-1051. Add a system for creating reservations of cluster capacity. 
Contributed by Subru Krishnan and Carlo Curino.
(cherry picked from commit c8212bacb1b2a7e6ee83cc56f72297465ce99390)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit ebf4aae60f7a641ac7406e27d43528cace20d4ff)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-06 10:30:04 -07:00
subru
114f09226e YARN-2611. Fixing jenkins findbugs warning and TestRMWebServicesCapacitySched for branch YARN-1051. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit c47464aba407d1dafe10be23fe454f0489cc4367)
(cherry picked from commit a2986234be4e02f9ccb589f9ff5f7ffb28bc6400)
(cherry picked from commit fb5e9df7fd09a5cbc2c98bc9c875faf6a3b8f7c0)
 2014-10-06 10:29:14 -07:00
subru
bf4394e160 YARN-2576. Making test patch pass in branch. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit 90ac0be86b898aefec5471db4027554c8e1b310c)
(cherry picked from commit 5e10a13bb4759984494c6a870c7f08fb6693c9c0)
(cherry picked from commit ae0f16ccc837de2e32d7137cf48d5b17cb7c18d4)
 2014-10-06 10:29:14 -07:00