b244701fad
YARN-1402. Update related Web UI and CLI with exposing client API to check log aggregation status. Contributed by Xuan Gong.
...
(cherry picked from commit 1db355a875
2015-04-17 13:20:27 -07:00
42cf8e0554
YARN-3021. YARN's delegation-token handling disallows certain trust setups to operate properly over DistCp. Contributed by Yongjun Zhang
...
(cherry picked from commit bb6dde68f1
2015-04-16 19:44:32 -07:00
51c458a5c4
MAPREDUCE-6300. Task list sort by task id broken. Contributed by Siqi Li.
...
(cherry picked from commit 1fa8075794
2015-04-16 10:06:03 -05:00
6fed2c2a79
YARN-3354. Add node label expression in ContainerTokenIdentifier to support RM recovery. Contributed by Wangda Tan
...
(cherry picked from commit 1b89a3e173
2015-04-15 14:03:29 -07:00
e977247f3e
YARN-3326. Support RESTful API for getLabelsToNodes. Contributed by Naganarasimha G R.
...
(cherry picked from commit e48cedc663
2015-04-15 14:04:24 -05:00
89a7c98436
YARN-3318. Create Initial OrderingPolicy Framework and FifoOrderingPolicy. (Craig Welch via wangda)
...
(cherry picked from commit 5004e75332
2015-04-15 09:57:52 -07:00
9ebbf1bfce
YARN-3361. CapacityScheduler side changes to support non-exclusive node labels. Contributed by Wangda Tan
...
(cherry picked from commit 0fefda645b
2015-04-14 11:46:35 -07:00
81bbee6852
YARN-3266. RMContext#inactiveNodes should have NodeId as map key. Contributed by Chengbing Liu
...
(cherry picked from commit b46ee1e7a3
2015-04-14 11:31:42 -07:00
d58f5c8894
YARN-3472. Fixed possible leak in DelegationTokenRenewer#allTokens. Contributed by Rohith Sharmaks
...
(cherry picked from commit a1afbc48b5
2015-04-13 14:07:51 -07:00
32b01263d3
YARN-3382. Some of UserMetricsInfo metrics are incorrectly set to root queue metrics. Contributed by Rohit Agarwal
...
(cherry picked from commit 944a16579f
2015-04-13 10:54:15 -07:00
ad99d268e5
YARN-3469. ZKRMStateStore: Avoid setting watches that are not required. (Jun Hong via kasha)
...
(cherry picked from commit e516706b89
2015-04-10 11:21:30 -07:00
d3daf9665c
YARN-3348. Add a 'yarn top' tool to help understand cluster usage. Contributed by Varun Vasudev
2015-04-10 10:40:31 -07:00
0891de0712
YARN-1376. NM need to notify the log aggregation status to RM through Node heartbeat. Contributed by Xuan Gong.
...
(cherry picked from commit 92431c9617
2015-04-10 08:58:07 -07:00
f5b49160d9
YARN-3293. Track and display capacity scheduler health metrics in web
...
UI. Contributed by Varun Vasudev
(cherry picked from commit afa5d4715a
2015-04-09 23:40:02 -07:00
1ff3fd33ed
YARN-3055. Fixed ResourceManager's DelegationTokenRenewer to not stop token renewal of applications part of a bigger workflow. Contributed by Daryn Sharp.
...
(cherry picked from commit 9c5911294e
2015-04-09 13:10:06 -07:00
73d9e4c77b
YARN-3466. Fix RM nodes web page to sort by node HTTP-address, #containers and node-label column. (Jason Lowe via wangda)
...
(cherry picked from commit 1885141e90
2015-04-09 10:36:10 -07:00
0522d6970d
YARN-3294. Allow dumping of Capacity Scheduler debug logs via web UI for
...
a fixed time period. Contributed by Varun Vasudev
(cherry picked from commit d27e9241e8
2015-04-07 09:54:29 -07:00
354d2c75cf
YARN-2429. TestAMRMTokens.testTokenExpiry fails Intermittently with error message:Invalid AMRMToken (zxu via rkanter)
...
(cherry picked from commit 99b08a748e
2015-04-06 14:11:50 -07:00
d2b2d76cce
TestFairScheduler.testContinuousScheduling fails Intermittently. Contributed by Zhihai Xu.
...
(cherry picked from commit 53959e69f7
2015-04-06 20:19:47 +09:00
1eb6d74967
YARN-3435. AM container to be allocated Appattempt AM container shown as
...
null. Contributed by Bibin A Chundatt
(cherry picked from commit 96d72118f5
2015-04-05 00:43:22 -07:00
405aead879
YARN-2901. Add errors and warning metrics page to RM, NM web UI. (Varun Vasudev via wangda)
...
(cherry picked from commit bad070fe15
2015-04-02 17:23:44 -07:00
6286cfd711
YARN-3415. Non-AM containers can be counted towards amResourceUsage of a fairscheduler queue (Zhihai Xu via Sandy Ryza)
2015-04-02 13:57:53 -07:00
6d0bebc912
YARN-3430. Made headroom data available on app attempt page of RM WebUI. Contributed by Xuan Gong.
...
(cherry picked from commit 8366a36ad3
2015-04-01 13:49:00 -07:00
e26b6e55e9
YARN-3248. Display count of nodes blacklisted by apps in the web UI.
...
Contributed by Varun Vasudev
(cherry picked from commit 4728bdfa15
2015-04-01 04:20:37 -07:00
85c420a4d4
YARN-3412. RM tests should use MockRM where possible. (kasha)
...
(cherry picked from commit 79f7f2aabf
2015-03-31 09:22:16 -07:00
99b8255693
YARN-3258. FairScheduler: Need to add more logging to investigate allocations. Contributed by Anubhav Dhoot.
...
(cherry picked from commit b5a22e9838
2015-03-31 17:43:07 +09:00
cba4ed1678
YARN-2495. Allow admin specify labels from each NM (Distributed configuration for node label). (Naganarasimha G R via wangda)
...
(cherry picked from commit 2a945d24f7
2015-03-30 12:05:54 -07:00
29629020c1
YARN-3383. AdminService should use warn instead of info to log exception when operation fails. (Li Lu via wangda)
...
(cherry picked from commit 97a7277a2d
2015-03-24 10:39:45 -07:00
4e0c48703e
YARN-2868. FairScheduler: Metric for latency to allocate first container for an application. (Ray Chiang via kasha)
...
(cherry picked from commit 972f1f1ab9
2015-03-23 14:10:23 -07:00
75591e4131
YARN-3241. FairScheduler handles invalid queue names inconsistently. (Zhihai Xu via kasha)
...
(cherry picked from commit 2bc097cd14
2015-03-23 13:24:22 -07:00
342c525eaa
YARN-3336. FileSystem memory leak in DelegationTokenRenewer.
...
(cherry picked from commit 6ca1f12024
2015-03-23 10:46:06 -07:00
9c494cedaf
YARN-3345. Add non-exclusive node label API. Contributed by Wangda Tan
...
(cherry picked from commit e1feb4ea1a
2015-03-20 19:05:14 -07:00
a3a155a34e
YARN-3356. Capacity Scheduler FiCaSchedulerApp should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan
...
(cherry picked from commit 586348e4cb
2015-03-20 13:54:46 -07:00
8e142d27cb
YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die. (Brahma Reddy Battula via wangda)
...
(cherry picked from commit 6bc7710ec7
2015-03-20 10:42:51 -07:00
3f0c9e5fe3
YARN-3379. Fixed missing data in localityTable and ResourceRequests table in RM WebUI. Contributed by Xuan Gong
...
(cherry picked from commit 4e886eb9cb
2015-03-19 22:27:57 -07:00
a888dac344
YARN-3357. Move TestFifoScheduler to FIFO package. Contributed by Rohith
...
Sharmaks.
(cherry picked from commit 93d0f4acc8
2015-03-19 12:18:08 +05:30
15ebacf03a
YARN-3273. Improve scheduler UI to facilitate scheduling analysis and debugging. Contributed Rohith Sharmaks
...
(cherry picked from commit 658097d6da
2015-03-17 21:31:20 -07:00
b00b216a91
YARN-3205. FileSystemRMStateStore should disable FileSystem Cache to avoid get a Filesystem with an old configuration. Contributed by Zhihai Xu.
...
(cherry picked from commit 3bc72cc16d
2015-03-18 11:53:40 +09:00
ab34e6975b
YARN-3305. Normalize AM resource request on app submission. Contributed by Rohith Sharmaks
...
(cherry picked from commit 968425e9f7
2015-03-17 13:50:44 -07:00
1e77d92d62
Revert "YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)"
...
This reverts commit c2b185def832b4330456
2015-03-17 12:31:44 -07:00
1c601e492f
YARN-3243. CapacityScheduler should pass headroom from parent to children to make sure ParentQueue obey its capacity limits. Contributed by Wangda Tan.
...
(cherry picked from commit 487374b7fe
2015-03-17 10:25:07 -07:00
895588b439
YARN-3197. Confusing log generated by CapacityScheduler. Contributed by
...
Varun Saxena.
(cherry picked from commit 7179f94f9d
2015-03-17 15:59:19 +05:30
a20427d339
YARN-3349. Treat all exceptions as failure in TestFSRMStateStore#testFSRMStateStoreClientRetry. Contributed by Zhihai Xu.
...
(cherry picked from commit 7522a643fa
2015-03-17 08:10:12 +09:00
d6e3ef2506
YARN-1453. [JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments. Contributed by Akira AJISAKA, Andrew Purtell, and Allen Wittenauer.
...
(cherry picked from commit 3da9a97cfb
2015-03-16 23:20:17 +09:00
53aa3a4d1f
YARN-3154. Added additional APIs in LogAggregationContext to avoid aggregating running logs of application when rolling is enabled. Contributed by Xuan Gong.
...
(cherry picked from commit 863079bb87
2015-03-12 13:33:42 -07:00
426535007b
YARN-1884. Added nodeHttpAddress into ContainerReport and fixed the link to NM web page. Contributed by Xuan Gong.
...
(cherry picked from commit 85f6d67fa7
2015-03-11 19:39:46 -07:00
4021451746
YARN-3300. Outstanding_resource_requests table should not be shown in AHS. Contributed by Xuan Gong
...
(cherry picked from commit c3003eba6f
2015-03-09 20:47:33 -07:00
b46f9e72db
HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa)
2015-03-09 19:53:55 +09:00
d9416317a9
Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2.
2015-03-08 20:32:51 -07:00
ef3d9bdf6b
YARN-3275. CapacityScheduler: Preemption happening on non-preemptable queues. Contributed by Eric Payne
...
(cherry picked from commit 27e8ea820f
2015-03-06 22:45:09 +00:00
a5f3fb4dc1
YARN-1809. Synchronize RM and TimeLineServer Web-UIs. Contributed by Zhijie Shen and Xuan Gong
...
(cherry picked from commit 95bfd087dc
2015-03-05 21:27:03 -08:00
6660c2f83b
YARN-3249. Add a 'kill application' button to Resource Manager's Web UI. Contributed by Ryu Kobayashi.
...
(cherry picked from commit 1b67209612
2015-03-05 19:56:59 +09:00
0d62e94887
YARN-3242. Asynchrony in ZK-close can lead to ZKRMStateStore watcher receiving events for old client. (Zhihai Xu via kasha)
...
(cherry picked from commit 8d88691d16
2015-03-04 19:49:21 -08:00
721d7b5741
YARN-3231. FairScheduler: Changing queueMaxRunningApps interferes with pending jobs. (Siqi Li via kasha)
...
(cherry picked from commit 22426a1c9f
2015-03-04 18:07:33 -08:00
888a445638
YARN-3222. Fixed NPE on RMNodeImpl#ReconnectNodeTransition when a node is reconnected with a different port. Contributed by Rohith Sharmaks
...
(cherry picked from commit b2f1ec312e
2015-03-03 16:28:55 -08:00
0a502c665b
YARN-3272. Surface container locality info in RM web UI (Jian He via wangda)
...
(cherry picked from commit e17e5ba9d7
2015-03-03 11:49:45 -08:00
253c78548b
YARN-3265. Fixed a deadlock in CapacityScheduler by always passing a queue's available resource-limit from the parent queue. Contributed by Wangda Tan.
...
(cherry picked from commit 14dd647c55
2015-03-02 17:54:01 -08:00
76b7283fa1
YARN-3281. Added RMStateStore to StateMachine visualization list. Contributed by Chengbing Liu
...
(cherry picked from commit 5d0bae550f
2015-03-02 14:43:12 -08:00
0b0be0056b
YARN-3262. Surface application outstanding resource requests table in RM web UI. (Jian He via wangda)
...
(cherry picked from commit edcecedc1c
2015-02-27 16:14:35 -08:00
79f73f4613
YARN-2820. Retry in FileSystemRMStateStore when FS's operations fail due to IOException. Contributed by Zhihai Xu.
...
(cherry picked from commit 01a1621930
2015-02-28 00:57:01 +09:00
9e67f2cb0d
YARN-3255. RM, NM, JobHistoryServer, and WebAppProxyServer's main() should support generic options. Contributed by Konstantin Shvachko.
2015-02-26 17:26:08 -08:00
f83d2e4410
YARN-3256. TestClientToAMTokens#testClientTokenRace is not running against
...
all Schedulers even when using ParameterizedSchedulerTestBase. Contributed
by Anubhav Dhoot.
(cherry picked from commit 0d4296f0e0
2015-02-26 15:47:32 +05:30
13e213c41a
YARN-3247. TestQueueMappings should use CapacityScheduler explicitly. Contributed by Zhihai Xu.
...
(cherry picked from commit 6cbd9f1113
2015-02-25 11:23:34 +09:00
0ca974d0b0
YARN-2797. TestWorkPreservingRMRestart should use
...
ParametrizedSchedulerTestBase. Contributed by Karthik Kambatla
(cherry picked from commit fe7a302473
2015-02-21 19:19:08 -08:00
a1963968d2
YARN-3230. Clarify application states on the web UI. (Jian He via wangda)
...
(cherry picked from commit ce5bf927c3
2015-02-20 10:40:04 -08:00
3e8dfd1299
YARN-3194. RM should handle NMContainerStatuses sent by NM while registering if NM is Reconnected node. Contributed by Rohith
...
(cherry picked from commit a64dd3d24b
2015-02-20 15:12:08 +00:00
eaeaf80d39
YARN-933. Fixed InvalidStateTransitonException at FINAL_SAVING state in RMApp. Contributed by Rohith Sharmaks
...
(cherry picked from commit c0d9b93953
2015-02-19 15:43:14 -08:00
cd5eb9c1de
YARN-3076. Add API/Implementation to YarnClient to retrieve label-to-node mapping (Varun Saxena via wangda)
...
(cherry picked from commit d49ae725d5
2015-02-19 11:01:47 -08:00
fef5efa0cb
YARN-1615. Fix typos in delay scheduler's description. Contributed by Akira Ajisaka.
...
(cherry picked from commit b8a14efdf5
2015-02-19 10:24:40 +09:00
b8fe1a747e
YARN-1514. Utility to benchmark ZKRMStateStore#loadState for RM HA. Contributed by Tsuyoshi OZAWA
...
(cherry picked from commit 1c03376300
2015-02-18 16:34:42 -08:00
8f06eaa251
YARN-3132. RMNodeLabelsManager should remove node from node-to-label mapping when node becomes deactivated. Contributed by Wangda Tan
...
(cherry picked from commit f5da5566d9
2015-02-18 11:52:28 -08:00
fabac2ff30
YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and Devaraj K.
...
(cherry picked from commit 556386a070
2015-02-17 01:54:09 +09:00
42368b91c4
Revert "YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj."
...
This reverts commit 67e3a77720
2015-02-17 01:53:58 +09:00
67e3a77720
YARN-1299. Improve a log message in AppSchedulingInfo by adding application id. Contributed by Ashutosh Jindal and devaraj.
...
(cherry picked from commit 9aae81c934
2015-02-17 01:48:53 +09:00
613065d75a
YARN-3182. Cleanup switch statement in ApplicationMasterLauncher#handle(). Contributed by Ray Chiang.
...
(cherry picked from commit 110cf6b9a3
2015-02-13 14:21:43 +09:00
5f0495689f
YARN-3158. Correct log messages in ResourceTrackerService. Contributed
...
by Varun Saxena
(cherry picked from commit 99f6bd4f7a
2015-02-12 16:43:28 -08:00
b5d6f76c6a
YARN-3104. Fixed RM to not generate new AMRM tokens on every heartbeat between rolling and activation. Contributed by Jason Lowe
...
(cherry picked from commit 18297e0972
2015-02-12 16:03:12 -08:00
fcfbdf6440
YARN-3181. FairScheduler: Fix up outdated findbugs issues. (kasha)
...
(cherry picked from commit c2b185def8
2015-02-12 15:39:50 -08:00
1be2d64ddd
YARN-3124. Fixed CS LeafQueue/ParentQueue to use QueueCapacities to track capacities-by-label. Contributed by Wangda Tan
...
(cherry picked from commit 18a594257e
2015-02-12 15:01:14 -08:00
cac6fe217a
YARN-3151. On Failover tracking url wrong in application cli for KILLED
...
application. Contributed by Rohith
(cherry picked from commit 65c69e296e
2015-02-11 21:21:06 -08:00
f2c3058b8d
YARN-3160. Fix non-atomic operation on nodeUpdateQueue in RMNodeImpl. (Contributed by Chengbing Liu)
...
(cherry picked from commit c541a374d8
2015-02-11 03:46:40 -08:00
fd75b8c9ca
YARN-2246. Made the proxy tracking URL always be http(s)://proxy addr:port/proxy/<appId> to avoid duplicate sections. Contributed by Devaraj K.
...
(cherry picked from commit d5855c0e46
2015-02-10 15:29:09 -08:00
24d4d933f7
YARN-3100. Made YARN authorization pluggable. Contributed by Jian He.
...
(cherry picked from commit 23bf6c7207
2015-02-09 20:37:58 -08:00
6146680955
YARN-3094. Reset timer for liveness monitors after RM recovery. Contributed by Jun Gong
...
(cherry picked from commit 0af6a99a3f
2015-02-09 13:47:33 -08:00
1cb1c8b615
YARN-2990. FairScheduler's delay-scheduling always waits for node-local and rack-local delays, even for off-rack-only requests. (kasha)
...
(cherry picked from commit 7e42088abf
2015-02-08 22:48:29 -08:00
61fdd862be
YARN-3143. RM Apps REST API can return NPE or entries missing id and other fields. Contributed by Jason Lowe
...
(cherry picked from commit da2fb2bc46
2015-02-06 21:51:49 +00:00
3ddafaa7c8
YARN-2694. Ensure only single node label specified in ResourceRequest. Contributed by Wangda Tan
...
(cherry picked from commit c1957fef29
2015-02-06 11:34:56 -08:00
29c77de297
YARN-1904. Ensure exceptions thrown in ClientRMService & ApplicationHistoryClientService are uniform when application-attempt is not found. Contributed by Zhijie Shen.
2015-02-05 23:49:43 -08:00
b109051145
YARN-3145. Fixed ConcurrentModificationException on CapacityScheduler ParentQueue#getQueueUserAclInfo. Contributed by Tsuyoshi OZAWA
...
(cherry picked from commit 4641196fe0
2015-02-05 16:13:23 -08:00
ebdd88ec81
YARN-1582. Capacity Scheduler: add a maximum-allocation-mb setting per queue. Contributed by Thomas Graves
...
(cherry picked from commit 69c8a7f45b
2015-02-05 19:30:06 +00:00
84df660af4
YARN-3101. In Fair Scheduler, fix canceling of reservations for exceeding max share (Anubhav Dhoot via Sandy Ryza)
2015-02-05 09:45:17 -08:00
cdd76d5922
YARN-3058. Fix error message of tokens' activation delay configuration. Contributed by Yi Liu.
...
(cherry picked from commit 26dee1486b
2015-02-04 11:41:48 +09:00
3ab3caa9ea
YARN-2543. Made resource usage be published to the timeline server too. Contributed by Naganarasimha G R.
...
(cherry picked from commit e5b56e2af6
2015-02-03 17:35:40 -08:00
205e15c4a3
YARN-3075. NodeLabelsManager implementation to retrieve label to node mapping (Varun Saxena via wangda)
...
(cherry picked from commit 5bd984691b
2015-02-03 12:55:25 -08:00
c0b1311a93
YARN-3098. Created common QueueCapacities class in Capacity Scheduler to track capacities-by-labels of queues. Contributed by Wangda Tan
...
(cherry picked from commit 21d80b3dd9
2015-02-03 11:43:40 -08:00
3828fb5fab
YARN-3085. Application summary should include the application type. Contributed by Rohith
...
(cherry picked from commit a761bf8726
2015-02-03 14:57:25 +00:00
4941b8f30d
YARN-3077. Fixed RM to create zk root path recursively. Contributed by Chun Chen
...
(cherry picked from commit 054a947989
2015-01-30 17:35:26 -08:00
cabf97ae4f
YARN-3099. Capacity Scheduler LeafQueue/ParentQueue should use ResourceUsage to track used-resources-by-label. Contributed by Wangda Tan
...
(cherry picked from commit 86358221fc
2015-01-30 15:16:22 -08:00
1081b4c4a7
HADOOP-9907. Webapp http://hostname:port/metrics link is not working. (aajisaka)
...
(cherry picked from commit 342efa110a
2015-01-30 02:50:46 +09:00
58d7d1efc6
YARN-3079. Scheduler should also update maximumAllocation when updateNodeResource. (Zhihai Xu via wangda)
...
(cherry picked from commit 7882bc0f14
2015-01-28 21:56:17 -08:00
4718af4352
YARN-2932. Add entry for preemptable status (enabled/disabled) to scheduler web UI and queue initialize/refresh logging. (Eric Payne via wangda)
...
(cherry picked from commit 18741adf97
2015-01-27 15:37:02 -08:00
61b4116b4b
YARN-3092. Created a common ResourceUsage class to track labeled resource usages in Capacity Scheduler. Contributed by Wangda Tan
...
(cherry picked from commit 6f9fe76918
2015-01-26 15:38:27 -08:00
25f2cf8a4e
YARN-2800. Remove MemoryNodeLabelsStore and add a way to enable/disable node labels feature. Contributed by Wangda Tan.
...
(cherry picked from commit 24aa462673
2015-01-23 20:37:26 +09:00
ef6fc24dfb
YARN-2933. Capacity Scheduler preemption policy should only consider capacity without labels temporarily. Contributed by Mayank Bansal
...
(cherry picked from commit 0a2d3e717d
2015-01-19 16:57:17 -08:00
7cddec31d7
YARN-3064. TestRMRestart/TestContainerResourceUsage/TestNodeManagerResync failure with allocation timeout. (Contributed by Jian He)
...
(cherry picked from commit 5d1cca34fa
2015-01-16 00:10:36 -08:00
ed5a0d0aa8
YARN-2861. Fixed Timeline DT secret manager to not reuse RM's configs. Contributed by Zhijie Shen
...
(cherry picked from commit 9e33116d1d
2015-01-15 11:08:19 -08:00
4931600030
YARN-2637. Fixed max-am-resource-percent calculation in CapacityScheduler when activating applications. Contributed by Craig Welch
...
(cherry picked from commit c53420f583
2015-01-13 17:32:36 -08:00
13cdcf28e0
YARN-2643. Don't create a new DominantResourceCalculator on every FairScheduler.allocate call. (kasha via rkanter)
...
(cherry picked from commit 51881535e6
2015-01-12 17:51:45 -08:00
e5059b91bb
YARN-3027. Scheduler should use totalAvailable resource from node instead of availableResource for maxAllocation. (adhoot via rkanter)
...
(cherry picked from commit ae7bf31fe1
2015-01-12 10:48:45 -08:00
36b3dcaab2
YARN-3014. Replaces labels on a host should update all NM's labels on that host. Contributed by Wangda Tan
...
(cherry picked from commit a260406268
2015-01-09 17:50:18 -08:00
d2fbba790a
YARN-2996. Improved synchronization and I/O operations of FS- and Mem- RMStateStore. Contributed by Yi Liu.
...
(cherry picked from commit dc2eaa26b2
2015-01-08 09:49:27 -08:00
d3b26454e4
YARN-3010. Fixed findbugs warning in AbstractYarnScheduler. Contributed by Yi Liu
...
(cherry picked from commit e13a484a2b
2015-01-07 14:12:54 -08:00
8a81312062
YARN-2427. Added the API of moving apps between queues in RM web services. Contributed by Varun Vasudev.
...
(cherry picked from commit 60103fca04
2015-01-06 14:39:44 -08:00
c61e8a7bfa
YARN-2978. Fixed potential NPE while getting queue info. Contributed by Varun Saxena
...
(cherry picked from commit dd57c2047b
2015-01-06 14:16:43 -08:00
7adffad2bb
YARN-2881. [YARN-2574] Implement PlanFollower for FairScheduler. (Anubhav Dhoot via kasha)
...
(cherry picked from commit 0c4b112677
2015-01-06 04:42:55 +05:30
d070597a0e
YARN-2360. Fair Scheduler: Display dynamic fair share for queues on the scheduler page. (Ashwin Shankar and Wei Yan via kasha)
...
(cherry picked from commit 270a271f53
2015-01-06 03:28:48 +05:30
c6cf748985
YARN-2958. Made RMStateStore not update the last sequence number when updating the delegation token. Contributed by Varun Saxena.
...
(cherry picked from commit 562a701945
2015-01-05 13:35:09 -08:00
c116743bdd
YARN-2922. ConcurrentModificationException in CapacityScheduler's LeafQueue. Contributed by Rohith Sharmaks.
...
(cherry picked from commit ddc5be48fc
2015-01-05 00:11:01 +09:00
798ab51289
YARN-2998. Abstract out scheduler independent PlanFollower components. (Anubhav Dhoot via kasha)
...
(cherry picked from commit e7257acd8a
2014-12-30 19:55:50 -08:00
cde5bfe3ec
YARN-2987. Fixed ClientRMService#getQueueInfo to check against queue and app ACLs. Contributed by Varun Saxena
...
(cherry picked from commit e2351c7ae2
2014-12-30 17:16:05 -08:00
143e48c25a
YARN-2493. Added node-labels page on RM web UI. Contributed by Wangda Tan
...
(cherry picked from commit b7442bf92e
2014-12-30 16:49:59 -08:00
e5cb52ec1f
YARN-2938. Fixed new findbugs warnings in hadoop-yarn-resourcemanager and hadoop-yarn-applicationhistoryservice. Contributed by Varun Saxena.
...
(cherry picked from commit 241d3b3a50
2014-12-29 10:07:54 -08:00
ca0349b87a
YARN-2992. ZKRMStateStore crashes due to session expiry. Contributed by Karthik Kambatla
...
(cherry picked from commit 1454efe5d4
2014-12-26 18:00:32 -08:00
fc3e810a19
YARN-2993. Several fixes (missing acl check, error log msg ...) and some refinement in AdminService. (Contributed by Yi Liu)
...
(cherry picked from commit 40ee4bff65
2014-12-26 07:01:44 -08:00
60530a6c4f
YARN-2946. Fixed potential deadlock in RMStateStore. Contributed by Rohith Sharmaks
...
(cherry picked from commit 4f18018b7a
2014-12-23 22:15:28 -08:00
e3a54b1b8b
YARN-2837. Support TimeLine server to recover delegation token when restarting. Contributed by Zhijie Shen
...
(cherry picked from commit 149512a837
2014-12-23 18:26:37 -08:00
8b398a66ca
YARN-2340. Fixed NPE when queue is stopped during RM restart. Contributed by Rohith Sharmaks
...
(cherry picked from commit 0d89859b51
2014-12-22 21:55:06 -08:00
411836b74c
YARN-2920. Changed CapacityScheduler to kill containers on nodes where node labels are changed. Contributed by Wangda Tan
...
(cherry picked from commit fdf042dfff
2014-12-22 16:52:15 -08:00
6f252f7f36
YARN-2939. Fix new findbugs warnings in hadoop-yarn-common. (Li Lu via junping_du)
...
(cherry picked from commit a696fbb001
2014-12-22 03:08:25 -08:00
2abec14ec6
YARN-2975. FSLeafQueue app lists are accessed without required locks. (kasha)
...
(cherry picked from commit 24ee9e3431
2014-12-20 12:20:29 -08:00
9180d11b3b
YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks
...
(cherry picked from commit 808cba3821
2014-12-19 16:56:55 -08:00
8ee40a1580
YARN-2738. [YARN-2574] Add FairReservationSystem for FairScheduler. (Anubhav Dhoot via kasha)
...
(cherry picked from commit a22ffc3188
2014-12-19 15:38:27 -08:00
173664d70f
YARN-2964. RM prematurely cancels tokens for jobs that submit jobs (oozie). Contributed by Jian He
...
(cherry picked from commit 0402bada19
2014-12-18 23:31:11 +00:00
11e4ec52a1
YARN-2972. DelegationTokenRenewer thread pool never expands. Contributed by Jason Lowe
...
(cherry picked from commit 2b4b0e8847
2014-12-17 17:41:21 -08:00
2d832ad2eb
YARN-2964. FSLeafQueue#assignContainer - document the reason for using both write and read locks. (Tsuyoshi Ozawa via kasha)
...
(cherry picked from commit f2d150ea12
2014-12-17 16:24:17 -08:00
a858d726c8
YARN-2912 Jersey Tests failing with port in use. (varun saxena via stevel)
2014-12-12 17:10:54 +00:00
10ff600b35
YARN-2243. Order of arguments for Preconditions.checkNotNull() is wrong in
...
SchedulerApplicationAttempt ctor. Contributed by Devaraj K.
(cherry picked from commit bda748ac3a
2014-12-12 12:37:13 +05:30
0f47e4e745
YARN-2930. Fixed TestRMRestart#testRMRestartRecoveringNodeLabelManager intermittent failure. Contributed by Wangda Tan
...
(cherry picked from commit 2ed90a57fd
2014-12-09 16:48:46 -08:00
1986ea8dd2
YARN-2910. FSLeafQueue can throw ConcurrentModificationException. (Wilfred Spiegelenburg via kasha)
...
(cherry picked from commit a2e07a5456
2014-12-09 14:00:52 -08:00
4cc0abe4fe
YARN-2869. CapacityScheduler should trim sub queue names when parse configuration. Contributed by Wangda Tan
...
(cherry picked from commit e69af836f3
2014-12-05 17:34:15 -08:00
b72fb6c774
YARN-2056. Disable preemption at Queue level. Contributed by Eric Payne
...
(cherry picked from commit 4b13082199
2014-12-05 21:08:14 +00:00
1d1e7682c9
YARN-2301. Improved yarn container command. Contributed by Naganarasimha G R
...
(cherry picked from commit 258623ff8b
2014-12-04 12:53:52 -08:00
58c971164c
YARN-2880. Added a test to make sure node labels will be recovered if RM restart is enabled. Contributed by Rohith Sharmaks
...
(cherry picked from commit 73fbb3c66b
2014-12-03 17:17:40 -08:00
25be97808b
YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps. (Naganarasimha G R via kasha)
...
(cherry picked from commit 799353e2c7
2014-12-03 13:45:03 -08:00
d6f3d4893d
YARN-2894. Fixed a bug regarding application view acl when RM fails over. Contributed by Rohith Sharmaks
...
(cherry picked from commit 392c3aaea8
2014-12-02 17:17:16 -08:00
5067ac098b
YARN-2136. Changed RMStateStore to ignore store opearations when fenced. Contributed by Varun Saxena
...
(cherry picked from commit 52bcefca8b
2014-12-02 11:02:00 -08:00
d208c90148
YARN-2765. Added leveldb-based implementation for RMStateStore. Contributed by Jason Lowe
...
(cherry picked from commit a7fba0bc28
2014-12-01 16:45:17 -08:00
c7bd22974a
YARN-2907. SchedulerNode#toString should print all resource detail instead of only memory. (Contributed by Rohith)
...
(cherry picked from commit c732ed760e
2014-12-01 05:41:56 -08:00
ae35b0e14d
YARN-2906. CapacitySchedulerPage shows HTML tags for a queue's Active Users. Contributed by Jason Lowe
...
(cherry picked from commit 8a7ca13b13
2014-11-25 16:18:02 -08:00
2863056530
YARN-2404. Removed ApplicationAttemptState and ApplicationState class in RMStateStore. Contributed by Tsuyoshi OZAWA
...
(cherry picked from commit 5805a81efb
2014-11-25 12:51:03 -08:00
57d62d4ded
YARN-2697. Remove useless RMAuthenticationHandler. Contributed by Haosong Huang.
...
(cherry picked from commit e37a4ff0c1
2014-11-24 15:06:44 -08:00
3e2e8eac22
YARN-2669. FairScheduler: queue names shouldn't allow periods (Wei Yan via Sandy Ryza)
2014-11-21 16:08:22 -08:00
e9db0aa35c
YARN-2604. Scheduler should consider max-allocation-* in conjunction with the largest node. (Robert Kanter via kasha)
...
(cherry picked from commit 3114d4731d
2014-11-21 10:42:47 -08:00
7ba5bb0c5b
YARN-2315. FairScheduler: Set current capacity in addition to capacity. (Zhihai Xu via kasha)
...
(cherry picked from commit a9a0cc3679
2014-11-19 20:19:54 -08:00
564deb75a8
YARN-2802. ClusterMetrics to include AM launch and register delays. (Zhihai Xu via kasha)
...
(cherry picked from commit f9f8e750edbe6db453f0a845e2ed49ede66e0e8a)
2014-11-19 19:50:35 -08:00
db31ef7e7f
YARN-2865. Fixed RM to always create a new RMContext when transtions from StandBy to Active. Contributed by Rohith Sharmaks
...
(cherry picked from commit 9cb8b75ba5
2014-11-19 19:49:44 -08:00
8df4c04c3a
YARN-2690. [YARN-2574] Make ReservationSystem and its dependent classes independent of Scheduler type. (Anubhav Dhoot via kasha)
...
(cherry picked from commit 2fce6d6141
2014-11-17 16:46:18 -08:00
242fd0e39a
YARN-2414. RM web UI: app page will crash if app is failed before any attempt has been created. Contributed by Wangda Tan
...
(cherry picked from commit 81c9d17af8
2014-11-17 21:16:54 +00:00
21054c5d14
YARN-2432. RMStateStore should process the pending events before close. Contributed by Varun Saxena
...
(cherry picked from commit 9b860661a5
2014-11-14 17:25:21 -08:00
c4c77669f0
YARN-2811. In Fair Scheduler, reservation fulfillments shouldn't ignore max share (Siqi Li via Sandy Ryza)
2014-11-14 15:20:37 -08:00
beb184ac58
YARN-2856. Fixed RMAppImpl to handle ATTEMPT_KILLED event at ACCEPTED state on app recovery. Contributed by Rohith Sharmaks
...
(cherry picked from commit d005404ef7
2014-11-13 15:34:56 -08:00
ed3e5cb164
YARN-2853. Fixed a bug in ResourceManager causing apps to hang when the user kill request races with ApplicationMaster finish. Contributed by Jian He.
...
(cherry picked from commit 3651fe1b08
2014-11-13 08:13:57 -08:00
6984d899e3
YARN-2780. Log aggregated resource allocation in rm-appsummary.log. Contributed by Eric Payne
...
(cherry picked from commit f8aefa5e9c
2014-11-12 17:02:30 +00:00
adfb830a2b
Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT.
2014-11-09 19:19:02 -08:00
d76fc94b21
YARN-2834. Fixed ResourceManager to ignore token-renewal failures on recovery consistent with the (somewhat incorrect) behaviour in the non-recovery case. Contributed by Jian He.
...
(cherry picked from commit e76faebc95
2014-11-09 18:57:48 -08:00
175d222bfc
YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles.
...
(cherry picked from commit 43cd07b408
2014-11-09 15:03:59 -08:00
1880a5a7c3
YARN-2505. Supported get/add/remove/change labels in RM REST API. Contributed by Craig Welch.
...
(cherry picked from commit 9a4e0d343e
2014-11-07 20:39:53 -08:00
b865a95f06
YARN-2826. Fixed user-groups mappings' refresh bug caused by YARN-2826. Contributed by Wangda Tan.
...
(cherry picked from commit df36edf751
2014-11-07 19:44:52 -08:00
a50345f654
YARN-2753. Fixed a bunch of bugs in the NodeLabelsManager classes. Contributed by Zhihai xu.
...
(cherry picked from commit 4cfd5bc7c1
2014-11-07 14:16:54 -08:00
d5d2905262
YARN-2824. Fixed Capacity Scheduler to not crash when some node-labels are not mapped to queues by making default capacities per label to be zero. Contributed by Wangda Tan.
...
(cherry picked from commit 2ac1be7dec
2014-11-07 10:44:25 -08:00
bf79541868
YARN-2810. TestRMProxyUsersConf fails on Windows VMs. Contributed by Varun Vasudev
...
(cherry picked from commit 1e97f2f094
2014-11-07 09:46:04 -08:00
eb01c602ed
YARN-2823. Fixed ResourceManager app-attempt state machine to inform schedulers about previous finished attempts of a running appliation to avoid expectation mismatch w.r.t transferred containers. Contributed by Jian He.
...
(cherry picked from commit a5657182a7
2014-11-07 09:30:04 -08:00
52e57a95d9
YARN-2744. Fixed CapacityScheduler to validate node-labels correctly against queues. Contributed by Wangda Tan.
...
(cherry picked from commit a3839a9fbf
2014-11-06 17:29:12 -08:00
f92ff24f5e
YARN-2579. Fixed a deadlock issue when EmbeddedElectorService and FatalEventDispatcher try to transition RM to StandBy at the same time. Contributed by Rohith Sharmaks
...
(cherry picked from commit 395275af86
2014-11-05 17:03:01 -08:00
3bc31e35a6
YARN-2805. Fixed ResourceManager to load HA configs correctly before kerberos login. Contributed by Wangda Tan.
...
(cherry picked from commit 834e931d8e
2014-11-05 15:31:41 -08:00
f3ed97de3a
YARN-2767. Added a test case to verify that http static user cannot kill or submit apps in the secure mode. Contributed by Varun Vasudev.
...
(cherry picked from commit 7a4c92a9d55fcecef066053ac30dff0fcd4ec90c)
2014-11-05 10:59:05 -08:00
3e4b280de7
YARN-2010. Handle app-recovery failures gracefully. (Jian He and Karthik Kambatla via kasha)
...
(cherry picked from commit b2cd269802
2014-11-04 17:48:49 -08:00
aa09c4c342
HADOOP-10563. Remove the dependency of jsp in trunk. Contributed by Haohui Mai.
2014-11-04 14:52:51 -08:00
277141b82d
YARN-2795. Fixed ResourceManager to not crash loading node-label data from HDFS in secure mode. Contributed by Wangda Tan.
...
(cherry picked from commit ec6cbece8e
2014-11-03 13:45:31 -08:00
caae0a0a08
YARN-2785. Fixed intermittent TestContainerResourceUsage failure. Contributed by Varun Vasudev.
...
(cherry picked from commit 27715ec63b
2014-11-02 15:22:54 -08:00
97694e78ad
YARN-2779. Fixed ResourceManager to not require delegation tokens for communicating with Timeline Service. Contributed by Zhijie Shen.
...
(cherry picked from commit d1828d9443
2014-10-30 23:17:16 -07:00
0ba8d8c19c
YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan.
...
(cherry picked from commit e0233c16eb
2014-10-30 23:02:06 -07:00
d2ba115f06
YARN-2712. TestWorkPreservingRMRestart: Augment FS tests with queue and headroom checks. (Tsuyoshi Ozawa via kasha)
...
(cherry picked from commit 179cab81e0
2014-10-30 00:39:34 -07:00
e88832dfb3
YARN-2742. FairSchedulerConfiguration should allow extra spaces between value and unit. (Wei Yan via kasha)
...
(cherry picked from commit 782971ae7a
2014-10-29 10:25:29 -07:00
720de7eb4c
YARN-2503. Added node lablels in web UI. Contributed by Wangda Tan
...
(cherry picked from commit d5e0a09721
2014-10-28 18:00:59 -07:00
0ad33e1483
YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the sake of localization and log-aggregation for long-running services. Contributed by Jian He.
...
(cherry picked from commit a16d022ca4
2014-10-27 15:50:51 -07:00
3a77838535
YARN-2726. CapacityScheduler should explicitly log when an accessible label has no capacity. Contributed by Wangda Tan
2014-10-27 10:28:45 -07:00
36d3f1fcca
YARN-2743. Fixed a bug in ResourceManager that was causing RMDelegationToken identifiers to be tampered and thus causing app submission failures in secure mode. Contributed by Jian He.
...
(cherry picked from commit 0186645505
2014-10-26 11:15:45 -07:00
96a6e02d16
YARN-1915. Fixed a race condition that client could use the ClientToAMToken to contact with AM before AM actually receives the ClientToAMTokenMasterKey. Contributed by Jason Lowe
...
(cherry picked from commit 5864dd99a4
2014-10-24 22:48:48 -07:00
3b03ea6b50
YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He.
...
(cherry picked from commit 0f3b6900be
2014-10-23 22:03:30 -07:00
4b9de2cdfe
YARN-2721. Suppress NodeExist exception thrown by ZKRMStateStore when it retries creating znode. Contributed by Jian He.
...
(cherry picked from commit 7e3b5e6f5c
2014-10-21 21:52:01 -07:00
77dc9afe86
YARN-2715. Fixed ResourceManager to respect common configurations for proxy users/groups beyond just the YARN level config. Contributed by Zhijie Shen.
...
(cherry picked from commit c0e034336c
2014-10-21 20:10:45 -07:00
9915d52185
YARN-2504. Enhanced RM Admin CLI to support management of node-labels. Contribyted by Wangda Tan.
...
(cherry picked from commit 8256766498
2014-10-18 12:08:28 -07:00
ea841eb25d
YARN-2676. Enhanced Timeline auth-filter to support proxy users. Contributed by Zhijie Shen.
...
(cherry picked from commit 39063cd36f
2014-10-17 22:03:43 -07:00
1c52b6551b
YARN-2705. Fixed bugs in ResourceManager node-label manager that were causing test-failures: added a dummy in-memory labels-manager. Contributed by Wangda Tan.
...
(cherry picked from commit e9c66e8fd2
2014-10-17 18:27:04 -07:00
78a57087bf
YARN-1879. Marked Idempotent/AtMostOnce annotations to ApplicationMasterProtocol for RM fail over. Contributed by Tsuyoshi OZAWA
...
(cherry picked from commit c3de2412eb
2014-10-17 16:36:15 -07:00
79428e021f
YARN-2588. Standby RM fails to transitionToActive if previous transitionToActive failed with ZK exception. Contributed by Rohith Sharmaks
...
(cherry picked from commit a6aa6e42ca
2014-10-17 10:55:04 -07:00
99ce4277a8
YARN-2699. Fixed a bug in CommonNodeLabelsManager that caused tests to fail when using ephemeral ports on NodeIDs. Contributed by Wangda Tan.
...
(cherry picked from commit abae63caf9
2014-10-17 08:59:22 -07:00
76cf2250de
YARN-2685. Fixed a bug in CommonNodeLabelsManager that caused wrong resource tracking per label when a host runs multiple node-managers. Contributed by Wangda Tan.
...
(cherry picked from commit b3056c266a
2014-10-15 18:48:36 -07:00
e8e3a36213
YARN-2496. Enhanced Capacity Scheduler to have basic support for allocating resources based on node-labels. Contributed by Wangda Tan.
...
YARN-2500. Ehnaced ResourceManager to support schedulers allocating resources based on node-labels. Contributed by Wangda Tan.
(cherry picked from commit f2ea555ac6
2014-10-15 18:34:19 -07:00
Junping Du
Jian He
Akira Ajisaka
Tsuyoshi Ozawa
Wangda Tan
Karthik Kambatla
Xuan
Vinod Kumar Vavilapalli
Robert Kanter
Sandy Ryza
Zhijie Shen
cnauroth
Devaraj K
Jason Lowe
Konstantin V Shvachko
Arun C. Murthy
Steve Loughran
Haohui Mai