Commit Graph

2058 Commits

Author SHA1 Message Date
Devaraj K 2a01c01af7 YARN-3780. Should use equals when compare Resource in
RMNodeImpl#ReconnectNodeTransition. Contributed by zhihai xu.

(cherry picked from commit c7ee6c151c)
2015-06-08 11:56:58 +05:30
Karthik Kambatla 322e7d0d5e YARN-3655. FairScheduler: potential livelock due to maxAMShare limitation and container reservation. (Zhihai Xu via kasha)
(cherry picked from commit bd69ea408f)
2015-06-07 11:38:14 -07:00
Xuan 4d385b48f0 YARN-1462. AHS API and other AHS changes to handle tags for completed MR jobs. Contributed by Xuan Gong
(cherry picked from commit 3e000a919f)
2015-06-05 12:49:51 -07:00
Karthik Kambatla 429754de09 YARN-3259. FairScheduler: Trigger fairShare updates on node events. (Anubhav Dhoot via kasha)
(cherry picked from commit 75885852cc)
2015-06-05 09:42:12 -07:00
Zhijie Shen 8f04ac9c34 YARN-3766. Fixed the apps table column error of generic history web UI. Contributed by Xuan Gong.
(cherry picked from commit 18dd01d6bf)
2015-06-04 14:48:05 -07:00
Wangda Tan 344b750915 Add missing test file of YARN-3733
(cherry picked from commit 405bbcf68c)
2015-06-04 13:20:49 -07:00
Jian He 2e99210e4e YARN-2392. Add more diags about app retry limits on AM failures. Contributed by Steve Loughran
(cherry picked from commit 1970ca7cbc)
2015-06-04 11:14:36 -07:00
Jian He 6325e4b7dd YARN-3764. CapacityScheduler should forbid moving LeafQueue from one parent to another. Contributed by Wangda Tan
(cherry picked from commit 6ad4e59cfc)
2015-06-04 10:53:24 -07:00
Wangda Tan 78d626fa89 YARN-3733. Fix DominantRC#compare() does not work as expected if cluster resource is empty. (Rohith Sharmaks via wangda)
(cherry picked from commit ebd797c48f)
2015-06-04 10:23:33 -07:00
Junping Du d8c7ee1b53 YARN-41. The RM should handle the graceful shutdown of the NM. Contributed by Devaraj K.
(cherry picked from commit d7e7f6aa03)
2015-06-04 05:00:32 -07:00
Xuan 3f0573b059 YARN-3749. We should make a copy of configuration when init
MiniYARNCluster with multiple RMs. Contributed by Chun Chen

(cherry picked from commit 5766a04428)
2015-06-03 17:21:48 -07:00
Zhijie Shen 2fcd3fe6ba Revert "YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong."
This reverts commit 4e2ed34d20.
2015-06-03 14:16:51 -07:00
Zhijie Shen 5e976fffe3 Revert "YARN-1462. Correct fix version from branch-2.7.1 to branch-2.8 in"
This reverts commit 26ad3293ec.
2015-06-03 14:16:51 -07:00
Karthik Kambatla 62d51b889a YARN-3762. FairScheduler: CME on FSParentQueue#getQueueUserAclInfo. (kasha)
(cherry picked from commit edb9cd0f7a)
2015-06-03 13:50:02 -07:00
Jason Lowe 752caa95a4 YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. Contributed by Rohith Sharmaks
(cherry picked from commit e13b671aa5)
2015-06-03 19:45:18 +00:00
Zhijie Shen aaa567514a YARN-3751. Fixed AppInfo to check if used resources are null. Contributed by Sunil G.
(cherry picked from commit dbc4f64937)
2015-06-03 11:52:39 -07:00
Xuan 26ad3293ec YARN-1462. Correct fix version from branch-2.7.1 to branch-2.8 in
CHANGES.txt

(cherry picked from commit 0b5cfacde6)
2015-06-01 11:40:24 -07:00
Wangda Tan a3734f67d3 YARN-3725. App submission via REST API is broken in secure mode due to Timeline DT service address is empty. (Zhijie Shen via wangda)
(cherry picked from commit 5cc3fced95)
2015-05-31 16:33:50 -07:00
Xuan 4563411e04 YARN-2900. Application (Attempt and Container) Not Found in AHS results
in Internal Server Error (500). Contributed by Zhijie Shen and Mit Desai

(cherry picked from commit 9686261ecb)
2015-05-31 15:38:39 -07:00
Xuan 1ebaff0b86 Revert "YARN-2900. Application (Attempt and Container) Not Found in AHS results"
This reverts commit 4fee8b3202.
2015-05-31 00:24:08 -07:00
Xuan 4fee8b3202 YARN-2900. Application (Attempt and Container) Not Found in AHS results
in Internal Server Error (500). Contributed by Zhijie Shen and Mit Desai

(cherry picked from commit 06f8e9caba)
2015-05-30 21:58:04 -07:00
Zhijie Shen 4e2ed34d20 YARN-1462. Made RM write application tags to timeline server and exposed them to users via generic history web UI and REST API. Contributed by Xuan Gong.
(cherry picked from commit a9f986bc5a41ab74171da1076d10e83c5adc284f)
2015-05-30 21:06:32 -07:00
Karthik Kambatla 6c80e599d1 YARN-3467. Expose allocatedMB, allocatedVCores, and runningContainers metrics on running Applications in RM Web UI. (Anubhav Dhoot via kasha)
(cherry picked from commit a8acdd65b3)
2015-05-30 13:36:49 -07:00
Zhijie Shen 68cddb894a YARN-3740. Fixed the typo in the configuration name: APPLICATION_HISTORY_PREFIX_MAX_APPS. Contributed by Xuan Gong.
(cherry picked from commit eb6bf91eea)
2015-05-29 18:18:47 -07:00
Robert Kanter bde4031a02 YARN-3713. Remove duplicate function call storeContainerDiagnostics in ContainerDiagnosticsUpdateTransition (zxu via rkanter)
(cherry picked from commit 6aec13cb33)
2015-05-29 15:35:08 -07:00
Karthik Kambatla b9c5c5de68 YARN-3547. FairScheduler: Apps that have no resource demand should not participate scheduling. (Xianyin Xin via kasha)
(cherry picked from commit 3ae2a62501)
2015-05-29 15:17:43 -07:00
Andrew Wang bdaa0fed13 HADOOP-11885. hadoop-dist dist-layout-stitching.sh does not work with dash. (wang)
(cherry picked from commit 7673d4f205)

 Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
	hadoop-mapreduce-project/pom.xml
2015-05-29 14:55:52 -07:00
Wangda Tan 2fb794ecaf YARN-3716. Node-label-expression should be included by ResourceRequestPBImpl.toString. (Xianyin Xin via wangda)
(cherry picked from commit 788bfa0359)
2015-05-28 16:35:17 -07:00
Vinod Kumar Vavilapalli 148412bb30 Fixed more FilesSystemRMStateStore issues. Contributed by Vinod Kumar Vavilapalli.
(cherry picked from commit 9acd24fec4)
2015-05-28 15:27:01 -07:00
Xuan 550b55146d YARN-3723. Need to clearly document primaryFilter and otherInfo value
type. Contributed by Zhijie Shen

(cherry picked from commit 3077c299da)
2015-05-28 10:20:33 -07:00
Devaraj K 7116141d7d YARN-3722. Merge multiple TestWebAppUtils into
o.a.h.yarn.webapp.util.TestWebAppUtils. Contributed by Masatake Iwasaki.

(cherry picked from commit 7e509f5843)
2015-05-28 15:36:13 +05:30
Zhijie Shen 839f81a632 YARN-3700. Made generic history service load a number of latest applications according to the parameter or the configuration. Contributed by Xuan Gong.
(cherry picked from commit 54504133f4)
2015-05-27 16:53:47 -07:00
Wangda Tan 4790495538 YARN-3581. Deprecate -directlyAccessNodeLabelStore in RMAdminCLI. (Naganarasimha G R via wangda)
(cherry picked from commit cab7674e54)
2015-05-27 15:37:14 -07:00
cnauroth 6ed8989a6f YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch.
(cherry picked from commit 7bc33b63c78c6ee9a04c0c1511ef16e63f7f3074)
2015-05-27 14:21:15 -07:00
Wangda Tan 8857cba72e YARN-3647. RMWebServices api's should use updated api from CommonNodeLabelsManager to get NodeLabel object. (Sunil G via wangda)
(cherry picked from commit ec0a852a37)
2015-05-27 13:15:55 -07:00
Wangda Tan 0728a86b3a YARN-3686. CapacityScheduler should trim default_node_label_expression. (Sunil G via wangda)
(cherry picked from commit cdbd66be11)
2015-05-26 16:00:17 -07:00
Jian He 5e7be094ec YARN-3632. Ordering policy should be allowed to reorder an application when demand changes. Contributed by Craig Welch
(cherry picked from commit 10732d515f)
2015-05-26 12:02:36 -07:00
Vinod Kumar Vavilapalli a67cb4826b YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev.
(cherry picked from commit 500a1d9c76)
2015-05-26 11:39:40 -07:00
Tsuyoshi Ozawa cf5c1adf4f YARN-2336. Fair scheduler's REST API returns a missing '[' bracket JSON for deep queue tree. Contributed by Kenji Kikushima and Akira Ajisaka.
(cherry picked from commit 9a3d617b63)
2015-05-26 19:08:01 +09:00
Xuan 84245ff3b2 YARN-2238. Filtering on UI sticks even if I move away from the page.
Contributed by Jian He

(cherry picked from commit 39077dba2e)
2015-05-25 22:41:02 -07:00
Jian He 03af057a47 YARN-3707. RM Web UI queue filter doesn't work. Contributed by Wangda Tan
(cherry picked from commit 446d51591e)
2015-05-22 15:59:00 -07:00
Xuan 33be070a5e YARN-3701. Isolating the error of generating a single app report when
getting all apps from generic history service. Contributed by Zhijie
Shen

(cherry picked from commit 455b3acf0e)
2015-05-22 14:35:51 -07:00
Vinayakumar B 19895a41ad HADOOP-11743. maven doesn't clean all the site files (Contributed by ramtin)
(cherry picked from commit 55ed6558ba)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
2015-05-22 16:46:50 +05:30
Junping Du 5fb61d71b1 YARN-3594. WintuilsProcessStubExecutor.startStreamReader leaks streams. Contributed by Lars Francke.
(cherry picked from commit 132d909d4a)
2015-05-22 04:25:18 -07:00
Vinod Kumar Vavilapalli 673bd9708f YARN-3684. Changed ContainerExecutor's primary lifecycle methods to use a more extensible mechanism of context objects. Contributed by Sidharta Seethana.
(cherry picked from commit 53fafcf061)
2015-05-21 15:51:19 -07:00
Karthik Kambatla e8ac88d4fe YARN-3675. FairScheduler: RM quits when node removal races with continuous-scheduling on the same node. (Anubhav Dhoot via kasha)
(cherry picked from commit a8b50e46737c11936ba72c427da69b2365a07aac)
2015-05-21 13:39:40 -07:00
Devaraj K 157ecb2241 YARN-3646. Applications are getting stuck some times in case of retry
policy forever. Contributed by Raju Bairishetti.

(cherry picked from commit 0305316d69)
2015-05-21 20:16:09 +05:30
Akira Ajisaka 6d7e7ef1c4 YARN-3694. Fix dead link for TimelineServer REST API. Contributed by Jagadesh Kiran N.
(cherry picked from commit a5def58087)
2015-05-21 23:15:17 +09:00
Jian He 7c3895c8f6 YARN-3654. ContainerLogsPage web UI should not have meta-refresh. Contributed by Xuan Gong
(cherry picked from commit 6329bd00fa)
2015-05-20 17:21:19 -07:00
Jian He 521378ffa3 YARN-3609. Load node labels from storage inside RM serviceStart. Contributed by Wangda Tan
(cherry picked from commit 8966d42179)
2015-05-20 16:30:44 -07:00
Xuan a665b22cfa YARN-3681. yarn cmd says "could not find main class 'queue'" in windows. Contributed by Craig Welch and Varun Saxena 2015-05-20 14:42:59 -07:00
Wangda Tan d3193fd1d7 Move YARN-2918 from 2.8.0 to 2.7.1
(cherry picked from commit 03f897fd1a)
2015-05-20 13:50:11 -07:00
Wangda Tan b5ba6979b7 YARN-3583. Support of NodeLabel object instead of plain String in YarnClient side. (Sunil G via wangda)
(cherry picked from commit 563eb1ad2a)
(cherry picked from commit b0d22b0c606fad6b4ab5443c0aed07c829b46726)
2015-05-19 17:37:24 -07:00
Tsuyoshi Ozawa d022c5aba8 YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. Contributed by Vinod Kumar Vavilapalli.
(cherry picked from commit 7401e5b5e8)
2015-05-20 09:01:42 +09:00
Wangda Tan 7b97f049ee YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String. (Naganarasimha G R via wangda)
(cherry picked from commit b37da52a1c)
2015-05-19 16:35:23 -07:00
Jian He adb90c7f52 YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted. Contributed by Varun Vasudev
(cherry picked from commit 7438966586)
2015-05-19 14:21:48 -07:00
Colin Patrick Mccabe 9534e533dc HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7 ThreadLocalRandom (Sean Busbey via Colin P. McCabe)
(cherry picked from commit 470c87dbc6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
2015-05-19 11:15:04 -07:00
Ravi Prakash 084e453629 YARN-3302. TestDockerContainerExecutor should run automatically if it can detect docker in the usual place (Ravindra Kumar Naik via raviprak)
(cherry picked from commit c97f32e7b9)
2015-05-19 10:30:24 -07:00
Xuan d39039d54d YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. Contributed by Weiwei Yang
(cherry picked from commit 5009ad4a7f)
2015-05-19 09:57:13 -07:00
Xuan 3ceb2ffe54 YARN-3541. Add version info on timeline service / generic history web UI and REST API. Contributed by Zhijie Shen
(cherry picked from commit 76afd28862)
2015-05-18 13:19:01 -07:00
Xuan 802676e1be YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang
(cherry picked from commit b0ad644083)
2015-05-15 22:41:13 -07:00
Jason Lowe ce64720516 YARN-2421. RM still allocates containers to an app in the FINISHING state. Contributed by Chang Li
(cherry picked from commit f7e051c431)
2015-05-15 22:10:16 +00:00
Junping Du cc671349bc YARN-3505 addendum: fix an issue in previous patch.
(cherry picked from commit 03a293aed6)
2015-05-15 06:40:28 -07:00
Vinod Kumar Vavilapalli ad3196e016 Fixing HDFS state-store. Contributed by Arun Suresh.
(cherry picked from commit 9a2a9553ee)
2015-05-14 16:57:03 -07:00
Ravi Prakash 4caadaa8bb YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak)
(cherry picked from commit 53fe4eff09)
2015-05-14 15:56:36 -07:00
Junping Du bc13c7d84b YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in RMApps. Contributed by Xuan Gong.
(cherry picked from commit 15ccd967ee)
2015-05-14 10:59:48 -07:00
Wangda Tan 3a11baa15f YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda)
(cherry picked from commit 0e85044e26)
2015-05-13 17:01:23 -07:00
Jason Lowe a81ad81461 YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du
(cherry picked from commit 711d77cc54)
2015-05-13 21:08:45 +00:00
Jason Lowe aefc2bae9e Update fix version for YARN-3457 and YARN-3537.
(cherry picked from commit f7de6198da)
2015-05-13 20:54:21 +00:00
Wangda Tan d49262a172 YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda)
(cherry picked from commit 7f19e7a254)
2015-05-13 13:43:57 -07:00
Wangda Tan 8555a5146d YARN-3579. CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings. (Sunil G via wangda)
(cherry picked from commit d4f53fc963)
2015-05-13 13:33:38 -07:00
Xuan 487d9b0f3f YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch
(cherry picked from commit 0f95921447)
2015-05-13 13:12:04 -07:00
Wangda Tan 3df8880bbe YARN-2921. Fix MockRM/MockAM#waitForState sleep too long. (Tsuyoshi Ozawa via wangda)
(cherry picked from commit 341a476812)
2015-05-13 13:07:05 -07:00
cnauroth ddc4afeff2 HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. 2015-05-13 12:25:49 -07:00
Zhijie Shen 6043224458 YARN-3539. Updated timeline server documentation and marked REST APIs evolving. Contributed by Steve Loughran.
(cherry picked from commit fcd0702c10)
2015-05-12 21:13:47 -07:00
Karthik Kambatla 9d6c63f8be YARN-3613. TestContainerManagerSecurity should init and start Yarn cluster in setup instead of individual methods. (nijel via kasha)
(cherry picked from commit fe0df59627)
2015-05-12 10:45:51 -07:00
Devaraj K 5fceebd696 YARN-3629. NodeID is always printed as "null" in node manager
initialization log. Contributed by nijel.

(cherry picked from commit 5c2f05cd9b)
2015-05-12 22:22:40 +05:30
Devaraj K f887243f88 YARN-3513. Remove unused variables in ContainersMonitorImpl and add debug
log for overall resource usage by all containers.  Contributed by
Naganarasimha G R.

(cherry picked from commit 8badd82ce2)
2015-05-12 16:56:26 +05:30
Wangda Tan 12584ac10e Move YARN-3493 from 2.8 to 2.7.1 in CHANGES.txt 2015-05-11 18:08:25 -07:00
Wangda Tan d06d2a96f5 YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda)
(cherry picked from commit d6f6741296)
2015-05-11 17:35:01 -07:00
Wangda Tan 2c80a3fe19 Moved YARN-3434. (Interaction between reservations and userlimit can result in significant ULF violation.) From 2.8.0 to 2.7.1
(cherry picked from commit 1952f88889)
2015-05-11 15:22:25 -07:00
Junping Du a1a7deebf8 YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. projects. Contributed by Gabor Liptak.
(cherry picked from commit 7e543c27fa)
2015-05-11 05:28:25 -07:00
Karthik Kambatla 3cf4f24a9f YARN-3395. FairScheduler: Trim whitespaces when using username for queuename. (Zhihai Xu via kasha)
(cherry picked from commit a60f78e98e)
2015-05-09 15:41:49 -07:00
Karthik Kambatla e8dd0bc4fd YARN-1287. Consolidate MockClocks. (Sebastian Wong and Anubhav Dhoot via kasha)
(cherry picked from commit 70fb37cd79)
2015-05-09 14:35:13 -07:00
Xuan acb342b77c YARN-1912. ResourceLocalizer started without any jvm memory control.
Contributed by Masatake Iwasaki

(cherry picked from commit 6471d18bc7)
2015-05-08 20:02:23 -07:00
Xuan fc980247a6 YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IOException from cleanup. Contributed by zhihai xu
(cherry picked from commit 333f9a896d)
2015-05-08 17:54:46 -07:00
Zhijie Shen ef096599c3 YARN-2206. Updated document for applications REST API response examples. Contributed by Kenji Kikushima and Brahma Reddy Battula.
(cherry picked from commit 08f0ae403c)
2015-05-08 16:41:29 -07:00
Karthik Kambatla a7d8f21630 YARN-3271. FairScheduler: Move tests related to max-runnable-apps from TestFairScheduler to TestAppRunnability. (nijel via kasha)
(cherry picked from commit 2fb44c8aaf)
2015-05-08 16:40:36 -07:00
Karthik Kambatla 21a07769d4 YARN-1050. Document the Fair Scheduler REST API. (Kenji Kikushima and Roman Shaposhnik via kasha)
(cherry picked from commit 96473bdc2b)
2015-05-08 16:31:30 -07:00
Robert Kanter 453488cff2 YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter)
(cherry picked from commit 5658998845)
2015-05-08 16:23:20 -07:00
Jason Lowe 743ca1a056 YARN-3476. Nodemanager can fail to delete local logs if log aggregation fails. Contributed by Rohith
(cherry picked from commit 25e2b02122)
2015-05-08 22:46:34 +00:00
Jian He 5a1c2fed34 YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch. Contributed Zhihai Xu
(cherry picked from commit d0b73287c2)
2015-05-08 15:13:43 -07:00
Xuan 8d07116266 YARN-2331. Distinguish shutdown during supervision vs. shutdown for
rolling upgrade. Contributed by Jason Lowe

(cherry picked from commit 088156de43)
2015-05-08 15:11:55 -07:00
Vinod Kumar Vavilapalli 24c8d3e54b YARN-3018. Unified the default value for the configuration property yarn.scheduler.capacity.node-locality-delay in code and default xml file. Contributed by Nijel SF.
(cherry picked from commit f7a74d2491)
2015-05-08 14:45:55 -07:00
Wangda Tan f8807839c6 YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page. (Naganarasimha G R via wangda)
(cherry picked from commit f55d21d894)
2015-05-08 12:05:25 -07:00
Thomas Graves 638feaaa35 YARN-20. More information for yarn.resourcemanager.webapp.address in yarn-default.xml (Bartosz Ługowski vai tgraves)
(cherry picked from commit f0f5e3c075)
2015-05-08 17:22:33 +00:00
Thomas Graves 547b069887 YARN-3600. AM container link is broken (Naganarasimha G R via tgraves
(cherry picked from commit 5d708a4725)
2015-05-08 16:37:20 +00:00
Robert (Bobby) Evans 28e0593b96 YARN-644: Basic null check is not performed on passed in arguments before using them in ContainerManagerImpl.startContainer
(cherry picked from commit bcf2890502)
2015-05-08 11:11:53 -05:00
Devaraj K bf8be56292 YARN-2784. Make POM project names consistent. Contributed by Rohith. 2015-05-08 19:26:11 +05:30
Jason Lowe bcb863b156 YARN-3554. Default value for maximum nodemanager connect wait time is too high. Contributed by Naganarasimha G R
(cherry picked from commit 9757864fd6)
2015-05-08 13:47:48 +00:00