Commit Graph

2006 Commits

Author SHA1 Message Date
Wangda Tan b5ba6979b7 YARN-3583. Support of NodeLabel object instead of plain String in YarnClient side. (Sunil G via wangda)
(cherry picked from commit 563eb1ad2a)
(cherry picked from commit b0d22b0c606fad6b4ab5443c0aed07c829b46726)
2015-05-19 17:37:24 -07:00
Tsuyoshi Ozawa d022c5aba8 YARN-3677. Fix findbugs warnings in yarn-server-resourcemanager. Contributed by Vinod Kumar Vavilapalli.
(cherry picked from commit 7401e5b5e8)
2015-05-20 09:01:42 +09:00
Wangda Tan 7b97f049ee YARN-3565. NodeHeartbeatRequest/RegisterNodeManagerRequest should use NodeLabel object instead of String. (Naganarasimha G R via wangda)
(cherry picked from commit b37da52a1c)
2015-05-19 16:35:23 -07:00
Jian He adb90c7f52 YARN-2821. Fixed a problem that DistributedShell AM may hang if restarted. Contributed by Varun Vasudev
(cherry picked from commit 7438966586)
2015-05-19 14:21:48 -07:00
Colin Patrick Mccabe 9534e533dc HADOOP-11970. Replace uses of ThreadLocal<Random> with JDK7 ThreadLocalRandom (Sean Busbey via Colin P. McCabe)
(cherry picked from commit 470c87dbc6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java
        hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/JspHelper.java
2015-05-19 11:15:04 -07:00
Ravi Prakash 084e453629 YARN-3302. TestDockerContainerExecutor should run automatically if it can detect docker in the usual place (Ravindra Kumar Naik via raviprak)
(cherry picked from commit c97f32e7b9)
2015-05-19 10:30:24 -07:00
Xuan d39039d54d YARN-3601. Fix UT TestRMFailover.testRMWebAppRedirect. Contributed by Weiwei Yang
(cherry picked from commit 5009ad4a7f)
2015-05-19 09:57:13 -07:00
Xuan 3ceb2ffe54 YARN-3541. Add version info on timeline service / generic history web UI and REST API. Contributed by Zhijie Shen
(cherry picked from commit 76afd28862)
2015-05-18 13:19:01 -07:00
Xuan 802676e1be YARN-3526. ApplicationMaster tracking URL is incorrectly redirected on a QJM cluster. Contributed by Weiwei Yang
(cherry picked from commit b0ad644083)
2015-05-15 22:41:13 -07:00
Jason Lowe ce64720516 YARN-2421. RM still allocates containers to an app in the FINISHING state. Contributed by Chang Li
(cherry picked from commit f7e051c431)
2015-05-15 22:10:16 +00:00
Junping Du cc671349bc YARN-3505 addendum: fix an issue in previous patch.
(cherry picked from commit 03a293aed6)
2015-05-15 06:40:28 -07:00
Vinod Kumar Vavilapalli ad3196e016 Fixing HDFS state-store. Contributed by Arun Suresh.
(cherry picked from commit 9a2a9553ee)
2015-05-14 16:57:03 -07:00
Ravi Prakash 4caadaa8bb YARN-1519. Check in container-executor if sysconf is implemented before using it (Radim Kolar and Eric Payne via raviprak)
(cherry picked from commit 53fe4eff09)
2015-05-14 15:56:36 -07:00
Junping Du bc13c7d84b YARN-3505. Node's Log Aggregation Report with SUCCEED should not cached in RMApps. Contributed by Xuan Gong.
(cherry picked from commit 15ccd967ee)
2015-05-14 10:59:48 -07:00
Wangda Tan 3a11baa15f YARN-3362. Add node label usage in RM CapacityScheduler web UI. (Naganarasimha G R via wangda)
(cherry picked from commit 0e85044e26)
2015-05-13 17:01:23 -07:00
Jason Lowe a81ad81461 YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du
(cherry picked from commit 711d77cc54)
2015-05-13 21:08:45 +00:00
Jason Lowe aefc2bae9e Update fix version for YARN-3457 and YARN-3537.
(cherry picked from commit f7de6198da)
2015-05-13 20:54:21 +00:00
Wangda Tan d49262a172 YARN-3521. Support return structured NodeLabel objects in REST API (Sunil G via wangda)
(cherry picked from commit 7f19e7a254)
2015-05-13 13:43:57 -07:00
Wangda Tan 8555a5146d YARN-3579. CommonNodeLabelsManager should support NodeLabel instead of string label name when getting node-to-label/label-to-label mappings. (Sunil G via wangda)
(cherry picked from commit d4f53fc963)
2015-05-13 13:33:38 -07:00
Xuan 487d9b0f3f YARN-3626. On Windows localized resources are not moved to the front of the classpath when they should be. Contributed by Craig Welch
(cherry picked from commit 0f95921447)
2015-05-13 13:12:04 -07:00
Wangda Tan 3df8880bbe YARN-2921. Fix MockRM/MockAM#waitForState sleep too long. (Tsuyoshi Ozawa via wangda)
(cherry picked from commit 341a476812)
2015-05-13 13:07:05 -07:00
cnauroth ddc4afeff2 HADOOP-11966. Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh. Contributed by Chris Nauroth. 2015-05-13 12:25:49 -07:00
Zhijie Shen 6043224458 YARN-3539. Updated timeline server documentation and marked REST APIs evolving. Contributed by Steve Loughran.
(cherry picked from commit fcd0702c10)
2015-05-12 21:13:47 -07:00
Karthik Kambatla 9d6c63f8be YARN-3613. TestContainerManagerSecurity should init and start Yarn cluster in setup instead of individual methods. (nijel via kasha)
(cherry picked from commit fe0df59627)
2015-05-12 10:45:51 -07:00
Devaraj K 5fceebd696 YARN-3629. NodeID is always printed as "null" in node manager
initialization log. Contributed by nijel.

(cherry picked from commit 5c2f05cd9b)
2015-05-12 22:22:40 +05:30
Devaraj K f887243f88 YARN-3513. Remove unused variables in ContainersMonitorImpl and add debug
log for overall resource usage by all containers.  Contributed by
Naganarasimha G R.

(cherry picked from commit 8badd82ce2)
2015-05-12 16:56:26 +05:30
Wangda Tan 12584ac10e Move YARN-3493 from 2.8 to 2.7.1 in CHANGES.txt 2015-05-11 18:08:25 -07:00
Wangda Tan d06d2a96f5 YARN-3489. RMServerUtils.validateResourceRequests should only obtain queue info once. (Varun Saxena via wangda)
(cherry picked from commit d6f6741296)
2015-05-11 17:35:01 -07:00
Wangda Tan 2c80a3fe19 Moved YARN-3434. (Interaction between reservations and userlimit can result in significant ULF violation.) From 2.8.0 to 2.7.1
(cherry picked from commit 1952f88889)
2015-05-11 15:22:25 -07:00
Junping Du a1a7deebf8 YARN-3587. Fix the javadoc of DelegationTokenSecretManager in yarn, etc. projects. Contributed by Gabor Liptak.
(cherry picked from commit 7e543c27fa)
2015-05-11 05:28:25 -07:00
Karthik Kambatla 3cf4f24a9f YARN-3395. FairScheduler: Trim whitespaces when using username for queuename. (Zhihai Xu via kasha)
(cherry picked from commit a60f78e98e)
2015-05-09 15:41:49 -07:00
Karthik Kambatla e8dd0bc4fd YARN-1287. Consolidate MockClocks. (Sebastian Wong and Anubhav Dhoot via kasha)
(cherry picked from commit 70fb37cd79)
2015-05-09 14:35:13 -07:00
Xuan acb342b77c YARN-1912. ResourceLocalizer started without any jvm memory control.
Contributed by Masatake Iwasaki

(cherry picked from commit 6471d18bc7)
2015-05-08 20:02:23 -07:00
Xuan fc980247a6 YARN-3602. TestResourceLocalizationService.testPublicResourceInitializesLocalDir fails Intermittently due to IOException from cleanup. Contributed by zhihai xu
(cherry picked from commit 333f9a896d)
2015-05-08 17:54:46 -07:00
Zhijie Shen ef096599c3 YARN-2206. Updated document for applications REST API response examples. Contributed by Kenji Kikushima and Brahma Reddy Battula.
(cherry picked from commit 08f0ae403c)
2015-05-08 16:41:29 -07:00
Karthik Kambatla a7d8f21630 YARN-3271. FairScheduler: Move tests related to max-runnable-apps from TestFairScheduler to TestAppRunnability. (nijel via kasha)
(cherry picked from commit 2fb44c8aaf)
2015-05-08 16:40:36 -07:00
Karthik Kambatla 21a07769d4 YARN-1050. Document the Fair Scheduler REST API. (Kenji Kikushima and Roman Shaposhnik via kasha)
(cherry picked from commit 96473bdc2b)
2015-05-08 16:31:30 -07:00
Robert Kanter 453488cff2 YARN-3473. Fix RM Web UI configuration for some properties (rchiang via rkanter)
(cherry picked from commit 5658998845)
2015-05-08 16:23:20 -07:00
Jason Lowe 743ca1a056 YARN-3476. Nodemanager can fail to delete local logs if log aggregation fails. Contributed by Rohith
(cherry picked from commit 25e2b02122)
2015-05-08 22:46:34 +00:00
Jian He 5a1c2fed34 YARN-3604. Fixed ZKRMStateStore#removeApplication to also disable watch. Contributed Zhihai Xu
(cherry picked from commit d0b73287c2)
2015-05-08 15:13:43 -07:00
Xuan 8d07116266 YARN-2331. Distinguish shutdown during supervision vs. shutdown for
rolling upgrade. Contributed by Jason Lowe

(cherry picked from commit 088156de43)
2015-05-08 15:11:55 -07:00
Vinod Kumar Vavilapalli 24c8d3e54b YARN-3018. Unified the default value for the configuration property yarn.scheduler.capacity.node-locality-delay in code and default xml file. Contributed by Nijel SF.
(cherry picked from commit f7a74d2491)
2015-05-08 14:45:55 -07:00
Wangda Tan f8807839c6 YARN-3593. Add label-type and Improve "DEFAULT_PARTITION" in Node Labels Page. (Naganarasimha G R via wangda)
(cherry picked from commit f55d21d894)
2015-05-08 12:05:25 -07:00
Thomas Graves 638feaaa35 YARN-20. More information for yarn.resourcemanager.webapp.address in yarn-default.xml (Bartosz Ługowski vai tgraves)
(cherry picked from commit f0f5e3c075)
2015-05-08 17:22:33 +00:00
Thomas Graves 547b069887 YARN-3600. AM container link is broken (Naganarasimha G R via tgraves
(cherry picked from commit 5d708a4725)
2015-05-08 16:37:20 +00:00
Robert (Bobby) Evans 28e0593b96 YARN-644: Basic null check is not performed on passed in arguments before using them in ContainerManagerImpl.startContainer
(cherry picked from commit bcf2890502)
2015-05-08 11:11:53 -05:00
Devaraj K bf8be56292 YARN-2784. Make POM project names consistent. Contributed by Rohith. 2015-05-08 19:26:11 +05:30
Jason Lowe bcb863b156 YARN-3554. Default value for maximum nodemanager connect wait time is too high. Contributed by Naganarasimha G R
(cherry picked from commit 9757864fd6)
2015-05-08 13:47:48 +00:00
Tsuyoshi Ozawa 5ddc3adb0d YARN-3169. Drop YARN's overview document. Contributed by Brahma Reddy Battula.
(cherry picked from commit b419c1b2ec)
2015-05-08 19:58:47 +09:00
Tsuyoshi Ozawa b1f3590b22 YARN-3589. RM and AH web UI display DOCTYPE wrongly. Contbituted by Rohith.
(cherry picked from commit f26700f287)
2015-05-08 17:13:56 +09:00
Tsuyoshi Ozawa 1ae8bf5b21 MAPREDUCE-6284. Add Task Attempt State API to MapReduce Application Master REST API. Contributed by Ryu Kobayashi.
(cherry picked from commit d18f10ad1b)
2015-05-08 16:00:41 +09:00
Devaraj K 0d286d1041 YARN-3358. Audit log not present while refreshing Service ACLs.
Contributed by Varun Saxena.

(cherry picked from commit ef3d66d462)
2015-05-08 12:15:45 +05:30
Devaraj K dce2381dc4 YARN-3592. Fix typos in RMNodeLabelsManager. Contributed by Sunil G.
(cherry picked from commit a28cd02ba0)
2015-05-08 10:55:53 +05:30
Akira Ajisaka c7f3dd1eb3 YARN-3572. Correct typos in WritingYarnApplications.md. Contributed by Gabor Liptak.
(cherry picked from commit a521b50955)
2015-05-08 13:25:07 +09:00
Akira Ajisaka 170f1b0afd YARN-1832. Fix wrong MockLocalizerStatus#equals implementation. Contributed by Hong Zhiguo.
(cherry picked from commit b167fe7605)
2015-05-08 11:15:28 +09:00
Jian He d817fbb34d YARN-2918. RM should not fail on startup if queue's configured labels do not exist in cluster-node-labels. Contributed by Wangda Tan
(cherry picked from commit f489a4ec96)
2015-05-07 17:36:24 -07:00
Jian He c35114a1e6 YARN-3584. Fixed attempt diagnostics format shown on the UI. Contributed by nijel
(cherry picked from commit b88700dcd0)
2015-05-07 13:56:25 -07:00
Zhijie Shen bb035ff087 YARN-3448. Added a rolling time-to-live LevelDB timeline store implementation. Contributed by Jonathan Eagles.
(cherry picked from commit daf3e4ef8b)
2015-05-07 10:10:13 -07:00
Junping Du 684a5a6aeb YARN-3523. Cleanup ResourceManagerAdministrationProtocol interface audience. Contributed by Naganarasimha G R
(cherry picked from commit 8e991f4b1d)
2015-05-07 05:40:13 -07:00
Akira Ajisaka 67bebabb21 YARN-3577. Misspelling of threshold in log4j.properties for tests. Contributed by Brahma Reddy Battula.
(cherry picked from commit 995f1b5831fc33042de90267a689333dc110c4c3)
2015-05-07 13:33:17 +09:00
Vinod Kumar Vavilapalli d127dd64a7 YARN-3385. Fixed a race-condition in ResourceManager's ZooKeeper based state-store to avoid crashing on duplicate deletes. Contributed by Zhihai Xu.
(cherry picked from commit 4c7b9b6abe)
2015-05-06 17:52:13 -07:00
Junping Du 79babc4bd2 YARN-3580. [JDK8] TestClientRMService.testGetLabelsToNodes fails. Contributed by Robert Kanter.
(cherry picked from commit 31b627b2a8)
2015-05-06 16:51:56 -07:00
Robert Kanter 28b40aca98 YARN-3491. PublicLocalizer#addResource is too slow. (zxu via rkanter)
(cherry picked from commit b72507810a)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-05-06 14:20:08 -07:00
Jian He f8488b5aeb YARN-3301. Fixed the format issue of the new RM attempt web page. Contributed by Xuan Gong
(cherry picked from commit 0d3188fd25)
2015-05-06 13:51:44 -07:00
Vinod Kumar Vavilapalli da410ea02a YARN-3243. Moving CHANGES.txt entry to the right release.
(cherry picked from commit 185e63a726)
2015-05-06 10:44:19 -07:00
Xuan a0445eae5c YARN-3582. NPE in WebAppProxyServlet. Contributed by Jian He
(cherry picked from commit a583a40693)
2015-05-05 21:43:35 -07:00
Jian He eb0c6d2ee7 YARN-3343. Increased TestCapacitySchedulerNodeLabelUpdate#testNodeUpdate timeout. Contributed by Rohith Sharmaks
(cherry picked from commit e4c3b52c89)
2015-05-05 11:34:11 -07:00
Xuan 02cdcaec58 YARN-2123. Progress bars in Web UI always at 100% due to non-US locale.
Contributed by Akira AJISAKA

(cherry picked from commit b7dd3a4f04)
2015-05-05 10:45:18 -07:00
Junping Du 6ea007ba91 YARN-3396. Handle URISyntaxException in ResourceLocalizationService. (Contributed by Brahma Reddy Battula)
(cherry picked from commit 3810242062)
2015-05-05 10:19:16 -07:00
Jason Lowe 0f30913ea8 YARN-3552. RM Web UI shows -1 running containers for completed apps. Contributed by Rohith
(cherry picked from commit 9356cf8676)
2015-05-05 13:43:57 +00:00
Jian He f71c49c5e3 YARN-2725. Added test cases of retrying creating znode in ZKRMStateStore. Contributed by Tsuyoshi Ozawa
(cherry picked from commit d701acc9c6)
2015-05-04 16:17:01 -07:00
Wangda Tan 4e77ee2f6f YARN-3375. NodeHealthScriptRunner.shouldRun() check is performing 3 times for starting NodeHealthScriptRunner (Devaraj K via wangda)
(cherry picked from commit 71f4de220c)
2015-05-04 16:02:57 -07:00
Allen Wittenauer 02e650248d YARN-2980. Move health check script related functionality to hadoop-common (Varun Saxena via aw)
(cherry picked from commit d4ac6822e1)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
2015-05-04 16:00:20 -07:00
Jason Lowe 99b49a8181 YARN-3097. Logging of resource recovery on NM restart has redundancies. Contributed by Eric Payne
(cherry picked from commit 8f65c793f2)
2015-05-04 10:32:33 -05:00
Tsuyoshi Ozawa f382602de1 YARN-1993. Cross-site scripting vulnerability in TextView.java. Contributed byKenji Kikushima.
(cherry picked from commit e8d0ee5fc9)
2015-05-03 10:51:39 +09:00
Junping Du 8068e79989 YARN-2454. Fix compareTo of variable UNBOUNDED in o.a.h.y.util.resource.Resources. Contributed by Xu Yang.
(cherry picked from commit 57d9a972cb)
2015-05-02 10:20:30 -07:00
Gera Shegalov 9325dd863c YARN-2893. AMLaucher: sporadic job failures due to EOFException in readTokenStorageStream. (Zhihai Xu via gera)
(cherry picked from commit f8204e241d)
2015-05-01 19:04:48 -07:00
Wangda Tan 4151e5729a YARN-3006. Improve the error message when attempting manual failover with auto-failover enabled. (Akira AJISAKA via wangda)
(cherry picked from commit 7d46a806e7)
2015-05-01 16:44:49 -07:00
Robert Kanter 32b3b8e2ef YARN-3363. add localization and container launch time to ContainerMetrics at NM to show these timing information for each active container. (zxu via rkanter)
(cherry picked from commit ac7d152901)
2015-05-01 16:39:59 -07:00
Vinod Kumar Vavilapalli 90f6e8c90f YARN-2619. Added NodeManager support for disk io isolation through cgroups. Contributed by Varun Vasudev and Wei Yan.
(cherry picked from commit 1b3b9e5c31)
2015-04-30 21:43:22 -07:00
Zhijie Shen c9ee316045 YARN-3544. Got back AM logs link on the RM web UI for a completed app. Contributed by Xuan Gong.
(cherry picked from commit 21bf2cdcb77f69abc906e6cd401a8fb221f250e9)
2015-04-30 13:06:46 -07:00
Wangda Tan 100b41add6 YARN-3564. Fix TestContainerAllocation.testAMContainerAllocationWhenDNSUnavailable fails randomly. (Jian He via wangda)
(cherry picked from commit e2e8f77118)
2015-04-30 11:06:47 -07:00
Jian He a7a5737bc4 YARN-3533. Test: Fix launchAM in MockRM to wait for attempt to be scheduled. Contributed by Anubhav Dhoot
(cherry picked from commit 4c1af156ae)
2015-04-29 14:51:04 -07:00
tgraves 2e13183f60 YARN-3517. RM web ui for dumping scheduler logs should be for admins only (Varun Vasudev via tgraves)
(cherry picked from commit 2e215484bd)
2015-04-29 21:27:16 +00:00
Karthik Kambatla 4cc38df7de YARN-3485. FairScheduler headroom calculation doesn't consider maxResources for Fifo and FairShare policies. (kasha)
(cherry picked from commit 8f82970e0c)
2015-04-28 21:03:12 -07:00
Wangda Tan 8ee632caa7 YARN-2740. Fix NodeLabelsManager to properly handle node label modifications when distributed node label configuration enabled. (Naganarasimha G R via wangda)
(cherry picked from commit db1b674b50)
2015-04-27 16:42:36 -07:00
Karthik Kambatla d2d8bc243b Update CHANGES.txt - Pulled in YARN-3465, YARN-3516, and YARN-3464 to branch-2.7 (for 2.7.1)
(cherry picked from commit 32cd2c8d42)
2015-04-27 13:44:06 -07:00
Xuan 0e8af401d7 YARN-3530. ATS throws exception on trying to filter results without
otherinfo. Contributed by zhijie shen

(cherry picked from commit 7f07c4d810)
2015-04-27 10:37:40 -07:00
Karthik Kambatla 4045c41afe YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha)
(cherry picked from commit 47279c3228)
2015-04-26 09:14:12 -07:00
Vinod Kumar Vavilapalli db1a33e956 Moving YARN-3351, YARN-3382, YARN-3472, MAPREDUCE-6238 to the 2.7.1 CHANGES.txt
sections given the recent merge into branch-2.7.

(cherry picked from commit 2f82ae042a)
2015-04-24 17:20:02 -07:00
Jian He 9bf09b334d YARN-2498. Respect labels in preemption policy of capacity scheduler for inter-queue preemption. Contributed by Wangda Tan
(cherry picked from commit d497f6ea2b)
2015-04-24 17:03:57 -07:00
Tsuyoshi Ozawa 3884948d6c YARN-3406. Display count of running containers in the RM's Web UI. Contributed by Ryu Kobayashi.
(cherry picked from commit 4a3dabd94f)
2015-04-25 07:17:42 +09:00
Jason Lowe bd750f160b Fix commit version for YARN-3537
(cherry picked from commit 78fe6e57c7)
2015-04-24 22:09:01 +00:00
Jason Lowe cf4154676b YARN-3537. NPE when NodeManager.serviceInit fails and stopRecoveryStore invoked. Contributed by Brahma Reddy Battula
(cherry picked from commit 5e093f0d40)
2015-04-24 22:04:03 +00:00
Akira Ajisaka 73ba3ebe7c YARN-3444. Fix typo capabililty. Contributed by Gabor Liptak.
(cherry picked from commit 5ce3a77f3c)
2015-04-25 06:09:03 +09:00
Jian He 0583c27fb1 YARN-3387. Previous AM's container completed status couldn't pass to current AM if AM and RM restarted during the same time. Contributed by Sandflee
(cherry picked from commit d03dcb9635)
2015-04-24 12:14:17 -07:00
Xuan c184466939 YARN-3511. Add errors and warnings page to ATS. Contributed by Varun Vasudev
(cherry picked from commit eee9facbba)
2015-04-24 09:43:35 -07:00
Xuan 8f6053ae51 YARN-3516. killing ContainerLocalizer action doesn't take effect when
private localizer receives FETCH_FAILURE status. Contributed by zhihai
xu

(cherry picked from commit 0b3f8957a8)
2015-04-23 16:41:52 -07:00
Vinod Kumar Vavilapalli 0ec6e7e7f5 YARN-3413. Changed Nodelabel attributes (like exclusivity) to be settable only via addToClusterNodeLabels but not changeable at runtime. (Wangda Tan via vinodkv)
(cherry picked from commit f5fe35e297)
2015-04-23 11:25:30 -07:00
Jian He 48d8d88245 YARN-3522. Fixed DistributedShell to instantiate TimeLineClient as the correct user. Contributed by Zhijie Shen
(cherry picked from commit aa4a192feb)
2015-04-23 11:07:54 -07:00