Commit Graph

1012 Commits

Author SHA1 Message Date
Yufei Gu ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00
Jason Lowe e76e7b9f0b YARN-6837. Null LocalResource visibility or resource type can crash the nodemanager. Contributed by Jinjiang Ling
(cherry picked from commit c8df3668ec)
2017-07-20 11:06:39 -05:00
Sunil G d83e8712f6 YARN-5892. Support user-specific minimum user limit percentage in Capacity Scheduler. Contributed by Eric Payne.
(cherry picked from commit ca13b224b2)
2017-07-13 14:50:28 -05:00
Wangda Tan cbb5f601f1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Sunil G via wangda)
Change-Id: I9b19f69788068be05b3295247cdd7b972f8a573c
(cherry picked from commit c583ab02c7)
2017-07-13 10:44:07 -05:00
Xuan edc0aa95f1 YARN-6689. PlacementRule should be configurable. (Jonathan Hung via xgong) 2017-07-12 15:45:51 -07:00
Jason Lowe 401b191a3c YARN-6797. TimelineWriter does not fully consume the POST response. Contributed by Jason Lowe
(cherry picked from commit 655110393b)
2017-07-12 15:54:17 -05:00
Eric Payne e6cdf770ca Revert "YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G)"
This reverts commit eda4ac07c1.
Commit eda4ac07c1 was a separate patch from trunk rather than a cherry-pick. I will cherryp-ick dependencies and then cherry-pick the trunk commit for YARN-2113.
2017-07-12 08:56:40 -05:00
Naganarasimha b08ef44d9e YARN-6428. Queue AM limit is not honored in CS always. Contributed by Bibin A Chundatt.
(cherry picked from commit 626d730bfc)
2017-07-09 17:27:50 +05:30
Eric Payne eda4ac07c1 YARN-2113. Add cross-user preemption within CapacityScheduler's leaf-queue. (Contributed by Sunil G) 2017-07-06 10:43:00 -05:00
Daniel Templeton 15242786b6 YARN-6743. yarn.resourcemanager.zk-max-znode-size.bytes description needs spaces in yarn-default.xml
(Contributed by Lori Loberg via Daniel Templeton)

(cherry picked from commit 25d891a784)
2017-06-28 17:17:56 -07:00
Naganarasimha 92e266c2b6 YARN-5006. ResourceManager quit due to ApplicationStateData exceed the limit size of znode in zk. Contributed by Bibin A Chundatt. 2017-06-24 23:14:51 +05:30
Naganarasimha 578318473c YARN-6680. Avoid locking overhead for NO_LABEL lookups. Contributed by Daryn Sharp
(cherry picked from commit ee89ac84e6)
2017-06-19 22:45:21 +05:30
Naganarasimha 3106f256e2 YARN-6517. Fix warnings from Spotbugs in hadoop-yarn-common(addendum). Contributed by Weiwei Yang.
(cherry picked from commit 7582dedad1)
2017-06-18 19:56:11 +05:30
Eric Payne 33605e0896 YARN-6585. RM fails to start when upgrading from 2.7 for clusters with node labels. Contributed by Sunil G.
(cherry picked from commit 5578af8603)
2017-06-09 16:18:16 -05:00
Daniel Templeton c5b28c03a1 YARN-6679. Reduce Resource instance overhead via non-PBImpl
(Contributed by Daryn Sharp via Daniel Templeton)
2017-06-08 10:41:32 -07:00
Robert Kanter 2719cc0388 YARN-6602. Impersonation does not work if standby RM is contacted first (rkanter)
(cherry picked from commit 9855225a79)
2017-05-19 15:43:43 -07:00
Haibo Chen 86fcb8a66c YARN-6457. Allow custom SSL configuration to be supplied in WebApps. (Sanjay M Pujare via Haibo Chen)
(cherry picked from commit 1769b12a77)
2017-05-08 15:47:01 -07:00
Jason Lowe 6ca0c134e4 YARN-3839. Quit throwing NMNotYetReadyException. Contributed by Manikandan R
(cherry picked from commit 424887ecb7)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/TestContainerManagerRecovery.java
2017-05-08 17:25:11 -05:00
Steve Loughran 5ca4f0f82e YARN-524 TestYarnVersionInfo failing if generated properties doesn't include an SVN URL
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1463300 13f79535-47bb-0310-9956-ffa450edef68
(cherry picked from commit 7c65dbb938)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2017-05-01 08:58:12 -05:00
Naganarasimha 29370e7784 YARN-6517. Fix warnings from Spotbugs in hadoop-yarn-common. Contributed by Weiwei Yang
(cherry picked from commit 4b5bd73ac5)
2017-05-01 16:35:45 +05:30
Daniel Templeton a8512d5aeb YARN-2962. ZKRMStateStore: Limit the number of znodes under a znode (Contributed by Varun Sexena via Daniel Templeton)
(cherry picked from commit 2e52789edf)
2017-04-28 13:39:11 -07:00
Haibo Chen 894521673b YARN-6510. Fix profs stat file warning caused by process names that includes parenthesis. (Wilfred Spiegelenburg via Haibo Chen)
(cherry picked from commit 4f3ca0396a)
2017-04-26 11:47:23 -07:00
Sunil G a9495b206b YARN-6164. Expose Queue Configurations per Node Label through YARN client api. Contributed by Benson Qiu. 2017-04-24 12:16:18 +05:30
Junping Du d77cada182 YARN-3760. FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close(). Contributed by Haibo Chen.
(cherry picked from commit c26ccf1adb3a72df3f68e1150b86b813c691203a)
(cherry picked from commit 9d9087a67a)
2017-04-12 13:54:58 -07:00
Jason Lowe 84b7f2e956 YARN-6288. Exceptions during aggregated log writes are mishandled. Contributed by Akira Ajisaka
(cherry picked from commit 1b081ca27e)
2017-04-06 16:25:41 -05:00
Jason Lowe 903278559a YARN-6403. Invalid local resource request can raise NPE and make NM exit. Contributed by Tao Yang
(cherry picked from commit e8071aa249)
2017-04-05 12:31:48 -05:00
Daniel Templeton c2636468d7 YARN-5685. RM configuration allows all failover methods to disabled when automatic failover is enabled
(cherry picked from commit 640ba1d23f)
2017-03-29 12:38:50 -07:00
Robert Kanter a65011a128 YARN-6050. AMs can't be scheduled on racks or nodes (rkanter) 2017-03-28 14:33:08 -07:00
Wangda Tan c97da481d7 YARN-6339. Improve performance for createAndGetApplicationReport. (Yunjiong Zhao via wangda)
(cherry picked from commit cd014d57aa)
2017-03-27 13:59:36 -07:00
Junping Du 8b31ff6dba YARN-6294. ATS client should better handle Socket closed case. Contributed by Li Lu. 2017-03-15 11:49:51 -07:00
Junping Du 0e7879052a YARN-6313. YARN logs cli should provide logs for a completed container even when application is still running. Contributed by Xuan Gong.
(cherry picked from commit b88f5e0f7858d1d89b79dfd325b767c34416052d)
2017-03-14 12:58:41 -07:00
Jason Lowe ef95e74eb1 YARN-6321. TestResources test timeouts are too aggressive. Contributed by Eric Badger
(cherry picked from commit 9649c27864)
2017-03-10 13:07:20 -06:00
Jason Lowe 1a0358b59a YARN-6310. OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions. Contributed by Haibo Chen
(cherry picked from commit deb9f56946)
2017-03-10 11:09:04 -06:00
Karthik Kambatla 4218671f3c YARN-6264. AM not launched when a single vcore is available on the cluster. (Yufei Gu via kasha)
(cherry picked from commit a96afae125)
2017-03-09 23:16:19 -08:00
Jian He 712434e744 YARN-6270. WebUtils.getRMWebAppURLWithScheme() needs to honor RM HA setting. Contributed by Xuan Gong 2017-03-03 15:24:23 -08:00
Gera Shegalov df35ba81fe YARN-1728. Workaround guice3x-undecoded pathInfo in YARN WebApp. (Yuanbo Liu via gera) 2017-02-28 09:46:57 -08:00
Xuan 19c5b3d622 YARN-6228: EntityGroupFSTimelineStore should allow configurable cache
stores. Contributed by Li Lu

(cherry picked from commit 53d372a255)
2017-02-24 11:00:16 -08:00
Wangda Tan a82480c7d4 YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda) 2017-02-23 13:59:20 -08:00
Karthik Kambatla ca7a6a7365 YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha)
(cherry picked from commit 718ad9f6ee)
2017-02-22 15:52:44 -08:00
Daniel Templeton 93219b6454 YARN-6125. The application attempt's diagnostic message should have a maximum size
(Contributed by Andras Piros via Daniel Templeton)

(cherry picked from commit c7a36e6130)
2017-02-17 14:38:06 -08:00
Karthik Kambatla 46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha)
(cherry picked from commit 6c25dbcdc0)
2017-02-15 23:23:18 -08:00
Junping Du 323782b389 Addendum patch for YARN-6174 - remove deleted file.
(cherry picked from commit 0fc6f38379)
2017-02-15 09:09:27 -08:00
Junping Du 2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong.
(cherry picked from commit ce2d5bfa5f)
2017-02-15 09:08:54 -08:00
Junping Du 205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong.
(cherry picked from commit 464ff479ce)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
2017-02-13 06:16:04 -08:00
Sunil G 7b2b6caf12 YARN-6118. Add javadoc for Resources.isNone. Contributed by Andres Perez.
(cherry picked from commit 3a0a0a485a)
2017-02-10 15:54:59 +05:30
Junping Du 33e7c3044f YARN-6145. Improve log message on fail over. Contributed by Jian He.
(cherry picked from commit eec52e158b)
2017-02-08 02:15:11 -08:00
Junping Du ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong.
(cherry picked from commit 9dbfab1284)
2017-02-07 02:14:09 -08:00
Junping Du b7d59e790b HADOOP-12097. Allow port range to be specified while starting webapp. Contributed by Varun Saxena.
(cherry picked from commit cce35c3815)

Conflicts:

	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
2017-02-05 20:13:49 -08:00
Junping Du c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton 443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton)
(cherry picked from commit b3befc021b)
2017-01-26 14:07:46 -08:00
Sangjin Lee c4d3b28c2d YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo.
(cherry picked from commit dc6ec97048)
2017-01-25 15:36:07 -08:00
Karthik Kambatla 0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du 8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Jason Lowe 94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He
(cherry picked from commit 69fa81679f)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
2017-01-23 17:23:32 +00:00
bibinchundatt 580275628b YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are incorrect about behavior when a request is out of bounds. Contributed by Julia Sommer.
(cherry picked from commit a9d946d4df)
2017-01-17 14:18:41 +05:30
Daniel Templeton 718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton)
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
2017-01-11 15:36:18 -08:00
Junping Du ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay.
(cherry picked from commit 4db119b7b5)
2017-01-10 18:22:45 -08:00
Naganarasimha ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
(cherry picked from commit 287d3d6804)
2017-01-09 08:42:38 -08:00
Junping Du f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Arun Suresh 81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh)
(cherry picked from commit ac1e5d4f77)
2017-01-06 11:25:49 -08:00
Arun Suresh 2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 2273a74c1f)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
2017-01-06 11:15:10 -08:00
Arun Suresh 3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh)
(cherry picked from commit b0aace21b1)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
2017-01-06 11:15:09 -08:00
Arun Suresh 368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh)
(cherry picked from commit 3219b7b4ac)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
2017-01-06 11:15:09 -08:00
Daniel Templeton 8cb7aa2b52 YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton)
(cherry picked from commit 165d01a73e)
2016-12-30 08:56:03 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Robert Kanter 88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter)
(cherry picked from commit 9262797e86)
2016-12-27 14:14:41 -08:00
Sunil G 23bd68a4a4 YARN-5970. Validate application update timeout request parameters. Contributed by Rohith Sharma K S. 2016-12-12 11:59:14 +05:30
Robert Kanter 9cba7c6d71 YARN-5963. Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code (gsohn via rkanter)
(cherry picked from commit 72fe546841)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
2016-12-07 15:09:37 -08:00
Sunil a636a87270 YARN-5965. Retrospect ApplicationReport#getApplicationTimeouts. Contributed by Rohith Sharma K S
(cherry picked from commit ab923a53fc)
2016-12-08 00:34:45 +05:30
Jian He 0f6fbfc0db YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. Contributed by Akira Ajisaka & Wangda Tan
(cherry picked from commit 43ebff2e35)
2016-12-05 11:40:26 -08:00
Jason Lowe 6b17d449bf YARN-5915. ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write. Contributed by Atul Sikaria 2016-12-02 16:58:43 +00:00
Robert Kanter 99b046f8a9 MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter)
(cherry picked from commit c87b3a448a)
2016-12-01 17:30:02 -08:00
Daniel Templeton b8bebb8607 YARN-5774. MR Job stuck in ACCEPTED status without any progress in Fair Scheduler
if set yarn.scheduler.minimum-allocation-mb to 0. (Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 25f9872be6)
2016-11-29 10:12:10 -08:00
Sunil c28dc5e5a4 YARN-4206. Add Application timeouts in Application report and CLI. Contributed by Rohith Sharma K S.
(cherry picked from commit eb0a483ed0)
2016-11-24 18:27:50 +05:30
Naganarasimha 2d94f938c5 YARN-4330. MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages. Contributed by Varun Saxena
(cherry picked from commit 3541ed8068)
2016-11-23 14:14:56 +05:30
Naganarasimha b96ed43897 YARN-5911. DrainDispatcher does not drain all events on stop even if setDrainEventsOnStop is true. Contributed by Varun Saxena.
(cherry picked from commit 4667564162)
2016-11-23 09:36:43 +05:30
Rohith Sharma K S 74ac78b3c0 YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee.
(cherry picked from commit d65603517e)
2016-11-16 15:17:46 +05:30
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu
(cherry picked from commit 5af572b644)
2016-11-15 11:36:08 -08:00
Wangda Tan a76e214556 YARN-5552. Add Builder methods for common yarn API records. (Tao Jie via wangda)
(cherry picked from commit ede1a473f5)
2016-11-11 13:42:45 -08:00
Eric Payne dd5b9dabf9 YARN-4218. Metric for resource*time that was preempted. Contributed by Chang Li. 2016-11-10 23:02:42 +00:00
Jian He a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Arun Suresh 4a023b4414 YARN-5833. Addendum patch to include missing changes to yarn-default.xml
(cherry picked from commit 280357c29f)
2016-11-09 07:16:39 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey.
(cherry picked from commit e1c6ef2efa)
2016-11-08 16:33:58 -08:00
Jason Lowe 8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri
(cherry picked from commit 3f93ac0733)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
2016-11-08 22:16:53 +00:00
Varun Saxena 92b7491f23 YARN-5809. AsyncDispatcher possibly invokes multiple shutdown threads when handling exception (Jian He via Varun Saxena) 2016-11-01 04:37:56 +05:30
Naganarasimha 62d8536df6 Revert "YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin"
This reverts commit b92ad5d857.
2016-10-28 12:58:42 +05:30
Naganarasimha b92ad5d857 YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin
(cherry picked from commit 3de130252d)
2016-10-28 12:44:14 +05:30
Li Lu 2b0fd1f4e2 YARN-4668. Reuse objectMapper instance in Yarn. (Yiqun Lin via gtcarrera9)
(cherry picked from commit 221582c4ab)
2016-10-27 15:26:05 -07:00
Varun Saxena f572448809 YARN-4388. Cleanup mapreduce.job.hdfs-servers from yarn-default.xml (Junping Du via Varun Saxena) 2016-10-28 02:24:24 +05:30
Subru Krishnan 81595a127f YARN-5711. Propogate exceptions back to client when using hedging RM failover provider.
(cherry picked from commit 0a166b1347)
2016-10-24 19:02:03 -07:00
Varun Saxena e9a58691ab YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun Saxena) 2016-10-14 23:32:29 +05:30
Jason Lowe 28bd08798d YARN-5551. Ignore file backed pages from memory computation when smaps is enabled. Contributed by Rajesh Balamohan
(cherry picked from commit ecb51b857a)
2016-10-11 15:14:08 +00:00
Chris Douglas 539c3e4dc0 YARN-5702. Refactor TestPBImplRecords for reuse in other YARN modules. Contributed by Subru Krishnan
(cherry picked from commit d65b957776)
2016-10-05 12:19:21 -07:00
Karthik Kambatla 1578781fe9 YARN-4767. Network issues can cause persistent RM UI outage. (Daniel Templeton via kasha)
(cherry picked from commit 736d33cddd)
2016-10-03 14:36:14 -07:00
Naganarasimha 04f594dbea YARN-4855. Should check if node exists when replace nodelabels. Contributeed by Tao Jie
(cherry picked from commit 6e130c308c)
2016-10-03 02:13:46 -04:00
Subru Krishnan 9f43f89899 YARN-5384. Expose priority in ReservationSystem submission APIs. (Sean Po via Subru).
(cherry picked from commit 3a3697deab)
2016-09-30 19:43:29 -07:00
Jian He 39f896e1a1 YARN-4205. Add a service for monitoring application life time out. Contributed by Rohith Sharma K S
(cherry picked from commit 2ae5a3a5bf)
2016-09-29 22:06:36 +08:00
Varun Vasudev 3f1d3e3177 YARN-5662. Provide an option to enable ContainerMonitor. Contributed by Jian He.
(cherry picked from commit bc2656f09f)
2016-09-28 15:27:09 +05:30
Arun Suresh 7484d0b1b9 YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh 2016-09-26 23:55:13 +08:00
Naganarasimha 0e4a5611ff YARN-3692. Allow REST API to set a user generated message when killing an application. Contributed by Rohith Sharma K S 2016-09-24 21:13:01 +05:30
Varun Saxena b58651d8d4 YARN-5539. TimelineClient failed to retry on java.net.SocketTimeoutException: Read timed out (Junping Du via Varun Saxena) 2016-09-23 13:30:56 +05:30
Robert Kanter 0e4f2a0edf YARN-4973. YarnWebParams next.fresh.interval should be next.refresh.interval (templedf via rkanter)
(cherry picked from commit 5ffd4b7c1e)
2016-09-22 16:46:27 -07:00
Wangda Tan b9b6396e09 YARN-4591. YARN Web UIs should provide a robots.txt. (Sidharta Seethana via wangda)
(cherry picked from commit 5a58bfee30)
2016-09-20 17:23:01 -07:00
Naganarasimha 20729ead91 YARN-5642. Typos in 9 log messages. Contributed by Mehran Hassani 2016-09-17 10:37:42 +05:30
Naganarasimha 656b520dc6 YARN-4232. TopCLI console support for HA mode. Contributed by Bibin A Chundatt
(cherry picked from commit ade7c2bc9c)
2016-09-17 09:54:29 +05:30
Ray Chiang 4ee1729cb0 YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang) 2016-09-02 14:57:05 -07:00
Arun Suresh b279f42d79 YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh)
(cherry picked from commit d6d9cff21b)
2016-08-30 16:33:31 -07:00
Subru Krishnan 3ad7cb3e1c YARN-5327. API changes required to support recurring reservations in the YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru).
(cherry picked from commit b930dc3ec0)
2016-08-26 17:01:37 -07:00
Junping Du af1502fcab YARN-5557. Add localize API to the ContainerManagementProtocol. Contributed by Jian He.
(cherry picked from commit 9ef632f3b0)
2016-08-26 09:05:33 -07:00
Varun Vasudev 49a97a76f5 YARN-5430. Return container's ip and host from NM ContainerStatus call. Contributed by Jian He. 2016-08-26 16:33:21 +05:30
Junping Du d464483bf7 YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 2016-08-18 07:23:29 -07:00
Varun Saxena 5fa7a14da9 YARN=5526. DrainDispacher#serviceStop blocked if setDrainEventsOnStop invoked (sandflee via Varun Saxena) 2016-08-18 12:23:34 +05:30
Varun Saxena 96e9f8bec9 YARN-5475. Fix test failure of TestAggregatedLogFormat#testReadAcontainerLogs1 (Jun Gong via Varun Saxena) 2016-08-16 20:28:02 +05:30
Ray Chiang 372f6f8456 YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang)
(cherry picked from commit dbe9e70cc084220ea1f68da850cdb092281b5e96)
2016-08-10 19:26:49 -07:00
Arun Suresh 5f7edb79d1 YARN-5457. Refactor DistributedScheduling framework to pull out common functionality. (asuresh)
(cherry picked from commit 82c9e06101)
2016-08-09 00:46:08 -07:00
Junping Du adcecd417b YARN-5470. Differentiate exactly match with regex in yarn log CLI. Contributed by Xuan Gong.
(cherry picked from commit e605d47df0)
2016-08-05 11:00:13 -07:00
Arun Suresh f197378f81 YARN-5113. Refactoring and other clean-up for distributed scheduling. (Konstantinos Karanasos via asuresh)
(cherry picked from commit e5766b1dbe)
2016-07-31 11:49:09 -07:00
Li Lu 0b5d96abb5 YARN-5436. Race in AsyncDispatcher can cause random test failures in Tez (probably YARN also). (Zhiyuan Yang via gtcarrera9)
(cherry picked from commit 7086fc72ee)
2016-07-28 16:52:21 -07:00
Jason Lowe 2f854b8386 YARN-5438. TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM. Contributed by Rohith Sharma K S
(cherry picked from commit a1890c32c5)
2016-07-28 21:48:40 +00:00
Junping Du 932bf79244 YARN-5432. Lock already held by another process while LevelDB cache store creation for dag. Contributed by Li Lu.
(cherry picked from commit 7f3c306e2e)
2016-07-28 06:36:31 -07:00
Subru Krishnan 4ecfd05095 YARN-5441. Fixing minor Scheduler test case failures
(cherry picked from commit d2cbfd7de3)
2016-07-27 18:26:29 -07:00
Arun Suresh 0f6ff20652 YARN-5351. ResourceRequest should take ExecutionType into account during comparison. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 2d8d183b19)
2016-07-26 19:09:22 -07:00
Varun Vasudev 9e326f4eeb YARN-5309. Fix SSLFactory truststore reloader thread leak in TimelineClientImpl. Contributed by Weiwei Yang.
(cherry picked from commit 9ccf935b2c)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
2016-07-20 12:55:38 +05:30
Junping Du ad8180fc6c YARN-5213. Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently. Contributed by Xuan Gong.
(cherry picked from commit dc2f4b6ac8)
2016-07-19 07:19:17 -07:00
Junping Du e3bc4faa96 YARN-5339. Passing file to -out for YARN log CLI doesnt give warning or error code. Contributed by Xuan Gong.
(cherry picked from commit 7e5355c14e)
2016-07-15 09:24:03 -07:00
Vinod Kumar Vavilapalli 4a6cef8c91 YARN-5200. Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option. Contributed by Xuan Gong.
(cherry picked from commit eb47163234)
2016-07-13 10:55:18 -07:00
Robert Kanter 7aaf82d469 YARN-4366. Fix Lint Warnings in YARN Common (templedf via rkanter)
(cherry picked from commit bf6f4a3b98)
2016-07-12 15:13:00 -07:00
Jian He e94e6be772 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:37:52 -07:00
Jian He b7604a22e6 YARN-5233. Support for specifying a path for ATS plugin jars. Contributed by Li Lu
(cherry picked from commit 8a9d293dd6)
2016-07-06 16:45:32 -07:00
Wangda Tan e246cf74c9 YARN-5294. Pass remote ip address down to YarnAuthorizationProvider. (Jian He via wangda) 2016-07-06 16:42:35 -07:00
Xuan b7f18f2815 YARN-5266. Wrong exit code while trying to get app logs using regex via CLI. Contributed by Xuan Gong
(cherry picked from commit 6ab5aa1c1f)
2016-06-22 21:50:09 -07:00
Arun Suresh e64d21b297 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh)
(cherry picked from commit 99e5dd68d0)
2016-06-22 19:05:28 -07:00
Junping Du ea4cab2330 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev.
(cherry picked from commit d0162f2040)
2016-06-19 17:45:57 -07:00
Junping Du 6afa044dda YARN-5251. Yarn CLI to obtain App logs for last 'n' bytes fails. Contributed by Xuan Gong.
(cherry picked from commit c35fa4a0e5)
2016-06-17 08:25:33 -07:00
Junping Du 6dd34baf3c YARN-5237. Fix missing log files issue in rolling log aggregation. Contributed by Xuan Gong.
(cherry picked from commit 5dfc38ff57)
2016-06-15 16:20:07 -07:00
Varun Vasudev 75235149af YARN-5223. Container line in yarn logs output for a live application should include the hostname for the container. Contributed by Xuan Gong.
(cherry picked from commit 25064fb2fb)
2016-06-15 12:43:26 +05:30
Wangda Tan 863bfa4d6a YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:12:00 -07:00
Wangda Tan cd3bdbc70a YARN-4887. Add allocation request ID to AM-RM protocol for identifying resource-requests explicitly. (Subru Krishnan via wangda)
(cherry picked from commit eec835ec17)
2016-06-13 21:57:55 -07:00
Arun Suresh fccb641942 YARN-5124. Modify AMRMClient to set the ExecutionType in the ResourceRequest. (asuresh)
(cherry picked from commit 5143277958)
2016-06-12 09:43:19 -07:00
Wangda Tan e91e9e596d YARN-3426. Add jdiff support to YARN. (vinodkv via wangda)
(cherry picked from commit 03fc6b1bb0)
2016-06-10 09:53:40 -07:00
Vinod Kumar Vavilapalli f6cd1bcf89 YARN-5191. Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option. (Xuan Gong via vinodkv)
(cherry picked from commit 9378d9428f)
2016-06-09 12:32:53 -07:00
Wangda Tan aef6e455b1 YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-06-09 10:08:31 -07:00
Naganarasimha 27fc225558 YARN-4308. ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats. Contributed by Sunil G
(cherry picked from commit 1500a0a300)
2016-06-09 05:44:41 +05:30
Ming Ma 74e2b5efa2 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma)
(cherry picked from commit 4a1cedc010)
2016-06-06 14:34:47 -07:00
Arun Suresh 384a84828f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh)
(cherry picked from commit dc26601d8f)
2016-06-02 09:14:30 -07:00
Wangda Tan b63ebca9dd Addendum patch for YARN-4844, to supress few more deprecation warnings. 2016-06-01 14:37:20 -07:00
Xuan 04062593b4 YARN-5088. Improve "yarn log" command-line to read the last K bytes for the log files. Contributed by Xuan Gong
(cherry picked from commit 0bc05e40fa)
2016-06-01 13:45:24 -07:00
Varun Vasudev d7bb28f15b YARN-4007. Add support for different network setups when launching the docker container. Contributed by Sidharta Seethana.
(cherry picked from commit 86fb58b7dc)
2016-05-29 21:33:43 +05:30
Varun Vasudev 4f36c3d214 YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:55:39 +05:30
Arun Suresh 0b1472fbc0 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh)
(cherry picked from commit aa975bc781)
2016-05-27 14:07:15 -07:00
Xuan 38ae5954df YARN-5089. Improve "yarn log" command-line "logFiles" option to support
regex. Contributed by Xuan Gong

(cherry picked from commit bde819abbb)
2016-05-26 23:50:14 -07:00
Rohith Sharma K S ef3e0c6fd6 YARN-5005. TestRMWebServices#testDumpingSchedulerLogs fails randomly. Contributed by Bibin A Chundatt.
(cherry picked from commit 0a544f8a3e)
2016-05-27 10:48:47 +05:30
Arun Suresh 44cbf5b7f5 YARN-5162. Fix Exceptions thrown during in registerAM call when Distributed Scheduling is Enabled (Hitesh Sharma via asuresh)
(cherry picked from commit 5b41b288d0)
2016-05-26 14:58:15 -07:00
Carlo Curino c656977961 YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino)
(cherry picked from commit 013532a95e)
2016-05-25 16:58:19 -07:00
Karthik Kambatla 0ac8c098e8 YARN-3344. Fix warning - procfs stat file is not in the expected format. (Ravindra Kumar Naik and Akira Ajisaka via kasha)
(cherry picked from commit 77d5ce9069)
2016-05-25 10:51:09 -07:00
Robert Kanter 71e5b36763 YARN-4766. NM should not aggregate logs older than the retention policy (haibochen via rkanter)
(cherry picked from commit e07519b8db)
2016-05-25 10:25:59 -07:00
Naganarasimha 155f25061e YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt
(cherry picked from commit b4078bd17b)
2016-05-24 08:14:53 +08:00
Varun Vasudev e7f396fd47 YARN-5016. Add support for a minimum retry interval for container retries. Contributed by Jun Gong.
(cherry picked from commit 0287c49107)
2016-05-20 12:32:29 +05:30
Junping Du 2703ec6871 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 2016-05-19 14:10:15 -07:00
Arun Suresh 1b4600abcf YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
(cherry picked from commit 341888a0aa)
2016-05-18 22:35:55 -07:00
Arun Suresh d80d24aabf YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
(cherry picked from commit c282a08f38)
2016-05-18 22:08:55 -07:00
Karthik Kambatla b56fc51b70 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha)
(cherry picked from commit c8172f5f14)
2016-05-18 22:08:20 -07:00
Arun Suresh 68964bbaa2 YARN-4335. Allow ResourceRequests to specify ExecutionType of a request ask (kkaranasos via asuresh)
(cherry picked from commit 8ffabfdf4f)
(cherry picked from commit b2a654c5ee)
2016-05-18 22:02:43 -07:00
Karthik Kambatla 2c6c701aff YARN-2882. Add an OPPORTUNISTIC ExecutionType. (Konstantinos Karanasos and Inigo Goiri via kasha)
(cherry picked from commit fb00794368)
2016-05-18 22:02:28 -07:00
Varun Vasudev 3c9392a31d YARN-4913. Yarn logs should take a -out option to write to a directory. Contributed by Xuan Gong.
(cherry picked from commit ef1757790d)
2016-05-18 22:45:06 +05:30
Vinod Kumar Vavilapalli e0d6a9632d YARN-4842. Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs. Contributed by Ram Venkatesh and Xuan Gong.
(cherry picked from commit 87f5e35133)
2016-05-09 22:47:58 -07:00
Jason Lowe ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit d0da13229c)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Vinod Kumar Vavilapalli 8262ef8318 YARN-4905. Improved "yarn logs" command-line to optionally show log metadata also. Contributed by Xuan Gong.
(cherry picked from commit 9e37fe3b7a)
2016-05-04 14:17:09 -07:00
Jason Lowe d8fcf02069 YARN-4834. ProcfsBasedProcessTree doesn't track daemonized processes. Contributed by Nathan Roberts
(cherry picked from commit c6b4839168)
2016-05-03 22:14:50 +00:00
Varun Vasudev 6561e3b500 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong.
(cherry picked from commit 0f25a1bb52)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
2016-04-29 16:22:06 +05:30
Jian He 6ba39a1597 YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe
(cherry picked from commit 4a8508501b)
2016-04-28 21:54:30 -07:00
Jian He a9707dceaf YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:28:03 -07:00
Varun Vasudev cf3e93ee73 YARN-4966. Improve yarn logs to fetch container logs without specifying nodeId. Contributed by Xuan Gong.
(cherry picked from commit 66b07d8374)
2016-04-27 15:57:17 +05:30
Chris Douglas 878924189d YARN-4976. Missing NullPointer check in ContainerLaunchContextPBImpl. Contributed by Giovanni Matteo Fumarola
(cherry picked from commit 95a5046607)
2016-04-21 10:46:35 -07:00
Vinod Kumar Vavilapalli 342bb438f2 Fixed TimelineClient to retry SocketTimeoutException too. Contributed by Xuan Gong.
(cherry picked from commit 477003730e)
2016-04-18 11:49:00 -07:00
Xuan 94e4d349b0 YARN-4965. Distributed shell AM failed due to ClientHandlerException thrown by jersey. Contributed by Junping Du
(cherry picked from commit e6c0742012)
2016-04-16 19:40:11 -07:00
Naganarasimha 2213a40f30 YARN-4909. Fix intermittent failures of TestRMWebServices And TestRMWithCSRFFilter. Contributed by Bibin A Chundatt
(cherry picked from commit fdbafbc9e5)
2016-04-15 23:40:07 +05:30
Rohith Sharma K S d93cfefea5 YARN-4897. dataTables_wrapper change min height. Contributed by Bibin A Chundatt.
(cherry picked from commit 4ee7f0a3e7)
2016-04-12 10:28:18 +05:30
Jason Lowe f1a370ce8b Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla"
This reverts commit 814ceeb489.
2016-04-11 15:56:29 +00:00
Akira Ajisaka 8cf6630fc6 YARN-4630. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit 1ff27f9d12)
2016-04-11 14:55:37 +09:00
Varun Vasudev 4baee3eb67 YARN-4562. YARN WebApp ignores the configuration passed to it for keystore settings. Contributed by Sergey Shelukhin.
(cherry picked from commit 07920acc16)
2016-04-11 09:40:28 +05:30
Jason Lowe 814ceeb489 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit 1cbcd4a491)
2016-04-05 13:41:18 +00:00
naganarasimha 3772602848 YARN-4746. yarn web services should convert parse failures of appId, appAttemptId and containerId to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 5092c94195)
2016-04-04 18:08:18 +05:30
Karthik Kambatla ecfd25dc65 YARN-4657. Javadoc comment is broken for Resources.multiplyByAndAddTo(). (Daniel Templeton via kasha)
(cherry picked from commit 81d04cae41)
2016-04-01 16:20:18 -07:00
Arun Suresh f56266fccc Missing file for YARN-4895.
(cherry picked from commit 5686caa9fc)
2016-04-01 15:58:38 -07:00
Varun Vasudev 10d8f8a39c Revert "YARN-4857. Add missing default configuration regarding preemption of CapacityScheduler. Contributed by Kai Sasaki."
This reverts commit 0064cba169.

(cherry picked from commit 3488c4f2c9)
2016-04-01 12:21:49 +05:30
Sangjin Lee b91928cd26 YARN-4183. Clarify the behavior of timeline service config properties (Naganarasimha G R via sjlee)
(cherry picked from commit 6d67420dbc)
2016-03-31 10:49:45 -07:00
Varun Vasudev dcad99892d YARN-4857. Add missing default configuration regarding preemption of CapacityScheduler. Contributed by Kai Sasaki.
(cherry picked from commit 0064cba169)
2016-03-31 14:06:45 +05:30
Jason Lowe 3cea00a294 YARN-4814. ATS 1.5 timelineclient impl call flush after every event write. Contributed by Xuan Gong
(cherry picked from commit e8fc81f9c8)
2016-03-25 20:19:15 +00:00
Junping Du 76602161c0 YARN-4814. ATS 1.5 timelineclient impl call flush after every event write. Contributed by Xuan Gong.
(cherry picked from commit af1d125f9c)
2016-03-23 08:58:05 -07:00
Karthik Kambatla 95b59c17a7 YARN-4732. *ProcessTree classes have too many whitespace issues
(cherry picked from commit 7fae4c68e6)
2016-03-20 10:47:56 -07:00
Xuan 398dbde538 Revert "YARN-4814. ATS 1.5 timelineclient impl call flush after every event write. (Xuan Gong via gtcarrera9)"
This reverts commit 3a0b695441.

(cherry picked from commit a888b5bc53)
2016-03-15 22:11:16 -07:00
Li Lu 94649b9cc0 YARN-4817. Change Log Level to DEBUG for putDomain call in ATS 1.5. (Xuan Gong via gtcarrera9)
(cherry-picked from commit 5176a69868)
2016-03-15 11:48:37 -07:00
Li Lu 14a4110b75 YARN-4814. ATS 1.5 timelineclient impl call flush after every event write. (Xuan Gong via gtcarrera9)
(cherry picked from commit 3a0b695441)
2016-03-15 11:42:30 -07:00
Siddharth Seth edd9abf302 YARN-4816. Fix incompatible change in SystemClock.
(cherry picked from commit eba66a64d2)
2016-03-14 20:15:41 -07:00
Junping Du 3d5ac829da YARN-4545. Allow YARN distributed shell to use ATS v1.5 APIs. Li Lu via junping_du
(cherry picked from commit f291d82cd4)
2016-03-14 08:30:07 -07:00
Li Lu 76ef097fd1 YARN-4696. Improving EntityGroupFSTimelineStore on exception handling, test setup, and concurrency. (Steve Loughran via gtcarrera9)
(cherry-picked from commit d49cfb3504)
2016-03-10 10:56:51 -08:00
Varun Vasudev e9a0ffc7f1 YARN-4737. Add CSRF filter support in YARN. Contributed by Jonathan Maron.
(cherry picked from commit 43416187c07afb35e3267f94d0a41d8d3cfb5735)
2016-03-07 15:23:36 +05:30
Robert Kanter 872b8d90a6 YARN-4579. Allow DefaultContainerExecutor container log directory permissions to be configurable (rchiang via rkanter)
(cherry picked from commit d7fdec1e6b)
2016-02-25 16:40:05 -08:00