Commit Graph

3174 Commits

Author SHA1 Message Date
Karthik Kambatla 5575328bb6 YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha)
(cherry picked from commit dbbfcf74ab)
2017-02-17 14:07:47 -08:00
Daniel Templeton 10fd5627bb YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)

(cherry picked from commit 585168cc1bdb87398d4f388f3eeccd1c71114982)
2017-02-17 13:14:37 -08:00
Karthik Kambatla afc8124ff6 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha)
(cherry picked from commit a77f432449)
2017-02-16 15:03:35 -08:00
Li Lu 8fc67e5973 YARN-6177. Yarn client should exit with an informative error message if an incompatible Jersey library is used at client. Contributed by Weiwei Yang.
(cherry picked from commit 5d339c46f5)
2017-02-16 13:46:30 -08:00
Rohith Sharma K S f07b52dc5a YARN-6200. Reverting since the same functionality achieved by YARN-1623.
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit a37b3694ea.
2017-02-16 17:58:38 +05:30
Karthik Kambatla 9fecd5063f YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha)
(cherry picked from commit 74dd142250)
2017-02-16 00:03:26 -08:00
Karthik Kambatla fefac1276a YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha)
(cherry picked from commit 11be3f70e0)
2017-02-15 23:51:51 -08:00
Karthik Kambatla 46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha)
(cherry picked from commit 6c25dbcdc0)
2017-02-15 23:23:18 -08:00
Junping Du 323782b389 Addendum patch for YARN-6174 - remove deleted file.
(cherry picked from commit 0fc6f38379)
2017-02-15 09:09:27 -08:00
Junping Du 2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong.
(cherry picked from commit ce2d5bfa5f)
2017-02-15 09:08:54 -08:00
Varun Saxena 9cab25a63d YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena)
(cherry picked from commit b7613e0f40)
2017-02-15 14:58:39 +05:30
Masatake Iwasaki cba7e2c346 YARN-4753. Use doxia macro to generate in-page TOC of YARN site documentation. (iwasakims) 2017-02-15 13:37:25 +09:00
Karthik Kambatla 58acb676a2 YARN-6061. Addendum. Remove extraneous change.
(cherry picked from commit 353a9b2d91)
2017-02-14 15:20:11 -08:00
Karthik Kambatla 37921b3fef YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha)
(cherry picked from commit 652679aa8a)
2017-02-14 13:40:06 -08:00
Karthik Kambatla 30ff5bff1a YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha)
(cherry picked from commit 646c6d6509)
2017-02-13 11:27:02 -08:00
Junping Du 205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong.
(cherry picked from commit 464ff479ce)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
2017-02-13 06:16:04 -08:00
Daniel Templeton 7c3578ca17 YARN-6166. Unnecessary INFO logs in AMRMClientAsyncImpl$CallbackHandlerThread.run (Contributed by Grant Whiteheart via Daniel Templeton)
(cherry picked from commit 1238fa71f8)
2017-02-10 09:48:50 -08:00
Sunil G 7b2b6caf12 YARN-6118. Add javadoc for Resources.isNone. Contributed by Andres Perez.
(cherry picked from commit 3a0a0a485a)
2017-02-10 15:54:59 +05:30
Karthik Kambatla e259a818ad YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha)
(cherry picked from commit 08f93978f3)
2017-02-09 14:45:21 -08:00
Karthik Kambatla 41361ec39e YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:12:21 -08:00
Jason Lowe e330c3806c YARN-6137. Yarn client implicitly invoke ATS client which accesses HDFS. Contributed by Li Lu
(cherry picked from commit 37b4acf7ce)
2017-02-08 15:00:50 -06:00
Junping Du 33e7c3044f YARN-6145. Improve log message on fail over. Contributed by Jian He.
(cherry picked from commit eec52e158b)
2017-02-08 02:15:11 -08:00
Sidharta S 127ed38d13 YARN-5258. Document Use of Docker with LinuxContainerExecutor (Contributed by Daniel Templeton) 2017-02-07 16:35:56 -08:00
Junping Du ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong.
(cherry picked from commit 9dbfab1284)
2017-02-07 02:14:09 -08:00
Junping Du b7d59e790b HADOOP-12097. Allow port range to be specified while starting webapp. Contributed by Varun Saxena.
(cherry picked from commit cce35c3815)

Conflicts:

	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
2017-02-05 20:13:49 -08:00
Junping Du c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton)
(cherry picked from commit 3e06475307)
2017-01-31 16:26:41 -08:00
Daniel Templeton ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton)
(cherry picked from commit 87852b6ef4)
2017-01-30 21:46:38 -08:00
Ray Chiang 9f0b813350 YARN-6106. Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei Gu via rchiang)
(cherry picked from commit 3f970ac363dc2faf4f54180ee5b64d0c60807097)
2017-01-30 13:42:11 -08:00
Karthik Kambatla c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha)
(cherry picked from commit 11e44bdda7)
2017-01-30 13:39:35 -08:00
Jason Lowe 4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Junping Du 660f4d8631 YARN-6126. Obtaining app logs for Running application fails with json parse error. Contributed by Xuan Gong.
(cherry picked from commit 1a16431bd0)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
2017-01-27 06:33:44 -08:00
Karthik Kambatla 1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha)
(cherry picked from commit 4d3f73acc0)
2017-01-26 14:11:50 -08:00
Daniel Templeton c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9)
2017-01-26 14:08:46 -08:00
Daniel Templeton 443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton)
(cherry picked from commit b3befc021b)
2017-01-26 14:07:46 -08:00
Jason Lowe b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger"
This reverts commit 45407acaed.
2017-01-26 14:42:22 +00:00
Sangjin Lee b799ea7641 YARN-3637. Handle localization sym-linking correctly at the YARN level. Contributed by Chris Trezzo.
(cherry picked from commit 425a7e5028)
2017-01-25 15:52:05 -08:00
Sangjin Lee c4d3b28c2d YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo.
(cherry picked from commit dc6ec97048)
2017-01-25 15:36:07 -08:00
Jason Lowe 45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger
(cherry picked from commit 9e19f758c1)
2017-01-25 21:54:17 +00:00
Karthik Kambatla 73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha)
(cherry picked from commit abedb8a9d8)
2017-01-25 12:19:27 -08:00
Karthik Kambatla 71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha)
(cherry picked from commit e224c96234)
2017-01-25 12:19:20 -08:00
Karthik Kambatla 0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du 8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Wangda Tan 1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Jason Lowe 1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Rohith Sharma K S 26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G.
(cherry picked from commit 0101267d9d)
2017-01-24 11:01:52 +05:30
Jason Lowe 94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He
(cherry picked from commit 69fa81679f)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
2017-01-23 17:23:32 +00:00
Sunil G f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang.
(cherry picked from commit 3fa0d540df)
2017-01-23 12:24:39 +05:30
Jian He d374087670 YARN-6104. RegistrySecurity overrides zookeeper sasl system properties. Contributed by Billie Rinaldi 2017-01-19 10:19:24 -08:00
Chris Nauroth 5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer.
(cherry picked from commit b913677365)
2017-01-19 20:48:27 +05:30
Chris Nauroth 198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer.
(cherry picked from commit ef501b1a0b)
2017-01-19 20:48:27 +05:30
Arun Suresh 3c6ec20955 YARN-6110. Fix opportunistic containers documentation. (Akira Ajisaka via asuresh)
(cherry picked from commit 7e8d32147c)
2017-01-19 06:16:30 -08:00
Wangda Tan 8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda)
(cherry picked from commit 72054a817d)
2017-01-18 13:35:30 -08:00
Subru Krishnan fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru).
(cherry picked from commit 4d1f3d9020)
2017-01-17 14:51:27 -08:00
bibinchundatt 580275628b YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are incorrect about behavior when a request is out of bounds. Contributed by Julia Sommer.
(cherry picked from commit a9d946d4df)
2017-01-17 14:18:41 +05:30
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore.
(cherry picked from commit e407449ddb)
2017-01-17 10:22:29 +09:00
Junping Du 861e275646 YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong.
(cherry picked from commit cf695577aa)
2017-01-16 16:41:16 -08:00
Sunil G f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan.
(cherry picked from commit d3170f9eba)
2017-01-13 18:30:31 +05:30
Naganarasimha 4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S.
(cherry picked from commit a6b06f7179)
2017-01-12 06:42:18 +05:30
Daniel Templeton 718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton)
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
2017-01-11 15:36:18 -08:00
Jason Lowe cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du
(cherry picked from commit 357eab9566)
2017-01-11 17:15:14 +00:00
Junping Du ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay.
(cherry picked from commit 4db119b7b5)
2017-01-10 18:22:45 -08:00
Wangda Tan c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha a7cdd39234 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash.
(cherry picked from commit 4c431a6940)
2017-01-10 15:55:46 +05:30
Naganarasimha ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Sunil G 3190a4ba44 YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang.
(cherry picked from commit 9594c35dcb)
2017-01-10 14:10:25 +05:30
Daniel Templeton 74bf9c6e4b YARN-6073. Misuse of format specifier in Preconditions.checkArgument (Contributed by Yuanbo Liu via Daniel Templeton)
(cherry picked from commit 6332a318bc)
2017-01-09 22:14:46 -08:00
Junping Du b600577095 YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe.
(cherry picked from commit 945db55f2e)
2017-01-09 18:15:21 -08:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda)
(cherry picked from commit 287d3d6804)
2017-01-09 08:42:38 -08:00
Varun Saxena 4fab250fc0 YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena)
(cherry picked from commit f59e36b4ce)
2017-01-09 11:31:39 +05:30
Arun Suresh 4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Junping Du f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Jian He ba5d482366 YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S
(cherry picked from commit 020316458d)
2017-01-06 18:27:36 -08:00
Arun Suresh 81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh)
(cherry picked from commit ac1e5d4f77)
2017-01-06 11:25:49 -08:00
Arun Suresh 2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 2273a74c1f)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
2017-01-06 11:15:10 -08:00
Arun Suresh 3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh)
(cherry picked from commit b0aace21b1)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
2017-01-06 11:15:09 -08:00
Arun Suresh 44774eb21c YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh)
(cherry picked from commit 005850b28f)
(cherry picked from commit cbff10b4147f98a89b393519b17e16385294af07)
2017-01-06 11:15:09 -08:00
Arun Suresh 368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh)
(cherry picked from commit 3219b7b4ac)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
2017-01-06 11:15:09 -08:00
Arun Suresh 047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 283fa33feb)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
2017-01-06 11:15:09 -08:00
Arun Suresh 0e60c7cd33 YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. (Konstantinos Karanasos via asuresh)
(cherry picked from commit f38a6d03a1)
(cherry picked from commit 83ccab8a653a4ef943276651c9604bd434675980)
2017-01-06 11:15:09 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
2017-01-06 11:15:09 -08:00
Arun Suresh 28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh)
(cherry picked from commit aa3cab1eb2)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
2017-01-06 11:15:09 -08:00
Arun Suresh f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 10be45986c)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
2017-01-06 11:15:09 -08:00
Sunil G 65e7ae5dcf YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2017-01-06 21:30:52 +05:30
Rohith Sharma K S 3207762192 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S.
(cherry picked from commit e49e0a6e37)
2017-01-04 12:10:05 +05:30
Junping Du dc2e44e8f8 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong.
(cherry picked from commit 8fadd69047)
2017-01-03 15:19:36 -08:00
Wangda Tan 23f6a33156 YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)
(cherry picked from commit e0f2379312)
(cherry picked from commit 554e6656bc)
2017-01-03 14:52:31 -08:00
Wangda Tan cb64e8eb19 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" because forgot to add JIRA Number
This reverts commit 554e6656bc.
2017-01-03 14:51:59 -08:00
Wangda Tan 554e6656bc Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)
(cherry picked from commit e0f2379312)
2017-01-03 14:50:19 -08:00
Daniel Templeton 8cb7aa2b52 YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton)
(cherry picked from commit 165d01a73e)
2016-12-30 08:56:03 -08:00
Robert Kanter f9b48206dd YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter)
(cherry picked from commit f216276d21)
2016-12-28 15:29:06 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Sunil G b3231b480d YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max. Contributed by Wangda Tan.
(cherry picked from commit f6715b26b6)
2016-12-28 16:00:52 +05:30
Rohith Sharma K S cb40e3cffa YARN-6001. Improve moveApplicationQueues command line. Contributed by Sunil G.
(cherry picked from commit e297be74ed)
2016-12-28 12:17:38 +05:30
Wangda Tan 42f571728c YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda)
(cherry picked from commit 0840b4329b)
2016-12-27 21:19:10 -08:00
Robert Kanter 88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter)
(cherry picked from commit 9262797e86)
2016-12-27 14:14:41 -08:00
Robert Kanter 8fbd36c60f YARN-5962. Spelling errors in logging and exceptions for resource manager code (gsohn via rkanter)
(cherry picked from commit 1bbd023275)
2016-12-27 13:35:55 -08:00
Karthik Kambatla 10a154cedd YARN-5969. FairShareComparator: Cache value of getResourceUsage for better performance. (zhangshilong via kasha)
(cherry picked from commit c3973e7080)
2016-12-27 12:30:50 -08:00
Daniel Templeton 28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton)
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
2016-12-27 07:19:37 -08:00
Naganarasimha 4a9f965314 YARN-6026. A couple of spelling errors in the docs. Contributed by Grant Sohn. 2016-12-26 06:50:06 +05:30