Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,127 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3253 Commits

Author SHA1 Message Date
Junping Du 33e7c3044f YARN-6145. Improve log message on fail over. Contributed by Jian He. 
(cherry picked from commit eec52e158b)
 2017-02-08 02:15:11 -08:00
Sidharta S 127ed38d13 YARN-5258. Document Use of Docker with LinuxContainerExecutor (Contributed by Daniel Templeton) 2017-02-07 16:35:56 -08:00
Junping Du ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 
(cherry picked from commit 9dbfab1284)
 2017-02-07 02:14:09 -08:00
Junping Du b7d59e790b HADOOP-12097. Allow port range to be specified while starting webapp. Contributed by Varun Saxena. 
(cherry picked from commit cce35c3815)

Conflicts:

	hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java
 2017-02-05 20:13:49 -08:00
Junping Du c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 
(cherry picked from commit 3e06475307)
 2017-01-31 16:26:41 -08:00
Daniel Templeton ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 
(cherry picked from commit 87852b6ef4)
 2017-01-30 21:46:38 -08:00
Ray Chiang 9f0b813350 YARN-6106. Document FairScheduler 'allowPreemptionFrom' queue property. (Yufei Gu via rchiang) 
(cherry picked from commit 3f970ac363dc2faf4f54180ee5b64d0c60807097)
 2017-01-30 13:42:11 -08:00
Karthik Kambatla c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 
(cherry picked from commit 11e44bdda7)
 2017-01-30 13:39:35 -08:00
Jason Lowe 4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Junping Du 660f4d8631 YARN-6126. Obtaining app logs for Running application fails with json parse error. Contributed by Xuan Gong. 
(cherry picked from commit 1a16431bd0)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/LogsCLI.java
 2017-01-27 06:33:44 -08:00
Karthik Kambatla 1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 
(cherry picked from commit 4d3f73acc0)
 2017-01-26 14:11:50 -08:00
Daniel Templeton c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9)
 2017-01-26 14:08:46 -08:00
Daniel Templeton 443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 
(cherry picked from commit b3befc021b)
 2017-01-26 14:07:46 -08:00
Jason Lowe b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed.
 2017-01-26 14:42:22 +00:00
Sangjin Lee b799ea7641 YARN-3637. Handle localization sym-linking correctly at the YARN level. Contributed by Chris Trezzo. 
(cherry picked from commit 425a7e5028)
 2017-01-25 15:52:05 -08:00
Sangjin Lee c4d3b28c2d YARN-6117. SharedCacheManager does not start up. Contributed by Chris Trezzo. 
(cherry picked from commit dc6ec97048)
 2017-01-25 15:36:07 -08:00
Jason Lowe 45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1)
 2017-01-25 21:54:17 +00:00
Karthik Kambatla 73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 
(cherry picked from commit abedb8a9d8)
 2017-01-25 12:19:27 -08:00
Karthik Kambatla 71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 
(cherry picked from commit e224c96234)
 2017-01-25 12:19:20 -08:00
Karthik Kambatla 0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du 8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Wangda Tan 1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Jason Lowe 1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Rohith Sharma K S 26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 
(cherry picked from commit 0101267d9d)
 2017-01-24 11:01:52 +05:30
Jason Lowe 94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Sunil G f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 
(cherry picked from commit 3fa0d540df)
 2017-01-23 12:24:39 +05:30
Jian He d374087670 YARN-6104. RegistrySecurity overrides zookeeper sasl system properties. Contributed by Billie Rinaldi 2017-01-19 10:19:24 -08:00
Chris Nauroth 5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer. 
(cherry picked from commit b913677365)
 2017-01-19 20:48:27 +05:30
Chris Nauroth 198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer. 
(cherry picked from commit ef501b1a0b)
 2017-01-19 20:48:27 +05:30
Arun Suresh 3c6ec20955 YARN-6110. Fix opportunistic containers documentation. (Akira Ajisaka via asuresh) 
(cherry picked from commit 7e8d32147c)
 2017-01-19 06:16:30 -08:00
Wangda Tan 8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 
(cherry picked from commit 72054a817d)
 2017-01-18 13:35:30 -08:00
Subru Krishnan fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit 4d1f3d9020)
 2017-01-17 14:51:27 -08:00
bibinchundatt 580275628b YARN-6057. yarn.scheduler.minimum-allocation-* descriptions are incorrect about behavior when a request is out of bounds. Contributed by Julia Sommer. 
(cherry picked from commit a9d946d4df)
 2017-01-17 14:18:41 +05:30
Akira Ajisaka 2d6be7ea23 HADOOP-13933. Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit e407449ddb)
 2017-01-17 10:22:29 +09:00
Junping Du 861e275646 YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong. 
(cherry picked from commit cf695577aa)
 2017-01-16 16:41:16 -08:00
Sunil G f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 
(cherry picked from commit d3170f9eba)
 2017-01-13 18:30:31 +05:30
Naganarasimha 4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 
(cherry picked from commit a6b06f7179)
 2017-01-12 06:42:18 +05:30
Daniel Templeton 718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
 2017-01-11 15:36:18 -08:00
Jason Lowe cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 
(cherry picked from commit 357eab9566)
 2017-01-11 17:15:14 +00:00
Junping Du ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b5)
 2017-01-10 18:22:45 -08:00
Wangda Tan c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha a7cdd39234 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash. 
(cherry picked from commit 4c431a6940)
 2017-01-10 15:55:46 +05:30
Naganarasimha ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Sunil G 3190a4ba44 YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 
(cherry picked from commit 9594c35dcb)
 2017-01-10 14:10:25 +05:30
Daniel Templeton 74bf9c6e4b YARN-6073. Misuse of format specifier in Preconditions.checkArgument (Contributed by Yuanbo Liu via Daniel Templeton) 
(cherry picked from commit 6332a318bc)
 2017-01-09 22:14:46 -08:00
Junping Du b600577095 YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 
(cherry picked from commit 945db55f2e)
 2017-01-09 18:15:21 -08:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804)
 2017-01-09 08:42:38 -08:00
Varun Saxena 4fab250fc0 YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena) 
(cherry picked from commit f59e36b4ce)
 2017-01-09 11:31:39 +05:30
Arun Suresh 4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Junping Du f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Jian He ba5d482366 YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 
(cherry picked from commit 020316458d)
 2017-01-06 18:27:36 -08:00
Arun Suresh 81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 
(cherry picked from commit ac1e5d4f77)
 2017-01-06 11:25:49 -08:00
Arun Suresh 2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh 3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit b0aace21b1)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
 2017-01-06 11:15:09 -08:00
Arun Suresh 44774eb21c YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh) 
(cherry picked from commit 005850b28f)
(cherry picked from commit cbff10b4147f98a89b393519b17e16385294af07)
 2017-01-06 11:15:09 -08:00
Arun Suresh 368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh 047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 283fa33feb)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
 2017-01-06 11:15:09 -08:00
Arun Suresh 0e60c7cd33 YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit f38a6d03a1)
(cherry picked from commit 83ccab8a653a4ef943276651c9604bd434675980)
 2017-01-06 11:15:09 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Arun Suresh 28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 
(cherry picked from commit aa3cab1eb2)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
 2017-01-06 11:15:09 -08:00
Arun Suresh f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 10be45986c)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
 2017-01-06 11:15:09 -08:00
Sunil G 65e7ae5dcf YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2017-01-06 21:30:52 +05:30
Rohith Sharma K S 3207762192 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S. 
(cherry picked from commit e49e0a6e37)
 2017-01-04 12:10:05 +05:30
Junping Du dc2e44e8f8 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong. 
(cherry picked from commit 8fadd69047)
 2017-01-03 15:19:36 -08:00
Wangda Tan 23f6a33156 YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
(cherry picked from commit 554e6656bc)
 2017-01-03 14:52:31 -08:00
Wangda Tan cb64e8eb19 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" because forgot to add JIRA Number 
This reverts commit 554e6656bc.
 2017-01-03 14:51:59 -08:00
Wangda Tan 554e6656bc Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
 2017-01-03 14:50:19 -08:00
Daniel Templeton 8cb7aa2b52 YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton) 
(cherry picked from commit 165d01a73e)
 2016-12-30 08:56:03 -08:00
Robert Kanter f9b48206dd YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter) 
(cherry picked from commit f216276d21)
 2016-12-28 15:29:06 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Sunil G b3231b480d YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max. Contributed by Wangda Tan. 
(cherry picked from commit f6715b26b6)
 2016-12-28 16:00:52 +05:30
Rohith Sharma K S cb40e3cffa YARN-6001. Improve moveApplicationQueues command line. Contributed by Sunil G. 
(cherry picked from commit e297be74ed)
 2016-12-28 12:17:38 +05:30
Wangda Tan 42f571728c YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda) 
(cherry picked from commit 0840b4329b)
 2016-12-27 21:19:10 -08:00
Robert Kanter 88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter) 
(cherry picked from commit 9262797e86)
 2016-12-27 14:14:41 -08:00
Robert Kanter 8fbd36c60f YARN-5962. Spelling errors in logging and exceptions for resource manager code (gsohn via rkanter) 
(cherry picked from commit 1bbd023275)
 2016-12-27 13:35:55 -08:00
Karthik Kambatla 10a154cedd YARN-5969. FairShareComparator: Cache value of getResourceUsage for better performance. (zhangshilong via kasha) 
(cherry picked from commit c3973e7080)
 2016-12-27 12:30:50 -08:00
Daniel Templeton 28b605dd1a HADOOP-13940. Document the missing envvars commands (Contributed by Yiqun Lin via Daniel Templeton) 
(cherry picked from commit d432173737a532b7106682116013b751ba162ff0)
 2016-12-27 07:19:37 -08:00
Naganarasimha 4a9f965314 YARN-6026. A couple of spelling errors in the docs. Contributed by Grant Sohn. 2016-12-26 06:50:06 +05:30
Akira Ajisaka 522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
 2016-12-23 04:06:53 +09:00
Varun Saxena 3ba589cd1e YARN-5903. Fix race condition in TestResourceManagerAdministrationProtocolPBClientImpl beforeclass setup method (Haibo Chen via Varun Saxena) 
(cherry picked from commit 38e66d4d64)
 2016-12-22 23:11:18 +05:30
Junping Du cc42dbb703 YARN-4990. Re-direction of a particular log file within in a container in NM UI does not redirect properly. Contributed by Xuan Gong. 
(cherry picked from commit 736f54b727)
 2016-12-21 14:16:19 -08:00
Sunil G 238863c11b YARN-5650. Render Application Timeout value in web UI. Contributed by Akhil PB. 2016-12-20 11:03:57 +05:30
Naganarasimha e9ce2df1f2 YARN-5524. Yarn live log aggregation does not throw if command line arg is wrong. Contributed by Xuan Gong. 
(cherry picked from commit 358341398a)
 2016-12-20 06:45:52 +05:30
Xuan 236dbe3485 YARN-5999. AMRMClientAsync will stop if any exceptions thrown on allocate call. Contributed by Jian He 2016-12-14 15:05:42 -08:00
Sunil G 23bd68a4a4 YARN-5970. Validate application update timeout request parameters. Contributed by Rohith Sharma K S. 2016-12-12 11:59:14 +05:30
Jian He 71fdca4600 YARN-5709. Cleanup leader election configs and pluggability. Contribtued by Karthik Kambatla 
(cherry picked from commit b817c565c8be1d4a682d119bfac6f43ee09e87f0)
 2016-12-09 16:59:52 -08:00
Robert Kanter 9cba7c6d71 YARN-5963. Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code (gsohn via rkanter) 
(cherry picked from commit 72fe546841)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
 2016-12-07 15:09:37 -08:00
Sunil a636a87270 YARN-5965. Retrospect ApplicationReport#getApplicationTimeouts. Contributed by Rohith Sharma K S 
(cherry picked from commit ab923a53fc)
 2016-12-08 00:34:45 +05:30
Daniel Templeton d8b4961515 YARN-5136. Error in handling event type APP_ATTEMPT_REMOVED to the scheduler 
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
 2016-12-07 11:01:09 -08:00
Rohith Sharma K S 602c998443 YARN-5932. Retrospect moveApplicationToQueue in align with YARN-5611. Contributed by Sunil G. 2016-12-07 22:45:25 +05:30
Junping Du 3350d0c080 YARN-5184. Fix up incompatible changes introduced on ContainerStatus and NodeReport. Contributed by Sangjin Lee. 
(cherry picked from commit a7288da595)
 2016-12-06 14:37:46 -08:00
Karthik Kambatla 8773cd5f76 HADOOP-13859. TestConfigurationFieldsBase fails for fields that are DEFAULT values of skipped properties. (Haibo Chen via kasha) 
(cherry picked from commit c8d0a049b0)
 2016-12-06 14:16:38 -08:00
Naganarasimha 54c5880cf7 YARN-5921. Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState. Contributed by Varun Saxena 
(cherry picked from commit f3b8ff54ab)
 2016-12-06 11:01:14 +05:30
Jian He 0f6fbfc0db YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. Contributed by Akira Ajisaka & Wangda Tan 
(cherry picked from commit 43ebff2e35)
 2016-12-05 11:40:26 -08:00
Jian He d58fca0102 YARN-5746. The state of the parentQueue and its childQueues should be synchronized. Contributed by Xuan Gong 2016-12-02 16:18:39 -08:00
Daniel Templeton 1216bad0f7 YARN-5929. Missing scheduling policy in the FS queue metric. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 9b076327eac9da244f94e226816465e7d17fda54)
 2016-12-02 14:02:35 -08:00
Daniel Templeton eb301bb3ec YARN-5901. Fix race condition in TestGetGroups beforeclass setup() (Contributed by Haibo Chen via Daniel Templeton) 2016-12-02 12:05:29 -08:00
Jason Lowe 6b17d449bf YARN-5915. ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write. Contributed by Atul Sikaria 2016-12-02 16:58:43 +00:00
Robert Kanter 99b046f8a9 MAPREDUCE-6787. Allow job_conf.xml to be downloadable on the job overview page in JHS (haibochen via rkanter) 
(cherry picked from commit c87b3a448a)
 2016-12-01 17:30:02 -08:00
Li Lu 8ffe86f780 YARN-5761. Separate QueueManager from Scheduler. (Xuan Gong via gtcarrera9) 
(cherry picked from commit 69fb70c31a)
 2016-11-30 14:02:31 -08:00
Daniel Templeton 3013b028ba YARN-5942. "Overridden" is misspelled as "overriden" in FairScheduler.md 
(Contributed by Heather Sutherland via Daniel Templeton)

(cherry picked from commit 57194889fc39a36dc0697b29eafc4ac07ee42d00)
 2016-11-30 11:24:44 -08:00
Daniel Templeton c22ceb0a9b YARN-5851. TestContainerManagerSecurity testContainerManager[1] failed (Contributed by Haibo Chen via Daniel Templeton) 
(cherry picked from commit dd9a96cb17)
 2016-11-29 14:23:42 -08:00
Daniel Templeton 803aba03f2 YARN-5890. FairScheduler should log information about AM-resource-usage and max-AM-share for queues 
(Contributed by Yufei Gu via Daniel Templeton)
 2016-11-29 14:14:43 -08:00
Daniel Templeton b8bebb8607 YARN-5774. MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
if set yarn.scheduler.minimum-allocation-mb to 0. (Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 25f9872be6)
 2016-11-29 10:12:10 -08:00
Sunil c28dc5e5a4 YARN-4206. Add Application timeouts in Application report and CLI. Contributed by Rohith Sharma K S. 
(cherry picked from commit eb0a483ed0)
 2016-11-24 18:27:50 +05:30
Sunil 7b676c5632 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-24 17:37:36 +05:30
Rohith Sharma K S 6b47a7fb8e YARN-5920. Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang. Contributed by Varun Saxena. 
(cherry picked from commit e15c20edba)
 2016-11-24 12:19:19 +05:30
Rohith Sharma K S 87bb3c51d6 Reverting for fixing compilation errors in branch-2. 
Revert "YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S"

This reverts commit 4379658042.
 2016-11-24 09:18:07 +05:30
Jian He 4379658042 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-23 16:27:12 -08:00
Rohith Sharma K S 2d068f1793 YARN-5865. Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611. Contributed by Sunil G. 2016-11-23 23:29:39 +05:30
Naganarasimha 2d94f938c5 YARN-4330. MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages. Contributed by Varun Saxena 
(cherry picked from commit 3541ed8068)
 2016-11-23 14:14:56 +05:30
Naganarasimha b96ed43897 YARN-5911. DrainDispatcher does not drain all events on stop even if setDrainEventsOnStop is true. Contributed by Varun Saxena. 
(cherry picked from commit 4667564162)
 2016-11-23 09:36:43 +05:30
Daniel Templeton 06f5d27bd6 YARN-5722. FairScheduler hides group resolution exceptions when assigning queue (Contributed by Wilfred Spiegelenburg via Daniel Templeton) 
(cherry picked from commit 83cc7263af)
 2016-11-22 11:17:39 -08:00
Jason Lowe 38024eb8f7 YARN-5859. TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails. Contributed by Eric Badger 
(cherry picked from commit 009452bb6d)
 2016-11-21 16:37:59 +00:00
Wei-Chiu Chuang c5a3ec493b YARN-5271. ATS client doesn't work with Jersey 2 on the classpath. Contributed by Weiwei Yang. 
(cherry picked from commit 09520cb439)
 2016-11-17 22:23:58 -06:00
Subru Krishnan e5a5a5a618 YARN-5904. Reduce the number of default server threads for AMRMProxyService. 
(cherry picked from commit 140b9939da)
 2016-11-17 18:27:17 -08:00
Junping Du a8ad6b6171 YARN-3538. TimelineWebService doesn't catch runtime exception. Contributed by Steve Loughran and Tsuyoshi Ozawa. 
(cherry picked from commit f05a9ceb4a)
 2016-11-17 16:56:21 -08:00
Rohith Sharma K S bbe993238d YARN-5870. Expose getApplications API in YarnClient with GetApplicationsRequest parameter. Contributed by Jian He. 
(cherry picked from commit ce2847e794)
 2016-11-17 08:54:33 +05:30
Jason Lowe 46b7d6233c YARN-5836. Malicious AM can kill containers of other apps running in any node its containers are running. Contributed by Botong Huang 
(cherry picked from commit 59bfcbf357)
 2016-11-16 22:25:15 +00:00
Varun Saxena 6cb1fda99d YARN-5891. yarn rmadmin -help contains a misspelled ResourceManager (Grant Sohn via Varun Saxena) 
(cherry picked from commit 6a11877ab5)
 2016-11-17 03:28:08 +05:30
Varun Saxena 2893316dd0 YARN-5843. Incorrect documentation for timeline service entityType/events REST end points (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit c8bc7a8475)
 2016-11-17 00:52:08 +05:30
Rohith Sharma K S 74ac78b3c0 YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee. 
(cherry picked from commit d65603517e)
 2016-11-16 15:17:46 +05:30
Xiao Chen 3497221fcc YARN-5875. TestTokenClientRMService#testTokenRenewalWrongUser fails. Contributed by Gergely Novák. 
(cherry picked from commit d5c396ccb361e1a0cf4307ce1d1b37fae0eec250)
 2016-11-15 13:58:02 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Naganarasimha 260f3a9dc9 YARN-4355. NPE while processing localizer heartbeat. Contributed by Varun Saxena & Jonathan Hung. 
(cherry picked from commit 7ffb9943b8)
 2016-11-15 15:44:25 +05:30
Naganarasimha 7e9a6b653e Reverting the patch due to the issue raised in YARN-5765 
Revert "YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang."

This reverts commit 93b768d0d1.
 2016-11-15 11:05:03 +05:30
Rohith Sharma K S 528ef2407c YARN-5874. RM -format-state-store and -remove-application-from-state-store commands fail with NPE. Contributed by Varun Saxena. 
(cherry picked from commit b7070f3308)
 2016-11-15 11:03:38 +05:30
Rohith Sharma K S fad6de3080 YARN-5873. RM crashes with NPE if generic application history is enabled. Contributed by Varun Saxena. 
(cherry picked from commit 296c5de0cf)
 2016-11-15 11:03:27 +05:30
Karthik Kambatla ea74c7b7ae YARN-4033. In FairScheduler, parent queues should also display queue status. (Siqi Li via kasha) 
(cherry picked from commit 79448d4ab1)
 2016-11-13 21:39:44 -08:00
Jian He 9b0a2cb321 YARN-5825. ProportionalPreemptionalPolicy should use readLock over LeafQueue instead of synchronized block. Contributed by Sunil G 
(cherry picked from commit fad9609d13)
 2016-11-11 15:16:43 -08:00
Wangda Tan a76e214556 YARN-5552. Add Builder methods for common yarn API records. (Tao Jie via wangda) 
(cherry picked from commit ede1a473f5)
 2016-11-11 13:42:45 -08:00
Naganarasimha bde95f90e7 YARN-5545, Fix issues related to Max App in capacity scheduler. Contributed by Bibin A Chundatt 2016-11-12 00:52:35 +05:30
Eric Payne dd5b9dabf9 YARN-4218. Metric for resource*time that was preempted. Contributed by Chang Li. 2016-11-10 23:02:42 +00:00
Karthik Kambatla f4c8dc4783 YARN-5834. TestNodeStatusUpdater.testNMRMConnectionConf compares nodemanager wait time to the incorrect value. (Chang Li via kasha) 
(cherry picked from commit 3a98419532)
 2016-11-10 14:09:33 -08:00
Karthik Kambatla f7b25420a1 YARN-5453. FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource. (sandflee via kasha) 
(cherry picked from commit 86ac1ad9fd)
 2016-11-09 23:44:26 -08:00
Varun Saxena 06c8ebdb83 YARN-5862. TestDiskFailures.testLocalDirsFailures failed (Yufei Gu via Varun Saxena) 
(cherry picked from commit c202a10923)
 2016-11-10 11:43:57 +05:30
Jian He a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Naganarasimha 8a2998c08c YARN-4498. Application level node labels stats to be available in REST (addendum patch). Contributed by Bibin A Chundatt. 2016-11-10 05:21:07 +05:30
Daniel Templeton 42147bbbf7 YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted app 
is in ACCEPTED state in Fair Scheduler (Contributed by Yufei Gu)

(cherry picked from commit 59ee8b7a88)
 2016-11-09 13:18:28 -08:00
Arun Suresh 4a023b4414 YARN-5833. Addendum patch to include missing changes to yarn-default.xml 
(cherry picked from commit 280357c29f)
 2016-11-09 07:16:39 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa)
 2016-11-08 16:33:58 -08:00
Subru Krishnan 797c7bfab9 YARN-5833. Add validation to ensure default ports are unique in Configuration. (Konstantinos Karanasos via Subru). 
(cherry picked from commit 29e3b3417c)
 2016-11-08 14:39:09 -08:00
Jason Lowe 8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00
Jian He 6cdcab9078 YARN-5716. Add global scheduler interface definition and update CapacityScheduler to use it. Contributed by Wangda Tan 2016-11-07 10:31:43 -08:00
Jason Lowe 27ed2c526a YARN-5837. NPE when getting node status of a decommissioned node after an RM restart. Contributed by Robert Kanter 
(cherry picked from commit 6bb741ff0e)
 2016-11-04 22:21:45 +00:00
Sunil 3eed75a9ab YARN-5802. updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy. Contributred by Bibin A Chundatt 
(cherry picked from commit 19b3779ae7)
 2016-11-04 17:09:19 +05:30
Varun Vasudev 7676b89f43 YARN-5822. Log ContainerRuntime initialization error in LinuxContainerExecutor. Contributed by Sidharta Seethana. 
(cherry picked from commit 9ee0e3172e)
 2016-11-03 22:34:38 +05:30
Jason Lowe e2917180ee YARN-4862. Handle duplicate completed containers in RMNodeImpl. Contributed by Rohith Sharma K S 
(cherry picked from commit 352cbaa7a5)
 2016-11-03 14:03:56 +00:00
Naganarasimha 0c7caba087 YARN-5720. Update document for "rmadmin -replaceLabelOnNode". Contributred by Tao Jie 
(cherry picked from commit 0e75496049)
 2016-11-03 12:50:38 +05:30
Varun Saxena b818773919 YARN-5815. Random failure of TestApplicationPriority.testOrderOfActivatingThePriorityApplicationOnRMRestart (Bibin A Chundatt via Varun Saxena) 2016-11-03 00:41:53 +05:30
Naganarasimha 2612b509ef YARN-5697. Use CliParser to parse options in RMAdminCLI. Contributed by Tao jie 
(cherry picked from commit aacf214a27)
 2016-11-02 04:46:24 +05:30
Jason Lowe f8d653ef34 YARN-5001. Aggregated Logs root directory is created with wrong group if nonexistent. Contributed by Haibo Chen 
(cherry picked from commit 76893a4100)
 2016-11-01 20:24:22 +00:00
Varun Saxena 91ddea5bfb YARN-5788. Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:26:38 +05:30
Varun Saxena 2f0a101ab9 YARN-5805. Add isDebugEnabled check for debug logs in nodemanager (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:23:12 +05:30
Karthik Kambatla 2226ae4924 YARN-4998. Minor cleanup to UGI use in AdminService. (Daniel Templeton via kasha) 
(cherry picked from commit 733aa99313)
 2016-10-31 16:26:15 -07:00
Varun Saxena 92b7491f23 YARN-5809. AsyncDispatcher possibly invokes multiple shutdown threads when handling exception (Jian He via Varun Saxena) 2016-11-01 04:37:56 +05:30
Wangda Tan cef281abe6 YARN-2009. CapacityScheduler: Add intra-queue preemption for app priority support. (Sunil G via wangda) 
(cherry picked from commit 90dd3a8148)
 2016-10-31 15:23:16 -07:00
Daniel Templeton 9d13a13160 YARN-5793. Trim configuration values in DockerLinuxContainerRuntime (Contributed by Tianyin Xu via Daniel Templeton) 
(cherry picked from commit f3eb4c3c73)
 2016-10-31 14:01:15 -07:00
Daniel Templeton ce13463e7a YARN-4396. Log the trace information on FSAppAttempt#assignContainer (Contributed by Yiqun Li via Daniel Templeton) 
(cherry picked from commit 2528bea67f)
 2016-10-31 13:36:24 -07:00
Daniel Templeton adceebc400 YARN-4907. Make all MockRM#waitForState consistent. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 155f06e0c1a5a15365fb620f3802dca8aa46c287)
 2016-10-31 13:21:45 -07:00
Naganarasimha e4023f0d81 YARN-4498. Application level node labels stats to be available in REST. Contributed by Bibin A Chundatt. 2016-10-31 19:07:56 +05:30
Varun Saxena 0e6fcd3bd2 YARN-5773. RM recovery too slow due to LeafQueue#activateApplications (Bibin A Chundatt via Varun Saxena) 2016-10-29 13:51:02 +05:30
Jason Lowe b85b5c6b94 YARN-4963. capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable. Contributed by Nathan Roberts 
(cherry picked from commit 1eae719bce)
 2016-10-28 17:31:29 +00:00
Jason Lowe ea9a1be109 YARN-5767. Fix the order that resources are cleaned up from the local Public/Private caches. Contributed by Chris Trezzo 
(cherry picked from commit 1b79c417dc)
 2016-10-28 15:59:57 +00:00
Jason Lowe 2ab80a148a YARN-5027. NM should clean up app log dirs after NM restart. Contributed by sandflee 
(cherry picked from commit 7146359bfd)
 2016-10-28 15:49:54 +00:00
Jason Lowe 0f224d405f HADOOP-13770. Shell.checkIsBashSupported swallowed an interrupted exception. Contributed by Wei-Chiu Chuang 
(cherry picked from commit c017171da0)
 2016-10-28 15:06:23 +00:00
Naganarasimha 62d8536df6 Revert "YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin" 
This reverts commit b92ad5d857.
 2016-10-28 12:58:42 +05:30
Naganarasimha b92ad5d857 YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin 
(cherry picked from commit 3de130252d)
 2016-10-28 12:44:14 +05:30
Karthik Kambatla 950bfed1d3 YARN-4743. FairSharePolicy breaks TimSort assumption. (Zephyr Guo and Yufei Gu via kasha) 
(cherry picked from commit 4df8ed63ed)
 2016-10-27 17:48:24 -07:00
Subru Krishnan 334fd9e83f YARN-2306. Add test for leakage of reservation metrics in fair scheduler. (Hong Zhiguo and Yufei Gu via subru). 
(cherry picked from commit b2c4f24c31)
 2016-10-27 17:44:05 -07:00
Li Lu 2b0fd1f4e2 YARN-4668. Reuse objectMapper instance in Yarn. (Yiqun Lin via gtcarrera9) 
(cherry picked from commit 221582c4ab)
 2016-10-27 15:26:05 -07:00
Carlo Curino 58ac40b552 YARN-3432. Cluster metrics have wrong Total Memory when there is reserved memory on CS. (Brahma Reddy Battula via curino) 
(cherry picked from commit 892a8348fc)
 2016-10-27 15:22:04 -07:00
Subru Krishnan 4274600b95 YARN-3568. TestAMRMTokens should use some random port. (Takashi Ohnishi via Subru). 
(cherry picked from commit 79ae78dcbe)
 2016-10-27 15:13:01 -07:00
Varun Saxena f572448809 YARN-4388. Cleanup mapreduce.job.hdfs-servers from yarn-default.xml (Junping Du via Varun Saxena) 2016-10-28 02:24:24 +05:30
Varun Saxena c34bc3d661 YARN-5308. FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling (Kai Sasaki via Varun Saxena) 2016-10-28 02:23:25 +05:30
Jason Lowe e50215306d YARN-4831. Recovered containers will be killed after NM stateful restart. Contributed by Siqi Li 
(cherry picked from commit 7e3c327d31)
 2016-10-27 20:44:17 +00:00
Jason Lowe 8935b23706 YARN-5172. Update yarn daemonlog documentation due to HADOOP-12847. Contributed by Wei-Chiu Chuang 
(cherry picked from commit b4a8fbcbbc)
 2016-10-27 19:45:00 +00:00
Robert Kanter b7f7d42c27 YARN-4456. Clean up Lint warnings in nodemanager (templedf via rkanter) 
(cherry picked from commit 6fbfb501f2)
 2016-10-27 12:37:20 -07:00
Naganarasimha 15f9c18ae4 YARN-5420. Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary. Contributed by Sunil G. 
(cherry picked from commit b3c15e4ef7)
 2016-10-27 18:23:27 +05:30
Varun Saxena 65fa9827fc YARN-5757. RM REST API documentation is not up to date (Miklos Szegedi via Varun Saxena) 2016-10-27 17:24:24 +05:30
Naganarasimha 3ff80a800d YARN-3848. TestNodeLabelContainerAllocation is timing out. Contributed by Varun Saxena 
(cherry picked from commit 6c8830992c)
 2016-10-27 17:14:34 +05:30
Varun Saxena 65c009ca00 YARN-5686. DefaultContainerExecutor random working dir algorigthm skews results (Vrushali C via Varun Saxena) 2016-10-27 14:03:02 +05:30
Varun Saxena a8cfa5054c YARN-5710. Fix inconsistent naming in class ResourceRequest (Yufei Gu via Varun Saxena) 2016-10-27 13:56:59 +05:30
Varun Saxena daf9a656b9 YARN-5752. TestLocalResourcesTrackerImpl#testLocalResourceCache times out (Eric Badger via Varun Saxena) 2016-10-27 13:46:13 +05:30
Rohith Sharma K S 3a167669a5 YARN-4363. In TestFairScheduler, testcase should not create FairScheduler redundantly. Conntributed by Tao Jie. 
(cherry picked from commit e29cba61a0)
 2016-10-27 11:59:10 +05:30
Rohith Sharma K S 07b531c10a YARN-4555. TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment. Contributed by Takashi Ohnishi. 
(cherry picked from commit b110c4b5e8)
 2016-10-27 11:35:38 +05:30
Akira Ajisaka e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a667)
 2016-10-26 15:34:40 +09:00
Karthik Kambatla e4ba92abee YARN-5388. Deprecate and remove DockerContainerExecutor. (Daniel Templeton via kasha) 2016-10-25 15:26:27 -07:00
Karthik Kambatla aedd5c4c1b YARN-5677. RM should transition to standby when connection is lost for an extended period. (Daniel Templeton via kasha) 2016-10-25 13:15:58 -07:00
Karthik Kambatla ab9105d814 YARN-5754. Null check missing for earliest in FifoPolicy. (Yufei Gu via kasha) 
(cherry picked from commit a71fc81655)
 2016-10-24 22:19:56 -07:00
Subru Krishnan 81595a127f YARN-5711. Propogate exceptions back to client when using hedging RM failover provider. 
(cherry picked from commit 0a166b1347)
 2016-10-24 19:02:03 -07:00
Jian He bb8214cab8 Revert YARN-4126. RM should not issue delegation tokens in unsecure mode. 2016-10-23 15:56:22 -07:00
Karthik Kambatla 440a6326d1 YARN-5047. Refactor nodeUpdate across schedulers. (Ray Chiang via kasha) 
(cherry picked from commit 754cb4e30f)
 2016-10-20 21:18:13 -07:00
Karthik Kambatla c2cabce2e8 YARN-4911. Bad placement policy in FairScheduler causes the RM to crash 
(cherry picked from commit a064865abf)
 2016-10-20 20:57:31 -07:00
Robert Kanter e02c756f16 YARN-5466. DefaultContainerExecutor needs JavaDocs (templedf via rkanter) 
(cherry picked from commit f5d9235914)
 2016-10-17 14:29:42 -07:00
Varun Saxena e9a58691ab YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun Saxena) 2016-10-14 23:32:29 +05:30
Chris Douglas f0f2681b4f YARN-5717. Add tests for container-executor is_feature_enabled. Contributed by Sidharta Seethana 
(cherry picked from commit cf3f43e95b)
 2016-10-13 20:49:50 -07:00
Jason Lowe 28bd08798d YARN-5551. Ignore file backed pages from memory computation when smaps is enabled. Contributed by Rajesh Balamohan 
(cherry picked from commit ecb51b857a)
 2016-10-11 15:14:08 +00:00
Naganarasimha dc5f7a9d5e YARN-5057. Resourcemanager.security.TestDelegationTokenRenewer fails in trunk. Contributed by Jason Lowe. 
(cherry picked from commit 0773ffd0f8)
 2016-10-10 18:06:28 -04:00