Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,678 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2567 Commits

Author SHA1 Message Date
Junping Du f254002f1d YARN-6314. Potential infinite redirection on YARN log redirection web service. Contributed by Xuan Gong. 
(cherry picked from commit 5a9dda796f0e73060ada794ad5752cc6a237ab2e)
 2017-03-14 02:58:44 -07:00
Jason Lowe ef95e74eb1 YARN-6321. TestResources test timeouts are too aggressive. Contributed by Eric Badger 
(cherry picked from commit 9649c27864)
 2017-03-10 13:07:20 -06:00
Karthik Kambatla 4218671f3c YARN-6264. AM not launched when a single vcore is available on the cluster. (Yufei Gu via kasha) 
(cherry picked from commit a96afae125)
 2017-03-09 23:16:19 -08:00
Daniel Templeton c60cd88ad1 YARN-1047. Expose # of pre-emptions as a queue counter (Contributed by Karthik Kambatla via Daniel Templeton) 2017-03-09 18:18:03 -08:00
Daniel Templeton 6774f9c3c2 YARN-6300. NULL_UPDATE_REQUESTS is redundant in TestFairScheduler (Contributed by Yuanbo Liu via Daniel Templeton) 
(cherry picked from commit 8bc8804be1b96f8a4fc33b5d24dd7f7af93a437e)
 2017-03-09 17:53:24 -08:00
Sidharta S 3b6e5ef917 YARN-5669. Add support for docker pull command (Contribtued by luhuichun) 2017-03-09 16:23:21 -08:00
Jason Lowe 4a1187238a YARN-6165. Intra-queue preemption occurs even when preemption is turned off for a specific queue. Contributed by Eric Payne 
(cherry picked from commit d7762a5511)
 2017-03-08 16:47:31 -06:00
Robert Kanter 11d4cdc3a6 YARN-6297. TestAppLogAggregatorImp.verifyFilesUploaded() should check # of filed uploaded with that of files expected (haibochen via rkanter) 
(cherry picked from commit 287ba4ffa6)
 2017-03-08 10:45:59 -08:00
Sunil G ec4c2d42fe YARN-6207. Move application across queues should handle delayed event processing. Contributed by Bibin A Chundatt. 
(cherry picked from commit 1eb8186703)
 2017-03-08 12:13:14 +05:30
Daniel Templeton 83fbb051bb YARN-6287. RMCriticalThreadUncaughtExceptionHandler.rmContext should be final (Contributed by Corey Barker via Daniel Templeton) 
(cherry picked from commit e0c239cdbd)
 2017-03-07 12:01:38 -08:00
Junping Du a58dfcb421 YARN-6271. yarn rmadin -getGroups returns information from standby RM. Contributed by Jian He. 
(cherry picked from commit 8db7a8c3ae)
 2017-03-03 14:40:32 -08:00
Ray Chiang 303ee13e3c YARN-6218. Fix TestAMRMClient when using FairScheduler. (Miklos Szegedi via rchiang) 
(cherry picked from commit 30b91ff9540e35195af834d1bd5983114a556c6d)
 2017-03-03 12:57:17 -08:00
Jason Lowe 5eca427da2 YARN-6263. NMTokenSecretManagerInRM.createAndGetNMToken is not thread safe. Contributed by Haibo Chen 
(cherry picked from commit e58fc76030)
 2017-03-03 08:24:05 -06:00
Jian He c7ddf95d4a YARN-6153. KeepContainer does not work when AM retry window is set. Contributed by kyungwan nam 
(cherry picked from commit 235203dffd)
 2017-02-28 13:26:55 -08:00
Xuan 95bd3c3d55 YARN-6189: Improve application status log message when RM restarted when 
app is in NEW state. Contributed by Junping Du

(cherry picked from commit e0bb867c3f)
 2017-02-28 11:06:07 -08:00
Karthik Kambatla 58eefc71c7 YARN-6231. FairSchedulerTestBase helper methods should call scheduler.update to avoid flakiness. (kasha) 
(cherry picked from commit f187d63816)
 2017-02-27 22:16:32 -08:00
Naganarasimha 1eec911cd9 YARN-5703. ReservationAgents are not correctly configured. Contributed by Manikandan R. 
(cherry picked from commit 5f5b031d1f)
 2017-02-27 20:42:10 +05:30
Sunil G 68b08e96a0 YARN-6123. [YARN-5864] Add a test to make sure queues of orderingPolicy will be updated when childQueues is added or removed. Contributed by Wangda Tan. 2017-02-27 17:54:47 +05:30
Karthik Kambatla c24325bbcb YARN-6172. FSLeafQueue demand update needs to be atomic. (Miklos Szegedi via kasha) 
(cherry picked from commit fbfe86deea)
 2017-02-26 20:37:18 -08:00
Karthik Kambatla a95d3e1fe7 YARN-6215. FairScheduler preemption and update should not run concurrently. (Tao Jie via kasha) 
(cherry picked from commit 815d53506f)
 2017-02-26 20:31:03 -08:00
Xuan 19c5b3d622 YARN-6228: EntityGroupFSTimelineStore should allow configurable cache 
stores. Contributed by Li Lu

(cherry picked from commit 53d372a255)
 2017-02-24 11:00:16 -08:00
Wangda Tan 650ff95e00 YARN-5959. RM changes to support change of container ExecutionType. (Arun Suresh via wangda) 
(cherry picked from commit 0a55bd841e)
 2017-02-23 17:22:25 -08:00
Karthik Kambatla 40bc9e7ddb YARN-6222. TestFairScheduler.testReservationMetrics is flaky. (Yufei Gu via kasha) 
(cherry picked from commit 694e680d20)
 2017-02-23 15:30:29 -08:00
Wangda Tan a82480c7d4 YARN-6143. Fix incompatible issue caused by YARN-3583. (Sunil G via wangda) 2017-02-23 13:59:20 -08:00
Sunil G 801357726d YARN-6211. Synchronization improvement for moveApplicationAcrossQueues and updateApplicationPriority. Contributed by Bibin A Chundatt. 
(cherry picked from commit a207aa9930)
 2017-02-23 14:25:14 +05:30
Karthik Kambatla 37edbd35f1 YARN-6194. Cluster capacity in SchedulingPolicy is updated only on allocation file reload. (Yufei Gu via kasha) 
(cherry picked from commit b10e962224)
 2017-02-22 15:59:11 -08:00
Karthik Kambatla ca7a6a7365 YARN-6210. FairScheduler: Node reservations can interfere with preemption. (kasha) 
(cherry picked from commit 718ad9f6ee)
 2017-02-22 15:52:44 -08:00
Daniel Templeton 93219b6454 YARN-6125. The application attempt's diagnostic message should have a maximum size 
(Contributed by Andras Piros via Daniel Templeton)

(cherry picked from commit c7a36e6130)
 2017-02-17 14:38:06 -08:00
Karthik Kambatla 5575328bb6 YARN-6193. FairScheduler might not trigger preemption when using DRF. (kasha) 
(cherry picked from commit dbbfcf74ab)
 2017-02-17 14:07:47 -08:00
Daniel Templeton 10fd5627bb YARN-6188. Fix OOM issue with decommissioningNodesWatcher in the case of clusters with 
large number of nodes (Contributed by Ajay Jadhav via Daniel Templeton)

(cherry picked from commit 585168cc1bdb87398d4f388f3eeccd1c71114982)
 2017-02-17 13:14:37 -08:00
Karthik Kambatla afc8124ff6 YARN-6171. ConcurrentModificationException on FSAppAttempt.containersToPreempt. (Miklos Szegedi via kasha) 
(cherry picked from commit a77f432449)
 2017-02-16 15:03:35 -08:00
Rohith Sharma K S f07b52dc5a YARN-6200. Reverting since the same functionality achieved by YARN-1623. 
Revert "YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks)"

This reverts commit a37b3694ea.
 2017-02-16 17:58:38 +05:30
Karthik Kambatla 9fecd5063f YARN-5798. Set UncaughtExceptionHandler for all FairScheduler threads. (Yufei Gu via kasha) 
(cherry picked from commit 74dd142250)
 2017-02-16 00:03:26 -08:00
Karthik Kambatla fefac1276a YARN-4212. FairScheduler: Can't create a DRF queue under a FAIR policy queue. (Yufei Gu via kasha) 
(cherry picked from commit 11be3f70e0)
 2017-02-15 23:51:51 -08:00
Karthik Kambatla 46b6c95e0a YARN-6163. FS Preemption is a trickle for severely starved applications. (kasha) 
(cherry picked from commit 6c25dbcdc0)
 2017-02-15 23:23:18 -08:00
Junping Du 2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong. 
(cherry picked from commit ce2d5bfa5f)
 2017-02-15 09:08:54 -08:00
Varun Saxena 9cab25a63d YARN-6156. AM blacklisting to consider node label partition (Bibin A Chundatt via Varun Saxena) 
(cherry picked from commit b7613e0f40)
 2017-02-15 14:58:39 +05:30
Karthik Kambatla 58acb676a2 YARN-6061. Addendum. Remove extraneous change. 
(cherry picked from commit 353a9b2d91)
 2017-02-14 15:20:11 -08:00
Karthik Kambatla 37921b3fef YARN-6061. Add an UncaughtExceptionHandler for critical threads in RM. (Yufei Gu via kasha) 
(cherry picked from commit 652679aa8a)
 2017-02-14 13:40:06 -08:00
Karthik Kambatla 30ff5bff1a YARN-3933. FairScheduler: Multiple calls to completedContainer are not safe. (Shiwei Guo and Miklos Szegedi via kasha) 
(cherry picked from commit 646c6d6509)
 2017-02-13 11:27:02 -08:00
Junping Du 205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong. 
(cherry picked from commit 464ff479ce)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
 2017-02-13 06:16:04 -08:00
Karthik Kambatla e259a818ad YARN-6144. FairScheduler: preempted resources can become negative. (Miklos Szegedi via kasha) 
(cherry picked from commit 08f93978f3)
 2017-02-09 14:45:21 -08:00
Karthik Kambatla 41361ec39e YARN-6112. UpdateCallDuration is calculated only when debug logging is enabled. (Yufei Gu via kasha) 2017-02-09 13:12:21 -08:00
Junping Du ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 
(cherry picked from commit 9dbfab1284)
 2017-02-07 02:14:09 -08:00
Junping Du c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Daniel Templeton be14bda5c0 YARN-4658. Typo in o.a.h.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler comment (Contributed by Udai Potluri via Daniel Templeton) 
(cherry picked from commit 3e06475307)
 2017-01-31 16:26:41 -08:00
Daniel Templeton ccf33bc36c YARN-6103. Log updates for ZKRMStateStore (Contributed by Daniel Sturman via Daniel Templeton) 
(cherry picked from commit 87852b6ef4)
 2017-01-30 21:46:38 -08:00
Karthik Kambatla c9824c5ff6 YARN-6131. FairScheduler: Lower update interval for faster tests. (kasha) 
(cherry picked from commit 11e44bdda7)
 2017-01-30 13:39:35 -08:00
Jason Lowe 4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Karthik Kambatla 1b1a25cca6 YARN-6000. Make AllocationFileLoaderService.Listener public. (Tao Jie via kasha) 
(cherry picked from commit 4d3f73acc0)
 2017-01-26 14:11:50 -08:00
Daniel Templeton c4eeac02de YARN-4975. Fair Scheduler: exception thrown when a parent queue marked 'parent' has configured child queues 
(Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit f85b74ccf9)
 2017-01-26 14:08:46 -08:00
Daniel Templeton 443f2803ac YARN-4997. Update fair scheduler to use pluggable auth provider (Contributed by Tao Jie via Daniel Templeton) 
(cherry picked from commit b3befc021b)
 2017-01-26 14:07:46 -08:00
Jason Lowe b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed.
 2017-01-26 14:42:22 +00:00
Jason Lowe 45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1)
 2017-01-25 21:54:17 +00:00
Karthik Kambatla 73497f08fc YARN-5830. FairScheduler: Avoid preempting AM containers. (Yufei Gu via kasha) 
(cherry picked from commit abedb8a9d8)
 2017-01-25 12:19:27 -08:00
Karthik Kambatla 71026cc3f6 YARN-5831. FairScheduler: Propagate allowPreemptionFrom flag all the way down to the app. (Yufei Gu via kasha) 
(cherry picked from commit e224c96234)
 2017-01-25 12:19:20 -08:00
Karthik Kambatla 0ceb7149d9 YARN-4752. Improved preemption in FairScheduler. (kasha) 2017-01-25 10:45:02 -08:00
Junping Du 8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Wangda Tan 1309accd68 YARN-5864. Capacity Scheduler - Queue Priorities. (wangda) 2017-01-24 14:44:42 -08:00
Jason Lowe 1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Rohith Sharma K S 26c4cfb653 YARN-6082. Invalid REST api response for getApps since queueUsagePercentage is coming as INF. Contributed by Sunil G. 
(cherry picked from commit 0101267d9d)
 2017-01-24 11:01:52 +05:30
Jason Lowe 94b326f2c8 YARN-5910. Support for multi-cluster delegation tokens. Contributed by Jian He 
(cherry picked from commit 69fa81679f)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAppManager.java
 2017-01-23 17:23:32 +00:00
Sunil G f8e49bb343 YARN-6031. Application recovery has failed when node label feature is turned off during RM recovery. Contributed by Ying Zhang. 
(cherry picked from commit 3fa0d540df)
 2017-01-23 12:24:39 +05:30
Chris Nauroth 5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer. 
(cherry picked from commit b913677365)
 2017-01-19 20:48:27 +05:30
Chris Nauroth 198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer. 
(cherry picked from commit ef501b1a0b)
 2017-01-19 20:48:27 +05:30
Wangda Tan 8d6fa4569b YARN-5556. CapacityScheduler: Support deleting queues without requiring a RM restart. (Naganarasimha G R via wangda) 
(cherry picked from commit 72054a817d)
 2017-01-18 13:35:30 -08:00
Subru Krishnan fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit 4d1f3d9020)
 2017-01-17 14:51:27 -08:00
Junping Du 861e275646 YARN-6011. Add a new web service to list the files on a container in AHSWebService. Contributed by Xuan Gong. 
(cherry picked from commit cf695577aa)
 2017-01-16 16:41:16 -08:00
Sunil G f166bb8f09 YARN-6081. LeafQueue#getTotalPendingResourcesConsideringUserLimit should deduct reserved from pending to avoid unnecessary preemption of reserved container. Contributed by Wangda Tan. 
(cherry picked from commit d3170f9eba)
 2017-01-13 18:30:31 +05:30
Naganarasimha 4bd05840fc YARN-6072. RM unable to start in secure mode. Contributed by Ajith S. 
(cherry picked from commit a6b06f7179)
 2017-01-12 06:42:18 +05:30
Daniel Templeton 718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
 2017-01-11 15:36:18 -08:00
Jason Lowe cc09b1a49a YARN-5416. TestRMRestart#testRMRestartWaitForPreviousAMToFinish failed intermittently due to not wait SchedulerApplicationAttempt to be stopped. Contributed by Junping Du 
(cherry picked from commit 357eab9566)
 2017-01-11 17:15:14 +00:00
Junping Du ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b5)
 2017-01-10 18:22:45 -08:00
Wangda Tan c00a468b0b YARN-6022. Revert changes of AbstractResourceRequest. (Contributed by Wangda Tan / Daniel Templeton) 2017-01-10 10:17:33 -08:00
Naganarasimha a7cdd39234 YARN-6054. TimelineServer fails to start when some LevelDb state files are missing. Contributed by Ravi Prakash. 
(cherry picked from commit 4c431a6940)
 2017-01-10 15:55:46 +05:30
Naganarasimha ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Sunil G 3190a4ba44 YARN-5899. Debug log in AbstractCSQueue#canAssignToThisQueue needs improvement. Contributed by Ying Zhang. 
(cherry picked from commit 9594c35dcb)
 2017-01-10 14:10:25 +05:30
Junping Du b600577095 YARN-4148. When killing app, RM releases app's resource before they are released by NM. Contributed by Jason Lowe. 
(cherry picked from commit 945db55f2e)
 2017-01-09 18:15:21 -08:00
Wangda Tan 447f71326b YARN-3955. Support for application priority ACLs in queues of CapacityScheduler. (Sunil G via wangda) 
(cherry picked from commit 287d3d6804)
 2017-01-09 08:42:38 -08:00
Varun Saxena 4fab250fc0 YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena) 
(cherry picked from commit f59e36b4ce)
 2017-01-09 11:31:39 +05:30
Arun Suresh 4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Junping Du f615c1fbbc Addendum patch to YARN-3866: fix incompatible API change. Contributed by Wangda Tan. 2017-01-07 19:58:42 -08:00
Jian He ba5d482366 YARN-6009. Skip validating app timeout value on recovery. Contributed by Rohith Sharma K S 
(cherry picked from commit 020316458d)
 2017-01-06 18:27:36 -08:00
Arun Suresh 81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 
(cherry picked from commit ac1e5d4f77)
 2017-01-06 11:25:49 -08:00
Arun Suresh 2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh 3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit b0aace21b1)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
 2017-01-06 11:15:09 -08:00
Arun Suresh 44774eb21c YARN-5918. Handle Opportunistic scheduling allocate request failure when NM is lost. (Bibin A Chundatt via asuresh) 
(cherry picked from commit 005850b28f)
(cherry picked from commit cbff10b4147f98a89b393519b17e16385294af07)
 2017-01-06 11:15:09 -08:00
Arun Suresh 368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh 047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 283fa33feb)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
 2017-01-06 11:15:09 -08:00
Arun Suresh 0e60c7cd33 YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit f38a6d03a1)
(cherry picked from commit 83ccab8a653a4ef943276651c9604bd434675980)
 2017-01-06 11:15:09 -08:00
Arun Suresh 7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d4)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Arun Suresh 28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 
(cherry picked from commit aa3cab1eb2)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
 2017-01-06 11:15:09 -08:00
Arun Suresh f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 10be45986c)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
 2017-01-06 11:15:09 -08:00
Sunil G 65e7ae5dcf YARN-5906. Update AppSchedulingInfo to use SchedulingPlacementSet. Contributed by Wangda Tan. 2017-01-06 21:30:52 +05:30
Rohith Sharma K S 3207762192 YARN-5988. RM unable to start in secure setup. Contributed by Ajith S. 
(cherry picked from commit e49e0a6e37)
 2017-01-04 12:10:05 +05:30
Junping Du dc2e44e8f8 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong. 
(cherry picked from commit 8fadd69047)
 2017-01-03 15:19:36 -08:00
Wangda Tan 23f6a33156 YARN-6025. Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
(cherry picked from commit 554e6656bc)
 2017-01-03 14:52:31 -08:00
Wangda Tan cb64e8eb19 Revert "Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda)" because forgot to add JIRA Number 
This reverts commit 554e6656bc.
 2017-01-03 14:51:59 -08:00
Wangda Tan 554e6656bc Fix synchronization issues of AbstractYarnScheduler#nodeUpdate and its implementations. (Naganarasimha G R via wangda) 
(cherry picked from commit e0f2379312)
 2017-01-03 14:50:19 -08:00
Daniel Templeton 8cb7aa2b52 YARN-5931. Document timeout interfaces CLI and REST APIs (Contributed by Rohith Sharma K S via Daniel Templeton) 
(cherry picked from commit 165d01a73e)
 2016-12-30 08:56:03 -08:00
Robert Kanter f9b48206dd YARN-4882. Change the log level to DEBUG for recovering completed applications (templedf via rkanter) 
(cherry picked from commit f216276d21)
 2016-12-28 15:29:06 -08:00
Daniel Templeton 47f0a0e15b HADOOP-13932. Fix indefinite article in comments (Contributed by LiXin Ge via Daniel Templeton) 2016-12-28 07:03:51 -08:00
Sunil G b3231b480d YARN-6024. Capacity Scheduler 'continuous reservation looking' doesn't work when sum of queue's used and reserved resources is equal to max. Contributed by Wangda Tan. 
(cherry picked from commit f6715b26b6)
 2016-12-28 16:00:52 +05:30
Wangda Tan 42f571728c YARN-5756. Add state-machine implementation for scheduler queues. (Xuan Gong via wangda) 
(cherry picked from commit 0840b4329b)
 2016-12-27 21:19:10 -08:00
Robert Kanter 88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter) 
(cherry picked from commit 9262797e86)
 2016-12-27 14:14:41 -08:00
Robert Kanter 8fbd36c60f YARN-5962. Spelling errors in logging and exceptions for resource manager code (gsohn via rkanter) 
(cherry picked from commit 1bbd023275)
 2016-12-27 13:35:55 -08:00
Karthik Kambatla 10a154cedd YARN-5969. FairShareComparator: Cache value of getResourceUsage for better performance. (zhangshilong via kasha) 
(cherry picked from commit c3973e7080)
 2016-12-27 12:30:50 -08:00
Akira Ajisaka 522bc98026 YARN-4994. Use MiniYARNCluster with try-with-resources in tests. Contributed by Andras Bokor. 
(cherry picked from commit ae401539ea)
 2016-12-23 04:06:53 +09:00
Junping Du cc42dbb703 YARN-4990. Re-direction of a particular log file within in a container in NM UI does not redirect properly. Contributed by Xuan Gong. 
(cherry picked from commit 736f54b727)
 2016-12-21 14:16:19 -08:00
Sunil G 238863c11b YARN-5650. Render Application Timeout value in web UI. Contributed by Akhil PB. 2016-12-20 11:03:57 +05:30
Sunil G 23bd68a4a4 YARN-5970. Validate application update timeout request parameters. Contributed by Rohith Sharma K S. 2016-12-12 11:59:14 +05:30
Jian He 71fdca4600 YARN-5709. Cleanup leader election configs and pluggability. Contribtued by Karthik Kambatla 
(cherry picked from commit b817c565c8be1d4a682d119bfac6f43ee09e87f0)
 2016-12-09 16:59:52 -08:00
Robert Kanter 9cba7c6d71 YARN-5963. Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code (gsohn via rkanter) 
(cherry picked from commit 72fe546841)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
 2016-12-07 15:09:37 -08:00
Sunil a636a87270 YARN-5965. Retrospect ApplicationReport#getApplicationTimeouts. Contributed by Rohith Sharma K S 
(cherry picked from commit ab923a53fc)
 2016-12-08 00:34:45 +05:30
Daniel Templeton d8b4961515 YARN-5136. Error in handling event type APP_ATTEMPT_REMOVED to the scheduler 
(Contributed by Wilfred Spiegelenburg via Daniel Templeton)
 2016-12-07 11:01:09 -08:00
Rohith Sharma K S 602c998443 YARN-5932. Retrospect moveApplicationToQueue in align with YARN-5611. Contributed by Sunil G. 2016-12-07 22:45:25 +05:30
Naganarasimha 54c5880cf7 YARN-5921. Incorrect synchronization in RMContextImpl#setHAServiceState/getHAServiceState. Contributed by Varun Saxena 
(cherry picked from commit f3b8ff54ab)
 2016-12-06 11:01:14 +05:30
Jian He 0f6fbfc0db YARN-5559. Analyse 2.8.0/3.0.0 jdiff reports and fix any issues. Contributed by Akira Ajisaka & Wangda Tan 
(cherry picked from commit 43ebff2e35)
 2016-12-05 11:40:26 -08:00
Jian He d58fca0102 YARN-5746. The state of the parentQueue and its childQueues should be synchronized. Contributed by Xuan Gong 2016-12-02 16:18:39 -08:00
Daniel Templeton 1216bad0f7 YARN-5929. Missing scheduling policy in the FS queue metric. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 9b076327eac9da244f94e226816465e7d17fda54)
 2016-12-02 14:02:35 -08:00
Li Lu 8ffe86f780 YARN-5761. Separate QueueManager from Scheduler. (Xuan Gong via gtcarrera9) 
(cherry picked from commit 69fb70c31a)
 2016-11-30 14:02:31 -08:00
Daniel Templeton c22ceb0a9b YARN-5851. TestContainerManagerSecurity testContainerManager[1] failed (Contributed by Haibo Chen via Daniel Templeton) 
(cherry picked from commit dd9a96cb17)
 2016-11-29 14:23:42 -08:00
Daniel Templeton 803aba03f2 YARN-5890. FairScheduler should log information about AM-resource-usage and max-AM-share for queues 
(Contributed by Yufei Gu via Daniel Templeton)
 2016-11-29 14:14:43 -08:00
Daniel Templeton b8bebb8607 YARN-5774. MR Job stuck in ACCEPTED status without any progress in Fair Scheduler 
if set yarn.scheduler.minimum-allocation-mb to 0. (Contributed by Yufei Gu via Daniel Templeton)

(cherry picked from commit 25f9872be6)
 2016-11-29 10:12:10 -08:00
Sunil c28dc5e5a4 YARN-4206. Add Application timeouts in Application report and CLI. Contributed by Rohith Sharma K S. 
(cherry picked from commit eb0a483ed0)
 2016-11-24 18:27:50 +05:30
Sunil 7b676c5632 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-24 17:37:36 +05:30
Rohith Sharma K S 6b47a7fb8e YARN-5920. Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang. Contributed by Varun Saxena. 
(cherry picked from commit e15c20edba)
 2016-11-24 12:19:19 +05:30
Rohith Sharma K S 87bb3c51d6 Reverting for fixing compilation errors in branch-2. 
Revert "YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S"

This reverts commit 4379658042.
 2016-11-24 09:18:07 +05:30
Jian He 4379658042 YARN-5649. Add REST endpoints for updating application timeouts. Contributed by Rohith Sharma K S 2016-11-23 16:27:12 -08:00
Rohith Sharma K S 2d068f1793 YARN-5865. Retrospect updateApplicationPriority api to handle state store exception in align with YARN-5611. Contributed by Sunil G. 2016-11-23 23:29:39 +05:30
Naganarasimha 2d94f938c5 YARN-4330. MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages. Contributed by Varun Saxena 
(cherry picked from commit 3541ed8068)
 2016-11-23 14:14:56 +05:30
Daniel Templeton 06f5d27bd6 YARN-5722. FairScheduler hides group resolution exceptions when assigning queue (Contributed by Wilfred Spiegelenburg via Daniel Templeton) 
(cherry picked from commit 83cc7263af)
 2016-11-22 11:17:39 -08:00
Jason Lowe 38024eb8f7 YARN-5859. TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails. Contributed by Eric Badger 
(cherry picked from commit 009452bb6d)
 2016-11-21 16:37:59 +00:00
Junping Du a8ad6b6171 YARN-3538. TimelineWebService doesn't catch runtime exception. Contributed by Steve Loughran and Tsuyoshi Ozawa. 
(cherry picked from commit f05a9ceb4a)
 2016-11-17 16:56:21 -08:00
Jason Lowe 46b7d6233c YARN-5836. Malicious AM can kill containers of other apps running in any node its containers are running. Contributed by Botong Huang 
(cherry picked from commit 59bfcbf357)
 2016-11-16 22:25:15 +00:00
Rohith Sharma K S 74ac78b3c0 YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee. 
(cherry picked from commit d65603517e)
 2016-11-16 15:17:46 +05:30
Xiao Chen 3497221fcc YARN-5875. TestTokenClientRMService#testTokenRenewalWrongUser fails. Contributed by Gergely Novák. 
(cherry picked from commit d5c396ccb361e1a0cf4307ce1d1b37fae0eec250)
 2016-11-15 13:58:02 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Naganarasimha 260f3a9dc9 YARN-4355. NPE while processing localizer heartbeat. Contributed by Varun Saxena & Jonathan Hung. 
(cherry picked from commit 7ffb9943b8)
 2016-11-15 15:44:25 +05:30
Naganarasimha 7e9a6b653e Reverting the patch due to the issue raised in YARN-5765 
Revert "YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang."

This reverts commit 93b768d0d1.
 2016-11-15 11:05:03 +05:30
Rohith Sharma K S 528ef2407c YARN-5874. RM -format-state-store and -remove-application-from-state-store commands fail with NPE. Contributed by Varun Saxena. 
(cherry picked from commit b7070f3308)
 2016-11-15 11:03:38 +05:30
Rohith Sharma K S fad6de3080 YARN-5873. RM crashes with NPE if generic application history is enabled. Contributed by Varun Saxena. 
(cherry picked from commit 296c5de0cf)
 2016-11-15 11:03:27 +05:30
Karthik Kambatla ea74c7b7ae YARN-4033. In FairScheduler, parent queues should also display queue status. (Siqi Li via kasha) 
(cherry picked from commit 79448d4ab1)
 2016-11-13 21:39:44 -08:00
Jian He 9b0a2cb321 YARN-5825. ProportionalPreemptionalPolicy should use readLock over LeafQueue instead of synchronized block. Contributed by Sunil G 
(cherry picked from commit fad9609d13)
 2016-11-11 15:16:43 -08:00
Wangda Tan a76e214556 YARN-5552. Add Builder methods for common yarn API records. (Tao Jie via wangda) 
(cherry picked from commit ede1a473f5)
 2016-11-11 13:42:45 -08:00
Naganarasimha bde95f90e7 YARN-5545, Fix issues related to Max App in capacity scheduler. Contributed by Bibin A Chundatt 2016-11-12 00:52:35 +05:30
Eric Payne dd5b9dabf9 YARN-4218. Metric for resource*time that was preempted. Contributed by Chang Li. 2016-11-10 23:02:42 +00:00
Karthik Kambatla f4c8dc4783 YARN-5834. TestNodeStatusUpdater.testNMRMConnectionConf compares nodemanager wait time to the incorrect value. (Chang Li via kasha) 
(cherry picked from commit 3a98419532)
 2016-11-10 14:09:33 -08:00
Karthik Kambatla f7b25420a1 YARN-5453. FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource. (sandflee via kasha) 
(cherry picked from commit 86ac1ad9fd)
 2016-11-09 23:44:26 -08:00
Varun Saxena 06c8ebdb83 YARN-5862. TestDiskFailures.testLocalDirsFailures failed (Yufei Gu via Varun Saxena) 
(cherry picked from commit c202a10923)
 2016-11-10 11:43:57 +05:30
Jian He a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Naganarasimha 8a2998c08c YARN-4498. Application level node labels stats to be available in REST (addendum patch). Contributed by Bibin A Chundatt. 2016-11-10 05:21:07 +05:30
Daniel Templeton 42147bbbf7 YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted app 
is in ACCEPTED state in Fair Scheduler (Contributed by Yufei Gu)

(cherry picked from commit 59ee8b7a88)
 2016-11-09 13:18:28 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa)
 2016-11-08 16:33:58 -08:00
Jason Lowe 8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00
Jian He 6cdcab9078 YARN-5716. Add global scheduler interface definition and update CapacityScheduler to use it. Contributed by Wangda Tan 2016-11-07 10:31:43 -08:00
Jason Lowe 27ed2c526a YARN-5837. NPE when getting node status of a decommissioned node after an RM restart. Contributed by Robert Kanter 
(cherry picked from commit 6bb741ff0e)
 2016-11-04 22:21:45 +00:00
Sunil 3eed75a9ab YARN-5802. updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy. Contributred by Bibin A Chundatt 
(cherry picked from commit 19b3779ae7)
 2016-11-04 17:09:19 +05:30
Varun Vasudev 7676b89f43 YARN-5822. Log ContainerRuntime initialization error in LinuxContainerExecutor. Contributed by Sidharta Seethana. 
(cherry picked from commit 9ee0e3172e)
 2016-11-03 22:34:38 +05:30
Jason Lowe e2917180ee YARN-4862. Handle duplicate completed containers in RMNodeImpl. Contributed by Rohith Sharma K S 
(cherry picked from commit 352cbaa7a5)
 2016-11-03 14:03:56 +00:00
Varun Saxena b818773919 YARN-5815. Random failure of TestApplicationPriority.testOrderOfActivatingThePriorityApplicationOnRMRestart (Bibin A Chundatt via Varun Saxena) 2016-11-03 00:41:53 +05:30
Jason Lowe f8d653ef34 YARN-5001. Aggregated Logs root directory is created with wrong group if nonexistent. Contributed by Haibo Chen 
(cherry picked from commit 76893a4100)
 2016-11-01 20:24:22 +00:00
Varun Saxena 91ddea5bfb YARN-5788. Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:26:38 +05:30
Varun Saxena 2f0a101ab9 YARN-5805. Add isDebugEnabled check for debug logs in nodemanager (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:23:12 +05:30
Karthik Kambatla 2226ae4924 YARN-4998. Minor cleanup to UGI use in AdminService. (Daniel Templeton via kasha) 
(cherry picked from commit 733aa99313)
 2016-10-31 16:26:15 -07:00
Wangda Tan cef281abe6 YARN-2009. CapacityScheduler: Add intra-queue preemption for app priority support. (Sunil G via wangda) 
(cherry picked from commit 90dd3a8148)
 2016-10-31 15:23:16 -07:00
Daniel Templeton 9d13a13160 YARN-5793. Trim configuration values in DockerLinuxContainerRuntime (Contributed by Tianyin Xu via Daniel Templeton) 
(cherry picked from commit f3eb4c3c73)
 2016-10-31 14:01:15 -07:00
Daniel Templeton ce13463e7a YARN-4396. Log the trace information on FSAppAttempt#assignContainer (Contributed by Yiqun Li via Daniel Templeton) 
(cherry picked from commit 2528bea67f)
 2016-10-31 13:36:24 -07:00
Daniel Templeton adceebc400 YARN-4907. Make all MockRM#waitForState consistent. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 155f06e0c1a5a15365fb620f3802dca8aa46c287)
 2016-10-31 13:21:45 -07:00
Naganarasimha e4023f0d81 YARN-4498. Application level node labels stats to be available in REST. Contributed by Bibin A Chundatt. 2016-10-31 19:07:56 +05:30
Varun Saxena 0e6fcd3bd2 YARN-5773. RM recovery too slow due to LeafQueue#activateApplications (Bibin A Chundatt via Varun Saxena) 2016-10-29 13:51:02 +05:30
Jason Lowe b85b5c6b94 YARN-4963. capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable. Contributed by Nathan Roberts 
(cherry picked from commit 1eae719bce)
 2016-10-28 17:31:29 +00:00
Jason Lowe ea9a1be109 YARN-5767. Fix the order that resources are cleaned up from the local Public/Private caches. Contributed by Chris Trezzo 
(cherry picked from commit 1b79c417dc)
 2016-10-28 15:59:57 +00:00
Jason Lowe 2ab80a148a YARN-5027. NM should clean up app log dirs after NM restart. Contributed by sandflee 
(cherry picked from commit 7146359bfd)
 2016-10-28 15:49:54 +00:00
Jason Lowe 0f224d405f HADOOP-13770. Shell.checkIsBashSupported swallowed an interrupted exception. Contributed by Wei-Chiu Chuang 
(cherry picked from commit c017171da0)
 2016-10-28 15:06:23 +00:00
Karthik Kambatla 950bfed1d3 YARN-4743. FairSharePolicy breaks TimSort assumption. (Zephyr Guo and Yufei Gu via kasha) 
(cherry picked from commit 4df8ed63ed)
 2016-10-27 17:48:24 -07:00
Subru Krishnan 334fd9e83f YARN-2306. Add test for leakage of reservation metrics in fair scheduler. (Hong Zhiguo and Yufei Gu via subru). 
(cherry picked from commit b2c4f24c31)
 2016-10-27 17:44:05 -07:00
Carlo Curino 58ac40b552 YARN-3432. Cluster metrics have wrong Total Memory when there is reserved memory on CS. (Brahma Reddy Battula via curino) 
(cherry picked from commit 892a8348fc)
 2016-10-27 15:22:04 -07:00
Subru Krishnan 4274600b95 YARN-3568. TestAMRMTokens should use some random port. (Takashi Ohnishi via Subru). 
(cherry picked from commit 79ae78dcbe)
 2016-10-27 15:13:01 -07:00
Varun Saxena c34bc3d661 YARN-5308. FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling (Kai Sasaki via Varun Saxena) 2016-10-28 02:23:25 +05:30
Jason Lowe e50215306d YARN-4831. Recovered containers will be killed after NM stateful restart. Contributed by Siqi Li 
(cherry picked from commit 7e3c327d31)
 2016-10-27 20:44:17 +00:00
Robert Kanter b7f7d42c27 YARN-4456. Clean up Lint warnings in nodemanager (templedf via rkanter) 
(cherry picked from commit 6fbfb501f2)
 2016-10-27 12:37:20 -07:00
Naganarasimha 15f9c18ae4 YARN-5420. Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary. Contributed by Sunil G. 
(cherry picked from commit b3c15e4ef7)
 2016-10-27 18:23:27 +05:30
Naganarasimha 3ff80a800d YARN-3848. TestNodeLabelContainerAllocation is timing out. Contributed by Varun Saxena 
(cherry picked from commit 6c8830992c)
 2016-10-27 17:14:34 +05:30
Varun Saxena 65c009ca00 YARN-5686. DefaultContainerExecutor random working dir algorigthm skews results (Vrushali C via Varun Saxena) 2016-10-27 14:03:02 +05:30
Varun Saxena daf9a656b9 YARN-5752. TestLocalResourcesTrackerImpl#testLocalResourceCache times out (Eric Badger via Varun Saxena) 2016-10-27 13:46:13 +05:30
Rohith Sharma K S 3a167669a5 YARN-4363. In TestFairScheduler, testcase should not create FairScheduler redundantly. Conntributed by Tao Jie. 
(cherry picked from commit e29cba61a0)
 2016-10-27 11:59:10 +05:30
Rohith Sharma K S 07b531c10a YARN-4555. TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment. Contributed by Takashi Ohnishi. 
(cherry picked from commit b110c4b5e8)
 2016-10-27 11:35:38 +05:30
Akira Ajisaka e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a667)
 2016-10-26 15:34:40 +09:00
Karthik Kambatla e4ba92abee YARN-5388. Deprecate and remove DockerContainerExecutor. (Daniel Templeton via kasha) 2016-10-25 15:26:27 -07:00
Karthik Kambatla aedd5c4c1b YARN-5677. RM should transition to standby when connection is lost for an extended period. (Daniel Templeton via kasha) 2016-10-25 13:15:58 -07:00
Karthik Kambatla ab9105d814 YARN-5754. Null check missing for earliest in FifoPolicy. (Yufei Gu via kasha) 
(cherry picked from commit a71fc81655)
 2016-10-24 22:19:56 -07:00
Jian He bb8214cab8 Revert YARN-4126. RM should not issue delegation tokens in unsecure mode. 2016-10-23 15:56:22 -07:00
Karthik Kambatla 440a6326d1 YARN-5047. Refactor nodeUpdate across schedulers. (Ray Chiang via kasha) 
(cherry picked from commit 754cb4e30f)
 2016-10-20 21:18:13 -07:00
Karthik Kambatla c2cabce2e8 YARN-4911. Bad placement policy in FairScheduler causes the RM to crash 
(cherry picked from commit a064865abf)
 2016-10-20 20:57:31 -07:00
Robert Kanter e02c756f16 YARN-5466. DefaultContainerExecutor needs JavaDocs (templedf via rkanter) 
(cherry picked from commit f5d9235914)
 2016-10-17 14:29:42 -07:00
Varun Saxena e9a58691ab YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun Saxena) 2016-10-14 23:32:29 +05:30
Chris Douglas f0f2681b4f YARN-5717. Add tests for container-executor is_feature_enabled. Contributed by Sidharta Seethana 
(cherry picked from commit cf3f43e95b)
 2016-10-13 20:49:50 -07:00
Naganarasimha dc5f7a9d5e YARN-5057. Resourcemanager.security.TestDelegationTokenRenewer fails in trunk. Contributed by Jason Lowe. 
(cherry picked from commit 0773ffd0f8)
 2016-10-10 18:06:28 -04:00
Rohith Sharma K S 94a6f65989 YARN-5101. YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order. Contributed by Sunil G. 
(cherry picked from commit 4d2f380d78)
 2016-10-06 20:43:47 +05:30
Jian He caafa980af YARN-3139. Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler. Contributed by Wangda Tan 2016-10-06 07:55:14 -07:00
Akira Ajisaka 5f1432d98e MAPREDUCE-6789. Fix TestAMWebApp failure. Contributed by Daniel Templeton. 
(cherry picked from commit 272a21747e)
 2016-10-06 15:59:19 +09:00
Varun Vasudev 3ce02387ea YARN-5704. Provide config knobs to control enabling/disabling new/work in progress features in container-executor. Contributed by Sidharta Seethana. 
(cherry picked from commit 0992708d79)
 2016-10-05 15:32:50 +05:30
Karthik Kambatla 1578781fe9 YARN-4767. Network issues can cause persistent RM UI outage. (Daniel Templeton via kasha) 
(cherry picked from commit 736d33cddd)
 2016-10-03 14:36:14 -07:00
Karthik Kambatla f8c2c7b2c0 YARN-5672. FairScheduler: Wrong queue name in log when adding application. (Wilfred Spiegelenburg via kasha) 
(cherry picked from commit 0da54e8848)
 2016-10-03 06:04:09 -07:00
Naganarasimha 04f594dbea YARN-4855. Should check if node exists when replace nodelabels. Contributeed by Tao Jie 
(cherry picked from commit 6e130c308c)
 2016-10-03 02:13:46 -04:00
Karthik Kambatla 57726eed83 YARN-5678. Log demand as demand in FSLeafQueue and FSParentQueue. (Yufei Gu via kasha) 
(cherry picked from commit 82857037b6)
 2016-10-02 22:10:07 -07:00
Subru Krishnan 9f43f89899 YARN-5384. Expose priority in ReservationSystem submission APIs. (Sean Po via Subru). 
(cherry picked from commit 3a3697deab)
 2016-09-30 19:43:29 -07:00
Li Lu 919c0802fa YARN-4205. Addendum patch to fix branch-2. 2016-09-29 15:41:31 -07:00
Jian He 39f896e1a1 YARN-4205. Add a service for monitoring application life time out. Contributed by Rohith Sharma K S 
(cherry picked from commit 2ae5a3a5bf)
 2016-09-29 22:06:36 +08:00
Robert Kanter 4f8c7f819c YARN-5400. addendum patch to fix JDK 7 issue 2016-09-28 16:01:18 -07:00
Robert Kanter bc91e33d5e YARN-5400. Light cleanup in ZKRMStateStore (templedf via rkanter) 
(cherry picked from commit bcb2528a51)
 2016-09-28 14:57:14 -07:00
Varun Vasudev 3f1d3e3177 YARN-5662. Provide an option to enable ContainerMonitor. Contributed by Jian He. 
(cherry picked from commit bc2656f09f)
 2016-09-28 15:27:09 +05:30
Wangda Tan 379c2b3e5b YARN-3142. Improve locks in AppSchedulingInfo. (Varun Saxena via wangda) 
(cherry picked from commit 1831be8e73)
 2016-09-27 12:05:00 -07:00
Naganarasimha 86ec99e82d YARN-5660. Wrong audit constants are used in Get/Put of priority in RMWebService. Contributed by Rohith Sharma K S 
(cherry picked from commit ebf528c443)
 2016-09-27 16:42:21 +05:30
Arun Suresh 7484d0b1b9 YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh 2016-09-26 23:55:13 +08:00
Akira Ajisaka 74f2df16a9 YARN-5663. Small refactor in ZKRMStateStore. Contributed by Oleksii Dymytrov. 
(cherry picked from commit 14a696f369)
 2016-09-26 15:01:13 +09:00
Naganarasimha 0e4a5611ff YARN-3692. Allow REST API to set a user generated message when killing an application. Contributed by Rohith Sharma K S 2016-09-24 21:13:01 +05:30
Arun Suresh 2a9f809c96 YARN-5656. Fix ReservationACLsTestBase. (Sean Po via asuresh) 
(cherry picked from commit 9f03b403ec)
 2016-09-20 12:27:50 -07:00
Wangda Tan 679652fcbd Addendum patch for fix javadocs failure which is caused by YARN-3141. (wangda) 
(cherry picked from commit e45307c9a0)
 2016-09-20 11:21:47 -07:00
Jason Lowe 7231000149 YARN-5655. TestContainerManagerSecurity#testNMTokens is asserting. Contributed by Robert Kanter 
(cherry picked from commit c6d1d742e7)
 2016-09-20 14:16:47 +00:00
Jian He 3acd30df71 YARN-3140. Improve locks in AbstractCSQueue/LeafQueue/ParentQueue. Contributed by Wangda Tan 
(cherry picked from commit 2b66d9ec5b)
 2016-09-20 15:05:54 +08:00
Jason Lowe 035f5f8f1d YARN-5540. Scheduler spends too much time looking at empty priorities. Contributed by Jason Lowe 
(cherry picked from commit 7558dbbb48)
 2016-09-19 20:33:59 +00:00
Jian He 9942ca2bf0 YARN-3141. Improve locks in SchedulerApplicationAttempt/FSAppAttempt/FiCaSchedulerApp. Contributed by Wangda Tan 
(cherry picked from commit b8a30f2f17)
 2016-09-19 17:08:57 +08:00
Arun Suresh dfaac56433 YARN-5637. Changes in NodeManager to support Container rollback and commit. (asuresh) 
(cherry picked from commit 3552c2b99d)
 2016-09-18 11:06:38 -07:00
Arun Suresh 00759f50be YARN-5657. Fix TestDefaultContainerExecutor. (asuresh) 
(cherry picked from commit f67237cbe7)
 2016-09-17 09:32:55 -07:00
Naganarasimha 20729ead91 YARN-5642. Typos in 9 log messages. Contributed by Mehran Hassani 2016-09-17 10:37:42 +05:30
Wangda Tan ee101e49fa YARN-4091. Add REST API to retrieve scheduler activity. (Chen Ge via wangda) 2016-09-15 14:49:03 -07:00
Arun Suresh 16aecc699d YARN-5620. Core changes in NodeManager to support re-initialization of Containers with new launchContext. (asuresh) 
(cherry picked from commit 40b5a59b72)
 2016-09-15 08:11:14 -07:00
Jason Lowe 4f2f0eb0bb YARN-5630. NM fails to start after downgrade from 2.8 to 2.7. Contributed by Jason Lowe 
(cherry picked from commit e793309735)
 2016-09-13 14:43:21 +00:00
Karthik Kambatla a064375e68 YARN-5616. Clean up WeightAdjuster. (Yufei Gu via kasha) 
(cherry picked from commit c0e492e50f)
 2016-09-06 22:40:49 -07:00
Varun Vasudev ec3a651b17 YARN-5576. Allow resource localization while container is running. Contributed by Jian He. 2016-09-06 20:07:54 +05:30
Karthik Kambatla a5748eec7b YARN-5264. Store all queue-specific information in FSQueue. (Yufei Gu via kasha) 
(cherry picked from commit f6ea9be547)
 2016-09-02 15:02:20 -07:00
Ray Chiang 4ee1729cb0 YARN-5549. AMLauncher#createAMContainerLaunchContext() should not log the command to be launched indiscriminately. (Daniel Templeton via rchiang) 2016-09-02 14:57:05 -07:00
Varun Vasudev 1222433729 YARN-5555. Scheduler UI: "% of Queue" is inaccurate if leaf queue is hierarchically nested. Contributed by Eric Payne. 
(cherry picked from commit 05f5c0f631)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java
 2016-09-02 16:16:31 +05:30
Karthik Kambatla 736dd1646d YARN-5566. Client-side NM graceful decom is not triggered when jobs finish. (Robert Kanter via kasha) 
(cherry picked from commit 74f4bae455)
 2016-09-01 14:44:41 -07:00
Varun Vasudev 34f9330651 YARN-5596. Fix failing unit test in TestDockerContainerRuntime. Contributed by Sidharta Seethana. 
(cherry picked from commit 08f55ccbb0)
 2016-09-01 14:10:18 +05:30
Arun Suresh b279f42d79 YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 
(cherry picked from commit d6d9cff21b)
 2016-08-30 16:33:31 -07:00
Subru Krishnan 3ad7cb3e1c YARN-5327. API changes required to support recurring reservations in the YARN ReservationSystem. (Sangeetha Abdu Jyothi via Subru). 
(cherry picked from commit b930dc3ec0)
 2016-08-26 17:01:37 -07:00
Karthik Kambatla d9b42248de YARN-5373. NPE listing wildcard directory in containerLaunch. (Daniel Templeton via kasha) 
(cherry picked from commit cde3a00526)
 2016-08-26 11:08:16 -07:00
Junping Du af1502fcab YARN-5557. Add localize API to the ContainerManagementProtocol. Contributed by Jian He. 
(cherry picked from commit 9ef632f3b0)
 2016-08-26 09:05:33 -07:00
Naganarasimha 026027166e YARN-3940. Application moveToQueue should check NodeLabel permission. Contributed by Bibin A Chundatt 
(cherry picked from commit 46e02ab719)
 2016-08-26 20:23:01 +05:30
Varun Vasudev 49a97a76f5 YARN-5430. Return container's ip and host from NM ContainerStatus call. Contributed by Jian He. 2016-08-26 16:33:21 +05:30
Naganarasimha ee3358402a YARN-5564. Fix typo in RM_SCHEDULER_RESERVATION_THRESHOLD_INCREMENT_MULTIPLE. Contributed by Ray Chiang 
(cherry picked from commit 27c3b86252)
 2016-08-26 08:53:42 +05:30
Varun Vasudev 952c309dd9 YARN-5042. Mount /sys/fs/cgroup into Docker containers as read only mount. Contributed by luhuichun. 
(cherry picked from commit 42d9876471ff0bf0ea240cd48fe483cda8aa1ec7)
 2016-08-25 14:18:12 +05:30
Arun Suresh bd348d20b2 Revert "YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh)" 
This reverts commit 307cda70db.
 2016-08-24 10:23:06 -07:00
Rohith Sharma K S a19859c3f5 YARN-5544. TestNodeBlacklistingOnAMFailures fails on trunk. Contributed by Sunil G. 
(cherry picked from commit 0d5997d2b9)
 2016-08-23 14:39:19 +05:30
Wangda Tan c7d782d2f6 YARN-3388. Allocation in LeafQueue could get stuck because DRF calculator isn't well supported when computing user-limit. (Nathan Roberts via wangda) 
(cherry picked from commit 444b2ea7af)
 2016-08-19 16:47:03 -07:00
Varun Saxena 928cccb9f9 YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena) 2016-08-19 17:32:23 +05:30
Varun Saxena a746775e9a Revert "YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena)" 
This reverts commit 613048a8a2.
 2016-08-19 16:40:48 +05:30
Varun Saxena 613048a8a2 YARN-5533. JMX AM Used metrics for queue wrong when app submited to nodelabel partition (Bibin A Chundatt via Varun Saxena) 2016-08-19 15:18:42 +05:30
Junping Du d464483bf7 YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 2016-08-18 07:23:29 -07:00
Karthik Kambatla 07d5ab16df YARN-4702. FairScheduler: Allow setting maxResources for ad hoc queues. (Daniel Templeton via kasha) 
(cherry picked from commit 20f0eb871c)
 2016-08-17 17:54:28 -07:00
Varun Vasudev 3ee1fb774e YARN-5455. Update Javadocs for LinuxContainerExecutor. Contributed by Daniel Templeton. 
(cherry picked from commit 7f05ff7a4e)
 2016-08-17 15:35:23 +05:30
Varun Saxena 354afbb9df YARN-5521. Fix random failure of TestCapacityScheduler#testKillAllAppsInQueue (sandflee via Varun Saxena) 2016-08-16 00:04:40 +05:30
Varun Saxena 3c7cf5cd77 YARN-5491. Fix random failure of TestCapacityScheduler#testCSQueueBlocked (Bibin A Chundatt via Varun Saxena) 2016-08-15 03:56:31 +05:30
Varun Saxena 918a0f12fa YARN-5476. Non existent application reported as ACCEPTED by YarnClientImpl (Junping Du via Varun Saxena) 2016-08-13 00:19:00 +05:30
Naganarasimha b63d3dcac9 YARN-4833. For Queue AccessControlException client retries multiple times on both RM. Contributed by Bibin A Chundatt 2016-08-12 20:16:15 +05:30
Rohith Sharma K S 0937715f7f YARN-5492. TestSubmitApplicationWithRMHA is failing sporadically during precommit builds. Contributed by Vrushali C. 
(cherry picked from commit 5199db387d)
 2016-08-11 11:51:56 +05:30
Ray Chiang 372f6f8456 YARN-5137. Make DiskChecker pluggable in NodeManager. (Yufei Gu via rchiang) 
(cherry picked from commit dbe9e70cc084220ea1f68da850cdb092281b5e96)
 2016-08-10 19:26:49 -07:00
Jason Lowe 2b31f6fb2c YARN-5382. RM does not audit log kill request for active applications. Contributed by Vrushali C 
(cherry picked from commit 5c95bb315b)
 2016-08-10 18:27:25 +00:00
Jason Lowe 3d401206cf YARN-5483. Optimize RMAppAttempt#pullJustFinishedContainers. Contributed by sandflee 2016-08-10 18:06:56 +00:00
Karthik Kambatla 447ba47640 YARN-5343. TestContinuousScheduling#testSortedNodes fails intermittently. (Yufei Gu via kasha) 
(cherry picked from commit 7992c0b42c)
 2016-08-09 16:51:27 -07:00
Varun Vasudev c599f6215e YARN-5394. Remove bind-mount /etc/passwd for Docker containers. Contributed by Zhankun Tang. 
(cherry picked from commit 522ddbde79)
 2016-08-09 16:05:03 +05:30
Arun Suresh 5f7edb79d1 YARN-5457. Refactor DistributedScheduling framework to pull out common functionality. (asuresh) 
(cherry picked from commit 82c9e06101)
 2016-08-09 00:46:08 -07:00
Naganarasimha 93b768d0d1 YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang. 2016-08-09 06:48:39 +05:30
Varun Saxena ce8df272f4 YARN-4910. Fix incomplete log info in ResourceLocalizationService (Jun Gong via Varun Saxena) 2016-08-08 22:14:56 +05:30
Naganarasimha 991823482f YARN-4624. NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI. Contributed by Brahma Reddy Battula 
(cherry picked from commit d81b8163b4)
 2016-08-06 07:04:30 +05:30
Wangda Tan a73e118437 YARN-4888. Changes in scheduler to identify resource-requests explicitly by allocation-id. (Subru Krishnan via wangda) 
(cherry picked from commit 3f100d76ff)
 2016-08-05 10:44:07 -07:00
Rohith Sharma K S 69da77c357 YARN-5333. Some recovered apps are put into default queue when RM HA. Contributed by Jun Gong. 
(cherry picked from commit d9a354c2f3)
 2016-08-05 21:37:05 +05:30
Varun Vasudev 19d894538d YARN-5459. Add support for docker rm. Contributed by Shane Kumpf. 
(cherry picked from commit 08e33381f5)
 2016-08-04 14:18:01 +05:30
Jason Lowe 65a91c8527 YARN-5469. Increase timeout of TestAmFilter.testFilter. Contributed by Eric Badger 
(cherry picked from commit db4a61dc61)
 2016-08-03 19:53:28 +00:00
Jason Lowe 1ea03411c2 YARN-5462. TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently. Contributed by Eric Badger 
(cherry picked from commit db646540f0)
 2016-08-03 19:18:50 +00:00
Jason Lowe 9a0ac56a5c YARN-4280. CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster. Contributed by Kuhu Shukla 
(cherry picked from commit 4d92aefd35)
 2016-08-03 19:00:06 +00:00
Varun Vasudev 1087745d79 YARN-5460. Change container runtime type logging in DelegatingLinuxContainerRuntime to debug. Contributed by Shane Kumpf. 
(cherry picked from commit d848184e90)
 2016-08-03 12:03:31 +05:30
Varun Vasudev b63e6c8027 YARN-5458. Rename DockerStopCommandTest to TestDockerStopCommand. Contributed by Shane Kumpf. 
(cherry picked from commit 7fc70c6422)
 2016-08-02 17:02:17 +05:30
Varun Vasudev 4e5409a2bb YARN-5443. Add support for docker inspect command. Contributed by Shane Kumpf. 
(cherry picked from commit 2e7c2a13a8)
 2016-08-02 17:01:59 +05:30
Varun Vasudev 7ecb1635b4 YARN-5444. Fix failing unit tests in TestLinuxContainerExecutorWithMocks. Contributed by Yufei Gu. 
(cherry picked from commit 4455aa7c86a36b6e017e288479a7d4ced889faa0)
 2016-08-01 16:01:10 +05:30
Arun Suresh f197378f81 YARN-5113. Refactoring and other clean-up for distributed scheduling. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit e5766b1dbe)
 2016-07-31 11:49:09 -07:00
Subru Krishnan 2f4df2b851 YARN-5203.Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API. Contributed by Ellen Hui. 
(cherry picked from commit 4e756d7271)
 2016-07-28 16:04:30 -07:00
Junping Du 932bf79244 YARN-5432. Lock already held by another process while LevelDB cache store creation for dag. Contributed by Li Lu. 
(cherry picked from commit 7f3c306e2e)
 2016-07-28 06:36:31 -07:00
Subru Krishnan 4ecfd05095 YARN-5441. Fixing minor Scheduler test case failures 
(cherry picked from commit d2cbfd7de3)
 2016-07-27 18:26:29 -07:00
Wangda Tan 77e0b6d1bf YARN-5195. RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. (sandflee via wangda) 
(cherry picked from commit d62e121ffc)
 2016-07-26 21:36:37 -07:00
Wangda Tan 55fa19d576 YARN-5342. Improve non-exclusive node partition resource allocation in Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit 49969b16cd)
 2016-07-26 21:16:12 -07:00
Arun Suresh df509a8066 YARN-5392. Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerRequestKey. (asuresh and subru) 
(cherry picked from commit 5aace38b74)
 2016-07-26 14:58:54 -07:00
Rohith Sharma K S eff65463de YARN-5092. TestRMDelegationTokens fails intermittently. Contributed by Jason Lowe. 
(cherry picked from commit 557a245d83)
 2016-07-21 12:49:06 +05:30
Akira Ajisaka 2bf5c4eb1a YARN-4883. Make consistent operation name in AdminService. Contributed by Kai Sasaki. 
(cherry picked from commit c63afdbe14)
 2016-07-20 16:51:44 -07:00
Vinod Kumar Vavilapalli 5641c28c1c YARN-5340. Fixed a race condition in RollingLevelDBTimelineStore that caused loss of Timeline events. Contributed by Li Lu. 
(cherry picked from commit 1c9d2ab503)
 2016-07-20 08:38:56 -07:00
Arun Suresh 586cd0daa1 YARN-5350. Distributed Scheduling: Ensure sort order of allocatable nodes returned by the RM is not lost. (asuresh) 
(cherry picked from commit 8fbe6ece24)
 2016-07-19 23:22:47 -07:00
Varun Saxena 2153c9f04c YARN-4996. Make TestNMReconnect.testCompareRMNodeAfterReconnect() scheduler agnostic (Kai Sasaki via Varun Saxena) 2016-07-19 16:07:56 +05:30
Ray Chiang 66a501be5e YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang) 
(cherry picked from commit 2df1b1e72b18422606e38a9fbe311988637cb9a5)
 2016-07-15 14:39:28 -07:00
Arun Suresh 9d31c0b53f YARN-5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh) 
(cherry picked from commit e905a42a2c)
 2016-07-15 14:38:26 -07:00
Wangda Tan e34e1aa4fe YARN-4484. Available Resource calculation for a queue is not correct when used with labels. (Sunil G via wangda) 
(cherry picked from commit 24db9167f1)
 2016-07-15 11:41:35 -07:00
Varun Saxena 8b0337b499 YARN-5383. Fix findbugs and checkstyle issues in ContainerExecutor. (Vrushali C via Varun Saxena) 2016-07-15 23:30:36 +05:30
Akira Ajisaka 6664077727 YARN-5159. Wrong Javadoc tag in MiniYarnCluster. Contributed by Andras Bokor. 
(cherry picked from commit 6cf017558a)
 2016-07-14 10:45:39 -07:00
Varun Vasudev d5d68d98c6 YARN-5303. Clean up ContainerExecutor JavaDoc. Contributed by Daniel Templeton. 
(cherry picked from commit 54bf14f80b)
 2016-07-14 19:32:09 +05:30
Varun Vasudev b4124c6a5a YARN-4759. Fix signal handling for docker containers. Contributed by Shane Kumpf. 
(cherry picked from commit e5e558b0a3)
 2016-07-14 19:32:00 +05:30
Varun Vasudev abfaf0e0e7 YARN-5298. Mount usercache and NM filecache directories into Docker container. Contributed by Sidharta Seethana. 
(cherry-picked from commit 58e1850801)
 2016-07-14 16:48:22 +05:30
Rohith Sharma K S 356c09a509 YARN-5299. Log Docker run command when container fails. Contributed by Varun Vasudev. 
(cherry picked from commit dbe97aa768)
 2016-07-14 11:49:48 +05:30
Rohith Sharma K S 694b4abbc0 YARN-5362. TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail. Contributed by sandflee. 
(cherry picked from commit d6d41e820a)
 2016-07-13 19:15:41 +05:30
Varun Saxena dc3e077301 YARN-5353. ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena). 2016-07-13 07:46:00 +05:30
Jason Lowe 1f96baf4ad YARN-5317. testAMRestartNotLostContainerCompleteMsg may fail. Contributed by sandflee 
(cherry picked from commit 10b704c594)
 2016-07-12 20:38:26 +00:00
Jian He e94e6be772 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:37:52 -07:00
Varun Saxena 5ef27b65b9 YARN-5037. Fix random failure of TestRMRestart#testQueueMetricsOnRMRestart (sandflee via Varun Saxena). 2016-07-10 22:13:53 +05:30
Varun Saxena f515678b67 YARN-5318. Fix intermittent test failure of TestRMAdminService#testRefreshNodesResourceWithFileSystemBasedConfigurationProvider. (Jun Gong via Varun Saxena). 2016-07-09 01:43:09 +05:30
Varun Saxena 550abb53e5 YARN-5297. Avoid printing a stack trace when recovering an app after the RM restarts. (Junping Du via Varun Saxena). 2016-07-09 00:14:51 +05:30
Junping Du 5c0386d535 YARN-4939. The decommissioning Node should keep alive during NM restart. Contributed by sandflee. 
(cherry picked from commit 30ee57ceb1)
 2016-07-08 04:15:47 -07:00
Vinod Kumar Vavilapalli 687185feb2 YARN-5314. Fixed a ConcurrentModificationException in ATS v1.5 EntityGroupFSTimelineStore. Contributed by Li Lu. 
(cherry picked from commit 673e5e02fe)
 2016-07-07 22:38:41 -07:00
Jian He b7604a22e6 YARN-5233. Support for specifying a path for ATS plugin jars. Contributed by Li Lu 
(cherry picked from commit 8a9d293dd6)
 2016-07-06 16:45:32 -07:00
Wangda Tan e246cf74c9 YARN-5294. Pass remote ip address down to YarnAuthorizationProvider. (Jian He via wangda) 2016-07-06 16:42:35 -07:00
Vinod Kumar Vavilapalli c3d9ac82af YARN-5224. Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application. Contributed by Xuan Gong. 2016-07-06 14:19:16 -07:00
Jian He 46f8204992 YARN-5296. NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl. Contributed by Junping Du 
(cherry picked from commit d792a90206)
 2016-07-05 18:06:03 -07:00
Vinod Kumar Vavilapalli 56a08983e2 YARN-5214. Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks. Contributed by Junping Du. 
(cherry picked from commit ce9c006430)
 2016-07-05 16:08:57 -07:00
Varun Saxena 516aa58c1b YARN-5286. Add RPC port info in RM web service's response when getting app status. (Jun Gong via Varun Saxena). 2016-07-05 23:10:08 +05:30
Ray Chiang fb9327edae YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang) 
(cherry picked from commit 0a5def155e)
 2016-07-01 15:55:56 -07:00
Jian He d2559ca307 YARN-5023. TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry fails. Contributed by sandflee 
(cherry picked from commit c35a5a7a8d)
 2016-07-01 14:29:28 -07:00
Varun Saxena 527cd9852a YARN-5182. MockNodes.newNodes creates one more node per rack than requested. (Karthik Kambatla via Varun Saxena). 2016-06-30 00:06:17 +05:30
Rohith Sharma K S d3fc1bddea YARN-5262. Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat. 
(cherry picked from commit 26b5e6116f)
 2016-06-29 10:13:17 +05:30
Arun Suresh e64d21b297 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh) 
(cherry picked from commit 99e5dd68d0)
 2016-06-22 19:05:28 -07:00
Sangjin Lee 2a79910c22 YARN-4958. The file localization process should allow for wildcards to reduce the application footprint in the state store (Daniel Templeton via sjlee) 
(cherry picked from commit 5107a967fa)
 2016-06-20 10:00:41 -07:00
Junping Du ea4cab2330 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev. 
(cherry picked from commit d0162f2040)
 2016-06-19 17:45:57 -07:00
Karthik Kambatla 25be8105cd YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha) 
(cherry picked from commit 20f2799938)
 2016-06-17 22:25:04 -07:00
Karthik Kambatla e9c1155233 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit. Addendum to fix javac warning. (Arun Suresh via kasha) 
(cherry picked from commit fbbe0bb627)
 2016-06-17 22:13:18 -07:00
Junping Du 6afa044dda YARN-5251. Yarn CLI to obtain App logs for last 'n' bytes fails. Contributed by Xuan Gong. 
(cherry picked from commit c35fa4a0e5)
 2016-06-17 08:25:33 -07:00
Junping Du 6dd34baf3c YARN-5237. Fix missing log files issue in rolling log aggregation. Contributed by Xuan Gong. 
(cherry picked from commit 5dfc38ff57)
 2016-06-15 16:20:07 -07:00
Wangda Tan 863bfa4d6a YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:12:00 -07:00
Rohith Sharma K S c540b9bd7b YARN-4989. TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently. Contributed by Ajith S. 
(cherry picked from commit 28b66ae919)
 2016-06-13 11:23:10 +05:30
Arun Suresh f3c7a99e33 YARN-5212. Run existing ContainerManager tests using QueuingContainerManagerImpl. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 7dae2b3bc4)
 2016-06-12 10:06:18 -07:00
Arun Suresh 5985221b46 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh) 
(cherry picked from commit 5279af7cd4)
 2016-06-10 22:34:33 -07:00
Rohith Sharma K S efdfa68911 YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe. 
(cherry picked from commit e0f4620cc7)
 2016-06-11 10:31:00 +05:30
Wangda Tan e91e9e596d YARN-3426. Add jdiff support to YARN. (vinodkv via wangda) 
(cherry picked from commit 03fc6b1bb0)
 2016-06-10 09:53:40 -07:00
Wangda Tan a4094ddb5a YARN-5208. Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled. (Rohith Sharma K S via wangda) 
(cherry picked from commit 244506f9c8)
 2016-06-10 09:37:35 -07:00
Vinod Kumar Vavilapalli f6cd1bcf89 YARN-5191. Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option. (Xuan Gong via vinodkv) 
(cherry picked from commit 9378d9428f)
 2016-06-09 12:32:53 -07:00
Wangda Tan aef6e455b1 YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-06-09 10:08:31 -07:00
Naganarasimha 27fc225558 YARN-4308. ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats. Contributed by Sunil G 
(cherry picked from commit 1500a0a300)
 2016-06-09 05:44:41 +05:30
Arun Suresh c29fbdd8ac YARN-5204. Properly report status of killed/stopped queued containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 3344ba70e0)
 2016-06-08 08:32:17 -07:00
Arun Suresh 154c7c343b YARN-5176. More test cases for queuing of containers at the NM. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 76f0800c21)
 2016-06-07 17:17:22 -07:00
Xuan 10f0c0475e YARN-5199. Close LogReader in in AHSWebServices#getStreamingOutput and 
FileInputStream in NMWebServices#getLogs. Contributed by Xuan Gong

(cherry picked from commit 58be55b6e0)
 2016-06-07 16:08:54 -07:00
Rohith Sharma K S a29f8bdc3d YARN-5118. Tests fails with localizer port bind exception. Contributed by Brahma Reddy Battula. 
(cherry picked from commit bddea5fe5f)
 2016-06-07 11:23:02 +05:30
Arun Suresh d3655f4123 YARN-4525. Fix bug in RLESparseResourceAllocation.getRangeOverlapping(). (Ishai Menache and Carlo Curino via asuresh) 
(cherry picked from commit 3a154f75ed)
 2016-06-06 21:18:53 -07:00
Arun Suresh f0a869b52a YARN-5185. StageAllocaterGreedyRLE: Fix NPE in corner case. (Carlo Curino via asuresh) 
(cherry picked from commit 7a9b7372a1)
 2016-06-06 21:07:24 -07:00
Ming Ma 74e2b5efa2 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma) 
(cherry picked from commit 4a1cedc010)
 2016-06-06 14:34:47 -07:00
Arun Suresh 8ebd89dcfb YARN-5165. Fix NoOvercommitPolicy to take advantage of RLE representation of plan. (Carlo Curino via asuresh) 
(cherry picked from commit db54670e83)
 2016-06-03 14:50:51 -07:00
Vinod Kumar Vavilapalli 029888871f YARN-5098. Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time. Contributed by Jian He. 
Made one minor edit for branch-2 patch.

(cherry picked from commit f10ebc67f5)
 2016-06-03 13:11:17 -07:00
Jian He 7f865bf438 YARN-5190. Registering/unregistering container metrics in ContainerMonitorImpl and ContainerImpl causing uncaught exception in ContainerMonitorImpl. Contributed by Junping Du 
(cherry picked from commit 99cc439e29)
 2016-06-03 11:11:20 -07:00
Jian He a022776c2e YARN-1815. Work preserving recovery of Unmanged AMs. Contributed by Subru Krishnan 
(cherry picked from commit 097baaaeba)
 2016-06-03 10:49:54 -07:00
Arun Suresh 384a84828f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh) 
(cherry picked from commit dc26601d8f)
 2016-06-02 09:14:30 -07:00
Xuan 04062593b4 YARN-5088. Improve "yarn log" command-line to read the last K bytes for the log files. Contributed by Xuan Gong 
(cherry picked from commit 0bc05e40fa)
 2016-06-01 13:45:24 -07:00
Varun Vasudev d7bb28f15b YARN-4007. Add support for different network setups when launching the docker container. Contributed by Sidharta Seethana. 
(cherry picked from commit 86fb58b7dc)
 2016-05-29 21:33:43 +05:30
Varun Vasudev 4f36c3d214 YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:55:39 +05:30