Commit Graph

2004 Commits

Author SHA1 Message Date
Varun Saxena 516aa58c1b YARN-5286. Add RPC port info in RM web service's response when getting app status. (Jun Gong via Varun Saxena). 2016-07-05 23:10:08 +05:30
Ray Chiang fb9327edae YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang)
(cherry picked from commit 0a5def155e)
2016-07-01 15:55:56 -07:00
Jian He d2559ca307 YARN-5023. TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry fails. Contributed by sandflee
(cherry picked from commit c35a5a7a8d)
2016-07-01 14:29:28 -07:00
Varun Saxena 527cd9852a YARN-5182. MockNodes.newNodes creates one more node per rack than requested. (Karthik Kambatla via Varun Saxena). 2016-06-30 00:06:17 +05:30
Rohith Sharma K S d3fc1bddea YARN-5262. Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat.
(cherry picked from commit 26b5e6116f)
2016-06-29 10:13:17 +05:30
Arun Suresh e64d21b297 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh)
(cherry picked from commit 99e5dd68d0)
2016-06-22 19:05:28 -07:00
Sangjin Lee 2a79910c22 YARN-4958. The file localization process should allow for wildcards to reduce the application footprint in the state store (Daniel Templeton via sjlee)
(cherry picked from commit 5107a967fa)
2016-06-20 10:00:41 -07:00
Junping Du ea4cab2330 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev.
(cherry picked from commit d0162f2040)
2016-06-19 17:45:57 -07:00
Karthik Kambatla 25be8105cd YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha)
(cherry picked from commit 20f2799938)
2016-06-17 22:25:04 -07:00
Karthik Kambatla e9c1155233 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit. Addendum to fix javac warning. (Arun Suresh via kasha)
(cherry picked from commit fbbe0bb627)
2016-06-17 22:13:18 -07:00
Junping Du 6afa044dda YARN-5251. Yarn CLI to obtain App logs for last 'n' bytes fails. Contributed by Xuan Gong.
(cherry picked from commit c35fa4a0e5)
2016-06-17 08:25:33 -07:00
Junping Du 6dd34baf3c YARN-5237. Fix missing log files issue in rolling log aggregation. Contributed by Xuan Gong.
(cherry picked from commit 5dfc38ff57)
2016-06-15 16:20:07 -07:00
Wangda Tan 863bfa4d6a YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:12:00 -07:00
Rohith Sharma K S c540b9bd7b YARN-4989. TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently. Contributed by Ajith S.
(cherry picked from commit 28b66ae919)
2016-06-13 11:23:10 +05:30
Arun Suresh f3c7a99e33 YARN-5212. Run existing ContainerManager tests using QueuingContainerManagerImpl. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 7dae2b3bc4)
2016-06-12 10:06:18 -07:00
Arun Suresh 5985221b46 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh)
(cherry picked from commit 5279af7cd4)
2016-06-10 22:34:33 -07:00
Rohith Sharma K S efdfa68911 YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe.
(cherry picked from commit e0f4620cc7)
2016-06-11 10:31:00 +05:30
Wangda Tan e91e9e596d YARN-3426. Add jdiff support to YARN. (vinodkv via wangda)
(cherry picked from commit 03fc6b1bb0)
2016-06-10 09:53:40 -07:00
Wangda Tan a4094ddb5a YARN-5208. Run TestAMRMClient TestNMClient TestYarnClient TestClientRMTokens TestAMAuthorization tests with hadoop.security.token.service.use_ip enabled. (Rohith Sharma K S via wangda)
(cherry picked from commit 244506f9c8)
2016-06-10 09:37:35 -07:00
Vinod Kumar Vavilapalli f6cd1bcf89 YARN-5191. Renamed the newly added “download=true” option for getting logs via NMWebServices and AHSWebServices to be a better "format" option. (Xuan Gong via vinodkv)
(cherry picked from commit 9378d9428f)
2016-06-09 12:32:53 -07:00
Wangda Tan aef6e455b1 YARN-4837. User facing aspects of 'AM blacklisting' feature need fixing. (vinodkv via wangda) 2016-06-09 10:08:31 -07:00
Naganarasimha 27fc225558 YARN-4308. ContainersAggregated CPU resource utilization reports negative usage in first few heartbeats. Contributed by Sunil G
(cherry picked from commit 1500a0a300)
2016-06-09 05:44:41 +05:30
Arun Suresh c29fbdd8ac YARN-5204. Properly report status of killed/stopped queued containers. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 3344ba70e0)
2016-06-08 08:32:17 -07:00
Arun Suresh 154c7c343b YARN-5176. More test cases for queuing of containers at the NM. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 76f0800c21)
2016-06-07 17:17:22 -07:00
Xuan 10f0c0475e YARN-5199. Close LogReader in in AHSWebServices#getStreamingOutput and
FileInputStream in NMWebServices#getLogs. Contributed by Xuan Gong

(cherry picked from commit 58be55b6e0)
2016-06-07 16:08:54 -07:00
Rohith Sharma K S a29f8bdc3d YARN-5118. Tests fails with localizer port bind exception. Contributed by Brahma Reddy Battula.
(cherry picked from commit bddea5fe5f)
2016-06-07 11:23:02 +05:30
Arun Suresh d3655f4123 YARN-4525. Fix bug in RLESparseResourceAllocation.getRangeOverlapping(). (Ishai Menache and Carlo Curino via asuresh)
(cherry picked from commit 3a154f75ed)
2016-06-06 21:18:53 -07:00
Arun Suresh f0a869b52a YARN-5185. StageAllocaterGreedyRLE: Fix NPE in corner case. (Carlo Curino via asuresh)
(cherry picked from commit 7a9b7372a1)
2016-06-06 21:07:24 -07:00
Ming Ma 74e2b5efa2 MAPREDUCE-5044. Have AM trigger jstack on task attempts that timeout before killing them. (Eric Payne and Gera Shegalov via mingma)
(cherry picked from commit 4a1cedc010)
2016-06-06 14:34:47 -07:00
Arun Suresh 8ebd89dcfb YARN-5165. Fix NoOvercommitPolicy to take advantage of RLE representation of plan. (Carlo Curino via asuresh)
(cherry picked from commit db54670e83)
2016-06-03 14:50:51 -07:00
Vinod Kumar Vavilapalli 029888871f YARN-5098. Fixed ResourceManager's DelegationTokenRenewer to replace expiring system-tokens if RM stops and only restarts after a long time. Contributed by Jian He.
Made one minor edit for branch-2 patch.

(cherry picked from commit f10ebc67f5)
2016-06-03 13:11:17 -07:00
Jian He 7f865bf438 YARN-5190. Registering/unregistering container metrics in ContainerMonitorImpl and ContainerImpl causing uncaught exception in ContainerMonitorImpl. Contributed by Junping Du
(cherry picked from commit 99cc439e29)
2016-06-03 11:11:20 -07:00
Jian He a022776c2e YARN-1815. Work preserving recovery of Unmanged AMs. Contributed by Subru Krishnan
(cherry picked from commit 097baaaeba)
2016-06-03 10:49:54 -07:00
Arun Suresh 384a84828f YARN-5180. Allow ResourceRequest to specify an enforceExecutionType flag. (asuresh)
(cherry picked from commit dc26601d8f)
2016-06-02 09:14:30 -07:00
Xuan 04062593b4 YARN-5088. Improve "yarn log" command-line to read the last K bytes for the log files. Contributed by Xuan Gong
(cherry picked from commit 0bc05e40fa)
2016-06-01 13:45:24 -07:00
Varun Vasudev d7bb28f15b YARN-4007. Add support for different network setups when launching the docker container. Contributed by Sidharta Seethana.
(cherry picked from commit 86fb58b7dc)
2016-05-29 21:33:43 +05:30
Varun Vasudev 4f36c3d214 YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:55:39 +05:30
Arun Suresh de28ca1e92 YARN-5117. QueuingContainerManager does not start GUARANTEED Container even if Resources are available. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 4fc09a897b)
2016-05-27 17:50:34 -07:00
Arun Suresh 0b1472fbc0 YARN-5127. Expose ExecutionType in Container api record. (Hitesh Sharma via asuresh)
(cherry picked from commit aa975bc781)
2016-05-27 14:07:15 -07:00
Junping Du dea79a7dd6 YARN-4987. Read cache concurrency issue between read and evict in EntityGroupFS timeline store. Contributed by Li Lu.
(cherry picked from commit 705286ccae)
2016-05-27 06:59:15 -07:00
Rohith Sharma K S ef3e0c6fd6 YARN-5005. TestRMWebServices#testDumpingSchedulerLogs fails randomly. Contributed by Bibin A Chundatt.
(cherry picked from commit 0a544f8a3e)
2016-05-27 10:48:47 +05:30
Karthik Kambatla 59335b4d7a YARN-5035. FairScheduler: Adjust maxAssign dynamically when assignMultiple is turned on. (kasha)
(cherry picked from commit 04ded558b0)
2016-05-26 15:09:47 -07:00
Arun Suresh 44cbf5b7f5 YARN-5162. Fix Exceptions thrown during in registerAM call when Distributed Scheduling is Enabled (Hitesh Sharma via asuresh)
(cherry picked from commit 5b41b288d0)
2016-05-26 14:58:15 -07:00
Karthik Kambatla 999cdd91b7 YARN-4866. FairScheduler: AMs can consume all vcores leading to a livelock when using FAIR policy. (Yufei Gu via kasha)
(cherry picked from commit 4f513a4a8e)
2016-05-25 22:13:42 -07:00
Carlo Curino c656977961 YARN-4957. Add getNewReservation in ApplicationClientProtocol (Sean Po via curino)
(cherry picked from commit 013532a95e)
2016-05-25 16:58:19 -07:00
Jason Lowe da77040793 YARN-4459. container-executor should only kill process groups. Contributed by Jun Gong
(cherry picked from commit 1ba31fe9e9)
2016-05-25 21:29:27 +00:00
Robert Kanter 71e5b36763 YARN-4766. NM should not aggregate logs older than the retention policy (haibochen via rkanter)
(cherry picked from commit e07519b8db)
2016-05-25 10:25:59 -07:00
Varun Vasudev 5361a91909 YARN-5115. Avoid setting CONTENT-DISPOSITION header in the container-logs web-service. Contributed by Xuan Gong.
(cherry picked from commit c52c15dd2df5050c6c3ddbe5ee708404f1920721)
2016-05-25 19:01:22 +05:30
Varun Vasudev 36f2ae0692 YARN-857. Localization failures should be available in container diagnostics. Contributed by Vinod Kumar Vavilapalli.
(cherry picked from commit f440a9d8c4a177bc5062d21d4b4bc4d9b2944344)
2016-05-25 19:01:22 +05:30
Rohith Sharma K S b9567b89b9 YARN-5024. TestContainerResourceUsage#testUsageAfterAMRestartWithMultipleContainers random failure. Contributed by Bibin A Chundatt
(cherry picked from commit 28bd63e92b)
2016-05-25 10:24:59 +05:30
Naganarasimha 02323603e2 YARN-5114. Add additional tests in TestRMWebServicesApps and rectify testInvalidAppAttempts failure in 2.8. Contributed by Bibin A Chundatt
(cherry picked from commit edd716e99c)
2016-05-25 06:15:50 +08:00
Karthik Kambatla 108db38788 YARN-4878. Expose scheduling policy and max running apps over JMX for Yarn queues. (Yufei Gu via kasha)
(cherry picked from commit f979d779e1)
2016-05-24 10:54:33 -07:00
Naganarasimha 155f25061e YARN-3971. Skip RMNodeLabelsManager#checkRemoveFromClusterNodeLabelsOfQueue on nodelabel recovery. (addendum patch). Contributed by Bibin A chundatt
(cherry picked from commit b4078bd17b)
2016-05-24 08:14:53 +08:00
Karthik Kambatla f33f0b697b YARN-4979. FSAppAttempt demand calculation considers demands at multiple locality levels different. (Zhihai Xu via kasha)
(cherry picked from commit 6d043aa4cf)
2016-05-23 14:30:05 -07:00
Jason Lowe 7602070457 YARN-5055. max apps per user can be larger than max per queue. Contributed by Eric Badger
(cherry picked from commit ac954486c5)
2016-05-23 15:56:00 +00:00
Jason Lowe e8bd2a5be6 YARN-5103. With NM recovery enabled, restarting NM multiple times results in AM restart. Contributed by Junping Du
(cherry picked from commit d1df0266cf)
2016-05-23 15:16:52 +00:00
Junping Du 867cd2f5a2 YARN-5112. Excessive log warnings for directory permission issue on NM recovery. Contributed by Jian He. 2016-05-22 16:02:57 -07:00
Varun Vasudev e7f396fd47 YARN-5016. Add support for a minimum retry interval for container retries. Contributed by Jun Gong.
(cherry picked from commit 0287c49107)
2016-05-20 12:32:29 +05:30
Junping Du 2703ec6871 YARN-5076. YARN web interfaces lack XFS protection. Contributed by Jonathan Maron. 2016-05-19 14:10:15 -07:00
Jian He 2d846fdcbd YARN-4002. Make ResourceTrackerService#nodeHeartbeat more concurrent. Contributed by Rohith Sharma K S & Zhiguo Hong
(cherry picked from commit feb90ffcca)
2016-05-19 13:02:05 -07:00
Junping Du 416274b53e YARN-5100. The YarnApplicationState is always running in ATS even application is finished. Contributed by Xuan Gong.
(cherry picked from commit 141873ca7d)
2016-05-19 09:14:42 -07:00
Steve Loughran 151ecdf26d HADOOP-12767. Update apache httpclient version to 4.5.2; httpcore to 4.4.4. Artem Aliev via stevel. 2016-05-19 14:44:44 +01:00
Arun Suresh 15808717f3 YARN-5110. Fix OpportunisticContainerAllocator to insert complete HostAddress in issued ContainerTokenIds. (Konstantinos Karanasos via asuresh)
(cherry picked from commit 1597630681)
2016-05-18 23:20:54 -07:00
Arun Suresh a555a320e8 YARN-5090. Add End-to-End test-cases for DistributedScheduling using MiniYarnCluster. (asuresh)
(cherry picked from commit 8a9ecb7584)
2016-05-18 23:20:36 -07:00
Arun Suresh f31602893e YARN-5075. Fix findbugs warnings in hadoop-yarn-common module. (asuresh)
(cherry picked from commit ccc93e7812)
2016-05-18 23:20:08 -07:00
Arun Suresh bbcea3abf3 YARN-4738. Notify the RM about the status of OPPORTUNISTIC containers (Konstantinos Karanasos via asuresh)
(cherry picked from commit f45bc5a83e)
2016-05-18 23:12:19 -07:00
Arun Suresh 69a9ce3494 YARN-2888. Corrective mechanisms for rebalancing NM container queues. (asuresh)
(cherry picked from commit f0ac18d001)
2016-05-18 23:09:11 -07:00
Arun Suresh 005dae521c YARN-5073. Refactor startContainerInternal() in ContainerManager to remove unused parameter. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d971bf252f)
2016-05-18 22:36:28 -07:00
Arun Suresh 1b4600abcf YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
(cherry picked from commit 341888a0aa)
2016-05-18 22:35:55 -07:00
Arun Suresh 307cda70db YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d464f4d1c4)
2016-05-18 22:09:30 -07:00
Arun Suresh d80d24aabf YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
(cherry picked from commit c282a08f38)
2016-05-18 22:08:55 -07:00
Karthik Kambatla b56fc51b70 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha)
(cherry picked from commit c8172f5f14)
2016-05-18 22:08:20 -07:00
Jian He 264c06a438 YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 15:02:14 -07:00
Eric Payne 42b2a34ed8 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger.
(cherry picked from commit 1217c8f6b4)
2016-05-16 20:38:10 +00:00
Jason Lowe 575c056357 YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du
(cherry picked from commit 81effb7dcd)
2016-05-16 15:43:20 +00:00
Rohith Sharma K S a37b3694ea YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-16 15:25:15 +05:30
Jason Lowe bb95a5e18f YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger
(cherry picked from commit 013000fbc2)
2016-05-12 20:30:10 +00:00
Sangjin Lee d6d13ec670 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale)
(cherry picked from commit 0bbe01f8d5)
2016-05-12 10:14:24 -07:00
Karthik Kambatla f96b2b6fac YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha)
(cherry picked from commit 4b4e4c6ba8)
2016-05-11 17:36:42 -07:00
Junping Du 9dc3683d87 YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong.
(cherry picked from commit 39f2bac38b)
2016-05-11 09:33:16 -07:00
Naganarasimha 3732a1e985 YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 2750fb900f)
2016-05-08 23:02:07 +05:30
Yongjun Zhang 5172d0e7b1 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang)
(cherry picked from commit 47c41e7ac7)
2016-05-06 22:38:56 -07:00
Jason Lowe 3895058a67 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena
(cherry picked from commit b2ed6ae731)
2016-05-06 23:00:25 +00:00
Wangda Tan b68e6b1d6d getApplicationReport call may raise NPE for removed queues. (Jian He via wangda)
(cherry picked from commit 23248f63aa)
2016-05-06 15:32:15 -07:00
Jian He a6b24c62ab YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan
(cherry picked from commit bb62e05925)
2016-05-05 12:56:55 -07:00
Jason Lowe ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit d0da13229c)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Varun Vasudev 38a3b86141 YARN-4595. Add support for configurable read-only mounts when launching Docker containers. Contributed by Billie Rinaldi.
(cherry picked from commit 72b047715c)
2016-05-05 13:02:38 +05:30
Wangda Tan 585299146a YARN-4984. LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak. (Junping Du via wangda)
(cherry picked from commit 7bd418e48c)
2016-05-04 11:39:25 -07:00
Junping Du 1ffb0c43d6 YARN-4920. ATS/NM should support a link to dowload/get the logs in text format. Contributed by Xuan Gong.
(cherry picked from commit 3cf223166d452a0f58f92676837a9edb8ddc1139)
2016-05-04 10:36:31 -07:00
Rohith Sharma K S 5aad4070b2 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt
(cherry picked from commit 75e0450593)
2016-05-04 10:26:25 +05:30
Jason Lowe baac4e7db1 YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts
(cherry picked from commit ed54f5f1ff)
2016-05-03 22:16:17 +00:00
Junping Du 47f67ae447 YARN-4851. Metric improvements for ATS v1.5 storage components. Li Lu via junping_du.
(cherry picked from commit 06413da72e)
2016-05-03 04:18:01 -07:00
Robert Kanter ac8fb579c6 Remove parent's env vars from child processes 2016-04-29 09:26:09 -07:00
Varun Vasudev 6561e3b500 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong.
(cherry picked from commit 0f25a1bb52)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
2016-04-29 16:22:06 +05:30
Jian He 6ba39a1597 YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe
(cherry picked from commit 4a8508501b)
2016-04-28 21:54:30 -07:00
Jian He a9707dceaf YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:28:03 -07:00
Li Lu 50b7a35d56 YARN-4956. findbug issue on LevelDBCacheTimelineStore. (Zhiyuan Yang via gtcarrera9)
(cherry picked from commit f16722d2ef)
2016-04-27 10:58:12 -07:00
Karthik Kambatla 864ecb4434 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha)
(cherry picked from commit 185c3d4de1)
2016-04-27 09:43:42 -07:00
Jian He 9d3ddb0b4d YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode
(cherry picked from commit 4beff01354)
2016-04-26 21:02:04 -07:00
Karthik Kambatla 52bfa90fed YARN-4795. ContainerMetrics drops records. (Daniel Templeton via kasha)
(cherry picked from commit 1a3f1482e2)
2016-04-26 06:18:27 -07:00