2802 Commits

Author SHA1 Message Date
Arun Suresh
005dae521c YARN-5073. Refactor startContainerInternal() in ContainerManager to remove unused parameter. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d971bf252f36e902a0fe446cb11201835ad16bcb)
2016-05-18 22:36:28 -07:00
Arun Suresh
1b4600abcf YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
(cherry picked from commit 341888a0aa23f24458b4e6e34868794b9735c06a)
2016-05-18 22:35:55 -07:00
Arun Suresh
307cda70db YARN-5049. Extend NMStateStore to save queued container information. (Konstantinos Karanasos via asuresh)
(cherry picked from commit d464f4d1c4dec483852fc8c0496787cba0af8f57)
2016-05-18 22:09:30 -07:00
Arun Suresh
d80d24aabf YARN-2885. Create AMRMProxy request interceptor and ContainerAllocator to distribute OPPORTUNISTIC containers to appropriate Nodes (asuresh)
(cherry picked from commit 2bf025278a318b0452fdc9ece4427b4c42124e39)
(cherry picked from commit c282a08f3892e2e8ceb58e1e9a411062fbd1fb2b)
2016-05-18 22:08:55 -07:00
Karthik Kambatla
b56fc51b70 YARN-2883. Queuing of container requests in the NM. (Konstantinos Karanasos and Arun Suresh via kasha)
(cherry picked from commit c8172f5f143d2fefafa5a412899ab7cd081b406d)
2016-05-18 22:08:20 -07:00
Arun Suresh
68964bbaa2 YARN-4335. Allow ResourceRequests to specify ExecutionType of a request ask (kkaranasos via asuresh)
(cherry picked from commit 8ffabfdf4fec104cdaa68133385cfe9cf88b95a8)
(cherry picked from commit b2a654c5ee6524f81c971ea0b70e58ea0a455f1d)
2016-05-18 22:02:43 -07:00
Karthik Kambatla
2c6c701aff YARN-2882. Add an OPPORTUNISTIC ExecutionType. (Konstantinos Karanasos and Inigo Goiri via kasha)
(cherry picked from commit fb00794368e0aa7aafa9dfc8d453810f641b82b2)
2016-05-18 22:02:28 -07:00
Naganarasimha
a166286b9e YARN-4925. ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression. Contributed by Bibin A Chundatt
(cherry picked from commit f04c81c9ce93512bc714531a7731debbe6b794ce)
2016-05-19 01:35:47 +08:00
Varun Vasudev
3c9392a31d YARN-4913. Yarn logs should take a -out option to write to a directory. Contributed by Xuan Gong.
(cherry picked from commit ef1757790d89cc72f88f5330761b1c8901c59e94)
2016-05-18 22:45:06 +05:30
Jian He
264c06a438 YARN-4832. NM side resource value should get updated if change applied in RM side. Contributed by Junping Du 2016-05-17 15:02:14 -07:00
Eric Payne
42b2a34ed8 YARN-5069. TestFifoScheduler.testResourceOverCommit race condition. Contributed by Eric Badger.
(cherry picked from commit 1217c8f6b49e3afd8ca008ffddcf6615f0accfc5)
2016-05-16 20:38:10 +00:00
Jason Lowe
575c056357 YARN-4325. Nodemanager log handlers fail to send finished/failed events in some cases. Contributed by Junping Du
(cherry picked from commit 81effb7dcde2b31423438d6f1b8b8204d4ca05b3)
2016-05-16 15:43:20 +00:00
Rohith Sharma K S
a37b3694ea YARN-5068. Expose scheduler queue to application master. (Harish Jaiprakash via rohithsharmaks) 2016-05-16 15:25:15 +05:30
Xuan
3fe4107909 YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or
RUNNING AM. Contributed by Xuan Gong

(cherry picked from commit 7be53b65f95dae89c760cde354b42f15f5ee3b5f)
2016-05-14 12:22:06 -07:00
Jason Lowe
bb95a5e18f YARN-5053. More informative diagnostics when applications killed by a user. Contributed by Eric Badger
(cherry picked from commit 013000fbc2081c2b7e78b790fc6d847de92b9384)
2016-05-12 20:30:10 +00:00
Sangjin Lee
d6d13ec670 YARN-4577. Enable aux services to have their own custom classpath/jar file (Xuan Gong via sale)
(cherry picked from commit 0bbe01f8d56191edfba3b50fb9f8859a0b3f826f)
2016-05-12 10:14:24 -07:00
Karthik Kambatla
f96b2b6fac YARN-4995. FairScheduler: Display per-queue demand on the scheduler page. (xupeng via kasha)
(cherry picked from commit 4b4e4c6ba83bc5c41d7bb69bb2483bcfe894a260)
2016-05-11 17:36:42 -07:00
Junping Du
9dc3683d87 YARN-5029. RM needs to send update event with YarnApplicationState as Running to ATS/AHS. Contributed by Xuan Gong.
(cherry picked from commit 39f2bac38b111f90d3402229201cdb4315f5d4f5)
2016-05-11 09:33:16 -07:00
Vinod Kumar Vavilapalli
e0d6a9632d YARN-4842. Fixed "yarn logs" command to guess (and thus not require) the appOwner argument when viewing another user's logs. Contributed by Ram Venkatesh and Xuan Gong.
(cherry picked from commit 87f5e351337a905af5215af76c72b9312616cd4f)
2016-05-09 22:47:58 -07:00
Naganarasimha
3732a1e985 YARN-4926. Change nodelabel rest API invalid reponse status to 400. Contributed by Bibin A Chundatt
(cherry picked from commit 2750fb900fc9c468570d051cb19e3a379a898c38)
2016-05-08 23:02:07 +05:30
Yongjun Zhang
5172d0e7b1 YARN-5048. DelegationTokenRenewer#skipTokenRenewal may throw NPE (Jian He via Yongjun Zhang)
(cherry picked from commit 47c41e7ac7e6b905a58550f8899f629c1cf8b138)
2016-05-06 22:38:56 -07:00
Jason Lowe
3895058a67 YARN-4747. AHS error 500 due to NPE when container start event is missing. Contributed by Varun Saxena
(cherry picked from commit b2ed6ae73197990a950ce71ece80c0f23221c384)
2016-05-06 23:00:25 +00:00
Wangda Tan
b68e6b1d6d getApplicationReport call may raise NPE for removed queues. (Jian He via wangda)
(cherry picked from commit 23248f63aab74a19dba38d348f2b231c8360770a)
2016-05-06 15:32:15 -07:00
Jian He
a6b24c62ab YARN-4390. Do surgical preemption based on reserved container in CapacityScheduler. Contributed by Wangda Tan
(cherry picked from commit bb62e0592566b2fcae7136b30972aad2d3ac55b0)
2016-05-05 12:56:55 -07:00
Jason Lowe
ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
(cherry picked from commit d0da13229cf692579c8c9db47a93f6c6255392c8)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Varun Vasudev
38a3b86141 YARN-4595. Add support for configurable read-only mounts when launching Docker containers. Contributed by Billie Rinaldi.
(cherry picked from commit 72b047715c1ae89dff3dfe76c1af91dfe255ed70)
2016-05-05 13:02:38 +05:30
Vinod Kumar Vavilapalli
8262ef8318 YARN-4905. Improved "yarn logs" command-line to optionally show log metadata also. Contributed by Xuan Gong.
(cherry picked from commit 9e37fe3b7a3b5f0a193d228bb5e065f41acd2835)
2016-05-04 14:17:09 -07:00
Wangda Tan
585299146a YARN-4984. LogAggregationService shouldn't swallow exception in handling createAppDir() which cause thread leak. (Junping Du via wangda)
(cherry picked from commit 7bd418e48c71590fc8026d69f9b8f8ad42f2aade)
2016-05-04 11:39:25 -07:00
Junping Du
1ffb0c43d6 YARN-4920. ATS/NM should support a link to dowload/get the logs in text format. Contributed by Xuan Gong.
(cherry picked from commit 3cf223166d452a0f58f92676837a9edb8ddc1139)
2016-05-04 10:36:31 -07:00
Rohith Sharma K S
5aad4070b2 YARN-4947. Test timeout is happening for TestRMWebServicesNodes. Contributed by Bibin A Chundatt
(cherry picked from commit 75e0450593680fb6e57fbef4f4f3af1487023b5d)
2016-05-04 10:26:25 +05:30
Jason Lowe
baac4e7db1 YARN-5003. Add container resource to RM audit log. Contributed by Nathan Roberts
(cherry picked from commit ed54f5f1ff7862f8216f77d5ea8f9ccea674ccd1)
2016-05-03 22:16:17 +00:00
Jason Lowe
d8fcf02069 YARN-4834. ProcfsBasedProcessTree doesn't track daemonized processes. Contributed by Nathan Roberts
(cherry picked from commit c6b48391680c1b81a86aabc3ad4c725bfade6d2e)
2016-05-03 22:14:50 +00:00
Junping Du
47f67ae447 YARN-4851. Metric improvements for ATS v1.5 storage components. Li Lu via junping_du.
(cherry picked from commit 06413da72efed9a50e49efaf7110c220c88a7f4a)
2016-05-03 04:18:01 -07:00
Robert Kanter
ac8fb579c6 Remove parent's env vars from child processes 2016-04-29 09:26:09 -07:00
Varun Vasudev
6561e3b500 YARN-3998. Add support in the NodeManager to re-launch containers. Contributed by Jun Gong.
(cherry picked from commit 0f25a1bb52bc56661fd020a6ba82df99f8c6ef1f)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ProtoUtils.java
2016-04-29 16:22:06 +05:30
Jian He
6ba39a1597 YARN-5009. NMLeveldbStateStoreService database can grow substantially leading to longer recovery times. Contributed by Jason Lowe
(cherry picked from commit 4a8508501bc753858693dacdafba61d604702f71)
2016-04-28 21:54:30 -07:00
Jian He
a9707dceaf YARN-5008. LeveldbRMStateStore database can grow substantially leading to long recovery times. Contributed by Jason Lowe 2016-04-28 21:28:03 -07:00
Li Lu
50b7a35d56 YARN-4956. findbug issue on LevelDBCacheTimelineStore. (Zhiyuan Yang via gtcarrera9)
(cherry picked from commit f16722d2ef31338a57a13e2c8d18c1c62d58bbaf)
2016-04-27 10:58:12 -07:00
Karthik Kambatla
864ecb4434 YARN-4807. MockAM#waitForState sleep duration is too long. (Yufei Gu via kasha)
(cherry picked from commit 185c3d4de1ac4cf10cc1aa00aaaaf367b3880b80)
2016-04-27 09:43:42 -07:00
Varun Vasudev
cf3e93ee73 YARN-4966. Improve yarn logs to fetch container logs without specifying nodeId. Contributed by Xuan Gong.
(cherry picked from commit 66b07d83740a2ec3e6bfb2bfd064863bae37a1b5)
2016-04-27 15:57:17 +05:30
Jian He
9d3ddb0b4d YARN-4983. JVM and UGI metrics disappear after RM transitioned to standby mode
(cherry picked from commit 4beff013546dbc29c004a1bad1b019dc9d2b751f)
2016-04-26 21:02:04 -07:00
Karthik Kambatla
52bfa90fed YARN-4795. ContainerMetrics drops records. (Daniel Templeton via kasha)
(cherry picked from commit 1a3f1482e2738c7f9a983bc55189116930388d7b)
2016-04-26 06:18:27 -07:00
Karthik Kambatla
a5edb45b18 YARN-1297. FairScheduler: Move some logs to debug and check if debug logging is enabled
(cherry picked from commit 4b1dcbbe0c1d0036f65283be4b25d9b2211abed3)
2016-04-26 05:10:29 -07:00
Wangda Tan
45ff579bfa YARN-4846. Fix random failures for TestCapacitySchedulerPreemption#testPreemptionPolicyShouldRespectAlreadyMarkedKillableContainers. (Bibin A Chundatt via wangda)
(cherry picked from commit 7cb3a3da96e59fc9b6528644dae5fb0ac1e44eac)
2016-04-22 11:41:08 -07:00
Eric Payne
a7f903b2ba YARN-4556. TestFifoScheduler.testResourceOverCommit fails. Contributed by Akihiro Suda
(cherry picked from commit 3dce486d88895dcdf443f4d0064d1fb6e9116045)
2016-04-21 21:27:10 +00:00
Li Lu
054fa104c5 YARN-4968. A couple of AM retry unit tests need to wait SchedulerApplicationAttempt stopped. (Wangda Tan via gtcarrera9)
(cherry picked from commit 7c6339f66ac301406504be28841bc3f3bfebc8ae)
2016-04-21 13:27:47 -07:00
Chris Douglas
878924189d YARN-4976. Missing NullPointer check in ContainerLaunchContextPBImpl. Contributed by Giovanni Matteo Fumarola
(cherry picked from commit 95a50466075c28110fa7c297e9c5246892076ca8)
2016-04-21 10:46:35 -07:00
Karthik Kambatla
75cf238354 YARN-4784. Fairscheduler: defaultQueueSchedulingPolicy should not accept FIFO. (Yufei Gu via kasha)
(cherry picked from commit 170c4fd4cdbb8b4c1c4bb219e4c74fa0ba81b055)
2016-04-20 23:58:30 -07:00
Karthik Kambatla
cd74cf9525 YARN-4935. TestYarnClient#testSubmitIncorrectQueue fails when FairScheduler is the default. (Yufei Gu via kasha)
(cherry picked from commit 6e297836d64498d90d0cd343789db2f2d490378a)
2016-04-20 23:41:03 -07:00
Wangda Tan
83a5cdc400 YARN-4890. Unit test intermittent failure: TestNodeLabelContainerAllocation#testQueueUsedCapacitiesUpdate. (Sunil G via wangda)
(cherry picked from commit 33fd95a99c4df6c4de6b0a53dc6679f3cab629e0)
2016-04-20 17:38:22 -07:00