4147 Commits

Author SHA1 Message Date
Yufei Gu
9bb9831347 YARN-2162. Add ability in Fair Scheduler to optionally configure maxResources in terms of percentage. (Yufei Gu) 2017-10-06 00:29:15 -07:00
Haibo Chen
c021c8a19f YARN-7009. TestNMClient.testNMClientNoCleanupOnStop is flaky by design. (Miklos Szegedi via Haibo Chen)
(cherry picked from commit c071aad5da6f6601978e73d38ce5813958848bc4)
2017-10-05 15:51:08 -07:00
Chris Trezzo
e5af16cf6c YARN-2960. Add documentation for the YARN shared cache.
(cherry picked from commit 7e76f85bc68166b01b51fcf6ba4b3fd9281d4a03)
2017-10-05 11:15:39 -07:00
Chris Trezzo
4b9b66f921 Revert "YARN-2960. Add documentation for the YARN shared cache."
This reverts commit 54a01c28cc153872aa7eed68000ab0ddf010054a.
2017-10-05 11:11:02 -07:00
Chris Trezzo
54a01c28cc YARN-2960. Add documentation for the YARN shared cache.
(cherry picked from commit 7e76f85bc68166b01b51fcf6ba4b3fd9281d4a03)
2017-10-05 10:54:44 -07:00
Arun Suresh
b61d12e300 YARN-7258. Add Node and Rack Hints to Opportunistic Scheduler. (Kartheek Muthyala via asuresh).
(cherry picked from commit b733348dde18a242e6c9074c512116a8baf1d281)
2017-10-05 10:06:30 -07:00
Sunil G
807d8b41ad YARN-6182. Fix alignment issues and missing information in new YARN UI's Queue page. Contributed by Akhil PB.
(cherry picked from commit f702c9575495e3af99d4b5cae9f73969e1a6dfc2)
2017-10-05 11:35:36 +05:30
Sunil G
6a152b3e91 YARN-7279. Fix typo in helper message of ContainerLauncher. Contributed by Elek, Marton.
(cherry picked from commit 592bf2d550a07ea5c5df3ba0ab2952c34d941b4b)
2017-10-05 10:52:49 +05:30
Subru Krishnan
829a8e26b9 YARN-5329. Placement Agent enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru).
(cherry picked from commit e6e614e380ed1d746973b50f666a9c40d272073e)
2017-10-04 19:29:03 -07:00
Subru Krishnan
8d59b1fbee YARN-5330. SharingPolicy enhancements required to support recurring reservations in ReservationSystem. (Carlo Curino via Subru).
(cherry picked from commit fa6137501c1499ae33f6e0e2adc31671a7e782dc)
2017-10-04 16:41:18 -07:00
Subru Krishnan
eb0a00c9b5 YARN-5328. Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem.
(cherry picked from commit b6e7d1369690eaf50ce9ea7968f91a72ecb74de0)
2017-10-04 16:41:18 -07:00
Subru Krishnan
b5acbfef48 YARN-2037. Add work preserving restart support for Unmanaged AMs. (Botong Huang via Subru).
(cherry picked from commit d4d2fd1acd2fdddf04f45e67897804eea30d79a1)
2017-10-04 16:33:39 -07:00
Akira Ajisaka
fe0d6f5b1c
YARN-7044. TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails.
(cherry picked from commit 2df1b2ac0509ba10fff606ada7e9b3562c12dd16)
2017-10-05 06:08:05 +09:00
Akira Ajisaka
7ba174791b
YARN-6975. Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router. Contributed by Yeliang Cang.
(cherry picked from commit acf5b880d8283f5a96455c89a6b1b548d0d8c0e7)
2017-10-04 14:54:15 +09:00
Ray Chiang
01b047b41b YARN-7219. Make AllocateRequestProto compatible with branch-2/branch-2.8. (rchiang)
(cherry picked from commit 67e12169b0ae8f3a9edb6511d3ea7ffd57b82e75)
2017-10-03 15:16:31 -07:00
Akira Ajisaka
b658ae002f
YARN-6943. Update Yarn to YARN in documentation. Contributed by Chetna Chaudhari.
(cherry picked from commit c9b525bd943fe7e098e72fb9be5623862c7ab5ff)
2017-10-04 07:02:31 +09:00
Akira Ajisaka
2fa1d39952
YARN-6916. Moving logging APIs over to slf4j in hadoop-yarn-server-common. Contributed by Bibin A Chundatt and Akira Ajisaka.
(cherry picked from commit 4a877737182808fe3392a116762ba59973a477a6)
2017-10-04 06:08:00 +09:00
Jian He
d2c3b93ee5 YARN-7269. Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications. Contributed by Wangda Tan
(cherry picked from commit 4111e6c7819d66e17e0932302c589777a355dbae)
2017-10-03 12:23:03 -07:00
Sidharta S
dcc454b4a0 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe
(cherry picked from commit 7eb8499996869cdb63743f1c9eca2ba91d57ad08)
2017-10-02 19:32:19 -07:00
Wangda Tan
c2f751cb0f YARN-7259. Add size-based rolling policy to LogAggregationIndexedFileController. (xgong via wangda)
Change-Id: Ifaf82c0aee6b73b9b6ebf103aa72e131e3942f31
(cherry picked from commit 280080fad01304c85a9ede4d4f7b707eb36c0155)
2017-10-02 16:37:27 -07:00
Junping Du
aef3e4b6af YARN-7194. Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat. Contributed by Xuan Gong.
(cherry picked from commit c92c1d521eadfd8a4cd8205cc6aee74816f353f4)
2017-10-02 16:35:27 -07:00
Wangda Tan
d58d75ca50 YARN-6509. Add a size threshold beyond which yarn logs will require a force option. (Xuan Gong via wangda)
Change-Id: I755fe903337d4ff9ec35dae5b9cce638794e1d0f
(cherry picked from commit ec2ae3060a807c8754826af2135a68c08b2e4f13)
2017-09-29 16:49:59 -07:00
Junping Du
c3eade44d0 YARN-7072. Add a new log aggregation file format controller. Contributed by Xuan Gong.
(cherry picked from commit 3fddabc2fe4fbdb8ef3f9ce7558955c4f0794dcc)
2017-09-29 16:49:44 -07:00
Wangda Tan
cdaab892d8 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda)
Change-Id: I0ee0b1bb459437432a22cf68861a6354f0decabb
(cherry picked from commit febeead5f95c6fc245ea3735f5b538d4bb4dc8a4)
2017-09-29 16:46:34 -07:00
Eric Payne
7a9479597f YARN-7084. TestSchedulingMonitor#testRMStarts fails sporadically. Contributed by Jason Lowe.
(cherry picked from commit 7f6118f918e0fbe81cb64e8b9f6959b8d66679f6)
2017-09-29 10:39:01 -05:00
Wangda Tan
091fc32ce8 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda)
Change-Id: I76aec68cbfb42d239279d7245a03290d691e87a4
(cherry picked from commit d3b1c6319546706c41a2011ead6c3fe208883200)
2017-09-28 16:46:23 -07:00
Chris Trezzo
3d23522110 YARN-7250. Update Shared cache client api to use URLs.
(cherry picked from commit c114da5e64d14b1d9e614081c4171ea0391cb1aa)
2017-09-28 15:31:23 -07:00
Jason Lowe
a1e9739730 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh
(cherry picked from commit 85d81ae58ec4361a944c84753a900460a0888b9b)
2017-09-28 14:14:14 -05:00
Andrew Wang
5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Arun Suresh
59453dad8c YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh)
(cherry picked from commit df800f6cf3ea663daf4081ebe784808b08d9366d)
2017-09-28 11:29:26 -07:00
Arun Suresh
ed855e2eef YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh)
(cherry picked from commit 66ca0a65408521d5f9b080dd16b353b49fb8eaea)
2017-09-28 11:27:43 -07:00
Arun Suresh
8eebee5094 YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh)
(cherry picked from commit 4f8194430fc6a69d9cc99b78828fd7045d5683e8)
2017-09-28 11:24:20 -07:00
Arun Suresh
dd51a74601 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh)
(cherry picked from commit 864fbacd4548004b1de8b0812627976acd22aff5)
2017-09-28 11:22:58 -07:00
Daniel Templeton
7b49de5e30 YARN-6622. Document Docker work as experimental (Contributed by Varun Vasudev)
(cherry picked from commit 6651cbcc72d92caf86b744fa76fba513b36b12c7)
2017-09-28 10:16:45 -07:00
Xuan
81e7872793 YARN-7257. AggregatedLogsBlock reports a bad 'end' value as a bad
'start' value. Contributed by Jason Lowe

(cherry picked from commit 28c4957fccebe2d7e63ec9fe9af58313b4f21d4f)
2017-09-27 15:47:45 -07:00
Chris Trezzo
ad7ed9d044 YARN-7253. Shared Cache Manager daemon command listed as admin subcmd in yarn script.
(cherry picked from commit c87db8d154ab2501e786b4f1669b205759ece5c3)
2017-09-27 11:36:42 -07:00
Jason Lowe
8526bfa19d Revert "YARN-6570. No logs were found for running application, running"
This reverts commit e91447ccc9ff438dd300e2564c251b2c4b0f369a.
2017-09-26 10:41:44 -05:00
Rohith Sharma K S
52571d9fa3 YARN-65. Reduce RM app memory footprint once app has completed. Contributed by Manikandan R.
(cherry picked from commit 06e5a7b5cf141420d3a411088b87acba72e68cad)
2017-09-26 14:46:47 +05:30
Akira Ajisaka
149d3ad6ba
YARN-7153. Remove duplicated code in AMRMClientAsyncImpl.java. Contributed by Sen Zhao.
(cherry picked from commit e928ee583c5a1367e24eab34057f8d8496891452)
2017-09-26 02:04:23 +09:00
Junping Du
b58cc16171 YARN-7118. AHS REST API can return NullPointerException. Contributed by Billie Rinaldi.
(cherry picked from commit 4002bf0a9e7e3619a3bdcff071f0369cbf2873ad)
2017-09-22 14:00:40 -07:00
Jason Lowe
a294bfd80e YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger.
(cherry picked from commit bfd1a72ba8fbb06da73fede2a85e0b544d6ab43f)
2017-09-21 17:43:51 -05:00
Junping Du
9e3313f8a1 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi.
(cherry picked from commit e5e1851d803bf8d8b96fec1b5c0058014e9329d0)
2017-09-21 14:06:45 -07:00
Wangda Tan
4dbf7a501a YARN-6991. "Kill application" button does not show error if other user tries to kill the application for secure cluster. (Suma Shivaprasad via wangda)
Change-Id: I7e7894b24609709f89064ee5882f055dbb09080b
(cherry picked from commit 263e2c692a4b0013766cd1f6b6d7ed674b2b1040)
2017-09-21 12:27:18 -07:00
Jason Lowe
1e55972273 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger
(cherry picked from commit 10d7493587643b52cee5fde87eca9ef99c422a70)
2017-09-21 08:44:16 -05:00
Sangjin Lee
f919bcadb7 YARN-6771. Use classloader inside configuration class to make new
classes. Contributed by Jongyoul Lee.

(cherry picked from commit 8b336632acad10e45d029596c5e3196e1857d891)
2017-09-20 22:54:30 -07:00
Junping Du
974605c025 YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh.
(cherry picked from commit 647b7527a9cdf4717e7dcbbb660e5812b67a17f1)
2017-09-19 18:31:54 -07:00
Arun Suresh
2ccfe92a4e YARN-7203. Add container ExecutionType into ContainerReport. (Botong Huang via asuresh)
(cherry picked from commit 56ef5279c1db93d03b2f1e04badbfe804f548918)
2017-09-18 15:50:39 -07:00
Andrew Wang
531290c2c8 HADOOP-14835. mvn site build throws SAX errors. Contributed by Andrew Wang and Sean Mackrory.
(cherry picked from commit 3cf3540f19b5fd1a174690db9f1b7be2977d96ba)
2017-09-18 15:13:45 -07:00
Xuan
e91447ccc9 YARN-6570. No logs were found for running application, running
container. Contributed by Junping Du

(cherry picked from commit 7c732924a889cd280e972882619a1827877fbafa)
2017-09-18 14:05:02 -07:00
Jason Lowe
7e72be9456 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh
(cherry picked from commit a4f9c7c9247801dd37beec6fc195622af1b884ad)
2017-09-18 10:23:22 -05:00