Commit Graph

868 Commits

Author SHA1 Message Date
Miklos Szegedi 86a2ac94b1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-17 12:56:10 -08:00
Miklos Szegedi 09428b137f YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu.
(cherry picked from commit 2dcfc18)
2018-01-12 17:44:31 -08:00
Miklos Szegedi 35527db157 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang.
(cherry picked from commit bc285da)
2018-01-11 10:30:15 -08:00
Jason Lowe 42964c2a66 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:21:40 -06:00
Jason Lowe 31a06baca0 YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang
(cherry picked from commit 811fabdebe)
2017-12-18 15:27:27 -06:00
Jason Lowe 0f8de8cd91 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-15 15:54:56 -06:00
Weiwei Yang d69b7358b6 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang.
(cherry picked from commit 7efc4f7688)
2017-12-13 13:28:48 +08:00
Subru Krishnan 850bd0ed7c YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru).
(cherry picked from commit 670e8d4ec7)
2017-12-08 15:44:45 -08:00
Weiwei Yang 029714e3ec YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-08 13:53:33 +08:00
Jason Lowe 3fb83cbeb2 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan
(cherry picked from commit 60f95fb719)
2017-12-01 15:51:49 -06:00
Yufei Gu de144175a4 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu)
(cherry picked from commit d8923cdbf1)
(cherry picked from commit c0ab3c8407)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
2017-11-27 14:54:27 -08:00
Subru Krishnan ed31091361 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-20 14:21:58 -08:00
Junping Du a72dcb9cad YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 18:22:30 -08:00
Arun Suresh c153bed6e3 YARN-5049. Addendum patch for branch-2. (asuresh) 2017-11-12 10:16:00 -08:00
Eric Yang 90ac8cd5a1 YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger)
(cherry picked from commit 7a49ddfdde)
2017-11-12 09:59:39 -08:00
Subru Krishnan 0282efab14 YARN-7476. Fix miscellaneous issues in ATSv2 after merge to branch-2. (Varun Saxena via Subru). 2017-11-12 09:19:30 -08:00
bibinchundatt fd4868bdb4 Add containerId to Localizer failed logs. Contributed by Prabhu Joseph
(cherry picked from commit dcd99c4b9a)
2017-11-06 23:09:41 +05:30
Jason Lowe 6ad6882343 YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger
(cherry picked from commit b7dee1f060)
2017-11-02 09:50:55 -05:00
Arun Suresh 71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Arun Suresh 2654d9973e YARN-7299. Fix TestDistributedScheduler. (asuresh)
(cherry picked from commit 9c5c68745e)
2017-10-27 23:11:34 -07:00
Jason Lowe 9093ad6168 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla
(cherry picked from commit 665bb147aa)
2017-10-27 17:09:29 -05:00
Eric Yang f6d3c101a6 YARN-7353. Improved volume mount check for directories and unit test compatibility on RHEL7. Contributed by Eric Badger. 2017-10-24 17:01:08 -04:00
Yufei Gu c3946f29e7 YARN-7261. Add debug message for better download latency monitoring. (Yufei Gu)
(cherry picked from commit 0799fde35e)
2017-10-20 10:02:35 -07:00
Wangda Tan 2e3b7130c5 YARN-6623. Add support to turn off launching privileged containers in the container-executor. (Varun Vasudev via wangda) 2017-10-19 15:11:05 -07:00
Varun Saxena 56e0873e22 Fix javac, javadoc and checkstyle issues before branch-2 merge. 2017-10-17 23:36:41 +05:30
Haibo Chen 0f51f15097 YARN-6323. Rolling upgrade/config change is broken on timeline v2. (Vrushali C via Haibo Chen) 2017-10-17 23:36:40 +05:30
Jian He cbd5e89b24 MAPREDUCE-6838. [ATSv2 Security] Add timeline delegation token received in allocate response to UGI. Contributed by Varun Saxena
(cherry picked from commit c6ce422a3423305ac665f80f039f61294adaa32c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
2017-10-17 23:36:40 +05:30
Jian He 53ddf1b15f YARN-7038. Addendum patch. Fix compilation error. Contributed by Varun Saxena 2017-10-17 23:36:39 +05:30
Rohith Sharma K S 4e1af68ae1 YARN-7038. [Atsv2 Security] CollectorNodemanagerProtocol RPC interface doesn't work when service authorization is enabled. Contributed by Varun Saxena.
(cherry picked from commit 387dd3da87759860dd18bd8aa95c60d962b1e6a9)
2017-10-17 23:36:39 +05:30
Jian He 55e5742394 YARN-7006. [ATSv2 Security] Changes for authentication for CollectorNodemanagerProtocol. Contributed by Varun Saxena 2017-10-17 23:36:39 +05:30
Rohith Sharma K S 5a7c272e0a YARN-6130. [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM. Contributed by Varun Saxena. 2017-10-17 23:36:39 +05:30
Haibo Chen d53038626e YARN-6555. Store application flow context in NM state store for work-preserving restart. (Rohith Sharma K S via Haibo Chen)
(cherry picked from commit 47474fffac)
(cherry picked from commit 8817cb5c8424359b880c6d700e53092f0269c1bb)
2017-10-17 23:36:38 +05:30
Varun Saxena 16d8df68f6 YARN-6377. NMTimelinePublisher#serviceStop does not stop timeline clients (Haibo Chen via Varun Saxena)
(cherry picked from commit a4b5aa8493)
2017-10-17 23:36:37 +05:30
Sangjin Lee bb100d35fd YARN-4675. Reorganize TimelineClient and TimelineClientImpl into separate classes for ATSv1.x and ATSv2. Contributed by Naganarasimha G R.
(cherry picked from commit 73235ab30361b41293846189f3c5fef321ae7cac)
2017-10-17 23:36:36 +05:30
Sangjin Lee b8cfb4fcb3 YARN-5792. Adopt the id prefix for YARN, MR, and DS entities. Contributed by Varun Saxena.
(cherry picked from commit f734977b27a514ce0561638c0a6a17b1ef093026)
2017-10-17 23:36:35 +05:30
Varun Saxena 684c2d149f YARN-3359. Addendum for Recover collector list when RM fails over (Li Lu via Varun Saxena)
(cherry picked from commit 25b19178dd53bc0d57ca5fd64d3464af7b59e588)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
2017-10-17 23:36:35 +05:30
Varun Saxena 6562a354a9 YARN-3359. Recover collector list when RM fails over (Li Lu via Varun Saxena)
(cherry picked from commit 1c8a57550b)
2017-10-17 23:36:34 +05:30
Sangjin Lee 853a5da041 YARN-5699. Retrospect yarn entity fields which are publishing in events info fields. Contributed by Rohith Sharma K S.
(cherry picked from commit 787d5942003a5441ba455a79fa08da28f60d5777)
2017-10-17 23:36:34 +05:30
Sangjin Lee 4675f22e6a YARN-5638. Introduce a collector timestamp to uniquely identify collectors creation order in collector discovery. Contributed by Li Lu.
(cherry picked from commit 0c1863144649ea265da65ce25158707cc3a3fb4a)
2017-10-17 23:36:34 +05:30
Sangjin Lee 51102019f5 YARN-5156. YARN_CONTAINER_FINISHED of YARN_CONTAINERs will always have running state (Vrushali C via sjlee)
(cherry picked from commit d0a62d8adf86f7cb2481940d212788cc5f327367)
2017-10-17 23:36:34 +05:30
Sangjin Lee e212e562ec YARN-5355. Backported YARN-2928 into our branch-2 feature branch. 2017-10-17 23:36:33 +05:30
Nathan Roberts b876a9314e YARN-7333. container-executor fails to remove entries from a directory that is not writable or executable. Contributed by Jason Lowe.
(cherry picked from commit 8620140a6a3ec0117675ede06d92d830da3da551)
2017-10-16 16:44:34 -05:00
Arun Suresh 53cd33cc4a YARN-7275. NM Statestore cleanup for Container updates. (Kartheek Muthyala via asuresh)
(cherry picked from commit a50be1b8f4)
2017-10-16 13:27:08 -07:00
Wangda Tan 2c1381300f YARN-7321. Backport container-executor changes from YARN-6852 to branch-2. (Varun Vasudev via wangda)
Change-Id: Ib6434b5a5ebe591050c3949cac12ff9e2a2bbaa5
2017-10-14 10:48:26 -07:00
Wangda Tan 1ab6b4a08e YARN-6033. Add support for sections in container-executor configuration file. (Varun Vasudev via wangda)
Change-Id: Id1ded3ee11e6242dfed5616903580f329f3bb26a
2017-10-12 09:14:08 -07:00
Naganarasimha 5cb3bd3b06 YARN-7285. ContainerExecutor always launches with priorities due to yarn-default property. Contributed by Jason Lowe.
(cherry picked from commit d0046bb5c6)
2017-10-06 00:09:46 +05:30
Sidharta S 3ecbfff937 YARN-7226. Whitelisted variables do not support delayed variable expansion. Contributed by Jason Lowe 2017-10-04 14:05:01 -07:00
Wangda Tan 598329e6f8 YARN-6550. Capture launch_container.sh logs to a separate log file. (Suma Shivaprasad via wangda)
Change-Id: I66c9b50ae3955dda0b5bf0674a9bfb728db27acb
2017-10-03 09:46:35 -07:00
Subru Krishnan ffcf5ba1ce YARN-6962. Add support for updateContainers when allocating using FederationInterceptor. (Botong Huang via Subru).
(cherry picked from commit ca669f9f8b)
2017-09-28 13:11:32 -07:00
Subru Krishnan d6da014f67 YARN-6955. Handle concurrent register AM requests in FederationInterceptor. (Botong Huang via Subru).
(cherry picked from commit c61f2c4198)
2017-09-28 13:11:19 -07:00
Jason Lowe 913a64e4c9 YARN-7248. NM returns new SCHEDULED container status to older clients. Contributed by Arun Suresh
(cherry picked from commit 85d81ae58e)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
2017-09-28 14:25:01 -05:00
Jason Lowe c4de765fcb Revert "YARN-6570. No logs were found for running application, running"
This reverts commit 6aaca3d08e.
2017-09-26 10:44:16 -05:00
Arun Suresh e7199cb494 YARN-7240. Add more states and transitions to stabilize the NM Container state machine. (Kartheek Muthyala via asuresh)
(cherry picked from commit df800f6cf3)
2017-09-25 16:36:58 -07:00
Subru Krishnan 35a38330ef YARN-6511. Federation: transparently spanning application across multiple sub-clusters. (Botong Huang via Subru).
(cherry picked from commit 8c988d235e)
(cherry picked from commit 70b1a757f1)
2017-09-21 16:49:55 -07:00
Subru Krishnan 7ede8c1a53 YARN-3666. Federation Intercepting and propagating AM- home RM communications. (Botong Huang via Subru).
(cherry picked from commit 2399eb8200)
(cherry picked from commit bed1832c93)
2017-09-21 16:49:54 -07:00
Subru Krishnan 9476d86ce8 YARN-5531. UnmanagedAM pool manager for federating application across clusters. (Botong Huang via Subru).
(cherry picked from commit 73bb2102ce)
(cherry picked from commit 859aa1f9d6)
2017-09-21 16:47:43 -07:00
Subru Krishnan dc0a2e6c59 YARN-5411. Create a proxy chain for ApplicationClientProtocol in the Router. (Giovanni Matteo Fumarola via Subru).
(cherry picked from commit 4846069061)
(cherry picked from commit 80e1904000)
2017-09-21 16:43:08 -07:00
Jason Lowe fc6d9612c7 YARN-4266. Allow users to enter containers as UID:GID pair instead of by username. Contributed by luhuichun, Zhankun Tang, and Eric Badger.
(cherry picked from commit bfd1a72ba8)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
2017-09-21 17:52:11 -05:00
Junping Du e67c8347c4 YARN-7034. DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor. Contributed by Miklos Szegedi. 2017-09-21 14:12:57 -07:00
Jason Lowe ae42bf73c0 YARN-6968. Hardcoded absolute pathname in DockerLinuxContainerRuntime. Contributed by Eric Badger
(cherry picked from commit 10d7493587)
2017-09-21 08:39:31 -05:00
Junping Du bdd8433d4c YARN-7186. Fix finicky TestContainerManager tests. Contributed by Arun Suresh.
(cherry picked from commit 647b7527a9)
2017-09-19 18:36:25 -07:00
Xuan 6aaca3d08e YARN-6570. No logs were found for running application, running
container. Contributed by Junping Du

(cherry picked from commit 7c732924a8)
(cherry picked from commit e91447ccc9)
2017-09-18 14:05:36 -07:00
Jason Lowe cd6cf0caf6 YARN-7192. Add a pluggable StateMachine Listener that is notified of NM Container State changes. Contributed by Arun Suresh
(cherry picked from commit a4f9c7c924)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/ContainerImpl.java
2017-09-18 10:34:53 -05:00
Arun Suresh b93a23f411 YARN-6059. Update paused container state in the NM state store. (Hitesh Sharma via asuresh)
(cherry picked from commit 66ca0a6540)
2017-09-14 10:30:30 -07:00
Arun Suresh 52bf458616 YARN-5216. Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM. (Hitesh Sharma via asuresh)
(cherry picked from commit 4f8194430f)
2017-09-14 10:29:54 -07:00
Arun Suresh 1af9f962c9 YARN-5292. NM Container lifecycle and state transitions to support for PAUSED container state. (Hitesh Sharma via asuresh)
(cherry picked from commit 864fbacd45)
2017-09-14 10:26:34 -07:00
Eric Payne fb3d53e001 YARN-4727. Unable to override the $HADOOP_CONF_DIR env variable for container. Contributed by Jason Lowe.
(cherry picked from commit 3860be7961)
2017-09-13 15:07:27 -05:00
Nathan Roberts 1d33290a0a YARN-7014. Fix off-by-one error causing heap corruption (Jason Lowe via nroberts)
(cherry picked from commit d265459024)
2017-09-13 09:47:55 -07:00
Varun Vasudev 4647761341 YARN-5719. Enforce a C standard for native container-executor. Contributed by Chris Douglas.
(cherry picked from commit 972da46cb4)
2017-09-13 18:15:02 +05:30
Arun Suresh 0d32a38ac3 YARN-7185. [Addendum patch] Minor javadoc and checkstyle fix.
(cherry picked from commit a1c398e0dc)
2017-09-12 17:05:01 -07:00
Arun Suresh d735c1f26b YARN-7185. ContainerScheduler should only look at availableResource for GUARANTEED containers when OPPORTUNISTIC container queuing is enabled. (Wangda Tan via asuresh)
(cherry picked from commit 2ae72692fc)
2017-09-12 16:33:05 -07:00
Arun Suresh 7803a93fff YARN-6706. Refactor ContainerScheduler to make oversubscription change easier. (Haibo Chen via asuresh)
(cherry picked from commit 5b007921cd)
2017-09-12 16:33:05 -07:00
Wangda Tan c093e4622c YARN-6726. Fix issues with docker commands executed by container-executor. (Shane Kumpf via wangda)
Change-Id: I8fa7d70526505d4082ac9b51cd6066376a599108
2017-09-11 11:34:51 -07:00
Yufei Gu 416a44004f YARN-6799. Remove the duplicated code in CGroupsHandlerImp.java. (Contributed by Weiyuan via Yufei Gu)
(cherry picked from commit 5ab21dfe9c)
2017-09-10 23:24:14 -07:00
Xuan 02ae3f8202 YARN-7072. Add a new log aggregation file format controller (xgong) 2017-09-10 23:18:31 -07:00
Nathan Roberts b3a478832a YARN-6219. Fix conf usage in NMWebServer unit tests. Contributed by Jason Lowe. 2017-09-08 16:01:06 -05:00
Arun Suresh 2b09cec218 YARN-6849. NMContainerStatus should have the Container ExecutionType. (Kartheek Muthyala via asuresh)
(cherry picked from commit 1f53ae7972)
2017-09-08 09:26:50 -07:00
Jason Lowe c004d57d47 YARN-6930. Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager. Contributed by Shane Kumpf 2017-09-07 16:24:55 -05:00
Junping Du 119220b88f YARN-6877. Create an abstract log reader for extendability. Contributed by Xuan Gong. 2017-09-01 03:04:55 -07:00
Wangda Tan 9bcd64d233 YARN-5219. When an export var command fails in launch_container.sh, the full container launch should fail. (Sunil G via wangda)
Change-Id: Iaa6b978bb89482e9d1d77ba57f4adfdc48e39a3c
(cherry picked from commit f59332b97b)
2017-08-29 15:29:53 -07:00
Junping Du b906bddd95 YARN-7037. Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices. Contributed by Tao Yang.
(cherry picked from commit ad45d19998)
2017-08-29 15:17:21 -07:00
Yufei Gu 221c1bdf7e YARN-7099. ResourceHandlerModule.parseConfiguredCGroupPath only works for privileged yarn users. (Contributed by Miklos Szegedi via Yufei Gu)
(cherry picked from commit d5bc958279)
2017-08-28 10:11:01 -07:00
Li Lu b7f319c53c YARN-6999. Add log about how to solve Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster. (Linlin Zhou via gtcarrera9)
(cherry picked from commit 3bee7046c5)
2017-08-25 22:45:20 -07:00
Eric Payne 1617744801 YARN-7087. NM failed to perform log aggregation due to absent container. Contributed by Jason Lowe.
(cherry picked from commit e864f81471)
2017-08-25 15:52:29 -05:00
Junping Du e12358c5ce YARN-6876. Create an abstract log writer for extendability. Contributed by Xuan Gong.
(cherry picked from commit c2cb7ea1ef)
2017-08-24 15:27:40 -07:00
Akira Ajisaka bf903396a0
YARN-7047. Moving logging APIs over to slf4j in hadoop-yarn-server-nodemanager. Contributed by Yeliang Cang. 2017-08-25 03:28:02 +09:00
bibinchundatt cc50ca0722 YARN-7074. Fix NM state store update comment. Contributed by Botong Huang
(cherry picked from commit de0cba700b)
2017-08-24 16:40:18 +05:30
Ray Chiang 2299c8dffa YARN-6798. Fix NM startup failure with old state store due to version mismatch. (Botong Huang via rchiang)
(cherry picked from commit f5f14a2ad6)
2017-08-22 15:24:02 -07:00
Jason Lowe b15a393440 YARN-6988. container-executor fails for docker when command length > 4096 B. Contributed by Eric Badger 2017-08-22 16:28:08 -05:00
Arun Suresh e0ba8a9ed8 YARN-6979. [Addendum patch] Fixed classname and added javadocs. (Kartheek Muthyala via asuresh)
(cherry picked from commit 7a82d7bcea)
2017-08-20 10:28:42 -07:00
Arun Suresh f8f193c8e8 YARN-6979. Add flag to notify all types of container updates to NM via NodeHeartbeatResponse. (Kartheek Muthyala via asuresh)
(cherry picked from commit 8410d862d3)
2017-08-20 08:31:17 -07:00
Sunil G 5511cea86b YARN-3254. HealthReport should include disk full information. Contributed by Suma Shivaprasad.
(cherry picked from commit f9a0e23381)
2017-08-17 15:10:07 +05:30
Arun Suresh 7b22df3da6 YARN-5978. ContainerScheduler and ContainerManager changes to support ExecType update. (Kartheek Muthyala via asuresh)
(cherry picked from commit 4d7be1d857)
2017-08-14 19:57:34 -07:00
Karthik Kambatla f6c74ac5ea YARN-5927. BaseContainerManagerTest::waitForNMContainerState timeout accounting is not accurate. (Kai Sasaki via kasha)
(cherry picked from commit 8c4b6d16a5)
2017-08-11 12:18:29 -07:00
Jian He 8b7590057e YARN-6804. Allow custom hostname for docker containers in native services. Contributed by Billie Rinaldi 2017-08-09 11:12:38 -07:00
Daniel Templeton 675d368950 YARN-6757. Refactor the usage of yarn.nodemanager.linux-container-executor.cgroups.mount-path
(Contributed by Miklos Szegedi via Daniel Templeton)

(cherry picked from commit 47b145b9b4)
2017-08-08 11:12:34 -07:00
Arun Suresh 3f50800d41 YARN-6920. Fix resource leak that happens during container re-initialization. (asuresh)
(cherry picked from commit 8d3fd81980)
2017-08-07 19:01:13 -07:00
Sunil G d15bac141e YARN-6951. Fix debug log when Resource Handler chain is enabled. Contributed by Yang Wang.
(cherry picked from commit 46b7054fa7)
2017-08-07 13:17:04 +05:30
Arun Suresh 88d6adf419 YARN-5977. ContainerManagementProtocol changes to support change of container ExecutionType. (Kartheek Muthyala via asuresh)
(cherry picked from commit 35dc782923)
2017-08-03 21:18:43 -07:00
Yufei Gu ce609d06c0 YARN-6832. Tests use assertTrue(....equals(...)) instead of assertEquals()). (Daniel Templeton via Yufei Gu) 2017-08-03 13:52:58 -07:00