Commit Graph

567 Commits

Author SHA1 Message Date
Varun Vasudev 78919f8c34 YARN-4245. Generalize config file handling in container-executor. Contributed by Sidharta Seethana.
(cherry picked from commit 8ed2e060e8)
2016-03-07 16:19:27 +05:30
Varun Vasudev e9a0ffc7f1 YARN-4737. Add CSRF filter support in YARN. Contributed by Jonathan Maron.
(cherry picked from commit 43416187c07afb35e3267f94d0a41d8d3cfb5735)
2016-03-07 15:23:36 +05:30
Haohui Mai c5db4ab0b4 HADOOP-12813. Migrate TestRPC and related codes to rebase on ProtobufRpcEngine. Contributed by Kai Zheng. 2016-02-29 14:10:18 -08:00
Jason Lowe bd0f5085e3 YARN-4731. container-executor should not follow symlinks in recursive_unlink_children. Contributed by Colin Patrick McCabe
(cherry picked from commit c58a6d53c5)
2016-02-29 15:26:26 +00:00
Ming Ma 1656bcec5f YARN-4720. Skip unnecessary NN operations in log aggregation. (Jun Gong via mingma)
(cherry picked from commit 7f3139e54d)
2016-02-26 08:43:14 -08:00
Robert Kanter 872b8d90a6 YARN-4579. Allow DefaultContainerExecutor container log directory permissions to be configurable (rchiang via rkanter)
(cherry picked from commit d7fdec1e6b)
2016-02-25 16:40:05 -08:00
Robert Kanter c2098d2470 YARN-4697. NM aggregation thread pool is not bound by limits (haibochen via rkanter)
(cherry picked from commit 954dd57043)
2016-02-24 15:00:48 -08:00
Jason Lowe acffe82353 YARN-2046. Out of band heartbeats are sent only on container kill and possibly too early. Contributed by Ming Ma
(cherry picked from commit d284e187b8)
2016-02-23 20:51:01 +00:00
Varun Vasudev 2c218ca8a8 YARN-4709. NMWebServices produces incorrect JSON for containers. Contributed by Varun Saxena.
(cherry picked from commit 140cb5d745)
2016-02-23 12:32:16 +05:30
Varun Vasudev 3a5b39e931 YARN-4655. Log uncaught exceptions/errors in various thread pools in YARN. Contributed by Sidharta Seethana.
(cherry picked from commit fa00d3e205)
2016-02-11 12:09:29 +05:30
Jason Lowe b12f16ab73 YARN-4594. container-executor fails to remove directory tree when chmod required. Contributed by Colin Patrick McCabe
(cherry picked from commit fa328e2d39)
2016-02-03 17:22:12 +00:00
Varun Vasudev bbda446889 YARN-4649. Add additional logging to some NM state store operations. Contributed by Sidharta Seethana
(cherry picked from commit 1cd55e0c17)
2016-02-02 09:26:22 +05:30
Wangda Tan 9591363ad8 YARN-4340. Add list API to reservation system. (Sean Po via wangda)
(cherry picked from commit 9875325d5c)
2016-02-02 10:19:19 +08:00
Rohith Sharma K S b8d5df06f2 YARN-4543. Fix random test failure in TestNodeStatusUpdater.testStopReentrant. (Akihiro Suda via rohithsharmaks)
(cherry picked from commit ac68666803)
2016-01-29 12:30:53 +05:30
Jason Lowe 29195c7668 YARN-4643. Container recovery is broken with delegating container runtime. Contributed by Sidharta Seethana
(cherry picked from commit 61382ff8fa)
2016-01-28 19:00:51 +00:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 47b20d0c74 YARN-3542. Refactored existing CPU cgroups support to use the newer and integrated ResourceHandler mechanism, and also deprecated the old LCEResourceHandler inteface hierarchy. Contributed by Varun Vasudev.
(cherry picked from commit 2085e60a96)
2016-01-25 16:22:47 -08:00
Jian He 2d236c9de2 YARN-4520. Finished app info is unnecessarily persisted in NM state-store if container is acquired but not lunched on this node. Contributed by sandflee 2016-01-25 15:36:40 -08:00
Xuan 1d19557dda YARN-4496. Improve HA ResourceManager Failover detection on the client.
Contributed by Jian He

(cherry picked from commit 618bfd6ac2)
2016-01-22 18:21:50 -08:00
Jason Lowe 635849fa45 YARN-4598. Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL. Contributed by tangshangwen
(cherry picked from commit 46e5ea81e0)
2016-01-22 21:55:59 +00:00
Varun Vasudev 85561207f9 YARN-4578. Directories that are mounted in docker containers need to be more restrictive/container-specific. Contributed by Sidharta Seethana.
(cherry picked from commit b41a7e89d1)
2016-01-22 14:44:22 +05:30
Wangda Tan 23deeb4230 HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda)
(cherry picked from commit 89d1fd5dac)
2016-01-19 21:28:51 +08:00
Karthik Kambatla ef59521094 YARN-4526. Make SystemClock singleton so AppSchedulingInfo could use it. (kasha)
(cherry picked from commit d40859fab1)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapred/TestTaskAttemptListenerImpl.java
2016-01-18 10:59:34 +01:00
Colin Patrick Mccabe 48b8c99479 HADOOP-12712. Fix some cmake plugin and native build warnings (cmccabe)
(cherry picked from commit b2c155f810)
2016-01-15 13:22:14 -08:00
Colin Patrick Mccabe cea1c71087 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe)
(cherry picked from commit b1ed28fa77)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
2016-01-14 11:13:13 -08:00
Varun Vasudev 6220a024b1 YARN-4553. Add cgroups support for docker containers. Contributed by Sidharta Seethana.
(cherry picked from commit 3ddb92bd30)
2016-01-14 14:30:19 +05:30
Jason Lowe d164cdcda1 YARN-4414. Nodemanager connection errors are retried at multiple levels. Contributed by Chang Li
(cherry picked from commit 13de8359a1)
2016-01-12 15:57:07 +00:00
Akira Ajisaka 3905435d3d YARN-4544. All the log messages about rolling monitoring interval are shown with WARN level. Contributed by Takashi Ohnishi.
(cherry picked from commit 52b77577c4)
2016-01-08 01:13:30 +09:00
Steve Loughran d9459ffa00 YARN-4550. Some tests in TestContainerLanch fails on non-english locale environment. (Takashi Ohnishi via stevel) 2016-01-07 14:30:07 +00:00
rohithsharmaks 4865642e7c YARN-4393. Fix intermittent test failure for TestResourceLocalizationService#testFailedDirsResourceRelease (Varun Saxana via rohithsharmaks)
(cherry picked from commit 791c1639ae)
2016-01-07 09:39:37 +05:30
Rohith Sharma K S ae25e283cf YARN-4530. LocalizedResource trigger a NPE Cause the NodeManager exit. (tangshangwen via rohithsharmaks)
(cherry picked from commit f9e36dea96)
2016-01-04 10:04:27 +05:30
Gera Shegalov 8baf9369bb YARN-2934. Improve handling of container's stderr. (Naganarasimha G R via gera)
(cherry picked from commit 2c17b81569)
2015-12-24 23:52:43 -08:00
Uma Mahesh fc9e130825 YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh)
(cherry picked from commit 0f82b5d878)
2015-12-19 23:17:33 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) d9b9a75928 YARN-1856. Added cgroups based memory monitoring for containers as another alternative to custom memory-monitoring. Contributed by Varun Vasudev.
(cherry picked from commit 4e7d32c0db)
2015-12-17 12:13:56 -08:00
Wangda Tan 9f4a6ec663 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda)
(cherry picked from commit 79c41b1d83)
2015-12-16 13:28:56 -08:00
Jian He 72544db3cd YARN-4402. TestNodeManagerShutdown And TestNodeManagerResync fails with bind exception. Contributed by Brahma Reddy Battula 2015-12-14 14:59:18 -08:00
Wangda Tan 9f979107c2 YARN-4309. Add container launch related debug information to container logs when a container fails. (Varun Vasudev via wangda)
(cherry picked from commit dfcbbddb09)
2015-12-14 11:13:51 -08:00
rohithsharmaks f626d18cc3 YARN-4431. Not necessary to do unRegisterNM() if NM get stop due to failed to connect to RM. (Junpin Du via rohithsharmaks)
(cherry picked from commit 15c3e7ffe3)
2015-12-09 10:51:57 +05:30
Steve Loughran af14458de7 HADOOP-12321. Make JvmPauseMonitor an AbstractService. (Sunil G via Stevel) [includes HDFS-8947 MAPREDUCE-6462 and YARN-4072] 2015-12-06 17:42:56 +00:00
Jian He 23fe1abc02 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999). Contributed by Mohammad Shahid Khan and Varun Saxena
(cherry picked from commit 9f77ccad73)
2015-12-03 12:49:22 -08:00
Junping Du e76ba91fa5 YARN-4408. Fix issue that NodeManager still reports negative running containers. Contributed by Robert Kanter.
(cherry picked from commit 62e9348bc1)
2015-12-03 06:37:31 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Tsuyoshi Ozawa d76b523b02 YARN-4380. TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently. Contributed by Varun Saxena.
(cherry picked from commit 0656d2dc83)
2015-11-26 01:10:41 +09:00
Jason Lowe 9e54433c6c YARN-4132. Separate configs for nodemanager to resourcemanager connection timeout and retries. Contributed by Chang Li
(cherry picked from commit 4ac6799d4a)
2015-11-24 22:36:19 +00:00
Karthik Kambatla 49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha)
(cherry picked from commit 52948bb20b)
2015-11-24 13:52:29 +05:30
Jian He 2cf084ac76 Revert "YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan"
This reverts commit a4ff03b200.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
2015-11-16 20:19:31 -08:00
Junping Du 1d107d8058 YARN-4354. Public resource localization fails with NPE. Contributed by Jason Lowe.
(cherry picked from commit 855d52927b)
2015-11-15 04:44:59 -08:00
Jian He a4ff03b200 YARN-3840. Resource Manager web ui issue when sorting application by id (with application having id > 9999) Contributed by Mohammad Shahid Khan
(cherry picked from commit 8fbea531d7)
2015-11-09 11:05:12 -08:00
Jason Lowe c75d8b164f YARN-2902. Killing a container that is localizing can orphan resources in the DOWNLOADING state. Contributed by Varun Saxena
(cherry picked from commit e2267de207)
2015-10-29 16:39:55 +00:00
Akira Ajisaka d9156c1245 YARN-4300. [JDK8] Fix javadoc errors caused by wrong tags. (aajisaka)
(cherry picked from commit 8a68630dd1)
2015-10-27 10:28:30 +09:00
Wangda Tan 9bf1e74171 YARN-4169. Fix racing condition of TestNodeStatusUpdaterForLabels. (Naganarasimha G R via wangda)
(cherry picked from commit 6f606214e7)
2015-10-26 16:38:39 -07:00
Rohith Sharma K S 33455fa415 YARN-2729. Support script based NodeLabelsProvider Interface in Distributed Node Label Configuration Setup. (Naganarasimha G R via rohithsharmaks)
(cherry picked from commit 5acdde4744)
2015-10-26 15:45:22 +05:30
Tsuyoshi Ozawa e85ebf0825 YARN-3528. Tests with 12345 as hard-coded port break jenkins. Contributed by Brahma Reddy Battula.
Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestNodeManagerShutdown.java
2015-10-26 17:06:46 +09:00
Allen Wittenauer 599653ad92 YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) 2015-10-25 21:43:51 -07:00
Jonathan Eagles 6db7bfbcfd YARN-4009. CORS support for ResourceManager REST API. ( Varun Vasudev via jeagles)
(cherry picked from commit f8adeb712d)
2015-10-23 10:43:59 -05:00
Varun Vasudev e4429de86f YARN-4267. Add additional logging to container launch implementations in container-executor. Contributed by Sidharta Seethana.
(cherry picked from commit 151eab2a12)
2015-10-19 20:56:43 +05:30
Varun Vasudev 196d549e2b YARN-4262. Allow whitelisted users to run privileged docker containers. Contributed by Sidharta Seethana.
(cherry picked from commit e39ae0e676)
2015-10-19 20:21:48 +05:30
Steve Loughran 390f29ab99 YARN-4155. TestLogAggregationService.testLogAggregationServiceWithInterval failing. (Bibin A Chundatt via stevel) 2015-10-18 12:03:46 +01:00
Varun Vasudev 342870b70f YARN-4258. Add support for controlling capabilities for docker containers. Contributed by Sidharta Seethana.
(cherry picked from commit 63020c54c1)
2015-10-15 13:41:58 +05:30
Steve Loughran 0e4fb329ed HADOOP-12478. Shell.getWinUtilsPath() has been renamed Shell.getWinutilsPath(). (stevel) 2015-10-14 20:25:33 +01:00
Varun Vasudev 2dd9c475c4 YARN-4255. container-executor does not clean up docker operation command files. Contributed by Sidharta Seethana.
(cherry picked from commit da1ee078f9)
2015-10-14 15:27:18 +05:30
Varun Vasudev 0aeedfd572 YARN-4252. Log container-executor invocation details when exit code is non-zero. Contributed by Sidharta Seethana.
(cherry picked from commit fb9c1519b3)
2015-10-14 15:19:19 +05:30
Varun Vasudev a9b03b5afb YARN-4253. Standardize on using PrivilegedOperationExecutor for all invocations of container-executor in LinuxContainerExecutor. Contributed by Sidharta Seethana.
(cherry picked from commit  8d59293089)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
2015-10-14 15:04:22 +05:30
Steve Loughran 5eab51a24f HADOOP-10775. Shell operations to fail with meaningful errors on windows if winutils.exe not found. (stevel) 2015-10-13 21:47:54 +01:00
Varun Vasudev 34b8142802 YARN-4017. container-executor overuses PATH_MAX. Contributed by Sidharta Seethana
(cherry picked from commit 5b6bae0094)
2015-10-13 12:50:28 +05:30
Jason Lowe 0139517811 YARN-261. Ability to fail AM attempts. Contributed by Andrey Klochkov and Rohith Sharma K S
(cherry picked from commit a0bca2b5ad)
2015-10-09 14:19:41 +00:00
Jason Lowe ec74b0cd4e YARN-3943. Use separate threshold configurations for disk-full detection and disk-not-full detection. Contributed by Zhihai Xu
(cherry picked from commit 8d226225d0)
2015-10-08 22:26:35 +00:00
Wangda Tan d24a3b9a3c YARN-4176. Resync NM nodelabels with RM periodically for distributed nodelabels. (Bibin A Chundatt via wangda)
(cherry picked from commit 30ac69c6bd)
2015-10-05 15:48:40 -07:00
Vinod Kumar Vavilapalli b876babdc7 HADOOP-12441. Fixed shell-kill command behaviour to work correctly on some Linux distributions after HADOOP-12317. Contributed by Wangda Tan.
(cherry picked from commit f1c19b9365)
2015-10-05 10:57:55 -07:00
Xuan 53bddc410f YARN-1897. CLI and core support for signal container functionality. Contributed by Ming Ma
(cherry picked from commit 8f08532bde)
2015-10-02 18:53:04 -07:00
Jason Lowe 3e3733437f YARN-3619. ContainerMetrics unregisters during getMetrics and leads to ConcurrentModificationException. Contributed by Zhihai Xu
(cherry picked from commit fdf02d1f26)
2015-10-02 20:21:32 +00:00
Jason Lowe d645ee1d62 YARN-3727. For better error recovery, check if the directory exists before using it for localization. Contributed by Zhihai Xu
(cherry picked from commit 854d25b0c3)
2015-09-30 15:01:01 +00:00
Jian He d592aa115b YARN-3868. Recovery support for container resizing. Contributed by Meng Ding
(cherry picked from commit c57eac5dfe)
2015-09-24 10:24:13 -07:00
Jian He b6c594c6ea YARN-1644. RM-NM protocol changes and NodeStatusUpdater implementation to support container resizing. Contributed by Meng Ding
(cherry picked from commit c3dc1af072)
2015-09-24 10:24:04 -07:00
Jian He b8955d81c5 YARN-1643. Make ContainersMonitor support changing monitoring size of an allocated container. Contributed by Meng Ding and Wangda Tan
(cherry picked from commit c59ae4eeb1)
2015-09-24 10:23:55 -07:00
Jian He fa0a554ae0 YARN-3867. ContainerImpl changes to support container resizing. Contributed by Meng Ding
(cherry picked from commit 5f5a968d65)
2015-09-24 10:23:46 -07:00
Jian He afe4afd0c9 YARN-1645. ContainerManager implementation to support container resizing. Contributed by Meng Ding & Wangda Tan
(cherry picked from commit ffd820c27a)
2015-09-24 10:23:35 -07:00
Jian He c19e8ffdc7 YARN-1449. AM-NM protocol changes to support container resizing. Contributed by Meng Ding & Wangda Tan)
(cherry picked from commit 83a18add10)
2015-09-24 10:23:25 -07:00
Rohith Sharma K S ace4d26936 YARN-4152. NodeManager crash with NPE when LogAggregationService#stopContainer called for absent container. (Bibin A Chundatt via rohithsharmaks) 2015-09-24 11:31:47 +05:30
Jason Lowe c829be5133 YARN-4095. Avoid sharing AllocatorPerContext object in LocalDirAllocator between ShuffleHandler and LocalDirsHandlerService. Contributed by Zhihai Xu
(cherry picked from commit c890c51a91)
2015-09-23 15:44:13 +00:00
Xuan f3c0a210b6 YARN-4149. yarn logs -am should provide an option to fetch all the log
files. Contributed by Varun Vasudev

(cherry picked from commit 34ef1a092b)
2015-09-15 14:37:59 -07:00
Jason Lowe 181bda0421 YARN-4158. Remove duplicate close for LogWriter in AppLogAggregatorImpl#uploadLogsForContainers. Contributed by Zhihai Xu
(cherry picked from commit 8c1cdb17a0)
2015-09-15 20:22:23 +00:00
Varun Vasudev a4288f1262 YARN-4136. LinuxContainerExecutor loses info when forwarding ResourceHandlerException. Contributed by Bibin A Chundatt.
(cherry picked from commit 486d5cb803)
2015-09-11 14:39:01 +05:30
Wangda Tan 5decd8f9f7 YARN-4106. NodeLabels for NM in distributed mode is not updated even after clusterNodelabel addition in RM. (Bibin A Chundatt via wangda)
(cherry picked from commit 77666105b4)
2015-09-10 09:30:34 -07:00
Zhihai Xu 214338bd68 YARN-4096. App local logs are leaked if log aggregation fails to initialize for the app. Contributed by Jason Lowe.
(cherry picked from commit 16b9037dc1)
2015-09-08 12:33:29 -07:00
Jian He f23be93dd1 YARN-2884. Added a proxy service in NM to proxy the the communication between AM and RM. Contributed by Kishore Chaliparambil
(cherry picked from commit 6f72f1e600)
2015-09-08 11:24:07 +08:00
Varun Vasudev 70575286b7 YARN-3591. Resource localization on a bad disk causes subsequent containers failure. Contributed by Lavkesh Lahngir.
(cherry picked from commit 1dbd8e34a7)
2015-09-07 11:34:14 +05:30
Rohith Sharma K S 7ecdf51441 YARN-4073. Removed unused ApplicationACLsManager in ContainerManagerImpl constructor. (Naganarasimha G R via rohithsharmaks)
(cherry picked from commit 095ab9ab5f)
2015-09-02 14:28:50 +05:30
Xuan 85363ea4bd YARN-221. Addendum patch to compilation issue which is caused by missing
AllContainerLogAggregationPolicy. Contributed by Xuan Gong

(cherry picked from commit b71c6006f5)
2015-08-23 16:50:02 -07:00
Xuan 205b98c059 YARN-221. NM should provide a way for AM to tell it not to aggregate
logs. Contributed by Ming Ma

(cherry picked from commit 37e1c3d82a)
2015-08-22 16:26:39 -07:00
Wangda Tan 6452b31760 YARN-2923. Support configuration based NodeLabelsProvider Service in Distributed Node Label Configuration Setup. (Naganarasimha G R)
(cherry picked from commit fc07464d1a)
2015-08-20 16:52:57 -07:00
Karthik Kambatla b567aa2b4f YARN-4055. Report node resource utilization in heartbeat. (Inigo Goiri via kasha)
(cherry picked from commit 13604bd5f1)
2015-08-20 16:52:33 -07:00
Zhihai Xu 051a622b57 YARN-4057. If ContainersMonitor is not enabled, only print related log info one time. Contributed by Jun Gong.
(cherry picked from commit 14215c8ef8)
2015-08-18 11:47:41 -07:00
Karthik Kambatla d2ff763533 YARN-3534. Collect memory/cpu usage on the node. (Inigo Goiri via kasha)
(cherry picked from commit def12933b3)
2015-08-16 06:26:21 -07:00
Jian He 4a5fdb7c23 YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. Contributed by Jun Gong
(cherry picked from commit 38aed1a94e)
2015-08-13 14:46:34 -07:00
Junping Du 7af531d2e3 YARN-4019. Add JvmPauseMonitor to ResourceManager and NodeManager. Contributed by Robert Kanter.
(cherry picked from commit cfee02b3bd)
2015-08-06 06:50:40 -07:00
Xuan 89ccfd9fcc YARN-4004. container-executor should print output of docker logs if the
docker container exits with non-0 exit status. Contributed by Varun
Vasudev

(cherry picked from commit c3364ca8e7)
2015-08-03 18:11:16 -07:00
Jason Lowe b264217bc3 YARN-3965. Add startup timestamp to nodemanager UI. Contributed by Hong Zhiguo
(cherry picked from commit 469cfcd695)
2015-08-03 15:55:04 +00:00
Xuan 2ec008eb38 YARN-3982. container-executor parsing of container-executor.cfg broken
in trunk and branch-2. Contributed by Varun Vasudev

(cherry picked from commit f170934215)
2015-07-27 23:47:31 -07:00
Varun Vasudev 9da487e0fd YARN-3853. Add docker container runtime support to LinuxContainterExecutor. Contributed by Sidharta Seethana.
(cherry picked from commit 3e6fce91a4)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
2015-07-27 12:03:36 -07:00
Varun Vasudev ec0f801f52 YARN-3852. Add docker container support to container-executor. Contributed by Abin Shahab.
(cherry picked from commit f36835ff9b)
2015-07-27 10:18:06 -07:00