Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,937 Commits 326 Branches 371 Tags
Commit Graph

319 Commits

Author SHA1 Message Date
Tsuyoshi Ozawa
4be18edf2f YARN-4380. TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently. Contributed by Varun Saxena. 
(cherry picked from commit 0656d2dc83af6a48a8d8d0e37cdf1f813124f366)
(cherry picked from commit d76b523b020fcd76e9abdd661967ca4a931d9863)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2016-01-06 14:34:15 -08:00
Junping Du
cfbb035420 YARN-4354. Public resource localization fails with NPE. Contributed by Jason Lowe. 
(cherry picked from commit 855d52927b6115e2cfbd97a94d6c1a3ddf0e94bb)
(cherry picked from commit 1d107d805841e9b39db01aa236068a0ff1ca46d1)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
 2016-01-06 14:15:55 -08:00
Junping Du
9da7b1fdd2 YARN-2902. Killing a container that is localizing can orphan resources in the DOWNLOADING state. Contributed by Varun Saxena. 2016-01-06 13:50:35 -08:00
Sangjin Lee
51a2e6304a Preparing for 2.6.4 development 2016-01-05 15:40:31 -08:00
Jason Lowe
271875a426 YARN-3925. ContainerLogsUtils#getContainerLogFile fails to read container log files from full disks. Contributed by zhihai xu 
(cherry picked from commit ff9c13e0a739bb13115167dc661b6a16b2ed2c04)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestContainerLogsPage.java
(cherry picked from commit e8410c0175518a7ca4f44921e950050854895563)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-11-23 20:57:57 +00:00
Sangjin Lee
6466ead9e0 Preparing for 2.6.3 development 2015-10-21 10:58:06 -07:00
Jason Lowe
2ecd173426 YARN-4005. Completed container whose app is finished is possibly not removed from NMStateStore. Contributed by Jun Gong 
(cherry picked from commit 38aed1a94ed7b6da62e2445b5610bc02b1cddeeb)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-10-08 15:16:11 +00:00
Jason Lowe
49335d9b2b YARN-3727. For better error recovery, check if the directory exists before using it for localization. Contributed by Zhihai Xu 
(cherry picked from commit 854d25b0c30fd40f640c052e79a8747741492042)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/localizer/TestLocalResourcesTrackerImpl.java
(cherry picked from commit 493f07200837969d6e4d6af2cccc732e60f119ba)

Conflicts:

	hadoop-yarn-project/CHANGES.txt
 2015-09-30 16:12:09 +00:00
Sangjin Lee
4cb7dbaead Preparing for 2.6.2 development: mvn versions:set -DnewVersion=2.6.2 2015-09-25 15:51:13 -07:00
Vinod Kumar Vavilapalli
3462a00dd2 Preparing for release 2.6.1: mvn versions:set -DnewVersion=2.6.1 2015-09-09 15:29:57 -07:00
Jason Lowe
fe5877a49e YARN-3850. NM fails to read files from full disks which can lead to container logs being lost and other issues. Contributed by Varun Saxena 
(cherry picked from commit 40b256949ad6f6e0dbdd248f2d257b05899f4332)

(cherry picked from commit 0221d19f4e398c386f4ca3990b0893562aa8dacf)
(cherry picked from commit 87d2204f28f192a964c04a5fa1e2e31644d74b59)
 2015-09-03 14:35:01 -07:00
Jason Lowe
f21fb808f1 YARN-3832. Resource Localization fails on a cluster due to existing cache directories. Contributed by Brahma Reddy Battula 
(cherry picked from commit 8d58512d6e6d9fe93784a9de2af0056bcc316d96)

(cherry picked from commit 15b1800b1289d239cbebc5cfd66cfe156d45a2d3)
(cherry picked from commit 38400507e3352d83c2a1f364de137366249b7983)
 2015-09-03 14:26:43 -07:00
Jason Lowe
193d8d3667 YARN-3585. NodeManager cannot exit on SHUTDOWN event triggered and NM recovery is enabled. Contributed by Rohith Sharmaks 
(cherry picked from commit e13b671aa510f553f4a6a232b4694b6a4cce88ae)

(cherry picked from commit 752caa95a40d899e1bf98bc907e91aec2bb57073)
(cherry picked from commit 13c4db632b0e7f19dcfa883c2492431c2c7d0799)
 2015-09-03 14:09:16 -07:00
Jason Lowe
778da79e6f YARN-3641. NodeManager: stopRecoveryStore() shouldn't be skipped when exceptions happen in stopping NM's sub-services. Contributed by Junping Du 
(cherry picked from commit 711d77cc54a64b2c3db70bdacc6bf2245c896a4b)

(cherry picked from commit a81ad814610936a02e55964fbe08f7b33fe29b23)
(cherry picked from commit aa82b0684554be8d09f6fcd88826f167922280cc)
 2015-09-03 11:50:30 -07:00
Karthik Kambatla
6ade6b5051 YARN-3464. Race condition in LocalizerRunner kills localizer before localizing all resources. (Zhihai Xu via kasha) 
(cherry picked from commit 47279c3228185548ed09c36579b420225e4894f5)
(cherry picked from commit 4045c41afe440b773d006e962bf8a5eae3fdc284)
(cherry picked from commit 6f2cc0dfa8f21984ecdab59dc087ccf525934930)
 2015-09-02 15:03:51 -07:00
Xuan
9af5b1dcd0 YARN-3024. LocalizerRunner should give DIE action when all resources are 
localized. Contributed by Chengbing Liu

(cherry picked from commit 0d6bd62102f94c55d59f7a0a86a684e99d746127)
(cherry picked from commit a7696b3fbfacd98a892bbb3678663658c7b9d2bd)
(cherry picked from commit 9e30232004ab7c3c3bfde3b8b27c37fa7065f6be)
 2015-09-02 14:52:06 -07:00
Jian He
994c3d049a YARN-3011. Possible IllegalArgumentException in ResourceLocalizationService might lead NM to crash. Contributed by Varun Saxena 
(cherry picked from commit 4e15fc08411318e11152fcd5a4648ed1d6fbb480)

(cherry picked from commit 8100c8a68c32978a177af9a3e6639f6de533886d)
(cherry picked from commit 10a6c4f349e6f32ed2a520bf669a0cbfff31c824)
 2015-08-31 15:38:45 -07:00
Jian He
3f8da2a9eb YARN-2997. Fixed NodeStatusUpdater to not send alreay-sent completed container statuses on heartbeat. Contributed by Chengbing Liu 
(cherry picked from commit cc2a745f7e82c9fa6de03242952347c54c52dccc)

(cherry picked from commit e7e6173049adca2a2ae0e1231adcaca8168bec27)
(cherry picked from commit 3c4ed2497b14140f09b3cae4959be6474c4cdc99)
 2015-08-30 20:45:45 -07:00
Jian He
6dfdcf094d YARN-2952. Fixed incorrect version check in StateStore. Contributed by Rohith Sharmaks 
(cherry picked from commit 808cba3821d5bc4267f69d14220757f01cd55715)

(cherry picked from commit 9180d11b3bbb2a49127d5d25f53b38c5113bf7ea)
 2015-08-30 20:34:14 -07:00
Jason Lowe
f83d898944 YARN-2816. NM fail to start with NPE during container recovery. Contributed by Zhihai Xu 
(cherry picked from commit 49c38898b0be64fc686d039ed2fb2dea1378df02)

(cherry picked from commit ad140d1fc831735fb9335e27b38d2fc040847af1)
(cherry picked from commit 85b23c323c80c5303bd0b7bdb066258792ca67d8)
 2015-08-27 18:32:59 -07:00
Jason Lowe
685790e027 YARN-2846. Incorrect persist exit code for running containers in reacquireContainer() that interrupted by NodeManager restart. Contributed by Junping Du 
(cherry picked from commit 33ea5ae92b9dd3abace104903d9a94d17dd75af5)
 2014-11-13 16:12:30 +00:00
Zhijie Shen
1535e7211e YARN-2794. Fixed log messages about distributing system-credentials. Contributed by Jian He. 
(cherry picked from commit be7bf956e96dd0fd9b521ca71df9124b9cc5ebd0)
 2014-11-12 11:10:47 -08:00
Xuan
a690629ffc YARN-2841. RMProxy should retry EOFException. Contributed by Jian He 2014-11-12 10:28:55 -08:00
Ravi Prakash
9613a57e83 YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN 2014-11-12 09:33:42 -08:00
Arun C. Murthy
60584c732f Preparing to release hadoop-2.6.0: Set version in branch-2.6 to 2.6.1-SNAPSHOT. 2014-11-09 19:21:11 -08:00
Arun C. Murthy
c2cc879f01 YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 
(cherry picked from commit 43cd07b408c6613d2c9aa89203cfa3110d830538)
 2014-11-09 15:04:55 -08:00
Jason Lowe
e5c0fa2c95 YARN-2825. Container leak on NM. Contributed by Jian He 
(cherry picked from commit c3d475070a1ec54c4b05923f4782cef204effd2c)
 2014-11-07 23:17:49 +00:00
cnauroth
9adb31b6a8 YARN-2803. MR distributed cache not working correctly on Windows after NodeManager privileged account changes. Contributed by Craig Welch. 
(cherry picked from commit 06b797947c980d7d21864eb8b700cf565756aac1)
(cherry picked from commit c16f7182937dc4e8bbf46dfedd9db427a0a33357)
 2014-11-07 12:45:25 -08:00
Zhijie Shen
1550617e48 YARN-2752. Made ContainerExecutor append "nice -n" arg only when priority adjustment flag is set. Contributed by Xuan Gong. 
(cherry picked from commit e06c23a6c92ef783cdb45447fa2abd1ab48d166f)
 2014-11-04 15:54:28 -08:00
Vinod Kumar Vavilapalli
0b73606b1c YARN-1922. Fixed NodeManager to kill process-trees correctly in the presence of races between the launch and the stop-container call and when root processes crash. Contributed by Billie Rinaldi. 
(cherry picked from commit c5a46d4c8ca236ff641a309f256bbbdf4dd56db5)
 2014-11-03 16:41:21 -08:00
Vinod Kumar Vavilapalli
c205841d49 YARN-2788. Fixed backwards compatiblity issues with log-aggregation feature that were caused when adding log-upload-time via YARN-2703. Contributed by Xuan Gong. 
(cherry picked from commit 58e9f24e0f06efede21085b7ffe36af042fa7b38)
 2014-11-03 13:20:01 -08:00
Jason Lowe
2098c68acb YARN-2730. DefaultContainerExecutor runs only one localizer at a time. Contributed by Siqi Li 
(cherry picked from commit 6157ace5475fff8d2513fd3cd99134b532b0b406)
 2014-11-03 20:40:02 +00:00
Vinod Kumar Vavilapalli
5a0aac5506 YARN-2790. Fixed a NodeManager bug that was causing log-aggregation to fail beyond HFDS delegation-token expiry even when RM is a proxy-user (YARN-2704). Contributed by Jian He. 
(cherry picked from commit 5c0381c96aa79196829edbca497c649eb6776944)
 2014-11-01 16:34:11 -07:00
Zhijie Shen
5492370a4c YARN-2711. Fixed TestDefaultContainerExecutor#testContainerLaunchError failure on Windows. Contributed by Varun Vasudev. 
(cherry picked from commit 1cd088fd9dac3015df0b6281974fc6b6c3ece20d)
 2014-10-31 17:51:03 -07:00
Xuan
aa13977001 YARN-2701. Addendum patch. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 2014-10-31 14:39:49 -07:00
Jason Lowe
773ba8b606 YARN-2755. NM fails to clean up usercache_DEL_<timestamp> dirs after YARN-661. Contributed by Siqi Li 
(cherry picked from commit 73e626ad91cd5c06a005068d8432fd16e06fe6a0)
 2014-10-30 15:12:35 +00:00
Zhijie Shen
ae62fe2ebd YARN-2741. Made NM web UI serve logs on the drive other than C: on Windows. Contributed by Craig Welch. 
(cherry picked from commit 8984e9b1774033e379b57da1bd30a5c81888c7a3)
 2014-10-28 14:13:31 -07:00
Vinod Kumar Vavilapalli
0af8cc9ca2 YARN-2704. Changed ResourceManager to optionally obtain tokens itself for the sake of localization and log-aggregation for long-running services. Contributed by Jian He. 
(cherry picked from commit a16d022ca4313a41425c8e97841c841a2d6f2f54)
 2014-10-27 15:51:20 -07:00
Zhijie Shen
a41371e173 YARN-2703. Added logUploadedTime into LogValue for better display. Contributed by Xuan Gong. 
(cherry picked from commit f81dc3f995579c1b94b11d60e9fc6da56c8a9496)
 2014-10-24 14:13:28 -07:00
Jian He
29d0164ee7 YARN-2198. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor. Contributed by Remus Rusanu 
(cherry picked from commit 3b12fd6cfbf4cc91ef8e8616c7aafa9de006cde5)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-22 16:22:08 -07:00
Jian He
a057552468 YARN-2682. Updated WindowsSecureContainerExecutor to not use DefaultContainerExecutor#getFirstApplicationDir and use getWorkingDir() instead. Contributed by Zhihai Xu 
(cherry picked from commit 0fd0ebae645e671699f6a6a56a012ebe6dfb5b2a)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
 2014-10-22 16:19:40 -07:00
cnauroth
8727f9e1b0 YARN-2720. Windows: Wildcard classpath variables not expanded against resources contained in archives. Contributed by Craig Welch. 
(cherry picked from commit 6637e3cf95b3a9be8d6b9cd66bc849a0607e8ed5)
(cherry picked from commit a066134277716a9709b63cfb11b4be5627a9bd3f)
 2014-10-21 12:36:43 -07:00
Jason Lowe
413fe4cb02 YARN-90. NodeManager should identify failed disks becoming good again. Contributed by Varun Vasudev 
(cherry picked from commit 6f2028bd1514d90b831f889fd0ee7f2ba5c15000)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDirectoryCollection.java
 2014-10-21 17:43:22 +00:00
Jian He
23b4af488e Missing file for YARN-2701 
(cherry picked from commit 4fa1fb3193bf39fcb1bd7f8f8391a78f69c3c302)
 2014-10-20 20:53:28 -07:00
Jian He
c7db30522c YARN-2701. Potential race condition in startLocalizer when using LinuxContainerExecutor. Contributed by Xuan Gong 
(cherry picked from commit 2839365f230165222f63129979ea82ada79ec56e)
 2014-10-20 20:53:28 -07:00
Vinod Kumar Vavilapalli
debb3686d3 YARN-1972. Added a secure container-executor for Windows. Contributed by Remus Rusanu. 
commit ba7f31c2ee8d23ecb183f88920ef06053c0b9769 is the corresponding trunk commit, this is a slightly different patch for branch-2.
(cherry picked from commit 3326fba382350db88ccaa67625f92fa1dc5fd3e3)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-20 20:47:49 -07:00
Jian He
bd45dba807 YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit f93d2ea27e7554272845fb11d97b3564704834b3)
 2014-10-15 15:31:57 -07:00
Karthik Kambatla
e6102b1828 YARN-2566. DefaultContainerExecutor should pick a working directory randomly. (Zhihai Xu via kasha) 
(cherry picked from commit cc93e7e683fa74eb1a7aa2b357a36667bd21086a)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DefaultContainerExecutor.java
 2014-10-13 16:42:47 -07:00
Jason Lowe
9d73c1b15f YARN-2377. Localization exception stack traces are not passed as diagnostic info. Contributed by Gera Shegalov 
(cherry picked from commit a56ea0100215ecf2e1471a18812b668658197239)
 2014-10-13 18:34:28 +00:00
Zhijie Shen
6da637ec12 YARN-2651. Spun off LogRollingInterval from LogAggregationContext. Contributed by Xuan Gong. 
(cherry picked from commit 4aed2d8e91c7dccc78fbaffc409d3076c3316289)
 2014-10-13 10:56:22 -07:00