Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,970 Commits 326 Branches 371 Tags
Commit Graph

687 Commits

Author SHA1 Message Date
Ming Ma
db5b4c292b YARN-5797. Add metrics to the node manager for cleaning the PUBLIC and PRIVATE caches. (Chris Trezzo via mingma) 2017-04-06 17:08:59 -07:00
Jason Lowe
84b7f2e956 YARN-6288. Exceptions during aggregated log writes are mishandled. Contributed by Akira Ajisaka 
(cherry picked from commit 1b081ca27e05e97d8b7d284ca24200d43763e481)
 2017-04-06 16:25:41 -05:00
Jason Lowe
903278559a YARN-6403. Invalid local resource request can raise NPE and make NM exit. Contributed by Tao Yang 
(cherry picked from commit e8071aa249c7b21b1de084ee5a9ca2a44efd3bf0)
 2017-04-05 12:31:48 -05:00
Ming Ma
7507ccd38a YARN-6004. Refactor TestResourceLocalizationService#testDownloadingResourcesOnContainer so that it is less than 150 lines. (Chris Trezzo via mingma) 2017-04-04 18:05:09 -07:00
Akira Ajisaka
554bc8342b
YARN-6329. Remove unnecessary TODO comment from AppLogAggregatorImpl.java. Contributed by victor bertschinger. 
(cherry picked from commit 82fb9ce8df7a534a4cbcec624d7b6c0b33e79123)
 2017-03-29 10:37:52 +09:00
Jason Lowe
75e6b7428e YARN-6217. TestLocalCacheDirectoryManager test timeout is too aggressive. Contributed by Miklos Szegedi 
(cherry picked from commit 4a8e3045027036afebbcb80f23b7a2886e56c255)
 2017-03-17 14:07:48 -05:00
Jason Lowe
a16ba4296e YARN-4051. ContainerKillEvent lost when container is still recovering and application finishes. Contributed by sandflee 2017-03-16 09:31:20 -05:00
Junping Du
f254002f1d YARN-6314. Potential infinite redirection on YARN log redirection web service. Contributed by Xuan Gong. 
(cherry picked from commit 5a9dda796f0e73060ada794ad5752cc6a237ab2e)
 2017-03-14 02:58:44 -07:00
Sidharta S
3b6e5ef917 YARN-5669. Add support for docker pull command (Contribtued by luhuichun) 2017-03-09 16:23:21 -08:00
Robert Kanter
11d4cdc3a6 YARN-6297. TestAppLogAggregatorImp.verifyFilesUploaded() should check # of filed uploaded with that of files expected (haibochen via rkanter) 
(cherry picked from commit 287ba4ffa66212c02e1b1edc8fca53f6368a9efc)
 2017-03-08 10:45:59 -08:00
Junping Du
2aa6f317f6 YARN-6174. Log files pattern should be same for both running and finished container. Contributed by Xuan Gong. 
(cherry picked from commit ce2d5bfa5f84e7e563980796549b56ef1e4bbf1e)
 2017-02-15 09:08:54 -08:00
Junping Du
205fc1f10c YARN-6113. Re-direct NM Web Service to get container logs for finished applications. Contributed by Xuan Gong. 
(cherry picked from commit 464ff479ceec76609dca3539081de6b503b17325)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/webapp/TestNMWebServices.java
 2017-02-13 06:16:04 -08:00
Junping Du
ab41da7e64 YARN-6108. Improve AHS webservice to accept NM address as a parameter to get container logs. Contributed by Xuan Gong. 
(cherry picked from commit 9dbfab1284015af6983b4a55b79c9e65bc0194e5)
 2017-02-07 02:14:09 -08:00
Junping Du
c699ce70ea YARN-6100. Improve YARN webservice to output aggregated container logs. Contributed by Xuan Gong. 2017-02-02 00:45:27 -08:00
Jason Lowe
4703f5d20b YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 2017-01-27 15:25:57 +00:00
Jason Lowe
b806201f0a Revert "YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger" 
This reverts commit 45407acaed05284ad897d04f4575ab0a4ee7a88b.
 2017-01-26 14:42:22 +00:00
Jason Lowe
45407acaed YARN-5641. Localizer leaves behind tarballs after container is complete. Contributed by Eric Badger 
(cherry picked from commit 9e19f758c1950cbcfcd1969461a8a910efca0767)
 2017-01-25 21:54:17 +00:00
Junping Du
8027c3e8b9 YARN-6099. Improve webservice to list aggregated log files. Contributed by Xuan Gong. 2017-01-24 15:28:52 -08:00
Jason Lowe
1672a06135 YARN-5547. NMLeveldbStateStore should be more tolerant of unknown keys. Contributed by Ajith S 2017-01-24 16:19:02 +00:00
Chris Nauroth
5251de00fa YARN-5456. container-executor support for FreeBSD, NetBSD, and others if conf path is absolute. Contributed by Allen Wittenauer. 
(cherry picked from commit b913677365ad77ca7daa5741c04c14df1a0313cd)
 2017-01-19 20:48:27 +05:30
Chris Nauroth
198bd84b33 YARN-5121. fix some container-executor portability issues. Contributed by Allen Wittenauer. 
(cherry picked from commit ef501b1a0b4c34a2cc43eb082d1c2364684cd7f1)
 2017-01-19 20:48:27 +05:30
Subru Krishnan
fb639aed90 YARN-6016. Fix minor bugs in handling of local AMRMToken in AMRMProxy. (Botong Huang via Subru). 
(cherry picked from commit 4d1f3d9020b8a8bf1d2a81e4d6ad20418ed9bcc2)
 2017-01-17 14:51:27 -08:00
Daniel Templeton
718ae2d477 YARN-5849. Automatically create YARN control group for pre-mounted cgroups (Contributed by Miklos Szegedi via Daniel Templeton) 
(cherry picked from commit 0fffebe51a91023c4949ab560f5c828f1b568072)
 2017-01-11 15:36:18 -08:00
Junping Du
ba6a013341 YARN-6079. Fix simple spelling errors in yarn test code. Contributed by vijay. 
(cherry picked from commit 4db119b7b55213929e5b86f2abb0ed84a21719b5)
 2017-01-10 18:22:45 -08:00
Naganarasimha
ea38256fab YARN-6015. AsyncDispatcher thread name can be set to improved debugging. Contributed by Ajith S. 2017-01-10 15:10:54 +05:30
Varun Saxena
4fab250fc0 YARN-6068. Log aggregation get failed when NM restart even with recovery (Junping Du via Varun Saxena) 
(cherry picked from commit f59e36b4ce71d3019ab91b136b6d7646316954e7)
 2017-01-09 11:31:39 +05:30
Arun Suresh
4985217de4 YARN-6066. Opportunistic containers Minor fixes : API annotations, parameter name changes, checkstyles. (asuresh) 2017-01-08 08:27:10 -08:00
Arun Suresh
81da7d1d30 YARN-5938. Refactoring OpportunisticContainerAllocator to use SchedulerRequestKey instead of Priority and other misc fixes (asuresh) 
(cherry picked from commit ac1e5d4f77e3b9df8dcacb0b1f72eecc27931eb8)
 2017-01-06 11:25:49 -08:00
Arun Suresh
2b4d3e8506 YARN-5646. Add documentation and update config parameter names for scheduling of OPPORTUNISTIC containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2273a74c1f3895163046cca09ff5e983df301d22)
(cherry picked from commit 9e17ffe599e12c59c48f4355de645c31a4735d04)
 2017-01-06 11:15:10 -08:00
Arun Suresh
3d3bb30a25 YARN-5982. Simplify opportunistic container parameters and metrics. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit b0aace21b1ef3436ba9d516186208fee9a9ceef2)
(cherry picked from commit 3faa0b0d682687099b253feee9b41340edef7111)
 2017-01-06 11:15:09 -08:00
Arun Suresh
368565f0ce YARN-4597. Introduce ContainerScheduler and a SCHEDULED state to NodeManager container lifecycle. (asuresh) 
(cherry picked from commit 3219b7b4ac7d12aee343f6ab2980b3357fc618b6)
(cherry picked from commit 49921678858a17ca58ca2ae704ced1ff584e0859)
 2017-01-06 11:15:09 -08:00
Arun Suresh
047772f15f YARN-5823. Update NMTokens in case of requests with only opportunistic containers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 283fa33febe043bd7b4fa87546be26c9c5a8f8b5)
(cherry picked from commit 1c4cc88a754ac9f557cdc8c859b8aadec19a5067)
 2017-01-06 11:15:09 -08:00
Arun Suresh
0e60c7cd33 YARN-5377. Fix TestQueuingContainerManager.testKillMultipleOpportunisticContainers. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit f38a6d03a11ca6de93a225563ddf55ec99d5063c)
(cherry picked from commit 83ccab8a653a4ef943276651c9604bd434675980)
 2017-01-06 11:15:09 -08:00
Arun Suresh
7437252102 YARN-2995. Enhance UI to show cluster resource utilization of various container Execution types. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 0aafc122d401a2f49d0f9ca556343f9ba871e979)
(cherry picked from commit 25598b6e7118bacd33a167fb496e9cda5e9f9ad0)
 2017-01-06 11:15:09 -08:00
Arun Suresh
28f3bb38bf YARN-5799. Fix Opportunistic Allocation to set the correct value of Node Http Address. (asuresh) 
(cherry picked from commit aa3cab1eb29c56368d15882d7260a994e615e8d8)
(cherry picked from commit ce10e98826884245e733431297d0275177965f17)
 2017-01-06 11:15:09 -08:00
Arun Suresh
f9ea753414 YARN-5486. Update OpportunisticContainerAllocatorAMService::allocate method to handle OPPORTUNISTIC container requests. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 10be45986cdf86a89055065b752959bd6369d54f)
(cherry picked from commit e3baa0988b4dfb592afe426ffeba3c7091b271c5)
 2017-01-06 11:15:09 -08:00
Junping Du
dc2e44e8f8 YARN-5923. Unable to access logs for a running application if YARN_ACL_ENABLE is enabled. Contributed by Xuan Gong. 
(cherry picked from commit 8fadd69047143c9c389cc09ca24100b5f90f79d2)
 2017-01-03 15:19:36 -08:00
Robert Kanter
88597991b0 YARN-5257. Fix unreleased resources and null dereferences (yufeigu via rkanter) 
(cherry picked from commit 9262797e86453fc04b7ca3710b73b21fcdf9e6b4)
 2016-12-27 14:14:41 -08:00
Junping Du
cc42dbb703 YARN-4990. Re-direction of a particular log file within in a container in NM UI does not redirect properly. Contributed by Xuan Gong. 
(cherry picked from commit 736f54b727c3f0ecc8fb9a594f2281c240c89cb8)
 2016-12-21 14:16:19 -08:00
Robert Kanter
9cba7c6d71 YARN-5963. Spelling errors in logging and exceptions for node manager, client, web-proxy, common, and app history code (gsohn via rkanter) 
(cherry picked from commit 72fe54684198b7df5c5fb2114616dff6d17a4402)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
 2016-12-07 15:09:37 -08:00
Naganarasimha
2d94f938c5 YARN-4330. MiniYARNCluster is showing multiple Failed to instantiate default resource calculator warning messages. Contributed by Varun Saxena 
(cherry picked from commit 3541ed80685f25486f33ef0f553854ccbdeb51d4)
 2016-11-23 14:14:56 +05:30
Jason Lowe
38024eb8f7 YARN-5859. TestResourceLocalizationService#testParallelDownloadAttemptsForPublicResource sometimes fails. Contributed by Eric Badger 
(cherry picked from commit 009452bb6dbe5dffb0b304d67a2f360fe0eee1e2)
 2016-11-21 16:37:59 +00:00
Jason Lowe
46b7d6233c YARN-5836. Malicious AM can kill containers of other apps running in any node its containers are running. Contributed by Botong Huang 
(cherry picked from commit 59bfcbf3579e45ddf96db3aafccf669c8e03648f)
 2016-11-16 22:25:15 +00:00
Mingliang Liu
4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b6443715b7a741296c1bd520a1840f9a7c)
 2016-11-15 11:36:08 -08:00
Naganarasimha
260f3a9dc9 YARN-4355. NPE while processing localizer heartbeat. Contributed by Varun Saxena & Jonathan Hung. 
(cherry picked from commit 7ffb9943b8838a3bb56684e0722db40d800743a2)
 2016-11-15 15:44:25 +05:30
Naganarasimha
7e9a6b653e Reverting the patch due to the issue raised in YARN-5765 
Revert "YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang."

This reverts commit 93b768d0d16ea7d1123905e38f8a93430d0e70db.
 2016-11-15 11:05:03 +05:30
Karthik Kambatla
f4c8dc4783 YARN-5834. TestNodeStatusUpdater.testNMRMConnectionConf compares nodemanager wait time to the incorrect value. (Chang Li via kasha) 
(cherry picked from commit 3a98419532687e4362ffc26abbc1264232820db7)
 2016-11-10 14:09:33 -08:00
Jian He
a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Andrew Wang
4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa9d87fdfc7474ca63998a13a3929874)
 2016-11-08 16:33:58 -08:00
Jason Lowe
8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733058238a2c8f23960c986c71dca0e02)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00