Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,026 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2864 Commits

Author SHA1 Message Date
Jason Lowe 3d401206cf YARN-5483. Optimize RMAppAttempt#pullJustFinishedContainers. Contributed by sandflee 2016-08-10 18:06:56 +00:00
Karthik Kambatla 447ba47640 YARN-5343. TestContinuousScheduling#testSortedNodes fails intermittently. (Yufei Gu via kasha) 
(cherry picked from commit 7992c0b42c)
 2016-08-09 16:51:27 -07:00
Varun Vasudev c599f6215e YARN-5394. Remove bind-mount /etc/passwd for Docker containers. Contributed by Zhankun Tang. 
(cherry picked from commit 522ddbde79)
 2016-08-09 16:05:03 +05:30
Arun Suresh 5f7edb79d1 YARN-5457. Refactor DistributedScheduling framework to pull out common functionality. (asuresh) 
(cherry picked from commit 82c9e06101)
 2016-08-09 00:46:08 -07:00
Naganarasimha 93b768d0d1 YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang. 2016-08-09 06:48:39 +05:30
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 31bb302c91 YARN-5470. Addedum to differentiate exactly matching of log-files with regex in yarn log CLI. Contributed by Xuan Gong. 
(cherry picked from commit 83a2ffec40)
 2016-08-08 14:29:24 -07:00
Varun Saxena ce8df272f4 YARN-4910. Fix incomplete log info in ResourceLocalizationService (Jun Gong via Varun Saxena) 2016-08-08 22:14:56 +05:30
Naganarasimha 991823482f YARN-4624. NPE in PartitionQueueCapacitiesInfo while accessing Schduler UI. Contributed by Brahma Reddy Battula 
(cherry picked from commit d81b8163b4)
 2016-08-06 07:04:30 +05:30
Varun Saxena 0e48f29937 YARN-5429. Fix return related javadoc warnings in yarn-api (Vrushali C via Varun Saxena) 2016-08-06 01:13:30 +05:30
Junping Du adcecd417b YARN-5470. Differentiate exactly match with regex in yarn log CLI. Contributed by Xuan Gong. 
(cherry picked from commit e605d47df0)
 2016-08-05 11:00:13 -07:00
Wangda Tan a73e118437 YARN-4888. Changes in scheduler to identify resource-requests explicitly by allocation-id. (Subru Krishnan via wangda) 
(cherry picked from commit 3f100d76ff)
 2016-08-05 10:44:07 -07:00
Rohith Sharma K S 69da77c357 YARN-5333. Some recovered apps are put into default queue when RM HA. Contributed by Jun Gong. 
(cherry picked from commit d9a354c2f3)
 2016-08-05 21:37:05 +05:30
Varun Vasudev 19d894538d YARN-5459. Add support for docker rm. Contributed by Shane Kumpf. 
(cherry picked from commit 08e33381f5)
 2016-08-04 14:18:01 +05:30
Jason Lowe 65a91c8527 YARN-5469. Increase timeout of TestAmFilter.testFilter. Contributed by Eric Badger 
(cherry picked from commit db4a61dc61)
 2016-08-03 19:53:28 +00:00
Jason Lowe 1ea03411c2 YARN-5462. TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently. Contributed by Eric Badger 
(cherry picked from commit db646540f0)
 2016-08-03 19:18:50 +00:00
Jason Lowe 9a0ac56a5c YARN-4280. CapacityScheduler reservations may not prevent indefinite postponement on a busy cluster. Contributed by Kuhu Shukla 
(cherry picked from commit 4d92aefd35)
 2016-08-03 19:00:06 +00:00
Varun Vasudev 1087745d79 YARN-5460. Change container runtime type logging in DelegatingLinuxContainerRuntime to debug. Contributed by Shane Kumpf. 
(cherry picked from commit d848184e90)
 2016-08-03 12:03:31 +05:30
Junping Du 7e213844a7 YARN-5226. Remove AHS enable check from LogsCLI#fetchAMContainerLogs. Contributed by Xuan Gong. 
(cherry picked from commit 3818393297)
 2016-08-02 10:42:31 -07:00
Varun Vasudev b63e6c8027 YARN-5458. Rename DockerStopCommandTest to TestDockerStopCommand. Contributed by Shane Kumpf. 
(cherry picked from commit 7fc70c6422)
 2016-08-02 17:02:17 +05:30
Varun Vasudev 4e5409a2bb YARN-5443. Add support for docker inspect command. Contributed by Shane Kumpf. 
(cherry picked from commit 2e7c2a13a8)
 2016-08-02 17:01:59 +05:30
Varun Vasudev 7ecb1635b4 YARN-5444. Fix failing unit tests in TestLinuxContainerExecutorWithMocks. Contributed by Yufei Gu. 
(cherry picked from commit 4455aa7c86a36b6e017e288479a7d4ced889faa0)
 2016-08-01 16:01:10 +05:30
Arun Suresh f197378f81 YARN-5113. Refactoring and other clean-up for distributed scheduling. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit e5766b1dbe)
 2016-07-31 11:49:09 -07:00
Junping Du 47dd871f96 YARN-5434. Add -client|server argument for graceful decommmission. Contributed by Robert Kanter. 
(cherry picked from commit 95f2b98597)
 2016-07-29 10:27:11 -07:00
Li Lu 0b5d96abb5 YARN-5436. Race in AsyncDispatcher can cause random test failures in Tez (probably YARN also). (Zhiyuan Yang via gtcarrera9) 
(cherry picked from commit 7086fc72ee)
 2016-07-28 16:52:21 -07:00
Subru Krishnan 2f4df2b851 YARN-5203.Return ResourceRequest JAXB object in ResourceManager Cluster Applications REST API. Contributed by Ellen Hui. 
(cherry picked from commit 4e756d7271)
 2016-07-28 16:04:30 -07:00
Jason Lowe 2f854b8386 YARN-5438. TimelineClientImpl leaking FileSystem Instances causing Long running services like HiverServer2 daemon going OOM. Contributed by Rohith Sharma K S 
(cherry picked from commit a1890c32c5)
 2016-07-28 21:48:40 +00:00
Li Lu 310245063b YARN-5440. Use AHSClient in YarnClient when TimelineServer is running. (Xuan Gong via gtcarrera9) 
(cherry picked from commit 26de4f0de7)
 2016-07-28 10:25:55 -07:00
Junping Du 932bf79244 YARN-5432. Lock already held by another process while LevelDB cache store creation for dag. Contributed by Li Lu. 
(cherry picked from commit 7f3c306e2e)
 2016-07-28 06:36:31 -07:00
Subru Krishnan 4ecfd05095 YARN-5441. Fixing minor Scheduler test case failures 
(cherry picked from commit d2cbfd7de3)
 2016-07-27 18:26:29 -07:00
Wangda Tan 77e0b6d1bf YARN-5195. RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler. (sandflee via wangda) 
(cherry picked from commit d62e121ffc)
 2016-07-26 21:36:37 -07:00
Wangda Tan 55fa19d576 YARN-5342. Improve non-exclusive node partition resource allocation in Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit 49969b16cd)
 2016-07-26 21:16:12 -07:00
Arun Suresh 0f6ff20652 YARN-5351. ResourceRequest should take ExecutionType into account during comparison. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 2d8d183b19)
 2016-07-26 19:09:22 -07:00
Arun Suresh df509a8066 YARN-5392. Replace use of Priority in the Scheduling infrastructure with an opaque ShedulerRequestKey. (asuresh and subru) 
(cherry picked from commit 5aace38b74)
 2016-07-26 14:58:54 -07:00
mattf-horton ea10e1384f HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 15:58:05 -07:00
Rohith Sharma K S eff65463de YARN-5092. TestRMDelegationTokens fails intermittently. Contributed by Jason Lowe. 
(cherry picked from commit 557a245d83)
 2016-07-21 12:49:06 +05:30
Rohith Sharma K S d9ed29e8c6 YARN-1126. Add validation of users input nodes-states options to nodes CLI. Contributed by Wei Yan. 
(cherry picked from commit be34b2a8fd)
 2016-07-21 12:22:34 +05:30
Akira Ajisaka 2bf5c4eb1a YARN-4883. Make consistent operation name in AdminService. Contributed by Kai Sasaki. 
(cherry picked from commit c63afdbe14)
 2016-07-20 16:51:44 -07:00
Vinod Kumar Vavilapalli 5641c28c1c YARN-5340. Fixed a race condition in RollingLevelDBTimelineStore that caused loss of Timeline events. Contributed by Li Lu. 
(cherry picked from commit 1c9d2ab503)
 2016-07-20 08:38:56 -07:00
Varun Vasudev 9e326f4eeb YARN-5309. Fix SSLFactory truststore reloader thread leak in TimelineClientImpl. Contributed by Weiwei Yang. 
(cherry picked from commit 9ccf935b2c)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/TimelineClientImpl.java
 2016-07-20 12:55:38 +05:30
Arun Suresh 586cd0daa1 YARN-5350. Distributed Scheduling: Ensure sort order of allocatable nodes returned by the RM is not lost. (asuresh) 
(cherry picked from commit 8fbe6ece24)
 2016-07-19 23:22:47 -07:00
Junping Du ad8180fc6c YARN-5213. Fix a bug in LogCLIHelpers which cause TestLogsCLI#testFetchApplictionLogs fails intermittently. Contributed by Xuan Gong. 
(cherry picked from commit dc2f4b6ac8)
 2016-07-19 07:19:17 -07:00
Varun Saxena 2153c9f04c YARN-4996. Make TestNMReconnect.testCompareRMNodeAfterReconnect() scheduler agnostic (Kai Sasaki via Varun Saxena) 2016-07-19 16:07:56 +05:30
Ray Chiang 66a501be5e YARN-5272. Handle queue names consistently in FairScheduler. (Wilfred Spiegelenburg via rchiang) 
(cherry picked from commit 2df1b1e72b18422606e38a9fbe311988637cb9a5)
 2016-07-15 14:39:28 -07:00
Arun Suresh 9d31c0b53f YARN-5181. ClusterNodeTracker: add method to get list of nodes matching a specific resourceName. (kasha via asuresh) 
(cherry picked from commit e905a42a2c)
 2016-07-15 14:38:26 -07:00
Wangda Tan e34e1aa4fe YARN-4484. Available Resource calculation for a queue is not correct when used with labels. (Sunil G via wangda) 
(cherry picked from commit 24db9167f1)
 2016-07-15 11:41:35 -07:00
Varun Saxena 8b0337b499 YARN-5383. Fix findbugs and checkstyle issues in ContainerExecutor. (Vrushali C via Varun Saxena) 2016-07-15 23:30:36 +05:30
Junping Du e3bc4faa96 YARN-5339. Passing file to -out for YARN log CLI doesnt give warning or error code. Contributed by Xuan Gong. 
(cherry picked from commit 7e5355c14e)
 2016-07-15 09:24:03 -07:00
Akira Ajisaka 6664077727 YARN-5159. Wrong Javadoc tag in MiniYarnCluster. Contributed by Andras Bokor. 
(cherry picked from commit 6cf017558a)
 2016-07-14 10:45:39 -07:00
Xuan 58088d27e6 YARN-5363. For AM containers, or for containers of running-apps, "yarn logs" incorrectly only (tries to) shows syslog file-type by default. (Contributed by Vinod Kumar Vavilapalli) 
(cherry picked from commit 429347289c)
 2016-07-14 10:21:55 -07:00
Varun Vasudev d5d68d98c6 YARN-5303. Clean up ContainerExecutor JavaDoc. Contributed by Daniel Templeton. 
(cherry picked from commit 54bf14f80b)
 2016-07-14 19:32:09 +05:30
Varun Vasudev b4124c6a5a YARN-4759. Fix signal handling for docker containers. Contributed by Shane Kumpf. 
(cherry picked from commit e5e558b0a3)
 2016-07-14 19:32:00 +05:30
Varun Vasudev abfaf0e0e7 YARN-5298. Mount usercache and NM filecache directories into Docker container. Contributed by Sidharta Seethana. 
(cherry-picked from commit 58e1850801)
 2016-07-14 16:48:22 +05:30
Rohith Sharma K S 356c09a509 YARN-5299. Log Docker run command when container fails. Contributed by Varun Vasudev. 
(cherry picked from commit dbe97aa768)
 2016-07-14 11:49:48 +05:30
Vinod Kumar Vavilapalli cbd2970be7 YARN-5339. Fixed "yarn logs" to fail when a file is passed to -out option instead of a directory. Contributed by Xuan Gong. 
(cherry picked from commit d18050522c)
 2016-07-13 14:55:44 -07:00
Vinod Kumar Vavilapalli 4a6cef8c91 YARN-5200. Enhanced "yarn logs" to be able to get a list of containers whose logs are aggregated via a "show_container_log_info" option. Contributed by Xuan Gong. 
(cherry picked from commit eb47163234)
 2016-07-13 10:55:18 -07:00
Rohith Sharma K S 694b4abbc0 YARN-5362. TestRMRestart#testFinishedAppRemovalAfterRMRestart can fail. Contributed by sandflee. 
(cherry picked from commit d6d41e820a)
 2016-07-13 19:15:41 +05:30
Varun Saxena dc3e077301 YARN-5353. ResourceManager can leak delegation tokens when they are shared across apps. (Jason Lowe via Varun Saxena). 2016-07-13 07:46:00 +05:30
Robert Kanter 7aaf82d469 YARN-4366. Fix Lint Warnings in YARN Common (templedf via rkanter) 
(cherry picked from commit bf6f4a3b98)
 2016-07-12 15:13:00 -07:00
Jason Lowe 1f96baf4ad YARN-5317. testAMRestartNotLostContainerCompleteMsg may fail. Contributed by sandflee 
(cherry picked from commit 10b704c594)
 2016-07-12 20:38:26 +00:00
Junping Du 54812841e2 New addendum to YARN-5080. Cannot obtain logs using YARN CLI -am for either KILLED or RUNNING AM. 
(cherry picked from commit 7705812c7e)
 2016-07-12 08:32:26 -07:00
Jian He e94e6be772 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:37:52 -07:00
Varun Saxena 5ef27b65b9 YARN-5037. Fix random failure of TestRMRestart#testQueueMetricsOnRMRestart (sandflee via Varun Saxena). 2016-07-10 22:13:53 +05:30
Junping Du 7db796562c Fix OOM issue in DistributedShell. Contributed by Jian He. 
(cherry picked from commit 9bdb5bebea)
 2016-07-09 05:11:29 -07:00
Varun Saxena f515678b67 YARN-5318. Fix intermittent test failure of TestRMAdminService#testRefreshNodesResourceWithFileSystemBasedConfigurationProvider. (Jun Gong via Varun Saxena). 2016-07-09 01:43:09 +05:30
Varun Saxena 550abb53e5 YARN-5297. Avoid printing a stack trace when recovering an app after the RM restarts. (Junping Du via Varun Saxena). 2016-07-09 00:14:51 +05:30
Junping Du 5c0386d535 YARN-4939. The decommissioning Node should keep alive during NM restart. Contributed by sandflee. 
(cherry picked from commit 30ee57ceb1)
 2016-07-08 04:15:47 -07:00
Vinod Kumar Vavilapalli 687185feb2 YARN-5314. Fixed a ConcurrentModificationException in ATS v1.5 EntityGroupFSTimelineStore. Contributed by Li Lu. 
(cherry picked from commit 673e5e02fe)
 2016-07-07 22:38:41 -07:00
Jian He b7604a22e6 YARN-5233. Support for specifying a path for ATS plugin jars. Contributed by Li Lu 
(cherry picked from commit 8a9d293dd6)
 2016-07-06 16:45:32 -07:00
Wangda Tan e246cf74c9 YARN-5294. Pass remote ip address down to YarnAuthorizationProvider. (Jian He via wangda) 2016-07-06 16:42:35 -07:00
Vinod Kumar Vavilapalli c3d9ac82af YARN-5224. Added new web-services /containers/{containerid}/logs & /containers/{containerid}/logs/{filename} and using them in "yarn logs" CLI to get logs of finished containers of a running application. Contributed by Xuan Gong. 2016-07-06 14:19:16 -07:00
Jian He ccb0791904 YARN-5227. Yarn logs command: no need to specify applicationId when 
specifying containerId. Contributed by Gergely Novák

(cherry picked from commit d169f5052f)
 2016-07-06 13:29:12 -07:00
Jian He 46f8204992 YARN-5296. NMs going OutOfMemory because ContainerMetrics leak in ContainerMonitorImpl. Contributed by Junping Du 
(cherry picked from commit d792a90206)
 2016-07-05 18:06:03 -07:00
Vinod Kumar Vavilapalli 56a08983e2 YARN-5214. Fixed locking in DirectoryCollection to avoid hanging NMs when various code-paths hit slow disks. Contributed by Junping Du. 
(cherry picked from commit ce9c006430)
 2016-07-05 16:08:57 -07:00
Varun Saxena 516aa58c1b YARN-5286. Add RPC port info in RM web service's response when getting app status. (Jun Gong via Varun Saxena). 2016-07-05 23:10:08 +05:30
Ray Chiang fb9327edae YARN-4568. Fix message when NodeManager runs into errors initializing the recovery directory. (rchiang) 
(cherry picked from commit 0a5def155e)
 2016-07-01 15:55:56 -07:00
Jian He d2559ca307 YARN-5023. TestAMRestart#testShouldNotCountFailureToMaxAttemptRetry fails. Contributed by sandflee 
(cherry picked from commit c35a5a7a8d)
 2016-07-01 14:29:28 -07:00
Varun Saxena 4837aebca4 YARN-5282. Fix typos in CapacityScheduler documentation. (Ray Chiang via Varun Saxena). 2016-07-01 11:39:26 +05:30
Varun Saxena 527cd9852a YARN-5182. MockNodes.newNodes creates one more node per rack than requested. (Karthik Kambatla via Varun Saxena). 2016-06-30 00:06:17 +05:30
Rohith Sharma K S d3fc1bddea YARN-5262. Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat. 
(cherry picked from commit 26b5e6116f)
 2016-06-29 10:13:17 +05:30
Xuan b7f18f2815 YARN-5266. Wrong exit code while trying to get app logs using regex via CLI. Contributed by Xuan Gong 
(cherry picked from commit 6ab5aa1c1f)
 2016-06-22 21:50:09 -07:00
Arun Suresh e64d21b297 YARN-5171. Extend DistributedSchedulerProtocol to notify RM of containers allocated by the Node. (Inigo Goiri via asuresh) 
(cherry picked from commit 99e5dd68d0)
 2016-06-22 19:05:28 -07:00
Sangjin Lee 2a79910c22 YARN-4958. The file localization process should allow for wildcards to reduce the application footprint in the state store (Daniel Templeton via sjlee) 
(cherry picked from commit 5107a967fa)
 2016-06-20 10:00:41 -07:00
Junping Du ea4cab2330 YARN-5246. NMWebAppFilter web redirects drop query parameters. Contributed by Varun Vasudev. 
(cherry picked from commit d0162f2040)
 2016-06-19 17:45:57 -07:00
Karthik Kambatla 25be8105cd YARN-5077. Fix FSLeafQueue#getFairShare() for queues with zero fairshare. (Yufei Gu via kasha) 
(cherry picked from commit 20f2799938)
 2016-06-17 22:25:04 -07:00
Karthik Kambatla e9c1155233 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit. Addendum to fix javac warning. (Arun Suresh via kasha) 
(cherry picked from commit fbbe0bb627)
 2016-06-17 22:13:18 -07:00
Junping Du 6afa044dda YARN-5251. Yarn CLI to obtain App logs for last 'n' bytes fails. Contributed by Xuan Gong. 
(cherry picked from commit c35fa4a0e5)
 2016-06-17 08:25:33 -07:00
Wangda Tan 7eb524ddee YARN-4844. Addendum patch, revert few changes to public and evolving to avoid breaking downstream projects as much as possible. 2016-06-16 11:09:56 -07:00
Junping Du eec703f36f YARN-5083. YARN CLI for AM logs does not give any error message if entered invalid am value. Contributed by Jian He. 
(cherry picked from commit e14ee0d3b5)
 2016-06-16 08:56:46 -07:00
Junping Du 6dd34baf3c YARN-5237. Fix missing log files issue in rolling log aggregation. Contributed by Xuan Gong. 
(cherry picked from commit 5dfc38ff57)
 2016-06-15 16:20:07 -07:00
Varun Vasudev 75235149af YARN-5223. Container line in yarn logs output for a live application should include the hostname for the container. Contributed by Xuan Gong. 
(cherry picked from commit 25064fb2fb)
 2016-06-15 12:43:26 +05:30
Varun Vasudev 5bd39d2f3c YARN-5122. "yarn logs" for running containers should print an explicit footer saying that the log may be incomplete. Contributed by Jian He. 
(cherry picked from commit d6284428e8)
 2016-06-15 12:37:51 +05:30
Wangda Tan 863bfa4d6a YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:12:00 -07:00
Wangda Tan cd3bdbc70a YARN-4887. Add allocation request ID to AM-RM protocol for identifying resource-requests explicitly. (Subru Krishnan via wangda) 
(cherry picked from commit eec835ec17)
 2016-06-13 21:57:55 -07:00
Rohith Sharma K S c540b9bd7b YARN-4989. TestWorkPreservingRMRestart#testCapacitySchedulerRecovery fails intermittently. Contributed by Ajith S. 
(cherry picked from commit 28b66ae919)
 2016-06-13 11:23:10 +05:30
Karthik Kambatla a92be02c2e YARN-5124. Modify AMRMClient to set the ExecutionType in the ResourceRequest. Addendum to fix compilation. (Arun Suresh via kasha) 2016-06-12 14:28:14 -07:00
Arun Suresh f3c7a99e33 YARN-5212. Run existing ContainerManager tests using QueuingContainerManagerImpl. (Konstantinos Karanasos via asuresh) 
(cherry picked from commit 7dae2b3bc4)
 2016-06-12 10:06:18 -07:00
Arun Suresh fccb641942 YARN-5124. Modify AMRMClient to set the ExecutionType in the ResourceRequest. (asuresh) 
(cherry picked from commit 5143277958)
 2016-06-12 09:43:19 -07:00
Arun Suresh 5985221b46 YARN-5082. Limit ContainerId increase in fair scheduler if the num of node app reserved reached the limit (sandflee via asuresh) 
(cherry picked from commit 5279af7cd4)
 2016-06-10 22:34:33 -07:00
Rohith Sharma K S efdfa68911 YARN-5197. RM leaks containers if running container disappears from node update. Contributed by Jason Lowe. 
(cherry picked from commit e0f4620cc7)
 2016-06-11 10:31:00 +05:30
Wangda Tan e91e9e596d YARN-3426. Add jdiff support to YARN. (vinodkv via wangda) 
(cherry picked from commit 03fc6b1bb0)
 2016-06-10 09:53:40 -07:00