Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,541 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2838 Commits

Author SHA1 Message Date
Rohith Sharma K S 74ac78b3c0 YARN-5375. invoke MockRM#drainEvents implicitly in MockRM methods to reduce test failures. Contributed by sandflee. 
(cherry picked from commit d65603517e)
 2016-11-16 15:17:46 +05:30
Xiao Chen 3497221fcc YARN-5875. TestTokenClientRMService#testTokenRenewalWrongUser fails. Contributed by Gergely Novák. 
(cherry picked from commit d5c396ccb361e1a0cf4307ce1d1b37fae0eec250)
 2016-11-15 13:58:02 -08:00
Mingliang Liu 4e0fcff8ab HADOOP-13427. Eliminate needless uses of FileSystem#{exists(), isFile(), isDirectory()}. Contributed by Steve Loughran and Mingliang Liu 
(cherry picked from commit 5af572b644)
 2016-11-15 11:36:08 -08:00
Naganarasimha 260f3a9dc9 YARN-4355. NPE while processing localizer heartbeat. Contributed by Varun Saxena & Jonathan Hung. 
(cherry picked from commit 7ffb9943b8)
 2016-11-15 15:44:25 +05:30
Naganarasimha 7e9a6b653e Reverting the patch due to the issue raised in YARN-5765 
Revert "YARN-5287. LinuxContainerExecutor fails to set proper permission. Contributed by Ying Zhang."

This reverts commit 93b768d0d1.
 2016-11-15 11:05:03 +05:30
Rohith Sharma K S 528ef2407c YARN-5874. RM -format-state-store and -remove-application-from-state-store commands fail with NPE. Contributed by Varun Saxena. 
(cherry picked from commit b7070f3308)
 2016-11-15 11:03:38 +05:30
Rohith Sharma K S fad6de3080 YARN-5873. RM crashes with NPE if generic application history is enabled. Contributed by Varun Saxena. 
(cherry picked from commit 296c5de0cf)
 2016-11-15 11:03:27 +05:30
Karthik Kambatla ea74c7b7ae YARN-4033. In FairScheduler, parent queues should also display queue status. (Siqi Li via kasha) 
(cherry picked from commit 79448d4ab1)
 2016-11-13 21:39:44 -08:00
Jian He 9b0a2cb321 YARN-5825. ProportionalPreemptionalPolicy should use readLock over LeafQueue instead of synchronized block. Contributed by Sunil G 
(cherry picked from commit fad9609d13)
 2016-11-11 15:16:43 -08:00
Wangda Tan a76e214556 YARN-5552. Add Builder methods for common yarn API records. (Tao Jie via wangda) 
(cherry picked from commit ede1a473f5)
 2016-11-11 13:42:45 -08:00
Naganarasimha bde95f90e7 YARN-5545, Fix issues related to Max App in capacity scheduler. Contributed by Bibin A Chundatt 2016-11-12 00:52:35 +05:30
Eric Payne dd5b9dabf9 YARN-4218. Metric for resource*time that was preempted. Contributed by Chang Li. 2016-11-10 23:02:42 +00:00
Karthik Kambatla f4c8dc4783 YARN-5834. TestNodeStatusUpdater.testNMRMConnectionConf compares nodemanager wait time to the incorrect value. (Chang Li via kasha) 
(cherry picked from commit 3a98419532)
 2016-11-10 14:09:33 -08:00
Karthik Kambatla f7b25420a1 YARN-5453. FairScheduler#update may skip update demand resource of child queue/app if current demand reached maxResource. (sandflee via kasha) 
(cherry picked from commit 86ac1ad9fd)
 2016-11-09 23:44:26 -08:00
Varun Saxena 06c8ebdb83 YARN-5862. TestDiskFailures.testLocalDirsFailures failed (Yufei Gu via Varun Saxena) 
(cherry picked from commit c202a10923)
 2016-11-10 11:43:57 +05:30
Jian He a422740bd7 YARN-5611. Provide an API to update lifetime of an application. Contributed by Rohith Sharma K S 2016-11-09 16:39:46 -08:00
Naganarasimha 8a2998c08c YARN-4498. Application level node labels stats to be available in REST (addendum patch). Contributed by Bibin A Chundatt. 2016-11-10 05:21:07 +05:30
Daniel Templeton 42147bbbf7 YARN-4329. [YARN-5437] Allow fetching exact reason as to why a submitted app 
is in ACCEPTED state in Fair Scheduler (Contributed by Yufei Gu)

(cherry picked from commit 59ee8b7a88)
 2016-11-09 13:18:28 -08:00
Arun Suresh 4a023b4414 YARN-5833. Addendum patch to include missing changes to yarn-default.xml 
(cherry picked from commit 280357c29f)
 2016-11-09 07:16:39 -08:00
Andrew Wang 4f25fd975e HADOOP-13789. Hadoop Common includes generated test protos in both jar and test-jar. Contributed by Sean Busbey. 
(cherry picked from commit e1c6ef2efa)
 2016-11-08 16:33:58 -08:00
Subru Krishnan 797c7bfab9 YARN-5833. Add validation to ensure default ports are unique in Configuration. (Konstantinos Karanasos via Subru). 
(cherry picked from commit 29e3b3417c)
 2016-11-08 14:39:09 -08:00
Jason Lowe 8bb264c70d YARN-5356. NodeManager should communicate physical resource capability to ResourceManager. Contributed by Inigo Goiri 
(cherry picked from commit 3f93ac0733)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/NodeStatusUpdaterImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
 2016-11-08 22:16:53 +00:00
Jian He 6cdcab9078 YARN-5716. Add global scheduler interface definition and update CapacityScheduler to use it. Contributed by Wangda Tan 2016-11-07 10:31:43 -08:00
Jason Lowe 27ed2c526a YARN-5837. NPE when getting node status of a decommissioned node after an RM restart. Contributed by Robert Kanter 
(cherry picked from commit 6bb741ff0e)
 2016-11-04 22:21:45 +00:00
Sunil 3eed75a9ab YARN-5802. updateApplicationPriority api in scheduler should ensure to re-insert app to correct ordering policy. Contributred by Bibin A Chundatt 
(cherry picked from commit 19b3779ae7)
 2016-11-04 17:09:19 +05:30
Varun Vasudev 7676b89f43 YARN-5822. Log ContainerRuntime initialization error in LinuxContainerExecutor. Contributed by Sidharta Seethana. 
(cherry picked from commit 9ee0e3172e)
 2016-11-03 22:34:38 +05:30
Jason Lowe e2917180ee YARN-4862. Handle duplicate completed containers in RMNodeImpl. Contributed by Rohith Sharma K S 
(cherry picked from commit 352cbaa7a5)
 2016-11-03 14:03:56 +00:00
Naganarasimha 0c7caba087 YARN-5720. Update document for "rmadmin -replaceLabelOnNode". Contributred by Tao Jie 
(cherry picked from commit 0e75496049)
 2016-11-03 12:50:38 +05:30
Varun Saxena b818773919 YARN-5815. Random failure of TestApplicationPriority.testOrderOfActivatingThePriorityApplicationOnRMRestart (Bibin A Chundatt via Varun Saxena) 2016-11-03 00:41:53 +05:30
Naganarasimha 2612b509ef YARN-5697. Use CliParser to parse options in RMAdminCLI. Contributed by Tao jie 
(cherry picked from commit aacf214a27)
 2016-11-02 04:46:24 +05:30
Jason Lowe f8d653ef34 YARN-5001. Aggregated Logs root directory is created with wrong group if nonexistent. Contributed by Haibo Chen 
(cherry picked from commit 76893a4100)
 2016-11-01 20:24:22 +00:00
Varun Saxena 91ddea5bfb YARN-5788. Apps not activiated and AM limit resource in UI and REST not updated after -replaceLabelsOnNode (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:26:38 +05:30
Varun Saxena 2f0a101ab9 YARN-5805. Add isDebugEnabled check for debug logs in nodemanager (Bibin A Chundatt via Varun Saxena) 2016-11-01 16:23:12 +05:30
Karthik Kambatla 2226ae4924 YARN-4998. Minor cleanup to UGI use in AdminService. (Daniel Templeton via kasha) 
(cherry picked from commit 733aa99313)
 2016-10-31 16:26:15 -07:00
Varun Saxena 92b7491f23 YARN-5809. AsyncDispatcher possibly invokes multiple shutdown threads when handling exception (Jian He via Varun Saxena) 2016-11-01 04:37:56 +05:30
Wangda Tan cef281abe6 YARN-2009. CapacityScheduler: Add intra-queue preemption for app priority support. (Sunil G via wangda) 
(cherry picked from commit 90dd3a8148)
 2016-10-31 15:23:16 -07:00
Daniel Templeton 9d13a13160 YARN-5793. Trim configuration values in DockerLinuxContainerRuntime (Contributed by Tianyin Xu via Daniel Templeton) 
(cherry picked from commit f3eb4c3c73)
 2016-10-31 14:01:15 -07:00
Daniel Templeton ce13463e7a YARN-4396. Log the trace information on FSAppAttempt#assignContainer (Contributed by Yiqun Li via Daniel Templeton) 
(cherry picked from commit 2528bea67f)
 2016-10-31 13:36:24 -07:00
Daniel Templeton adceebc400 YARN-4907. Make all MockRM#waitForState consistent. (Contributed by Yufei Gu via Daniel Templeton) 
(cherry picked from commit 155f06e0c1a5a15365fb620f3802dca8aa46c287)
 2016-10-31 13:21:45 -07:00
Naganarasimha e4023f0d81 YARN-4498. Application level node labels stats to be available in REST. Contributed by Bibin A Chundatt. 2016-10-31 19:07:56 +05:30
Varun Saxena 0e6fcd3bd2 YARN-5773. RM recovery too slow due to LeafQueue#activateApplications (Bibin A Chundatt via Varun Saxena) 2016-10-29 13:51:02 +05:30
Jason Lowe b85b5c6b94 YARN-4963. capacity scheduler: Make number of OFF_SWITCH assignments per heartbeat configurable. Contributed by Nathan Roberts 
(cherry picked from commit 1eae719bce)
 2016-10-28 17:31:29 +00:00
Jason Lowe ea9a1be109 YARN-5767. Fix the order that resources are cleaned up from the local Public/Private caches. Contributed by Chris Trezzo 
(cherry picked from commit 1b79c417dc)
 2016-10-28 15:59:57 +00:00
Jason Lowe 2ab80a148a YARN-5027. NM should clean up app log dirs after NM restart. Contributed by sandflee 
(cherry picked from commit 7146359bfd)
 2016-10-28 15:49:54 +00:00
Jason Lowe 0f224d405f HADOOP-13770. Shell.checkIsBashSupported swallowed an interrupted exception. Contributed by Wei-Chiu Chuang 
(cherry picked from commit c017171da0)
 2016-10-28 15:06:23 +00:00
Naganarasimha 62d8536df6 Revert "YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin" 
This reverts commit b92ad5d857.
 2016-10-28 12:58:42 +05:30
Naganarasimha b92ad5d857 YARN-5794. Fix the asflicense warnings. Contributed by Yiqun Lin 
(cherry picked from commit 3de130252d)
 2016-10-28 12:44:14 +05:30
Karthik Kambatla 950bfed1d3 YARN-4743. FairSharePolicy breaks TimSort assumption. (Zephyr Guo and Yufei Gu via kasha) 
(cherry picked from commit 4df8ed63ed)
 2016-10-27 17:48:24 -07:00
Subru Krishnan 334fd9e83f YARN-2306. Add test for leakage of reservation metrics in fair scheduler. (Hong Zhiguo and Yufei Gu via subru). 
(cherry picked from commit b2c4f24c31)
 2016-10-27 17:44:05 -07:00
Li Lu 2b0fd1f4e2 YARN-4668. Reuse objectMapper instance in Yarn. (Yiqun Lin via gtcarrera9) 
(cherry picked from commit 221582c4ab)
 2016-10-27 15:26:05 -07:00
Carlo Curino 58ac40b552 YARN-3432. Cluster metrics have wrong Total Memory when there is reserved memory on CS. (Brahma Reddy Battula via curino) 
(cherry picked from commit 892a8348fc)
 2016-10-27 15:22:04 -07:00
Subru Krishnan 4274600b95 YARN-3568. TestAMRMTokens should use some random port. (Takashi Ohnishi via Subru). 
(cherry picked from commit 79ae78dcbe)
 2016-10-27 15:13:01 -07:00
Varun Saxena f572448809 YARN-4388. Cleanup mapreduce.job.hdfs-servers from yarn-default.xml (Junping Du via Varun Saxena) 2016-10-28 02:24:24 +05:30
Varun Saxena c34bc3d661 YARN-5308. FairScheduler: Move continuous scheduling related tests to TestContinuousScheduling (Kai Sasaki via Varun Saxena) 2016-10-28 02:23:25 +05:30
Jason Lowe e50215306d YARN-4831. Recovered containers will be killed after NM stateful restart. Contributed by Siqi Li 
(cherry picked from commit 7e3c327d31)
 2016-10-27 20:44:17 +00:00
Jason Lowe 8935b23706 YARN-5172. Update yarn daemonlog documentation due to HADOOP-12847. Contributed by Wei-Chiu Chuang 
(cherry picked from commit b4a8fbcbbc)
 2016-10-27 19:45:00 +00:00
Robert Kanter b7f7d42c27 YARN-4456. Clean up Lint warnings in nodemanager (templedf via rkanter) 
(cherry picked from commit 6fbfb501f2)
 2016-10-27 12:37:20 -07:00
Naganarasimha 15f9c18ae4 YARN-5420. Delete org.apache.hadoop.yarn.server.resourcemanager.resource.Priority as its not necessary. Contributed by Sunil G. 
(cherry picked from commit b3c15e4ef7)
 2016-10-27 18:23:27 +05:30
Varun Saxena 65fa9827fc YARN-5757. RM REST API documentation is not up to date (Miklos Szegedi via Varun Saxena) 2016-10-27 17:24:24 +05:30
Naganarasimha 3ff80a800d YARN-3848. TestNodeLabelContainerAllocation is timing out. Contributed by Varun Saxena 
(cherry picked from commit 6c8830992c)
 2016-10-27 17:14:34 +05:30
Varun Saxena 65c009ca00 YARN-5686. DefaultContainerExecutor random working dir algorigthm skews results (Vrushali C via Varun Saxena) 2016-10-27 14:03:02 +05:30
Varun Saxena a8cfa5054c YARN-5710. Fix inconsistent naming in class ResourceRequest (Yufei Gu via Varun Saxena) 2016-10-27 13:56:59 +05:30
Varun Saxena daf9a656b9 YARN-5752. TestLocalResourcesTrackerImpl#testLocalResourceCache times out (Eric Badger via Varun Saxena) 2016-10-27 13:46:13 +05:30
Rohith Sharma K S 3a167669a5 YARN-4363. In TestFairScheduler, testcase should not create FairScheduler redundantly. Conntributed by Tao Jie. 
(cherry picked from commit e29cba61a0)
 2016-10-27 11:59:10 +05:30
Rohith Sharma K S 07b531c10a YARN-4555. TestDefaultContainerExecutor#testContainerLaunchError fails on non-english locale environment. Contributed by Takashi Ohnishi. 
(cherry picked from commit b110c4b5e8)
 2016-10-27 11:35:38 +05:30
Akira Ajisaka e29d0a2fd8 YARN-5575. Many classes use bare yarn. properties instead of the defined constants. Contributed by Daniel Templeton. 
(cherry picked from commit d3bb69a667)
 2016-10-26 15:34:40 +09:00
Karthik Kambatla e4ba92abee YARN-5388. Deprecate and remove DockerContainerExecutor. (Daniel Templeton via kasha) 2016-10-25 15:26:27 -07:00
Karthik Kambatla aedd5c4c1b YARN-5677. RM should transition to standby when connection is lost for an extended period. (Daniel Templeton via kasha) 2016-10-25 13:15:58 -07:00
Karthik Kambatla ab9105d814 YARN-5754. Null check missing for earliest in FifoPolicy. (Yufei Gu via kasha) 
(cherry picked from commit a71fc81655)
 2016-10-24 22:19:56 -07:00
Subru Krishnan 81595a127f YARN-5711. Propogate exceptions back to client when using hedging RM failover provider. 
(cherry picked from commit 0a166b1347)
 2016-10-24 19:02:03 -07:00
Jian He bb8214cab8 Revert YARN-4126. RM should not issue delegation tokens in unsecure mode. 2016-10-23 15:56:22 -07:00
Karthik Kambatla 440a6326d1 YARN-5047. Refactor nodeUpdate across schedulers. (Ray Chiang via kasha) 
(cherry picked from commit 754cb4e30f)
 2016-10-20 21:18:13 -07:00
Karthik Kambatla c2cabce2e8 YARN-4911. Bad placement policy in FairScheduler causes the RM to crash 
(cherry picked from commit a064865abf)
 2016-10-20 20:57:31 -07:00
Robert Kanter e02c756f16 YARN-5466. DefaultContainerExecutor needs JavaDocs (templedf via rkanter) 
(cherry picked from commit f5d9235914)
 2016-10-17 14:29:42 -07:00
Varun Saxena e9a58691ab YARN-5599. Publish AM launch command to ATS (Rohith Sharma K S via Varun Saxena) 2016-10-14 23:32:29 +05:30
Chris Douglas f0f2681b4f YARN-5717. Add tests for container-executor is_feature_enabled. Contributed by Sidharta Seethana 
(cherry picked from commit cf3f43e95b)
 2016-10-13 20:49:50 -07:00
Jason Lowe 28bd08798d YARN-5551. Ignore file backed pages from memory computation when smaps is enabled. Contributed by Rajesh Balamohan 
(cherry picked from commit ecb51b857a)
 2016-10-11 15:14:08 +00:00
Naganarasimha dc5f7a9d5e YARN-5057. Resourcemanager.security.TestDelegationTokenRenewer fails in trunk. Contributed by Jason Lowe. 
(cherry picked from commit 0773ffd0f8)
 2016-10-10 18:06:28 -04:00
Junping Du 8bee319130 YARN-5659. getPathFromYarnURL should use standard methods. Contributed by Sergey Shelukhin. 
(cherry picked from commit 459a4833a9)
 2016-10-07 07:47:16 -07:00
Rohith Sharma K S 94a6f65989 YARN-5101. YARN_APPLICATION_UPDATED event is parsed in ApplicationHistoryManagerOnTimelineStore#convertToApplicationReport with reversed order. Contributed by Sunil G. 
(cherry picked from commit 4d2f380d78)
 2016-10-06 20:43:47 +05:30
Jian He caafa980af YARN-3139. Improve locks in AbstractYarnScheduler/CapacityScheduler/FairScheduler. Contributed by Wangda Tan 2016-10-06 07:55:14 -07:00
Akira Ajisaka 5f1432d98e MAPREDUCE-6789. Fix TestAMWebApp failure. Contributed by Daniel Templeton. 
(cherry picked from commit 272a21747e)
 2016-10-06 15:59:19 +09:00
Chris Douglas 539c3e4dc0 YARN-5702. Refactor TestPBImplRecords for reuse in other YARN modules. Contributed by Subru Krishnan 
(cherry picked from commit d65b957776)
 2016-10-05 12:19:21 -07:00
Varun Vasudev 3ce02387ea YARN-5704. Provide config knobs to control enabling/disabling new/work in progress features in container-executor. Contributed by Sidharta Seethana. 
(cherry picked from commit 0992708d79)
 2016-10-05 15:32:50 +05:30
Karthik Kambatla 1578781fe9 YARN-4767. Network issues can cause persistent RM UI outage. (Daniel Templeton via kasha) 
(cherry picked from commit 736d33cddd)
 2016-10-03 14:36:14 -07:00
Karthik Kambatla f8c2c7b2c0 YARN-5672. FairScheduler: Wrong queue name in log when adding application. (Wilfred Spiegelenburg via kasha) 
(cherry picked from commit 0da54e8848)
 2016-10-03 06:04:09 -07:00
Naganarasimha 04f594dbea YARN-4855. Should check if node exists when replace nodelabels. Contributeed by Tao Jie 
(cherry picked from commit 6e130c308c)
 2016-10-03 02:13:46 -04:00
Karthik Kambatla 57726eed83 YARN-5678. Log demand as demand in FSLeafQueue and FSParentQueue. (Yufei Gu via kasha) 
(cherry picked from commit 82857037b6)
 2016-10-02 22:10:07 -07:00
Subru Krishnan 9f43f89899 YARN-5384. Expose priority in ReservationSystem submission APIs. (Sean Po via Subru). 
(cherry picked from commit 3a3697deab)
 2016-09-30 19:43:29 -07:00
Robert Kanter 56ec975ea9 YARN-5693. Reduce loglevel to Debug in ContainerManagementProtocolProxy and AMRMClientImpl (yufeigu via rkanter) 
(cherry picked from commit 2549ee9d4c)
 2016-09-30 13:35:58 -07:00
Li Lu 919c0802fa YARN-4205. Addendum patch to fix branch-2. 2016-09-29 15:41:31 -07:00
Jian He 39f896e1a1 YARN-4205. Add a service for monitoring application life time out. Contributed by Rohith Sharma K S 
(cherry picked from commit 2ae5a3a5bf)
 2016-09-29 22:06:36 +08:00
Robert Kanter 4f8c7f819c YARN-5400. addendum patch to fix JDK 7 issue 2016-09-28 16:01:18 -07:00
Robert Kanter bc91e33d5e YARN-5400. Light cleanup in ZKRMStateStore (templedf via rkanter) 
(cherry picked from commit bcb2528a51)
 2016-09-28 14:57:14 -07:00
Varun Vasudev 3f1d3e3177 YARN-5662. Provide an option to enable ContainerMonitor. Contributed by Jian He. 
(cherry picked from commit bc2656f09f)
 2016-09-28 15:27:09 +05:30
Wangda Tan 379c2b3e5b YARN-3142. Improve locks in AppSchedulingInfo. (Varun Saxena via wangda) 
(cherry picked from commit 1831be8e73)
 2016-09-27 12:05:00 -07:00
Wangda Tan 269401dc83 HADOOP-13544. JDiff reports unncessarily show unannotated APIs and cause confusion while our javadocs only show annotated and public APIs. (vinodkv via wangda) 
(cherry picked from commit 875062b5bc)
 2016-09-27 11:27:25 -07:00
Naganarasimha 86ec99e82d YARN-5660. Wrong audit constants are used in Get/Put of priority in RMWebService. Contributed by Rohith Sharma K S 
(cherry picked from commit ebf528c443)
 2016-09-27 16:42:21 +05:30
Arun Suresh 7484d0b1b9 YARN-5609. Expose upgrade and restart API in ContainerManagementProtocol. Contributed by Arun Suresh 2016-09-26 23:55:13 +08:00
Akira Ajisaka 74f2df16a9 YARN-5663. Small refactor in ZKRMStateStore. Contributed by Oleksii Dymytrov. 
(cherry picked from commit 14a696f369)
 2016-09-26 15:01:13 +09:00