Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,848 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

5553 Commits

Author SHA1 Message Date
Jim Brennan 4ef9cf9d71 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32ba)
 2020-10-08 19:01:38 +00:00
Jim Brennan ecf91638a8 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:36:51 +00:00
Adam Antal b7420eb4b0 YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760df)
 2020-10-05 10:39:14 +02:00
Eric E Payne 947b0a154a YARN-9809. Added node manager health status to resource manager registration call. Contributed by Eric Badger (ebadger). 2020-09-28 18:50:44 +00:00
Jim Brennan 1efb54bd52 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-15 16:27:08 +00:00
Eric E Payne 5b14af6d09 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed17)
 2020-09-11 16:46:28 +00:00
bibinchundatt b5d24d646c YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80a)
 2020-09-08 21:05:26 +00:00
Eric Badger 01ada576f3 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d)
 2020-09-08 16:14:26 +00:00
Adam Antal 696494d663 YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 
(cherry picked from commit 34fe74da0e)
 2020-09-07 12:01:35 +02:00
Sunil G 94723bff64 Revert "YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke." 
This reverts commit 2a40a33dfe.
 2020-08-20 19:15:10 +05:30
Sunil G 2a40a33dfe YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f442)
 2020-08-19 12:00:33 +05:30
Jonathan Hung 17d18a2a3a YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:52 -07:00
Eric Badger 9a1db93b1b YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925e)
 2020-08-05 19:03:48 +00:00
Eric E Payne 863689ff9a YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166)
 2020-07-30 17:08:02 +00:00
Jonathan Hung ffb920de2a YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:44:17 -07:00
Eric Badger 7350773b69 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
 2020-07-24 22:55:08 +00:00
Ayush Saxena 27a97e4f28 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:39:49 +05:30
Akira Ajisaka 0d949d375e HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 
(cherry picked from commit 1defe3a65a)
 2020-07-20 13:20:34 -07:00
Ahmed Hussein 8fd3dcc9ce HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 12:05:49 -05:00
Ahmed Hussein 43a865dc07 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 10:18:47 -05:00
Eric Badger 09f1547697 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan.
 2020-07-14 18:26:15 +00:00
Eric E Payne 52f2303b5a YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b75)
 2020-07-13 21:34:21 +00:00
Masatake Iwasaki 936dece92b YARN-10347. Fix double locking in CapacityScheduler#reinitialize in branch-3.1. 
(cherry picked from commit 4fa8055aa4)
 2020-07-09 14:19:22 +09:00
Eric E Payne e6794f2fc4 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 2020-06-29 19:21:04 +00:00
Eric Badger 7363931942 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan
 2020-06-12 19:08:36 +00:00
Szilard Nemeth 30d7a06686 YARN-10295. CapacityScheduler NPE can cause apps to get stuck without resources. Contributed by Benjamin Teke 2020-06-10 18:16:21 +02:00
Szilard Nemeth 8bef26a607 YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:01:20 +02:00
Eric E Payne 034d458511 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db302)
 2020-06-09 21:09:11 +00:00
Szilard Nemeth 54c89ffad4 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:49:54 +02:00
Jonathan Hung f31146bc1f YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb66)
(cherry picked from commit 7a323a45aa)
 2020-05-29 10:43:33 -07:00
Jonathan Hung a7ea55e015 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c122)
(cherry picked from commit 564d3211f2)
 2020-05-12 10:52:58 -07:00
Szilard Nemeth d345994468 YARN-9444. YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource. Contributed by Gergely Pollak 
(cherry picked from commit 52e9ee39a1)
 2020-05-07 18:17:06 +00:00
Ahmed Hussein 7740b88ee9 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:32:36 -05:00
Ahmed Hussein b23a585cb1 YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:49:45 -05:00
Szilard Nemeth 1dabbd5006 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 2020-04-28 17:52:14 +02:00
Szilard Nemeth f445487d50 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-27 09:59:13 +02:00
Szilard Nemeth 0dbb02a76c YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:31:33 +02:00
Szilard Nemeth 3b67dc24aa YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-04-24 11:15:53 +02:00
Akira Ajisaka 7b036c512f
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961)
 2020-04-24 10:47:36 +09:00
Wei-Chiu Chuang 48f1c8ffb6 Revert "YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja" 
This reverts commit a2067aafa9.
 2020-04-23 12:37:21 -07:00
Szilard Nemeth c81844d8a5 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:57:18 +02:00
Szilard Nemeth 764fa92c9f YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-04-23 14:51:07 +02:00
Szilard Nemeth 73cb3d3cb3 YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 2020-04-18 09:38:22 +02:00
Jonathan Hung d1af4e0fae YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit 49ae9b2137)
 2020-04-17 10:36:16 -07:00
Szilard Nemeth 2f01a91428 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-04-15 08:24:15 +02:00
Szilard Nemeth 58e559b5ac YARN-9354. Resources should be created with ResourceTypesTestHelper instead of TestUtils. Contributed by Andras Gyori 2020-04-15 08:16:15 +02:00
Szilard Nemeth a7eb95f114 YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. 2020-04-15 08:05:07 +02:00
Szilard Nemeth 9af7d905f9 YARN-5277. When localizers fail due to resource timestamps being out, provide more diagnostics. Contributed by Siddharth Ahuja 2020-04-15 07:55:49 +02:00
Jonathan Hung 54599b177c YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
(cherry picked from commit e3a52804b03d646f15048c078f8c5292d5cbecfa)
 2020-04-09 10:37:36 -07:00
Szilard Nemeth d2853d1bb0 YARN-10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-04-09 17:40:26 +02:00
Szilard Nemeth 9a66477e4d YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI. Contributed by Siddharth Ahuja 2020-04-09 12:22:12 +02:00
Akira Ajisaka 2a2c3a4094
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 
(cherry picked from commit e53d472bb0)
 2020-04-09 01:49:28 +09:00
Wilfred Spiegelenburg a2067aafa9
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 
Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c

(cherry picked from commit 2214005c0f)
 2020-04-08 13:12:31 +10:00
Akira Ajisaka 6d5f87b228
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen Zhao. 
(cherry picked from commit c162648aff)
 2020-04-01 16:06:41 +09:00
Jonathan Hung 5d3fb0ebe9 YARN-10200. Add number of containers to RMAppManager summary 
(cherry picked from commit 2de0572cdc1c6fdbfaab108b169b2d5b0c077e86)
 2020-03-25 10:27:48 -07:00
Siyao Meng 29e1880d27
HADOOP-16935. Backport HADOOP-10848. Cleanup calling of sun.security.krb5.Config. (#1912) 
(cherry picked from commit 0d47d283a6)

Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
 2020-03-24 16:01:33 -07:00
Szilard Nemeth 9e0d742025 YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-10 16:39:03 +01:00
Eric E Payne 153eac1d21 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1)
 2020-03-10 14:28:13 +00:00
Inigo Goiri 733f9b76b6 YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d)
 2020-02-27 13:19:43 -08:00
Elixir Kook 7d0ff2dc85
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f92)
 2020-02-26 17:32:09 +09:00
Szilard Nemeth 92ad3bd099 YARN-10143. YARN-10101 broke Yarn logs CLI. Contributed by Adam Antal 2020-02-24 21:27:31 +01:00
Sunil G de63115a2a YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 
(cherry picked from commit 6526f95bd2)
 2020-02-19 11:18:19 +05:30
Szilard Nemeth 6aec712c6c YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-11 09:18:44 +01:00
Sunil G 95b1cbcbd4 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 
(cherry picked from commit 28f730b317)
 2020-02-09 21:15:31 +05:30
Jonathan Hung aca930402c YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e)
 2020-02-05 11:17:03 -08:00
Prabhu Joseph 7136ebbb7a YARN-10022. Add RM Rest API to validate a CapacityScheduler Config with delta change 
Contributed by Kinga Marton.

(cherry-picked from commit 1ab9c692fa)
 2020-02-04 14:06:23 +05:30
Eric Badger 5736ecd123 YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 04:07:28 +00:00
Abhishek Modi be412546be YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 
(cherry picked from commit d2d963f3d4)
 2020-01-23 15:23:45 +00:00
Szilard Nemeth dcc16b07fa YARN-10083. Addendum to fix compilation error due to missing import 2020-01-22 17:18:03 +01:00
Szilard Nemeth 1e7679035f YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-22 16:50:15 +01:00
Szilard Nemeth da416c826f YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:18:35 +01:00
Szilard Nemeth bbdc39c13e YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-22 15:49:35 +01:00
Szilard Nemeth c411a1468f YARN-8148. Update decimal values for queue capacities shown on queue status CLI. Contributed by Prabhu Joseph 2020-01-20 09:29:52 +01:00
Szilard Nemeth 805589fc71 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-16 18:51:56 +01:00
Szilard Nemeth 0c2e312fef YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-15 10:51:02 +01:00
Szilard Nemeth 6a7dfb3bf3 YARN-10026. Pull out common code pieces from ATS v1.5 and v2. Contributed by Adam Antal 2020-01-12 13:54:08 +01:00
Eric E Payne 3ba0fd1e50 YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 
(cherry picked from commit 93233a7d6e)
 2020-01-09 17:22:10 +00:00
Eric Badger 58db04ce15 YARN-8672. TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out. Contributed by Chandni Singh and Jim Brennan. 2020-01-08 19:44:43 +00:00
Eric E Payne 5e2be81fcb YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc)
 2020-01-08 19:30:09 +00:00
Eric E Payne b20ce118da YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a17)
 2020-01-08 17:42:41 +00:00
Prabhu Joseph ad98a30810 YARN-10053. Use Shared Group Mapping Service in Placement Rules. Contributed by Wilfred Spiegelenburg. 
(cherry Picked from commit 217b56ffdd)
 2020-01-02 14:30:01 +05:30
Akira Ajisaka 85da5cb870
YARN-10055. bower install fails. (#1778) 
(cherry picked from commit 34ff7dbaf5)
 2019-12-23 20:26:40 +09:00
Eric Badger 355ec33416 YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne 2019-12-20 19:32:36 +00:00
Jonathan Hung 0707d0a0ae YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92)
 2019-12-18 13:18:22 -08:00
Jonathan Hung 01e2c996ff YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit fddc3d55c3)
 2019-12-18 10:51:57 -08:00
Eric Badger 18b515322c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 
(cherry picked from commit f47dcf2d4c)
 2019-12-17 17:36:52 +00:00
Akira Ajisaka a3cdba7713
YARN-9985. Unsupported transitionToObserver option displaying for rmadmin command. Contributed by Ayush Saxena. 
(cherry picked from commit dc66de7448)
 2019-12-09 18:38:47 +09:00
Jonathan Hung 9228e3f0ad YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc)
 2019-12-08 16:36:08 -08:00
Jonathan Hung 8badcd989e Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit b5235f1ed0.
 2019-12-08 16:36:00 -08:00
Jonathan Hung b5235f1ed0 YARN-10012. Guaranteed and max capacity queue metrics for custom resources 
(cherry picked from commit 92bce918dc)
 2019-12-08 16:03:34 -08:00
Sunil G 69dc329acc YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf)
 2019-12-08 14:40:45 -08:00
Szilard Nemeth 2ad7b90505 YARN-9993. Remove incorrectly committed files from YARN-9011. Contributed by Wilfred Spiegelenburg 2019-11-28 12:39:54 +01:00
Sunil G f9b872b6ec YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. 
Contributed by Prabhu Joseph.
 2019-11-27 23:14:33 +05:30
Szilard Nemeth 8eda9fcab8 YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. 
Contributed by Prabhu Joseph.
 2019-11-27 22:24:06 +05:30
Szilard Nemeth 3fc8930129 YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-26 14:26:58 +01:00
HUAN-PING SU 59a6261e81
YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 
(cherry picked from commit f8e36e03b4)
 2019-11-25 15:29:37 +09:00
Szilard Nemeth dcc453b4b8 YARN-9968. Public Localizer is exiting in NodeManager due to NullPointerException. Contributed by Tarun Parimi 2019-11-22 12:59:35 +01:00
Tao Yang af495192a5 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:14:16 +08:00
Eric Yang 6951689f4c YARN-9983. Fixed typo in YARN Service overview. 
Contributed by Denes Gerencser
 2019-11-19 14:19:49 -05:00
Abhishek Modi 31591bb296 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 
(cherry picked from commit 751b5a1ac8)
 2019-11-19 16:49:14 +05:30
Sunil G b04f152876 YARN-9909. Offline Format of YarnConfigurationStore. Contributed by Prabhu Joseph 2019-11-19 16:10:02 +05:30
Sunil G c1ec51696c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit ea68756c0c)
 2019-11-19 14:12:03 +05:30
Sunil G 049279bb66 YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 215f2052fc)
 2019-11-19 14:05:11 +05:30
Jonathan Eagles 254e18dcaf Revert "YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph." 
This reverts commit 11c763c220.
 2019-11-05 15:10:01 -06:00
Sunil G 597b315811 YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 
(cherry picked from commit 51e7d1b37e)
 2019-11-04 23:28:39 +05:30
Sunil G 11c763c220 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 
(cherry picked from commit d462308e04)
 2019-11-03 08:48:04 +05:30
Jonathan Hung 5d2ffcc7aa Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
 2019-10-30 20:59:20 -07:00
Eric Badger fa6b27ea8d YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:15:48 +00:00
Eric E Payne ea574087d1 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f)
 2019-10-21 20:56:40 +00:00
Eric E Payne 23b72d8ae1 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988)
 2019-10-16 21:13:02 +00:00
Sunil G 9672b81fa3 YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 
(cherry picked from commit 090f73a9aa)
 2019-10-16 18:15:34 +05:30
Haibo Chen 3a5474c61e YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af4)
 2019-10-15 22:14:33 +00:00
Haibo Chen 1344823d4d YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88)
 2019-10-15 15:32:01 +00:00
Szilard Nemeth b10fdd136a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:24:59 +02:00
Akira Ajisaka eb4bd54938
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. (#1634) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e40e2d6ad5)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManagerImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/lifecycle/VolumeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterServiceInterceptor.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
 2019-10-10 15:27:54 +09:00
Szilard Nemeth da35a22083 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 42177e8b78.
 2019-10-09 19:58:46 +02:00
Szilard Nemeth 05966ce204 YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-10-09 14:18:06 +02:00
Szilard Nemeth 42177e8b78 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab95168)
 2019-10-09 13:28:01 +02:00
Szilard Nemeth 0ddb48a303 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3)
 2019-10-09 13:12:03 +02:00
Sunil G 9cb3ab058f YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit be901f4962)
 2019-10-09 15:54:09 +05:30
Jonathan Hung ad3c98456d YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 
(cherry picked from commit eebd313d76ed742fe82292bd8c0184970cdc5692)
 2019-10-08 11:17:05 -07:00
Szilard Nemeth 9b4aba49d1 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:41:46 +02:00
Sunil G 5704f15589 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 3a0afcfb7f.
 2019-10-05 09:16:04 +05:30
Sunil G 3a0afcfb7f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit 4510970e2f)
 2019-10-04 21:49:49 +05:30
Sunil G 312cfa994a YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 
(cherry picked from commit 99cd7572f1)
 2019-10-01 20:07:18 +05:30
bibinchundatt 5cd6eb2a18 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:04:58 +05:30
Sunil G 52f815d39d YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 
(cherry picked from commit 137546a78a)
 2019-10-01 09:09:22 +05:30
Jonathan Hung f4f210d2e5 Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
 2019-09-25 17:49:37 -07:00
Jonathan Hung 806c7b7dfb YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
 2019-09-24 13:51:54 -07:00
Jonathan Hung a1fa9a8a7f YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d)
 2019-09-23 13:12:57 -07:00
Sunil G 3e0025d877 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b)
 2019-09-18 09:22:48 +05:30
Weiwei Yang 7ec229244a YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 
(cherry picked from commit f8c14326ee)
 2019-09-18 09:19:38 +08:00
Eric Yang 345ef049df YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k. 
Contributed by Tarun Parimi

(cherry picked from commit eefe9bc85c)
 2019-09-17 13:22:53 -04:00
Jonathan Hung 1dbf87c9ff YARN-9824. Fall back to configured queue ordering policy class name 
(cherry picked from commit f8f8598ea5)
 2019-09-10 15:26:57 -07:00
bibinchundatt e10050678d YARN-8948. PlacementRule interface should be for all YarnSchedulers. Contributed by Bibin A Chundatt. 
(cherry picked from commit a68d766e87)
 2019-09-09 19:03:40 -07:00
Abhishek Modi f6cc887f35 YARN-9821. NM hangs at serviceStop when ATSV2 Backend Hbase is Down. Contributed by Prabhu Joseph. 2019-09-09 15:44:45 +05:30
Jonathan Hung 1f0449ddfb YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:24:17 -07:00
Jonathan Hung 45220d1157 YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 
(cherry picked from commit 43e389b980)
 2019-09-08 19:11:47 -07:00
Rohith Sharma K S 7d5bb2ebb7 Preparing for 3.2.2-SNAPSHOT development. 2019-09-07 08:52:08 +05:30
Wangda Tan 0e77347972 YARN-9813. RM does not start on JDK11 when UIv2 is enabled. (Adam Antal/Eric Yang via wangda) 
Change-Id: I18b8edc930b2efa0652f59c246931ad0d46827f3
(cherry picked from commit 34b82e6da0)
 2019-09-06 19:19:05 -07:00
Tao Yang 9ee257e353 YARN-8995. Log events info in AsyncDispatcher when event queue size cumulatively reaches a certain number every time(addendum). Contributed by Jonathan Hung. 2019-09-07 08:25:15 +08:00
Tao Yang cfce39023d YARN-9795. ClusterMetrics to include AM allocation delay. Contributed by Fengnan Li. 2019-09-07 07:53:46 +08:00
Jonathan Hung 9c9ff07249 YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 10:49:43 -07:00
Jonathan Hung 1f685efc73 YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:13:12 -07:00
Jonathan Hung 79ca399a30 YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 
(cherry picked from commit 0ccf4b0fe1)
 2019-09-05 14:05:45 -07:00
Billie Rinaldi 66627749d0 YARN-9718. Fixed yarn.service.am.java.opts shell injection. Contributed by Eric Yang 
(cherry picked from commit 2e2e5401f2)
 2019-09-05 12:54:20 -07:00
Rohith Sharma K S 4d9c5300e2 YARN-8567. Fetching yarn logs fails for long running application if it is not present in timeline store. Contributed by Tarun Parimi. 2019-09-05 18:16:35 +05:30
Eric Yang b87a727ff4 YARN-9374. Improve Timeline service resilience when HBase is unavailable. 
Contributed by Prabhu Joseph and Szilard Nemeth
 2019-09-05 16:32:18 +05:30
Eric Yang 02779cdc3a YARN-8499 ATSv2 Generalize TimelineStorageMonitor. 
Contributed by Prabhu Joseph
 2019-09-05 16:31:35 +05:30
Eric Yang 6110af2d1d YARN-7537. Add ability to load hbase config from distributed file system. 
Contributed by Prabhu Joseph
 2019-09-05 16:28:04 +05:30