Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,066 Commits 326 Branches 371 Tags
Commit Graph

5370 Commits

Author SHA1 Message Date
Jonathan Hung
0707d0a0ae YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92e8bad889c1ef15e0baaade6de6de4d)
 2019-12-18 13:18:22 -08:00
Jonathan Hung
01e2c996ff YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit fddc3d55c3e309936216b8c61944e113509999e2)
 2019-12-18 10:51:57 -08:00
Eric Badger
18b515322c YARN-10033. TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources. Contributed by Eric Payne. 
(cherry picked from commit f47dcf2d4cf437dd67d14dd05c60648d9ff87843)
 2019-12-17 17:36:52 +00:00
Akira Ajisaka
a3cdba7713
YARN-9985. Unsupported transitionToObserver option displaying for rmadmin command. Contributed by Ayush Saxena. 
(cherry picked from commit dc66de744826e0501040f8c2ca9e1edc076a80cf)
 2019-12-09 18:38:47 +09:00
Jonathan Hung
9228e3f0ad YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc5d03560169642b71636800680a3292)
 2019-12-08 16:36:08 -08:00
Jonathan Hung
8badcd989e Revert "YARN-10012. Guaranteed and max capacity queue metrics for custom resources" 
This reverts commit b5235f1ed05f6506aa2c40825b2b647475a4d91a.
 2019-12-08 16:36:00 -08:00
Jonathan Hung
b5235f1ed0 YARN-10012. Guaranteed and max capacity queue metrics for custom resources 
(cherry picked from commit 92bce918dc5d03560169642b71636800680a3292)
 2019-12-08 16:03:34 -08:00
Sunil G
69dc329acc YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf79bbaa5e603565640d8289991d781f)
 2019-12-08 14:40:45 -08:00
Szilard Nemeth
2ad7b90505 YARN-9993. Remove incorrectly committed files from YARN-9011. Contributed by Wilfred Spiegelenburg 2019-11-28 12:39:54 +01:00
Sunil G
f9b872b6ec YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. 
Contributed by Prabhu Joseph.
 2019-11-27 23:14:33 +05:30
Szilard Nemeth
8eda9fcab8 YARN-9937. Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo. 
Contributed by Prabhu Joseph.
 2019-11-27 22:24:06 +05:30
Szilard Nemeth
3fc8930129 YARN-9011. Race condition during decommissioning. Contributed by Peter Bacsko 2019-11-26 14:26:58 +01:00
HUAN-PING SU
59a6261e81
YARN-9966. Code duplication in UserGroupMappingPlacementRule (#1709) 
(cherry picked from commit f8e36e03b4e65f173bf2330715dceafdb6111190)
 2019-11-25 15:29:37 +09:00
Szilard Nemeth
dcc453b4b8 YARN-9968. Public Localizer is exiting in NodeManager due to NullPointerException. Contributed by Tarun Parimi 2019-11-22 12:59:35 +01:00
Tao Yang
af495192a5 YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 2019-11-22 16:14:16 +08:00
Eric Yang
6951689f4c YARN-9983. Fixed typo in YARN Service overview. 
Contributed by Denes Gerencser
 2019-11-19 14:19:49 -05:00
Abhishek Modi
31591bb296 YARN-9791. Queue Mutation API does not allow to remove a config. Contributed by Prabhu Joseph. 
(cherry picked from commit 751b5a1ac871734c46cb934d100a5367f1dbf791)
 2019-11-19 16:49:14 +05:30
Sunil G
b04f152876 YARN-9909. Offline Format of YarnConfigurationStore. Contributed by Prabhu Joseph 2019-11-19 16:10:02 +05:30
Sunil G
c1ec51696c YARN-8373. RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit ea68756c0cd3e6e20b8e7045a8b7edd180ea4708)
 2019-11-19 14:12:03 +05:30
Sunil G
049279bb66 YARN-9984. FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 215f2052fc3b7e366e8bd1bd332663966fa9206c)
 2019-11-19 14:05:11 +05:30
Jonathan Eagles
254e18dcaf Revert "YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph." 
This reverts commit 11c763c22055fea367b19b338a3d8067f9386ba4.
 2019-11-05 15:10:01 -06:00
Sunil G
597b315811 YARN-9950. Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively. Contributed by Prabhu Joseph. 
(cherry picked from commit 51e7d1b37eb06165657befdae82c015055c58334)
 2019-11-04 23:28:39 +05:30
Sunil G
11c763c220 YARN-9949. Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo. Contributed by Prabhu Joseph. 
(cherry picked from commit d462308e0476053d1aae65eca2752eacce797e22)
 2019-11-03 08:48:04 +05:30
Jonathan Hung
5d2ffcc7aa Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
 2019-10-30 20:59:20 -07:00
Eric Badger
fa6b27ea8d YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:15:48 +00:00
Eric E Payne
ea574087d1 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-10-21 20:56:40 +00:00
Eric E Payne
23b72d8ae1 YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-10-16 21:13:02 +00:00
Sunil G
9672b81fa3 YARN-9900. Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format. Contributed by Prabhu Joseph. 
(cherry picked from commit 090f73a9aaeb4c03032a78c0a837478b527c27e0)
 2019-10-16 18:15:34 +05:30
Haibo Chen
3a5474c61e YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit 84e22a6af46db2859d7d2caf192861cae9b6a1a8)
 2019-10-15 22:14:33 +00:00
Haibo Chen
1344823d4d YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:32:01 +00:00
Szilard Nemeth
b10fdd136a YARN-8453. Additional Unit tests to verify queue limit and max-limit with multiple resource types. Contributed by Adam Antal 2019-10-15 13:24:59 +02:00
Akira Ajisaka
eb4bd54938
YARN-7243. Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanager. (#1634) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e40e2d6ad5cbe782c3a067229270738b501ed27e)

 Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/VolumeManagerImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/lifecycle/VolumeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterServiceTestBase.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterLauncher.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterServiceInterceptor.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceManager.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/TestRMAppTransitions.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairSchedulerConfiguration.java
 2019-10-10 15:27:54 +09:00
Szilard Nemeth
da35a22083 Revert "YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl" 
This reverts commit 42177e8b786efd7fa1365b68617382de022a5146.
 2019-10-09 19:58:46 +02:00
Szilard Nemeth
05966ce204 YARN-9552. FairScheduler: NODE_UPDATE can cause NoSuchElementException. Contributed by Peter Bacsko. 2019-10-09 14:18:06 +02:00
Szilard Nemeth
42177e8b78 YARN-9128. Use SerializationUtils from apache commons to serialize / deserialize ResourceMappings. Contributed by Zoltan Siegl 
(cherry picked from commit 6f1ab9516869dbfdee034f4a17bccc379a39fc31)
 2019-10-09 13:28:01 +02:00
Szilard Nemeth
0ddb48a303 YARN-9356. Add more tests to ratio method in TestResourceCalculator. Contributed by Zoltan Siegl 
(cherry picked from commit 35f093f5b3f310af49a8bb476e62173862f5af51)
 2019-10-09 13:12:03 +02:00
Sunil G
9cb3ab058f YARN-9873. Mutation API Config Change need to update Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit be901f49628a553cfe6a3942478cb528cce2b266)
 2019-10-09 15:54:09 +05:30
Jonathan Hung
ad3c98456d YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 
(cherry picked from commit eebd313d76ed742fe82292bd8c0184970cdc5692)
 2019-10-08 11:17:05 -07:00
Szilard Nemeth
9b4aba49d1 YARN-6715. Fix documentation about NodeHealthScriptRunner. Contributed by Peter Bacsko 2019-10-08 17:41:46 +02:00
Sunil G
5704f15589 Revert "YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph" 
This reverts commit 3a0afcfb7f196bee8e534a3e80211bfa077b2aee.
 2019-10-05 09:16:04 +05:30
Sunil G
3a0afcfb7f YARN-9873. Mutation API Config Change updates Version Number. Contributed by Prabhu Joseph 
(cherry picked from commit 4510970e2f7728d036c750b596985e5ffa357b60)
 2019-10-04 21:49:49 +05:30
Sunil G
312cfa994a YARN-9801. SchedConfCli does not work wiwith https mode. Contributed by Prabhu Joseph 
(cherry picked from commit 99cd7572f113cbbb8f5ef89af60af3734fe8cfa7)
 2019-10-01 20:07:18 +05:30
bibinchundatt
5cd6eb2a18 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:04:58 +05:30
Sunil G
52f815d39d YARN-9864. Format CS Configuration present in Configuration Store. Contributeed by Prabhu Joseph 
(cherry picked from commit 137546a78a45adbfb5ef338c1c2984a97b3cecc2)
 2019-10-01 09:09:22 +05:30
Jonathan Hung
f4f210d2e5 Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
 2019-09-25 17:49:37 -07:00
Jonathan Hung
806c7b7dfb YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
 2019-09-24 13:51:54 -07:00
Jonathan Hung
a1fa9a8a7f YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d3fdc29d500803cefd2931b54f44928)
 2019-09-23 13:12:57 -07:00
Sunil G
3e0025d877 YARN-9833. Race condition when DirectoryCollection.checkDirs() runs during container launch. Contributed by Peter Bacsko. 
(cherry picked from commit c474e24c0b73c0f52a7d5af2495355f4a0799344)
 2019-09-18 09:22:48 +05:30
Weiwei Yang
7ec229244a YARN-2255. YARN Audit logging not added to log4j.properties. Contributed by Aihua Xu. 
(cherry picked from commit f8c14326ee0026f2c57a4d061bff5fadec74b74d)
 2019-09-18 09:19:38 +08:00
Eric Yang
345ef049df YARN-9837. Fixed reading YARN Service JSON spec file larger than 128k. 
Contributed by Tarun Parimi

(cherry picked from commit eefe9bc85ccdabc2b7303969934dbce98f2b31b5)
 2019-09-17 13:22:53 -04:00