Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,138 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4100 Commits

Author SHA1 Message Date
Eric E Payne c296e3c0aa YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 
(cherry picked from commit 93233a7d6e)
 2020-01-09 17:52:26 +00:00
Eric Badger ab29b54b19 YARN-8672. TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out. Contributed by Chandni Singh and Jim Brennan. 2020-01-08 20:13:13 +00:00
Eric E Payne 5cca5ca81b YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc)
 2020-01-08 19:59:13 +00:00
Eric E Payne 2ae1b3568b YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a17)
 2020-01-08 18:04:12 +00:00
Eric Badger cb5b80d6cb YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 19:40:55 +00:00
Jonathan Hung 041fe5fb57 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92)
(cherry picked from commit 0707d0a0ae)
(cherry picked from commit 750fb4c321)
 2019-12-18 13:31:52 -08:00
Jonathan Hung c4a8c834e5 YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit 406c35dd12)
 2019-12-18 11:01:29 -08:00
Jonathan Hung d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Jonathan Hung b04f1df4cf YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc)
(cherry picked from commit 9228e3f0ad)
(cherry picked from commit 9b4368a62f)
 2019-12-08 16:44:38 -08:00
Sunil G 5401d7f024 YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf)
(cherry picked from commit 69dc329acc)
(cherry picked from commit 67cf1f94cd)
 2019-12-08 14:44:57 -08:00
Weiwei Yang e7ecb4f51f YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 
(cherry picked from commit bc6374f282)
 2019-12-03 13:56:54 -08:00
Eric E Payne 3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f)
 2019-11-22 19:03:42 +00:00
Eric E Payne bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988)
 2019-11-22 18:57:05 +00:00
Tao Yang 5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4)
 2019-11-22 16:18:59 +08:00
Jason Lowe dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df505421)
 2019-11-21 11:55:20 -08:00
Eric E Payne a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan 835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00
Eric E Payne a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c.
 2019-11-15 22:01:28 +00:00
Eric E Payne 6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2019-11-15 19:00:06 +00:00
Wangda Tan ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)
 2019-11-15 17:21:30 +00:00
Robert Kanter 235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c4364)
(cherry picked from commit 0506c762b2)
 2019-11-14 15:47:24 -08:00
Daniel Templeton 7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd)
 2019-11-14 14:56:23 -08:00
Wangda Tan b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4)
 2019-11-14 14:44:30 -08:00
Sunil G ec5c102ba7 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218d)
 2019-11-05 16:03:52 +00:00
Jonathan Hung ae3f5d66e4 YARN-9945. Fix javadoc in FederationProxyProviderUtil in branch-2 2019-10-31 15:30:23 -07:00
Jonathan Hung 915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
(cherry picked from commit 4d274f60bc)
 2019-10-30 21:01:40 -07:00
Eric Badger 128631c5ac YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:45:40 +00:00
Jonathan Hung 108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Haibo Chen 600d54fcd6 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88)
 2019-10-15 15:52:25 +00:00
Jonathan Hung ab7181e9af YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:19:39 -07:00
bibinchundatt c57e6bc385 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:13:53 +05:30
Jonathan Hung a0e592db5a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
(cherry picked from commit f4f210d2e5)
(cherry picked from commit 95ec7050b5)
 2019-09-25 17:50:05 -07:00
Jonathan Hung eedbf9d195 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
(cherry picked from commit dd094d79023f6598e47146166aa8c213e03d41b7)
(cherry picked from commit 10bdcb6f1da3b86146efa479c0bbc8d1da505789)
 2019-09-24 13:51:11 -07:00
Jonathan Hung 90fbfbbe71 YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d)
(cherry picked from commit a1fa9a8a7f)
(cherry picked from commit 6a1d2d56bd)
 2019-09-23 13:13:24 -07:00
Wangda Tan 727cbcb41d YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. (wangda) 
(cherry picked from commit df27824b80)
 2019-09-19 15:27:30 -07:00
Weiwei Yang c351230d0e YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 
(cherry picked from commit bff858e910)
 2019-09-19 20:25:31 +00:00
Wangda Tan e56f2d4993 YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
(cherry picked from commit e0cfb0a31a)
 2019-09-19 20:21:42 +00:00
Varun Saxena c9a46308a1 YARN-9825. Changes for initializing placement rules with ResourceScheduler in branch-2. Contributed by Jonathan Hung. 2019-09-13 17:03:47 +05:30
Jonathan Hung c412fab728 YARN-9824. Fall back to configured queue ordering policy class name 
(cherry picked from commit f8f8598ea5)
(cherry picked from commit 1dbf87c9ff)
(cherry picked from commit d75693bd6e)
 2019-09-10 15:36:45 -07:00
Jonathan Hung 1476cf5fbd YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:30:20 -07:00
Jonathan Hung d2eabbd7fd YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:20:51 -07:00
Jonathan Hung 6e7dfc9981 YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 10:59:14 -07:00
Jonathan Hung fb78b7c45c YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:03:33 -07:00
Jonathan Hung 6e02d8fcfb YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 
(cherry picked from commit 0ccf4b0fe1)
(cherry picked from commit cb806988d72bde1f9837c9e0fb82a3a6c032542c)
(cherry picked from commit 37d1f8c81e)
 2019-09-05 14:09:08 -07:00
Akira Ajisaka 758085a87e YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 
(cherry picked from commit 5db7c49062)
(cherry picked from commit a453f38015)
(cherry picked from commit 3c9d2f5317)
 2019-08-30 17:57:39 -07:00
Miklos Szegedi 655154cb45 YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 7f515f57ed)
 2019-08-29 19:52:36 -07:00
Wangda Tan 056b54c62c YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2019-08-29 17:02:47 -07:00
Jonathan Hung 87ad52f076 YARN-9409. Port resource type changes from YARN-7237 to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung 28fbc4e344 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types 2019-08-29 17:02:47 -07:00
Jonathan Hung 8e5ce8055e YARN-9271. Backport YARN-6927 for resource type support in MapReduce 2019-08-29 17:02:47 -07:00