Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,230 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

4122 Commits

Author SHA1 Message Date
Ahmed Hussein 712da57658 YARN-10255. fix intermittent failure TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic in branch-2.10 (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
 2020-04-30 17:50:55 -05:00
Jonathan Hung 27ad054696 YARN-8193. YARN RM hangs abruptly (stops allocating resources) when running successive applications. (Zian Chen via wangda) 2020-04-30 12:16:15 -07:00
Miklos Szegedi 71063434a6 YARN-8382. cgroup file leak in NM. Contributed by Hu Ziqian. 
(cherry picked from commit 925fdf761a513130e23c10575c7328c8681cff1d)
(cherry picked from commit 21948b048e)
 2020-04-27 14:14:42 -07:00
Wangda Tan 34804679e3 YARN-8369. Javadoc build failed due to 'bad use of >'. (Takanobu Asanuma via wangda) 
Change-Id: I79a42154e8f86ab1c3cc939b3745024b8eebe5f4
(cherry picked from commit 17aa40f669)
 2020-04-19 12:51:56 +09:00
Jonathan Hung ebce5c74e6 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 
(cherry picked from commit cd6c10de442fc3a53c9ed5521ac1d944a6ac95c6)
(cherry picked from commit 2c79865b951d0fdea7f576ce31e310b4074ecedd)
 2020-04-17 10:35:39 -07:00
Eric Yang 7945ed40f0 HADOOP-16361. Fixed TestSecureLogins#testValidKerberosName on branch-2. 
Contributed by Jim Brennan via eyang
 2020-04-16 12:12:21 -04:00
Eric Badger 0e05b02136 YARN-8680. YARN NM: Implement Iterable Abstraction for LocalResourceTracker 
state. Contributed by Pradeep Ambati.
 2020-04-14 18:30:56 +00:00
Jonathan Hung c0394c5434 YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
(cherry picked from commit e3a52804b03d646f15048c078f8c5292d5cbecfa)
(cherry picked from commit 54599b177c)
(cherry picked from commit 6271a2852e)
 2020-04-09 11:07:59 -07:00
Eric E Payne 0ce55c699c YARN-8242. YARN NM: OOM error while reading back the state store on recovery. Contributed by Pradeep Ambati and Kanwaljeet Sachdev 
YARN-10227. Pull YARN-8242 back to branch-2.10. Contributed by Jim Brennan
 2020-04-09 17:18:07 +00:00
Eric Badger a3343c4097 YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed 
Hussein.
 2020-04-09 00:52:45 +00:00
Jonathan Hung a7556f1ec2 YARN-8213. Add Capacity Scheduler performance metrics. (Weiwei Yang via wangda) 2020-03-27 16:10:39 -07:00
Jonathan Hung 1c8529f030 YARN-10200. Add number of containers to RMAppManager summary 
(cherry picked from commit 2de0572cdc1c6fdbfaab108b169b2d5b0c077e86)
(cherry picked from commit 5d3fb0ebe9)
(cherry picked from commit 9c6dd8c83a)
 2020-03-25 10:39:45 -07:00
Eric Badger d01e9c125b Revert "YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed" 
This reverts commit 85eac8b0df.
 2020-03-17 22:15:38 +00:00
Eric Badger 85eac8b0df YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed 
Hussein.
 2020-03-16 22:31:43 +00:00
Eric E Payne e9a7b30fa0 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 
(cherry picked from commit ede05b19d1)
 2020-03-10 15:48:49 +00:00
Inigo Goiri 2d44d7f139 YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 
(cherry picked from commit a43510e21d)
 2020-02-27 13:22:16 -08:00
Elixir Kook e844f6ffaa
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 
(cherry picked from commit d608e94f92)
 2020-02-26 17:33:05 +09:00
Eric E Payne 254b94bf5e YARN-10140: TestTimelineAuthFilterForV2 fails due to login failures in branch-2.10. Contributed by Ahmed Hussein (ahussein) 2020-02-25 16:36:50 +00:00
Jonathan Hung c34c87b1a8 YARN-8292. Fix the dominant resource preemption cannot happen when some of the resource vector becomes negative. Contributed by Wangda Tan and Eric Payne 2020-02-07 17:29:32 -08:00
Jonathan Hung 4fce8c8023 YARN-10116. Expose diagnostics in RMAppManager summary 
(cherry picked from commit 314e2f9d2e)
(cherry picked from commit 147897da4b420b4749f3c7b410f4c329632c3352)
(cherry picked from commit fa35b8370ce14c9b8ee911b73fda380817b964fd)
 2020-02-05 11:16:09 -08:00
Eric Badger 21970f6f67 YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-30 21:29:33 +00:00
Abhishek Modi 296786a647 YARN-9790. Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero. Contributed by kyungwan nam. 
(cherry picked from commit d2d963f3d4)
 2020-01-23 17:12:25 +00:00
Eric E Payne c296e3c0aa YARN-9018. Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path. Contributed by Kuhu Shukla (kshukla) 
(cherry picked from commit 93233a7d6e)
 2020-01-09 17:52:26 +00:00
Eric Badger ab29b54b19 YARN-8672. TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out. Contributed by Chandni Singh and Jim Brennan. 2020-01-08 20:13:13 +00:00
Eric E Payne 5cca5ca81b YARN-7387: org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit b1e07d27cc)
 2020-01-08 19:59:13 +00:00
Eric E Payne 2ae1b3568b YARN-10072: TestCSAllocateCustomResource failures. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 6899be5a17)
 2020-01-08 18:04:12 +00:00
Eric Badger cb5b80d6cb YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 19:40:55 +00:00
Jonathan Hung 041fe5fb57 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92)
(cherry picked from commit 0707d0a0ae)
(cherry picked from commit 750fb4c321)
 2019-12-18 13:31:52 -08:00
Jonathan Hung c4a8c834e5 YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit 406c35dd12)
 2019-12-18 11:01:29 -08:00
Jonathan Hung d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Jonathan Hung b04f1df4cf YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc)
(cherry picked from commit 9228e3f0ad)
(cherry picked from commit 9b4368a62f)
 2019-12-08 16:44:38 -08:00
Sunil G 5401d7f024 YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf)
(cherry picked from commit 69dc329acc)
(cherry picked from commit 67cf1f94cd)
 2019-12-08 14:44:57 -08:00
Weiwei Yang e7ecb4f51f YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 
(cherry picked from commit bc6374f282)
 2019-12-03 13:56:54 -08:00
Eric E Payne 3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f)
 2019-11-22 19:03:42 +00:00
Eric E Payne bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988)
 2019-11-22 18:57:05 +00:00
Tao Yang 5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4)
 2019-11-22 16:18:59 +08:00
Jason Lowe dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df505421)
 2019-11-21 11:55:20 -08:00
Eric E Payne a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan 835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00
Eric E Payne a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c.
 2019-11-15 22:01:28 +00:00
Eric E Payne 6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2019-11-15 19:00:06 +00:00
Wangda Tan ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)
 2019-11-15 17:21:30 +00:00
Robert Kanter 235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c4364)
(cherry picked from commit 0506c762b2)
 2019-11-14 15:47:24 -08:00
Daniel Templeton 7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd)
 2019-11-14 14:56:23 -08:00
Wangda Tan b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4)
 2019-11-14 14:44:30 -08:00
Sunil G ec5c102ba7 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218d)
 2019-11-05 16:03:52 +00:00
Jonathan Hung ae3f5d66e4 YARN-9945. Fix javadoc in FederationProxyProviderUtil in branch-2 2019-10-31 15:30:23 -07:00
Jonathan Hung 915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
(cherry picked from commit 4d274f60bc)
 2019-10-30 21:01:40 -07:00
Eric Badger 128631c5ac YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:45:40 +00:00
Jonathan Hung 108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00