Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,312 Commits 326 Branches 371 Tags
Commit Graph

3953 Commits

Author SHA1 Message Date
Eric Badger
cb5b80d6cb YARN-10009. In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined. Contributed by Eric Payne. 2019-12-20 19:40:55 +00:00
Jonathan Hung
041fe5fb57 YARN-9894. CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues. Contributed by Eric Payne 
(cherry picked from commit 7b93575b92e8bad889c1ef15e0baaade6de6de4d)
(cherry picked from commit 0707d0a0ae36456f3467cbb408c3a9a0073c70f7)
(cherry picked from commit 750fb4c3212e7c197f418ea2df711be4467ee27a)
 2019-12-18 13:31:52 -08:00
Jonathan Hung
c4a8c834e5 YARN-10039. Allow disabling app submission from REST endpoints 
(cherry picked from commit 406c35dd1213ef354570f8337e54a4fb4bbf5bad)
 2019-12-18 11:01:29 -08:00
Jonathan Hung
d74f64aa99 Setting version to 2.10.1-SNAPSHOT after branch-2 to branch-2.10 rename 2019-12-09 11:49:59 -08:00
Jonathan Hung
b04f1df4cf YARN-10012. Guaranteed and max capacity queue metrics for custom resources. Contributed by Manikandan R 
(cherry picked from commit 92bce918dc5d03560169642b71636800680a3292)
(cherry picked from commit 9228e3f0ad4e94cedd59c5b02a87ce6c267db5b5)
(cherry picked from commit 9b4368a62ff46db9529e333fdc2bc9db4863195d)
 2019-12-08 16:44:38 -08:00
Sunil G
5401d7f024 YARN-4901. QueueMetrics needs to be cleared before MockRM is initialized. Contributed by Peter Bacsko. 
(cherry picked from commit 002dcc4ebf79bbaa5e603565640d8289991d781f)
(cherry picked from commit 69dc329acc0174b133e8b68f16b3e894329a6fd0)
(cherry picked from commit 67cf1f94cd321761d676c5c9a8a07913fc3c2a70)
 2019-12-08 14:44:57 -08:00
Weiwei Yang
e7ecb4f51f YARN-9205. When using custom resource type, application will fail to run due to the CapacityScheduler throws InvalidResourceRequestException(GREATER_THEN_MAX_ALLOCATION). Contributed by Zhankun Tang. 
(cherry picked from commit bc6374f282dbff3b9ed91fb5d7825d57e6720f5e)
 2019-12-03 13:56:54 -08:00
Eric E Payne
3a7121b0b6 YARN-9915: Fix FindBug issue in QueueMetrics. Contributed by Prabhu Joseph. 
(cherry picked from commit 83d148074f9299de02d5c896a3ed4e11292cba73)
 2019-11-22 19:03:42 +00:00
Eric E Payne
bfb76e60cf YARN-9773: Add QueueMetrics for Custom Resources. Contributed by Manikandan R. 
(cherry picked from commit a5034c7988b6bf54bb7dab208100a2d205e3929e)
 2019-11-22 18:57:05 +00:00
Tao Yang
5b2a1b641a YARN-9838. Fix resource inconsistency for queues when moving app with reserved container to another queue. Contributed by jiulongzhu. 
(cherry picked from commit 8afabe41e4175551ae71217a45f7c4276f10b7da)
 2019-11-22 16:18:59 +08:00
Jason Lowe
dbbe23f269 YARN-7589. TestPBImplRecords fails with NullPointerException. Contributed by Daniel Templeton 
(cherry picked from commit 25df5054216a6a76d09d9c49984f8075ebc6a197)
 2019-11-21 11:55:20 -08:00
Eric E Payne
a9f51e0372 YARN-8842. Expose metrics for custom resource types in QueueMetrics. (Contributed by Szilard Nemeth) (refactored patch for branch-2) 2019-11-21 19:23:48 +00:00
Wangda Tan
835bc686e0 YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f19024d2eabd377210d17d4080ef70e)

Conflicts:
  hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
  I did not backport the changes to ResourcePBImpl.java because the code was removed by mistake and soon added back as part of YARN-7483 (Eric Payne).
 2019-11-19 22:39:14 +00:00
Eric E Payne
a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c587f8378daed741f7d05598d076188.
 2019-11-15 22:01:28 +00:00
Eric E Payne
6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdeef62945b592d5761666ad026b629c0b)
 2019-11-15 19:00:06 +00:00
Wangda Tan
ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f19024d2eabd377210d17d4080ef70e)
 2019-11-15 17:21:30 +00:00
Robert Kanter
235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c43644b4ad22d5385c22cad8ed573c0b1ba)
(cherry picked from commit 0506c762b2f96ccb2c12cce8fd4659536236354e)
 2019-11-14 15:47:24 -08:00
Daniel Templeton
7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd3beddcf8fe19625227e09982ec4be2)
 2019-11-14 14:56:23 -08:00
Wangda Tan
b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4a08a47ed2835f7a4a100daaa95833f)
 2019-11-14 14:44:30 -08:00
Sunil G
ec5c102ba7 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218db59cab0b03bbba427e5e9ef5b3003c)
 2019-11-05 16:03:52 +00:00
Jonathan Hung
ae3f5d66e4 YARN-9945. Fix javadoc in FederationProxyProviderUtil in branch-2 2019-10-31 15:30:23 -07:00
Jonathan Hung
915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aac608746946dc2ed5711549a01a2055)
(cherry picked from commit 4d274f60bc4bf2800477800ec58b2b2c6f4f656e)
 2019-10-30 21:01:40 -07:00
Eric Badger
128631c5ac YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87a75e16b2cca870d99a5e1e28c31d9b)
 2019-10-25 17:45:40 +00:00
Jonathan Hung
108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Haibo Chen
600d54fcd6 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88541f94cecc7fdbcaad010c4742cfdb)
 2019-10-15 15:52:25 +00:00
Jonathan Hung
ab7181e9af YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:19:39 -07:00
bibinchundatt
c57e6bc385 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:13:53 +05:30
Jonathan Hung
a0e592db5a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
(cherry picked from commit f4f210d2e57e49999579efb7db039cc623798267)
(cherry picked from commit 95ec7050b55490454e1de894da572a7e25374907)
 2019-09-25 17:50:05 -07:00
Jonathan Hung
eedbf9d195 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
(cherry picked from commit dd094d79023f6598e47146166aa8c213e03d41b7)
(cherry picked from commit 10bdcb6f1da3b86146efa479c0bbc8d1da505789)
 2019-09-24 13:51:11 -07:00
Jonathan Hung
90fbfbbe71 YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d3fdc29d500803cefd2931b54f44928)
(cherry picked from commit a1fa9a8a7f79a1a711cd881b526724b502e03456)
(cherry picked from commit 6a1d2d56bd6b3cd2f535a732cc07a78ea52062f8)
 2019-09-23 13:13:24 -07:00
Wangda Tan
727cbcb41d YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. (wangda) 
(cherry picked from commit df27824b80b8eca03d02837efe2da31f089e67ec)
 2019-09-19 15:27:30 -07:00
Weiwei Yang
c351230d0e YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 
(cherry picked from commit bff858e910f69991e6c75151d9b89c3e78210f16)
 2019-09-19 20:25:31 +00:00
Wangda Tan
e56f2d4993 YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
(cherry picked from commit e0cfb0a31a1c6ad0f9dcf0705b44229593a0915f)
 2019-09-19 20:21:42 +00:00
Varun Saxena
c9a46308a1 YARN-9825. Changes for initializing placement rules with ResourceScheduler in branch-2. Contributed by Jonathan Hung. 2019-09-13 17:03:47 +05:30
Jonathan Hung
c412fab728 YARN-9824. Fall back to configured queue ordering policy class name 
(cherry picked from commit f8f8598ea5e86330fa5b553963f5632da286409d)
(cherry picked from commit 1dbf87c9ff33b5e2f6cbb0ce8efd348c4314a0e2)
(cherry picked from commit d75693bd6ed4a9547f4932d41014a21de1398115)
 2019-09-10 15:36:45 -07:00
Jonathan Hung
1476cf5fbd YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:30:20 -07:00
Jonathan Hung
d2eabbd7fd YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:20:51 -07:00
Jonathan Hung
6e7dfc9981 YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 10:59:14 -07:00
Jonathan Hung
fb78b7c45c YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:03:33 -07:00
Jonathan Hung
6e02d8fcfb YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 
(cherry picked from commit 0ccf4b0fe16a8c879a560f2a612a3185eb2df72b)
(cherry picked from commit cb806988d72bde1f9837c9e0fb82a3a6c032542c)
(cherry picked from commit 37d1f8c81eeee79b1ef5ce6316afbcdbc76acd8e)
 2019-09-05 14:09:08 -07:00
Akira Ajisaka
758085a87e YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 
(cherry picked from commit 5db7c4906235dd1070b9706ca4971f1eb5d74139)
(cherry picked from commit a453f38015fec454f63e9f5459b0330ac3bc7eb3)
(cherry picked from commit 3c9d2f53175e11ae12fe5857289980ee612124a8)
 2019-08-30 17:57:39 -07:00
Miklos Szegedi
655154cb45 YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 7f515f57ede74dae787994f37bfafd5d20c9aa4c)
 2019-08-29 19:52:36 -07:00
Wangda Tan
056b54c62c YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2019-08-29 17:02:47 -07:00
Jonathan Hung
87ad52f076 YARN-9409. Port resource type changes from YARN-7237 to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung
28fbc4e344 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types 2019-08-29 17:02:47 -07:00
Jonathan Hung
8e5ce8055e YARN-9271. Backport YARN-6927 for resource type support in MapReduce 2019-08-29 17:02:47 -07:00
Wangda Tan
6f4ebe01b7 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda) 
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
(cherry picked from commit 3043a93d461fd8b9ccc2ff4b8d17e5430ed77615)
 2019-08-29 17:02:47 -07:00
Sunil G
1875231be5 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 2019-08-29 17:02:47 -07:00
Daniel Templeton
062fbd084d YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
(cherry picked from commit db82a41d94872cea4d0c1bb1336916cebc2faeec)
 2019-08-29 17:02:47 -07:00
Wangda Tan
7025333d31 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2019-08-29 17:02:47 -07:00