Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,098 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3890 Commits

Author SHA1 Message Date
Eric E Payne a74411ebaa Revert "YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda)" 
Pulling this back into branch-2 prevented nodemanagers from coming up if extended resources are present.

This reverts commit ab83765a6c.
 2019-11-15 22:01:28 +00:00
Eric E Payne 6973f78a73 YARN-8179: Preemption does not happen due to natural_termination_factor when DRF is used. Contributed by Kyungwan Nam. 
(cherry picked from commit 0b4c44bdee)
 2019-11-15 19:00:06 +00:00
Wangda Tan ab83765a6c YARN-7411. Inter-Queue preemption's computeFixpointAllocation need to handle absolute resources while computing normalizedGuarantee. (Sunil G via wangda) 
Change-Id: I41b1d7558c20fc4eb2050d40134175a2ef6330cb
(cherry picked from commit 034b312d9f)
 2019-11-15 17:21:30 +00:00
Robert Kanter 235d6c73e5 YARN-8202. DefaultAMSProcessor should properly check units of requested custom resource types against minimum/maximum allocation (snemeth via rkanter) 
(cherry picked from commit c8b53c4364)
(cherry picked from commit 0506c762b2)
 2019-11-14 15:47:24 -08:00
Daniel Templeton 7f1d2e196c YARN-7541. Node updates don't update the maximum cluster capability for resources other than CPU and memory 
(cherry picked from commit 8498d287cd)
 2019-11-14 14:56:23 -08:00
Wangda Tan b5128b2ae2 YARN-7739. DefaultAMSProcessor should properly check customized resource types against minimum/maximum allocation. (wangda) 
Change-Id: I10cc9341237d9a2fc0f8c855efb98a36b91389e2
(cherry picked from commit d02e42cee4)
 2019-11-14 14:44:30 -08:00
Sunil G ec5c102ba7 YARN-8004. Add unit tests for inter queue preemption for dominant resource calculator. Contributed by Zian Chen. 
(cherry picked from commit 71220d218d)
 2019-11-05 16:03:52 +00:00
Jonathan Hung ae3f5d66e4 YARN-9945. Fix javadoc in FederationProxyProviderUtil in branch-2 2019-10-31 15:30:23 -07:00
Jonathan Hung 915b795358 Make upstream aware of 2.10.0 release 
(cherry picked from commit 7663db59c097c82eeed2df7a91168a4d7123c96b)
(cherry picked from commit 5d2ffcc7aa)
(cherry picked from commit 4d274f60bc)
 2019-10-30 21:01:40 -07:00
Eric Badger 128631c5ac YARN-9914. Use separate configs for free disk space checking for full and not-full disks. Contributed by Jim Brennan 
(cherry picked from commit eef34f2d87)
 2019-10-25 17:45:40 +00:00
Jonathan Hung 108127238b Preparing for 2.11.0 development 2019-10-15 10:01:45 -07:00
Haibo Chen 600d54fcd6 YARN-8750. Refactor TestQueueMetrics. (Contributed by Szilard Nemeth) 
(cherry picked from commit e60b797c88)
 2019-10-15 15:52:25 +00:00
Jonathan Hung ab7181e9af YARN-9760. Support configuring application priorities on a workflow level. Contributed by Varun Saxena 2019-10-08 11:19:39 -07:00
bibinchundatt c57e6bc385 YARN-9858. Optimize RMContext getExclusiveEnforcedPartitions. Contributed by Jonathan Hung. 2019-10-01 16:13:53 +05:30
Jonathan Hung a0e592db5a Addendum to YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit d86a1acc866cbda845fb3896dc824baf12217383)
(cherry picked from commit f4f210d2e5)
(cherry picked from commit 95ec7050b5)
 2019-09-25 17:50:05 -07:00
Jonathan Hung eedbf9d195 YARN-9730. Support forcing configured partitions to be exclusive based on app node label 
(cherry picked from commit 73a044a63822303f792183244e25432528ecfb1e)
(cherry picked from commit dd094d79023f6598e47146166aa8c213e03d41b7)
(cherry picked from commit 10bdcb6f1da3b86146efa479c0bbc8d1da505789)
 2019-09-24 13:51:11 -07:00
Jonathan Hung 90fbfbbe71 YARN-9762. Add submission context label to audit logs. Contributed by Manoj Kumar 
(cherry picked from commit 3d78b1223d)
(cherry picked from commit a1fa9a8a7f)
(cherry picked from commit 6a1d2d56bd)
 2019-09-23 13:13:24 -07:00
Wangda Tan 727cbcb41d YARN-7410. Cleanup FixedValueResource to avoid dependency to ResourceUtils. (wangda) 
(cherry picked from commit df27824b80)
 2019-09-19 15:27:30 -07:00
Weiwei Yang c351230d0e YARN-7860. Fix UT failure TestRMWebServiceAppsNodelabel#testAppsRunning. Contributed by Sunil G. 
(cherry picked from commit bff858e910)
 2019-09-19 20:25:31 +00:00
Wangda Tan e56f2d4993 YARN-7817. Add Resource reference to RM's NodeInfo object so REST API can get non memory/vcore resource usages. (Sunil G via wangda) 
Change-Id: Ia7ceeabd82046645ddeaf487c763288f36cfbdee
(cherry picked from commit e0cfb0a31a)
 2019-09-19 20:21:42 +00:00
Varun Saxena c9a46308a1 YARN-9825. Changes for initializing placement rules with ResourceScheduler in branch-2. Contributed by Jonathan Hung. 2019-09-13 17:03:47 +05:30
Jonathan Hung c412fab728 YARN-9824. Fall back to configured queue ordering policy class name 
(cherry picked from commit f8f8598ea5)
(cherry picked from commit 1dbf87c9ff)
(cherry picked from commit d75693bd6e)
 2019-09-10 15:36:45 -07:00
Jonathan Hung 1476cf5fbd YARN-9820. RM logs InvalidStateTransitionException when app is submitted. Contributed by Prabhu Joseph 2019-09-09 00:30:20 -07:00
Jonathan Hung d2eabbd7fd YARN-9764. Print application submission context label in application summary. Contributed by Manoj Kumar 2019-09-08 19:20:51 -07:00
Jonathan Hung 6e7dfc9981 YARN-9763. Print application tags in application summary. Contributed by Manoj Kumar 2019-09-06 10:59:14 -07:00
Jonathan Hung fb78b7c45c YARN-9761. Allow overriding application submissions based on server side configs. Contributed by Pralabh Kumar 2019-09-06 10:03:33 -07:00
Jonathan Hung 6e02d8fcfb YARN-9810. Add queue capacity/maxcapacity percentage metrics. Contributed by Shubham Gupta 
(cherry picked from commit 0ccf4b0fe1)
(cherry picked from commit cb806988d72bde1f9837c9e0fb82a3a6c032542c)
(cherry picked from commit 37d1f8c81e)
 2019-09-05 14:09:08 -07:00
Akira Ajisaka 758085a87e YARN-9162. Fix TestRMAdminCLI#testHelp. Contributed by Ayush Saxena. 
(cherry picked from commit 5db7c49062)
(cherry picked from commit a453f38015)
(cherry picked from commit 3c9d2f5317)
 2019-08-30 17:57:39 -07:00
Miklos Szegedi 655154cb45 YARN-7585. NodeManager should go unhealthy when state store throws DBException. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 7f515f57ed)
 2019-08-29 19:52:36 -07:00
Wangda Tan 056b54c62c YARN-6909. Use LightWeightedResource when number of resource types more than two. (Sunil G via wangda) 
Change-Id: I90e021c5dea7abd9ec6bd73b2287c8adebe14595
 2019-08-29 17:02:47 -07:00
Jonathan Hung 87ad52f076 YARN-9409. Port resource type changes from YARN-7237 to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung 28fbc4e344 YARN-9272. Backport YARN-7738 for refreshing max allocation for multiple resource types 2019-08-29 17:02:47 -07:00
Jonathan Hung 8e5ce8055e YARN-9271. Backport YARN-6927 for resource type support in MapReduce 2019-08-29 17:02:47 -07:00
Wangda Tan 6f4ebe01b7 YARN-8183. Fix ConcurrentModificationException inside RMAppAttemptMetrics#convertAtomicLongMaptoLongMap. (Suma Shivaprasad via wangda) 
Change-Id: I347871d672001653a3afe2e99adefd74e0d798cd
(cherry picked from commit bb3c504764f807fccba7f28298a12e2296f284cb)
(cherry picked from commit 3043a93d46)
 2019-08-29 17:02:47 -07:00
Sunil G 1875231be5 YARN-7383. Node resource is not parsed correctly for resource names containing dot. Contributed by Gergely Novák. 2019-08-29 17:02:47 -07:00
Daniel Templeton 062fbd084d YARN-7143. FileNotFound handling in ResourceUtils is inconsistent 
Change-Id: Ib1bb487e14a15edd2b5a42cf5078c5a2b295f069
(cherry picked from commit db82a41d94872cea4d0c1bb1336916cebc2faeec)
 2019-08-29 17:02:47 -07:00
Wangda Tan 7025333d31 YARN-7345. GPU Isolation: Incorrect minor device numbers written to devices.deny file. (Jonathan Hung via wangda) 2019-08-29 17:02:47 -07:00
Jonathan Hung 22ba6939eb YARN-9291. Backport YARN-7637 to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung e102c3d74e YARN-9397. Fix empty NMResourceInfo object test failures in branch-2 2019-08-29 17:02:47 -07:00
Sunil G 86709564cb YARN-7223. Document GPU isolation feature. Contributed by Wangda Tan. 2019-08-29 17:02:47 -07:00
Sunil G c539427696 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2019-08-29 17:02:47 -07:00
Wangda Tan ca7d0fda4f YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2019-08-29 17:02:47 -07:00
Jonathan Hung 7939ae1695 YARN-9289. Backport YARN-7330 for GPU in UI to branch-2 2019-08-29 17:02:47 -07:00
Jian He 4d82318eb1 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2019-08-29 17:02:47 -07:00
Jonathan Hung 16faceb0da YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class 2019-08-29 17:02:47 -07:00
Jonathan Hung 7ec4d7c6ce YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation 2019-08-29 17:02:47 -07:00
Jonathan Hung 631dfc7277 YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung 56259bcecb YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung 078dfb09fe YARN-9175. Null resources check in ResourceInfo for branch-3.0 
(cherry picked from commit a0291a015c)
 2019-08-29 17:02:47 -07:00
Sunil G 4d30c90a1b YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 
(cherry picked from commit da0b6a354b)
(cherry picked from commit 74030d808c)
 2019-08-29 17:02:47 -07:00