Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,464 Commits 326 Branches 371 Tags
Commit Graph

4193 Commits

Author SHA1 Message Date
Sunil G
c539427696 YARN-7594. TestNMWebServices#testGetNMResourceInfo fails on trunk. Contributed by Gergely Novák. 2019-08-29 17:02:47 -07:00
Wangda Tan
ca7d0fda4f YARN-7573. Gpu Information page could be empty for nodes without GPU. (Sunil G via wangda) 
Change-Id: I7f614e5a589a09ce4e4286c84b706e05c29abd14
 2019-08-29 17:02:47 -07:00
Jonathan Hung
7939ae1695 YARN-9289. Backport YARN-7330 for GPU in UI to branch-2 2019-08-29 17:02:47 -07:00
Jian He
4d82318eb1 YARN-7396. NPE when accessing container logs due to null dirsHandler. Contributed by Jonathan Hung 2019-08-29 17:02:47 -07:00
Jonathan Hung
16faceb0da YARN-9174. Backport YARN-7224 for refactoring of GpuDevice class 2019-08-29 17:02:47 -07:00
Jonathan Hung
7ec4d7c6ce YARN-9280. Backport YARN-6620 to YARN-8200/branch-2 for NodeManager-side GPU isolation 2019-08-29 17:02:47 -07:00
Jonathan Hung
631dfc7277 YARN-9180. Port YARN-7033 NM recovery of assigned resources to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung
56259bcecb YARN-9187. Backport YARN-6852 for GPU-specific native changes to branch-2 2019-08-29 17:02:47 -07:00
Jonathan Hung
078dfb09fe YARN-9175. Null resources check in ResourceInfo for branch-3.0 
(cherry picked from commit a0291a015c1af0ea1282849bd8fb32824d7452fa)
 2019-08-29 17:02:47 -07:00
Sunil G
4d30c90a1b YARN-7137. [YARN-3926] Move newly added APIs to unstable in YARN-3926 branch. Contributed by Wangda Tan. 
(cherry picked from commit da0b6a354bf6f6bf37ca5a05a4a8eece09aa4893)
(cherry picked from commit 74030d808cd95e26a0c48500c08d269fcb4150ee)
 2019-08-29 17:02:47 -07:00
Daniel Templeton
72d90e750b YARN-7270 addendum: Reapplied changes after YARN-3926 backports 2019-08-29 17:02:47 -07:00
Jonathan Hung
d6673c5491 YARN-9188. Port YARN-7136 to branch-2 2019-08-29 17:02:47 -07:00
Wangda Tan
26e87be3f2 YARN-7056. [Partial backport] Document Resource Profiles feature. (Sunil G via wangda) 
Change-Id: I5f5d52cdf7aba4ac09684aab573e6916dbad183e
(cherry picked from commit 6b88cd1901a65b977fed759e322dcd75a2cd33b6)
 2019-08-29 17:02:47 -07:00
Sunil G
45b50c45d6 YARN-6933. [YARN-3926] ResourceUtils.DISALLOWED_NAMES check is duplicated. Contributed by Manikandan R. 
(cherry picked from commit 805095496dfd8e50f71b70cf20845e954d3ba47c)
 2019-08-29 17:02:47 -07:00
Wangda Tan
8cea242e94 YARN-7093. Improve log message in ResourceUtils. (Sunil G via wangda) 
Change-Id: I88928a747ee3eec17bc76fb71e9aaa632d091f0f
(cherry picked from commit 53df3eac503064b8c4cd8eb2b659567622f4d8a7)
 2019-08-29 17:02:46 -07:00
Wangda Tan
f340726c9a YARN-7039. Fix javac and javadoc errors in YARN-3926 branch. (Sunil G via wangda) 
Change-Id: I442bf6d838b3aba83f1f6779cf9dcf8596a2102d
(cherry picked from commit e490602e9b306d5b8a543b93fb15a7395bb9a03d)
 2019-08-29 17:02:46 -07:00
Sunil G
74bb2d6ae7 YARN-7067. [YARN-3926] Optimize ResourceType information display in UI. Contributed by Wangda Tan. 
(cherry picked from commit a333ba54e3f5a1ca57b0fc94bfd008f3cbd5b14b)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/MetricsOverviewTable.java
 2019-08-29 17:02:46 -07:00
Wangda Tan
1f05fa041a YARN-7043. [Partial backport] Cleanup ResourceProfileManager. (wangda) 
Change-Id: I463356f37bf1f6a3f1fc3c594c79916e8c0ab913
(cherry picked from commit 39240b61a163b127eec385decd30ffd96e694c28)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ClientRMService.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/OpportunisticContainerAllocatorAMService.java
 2019-08-29 17:02:46 -07:00
Sunil G
7c37fbef12 YARN-6781. [YARN-3926] ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter. Contributed by Yu-Tang Lin. 
(cherry picked from commit 758b7719430c981a6d4d1644d9d03cd16b981cf3)
 2019-08-29 17:02:46 -07:00
Wangda Tan
c69dd9eea6 YARN-6789. Add Client API to get all supported resource types from RM. (Sunil G via wangda) 
Change-Id: I366d8db6f6700acd087db5acb7a1be7e41b2b68d
(cherry picked from commit df3855541af98a3805958f2b5b9db6be705d52ab)
 2019-08-29 17:02:46 -07:00
Wangda Tan
43030aa423 YARN-7042. Clean up unit tests after YARN-6610. (Daniel Templeton via wangda) 
Change-Id: I8e40f704b6fcdd5b14faa9548a27986501044fa1
(cherry picked from commit b1fe3a222e7673fd84a878622969f958022061e9)
 2019-08-29 17:02:46 -07:00
Sunil G
7b1c58b037 YARN-7030. [YARN-3926] Performance optimizations in Resource and ResourceUtils class. Contributed by Wangda Tan. 
(cherry picked from commit a6989af1d91d8b6795862c39d154823f8488c8de)
 2019-08-29 17:02:46 -07:00
Sunil G
2fafdc7f8b YARN-6610. [YARN-3926] DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources. Contributed by Daniel Templeton. 
(cherry picked from commit a0030c8c6f5489d8285632a651394b2b2320255c)
 2019-08-29 17:02:46 -07:00
Sunil G
06c38ee7a4 YARN-6892. [YARN-3926] Improve API implementation in Resources and DominantResourceCalculator class. Contributed by Sunil G. 
(cherry picked from commit 2b51b262aba0191b80dc93799574c0b959cb4f4e)
 2019-08-29 17:02:46 -07:00
Yufei Gu
59cfecd60e YARN-6994. [YARN-3926] Remove last uses of Long from resource types code. (Daniel Templeton via Yufei Gu) 
(cherry picked from commit d5e9939ebb59e4f6f8e5e2fe7f619fb032f2911c)
 2019-08-29 17:02:46 -07:00
Daniel Templeton
d8f388652e YARN-6788. [YARN-3926] Improve performance of resource profile branch 
(Contributed by Sunil Govindan via Daniel Templeton)

(cherry picked from commit 3aeaafecb823ef6c175ea5d0d9cb726faacaf32d)
 2019-08-29 17:02:46 -07:00
Sunil G
1dc66dbb3f YARN-6786. [YARN-3926] ResourcePBImpl imports cleanup. Contributed by Yeliang Cang. 
(cherry picked from commit 4e5632d28eb8f6b1b8830c2aa89ec6321d477977)
 2019-08-29 17:02:46 -07:00
Sunil G
ad6f095774 YARN-6761. Fix build for YARN-3926 branch. Contributed by Varun Vasudev. 
(cherry picked from commit ca2808d8bbe49766029e5dba88ff863f6c993287)
 2019-08-29 17:02:46 -07:00
Jonathan Hung
d5963b2be9 YARN-9182. Backport YARN-6445 resource profile performance improvements to branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung
08923e324a YARN-9177. Use resource map for app metrics in TestCombinedSystemMetricsPublisher for branch-2 2019-08-29 17:02:46 -07:00
Jonathan Hung
7f614f7cee YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2 2019-08-29 17:02:46 -07:00
Sunil G
c5e77e8b84 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 
(cherry picked from commit 7805deed4896e470ebd2f6bbd1ba9962947c63cd)
 2019-08-29 17:02:46 -07:00
Arun Suresh
6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac330147b2d3816a31f2ee83e09c41fe0dd9)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a07fede71dd751ed2c1f40036b22170e)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
015a55a5c6 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 
(cherry picked from commit 9e4ba6aff595b7a5b53d016c97c8334e10f1f2cb)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b0063907d4c07ea6ee261e861bf5cc3a9a)
 2019-08-29 17:02:46 -07:00
Arun Suresh
817837e65c YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 
(cherry picked from commit 7ba698997b6845b664b2f8c82e384b457c30d176)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
133eb31d54 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 
(cherry picked from commit ceb12c59a34677518a198e19acb4e29cdfc8a0e1)
 2019-08-29 17:02:46 -07:00
Wangda Tan
be2afb4a5d YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 
(cherry picked from commit 32c91223f1bd06561ea4ce2d1944e8d9a847f18c)
 2019-08-29 17:02:46 -07:00
Wangda Tan
afcf9c55b5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 
(cherry picked from commit 1bbab7c1570a2438b2fa6da70397dd1d5211a137)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
 2019-08-29 17:02:46 -07:00
Eric E Payne
8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:26:03 +00:00
Jonathan Hung
b6be5f869a YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e3e5daff2db4d30771a194f3bc47d6d)
 2019-08-27 15:49:58 -07:00
Jonathan Hung
f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
(cherry picked from commit 6baa0d1e4d78c15833257d98bdd2b38806faed9f)
 2019-08-26 16:05:25 -07:00
Weiwei Yang
97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f676237290d18ddbbc9aec369267692)

(cherry picked from commit a0b7abf278c147d3ae7821c6783a85cb972c6ec2)
 2019-08-23 09:59:47 -07:00
Eric Badger
afa9a4084d YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

(cherry picked from commit db88224e8f9d164ac811fcca9efe4a350cebecd1)
 2019-08-13 17:27:43 +00:00
Erik Krogen
d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Haibo Chen
4a9fc45f6f YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d5398531835b24d812f6f95094a0e0493e)
(cherry picked from commit 8d357343c4bc9f18e25543583f8f217b8a2f621b)
(cherry picked from commit f943bff2546330bc9eb0914d1a0fa1ef4c709b72)
(cherry picked from commit 6109ecf16d6f0ebf1e314e163b5c41ad75758be1)
 2019-08-06 15:14:45 -07:00
Weiwei Yang
19af228957 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
(cherry picked from commit 70efe253f369f579f05d833d45e6a1bae3e41dfe)
(cherry picked from commit cd8e9fa80b2903935ca9c337631e60161a6053b3)
 2019-08-05 10:04:09 -07:00
Weiwei Yang
4e5ba315c6 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b6701c82799a6231103ca7ee1f8b4efde)
(cherry picked from commit 7dd8eafe337a33689ff6f2f947ad152e06dbc55f)
(cherry picked from commit d095325234fe8fa8d2513c2f03c03347ef5f3606)
 2019-08-05 10:04:00 -07:00
Wangda Tan
75fc058290 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
(cherry picked from commit 46baafedf13446f5c1b973d667f437bf7158d3c2)
(cherry picked from commit bf4da6aacee62be906e71eca6e33821f5ee79a2d)
 2019-08-05 10:03:56 -07:00