Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,040 Commits 326 Branches 371 Tags
Commit Graph

4063 Commits

Author SHA1 Message Date
Jonathan Hung
7f614f7cee YARN-9181. Backport YARN-6232 for generic resource type usage to branch-2 2019-08-29 17:02:46 -07:00
Sunil G
c5e77e8b84 YARN-5588. [Partial backport] [YARN-3926] Add support for resource profiles in distributed shell. Contributed by Varun Vasudev. 
(cherry picked from commit 7805deed4896e470ebd2f6bbd1ba9962947c63cd)
 2019-08-29 17:02:46 -07:00
Arun Suresh
6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac330147b2d3816a31f2ee83e09c41fe0dd9)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a07fede71dd751ed2c1f40036b22170e)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S
015a55a5c6 YARN-5242. Update DominantResourceCalculator to consider all resource types in calculations. Contributed by Varun Vasudev. 
(cherry picked from commit 9e4ba6aff595b7a5b53d016c97c8334e10f1f2cb)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b0063907d4c07ea6ee261e861bf5cc3a9a)
 2019-08-29 17:02:46 -07:00
Arun Suresh
817837e65c YARN-4829. Add support for binary units in Resource class.(vvasudev via asuresh) 
(cherry picked from commit 7ba698997b6845b664b2f8c82e384b457c30d176)
 2019-08-29 17:02:46 -07:00
Varun Vasudev
133eb31d54 YARN-4715. Add support to read resource types from a config file. Contributed by Varun Vasudev. 
(cherry picked from commit ceb12c59a34677518a198e19acb4e29cdfc8a0e1)
 2019-08-29 17:02:46 -07:00
Wangda Tan
be2afb4a5d YARN-4172. Extend DominantResourceCalculator to account for all resources. (Varun Vasudev via wangda) 
(cherry picked from commit 32c91223f1bd06561ea4ce2d1944e8d9a847f18c)
 2019-08-29 17:02:46 -07:00
Wangda Tan
afcf9c55b5 YARN-4081. Add support for multiple resource types in the Resource class. (Varun Vasudev via wangda) 
(cherry picked from commit 1bbab7c1570a2438b2fa6da70397dd1d5211a137)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ResourcePBImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java
 2019-08-29 17:02:46 -07:00
Eric E Payne
8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec83a2bc2ffb6d109ed3d64b394b870d)
 2019-08-28 21:26:03 +00:00
Jonathan Hung
b6be5f869a YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e3e5daff2db4d30771a194f3bc47d6d)
 2019-08-27 15:49:58 -07:00
Jonathan Hung
f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca15098e891c560beb3181c22ef3f80ff)
(cherry picked from commit e4249c320257586384035ea3fc286fe54cc699a1)
(cherry picked from commit 6baa0d1e4d78c15833257d98bdd2b38806faed9f)
 2019-08-26 16:05:25 -07:00
Weiwei Yang
97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f676237290d18ddbbc9aec369267692)

(cherry picked from commit a0b7abf278c147d3ae7821c6783a85cb972c6ec2)
 2019-08-23 09:59:47 -07:00
Eric Badger
afa9a4084d YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949f041da2ee04da441c5f9f85e1f2c64)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be76577718b22f936aea9e2b2cd100ea)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

(cherry picked from commit db88224e8f9d164ac811fcca9efe4a350cebecd1)
 2019-08-13 17:27:43 +00:00
Erik Krogen
d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Haibo Chen
4a9fc45f6f YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d5398531835b24d812f6f95094a0e0493e)
(cherry picked from commit 8d357343c4bc9f18e25543583f8f217b8a2f621b)
(cherry picked from commit f943bff2546330bc9eb0914d1a0fa1ef4c709b72)
(cherry picked from commit 6109ecf16d6f0ebf1e314e163b5c41ad75758be1)
 2019-08-06 15:14:45 -07:00
Weiwei Yang
19af228957 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23c5d1bd52d04c303506806d0dc61cd2)
(cherry picked from commit 70efe253f369f579f05d833d45e6a1bae3e41dfe)
(cherry picked from commit cd8e9fa80b2903935ca9c337631e60161a6053b3)
 2019-08-05 10:04:09 -07:00
Weiwei Yang
4e5ba315c6 YARN-8915. Update the doc about the default value of maximum-container-assignments for capacity scheduler. Contributed by Zhankun Tang. 
(cherry picked from commit 78ea897b6701c82799a6231103ca7ee1f8b4efde)
(cherry picked from commit 7dd8eafe337a33689ff6f2f947ad152e06dbc55f)
(cherry picked from commit d095325234fe8fa8d2513c2f03c03347ef5f3606)
 2019-08-05 10:04:00 -07:00
Wangda Tan
75fc058290 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07df2a3ed6273b96ae857c278fd72718)
(cherry picked from commit 46baafedf13446f5c1b973d667f437bf7158d3c2)
(cherry picked from commit bf4da6aacee62be906e71eca6e33821f5ee79a2d)
 2019-08-05 10:03:56 -07:00
Eric Badger
9aae72af84 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3ba302327baf2e98d1e07b953dcbbfd)
 2019-08-02 21:06:54 +00:00
Eric E Payne
77d1aa9376 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 
(cherry picked from commit c04ccf296dab6eaa53f008342e50f8729935a50d)
 2019-07-30 20:31:02 +00:00
Ahmed Hussein
dc4d7f69ce YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 9d63fc7ed61ca6c3d901fe9984a7cca98ce07ea9)
 2019-07-24 11:21:29 -05:00
Haibo Chen
68c24b322d YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 2019-07-19 11:00:38 -07:00
Eric E Payne
441339a716 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 2019-06-04 18:57:44 +00:00
Takanobu Asanuma
e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka
cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Haibo Chen
dc28c86270 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 
(cherry picked from commit 597fa47ad125c0871f5c4deb3a883e5b3341c67b)
(cherry picked from commit c6573562cbc43832e9332989996e5d07b8bce9b0)
(cherry picked from commit ea1f0f282bee7ca64ce9fbe1a59bb5e231420d29)
(cherry picked from commit aa7ff8552e0f1b9439926ae3476a3227dcf339f3)
 2019-05-06 12:01:26 -07:00
Eric E Payne
5bf40f592e YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 2019-05-03 16:17:43 +00:00
Vrushali C
396fcee0b0 YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks 2019-03-07 23:04:36 -08:00
Vrushali C
b83ec96767 YARN-8549 Adding a NoOp timeline writer and reader plugin classes for ATSv2. Contributed by Prabha Manepalli 2019-03-07 22:54:32 -08:00
Eric E Payne
6e6ebc368c YARN-5714. ContainerExecutor does not order environment map. Contributed by Remi Catherinot and Jim Brennan. 2019-03-07 16:22:36 +00:00
Adam Antal
3c7c3366ff
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9385ec45d75109a2e6565faa10527cc56637bf5f)
 2019-02-15 19:00:48 +09:00
Masatake Iwasaki
83a9369f7d YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 
(cherry picked from commit e0ab1bdecec3d7ba7ddc0849781d7f71714f8687)
 2019-02-09 00:34:57 +09:00
Eric E Payne
2a7dcc509b YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 
(cherry picked from commit d1ca9432dd0b3e9b46b4903e8c9d33f5c28fcc1b)
 2019-02-07 21:29:23 +00:00
Sunil G
a91933620d YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 
(cherry picked from commit 6ffe6ea8999c1216b6e510e350535754431bac2d)
 2019-02-07 19:11:12 +05:30
Eric E Payne
c6c5e94de4 YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 
(cherry picked from commit 04105bbfdb041a41062c856632641140de84fba8)
 2019-01-29 18:22:12 +00:00
Jonathan Hung
338dbbedf7 YARN-9222. Print launchTime in ApplicationSummary 
(cherry picked from commit 6cace58e212d3ee0aec988926a5a17c9cc58e645)
(cherry picked from commit bf760e7e81f8e02ad413c470fccf78aaa9cb9f86)
(cherry picked from commit 80a08c29adc9d07974bbe8087abcec750d5f7916)
(cherry picked from commit c71054c428b3abe00507792099e13a2dad337697)
 2019-01-25 13:40:51 -08:00
Haibo Chen
c02c19121e YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 
(cherry picked from commit bb92bfb4ef96baa234966b60e464d1773fbf3f22)
(cherry picked from commit 3a764250262e745a6762dd6551b0d0a6af39e891)
(cherry picked from commit 75df8bed6eed86ad7c34460a1167c35d48b559d7)
 2019-01-23 12:18:42 -08:00
Weiwei Yang
c77f7e25c9 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 2019-01-23 11:02:15 +08:00
Akira Ajisaka
300d65c4d7
YARN-8747. [UI2] YARN UI2 page loading failed due to js error under some time zone configuration. Contributed by collinma. 
(cherry picked from commit 104ef5df36fd45f1c2d07a2f3d441263aa85e22e)
 2019-01-16 14:39:17 +09:00
Weiwei Yang
c54a689d96 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 2019-01-09 14:33:06 +08:00
Botong Huang
15286d9166 YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi. 2018-12-22 12:46:13 -08:00
Jonathan Hung
eb8b1eaa9a YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 
(cherry picked from commit cc51607ccdf60f4ebecf7afa3e2056297ca0ac5f)
 2018-12-07 10:47:45 -08:00
Jonathan Hung
7456da99c1 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:18:17 -08:00
Botong Huang
b7ca20e16e YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang. 2018-11-11 19:41:08 -08:00
Akira Ajisaka
ee1cb3b6e8
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-10 21:44:46 +09:00
bibinchundatt
731b90d44f YARN-8905. [Router] Add JvmMetricsInfo and pause monitor. Contributed by Bilwa S T. 
(cherry picked from commit f84a278baad6cb871ea4196257f23a938826ca23)
 2018-11-03 20:50:28 +05:30
Giovanni Matteo Fumarola
81da8b262b YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang. 2018-11-02 16:02:56 -07:00
Jonathan Hung
62ec800bd4 YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 2018-11-01 14:13:15 -07:00