Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
15,484 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3095 Commits

Author SHA1 Message Date
Arun Suresh 6db8c36ba9 YARN-5587. [Partial backport] Add support for resource profiles. (vvasudev via asuresh) 
(cherry picked from commit 6708ac3301)
 2019-08-29 17:02:46 -07:00
Rohith Sharma K S f99d603aaf YARN-5586. Update the Resources class to consider all resource types. Contributed by Varun Vasudev. 
(cherry picked from commit 239c1824a0)
 2019-08-29 17:02:46 -07:00
Varun Vasudev 7d5050a768 YARN-4830. Add support for resource types in the nodemanager. Contributed by Varun Vasudev. 
(cherry picked from commit 759114b006)
 2019-08-29 17:02:46 -07:00
Eric E Payne 8e8c16dbc6 YARN-9756: Create metric that sums total memory/vcores preempted per round. Contributed by Manikandan R (manirajv06). 
(cherry picked from commit d562050cec)
 2019-08-28 21:26:03 +00:00
Jonathan Hung b6be5f869a YARN-9438. launchTime not written to state store for running applications 
(cherry picked from commit 9568656cd21d9c02168e18ce35c6726077bbf3a1)
(cherry picked from commit 0c498de6e87c6bdc959afa31deb03d0907e0e1a1)
(cherry picked from commit f73842780e)
 2019-08-27 15:49:58 -07:00
Jonathan Hung f083f7fef1 YARN-9775. RMWebServices /scheduler-conf GET returns all hadoop configurations for ZKConfigurationStore. Contributed by Prabhu Joseph 
(cherry picked from commit 8660e48ca1)
(cherry picked from commit e4249c3202)
(cherry picked from commit 6baa0d1e4d)
 2019-08-26 16:05:25 -07:00
Weiwei Yang 97f207d1a9 YARN-8025. UsersManangers#getComputedResourceLimitForActiveUsers throws NPE due to preComputedActiveUserLimit is empty. Contributed by Tao Yang. 
(Cherry picked from commit 67f239c42f)

(cherry picked from commit a0b7abf278)
 2019-08-23 09:59:47 -07:00
Eric Badger afa9a4084d YARN-9442. container working directory has group read permissions. Contributed by Jim Brennan. 
(cherry picked from commit 2ac029b949)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c

(cherry picked from commit cec71691be)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c

(cherry picked from commit db88224e8f)
 2019-08-13 17:27:43 +00:00
Erik Krogen d5da529acf HDFS-13286. [SBN read] Add haadmin commands to transition between standby and observer. Contributed by Chao Sun. 2019-08-09 15:49:55 -07:00
Haibo Chen 4a9fc45f6f YARN-9559. Create AbstractContainersLauncher for pluggable ContainersLauncher logic. (Contributed by Jonathan Hung) 
(cherry picked from commit f51702d539)
(cherry picked from commit 8d357343c4)
(cherry picked from commit f943bff254)
(cherry picked from commit 6109ecf16d)
 2019-08-06 15:14:45 -07:00
Weiwei Yang 19af228957 YARN-8944. TestContainerAllocation.testUserLimitAllocationMultipleContainers failure after YARN-8896. Contributed by Wilfred Spiegelenburg. 
(cherry picked from commit 1d90a0dd23)
(cherry picked from commit 70efe253f3)
(cherry picked from commit cd8e9fa80b2903935ca9c337631e60161a6053b3)
 2019-08-05 10:04:09 -07:00
Wangda Tan 75fc058290 YARN-8896. Limit the maximum number of container assignments per heartbeat. (Zhankun Tang via wangda) 
Change-Id: I6e72f8362bd7f5c2a844cb9e3c4732492314e9f1
(cherry picked from commit 780be14f07)
(cherry picked from commit 46baafedf1)
(cherry picked from commit bf4da6aacee62be906e71eca6e33821f5ee79a2d)
 2019-08-05 10:03:56 -07:00
Eric Badger 9aae72af84 YARN-8045. Reduce log output from container status calls. Contributed by Craig Condit 
(cherry picked from commit 144a55f0e3)
 2019-08-02 21:06:54 +00:00
Eric E Payne 77d1aa9376 YARN-9596: QueueMetrics has incorrect metrics when labelled partitions are involved. Contributed by Muhammad Samir Khan. 
(cherry picked from commit c04ccf296d)
 2019-07-30 20:31:02 +00:00
Ahmed Hussein dc4d7f69ce YARN-9563. Resource report REST API could return NaN or Inf (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 9d63fc7ed6)
 2019-07-24 11:21:29 -05:00
Haibo Chen 68c24b322d YARN-9668. UGI conf doesn't read user overridden configurations on RM and NM startup. (Contributed by Jonanthan Hung) 2019-07-19 11:00:38 -07:00
Eric E Payne 441339a716 YARN-8625. Aggregate Resource Allocation for each job is not present in ATS. Contributed by Prabhu Joseph. 2019-06-04 18:57:44 +00:00
Takanobu Asanuma e3d4f6bfed HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2019-05-29 17:41:21 +09:00
Akira Ajisaka cea1b0000c
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:28:54 +09:00
Haibo Chen dc28c86270 YARN-9529. Log correct cpu controller path on error while initializing CGroups. (Contributed by Jonathan Hung) 
(cherry picked from commit 597fa47ad1)
(cherry picked from commit c6573562cb)
(cherry picked from commit ea1f0f282b)
(cherry picked from commit aa7ff8552e)
 2019-05-06 12:01:26 -07:00
Eric E Payne 5bf40f592e YARN-9285: RM UI progress column is of wrong type. Contributed by Ahmed Hussein. 2019-05-03 16:17:43 +00:00
Vrushali C 396fcee0b0 YARN-9150 Making TimelineSchemaCreator support different backends for Timeline Schema Creation in ATSv2. Contributed by Sushil Ks 2019-03-07 23:04:36 -08:00
Vrushali C b83ec96767 YARN-8549 Adding a NoOp timeline writer and reader plugin classes for ATSv2. Contributed by Prabha Manepalli 2019-03-07 22:54:32 -08:00
Eric E Payne 6e6ebc368c YARN-5714. ContainerExecutor does not order environment map. Contributed by Remi Catherinot and Jim Brennan. 2019-03-07 16:22:36 +00:00
Adam Antal 3c7c3366ff
YARN-9283. Javadoc of LinuxContainerExecutor#addSchedPriorityCommand has a wrong property name as reference 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9385ec45d7)
 2019-02-15 19:00:48 +09:00
Masatake Iwasaki 83a9369f7d YARN-9282. Typo in javadoc of class LinuxContainerExecutor: hadoop.security.authetication should be 'authentication'. Contributed by Charan Hebri. 
(cherry picked from commit e0ab1bdece)
 2019-02-09 00:34:57 +09:00
Eric E Payne 2a7dcc509b YARN-7171: RM UI should sort memory / cores numerically. Contributed by Ahmed Hussein 
(cherry picked from commit d1ca9432dd)
 2019-02-07 21:29:23 +00:00
Sunil G a91933620d YARN-9206. RMServerUtils does not count SHUTDOWN as an accepted state. Contributed by Kuhu Shukla. 
(cherry picked from commit 6ffe6ea899)
 2019-02-07 19:11:12 +05:30
Eric E Payne c6c5e94de4 YARN-6616: YARN AHS shows submitTime for jobs same as startTime. Contributed by Prabhu Joseph 
(cherry picked from commit 04105bbfdb)
 2019-01-29 18:22:12 +00:00
Jonathan Hung 338dbbedf7 YARN-9222. Print launchTime in ApplicationSummary 
(cherry picked from commit 6cace58e21)
(cherry picked from commit bf760e7e81)
(cherry picked from commit 80a08c29adc9d07974bbe8087abcec750d5f7916)
(cherry picked from commit c71054c428b3abe00507792099e13a2dad337697)
 2019-01-25 13:40:51 -08:00
Haibo Chen c02c19121e YARN-7088. Add application launch time to Resource Manager REST API. (Kanwaljeet Sachdev via Haibo Chen) 
(cherry picked from commit bb92bfb4ef)
(cherry picked from commit 3a764250262e745a6762dd6551b0d0a6af39e891)
(cherry picked from commit 75df8bed6eed86ad7c34460a1167c35d48b559d7)
 2019-01-23 12:18:42 -08:00
Weiwei Yang c77f7e25c9 YARN-9210. RM nodes web page can not display node info. Contributed by Jiandan Yang. 2019-01-23 11:02:15 +08:00
Weiwei Yang c54a689d96 YARN-8833. Avoid potential integer overflow when computing fair shares. Contributed by liyakun. 2019-01-09 14:33:06 +08:00
Botong Huang 15286d9166 YARN 9108. Fix FederationIntercepter merge home and secondary allocate response typo. Contributed by Abhishek Modi. 2018-12-22 12:46:13 -08:00
Jonathan Hung eb8b1eaa9a YARN-9085. Add Guaranteed and MaxCapacity to CSQueueMetrics 
(cherry picked from commit cc51607ccd)
 2018-12-07 10:47:45 -08:00
Jonathan Hung 7456da99c1 YARN-9036. Escape newlines in health report in YARN UI. Contributed by Keqiu Hu 2018-11-30 10:18:17 -08:00
Botong Huang b7ca20e16e YARN-8933. [AMRMProxy] Fix potential empty fields in allocation response, move SubClusterTimeout to FederationInterceptor. Contributed by Botong Huang. 2018-11-11 19:41:08 -08:00
Akira Ajisaka ee1cb3b6e8
YARN-8233. NPE in CapacityScheduler#tryCommit when handling allocate/reserve proposal whose allocatedOrReservedContainer is null. Contributed by Tao Yang. 2018-11-10 21:44:46 +09:00
bibinchundatt 731b90d44f YARN-8905. [Router] Add JvmMetricsInfo and pause monitor. Contributed by Bilwa S T. 
(cherry picked from commit f84a278baa)
 2018-11-03 20:50:28 +05:30
Giovanni Matteo Fumarola 81da8b262b YARN-8893. [AMRMProxy] Fix thread leak in AMRMClientRelayer and UAM client. Contributed by Botong Huang. 2018-11-02 16:02:56 -07:00
Jonathan Hung 62ec800bd4 YARN-7225. Add queue and partition info to RM audit log. Contributed by Eric Payne 2018-11-01 14:13:15 -07:00
Rohith Sharma K S dbf36dbb8d YARN-8826. Fix lingering timeline collector after serviceStop in TimelineCollectorManager. Contributed by Prabha Manepalli. 
(cherry picked from commit 0b62983c5a)
 2018-10-23 14:10:24 +05:30
Vrushali C 7ed627af6b YARN-3879 [Storage implementation] Create HDFS backing storage implementation for ATS reads. Contributed by Abhishek Modi. 2018-10-11 21:23:34 -07:00
Wangda Tan 91f18a5148 YARN-7274. Ability to disable elasticity at leaf queue level. (Zian Chen via wangda) 
Change-Id: Ic8d43e297f0f5de788b562f7eff8106c5c35e8d2
(cherry picked from commit 74665e3a7d)
 2018-10-11 16:58:07 +00:00
Vrushali C 7a5d27dde4 YARN-6989 Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way. Contributed by Abhishek Modi 2018-10-10 15:25:58 -07:00
Weiwei Yang cc1bf7f894 YARN-8858. CapacityScheduler should respect maximum node resource when per-queue maximum-allocation is being used. Contributed by Wangda Tan. 
(cherry picked from commit edce866489)
 2018-10-10 10:17:18 +08:00
Weiwei Yang d412785175 YARN-8720. CapacityScheduler does not enforce max resource allocation check at queue level. Contributed by Tarun Parimi. 
(cherry picked from commit f1a893fdbc)
 2018-10-10 10:16:16 +08:00
Inigo Goiri b0900ad310 YARN-7652. Handle AM register requests asynchronously in FederationInterceptor. Contributed by Botong Huang. 
(cherry picked from commit c3d22d3b45)
 2018-10-09 10:30:42 -07:00
Giovanni Matteo Fumarola 57caab6f9b YARN-8760. [AMRMProxy] Fix concurrent re-register due to YarnRM failover in AMRMClientRelayer. Contributed by Botong Huang. 2018-10-01 14:27:01 -07:00
Eric E Payne d7d0e55e0a YARN-8774. Memory leak when CapacityScheduler allocates from reserved container with non-default label. Contributed by Tao Yang. 2018-09-28 16:10:31 +00:00
Jason Lowe 1b0a110501 YARN-8804. resourceLimits may be wrongly calculated when leaf-queue is blocked in cluster with 3+ level queues. Contributed by Tao Yang 
(cherry picked from commit 6b988d821e)
 2018-09-26 17:06:07 -07:00
Arun Suresh 4d69741a61 YARN-7974. Allow updating application tracking url after registration. (Jonathan Hung via asuresh) 2018-09-26 00:08:10 -07:00
Giovanni Matteo Fumarola 6937925838 YARN-8696. [AMRMProxy] FederationInterceptor upgrade: home sub-cluster heartbeat async. Contributed by Botong Huang. 2018-09-24 11:40:07 -07:00
Giovanni Matteo Fumarola 60565976e1 YARN-8658. [AMRMProxy] Metrics for AMRMClientRelayer inside FederationInterceptor. Contributed by Young Chen. 2018-09-21 10:36:36 -07:00
Eric E Payne 121cefd472 YARN-8709: CS preemption monitor always fails since one under-served queue was deleted. Contributed by Tao Yang. 
(cherry picked from commit 987d8191ad)
 2018-09-10 20:32:20 +00:00
Eric E Payne 21aa7f1d82 YARN-8051: TestRMEmbeddedElector#testCallbackSynchronization is flakey. Contributed by Robert Kanter and Jason Lowe. 2018-08-29 21:30:38 +00:00
Giovanni Matteo Fumarola bb0e75f72f YARN-8697. LocalityMulticastAMRMProxyPolicy should fallback to random sub-cluster when cannot resolve resource. Contributed by Botong Huang. 2018-08-28 16:07:31 -07:00
Giovanni Matteo Fumarola d4a3be9591 HADOOP-15699. Fix some of testContainerManager failures in Windows. Contributed by Botong Huang. 2018-08-27 12:28:16 -07:00
Giovanni Matteo Fumarola 548a595027 YARN-8705. Refactor the UAM heartbeat thread in preparation for YARN-8696. Contributed by Botong Huang. 2018-08-27 11:26:31 -07:00
Jason Lowe c4e3df2261 YARN-8649. NPE in localizer hearbeat processing if a container is killed while localizing. Contributed by lujie 
(cherry picked from commit 585ebd873a)
 2018-08-23 09:43:03 -05:00
Giovanni Matteo Fumarola 8e6807ef4a YARN-8673. [AMRMProxy] More robust responseId resync after an YarnRM master slave switch. Contributed by Botong Huang. 2018-08-21 13:09:33 -07:00
Giovanni Matteo Fumarola 89da0e9901 YARN-8581. [AMRMProxy] Add sub-cluster timeout in LocalityMulticastAMRMProxyPolicy. Contributed by Botong Huang. 2018-08-21 13:04:49 -07:00
Rohith Sharma K S 1a53aab4d6 YARN-8129. Improve error message for invalid value in fields attribute. Contributed by Abhishek Modi. 
(cherry picked from commit d3fef7a5c5)
 2018-08-21 12:11:31 +05:30
Rohith Sharma K S c68d1d49ca YARN-8679. [ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked. Contributed by Wangda Tan. 
(cherry picked from commit 4aacbfff60)
 2018-08-18 11:04:20 +05:30
Jason Lowe a44e53a314 YARN-8640. Restore previous state in container-executor after failure. Contributed by Jim Brennan 
(cherry picked from commit d1d129aa9d)
 2018-08-14 10:33:27 -05:00
Jonathan Hung 7abffe4529 YARN-8559. Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint. Contributed by Weiwei Yang. 2018-08-10 15:34:21 -07:00
Jason Lowe 2024260af6 YARN-8331. Race condition in NM container launched after done. Contributed by Pradeep Ambati 
(cherry picked from commit cd04e954d2)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/container/TestContainer.java
 2018-08-09 10:35:07 -05:00
Haibo Chen 1991a1d760 YARN-6966. NodeManager metrics may return wrong negative values when NM restart. (Szilard Nemeth via Haibo Chen) 2018-08-02 10:06:16 -07:00
Arun Suresh e2b82b82e2 YARN-7542. Fix issue that causes some Running Opportunistic Containers to be recovered as PAUSED. (Sampada Dehankar via asuresh) 
(cherry picked from commit a55884c68e)
(cherry picked from commit bd4dcc7772)
 2018-08-02 09:59:04 -07:00
Rohith Sharma K S 21e416ad27 YARN-8155. Improve ATSv2 client logging in RM and NM publisher. Contributed by Abhishek Modi. 2018-08-01 22:25:53 +05:30
Eric E Payne 3aa17bd737 YARN-4606. CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps. Contributed by Manikandan R 
(cherry picked from commit 9485c9aee6)
 2018-07-25 17:06:47 +00:00
Robert Kanter edb9d8b554 YARN-8518. test-container-executor test_is_empty() is broken (Jim_Brennan via rkanter) 
(cherry picked from commit 1bc106a738)
(cherry picked from commit 6e0db6fe1a)
 2018-07-22 05:28:21 +00:00
Robert Kanter 92f02f97fd Only mount non-empty directories for cgroups (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 0838fe8337)
(cherry picked from commit c1dc4ca2c6)
 2018-07-22 05:28:20 +00:00
Robert Kanter f5fd5aa025 Disable mounting cgroups by default (miklos.szegedi@cloudera.com via rkanter) 
(cherry picked from commit 351cf87c92)
(cherry picked from commit d61d84279f)
 2018-07-22 05:28:20 +00:00
Eric E Payne 8ee439d791 YARN-8421: when moving app, activeUsers is increased, even though app does not have outstanding request. Contributed by Kyungwan Nam 
(cherry picked from commit 937ef39b3f)
 2018-07-16 17:01:38 +00:00
Jason Lowe 0e6efe06ea YARN-8515. container-executor can crash with SIGPIPE after nodemanager restart. Contributed by Jim Brennan 
(cherry picked from commit 17118f446c)
 2018-07-13 10:11:57 -05:00
Sunil G 6cc5d49fa3 YARN-8473. Containers being launched as app tears down can leave containers in NEW state. Contributed by Jason Lowe. 
(cherry picked from commit 705e2c1f7c)
 2018-07-10 20:13:56 +05:30
Giovanni Matteo Fumarola aab9bfc13c YARN-7899. [AMRMProxy] Stateful FederationInterceptor for pending requests. Contributed by Botong Huang. 2018-07-09 16:47:44 -07:00
Giovanni Matteo Fumarola 64baa9ec89 YARN-8481. AMRMProxyPolicies should accept heartbeat response from new/unknown subclusters. Contributed by Botong Huang. 2018-06-29 11:51:50 -07:00
Jason Lowe 14c7dc3c1e YARN-8451. Multiple NM heartbeat thread created when a slow NM resync with RM. Contributed by Botong Huang 
(cherry picked from commit 100470140d)
 2018-06-29 13:17:14 -05:00
Inigo Goiri 82874e7895 YARN-8412. Move ResourceRequest.clone logic everywhere into a proper API. Contributed by Botong Huang. 2018-06-21 18:25:30 -07:00
Sunil G 96a6798c1d YARN-8404. Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down. Contributed by Rohith Sharma K S 
(cherry picked from commit 6307962b93)
 2018-06-13 16:10:57 +05:30
Inigo Goiri 85f3105e61 HADOOP-15529. ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit 6e756e8a62)
 2018-06-12 10:26:01 -07:00
Rohith Sharma K S 0af3bea05d YARN-8405. RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773. Contributed by Íñigo Goiri. 
(cherry picked from commit 2df73dace0)
 2018-06-12 17:30:02 +05:30
Inigo Goiri 8be1640bf6 YARN-8370. Some Node Manager tests fail on Windows due to improper path/file separator. Contributed by Anbang Hu. 
(cherry picked from commit 2b2f672022)
 2018-06-11 19:27:34 -07:00
Inigo Goiri b991b38f51 YARN-8359. Exclude containermanager.linux test classes on Windows. Contributed by Jason Lowe. 
(cherry picked from commit 3492a1db2c0654ce5375360caa74a34f928f23be)
 2018-06-07 17:11:12 -07:00
Robert Kanter f97bd6bb7f YARN-4677. RMNodeResourceUpdateEvent update from scheduler can lead to race condition (wilfreds and gphillips via rkanter) 2018-06-04 15:59:27 -07:00
Sunil G d47a525163 YARN-4781. Support intra-queue preemption for fairness ordering policy. Contributed by Eric Payne. 2018-06-02 08:30:39 +05:30
Wangda Tan 911852e932 YARN-8068. Application Priority field causes NPE in app timeline publish when Hadoop 2.7 based clients to 2.8+ (Sunil G via wangda) 
Change-Id: I7910bd1064a1b4dbbe2084080c060822ea6f3b48
(cherry picked from commit 9eef19b2ad)
 2018-05-24 13:05:23 -05:00
Inigo Goiri 8f43ade46a YARN-8344. Missing nm.stop() in TestNodeManagerResync to fix testKillContainersOnResync. Contributed by Giovanni Matteo Fumarola. 
(cherry picked from commit e99e5bf104)
 2018-05-23 14:17:20 -07:00
Wangda Tan 777743beb6 YARN-8232. RMContainer lost queue name when RM HA happens. (Hu Ziqian via wangda) 
Change-Id: Ia21e1da6871570c993bbedde76ce32929e95970f
(cherry picked from commit 6b96a73bb0)
 2018-05-22 10:34:43 -05:00
Arun Suresh 113e2d6801 YARN-7900. [AMRMProxy] AMRMClientRelayer for stateful FederationInterceptor. (Botong Huang via asuresh) 2018-05-21 11:26:32 -07:00
Sunil G 19cf706711 YARN-8249. Few REST api's in RMWebServices are missing static user check. Contributed by Sunil G. 2018-05-16 12:18:25 +05:30
Haibo Chen e28af8b0eb YARN-8130 Race condition when container events are published for KILLED applications. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 2d00a0c71b)
 2018-05-15 11:58:56 +05:30
Vrushali C c2b05339cf YARN-8247 Incorrect HTTP status code returned by ATSv2 for non-whitelisted users. Contributed by Rohith Sharma K S 
(cherry picked from commit 3c95ca4f21)
 2018-05-14 11:30:00 +05:30
Jason Lowe 5e3369ccd2 YARN-8244. TestContainerSchedulerQueuing.testStartMultipleContainers failed. Contributed by Jim Brennan 
(cherry picked from commit dc912994a1)
 2018-05-11 14:20:53 -05:00
Rohith Sharma K S 3cb9888d56 YARN-8253. HTTPS Ats v2 api call fails with 'bad HTTP parsed'. Contributed by Charan Hebri. 
(cherry picked from commit 7450583721)
 2018-05-08 12:33:23 +05:30
Robert Kanter 312462532e HADOOP-15390. Yarn RM logs flooded by DelegationTokenRenewer trying to renew KMS tokens (xiaochen via rkanter) 
(cherry picked from commit 7ab08a9c37)
(cherry picked from commit 74e5b4b437)
 2018-05-07 12:39:57 -07:00
Rohith Sharma K S 250ea47985 YARN-8221. RMWebServices also need to honor yarn.resourcemanager.display.per-user-apps. Contributed by Sunil G. 
(cherry picked from commit ef3ecc308d)
 2018-04-27 23:00:19 +05:30
Sunil G a91d5c7e2c YARN-8215. ATS v2 returns invalid YARN_CONTAINER_ALLOCATED_HOST_HTTP_ADDRESS from NM. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7fc09c4f72)
 2018-04-27 11:19:12 +05:30
Sunil G ab2b4290a0 YARN-8205. Application State is not updated to ATS if AM launching is delayed. Contributed by Rohith Sharma K S. 
(cherry picked from commit 1634de0fc1)
 2018-04-27 11:08:27 +05:30
Jason Lowe 06f3f453c2 YARN-7786. NullPointerException while launching ApplicationMaster. Contributed by lujie 2018-04-20 13:31:19 -05:00
Sunil G 733248cec7 YARN-6827. [ATS1/1.5] NPE exception while publishing recovering applications into ATS during RM restart. Contributed by Rohith Sharma K S. 
(cherry picked from commit 7d06806dfd)
 2018-04-20 12:12:26 +05:30
Jason Lowe 5ec195edbc YARN-7189. Container-executor doesn't remove Docker containers that error out early. Contributed by Eric Badger 
(cherry picked from commit 391ac5cdd2)
 2018-04-17 09:53:19 -05:00
Inigo Goiri 93fc813329 YARN-8164. Fix a potential NPE in AbstractSchedulerPlanFollower. Contributed by lujie. 
(cherry picked from commit f1461b2487)
 2018-04-16 17:32:50 -07:00
Inigo Goiri f37f680a4e YARN-8165. Incorrect queue name logging in AbstractContainerAllocator. Contributed by Weiwei Yan. 
(cherry picked from commit dd5e18c4ae)
 2018-04-16 17:07:38 -07:00
Eric Yang 0731756293 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)
 2018-04-16 13:39:21 -04:00
Rohith Sharma K S 900e6b435a YARN-8156. Increase the default value of yarn.timeline-service.app-collector.linger-period.ms. Contributed by Charan Hebri. 
(cherry picked from commit 669eb7bdea)
 2018-04-14 11:15:30 +05:30
Wei-Chiu Chuang cc2a2a8e06 Revert "YARN-7810. Update TestDockerContainerRuntime to test with current user credential." 
This reverts commit 724bffdb89.
 2018-04-13 10:13:59 -07:00
Eric E Payne 7393020cb3 YARN-8147. TestClientRMService#testGetApplications sporadically fails. Contributed by Jason Lowe 
(cherry picked from commit 18844599ae)
 2018-04-12 18:19:12 +00:00
Eric E Payne b0dfb18daa YARN-8120. JVM can crash with SIGSEGV when exiting due to custom leveldb logger. Contributed by Jason Lowe. 
(cherry picked from commit 6bb128dfb8)
 2018-04-12 16:38:50 +00:00
Weiwei Yang a48deb1552 YARN-7527. Over-allocate node resource in async-scheduling mode of CapacityScheduler. Contributed by Tao Yang. 2018-04-12 10:12:46 +08:00
Eric Yang 724bffdb89 YARN-7810. Update TestDockerContainerRuntime to test with current user credential. 
(contributed by Shane Kumpf)

            Cherry-picked from: 59828be197
 2018-04-11 20:06:33 -04:00
Weiwei Yang 2b2e2ac5f4 YARN-6629. NPE occurred when container allocation proposal is applied but its resource requests are removed before. Contributed by Tao Yang. 2018-04-11 20:21:05 +08:00
Vrushali C fe4c7ee94c YARN-8073 TimelineClientImpl doesn't honor yarn.timeline-service.versions configuration. Contributed by Rohith Sharma K S 
(cherry picked from commit 345e7624d5)
 2018-04-11 09:51:42 +05:30
Subru Krishnan 05246b7585 YARN-8110. AMRMProxy recover should catch for all throwable to avoid premature exit. (Botong Huang via Subru). 
(cherry picked from commit 00905efab2)
 2018-04-06 16:38:28 -07:00
Haibo Chen b19267259b YARN-8107. Give an informative message when incorrect format is used in ATSv2 filter attributes. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit 024d7c0870)
(cherry picked from commit b232dcab33)
(cherry picked from commit ca105d1c8e)
 2018-04-06 09:47:25 -07:00
Haibo Chen a7d8b4d775 YARN-6936. [Atsv2] Retrospect storing entities into sub application table from client perspective. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit f8b8bd53c4)
(cherry picked from commit 6658018410)
(cherry picked from commit 9cb378f5c7)
 2018-04-05 11:45:01 -07:00
Subru Krishnan 304ce18714 YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 
(cherry picked from commit 09999d7e01)
 2018-03-28 11:51:59 -07:00
Subru Krishnan b36b438043 Revert "YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru)." 
This reverts commit 0ab1d2f25b.
 2018-03-28 11:51:59 -07:00
Subru Krishnan 0ab1d2f25b YARN-8010. Add config in FederationRMFailoverProxy to not bypass facade cache when failing over. (Botong Huang via Subru). 
(cherry picked from commit 2a2ef15caf)
 2018-03-27 17:47:37 -07:00
Rohith Sharma K S 5fba8c67de YARN-7581. HBase filters are not constructed correctly in ATSv2. Contributed by Habio Chen. 2018-03-23 12:05:11 +05:30
Jason Lowe 13f74a3601 YARN-8054. Improve robustness of the LocalDirsHandlerService MonitoringTimerTask thread. Contributed by Jonathan Eagles 
(cherry picked from commit 5aa7052e31)
 2018-03-21 17:40:05 -05:00
Jason Lowe a253925862 Revert "YARN-7581. HBase filters are not constructed correctly in ATSv2. Contributed by Habio Chen." 
This reverts commit a01091e90c.
 2018-03-21 17:36:24 -05:00
Rohith Sharma K S a01091e90c YARN-7581. HBase filters are not constructed correctly in ATSv2. Contributed by Habio Chen. 
(cherry picked from commit 29acea5000)
 2018-03-21 08:30:03 +05:30
Yufei Gu 204674f414 YARN-8039. Clean up log dir configuration in TestLinuxContainerExecutorWithMocks.testStartLocalizer. Contributed by Miklos Szegedi. 
(cherry picked from commit 49c747ab18)
 2018-03-16 16:34:14 -07:00
Rohith Sharma K S 207daabbcc YARN-8003. Backport the code structure changes in YARN-7346 to branch-2. Contributed by Haibo Chen. 2018-03-07 13:25:28 +05:30
Jason Lowe 0a3a878428 YARN-7511. NPE in ContainerLocalizer when localization failed for running container. Contributed by Tao Yang 
(cherry picked from commit 83798f15f8)
 2018-03-02 10:08:32 -06:00
Haibo Chen 45d012f208 YARN-7835. Race condition in NM while publishing events if second attempt is launched on the same node. (Rohith Sharma K S via Haibo Chen) 
(cherry picked from commit d1274c3b71)
 2018-02-28 21:24:17 -08:00
Jason Lowe 7a6aa7bf99 YARN-7945. Java Doc error in UnmanagedAMPoolManager for branch-2. Contributed by Botong Huang 2018-02-22 10:45:16 -06:00
Sunil G f7e5e45b72 YARN-7947. Capacity Scheduler intra-queue preemption can NPE for non-schedulable apps. Contributed by Eric Payne. 
(cherry picked from commit bdd2a184d7)
 2018-02-21 15:26:20 +05:30
Jason Lowe 51ad923f51 YARN-7813. Capacity Scheduler Intra-queue Preemption should be configurable for each queue. Contributed by Eric Payne 
(cherry picked from commit 85c611ad7d)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/CapacitySchedulerPage.java
 2018-02-19 15:44:56 -06:00
Rohith Sharma K S da0660ec84 YARN-7919. Refactor timelineservice-hbase module into submodules. Contributed by Haibo Chen. 2018-02-18 13:26:10 +05:30
Billie Rinaldi aa64a50746 Revert "YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi." 
This reverts commit a72dcb9cad.
 2018-02-07 08:58:28 -08:00
Billie Rinaldi 93128c1904 YARN-7890. NPE during container relaunch. Contributed by Jason Lowe 
(cherry picked from commit a196ee9362)
 2018-02-06 12:04:52 -08:00
Jason Lowe 260f30f063 YARN-7849. TestMiniYarnClusterNodeUtilization#testUpdateNodeUtilization fails due to heartbeat sync error. Contributed by Botong Huang 
(cherry picked from commit 60656bceca)
 2018-02-05 17:22:42 -06:00
Haibo Chen c25c082961 YARN-6868. Add test scope to certain entries in hadoop-yarn-server-resourcemanager pom.xml. (Ray Chiang via Haibo Chen) 
(cherry picked from commit a20e7105ea)
 2018-02-05 10:51:03 -08:00
Jinjiang Ling 987a8972ac
YARN-7698. A misleading variable's name in ApplicationAttemptEventDispatcher 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3400d0c535)
 2018-01-30 00:13:04 +09:00
Jian He 0497e09339 YARN-7765. Fixed an issue that kerberos tgt not found when NM posting timeline events. Contributed by Rohith Sharma K S 2018-01-28 13:24:49 -08:00
Jason Lowe 4d7bbee0a1 YARN-7102. NM heartbeat stuck when responseId overflows MAX_INT. Contributed by Botong Huang 2018-01-25 17:56:36 -06:00
Sunil G bed683d300 YARN-7728. Expose container preemptions related information in Capacity Scheduler queue metrics. Contributed by Eric Payne. 
(cherry picked from commit 82cc6f6968)
 2018-01-25 14:34:31 +05:30
Zhe Zhang e9a338ff64 YARN-7737. prelaunch.err file not found exception on container failure. Contributed by Keqiu Hu. 
(cherry picked from commit fa8cf4d1b4)
(cherry picked from commit 83a4f92a3f)
 2018-01-24 13:10:35 -08:00
Jian He c611707e28 YARN-7801. AmFilterInitializer should addFilter after fill all parameters. Contributed by Wangda Tan 2018-01-24 11:19:46 -08:00
Rohith Sharma K S f036609cea Revert "YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma" 
This reverts commit 41e10161b6.
 2018-01-24 15:05:53 +05:30
Vrushali C 41e10161b6 YARN-7537 [Atsv2] load hbase configuration from filesystem rather than URL. Contributed by Rohith Sharma 2018-01-19 15:59:47 -08:00
Haibo Chen 895a0391fe YARN-5094. some YARN container events have timestamp of -1. 
(cherry picked from commit 4aca4ff759)
(cherry picked from commit 8793e45f53)
 2018-01-19 10:35:40 -08:00
Miklos Szegedi 86a2ac94b1 YARN-7758. Add an additional check to the validity of container and application ids passed to container-executor. Contributed by Yufei Gu. 2018-01-17 12:56:10 -08:00
Rohith Sharma K S 8985fccbbc YARN-6736. Consider writing to both ats v1 & v2 from RM for smoother upgrades. Contributed by Aaron Gresch. 
(cherry picked from commit d09058b2fd)
 2018-01-16 09:01:02 +05:30
Miklos Szegedi 09428b137f YARN-7705. Create the container log directory with correct sticky bit in C code. Contributed by Yufei Gu. 
(cherry picked from commit 2dcfc18)
 2018-01-12 17:44:31 -08:00
Sunil G e7a8d6bd1e YARN-7727. Incorrect log levels in few logs with QueuePriorityContainerCandidateSelector. Contributed by Prabhu Joseph. 
(cherry picked from commit 128d773a23)
 2018-01-12 15:56:08 +05:30
Miklos Szegedi 35527db157 YARN-7590. Improve container-executor validation check. Contributed by Eric Yang. 
(cherry picked from commit bc285da)
 2018-01-11 10:30:15 -08:00
Robert Kanter ff67c68d3c YARN-7622. Allow fair-scheduler configuration on HDFS (gphillips via rkanter) 2018-01-10 15:03:33 -08:00
bibinchundatt 73478e3bbe YARN-7508. NPE in FiCaSchedulerApp when debug log enabled in async-scheduling mode. Contributed by Tao Yang. 
(cherry picked from commit b26e30ab16)
 2018-01-10 10:31:46 +05:30
Jason Lowe c892789896 YARN-6948. Invalid event: ATTEMPT_ADDED at FINAL_SAVING. Contributed by lujie 
(cherry picked from commit 90c86d6e09)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
 2018-01-09 10:59:33 -06:00
Jason Lowe f15c7e86e8 YARN-7663. RMAppImpl:Invalid event: START at KILLED. Contributed by lujie 
(cherry picked from commit d2c2d8c913)
 2018-01-09 10:01:06 -06:00
Ray Chiang 18c3982851 YARN-4227. Ignore expired containers from removed nodes in FairScheduler. (Wilfred Spiegelenburg via rchiang) 
(cherry picked from commit bc2d67d6c10619716ef7acce263f3269a86c3150)
 2018-01-08 16:23:29 -08:00
Rohith Sharma K S 37a8224101 YARN-7699. queueUsagePercentage is coming as INF for getApp REST api call. Contributed by Sunil G. 
(cherry picked from commit 281350674c)
 2018-01-08 15:52:41 +05:30
Sunil G cac8a8616e YARN-7619. Max AM Resource value in Capacity Scheduler UI has to be refreshed for every user. Contributed by Eric Payne. 
(cherry picked from commit a79abbc03e)
 2018-01-05 14:51:29 +05:30
Jason Lowe 42964c2a66 YARN-7678. Ability to enable logging of container memory stats. Contributed by Jim Brennan 2018-01-04 10:21:40 -06:00
Rohith Sharma K S e46142a8c2 YARN-7692. Skip validating priority acls while recovering applications. Contributed by Sunil G. 
(cherry picked from commit c9bf813c9a)
 2018-01-03 18:29:38 +05:30
Sunil G 320f564bae YARN-7032. [ATSv2] NPE while starting hbase co-processor when HBase authorization is enabled. Contributed by Rohith Sharma K S. 
(cherry picked from commit d62932c3b2)
 2017-12-20 11:39:38 +05:30
Varun Saxena d72d2440f4 YARN-7662. [ATSv2] Define new set of configurations for reader and collectors to bind (Rohith Sharma K S via Varun Saxena) 
(cherry picked from commit c0aeb666a4)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2017-12-19 22:51:56 +05:30
Jason Lowe 31a06baca0 YARN-7661. NodeManager metrics return wrong value after update node resource. Contributed by Yang Wang 
(cherry picked from commit 811fabdebe)
 2017-12-18 15:27:27 -06:00
Jason Lowe 0f8de8cd91 YARN-7595. Container launching code suppresses close exceptions after writes. Contributed by Jim Brennan 2017-12-15 15:54:56 -06:00
Subru Krishnan 8d3d7fa1c5 YARN-7630. Fix AMRMToken rollover handling in AMRMProxy. Contributed by Botong Huang. 
(cherry picked from commit 17ba74be29)
 2017-12-14 14:07:44 -08:00
Weiwei Yang d69b7358b6 YARN-7647. NM print inappropriate error log when node-labels is enabled. Contributed by Yang Wang. 
(cherry picked from commit 7efc4f7688)
 2017-12-13 13:28:48 +08:00
Weiwei Yang 7b3c64aece YARN-7608. Incorrect sTarget column causing DataTable warning on RM application and scheduler web page. Contributed by Gergely Novák. 2017-12-11 13:06:13 +08:00
Wangda Tan c27a5ba65c YARN-7591. NPE in async-scheduling mode of CapacityScheduler. (Tao Yang via wangda) 
Change-Id: I67e086d6d35c3166ec0f6305490ea6527bd6d799
(cherry picked from commit 41796ea60a)
 2017-12-08 21:50:41 -08:00
Subru Krishnan 850bd0ed7c YARN-6704. Add support for work preserving NM restart when FederationInterceptor is enabled in AMRMProxyService. (Botong Huang via Subru). 
(cherry picked from commit 670e8d4ec7)
 2017-12-08 15:44:45 -08:00
Weiwei Yang 029714e3ec YARN-7607. Remove the trailing duplicated timestamp in container diagnostics message. Contributed by Weiwei Yang. 2017-12-08 13:53:33 +08:00
Robert Kanter 54b5da83ec YARN-5594. Handle old RMDelegationToken format when recovering RM (rkanter) 
(cherry picked from commit d8863fc16f)
 2017-12-04 13:18:04 -08:00
Jason Lowe 3fb83cbeb2 YARN-7455. quote_and_append_arg can overflow buffer. Contributed by Jim Brennan 
(cherry picked from commit 60f95fb719)
 2017-12-01 15:51:49 -06:00
Robert Kanter edf9fd51fb YARN-4813. TestRMWebServicesDelegationTokenAuthentication.testDoAs fails intermittently (grepas via rkanter) 
(cherry picked from commit c83fe44917)
 2017-12-01 12:21:18 -08:00
Yufei Gu de144175a4 YARN-7363. ContainerLocalizer don't have a valid log4j config in case of Linux container executor. (Contributed by Yufei Gu) 
(cherry picked from commit d8923cdbf1)
(cherry picked from commit c0ab3c8407)

Conflicts:
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
 2017-11-27 14:54:27 -08:00
Wangda Tan 94f7b0f99f YARN-7509. AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby. (Tao Yang via wangda) 
Change-Id: I7477fe355419fd4a0a6e2bdda7319abad4c4c748
(cherry picked from commit 834e91ee91)
 2017-11-23 20:06:16 -08:00
Subru Krishnan ed31091361 YARN-6128. Add support for AMRMProxy HA. (Botong Huang via Subru). 2017-11-20 14:21:58 -08:00
bibinchundatt 5c2986e58d YARN-7489. ConcurrentModificationException in RMAppImpl#getRMAppMetrics. Contributed by Tao Yang. 2017-11-20 21:31:50 +05:30
Sunil G a081d91480 YARN-7469. Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit. Contributed by Eric Payne. 
(cherry picked from commit 61ace174cd)
 2017-11-16 22:42:52 +05:30
Junping Du a72dcb9cad YARN-6078. Containers stuck in Localizing state. Contributed by Billie Rinaldi. 2017-11-13 18:22:30 -08:00
Arun Suresh c153bed6e3 YARN-5049. Addendum patch for branch-2. (asuresh) 2017-11-12 10:16:00 -08:00
Eric Yang 90ac8cd5a1 YARN-7412. Fix unit test for docker mount check on ubuntu. (Contributed by Eric Badger) 
(cherry picked from commit 7a49ddfdde)
 2017-11-12 09:59:39 -08:00
Subru Krishnan 0282efab14 YARN-7476. Fix miscellaneous issues in ATSv2 after merge to branch-2. (Varun Saxena via Subru). 2017-11-12 09:19:30 -08:00
Arun Suresh 5cd92ed555 YARN-6627. Addendum patch for branch-2. (Sunil G via asuresh) 2017-11-11 09:10:17 -08:00
Arun Suresh 0411c710d2 YARN-6102. Addendum patch for branch-2. (Rohith Sharma K S via asuresh) 2017-11-10 13:44:03 -08:00
Haibo Chen 3fbe67d3e1 YARN-7388. TestAMRestart should be scheduler agnostic. 
(cherry picked from commit a1382a18df)
(cherry picked from commit 91a7f8d246)
 2017-11-09 10:57:50 -08:00
Arun Suresh 5486542074 YARN-7453. Fix issue where RM fails to switch to active after first successful start. (Rohith Sharma K S via asuresh) 
(cherry picked from commit a9c70b0e84)
 2017-11-08 08:02:23 -08:00
bibinchundatt fd4868bdb4 Add containerId to Localizer failed logs. Contributed by Prabhu Joseph 
(cherry picked from commit dcd99c4b9a)
 2017-11-06 23:09:41 +05:30
Subru Krishnan 656ab06873 YARN-6932. Fix TestFederationRMFailoverProxyProvider test case failure. 
(cherry picked from commit 0542e6f86e)
 2017-11-03 11:11:25 -07:00
Inigo Goiri 2d8d420f26 YARN-7434. Router getApps REST invocation fails with multiple RMs. Contributed by Inigo Goiri. 
(cherry picked from commit 6e2259264ad9525eeec2a14055d53236711659d7)
 2017-11-02 21:34:34 -07:00
Eric Payne 1516a9ef61 YARN-7370: Preemption properties should be refreshable. Contrubted by Gergely Novák. 
(cherry picked from commit e6ec02001f)
 2017-11-02 12:56:41 -05:00
Jason Lowe 6ad6882343 YARN-7286. Add support for docker to have no capabilities. Contributed by Eric Badger 
(cherry picked from commit b7dee1f060)
 2017-11-02 09:50:55 -05:00
Rohith Sharma K S e53fcd52ee addendum patch for YARN-7289. 
(cherry picked from commit 940ffe3f9c)
 2017-11-02 14:07:28 +05:30
Haibo Chen 32a712a740 YARN-7389. Make TestResourceManager Scheduler agnostic. (Robert Kanter via Haibo Chen) 
(cherry picked from commit d7f3737f3b)
 2017-11-01 14:19:59 -07:00
Daniel Templeton 956d814034 YARN-7146. Many RM unit tests failing with FairScheduler 
(Contributed by Robert Kanter via Daniel Templeton)
 2017-11-01 14:11:26 -07:00
Inigo Goiri 8180ab436a YARN-7276. Federation Router Web Service fixes. Contributed by Inigo Goiri. 2017-11-01 13:21:15 -07:00
Arun Suresh 71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Yufei Gu 243b70eca8 YARN-6747. TestFSAppStarvation.testPreemptionEnable fails intermittently. (Contributed by Miklos Szegedi) 
(cherry picked from commit d4811c8cfa)
 2017-10-29 16:47:22 -07:00
Varun Saxena e1b3bbde60 YARN-7380. Fix findbugs warning in timeline service on branch-2 (Vrushali C via Varun Saxena). 2017-10-29 16:38:20 +05:30
Arun Suresh 2654d9973e YARN-7299. Fix TestDistributedScheduler. (asuresh) 
(cherry picked from commit 9c5c68745e)
 2017-10-27 23:11:34 -07:00
Jason Lowe 9093ad6168 YARN-7244. ShuffleHandler is not aware of disks that are added. Contributed by Kuhu Shukla 
(cherry picked from commit 665bb147aa)
 2017-10-27 17:09:29 -05:00