Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
25,219 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

6124 Commits

Author SHA1 Message Date
Eric Badger e4dcc31114 YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by 
Qi Zhu

(cherry picked from commit fcdb1e7fec)
 2021-02-26 19:07:59 +00:00
Jonathan Hung be6e99963d YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 
(cherry picked from commit f348ab3f2f468751af329a1ffce4917cb000fcbf)
 2021-02-25 15:09:33 -08:00
Jim Brennan db457b056a [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 
(cherry picked from commit c373da9f88)
 2021-02-25 16:48:46 +00:00
Eric Badger 305c4e72c2 YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by 
Qi Zhu.

(cherry picked from commit 47420ae3ed)
 2021-02-22 18:59:45 +00:00
Eric Badger 27f00c7cd4 YARN-10501. Can't remove all node labels after add node label without 
nodemanager port. Contributed by caozhiqiang.

(cherry picked from commit 4891e68c2b)
 2021-02-19 23:25:25 +00:00
Inigo Goiri 8c8ef2f444 YARN-9017. PlacementRule order is not maintained in CS. Contributed by Bilwa S T. 
(cherry picked from commit 35010120fb)
 2021-02-18 20:42:26 +05:30
Inigo Goiri 97171b9b18 YARN-8942. PriorityBasedRouterPolicy throws exception if all sub-cluster weights have negative value. Contributed by Bilwa S T. 
(cherry picked from commit 108ecf992f)
 2021-02-18 20:32:32 +05:30
bibinchundatt a1bebfd85e YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 
(cherry picked from commit 5323e83edf)
 2021-02-18 20:23:56 +05:30
Prabhu Joseph 72904c014d YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver. 
Contributed by Bilwa ST.

(cherry picked from commit c7e71a6c0b)
 2021-02-18 14:25:16 +05:30
Prabhu Joseph 0c46ab51b5 YARN-8047. RMWebApp make external class pluggable. 
Contributed by Bilwa S T.

(cherry picked from commit 3a4d05b850)
 2021-02-18 13:59:50 +05:30
Jim Brennan 457c8404b9 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 
(cherry picked from commit 3fcc7a55b6)
 2021-02-16 16:47:46 +00:00
Masatake Iwasaki 4468378e4b YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 
(cherry picked from commit f9a073c6c1)
 2021-02-11 20:26:09 +00:00
Jim Brennan d3c7cb7c38 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 
(cherry picked from commit c22c77af43)
 2021-02-05 17:37:04 +00:00
Akira Ajisaka bdd22b61c2
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 (#2546) 
Reviewed-by: Steve Loughran <stevel@apache.org>
(cherry picked from commit 52217fce3d)
 2021-02-05 16:15:06 +09:00
bibinchundatt 1520b84b36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 
(cherry picked from commit 8bc2dfbf36)
 2021-01-22 08:30:12 +05:30
Eric Badger 7fc624aabc YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 
(cherry picked from commit 768e2f42ba)
 2021-01-13 23:43:15 +00:00
Eric Payne 4518ece4c9 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 
(cherry picked from commit d8fadec944)
 2021-01-13 17:17:07 +00:00
Ahmed Hussein 98690134d5 YARN-10334. Close clients in TestDistributedShell (#2571) 
(cherry picked from commit 513f1995ad)
 2021-01-09 21:18:00 +09:00
Ahmed Hussein a40eabd717 YARN-10536. Client in distributedShell swallows interrupt exceptions (#2554) 
(cherry picked from commit 7a88f45366)
 2021-01-09 21:18:00 +09:00
Masatake Iwasaki 35740a25d9 YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. (#2586) 
(cherry picked from commit 5abeecd33f)
 2021-01-09 21:18:00 +09:00
He Xiaoqiao e95ee67632
Make upstream aware of 3.2.2 release. 2021-01-09 18:07:10 +08:00
Neil cd5ee0014f YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 
(cherry picked from commit fa4cf91b57)
 2021-01-08 10:38:08 -08:00
Szilard Nemeth f6b9f82b3f YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:41:17 +01:00
srinivasst 98565b6c60 YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564) 
Contributed by Srinivas S T

(cherry picked from commit 1b1791075a)
 2021-01-08 10:57:37 +05:30
Akira Ajisaka 3748c258f9
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. (#2592) 
(cherry picked from commit ae4945fb2c)
 2021-01-06 12:14:09 +09:00
Eric Badger 1b5b2bea83 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api 
for nodes. Contributed by Jim Brennan.

(cherry picked from commit 4c5d88e230)
 2020-12-21 23:11:47 +00:00
Akira Ajisaka 71bda1a2e8
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2538) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2020-12-11 13:58:02 +09:00
Akira Ajisaka b84bc63aa2 HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. (#2434) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-12-10 05:51:45 +05:30
Ayush Saxena 8378ab9f92 HADOOP-17288. Use shaded guava from thirdparty. Contributed by Ayush Saxena. #2505 2020-12-10 05:50:55 +05:30
Andrea Scarpino c5b9c5dfe5
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs (#2512) 
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9170eb566b)
 2020-12-04 00:16:45 +09:00
Eric Payne 1fd6d81617 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 
(cherry picked from commit fa773a8326)
 2020-12-01 22:51:20 +00:00
kevinzhao1661 7d9c6edadf
YARN-10498. Fix typo in CapacityScheduler Markdown document (#2484) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 4d2ae5b398)
 2020-11-30 11:17:59 +09:00
Akira Ajisaka db04195afd
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs (#2488) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 2ce2198287)
 2020-11-26 11:34:39 +09:00
Akira Ajisaka bb4c6db9b3
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi. 
(cherry picked from commit c4ba0ab7df)
 2020-11-24 15:22:51 +09:00
Ahmed Hussein bc634c077b YARN-10485. TimelineConnector swallows InterruptedException (#2450). Contributed by Ahmed Hussein 
(cherry picked from commit 0b2510ee1f)
 2020-11-16 21:09:36 +00:00
Gabriel Medeiros Coelho 9a3f2bb545 YARN-10480. replace href tags with ng-href (#2426) 
(cherry picked from commit 61f8c5767e)
 2020-11-10 11:00:00 -08:00
Peter Bacsko 0361837c9e YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. Contributed by Peter Bacsko. 2020-11-10 13:16:52 +01:00
Eric E Payne 1e22929885 YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 55339c2bdd)
 2020-11-05 22:08:51 +00:00
Eric E Payne 2473e8b711 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 17:16:28 +00:00
Jim Brennan cdaef111d5 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 
(cherry picked from commit 8ee6bc2518)
 2020-10-29 19:52:43 +00:00
Jonathan Hung f95c0824b0 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 
(cherry picked from commit bab5bf9743)
 2020-10-28 10:33:41 -07:00
Eric Badger 8abf939152 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics. 
Contributed by Jim Brennan.

(cherry picked from commit 8b8c672780)
 2020-10-15 18:09:00 +00:00
Jim Brennan 76e223a320 YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 
(cherry picked from commit deb35a32ba)
 2020-10-08 18:59:25 +00:00
Jim Brennan c789e944b7 YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 
(cherry picked from commit b361f29dda)
 2020-10-06 18:19:53 +00:00
Adam Antal 3ae78e40bf YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 
(cherry picked from commit a1f7e760df)
 2020-10-05 10:10:46 +02:00
Eric Yang 9176e8fe5d YARN-9809. Added node manager health status to resource manager registration call. 
Contributed by Eric Badger via eyang

(cherry picked from commit e8dc862d38)
 2020-09-28 16:41:53 +00:00
Hui Fei ed19f63998
HADOOP-17277. Correct spelling errors for separator (#2322) 
Contributed by Hui Fei.

(cherry picked from commit 474fa80bfb)
 2020-09-23 15:39:51 +09:00
Jim Brennan 0ec21b9667 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 
(cherry picked from commit 90894ea641)
 2020-09-14 21:22:02 +00:00
Eric E Payne 87ff2f5597 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 
(cherry picked from commit 9afec2ed17)
 2020-09-11 16:19:09 +00:00
Brahma Reddy Battula ea37a05d4b YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 
(cherry picked from commit 43572fc7f8)
 2020-09-09 17:09:35 +05:30
bibinchundatt acb219b9c9 YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 
(cherry picked from commit 5d8600e80a)
 2020-09-08 20:42:27 +00:00
Eric Badger 59e8508ce2 [YARN-10353] Log vcores used and cumulative cpu in containers monitor. 
Contributed by Jim Brennan

(cherry picked from commit 736bed6d6d)
 2020-09-08 15:37:35 +00:00
Adam Antal 5710005a19 YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 
(cherry picked from commit 34fe74da0e)
 2020-09-07 11:46:30 +02:00
Sunil G ea1c048a61 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 
(cherry picked from commit 82ec28f442)
 2020-08-19 11:56:40 +05:30
Jim Brennan 392a62278a YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger 
(cherry picked from commit 4a7deae478)
 2020-08-17 16:55:35 +00:00
hemanthboyina 5ededc0cba YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T. 
(cherry picked from commit 86bbd38c8d)
 2020-08-13 23:47:50 +05:30
Jonathan Hung 706d14c41d YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 
(cherry picked from commit 64753addba)
 2020-08-07 17:43:16 -07:00
bibinchundatt 0c0ea4979e YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 
(cherry picked from commit 81da221c75)
 2020-08-07 09:01:53 +05:30
Eric Badger 723f1f61d6 YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource. 
Contributed by Bibin Chundatt and Eric Payne.

(cherry picked from commit 5edd8b925e)
 2020-08-05 18:56:56 +00:00
Brahma Reddy Battula 643ff4881d YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 
(cherry picked from commit eac558380f)
 2020-08-03 12:55:31 +05:30
Eric E Payne 0cb1b178ec YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 
(cherry picked from commit e0c9653166)
 2020-07-30 16:39:56 +00:00
Jonathan Hung d3c7f7cdbd YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 
(cherry picked from commit 3eaf62726f)
 2020-07-28 13:43:44 -07:00
Eric Badger 7ec692aa83 YARN-4771. Some containers can be skipped during log aggregation after NM 
restart. Contributed by Jason Lowe and Jim Brennan.

(cherry picked from commit ac5f21dbef)
 2020-07-24 22:46:57 +00:00
Ayush Saxena e3b8d4eb05 HADOOP-17100. Replace Guava Supplier with Java8+ Supplier in Hadoop. Contributed by Ahmed Hussein. 2020-07-22 18:21:14 +05:30
Ahmed Hussein 9e7266df6c HADOOP-17099. Replace Guava Predicate with Java8+ Predicate 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 1f71c4ae71)
 2020-07-15 11:40:13 -05:00
Ahmed Hussein 5969922305 HADOOP-17101. Replace Guava Function with Java8+ Function 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 98fcffe93f)
 2020-07-15 09:57:36 -05:00
Eric Badger 41bcef9486 YARN-10348. Allow RM to always cancel tokens after app completes. Contributed by 
Jim Brennan

(cherry picked from commit 48f90115b5)
 2020-07-13 23:12:18 +00:00
Eric E Payne 7044a007b3 YARN-10297. TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently. Contributed by Jim Brennan (Jim_Brennan) 
(cherry picked from commit 0427100b75)
 2020-07-13 19:02:40 +00:00
Brahma Reddy Battula 7b175739a9 YARN-10341. Yarn Service Container Completed event doesn't get processed. Contributed by Bilwa S T. 
(cherry picked from commit dfe60392c9)
 2020-07-09 12:36:21 +05:30
Akira Ajisaka 0aa2d7d506
YARN-10344. Sync netty versions in hadoop-yarn-csi. (#2126) 
(cherry picked from commit 10d218934c)
 2020-07-09 15:07:44 +09:00
Akira Ajisaka 20df70a895
HADOOP-17111. Replace Guava Optional with Java8+ Optional. Contributed by Ahmed Hussein. 
(cherry picked from commit 639acb6d89)
 2020-07-06 16:09:37 +09:00
Szilard Nemeth 439c51425e YARN-10330. Add missing test scenarios to TestUserGroupMappingPlacementRule and TestAppNameMappingPlacementRule. Contributed by Peter Bacsko 2020-07-01 17:42:45 +02:00
Szilard Nemeth cfb2084cba YARN-10325. Document max-parallel-apps for Capacity Scheduler. Contributed by Peter Bacsko 2020-07-01 13:40:36 +02:00
Szilard Nemeth d88a6eebf2 YARN-10318. ApplicationHistory Web UI incorrect column indexing. Contributed by Andras Gyori 2020-07-01 13:31:35 +02:00
Szilard Nemeth 8b482744e9 YARN-10277. CapacityScheduler test TestUserGroupMappingPlacementRule should build proper hierarchy. Contributed by Szilard Nemeth 2020-06-30 11:32:59 +02:00
Akira Ajisaka aa283fc2c2
YARN-10331. Upgrade node.js to 10.21.0. (#2106) 
(cherry picked from commit cd188ea9f0)
 2020-06-30 16:54:59 +09:00
Eric E Payne d7696453a0 YARN-9903: Support reservations continue looking for Node Labels. Contributed by Jim Brennan (Jim_Brennan). 
(cherry picked from commit 74fc13cf91)
 2020-06-29 18:59:52 +00:00
Szilard Nemeth fa41e38450 YARN-10279. Avoid unnecessary QueueMappingEntity creations. Contributed by Marton Hudaky 
(cherry picked from commit 6a8fd73b27)
 2020-06-25 17:28:48 +02:00
Szilard Nemeth 480919e42d YARN-10316. FS-CS converter: convert maxAppsDefault, maxRunningApps settings. Contributed by Peter Bacsko 2020-06-23 16:25:33 +02:00
Szilard Nemeth 8f1b70e367 YARN-9930. Support max running app logic for CapacityScheduler. Contributed by Peter Bacsko 2020-06-22 12:00:06 +02:00
Szilard Nemeth ec913398a9 YARN-10281. Redundant QueuePath usage in UserGroupMappingPlacementRule and AppNameMappingPlacementRule. Contributed by Gergely Pollak 2020-06-17 14:36:08 +02:00
Szilard Nemeth 8be302a3b8 YARN-10274. Merge QueueMapping and QueueMappingEntity. Contributed by Gergely Pollak 2020-06-16 18:25:47 +02:00
Szilard Nemeth 52efe48d79 YARN-10292. FS-CS converter: add an option to enable asynchronous scheduling in CapacityScheduler. Contributed by Benjamin Teke 2020-06-16 18:01:39 +02:00
Vinayakumar B 534b15caf9
HADOOP-17046. Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes (#2026) 2020-06-12 23:20:10 +05:30
Eric Badger fcd7ce53b5 YARN-10312. Add support for yarn logs -logFile to retain backward compatibility. 
Contributed by Jim Brennan.

(cherry picked from commit fed6fecd3a)
 2020-06-11 21:11:20 +00:00
Szilard Nemeth e35f619841 YARN-10296. Make ContainerPBImpl#getId/setId synchronized. Contributed by Benjamin Teke 2020-06-10 18:00:21 +02:00
Eric E Payne a7526ba9f7 YARN-10300: appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeat. Contributed by Eric Badger (ebadger). 
(cherry picked from commit 56247db302)
 2020-06-09 18:51:46 +00:00
Szilard Nemeth ac307fe20d YARN-10284. Add lazy initialization of LogAggregationFileControllerFactory in LogServlet. Contributed by Adam Antal 2020-06-05 12:40:57 +02:00
Szilard Nemeth a266e32d82 YARN-10286. PendingContainers bugs in the scheduler outputs. Contributed by Andras Gyori 2020-06-05 09:50:43 +02:00
Szilard Nemeth f65f64e8ae YARN-10254. CapacityScheduler incorrect User Group Mapping after leaf queue change. Contributed by Gergely Pollak 2020-06-02 18:32:06 +02:00
Prabhu Joseph 07b8963aa3 YARN-10259. Fix reservation logic in Multi Node Placement. 
Reviewed by Wangda Tan.

(cherry picked from commit 6ce295b787)
 2020-06-01 18:24:38 +05:30
Jonathan Hung 7a323a45aa YARN-6492. Generate queue metrics for each partition. Contributed by Manikandan R 
(cherry picked from commit c30c23cb66)
 2020-05-29 09:25:08 -07:00
Szilard Nemeth 94358dbc3c YARN-10108. FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e)
 2020-05-27 13:07:17 +02:00
Szilard Nemeth 2c60bcb1c2 Revert "FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak" 
This reverts commit 9ce913a7bf.
Reverting because of missing Jira-ID in the commit message
 2020-05-27 13:07:01 +02:00
Szilard Nemeth 9ce913a7bf FS-CS converter: nestedUserQueue with default rule results in invalid queue mapping. Contributed by Gergely Pollak 
(cherry picked from commit ac4540dd8e)
 2020-05-22 14:48:53 +02:00
Eric Yang d7cf19d7c0 YARN-10228. Relax restriction of file path character in yarn.service.am.java.opts. 
Contributed by Bilwa S T via eyang
 2020-05-20 09:31:40 -07:00
Surendra Singh Lilhore 7cef7109de YARN-10265. Upgrade Netty-all dependency to latest version 4.1.50 to fix ARM support issue. Contributed by liusheng 2020-05-14 15:17:13 +05:30
Ayush Saxena 74887ab2ad YARN-9898. Dependency netty-all-4.1.27.Final doesn't support ARM platform. Contributed by liusheng. 2020-05-14 00:37:09 +05:30
Jonathan Hung 564d3211f2 YARN-10260. Allow transitioning queue from DRAINING to RUNNING state. Contributed by Bilwa S T 
(cherry picked from commit fff1d2c122)
 2020-05-12 10:49:10 -07:00
Szilard Nemeth 95e4ed18a6 YARN-10160. Add auto queue creation related configs to RMWebService#CapacitySchedulerQueueInfo. Contributed by Prabhu Joseph 2020-05-12 09:03:50 +02:00
Ahmed Hussein 7aade932c7 YARN-8959. TestContainerResizing fails randomly (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit 92e3ebb401)
 2020-05-06 12:28:04 -05:00
Ahmed Hussein 1d8350055e YARN-10256. Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunistic (Ahmed Hussein via jeagles) 
Signed-off-by: Jonathan Eagles <jeagles@gmail.com>
(cherry picked from commit f5081a9a5d)
 2020-05-04 10:24:01 -05:00
Akira Ajisaka dfa7f160a5
Preparing for 3.3.1 development 2020-04-30 13:33:42 +09:00
Szilard Nemeth 8ffe1f313c YARN-10247. Application priority queue ACLs are not respected. Contributed by Sunil G 
(cherry picked from commit 410c605aec)
 2020-04-29 15:54:32 +02:00
Szilard Nemeth 75433a658d YARN-10237. Add isAbsoluteResource config for queue in scheduler response. Contributed by Prabhu Joseph 2020-04-29 09:36:37 +02:00
Szilard Nemeth e45faae1d4 YARN-10215. Endpoint for obtaining direct URL for the logs. Contributed by Andras Gyori 
(cherry picked from commit ab36429559)
 2020-04-28 18:56:29 +02:00
Akira Ajisaka 62ee5970f8
YARN-9848. Revert YARN-4946. Contributed by Steven Rand. 
(cherry picked from commit 6ef01646ba)
 2020-04-28 05:45:37 +09:00
Szilard Nemeth c0b7b38e22 YARN-10194. YARN RMWebServices /scheduler-conf/validate leaks ZK Connections. Contributed by Prabhu Joseph 
(cherry picked from commit f91e21ac10)
 2020-04-27 08:26:46 +02:00
Szilard Nemeth 7a3f190d89 YARN-10189. Code cleanup in LeveldbRMStateStore. Contributed by Benjamin Teke 2020-04-24 17:09:35 +02:00
Szilard Nemeth 61ca459c74 YARN-9999. TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanup. Contributed by Benjamin Teke 2020-04-24 11:28:52 +02:00
Akira Ajisaka 497c7a1680
YARN-10223. Remove jersey-test-framework-core dependency from yarn-server-common. (#1939) 
(cherry picked from commit 9827ff2961)
 2020-04-24 10:27:39 +09:00
Szilard Nemeth 1340518cd8 YARN-9996. Code cleanup in QueueAdminConfigurationMutationACLPolicy. Contributed by Siddharth Ahuja 2020-04-23 14:55:50 +02:00
Jonathan Hung 49ae9b2137 YARN-9954. Configurable max application tags and max tag length. Contributed by Bilwa S T 2020-04-17 10:34:59 -07:00
Szilard Nemeth 4bd37f2283 YARN-5277. When localizers fail due to resource timestamps being out, provide more diagnostics. Contributed by Siddharth Ahuja 2020-04-15 07:54:53 +02:00
Szilard Nemeth f473473355 YARN-10234. FS-CS converter: don't enable auto-create queue property for root. Contributed by Peter Bacsko 
(cherry picked from commit 55fcbcb5c2)
 2020-04-15 07:27:47 +02:00
Jonathan Hung e1dd78143b YARN-10212. Create separate configuration for max global AM attempts. Contributed by Bilwa S T 
(cherry picked from commit 57659422abbf6d9bf52e6e27fca775254bb77a56)
 2020-04-09 11:06:49 -07:00
Szilard Nemeth 061a7a8a41 YARN-9995. Code cleanup in TestSchedConfCLI. Contributed by Bilwa S T. 
(cherry picked from commit c04a484aed)
 2020-04-09 15:47:57 +02:00
Sunil G 94df6b513b YARN-10226. NPE in Capacity Scheduler while using %primary_group queue mapping. Contributed by Peter Bacsko. 
(cherry picked from commit 6fb29d0f17)
 2020-04-09 18:52:52 +05:30
Akira Ajisaka e8520fd132
HADOOP-14836. Upgrade maven-clean-plugin to 3.1.0 (#1933) 
(cherry picked from commit e53d472bb0)
 2020-04-09 01:49:01 +09:00
Wilfred Spiegelenburg 618a9208f7
YARN-10063. Add container-executor arguments --http/--https to usage. Contributed by Siddharth Ahuja 
(cherry picked from commit 2214005c0f)
 2020-04-08 13:02:51 +10:00
Szilard Nemeth d293e120eb YARN-10207. CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UI. Contributed by Siddharth Ahuja 
(cherry picked from commit bffb43b00e)
 2020-04-07 17:04:01 +02:00
Szilard Nemeth 7abc6221a3 YARN-10001. Add explanation of unimplemented methods in InMemoryConfigurationStore. Contributed by Siddharth Ahuja 
(cherry picked from commit 45362a9f4c)
 2020-04-07 16:01:47 +02:00
Akira Ajisaka a2f4dcb470
YARN-10202. Fix documentation about NodeAttributes. Contributed by Sen Zhao. 
(cherry picked from commit c162648aff)
 2020-04-01 16:06:16 +09:00
RogPodge 50f7f6dfd1
YARN-10210. Add a RMFailoverProxyProvider that does DNS resolution on failover. 2020-03-26 08:14:30 -07:00
Szilard Nemeth 3d5ade1839 YARN-10043. FairOrderingPolicy Improvements. Contributed by Manikandan R 2020-03-26 13:23:43 +01:00
Szilard Nemeth 9a297ff31d YARN-10197. FS-CS converter: fix emitted ordering policy string and max-am-resource percent value. Contributed by Peter Bacsko 2020-03-26 12:35:10 +01:00
Szilard Nemeth 348685dcb9 YARN-9998. Code cleanup in LeveldbConfigurationStore. Contributed by Benjamin Teke 2020-03-26 12:18:15 +01:00
Jonathan Hung 6ce189c621 YARN-10200. Add number of containers to RMAppManager summary 2020-03-25 10:41:59 -07:00
RogPodge 2d294bd575
HADOOP-16938. Make non-HA proxy providers pluggable 2020-03-25 08:06:58 -07:00
Sunil G cdb2107066 YARN-9879. Allow multiple leaf queues with the same name in CapacityScheduler. Contributed by Gergely Pollak. 2020-03-25 16:50:19 +05:30
Prabhu Joseph d312991ead YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue 
Contributed by Peter Bacsko. Reviewed by Manikandan R.
 2020-03-23 18:01:14 +05:30
Prabhu Joseph f2d3ac2a3f YARN-10034. Remove Allocation Tags from released container from Decommission node 
Contributed by Kyungwan Nam. Reviewed by Adam Antal.
 2020-03-19 12:55:14 +05:30
Eric Badger 1975479285 YARN-2710. RM HA tests failed intermittently on trunk. Contributed by Ahmed 
Hussein.
 2020-03-16 22:24:02 +00:00
Weiwei Yang b9d825f178 YARN-9567. Add diagnostics for outstanding resource requests on app attempts page. Contributed by Tao Yang. 2020-03-13 14:39:36 -07:00
Weiwei Yang d273f6a2b4 YARN-9538. Document scheduler/app activities and REST APIs. Contributed by Tao Yang. 2020-03-13 14:36:52 -07:00
Bilwa c83644deac YARN-10110. Adding RouterPolicyProvider for RM Federation. 
Contributed by Bilwa S T.
 2020-03-13 16:56:44 +05:30
Szilard Nemeth 5ead9c15ca YARN-9997. Code cleanup in ZKConfigurationStore. Contributed by Andras Gyori 2020-03-12 12:29:03 +01:00
Szilard Nemeth 38d87883b6 YARN-10193. FS-CS converter: fix incorrect capacity conversion. Contributed by Peter Bacsko 2020-03-12 12:18:37 +01:00
Sunil G 0fd8bf5f6b YARN-10191. FS-CS converter: call System.exit function call for every code path in main method. Contributed by Peter Bacsko. 2020-03-12 15:03:12 +05:30
Szilard Nemeth cf9cf83a43 YARN-9354. Resources should be created with ResourceTypesTestHelper instead of TestUtils. Contributed by Andras Gyori 2020-03-10 16:44:48 +01:00
Szilard Nemeth 61f4cf3055 YARN-10002. Code cleanup and improvements in ConfigurationStoreBaseTest. Contributed by Benjamin Teke 2020-03-10 16:35:04 +01:00
Szilard Nemeth 9314ef947f YARN-10168. FS-CS Converter: tool doesn't handle min/max resource conversion correctly. Contributed by Peter Bacsko 2020-03-10 16:07:46 +01:00
Eric E Payne ede05b19d1 YARN-942. TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically Contributed by Ahmed Hussein (ahussein) 2020-03-10 14:17:45 +00:00
Szilard Nemeth 44afe1154d YARN-9419. Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabled. Contribued by Andras Gyori 2020-03-09 16:08:24 +01:00
bibinchundatt 3859fa76d0 YARN-6924. Metrics for Federation AMRMProxy. Contributed by Young Chen 2020-03-07 09:34:42 +05:30
Szilard Nemeth ea0444851d YARN.10003. YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStore. Contributed by Benjamin Teke 2020-03-06 17:49:45 +01:00
Szilard Nemeth 004e955348 YARN-10167. FS-CS Converter: Need to validate c-s.xml after converting. Contributed by Peter Bacsko 2020-03-05 09:56:42 +01:00
Eric Badger 2649f8b327 YARN-10173. Make pid file generation timeout configurable in case of reacquired 
container. Contributed by Adam Antal.
 2020-03-04 23:31:57 +00:00
Szilard Nemeth c0d084247c YARN-10175. FS-CS converter: only convert placement rules if a cmd line switch is defined. Contributed by Peter Bacsko 2020-03-03 09:46:32 +01:00
Szilard Nemeth e9eecedf69 YARN-10148. addendum: Fix method call parameter order of setAdminAndSubmitACL in TestCapacitySchedulerQueueACLs. Contributed by Kinga Marton 2020-03-02 16:25:14 +01:00
Inigo Goiri a43510e21d YARN-10161. TestRouterWebServicesREST is corrupting STDOUT. Contributed by Jim Brennan. 2020-02-27 13:18:30 -08:00
Inigo Goiri b420ddeada YARN-10155. TestDelegationTokenRenewer.testTokenThreadTimeout fails in trunk. Contributed by Manikandan R. 2020-02-27 12:53:20 -08:00
Szilard Nemeth 10461e0193 YARN-10148. Add Unit test for queue ACL for both FS and CS. Contributed by Kinga Marton 2020-02-27 20:10:32 +01:00
Akira Ajisaka 7dfa37e8f0
YARN-10152. Fix findbugs warnings in hadoop-yarn-applications-mawo-core module (#1852) 2020-02-27 10:36:49 +09:00
Brahma Reddy Battula 3a9ccf7f6d YARN-10141.Interceptor in FederationInterceptorREST doesnt update on RM switchover. Contributed by D M Murali Krishna Reddy. 2020-02-26 23:24:00 +05:30
tanu.ajmera b5df1da3a1 YARN-9593. Support Comma in the value of Scheduler Configuration Mutation API. 2020-02-26 15:04:29 +05:30
Elixir Kook d608e94f92
YARN-10156. Fix typo 'complaint' which means quite different in Federation.md (#1856) 2020-02-26 17:29:26 +09:00
Szilard Nemeth d68616b045 YARN-10130. FS-CS converter: Do not allow output dir to be the same as input dir. Contributed by Adam Antal 2020-02-25 21:30:04 +01:00
Wei-Chiu Chuang dda00d3ff5
YARN-10074. Update netty to 4.1.42Final in yarn-csi. Contributed by Wei-Chiu Chuang. (#1807) 2020-02-25 13:47:52 +09:00
Szilard Nemeth 72fa9c7f1b YARN-10135. FS-CS converter tool: issue warning on dynamic auto-create mapping rules. Contributed by Peter Bacsko 2020-02-24 21:54:07 +01:00
Szilard Nemeth 34cf63c7d0 YARN-10157. FS-CS converter: initPropertyActions() is not called without rules file. Contributed by Peter Bacsko 2020-02-24 21:45:49 +01:00
Prabhu Joseph 181e6d033e YARN-10119. Option to reset AM failure count for YARN Service 
Contributed by kyungwan nam.
 2020-02-20 20:34:06 +05:30
Szilard Nemeth 4af2556b48 YARN-10143. YARN-10101 broke Yarn logs CLI. Contributed by Adam Antal 2020-02-20 15:27:15 +01:00
Szilard Nemeth 3f1aad05f0 YARN-10147. FPGA plugin can't find the localized aocx file. Contributed by Peter Bacsko 2020-02-19 15:54:25 +01:00
Sunil G 6526f95bd2 YARN-10139. ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB. Contributed by Prabhu Joseph. 2020-02-19 11:17:22 +05:30
Akira Ajisaka d6d7f8d8c5
YARN-8374. Upgrade objenesis to 2.6 (#1798) 2020-02-19 09:50:37 +09:00
Brahma Reddy Battula 20add89718 YARN-10136. [Router] : Application metrics are hardcode as N/A in UI. Contributed by Bilwa S T. 2020-02-14 16:50:29 +05:30
Szilard Nemeth da99ac7e93 YARN-10137. UIv2 build is broken in trunk. Contributed by Adam Antal 2020-02-13 16:31:35 +01:00
Szilard Nemeth f1b1b332f5 YARN-10029. Add option to UIv2 to get container logs from the new JHS API. Contributed by Adam Antal 2020-02-13 12:08:54 +01:00
Prabhu Joseph fe7d67a8a2 YARN-9521. Handle FileSystem close in ApiServiceClient 
Contributed by kyungwan nam. Reviewed by Eric Yang.
 2020-02-13 09:39:13 +05:30
Prabhu Joseph e637797211 YARN-10127. Remove setting App Ordering Policy to ParentQueue in FSQueueConverter 
Contributed by Peter Bacsko.
 2020-02-11 22:01:58 +05:30
Sunil G 28f730b317 YARN-10109. Allow stop and convert from leaf to parent queue in a single Mutation API call. Contributed by Prabhu Joseph 2020-02-09 21:14:53 +05:30
Ayush Saxena 3f0a7cd17a YARN-9624. Use switch case for ProtoUtils#convertFromProtoFormat containerState. Contributed by Bilwa S T 2020-02-09 19:14:18 +05:30
Vinayakumar B 7dac7e1d13
HADOOP-16596. [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency (#1635). Contributed by Vinayakumar B. 2020-02-07 14:51:24 +05:30
Szilard Nemeth 71b2c2ffe9 YARN-10101. Support listing of aggregated logs for containers belonging to an application attempt. Contributed by Adam Antal 2020-02-06 12:25:06 +01:00
Jonathan Hung 314e2f9d2e YARN-10116. Expose diagnostics in RMAppManager summary 2020-02-04 17:44:05 -08:00
Giovanni Matteo Fumarola bf8686f43f YARN-8982. [Router] Add locality policy. Contributed by Young Chen. 2020-01-30 16:59:36 -08:00
Szilard Nemeth a7d72c523a YARN-10099. FS-CS converter: handle allow-undeclared-pools and user-as-default-queue properly and fix misc issues. Contributed by Peter Bacsko 2020-01-30 16:03:38 +01:00
Akira Ajisaka a5ef08b619
YARN-9743. [JDK11] TestTimelineWebServices.testContextFactory fails. (#1824) Contributed by Akira Ajisaka and Kinga Marton. 2020-01-30 14:10:31 +09:00
Prabhu Joseph 825db8fe2a YARN-10107. Fix GpuResourcePlugin#getNMResourceInfo to honor Auto Discovery Enabled 
Contributed by Szilard Nemeth.
 2020-01-29 13:30:00 +05:30
Eric Badger e578e52aae YARN-10084. Allow inheritance of max app lifetime / default app lifetime. Contributed by Eric Payne. 2020-01-29 03:54:43 +00:00
Inigo Goiri 5abd0148eb YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-28 10:06:37 -08:00
Prabhu Joseph 1ab9c692fa YARN-10022. RM Rest API to validate the CapacityScheduler Configuration change 
Contributed by Kinga Marton.
 2020-01-28 23:16:04 +05:30
Szilard Nemeth 94f060242e YARN-10104. FS-CS converter: dry run should work without output defined. Contributed by Peter Bacsko 2020-01-28 15:36:04 +01:00
Szilard Nemeth ca29768035 YARN-10085. FS-CS converter: remove mixed ordering policy check. Contributed by Peter Bacsko 2020-01-28 15:22:48 +01:00
Szilard Nemeth 9520b2ad79 YARN-10083. Provide utility to ask whether an application is in final status. Contributed by Adam Antal 2020-01-22 16:25:07 +01:00
Inigo Goiri b4870bce3a Revert "YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R." 
This reverts commit 0696828a09.
 2020-01-21 17:45:17 -08:00
Inigo Goiri 0696828a09 YARN-9768. RM Renew Delegation token thread should timeout and retry. Contributed by Manikandan R. 2020-01-21 13:41:01 -08:00
Akira Ajisaka 1defe3a65a
HADOOP-16753. Refactor HAAdmin. Contributed by Xieming Li. 2020-01-21 10:58:32 +09:00
Szilard Nemeth 581072a8f0 YARN-7913. Improve error handling when application recovery fails with exception. Contributed by Wilfred Spiegelenburg 2020-01-20 13:10:32 +01:00
Szilard Nemeth 6d52bbbfcf YARN-9525. IFile format is not working against s3a remote folder. Contributed by Adam Antal 2020-01-20 12:36:55 +01:00
Szilard Nemeth 8b3ee2f7e9 YARN-9462. TestResourceTrackerService.testNodeRemovalGracefully fails sporadically. Contributed by Prabhu Joseph 2020-01-20 09:54:22 +01:00
Szilard Nemeth 57aad0f43a YARN-10081. Exception message from ClientRMProxy#getRMAddress is misleading. Contributed by Ravuri Sushma Sree 2020-01-20 09:41:06 +01:00
Szilard Nemeth 14d0f9a775 YARN-8148. Update decimal values for queue capacities shown on queue status CLI. Contributed by Prabhu Joseph 2020-01-20 09:28:23 +01:00
Prabhu Joseph a0ff42d761 YARN-10070. Fix NPE if no queue mapping defined for proxy user when using application tag based placement. 
Contributed by Kinga Marton. Reviewed by Adam Antal.
 2020-01-16 12:55:27 +05:30
Akira Ajisaka 14c2c3d69d
YARN-9512. [JDK11] TestAuxServices#testCustomizedAuxServiceClassPath fails because of ClassCastException. (#1806) 2020-01-16 12:21:50 +09:00
Szilard Nemeth 2aa065d98f YARN-10082. FS-CS converter: disable terminal placement rule checking. Contributed by Peter Bacsko 2020-01-15 12:42:13 +01:00
Szilard Nemeth 7c5cecc3b3 YARN-9970. Refactor TestUserGroupMappingPlacementRule#verifyQueueMapping. Contributed by Manikandan R 2020-01-15 09:55:41 +01:00
Szilard Nemeth 1c51f36be7 YARN-9788. Queue Management API does not support parallel updates. Contributed by Prabhu Joseph 2020-01-14 12:26:03 +01:00
Szilard Nemeth 13cea0412c YARN-10028. Integrate the new abstract log servlet to the JobHistory server. Contributed by Adam Antal 2020-01-14 11:00:08 +01:00