Commit Graph

4466 Commits

Author SHA1 Message Date
Szilard Nemeth 2541efa496 YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori 2021-05-25 13:55:59 +02:00
Viraj Jasani 996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project (#3033)
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 18:10:20 +09:00
zhuqi-lucas 59172ada90 YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju. 2021-05-24 23:12:07 +08:00
Szilard Nemeth 1e44bdb84c YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke 2021-05-22 14:55:01 +02:00
Peter Bacsko eb72628e15 YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B. 2021-05-17 21:07:29 +02:00
Peter Bacsko 8891e5c028 YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju. 2021-05-17 13:30:12 +02:00
lujiefsi d92a25b790
YARN-10555. Missing access check before getAppAttempts (#2608)
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-17 13:44:38 +09:00
zhuqi e7f0e8073b YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu. 2021-05-14 13:36:07 +08:00
zhuqi d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. (#2911)
Contributed by Qi Zhu.
2021-05-14 13:12:28 +08:00
Peter Bacsko 626be24c3e YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori. 2021-05-12 14:54:47 +02:00
Akira Ajisaka 7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. (#2960)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-06 23:37:45 +09:00
Wei-Chiu Chuang f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945)
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-02 11:10:32 +09:00
Eric Badger 803ac4b1a0 YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu 2021-04-29 17:42:13 +00:00
Peter Bacsko 9166bfeb74 YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu. 2021-04-24 10:10:10 +02:00
Eric Badger 6857a05d6a YARN-10479. Can't remove all node labels after add node label without
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
2021-04-23 22:02:04 +00:00
Jim Brennan 1cbe35946b YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu. 2021-04-23 21:35:09 +00:00
Peter Bacsko 14a84c47b0 YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja. 2021-04-23 17:38:47 +02:00
Szilard Nemeth 3b6b21e459 YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak 2021-04-23 16:17:29 +02:00
Szilard Nemeth 919daec36b YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak 2021-04-23 16:12:12 +02:00
Szilard Nemeth f76a2a7606 YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko 2021-04-23 16:07:58 +02:00
Eric Badger 6cb90005a7 YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu 2021-04-20 17:34:49 +00:00
Viraj Jasani 9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation (#2922)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-17 20:47:44 +09:00
Eric Badger 213d3deb26 YARN-10503. Support queue capacity in terms of absolute resources with custom
resourceType. Contributed by Qi Zhu.
2021-04-09 00:34:15 +00:00
Peter Bacsko ca9aa91d10 YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori. 2021-04-08 12:42:48 +02:00
Szilard Nemeth 9cd69c20c4 YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori 2021-04-07 11:52:21 +02:00
Eric Badger 26b8f678b2 YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor.
Contributed by Jim Brennan.
2021-04-06 01:16:14 +00:00
Peter Bacsko 158758c5bf YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu. 2021-04-01 16:09:52 +02:00
Peter Bacsko 9f1655baf2 YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu. 2021-04-01 11:39:40 +02:00
Peter Bacsko a0deda1a77 YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu. 2021-04-01 09:21:15 +02:00
Szilard Nemeth 6fd0c661b6 YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak 2021-03-31 16:14:21 +02:00
Peter Bacsko ff6ec20d84 YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu. 2021-03-31 10:55:14 +02:00
Akira Ajisaka 8ad77a26c9
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai. 2021-03-30 17:46:51 +09:00
Eric Badger 19e418c10d YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu. 2021-03-25 22:33:58 +00:00
Peter Bacsko ceb75e1e2a YARN-10674. fs2cs should generate auto-created queue deletion properties. Contributed by Qi Zhu. 2021-03-24 08:15:06 +01:00
Jim Brennan 174f3a96b1 YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T. 2021-03-23 18:21:45 +00:00
Jack Jiang d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof (#2804) 2021-03-23 08:46:11 -07:00
Cyrus Jackson cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications (#2674). Contributed by Cyrus Jackson 2021-03-19 15:49:05 +05:30
Eric Badger 3d4ea4b336 YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu. 2021-03-18 19:36:50 +00:00
Eric Badger 181b9ddc51 YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu. 2021-03-18 17:45:09 +00:00
Peter Bacsko ce6bfd5718 YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu. 2021-03-18 13:40:16 +01:00
Peter Bacsko 38495af325 YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu. 2021-03-18 12:46:11 +01:00
Szilard Nemeth a5745711dd YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak 2021-03-18 12:43:01 +01:00
Peter Bacsko d7eeca4d0c YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu. 2021-03-18 11:49:16 +01:00
Eric Badger 49f89f1d3d YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu. 2021-03-17 18:11:37 +00:00
Peter Bacsko 3e58d5611d YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu. 2021-03-17 13:38:20 +01:00
Wilfred Spiegelenburg f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.

Contributed by: Siddharth Ahuja
2021-03-17 10:55:05 +11:00
Peter Bacsko 73394fabc7 YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu. 2021-03-16 16:09:12 +01:00
Peter Bacsko b80588b688 YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu. 2021-03-16 15:23:27 +01:00
zhuqi e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. (#2760)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-16 13:50:29 +09:00
Eric Payne aa4c17b9d7 YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T 2021-03-15 19:09:40 +00:00
Jim Brennan 5f067cf0f3 [YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu. 2021-03-12 17:17:31 +00:00
Szilard Nemeth 5db4c0bf70 YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak 2021-03-12 14:10:16 +01:00
Akira Ajisaka 23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2753)
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
Brahma Reddy Battula b2a565629d YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy. 2021-03-09 20:26:07 +05:30
Peter Bacsko d5e035dbe1 YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu. 2021-03-09 14:33:14 +01:00
Peter Bacsko 3851994cd6 Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu."
This reverts commit 369f75b7a7.
2021-03-09 14:32:02 +01:00
Peter Bacsko 369f75b7a7 YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu. 2021-03-09 14:28:23 +01:00
Peter Bacsko 20416bc62d YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth. 2021-03-09 11:34:37 +01:00
Peter Bacsko e472ee2aa5 YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth. 2021-03-09 11:28:24 +01:00
Peter Bacsko 04cd3115ba YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu. 2021-03-08 12:29:44 +01:00
Peter Bacsko c6b30a59dd YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth. 2021-03-05 23:13:35 +01:00
Eric Badger 9d088639bf YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim
Brennan.
2021-03-05 19:46:40 +00:00
Peter Bacsko e82e7c597a YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu. 2021-03-05 15:56:56 +01:00
Peter Bacsko e19c00925f YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu. 2021-03-05 13:18:06 +01:00
Peter Bacsko 6699198b54 YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu. 2021-03-04 17:18:35 +01:00
Peter Bacsko a85aeee876 YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu. 2021-03-04 12:23:11 +01:00
Neil d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak (#2719). Contributed by Max Xie 2021-03-04 14:52:58 +05:30
Peter Bacsko 7e8040e6ad YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori. 2021-03-03 17:44:30 +01:00
JJiaguodong 8af56de1fa
HADOOP-17560. Fix some spelling errors (#2730)
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-03-03 11:41:05 +09:00
Peter Bacsko 947b50489d YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke. 2021-02-26 21:57:54 +01:00
Ahmed Hussein e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase (#2722). Contributed by Ahmed Hussein 2021-02-26 13:42:33 -06:00
Jonathan Hung ac37ca0b67 YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen 2021-02-25 16:01:18 -08:00
Jim Brennan c373da9f88 [YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne 2021-02-25 16:46:19 +00:00
Peter Bacsko d28b6f90c8 YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak. 2021-02-19 13:57:30 +01:00
Peter Bacsko 4383726d19 YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak. 2021-02-19 12:01:31 +01:00
Akira Ajisaka 9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)"
This reverts commit 4cf3531583.
2021-02-19 11:09:10 +09:00
Akira Ajisaka 4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin (#2454)
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.

Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
Szilard Nemeth 9882a22365 YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko 2021-02-16 18:06:57 +01:00
Jim Brennan 3fcc7a55b6 [YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger 2021-02-16 16:45:35 +00:00
Szilard Nemeth bad6038a48 YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori 2021-02-15 14:48:42 +01:00
Szilard Nemeth 852aac34f2 Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu 2021-02-15 14:20:50 +01:00
Masatake Iwasaki f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. (#2619) 2021-02-11 13:55:04 -06:00
Szilard Nemeth cacc870389 YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko 2021-02-10 13:09:32 +01:00
Szilard Nemeth a8bd516e39 YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko 2021-02-10 11:15:53 +01:00
Szilard Nemeth fc5f241535 YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak 2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai 0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread (#2665)
Amogh Rajesh Desai via Zhankun Tang
2021-02-09 11:05:14 +08:00
Jim Brennan c22c77af43 [YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger. 2021-02-05 17:33:01 +00:00
Szilard Nemeth 5aa9866ec2 YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori 2021-02-05 17:43:01 +01:00
Szilard Nemeth c19326c051 YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu 2021-02-05 17:34:45 +01:00
Szilard Nemeth 79a46599f7 YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori 2021-02-05 17:09:08 +01:00
Eric Badger 5e719bf588 YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu 2021-02-04 18:30:39 +00:00
Szilard Nemeth c17414111b YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak 2021-02-04 08:12:13 +01:00
bibinchundatt 6fc26ad539 YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu 2021-02-03 08:50:45 +05:30
Szilard Nemeth 9bf2ac07bb YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori 2021-02-02 09:13:46 +01:00
Szilard Nemeth 7d88953768 YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori 2021-02-02 09:07:25 +01:00
belugabehr 21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class (#2633) 2021-02-01 09:40:01 -08:00
Peter Bacsko fa15594ae6 YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke. 2021-01-28 20:58:51 +01:00
Szilard Nemeth 9628aa87bf YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke 2021-01-27 18:14:06 +01:00
Szilard Nemeth 7c4ef42837 YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko 2021-01-27 15:49:26 +01:00
Szilard Nemeth e2a7008d50 YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak 2021-01-26 18:31:39 +01:00
Wangda Tan 4f008153ef YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda)
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
2021-01-26 08:24:29 -08:00
Wangda Tan b7384a8d02 YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda)
Change-Id: I670e5525619b320745254609c48e7e1afb084835
2021-01-26 08:24:28 -08:00
Szilard Nemeth f1766e5bb4 YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori 2021-01-26 16:22:35 +01:00
Szilard Nemeth 8620984b8d YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko 2021-01-21 20:58:57 +01:00
Peter Bacsko 17fdac8de5 YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth. 2021-01-21 18:06:53 +01:00
Szilard Nemeth 06fef5ee43 YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth 2021-01-21 09:23:11 +01:00
bibinchundatt 8bc2dfbf36 YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal 2021-01-21 11:23:30 +05:30
Szilard Nemeth cfe6e1f7da YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori 2021-01-20 15:22:44 +01:00
Sunil G 4b5bc05a78 YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth. 2021-01-20 12:28:59 +05:30
Szilard Nemeth a326f22606 YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko 2021-01-19 09:42:40 +01:00
Szilard Nemeth 6abdb148e4 YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak 2021-01-18 20:19:36 +01:00
Szilard Nemeth acea862d59 YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak 2021-01-18 16:04:45 +01:00
Wangda Tan 3d46141583 YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori)
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
2021-01-15 14:20:08 -08:00
Eric Badger 768e2f42ba YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan. 2021-01-13 23:41:52 +00:00
Eric Payne d8fadec944 YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan) 2021-01-13 17:01:28 +00:00
Szilard Nemeth 836c630430 YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko 2021-01-13 14:24:12 +01:00
bibinchundatt 61f77b7674 YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal 2021-01-13 16:00:11 +05:30
Szilard Nemeth 85b1c017ee YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko 2021-01-12 15:29:20 +01:00
Szilard Nemeth 9c6e093783 YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko 2021-01-12 15:21:27 +01:00
Wangda Tan b0eec09097 YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori)
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
2021-01-11 17:46:09 -08:00
Szilard Nemeth 4176759909 YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko 2021-01-08 13:40:33 +01:00
Szilard Nemeth 580a6a75a3 YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja 2021-01-08 12:38:43 +01:00
srinivasst 1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM (#2564)
Contributed by Srinivas S T
2021-01-08 10:52:52 +05:30
Neil fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore (#2568) 2020-12-30 22:20:12 -08:00
Eric Badger 4c5d88e230 YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api
for nodes. Contributed by Jim Brennan.
2020-12-21 23:10:31 +00:00
zhuqi bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. (#2563)
Qi Zhu via Zhankun Tang
2020-12-21 10:04:16 +08:00
Akira Ajisaka df7f1e5199
YARN-10499. TestRouterWebServiceREST fails (#2490). Contributed by Akira Ajisaka 2020-12-16 17:26:15 +01:00
Szilard Nemeth df35c7f519 YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak 2020-12-15 17:30:14 +01:00
adamantal 3234e5eaf3 YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori 2020-12-12 09:42:22 +01:00
adamantal 0a45bd034e YARN-10520. Deprecated the residual nested class for the LCEResourceHandler (#2527). Contributed by Wanqiang Ji 2020-12-09 12:55:24 +01:00
zhuqi d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler (#2494)
Contributed by Qi Zhu.
2020-12-09 19:48:39 +08:00
Eric Badger 7d3c8ef606 YARN-10495. make the rpath of container-executor configurable. Contributed by
angerszhu.
2020-12-08 00:53:24 +00:00
adamantal 9969745343 YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy 2020-12-03 09:55:06 +01:00
Eric Payne fa773a8326 YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth) 2020-12-01 22:06:47 +00:00
Ankit Kumar 08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair (#2485)
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-11-25 13:00:09 +09:00
Ahmed Hussein 569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions (#2461) 2020-11-24 11:09:30 -08:00
Szilard Nemeth 0d3155a687 YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak 2020-11-18 17:02:47 +01:00
Szilard Nemeth 5ff70a59c4 YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko 2020-11-18 16:51:42 +01:00
Peter Bacsko 567600fd80 YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:10:15 +01:00
Peter Bacsko cd0490e8c6 Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak."
This reverts commit b0ab222a6c.
2020-11-11 17:06:12 +01:00
Peter Bacsko b0ab222a6c YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak. 2020-11-11 17:03:16 +01:00
Eric E Payne 55339c2bdd YARN-10479. RMProxy should retry on SocketTimeout Exceptions. Contributed by Jim Brennan (Jim_Brennan) 2020-11-05 21:50:46 +00:00
Eric E Payne 0461a07c01 YARN-10475: Scale RM-NM heartbeat interval based on node utilization. Contributed by Jim Brennan (Jim_Brennan). 2020-11-02 16:55:06 +00:00
Surendra Singh Lilhore 7169ec4509
YARN-10442. RM should make sure node label file highly available. (#2390)
* YARN-10442. RM should make sure node label file highly available.

* YARN-10442. Corrected method name.
            Added license header for file TestNodeLabelFileReplication.java
            Added Test code.

* YARN-10442. Changed property name and log.

* YARN-10442. Changed default replication for FS store file.
2020-10-30 23:05:16 +05:30
Wangda Tan c47c9fd65d YARN-10458. Hive On Tez queries fails upon submission to dynamically created pools. (Peter Bacsko via wangda)
Change-Id: I518dc925187ce55e9d35a37ba20878c0f4e37e5c
2020-10-30 09:49:48 -07:00
Jim Brennan 8ee6bc2518 YARN-10471. Prevent logs for any container from becoming larger than a configurable size. Contributed by Eric Payne 2020-10-29 19:49:18 +00:00
Jonathan Hung bab5bf9743 YARN-10467. ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers. Contributed by Haibo Chen 2020-10-28 10:33:18 -07:00
Sunil G 7435604a91 YARN-10453. Add partition resource info to get-node-labels and label-mappings api responses. Contributed Akhil PB. 2020-10-22 12:52:58 +05:30
Szilard Nemeth 31477a224f YARN-10460. Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail. Contributed by Peter Bacsko 2020-10-19 13:17:51 +02:00
Ayush Saxena 1e3a6efcef
HADOOP-17288. Use shaded guava from thirdparty. (#2342). Contributed by Ayush Saxena. 2020-10-17 12:01:18 +05:30
Eric Badger 8b8c672780 YARN-10450. Add cpu and memory utilization per node and cluster-wide metrics.
Contributed by Jim Brennan.
2020-10-15 17:57:43 +00:00
Szilard Nemeth 42358169f6 YARN-10454: Add applicationName policy. Contributed by Peter Bacsko 2020-10-13 10:11:10 +02:00
Jim Brennan deb35a32ba YARN-10455. TestNMProxy.testNMProxyRPCRetry is not consistent. Contributed by Ahmed Hussein 2020-10-08 18:46:47 +00:00
Jim Brennan b361f29dda YARN-10451. RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined. Contributed by Eric Payne 2020-10-06 18:18:08 +00:00
Adam Antal a1f7e760df YARN-10393. MR job live lock caused by completed state container leak in heartbeat between node manager and RM. Contributed by zhenzhao wang and Jim Brennan 2020-10-05 10:09:14 +02:00
Fei Hui d0d10f7e8f
HADOOP-17276. Extend CallerContext to make it include many items (#2327) 2020-10-04 02:02:28 +09:00
Adam Antal bb8446e80c YARN-10447. TestLeafQueue: ActivitiesManager thread might interfere with ongoing stubbing. Contributed by Peter Bacsko 2020-10-01 11:42:04 +02:00
Szilard Nemeth a7a1f1541a YARN-10413. Change fs2cs to generate mapping rules in the new format. Contributed by Peter Bacsko 2020-09-29 20:18:26 +02:00
Shubham Gupta e3cd627069
YARN-10438. Handle null containerId in ClientRMService#getContainerReport() (#2313)
Co-authored-by: Shubham Gupta <gshubham@microsoft.com>
2020-09-25 12:20:39 +05:30
Szilard Nemeth eacbe07b56 YARN-9333. TestFairSchedulerPreemption.testRelaxLocalityPreemptionWithNoLessAMInRemainingNodes fails intermittently. Contributed by Peter Bacsko 2020-09-17 19:57:19 +02:00
Jim Brennan 90894ea641 YARN-10430. Log improvements in NodeStatusUpdaterImpl. Contributed by Bilwa S T. 2020-09-14 21:19:14 +00:00
Eric E Payne 9afec2ed17 YARN-10390: LeafQueue: retain user limits cache across assignContainers() calls. Contributed by Samir Khan (samkhan). 2020-09-11 13:29:26 +00:00
Szilard Nemeth 773ac799c6 YARN-10387. Implement logic which returns MappingRule objects based on mapping rules. Contributed by Peter Bacsko 2020-09-09 15:12:58 +02:00
Brahma Reddy Battula 43572fc7f8 YARN-10397. SchedulerRequest should be forwarded to scheduler if custom scheduler supports placement constraints. Contributed by Bilwa S T. 2020-09-09 17:08:13 +05:30
Peter Bacsko aba4a506d6 YARN-10424. Adapt existing AppName and UserGroupMapping unittests to ensure backwards compatibility. Contributed by Benjamin Teke. 2020-09-09 09:41:11 +02:00
Peter Bacsko c4fb4044b2 YARN-10415. Create a group matcher which checks ALL groups of the user. Contributed by Gergely Pollak. 2020-09-08 10:57:00 +02:00
Adam Antal 34fe74da0e YARN-10332. RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING state. Contributed by yehuanhuan 2020-09-07 11:44:09 +02:00
Peter Bacsko 360bbcd3bc YARN-10411. Create an allowCreate flag for MappingRuleAction. Contributed by Gergely Pollak. 2020-09-07 11:39:03 +02:00
Peter Bacsko 1841a5bb03 YARN-10376. Create a class that covers the functionality of UserGroupMappingPlacementRule and AppNameMappingPlacementRule using the new mapping rules. Contributed by Gergely Pollak. 2020-09-05 10:00:33 +02:00
Szilard Nemeth 696e4fe50e Typos in YarnScheduler#allocate method's doc comment. Contributed by Siddharth Ahuja 2020-09-04 09:59:54 +02:00
Peter Bacsko 5c15815773 YARN-10375. CS Mapping rule config parser should return MappingRule objects. Contributed by Gergely Pollak. 2020-09-03 18:54:22 +02:00
Peter Bacsko 288dab7563 YARN-10372. Create MappingRule class to represent each CS mapping rule. Contributed by Gergely Pollak 2020-09-03 15:02:48 +02:00
Masatake Iwasaki 5e12dc51f0
YARN-10419. Javadoc error in hadoop-yarn-server-common module. (#2271) 2020-09-03 16:17:52 +09:00
Peter Bacsko c7dab2b23e YARN-10374. Create Actions for CS mapping rules. Contributed by Gergely Pollak. 2020-09-01 17:38:49 +02:00
Adam Antal 73a0d149e2 YARN-10386. Addendum patch: Fixing ASF Warnings. Contributed by Peter Bacsko. 2020-09-01 14:39:24 +02:00
Szilard Nemeth 4ffe26f9b8 [UI1] Provide a way to hide Tools section in Web UIv1. Contributed by Andras Gyori 2020-08-31 16:00:39 +02:00
Szilard Nemeth 56a5c360a1 YARN-10373. Create Matchers for CS mapping rules. Contributed by Gergely Pollak 2020-08-29 21:34:55 +02:00
Szilard Nemeth f4f872b778 YARN-10371. Create variable context class for CS queue mapping rules. Contributed by Gergely Pollak 2020-08-29 17:31:48 +02:00
Szilard Nemeth a888d580d8 YARN-10409. Improve MockQueueHierarchyBuilder to detect queue ambiguity. Contributed by Gergely Pollak 2020-08-29 15:24:50 +02:00
Szilard Nemeth 44542863f4 YARN-10408. Extract MockQueueHierarchyBuilder to a separate class. Contributed by Gergely Pollak 2020-08-28 17:35:45 +02:00
Szilard Nemeth 6e3d7050a9 YARN-10386. Create new JSON schema for Placement Rules. Contributed by Peter Bacsko and Benjamin Teke 2020-08-28 17:09:31 +02:00
Adam Antal 82a7505646 YARN-10304. Create an endpoint for remote application log directory path query. Contributed by Andras Gyori 2020-08-25 13:29:12 +02:00
Prabhu Joseph 5c927eb550 YARN-10360. Support Multi Node Placement in SingleConstraintAppPlacementAllocator
Reviewed by Sunil G.
2020-08-24 15:00:02 +05:30
Zhengbo Li 9b9f7ea16a
YARN-10399 Refactor NodeQueueLoadMonitor class to make it extendable (#2228)
Refactor NodeQueueLoadMonitor class to make it extendable
2020-08-19 10:14:50 -07:00
Sunil G 82ec28f442 YARN-10396. Max applications calculation per queue disregards queue level settings in absolute mode. Contributed by Benjamin Teke. 2020-08-19 11:54:48 +05:30
Jim Brennan 4a7deae478 YARN-10391. --module-gpu functionality is broken in container-executor. Contributed by Eric Badger 2020-08-17 16:41:15 +00:00
hemanthboyina 86bbd38c8d YARN-10336. RM page should throw exception when command injected in RM REST API to get applications. Contributed by Bilwa S T. 2020-08-13 23:44:42 +05:30
Prabhu Joseph 6c2ce3d56b YARN-10389. Option to override RMWebServices with custom WebService class
Contributed by Tanu Ajmera. Reviewed by Bilwa ST and Sunil G.
2020-08-11 15:46:47 +05:30
Prabhu Joseph 5e0f879779 YARN-10364. Fix logic of queue capacity is absolute resource or percentage.
Contributed by Bilwa ST. Reviewed by Sunil G.
2020-08-08 22:33:58 +05:30
Jonathan Hung 64753addba YARN-10251. Show extended resources on legacy RM UI. Contributed by Eric Payne 2020-08-07 17:43:01 -07:00
bibinchundatt 81da221c75 YARN-10388. RMNode updatedCapability flag not set while RecommissionNodeTransition. Contributed by Pranjal Protim Borah 2020-08-07 08:36:52 +05:30
Prabhu Joseph c7e71a6c0b YARN-10361. Make custom DAO classes configurable into RMWebApp#JAXBContextResolver.
Contributed by Bilwa ST.
2020-08-06 13:19:52 +05:30
Eric Badger 5edd8b925e YARN-4575. ApplicationResourceUsageReport should return ALL reserved resource.
Contributed by Bibin Chundatt and Eric Payne.
2020-08-05 18:47:49 +00:00
Prabhu Joseph 0277856738 YARN-10377. Fix filter index to show apps while clicking on queue in RM UI
Contributed by Tarun Parimi.
2020-08-04 23:10:45 +05:30
Prabhu Joseph aa5afa72c5 YARN-10381. Add application attempt state in AppAttempts RM REST API
Contributed by Siddharth Ahuja. Reviewed by Bilwa ST.
2020-08-04 23:07:28 +05:30
Brahma Reddy Battula eac558380f YARN-10229. [Federation] Client should be able to submit application to RM directly using normal client conf. Contributed by Bilwa S T. 2020-08-03 12:54:36 +05:30
bibinchundatt 5323e83edf YARN-10359. Log container report only if list is not empty. Contributed by Bilwa S T. 2020-08-01 13:03:46 +05:30
Eric E Payne e0c9653166 YARN-1529: Add Localization overhead metrics to NM. Contributed by Jim_Brennan. 2020-07-30 15:30:22 +00:00
bibinchundatt 5d8600e80a YARN-10369. Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUG. Contributed by Jim Brennan. 2020-07-29 23:26:08 +05:30
Jonathan Hung 3eaf62726f YARN-10343. Legacy RM UI should include labeled metrics for allocated, total, and reserved resources. Contributed by Eric Payne 2020-07-28 13:43:19 -07:00