f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it
...
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.
Contributed by: Siddharth Ahuja
2021-03-17 10:55:05 +11:00
73394fabc7
YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu.
2021-03-16 16:09:12 +01:00
b80588b688
YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu.
2021-03-16 15:23:27 +01:00
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. ( #2760 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-16 13:50:29 +09:00
aa4c17b9d7
YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T
2021-03-15 19:09:40 +00:00
5f067cf0f3
[YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu.
2021-03-12 17:17:31 +00:00
5db4c0bf70
YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak
2021-03-12 14:10:16 +01:00
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
b2a565629d
YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy.
2021-03-09 20:26:07 +05:30
d5e035dbe1
YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu.
2021-03-09 14:33:14 +01:00
3851994cd6
Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu."
...
This reverts commit 369f75b7a7
2021-03-09 14:32:02 +01:00
369f75b7a7
YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu.
2021-03-09 14:28:23 +01:00
20416bc62d
YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth.
2021-03-09 11:34:37 +01:00
e472ee2aa5
YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth.
2021-03-09 11:28:24 +01:00
04cd3115ba
YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu.
2021-03-08 12:29:44 +01:00
c6b30a59dd
YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth.
2021-03-05 23:13:35 +01:00
9d088639bf
YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim
...
Brennan.
2021-03-05 19:46:40 +00:00
e82e7c597a
YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu.
2021-03-05 15:56:56 +01:00
e7cad3811f
YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu.
2021-03-05 13:50:45 +01:00
e19c00925f
YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu.
2021-03-05 13:18:06 +01:00
6699198b54
YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu.
2021-03-04 17:18:35 +01:00
a85aeee876
YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu.
2021-03-04 12:23:11 +01:00
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak ( #2719 ). Contributed by Max Xie
2021-03-04 14:52:58 +05:30
7e8040e6ad
YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori.
2021-03-03 17:44:30 +01:00
8af56de1fa
HADOOP-17560. Fix some spelling errors ( #2730 )
...
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-03-03 11:41:05 +09:00
3101c39b49
YARN-10656. Parsing error in CapacityScheduler.md ( #2725 )
...
Reviewed-by: Peter Bacsko <pbacsko@apache.org>
2021-02-27 12:46:36 +09:00
947b50489d
YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke.
2021-02-26 21:57:54 +01:00
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 13:42:33 -06:00
fcdb1e7fec
YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by
...
Qi Zhu
2021-02-26 19:03:59 +00:00
ac37ca0b67
YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen
2021-02-25 16:01:18 -08:00
c373da9f88
[YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne
2021-02-25 16:46:19 +00:00
a89c624002
YARN-10609. Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue). Contributed by Qi Zhu.
2021-02-24 20:41:35 +01:00
47420ae3ed
YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by
...
Qi Zhu.
2021-02-22 18:48:14 +00:00
b722c52f4e
YARN-10513. CS Flexible Auto Queue Creation RM UIv2 modifications. Contributed by Andras Gyori.
2021-02-22 12:55:45 +01:00
4891e68c2b
YARN-10501. Can't remove all node labels after add node label without
...
nodemanager port. Contributed by caozhiqiang.
2021-02-19 22:29:10 +00:00
d28b6f90c8
YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak.
2021-02-19 13:57:30 +01:00
4383726d19
YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak.
2021-02-19 12:01:31 +01:00
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )"
...
This reverts commit 4cf3531583
2021-02-19 11:09:10 +09:00
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )
...
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
9882a22365
YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko
2021-02-16 18:06:57 +01:00
3fcc7a55b6
[YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger
2021-02-16 16:45:35 +00:00
bad6038a48
YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori
2021-02-15 14:48:42 +01:00
852aac34f2
Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu
2021-02-15 14:20:50 +01:00
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. ( #2619 )
2021-02-11 13:55:04 -06:00
98ca6afd17
YARN-10618. RM UI2 Application page shows the AM preempted containers instead of the nonAM ones. Contributed by Benjamin Teke
2021-02-11 09:14:37 +01:00
cacc870389
YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko
2021-02-10 13:09:32 +01:00
a8bd516e39
YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko
2021-02-10 11:15:53 +01:00
fc5f241535
YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak
2021-02-09 18:16:42 +01:00
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread ( #2665 )
...
Amogh Rajesh Desai via Zhankun Tang
2021-02-09 11:05:14 +08:00
c22c77af43
[YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger.
2021-02-05 17:33:01 +00:00
5aa9866ec2
YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori
2021-02-05 17:43:01 +01:00
c19326c051
YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu
2021-02-05 17:34:45 +01:00
79a46599f7
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori
2021-02-05 17:09:08 +01:00
52217fce3d
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 ( #2546 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-02-05 16:14:10 +09:00
5e719bf588
YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu
2021-02-04 18:30:39 +00:00
c17414111b
YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak
2021-02-04 08:12:13 +01:00
6fc26ad539
YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu
2021-02-03 08:50:45 +05:30
9bf2ac07bb
YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori
2021-02-02 09:13:46 +01:00
7d88953768
YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori
2021-02-02 09:07:25 +01:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
fa15594ae6
YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke.
2021-01-28 20:58:51 +01:00
9628aa87bf
YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke
2021-01-27 18:14:06 +01:00
7c4ef42837
YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko
2021-01-27 15:49:26 +01:00
e2a7008d50
YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak
2021-01-26 18:31:39 +01:00
4f008153ef
YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda)
...
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
2021-01-26 08:24:29 -08:00
b7384a8d02
YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda)
...
Change-Id: I670e5525619b320745254609c48e7e1afb084835
2021-01-26 08:24:28 -08:00
f1766e5bb4
YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori
2021-01-26 16:22:35 +01:00
8620984b8d
YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko
2021-01-21 20:58:57 +01:00
1ccba3734d
YARN-10490. yarn top command not quitting completely with ctrl+c. Contributed by Agshin Kazimli
2021-01-21 20:30:19 +01:00
17fdac8de5
YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth.
2021-01-21 18:06:53 +01:00
06fef5ee43
YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth
2021-01-21 09:23:11 +01:00
8bc2dfbf36
YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal
2021-01-21 11:23:30 +05:30
cfe6e1f7da
YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori
2021-01-20 15:22:44 +01:00
4b5bc05a78
YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth.
2021-01-20 12:28:59 +05:30
a326f22606
YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko
2021-01-19 09:42:40 +01:00
6abdb148e4
YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak
2021-01-18 20:19:36 +01:00
acea862d59
YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak
2021-01-18 16:04:45 +01:00
3d46141583
YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori)
...
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
2021-01-15 14:20:08 -08:00
768e2f42ba
YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan.
2021-01-13 23:41:52 +00:00
d8fadec944
YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan)
2021-01-13 17:01:28 +00:00
836c630430
YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko
2021-01-13 14:24:12 +01:00
61f77b7674
YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal
2021-01-13 16:00:11 +05:30
85b1c017ee
YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko
2021-01-12 15:29:20 +01:00
9c6e093783
YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko
2021-01-12 15:21:27 +01:00
b0eec09097
YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori)
...
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
2021-01-11 17:46:09 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
890f2da624
YARN-10553. Refactor TestDistributedShell ( #2581 )
2021-01-09 08:39:21 +09:00
4176759909
YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko
2021-01-08 13:40:33 +01:00
580a6a75a3
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
2021-01-08 12:38:43 +01:00
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM ( #2564 )
...
Contributed by Srinivas S T
2021-01-08 10:52:52 +05:30
ae4945fb2c
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. ( #2592 )
2021-01-06 12:12:46 +09:00
5abeecd33f
YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. ( #2586 )
2021-01-05 13:38:34 +09:00
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore ( #2568 )
2020-12-30 22:20:12 -08:00
513f1995ad
YARN-10334. Close clients in TestDistributedShell ( #2571 )
2020-12-28 11:09:10 -08:00
4c5d88e230
YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api
...
for nodes. Contributed by Jim Brennan.
2020-12-21 23:10:31 +00:00
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. ( #2563 )
...
Qi Zhu via Zhankun Tang
2020-12-21 10:04:16 +08:00
7a88f45366
YARN-10536. Client in distributedShell swallows interrupt exceptions ( #2554 )
2020-12-17 15:13:28 -08:00
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails ( #2490 ). Contributed by Akira Ajisaka
2020-12-16 17:26:15 +01:00
df35c7f519
YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak
2020-12-15 17:30:14 +01:00
3234e5eaf3
YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori
2020-12-12 09:42:22 +01:00
Wilfred Spiegelenburg
Peter Bacsko
zhuqi
Eric Payne
Jim Brennan
Szilard Nemeth
Akira Ajisaka
Brahma Reddy Battula
Eric Badger
Neil
JJiaguodong
Ahmed Hussein
Jonathan Hung
Masatake Iwasaki
Amogh Rajesh Desai
bibinchundatt
belugabehr
Wangda Tan
Sunil G
He Xiaoqiao
srinivasst
adamantal