Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
Weihao Zheng
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
Jack
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
Benjamin Teke
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
Benjamin Teke
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
Tamas Domok
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
Akshat Bordia
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
Akshat Bordia
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
Szilard Nemeth
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
Jack
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
Benjamin Teke
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files ( #3318 )
...
* YARN 10884 : Parse log files which has empty owner
* Removing the whitespace
* Added a test case for null user
* Fixed indentations
* Fixed the indentation for test cases
Co-authored-by: Swathi C <cswathi@microsoft.com>
2021-09-07 21:37:51 +05:30
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. ( #3314 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-27 23:09:54 +02:00
Szilard Nemeth
e06a5cb197
YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
2021-08-24 15:27:34 +02:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node ( #3287 )
...
* Account for scheduled AM containers before deactivating node
* Move AM container check to separate method.
* Fix UTs
* Fix UTs
* Remove unnecessary import
* Add timeout for UT
2021-08-17 14:18:55 +05:30
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests ( #3248 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 ( #3241 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5
YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang.
2021-08-02 18:23:33 +08:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues ( #3225 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. ( #3194 )
...
YARN-10841. Fix token reset synchronization for UAM response token. Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
Szilard Nemeth
1b9efe58c9
YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213
YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
2021-07-28 14:49:10 +02:00
Szilard Nemeth
fa0289b022
YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
2021-07-27 16:47:41 +02:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. ( #3203 )
...
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor ( #3135 )
...
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135 )
2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d
YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori.
2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc
YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger.
2021-07-21 15:31:44 +08:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 ( #3204 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-16 12:17:17 +09:00
Jim Brennan
632f64cadb
YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne.
2021-07-15 14:23:31 +00:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
Jim Brennan
dc6f456e95
YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne.
2021-06-28 14:52:19 +00:00
Jim Brennan
7c7d02edbd
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
2021-06-25 20:32:08 +00:00
Peter Bacsko
0934e783cf
YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori.
2021-06-24 20:15:10 +02:00
Szilard Nemeth
6562391737
YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
2021-06-16 18:55:09 +02:00
Szilard Nemeth
428478bbe2
YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori
2021-06-16 18:26:58 +02:00
Szilard Nemeth
e31d06032b
YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke
2021-06-14 22:33:04 +02:00
Szilard Nemeth
7003997e36
YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
2021-06-12 14:49:52 +02:00
Viraj Jasani
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project ( #3075 )
2021-06-09 15:15:47 +09:00
Viraj Jasani
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils ( #3081 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-06-08 19:37:23 -07:00
Prabhu Josephraj
9445abb500
YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar
2021-06-08 20:37:40 +05:30
zhuqi-lucas
ec16b1d3b9
YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke.
2021-06-08 21:03:43 +08:00
Hideyuki Furue
9c7b8cf54e
Fix container-executor
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-06 07:05:29 +09:00
Szilard Nemeth
200eec8f2e
YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko
2021-06-03 13:27:36 +02:00
Szilard Nemeth
2707f69251
YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak
2021-06-01 16:01:39 +02:00
Gergely Pollak
e9339aa376
YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth
2021-06-01 15:57:22 +02:00
Szilard Nemeth
b86a6eb871
YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori
2021-05-25 18:27:53 +02:00
Szilard Nemeth
2541efa496
YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori
2021-05-25 13:55:59 +02:00
Viraj Jasani
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project ( #3033 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 18:10:20 +09:00
zhuqi-lucas
59172ada90
YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju.
2021-05-24 23:12:07 +08:00
Szilard Nemeth
1e44bdb84c
YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke
2021-05-22 14:55:01 +02:00
Peter Bacsko
eb72628e15
YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B.
2021-05-17 21:07:29 +02:00
Peter Bacsko
8891e5c028
YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju.
2021-05-17 13:30:12 +02:00
lujiefsi
d92a25b790
YARN-10555. Missing access check before getAppAttempts ( #2608 )
...
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-17 13:44:38 +09:00
zhuqi
e7f0e8073b
YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu.
2021-05-14 13:36:07 +08:00
zhuqi
d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. ( #2911 )
...
Contributed by Qi Zhu.
2021-05-14 13:12:28 +08:00
Peter Bacsko
626be24c3e
YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori.
2021-05-12 14:54:47 +02:00
Akira Ajisaka
7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. ( #2960 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-06 23:37:45 +09:00
Wei-Chiu Chuang
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). ( #2945 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-02 11:10:32 +09:00
Eric Badger
803ac4b1a0
YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu
2021-04-29 17:42:13 +00:00
Peter Bacsko
9166bfeb74
YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu.
2021-04-24 10:10:10 +02:00
Eric Badger
6857a05d6a
YARN-10479. Can't remove all node labels after add node label without
...
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
2021-04-23 22:02:04 +00:00
Jim Brennan
1cbe35946b
YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu.
2021-04-23 21:35:09 +00:00
Peter Bacsko
14a84c47b0
YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja.
2021-04-23 17:38:47 +02:00
Szilard Nemeth
3b6b21e459
YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak
2021-04-23 16:17:29 +02:00
Szilard Nemeth
919daec36b
YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak
2021-04-23 16:12:12 +02:00
Szilard Nemeth
f76a2a7606
YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko
2021-04-23 16:07:58 +02:00
Eric Badger
6cb90005a7
YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu
2021-04-20 17:34:49 +00:00
Viraj Jasani
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation ( #2922 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-17 20:47:44 +09:00
Eric Badger
213d3deb26
YARN-10503. Support queue capacity in terms of absolute resources with custom
...
resourceType. Contributed by Qi Zhu.
2021-04-09 00:34:15 +00:00
Peter Bacsko
ca9aa91d10
YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori.
2021-04-08 12:42:48 +02:00
Szilard Nemeth
9cd69c20c4
YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori
2021-04-07 11:52:21 +02:00
Eric Badger
26b8f678b2
YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor.
...
Contributed by Jim Brennan.
2021-04-06 01:16:14 +00:00
Peter Bacsko
158758c5bf
YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu.
2021-04-01 16:09:52 +02:00
Peter Bacsko
9f1655baf2
YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu.
2021-04-01 11:39:40 +02:00
Peter Bacsko
a0deda1a77
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu.
2021-04-01 09:21:15 +02:00
Szilard Nemeth
6fd0c661b6
YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak
2021-03-31 16:14:21 +02:00
Peter Bacsko
ff6ec20d84
YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu.
2021-03-31 10:55:14 +02:00
Akira Ajisaka
8ad77a26c9
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai.
2021-03-30 17:46:51 +09:00
Eric Badger
19e418c10d
YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu.
2021-03-25 22:33:58 +00:00
Peter Bacsko
ceb75e1e2a
YARN-10674. fs2cs should generate auto-created queue deletion properties. Contributed by Qi Zhu.
2021-03-24 08:15:06 +01:00
Jim Brennan
174f3a96b1
YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T.
2021-03-23 18:21:45 +00:00
Jack Jiang
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof ( #2804 )
2021-03-23 08:46:11 -07:00
Cyrus Jackson
cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications ( #2674 ). Contributed by Cyrus Jackson
2021-03-19 15:49:05 +05:30
Eric Badger
3d4ea4b336
YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu.
2021-03-18 19:36:50 +00:00
Eric Badger
181b9ddc51
YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu.
2021-03-18 17:45:09 +00:00
Peter Bacsko
ce6bfd5718
YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu.
2021-03-18 13:40:16 +01:00
Peter Bacsko
38495af325
YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu.
2021-03-18 12:46:11 +01:00
Szilard Nemeth
a5745711dd
YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak
2021-03-18 12:43:01 +01:00
Peter Bacsko
d7eeca4d0c
YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu.
2021-03-18 11:49:16 +01:00
Eric Badger
49f89f1d3d
YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu.
2021-03-17 18:11:37 +00:00
Peter Bacsko
3e58d5611d
YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu.
2021-03-17 13:38:20 +01:00
Wilfred Spiegelenburg
f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it
...
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.
Contributed by: Siddharth Ahuja
2021-03-17 10:55:05 +11:00
Peter Bacsko
73394fabc7
YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu.
2021-03-16 16:09:12 +01:00
Peter Bacsko
b80588b688
YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu.
2021-03-16 15:23:27 +01:00
zhuqi
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. ( #2760 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-16 13:50:29 +09:00
Eric Payne
aa4c17b9d7
YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T
2021-03-15 19:09:40 +00:00
Jim Brennan
5f067cf0f3
[YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu.
2021-03-12 17:17:31 +00:00
Szilard Nemeth
5db4c0bf70
YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak
2021-03-12 14:10:16 +01:00
Akira Ajisaka
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
Brahma Reddy Battula
b2a565629d
YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy.
2021-03-09 20:26:07 +05:30
Peter Bacsko
d5e035dbe1
YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu.
2021-03-09 14:33:14 +01:00
Peter Bacsko
3851994cd6
Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu."
...
This reverts commit 369f75b7a7
.
2021-03-09 14:32:02 +01:00
Peter Bacsko
369f75b7a7
YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu.
2021-03-09 14:28:23 +01:00
Peter Bacsko
20416bc62d
YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth.
2021-03-09 11:34:37 +01:00
Peter Bacsko
e472ee2aa5
YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth.
2021-03-09 11:28:24 +01:00
Peter Bacsko
04cd3115ba
YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu.
2021-03-08 12:29:44 +01:00
Peter Bacsko
c6b30a59dd
YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth.
2021-03-05 23:13:35 +01:00
Eric Badger
9d088639bf
YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim
...
Brennan.
2021-03-05 19:46:40 +00:00
Peter Bacsko
e82e7c597a
YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu.
2021-03-05 15:56:56 +01:00
Peter Bacsko
e19c00925f
YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu.
2021-03-05 13:18:06 +01:00
Peter Bacsko
6699198b54
YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu.
2021-03-04 17:18:35 +01:00
Peter Bacsko
a85aeee876
YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu.
2021-03-04 12:23:11 +01:00
Neil
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak ( #2719 ). Contributed by Max Xie
2021-03-04 14:52:58 +05:30
Peter Bacsko
7e8040e6ad
YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori.
2021-03-03 17:44:30 +01:00
JJiaguodong
8af56de1fa
HADOOP-17560. Fix some spelling errors ( #2730 )
...
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-03-03 11:41:05 +09:00
Peter Bacsko
947b50489d
YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke.
2021-02-26 21:57:54 +01:00
Ahmed Hussein
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 13:42:33 -06:00
Jonathan Hung
ac37ca0b67
YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen
2021-02-25 16:01:18 -08:00
Jim Brennan
c373da9f88
[YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne
2021-02-25 16:46:19 +00:00
Peter Bacsko
d28b6f90c8
YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak.
2021-02-19 13:57:30 +01:00
Peter Bacsko
4383726d19
YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak.
2021-02-19 12:01:31 +01:00
Akira Ajisaka
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )"
...
This reverts commit 4cf3531583
.
2021-02-19 11:09:10 +09:00
Akira Ajisaka
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )
...
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
Szilard Nemeth
9882a22365
YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko
2021-02-16 18:06:57 +01:00
Jim Brennan
3fcc7a55b6
[YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger
2021-02-16 16:45:35 +00:00
Szilard Nemeth
bad6038a48
YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori
2021-02-15 14:48:42 +01:00
Szilard Nemeth
852aac34f2
Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu
2021-02-15 14:20:50 +01:00
Masatake Iwasaki
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. ( #2619 )
2021-02-11 13:55:04 -06:00
Szilard Nemeth
cacc870389
YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko
2021-02-10 13:09:32 +01:00
Szilard Nemeth
a8bd516e39
YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko
2021-02-10 11:15:53 +01:00
Szilard Nemeth
fc5f241535
YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak
2021-02-09 18:16:42 +01:00
Amogh Rajesh Desai
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread ( #2665 )
...
Amogh Rajesh Desai via Zhankun Tang
2021-02-09 11:05:14 +08:00
Jim Brennan
c22c77af43
[YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger.
2021-02-05 17:33:01 +00:00
Szilard Nemeth
5aa9866ec2
YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori
2021-02-05 17:43:01 +01:00
Szilard Nemeth
c19326c051
YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu
2021-02-05 17:34:45 +01:00
Szilard Nemeth
79a46599f7
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori
2021-02-05 17:09:08 +01:00
Eric Badger
5e719bf588
YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu
2021-02-04 18:30:39 +00:00
Szilard Nemeth
c17414111b
YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak
2021-02-04 08:12:13 +01:00
bibinchundatt
6fc26ad539
YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu
2021-02-03 08:50:45 +05:30
Szilard Nemeth
9bf2ac07bb
YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori
2021-02-02 09:13:46 +01:00
Szilard Nemeth
7d88953768
YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori
2021-02-02 09:07:25 +01:00
belugabehr
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
Peter Bacsko
fa15594ae6
YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke.
2021-01-28 20:58:51 +01:00
Szilard Nemeth
9628aa87bf
YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke
2021-01-27 18:14:06 +01:00
Szilard Nemeth
7c4ef42837
YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko
2021-01-27 15:49:26 +01:00
Szilard Nemeth
e2a7008d50
YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak
2021-01-26 18:31:39 +01:00