Szilard Nemeth
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
Szilard Nemeth
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
Andrew Chung
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
Szilard Nemeth
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
Szilard Nemeth
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
Viraj Jasani
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
Szilard Nemeth
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
Szilard Nemeth
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
Szilard Nemeth
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
Szilard Nemeth
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
Ashutosh Gupta
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing ( #3728 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
Andras Gyori
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
Viraj Jasani
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile ( #3722 )
2021-12-02 10:40:30 +08:00
Sushmasree-28
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files ( #3735 )
...
Contributed by sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
Viraj Jasani
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
Andrew Chung
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
Andrew Chung
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
Viraj Jasani
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
Szilard Nemeth
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
Szilard Nemeth
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00
Szilard Nemeth
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration ( #3570 ). Contributed by Szilard Nemeth
2021-10-28 15:19:32 +02:00
Szilard Nemeth
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue ( #3551 ) contributed by Szilard Nemeth
2021-10-27 19:03:45 +02:00
Szilard Nemeth
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler ( #3581 ) Contributed by Szilard Nemeth
2021-10-27 17:13:49 +02:00
Jack
9cfd8d0a83
YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth
2021-10-23 14:47:09 +02:00
9uapaw
32ecaed9c3
YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori
2021-10-22 17:32:33 +02:00
Adam Antal
23772d946b
YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal
2021-10-22 16:33:03 +02:00
Ahmed Hussein
d286994009
YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne)
2021-10-20 22:18:36 +00:00
Szilard Nemeth
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth
2021-10-20 15:52:37 +02:00
Andras Gyori
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori
2021-10-20 12:56:41 +02:00
Szilard Nemeth
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule ( #3560 )
...
* YARN-10958. Initial commit
* Fix javadoc + behaviour
* Fix review comments
* fix checkstyle + blanks
* fix checkstyle + blanks
* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
9uapaw
616cea2e80
YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori
2021-10-19 13:06:45 +02:00
Szilard Nemeth
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth
2021-10-19 12:24:58 +02:00
lujiefsi
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk ( #3552 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
Viraj Jasani
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules ( #3541 )
2021-10-14 18:03:01 +09:00
Viraj Jasani
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
Szilard Nemeth
8c271caa81
YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke
2021-10-12 17:35:06 +02:00
Benjamin Teke
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
9uapaw
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori
2021-10-07 17:09:38 +02:00
Tamas Domok
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
9uapaw
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori
2021-10-01 04:20:36 +02:00
Tamas Domok
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
Neil
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint ( #3480 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
Akshat Bordia
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics ( #3439 )
2021-09-28 09:51:18 -07:00
Tamas Domok
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9uapaw
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
2021-09-21 16:08:24 +02:00
Szilard Nemeth
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth
2021-09-20 16:47:46 +02:00
Adam Antal
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities ( #3454 )
2021-09-19 14:42:02 +02:00
Szilard Nemeth
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class ( #3420 )
2021-09-19 13:11:56 +02:00
Eric Badger
43f0a34dd4
YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne.
2021-09-15 20:03:45 +00:00
Benjamin Teke
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources ( #3418 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
Tamas Domok
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
Tamas Domok
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
Weihao Zheng
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
Jack
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
Benjamin Teke
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
9uapaw
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
Benjamin Teke
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
Tamas Domok
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
Akshat Bordia
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
Akshat Bordia
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
Szilard Nemeth
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
Jack
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
Benjamin Teke
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
Swathi Chandrashekar
9512c774c0
YARN-10884: Handle empty owners to parse log files ( #3318 )
...
* YARN 10884 : Parse log files which has empty owner
* Removing the whitespace
* Added a test case for null user
* Fixed indentations
* Fixed the indentation for test cases
Co-authored-by: Swathi C <cswathi@microsoft.com>
2021-09-07 21:37:51 +05:30
Tamas Domok
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. ( #3314 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-27 23:09:54 +02:00
Szilard Nemeth
e06a5cb197
YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
2021-08-24 15:27:34 +02:00
srinivasst
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node ( #3287 )
...
* Account for scheduled AM containers before deactivating node
* Move AM container check to separate method.
* Fix UTs
* Fix UTs
* Remove unnecessary import
* Add timeout for UT
2021-08-17 14:18:55 +05:30
Tamas Domok
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-04 15:27:46 +02:00
Tamas Domok
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests ( #3248 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-03 16:19:04 +02:00
Viraj Jasani
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 ( #3241 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
zhuqi-lucas
efb3fa2bf5
YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang.
2021-08-02 18:23:33 +08:00
Benjamin Teke
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues ( #3225 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
minni31
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. ( #3194 )
...
YARN-10841. Fix token reset synchronization for UAM response token. Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
Szilard Nemeth
1b9efe58c9
YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
2021-07-28 16:50:14 +02:00
Szilard Nemeth
8d0297c213
YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
2021-07-28 14:49:10 +02:00
Szilard Nemeth
fa0289b022
YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
2021-07-27 16:47:41 +02:00
Benjamin Teke
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. ( #3203 )
...
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
Akshat Bordia
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor ( #3135 )
...
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135 )
2021-07-23 17:07:51 +05:30
zhuqi-lucas
2da9b95d4d
YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori.
2021-07-22 20:30:43 +08:00
zhuqi-lucas
0441efe1fc
YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger.
2021-07-21 15:31:44 +08:00
Viraj Jasani
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 ( #3204 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-16 12:17:17 +09:00
Jim Brennan
632f64cadb
YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne.
2021-07-15 14:23:31 +00:00
Artem Smotrakov
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
Viraj Jasani
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
Jim Brennan
dc6f456e95
YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne.
2021-06-28 14:52:19 +00:00
Jim Brennan
7c7d02edbd
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
2021-06-25 20:32:08 +00:00
Peter Bacsko
0934e783cf
YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori.
2021-06-24 20:15:10 +02:00
Szilard Nemeth
6562391737
YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
2021-06-16 18:55:09 +02:00
Szilard Nemeth
428478bbe2
YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori
2021-06-16 18:26:58 +02:00
Szilard Nemeth
e31d06032b
YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke
2021-06-14 22:33:04 +02:00
Szilard Nemeth
7003997e36
YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
2021-06-12 14:49:52 +02:00
Viraj Jasani
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project ( #3075 )
2021-06-09 15:15:47 +09:00
Viraj Jasani
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils ( #3081 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-06-08 19:37:23 -07:00
Prabhu Josephraj
9445abb500
YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar
2021-06-08 20:37:40 +05:30
zhuqi-lucas
ec16b1d3b9
YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke.
2021-06-08 21:03:43 +08:00
Hideyuki Furue
9c7b8cf54e
Fix container-executor
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-06 07:05:29 +09:00
Szilard Nemeth
200eec8f2e
YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko
2021-06-03 13:27:36 +02:00
Szilard Nemeth
2707f69251
YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak
2021-06-01 16:01:39 +02:00
Gergely Pollak
e9339aa376
YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth
2021-06-01 15:57:22 +02:00
Szilard Nemeth
b86a6eb871
YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori
2021-05-25 18:27:53 +02:00