e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… ( #3741 )
...
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp ( #2591 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2022-01-28 15:51:21 +09:00
02f6bad1ff
Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )"
...
This reverts commit 1c01944f35
2022-01-28 00:36:25 +09:00
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 00:04:01 +09:00
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse ( #3766 )
2022-01-25 09:25:18 -08:00
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container ( #3779 )
2022-01-24 08:03:36 -08:00
dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui ( #3890 )
...
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects ) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases )
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7 )
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-20 21:44:10 +09:00
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c ( #3851 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-17 16:20:48 +09:00
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable ( #3859 )
2022-01-11 14:38:59 +08:00
3bf014d871
YARN-8859. Add audit logs for router service ( #3607 )
...
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-01-03 10:38:59 +05:30
4db27acf06
YARN-11053. AuxService should not use class name as default system classes ( #3816 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 11:08:04 +09:00
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch ( #3793 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
e2d6fd075d
YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).
2021-12-21 19:05:39 +00:00
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled ( #3802 )
2021-12-20 18:02:57 +08:00
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
07141426e0
YARN-11050 ( #3805 )
2021-12-18 11:52:03 +08:00
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class ( #3800 ). Contributed by Szilard Nemeth
2021-12-17 00:18:14 +01:00
aec9cdb467
YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok
2021-12-16 23:39:18 +01:00
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API ( #3799 ). Contributed by Szilard Nemeth
2021-12-16 15:53:08 +01:00
6ea1685abd
YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke
2021-12-16 00:01:09 +01:00
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster ( #3796 )
2021-12-15 16:47:51 +08:00
843f66f4dc
Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke
2021-12-14 22:00:46 +01:00
364d38f00d
YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke
2021-12-14 14:57:08 +01:00
a5bcf4c792
YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke
2021-12-13 21:57:46 +01:00
898055e204
YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke
2021-12-13 16:12:55 +01:00
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown ( #3768 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 14:03:32 +09:00
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
7a8c2e0097
YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori
2021-12-07 08:51:03 +01:00
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing ( #3728 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile ( #3722 )
2021-12-02 10:40:30 +08:00
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files ( #3735 )
...
Contributed by sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
c9d64bad37
YARN-11007. Correct words in YARN documents ( #3680 )
...
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-25 17:48:54 +09:00
08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method ( #3592 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 18:41:22 +09:00
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
b933f5f54b
YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang
2021-11-15 16:02:18 +01:00
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers ( #3603 )
2021-11-11 09:49:29 -08:00
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration ( #3570 ). Contributed by Szilard Nemeth
2021-10-28 15:19:32 +02:00
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue ( #3551 ) contributed by Szilard Nemeth
2021-10-27 19:03:45 +02:00
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler ( #3581 ) Contributed by Szilard Nemeth
2021-10-27 17:13:49 +02:00
9cfd8d0a83
YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth
2021-10-23 14:47:09 +02:00
32ecaed9c3
YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori
2021-10-22 17:32:33 +02:00
23772d946b
YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal
2021-10-22 16:33:03 +02:00
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
d286994009
YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne)
2021-10-20 22:18:36 +00:00
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth
2021-10-20 15:52:37 +02:00
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori
2021-10-20 12:56:41 +02:00
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule ( #3560 )
...
* YARN-10958. Initial commit
* Fix javadoc + behaviour
* Fix review comments
* fix checkstyle + blanks
* fix checkstyle + blanks
* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
616cea2e80
YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori
2021-10-19 13:06:45 +02:00
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth
2021-10-19 12:24:58 +02:00
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk ( #3552 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules ( #3541 )
2021-10-14 18:03:01 +09:00
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
8c271caa81
YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke
2021-10-12 17:35:06 +02:00
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori
2021-10-07 17:09:38 +02:00
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori
2021-10-01 04:20:36 +02:00
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint ( #3480 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics ( #3439 )
2021-09-28 09:51:18 -07:00
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
2021-09-21 16:08:24 +02:00
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth
2021-09-20 16:47:46 +02:00
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities ( #3454 )
2021-09-19 14:42:02 +02:00
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController ( #3450 )
2021-09-19 14:35:34 +02:00
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class ( #3420 )
2021-09-19 13:11:56 +02:00
f5c76c8e31
YARN-10936. Log typo corrected. ( #3449 )
...
Contributed by: Tibor Kovacs
2021-09-17 08:27:51 +02:00
43f0a34dd4
YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne.
2021-09-15 20:03:45 +00:00
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources ( #3418 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir ( #3355 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-08 17:04:32 +02:00
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
minni31
Akira Ajisaka
Wei-Chiu Chuang
Andrew Chung
dependabot[bot]
Gera Shegalov
luoyuan3471
Andras Gyori
Cheng Pan
Ashutosh Gupta
Eric Payne
Viraj Jasani
Szilard Nemeth
Szilard Nemeth
Sushmasree-28
GuoPhilipse
Jack
Adam Antal
Ahmed Hussein
lujiefsi
Szilard Nemeth
Benjamin Teke
Tamas Domok
Neil
Akshat Bordia
TiborKovacsCloudera
Eric Badger
Weihao Zheng