ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… ( #3195 )
2022-02-01 09:37:40 -08:00
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… ( #3632 )
2022-02-01 09:35:59 -08:00
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… ( #3741 )
...
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp ( #2591 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2022-01-28 15:51:21 +09:00
02f6bad1ff
Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )"
...
This reverts commit 1c01944f35
2022-01-28 00:36:25 +09:00
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 00:04:01 +09:00
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse ( #3766 )
2022-01-25 09:25:18 -08:00
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container ( #3779 )
2022-01-24 08:03:36 -08:00
dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui ( #3890 )
...
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects ) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases )
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7 )
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-20 21:44:10 +09:00
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c ( #3851 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-17 16:20:48 +09:00
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable ( #3859 )
2022-01-11 14:38:59 +08:00
3bf014d871
YARN-8859. Add audit logs for router service ( #3607 )
...
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-01-03 10:38:59 +05:30
4db27acf06
YARN-11053. AuxService should not use class name as default system classes ( #3816 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 11:08:04 +09:00
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch ( #3793 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
e2d6fd075d
YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).
2021-12-21 19:05:39 +00:00
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled ( #3802 )
2021-12-20 18:02:57 +08:00
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
07141426e0
YARN-11050 ( #3805 )
2021-12-18 11:52:03 +08:00
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class ( #3800 ). Contributed by Szilard Nemeth
2021-12-17 00:18:14 +01:00
aec9cdb467
YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok
2021-12-16 23:39:18 +01:00
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API ( #3799 ). Contributed by Szilard Nemeth
2021-12-16 15:53:08 +01:00
6ea1685abd
YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke
2021-12-16 00:01:09 +01:00
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster ( #3796 )
2021-12-15 16:47:51 +08:00
843f66f4dc
Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke
2021-12-14 22:00:46 +01:00
364d38f00d
YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke
2021-12-14 14:57:08 +01:00
a5bcf4c792
YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke
2021-12-13 21:57:46 +01:00
898055e204
YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke
2021-12-13 16:12:55 +01:00
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown ( #3768 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 14:03:32 +09:00
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
7a8c2e0097
YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori
2021-12-07 08:51:03 +01:00
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing ( #3728 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile ( #3722 )
2021-12-02 10:40:30 +08:00
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files ( #3735 )
...
Contributed by sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
c9d64bad37
YARN-11007. Correct words in YARN documents ( #3680 )
...
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-25 17:48:54 +09:00
08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method ( #3592 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 18:41:22 +09:00
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
b933f5f54b
YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang
2021-11-15 16:02:18 +01:00
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers ( #3603 )
2021-11-11 09:49:29 -08:00
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration ( #3570 ). Contributed by Szilard Nemeth
2021-10-28 15:19:32 +02:00
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue ( #3551 ) contributed by Szilard Nemeth
2021-10-27 19:03:45 +02:00
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler ( #3581 ) Contributed by Szilard Nemeth
2021-10-27 17:13:49 +02:00
9cfd8d0a83
YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth
2021-10-23 14:47:09 +02:00
32ecaed9c3
YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori
2021-10-22 17:32:33 +02:00
23772d946b
YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal
2021-10-22 16:33:03 +02:00
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
d286994009
YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne)
2021-10-20 22:18:36 +00:00
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth
2021-10-20 15:52:37 +02:00
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori
2021-10-20 12:56:41 +02:00
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule ( #3560 )
...
* YARN-10958. Initial commit
* Fix javadoc + behaviour
* Fix review comments
* fix checkstyle + blanks
* fix checkstyle + blanks
* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
616cea2e80
YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori
2021-10-19 13:06:45 +02:00
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth
2021-10-19 12:24:58 +02:00
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk ( #3552 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules ( #3541 )
2021-10-14 18:03:01 +09:00
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
8c271caa81
YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke
2021-10-12 17:35:06 +02:00
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori
2021-10-07 17:09:38 +02:00
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori
2021-10-01 04:20:36 +02:00
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint ( #3480 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics ( #3439 )
2021-09-28 09:51:18 -07:00
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
2021-09-21 16:08:24 +02:00
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth
2021-09-20 16:47:46 +02:00
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities ( #3454 )
2021-09-19 14:42:02 +02:00
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController ( #3450 )
2021-09-19 14:35:34 +02:00
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class ( #3420 )
2021-09-19 13:11:56 +02:00
f5c76c8e31
YARN-10936. Log typo corrected. ( #3449 )
...
Contributed by: Tibor Kovacs
2021-09-17 08:27:51 +02:00
43f0a34dd4
YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne.
2021-09-15 20:03:45 +00:00
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources ( #3418 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir ( #3355 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-08 17:04:32 +02:00
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
3024a47026
Add documentation for YARN-10623 auto refresh queue conf in CS ( #3279 )
2021-09-08 16:03:15 +02:00
d9cb69853b
YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler
2021-09-08 15:43:57 +02:00
9c8fe1e512
YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke
2021-09-08 15:17:27 +02:00
9512c774c0
YARN-10884: Handle empty owners to parse log files ( #3318 )
...
* YARN 10884 : Parse log files which has empty owner
* Removing the whitespace
* Added a test case for null user
* Fixed indentations
* Fixed the indentation for test cases
Co-authored-by: Swathi C <cswathi@microsoft.com>
2021-09-07 21:37:51 +05:30
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. ( #3314 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-27 23:09:54 +02:00
e06a5cb197
YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
2021-08-24 15:27:34 +02:00
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node ( #3287 )
...
* Account for scheduled AM containers before deactivating node
* Move AM container check to separate method.
* Fix UTs
* Fix UTs
* Remove unnecessary import
* Add timeout for UT
2021-08-17 14:18:55 +05:30
44bab51be4
YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth
2021-08-04 15:43:57 +02:00
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-04 15:27:46 +02:00
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests ( #3248 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-03 16:19:04 +02:00
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 ( #3241 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
efb3fa2bf5
YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang.
2021-08-02 18:23:33 +08:00
8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. ( #3236 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 19:15:27 +02:00
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues ( #3225 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. ( #3194 )
...
YARN-10841. Fix token reset synchronization for UAM response token. Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
f2b6c03fc1
YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph
2021-07-28 17:02:15 +02:00
1b9efe58c9
YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
2021-07-28 16:50:14 +02:00
8d0297c213
YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
2021-07-28 14:49:10 +02:00
b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent ( #3212 )
...
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-27 19:35:20 -07:00
fa0289b022
YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
2021-07-27 16:47:41 +02:00
b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 ( #2891 )
...
Bumps [http-proxy](https://github.com/http-party/node-http-proxy ) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases )
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md )
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-26 02:16:21 +09:00
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. ( #3203 )
...
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor ( #3135 )
...
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135 )
2021-07-23 17:07:51 +05:30
2da9b95d4d
YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori.
2021-07-22 20:30:43 +08:00
0441efe1fc
YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger.
2021-07-21 15:31:44 +08:00
e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution ( #3214 )
...
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-21 10:06:46 +08:00
17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. ( #3213 )
2021-07-20 11:52:52 +08:00
0ac443b1f8
YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan.
2021-07-19 10:11:20 +08:00
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 ( #3204 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-16 12:17:17 +09:00
632f64cadb
YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne.
2021-07-15 14:23:31 +00:00
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
dc6f456e95
YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne.
2021-06-28 14:52:19 +00:00
6b4987f09c
YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar.
2021-06-27 16:28:32 +05:30
7c7d02edbd
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
2021-06-25 20:32:08 +00:00
0934e783cf
YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori.
2021-06-24 20:15:10 +02:00
11d144d228
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. ( #3128 )
2021-06-23 19:33:03 +09:00
6562391737
YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
2021-06-16 18:55:09 +02:00
428478bbe2
YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori
2021-06-16 18:26:58 +02:00
9a6a11c452
YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy.
2021-06-15 18:58:42 +00:00
e31d06032b
YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke
2021-06-14 22:33:04 +02:00
f0bdc422aa
YARN-10816. Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun Parimi
2021-06-12 15:18:41 +02:00
7003997e36
YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
2021-06-12 14:49:52 +02:00
4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom ( #3087 )
2021-06-11 12:15:52 +09:00
9a17f97133
YARN-10803. [JDK 11] TestRMFailoverProxyProvider and TestNoHaRMFailoverProxyProvider fails by ClassCastException. ( #3068 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-06-10 14:29:47 +09:00
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project ( #3075 )
2021-06-09 15:15:47 +09:00
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils ( #3081 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-06-08 19:37:23 -07:00
9445abb500
YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar
2021-06-08 20:37:40 +05:30
ec16b1d3b9
YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke.
2021-06-08 21:03:43 +08:00
9c7b8cf54e
Fix container-executor
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-06 07:05:29 +09:00
200eec8f2e
YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko
2021-06-03 13:27:36 +02:00
2707f69251
YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak
2021-06-01 16:01:39 +02:00
e9339aa376
YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth
2021-06-01 15:57:22 +02:00
986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom ( #3049 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-26 17:14:31 +09:00
c4581827a9
YARN-10691. DominantResourceCalculator isInvalidDivisor should consider only countable resource types. Contributed by Bilwa S T.
2021-05-25 18:07:28 +00:00
b86a6eb871
YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori
2021-05-25 18:27:53 +02:00
2541efa496
YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori
2021-05-25 13:55:59 +02:00
1cbcde04f3
YARN-10770. container-executor permission is wrong in SecureContainer.md. Contributed by Siddharth Ahuja.
2021-05-25 18:28:09 +09:00
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project ( #3033 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 18:10:20 +09:00
59172ada90
YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju.
2021-05-24 23:12:07 +08:00
6bb0892f6e
YARN-10753. Document the removal of FS default queue creation. Contributed by Benjamin Teke
2021-05-22 15:00:08 +02:00
1e44bdb84c
YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke
2021-05-22 14:55:01 +02:00
d146ab3af7
YARN-10766. [UI2] Bump moment-timezone to 0.5.33. Contributed by Andras Gyori
2021-05-21 18:07:32 +02:00
43e77caa71
YARN-10779. Add option to disable lowercase conversion in GetApplicationsRequestPBImpl and ApplicationSubmissionContextPBImpl. Contributed by Peter Bacsko
2021-05-21 18:04:24 +02:00
c8073811b2
YARN-9279. Remove the old hamlet package. ( #2986 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-21 12:40:50 +09:00
f8b0063201
Bump node-sass ( #3004 )
...
Bumps [node-sass](https://github.com/sass/node-sass ) from 4.13.0 to 4.14.1.
- [Release notes](https://github.com/sass/node-sass/releases )
- [Changelog](https://github.com/sass/node-sass/blob/master/CHANGELOG.md )
- [Commits](https://github.com/sass/node-sass/compare/v4.13.0...v4.14.1 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
2021-05-18 20:14:18 -07:00
eb72628e15
YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B.
2021-05-17 21:07:29 +02:00
8891e5c028
YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju.
2021-05-17 13:30:12 +02:00
d92a25b790
YARN-10555. Missing access check before getAppAttempts ( #2608 )
...
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-17 13:44:38 +09:00
344cacce23
YARN-10545. Improve the readability of diagnostics log in yarn-ui2 web page. #2540
...
Contributed by kyoty.
2021-05-14 23:57:54 +08:00
e7f0e8073b
YARN-10761: Add more event type to RM Dispatcher event metrics. Contributed by Qi Zhu.
2021-05-14 13:36:07 +08:00
d2b0675d61
YARN-10737: Fix typos in CapacityScheduler#schedule. ( #2911 )
...
Contributed by Qi Zhu.
2021-05-14 13:12:28 +08:00
881ab4ed39
Bump underscore ( #2997 )
...
Bumps [underscore](https://github.com/jashkenas/underscore ) from 1.9.1 to 1.13.1.
- [Release notes](https://github.com/jashkenas/underscore/releases )
- [Commits](https://github.com/jashkenas/underscore/compare/1.9.1...1.13.1 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
2021-05-13 05:54:07 -07:00
2f67a2646e
Bump acorn ( #3003 )
...
Bumps [acorn](https://github.com/acornjs/acorn ) from 5.7.3 to 5.7.4.
- [Release notes](https://github.com/acornjs/acorn/releases )
- [Commits](https://github.com/acornjs/acorn/compare/5.7.3...5.7.4 )
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Gergely Pollak <shuzirra@cloudera.com>
2021-05-13 05:51:43 -07:00
626be24c3e
YARN-10571. Refactor dynamic queue handling logic. Contributed by Andras Gyori.
2021-05-12 14:54:47 +02:00
91430889a5
HADOOP-17665 Ignore missing keystore configuration in reloading mechanism
2021-05-10 13:31:48 -07:00
7cb7b8adc1
YARN-10756. Remove additional junit 4.11 dependency from javadoc. ( #2960 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Bilwa S T <stbilwa@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-06 23:37:45 +09:00
f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). ( #2945 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-02 11:10:32 +09:00
803ac4b1a0
YARN-10707. Support custom resources in ResourceUtilization, and update Node GPU Utilization to use. Contributed by Qi Zhu
2021-04-29 17:42:13 +00:00
5b09ba75ec
YARN-10747. Bump YARN CSI protobuf version to 3.7.1 ( #2946 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Change-Id: Ib2a4bf775a5804641a91c6f9a486772c9d87c17e
2021-04-28 12:44:13 +09:00
a967ab06f2
YARN-10739. GenericEventHandler.printEventQueueDetails causes RM recovery to take too much time. Contributed by Qi Zhu.
2021-04-27 14:01:53 +02:00
9166bfeb74
YARN-10637. fs2cs: add queue autorefresh policy during conversion. Contributed by Qi Zhu.
2021-04-24 10:10:10 +02:00
538ce9c354
HADOOP-17650. Bump solr to unblock build failure with Maven 3.8.1 ( #2939 )
...
Reviewed-by: Siyao Meng <siyao@apache.org>
2021-04-25 23:48:39 -07:00
6857a05d6a
YARN-10479. Can't remove all node labels after add node label without
...
nodemanager port, broken by YARN-10647. Contributed by D M Murali Krishna Reddy
2021-04-23 22:02:04 +00:00
1cbe35946b
YARN-10743. Add a policy for not aggregating for containers which are killed because exceeding container log size limit. Contributed by Qi Zhu.
2021-04-23 21:35:09 +00:00
ebf6b14c67
YARN-10123. Error message around yarn app -stop/start can be improved. Contributed by Siddharth Ahuja
...
This is required to highlight that an implementation at framework level is needed for the stop/start functionality to work.
2021-04-23 22:49:26 +02:00
14a84c47b0
YARN-10705. Misleading DEBUG log for container assignment needs to be removed when the container is actually reserved, not assigned in FairScheduler. Contributed by Siddharth Ahuja.
2021-04-23 17:38:47 +02:00
3b6b21e459
YARN-10750. TestMetricsInvariantChecker.testManyRuns is broken since HADOOP-17524. Contributed by Gergely Pollak
2021-04-23 16:17:29 +02:00
919daec36b
YARN-10746. RmWebApp add default-node-label-expression to the queue info. Contributed by Gergely Pollak
2021-04-23 16:12:12 +02:00
f76a2a7606
YARN-10654. Dots '.' in CSMappingRule path variables should be replaced. Contributed by Peter Bacsko
2021-04-23 16:07:58 +02:00
6cb90005a7
YARN-10723. Change CS nodes page in UI to support custom resource. Contributed by Qi Zhu
2021-04-20 17:34:49 +00:00
9a261b0a16
HADOOP-17642. Remove appender EventCounter to avoid instantiation ( #2922 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-17 20:47:44 +09:00
ba3bc53f4e
YARN-10736. Fix GetApplicationsRequest JavaDoc. Contributed by Miklos Gergely. ( #2906 )
2021-04-14 17:16:30 +02:00
6a9a5ae5fd
Bump y18n ( #2843 )
...
Bumps [y18n](https://github.com/yargs/y18n ) from 3.2.1 to 3.2.2.
- [Release notes](https://github.com/yargs/y18n/releases )
- [Changelog](https://github.com/yargs/y18n/blob/master/CHANGELOG.md )
- [Commits](https://github.com/yargs/y18n/commits )
Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-04-11 01:30:49 +09:00
213d3deb26
YARN-10503. Support queue capacity in terms of absolute resources with custom
...
resourceType. Contributed by Qi Zhu.
2021-04-09 00:34:15 +00:00
ca9aa91d10
YARN-10564. Support Auto Queue Creation template configurations. Contributed by Andras Gyori.
2021-04-08 12:42:48 +02:00
9cd69c20c4
YARN-10714. Remove dangling dynamic queues on reinitialization. Contributed by Andras Gyori
2021-04-07 11:52:21 +02:00
26b8f678b2
YARN-10702. Add cluster metric for amount of CPU used by RM Event Processor.
...
Contributed by Jim Brennan.
2021-04-06 01:16:14 +00:00
158758c5bf
YARN-10726. Log the size of DelegationTokenRenewer event queue in case of too many pending events. Contributed by Qi Zhu.
2021-04-01 16:09:52 +02:00
9f1655baf2
YARN-9618. NodesListManager event improvement. Contributed by Qi Zhu.
2021-04-01 11:39:40 +02:00
a0deda1a77
YARN-10720. YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging. Contributed by Qi Zhu.
2021-04-01 09:21:15 +02:00
2c482fbacf
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:46:35 -07:00
22961a615d
Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2"
...
This reverts commit a2975d2153
2021-03-31 10:43:09 -07:00
a2975d2153
HADOOP-16524. Automatic keystore reloading for HttpServer2
...
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
2021-03-31 10:40:20 -07:00
5183aaeda2
Revert "Hadoop 16524 - resubmission following some unit test fixes ( #2693 )"
...
Revert to fix the summary message.
This reverts commit 9509bebf7f
2021-03-31 10:39:55 -07:00
9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes ( #2693 )
...
Signed-off-by: stack <stack@apache.org>
2021-03-31 10:07:42 -07:00
6fd0c661b6
YARN-10597. CSMappingPlacementRule should not create new instance of Groups. Contributed by Gergely Pollak
2021-03-31 16:14:21 +02:00
ff6ec20d84
YARN-10718. Fix CapacityScheduler#initScheduler log error. Contributed by Qi Zhu.
2021-03-31 10:55:14 +02:00
8ad77a26c9
YARN-10716. Fix typo in ContainerRuntime. Contributed by xishuhai.
2021-03-30 17:46:51 +09:00
413a4c3c05
YARN-10466.Fix NullPointerException in yarn-services Component.java. Contributed by D M Murali Krishna Reddy
2021-03-30 13:52:07 +05:30
6577bf1891
YARN-10439. addendum fix for shaded guva.
2021-03-30 13:48:40 +05:30
b61f52ec56
YARN-10441. Add support for hadoop.http.rmwebapp.scheduler.page.class. Contributed by D M Murali Krishna Reddy
2021-03-30 09:51:47 +05:30
d0dcfc405c
YARN-10439. Yarn Service AM listens on all IP's on the machine. Contributed by D M Murali Krishna Reddy
2021-03-30 09:46:12 +05:30
2d62dced4b
YARN-10437. Destroy yarn service if any YarnException occurs during submitApp.Contributed by D M Murali Krishna Reddy
2021-03-30 09:39:00 +05:30
19e418c10d
YARN-10713. ClusterMetrics should support custom resource capacity related metrics. Contributed by Qi Zhu.
2021-03-25 22:33:58 +00:00
ceb75e1e2a
YARN-10674. fs2cs should generate auto-created queue deletion properties. Contributed by Qi Zhu.
2021-03-24 08:15:06 +01:00
174f3a96b1
YARN-10697. Resources are displayed in bytes in UI for schedulers other than capacity. Contributed by Bilwa S T.
2021-03-23 18:21:45 +00:00
d8ec8ab965
HADOOP-17599. Remove NULL checks before instanceof ( #2804 )
2021-03-23 08:46:11 -07:00
cd44e917d0
YARN-10476. Queue metrics for Unmanaged applications ( #2674 ). Contributed by Cyrus Jackson
2021-03-19 15:49:05 +05:30
3d4ea4b336
YARN-10701. The yarn.resource-types should support multi types without trimmed. Contributed by Qi Zhu.
2021-03-18 19:36:50 +00:00
181b9ddc51
YARN-10703. Fix potential null pointer error of gpuNodeResourceUpdateHandler in NodeResourceMonitorImpl. Contributed by Qi Zhu.
2021-03-18 17:45:09 +00:00
ce6bfd5718
YARN-10641. Refactor the max app related update, and fix maxApllications update error when add new queues. Contributed by Qi Zhu.
2021-03-18 13:40:16 +01:00
38495af325
YARN-10692. Add Node GPU Utilization and apply to NodeMetrics. Contributed by Qi Zhu.
2021-03-18 12:46:11 +01:00
a5745711dd
YARN-10659. Improve CS MappingRule %secondary_group evaluation. Contributed by Gergely Pollak
2021-03-18 12:43:01 +01:00
d7eeca4d0c
YARN-10685. Fix typos in AbstractCSQueue. Contributed by Qi Zhu.
2021-03-18 11:49:16 +01:00
49f89f1d3d
YARN-10688. ClusterMetrics should support GPU capacity related metrics.. Contributed by Qi Zhu.
2021-03-17 18:11:37 +00:00
3e58d5611d
YARN-10497. Fix an issue in CapacityScheduler which fails to delete queues. Contributed by Wangda Tan and Qi Zhu.
2021-03-17 13:38:20 +01:00
f276f1af80
YARN-10652. Capacity Scheduler fails to handle user weights for a user that has a "." (dot) in it
...
This only fixes the user name resolution for weights in the queues. It
does not add generic support for user names with dots in all use cases
in the capacity scheduler.
Contributed by: Siddharth Ahuja
2021-03-17 10:55:05 +11:00
73394fabc7
YARN-10686. Fix TestCapacitySchedulerAutoQueueCreation#testAutoQueueCreationFailsForEmptyPathWithAQCAndWeightMode. Contributed by Qi Zhu.
2021-03-16 16:09:12 +01:00
b80588b688
YARN-10682. The scheduler monitor policies conf should trim values separated by comma. Contributed by Qi Zhu.
2021-03-16 15:23:27 +01:00
e9c98548e9
YARN-10689. Fix the finding bugs in extractFloatValueFromWeightConfig. ( #2760 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-03-16 13:50:29 +09:00
aa4c17b9d7
YARN-10588. Percentage of queue and cluster is zero in WebUI . Contributed by Bilwa S T
2021-03-15 19:09:40 +00:00
5f067cf0f3
[YARN-10687] Add option to disable/enable free disk space checking and percentage checking for full and not-full disks. Contributed by Qi Zhu.
2021-03-12 17:17:31 +00:00
5db4c0bf70
YARN-10412. Move CS placement rule related changes to a separate package. Contributed by Gergely Pollak
2021-03-12 14:10:16 +01:00
23b343aed1
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2753 )
...
Removed findbugs from the hadoop build images and added spotbugs instead.
Upgraded SpotBugs to 4.2.2 and spotbugs-maven-plugin to 4.2.0.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-03-11 10:56:07 +09:00
b2a565629d
YARN-10671.Fix Typo in TestSchedulingRequestContainerAllocation. Contributed by D M Murali Krishna Reddy.
2021-03-09 20:26:07 +05:30
d5e035dbe1
YARN-9615. Add dispatcher metrics to RM. Contributed by Jonathan Hung and Qi Zhu.
2021-03-09 14:33:14 +01:00
3851994cd6
Revert "YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu."
...
This reverts commit 369f75b7a7
2021-03-09 14:32:02 +01:00
369f75b7a7
YARN-9615. Add dispatcher metrics to RM. Contributed by Qi Zhu.
2021-03-09 14:28:23 +01:00
20416bc62d
YARN-10675. Consolidate YARN-10672 and YARN-10447. Contributed by Szilard Nemeth.
2021-03-09 11:34:37 +01:00
e472ee2aa5
YARN-10676. Improve code quality in TestTimelineAuthenticationFilterForV1. Contributed by Szilard Nemeth.
2021-03-09 11:28:24 +01:00
04cd3115ba
YARN-10658. CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. Contributed by Qi Zhu.
2021-03-08 12:29:44 +01:00
c6b30a59dd
YARN-10672. All testcases in TestReservations are flaky. Contributed By Szilard Nemeth.
2021-03-05 23:13:35 +01:00
9d088639bf
YARN-10664. Allow parameter expansion in NM_ADMIN_USER_ENV. Contributed by Jim
...
Brennan.
2021-03-05 19:46:40 +00:00
e82e7c597a
YARN-10640. Adjust the queue Configured capacity to Configured weight number for weight mode in UI. Contributed by Qi Zhu.
2021-03-05 15:56:56 +01:00
e7cad3811f
YARN-10642. Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995. Contributed by zhengchenyu.
2021-03-05 13:50:45 +01:00
e19c00925f
YARN-10639. Queueinfo related capacity, should adjusted to weight mode. Contributed by Qi Zhu.
2021-03-05 13:18:06 +01:00
6699198b54
YARN-10532. Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used. Contributed by Qi Zhu.
2021-03-04 17:18:35 +01:00
a85aeee876
YARN-10623. Capacity scheduler should support refresh queue automatically by a thread policy. Contributed by Qi Zhu.
2021-03-04 12:23:11 +01:00
d615e2d3bd
YARN-10649. Fix RMNodeImpl.updateExistContainers leak ( #2719 ). Contributed by Max Xie
2021-03-04 14:52:58 +05:30
7e8040e6ad
YARN-10655. Limit queue creation depth relative to its first static parent. Contributed by Andras Gyori.
2021-03-03 17:44:30 +01:00
8af56de1fa
HADOOP-17560. Fix some spelling errors ( #2730 )
...
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
2021-03-03 11:41:05 +09:00
3101c39b49
YARN-10656. Parsing error in CapacityScheduler.md ( #2725 )
...
Reviewed-by: Peter Bacsko <pbacsko@apache.org>
2021-02-27 12:46:36 +09:00
947b50489d
YARN-10627. Extend logging to give more information about weight mode. Contributed by Benjamin Teke.
2021-02-26 21:57:54 +01:00
e04bcb3a06
MAPREDUCE-7320. organize test directories for ClusterMapReduceTestCase ( #2722 ). Contributed by Ahmed Hussein
2021-02-26 13:42:33 -06:00
fcdb1e7fec
YARN-10653. Fixed the findbugs issues introduced by YARN-10647. Contributed by
...
Qi Zhu
2021-02-26 19:03:59 +00:00
ac37ca0b67
YARN-10651. CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource(). Contributed by Haibo Chen
2021-02-25 16:01:18 -08:00
c373da9f88
[YARN-10613] Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF. Contributed by Eric Payne
2021-02-25 16:46:19 +00:00
a89c624002
YARN-10609. Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue). Contributed by Qi Zhu.
2021-02-24 20:41:35 +01:00
47420ae3ed
YARN-10647. Fix TestRMNodeLabelsManager failed after YARN-10501. Contributed by
...
Qi Zhu.
2021-02-22 18:48:14 +00:00
b722c52f4e
YARN-10513. CS Flexible Auto Queue Creation RM UIv2 modifications. Contributed by Andras Gyori.
2021-02-22 12:55:45 +01:00
4891e68c2b
YARN-10501. Can't remove all node labels after add node label without
...
nodemanager port. Contributed by caozhiqiang.
2021-02-19 22:29:10 +00:00
d28b6f90c8
YARN-10636. CS Auto Queue creation should reject submissions with empty path parts. Contributed by Gergely Pollak.
2021-02-19 13:57:30 +01:00
4383726d19
YARN-10635. CSMapping rule can return paths with empty parts. Contributed by Gergely Pollak.
2021-02-19 12:01:31 +01:00
9a298d180d
Revert "HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )"
...
This reverts commit 4cf3531583
2021-02-19 11:09:10 +09:00
4cf3531583
HADOOP-16870. Use spotbugs-maven-plugin instead of findbugs-maven-plugin ( #2454 )
...
Use spotbugs instead of findbugs. Removed findbugs from the hadoop build images,
and added spotbugs in the images instead.
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
2021-02-17 10:38:20 +09:00
9882a22365
YARN-10625. FairScheduler: add global flag to disable AM-preemption. Contributed by Peter Bacsko
2021-02-16 18:06:57 +01:00
3fcc7a55b6
[YARN-10626] Log resource allocation in NM log at container start time. Contributed by Eric Badger
2021-02-16 16:45:35 +00:00
bad6038a48
YARN-10622. Fix preemption policy to exclude childless ParentQueues. Contributed by Andras Gyori
2021-02-15 14:48:42 +01:00
852aac34f2
Support max queues limit configuration in new auto created queue, consistent with old auto created. Contributed by Qi Zhu
2021-02-15 14:20:50 +01:00
f9a073c6c1
YARN-10500. TestDelegationTokenRenewer fails intermittently. ( #2619 )
2021-02-11 13:55:04 -06:00
98ca6afd17
YARN-10618. RM UI2 Application page shows the AM preempted containers instead of the nonAM ones. Contributed by Benjamin Teke
2021-02-11 09:14:37 +01:00
cacc870389
YARN-10593. Fix incorrect string comparison in GpuDiscoverer. Contributed by Peter Bacsko
2021-02-10 13:09:32 +01:00
a8bd516e39
YARN-10620. fs2cs: parentQueue for certain placement rules are not set during conversion. Contributed by Peter Bacsko
2021-02-10 11:15:53 +01:00
fc5f241535
YARN-10619. CS Mapping Rule %specified rule catches default submissions. Contributed by Gergely Pollak
2021-02-09 18:16:42 +01:00
0b05dd8802
YARN-9650. Set thread names for CapacityScheduler AsyncScheduleThread ( #2665 )
...
Amogh Rajesh Desai via Zhankun Tang
2021-02-09 11:05:14 +08:00
c22c77af43
[YARN-10607] User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH. Contributed by Eric Badger.
2021-02-05 17:33:01 +00:00
5aa9866ec2
YARN-10615. Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name. Contributed by Andras Gyori
2021-02-05 17:43:01 +01:00
c19326c051
YARN-10610. Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath. Contributed by Qi Zhu
2021-02-05 17:34:45 +01:00
79a46599f7
YARN-10428. Zombie applications in the YARN queue using FAIR + sizebasedweight. Contributed by Guang Yang, Andras Gyori
2021-02-05 17:09:08 +01:00
52217fce3d
HADOOP-17432. [JDK 16] KerberosUtil#getOidInstance is broken by JEP 396 ( #2546 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
2021-02-05 16:14:10 +09:00
5e719bf588
YARN-10611. Fix that shaded should be used for google guava imports in YARN-10352. Contributed by Qi Zhu
2021-02-04 18:30:39 +00:00
c17414111b
YARN-10612. Fix findbugs issue introduced in YARN-10585. Contributed by Gergely Pollak
2021-02-04 08:12:13 +01:00
6fc26ad539
YARN-10352 Skip schedule on not heartbeated nodes in Multi Node Placement. Contributed by Prabhu Joseph and Qi Zhu
2021-02-03 08:50:45 +05:30
9bf2ac07bb
YARN-10605. Add queue-mappings-override.enable property in FS2CS conversions. Contributed by Andras Gyori
2021-02-02 09:13:46 +01:00
7d88953768
YARN-10604. Support auto queue creation without mapping rules. Contributed by Andras Gyori
2021-02-02 09:07:25 +01:00
21a3fc3d2d
HADOOP-17482: Remove Commons Logger from FileSystem Class ( #2633 )
2021-02-01 09:40:01 -08:00
fa15594ae6
YARN-10600. Convert root queue in fs2cs weight mode conversion. Contributed by Benjamin Teke.
2021-01-28 20:58:51 +01:00
9628aa87bf
YARN-10598. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information. Contributed by Benjamin Teke
2021-01-27 18:14:06 +01:00
7c4ef42837
YARN-10599. fs2cs should generate new 'auto-queue-creation-v2.enabled' properties for all parents. Contributed by Peter Bacsko
2021-01-27 15:49:26 +01:00
e2a7008d50
YARN-10585. Create a class which can convert from legacy mapping rule format to the new JSON format. Contributed by Gergely Pollak
2021-01-26 18:31:39 +01:00
4f008153ef
YARN-10587. Fix AutoCreateLeafQueueCreation cap related caculation when in absolute mode. (Qi Zhu via wangda)
...
Change-Id: I8889c393a9f08b633c72a2e873c6f45ca12cc82b
2021-01-26 08:24:29 -08:00
b7384a8d02
YARN-10531. Be able to disable user limit factor for CapacityScheduler Leaf Queue. (Qi Zhu via wangda)
...
Change-Id: I670e5525619b320745254609c48e7e1afb084835
2021-01-26 08:24:28 -08:00
f1766e5bb4
YARN-10596. Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled. Contributed by Andras Gyori
2021-01-26 16:22:35 +01:00
8620984b8d
YARN-10515. Fix flaky test TestCapacitySchedulerAutoQueueCreation.testDynamicAutoQueueCreationWithTags. Contributed by Peter Bacsko
2021-01-21 20:58:57 +01:00
1ccba3734d
YARN-10490. yarn top command not quitting completely with ctrl+c. Contributed by Agshin Kazimli
2021-01-21 20:30:19 +01:00
17fdac8de5
YARN-10581. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues. Contributed by Szilard Nemeth.
2021-01-21 18:06:53 +01:00
06fef5ee43
YARN-10579. CS Flexible AQC: Modify RM /scheduler endpoint to include weight values for queues. Contributed by Szilard Nemeth
2021-01-21 09:23:11 +01:00
8bc2dfbf36
YARN-10519. Refactor QueueMetricsForCustomResources class to move to yarn-common package. Contributed by Minni Mittal
2021-01-21 11:23:30 +05:30
cfe6e1f7da
YARN-10578. Fix Auto Queue Creation parent handling. Contributed by Andras Gyori
2021-01-20 15:22:44 +01:00
4b5bc05a78
YARN-10512. CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS. Contributed by Szilard Nemeth.
2021-01-20 12:28:59 +05:30
a326f22606
YARN-10573. Enhance placement rule conversion in fs2cs in weight mode and enable it by default. Contributed by Peter Bacsko
2021-01-19 09:42:40 +01:00
6abdb148e4
YARN-10535. Make queue placement in CapacityScheduler compliant with auto-queue-placement. Contributed by Gergely Pollak
2021-01-18 20:19:36 +01:00
acea862d59
YARN-10574. Fix the FindBugs warning introduced in YARN-10506. Contributed by Gergely Pollak
2021-01-18 16:04:45 +01:00
3d46141583
YARN-10506. Update queue creation logic to use weight mode and allow the flexible static/dynamic creation. (Contributed by Qi Zhu, Andras Gyori)
...
Change-Id: I118862fd5e11ee6888275e2bcf667fedfa56c5d7
2021-01-15 14:20:08 -08:00
768e2f42ba
YARN-10562. Follow up changes for YARN-9833. Contributed by Jim Brennan.
2021-01-13 23:41:52 +00:00
d8fadec944
YARN-4589: Diagnostics for localization timeouts is lacking. Contributed by Chang Li (lichangleo) and Jim Brennan (Jim_Brennan)
2021-01-13 17:01:28 +00:00
836c630430
YARN-10525. Add weight mode conversion to fs2cs. Contributed by Peter Bacsko
2021-01-13 14:24:12 +01:00
61f77b7674
YARN-8529. Add timeout to RouterWebServiceUtil#invokeRMWebService. Contributed by Minni Mittal
2021-01-13 16:00:11 +05:30
85b1c017ee
YARN-10563. Fix dependency exclusion problem in poms. Contributed by Peter Bacsko
2021-01-12 15:29:20 +01:00
9c6e093783
YARN-10570. Remove experimental warning message from fs2cs. Contributed by Peter Bacsko
2021-01-12 15:21:27 +01:00
b0eec09097
YARN-10504. Implement weight mode in Capacity Scheduler. (Contributed by Wangda Tan, Benjamin Teke, zhuqi, Andras Gyori)
...
Change-Id: Ic49c730b0ab502ba86527fb662d25c4c8b1c2588
2021-01-11 17:46:09 -08:00
ec22850dbe
Make upstream aware of 3.2.2 release.
2021-01-09 18:06:08 +08:00
890f2da624
YARN-10553. Refactor TestDistributedShell ( #2581 )
2021-01-09 08:39:21 +09:00
4176759909
YARN-10507. Add the capability to fs2cs to write the converted placement rules inside capacity-scheduler.xml. Contributed by Peter Bacsko
2021-01-08 13:40:33 +01:00
580a6a75a3
YARN-10528. maxAMShare should only be accepted for leaf queues, not parent queues. Contributed by Siddharth Ahuja
2021-01-08 12:38:43 +01:00
1b1791075a
YARN-10538: Add RECOMMISSIONING nodes to the list of updated nodes returned to the AM ( #2564 )
...
Contributed by Srinivas S T
2021-01-08 10:52:52 +05:30
ae4945fb2c
YARN-10560. Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2. ( #2592 )
2021-01-06 12:12:46 +09:00
5abeecd33f
YARN-10558. Fix failure of TestDistributedShell#testDSShellWithOpportunisticContainers. ( #2586 )
2021-01-05 13:38:34 +09:00
fa4cf91b57
YARN-10541. capture the performance metrics of ZKRMStateStore ( #2568 )
2020-12-30 22:20:12 -08:00
513f1995ad
YARN-10334. Close clients in TestDistributedShell ( #2571 )
2020-12-28 11:09:10 -08:00
4c5d88e230
YARN-10540. Node page is broken in YARN UI1 and UI2 including RMWebService api
...
for nodes. Contributed by Jim Brennan.
2020-12-21 23:10:31 +00:00
bb528e3239
YARN-10463: For Federation, we should support getApplicationAttemptReport. ( #2563 )
...
Qi Zhu via Zhankun Tang
2020-12-21 10:04:16 +08:00
7a88f45366
YARN-10536. Client in distributedShell swallows interrupt exceptions ( #2554 )
2020-12-17 15:13:28 -08:00
df7f1e5199
YARN-10499. TestRouterWebServiceREST fails ( #2490 ). Contributed by Akira Ajisaka
2020-12-16 17:26:15 +01:00
df35c7f519
YARN-10526. RMAppManager CS Placement ignores parent path. Contributed by Gergely Pollak
2020-12-15 17:30:14 +01:00
3234e5eaf3
YARN-10031. Create a general purpose log request with additional query parameters. Contributed by Andras Gyori
2020-12-12 09:42:22 +01:00
0a45bd034e
YARN-10520. Deprecated the residual nested class for the LCEResourceHandler ( #2527 ). Contributed by Wanqiang Ji
2020-12-09 12:55:24 +01:00
d67ccd03e3
YARN-10380: Import logic of multi-node allocation in CapacityScheduler ( #2494 )
...
Contributed by Qi Zhu.
2020-12-09 19:48:39 +08:00
7d3c8ef606
YARN-10495. make the rpath of container-executor configurable. Contributed by
...
angerszhu.
2020-12-08 00:53:24 +00:00
9170eb566b
YARN-10511. Update yarn.nodemanager.env-whitelist value in docs ( #2512 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-12-04 00:14:04 +09:00
9969745343
YARN-9883. Reshape SchedulerHealth class. Contributed by D M Murali Krishna Reddy
2020-12-03 09:55:06 +01:00
fa773a8326
YARN-10278: CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework. Contributed by Szilard Nemeth (snemeth)
2020-12-01 22:06:47 +00:00
4d2ae5b398
YARN-10498. Fix typo in CapacityScheduler Markdown document ( #2484 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-11-30 11:17:10 +09:00
2ce2198287
HADOOP-17394. [JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:32:10 +09:00
65002c9486
Revert "[JDK 11] Fix error in mvn package -Pdocs ( #2488 )" because JIRA ID is missing.
...
This reverts commit ce5b3d7975
2020-11-26 11:30:53 +09:00
ce5b3d7975
[JDK 11] Fix error in mvn package -Pdocs ( #2488 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2020-11-26 11:27:30 +09:00
08b2e285db
YARN-10488. Several typos in package: org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair ( #2485 )
...
Co-authored-by: akumar <akumar@cloudera.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2020-11-25 13:00:09 +09:00
569b20e31c
YARN-10468. Fix TestNodeStatusUpdater timeouts and broken conditions ( #2461 )
2020-11-24 11:09:30 -08:00
c4ba0ab7df
YARN-10470. When building new web ui with root user, the bower install should support it. Contributed by zhuqi.
2020-11-24 15:22:07 +09:00
0d3155a687
YARN-10457. Add a configuration switch to change between legacy and JSON placement rule format. Contributed by Gergely Pollak
2020-11-18 17:02:47 +01:00
5ff70a59c4
YARN-10486. FS-CS converter: handle case when weight=0 and allow more lenient capacity checks in Capacity Scheduler. Contributed by Peter Bacsko
2020-11-18 16:51:42 +01:00
0b2510ee1f
YARN-10485. TimelineConnector swallows InterruptedException ( #2450 ). Contributed by Ahmed Hussein
2020-11-16 14:55:40 -06:00
567600fd80
YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.
2020-11-11 17:10:15 +01:00
cd0490e8c6
Revert "YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak."
...
This reverts commit b0ab222a6c
2020-11-11 17:06:12 +01:00
b0ab222a6c
YARN-10425. Replace the legacy placement engine in CS with the new one. Contributed by Gergely Pollak.
2020-11-11 17:03:16 +01:00
61f8c5767e
YARN-10480. replace href tags with ng-href ( #2426 )
2020-11-10 10:54:28 -08:00
4331c88352
HADOOP-17344. Harmonize guava version and shade guava in yarn-csi. ( #2434 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-11-10 15:19:40 +09:00
minni31
Akira Ajisaka
Wei-Chiu Chuang
Andrew Chung
dependabot[bot]
Gera Shegalov
luoyuan3471
Andras Gyori
Cheng Pan
Ashutosh Gupta
Eric Payne
Viraj Jasani
Szilard Nemeth
Szilard Nemeth
Sushmasree-28
GuoPhilipse
Jack
Adam Antal
Ahmed Hussein
lujiefsi
Szilard Nemeth
Benjamin Teke
Tamas Domok
Neil
Akshat Bordia
TiborKovacsCloudera
Eric Badger
Weihao Zheng
zhuqi
Swathi Chandrashekar
srinivasst
Gergely Pollak
Masatake Iwasaki
Jim Brennan
Artem Smotrakov
Prabhu Joseph
Peter Bacsko
Akira Ajisaka
Prabhu Josephraj
Hideyuki Furue
Gergely Pollak
lujiefsi
kyoty
zhuqi
Borislav Iordanov
Siyao Meng
Miklos Gergely
stack
Brahma Reddy Battula
Jack Jiang
Cyrus Jackson
Wilfred Spiegelenburg
JJiaguodong
Ahmed Hussein
Jonathan Hung
Amogh Rajesh Desai
bibinchundatt
belugabehr
Wangda Tan
Sunil G
He Xiaoqiao
adamantal
Andrea Scarpino
kevinzhao1661
Ankit Kumar
Gabriel Medeiros Coelho