edeb99548a
YARN-11161. Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation ( #4610 )
2022-07-25 10:05:45 -07:00
63db1a85e3
YARN-11203. Fix typo in hadoop-yarn-server-router module. ( #4510 ). Contributed by fanshilun.
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-07-23 20:28:45 +05:30
5c84cb81ba
YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs ( #4543 )
2022-07-22 17:06:38 -07:00
838020ce3b
YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation ( #4540 )
2022-07-21 11:57:24 -07:00
f4b635c4dc
YARN-11211. QueueMetrics leaks Configuration objects when validation API is called multiple times. Contributed by Andras Gyori
2022-07-21 14:20:34 +02:00
34e548cb62
HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api ( #4547 )
...
This downgrades jackson from the version switched to in
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.
The 2.12.7 release still contains the fix for CVE-2020-36518.
Contributed by PJ Fanning
2022-07-17 21:37:54 +05:30
84ce592a85
YARN-11198. clean up numa resources from statestore ( #4546 )
...
* YARN-11198. clean up numa resources from levelDB
Co-authored-by: Deb <dbsamrat@3c22fba1b03f.ant.amazon.com>
2022-07-14 11:07:48 +05:30
161b1fac2e
YARN-11169. Support moveApplicationAcrossQueues, getQueueInfo API's for Federation. ( #4464 )
2022-07-05 11:24:29 -07:00
151bb31c47
YARN-9403.GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table ( #4516 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-07-02 21:59:28 +05:30
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping ( #4515 )
...
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-07-02 21:58:56 +05:30
3cad632709
YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok.
...
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
2022-07-01 10:50:48 +02:00
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' ( #4422 ). Contributed by fanshilun.
2022-06-29 15:20:41 +05:30
a177232ebc
YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down ( #4492 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-28 09:32:07 +05:30
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation ( #4487 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-23 21:59:27 +05:30
0af4bb3b42
YARN-11192. TestRouterWebServicesREST failing after YARN-9827. ( #4484 ). Contributed by fanshilun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-23 13:21:36 +05:30
3a66348fda
YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori
2022-06-20 15:03:58 +02:00
62e4476102
YARN-10122. Support signalToContainer API for Federation. ( #4421 )
2022-06-17 16:38:36 -07:00
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. ( #4408 )
...
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.
Contributed by zhengchenyu
2022-06-17 19:49:36 +01:00
020201cb65
Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu.
2022-06-17 19:28:32 +02:00
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation ( #4396 )
2022-06-08 17:34:43 -07:00
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation ( #4341 )
2022-06-02 12:54:29 -07:00
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project ( #4381 ). Contributed by Ashutosh Gupta.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-05-31 23:17:11 +05:30
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* ( #4271 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-26 15:51:55 +09:00
678768035d
YARN-11137. Improve log message in FederationClientInterceptor ( #4336 )
2022-05-24 17:21:04 -07:00
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. ( #4350 )
2022-05-24 14:07:19 +09:00
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation ( #4317 )
2022-05-20 13:19:42 -07:00
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 ( #4267 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-21 03:21:13 +09:00
0e6a6d1880
YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation
2022-05-18 18:22:42 +02:00
54cd0174c0
YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe
2022-05-18 18:16:34 +02:00
6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor ( #4274 )
2022-05-15 09:16:06 -07:00
6985f9aabe
YARN-10080. Support show app id on localizer thread pool ( #4283 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:41:06 +09:00
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases ( #4110 )
...
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:11:42 +09:00
eced5bec68
YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth
2022-05-13 13:16:12 +02:00
5e2f4339fa
YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok
2022-05-12 13:42:06 +02:00
9af3eabdca
YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth
2022-05-11 18:01:31 +02:00
b62d6ce6fd
YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori
2022-05-11 14:29:05 +02:00
09a69f996b
YARN-11130. removed unused import ( #4276 )
2022-05-11 12:14:52 +05:30
ae47846a5b
HADOOP-16515. Update the link to compatibility guide ( #4226 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-08 07:38:32 +09:00
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 ( #3988 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-28 14:18:19 +09:00
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. ( #4175 )
2022-04-15 17:50:49 +09:00
3b46aae977
YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1
2022-04-13 15:57:18 +02:00
e044a46f97
YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang
2022-03-31 12:39:56 +02:00
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 ( #3990 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-26 20:31:16 +09:00
da09d68056
YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok
2022-03-25 21:53:18 +01:00
921267ca31
YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang.
2022-03-23 08:57:00 +01:00
2beb7296fb
YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang.
2022-03-22 13:14:43 +01:00
c3124a3f6e
YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke.
2022-03-21 14:55:49 +01:00
e2701e227f
YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang.
2022-03-21 14:37:10 +01:00
4537b34e1c
YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang.
2022-03-21 14:18:27 +01:00
ed65aa2324
YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori
2022-03-10 22:22:58 +01:00
481da19494
YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke
2022-03-10 22:15:35 +01:00
383b73417d
YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-03-10 13:11:19 +01:00
9539ff108a
YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth
2022-03-10 08:40:43 +01:00
db8ae4b654
YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori
2022-03-09 19:36:09 +01:00
2ece95064b
YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori
2022-03-09 19:29:53 +01:00
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing ( #4027 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-08 21:32:51 +09:00
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies ( #4000 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-03-08 17:27:04 +09:00
851a4dc2d9
YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth
2022-03-07 09:01:18 +01:00
379baa5eb6
YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori
2022-03-03 16:44:12 +01:00
356d337d1e
YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok
2022-03-02 16:19:17 +01:00
9e475aede6
YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke
2022-03-02 11:16:24 +01:00
365375412f
YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori
2022-02-22 12:26:35 +01:00
0463498adc
YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori
2022-02-22 11:55:34 +01:00
7dd288ce8c
YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori
2022-02-17 19:58:14 +01:00
ad0a1dc897
YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke
2022-02-17 19:48:42 +01:00
4c05d257ba
YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke
2022-02-17 19:43:37 +01:00
48e02f7998
YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok
2022-02-17 19:17:31 +01:00
bb1135c77c
YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-02-17 19:12:51 +01:00
d58c10c3a5
YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok
2022-02-17 19:03:44 +01:00
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… ( #3195 )
2022-02-01 09:37:40 -08:00
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… ( #3632 )
2022-02-01 09:35:59 -08:00
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… ( #3741 )
...
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse ( #3766 )
2022-01-25 09:25:18 -08:00
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container ( #3779 )
2022-01-24 08:03:36 -08:00
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c ( #3851 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-17 16:20:48 +09:00
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable ( #3859 )
2022-01-11 14:38:59 +08:00
3bf014d871
YARN-8859. Add audit logs for router service ( #3607 )
...
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-01-03 10:38:59 +05:30
4db27acf06
YARN-11053. AuxService should not use class name as default system classes ( #3816 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 11:08:04 +09:00
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch ( #3793 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
e2d6fd075d
YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).
2021-12-21 19:05:39 +00:00
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled ( #3802 )
2021-12-20 18:02:57 +08:00
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
07141426e0
YARN-11050 ( #3805 )
2021-12-18 11:52:03 +08:00
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class ( #3800 ). Contributed by Szilard Nemeth
2021-12-17 00:18:14 +01:00
aec9cdb467
YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok
2021-12-16 23:39:18 +01:00
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API ( #3799 ). Contributed by Szilard Nemeth
2021-12-16 15:53:08 +01:00
6ea1685abd
YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke
2021-12-16 00:01:09 +01:00
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster ( #3796 )
2021-12-15 16:47:51 +08:00
843f66f4dc
Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke
2021-12-14 22:00:46 +01:00
364d38f00d
YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke
2021-12-14 14:57:08 +01:00
a5bcf4c792
YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke
2021-12-13 21:57:46 +01:00
898055e204
YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke
2021-12-13 16:12:55 +01:00
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown ( #3768 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 14:03:32 +09:00
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing ( #3728 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile ( #3722 )
2021-12-02 10:40:30 +08:00
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files ( #3735 )
...
Contributed by sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration ( #3570 ). Contributed by Szilard Nemeth
2021-10-28 15:19:32 +02:00
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue ( #3551 ) contributed by Szilard Nemeth
2021-10-27 19:03:45 +02:00
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler ( #3581 ) Contributed by Szilard Nemeth
2021-10-27 17:13:49 +02:00
9cfd8d0a83
YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth
2021-10-23 14:47:09 +02:00
32ecaed9c3
YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori
2021-10-22 17:32:33 +02:00
23772d946b
YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal
2021-10-22 16:33:03 +02:00
d286994009
YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne)
2021-10-20 22:18:36 +00:00
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth
2021-10-20 15:52:37 +02:00
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori
2021-10-20 12:56:41 +02:00
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule ( #3560 )
...
* YARN-10958. Initial commit
* Fix javadoc + behaviour
* Fix review comments
* fix checkstyle + blanks
* fix checkstyle + blanks
* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
616cea2e80
YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori
2021-10-19 13:06:45 +02:00
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth
2021-10-19 12:24:58 +02:00
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk ( #3552 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules ( #3541 )
2021-10-14 18:03:01 +09:00
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
8c271caa81
YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke
2021-10-12 17:35:06 +02:00
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori
2021-10-07 17:09:38 +02:00
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori
2021-10-01 04:20:36 +02:00
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint ( #3480 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics ( #3439 )
2021-09-28 09:51:18 -07:00
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
2021-09-21 16:08:24 +02:00
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth
2021-09-20 16:47:46 +02:00
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities ( #3454 )
2021-09-19 14:42:02 +02:00
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class ( #3420 )
2021-09-19 13:11:56 +02:00
43f0a34dd4
YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne.
2021-09-15 20:03:45 +00:00
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources ( #3418 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
9512c774c0
YARN-10884: Handle empty owners to parse log files ( #3318 )
...
* YARN 10884 : Parse log files which has empty owner
* Removing the whitespace
* Added a test case for null user
* Fixed indentations
* Fixed the indentation for test cases
Co-authored-by: Swathi C <cswathi@microsoft.com>
2021-09-07 21:37:51 +05:30
16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. ( #3314 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-27 23:09:54 +02:00
e06a5cb197
YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke
2021-08-24 15:27:34 +02:00
4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node ( #3287 )
...
* Account for scheduled AM containers before deactivating node
* Move AM container check to separate method.
* Fix UTs
* Fix UTs
* Remove unnecessary import
* Add timeout for UT
2021-08-17 14:18:55 +05:30
0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-04 15:27:46 +02:00
a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests ( #3248 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-03 16:19:04 +02:00
ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 ( #3241 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
efb3fa2bf5
YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang.
2021-08-02 18:23:33 +08:00
ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues ( #3225 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. ( #3194 )
...
YARN-10841. Fix token reset synchronization for UAM response token. Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
1b9efe58c9
YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori
2021-07-28 16:50:14 +02:00
8d0297c213
YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori
2021-07-28 14:49:10 +02:00
fa0289b022
YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang
2021-07-27 16:47:41 +02:00
05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. ( #3203 )
...
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor ( #3135 )
...
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135 )
2021-07-23 17:07:51 +05:30
2da9b95d4d
YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori.
2021-07-22 20:30:43 +08:00
0441efe1fc
YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger.
2021-07-21 15:31:44 +08:00
d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 ( #3204 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-16 12:17:17 +09:00
632f64cadb
YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne.
2021-07-15 14:23:31 +00:00
ba325a8ada
HADOOP-17793. Better token validation ( #3189 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility ( #3171 )
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
dc6f456e95
YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne.
2021-06-28 14:52:19 +00:00
7c7d02edbd
YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T.
2021-06-25 20:32:08 +00:00
0934e783cf
YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori.
2021-06-24 20:15:10 +02:00
6562391737
YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori
2021-06-16 18:55:09 +02:00
428478bbe2
YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori
2021-06-16 18:26:58 +02:00
e31d06032b
YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke
2021-06-14 22:33:04 +02:00
7003997e36
YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi
2021-06-12 14:49:52 +02:00
81d7069316
YARN-10805. Replace Guava Lists usage by Hadoop's own Lists in hadoop-yarn-project ( #3075 )
2021-06-09 15:15:47 +09:00
ca27744770
YARN-10809. Missing dependency causing NoClassDefFoundError in TestHBaseTimelineStorageUtils ( #3081 )
...
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
2021-06-08 19:37:23 -07:00
9445abb500
YARN-10792. Set Completed AppAttempt LogsLink to Log Server URL. Contributed by Abhinaba Sarkar
2021-06-08 20:37:40 +05:30
ec16b1d3b9
YARN-10807. Parents node labels are incorrectly added to child queues in weight mode. Contributed by Benjamin Teke.
2021-06-08 21:03:43 +08:00
9c7b8cf54e
Fix container-executor
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-06-06 07:05:29 +09:00
200eec8f2e
YARN-10796. Capacity Scheduler: dynamic queue cannot scale out properly if its capacity is 0%. Contributed by Peter Bacsko
2021-06-03 13:27:36 +02:00
2707f69251
YARN-10787. Queue submit ACL check is wrong when CS queue is ambiguous. Contributed by Gergely Pollak
2021-06-01 16:01:39 +02:00
e9339aa376
YARN-10797. Logging parameter issues in scheduler package. Contributed by Szilard Nemeth
2021-06-01 15:57:22 +02:00
b86a6eb871
YARN-10782. Extend /scheduler endpoint with template properties. Contributed by Andras Gyori
2021-05-25 18:27:53 +02:00
2541efa496
YARN-10783. Allow definition of auto queue template properties in root. Contributed by Andras Gyori
2021-05-25 13:55:59 +02:00
996d31f2dc
HADOOP-17721. Replace Guava Sets usage by Hadoop's own Sets in hadoop-yarn-project ( #3033 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-05-25 18:10:20 +09:00
59172ada90
YARN-10771. Add cluster metric for size of SchedulerEventQueue and RMEventQueue. Contributed by chaosju.
2021-05-24 23:12:07 +08:00
1e44bdb84c
YARN-7769. FS QueueManager should not create default queue at init. Contributed by Benjamin Teke
2021-05-22 14:55:01 +02:00
eb72628e15
YARN-10258. Add metrics for 'ApplicationsRunning' in NodeManager. Contributed by ANANDA G B.
2021-05-17 21:07:29 +02:00
8891e5c028
YARN-10763. Add the number of containers assigned per second metrics to ClusterMetrics. Contributed by chaosju.
2021-05-17 13:30:12 +02:00
d92a25b790
YARN-10555. Missing access check before getAppAttempts ( #2608 )
...
Co-authored-by: lujie <lujie@foxmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-05-17 13:44:38 +09:00
slfan1989
Szilard Nemeth
PJ Fanning
Samrat
Ashutosh Gupta
Tamas Domok
zhengchenyu
Benjamin Teke
Owen O'Malley
Viraj Jasani
9uapaw
Jian Chen
Szilard Nemeth
章锡平
Junfan Zhang
minni31
Andrew Chung
Gera Shegalov
Cheng Pan
Eric Payne
Szilard Nemeth
Sushmasree-28
Jack
Adam Antal
Ahmed Hussein
lujiefsi
Tamas Domok
Neil
Akshat Bordia
Eric Badger
Weihao Zheng
Swathi Chandrashekar
srinivasst
zhuqi-lucas
Jim Brennan
Artem Smotrakov
Peter Bacsko
Prabhu Josephraj
Hideyuki Furue
Gergely Pollak
lujiefsi