edeb99548a
YARN-11161. Support getAttributesToNodes, getClusterNodeAttributes, getNodesToAttributes API's for Federation ( #4610 )
2022-07-25 10:05:45 -07:00
63db1a85e3
YARN-11203. Fix typo in hadoop-yarn-server-router module. ( #4510 ). Contributed by fanshilun.
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-07-23 20:28:45 +05:30
5c84cb81ba
YARN-8900. [Router] Federation: routing getContainers REST invocations transparently to multiple RMs ( #4543 )
2022-07-22 17:06:38 -07:00
3cce41a1f6
Make upstream aware of 3.2.4 release.
...
(cherry picked from commit e1637a57df
2022-07-22 02:27:19 +00:00
838020ce3b
YARN-11160. Support getResourceProfiles, getResourceProfile API's for Federation ( #4540 )
2022-07-21 11:57:24 -07:00
f4b635c4dc
YARN-11211. QueueMetrics leaks Configuration objects when validation API is called multiple times. Contributed by Andras Gyori
2022-07-21 14:20:34 +02:00
34e548cb62
HADOOP-18332: remove rs-api dependency as it conflicts with jsr311-api ( #4547 )
...
This downgrades jackson from the version switched to in
HADOOP-18033 (2.13.0), to Jackson 2.12.7.
This removes the dependency on javax.ws.rs-api,
so avoiding runtime problems with applications using
jersey-core v1 and/or jsr311-api.
The 2.12.7 release still contains the fix for CVE-2020-36518.
Contributed by PJ Fanning
2022-07-17 21:37:54 +05:30
84ce592a85
YARN-11198. clean up numa resources from statestore ( #4546 )
...
* YARN-11198. clean up numa resources from levelDB
Co-authored-by: Deb <dbsamrat@3c22fba1b03f.ant.amazon.com>
2022-07-14 11:07:48 +05:30
161b1fac2e
YARN-11169. Support moveApplicationAcrossQueues, getQueueInfo API's for Federation. ( #4464 )
2022-07-05 11:24:29 -07:00
151bb31c47
YARN-9403.GET /apps/{appid}/entities/YARN_APPLICATION accesses application table instead of entity table ( #4516 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-07-02 21:59:28 +05:30
57cbde9abf
YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping ( #4515 )
...
* YARN-10287.Update scheduler-conf corrupts the CS configuration when removing queue which is referred in queue mapping
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-07-02 21:58:56 +05:30
3cad632709
YARN-11202. Optimize ClientRMService.getApplications. Contributed by Tamas Domok.
...
Change-Id: I55ddb46fd0e4cdb644747d6d43083215f10861b5
2022-07-01 10:50:48 +02:00
2d133a54ac
YARN-11204. Various MapReduce tests fail with NPE in AggregatedLogDeletionService.stopRMClient. Contributed by Szilard Nemeth.
2022-06-29 15:12:04 +02:00
073b8ea1d5
HADOOP-18284. Remove Unnecessary semicolon ';' ( #4422 ). Contributed by fanshilun.
2022-06-29 15:20:41 +05:30
7eefdf8642
YARN-11195. Adding document to enable numa ( #4501 )
...
Contributed by Samrat Deb.
2022-06-28 17:46:43 +05:30
a177232ebc
YARN-9822.TimelineCollectorWebService#putEntities blocked when ATSV2 HBase is down ( #4492 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-28 09:32:07 +05:30
734b6f19ad
YARN-9874.Remove unnecessary LevelDb write call in LeveldbConfigurationStore#confirmMutation ( #4487 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-23 21:59:27 +05:30
4abb2ba58c
YARN-10320.Replace FSDataInputStream#read with readFully in Log Aggregation ( #4486 )
...
* YARN-10320.Replace FSDataInputStream#read with readFully in Log Aggregation
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-23 21:58:32 +05:30
0af4bb3b42
YARN-11192. TestRouterWebServicesREST failing after YARN-9827. ( #4484 ). Contributed by fanshilun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-23 13:21:36 +05:30
e6ecc4f3e4
YARN-11188. Only files belong to the first file controller are removed even if multiple log aggregation file controllers are configured. Contributed by Szilard Nemeth.
2022-06-22 14:40:20 +02:00
cbdabe9ec8
YARN-9971.YARN Native Service HttpProbe logs THIS_HOST in error messages ( #4436 )
...
* YARN-9971.YARN Native Service HttpProbe logs THIS_HOST in error messages
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
2022-06-22 09:38:19 +05:30
3a66348fda
YARN-11185. Pending app metrics are increased doubly when a queue reaches its max-parallel-apps limit. Contributed by Andras Gyori
2022-06-20 15:03:58 +02:00
5d08ffa769
YARN-11182. Refactor TestAggregatedLogDeletionService: 2nd phase. Contributed by Szilard Nemeth.
2022-06-20 14:12:51 +02:00
4f425b641c
YARN-9827.Fix Http Response code in GenericExceptionHandler ( #4393 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed by Akira Ajisaka.
2022-06-20 10:44:47 +05:30
62e4476102
YARN-10122. Support signalToContainer API for Federation. ( #4421 )
2022-06-17 16:38:36 -07:00
80446dcd08
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. ( #4408 )
...
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.
Contributed by zhengchenyu
2022-06-17 19:49:36 +01:00
020201cb65
Queue filter in CS UI v1 does not work as expected. Contributed by Chengbing Liu.
2022-06-17 19:28:32 +02:00
dc5460d525
YARN-11078. Set env vars in a cross platform compatible way ( #4432 )
...
* Maven runs the ember build script.
The environment variable TMPDIR was
set as per bash syntax.
* This failed on Windows since the
Windows command prompt doesn't
support bash syntax.
* We're now detecting the OS and
setting a Maven property
"emberBuildScript" in a cross
platform compatible way.
2022-06-15 15:29:55 +05:30
75bc6cfced
YARN-11176. Refactor TestAggregatedLogDeletionService. Contributed by Szilard Nemeth.
2022-06-14 16:14:44 +02:00
c9a174a260
YARN-11175. Refactor LogAggregationFileControllerFactory
2022-06-13 13:58:13 +02:00
985161f6ec
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi ( #4417 )
...
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.
Contributed by Steve Loughran
2022-06-09 13:10:46 +01:00
98ca6fa10a
YARN-11159. Support failApplicationAttempt, updateApplicationPriority, updateApplicationTimeouts API's for Federation ( #4396 )
2022-06-08 17:34:43 -07:00
1cc2416f68
YARN-10487. Support getQueueUserAcls, listReservations, getApplicationAttempts, getContainerReport, getContainers, getResourceTypeInfo API's for Federation ( #4341 )
2022-06-02 12:54:29 -07:00
7c66266b4e
YARN-11142.Remove unused Imports in Hadoop YARN project ( #4381 ). Contributed by Ashutosh Gupta.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-05-31 23:17:11 +05:30
3228142f53
Make upstream aware of 2.10.2 release.
...
(cherry picked from commit 3dcb6367ed
2022-06-01 00:53:01 +09:00
e3e9369c1d
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* ( #4271 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-26 15:51:55 +09:00
678768035d
YARN-11137. Improve log message in FederationClientInterceptor ( #4336 )
2022-05-24 17:21:04 -07:00
f390edaec4
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. ( #4350 )
2022-05-24 14:07:19 +09:00
8dd3ef1f08
YARN-10465. Support getNodeToLabels, getLabelsToNodes, getClusterNodeLabels API's for Federation ( #4317 )
2022-05-20 13:19:42 -07:00
0733e968ab
HADOOP-18224. Upgrade maven compiler plugin to 3.10.1 ( #4267 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-21 03:21:13 +09:00
0e6a6d1880
YARN-11152. QueueMetrics is leaking memory when creating a new queue during reinitialisation
2022-05-18 18:22:42 +02:00
54cd0174c0
YARN-11147. ResourceUsage and QueueCapacities classes provide node label iterators that are not thread safe
2022-05-18 18:16:34 +02:00
6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
...
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version
Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:54 +01:00
79cc0caf9f
YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value ( #4280 )
...
* Update QueueConfigurationsPBImpl.java
* Update TestPBImplRecords.java
* Update TestPBImplRecords.java
* Update TestPBImplRecords.java
* Update TestPBImplRecords.java
2022-05-16 22:10:09 +05:30
931abbd927
YARN-11092. Upgrade jquery ui to 1.13.1 ( #4260 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-16 23:11:17 +09:00
6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor ( #4274 )
2022-05-15 09:16:06 -07:00
6985f9aabe
YARN-10080. Support show app id on localizer thread pool ( #4283 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:41:06 +09:00
d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases ( #4110 )
...
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:11:42 +09:00
eced5bec68
YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth
2022-05-13 13:16:12 +02:00
5e2f4339fa
YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok
2022-05-12 13:42:06 +02:00
9af3eabdca
YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth
2022-05-11 18:01:31 +02:00
eeedc0c7e4
YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke
2022-05-11 14:39:42 +02:00
b62d6ce6fd
YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori
2022-05-11 14:29:05 +02:00
09a69f996b
YARN-11130. removed unused import ( #4276 )
2022-05-11 12:14:52 +05:30
ae47846a5b
HADOOP-16515. Update the link to compatibility guide ( #4226 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-08 07:38:32 +09:00
d4a91bd0c0
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class ( #4242 )
...
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-03 03:14:26 +09:00
a74acc755e
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. ( #4222 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-03 03:08:30 +09:00
63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 ( #3988 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-28 14:18:19 +09:00
006767eb94
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md ( #4221 )
2022-04-26 09:51:11 +08:00
6999acf520
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2)
...
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.
As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.
This commit depends on the associated hadoop-common patch,
which must be committed first.
Contributed by Steve Loughran.
Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
2022-04-24 17:33:05 +01:00
b35b744670
YARN-11111. Recovery failure when node-label configure-type transit from delegated-centralized to centralized ( #4200 )
2022-04-21 22:37:28 -04:00
d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. ( #4175 )
2022-04-15 17:50:49 +09:00
3b46aae977
YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1
2022-04-13 15:57:18 +02:00
e044a46f97
YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang
2022-03-31 12:39:56 +02:00
6e00a799e7
YARN-11106. Fix the test failure due to missing conf of yarn.resourcemanager.node-labels.am.default-node-label-expression. Contributed by Junfan Zhang
2022-03-30 19:43:23 +02:00
61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 ( #3990 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-26 20:31:16 +09:00
da09d68056
YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok
2022-03-25 21:53:18 +01:00
921267ca31
YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang.
2022-03-23 08:57:00 +01:00
2beb7296fb
YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang.
2022-03-22 13:14:43 +01:00
c3124a3f6e
YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke.
2022-03-21 14:55:49 +01:00
e2701e227f
YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang.
2022-03-21 14:37:10 +01:00
4537b34e1c
YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang.
2022-03-21 14:18:27 +01:00
ed65aa2324
YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori
2022-03-10 22:22:58 +01:00
481da19494
YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke
2022-03-10 22:15:35 +01:00
383b73417d
YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-03-10 13:11:19 +01:00
9539ff108a
YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth
2022-03-10 08:40:43 +01:00
db8ae4b654
YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori
2022-03-09 19:36:09 +01:00
2ece95064b
YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori
2022-03-09 19:29:53 +01:00
56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing ( #4027 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-08 21:32:51 +09:00
66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies ( #4000 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-03-08 17:27:04 +09:00
851a4dc2d9
YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth
2022-03-07 09:01:18 +01:00
379baa5eb6
YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori
2022-03-03 16:44:12 +01:00
1c27c69f44
Revert "YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori"
...
This reverts commit de526e166chttps://issues.apache.org/jira/browse/YARN-10850 has been committed.
2022-03-03 16:38:56 +01:00
22fe79cee3
YARN-11076. Upgrade jQuery version in Yarn UI2. ( #4046 )
...
Change-Id: I3cb1677741df5a1978e83029443d4a2d5d7e3d7f
2022-03-03 23:54:03 +09:00
356d337d1e
YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok
2022-03-02 16:19:17 +01:00
9e475aede6
YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke
2022-03-02 11:16:24 +01:00
365375412f
YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori
2022-02-22 12:26:35 +01:00
0463498adc
YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori
2022-02-22 11:55:34 +01:00
de526e166c
YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori
2022-02-17 21:05:01 +01:00
7dd288ce8c
YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori
2022-02-17 19:58:14 +01:00
ad0a1dc897
YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke
2022-02-17 19:48:42 +01:00
4c05d257ba
YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke
2022-02-17 19:43:37 +01:00
51001fc7bf
YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok
2022-02-17 19:38:18 +01:00
48e02f7998
YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok
2022-02-17 19:17:31 +01:00
bb1135c77c
YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-02-17 19:12:51 +01:00
d58c10c3a5
YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok
2022-02-17 19:03:44 +01:00
7fd90cdcbe
YARN-10788. TestCsiClient fails ( #3989 )
...
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2022-02-15 01:13:13 +09:00
ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… ( #3195 )
2022-02-01 09:37:40 -08:00
87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… ( #3632 )
2022-02-01 09:35:59 -08:00
e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… ( #3741 )
...
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp ( #2591 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2022-01-28 15:51:21 +09:00
02f6bad1ff
Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )"
...
This reverts commit 1c01944f35
2022-01-28 00:36:25 +09:00
1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. ( #3936 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 00:04:01 +09:00
4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse ( #3766 )
2022-01-25 09:25:18 -08:00
3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container ( #3779 )
2022-01-24 08:03:36 -08:00
dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui ( #3890 )
...
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects ) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases )
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7 )
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-20 21:44:10 +09:00
a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c ( #3851 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-17 16:20:48 +09:00
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable ( #3859 )
2022-01-11 14:38:59 +08:00
3bf014d871
YARN-8859. Add audit logs for router service ( #3607 )
...
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-01-03 10:38:59 +05:30
4db27acf06
YARN-11053. AuxService should not use class name as default system classes ( #3816 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 11:08:04 +09:00
00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch ( #3793 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
e2d6fd075d
YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi).
2021-12-21 19:05:39 +00:00
6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled ( #3802 )
2021-12-20 18:02:57 +08:00
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
07141426e0
YARN-11050 ( #3805 )
2021-12-18 11:52:03 +08:00
8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class ( #3800 ). Contributed by Szilard Nemeth
2021-12-17 00:18:14 +01:00
aec9cdb467
YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok
2021-12-16 23:39:18 +01:00
a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API ( #3799 ). Contributed by Szilard Nemeth
2021-12-16 15:53:08 +01:00
6ea1685abd
YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke
2021-12-16 00:01:09 +01:00
f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster ( #3796 )
2021-12-15 16:47:51 +08:00
843f66f4dc
Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke
2021-12-14 22:00:46 +01:00
364d38f00d
YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke
2021-12-14 14:57:08 +01:00
a5bcf4c792
YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke
2021-12-13 21:57:46 +01:00
898055e204
YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke
2021-12-13 16:12:55 +01:00
4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown ( #3768 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 14:03:32 +09:00
f544ef4a51
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2021-12-10 15:09:53 +01:00
a0d8cde133
YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs
2021-12-09 17:51:44 +01:00
ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy ( #3717 )
2021-12-08 10:02:10 -08:00
66a08e933a
YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok
2021-12-08 16:41:35 +01:00
19430118c6
YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2021-12-08 16:03:42 +01:00
53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 ( #3749 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
126079612c
YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori
2021-12-07 15:55:16 +01:00
7284d23476
YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke
2021-12-07 15:43:55 +01:00
60440a836a
YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok
2021-12-07 15:26:07 +01:00
7a8c2e0097
YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori
2021-12-07 08:51:03 +01:00
d75f2db2f9
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2021-12-07 08:42:57 +01:00
5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing ( #3728 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars ( #3710 )
...
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile ( #3722 )
2021-12-02 10:40:30 +08:00
7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files ( #3735 )
...
Contributed by sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages ( #3712 )
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
c9d64bad37
YARN-11007. Correct words in YARN documents ( #3680 )
...
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-25 17:48:54 +09:00
08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method ( #3592 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 18:41:22 +09:00
d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 ( #3642 )
2021-11-23 13:21:51 -08:00
5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources ( #3646 )
2021-11-23 13:20:08 -08:00
c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules ( #3687 )
2021-11-23 13:36:22 +09:00
b933f5f54b
YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang
2021-11-15 16:02:18 +01:00
7cb887e6c2
YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori
2021-11-12 15:43:45 +01:00
e220e88eca
YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori
2021-11-12 15:33:39 +01:00
8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers ( #3603 )
2021-11-11 09:49:29 -08:00
2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration ( #3570 ). Contributed by Szilard Nemeth
2021-10-28 15:19:32 +02:00
d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue ( #3551 ) contributed by Szilard Nemeth
2021-10-27 19:03:45 +02:00
66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler ( #3581 ) Contributed by Szilard Nemeth
2021-10-27 17:13:49 +02:00
9cfd8d0a83
YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth
2021-10-23 14:47:09 +02:00
32ecaed9c3
YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori
2021-10-22 17:32:33 +02:00
23772d946b
YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal
2021-10-22 16:33:03 +02:00
516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom ( #3555 )
2021-10-21 16:54:25 +09:00
d286994009
YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne)
2021-10-20 22:18:36 +00:00
20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth
2021-10-20 15:52:37 +02:00
35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori
2021-10-20 12:56:41 +02:00
414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule ( #3560 )
...
* YARN-10958. Initial commit
* Fix javadoc + behaviour
* Fix review comments
* fix checkstyle + blanks
* fix checkstyle + blanks
* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
616cea2e80
YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori
2021-10-19 13:06:45 +02:00
025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth
2021-10-19 12:24:58 +02:00
ae95caa60e
YARN-10976. Fix resource leak due to Files.walk ( #3552 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules ( #3541 )
2021-10-14 18:03:01 +09:00
1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
8c271caa81
YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke
2021-10-12 17:35:06 +02:00
35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori
2021-10-07 17:09:38 +02:00
5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils ( #3515 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori
2021-10-01 04:20:36 +02:00
3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
4bd0c36189
YARN-10970. Standby RM should expose prom endpoint ( #3480 )
...
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics ( #3439 )
2021-09-28 09:51:18 -07:00
8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori
2021-09-21 16:08:24 +02:00
4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth
2021-09-20 16:47:46 +02:00
a9b2469a53
YARN-10950. Code cleanup in QueueCapacities ( #3454 )
2021-09-19 14:42:02 +02:00
5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController ( #3450 )
2021-09-19 14:35:34 +02:00
aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class ( #3420 )
2021-09-19 13:11:56 +02:00
f5c76c8e31
YARN-10936. Log typo corrected. ( #3449 )
...
Contributed by: Tibor Kovacs
2021-09-17 08:27:51 +02:00
43f0a34dd4
YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne.
2021-09-15 20:03:45 +00:00
5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources ( #3418 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy ( #3403 )
...
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic ( #3390 )
...
- capacityConfigType update is extracted to a separate method
- validation logic is extracted to a helper function
- min resource must not be greater than max resource is now checked
after the max resource is updated
Change-Id: I731c2639281721afed32c30854bafcf048d6ee28
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor ( #3389 )
...
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF ( #3352 )
...
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate ( #3396 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue ( #3392 )
2021-09-10 16:59:46 +02:00
b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic ( #3407 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue ( #3402 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor ( #3325 )
2021-09-09 09:20:57 -07:00
a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report ( #3252 )
2021-09-08 10:36:56 -07:00
2ff3fc50e4
YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak
2021-09-08 18:01:39 +02:00
5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir ( #3355 )
...
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-08 17:04:32 +02:00
4e209a31da
YARN-10919. Remove LeafQueue#scheduler field ( #3382 )
...
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario ( #3339 )
...
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
3024a47026
Add documentation for YARN-10623 auto refresh queue conf in CS ( #3279 )
2021-09-08 16:03:15 +02:00
slfan1989
Masatake Iwasaki
Szilard Nemeth
PJ Fanning
Samrat
Ashutosh Gupta
Tamas Domok
9uapaw
zhengchenyu
Benjamin Teke
Gautham B A
Steve Loughran
Owen O'Malley
Viraj Jasani
Zilong Zhu
Jian Chen
Szilard Nemeth
jteagles
Junfan Zhang
章锡平
Akira Ajisaka
minni31
Akira Ajisaka
Wei-Chiu Chuang
Andrew Chung
dependabot[bot]
Gera Shegalov
luoyuan3471
Cheng Pan
Eric Payne
Szilard Nemeth
Sushmasree-28
GuoPhilipse
Jack
Adam Antal
Ahmed Hussein
lujiefsi
Tamas Domok
Neil
Akshat Bordia
TiborKovacsCloudera
Eric Badger
Weihao Zheng
zhuqi