Commit Graph

6575 Commits

Author SHA1 Message Date
Steve Loughran 6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:54 +01:00
Zilong Zhu 79cc0caf9f
YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value (#4280)
* Update QueueConfigurationsPBImpl.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java

* Update TestPBImplRecords.java
2022-05-16 22:10:09 +05:30
Ashutosh Gupta 931abbd927
YARN-11092. Upgrade jquery ui to 1.13.1 (#4260)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-16 23:11:17 +09:00
slfan1989 6896c35a8d
YARN-11122. Support getClusterNodes API in FederationClientInterceptor (#4274) 2022-05-15 09:16:06 -07:00
Ashutosh Gupta 6985f9aabe
YARN-10080. Support show app id on localizer thread pool (#4283)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:41:06 +09:00
Jian Chen d2c9eb653a
YARN-11073. Avoid unnecessary preemption for tiny queues under certain corner cases (#4110)
Co-authored-by: Jian Chen <jian.chen@airbnb.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-14 01:11:42 +09:00
Szilard Nemeth eced5bec68 YARN-11123. ResourceManager webapps test failures due to org.apache.hadoop.metrics2.MetricsException and subsequent java.net.BindException: Address already in use. Contributed by Szilard Nemeth 2022-05-13 13:16:12 +02:00
Szilard Nemeth 5e2f4339fa YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok 2022-05-12 13:42:06 +02:00
Szilard Nemeth 9af3eabdca YARN-11114. RMWebServices returns only apps matching exactly the submitted queue name. Contributed by Szilard Nemeth 2022-05-11 18:01:31 +02:00
Szilard Nemeth eeedc0c7e4 YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke 2022-05-11 14:39:42 +02:00
Szilard Nemeth b62d6ce6fd YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori 2022-05-11 14:29:05 +02:00
slfan1989 09a69f996b
YARN-11130. removed unused import (#4276) 2022-05-11 12:14:52 +05:30
Ashutosh Gupta ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-08 07:38:32 +09:00
jteagles d4a91bd0c0
YARN-11116. Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class (#4242)
Co-authored-by: Jonathan Eagles <jeagles@verizonmedia.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-03 03:14:26 +09:00
Ashutosh Gupta a74acc755e
YARN-10187. Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained. (#4222)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-03 03:08:30 +09:00
PJ Fanning 63187083cc
HADOOP-15983. Use jersey-json that is built to use jackson2 (#3988)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-28 14:18:19 +09:00
Ashutosh Gupta 006767eb94
YARN-10303. Fixing rest api example, status code error and other typos in ResourceManagerRest.md (#4221) 2022-04-26 09:51:11 +08:00
Steve Loughran 6999acf520
HADOOP-16202. Enhanced openFile(): mapreduce and YARN changes. (#2584/2)
These changes ensure that sequential files are opened with the
right read policy, and split start/end is passed in.

As well as offering opportunities for filesystem clients to
choose fetch/cache/seek policies, the settings ensure that
processing text files on an s3 bucket where the default policy
is "random" will still be processed efficiently.

This commit depends on the associated hadoop-common patch,
which must be committed first.

Contributed by Steve Loughran.

Change-Id: Ic6713fd752441cf42ebe8739d05c2293a5db9f94
2022-04-24 17:33:05 +01:00
Junfan Zhang b35b744670
YARN-11111. Recovery failure when node-label configure-type transit from delegated-centralized to centralized (#4200) 2022-04-21 22:37:28 -04:00
章锡平 d5cba5c3d1
YARN-11107. Addendum. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. (#4175) 2022-04-15 17:50:49 +09:00
章锡平 3b46aae977 YARN-11107. When NodeLabel is enabled for a YARN cluster, AM blacklist program does not work properly. Contributed by zhangxiping1 2022-04-13 15:57:18 +02:00
Junfan Zhang e044a46f97 YARN-11088. Introduce the config to control the AM allocated to non-exclusive nodes. Contributed by Junfan Zhang 2022-03-31 12:39:56 +02:00
Junfan Zhang 6e00a799e7 YARN-11106. Fix the test failure due to missing conf of yarn.resourcemanager.node-labels.am.default-node-label-expression. Contributed by Junfan Zhang 2022-03-30 19:43:23 +02:00
PJ Fanning 61e809b245
HADOOP-13386. Upgrade Avro to 1.9.2 (#3990)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-26 20:31:16 +09:00
Tamas Domok da09d68056 YARN-11069. Dynamic Queue ACL handling in Legacy and Flexible Auto Created Queues. Contributed by Tamas Domok 2022-03-25 21:53:18 +01:00
9uapaw 921267ca31 YARN-11084. Introduce new config to specify AM default node-label when not specified. Contributed by Junfan Zhang. 2022-03-23 08:57:00 +01:00
9uapaw 2beb7296fb YARN-11087. Introduce the config to control the refresh interval in RMDelegatedNodeLabelsUpdater. Contributed by Junfan Zhang. 2022-03-22 13:14:43 +01:00
9uapaw c3124a3f6e YARN-10565. Refactor CS queue initialization to simplify weight mode calculation. Contributed by Benjamin Teke. 2022-03-21 14:55:49 +01:00
Szilard Nemeth e2701e227f YARN-11086. Add space in debug log of ParentQueue. Contributed by Junfan Zhang. 2022-03-21 14:37:10 +01:00
Szilard Nemeth 4537b34e1c YARN-11089. Fix typo in RM audit log. Contributed by Junfan Zhang. 2022-03-21 14:18:27 +01:00
Szilard Nemeth ed65aa2324 YARN-11067. Resource overcommitment due to incorrect resource normalisation logical order. Contributed by Andras Gyori 2022-03-10 22:22:58 +01:00
Szilard Nemeth 481da19494 YARN-10049. FIFOOrderingPolicy Improvements. Contributed by Benjamin Teke 2022-03-10 22:15:35 +01:00
9uapaw 383b73417d YARN-11036. Do not inherit from TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-03-10 13:11:19 +01:00
9uapaw 9539ff108a YARN-11052. Improve code quality in TestRMWebServicesNodeLabels. Contributed by Szilard Nemeth 2022-03-10 08:40:43 +01:00
Szilard Nemeth db8ae4b654 YARN-10918. Simplify method: CapacitySchedulerQueueManager#parseQueue. Contributed by Andras Gyori 2022-03-09 19:36:09 +01:00
Szilard Nemeth 2ece95064b YARN-10945. Add javadoc to all methods of AbstractCSQueue. Contributed by Andras Gyori 2022-03-09 19:29:53 +01:00
Viraj Jasani 56d807175d
YARN-11081. TestYarnConfigurationFields consistently keeps failing (#4027)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-08 21:32:51 +09:00
Viraj Jasani 66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-03-08 17:27:04 +09:00
9uapaw 851a4dc2d9 YARN-11049. MutableConfScheduler is referred as plain String instead of class name. Contributed by Szilard Nemeth 2022-03-07 09:01:18 +01:00
Szilard Nemeth 379baa5eb6 YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-03-03 16:44:12 +01:00
Szilard Nemeth 1c27c69f44 Revert "YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori"
This reverts commit de526e166c.

With this commit, the patch of
https://issues.apache.org/jira/browse/YARN-10850 has been committed.
2022-03-03 16:38:56 +01:00
Tamas Domok 22fe79cee3
YARN-11076. Upgrade jQuery version in Yarn UI2. (#4046)
Change-Id: I3cb1677741df5a1978e83029443d4a2d5d7e3d7f
2022-03-03 23:54:03 +09:00
Szilard Nemeth 356d337d1e YARN-11042. Fix testQueueSubmitWithACLsEnabledWithQueueMapping in TestAppManager. Contributed by Tamas Domok 2022-03-02 16:19:17 +01:00
9uapaw 9e475aede6 YARN-10983. Follow-up changes for YARN-10904. Contributed by Benjamin Teke 2022-03-02 11:16:24 +01:00
Szilard Nemeth 365375412f YARN-10590. Consider legacy auto queue creation absolute resource template to avoid rounding errors. Contributed by Andras Gyori 2022-02-22 12:26:35 +01:00
Szilard Nemeth 0463498adc YARN-10944. AbstractCSQueue: Eliminate code duplication in overloaded versions of setMaxCapacity. Contributed by Andras Gyori 2022-02-22 11:55:34 +01:00
Szilard Nemeth de526e166c YARN-10947. Simplify AbstractCSQueue#initializeQueueState. Contributed by Andras Gyori 2022-02-17 21:05:01 +01:00
Szilard Nemeth 7dd288ce8c YARN-11070. Minimum resource ratio is overridden by subsequent labels. Contributed by Andras Gyori 2022-02-17 19:58:14 +01:00
Szilard Nemeth ad0a1dc897 YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke 2022-02-17 19:48:42 +01:00
Szilard Nemeth 4c05d257ba YARN-10995. Move PendingApplicationComparator from GuaranteedOrZeroCapacityOverTimePolicy. Contributed by Benjamin Teke 2022-02-17 19:43:37 +01:00
Szilard Nemeth 51001fc7bf YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok 2022-02-17 19:38:18 +01:00
Szilard Nemeth 48e02f7998 YARN-10580. Fix some issues in TestRMWebServicesCapacitySchedDynamicConfig. Contributed by Tamas Domok 2022-02-17 19:17:31 +01:00
Szilard Nemeth bb1135c77c YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2022-02-17 19:12:51 +01:00
Szilard Nemeth d58c10c3a5 YARN-11071. AutoCreatedQueueTemplate incorrect wildcard level. Contributed by Tamas Domok 2022-02-17 19:03:44 +01:00
Akira Ajisaka 7fd90cdcbe
YARN-10788. TestCsiClient fails (#3989)
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.

Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2022-02-15 01:13:13 +09:00
minni31 ec2fd01333
YARN-10459. containerLaunchedOnNode method not need to hold scheduler… (#3195) 2022-02-01 09:37:40 -08:00
minni31 87abc437c7
YARN-10822. Containers going from New to Scheduled transition for kil… (#3632) 2022-02-01 09:35:59 -08:00
minni31 e8f767f2f4
YARN-11026. Make default AppPlacementAllocator configurable in AppSch… (#3741)
* YARN-11026. Make default AppPlacementAllocator configurable in AppSchedulingInfo
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-02-01 12:18:54 +05:30
Akira Ajisaka 9cb535caf2
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp (#2591)
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2022-01-28 15:51:21 +09:00
Akira Ajisaka 02f6bad1ff Revert "YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936)"
This reverts commit 1c01944f35.
2022-01-28 00:36:25 +09:00
Wei-Chiu Chuang 1c01944f35
YARN-11068. Exclude transitive log4j2 dependency coming from solr 8. (#3936)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 00:04:01 +09:00
minni31 4b26635a34
YARN-11034. Add enhanced headroom in AllocateResponse (#3766) 2022-01-25 09:25:18 -08:00
Andrew Chung 3ed3c74a6c
YARN-11015. Decouple queue capacity with ability to run OPPORTUNISTIC container (#3779) 2022-01-24 08:03:36 -08:00
dependabot[bot] dae33cf935
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui (#3890)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7)

---
updated-dependencies:
- dependency-name: follow-redirects
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-20 21:44:10 +09:00
Gera Shegalov a94e9fcbde
YARN-11055. Add missing newline in cgroups-operations.c (#3851)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-17 16:20:48 +09:00
luoyuan3471 e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791)
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
Andras Gyori 21fe0e63fd
YARN-10632. Make auto queue creation maximum allowed depth configurable (#3859) 2022-01-11 14:38:59 +08:00
minni31 3bf014d871
YARN-8859. Add audit logs for router service (#3607)
Co-authored-by: Minni Mittal <mimittal@microsoft.com>
2022-01-03 10:38:59 +05:30
Cheng Pan 4db27acf06
YARN-11053. AuxService should not use class name as default system classes (#3816)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 11:08:04 +09:00
Ashutosh Gupta 00e2405fbd
YARN-8234. Improve RM system metrics publisher's performance by pushing events to timeline server in batch (#3793)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 17:14:51 +09:00
Eric Payne e2d6fd075d YARN-10178: Global Scheduler async thread crash caused by 'Comparison method violates its general contract. Contributed by Andras Gyori (gandras) and Qi Zhu (zhuqi). 2021-12-21 19:05:39 +00:00
Viraj Jasani 6185056f4d
YARN-11047. ResourceManager and NodeManager unable to connect to Hbase when ATSv2 is enabled (#3802) 2021-12-20 18:02:57 +08:00
Viraj Jasani 04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
Szilard Nemeth 07141426e0
YARN-11050 (#3805) 2021-12-18 11:52:03 +08:00
Szilard Nemeth 8d214cb785
YARN-10951. CapacityScheduler: Move all fields and initializer code that belongs to async scheduling to a new class (#3800). Contributed by Szilard Nemeth 2021-12-17 00:18:14 +01:00
Szilard Nemeth aec9cdb467 YARN-10963. Split TestCapacityScheduler by test categories. Contributed by Tamas Domok 2021-12-16 23:39:18 +01:00
Szilard Nemeth a9a5830f31
YARN-11048. Add tests that shows how to delete config values with Mutation API (#3799). Contributed by Szilard Nemeth 2021-12-16 15:53:08 +01:00
Szilard Nemeth 6ea1685abd YARN-11044. Fix TestApplicationLimits.testLimitsComputation() ineffective asserts. Contributed by Benjamin Teke 2021-12-16 00:01:09 +01:00
Viraj Jasani f43ac31b44
YARN-11045. ATSv2 storage monitor fails to read from hbase cluster (#3796) 2021-12-15 16:47:51 +08:00
Szilard Nemeth 843f66f4dc Clean up checkstyle warnings from YARN-11024/10907/10929. Contributed by Benjamin Teke 2021-12-14 22:00:46 +01:00
Szilard Nemeth 364d38f00d YARN-10929. Do not use a separate config in legacy CS AQC. Contributed by Benjamin Teke 2021-12-14 14:57:08 +01:00
Szilard Nemeth a5bcf4c792 YARN-10907. Minimize usages of AbstractCSQueue#csContext. Contributed by Benjamin Teke 2021-12-13 21:57:46 +01:00
Szilard Nemeth 898055e204 YARN-11024. Create an AbstractLeafQueue to store the common LeafQueue + AutoCreatedLeafQueue functionality. Contributed by Benjamin Teke 2021-12-13 16:12:55 +01:00
Viraj Jasani 4e326f0c97
HADOOP-18039. Upgrade hbase2 version and fix TestTimelineWriterHBaseDown (#3768)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-13 14:03:32 +09:00
Szilard Nemeth f544ef4a51 YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok 2021-12-10 15:09:53 +01:00
Szilard Nemeth a0d8cde133 YARN-10982. Replace all occurences of queuePath with the new QueuePath class. Contributed by Tibor Kovacs 2021-12-09 17:51:44 +01:00
Andrew Chung ffee92bbf1
YARN-11005. Implement the core QUEUE_LENGTH_THEN_RESOURCES OContainer allocation policy (#3717) 2021-12-08 10:02:10 -08:00
Szilard Nemeth 66a08e933a YARN-11038. Fix testQueueSubmitWithACL* tests in TestAppManager. Contributed by Tamas Domok 2021-12-08 16:41:35 +01:00
Szilard Nemeth 19430118c6 YARN-11031. Improve the maintainability of RM webapp tests like TestRMWebServicesCapacitySched. Contributed by Tamas Domok 2021-12-08 16:03:42 +01:00
Viraj Jasani 53edd0de5a
HADOOP-18033. Upgrade fasterxml Jackson to 2.13.0 (#3749)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 16:52:22 +09:00
Szilard Nemeth 126079612c YARN-11016. Queue weight is incorrectly reset to zero. Contributed by Andras Gyori 2021-12-07 15:55:16 +01:00
Szilard Nemeth 7284d23476 YARN-11006. Allow overriding user limit factor and maxAMResourcePercent with AQCv2 templates. Contributed by Benjamin Teke 2021-12-07 15:43:55 +01:00
Szilard Nemeth 60440a836a YARN-11023. Extend the root QueueInfo with max-parallel-apps in CapacityScheduler. Contributed by Tamas Domok 2021-12-07 15:26:07 +01:00
Szilard Nemeth 7a8c2e0097 YARN-11020. [UI2] No container is found for an application attempt with a single AM container. Contributed by Andras Gyori 2021-12-07 08:51:03 +01:00
Szilard Nemeth d75f2db2f9 YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke 2021-12-07 08:42:57 +01:00
Ashutosh Gupta 5a950b8900
YARN-9063. ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing (#3728)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-06 19:45:44 +09:00
Andras Gyori 47ea0d734f
HADOOP-18021. Provide a public wrapper of Configuration#substituteVars (#3710)
Contributed by Andras Gyori
2021-12-03 16:44:58 +00:00
Viraj Jasani b34dcb5b3a
HADOOP-18025. Upgrade HBase version to 1.7.1 for hbase1 profile (#3722) 2021-12-02 10:40:30 +08:00
Sushmasree-28 7b840f2693
YARN-10975 EntityGroupFSTimelineStore#ActiveLogParser parses already processed files (#3735)
Contributed by  sravuri <sravuri@microsoft.com> and reviewed by Prabhu Joseph
2021-11-30 00:00:25 +05:30
Viraj Jasani 215388beea
HADOOP-18022. Add restrict-imports-enforcer-rule for Guava Preconditions and remove remaining usages (#3712)
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-11-29 17:37:30 +09:00
GuoPhilipse c9d64bad37
YARN-11007. Correct words in YARN documents (#3680)
Reviewed-by: cxorm <lianp964@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-25 17:48:54 +09:00
Ashutosh Gupta 08f3df3ea2
YARN-10991. Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method (#3592)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 18:41:22 +09:00
Andrew Chung d3f0b7eab7
YARN-10760. Number of allocated OPPORTUNISTIC containers can dip below 0 (#3642) 2021-11-23 13:21:51 -08:00
Andrew Chung 5b1b2c8ef6
YARN-11003. Make RMNode aware of all (OContainer inclusive) allocated resources (#3646) 2021-11-23 13:20:08 -08:00
Viraj Jasani c0bdba8fac
HADOOP-18017. unguava: remove Preconditions from hadoop-yarn-project modules (#3687) 2021-11-23 13:36:22 +09:00
Szilard Nemeth b933f5f54b YARN-11001. Add docs on removing node label mapping from a node. Contributed by Manu Zhang 2021-11-15 16:02:18 +01:00
Szilard Nemeth 7cb887e6c2 YARN-10997. Revisit allocation and reservation logging. Contributed by Andras Gyori 2021-11-12 15:43:45 +01:00
Szilard Nemeth e220e88eca YARN-10996. Fix race condition of User object acquisitions. Contributed by Andras Gyori 2021-11-12 15:33:39 +01:00
minni31 8b159d296d
[YARN-10998] Add YARN_ROUTER_HEAPSIZE to yarn-env for routers (#3603) 2021-11-11 09:49:29 -08:00
Szilard Nemeth 2e32cc68fc
YARN-10985. Add some tests to verify ACL behaviour in CapacitySchedulerConfiguration (#3570). Contributed by Szilard Nemeth 2021-10-28 15:19:32 +02:00
Szilard Nemeth d598904046
YARN-10904. Investigate: Remove unnecessary fields from AbstractCSQueue (#3551) contributed by Szilard Nemeth 2021-10-27 19:03:45 +02:00
Szilard Nemeth 66ac476b48
YARN-10924. Clean up CapacityScheduler#initScheduler (#3581) Contributed by Szilard Nemeth 2021-10-27 17:13:49 +02:00
Jack 9cfd8d0a83 YARN-10909. AbstractCSQueue: Annotate all methods with VisibleForTesting that are only used by test code. Contributed by JackWangCS, Szilard Nemeth 2021-10-23 14:47:09 +02:00
9uapaw 32ecaed9c3 YARN-10930. Introduce universal capacity resource vector. Contributed by Andras Gyori 2021-10-22 17:32:33 +02:00
Adam Antal 23772d946b YARN-10948. Rename SchedulerQueue#activeQueue to activateQueue. Contributed by Adam Antal 2021-10-22 16:33:03 +02:00
Viraj Jasani 516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
Ahmed Hussein d286994009 YARN-1115: Provide optional means for a scheduler to check real user ACLs. Contributed by Eric Payne (epayne) 2021-10-20 22:18:36 +00:00
Szilard Nemeth 20aeb5ecc3
YARN-10916. Investigate and simplify GuaranteedOrZeroCapacityOverTimePolicy#computeQueueManagementChanges. Contributed by Szilard Nemeth 2021-10-20 15:52:37 +02:00
Andras Gyori 35b8441fd9
YARN-10949. Simplify AbstractCSQueue#updateMaxAppRelatedField and find a more meaningful name for this method. Contributed by Andras Gyori 2021-10-20 12:56:41 +02:00
Szilard Nemeth 414d40155c
YARN-10958. Use correct configuration for Group service init in CSMappingPlacementRule (#3560)
* YARN-10958. Initial commit

* Fix javadoc + behaviour

* Fix review comments

* fix checkstyle + blanks

* fix checkstyle + blanks

* Fix checkstyle + blanks
2021-10-20 10:48:42 +02:00
9uapaw 616cea2e80 YARN-10954. Remove commented code block from CSQueueUtils#loadCapacitiesByLabelsFromConf. Contributed by Andras Gyori 2021-10-19 13:06:45 +02:00
Szilard Nemeth 025f97c8c2
YARN-10942. Move AbstractCSQueue fields to separate objects that are tracking usage. Contributed by Szilard Nemeth 2021-10-19 12:24:58 +02:00
lujiefsi ae95caa60e
YARN-10976. Fix resource leak due to Files.walk (#3552)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-18 15:24:15 +09:00
Viraj Jasani d336227e5c
HADOOP-17963. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-yarn-project modules (#3541) 2021-10-14 18:03:01 +09:00
Viraj Jasani 1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-14 13:07:24 +09:00
Szilard Nemeth 8c271caa81 YARN-6862. Nodemanager resource usage metrics sometimes are negative. Contributed by Benjamin Teke 2021-10-12 17:35:06 +02:00
Benjamin Teke 35eff54556
YARN-10934. Fix LeafQueue#activateApplication NPE when the user of the pending application is missing from usersManager. Contributed by Benjamin Teke
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-10-07 20:11:42 +02:00
9uapaw 4b1b6b858a
YARN-10953. Make CapacityScheduler#getOrCreateQueueFromPlacementConte… Contributed by Andras Gyori 2021-10-07 17:09:38 +02:00
Tamas Domok 5535d66fb5
YARN-10692. Do not extend from CapacitySchedulerTestBase when not needed. Contributed by Tamas Domok
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-07 16:21:05 +02:00
Viraj Jasani 8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-10-07 10:58:29 +09:00
9uapaw ed8e879320
YARN-10823. Expose all node labels for root without explicit configurations. Contributed by Andras Gyori 2021-10-01 04:20:36 +02:00
Tamas Domok 3c9e3d5321
YARN-10960. Extract test queues and related methods from TestCapacityScheduler. Contributed by Tamas Domok
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-10-01 04:05:52 +02:00
Neil 4bd0c36189
YARN-10970. Standby RM should expose prom endpoint (#3480)
Reviewed-by: Adam Antal <adamantal@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-09-29 15:46:55 +09:00
Akshat Bordia 51bd528bd3
YARN-10957. Using invokeConcurrent Overload with Collection in getClusterMetrics (#3439) 2021-09-28 09:51:18 -07:00
Tamas Domok 8f4456d4a1
YARN-10961. TestCapacityScheduler: reuse appHelper where feasible. Contributed by Tamas Domok
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-21 16:13:04 +02:00
9uapaw 9f6430c9ed
YARN-10897. Introduce QueuePath class. Contributed by Andras Gyori 2021-09-21 16:08:24 +02:00
Szilard Nemeth 4df4389325
YARN-10911. AbstractCSQueue: Create a separate class for usernames and weights that are travelling in a Map. Contributed by Szilard Nemeth 2021-09-20 16:47:46 +02:00
Adam Antal a9b2469a53
YARN-10950. Code cleanup in QueueCapacities (#3454) 2021-09-19 14:42:02 +02:00
TiborKovacsCloudera 5dcb23c1f0
YARN-10937. Fix log message arguments in LogAggregationFileController (#3450) 2021-09-19 14:35:34 +02:00
Szilard Nemeth aa74a303ed
YARN-10913. AbstractCSQueue: Group preemption methods and fields into a separate class (#3420) 2021-09-19 13:11:56 +02:00
adamantal f5c76c8e31 YARN-10936. Log typo corrected. (#3449)
Contributed by: Tibor Kovacs
2021-09-17 08:27:51 +02:00
Eric Badger 43f0a34dd4 YARN-10935. AM Total Queue Limit goes below per-user AM Limit if parent is full. Contributed by Eric Payne. 2021-09-15 20:03:45 +00:00
Benjamin Teke 5dc2f7b137
YARN-10915. AbstractCSQueue: Simplify complex logic in methods: deriveCapacityFromAbsoluteConfigurations and updateEffectiveResources (#3418)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-14 18:05:40 +02:00
Tamas Domok 783d94f5cd
YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy (#3403)
* YARN-10917. Investigate and simplify CapacitySchedulerConfigValidator#validateQueueHierarchy.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:54:25 +02:00
Tamas Domok 63c892278f
YARN-10912. AbstractCSQueue#updateConfigurableResourceRequirement: Separate validation logic from initialization logic (#3390)
- capacityConfigType update is extracted to a separate method
 - validation logic is extracted to a helper function
 - min resource must not be greater than max resource is now checked
   after the max resource is updated

Change-Id: I731c2639281721afed32c30854bafcf048d6ee28

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-14 17:30:44 +02:00
Weihao Zheng ad1d40970a
YARN-10928. Support default queue config for minimum-user-limit-percent/user-limit-factor (#3389)
Contributed by Weihao Zheng
2021-09-13 11:06:53 +08:00
Jack d8026e387e
YARN-10903. Fix the headroom check in ParentQueue and RegularContainerAllocator for DRF (#3352)
Contributed by Jie Wang <jie.wang@hulu.com>
2021-09-13 10:54:11 +08:00
Benjamin Teke 971f1b8b0a
YARN-10872. Replace getPropsWithPrefix calls in AutoCreatedQueueTemplate (#3396)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 17:32:42 +02:00
9uapaw 811fd23f23
YARN-10852. Optimise CSConfiguration getAllUserWeightsForQueue (#3392) 2021-09-10 16:59:46 +02:00
Benjamin Teke b229e5a345
YARN-10910. AbstractCSQueue#setupQueueConfigs: Separate validation logic from initialization logic (#3407)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-10 16:48:58 +02:00
Tamas Domok 29a6f141d4
YARN-10914. Simplify duplicated code for tracking ResourceUsage in AbstractCSQueue (#3402)
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-10 15:57:46 +02:00
Akshat Bordia dee6dc2f89
YARN-10893. Adding metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor (#3325) 2021-09-09 09:20:57 -07:00
Akshat Bordia a186460004
YARN-10829. Follow up: Adding null checks before merging ResourceUsage Report (#3252) 2021-09-08 10:36:56 -07:00
Szilard Nemeth 2ff3fc50e4 YARN-10870. Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page. Contributed by Gergely Pollak 2021-09-08 18:01:39 +02:00
Tamas Domok 5e166898aa
YARN-10901. Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir (#3355)
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-09-08 17:04:32 +02:00
Jack 4e209a31da
YARN-10919. Remove LeafQueue#scheduler field (#3382)
Co-authored-by: Jie Wang <jie.wang@hulu.com>
2021-09-08 16:19:29 +02:00
Benjamin Teke 40e639ad07
YARN-10646. TestCapacitySchedulerWeightMode test descriptor comments doesnt reflect the correct scenario (#3339)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-09-08 16:11:04 +02:00
zhuqi 3024a47026
Add documentation for YARN-10623 auto refresh queue conf in CS (#3279) 2021-09-08 16:03:15 +02:00
Benjamin Teke d9cb69853b
YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler 2021-09-08 15:43:57 +02:00
Szilard Nemeth 9c8fe1e512 YARN-10576. Update Capacity Scheduler documentation with JSON-based placement mapping. Contributed by Benjamin Teke 2021-09-08 15:17:27 +02:00
Swathi Chandrashekar 9512c774c0
YARN-10884: Handle empty owners to parse log files (#3318)
* YARN 10884 : Parse log files which has empty owner

* Removing the whitespace

* Added a test case for null user

* Fixed indentations

* Fixed the indentation for test cases

Co-authored-by: Swathi C <cswathi@microsoft.com>
2021-09-07 21:37:51 +05:30
Tamas Domok 16e6030e25
YARN-10891. Extend QueueInfo with max-parallel-apps in CS. (#3314)
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-27 23:09:54 +02:00
Szilard Nemeth e06a5cb197 YARN-10838. Implement an optimised version of Configuration getPropsWithPrefix. Contributed by Andras Gyori, Benjamin Teke 2021-08-24 15:27:34 +02:00
srinivasst 4f3f26ce09
YARN-10873: Account for scheduled AM containers before deactivating node (#3287)
* Account for scheduled AM containers before deactivating node

* Move AM container check to separate method.

* Fix UTs

* Fix UTs

* Remove unnecessary import

* Add timeout for UT
2021-08-17 14:18:55 +05:30
Gergely Pollak 44bab51be4 YARN-10849 Clarify testcase documentation for TestServiceAM#testContainersReleasedWhenPreLaunchFails. Contributed by Szilard Nemeth 2021-08-04 15:43:57 +02:00
Tamas Domok 0ba6f35dc2
YARN-10355. Refactor NM ContainerLaunch.java#orderEnvByDependencies
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-04 15:27:46 +02:00
Tamas Domok a5811dda7b
YARN-10874. Refactor NM ContainerLaunch#getEnvDependencies's unit tests (#3248)
Co-authored-by: Tamas Domok <tdomok@cloudera.com>
2021-08-03 16:19:04 +02:00
Viraj Jasani ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-08-03 14:44:00 +09:00
zhuqi-lucas efb3fa2bf5 YARN-10854. Support marking inactive node as untracked without configured include path. Contributed by Tao Yang. 2021-08-02 18:23:33 +08:00
Benjamin Teke 8f750c54aa
YARN-10856. Prevent ATS v2 health check REST API call if the ATS service itself is disabled. (#3236)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 19:15:27 +02:00
Benjamin Teke ac0a4e7f58
YARN-10869. CS considers only the default maximum-allocation-mb/vcore property as a maximum when it creates dynamic queues (#3225)
Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-29 17:56:14 +02:00
minni31 d78b300ed4
YARN-10841. Fix token reset synchronization for UAM response token. (#3194)
YARN-10841. Fix token reset synchronization for UAM response token.  Contributed by Minni Mittal
2021-07-29 14:55:39 +05:30
Szilard Nemeth f2b6c03fc1 YARN-6272. TestAMRMClient#testAMRMClientWithContainerResourceChange fails intermittently. Contributed by Andras Gyory & Prabhu Joseph 2021-07-28 17:02:15 +02:00
Szilard Nemeth 1b9efe58c9 YARN-10790. CS Flexible AQC: Add separate parent and leaf template property. Contributed by Andras Gyori 2021-07-28 16:50:14 +02:00
Szilard Nemeth 8d0297c213 YARN-10727. ParentQueue does not validate the queue on removal. Contributed by Andras Gyori 2021-07-28 14:49:10 +02:00
9uapaw b4a524722a
YARN-9551. TestTimelineClientV2Impl.testSyncCall fails intermittent (#3212)
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-27 19:35:20 -07:00
Szilard Nemeth fa0289b022 YARN-6221. Entities missing from ATS when summary log file info got returned to the ATS before the domain log. Contributed by Xiaomin Zhang 2021-07-27 16:47:41 +02:00
dependabot[bot] b7431c3ed5
[UI2] Bump http-proxy to 1.18.1 (#2891)
Bumps [http-proxy](https://github.com/http-party/node-http-proxy) from 1.18.0 to 1.18.1.
- [Release notes](https://github.com/http-party/node-http-proxy/releases)
- [Changelog](https://github.com/http-party/node-http-proxy/blob/master/CHANGELOG.md)
- [Commits](https://github.com/http-party/node-http-proxy/compare/1.18.0...1.18.1)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-26 02:16:21 +09:00
Benjamin Teke 05b6a1a06a
YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts. (#3203)
* YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

* fixup: YARN-10833. Set the X-FRAME-OPTIONS header for the default contexts.

Co-authored-by: Benjamin Teke <bteke@cloudera.com>
2021-07-24 11:44:21 +08:00
Akshat Bordia aa1a5dd413
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
YARN-10829. Support getApplications API in FederationClientInterceptor (#3135)
2021-07-23 17:07:51 +05:30
zhuqi-lucas 2da9b95d4d YARN-10657. We should make max application per queue to support node label. Contributed by Andras Gyori. 2021-07-22 20:30:43 +08:00
zhuqi-lucas 0441efe1fc YARN-10860. Make max container per heartbeat configs refreshable. Contributed by Eric Badger. 2021-07-21 15:31:44 +08:00
9uapaw e634bf33f3
YARN-10630. [UI2] Ambiguous queue name resolution (#3214)
Co-authored-by: Andras Gyori <gandras@cloudera.com>
2021-07-21 10:06:46 +08:00
Masatake Iwasaki 17bf2fcbc5
YARN-10858. [UI2] YARN-10826 breaks Queue view. (#3213) 2021-07-20 11:52:52 +08:00
zhuqi-lucas 0ac443b1f8 YARN-10855. yarn logs cli fails to retrieve logs if any TFile is corrupt or empty. Contributed by Jim Brennan. 2021-07-19 10:11:20 +08:00
Viraj Jasani d0ee065cc4
HADOOP-16272. Upgrade HikariCP to 4.0.3 (#3204)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-16 12:17:17 +09:00
Jim Brennan 632f64cadb YARN-10456. RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry. Contributed by Eric Payne. 2021-07-15 14:23:31 +00:00
Artem Smotrakov ba325a8ada
HADOOP-17793. Better token validation (#3189)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-07-10 13:42:31 +09:00
Viraj Jasani 618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171)
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-07-08 16:03:40 +09:00
Jim Brennan dc6f456e95 YARN-10834. Intra-queue preemption: apps that don't use defined custom resource won't be preempted. Contributed by Eric Payne. 2021-06-28 14:52:19 +00:00
Prabhu Joseph 6b4987f09c YARN-10820. Make GetClusterNodesRequestPBImpl thread safe. Contributed by Swathi Chandrashekar. 2021-06-27 16:28:32 +05:30
Jim Brennan 7c7d02edbd YARN-10824. Title not set for JHS and NM webpages. Contributed by Bilwa S T. 2021-06-25 20:32:08 +00:00
Peter Bacsko 0934e783cf YARN-10780. Optimise retrieval of configured node labels in CS queues. Contributed by Andras Gyori. 2021-06-24 20:15:10 +02:00
Masatake Iwasaki 11d144d228
YARN-10826. [UI2] Upgrade Node.js to v12.22.1. (#3128) 2021-06-23 19:33:03 +09:00
Szilard Nemeth 6562391737 YARN-10813. Set default capacity of root for node labels. Contributed by Andras Gyori 2021-06-16 18:55:09 +02:00
Szilard Nemeth 428478bbe2 YARN-10801. Fix Auto Queue template to properly set all configuration properties. Contributed by Andras Gyori 2021-06-16 18:26:58 +02:00
Jim Brennan 9a6a11c452 YARN-10767. Yarn Logs Command retrying on Standby RM for 30 times. Contributed by D M Murali Krishna Reddy. 2021-06-15 18:58:42 +00:00
Szilard Nemeth e31d06032b YARN-10802. Change Capacity Scheduler minimum-user-limit-percent to accept decimal values. Contributed by Benjamin Teke 2021-06-14 22:33:04 +02:00
Szilard Nemeth f0bdc422aa YARN-10816. Avoid doing delegation token ops when yarn.timeline-service.http-authentication.type=simple. Contributed by Tarun Parimi 2021-06-12 15:18:41 +02:00
Szilard Nemeth 7003997e36 YARN-10789. RM HA startup can fail due to race conditions in ZKConfigurationStore. Contributed by Tarun Parimi 2021-06-12 14:49:52 +02:00