856e483592
YARN-11022. Fix the documentation for max-parallel-apps in CS. Contributed by Tamas Domok
2022-03-02 16:13:35 +01:00
192f53283b
YARN-10894. Follow up YARN-10237: fix the new test case in TestRMWebServicesCapacitySched. Contributed by Tamas Domok
2022-03-02 16:04:38 +01:00
3ef3c5a05b
YARN-11033. isAbsoluteResource is not correct for dynamically created queues. Contributed by Tamas Domok
2022-03-02 14:45:31 +01:00
f06f44b1c2
YARN-11014. YARN incorrectly validates maximum capacity resources on the validation API. Contributed by Benjamin Teke
2022-03-02 14:23:00 +01:00
935619a28c
YARN-11075. Explicitly declare serialVersionUID in LogMutation class. Contributed by Benjamin Teke
2022-03-01 18:05:04 +01:00
36a50ba3e0
HADOOP-18075. ABFS: Fix failure caused by listFiles() in ITestAbfsRestOperationException ( #4040 )
...
Contributed by Sumangala Patki
Change-Id: I245c08dab050d59b90ac6fdcb4c03153db77be0b
2022-03-01 13:48:39 +00:00
0ed0375413
HADOOP-17862. ABFS: Fix unchecked cast compiler warning for AbfsListStatusRemoteIterator ( #3331 )
...
closes #3331
Contributed by Sumangala Patki
Change-Id: I6cca91c8bcc34052c5233035f14a576f23086067
2022-03-01 13:48:39 +00:00
5e109705ef
HADOOP-17765. ABFS: Use Unique File Paths in Tests. ( #3153 )
...
Contributed by Sumangala Patki
Change-Id: Ic8f34bf578069504f7a811a7729982b9c9f49729
2022-03-01 12:29:03 +00:00
74f5f90615
HDFS-16397. Reconfig slow disk parameters for datanode ( #3828 )
...
(cherry picked from commit 6b07c851f3
2022-02-26 02:22:28 +09:00
9c0bdc5aea
HDFS-16371. Exclude slow disks when choosing volume ( #3753 ) ( #4031 )
2022-02-26 02:21:07 +09:00
5601d0848a
HDFS-15854. Make some parameters configurable for SlowDiskTracker and SlowPeerTracker ( #2718 )
...
Authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit 32353eb38a
2022-02-25 11:19:27 +09:00
1a3060d41e
HADOOP-18139: Allow configuration of zookeeper server principal.
...
Fixes #4024
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-02-24 15:15:13 -08:00
a1319e2404
HADOOP-18071. ABFS: Set driver global timeout for ITestAzureBlobFileSystemBasics ( #3866 )
...
Contributed by Sumangala Patki
Change-Id: I05f0cd1f0bd277b90f06a71345c46bfde48d7e7e
2022-02-23 21:30:39 +00:00
fa30224e95
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang.
...
(cherry picked from commit e8cb2ae409
2022-02-23 11:41:19 +08:00
d763c99707
HADOOP-18125. Utility to identify git commit / Jira fixVersion discrepancies for RC preparation ( #3991 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 697e5d4636
2022-02-22 11:01:35 +08:00
94a0a04113
HADOOP-18136. Verify FileUtils.unTar() handling of missing .tar files.
...
Contributed by Steve Loughran
Change-Id: I3856afa821dbc8c2e3cb1cbe33793ec1734e2e24
2022-02-21 17:09:36 +00:00
a302a19b48
HADOOP-18126. update junit 5 version due to build issues ( #3993 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 5f6a294fab
2022-02-17 14:07:57 +09:00
d14a7c6ee5
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953)
...
(cherry picked from commit 19d90e62fb
2022-02-15 16:48:13 -08:00
db67952f9f
HDFS-16396. Reconfig slow peer parameters for datanode ( #3827 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0c194f2157
2022-02-16 09:45:07 +09:00
4c57fb4d6b
HDFS-15745. Make DataNodePeerMetrics#LOW_THRESHOLD_MS and MIN_OUTLIER_DETECTION_NODES configurable. Contributed by Haibin Huang. ( #3992 )
...
(cherry picked from commit 1cd96e8dd8
2022-02-16 09:42:43 +09:00
352656999f
YARN-10788. TestCsiClient fails ( #3989 )
...
Create unix domain socket in java.io.tmpdir instead of
test.build.dir to avoid 'File name too long' error.
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 7fd90cdcbe
2022-02-15 01:14:31 +09:00
7512714475
HDFS-16449. Fix hadoop web site release notes and changelog not available ( #3967 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit b68964336d
2022-02-14 05:40:16 +09:00
752a7b6d49
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
(cherry picked from commit e2d620192a
2022-02-11 23:18:25 +08:00
9071c9646c
Fix thread safety of EC decoding during concurrent preads ( #3881 )
...
(cherry picked from commit 0e74f1e467
2022-02-11 10:20:45 +08:00
5b47b9f360
HADOOP-18096. Distcp: Sync moves filtered file to home directory rather than deleting. ( #3940 ). Contributed by Ayush Saxena.
...
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: stack <stack@apache.org>
2022-02-11 02:05:14 +05:30
088684ec60
HADOOP-18091. S3A auditing leaks memory through ThreadLocal references ( #3930 )
...
Adds a new map type WeakReferenceMap, which stores weak
references to values, and a WeakReferenceThreadMap subclass
to more closely resemble a thread local type, as it is a
map of threadId to value.
Construct it with a factory method and optional callback
for notification on loss and regeneration.
WeakReferenceThreadMap<WrappingAuditSpan> activeSpan =
new WeakReferenceThreadMap<>(
(k) -> getUnbondedSpan(),
this::noteSpanReferenceLost);
This is used in ActiveAuditManagerS3A for span tracking.
Relates to
* HADOOP-17511. Add an Audit plugin point for S3A
* HADOOP-18094. Disable S3A auditing by default.
Contributed by Steve Loughran.
Change-Id: Ibf7bb082fd47298f7ebf46d92f56e80ca9b2aaf8
2022-02-10 12:33:40 +00:00
84de16028d
HADOOP-18114. Documentation correction in assumed_roles.md ( #3949 )
...
Fixes typo in hadoop-aws/assumed_roles.md
Contributed by Joey Krabacher
Change-Id: I2b77bd7793ae0433196b77042d5f400d0bcbe745
2022-02-09 10:47:24 +00:00
ce7cabb771
HDFS-16437 ReverseXML processor doesn't accept XML files without the … ( #3926 )
...
(cherry picked from commit 125e3b6160
2022-02-06 13:36:57 +08:00
709e617a84
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background ( #3842 )
...
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e
2022-02-06 13:06:35 +08:00
8b03514eaf
HADOOP-18100: Change scope of inner classes in InodeTree to make them accessible outside package
...
Fixes #3950
Signed-off-by: Owen O'Malley <omalley@apache.org>
Cherry-picked from 3684c7f6
2022-02-04 12:13:10 -08:00
87ff57765a
HADOOP-18085. S3 SDK Upgrade causes AccessPoint ARN endpoint mistranslation ( #3902 )
...
Part of HADOOP-17198. Support S3 Access Points.
HADOOP-18068. "upgrade AWS SDK to 1.12.132" broke the access point endpoint
translation.
Correct endpoints should start with "s3-accesspoint.", after SDK upgrade they start with
"s3.accesspoint-" which messes up tests + region detection by the SDK.
Contributed by Bogdan Stolojan
Change-Id: I0c0181628ab803afc39036003777eaec79aa378c
2022-02-04 16:22:24 +00:00
a8d7acf1a8
HADOOP-17951. Improve S3A checking of S3 Access Point existence ( #3516 )
...
Follow-on to HADOOP-17198. Support S3 Access Points
Contributed by Bogdan Stolojan
Change-Id: I0932476c64e1967eb0cb3e0f00060fac5d2bae72
2022-02-04 16:22:04 +00:00
664075f35d
HADOOP-17198. Support S3 Access Points ( #3260 )
...
Add support for S3 Access Points. This provides extra security as it
ensures applications are not working with buckets belong to third parties.
To bind a bucket to an access point, set the access point (ap) ARN,
which must be done for each specific bucket, using the pattern
fs.s3a.bucket.$BUCKET.accesspoint.arn = ARN
* The global/bucket option `fs.s3a.accesspoint.required` to
mandate that buckets must declare their access point.
* This is not compatible with S3Guard.
Consult the documentation for further details.
Contributed by Bogdan Stolojan
(this commit contains the changes to TestArnResource from HADOOP-18068,
"upgrade AWS SDK to 1.12.132" so that it works with the later SDK.)
Change-Id: I3fac213e52ca6ec1c813effb8496c353964b8e1b
2022-02-04 16:21:35 +00:00
7171e2190e
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception ( #3942 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21
2022-01-31 13:19:40 +09:00
5e2eac6c41
HDFS-16303. Improve handling of datanode lost while decommissioning ( #3921 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-31 13:18:36 +09:00
110104da38
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure ( #3850 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f
2022-01-28 17:05:32 +09:00
8032b680fb
YARN-10561. Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp ( #2591 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9cb535caf2
2022-01-28 15:52:33 +09:00
b5d2e00f81
HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped ( #3888 )
...
(cherry picked from commit 6136d630a3
2022-01-27 13:44:03 +09:00
d613776b64
HADOOP-18093. Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java ( #3918 ). Contributed by Xing Lin. ( #3929 )
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 0d17b629ff
2022-01-26 21:55:32 +05:30
ef1a2b478b
HDFS-16398. Reconfig block report parameters for datanode ( #3831 )
...
(cherry picked from commit c2ff39006f
2022-01-26 17:36:17 +09:00
ff3a88b9c2
HDFS-16423. Balancer should not get blocks on stale storages ( #3883 ) ( #3924 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit db2c3200e6
2022-01-26 11:54:13 +08:00
bd13d73334
HDFS-16262. Async refresh of cached locations in DFSInputStream ( #3527 )
...
(cherry picked from commit 94b884ae55
2022-01-25 11:43:47 +00:00
728ed10a7c
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy ( #3899 ). Contributed by daimin.
...
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e
2022-01-25 09:58:20 +08:00
4fd0389153
HADOOP-18094. Disable S3A auditing by default.
...
See HADOOP-18091. S3A auditing leaks memory through ThreadLocal references
* Adds a new option fs.s3a.audit.enabled to controls whether or not auditing
is enabled. This is false by default.
* When false, the S3A auditing manager is NoopAuditManagerS3A,
which was formerly only used for unit tests and
during filsystem initialization.
* When true, ActiveAuditManagerS3A is used for managing auditing,
allowing auditing events to be reported.
* updates documentation and tests.
This patch does not fix the underlying leak. When auditing is enabled,
long-lived threads will retain references to the audit managers
of S3A filesystem instances which have already been closed.
Contributed by Steve Loughran.
Change-Id: I671e594cd59e8ca77a1f65be791ad0ae9530b8d9
2022-01-24 14:04:23 +00:00
55192570a1
YARN-11065. Bump follow-redirects from 1.13.3 to 1.14.7 in hadoop-yarn-ui ( #3890 )
...
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects ) from 1.13.3 to 1.14.7.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases )
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.13.3...v1.14.7 )
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dae33cf935
2022-01-20 21:45:53 +09:00
d6ff60df65
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport ( #3714 ). Contributed by liubingxing.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a
2022-01-20 18:47:00 +08:00
9b221b9599
HADOOP-18084. ABFS: Add testfilePath while verifying test contents are read correctly ( #3903 )
...
Contributed by: Anmol Asrani
Change-Id: I6e71bf349a74032f453398c7ae66f9c3305be190
2022-01-19 10:18:05 +00:00
f9c0bc094a
HDFS-16399. Reconfig cache report parameters for datanode ( #3841 )
...
(cherry picked from commit e355646330
2022-01-19 18:43:15 +09:00
11fe5279b0
HDFS-16400. Reconfig DataXceiver parameters for datanode ( #3843 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit f02374df92
2022-01-19 18:42:48 +09:00
cdaf4d89f9
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable ( #3676 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 52ec65fd10
2022-01-19 18:40:41 +09:00
Szilard Nemeth
Steve Loughran
sumangala-patki
litao
litao
Owen O'Malley
Ayush Saxena
Viraj Jasani
PJ Fanning
Chentao Yu
Takanobu Asanuma
Akira Ajisaka
GuoPhilipse
luoyuan3471
daimin
Joey Krabacher
singer-bin
Abhishek Das
Petre Bogdan Stolojan
KevinWikant
secfree
Akira Ajisaka
Xing Lin
Wei-Chiu Chuang
Bryan Beaudreault
dependabot[bot]
liubingxing
Anmol Asrani