fb4e8172a0
MAPREDUCE-7391. TestLocalDistributedCacheManager failing after HADOOP-16202 ( #4472 )
...
Fixing a mockito-based test which broke when HADOOP-16202
changed the methods being invoked.
Contributed by Steve Loughran
2022-06-22 13:13:24 +01:00
53a530aa88
MAPREDUCE-7371. DistributedCache alternative APIs should not use DistributedCache APIs internally ( #3855 )
...
Contributed by Viraj Jasani
2022-06-22 13:13:05 +01:00
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 ( #4482 )
...
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.
Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.
Contributed by Steve Loughran
2022-06-22 13:09:50 +01:00
49342cffdb
HADOOP-18159. Bump cos_api-bundle to 5.6.69 to update public-suffix-list.txt ( #4444 )
...
Bump cos_api-bundle to 5.6.69
All copies of httpclient, including shaded ones in libraries used
by the s3a, gs and cos cloud connectors, turn out to load their
TLD list from the same resource mozilla/public-suffix-list.txt
Updating the hadoop-cos dependency ensures that its version
of public-suffix-list.txt is up to date -and so the s3a connector
able to talk to s3 resources if the cos-api-bundle JAR is where
the resource is loaded from.
Contributed by André Fonseca
2022-06-21 20:17:08 +01:00
aeb2a2f860
HADOOP-17833. Improve Magic Committer performance ( #3289 ) ( #4470 )
...
Speed up the magic committer with key changes being
* Writes under __magic always retain directory markers
* File creation under __magic skips all overwrite checks,
including the LIST call intended to stop files being
created over dirs.
* mkdirs under __magic probes the path for existence
but does not look any further.
Extra parallelism in task and job commit directory scanning
Use of createFile and openFile with parameters which all for
HEAD checks to be skipped.
The committer can write the summary _SUCCESS file to the path
`fs.s3a.committer.summary.report.directory`, which can be in a
different file system/bucket if desired, using the job id as
the filename.
Also: HADOOP-15460. S3A FS to add `fs.s3a.create.performance`
Application code can set the createFile() option
fs.s3a.create.performance to true to disable the same
safety checks when writing under magic directories.
Use with care.
The createFile option prefix `fs.s3a.create.header.`
can be used to add custom headers to S3 objects when
created.
Contributed by Steve Loughran.
2022-06-21 10:49:37 +01:00
7561dbd134
HDFS-16637. TestHDFSCLI#testAll consistently failing ( #4466 ). Contributed by Viraj Jasani.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-21 13:44:30 +05:30
4f860f8ac2
MAPREDUCE-7369. Fixed MapReduce tasks timing out when spends more time on MultipleOutputs#close ( #4247 )
...
Contributed by Ravuri Sushma sree.
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
(cherry picked from commit 36c4be819f
2022-06-20 08:02:58 +00:00
43f4a0e92d
MAPREDUCE-7387. Fix TestJHSSecurity#testDelegationToken AssertionError due to HDFS-16563 ( #4428 ). Contributed by fanshilun.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-20 12:16:33 +05:30
33ab84f2e2
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas ( #4410 )
...
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6
2022-06-20 11:24:45 +09:00
d7de378b22
YARN-11172. Fix TestClientRMTokens#testDelegationToken introduced by HDFS-16563. ( #4408 )
...
Regression caused by HDFS-16563; the hdfs exception text was changed, but because it was
a YARN test doing the check, Yetus didn't notice.
Contributed by zhengchenyu
2022-06-17 19:51:56 +01:00
18a5e843bc
HDFS-16581.Print node status when executing printTopology. ( #4321 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-16 19:20:34 +08:00
ee3ee98ee5
HDFS-16623. Avoid IllegalArgumentException in LifelineSender ( #4409 )
...
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473
2022-06-10 19:02:47 +00:00
de9f994338
YARN-11173. remove redeclaration of os-maven-plugin.version from yarn-csi ( #4417 )
...
This is a followup to HADOOP-18275 and its upgrade of os-maven-plugin.version
When that change is merged in, this MUST follow it.
Contributed by Steve Loughran
Change-Id: I61d087041561eeb8c9c42b5b7d8f0bb63f296b15
2022-06-10 17:03:25 +01:00
bdef321d52
HDFS-16576. Remove unused imports in HDFS project ( #4389 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170
2022-06-09 22:42:04 +09:00
a2f8a9e5d8
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig ( #4412 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-09 09:59:34 +08:00
7ec988d264
HADOOP-12020. Add s3a storage class option fs.s3a.create.storage.class ( #3877 )
...
Adds a new option fs.s3a.create.storage.class which can
be used to set the storage class for files created in AWS S3.
Consult the documentation for details and instructions on how
disable the relevant tests when testing against third-party
stores.
Contributed by Monthon Klongklaew
Change-Id: I8cdebadf294a89fde08d98729ad96f251d58411c
2022-06-08 20:02:07 +01:00
516a2a8e44
HDFS-16618. sync_file_range error should include more volume/file info ( #4402 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-07 16:56:07 +08:00
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits ( #4357 ) ( #4405 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-06-07 06:41:16 +08:00
03c2941d4b
HADOOP-18275. Update os-maven-plugin to 1.7.0 ( #4397 )
...
Contributed by Steve Loughran
Change-Id: Ic4d442a37299dc8098b0bca3cc51beca6f058283
2022-06-06 13:20:00 +01:00
0c15daa77a
HDFS-16563. Namenode WebUI prints sensitive information on Token expiry ( #4241 )
...
Contributed by Renukaprasad C
Change-Id: I5cd2cec1dd79917f810207821b3bdf4fe1a5d24c
2022-06-06 11:08:57 +01:00
7223a337f6
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery ( #4379 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1d
2022-06-06 18:02:44 +09:00
7d6b133af3
HDFS-16610. Make fsck read timeout configurable ( #4384 )
...
(cherry picked from commit 34a973a90e
2022-06-01 20:54:56 +01:00
de4c975710
HADOOP-18238. Fix reentrancy check in SFTPFileSystem.close() ( #4330 )
...
Contributed by Ashutosh Gupta
Change-Id: I2742675add74259a93b3762a80c7ab5ee6d08c37
2022-05-30 17:34:45 +01:00
dd9b8af9c4
HADOOP-18269. Misleading method name in DistCpOptions.( #4216 )
...
Contributed by guophilipse
Change-Id: I5e75d030406997339c20e970483825e529d9cd10
2022-05-30 14:04:33 +01:00
91f19bf8fa
HADOOP-18244. Fix Hadoop-Common JavaDoc Error on branch-3.3 ( #4327 ). Contributed by fanshilun.
2022-05-29 11:31:16 +05:30
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java ( #4337 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785
2022-05-26 15:28:27 -07:00
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock ( #4332 )
...
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a
2022-05-26 10:14:50 -07:00
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication ( #4126 ) ( #4304 )
...
(cherry picked from commit cee8c62498
2022-05-27 00:50:40 +08:00
e0732baeb8
YARN-11128. Fix comments in TestProportionalCapacityPreemptionPolicy* ( #4271 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e3e9369c1d
2022-05-26 15:58:47 +09:00
241fb6b2a7
HADOOP-18251. Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py. ( #4344 )
...
(cherry picked from commit 6b331dde31
2022-05-26 03:28:55 +00:00
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… ( #4347 )
...
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
2022-05-25 17:02:28 -07:00
8c492a1d65
HADOOP-18240. Upgrade Yetus to 0.14.0 ( #4328 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Chris Nauroth <cnauroth@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 84b0455cf8
2022-05-25 17:32:19 +09:00
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. ( #4342 ). Contributed by JiangHua Zhu.
...
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-05-24 23:47:45 +08:00
1f111d6a41
YARN-11162. Set the zk acl for nodes created by ZKConfigurationStore. ( #4350 )
...
(cherry picked from commit f390edaec4
2022-05-24 05:17:34 +00:00
ab3a9cedc9
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node ( #4323 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-21 09:47:18 +08:00
57fe613299
HDFS-16453. Upgrade okhttp from 2.7.5 to 4.9.3 ( #4229 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit fb910bd906
2022-05-21 03:17:15 +09:00
90ec4418c7
YARN-11141. Capacity Scheduler does not support ambiguous queue names when moving application across queues. Contributed by Andras Gyori
2022-05-18 14:34:08 +02:00
4f112e3138
YARN-11126. ZKConfigurationStore Java deserialisation vulnerability. Contributed by Tamas Domok
2022-05-18 14:25:35 +02:00
b4550b3356
YARN-10850. TimelineService v2 lists containers for all attempts when filtering for one. Contributed by Benjamin Teke
2022-05-18 14:08:41 +02:00
b0012ee578
HADOOP-18237. Upgrade Apache Xerces Java to 2.12.2 ( #4318 )
...
Upgrade Apache Xerces Java to 2.12.2 due to handle vulnerability CVE-2022-23437
Contributed by Ashutosh Gupta
2022-05-17 20:37:30 +01:00
2dd5852ad0
YARN-11133. YarnClient gets the wrong EffectiveMinCapacity value ( #4280 )
...
(cherry picked from commit 79cc0caf9f
2022-05-18 03:03:13 +09:00
fe306ce57e
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
...
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version
Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:09 +01:00
c2071947f8
HADOOP-18234. Fix s3a access point xml examples ( #4309 )
...
Contributed by Ashutosh Gupta
Change-Id: Ib99a4c7113d60690c1301d80cd5c27a43c463db0
2022-05-16 17:48:04 +01:00
a50ce18f4d
YARN-11092. Upgrade jquery ui to 1.13.1 ( #4260 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 931abbd927
2022-05-16 23:26:46 +09:00
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes ( #4170 ) ( #4246 )
...
Cherry-pick backport of 9ed8d60511
2022-05-15 21:32:48 -07:00
d58f9d438f
YARN-10080. Support show app id on localizer thread pool ( #4283 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6985f9aabe
2022-05-14 01:41:31 +09:00
e16ae55833
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes ( #4264 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-11 09:00:31 +08:00
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy ( #4296 )
2022-05-11 07:56:53 +08:00
1e043b937a
HADOOP-18222. Prevent DelegationTokenSecretManagerMetrics from registering multiple times
...
Fixes #4266
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-05-10 14:52:31 -07:00
277daca91f
HADOOP-17479. Fix the examples of hadoop config prefix ( #4197 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 40a8b9a6a5
2022-05-08 08:09:47 +09:00
Steve Loughran
Viraj Jasani
André Fonseca
Ashutosh Gupta
slfan1989
KevinWikant
zhengchenyu
jianghuazhu
xuzq
monthonk
Renukaprasad C
Samrat
Stephen O'Donnell
GuoPhilipse
Wei-Chiu Chuang
Masatake Iwasaki
Michael Stack
Owen O'Malley
Szilard Nemeth
Zilong Zhu
daimin
hchaverr