853ffb545a
HADOOP-18515. Backport HADOOP-17612 to branch-3.3(Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0) ( #5097 )
...
* HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241 )
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Co-authored-by: Viraj Jasani <vjasani@apache.org>
Co-authored-by: Melissa You <myou@myou-mn1.linkedin.biz>
2022-11-05 09:28:24 -07:00
cbac2c4875
HDFS-16716. Improve appendToFile command: support appending on file with new block ( #4697 )
...
Reviewed-by: xuzq <15040255127@163.com>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-10-27 19:11:51 +08:00
53143409a8
HDFS-16822. HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API. ( #5079 )
...
Reviewed-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
(cherry picked from commit 545a556883
2022-10-27 14:40:07 +09:00
36a0e818ec
HDFS-16016. BPServiceActor to provide new thread to handle IBR ( #2998 )
...
Contributed by Viraj Jasani
(cherry picked from commit c1bf3cb0da
2022-10-24 15:16:38 +09:00
198bc444de
HDFS-16566 Erasure Coding: Recovery may causes excess replicas when busy DN exsits ( #4252 ) ( #5059 )
...
(cherry picked from commit 9376b65989
2022-10-22 13:14:04 +09:00
237814a9b3
HDFS-16480. Fix typo: indicies -> indices ( #4020 )
...
(cherry picked from commit 5eab9719cb
2022-10-21 17:32:58 +09:00
0c2234fd8e
HDFS-15803. EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo. Contributed by huhaiyang
...
(cherry picked from commit 66ecee333e
2022-10-21 17:31:30 +09:00
75b04010a2
HDFS-16795. Use secure XML parsers ( #4979 )
...
Move construction of XML parsers in HDFS
modules to using the locked-down parser factory
of HADOOP-18469.
Contributed by P J Fanning
2022-10-20 17:48:58 +01:00
ceec19e61a
HDFS-16776 Erasure Coding: The length of targets should be checked when DN gets a reconstruction task ( #4901 )
...
(cherry picked from commit 9a29075f91
2022-09-23 12:29:39 +09:00
357c83db94
HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request ( #4724 ) ( #4794 )
2022-09-13 10:50:23 -07:00
3a6c8ff8bb
HDFS-16755. TestQJMWithFaults.testUnresolvableHostName() can fail due to unexpected host resolution ( #4833 )
...
Use ".invalid" domain from IETF RFC 2606 to ensure that the host doesn't resolve.
Contributed by Steve Vaughan Jr
2022-09-01 14:01:26 +01:00
833fc64558
HDFS-16684. Exclude the current JournalNode ( #4786 )
...
The JournalNodeSyncer will include the local instance in syncing when using a bind host (e.g. 0.0.0.0). There is a mechanism that is supposed to exclude the local instance, but it doesn't recognize the meta-address as a local address.
Running with bind addresses set to 0.0.0.0, the JournalNodeSyncer will log attempts to sync with itself as part of the normal syncing rotation. For an HA configuration running 3 JournalNodes, the "other" list used by the JournalNodeSyncer will include 3 proxies.
Exclude bound local addresses, including the use of a wildcard address in the bound host configurations, while still allowing multiple instances on the same host.
Allow sync attempts with unresolved addresses, so that sync attempts can drive resolution as servers become available.
Backport.
Signed-off-by: stack <stack@apache.org>
2022-08-28 11:15:04 -07:00
3edddaf9fc
HDFS-16732. [SBN READ] Avoid get location from observer when the block report is delayed ( #4756 )
...
Signed-off-by: Erik Krogen <xkrogen@apache.org>
(cherry picked from commit 231a4468cd
2022-08-25 10:41:04 -07:00
4138661010
HDFS-16625. Check assumption about PMDK availability ( #4788 )
...
Co-authored-by: Steve Vaughan Jr <s_vaughan@apple.com>
2022-08-23 19:35:59 +09:00
9c5228cf6b
HADOOP-18305. Release Hadoop 3.3.4: upstream changelog and jdiff files
...
Add the r3.3.4 changelog, release notes and jdiff xml files.
Change-Id: I98b0fed54da3b810c3f23fe5b12e673937916257
2022-08-05 14:02:28 +01:00
e024d1a3f8
HDFS-16712. Fix incorrect placeholder in DataNode.java ( #4672 ). Contributed by ZanderXu.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-08-03 13:02:42 +05:30
ff13f9ee8b
Make upstream aware of 3.2.4 release.
...
(cherry picked from commit e1637a57df
2022-07-22 02:31:34 +00:00
4ba463069b
HADOOP-18288. Total requests and total requests per sec served by RPC servers ( #4485 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-23 17:30:01 +08:00
9ca4ac0af0
HADOOP-18305. Preparing for 3.3.4 release: branch-3.3 version => 3.3.9 ( #4482 )
...
Updating the hadoop version of branch-3.3 to 3.3.9-SNAPSHOT
pending agreement on what number its future release should take.
Using 3.3.9-SNAPSHOT puts space in for other incremental releases,
while avoiding creating JIRA release ordering and autocompletion
confusion the way adding a 3.3.10 or higher version would do.
Contributed by Steve Loughran
2022-06-22 13:09:50 +01:00
7561dbd134
HDFS-16637. TestHDFSCLI#testAll consistently failing ( #4466 ). Contributed by Viraj Jasani.
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-21 13:44:30 +05:30
33ab84f2e2
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas ( #4410 )
...
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6
2022-06-20 11:24:45 +09:00
18a5e843bc
HDFS-16581.Print node status when executing printTopology. ( #4321 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-16 19:20:34 +08:00
ee3ee98ee5
HDFS-16623. Avoid IllegalArgumentException in LifelineSender ( #4409 )
...
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender
Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473
2022-06-10 19:02:47 +00:00
bdef321d52
HDFS-16576. Remove unused imports in HDFS project ( #4389 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6e11c94170
2022-06-09 22:42:04 +09:00
a2f8a9e5d8
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig ( #4412 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-09 09:59:34 +08:00
516a2a8e44
HDFS-16618. sync_file_range error should include more volume/file info ( #4402 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-07 16:56:07 +08:00
132fbbe228
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits ( #4357 ) ( #4405 )
...
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-06-07 06:41:16 +08:00
7223a337f6
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery ( #4379 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7f08ed0d1d
2022-06-06 18:02:44 +09:00
7d6b133af3
HDFS-16610. Make fsck read timeout configurable ( #4384 )
...
(cherry picked from commit 34a973a90e
2022-06-01 20:54:56 +01:00
d921cc71fd
HDFS-16585.Add @VisibleForTesting in Dispatcher.java ( #4337 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785
2022-05-26 15:28:27 -07:00
55ba3a7944
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock ( #4332 )
...
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
(cherry picked from commit 297f0f6d6a
2022-05-26 10:14:50 -07:00
ba856bff95
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication ( #4126 ) ( #4304 )
...
(cherry picked from commit cee8c62498
2022-05-27 00:50:40 +08:00
ae9d671232
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… ( #4347 )
...
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
2022-05-25 17:02:28 -07:00
fe6b050857
HDFS-16588. Backport HDFS-16584 to branch-3.3. ( #4342 ). Contributed by JiangHua Zhu.
...
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-05-24 23:47:45 +08:00
ab3a9cedc9
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node ( #4323 )
...
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-21 09:47:18 +08:00
fe306ce57e
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
...
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version
Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:09 +01:00
1d226874d2
HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes ( #4170 ) ( #4246 )
...
Cherry-pick backport of 9ed8d60511
2022-05-15 21:32:48 -07:00
e16ae55833
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes ( #4264 )
...
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-11 09:00:31 +08:00
bb398f78cc
Fix TestDFSStripedInputStreamWithRandomECPolicy ( #4296 )
2022-05-11 07:56:53 +08:00
c879230547
HDFS-16185. Fix comment in LowRedundancyBlocks.java ( #4194 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9
2022-05-08 08:06:14 +09:00
62c6a08ffd
HADOOP-16515. Update the link to compatibility guide ( #4226 )
...
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ae47846a5b
2022-05-08 07:39:17 +09:00
d711d200fe
HDFS-16520. Improve EC pread: avoid potential reading whole block ( #4104 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 29401c8203
2022-05-06 10:35:55 -07:00
4d935eaed7
HDFS-16521. DFS API to retrieve slow datanodes ( #4107 ) ( #4259 )
2022-05-06 04:55:46 +08:00
ee450bbbc6
HDFS-16528. Reconfigure slow peer enable for Namenode ( #4251 )
2022-05-02 08:03:02 +08:00
bb13e228bc
HADOOP-17956. Replace all default Charset usage with UTF-8 ( #3529 )
...
Change-Id: I0094a84619ce19acf340d8dd1040cfe9bd88184e
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-27 10:30:07 +01:00
b62a460fd9
HDFS-16519. Add throttler to EC reconstruction ( #4101 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit aebd55f788
2022-04-23 12:47:34 +09:00
9132eeb4dd
HDFS-16552. Fix NPE for TestBlockManager ( #4210 )
...
(cherry picked from commit 5ebbacc480
2022-04-23 12:15:52 +09:00
5e137ac33e
Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value ( #4148 ). Contributed by Stephen O'Donnell."
...
This reverts commit 8ae033d1a3
2022-04-20 20:45:17 +01:00
2ff91232bc
HDFS-16544. EC decoding failed due to invalid buffer ( #4179 )
...
(cherry picked from commit 76bbd17374
2022-04-20 15:07:38 +09:00
8ae033d1a3
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value ( #4148 ). Contributed by Stephen O'Donnell.
...
(cherry picked from commit dbeeee0363
2022-04-19 11:12:36 +01:00
Melissa You
M1eyu2018
Takanobu Asanuma
Viraj Jasani
SevenAddSix
Hui Fei
Steve Loughran
Kidd5368
Steve Vaughan
zhengchenyu
xuzq
Masatake Iwasaki
KevinWikant
jianghuazhu
Ashutosh Gupta
slfan1989
Samrat
Stephen O'Donnell
Wei-Chiu Chuang
Michael Stack
daimin
litao
S O'Donnell
qinyuren