Commit Graph

8842 Commits

Author SHA1 Message Date
KevinWikant cfceaebde6
HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410)
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-06-20 11:20:24 +09:00
Viraj Jasani cb0421095b
HDFS-16634. Dynamically adjust slow peer report size on JMX metrics (#4448)
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-20 09:21:00 +08:00
Viraj Jasani e38e13be03
HADOOP-18288. Total requests and total requests per sec served by RPC servers (#4431)
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-18 12:17:20 +08:00
xuzq 4893f00395
HDFS-16600. Fix deadlock of fine-grain lock for FsDatastImpl of DataNode. (#4367). Contributed by ZanderXu.
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-06-17 22:05:33 +08:00
caozhiqiang 9e3fc40ecb
HDFS-16613. EC: Improve performance of decommissioning dn with many ec blocks (#4398) 2022-06-17 02:11:25 +08:00
jianghuazhu 6cbeae2e52
HDFS-16581.Print node status when executing printTopology. (#4321)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-16 19:18:58 +08:00
xuzq d0715b1024
HDFS-16598. Fix DataNode FsDatasetImpl lock issue without GS checks. (#4366). Contributed by ZanderXu.
Reviewed-by: Mingxiang Li <liaiphag0@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-06-14 21:48:03 +08:00
Steve Vaughan bebf03a66e
HDFS-16625. Check assumption about PMDK availability (#4414)
Signed-off-by: Ashutosh Gupta <ashutosh.gupta@st.niituniversity.in>
Signed-off-by: stack <stack@apache.org>
2022-06-13 23:42:18 -07:00
slfan1989 170668b994
HDFS-16609. Fix Flakes Junit Tests that often report timeouts. (#4382). Contributed by fanshilun. 2022-06-11 20:11:21 +08:00
slfan1989 02c21ef8c3
HDFS-16627. Improve BPServiceActor#register log to add NameNode address. (#4419). Contributed by fanshilun. 2022-06-11 20:04:39 +08:00
xuzq af5003a473
HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409)
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
2022-06-10 12:00:56 -07:00
jianghuazhu 7f5a34dfaa
HDFS-16621.Remove unused JNStorage#getCurrentDir() (#4404). Contributed by JiangHua Zhu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-06-10 12:12:41 +05:30
Ashutosh Gupta 6e11c94170
HDFS-16576. Remove unused imports in HDFS project (#4389)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-06-09 22:39:00 +09:00
slfan1989 2ac01b273a
HDFS-16624. Fix flaky unit test TestDFSAdmin#testAllDatanodesReconfig (#4412)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-09 09:34:25 +08:00
Viraj Jasani 0a65883c50
HDFS-16618. sync_file_range error should include more volume/file info (#4402)
Signed-off-by: Tao Li <tomscut@apache.org>
2022-06-07 16:53:45 +08:00
Samrat 7f08ed0d1d
HDFS-16608. Fix the link in TestClientProtocolForPipelineRecovery (#4379)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-06-06 17:58:49 +09:00
Viraj Jasani 25591ef51b
HDFS-16595. Slow peer metrics - add median, mad and upper latency limits (#4357)
Reviewed-by: Tao Li <tomscut@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-06-04 06:10:14 +08:00
Stephen O'Donnell 34a973a90e
HDFS-16610. Make fsck read timeout configurable (#4384) 2022-06-01 20:36:01 +01:00
slfan1989 e85f827f0a
HDFS-16603. Improve DatanodeHttpServer With Netty recommended method. (#4372)
Co-authored-by: slfan1989 <louj1988@@>
2022-06-01 06:45:07 +08:00
Masatake Iwasaki 3228142f53 Make upstream aware of 2.10.2 release.
(cherry picked from commit 3dcb6367ed)

Conflicts:
	hadoop-project-dist/pom.xml
2022-06-01 00:53:01 +09:00
xuzq dc9b21feac
HDFS-16587. Allow configuring the number of JournalNodeRPCServer Handlers (#4339). Contributed by ZanderXu.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-05-28 09:43:37 +05:30
Ashutosh Gupta bee538f785
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-05-27 05:18:15 +08:00
Stephen O'Donnell 297f0f6d6a
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop holding the write lock (#4332)
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
2022-05-27 01:11:10 +08:00
slfan1989 f469b0e143
HADOOP-18249. Fix getUri() in HttpRequest has been deprecated. (#4335)
* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.
WebHdfsHandler.java req.getUri() replace uri(), req.getMethod() replace method()
HostRestrictingAuthorizationFilterHandler.java req.getUri() replace uri()
TestHostRestrictingAuthorizationFilterHandler.java remove throws Exception, channelResponse.getStatus() replace status().

* HADOOP-18249. Fix getUri() in HttpRequest has been deprecated.

* HADOOP-18249. Fix Some CheckStyle.

Co-authored-by: slfan1989 <louj1988@@>
2022-05-24 23:11:10 +05:30
Michael Stack 5ab747222b
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4338)
* HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor IllegalThreadStateException 'fatal exception and exit'

Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.

Signed-off-by: Hexiaoqiao <hexiaoqiao@apache.org>
2022-05-23 08:45:08 -07:00
jianghuazhu 4fd4208439
HDFS-16584.Record StandbyNameNode information when Balancer is running. (#4333). Contributed by JiangHua Zhu.
Signed-off-by: litao <tomleescut@gmail.com>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-05-22 21:59:38 +08:00
Viraj Jasani 93a13202d7
HDFS-16582. Expose aggregate latency of slow node as perceived by the reporting node (#4323)
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-21 09:28:59 +08:00
Steve Loughran 6f261ed4a2
HADOOP-18198. Release 3.3.3: release notes and jdiff files.
* Add the changelog and release notes
* add all jdiff XML files
* update the project pom with the new stable version

Change-Id: Iaea846c3e451bbd446b45de146845a48953d580d
2022-05-17 19:00:54 +01:00
singer-bin d74f328b00
HDFS-16525.System.err should be used when error occurs in multiple methods in DFSAdmin class (#4122) 2022-05-12 16:53:09 +08:00
Viraj Jasani 6983aef945
HDFS-16568. dfsadmin -reconfig option to start/query reconfig on all live datanodes (#4264)
Signed-off-by: Tao Li <tomscut@apache.org>
2022-05-11 08:10:03 +08:00
daimin d486ae8c0f
Fix TestDFSStripedInputStreamWithRandomECPolicy (#4291) 2022-05-10 08:01:16 +08:00
Ashutosh Gupta 01027e52a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-08 08:05:34 +09:00
Ashutosh Gupta ae47846a5b
HADOOP-16515. Update the link to compatibility guide (#4226)
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-05-08 07:38:32 +09:00
daimin 29401c8203
HDFS-16520. Improve EC pread: avoid potential reading whole block (#4104)
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-05-07 01:30:32 +08:00
Murali Krishna 9807dfd25c
HDFS-16562. Upgrade moment.min.js to 2.29.2 (#4240) 2022-05-03 17:42:45 +05:30
Viraj Jasani 2dfa928a20
HDFS-16521. DFS API to retrieve slow datanodes (#4107)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-05-03 05:05:40 +08:00
smarthan 7bd7725532
HDFS-16553. Fix checkstyle for the length of BlockManager construction method over limit. (#4211). Contributed by Chengwei Wang. 2022-04-29 22:59:23 +08:00
Viraj Jasani b9ade7a4cd
HDFS-16528. Reconfigure slow peer enable for Namenode (#4186) 2022-04-29 17:00:17 +08:00
stack 9ed8d60511 HDFS-16540. Data locality is lost when DataNode pod restarts in kubernetes (#4170)
This reverts the previous commit 4e47eb66d1
undone so I could reapply with the '.' after the HDFS-16540 as is done
in all other commits.
2022-04-28 10:28:01 -07:00
stack 4e47eb66d1 Revert "HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)"
Revert to add the '.' after HDFS-16540 so commit message format matches
precedent

This reverts commit bda0881bf9.
2022-04-28 10:26:46 -07:00
huaxiangsun bda0881bf9
HDFS-16540 Data locality is lost when DataNode pod restarts in kubernetes (#4170)
When DN with the same UUID is registered with a different IP, host2DatanodeMap needs to be updated accordingly.
2022-04-28 10:24:37 -07:00
smarthan f187e9bcd5
HDFS-16554. Remove unused configuration dfs.namenode.block.deletion.increment. (#4213). Contributed by Chengwei Wang. 2022-04-27 11:44:38 +08:00
litao f1e5f8e764
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4234) 2022-04-26 11:48:06 +08:00
Hui Fei 17d64ba495
Revert "HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035)" (#4232) 2022-04-24 17:18:31 +08:00
litao acc0e0a210
HDFS-16488. [SPS]: Expose metrics to JMX for external SPS (#4035) 2022-04-24 14:54:41 +08:00
daimin aebd55f788
HDFS-16519. Add throttler to EC reconstruction (#4101)
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-04-23 12:45:22 +09:00
litao 5ebbacc480
HDFS-16552. Fix NPE for TestBlockManager (#4210) 2022-04-23 12:14:06 +09:00
smarthan bf4730b4d6
HDFS-16500. Make asynchronous blocks deletion lock and unlock durtion threshold configurable. (#4061). Contributed by Chengwei Wang. 2022-04-21 12:32:12 +08:00
S O'Donnell a4683be65e Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell."
This reverts commit dbeeee0363.
2022-04-20 20:34:43 +01:00
qinyuren 76bbd17374
HDFS-16544. EC decoding failed due to invalid buffer (#4179) 2022-04-20 15:04:27 +09:00
qinyuren 52e152f8b0
HDFS-16538. EC decoding failed due to not enough valid inputs (#4167)
Co-authored-by: liubingxing <liubingxing@bigo.sg>
2022-04-19 13:37:28 +09:00
Happy-shi f74f241636
HDFS-16541. Fix a typo in NameNodeLayoutVersion (#4176) 2022-04-18 22:53:57 +05:30
jianghuazhu 900682e712
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819)
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-04-18 11:21:51 +08:00
Quanlong Huang 35d4c02bcc
HDFS-16535. SlotReleaser should reuse the domain socket based on socket paths (#4158)
Reviewed-by: Lisheng Sun <sunlisheng@apache.org>
2022-04-18 10:32:29 +08:00
Stephen O'Donnell dbeeee0363
HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 2022-04-17 20:05:11 +08:00
He Xiaoqiao cb975c3df6
HDFS-16534. Split FsDatasetImpl from block pool locks to volume grain locks. (#4141) Contributed by limingxiang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-04-17 19:21:27 +08:00
Renukaprasad C f14f305051
HDFS-16526. Add metrics for slow DataNode (#4162) 2022-04-15 21:37:05 +05:30
caozhiqiang cee8c62498
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication (#4126) 2022-04-14 18:42:39 +09:00
daimin c65c383b7e
HDFS-16509. Fix decommission UnsupportedOperationException (#4077). Contributed by daimin. 2022-04-14 11:07:06 +08:00
Takanobu Asanuma 2efab92959
HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 2022-04-14 11:23:38 +09:00
qinyuren 45394433a1
HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 2022-04-13 11:27:37 +09:00
Takanobu Asanuma 37650ced81 HDFS-16497. EC: Add param comment for liveBusyBlockIndices with HDFS-14768. Contributed by caozhiqiang. 2022-04-08 18:38:40 +09:00
litao 5412fbf6d4
HDFS-16460. [SPS]: Handle failure retries for moving tasks (#4001) 2022-04-08 12:26:39 +08:00
singer-bin 807a428b55
HDFS-16457.Make fs.getspaceused.classname reconfigurable (#4069) 2022-04-08 10:27:35 +09:00
Viraj Jasani f70935522b
HADOOP-18188. Support touch command for directory (#4135)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-07 17:29:45 +09:00
Viraj Jasani 7c20602b17
HDFS-16522. Set Http and Ipc ports for Datanodes in MiniDFSCluster (#4108)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-06 18:17:02 +09:00
wangzhaohui 61bbdfd3a7
HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131)
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-04-06 17:29:43 +09:00
Stephen O'Donnell bbfe3500cf
HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 2022-04-06 08:49:52 +01:00
litao 34b3275bf4
HDFS-16477. [SPS]: Add metric PendingSPSPaths for getting the number of paths to be processed by SPS (#4009). Contributed by tomscut.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-04-03 01:36:03 +05:30
Xing Lin 15a5ea2c95
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from fallback FS (#4094)
HADOOP-18169. getDelegationTokens in ViewFs should also fetch the token from the fallback FS
2022-03-31 21:59:09 +00:00
litao 9a4dddd640
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:01:48 -07:00
He Xiaoqiao 2bf78e2416
HDFS-16511. Improve lock type for ReplicaMap under fine-grain lock mode. (#4085). Contributed by limingxiang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-31 14:00:38 +08:00
litao ac50657c37
HDFS-16413. Reconfig dfs usage parameters for datanode (#3863) 2022-03-31 00:51:17 +09:00
He Xiaoqiao 6eea28c3f3
HDFS-16498. Fix NPE for checkBlockReportLease #4057. Contributed by tomscut.
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-30 14:56:04 +08:00
Lei Yang a9b43966c0
HDFS-16518: Add shutdownhook to invalidate the KeyProviders in the cache
Fixes #4100
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-03-28 13:15:28 -07:00
Masatake Iwasaki 0fbd96a244 Make upstream aware of 3.2.3 release. 2022-03-28 08:07:50 +00:00
GuoPhilipse 046a6204b4
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724)
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-27 22:23:48 +09:00
litao 565e848d88
HDFS-16434. Add opname to read/write lock for remaining operations (#3915) 2022-03-25 20:19:01 +09:00
qinyuren 45ce1cce50
HDFS-16501. Print the exception when reporting a bad block (#4062)
Reviewed-by: tomscut <litao@bigo.sg>
2022-03-23 15:03:17 +09:00
He Xiaoqiao ef8bff0df9
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-23 11:32:12 +08:00
He Xiaoqiao 26ba3846cc
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang."
This reverts commit 88975496d8.
2022-03-23 11:01:35 +08:00
He Xiaoqiao 88975496d8
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. (#2918). Contributed by Hongbing Wang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-22 22:28:09 +08:00
Owen O'Malley 1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.

Fixes #4081
2022-03-21 09:27:35 -07:00
litao e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS (#4067). Contributed by tomscut.
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-03-21 12:40:51 +05:30
He Xiaoqiao 62d59e516e
HDFS-16504. Add parameter for NameNode to process getBloks request. (#4068). Contributed by Max Xie.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-20 14:01:45 +08:00
litao 7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058) 2022-03-17 12:04:48 +08:00
He Xiaoqiao a237526988
HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by JiangHua Zhu. 2022-03-16 20:43:38 +08:00
Viraj Jasani 1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit (#4064)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-03-16 09:32:29 +08:00
Xing Lin 8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-03-14 11:29:48 -07:00
He Xiaoqiao a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941). Contributed by limingxiang.
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
2022-03-12 18:40:09 +08:00
Stephen O'Donnell 49e4e4d0a5
HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error (#4051) 2022-03-08 11:07:11 +00:00
Viraj Jasani 278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 2022-03-04 22:17:48 +05:30
Chao Sun f800b65b40 Make upstream aware of 3.3.2 release 2022-03-02 19:14:50 -08:00
litao 9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041) 2022-03-02 11:25:50 +08:00
SevenAddSix 5eab9719cb
HDFS-16480. Fix typo: indicies -> indices (#4020) 2022-02-28 21:12:18 +08:00
Hui Fei b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998)" (#4038) 2022-02-28 21:05:19 +08:00
litao 8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998) 2022-02-28 21:03:59 +08:00
litao 6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 2022-02-25 01:23:20 +09:00
Viraj Jasani e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
jianghuazhu 589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00