046a6204b4
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second ( #3724 )
...
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-03-27 22:23:48 +09:00
565e848d88
HDFS-16434. Add opname to read/write lock for remaining operations ( #3915 )
2022-03-25 20:19:01 +09:00
45ce1cce50
HDFS-16501. Print the exception when reporting a bad block ( #4062 )
...
Reviewed-by: tomscut <litao@bigo.sg>
2022-03-23 15:03:17 +09:00
ef8bff0df9
HDFS-15987. Improve oiv tool to parse fsimage file in parallel with delimited format. ( #2918 ). Contributed by Hongbing Wang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-23 11:32:12 +08:00
26ba3846cc
Revert "HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. ( #2918 ). Contributed by Hongbing Wang."
...
This reverts commit 88975496d8
2022-03-23 11:01:35 +08:00
88975496d8
HDFS-14617. Improve oiv tool to parse fsimage file in parallel with delimited format. ( #2918 ). Contributed by Hongbing Wang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-22 22:28:09 +08:00
1d5650c4d0
HDFS-13248: Namenode needs to use the actual client IP when going through the
...
RBF proxy. There is a new configuration knob dfs.namenode.ip-proxy-users that configures
the list of users than can set their client ip address using the client context.
Fixes #4081
2022-03-21 09:27:35 -07:00
e5549a2a68
HDFS-16503. Should verify whether the path name is valid in the WebHDFS ( #4067 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-03-21 12:40:51 +05:30
62d59e516e
HDFS-16504. Add parameter for NameNode to process getBloks request. ( #4068 ). Contributed by Max Xie.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-03-20 14:01:45 +08:00
7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running ( #4058 )
2022-03-17 12:04:48 +08:00
a237526988
HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). ( #4048 ). Contributed by JiangHua Zhu.
2022-03-16 20:43:38 +08:00
1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit ( #4064 )
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2022-03-16 09:32:29 +08:00
8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS.
...
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.
If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.
Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
2022-03-14 11:29:48 -07:00
a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. ( #3941 ). Contributed by limingxiang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
2022-03-12 18:40:09 +08:00
49e4e4d0a5
HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error ( #4051 )
2022-03-08 11:07:11 +00:00
278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster ( #4028 ). Contributed by Viraj Jasani.
2022-03-04 22:17:48 +05:30
f800b65b40
Make upstream aware of 3.3.2 release
2022-03-02 19:14:50 -08:00
9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode ( #4041 )
2022-03-02 11:25:50 +08:00
5eab9719cb
HDFS-16480. Fix typo: indicies -> indices ( #4020 )
2022-02-28 21:12:18 +08:00
b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode ( #3998 )" ( #4038 )
2022-02-28 21:05:19 +08:00
8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode ( #3998 )
2022-02-28 21:03:59 +08:00
6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode ( #3828 )
2022-02-25 01:23:20 +09:00
e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics ( #4002 )
2022-02-22 13:04:36 +09:00
589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. ( #3861 )
2022-02-22 10:15:19 +08:00
19d90e62fb
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953)
2022-02-15 15:58:24 -08:00
0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode ( #3827 )
...
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2022-02-15 13:40:46 +09:00
bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads ( #3847 )
2022-02-09 10:07:05 +08:00
125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … ( #3926 )
2022-02-06 13:05:57 +08:00
089e06de21
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception ( #3942 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-31 13:04:32 +09:00
bd50b9117e
HDFS-16444. Show start time of JournalNode on Web ( #3943 )
2022-01-30 16:13:58 +08:00
39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure ( #3850 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-28 17:04:11 +09:00
e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. ( #3900 ). Contributed by limingxiang.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-27 16:53:21 +08:00
6136d630a3
HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped ( #3888 )
2022-01-27 13:42:44 +09:00
c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode ( #3831 )
2022-01-26 17:24:09 +09:00
8c7c49d877
HDFS-16401.Remove the worthless DatasetVolumeChecker#numAsyncDatasetChecks. ( #3838 )
2022-01-25 21:51:17 +08:00
94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream ( #3527 )
2022-01-25 11:42:35 +00:00
43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename ( #3898 ). Contributed by lei w.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-25 15:26:18 +08:00
5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy ( #3899 ). Contributed by daimin.
...
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2022-01-24 12:04:26 +05:30
15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. ( #3839 ). Contributed by tomscut.
...
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-24 14:26:30 +08:00
db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages ( #3883 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-19 17:59:42 +09:00
e355646330
HDFS-16399. Reconfig cache report parameters for datanode ( #3841 )
2022-01-19 15:10:39 +09:00
fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force ( #3887 )
2022-01-19 13:42:33 +09:00
d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. ( #3880 )
...
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-14 22:38:11 +09:00
f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode ( #3843 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2022-01-14 14:48:10 +09:00
034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously ( #3063 ). Contributed by Xiangyi Zhu.
...
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2022-01-13 12:56:19 +08:00
e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 ( #3791 )
...
Co-authored-by: luoyuan <luoyuan@shopee.com>
2022-01-12 11:40:32 +08:00
46b02788bd
HDFS-16371. Exclude slow disks when choosing volume ( #3753 )
2022-01-06 09:31:00 +09:00
43e5218a86
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader ( #3854 )
...
Contributed by Ashutosh Gupta
2022-01-05 15:51:14 +00:00
e1d0aa9ee5
HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero ( #3856 )
2022-01-05 15:42:58 +00:00
9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue ( #3835 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2022-01-04 13:25:16 +09:00
b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline ( #3704 )
2021-12-30 00:27:11 +09:00
a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). ( #3823 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-24 17:24:12 +09:00
d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning ( #3675 )
...
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-23 22:59:08 +09:00
cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout ( #3821 )
2021-12-22 10:44:07 +08:00
9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. ( #3803 )
...
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
2021-12-21 11:04:02 +09:00
746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. ( #3806 )
2021-12-20 11:28:55 +00:00
32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout ( #3815 )
2021-12-20 18:16:22 +08:00
04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future ( #3789 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-20 16:01:34 +09:00
d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport ( #3714 ). Contributed by liubingxing.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-17 13:31:28 +08:00
22f5e1885d
HDFS-16377. Should CheckNotNull before access FsDatasetSpi ( #3784 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-16 13:49:50 +09:00
10cdde59b4
HDFS-16375. The FBR lease ID should be exposed to the log ( #3769 )
2021-12-16 13:29:32 +09:00
59c650802d
HDFS-16378. Add datanode address to BlockReportLeaseManager logs ( #3786 ). Contributed by tomscut.
...
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
2021-12-15 11:16:32 +08:00
d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. ( #3756 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-14 17:21:51 +09:00
c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable ( #3716 )
2021-12-14 11:19:06 +09:00
9b9e2ef87f
HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure ( #3774 )
...
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
2021-12-10 01:36:31 +09:00
35556ea043
HDFS-16333. fix balancer bug when transfer an EC block ( #3679 )
2021-12-09 13:24:09 +09:00
1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support ( #3761 )
2021-12-08 17:52:40 +05:30
00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode ( #3661 )
...
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-08 21:21:48 +09:00
25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy ( #3696 ). Contributed by guophilipse.
...
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-08 16:34:54 +05:30
e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. ( #3740 )
2021-12-07 21:39:04 +09:00
80cccc5376
HDFS-16351. Add path exception information in FSNamesystem ( #3713 ). Contributed by guophilipse.
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-07 14:19:27 +05:30
1509ea5de2
HDFS-16338. Correct fsimage error configuration message ( #3684 ). Contributed by guophilipse.
...
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-06 14:00:50 +05:30
e8e69de106
HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu.
2021-12-06 10:44:36 +09:00
c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo ( #3747 ). Contributed by tomscut.
...
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
2021-12-04 22:54:44 +05:30
c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable ( #3664 )
...
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-04 02:19:36 +09:00
dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake ( #3677 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-12-03 23:30:13 +09:00
0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 ( #3742 )
2021-12-03 16:51:04 +05:30
52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable ( #3676 )
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2021-12-03 14:12:05 +09:00
1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand ( #3732 ). Contributed by tomscut.
2021-12-01 12:48:21 +05:30
829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString ( #3695 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-29 17:48:10 +09:00
86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded ( #3689 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-26 19:52:14 +09:00
9c887e5b82
HADOOP-18014. CallerContext should not include some characters. ( #3698 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
2021-11-25 14:05:04 +09:00
cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully ( #3711 )
2021-11-25 10:23:04 +08:00
99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3708 )
2021-11-25 10:20:42 +08:00
e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback ( #3686 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-24 22:54:11 +09:00
faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. ( #3694 )
2021-11-23 13:36:14 +00:00
0ed817babb
HDFS-16337. Show start time of Datanode on Web ( #3682 )
...
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
2021-11-23 10:18:30 +08:00
c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode ( #3654 )
2021-11-22 18:04:32 +08:00
20b78c8f97
HDFS-16335. Fix HDFSCommands.md ( #3681 )
2021-11-22 17:07:52 +09:00
8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )" ( #3697 )
2021-11-22 10:05:21 +08:00
3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description ( #3678 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-22 07:41:57 +09:00
965e3874e9
HDFS-16326. Simplify the code for DiskBalancer ( #3663 )
...
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-11-19 11:37:10 +09:00
0c1eb0125b
HDFS-16328. Correct disk balancer desc ( #3669 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 01:02:01 +09:00
38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer ( #3672 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:55:39 +09:00
dc751df63b
HDFS-16329. Fix log format for BlockManager ( #3670 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2021-11-19 00:42:01 +09:00
5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router ( #3635 )
2021-11-18 13:46:35 +09:00
91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson ( #3630 )
2021-11-17 22:41:06 +08:00
646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers ( #3659 )
2021-11-17 22:17:09 +09:00
5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. ( #3320 )
2021-11-16 18:21:50 +08:00
c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode ( #3643 )
...
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
2021-11-16 11:19:14 +09:00
GuoPhilipse
litao
qinyuren
He Xiaoqiao
Owen O'Malley
Viraj Jasani
Xing Lin
Stephen O'Donnell
Chao Sun
SevenAddSix
Hui Fei
jianghuazhu
Chentao Yu
secfree
singer-bin
KevinWikant
Li MingXiang
Bryan Beaudreault
Thinker313
daimin
Jackson Wang
Xiangyi Zhu
luoyuan3471
Ashutosh Gupta
liever18
Symious
Ayush Saxena
Akira Ajisaka
PHILO-HE
Masatake Iwasaki
Takanobu Asanuma
huhaiyang
bitterfox
wangzhaohui
Mukul Kumar Singh
Neil