Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
26,687 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

9897 Commits

Author SHA1 Message Date
Gautham B A a631f45a99
HDFS-16470. Make HDFS find tool cross platform (#4076) 
* The source files for hdfs_find uses
   getopt for parsing the command
   line arguments. getopt is available
   only on Linux and thus, isn't cross
   platform.
* Thus, we need to replace getopt
   with boost::program_options to
   make hdfs_find cross platform.
 2022-03-18 21:41:01 +05:30
litao 7f6a891f03
HDFS-16499. [SPS]: Should not start indefinitely while another SPS process is running (#4058) 2022-03-17 12:04:48 +08:00
He Xiaoqiao a237526988
HDFS-16494.Removed reuse of AvailableSpaceVolumeChoosingPolicy#initLocks(). (#4048). Contributed by JiangHua Zhu. 2022-03-16 20:43:38 +08:00
Viraj Jasani 1c0bc35305
HDFS-16502. Reconfigure Block Invalidate limit (#4064) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-16 09:32:29 +08:00
Xing Lin 8b8158f02d
HADOOP-18144: getTrashRoot in ViewFileSystem should return a path in ViewFS. 
To get the new behavior, define fs.viewfs.trash.force-inside-mount-point to be true.

If the trash root for path p is in the same mount point as path p,
and one of:
* The mount point isn't at the top of the target fs.
* The resolved path of path is root (eg it is the fallback FS).
* The trash root isn't in user's target fs home directory.
get the corresponding viewFS path for the trash root and return it.
Otherwise, use <mnt>/.Trash/<user>.

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 11:29:48 -07:00
Owen O'Malley 7b5eac27ff
HDFS-16495: RBF should prepend the client ip rather than append it. 
Fixes #4054

Signed-off-by: Owen O'Malley <oomalley@linkedin.com>
 2022-03-14 10:21:35 -07:00
He Xiaoqiao a32cfc2169
HDFS-15382. Split one FsDatasetImpl lock to block pool grain locks. (#3941). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: litao <tomleescut@gmail.com>
 2022-03-12 18:40:09 +08:00
Stephen O'Donnell 49e4e4d0a5
HDFS-16496. Snapshot diff on snapshotable directory fails with not snapshottable error (#4051) 2022-03-08 11:07:11 +00:00
Viraj Jasani 66b72406bd
HADOOP-18131. Upgrade maven enforcer plugin and relevant dependencies (#4000) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-03-08 17:27:04 +09:00
Gautham B A 8f07081789
HDFS-16462. Make HDFS get tool cross platform (#4003) 2022-03-05 23:05:14 +05:30
Gautham B A 19561846df
HADOOP-18151. Switch the baseurl for Centos 8 (#4047) 
* Centos 8 has reached its
  End-of-Life and thus its
  packages are no longer
  accessible from
  mirror.centos.org.
* This PR switches the baseurl
  to vault.centos.org where
  the packages are archived.
 2022-03-05 07:11:22 +05:30
Viraj Jasani 278568203b
HDFS-16481. Provide support to set Http and Rpc ports in MiniJournalCluster (#4028). Contributed by Viraj Jasani. 2022-03-04 22:17:48 +05:30
Chao Sun f800b65b40 Make upstream aware of 3.3.2 release 2022-03-02 19:14:50 -08:00
litao 9199787215
HDFS-16458. [SPS]: Fix bug for unit test of reconfiguring SPS mode (#4041) 2022-03-02 11:25:50 +08:00
SevenAddSix 5eab9719cb
HDFS-16480. Fix typo: indicies -> indices (#4020) 2022-02-28 21:12:18 +08:00
Hui Fei b2f541a749
Revert "[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998)" (#4038) 2022-02-28 21:05:19 +08:00
litao 8aa568cea5
[SPS]: Fix bug for unit test of reconfiguring SPS mode (#3998) 2022-02-28 21:03:59 +08:00
litao 6b07c851f3
HDFS-16397. Reconfig slow disk parameters for datanode (#3828) 2022-02-25 01:23:20 +09:00
Viraj Jasani e363f51ffb
HDFS-16461. Expose JournalNode storage info in the jmx metrics (#4002) 2022-02-22 13:04:36 +09:00
jianghuazhu 589695c6a9
HDFS-16316.Improve DirectoryScanner: add regular file check related block. (#3861) 2022-02-22 10:15:19 +08:00
Neil 2f448acf39
HDFS-16459. RBF: register RBFMetrics in MetricsSystem for promethuessink (#3999) 2022-02-21 11:32:34 -08:00
Chentao Yu 19d90e62fb HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 2022-02-15 15:58:24 -08:00
YulongZ 48bef285a8
HDFS-16440. RBF: Support router get HAServiceStatus with Lifeline RPC address (#3971) 2022-02-15 08:44:17 -08:00
litao 0c194f2157
HDFS-16396. Reconfig slow peer parameters for datanode (#3827) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-15 13:40:46 +09:00
Neil c777142ed3
RBF: Registry HDFS Router's rpcserver & rpcclient metrics for PrometheusSink. (#3965) 2022-02-10 09:17:30 -08:00
Neil 390967f1f0
RBF: Add search box in Router's tab-mounttable web page (#3968) 2022-02-09 09:17:23 -08:00
secfree bf0cefb0d8
HDFS-16406. ReadsFromLocalClient counts short-circuit reads (#3847) 2022-02-09 10:07:05 +08:00
YulongZ 3e7a7c3c4a
HDFS-16411 RBF: RouterId is NULL when disable RourterRpcServer (#3878) 2022-02-08 08:40:57 -08:00
singer-bin 125e3b6160
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 2022-02-06 13:05:57 +08:00
litao b5b07af605
HDFS-16435. Remove no need TODO comment for ObserverReadProxyProvider (#3912). Contributed by tomscut. 
Reviewed-by: Chao Sun <sunchao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-02-03 08:21:06 +05:30
Gautham B A ed44662968
HDFS-16445. Make HDFS count, mkdir, rm cross platform (#3945) 2022-02-02 11:12:17 +05:30
KevinWikant 089e06de21
HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 13:04:32 +09:00
litao bd50b9117e
HDFS-16444. Show start time of JournalNode on Web (#3943) 2022-01-30 16:13:58 +08:00
secfree 39cad5f28f
HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-28 17:04:11 +09:00
Li MingXiang e17c96a40a
HDFS-16429. Add DataSetLockManager to manage fine-grain locks for FsDataSetImpl. (#3900). Contributed by limingxiang. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-27 16:53:21 +08:00
litao 6136d630a3
HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 2022-01-27 13:42:44 +09:00
litao c2ff39006f
HDFS-16398. Reconfig block report parameters for datanode (#3831) 2022-01-26 17:24:09 +09:00
jianghuazhu 8c7c49d877
HDFS-16401.Remove the worthless DatasetVolumeChecker#numAsyncDatasetChecks. (#3838) 2022-01-25 21:51:17 +08:00
Bryan Beaudreault 94b884ae55
HDFS-16262. Async refresh of cached locations in DFSInputStream (#3527) 2022-01-25 11:42:35 +00:00
Thinker313 43153e80cb
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-25 15:26:18 +08:00
daimin d69938994e
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-01-25 13:02:37 +08:00
daimin 5ef335da1e
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2022-01-24 12:04:26 +05:30
litao 15b820c83c
HDFS-16402. Improve HeartbeatManager logic to avoid incorrect stats. (#3839). Contributed by tomscut. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-24 14:26:30 +08:00
qinyuren db2c3200e6
HDFS-16423. Balancer should not get blocks on stale storages (#3883) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-19 17:59:42 +09:00
litao e355646330
HDFS-16399. Reconfig cache report parameters for datanode (#3841) 2022-01-19 15:10:39 +09:00
qinyuren fcb1076699
HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 2022-01-19 13:42:33 +09:00
Jackson Wang d8862822d2
HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 22:38:11 +09:00
litao f02374df92
HDFS-16400. Reconfig DataXceiver parameters for datanode (#3843) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 14:48:10 +09:00
Xiangyi Zhu 034dc8d03c
HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3063). Contributed by Xiangyi Zhu. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-13 12:56:19 +08:00
Gautham B A 271ac0e091
HDFS-16419. Make HDFS data transfer tools cross platform (#3873) 2022-01-12 19:57:27 +05:30
luoyuan3471 e2d620192a
HADOOP-18044. Hadoop - Upgrade to jQuery 3.6.0 (#3791) 
Co-authored-by: luoyuan <luoyuan@shopee.com>
 2022-01-12 11:40:32 +08:00
Felix Nguyen b88d5ae1c6
HDFS-16417. StaticRouterRpcFairnessPolicyController init (#3871) 2022-01-10 18:28:20 +08:00
Viraj Jasani f64fda0f00
HADOOP-18055. Async Profiler endpoint for Hadoop daemons (#3824) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-06 17:56:49 +08:00
litao 46b02788bd
HDFS-16371. Exclude slow disks when choosing volume (#3753) 2022-01-06 09:31:00 +09:00
Ashutosh Gupta 43e5218a86
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 15:51:14 +00:00
liever18 e1d0aa9ee5
HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 2022-01-05 15:42:58 +00:00
Gautham B A c3006be516
HDFS-16407. Make hdfs_du tool cross platform (#3848) 2022-01-04 22:29:54 +05:30
Ashutosh Gupta 9eea0e28f2
HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-04 13:25:16 +09:00
Ayush Saxena 61d424f385
HDFS-16393. RBF: Fix TestRouterRPCMultipleDestinationMountTableResolver. (#3849). Contributed by Ayush Saxena. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2022-01-04 07:50:01 +05:30
Symious b225287913
HDFS-16348. Mark slownode as badnode to recover pipeline (#3704) 2021-12-30 00:27:11 +09:00
jianghuazhu a5659fe0f7
HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-24 17:24:12 +09:00
KevinWikant d20b598f97
HDFS-16303. Improve handling of datanode lost while decommissioning (#3675) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-23 22:59:08 +09:00
secfree cc07a5a186
HDFS-16392. Fix TestWebHdfsFileSystemContract#testResponseCode timeout (#3821) 2021-12-22 10:44:07 +08:00
wangzhaohui 8d251bd629
HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 2021-12-21 11:28:57 +00:00
Jackson Wang 9420cb6f8a
HDFS-16385. Fix Datanode retrieve slownode information bug. (#3803) 
Reviewed-by: Symious <yiyang0203@gmail.com>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
 2021-12-21 11:04:02 +09:00
jianghuazhu 746b328554
HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 2021-12-20 11:28:55 +00:00
secfree 32a78e0b9e
HDFS-16168. Fix TestHDFSFileSystemContract.testAppend timeout (#3815) 2021-12-20 18:16:22 +08:00
Viraj Jasani 04b6b9a87b
HADOOP-16908. Prune Jackson 1 from the codebase and restrict it's usage for future (#3789) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-20 16:01:34 +09:00
liubingxing d8dea6f52a
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-17 13:31:28 +08:00
Wei-Chiu Chuang bdec546671
Revert "HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798)" 
This reverts commit a4557f9ed9.
 2021-12-16 21:27:08 +08:00
litao 22f5e1885d
HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-16 13:49:50 +09:00
Tamás Pénzes a4557f9ed9
HDFS-16384. Upgrade Netty to 4.1.72.Final (#3798) 2021-12-16 12:38:42 +08:00
litao 10cdde59b4
HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 2021-12-16 13:29:32 +09:00
litao 59c650802d
HDFS-16378. Add datanode address to BlockReportLeaseManager logs (#3786). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-12-15 11:16:32 +08:00
Ayush Saxena d29f0e83a9
HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:21:51 +09:00
litao c56a07f36b
HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 2021-12-14 11:19:06 +09:00
Akira Ajisaka 9b9e2ef87f
HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:36:31 +09:00
Gautham B A 6dddbd42ed
HDFS-16285. Make HDFS ownership tools cross platform (#3588) 2021-12-09 10:17:41 +05:30
qinyuren 35556ea043
HDFS-16333. fix balancer bug when transfer an EC block (#3679) 2021-12-09 13:24:09 +09:00
PHILO-HE 1364847fdd
HDFS-15788. Correct the statement for pmem cache to reflect cache persistence support (#3761) 2021-12-08 17:52:40 +05:30
GuoPhilipse 00d483d6de
HDFS-16324. Fix error log in BlockManagerSafeMode (#3661) 
Reviewed-by: tomscut <litao@bigo.sg>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 21:21:48 +09:00
lzx404243 7ba91c1235
HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-08 20:44:18 +09:00
GuoPhilipse 25849ff7b7
HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-08 16:34:54 +05:30
Masatake Iwasaki e9526042a2
HDFS-16354. Add description of GETSNAPSHOTDIFFLISTING to WebHDFS doc. (#3740) 2021-12-07 21:39:04 +09:00
GuoPhilipse 80cccc5376
HDFS-16351. Add path exception information in FSNamesystem (#3713). Contributed by guophilipse. 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-07 14:19:27 +05:30
GuoPhilipse 1509ea5de2
HDFS-16338. Correct fsimage error configuration message (#3684). Contributed by guophilipse. 
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-06 14:00:50 +05:30
Takanobu Asanuma e8e69de106 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 2021-12-06 10:44:36 +09:00
litao c2afb6a00b
HDFS-16370. Fix assert message for BlockInfo (#3747). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 22:54:44 +05:30
GuoPhilipse 2f55a6cced
HDFS-16357. Fix log format in DFSUtilClient (#3729). Contributed by guophilipse. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-04 18:35:05 +05:30
Ayush Saxena cab7086fbc
HDFS-16369. RBF: Fix the retry logic of RouterRpcServer#invokeAtAvailableNs. (#3745). Contributed by Ayush Saxena. 
Reviewed-by: litao <tomleescut@gmail.com>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-12-04 10:54:29 +05:30
huhaiyang c0f405a46b
HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-04 02:19:36 +09:00
bitterfox dd6b987c93
HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-12-03 23:30:13 +09:00
wangzhaohui 0cb6c28d19
HDFS-16364. Remove unnecessary brackets in NameNodeRpcServer#L453 (#3742) 2021-12-03 16:51:04 +05:30
litao 52ec65fd10
HDFS-16331. Make dfs.blockreport.intervalMsec reconfigurable (#3676) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-03 14:12:05 +09:00
Viraj Jasani 0c62a514f7
HDFS-16358. HttpFS implementation for getSnapshotDiffReportListing (#3730) 2021-12-03 13:25:32 +09:00
litao 196935a8d2
HDFS-16359. RBF: RouterRpcServer#invokeAtAvailableNs does not take effect when retrying (#3731). Contributed by tomscut. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-12-02 19:10:35 +05:30
litao 1ebdac2cd6
HDFS-16361. Fix log format for QueryCommand (#3732). Contributed by tomscut. 2021-12-01 12:48:21 +05:30
litao 829af89dc4
HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-29 17:48:10 +09:00
litao 86705eda82
HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-26 19:52:14 +09:00
Takanobu Asanuma 9c887e5b82
HADOOP-18014. CallerContext should not include some characters. (#3698) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-25 14:05:04 +09:00
Viraj Jasani cdc13e91b6
HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 2021-11-25 10:23:04 +08:00
huhaiyang 99b161dec7
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3708) 2021-11-25 10:20:42 +08:00
Viraj Jasani e8566b3812
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-24 22:54:11 +09:00
Mukul Kumar Singh faa4eeacb1
HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 2021-11-23 13:36:14 +00:00
litao 0ed817babb
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-11-23 10:18:30 +08:00
Symious c88640c4ad
HDFS-16320. Datanode retrieve slownode information from NameNode (#3654) 2021-11-22 18:04:32 +08:00
litao 20b78c8f97
HDFS-16335. Fix HDFSCommands.md (#3681) 2021-11-22 17:07:52 +09:00
Hui Fei 8e9ad2d7af
Revert "HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630)" (#3697) 2021-11-22 10:05:21 +08:00
GuoPhilipse 3d1ca6f6dc
HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-22 07:41:57 +09:00
litao 965e3874e9
HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-11-19 11:37:10 +09:00
GuoPhilipse 0c1eb0125b
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 01:02:01 +09:00
Viraj Jasani 38614d05e1
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:55:39 +09:00
litao dc751df63b
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-19 00:42:01 +09:00
litao 5b05068fdc
HDFS-16310. RBF: Add client port to CallerContext for Router (#3635) 2021-11-18 13:46:35 +09:00
huhaiyang 91af256a5b
HADOOP-17995. Stale record should be remove when DataNodePeerMetrics#dumpSendPacketDownstreamAvgInfoAsJson (#3630) 2021-11-17 22:41:06 +08:00
Viraj Jasani 646c470e5d
HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 2021-11-17 22:17:09 +09:00
Neil 5626734a36
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 2021-11-16 18:21:50 +08:00
litao c9f95b01ef
HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3643) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-16 11:19:14 +09:00
Symious 89fcbd84f9
HDFS-16313. RBF: Add metrics for each sub cluster (#3638) 2021-11-16 09:47:25 +08:00
Viraj Jasani e14a2dcbba
HADOOP-18006. maven-enforcer-plugin's execution of banned-illegal-imports gets overridden in child poms (#3648) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-11-15 22:57:24 +09:00
GuoPhilipse 6e6f2e4baa
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 19:59:15 +05:30
huhaiyang db89a9411e
HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 2021-11-12 10:07:34 +08:00
jianghuazhu 3d692c0936
HDFS-16305.Record the remote NameNode address when the rolling log is triggered. (#3629) 2021-11-12 09:52:02 +08:00
litao 1c1cf64616
HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-12 10:23:25 +09:00
litao 2a1a11c039
HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:22:32 +09:00
litao ea65fc26d8
HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-10 19:16:45 +09:00
jianghuazhu b307f18a58
HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 2021-11-10 11:32:52 +09:00
litao cbc7fb6bca
HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-09 17:06:28 +09:00
GuoPhilipse 9b2fa7e47a
HDFS-16307. Improve HdfsBlockPlacementPolicies docs readability 2021-11-08 10:21:39 -08:00
Gautham B A 0a952e4d1f
HDFS-16304. Locate OpenSSL libs for libhdfspp (#3625) 2021-11-08 08:58:07 -08:00
Symious d2b8d6b534
HDFS-16302. RBF: RouterRpcFairnessPolicyController record requests accepted by each nameservice (#3621) 2021-11-08 09:52:16 +08:00
Xiangyi Zhu e5cee76785
HDFS-16273. RBF: RouterRpcFairnessPolicyController add availableHandleOnPerNs metrics (#3553) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-11-08 09:30:53 +09:00
Symious 6f7b965808
HDFS-16296. RouterRpcFairnessPolicyController add rejected permits for each nameservice (#3613) 2021-11-05 11:03:07 +08:00
Gautham B A 1032724aa3
HDFS-16300. Use libcrypto in Windows for libhdfspp (#3617) 2021-11-04 09:19:11 -07:00
jianghuazhu 72718a8dc1
HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 2021-11-04 16:43:25 +09:00
jianghuazhu 18b2cca74c
HDFS-16291.Make the comment of INode#ReclaimContext more standardized. (#3602) 2021-11-04 09:51:11 +08:00
litao 359b03c83e
HDFS-16266. Add remote port information to HDFS audit log (#3538) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-11-04 10:16:03 +09:00
daimin a21895a5b3
HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 2021-11-03 20:19:56 +00:00
Stephen O'Donnell 2f35cc36cd
HDFS-16259. Catch and re-throw sub-classes of AccessControlException thrown by any permission provider plugins (eg Ranger) (#3598) 2021-11-02 11:14:48 +00:00
jianghuazhu 618fea27d2
HDFS-16269. [Fix] Improve NNThroughputBenchmark#blockReport operation. (#3544) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-02 00:55:50 +09:00
jianghuazhu 0a68413a4f
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600) 
HDFS-16290.Make log more standardized when executing verifyAndSetNamespaceInfo(). (#3600). Contributed by JiangHua Zhu.
 2021-10-31 08:48:52 +05:30
Masatake Iwasaki 634f8a2209
HDFS-16091. WebHDFS should support getSnapshotDiffReportListing. (#3374) 2021-10-31 09:29:34 +09:00
litao 6aa664c624
HDFS-16279. Print detail datanode info when process first storage report (#3564) 2021-10-29 10:27:00 +09:00
litao c1a8285363
HDFS-16281. Fix flaky unit tests failed due to timeout (#3574) 2021-10-26 14:54:11 +08:00
Ahmed Hussein 62c86eaa0e
HADOOP-17970. unguava: remove Preconditions from hdfs-projects modules (#3566) 2021-10-23 16:13:07 -05:00
daimin 6e57449350
HDFS-16282. Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:12:13 +05:30
Ayush Saxena f67e6c315b
Revert "Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin." 
Misses Jira ID in the commit Message: HDFS-16282
This reverts commit 7b913bc6bc.
 2021-10-22 21:10:38 +05:30
daimin 7b913bc6bc
Remove duplicate generic usage information to hdfs debug command (#3576). Contributed by daimin. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 21:09:37 +05:30
litao 2c37bebac4
HDFS-16280. Fix typo for ShortCircuitReplica#isStale (#3568). Contributed by tomscut. 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 08:21:30 +05:30
GuoPhilipse 1385cdfeb9
HDFS-16277. improve decision in AvailableSpaceBlockPlacementPolicy (#3559). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-10-22 07:59:27 +05:30
Michael Kuchenbecker a73ff6915a HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 2021-10-21 13:41:46 -07:00
Gautham B A a2f089df4d
HDFS-16278. Make HDFS snapshot tools cross platform (#3563) 2021-10-21 10:24:33 -07:00
Viraj Jasani 516f36c6f1
HADOOP-17967. Keep restrict-imports-enforcer-rule for Guava VisibleForTesting in hadoop-main pom (#3555) 2021-10-21 16:54:25 +09:00
daimin 5337bebcc5
HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 2021-10-18 11:55:55 +01:00
Xiangyi Zhu d09b380dc2
HDFS-16276. RBF: Remove the useless configuration of rpc isolation in md (#3557) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-18 13:53:41 +09:00
Ayush Saxena 263e4b595d
HDFS-16271. RBF: NullPointerException when setQuota through routers with quota disabled. Contributed by Chengwei Wang. 2021-10-16 14:35:18 +05:30
litao d72e2cb786
HDFS-16274. Improve error msg for FSNamesystem#startFileInt (#3554) 
* HDFS-16274. Improve log for FSNamesystem#startFileInt

* fix failed unit test
 2021-10-15 22:41:38 +05:30
Viraj Jasani 1151edf12e
HADOOP-17956. Replace all default Charset usage with UTF-8 (#3529) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-14 13:07:24 +09:00
LeonGao 7279fe8661
HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 2021-10-13 18:14:03 -07:00
Gautham B A 107fe227eb
HDFS-16267. Make hdfs_df tool cross platform (#3542) 2021-10-13 09:11:21 -07:00
jianghuazhu ff24fe6197
HDFS-16244.Add the necessary write lock in Checkpointer#doCheckpoint(). (#3497) 2021-10-13 17:21:03 +08:00
Gautham B A a05eb23a85
HDFS-16260. Make hdfs_deleteSnapshot tool cross platform (#3532) 2021-10-11 18:25:28 -07:00
jianghuazhu d7c01d083c
HDFS-16264.When adding block keys, the records come from the specific Block Pool. (#3533) 2021-10-11 19:49:58 +08:00
langlaile1221 88d8c3d0c8
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
 2021-10-11 16:11:31 +08:00
Viraj Jasani 3ac0cc9e6f
HADOOP-17957. Replace Guava VisibleForTesting by Hadoop's own annotation in hadoop-hdfs-project modules (#3530) 
Reviewed-by: Ahmed Hussein <ahussein@apache.org>
 2021-10-11 15:32:49 +09:00
Gautham B A 4f3dfb7c1c
HDFS-16265. Refactor HDFS tool tests for better reuse (#3536) 2021-10-09 11:00:43 -07:00
Gautham B A cc95fc4cc6
HDFS-16263. Add CMakeLists for hdfs_allowSnapshot (#3531) 2021-10-08 08:49:56 -07:00
Gautham B A 6bad3badf5
HDFS-16251. Make hdfs_cat tool cross platform (#3523) 2021-10-07 10:57:11 -07:00
Viraj Jasani 8071dbb9c6
HADOOP-17950. Provide replacement for deprecated APIs of commons-io IOUtils (#3515) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-10-07 10:58:29 +09:00
Gautham B A e12cd0c638
HDFS-16254. Cleanup protobuf on exit of hdfs_allowSnapshot (#3518) 2021-10-06 09:24:27 -07:00
S O'Donnell a30231ff8e HDFS-16252. Correct docs for dfs.http.client.retry.policy.spec. Contributed by Stephen O'Donnell 2021-10-06 15:04:04 +01:00
Gautham B A 84f10fd78b
HDFS-16250. Refactor AllowSnapshotMock using GMock (#3513) 2021-10-05 09:16:42 -07:00
huhaiyang 68c2accc20
HDFS-16247. RBF: Fix the ProcessingAvgTime and ProxyAvgTime code comments and document metrics describe ms unit (#3511) 2021-10-04 23:52:26 +08:00
Ayush Saxena 5f0452602f
HDFS-16222. Fix ViewDFS with mount points for HDFS only API. (#3422). Contributed by Ayush Saxena. 
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>
 2021-10-03 10:02:35 +05:30
Neil bf9106c812
HDFS-16242. JournalMetrics should add JournalId MetricTag. (#3494) 
Contributed by Max Xie
 2021-10-01 09:35:14 +02:00
jianghuazhu 1789c7c8c0
HDFS-16238.Improve comments related to EncryptionZoneManager. (#3483) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-10-01 10:30:41 +08:00
huhaiyang 1f8f53f7d7
HDFS-16231. Fix TestDataNodeMetrics#testReceivePacketSlowMetrics (#3471) 2021-09-28 13:09:18 +08:00
jianghuazhu 02993bc11e
HDFS-16237.Record the BPServiceActor information that communicates with Standby. (#3479) 2021-09-28 13:07:36 +08:00
Masatake Iwasaki 86e80752d1
HDFS-16240. Replace unshaded guava in HttpFSServerWebServer. (#3492) 2021-09-28 13:35:35 +09:00
Chao Sun 1397cf2496 Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the source and destination are the same" 
This reverts commit 59f01a548e.
 2021-09-25 21:32:43 -07:00
AngersZhuuuu 5f9321a5d4
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 2021-09-25 18:17:44 +08:00
Wei-Chiu Chuang 87632bbacf
HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 2021-09-24 08:35:23 -07:00
litao 3113a119af
HDFS-16232. Fix java doc for BlockReaderRemote#newBlockReader (#3456) 2021-09-24 13:14:25 +08:00
Ayush Saxena 98e2781dde
HDFS-16063. Add toString to EditLogFileInputStream. Contributed by Dionisii Iuzhakov. 2021-09-23 10:45:02 +05:30
Gautham B A 35a8d48872
HDFS-16205. Make hdfs_allowSnapshot tool cross platform (#3388) 2021-09-22 10:29:38 -07:00
Viraj Jasani c7e7b2f907
HDFS-16219. RBF: Set default map tasks and bandwidth in RouterFederationRename (#3452) 2021-09-22 15:34:51 +09:00
LeonGao b1431813d0
HDFS-10648. Expose Balancer metrics through Metrics2 (#3427) 2021-09-21 12:26:17 -07:00
Viraj Jasani feee41aa00
HDFS-16213. Flaky test TestFsDatasetImpl#testDnRestartWithHardLink (#3386) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-21 10:11:20 +09:00
Tamas Domok f93e8fbf2d
HDFS-16129. Fixing the signature secret file misusage in HttpFS. Contributed by Tamas Domok 
* HDFS-16129. Fixing the signature secret file misusage in HttpFS.

The signature secret file was not used in HttpFs.
 - if the configuration did not contain the deprecated
httpfs.authentication.signature.secret.file option then it
used the random secret provider
 - if both option (httpfs. and hadoop.http.) was set then
the HttpFSAuthenticationFilter could not read the file
because the file path was not substituted properly

!NOTE! behavioral change: the deprecated httpfs. configuration
values are overwritten with the hadoop.http. values.

The commit also contains a follow up change to the YARN-10814,
empty secret files will result in a random secret provider.

Co-authored-by: Tamas Domok <tdomok@cloudera.com>
 2021-09-20 14:29:50 +02:00
Thomas Leplus f92c6750ec
HDFS-16230. Remove irrelevant trim() call in TestStorageRestore. (#3455). Contributed by Thomas Leplus. 2021-09-20 00:09:05 +05:30
Viraj Jasani c9763a99c1
HDFS-16227. De-flake TestMover#testMoverWithStripedFile (#3429) 2021-09-18 19:02:45 +08:00
jianghuazhu cd5c6395e8
HDFS-16229.Remove the use of obsolete BLOCK_DELETION_INCREMENT. (#3451). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-18 11:06:51 +05:30
Viraj Jasani 7c25a77911
HDFS-16217. RBF: Set default value of hdfs.fedbalance.procedure.scheduler.journal.uri by adding appropriate config resources (#3434) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-17 16:39:29 +09:00
LeonGao deb52cdcbb
HDFS-16224. testBalancerWithObserverWithFailedNode times out (#3425) 2021-09-16 23:59:36 -07:00
lfengnan 4d21655d04
HDFS-16218. Use HdfsConfiguration for passing in Router principal (#3444) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-09-16 14:58:02 +09:00
litao 2d479309cc
HDFS-16203. Discover datanodes with unbalanced block pool usage by the standard deviation (#3366) 
Reviewed-by: Hui Fei <ferhui@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-09-16 10:00:02 +09:00
wangzhaohui 3ecaa39668
HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
 2021-09-15 09:56:51 -07:00
litao 4f563ff1ba
HDFS-16225. Fix typo for FederationTestUtils (#3428). Contributed by tomscut. 2021-09-14 14:28:19 +05:30
jianghuazhu 1032104b9a
HDFS-15920.Solve the problem that the value of SafeModeMonitor#RECHECK_INTERVAL can be configured. (#2831). Contributed by JiangHua Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-14 14:24:36 +05:30
EungsopYoo c4c5883d8b
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-09-14 13:18:15 +08:00
Ayush Saxena d9eb5ad6d3
HADOOP-17900. Move ClusterStorageCapacityExceededException to Public from LimitedPrivate. (#3404). Contributed by Ayush Saxena. 2021-09-13 22:50:39 +05:30
Ayush Saxena 17c58ac959
HDFS-16223. AvailableSpaceRackFaultTolerantBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. (#3424). Contributed by Ayush Saxena. 2021-09-13 17:54:38 +08:00
bshashikant 356ebbbe80
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) 2021-09-13 10:25:19 +05:30
jianghuazhu edfde6eebc
HDFS-16197.Simplify getting NNStorage in FSNamesystem. (#3357) 2021-09-12 19:58:30 +08:00
Symious 7f6553af75
HDFS-16221. RBF: Add usage of refreshCallQueue for Router (#3421) 2021-09-12 19:52:47 +08:00
LeonGao 90bc688c78
HDFS-16188. RBF: Router to support resolving monitored namenodes with DNS (#3346) Contributed by Leon Gao 
* Router to support resolving monitored namenodes with DNS

* Style

* fix style and test failure

* Add test for NNHAServiceTarget const

* Resolve comments

* Fix test

* Comments and style

* Create a simple function to extract port

* Use LambdaTestUtils.intercept

* fix javadoc

* Trigger Build
 2021-09-10 16:40:08 -07:00
Symious 99cb2b6b7f
HDFS-16065. RBF: Add metrics to record Router's operations (#3100) 2021-09-09 18:45:23 +08:00
Ahmed Hussein bddc9bf63c
HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 2021-09-08 21:21:16 -07:00
Symious c0890e6d04
HDFS-16210. RBF: Add the option of refreshCallQueue to RouterAdmin (#3379) 2021-09-09 09:57:27 +08:00
litao 5a306955a4
HDFS-16209. Add description for dfs.namenode.caching.enabled (#3378) 2021-09-08 12:38:55 +08:00
jianghuazhu 4e4d4fcc7c
HDFS-16204. Improve FSDirEncryptionZoneOp related parameter comments. (#3368) 2021-09-06 18:06:12 +08:00
Renukaprasad C dd6b2cebed
HDFS-16138. BlockReportProcessingThread exit doesn't print the actual stack (#3244). Contributed by Renukaprasad C. 2021-09-05 17:07:59 +08:00
litao 364a1fbfb3
HDFS-16194. Simplify the code with DatanodeID#getXferAddrWithHostname (#3354). Contributed by tomscut. 2021-09-05 14:19:01 +08:00
Viraj Jasani cc6b545365
HDFS-16199. Resolve log placeholders in NamenodeBeanMetrics (#3362). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-04 16:20:08 +05:30
Weison Wei a610f6d9c3
HDFS-16202. Use constants "HdfsClientConfigKeys.Failover.PREFIX" instead of "dfs.client.failover." (#3367). Contributed by Weison Wei. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-09-03 12:05:53 +05:30
langlaile1221 24af52efab
HDFS-15966. Empty the statistical parameters when emptying the redundant queue (#2894) 2021-09-01 19:37:34 +08:00
Uma Maheswara Rao G 164608b546
HDFS-16192: ViewDistributedFileSystem#rename wrongly using src in the place of dst. (#3353) 
Co-authored-by: Uma Maheswara Rao G <umagangumalla@cloudera.com>
 2021-08-31 12:25:03 +08:00
Viraj Jasani aa9cdf2af6
HDFS-16143. Add Timer in EditLogTailer and de-flake TestEditLogTailer#testStandbyTriggersLogRollsWhenTailInProgressEdits (#3235) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-26 00:37:38 -07:00
LeonGao b53cae0ffb
HDFS-16157. Support configuring DNS record to get list of journal nodes contributed by Leon Gao. (#3284) 
* Add DNS resolution for QJM

* Add log

* Resolve comments

* checkstyle

* typo
 2021-08-25 17:40:12 -07:00
Viraj Jasani 1b9927afe1
HDFS-16184. De-flake TestBlockScanner#testSkipRecentAccessFile (#3329) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-08-25 17:37:25 +09:00
Neil 9084c728eb
HDFS-16180.FsVolumeImpl.nextBlock should consider that the block meta file has been deleted. (#3315) 2021-08-24 11:15:47 +08:00
Gautham B A b6d1971820
HDFS-16178. Make recursive rmdir in libhdfs++ cross platform (#3311) 2021-08-20 10:04:55 -07:00
litao 07627ef19e
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-08-19 12:30:38 +08:00
litao 0e922aab41
HDFS-16160. Improve the parameter annotation in DatanodeProtocol#sendHeartbeat (#3291) 2021-08-19 09:51:00 +08:00
Gautham B A 409878ef80
HDFS-16174. Refactor TempFile and TempDir in libhdfs++ (#3303) 2021-08-17 09:49:52 -07:00
jianghuazhu 53a2c65694
HDFS-16162.Improve DFSUtil#checkProtectedDescendants() related parameter comments. (#3295) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-17 16:47:10 +08:00
Viraj Jasani 6342d5e523
HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-16 14:54:25 +09:00
Viraj Jasani 919a99bbb4
HDFS-16163. Avoid locking entire blockPinningFailures map (#3296) 2021-08-16 13:40:05 +08:00
Gautham B A 77383a4d55
HDFS-15976. Make mkdtemp cross platform (#2908) 2021-08-10 09:23:28 -07:00
Akira Ajisaka 4972e7a246
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 2021-08-07 11:27:25 +09:00
wangzhaohui e85c44657c
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-08-06 18:23:00 +08:00
wangzhaohui a73b64f86b
HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 2021-08-06 09:27:05 +01:00
zhangshuyan0 10a2526b0b
HDFS-16146. All three replicas are lost due to not adding a new DataN… (#3247) Contributed by Shuyan Zhang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-08-04 00:22:21 +08:00
Viraj Jasani ccfa072dc7
HADOOP-17612. Upgrade Zookeeper to 3.6.3 and Curator to 5.2.0 (#3241) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-08-03 14:44:00 +09:00
Wei-Chiu Chuang 6d77f3b6cd
HDFS-14529. SetTimes to throw FileNotFoundException if inode is not found (#3243) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-30 08:29:56 -07:00
jianghuazhu 6f730fd25c
HDFS-15936.Solve SocketTimeoutException#sendPacket() does not record SocketTimeout exception. (#2836) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
Reviewed-by: cxorm <lianp964@gmail.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-07-28 21:57:28 -07:00
He Xiaoqiao 683feaa1d4
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
 2021-07-29 11:19:17 +08:00
S O'Donnell 4eae284827 HDFS-16144. Revert HDFS-15372 (Files in snapshots no longer see attribute provider permissions). Contributed by Stephen O'Donnell 2021-07-28 14:49:23 +01:00
zhihaixu2012 ae20516ebc
HDFS-16111. Add a configuration to RoundRobinVolumeChoosingPolicy to avoid failed volumes at datanodes. (#3175) 
Change-Id: Iead25812d4073e3980893e3e76f7d2b03b57442a

Co-authored-by: Zhihai Xu <zxu@apache.org>
 2021-07-27 19:18:44 -07:00
JiaguodongF aecfcf165f
HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid. (#3185) 
* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

* HDFS-16119. start balancer with parameters -hotBlockTimeInterval xxx is invalid

Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-27 22:25:21 +05:30
Viraj Jasani b038042ece
HDFS-16139. Update BPServiceActor Scheduler's nextBlockReportTime atomically (#3228). Contributed by Viraj Jasani. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-27 14:57:12 +08:00
Akira Ajisaka d710ec8d85
HDFS-16140. TestBootstrapAliasmap fails by BindException. (#3229) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-07-26 16:46:07 +09:00
litao 5d765497c5
HDFS-16131. Show storage type for failed volumes on namenode web (#3211). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-07-26 12:45:01 +08:00
Akira Ajisaka 2f2f822488
HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml
 2021-07-26 01:53:39 +09:00
Eric Yin de41ce8a16
HDFS-16087. Fix stuck issue in rbfbalance tool (#3141). Contributed by Eric Yin. 2021-07-21 00:01:55 +08:00
He Xiaoqiao 6ed7670a93
HDFS-16067. Support Append API in NNThroughputBenchmark. Contributed by Renukaprasad C. 2021-07-17 21:57:00 +08:00
Kihwal Lee 47002719f2 HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 2021-07-16 14:22:39 -05:00
crossfire 4bb25c810b
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 2021-07-15 19:58:12 -07:00
Abhishek Das 1dd03cc4b5 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 2021-07-13 18:11:50 -07:00
LeonGao ea90c5117d
HDFS-15785. Datanode to support using DNS to resolve nameservices to IP addresses to get list of namenodes. (#2639) 
* Rebase trunk

* Fix to use FQDN and update config name

* Fix javac

* Style and trigger build

* Trigger Build after force push

* Trigger Build

* Fix config names
 2021-07-13 14:18:59 -07:00
Artem Smotrakov ba325a8ada
HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-10 13:42:31 +09:00
S O'Donnell b6c06c4b76 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 2021-07-09 10:03:57 +01:00
bshashikant 6450c1ea9a
HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 2021-07-09 10:26:42 +05:30
Viraj Jasani 618c9218ee
HADOOP-17788. Replace IOUtils#closeQuietly usages by Hadoop's own utility (#3171) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-07-08 16:03:40 +09:00
litao b4c2647d0d
HDFS-16088. Standby NameNode process getLiveDatanodeStorageReport req… (#3140) 2021-07-08 14:10:45 +08:00
JiaguodongF 9381071758
HDFS-16114. the balancer parameters print error (#3179) 
Co-authored-by: jiaguodong5 <jiaguodong5@jd.com>
 2021-07-07 14:13:05 +05:30
Ayush Saxena c85a18327d HDFS-16101. Remove unuse variable and IoException in ProvidedStorageMap. Contributed by lei w. 2021-07-06 23:26:52 +05:30
litao a447cfd78b
HDFS-16110. Remove unused method reportChecksumFailure in DFSClient (#3174) 
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-07-05 19:31:22 -07:00
Viraj Jasani da575a53cf
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 2021-07-05 09:23:24 +08:00
litao b87bac13e4
HDFS-16109. Fix flaky some unit tests since they offen timeout (#3172) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-07-05 08:14:37 +09:00
litao 390f8603d3
HDFS-16106. Fix flaky unit test TestDFSShell (#3168) 2021-07-02 15:31:25 +08:00
zhangshuyan0 9be17339eb
HDFS-16099. Make bpServiceToActive to be volatile. (#3159) 
Co-authored-by: zhangshuyan <zqingchai@gmail.com>
 2021-07-01 23:52:21 +08:00
litao 52b9319e57
HDFS-16104. Remove unused parameter and fix java doc for DiskBalancerCLI (#3160). Contributed by tomscut. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-07-01 18:44:10 +05:30
litao 56c7ada7a5
HDFS-16086. Add volume information to datanode log for tracing (#3136) 2021-06-30 22:06:56 -07:00
Viraj Jasani d2e7003400
HDFS-16090. Fine grained lock for datanodeNetworkCounts (#3148) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
    Signed-off-by: stack Chuang <stack@apache.org>
    Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-30 12:55:07 -07:00
Xiangyi Zhu 4cac6ec405
HDFS-16096. Delete useless method DirectoryWithQuotaFeature#setQuota (#3156) 
Co-authored-by: zhuxiangyi <43412979@qq.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-06-30 00:12:08 -07:00
litao 95454d821c
HDFS-16089. Add metric EcReconstructionValidateTimeMillis for StripedBlockReconstructor (#3146) 2021-06-29 03:15:12 -07:00
Viraj Jasani 93a1685073
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 2021-06-29 02:31:02 -07:00
He Xiaoqiao 748570b73c
Revert "HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun." 
This reverts commit 0d07837712.
 2021-06-25 22:38:34 +08:00
He Xiaoqiao 0d07837712
HDFS-16044. Fix getListing call getLocatedBlocks even source is a directory. Contributed by ludun. 2021-06-25 01:09:42 +08:00
litao f78b6d47f9
HDFS-16085. Move the getPermissionChecker out of the read lock (#3134). Contributed by tomscut. 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-24 15:59:42 +05:30
litao fdef2b4cca
HDFS-16076. Avoid using slow DataNodes for reading by sorting locations (#3117) 2021-06-24 11:27:40 +09:00
Gautham B A ef5dbc7283
HADOOP-17766. CI for Debian 10 (#3129) 2021-06-23 10:02:33 -07:00
Viraj Jasani d9fbb3c508
HDFS-16082. Atomic operations on exceptionsSinceLastBalance and failedTimesSinceLastSuccessfulBalance in Balancer (#3127) 2021-06-22 20:21:34 -07:00
He Xiaoqiao 10b79a26fe
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-22 00:32:55 +08:00
Viraj Jasani c488abbc79
HDFS-16075. Use empty array constants present in StorageType and DatanodeInfo to avoid creating redundant objects (#3115) 
Reviewed-by: Hui Fei <ferhui@apache.org>
 2021-06-21 10:25:12 +09:00
litao bdc9c8809e
HDFS-16079. Improve the block state change log (#3120). Contributed by tomscut. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:23:11 +05:30
Viraj Jasani d9eccb1eb3
HDFS-16080. RBF: Invoking method in all locations should break the loop after successful result (#3121). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 17:19:05 +05:30
Ahmed Hussein c255feb5d7
HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:22:47 +05:30
litao c491f81a30
HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:20:52 +05:30
LeonGao 643dfd60e4
HDFS-15842. HDFS mover to emit metrics. (#2738) 2021-06-19 15:39:46 -07:00
Wei-Chiu Chuang 1ad674e548
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
 2021-06-16 20:37:56 -07:00
hust_hhb 56d249759f
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3065) 2021-06-17 11:08:29 +08:00
Ayush Saxena 2b304ad645 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:53:12 +05:30
Wei-Chiu Chuang a77bf7cf07 Make upstream aware of 3.3.1 release 2021-06-15 15:43:36 +00:00
Takanobu Asanuma 839fcf7682
HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 2021-06-15 13:20:09 +05:30
Viraj Jasani c1bf3cb0da
HDFS-16016. BPServiceActor to provide new thread to handle IBR (#2998) 
Contributed by Viraj Jasani
 2021-06-14 14:23:50 -07:00
Siyao Meng ebee2aed00
HDFS-16055. Quota is not preserved in snapshot INode (#3078) 2021-06-14 10:48:22 -07:00
Ahmed Hussein 352949d070
HDFS-15671. testBalancerRPCDelayQpsDefault fails intermittently (#3097) 2021-06-11 11:00:42 -05:00
litao 51ebf7b2a0
HDFS-16057. Make sure the order for location in ENTERING_MAINTENANCE state (#3084) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-06-11 17:30:55 +09:00
Viraj Jasani 4ef27a596f
HADOOP-17753. Keep restrict-imports-enforcer-rule for Guava Lists in top level hadoop-main pom (#3087) 2021-06-11 12:15:52 +09:00
Viraj Jasani eefa664fea
HDFS-16054. Replace Guava Lists usage by Hadoop's own Lists in hadoop-hdfs-project (#3073) 2021-06-09 15:12:48 +09:00
Ayush Saxena a1a3184171
HDFS-15916. Addendum. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#3056) 2021-06-08 19:34:45 -07:00
Ahmed Hussein a2a0283c7b
HDFS-16042. DatanodeAdminMonitor scan should be delay based (#3058) 2021-06-08 11:09:31 -05:00
litao c748fce17a
HDFS-16048. RBF: Print network topology on the router web (#3062) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hemanth Boyina <hemanthboyina@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
 2021-06-08 16:14:06 +09:00
huhaiyang b38b00e528
HDFS-15998. Fix NullPointException In listOpenFiles (#3036) 
Co-authored-by: huhaiyang <huhaiyang@didichuxing.com>
 2021-06-01 00:26:47 -07:00
Ayush Saxena b8ab19373d Revert "HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927)" 
This reverts commit 041488e8f5.
 2021-06-01 08:38:13 +05:30
Ayush Saxena bce14e746b HDFS-16051. Misspelt words in DataXceiver.java line 881 and line 885. Contributed by Ning Sheng. 2021-06-01 08:34:56 +05:30
Jinglun e90c41af34 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 19:28:40 +08:00
Jinglun 2b11e160e2 Revert "HDFS-15973." because the user.name and user.email message is not correct. 
This reverts commit b7f69467c1.
 2021-05-30 19:27:10 +08:00
李经纶 b7f69467c1 HDFS-15973. RBF: Add permission check before doing router federation rename. Contributed by Jinglun. 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-30 18:50:03 +08:00
Konstantin V Shvachko 8ce30f51f9 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 2021-05-27 18:47:20 -07:00
Konstantin V Shvachko 1abd03d68f HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 2021-05-26 14:59:50 -07:00
Gautham B A bcaeb1ac8c
Revert "Revert "HDFS-15971. Make mkstemp cross platform (#2898)"" (#3044) 2021-05-26 14:12:53 -07:00
Hui Fei 98a74e2351
HDFS-16041. TestErasureCodingCLI fails (#3052) 2021-05-26 20:12:17 +09:00
Viraj Jasani 986d0a4f1d
HADOOP-17732. Keep restrict-imports-enforcer-rule for Guava Sets in hadoop-main pom (#3049) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-26 17:14:31 +09:00
Narges Shadab 3fdeb7435a
HDFS-16031. Possible Resource Leak in org.apache.hadoop.hdfs.server.aliasmap#InMemoryAliasMap (#3027) 2021-05-26 14:43:41 +09:00
zhuxiangyi e18d806212
HDFS-16024: RBF: Rename data to the Trash should be based on src loca… (#3009) 2021-05-26 13:41:49 +08:00
Viraj Jasani 028ec4704b
HADOOP-17720. Replace Guava Sets usage by Hadoop's own Sets in HDFS (#3031) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-05-25 10:55:58 +09:00
lfengnan 43bf009112
HDFS-15757 RBF: Improving Router Connection Management (#2651) 2021-05-19 10:53:42 -07:00
huhaiyang acd712ca92
HDFS-15877. BlockReconstructionWork should resetTargets() before BlockManager#validateReconstructionWork return false (#2747) 2021-05-19 09:04:41 +08:00
Ayush Saxena fcd4140e5f
HDFS-15988. Stabilise HDFS Pre-Commit. (#2860). Contributed by Ayush Saxena. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-05-16 00:45:11 +05:30
Viraj Jasani b944084b32
HDFS-16007. Deserialization of ReplicaState should avoid throwing ArrayIndexOutOfBoundsException (#2982) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-11 12:38:15 +09:00
Siyao Meng 8d5cc98b42
HDFS-15997. Implement dfsadmin -provisionSnapshotTrash -all (#2958) 2021-05-10 12:41:07 -07:00
Ayush Saxena 359c0c1d1e HDFS-16003. ProcessReport print invalidatedBlocks should judge debug level at first. Contributed by lei w. 2021-05-10 17:09:51 +05:30
Jinglun 1b6994287a HDFS-15923. RBF: Authentication failed when rename accross sub clusters (#2819). Contributed by zhuobin zheng. 
Reviewed-by: Jinglun <jinglun@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-05-08 19:50:17 +08:00
Akira Ajisaka 9aa651f0b4
HDFS-16001. TestOfflineEditsViewer.testStored() fails reading negative value of FSEditLogOpCodes. (#2980) 
Reviewed-by: Xiaoqiao He <hexiaoqiao@apache.org>
 2021-05-08 00:24:06 +09:00
zhuqi 76c69c39d9
HDFS-15934. Make DirectoryScanner reconcile blocks batch size and interval between batch configurable. Contributed by Qi Zhu. (#2833) 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-05-05 21:02:13 +05:30
Viraj Jasani 041488e8f5
HDFS-15982. Deleted data using HTTP API should be saved to the trash (#2927) 
Reviewed-by: Siyao Meng <siyao@apache.org>
 2021-05-04 01:31:23 -07:00
Akira Ajisaka 7c561e2a02
HDFS-15952. TestRouterRpcMultiDestination#testProxyGetTransactionID and testProxyVersionRequest are flaky. (#2970) 2021-05-04 00:25:29 +09:00
lfengnan 6e525ab81c
HDFS-15810. RBF: RBFMetrics's TotalCapacity out of bounds (#2910) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 19:18:47 +09:00
Wei-Chiu Chuang f1e1809029
HADOOP-17653. Do not use guava's Files.createTempDir(). (#2945) 
Reviewed-by: Steve Loughran <stevel@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-05-02 11:10:32 +09:00
Karthik Palanisamy bd3da73a0f
HDFS-15865. Interrupt DataStreamer thread if no ack (#2728) 2021-05-01 23:35:31 +05:30
lfengnan 552e9dcc6c
HDFS-15561. RBF: Remove NPE when local namenode is not configured (#2954). Contributed by Fengnan Li. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-29 21:14:18 +08:00
Wei-Chiu Chuang b2e54762a4
HDFS-15624. fix the function of setting quota by storage type (#2377) (#2955) 
1. puts NVDIMM to the end of storage type enum to make sure compatibility.
2. adds check to make sure the software layout version is satisfied

Co-authored-by: su xu <kevinbrandon@163.com>
Co-authored-by: huangtianhua <huangtianhua223@gmail.com>
Co-authored-by: YaYun-Wang <34060507+YaYun-Wang@users.noreply.github.com>

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Vinayakumar B <vinayakumarb@apache.org>

Change-Id: I3c58beef50730827a09b3c968e9ad637baa57d44
 2021-04-28 23:54:39 -07:00
Wei-Chiu Chuang f41a368c14
HADOOP-11245. Update NFS gateway to use Netty4 (#2832) 
Reviewed-by: Tsz-Wo Nicholas Sze <szetszwo@apache.org>
 2021-04-28 23:43:11 -07:00
bshashikant ef13f8ad6b
HDFS-15961. standby namenode failed to start ordered snapshot deletion is enabled while having snapshottable directories (#2881) 2021-04-27 09:15:06 +05:30
Stephen O'Donnell 605ed85c29
HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 2021-04-26 11:00:23 +01:00
litao b968fa0957
HDFS-15991. Add location into datanode info for NameNodeMXBean (#2933) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-26 18:38:43 +09:00
Takanobu Asanuma 01bad0e92a HDFS-15967. Improve the log for Short Circuit Local Reads. Contributed by Bhavik Patel. 2021-04-26 13:29:28 +09:00
Brahma Reddy Battula 2621d3f15b HDFS-15566. NN restart fails after RollingUpgrade from 3.1.3/3.2.1 to 3.3.0. Contributed by Brahma Reddy Battula. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-26 11:39:50 +08:00
Wei-Chiu Chuang 90c6caf650 Revert "HDFS-15624. fix the function of setting quota by storage type (#2377)" 
This reverts commit 394b9f7a5c.

Ref: HDFS-15995.
Had to revert this commit, so we can commit HDFS-15566 (a critical bug preventing rolling upgrade to Hadoop 3.3)
Will re-work this fix again later.
 2021-04-26 11:27:15 +08:00
jianghuazhu c486dc396c
HDFS-15978. Solve DatanodeManager#getBlockRecoveryCommand() printing IOException. (#2913) Contributed by JiangHua Zhu. 2021-04-25 14:31:51 +08:00
zhuxiangyi dff95c5eca
HDFS-15974. RBF: Unable to display the datanode UI of the router (#2915) 
Co-authored-by: zhuxiangyi <zhuxiangyi@360.cn>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 10:19:11 +09:00
Viraj Jasani acb49f2fc3
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2923) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-21 19:45:49 +09:00
litao 85a3532849
HDFS-15970. Print network topology on the web (#2896) 2021-04-19 20:42:30 +09:00
litao af0448d37b
HDFS-15975. Use LongAdder instead of AtomicLong (#2907) 2021-04-17 16:49:09 +09:00
Eric Badger aed13f0f42 Revert "HDFS-15971. Make mkstemp cross platform (#2898)" 
This reverts commit b088f46b68.
 2021-04-16 17:15:33 +00:00
zhangshuyan0 14816be0b1
HDFS-15963. Unreleased volume references cause an infinite loop. (#2889) Contributed by Shuyan Zhang. 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-17 00:08:31 +08:00
Akira Ajisaka f0241ec216
HDFS-15977. Call explicit_bzero only if it is available. (#2914) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
 2021-04-16 13:26:45 +09:00
lfengnan 17be99f9f9
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2903) 2021-04-15 16:11:47 -07:00
Vivek Ratnavel Subramanian c821008836
HDFS-15850. Superuser actions should be reported to external enforcers (#2784) 2021-04-15 14:45:51 -07:00
Masatake Iwasaki 2717203f85
HADOOP-17569. Building native code fails on Fedora 33. (#2886) 2021-04-15 21:33:18 +09:00
Ayush Saxena 7d6f0ca0bd Revert "HDFS-15884. RBF: Remove unused method getCreateLocation in RouterRpcServer (#2754). Contributed by tomscut." 
This reverts commit e565b05c80.

The removed method needs to be used in HDFS-15423.
 2021-04-13 23:28:42 +05:30
Gautham B A b088f46b68
HDFS-15971. Make mkstemp cross platform (#2898) 2021-04-13 08:54:45 -07:00
Inigo Goiri 82462739f8
Revert "HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605)" (#2900) 
This reverts commit cb3ed32fe0.
 2021-04-12 20:44:31 -07:00
lfengnan cb3ed32fe0
HDFS-15423 RBF: WebHDFS create shouldn't choose DN from all sub-clusters (#2605) 2021-04-12 12:42:33 -07:00
Gautham B A 3148791da4
HDFS-15962. Make strcasecmp cross platform (#2883) 2021-04-09 09:01:33 -07:00
Viraj Jasani 56bd968fb4
HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874) 2021-04-09 11:10:18 +09:00
Gautham B A bf66116407
HDFS-15955. Make explicit_bzero cross platform (#2875) 2021-04-08 09:44:47 -07:00
Ayush Saxena c6539e3289
HDFS-15916. DistCp: Backward compatibility: Distcp fails from Hadoop 3 to Hadoop 2 for snapshotdiff. (#2863). Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2021-04-08 20:49:08 +05:30
Stephen O'Donnell 4c567fcff7
HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 2021-04-08 11:59:02 +01:00
Gautham B A 42ddb5c6fe
HDFS-15909. Make fnmatch cross platform (#2792) 2021-04-06 09:23:21 -07:00
He Xiaoqiao 459df41997
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 2021-04-06 21:22:48 +08:00
Gautham B A 36014b8282
HDFS-15949. Fix integer overflow (#2857) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-06 18:22:54 +09:00
Viraj Jasani 3f2682b92b
HADOOP-17622. Avoid usage of deprecated IOUtils#cleanup API. (#2862) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 13:39:10 +09:00
Viraj Jasani de05cefbae
HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-06 00:19:21 +05:30
Gautham B A 6e1df1c048
HDFS-15947. Replace deprecated protobuf APIs (#2856) 2021-04-05 09:52:48 -07:00
litao 9e2d5d6d05
HDFS-15951. Remove unused parameters in NameNodeProxiesClient (#2859) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-05 17:47:10 +09:00
Gautham B A ca5e13fd74
HDFS-15950. Remove unused hdfs.proto import (#2858) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-04-05 11:42:05 +09:00
zhuqi 96e410a127
HDFS-15930: Fix some @param errors in DirectoryScanner. (#2829). Contributed by Qi Zhu. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-04-04 03:11:02 +05:30
litao 3cb76447f5
HDFS-15946. Fix java doc in FSPermissionChecker (#2855). Contributed by tomscut. 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-02 10:37:04 -07:00
Gautham B A 70536ba1f9
HDFS-15944. Prevent truncation by snprintf (#2853) 2021-04-02 09:37:33 -07:00
Brahma Reddy Battula ed74479ea5 HDFS-15222. Correct the "hdfs fsck -list-corruptfileblocks" command output. Contributed by Ravuri Sushma sree. 2021-04-02 09:47:20 +05:30
litao 7c83f140dc
HDFS-15938. Fix java doc in FSEditLog (#2837). Contributed by tomscut. 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 19:28:17 -07:00
litao 4bd04126d6
HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-02 10:33:40 +09:00
Viraj Jasani 4f28738010
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:34:31 -07:00
Brahma Reddy Battula 0665ce9930 HDFS-15494. TestReplicaCachingGetSpaceUsed#testReplicaCachingGetSpaceUsedByRBWReplica Fails on Windows. Contributed by Ravuri Sushma sree. 2021-04-01 09:19:39 +05:30
He Xiaoqiao 5eca748970
HDFS-15921. Improve the log for the Storage Policy Operations. Contributed by Bhavik Patel. 2021-04-01 11:32:03 +08:00
Borislav Iordanov 2c482fbacf HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted
(and yet again because of commit message format).

Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:46:35 -07:00
stack 22961a615d Revert "HADOOP-16524. Automatic keystore reloading for HttpServer2" 
This reverts commit a2975d2153.
 2021-03-31 10:43:09 -07:00
stack a2975d2153 HADOOP-16524. Automatic keystore reloading for HttpServer2 
Reapply of issue reverted first because it caused yarn failures and
then again because the commit message was incorrectly formatted.
 2021-03-31 10:40:20 -07:00
stack 5183aaeda2 Revert "Hadoop 16524 - resubmission following some unit test fixes (#2693)" 
Revert to fix the summary message.

This reverts commit 9509bebf7f.
 2021-03-31 10:39:55 -07:00
Borislav Iordanov 9509bebf7f
Hadoop 16524 - resubmission following some unit test fixes (#2693) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 10:07:42 -07:00
He Xiaoqiao da4ceba4aa
HDFS-15863. RBF: Validation message to be corrected in FairnessPolicyController. Contributed by Renukaprasad C. 2021-03-31 13:00:13 +08:00
Gautham B A 7dfff496fa
HDFS-15929. Replace RAND_pseudo_bytes in util.cc (#2826) 2021-03-30 13:36:02 -07:00
Gautham B A 8668abf87e
HDFS-15927. Catch polymorphic type by reference (#2824) 2021-03-30 13:35:45 -07:00
Gautham B A 50b1e1c0c4
HDFS-15928. Replace RAND_pseudo_bytes in rpc_engine.cc (#2825) 2021-03-30 10:37:10 -07:00
Viraj Jasani 90fdd04f54
HDFS-15926 : Removed duplicate dependency of hadoop-annotations (#2823). Contributed by Viraj Jasani. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-29 23:22:54 +05:30
Gautham B A 6a3e135316
HDFS-15922. Use memcpy for copying non-null terminated string. (#2827) 2021-03-29 10:38:59 -07:00
Harunobu Daikoku ea6595d3b6
HDFS-15900. RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode. (#2787) 
Co-authored-by: Harunobu Daikoku <hdaikoku@yahoo-corp.jp>
 2021-03-29 11:43:29 +09:00
Ayush Saxena cd2501e54b HDFS-15764. Notify Namenode missing or new block on disk as soon as possible. Contributed by Yang Yun. 2021-03-28 16:13:59 +05:30
litao 72037a63b1
HDFS-15879. Exclude slow nodes when choose targets for blocks (#2748) 
Reviewed-by: Dinesh Chitlangia <dineshc@apache.org>
Reviewed-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-03-27 17:57:09 +09:00
Gautham B A 1fed18bb2d
HDFS-15922. Use memcpy for copying non-null terminated string in jni_helper.c (#2818) 2021-03-26 09:26:35 -07:00
Akira Ajisaka af1f9f43ea
HADOOP-17133. Implement HttpServer2 metrics (#2145) 2021-03-25 12:09:43 -07:00
He Xiaoqiao ed5814deff
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:24:14 +08:00
Gautham B A 654555783d
HDFS-15918. Replace deprecated RAND_pseudo_bytes (#2811) 2021-03-24 10:52:33 -07:00
touchida 95e6892675
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) 2021-03-24 16:56:09 +08:00
Takanobu Asanuma b4d97a8dc7 HDFS-15902. Improve the log for HTTPFS server operation. Contributed by Bhavik Patel. 2021-03-24 17:51:35 +09:00
Viraj Jasani 4b4ccce02f
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2794) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 22:47:45 -07:00
Ayush Saxena 569e407f64 HDFS-15907. Reduce Memory Overhead of AclFeature by avoiding AtomicInteger. Contributed by Stephen O'Donnell. 2021-03-24 02:10:02 +05:30
Gautham B A 714427c414
HDFS-15910. Improve security with explicit_bzero (#2793) 2021-03-23 09:58:19 -07:00
Jack Jiang 9c2a712597
HDFS-15913. Remove useless NULL checks before instanceof (#2805) 2021-03-23 08:51:35 -07:00
Narges Shadab d77c7ab4e0
HDFS-15908. Possible Resource Leak in org.apache.hadoop.hdfs.qjournal.server.Journal (#2790). Contributed by Narges Shadab. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-23 17:06:58 +05:30
Gautham B A 61bfa42389
HDFS-15903. Refactor X-Platform lib (#2783) 2021-03-22 08:40:54 -07:00
litao 343ce8a8c9
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2788) 2021-03-22 20:29:17 +09:00
He Xiaoqiao 03a3a04c06
HDFS-15787. Remove unnecessary Lease Renew in FSNamesystem#internalReleaseLease. Contributed by Lisheng Sun. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-22 18:37:00 +08:00
Viraj Jasani 261191cbc0
HDFS-15904 : De-flake TestBalancer#testBalancerWithSortTopNodes() (#2785) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-19 20:37:44 -07:00
Narges Shadab f5f3fc6fe9
HDFS-15868. Possible Resource Leak in EditLogFileOutputStream (#2736). Contributed by Narges Shadab. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2021-03-19 16:36:48 +08:00
Viraj Jasani 9c43b60348
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 2021-03-17 15:12:36 +09:00