Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,793 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

9166 Commits

Author SHA1 Message Date
Masatake Iwasaki 877ef944f9
HADOOP-18088. Replace log4j 1.x with reload4j. (#4052) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-14 17:19:18 +01:00
litao d5845474e2
HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-04-14 17:19:18 +01:00
GuoPhilipse 63c07519de
HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)

Change-Id: I2cae5d1c27a492d896da5338a92c7a86f88a8b43
 2022-04-14 17:19:18 +01:00
qinyuren 376904e422
HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-04-14 17:19:18 +01:00
Thinker313 5c3fdf0f4b
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-04-14 17:19:17 +01:00
Ayush Saxena 51b3a5b22c
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)

Change-Id: I9f04082d650628bc1b8b62dacaaf472f8a578742
 2022-04-14 17:19:17 +01:00
Chentao Yu 38d448e40b
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-04-14 17:19:16 +01:00
singer-bin 980fab9168
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)

Change-Id: I03e4f2af17f0e4a8245c9c2c8ea1cb2cb41f777a
 2022-04-14 17:19:16 +01:00
Steve Loughran a1c0673526
HADOOP-18198. Preparing for 3.3.3 release 
Change-Id: Idebf79191dc91dad52073f2c63ee9ab3a99464d9
 2022-04-14 17:19:16 +01:00
Chao Sun f91452b289 Make upstream aware of 3.3.2 release 2022-03-02 19:09:35 -08:00
Xiangyi Zhu 6da346a358 HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 09:26:05 -08:00
Jackson Wang d7cacea07b HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 09:07:32 -08:00
Ashutosh Gupta 6ea9906a8b HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:01:38 -08:00
liever18 d167ef4850 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 10:01:20 -08:00
Wei-Chiu Chuang e42e483d00 Make upstream aware of 3.3.1 release 2022-01-04 14:51:49 -08:00
Ashutosh Gupta b9ed31f9ec HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 14:38:52 -08:00
jianghuazhu 2ddc66fc0c HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2022-01-04 14:38:19 -08:00
wangzhaohui bceab2e439 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2022-01-04 14:37:55 -08:00
jianghuazhu 82cf0aa296 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2022-01-04 14:37:52 -08:00
litao c0ae38c6b4 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2022-01-04 14:37:40 -08:00
litao 24917bae9f HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4)
 2022-01-04 14:37:37 -08:00
Ayush Saxena 57c9a8509f HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-04 14:37:26 -08:00
litao f16d9dfc31 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b)
 2022-01-04 14:37:22 -08:00
GuoPhilipse 45e6d22462 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2022-01-04 14:37:15 -08:00
Akira Ajisaka c257abca21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2022-01-04 14:37:08 -08:00
qinyuren 04f0025ad2 HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2022-01-04 14:37:04 -08:00
Takanobu Asanuma 1e3f94fa3c Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd.
 2022-01-04 14:36:57 -08:00
qinyuren 5214140123 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2022-01-04 14:36:54 -08:00
lzx404243 47436346b4 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c1235)
 2022-01-04 14:36:50 -08:00
LeonGao d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2022-01-04 14:36:42 -08:00
Takanobu Asanuma 906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106)
 2022-01-04 14:36:36 -08:00
huhaiyang e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b)
 2022-01-04 14:36:32 -08:00
huhaiyang 069426f5eb HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411e)
 2022-01-04 14:36:28 -08:00
litao ce0bc7b473 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4)
 2022-01-04 14:35:59 -08:00
litao 30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2022-01-04 14:35:56 -08:00
Akira Ajisaka 5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794.
 2022-01-04 14:35:49 -08:00
Viraj Jasani 544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2022-01-04 14:35:40 -08:00
Ayush Saxena 328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani 3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2022-01-04 14:35:33 -08:00
Mukul Kumar Singh 7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2022-01-04 14:34:39 -08:00
litao 8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
 2022-01-04 14:34:15 -08:00
litao 486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f97)
 2022-01-04 14:30:52 -08:00
GuoPhilipse abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc)
 2022-01-04 14:30:18 -08:00
litao c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2022-01-04 14:29:56 -08:00
GuoPhilipse 1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2022-01-04 14:29:19 -08:00
Viraj Jasani bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2022-01-04 14:28:54 -08:00
litao e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2022-01-04 14:28:28 -08:00
Viraj Jasani 1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5d)
 2022-01-04 14:27:52 -08:00
litao 3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00