Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,878 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8579 Commits

Author SHA1 Message Date
Masatake Iwasaki 0fbb62ab8a Revert "HADOOP-17196. Fix C/C++ standard warnings (#2208)" 
This reverts commit ff907b310d.
 2022-06-30 00:48:50 +00:00
KevinWikant 4dad353597 HDFS-16064. Determine when to invalidate corrupt replicas based on number of usable replicas (#4410) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit cfceaebde6)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDecommission.java
 2022-06-20 11:40:11 +09:00
xuzq aba4418981 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a473)
(cherry picked from commit ee3ee98ee5)
 2022-06-10 19:33:36 +00:00
Viraj Jasani 557ab95dc2 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 17:11:57 +08:00
Wei-Chiu Chuang 4ebdc38634
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication. (#4358) 2022-06-04 06:56:07 +08:00
Stephen O'Donnell 58513d367e HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 7d6b133af3)
 2022-06-01 21:05:21 +01:00
Stephen O'Donnell 52e405390e
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop (#4394) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-06-01 07:55:48 +08:00
Ashutosh Gupta cc52627ad4
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785)
(cherry picked from commit d921cc71fd)
 2022-05-26 15:28:42 -07:00
Michael Stack ad9b6e8d81
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4348) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 16:35:56 -07:00
litao 48ca153100
HDFS-16579. Fix build failure for TestBlockManager on branch-3.2 (#4310) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-15 17:59:23 +09:00
Ashutosh Gupta 4a532f77a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9)
 2022-05-08 08:06:39 +09:00
litao 0965822ae1 HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc480)
 2022-04-23 12:18:19 +09:00
Inigo Goiri 27621c9eef HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B. 
(cherry picked from commit f069d38c8d)
 2022-04-22 16:00:40 +09:00
daimin 181efebdfe
HDFS-16546. Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection (#4187) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-22 15:34:28 +09:00
S O'Donnell bacd0b9a92 Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 045d48530c.
 2022-04-20 20:57:23 +01:00
qinyuren 4fed01ef21 HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd17374)
 2022-04-20 15:08:05 +09:00
Stephen O'Donnell 045d48530c HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee0363)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2022-04-19 11:38:34 +01:00
qinyuren d993d22038 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0)
 2022-04-19 13:40:16 +09:00
jianghuazhu ed83a278c2
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e712)
(cherry picked from commit 4aa9698ee1e9a10a0f882229036bbb4db1ede939)
 2022-04-19 11:13:29 +08:00
daimin 248c548616
Fix decommission UnsupportedOperationException (#4172) 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 10:40:12 +08:00
Takanobu Asanuma b8c6ba6046 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959)
 2022-04-14 11:52:12 +09:00
qinyuren cb14e8de81 HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a1)
 2022-04-13 11:53:58 +09:00
Stephen O'Donnell 72e48ac84e HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf)
 2022-04-06 13:06:28 +01:00
wangzhaohui 545d3a6cb7 HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a7)
 2022-04-06 17:31:42 +09:00
litao 6b24b85e3f HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:04:04 -07:00
Masatake Iwasaki 228ad1ac25 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a244)
 2022-03-28 08:09:58 +00:00
GuoPhilipse ff03212145 HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4)
 2022-03-27 22:33:17 +09:00
qinyuren 66be0a713f HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50)
 2022-03-23 15:06:20 +09:00
Masatake Iwasaki e13408d9d2
HADOOP-18088. Replace log4j 1.x with reload4j. (#4084) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-23 12:57:27 +09:00
Surendra Singh Lilhore 58fe5b2f35 HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 
(cherry picked from commit 5ff76cb8bc)
 2022-03-18 18:39:20 +09:00
Thinker313 6f815a546a
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 43153e80cb)
 2022-03-12 20:10:44 +08:00
Ayush Saxena 551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
 2022-02-23 11:41:35 +08:00
Chentao Yu 5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:07:46 -08:00
singer-bin 0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
(cherry picked from commit ce7cabb771)
 2022-02-06 13:37:16 +08:00
daimin 337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
KevinWikant 3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant 543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:14:42 +09:00
litao fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:45:47 +09:00
daimin 0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c)
 2022-01-25 10:01:09 +08:00
liubingxing 0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma 53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229.
 2022-01-20 22:07:41 +09:00
qinyuren 96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:27 +09:00
Jackson Wang 74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta 3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:14:52 +09:00
liever18 1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta 278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:26:09 +09:00
jianghuazhu 20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:28:00 +09:00
wangzhaohui c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 12:02:25 +00:00
jianghuazhu b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 12:08:30 +00:00