Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,873 Commits 326 Branches 371 Tags
Commit Graph

8577 Commits

Author SHA1 Message Date
xuzq
aba4418981 HDFS-16623. Avoid IllegalArgumentException in LifelineSender (#4409) 
* HDFS-16623. Avoid IllegalArgumentException in LifelineSender

Co-authored-by: zengqiang.xu <zengqiang.xu@shopee.com>
(cherry picked from commit af5003a47311bad542964c42c1f776e4350446b9)
(cherry picked from commit ee3ee98ee5a4c34da7a4960b099686bdc1186a71)
 2022-06-10 19:33:36 +00:00
Viraj Jasani
557ab95dc2 HDFS-16618. sync_file_range error should include more volume/file info (#4402) 
Signed-off-by: Tao Li <tomscut@apache.org>
 2022-06-07 17:11:57 +08:00
Wei-Chiu Chuang
4ebdc38634
HDFS-16456. EC: Decommission a rack with only on dn will fail when the rack number is equal with replication. (#4358) 2022-06-04 06:56:07 +08:00
Stephen O'Donnell
58513d367e HDFS-16610. Make fsck read timeout configurable (#4384) 
(cherry picked from commit 34a973a90ef89b633c9b5c13a79aa1ac11c92eb5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 7d6b133af3f7ee4b986bdd2b266b83693614b6ab)
 2022-06-01 21:05:21 +01:00
Stephen O'Donnell
52e405390e
HDFS-16583. DatanodeAdminDefaultMonitor can get stuck in an infinite loop (#4394) 
Co-authored-by: S O'Donnell <sodonnell@cloudera.com>
 2022-06-01 07:55:48 +08:00
Ashutosh Gupta
cc52627ad4
HDFS-16585.Add @VisibleForTesting in Dispatcher.java (#4337) 
Co-authored-by: Ashutosh Gupta <ashugpt@amazon.com>

Reviewed-by: Tao Li <tomscut@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit bee538f785c012794a2dcf122dc8717b18beb0f0)
(cherry picked from commit d921cc71fdd2422fdf9f5417fd622580eb4438a2)
 2022-05-26 15:28:42 -07:00
Michael Stack
ad9b6e8d81
HDFS-16586. Purge FsDatasetAsyncDiskService threadgroup; it causes BP… (#4348) 
Remove the ThreadGroup used by executor factories; they are unused
and ThreadGroups auto-destroy when their Thread-member count goes to zero.
This behavior is incompatible with the configuration we have on the per-volume
executor which is set to let all threads die if no use inside the
keepalive time.
 2022-05-25 16:35:56 -07:00
litao
48ca153100
HDFS-16579. Fix build failure for TestBlockManager on branch-3.2 (#4310) 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-05-15 17:59:23 +09:00
Ashutosh Gupta
4a532f77a9
HDFS-16185. Fix comment in LowRedundancyBlocks.java (#4194) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 01027e52a9789eb5b386729f52b7bb9e52fa5352)
 2022-05-08 08:06:39 +09:00
litao
0965822ae1 HDFS-16552. Fix NPE for TestBlockManager (#4210) 
(cherry picked from commit 5ebbacc48023c0c71dd428df40a4352feede373b)
 2022-04-23 12:18:19 +09:00
Inigo Goiri
27621c9eef HDFS-13983. TestOfflineImageViewer crashes in windows. Contributed by Vinayakumar B. 
(cherry picked from commit f069d38c8d3c0bfa91b70a60e4e556ec708fc411)
 2022-04-22 16:00:40 +09:00
daimin
181efebdfe
HDFS-16546. Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection (#4187) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-04-22 15:34:28 +09:00
S O'Donnell
bacd0b9a92 Revert "HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell." 
This reverts commit 045d48530cbbd7baa2b67937e103b9400487f1a4.
 2022-04-20 20:57:23 +01:00
qinyuren
4fed01ef21 HDFS-16544. EC decoding failed due to invalid buffer (#4179) 
(cherry picked from commit 76bbd173749f2af4f17946fb37c4c72e2de26764)
 2022-04-20 15:08:05 +09:00
Stephen O'Donnell
045d48530c HDFS-16531. Avoid setReplication writing an edit record if old replication equals the new value (#4148). Contributed by Stephen O'Donnell. 
(cherry picked from commit dbeeee03639f41a022dd07d5fc04e3aa65a94b5f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
 2022-04-19 11:38:34 +01:00
qinyuren
d993d22038 HDFS-16538. EC decoding failed due to not enough valid inputs (#4167) 
Co-authored-by: liubingxing <liubingxing@bigo.sg>
(cherry picked from commit 52e152f8b0d5f522f3b799ea72c6c887d5d2c42d)
 2022-04-19 13:40:16 +09:00
jianghuazhu
ed83a278c2
HDFS-16389.Improve NNThroughputBenchmark test mkdirs. (#3819) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900682e7120e7af250ebf0b4bafb40b412d3e11c)
(cherry picked from commit 4aa9698ee1e9a10a0f882229036bbb4db1ede939)
 2022-04-19 11:13:29 +08:00
daimin
248c548616
Fix decommission UnsupportedOperationException (#4172) 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-04-18 10:40:12 +08:00
Takanobu Asanuma
b8c6ba6046 HDFS-16479. EC: NameNode should not send a reconstruction work when the source datanodes are insufficient (#4138) 
(cherry picked from commit 2efab92959ca0a68c52ed6a9c721704e57afbcc7)
 2022-04-14 11:52:12 +09:00
qinyuren
cb14e8de81 HDFS-16484. [SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread (#4032) 
(cherry picked from commit 45394433a112334e48087bd60674538af739922a)
 2022-04-13 11:53:58 +09:00
Stephen O'Donnell
72e48ac84e HDFS-16530. setReplication debug log creates a new string even if debug is disabled (#4142) 
(cherry picked from commit bbfe3500cf76ce661e0b10bb5d16cb419f9b9b10)
 2022-04-06 13:06:28 +01:00
wangzhaohui
545d3a6cb7 HDFS-16529. Remove unnecessary setObserverRead in TestConsistentReadsObserver (#4131) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 61bbdfd3a73a18cf05aeb2c40429c34c4b9b0732)
 2022-04-06 17:31:42 +09:00
litao
6b24b85e3f HDFS-16507. [SBN read] Avoid purging edit log which is in progress (#4082) 2022-03-30 23:04:04 -07:00
Masatake Iwasaki
228ad1ac25 Make upstream aware of 3.2.3 release. 
(cherry picked from commit 0fbd96a2449ec49f840d93e1c7d290c5218ef4ea)
 2022-03-28 08:09:58 +00:00
GuoPhilipse
ff03212145 HDFS-16355. Improve the description of dfs.block.scanner.volume.bytes.per.second (#3724) 
Co-authored-by: gf13871 <gf13871@ly.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 046a6204b4a895b98ccd41dde1c9524a6bb0ea31)
 2022-03-27 22:33:17 +09:00
qinyuren
66be0a713f HDFS-16501. Print the exception when reporting a bad block (#4062) 
Reviewed-by: tomscut <litao@bigo.sg>
(cherry picked from commit 45ce1cce50c3ff65676d946e96bbc7846ad3131a)
 2022-03-23 15:06:20 +09:00
Masatake Iwasaki
e13408d9d2
HADOOP-18088. Replace log4j 1.x with reload4j. (#4084) 
Co-authored-by: Wei-Chiu Chuang <weichiu@apache.org>
 2022-03-23 12:57:27 +09:00
Surendra Singh Lilhore
58fe5b2f35 HDFS-14630. Configuration.getTimeDurationHelper() should not log time unit warning in info log. Contributed by hemanthboyina. 
(cherry picked from commit 5ff76cb8bc69d68ba7c9487d00b1dc753d616bb2)
 2022-03-18 18:39:20 +09:00
Thinker313
6f815a546a
HDFS-16428. Source path with storagePolicy cause wrong typeConsumed while rename (#3898). Contributed by lei w. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 43153e80cbfc924fde380e6606c04455341edeab)
 2022-03-12 20:10:44 +08:00
Ayush Saxena
551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409bc1d62f23efef485d1c6f1ff21e86c)
(cherry picked from commit fa30224e95d920adaea0ec65c360359044e3f754)
 2022-02-23 11:41:35 +08:00
Chentao Yu
5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb28539f8c79bbb24f703301489825a6)
 2022-02-15 16:07:46 -08:00
singer-bin
0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b616040b4f98956aa946cc51e99f7d596c2)
(cherry picked from commit ce7cabb771363ce460ac1c174c6380710e733ce8)
 2022-02-06 13:37:16 +08:00
daimin
337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994eec9c835906d28b0ae755dfd6698dcd)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
KevinWikant
3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de2135988c326328e544f04b5129c53517)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant
543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree
b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28fec1d0de7e5a80dc3399f0ad1fd3fce)
 2022-01-28 17:14:42 +09:00
litao
fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a351fa1a5dd668c318b20437e6aaa35b)
 2022-01-27 13:45:47 +09:00
daimin
0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1ed49e06cc8973412952e09ed08bb9c0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c9d224727580f9ed4f754af5ab9f8f1)
 2022-01-25 10:01:09 +08:00
liubingxing
0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52ada03e099fa7dbb3efe62345ca27588)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma
53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229968b715e5c117bf0b35474ee0f05ec.
 2022-01-20 22:07:41 +09:00
qinyuren
96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699bd1360c109c18df0261897afea05ba)
 2022-01-19 13:44:27 +09:00
Jackson Wang
74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d22dbba8facb94e80333cea5ffeed5ba)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta
3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a869ca5c2bdc457375461ae89cdf97315)
 2022-01-11 02:14:52 +09:00
liever18
1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5851d3cdc3b125daebe5cc9305ad6f0)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta
278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2d79c3691bee29cb3fc4123062e5f7a)
 2022-01-04 13:26:09 +09:00
jianghuazhu
20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2021-12-24 17:28:00 +09:00
wangzhaohui
c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629c2db33f03c80e93c7e54fceedbfbe6)
 2021-12-21 12:02:25 +00:00
jianghuazhu
b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2021-12-20 12:08:30 +00:00
litao
3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2021-12-16 13:52:01 +09:00
Ayush Saxena
a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00