Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,820 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8548 Commits

Author SHA1 Message Date
Ayush Saxena 551f8d3ddc
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409)
(cherry picked from commit fa30224e95)
 2022-02-23 11:41:35 +08:00
Chentao Yu 5066722eb7 HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb)
 2022-02-15 16:07:46 -08:00
singer-bin 0d0c256de3
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b6160)
(cherry picked from commit ce7cabb771)
 2022-02-06 13:37:16 +08:00
daimin 337e689711
HDFS-16403. Improve FUSE IO performance by supporting FUSE parameter max_background (#3842) 
Reviewed-by: Istvan Fajth <pifta@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d69938994e)
(cherry picked from commit f4241834cf4bf3cdc43a9c012a4d526b0299d88e)
 2022-02-06 13:37:14 +08:00
KevinWikant 3ae6a5b58e HDFS-16443. Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception (#3942) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 089e06de21)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeAdminDefaultMonitor.java
 2022-01-31 16:17:03 +09:00
KevinWikant 543ff58127
HDFS-16303. Improve handling of datanode lost while decommissioning (#3920) 
Co-authored-by: Kevin Wikant <wikak@amazon.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2022-01-31 16:12:27 +09:00
secfree b46ecb7330 HDFS-16169. Fix TestBlockTokenWithDFSStriped#testEnd2End failure (#3850) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 39cad5f28f)
 2022-01-28 17:14:42 +09:00
litao fd6459c117 HDFS-16427. Add debug log for BlockManager#chooseExcessRedundancyStriped (#3888) 
(cherry picked from commit 6136d630a3)
 2022-01-27 13:45:47 +09:00
daimin 0f0ebf9439
HDFS-16430. Add validation to maximum blocks in EC group when adding an EC policy (#3899). Contributed by daimin. 
Reviewed-by: tomscut <litao@bigo.sg>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 5ef335da1e)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ErasureCodingPolicyManager.java

(cherry picked from commit 728ed10a7c)
 2022-01-25 10:01:09 +08:00
liubingxing 0aff87466d
HDFS-16352. return the real datanode numBlocks in #getDatanodeStorageReport (#3714). Contributed by liubingxing. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit d8dea6f52a)
(cherry picked from commit 2d7cbfd04b8fb918e1889a5f6e891890b942a2b4)
 2022-01-24 15:51:28 +08:00
Takanobu Asanuma 53c4d89250 Revert "HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887)" 
This reverts commit 96dd426229.
 2022-01-20 22:07:41 +09:00
qinyuren 96dd426229 HDFS-16426. Fix nextBlockReportTime when trigger full block report force (#3887) 
(cherry picked from commit fcb1076699)
 2022-01-19 13:44:27 +09:00
Jackson Wang 74d3e83be0 HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit d8862822d2)
 2022-01-14 22:45:30 +09:00
Ashutosh Gupta 3992ba5d8c
HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta

(cherry picked from commit 43e5218a86)
 2022-01-11 02:14:52 +09:00
liever18 1e1e4a2f51 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5)
 2022-01-05 16:02:49 +00:00
Ashutosh Gupta 278fc00871 HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2)
 2022-01-04 13:26:09 +09:00
jianghuazhu 20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f7)
 2021-12-24 17:28:00 +09:00
wangzhaohui c66bae3fc3 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629)
 2021-12-21 12:02:25 +00:00
jianghuazhu b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 12:08:30 +00:00
litao 3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:52:01 +09:00
Ayush Saxena a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00
GuoPhilipse 038759e444 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2021-12-10 16:36:40 +09:00
Akira Ajisaka 94ca965e21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml
 2021-12-10 01:41:44 +09:00
Akira Ajisaka 3854b59bd1
HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:27:27 +09:00
qinyuren 72ffbd956a HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:42:07 +09:00
Takanobu Asanuma 2315849b8f Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 2072a6a476.
 2021-12-09 16:37:43 +09:00
qinyuren 2072a6a476 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2021-12-09 13:27:33 +09:00
lzx404243 6114816b03 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c1235)
 2021-12-08 20:49:27 +09:00
LeonGao b4fc326acf HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2021-12-06 16:19:05 +09:00
bitterfox 7bdb50c793 HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:43:37 +09:00
Ananya Singh b8dbc24727
HDFS-16317 : Backport HDFS-14729 for branch-3.2 (#3692) 
Co-authored-by: Sunil G <sunilg@apache.org>,Ananya Singh <purpul90@gmail.com>
 2021-11-30 18:01:53 +05:30
litao 86084637db HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2021-11-26 19:54:25 +09:00
Akira Ajisaka 6a28496fb0 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.." 
This reverts commit 4231067d0b.
 2021-11-25 22:18:32 +09:00
Viraj Jasani 03c908fb04
HDFS-16171. De-flake testDecommissionStatus (#3280) (#3720) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 22:18:01 +09:00
Viraj Jasani 2ff303ea00 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:26:57 +08:00
Ayush Saxena 4231067d0b
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.. 2021-11-25 02:01:36 +05:30
Viraj Jasani af452d2fc5
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:44 +09:00
Mukul Kumar Singh f5ff4e1ca8 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:53:22 +00:00
litao 2b0339cd34
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
(cherry picked from commit d746e2cdd8)
 2021-11-23 15:43:36 +08:00
litao abf8572a16 HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:48 +09:00
GuoPhilipse 583a41090c
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:49 +09:00
Viraj Jasani 12960d7fd4
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:38 +09:00
litao 381e41b97c
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:42 +09:00
Takanobu Asanuma 210464b08f
HDFS-15963. Unreleased volume references cause an infinite loop. Contributed by Shuyan Zhang. (#2941, #3668) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fd)

Co-authored-by: zhangshuyan0 <81411509+zhangshuyan0@users.noreply.github.com>
 2021-11-18 16:32:54 +09:00
litao 81f6cbc715 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 
(cherry picked from commit 026d5860cb)
 2021-11-17 13:53:33 +09:00
Neil 78c56f39b4
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
(cherry picked from commit ce3a7155335416ed4665f90f0ddd7ac3a3edd7e6)
 2021-11-16 22:59:35 +08:00
Wei-Chiu Chuang 002b5fbdc0
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
(cherry picked from commit 5333e872e2)
 2021-11-12 15:03:15 +08:00
litao c95df2eb6f HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:26:19 +09:00
litao ee871a6afd HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:24:32 +09:00
litao ba61dd9b73 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d8)
 2021-11-10 19:18:56 +09:00