Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
24,779 Commits 326 Branches 371 Tags
Commit Graph

9161 Commits

Author SHA1 Message Date
Ayush Saxena
51b3a5b22c
HDFS-11041. Unable to unregister FsDatasetState MBean if DataNode is shutdown twice. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit e8cb2ae409bc1d62f23efef485d1c6f1ff21e86c)

Change-Id: I9f04082d650628bc1b8b62dacaaf472f8a578742
 2022-04-14 17:19:17 +01:00
Chentao Yu
38d448e40b
HADOOP-18109. Ensure that default permissions of directories under internal ViewFS directories are the same as directories on target filesystems. Contributed by Chentao Yu. (3953) 
(cherry picked from commit 19d90e62fb28539f8c79bbb24f703301489825a6)
 2022-04-14 17:19:16 +01:00
singer-bin
980fab9168
HDFS-16437 ReverseXML processor doesn't accept XML files without the … (#3926) 
(cherry picked from commit 125e3b616040b4f98956aa946cc51e99f7d596c2)

Change-Id: I03e4f2af17f0e4a8245c9c2c8ea1cb2cb41f777a
 2022-04-14 17:19:16 +01:00
Steve Loughran
a1c0673526
HADOOP-18198. Preparing for 3.3.3 release 
Change-Id: Idebf79191dc91dad52073f2c63ee9ab3a99464d9
 2022-04-14 17:19:16 +01:00
Chao Sun
f91452b289 Make upstream aware of 3.3.2 release 2022-03-02 19:09:35 -08:00
Xiangyi Zhu
6da346a358 HDFS-16043. Add markedDeleteBlockScrubberThread to delete blocks asynchronously (#3882). Contributed by Xiangyi Zhu. 
Signed-off-by: He Xiaoqiao <hexiaoqiao@apache.org>
 2022-01-15 09:26:05 -08:00
Jackson Wang
d7cacea07b HDFS-16420. Avoid deleting unique data blocks when deleting redundancy striped blocks. (#3880) 
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-14 09:07:32 -08:00
Ashutosh Gupta
6ea9906a8b HDFS-16410. Insecure Xml parsing in OfflineEditsXmlLoader (#3854) 
Contributed by Ashutosh Gupta
 2022-01-05 10:01:38 -08:00
liever18
d167ef4850 HDFS-16408. Ensure LeaseRecheckIntervalMs is greater than zero (#3856) 
(cherry picked from commit e1d0aa9ee5851d3cdc3b125daebe5cc9305ad6f0)
 2022-01-05 10:01:20 -08:00
Wei-Chiu Chuang
e42e483d00 Make upstream aware of 3.3.1 release 2022-01-04 14:51:49 -08:00
Ashutosh Gupta
b9ed31f9ec HDFS-16409. Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue (#3835) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 9eea0e28f2d79c3691bee29cb3fc4123062e5f7a)
 2022-01-04 14:38:52 -08:00
jianghuazhu
2ddc66fc0c HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2022-01-04 14:38:19 -08:00
wangzhaohui
bceab2e439 HDFS-16391. Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService (#3820) 
(cherry picked from commit 8d251bd629c2db33f03c80e93c7e54fceedbfbe6)
 2022-01-04 14:37:55 -08:00
jianghuazhu
82cf0aa296 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2022-01-04 14:37:52 -08:00
litao
c0ae38c6b4 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2022-01-04 14:37:40 -08:00
litao
24917bae9f HDFS-16375. The FBR lease ID should be exposed to the log (#3769) 
(cherry picked from commit 10cdde59b4666e7f70e0788409ca7b214539cedb)
 2022-01-04 14:37:37 -08:00
Ayush Saxena
57c9a8509f HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2022-01-04 14:37:26 -08:00
litao
f16d9dfc31 HDFS-16327. Make DFS_NAMENODE_MAX_SLOWPEER_COLLECT_NODES_KEY reconfigurable (#3716) 
(cherry picked from commit c56a07f36b561c9aacb5e74bd20195a817d5fd8d)
 2022-01-04 14:37:22 -08:00
GuoPhilipse
45e6d22462 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7430692d1f40f59e9cb24690cbda216)
 2022-01-04 14:37:15 -08:00
Akira Ajisaka
c257abca21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml
 2022-01-04 14:37:08 -08:00
qinyuren
04f0025ad2 HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2022-01-04 14:37:04 -08:00
Takanobu Asanuma
1e3f94fa3c Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 55c0e676fd533f0319e31357f037939339a6c156.
 2022-01-04 14:36:57 -08:00
qinyuren
5214140123 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043c063f548b81ad431d9452f010b18f7)
 2022-01-04 14:36:54 -08:00
lzx404243
47436346b4 HDFS-15862. Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent (#2724) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 7ba91c123544e1c1cbee759a11879af3b19260b3)
 2022-01-04 14:36:50 -08:00
LeonGao
d02bf2dcce HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661d694fb0775f4f84333c1317f4e6899)
 2022-01-04 14:36:42 -08:00
Takanobu Asanuma
906eb831c6 HDFS-16293. Client sleeps and holds 'dataQueue' when DataNodes are congested. Contributed by Yuanxin Zhu. 
(cherry picked from commit e8e69de106c03d041a0d280ea727e3f252460163)
 2022-01-04 14:36:36 -08:00
huhaiyang
e50c9224fe HDFS-16314. Support to make dfs.namenode.block-placement-policy.exclude-slow-nodes.enabled reconfigurable (#3664) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c0f405a46b402b4bdf5b7771f5124790fc866965)
 2022-01-04 14:36:32 -08:00
huhaiyang
069426f5eb HDFS-16287. Support to make dfs.namenode.avoid.read.slow.datanode reconfigurable (#3596) 
(cherry picked from commit db89a9411ebee11372314e82d7ea0606c348d014)
 2022-01-04 14:36:28 -08:00
litao
ce0bc7b473 HDFS-16344. Improve DirectoryScanner.Stats#toString (#3695) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 829af89dc4240347675e20df0ce047245f465c7b)
 2022-01-04 14:35:59 -08:00
litao
30f0629d6e HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82208b6b04551e637414d0d9d64a4173)
 2022-01-04 14:35:56 -08:00
Akira Ajisaka
5c18979782 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686)" 
This reverts commit 3b79e0f794657a77550318c83d3d7303d745740b.
 2022-01-04 14:35:49 -08:00
Viraj Jasani
544dffd179 HDFS-16171. De-flake testDecommissionStatus (#3280) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 6342d5e523941622a140fd877f06e9b59f48c48b)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestDecommissioningStatus.java
 2022-01-04 14:35:43 -08:00
Viraj Jasani
c6914b1cb6 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b619159dc4bf185ad53b950bdf5de52a)
 2022-01-04 14:35:40 -08:00
Ayush Saxena
328d3b84df HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686) 2022-01-04 14:35:36 -08:00
Viraj Jasani
3ae8d4ccb9 HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b38129bcef687506742b3d7d35e8e4947a9)
 2022-01-04 14:35:33 -08:00
Mukul Kumar Singh
7729505f8a HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1f708b7c3a0701886fa63068327e9b3)
 2022-01-04 14:34:39 -08:00
litao
8ae7691216 HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babbcaa0560c11e2065cff3fff0bf328c5)
 2022-01-04 14:34:15 -08:00
litao
486a57fb77 HDFS-16335. Fix HDFSCommands.md (#3681) 
(cherry picked from commit 20b78c8f978dc74f03a0dd5551904f8ec40ccf82)
 2022-01-04 14:30:52 -08:00
GuoPhilipse
abc48be5b4 HDFS-16334. Correct NameNode ACL description (#3678) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 3d1ca6f6dc1799e95feced466411743e9f9fddd4)
 2022-01-04 14:30:18 -08:00
litao
c236d34b3b HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e90f7ca7cf6d1418893340338eb8760f)
 2022-01-04 14:29:56 -08:00
GuoPhilipse
1e7a46b14d HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b1e0c8bdc3f4170295771d75c5ed776)
 2022-01-04 14:29:19 -08:00
Viraj Jasani
bf37c64d6f HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e118711852c58e9f0ebc3493fcb5ebeb)
 2022-01-04 14:28:54 -08:00
litao
e11dd859ff HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b4ab2c9c26a1efe7479c31fd1de80d5)
 2022-01-04 14:28:28 -08:00
Viraj Jasani
1e6e8a19ab HDFS-16323. DatanodeHttpServer doesn't require handler state map while retrieving filter handlers (#3659) 
(cherry picked from commit 646c470e5de956e570a81dcd4890f028965be33a)
 2022-01-04 14:27:52 -08:00
litao
3cbfef9130 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 2022-01-04 14:26:17 -08:00
bitterfox
ef462b21bf HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2022-01-04 14:20:25 -08:00
Chao Sun
ce74635dd4 Preparing for 3.3.1 release 2021-11-16 16:07:17 -08:00
Neil
c182bf65e0
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36b9794f4b1af8b9a7d8d0055914a094)
 2021-11-16 22:59:25 +08:00
Ahmed Hussein
25bcc7e3ee HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 2021-11-16 11:51:35 +09:00
GuoPhilipse
b11e017777
HDFS-16321. Fix invalid config in TestAvailableSpaceRackFaultTolerantBPP (#3655). Contributed by guo. 
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-11-14 20:00:28 +05:30