Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,754 Commits 326 Branches 371 Tags
Commit Graph

8514 Commits

Author SHA1 Message Date
Viraj Jasani
2ff303ea00 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b619159dc4bf185ad53b950bdf5de52a)
 2021-11-25 10:26:57 +08:00
Ayush Saxena
4231067d0b
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.. 2021-11-25 02:01:36 +05:30
Viraj Jasani
af452d2fc5
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b38129bcef687506742b3d7d35e8e4947a9)
 2021-11-24 22:56:44 +09:00
Mukul Kumar Singh
f5ff4e1ca8 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1f708b7c3a0701886fa63068327e9b3)
 2021-11-23 13:53:22 +00:00
litao
2b0339cd34
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babbcaa0560c11e2065cff3fff0bf328c5)
(cherry picked from commit d746e2cdd80335c86526b293f284187ce07758c4)
 2021-11-23 15:43:36 +08:00
litao
abf8572a16 HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e90f7ca7cf6d1418893340338eb8760f)
 2021-11-19 11:39:48 +09:00
GuoPhilipse
583a41090c
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b1e0c8bdc3f4170295771d75c5ed776)
 2021-11-19 01:02:49 +09:00
Viraj Jasani
12960d7fd4
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e118711852c58e9f0ebc3493fcb5ebeb)
 2021-11-19 00:56:38 +09:00
litao
381e41b97c
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b4ab2c9c26a1efe7479c31fd1de80d5)
 2021-11-19 00:43:42 +09:00
Takanobu Asanuma
210464b08f
HDFS-15963. Unreleased volume references cause an infinite loop. Contributed by Shuyan Zhang. (#2941, #3668) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fdcf1e61b39651a51f8223cfb8ce2e31)

Co-authored-by: zhangshuyan0 <81411509+zhangshuyan0@users.noreply.github.com>
 2021-11-18 16:32:54 +09:00
litao
81f6cbc715 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 
(cherry picked from commit 026d5860cbf3acd701437bfd7c8084aba3d43881)
 2021-11-17 13:53:33 +09:00
Neil
78c56f39b4
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36b9794f4b1af8b9a7d8d0055914a094)
(cherry picked from commit ce3a7155335416ed4665f90f0ddd7ac3a3edd7e6)
 2021-11-16 22:59:35 +08:00
Wei-Chiu Chuang
002b5fbdc0
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80aef991d564a6140e341ddd76176416)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
(cherry picked from commit 5333e872e2a07de8ff2431c9ff041bb582e8e173)
 2021-11-12 15:03:15 +08:00
litao
c95df2eb6f HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616f34d039cf9246da9613914aa870515)
 2021-11-12 10:26:19 +09:00
litao
ee871a6afd HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039518bab7e5b581ef23bc5a2e8c81544)
 2021-11-10 19:24:32 +09:00
litao
ba61dd9b73 HDFS-16298. Improve error msg for BlockMissingException (#3615) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit ea65fc26d80db478f7e49749065c69da7e241bf0)
 2021-11-10 19:18:56 +09:00
jianghuazhu
9ddafb2eee HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a584b207782124bee0a377b2dbfc13df0)
 2021-11-10 11:40:28 +09:00
litao
b103be470c HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bcad8c959830a84521ea642cbf973ea83)
 2021-11-09 17:10:01 +09:00
jianghuazhu
06c4058a61 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1d6d16249c1073c31f1a078d4e118f6)
 2021-11-04 16:51:14 +09:00
daimin
29fd36e2f1 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3644944fe04cf558d593b96da0263fd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
 2021-11-03 21:30:48 +00:00
litao
b1801e99de HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c62443b4061c26c27376e5d1e0e591a323)
 2021-10-29 10:29:32 +09:00
Michael Kuchenbecker
7de539f3ca HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915ae3e0ced1b4c814a94845f51e655a0c)
 2021-10-21 13:50:54 -07:00
daimin
9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:20:43 +01:00
langlaile1221
88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c88bcec5acca30aadca36b7d65c4237c)
(cherry picked from commit a35a6ec06bc9a95e1718938da9175a9a3511f4d1)
 2021-10-11 16:22:59 +08:00
AngersZhuuuu
1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao
cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
 2021-09-29 17:44:00 +08:00
Inigo Goiri
86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang
978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
 2021-09-24 09:15:04 -07:00
He Xiaoqiao
4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
 2021-09-17 22:16:11 +08:00
Ayush Saxena
f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
 2021-09-17 22:16:07 +08:00
wangzhaohui
928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
 2021-09-15 10:06:27 -07:00
EungsopYoo
4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
(cherry picked from commit 51a4a23e373abe9b2f6469ffcd5bc8294307f025)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein
a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00
LeonGao
3c03ade285
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)
 2021-09-14 12:31:44 +08:00
Ahmed Hussein
33c60ba060
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)
 2021-09-14 12:31:34 +08:00
Stephen O'Donnell
ad55d5d1d2
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)
 2021-09-14 12:31:20 +08:00
Stephen O'Donnell
7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao
ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69bc578db1ce95198aa7fa7bc7199320.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein
318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Ahmed Hussein
1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
litao
ce100b5807
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e2bf4c87f12b53e508edf8fee05856a)
(cherry picked from commit 0067c06606e8e3176148fa1856aeb3ae0930951c)
 2021-08-19 12:31:44 +08:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Akira Ajisaka
78c5fdd90d
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:28:22 +09:00
wangzhaohui
b98a8d259b
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c4d1368e1195132bb32a815cb12934c)
(cherry picked from commit 7068e18842117f33b5caceee244e8b27618c121a)
 2021-08-06 18:25:28 +08:00
wangzhaohui
6b10154123 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b733177cf9b8b10088527e34099d73f)
 2021-08-06 09:52:51 +01:00
Giovanni Matteo Fumarola
2f3c0b8dd1
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 
(cherry picked from commit 1a08302897bac18baf0f56b7d41ad54f63ae191b)
 2021-08-04 15:54:08 +09:00