Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,784 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

8050 Commits

Author SHA1 Message Date
jianghuazhu b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b328554)
 2021-12-20 12:08:30 +00:00
litao 3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d)
 2021-12-16 13:52:01 +09:00
Ayush Saxena a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00
GuoPhilipse 038759e444 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7)
 2021-12-10 16:36:40 +09:00
Akira Ajisaka 94ca965e21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87f)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml
 2021-12-10 01:41:44 +09:00
Akira Ajisaka 3854b59bd1
HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:27:27 +09:00
qinyuren 72ffbd956a HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:42:07 +09:00
Takanobu Asanuma 2315849b8f Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 2072a6a476.
 2021-12-09 16:37:43 +09:00
qinyuren 2072a6a476 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043)
 2021-12-09 13:27:33 +09:00
LeonGao b4fc326acf HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661)
 2021-12-06 16:19:05 +09:00
bitterfox 7bdb50c793 HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:43:37 +09:00
Ananya Singh b8dbc24727
HDFS-16317 : Backport HDFS-14729 for branch-3.2 (#3692) 
Co-authored-by: Sunil G <sunilg@apache.org>,Ananya Singh <purpul90@gmail.com>
 2021-11-30 18:01:53 +05:30
litao 86084637db HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82)
 2021-11-26 19:54:25 +09:00
Akira Ajisaka 6a28496fb0 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.." 
This reverts commit 4231067d0b.
 2021-11-25 22:18:32 +09:00
Viraj Jasani 03c908fb04
HDFS-16171. De-flake testDecommissionStatus (#3280) (#3720) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 22:18:01 +09:00
Viraj Jasani 2ff303ea00 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b6)
 2021-11-25 10:26:57 +08:00
Ayush Saxena 4231067d0b
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.. 2021-11-25 02:01:36 +05:30
Viraj Jasani af452d2fc5
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b3812)
 2021-11-24 22:56:44 +09:00
Mukul Kumar Singh f5ff4e1ca8 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1)
 2021-11-23 13:53:22 +00:00
litao 2b0339cd34
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babb)
(cherry picked from commit d746e2cdd8)
 2021-11-23 15:43:36 +08:00
litao abf8572a16 HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e9)
 2021-11-19 11:39:48 +09:00
GuoPhilipse 583a41090c
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b)
 2021-11-19 01:02:49 +09:00
Viraj Jasani 12960d7fd4
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e1)
 2021-11-19 00:56:38 +09:00
litao 381e41b97c
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b)
 2021-11-19 00:43:42 +09:00
Takanobu Asanuma 210464b08f
HDFS-15963. Unreleased volume references cause an infinite loop. Contributed by Shuyan Zhang. (#2941, #3668) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fd)

Co-authored-by: zhangshuyan0 <81411509+zhangshuyan0@users.noreply.github.com>
 2021-11-18 16:32:54 +09:00
litao 81f6cbc715 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 
(cherry picked from commit 026d5860cb)
 2021-11-17 13:53:33 +09:00
Neil 78c56f39b4
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36)
(cherry picked from commit ce3a7155335416ed4665f90f0ddd7ac3a3edd7e6)
 2021-11-16 22:59:35 +08:00
Wei-Chiu Chuang 002b5fbdc0
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
(cherry picked from commit 5333e872e2)
 2021-11-12 15:03:15 +08:00
litao c95df2eb6f HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616)
 2021-11-12 10:26:19 +09:00
litao ee871a6afd HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039)
 2021-11-10 19:24:32 +09:00
jianghuazhu 9ddafb2eee HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a58)
 2021-11-10 11:40:28 +09:00
litao b103be470c HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bca)
 2021-11-09 17:10:01 +09:00
jianghuazhu 06c4058a61 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1)
 2021-11-04 16:51:14 +09:00
daimin 29fd36e2f1 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
 2021-11-03 21:30:48 +00:00
litao b1801e99de HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c624)
 2021-10-29 10:29:32 +09:00
Michael Kuchenbecker 7de539f3ca HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915a)
 2021-10-21 13:50:54 -07:00
daimin 9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc5)
 2021-10-18 12:20:43 +01:00
langlaile1221 88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c8)
(cherry picked from commit a35a6ec06b)
 2021-10-11 16:22:59 +08:00
Inigo Goiri 86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dc)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang 978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf)
(cherry picked from commit bb08de559a)
 2021-09-24 09:15:04 -07:00
He Xiaoqiao 4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao 50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa1)
 2021-09-17 22:16:11 +08:00
Inigo Goiri 5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5)
 2021-09-17 22:16:11 +08:00
Ayush Saxena f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab)
 2021-09-17 22:16:11 +08:00
Inigo Goiri d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d)
 2021-09-17 22:16:07 +08:00
wangzhaohui 928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14)
 2021-09-15 10:06:27 -07:00
EungsopYoo 4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8b)
(cherry picked from commit 51a4a23e37)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00
LeonGao 3c03ade285
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc)
 2021-09-14 12:31:44 +08:00