Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,786 Commits 326 Branches 371 Tags
Commit Graph

8051 Commits

Author SHA1 Message Date
jianghuazhu
20f602481a HDFS-16395. Remove useless NNThroughputBenchmark#dummyActionNoSynch(). (#3823) 
Reviewed-by: Fei Hui <feihui.ustc@gmail.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit a5659fe0f784f113489aee375ee1c2da8332d629)
 2021-12-24 17:28:00 +09:00
jianghuazhu
b68084ab43 HDFS-16386. Reduce DataNode load when FsDatasetAsyncDiskService is working. (#3806) 
(cherry picked from commit 746b32855496c054854c3e02d9993d97f10073d2)
 2021-12-20 12:08:30 +00:00
litao
3257646975 HDFS-16377. Should CheckNotNull before access FsDatasetSpi (#3784) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 22f5e1885d21c9db9ceecf611128508542ec0f11)
 2021-12-16 13:52:01 +09:00
Ayush Saxena
a57e83a47b HDFS-16373. Fix MiniDFSCluster restart in case of multiple namenodes. (#3756) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-12-14 17:24:05 +09:00
GuoPhilipse
038759e444 HDFS-16345. Fix test case fail in TestBlockStoragePolicy (#3696). Contributed by guophilipse. 
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 25849ff7b7430692d1f40f59e9cb24690cbda216)
 2021-12-10 16:36:40 +09:00
Akira Ajisaka
94ca965e21 HADOOP-18040. Use maven.test.failure.ignore instead of ignoreTestFailure (#3774) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 9b9e2ef87ffe7b33dc01699892acf62965fb3550)

 Conflicts:
	hadoop-tools/hadoop-federation-balance/pom.xml

(cherry picked from commit 35c5c6bb83beb4f68e1ff18907589b062a0467e8)

 Conflicts:
	hadoop-common-project/hadoop-registry/pom.xml
 2021-12-10 01:41:44 +09:00
Akira Ajisaka
3854b59bd1
HADOOP-18035. Ignore unit test failures to run all the unit tests from root (#3765) 
Reviewed-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-12-10 01:27:27 +09:00
qinyuren
72ffbd956a HDFS-16333. fix balancer bug when transfer an EC block (#3777) 2021-12-10 00:42:07 +09:00
Takanobu Asanuma
2315849b8f Revert "HDFS-16333. fix balancer bug when transfer an EC block (#3679)" 
This reverts commit 2072a6a47654c140d4bc9e386a0bc333114d1c2c.
 2021-12-09 16:37:43 +09:00
qinyuren
2072a6a476 HDFS-16333. fix balancer bug when transfer an EC block (#3679) 
(cherry picked from commit 35556ea043c063f548b81ad431d9452f010b18f7)
 2021-12-09 13:27:33 +09:00
LeonGao
b4fc326acf HDFS-16268. Balancer stuck when moving striped blocks due to NPE (#3546) 
(cherry picked from commit 7279fe8661d694fb0775f4f84333c1317f4e6899)
 2021-12-06 16:19:05 +09:00
bitterfox
7bdb50c793 HDFS-16332. Handle invalid token exception in sasl handshake (#3677) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dd6b987c93e8319560b633360f30ac84fc48e403)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/datatransfer/sasl/SaslDataTransferServer.java
 2021-12-03 23:43:37 +09:00
Ananya Singh
b8dbc24727
HDFS-16317 : Backport HDFS-14729 for branch-3.2 (#3692) 
Co-authored-by: Sunil G <sunilg@apache.org>,Ananya Singh <purpul90@gmail.com>
 2021-11-30 18:01:53 +05:30
litao
86084637db HDFS-16339. Show the threshold when mover threads quota is exceeded (#3689) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 86705eda82208b6b04551e637414d0d9d64a4173)
 2021-11-26 19:54:25 +09:00
Akira Ajisaka
6a28496fb0 Revert "HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.." 
This reverts commit 4231067d0b1932f891410f7b461b986848b855d3.
 2021-11-25 22:18:32 +09:00
Viraj Jasani
03c908fb04
HDFS-16171. De-flake testDecommissionStatus (#3280) (#3720) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
 2021-11-25 22:18:01 +09:00
Viraj Jasani
2ff303ea00 HDFS-16350. Datanode start time should be set after RPC server starts successfully (#3711) 
(cherry picked from commit cdc13e91b619159dc4bf185ad53b950bdf5de52a)
 2021-11-25 10:26:57 +08:00
Ayush Saxena
4231067d0b
HDFS-16336. Addendum: De-flake TestRollingUpgrade#testRollback (#3686). Contributed by Viraj Jasani.. 2021-11-25 02:01:36 +05:30
Viraj Jasani
af452d2fc5
HDFS-16336. De-flake TestRollingUpgrade#testRollback (#3686) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit e8566b38129bcef687506742b3d7d35e8e4947a9)
 2021-11-24 22:56:44 +09:00
Mukul Kumar Singh
f5ff4e1ca8 HDFS-16343. Add some debug logs when the dfsUsed are not used during Datanode startup. (#3694) 
(cherry picked from commit faa4eeacb1f708b7c3a0701886fa63068327e9b3)
 2021-11-23 13:53:22 +00:00
litao
2b0339cd34
HDFS-16337. Show start time of Datanode on Web (#3682) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit 0ed817babbcaa0560c11e2065cff3fff0bf328c5)
(cherry picked from commit d746e2cdd80335c86526b293f284187ce07758c4)
 2021-11-23 15:43:36 +08:00
litao
abf8572a16 HDFS-16326. Simplify the code for DiskBalancer (#3663) 
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 965e3874e90f7ca7cf6d1418893340338eb8760f)
 2021-11-19 11:39:48 +09:00
GuoPhilipse
583a41090c
HDFS-16328. Correct disk balancer desc (#3669) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 0c1eb0125b1e0c8bdc3f4170295771d75c5ed776)
 2021-11-19 01:02:49 +09:00
Viraj Jasani
12960d7fd4
HDFS-16330. Fix incorrect placeholder for Exception logs in DiskBalancer (#3672) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 38614d05e118711852c58e9f0ebc3493fcb5ebeb)
 2021-11-19 00:56:38 +09:00
litao
381e41b97c
HDFS-16329. Fix log format for BlockManager (#3670) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dc751df63b4ab2c9c26a1efe7479c31fd1de80d5)
 2021-11-19 00:43:42 +09:00
Takanobu Asanuma
210464b08f
HDFS-15963. Unreleased volume references cause an infinite loop. Contributed by Shuyan Zhang. (#2941, #3668) 
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
(cherry picked from commit 9f2db2c9fdcf1e61b39651a51f8223cfb8ce2e31)

Co-authored-by: zhangshuyan0 <81411509+zhangshuyan0@users.noreply.github.com>
 2021-11-18 16:32:54 +09:00
litao
81f6cbc715 HDFS-16315. Add metrics related to Transfer and NativeCopy for DataNode (#3666) 
(cherry picked from commit 026d5860cbf3acd701437bfd7c8084aba3d43881)
 2021-11-17 13:53:33 +09:00
Neil
78c56f39b4
HDFS-16182.numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage. (#3320) 
(cherry picked from commit 5626734a36b9794f4b1af8b9a7d8d0055914a094)
(cherry picked from commit ce3a7155335416ed4665f90f0ddd7ac3a3edd7e6)
 2021-11-16 22:59:35 +08:00
Wei-Chiu Chuang
002b5fbdc0
HDFS-16187. SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing (#3340) (#3640) 
(cherry picked from commit 356ebbbe80aef991d564a6140e341ddd76176416)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZones.java

Change-Id: If76fd0d77fafc90fe2f2c19ab1d0c43a58510f6b

Co-authored-by: bshashikant <shashikant@apache.org>
(cherry picked from commit 5333e872e2a07de8ff2431c9ff041bb582e8e173)
 2021-11-12 15:03:15 +08:00
litao
c95df2eb6f HDFS-16311. Metric metadataOperationRate calculation error in DataNodeVolumeMetrics (#3636) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit 1c1cf64616f34d039cf9246da9613914aa870515)
 2021-11-12 10:26:19 +09:00
litao
ee871a6afd HDFS-16312. Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents (#3637) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 2a1a11c039518bab7e5b581ef23bc5a2e8c81544)
 2021-11-10 19:24:32 +09:00
jianghuazhu
9ddafb2eee HDFS-16301. Improve BenchmarkThroughput#SIZE naming standardization. (#3622) 
(cherry picked from commit b307f18a584b207782124bee0a377b2dbfc13df0)
 2021-11-10 11:40:28 +09:00
litao
b103be470c HDFS-16299. Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics (#3616) 
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit cbc7fb6bcad8c959830a84521ea642cbf973ea83)
 2021-11-09 17:10:01 +09:00
jianghuazhu
06c4058a61 HDFS-16294.Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED. (#3605) 
(cherry picked from commit 72718a8dc1d6d16249c1073c31f1a078d4e118f6)
 2021-11-04 16:51:14 +09:00
daimin
29fd36e2f1 HDFS-16286. Add a debug tool to verify the correctness of erasure coding on file (#3593) 
(cherry picked from commit a21895a5b3644944fe04cf558d593b96da0263fd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DebugAdmin.java
 2021-11-03 21:30:48 +00:00
litao
b1801e99de HDFS-16279. Print detail datanode info when process first storage report (#3564) 
(cherry picked from commit 6aa664c62443b4061c26c27376e5d1e0e591a323)
 2021-10-29 10:29:32 +09:00
Michael Kuchenbecker
7de539f3ca HDFS-7612: Fix default cache directory in TestOfflineEditsViewer.testStored. Contributed by Michael Kuchenbecker (#3571) 
(cherry picked from commit a73ff6915ae3e0ced1b4c814a94845f51e655a0c)
 2021-10-21 13:50:54 -07:00
daimin
9f2c89e690 HDFS-16272. Fix int overflow in computing safe length during EC block recovery (#3548) 
(cherry picked from commit 5337bebcc53a0144311877362787ed9ece0cb118)
 2021-10-18 12:20:43 +01:00
langlaile1221
88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c88bcec5acca30aadca36b7d65c4237c)
(cherry picked from commit a35a6ec06bc9a95e1718938da9175a9a3511f4d1)
 2021-10-11 16:22:59 +08:00
Inigo Goiri
86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang
978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
 2021-09-24 09:15:04 -07:00
He Xiaoqiao
4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
 2021-09-17 22:16:11 +08:00
Ayush Saxena
f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
 2021-09-17 22:16:07 +08:00
wangzhaohui
928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
 2021-09-15 10:06:27 -07:00
EungsopYoo
4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
(cherry picked from commit 51a4a23e373abe9b2f6469ffcd5bc8294307f025)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein
a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00