Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,719 Commits 326 Branches 371 Tags
Commit Graph

8491 Commits

Author SHA1 Message Date
langlaile1221
88744b8a04
HDFS-16241. Standby close reconstruction thread (#3493) 
Co-authored-by: zhanghuazong <Zhz@20068867>
(cherry picked from commit 88d8c3d0c88bcec5acca30aadca36b7d65c4237c)
(cherry picked from commit a35a6ec06bc9a95e1718938da9175a9a3511f4d1)
 2021-10-11 16:22:59 +08:00
AngersZhuuuu
1df3b3de46
HDFS-16235. Fix Deadlock in LeaseRenewer for static remove method (#3472) 
(cherry picked from commit 5f9321a5d4112e7b4ee9e6b948ce46a2b152a2db)
 2021-09-29 17:44:32 +08:00
He Xiaoqiao
cfd0265975
HDFS-14575. LeaseRenewer#daemon threads leak in DFSClient. Contributed by Renukaprasad C. 
Co-authored-by: Tao Yang <taoyang1@apache.org>
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 10b79a26fe0677b266acf237e8458e93743424a6)
 2021-09-29 17:44:00 +08:00
Inigo Goiri
86028d975c
HDFS-15092. TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails. Contributed by Fei Hui. 
(cherry picked from commit 8cfc3673dcbf1901ca6fad11b5c996e54e32ed6b)
 2021-09-27 14:52:27 +09:00
Wei-Chiu Chuang
978ec57445 HDFS-16233. Do not use exception handler to implement copy-on-write for EnumCounters. (#3468) 
(cherry picked from commit 87632bbacf064e9b07564ba66ed1fbb1d42290d1)
(cherry picked from commit bb08de559a36fd56c7d4023c6dcac66da3fbacc5)
 2021-09-24 09:15:04 -07:00
He Xiaoqiao
4944dafa29
HDFS-15113. Addendum: Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit af64ce2f4a72705c5b68ddfe4d29f0d208fd38e7)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
50b0f0dc42
HDFS-15113. Missing IBR when NameNode restart if open processCommand async feature. Contributed by Xiaoqiao He. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Brahma Reddy Battula <brahma@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
(cherry picked from commit e9955bb8ff44396eb478f709307d647ca884de99)
 2021-09-17 22:16:12 +08:00
He Xiaoqiao
fc5fe67b3f
HDFS-15651. Client could not obtain block when DN CommandProcessingThread exit. Contributed by Aiphago. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Yiqun Lin <yqlin@apache.org>
(cherry picked from commit 3067a25fa12a012709f43aa35cc606db6fb137f9)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
5da46e197b
HDFS-15075. Remove process command timing from BPServiceActor. Contributed by Xiaoqiao He. 
(cherry picked from commit cdcb77a2c5ca99502d2ac2fbf803f22463eb1343)
 2021-09-17 22:16:11 +08:00
Ayush Saxena
f77965b29e
HDFS-14997. Addendum: BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit 80f91d14ab0fb385252d4eeb19141bd059303d59)
 2021-09-17 22:16:11 +08:00
Inigo Goiri
d5501c125f
HDFS-14997. BPServiceActor processes commands from NameNode asynchronously. Contributed by Xiaoqiao He. 
(cherry picked from commit b86895485d95440de099831e0db38db037f16bdd)
 2021-09-17 22:16:07 +08:00
wangzhaohui
928dbac0a6 HDFS-16181. [SBN Read] Fix display of JournalNode metric RpcRequestCacheMissAmount (#3317) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>

(cherry picked from commit 232fd7cae170de8c6b52c14841a47dca8735c6d2)
(cherry picked from commit 2f73ac1c14dde86a5c0b86f9272cf98084478994)
 2021-09-15 10:06:27 -07:00
EungsopYoo
4eeb4d32e5
HDFS-16198. Short circuit read leaks Slot objects when InvalidToken exception is thrown (#3359) 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit c4c5883d8bf1fdc330e1da4d93eba760fa70c0e8)
(cherry picked from commit 51a4a23e373abe9b2f6469ffcd5bc8294307f025)
 2021-09-14 16:49:14 +08:00
Ahmed Hussein
a3920e89de
HDFS-15160. amend to fix javac error supressing unchecked warning 2021-09-14 12:31:49 +08:00
LeonGao
3c03ade285
HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679) 
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)
 2021-09-14 12:31:44 +08:00
Ahmed Hussein
33c60ba060
HDFS-15457. TestFsDatasetImpl fails intermittently (#2407) 
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)
 2021-09-14 12:31:34 +08:00
Stephen O'Donnell
ad55d5d1d2
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)
 2021-09-14 12:31:20 +08:00
Stephen O'Donnell
7039433146
HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)
 2021-09-14 12:31:00 +08:00
He Xiaoqiao
ca182b6e5f
Revert " HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200)" 
This reverts commit 318bc5ff69bc578db1ce95198aa7fa7bc7199320.
 2021-09-14 12:27:42 +08:00
Ahmed Hussein
318bc5ff69
HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. (#3200) 
* HDFS-15150. Introduce read write lock to Datanode. Contributed Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit d7c136b9ed6d99e1b03f5b89723b3a20df359ba8)

* HDFS-15160. ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 2a67e2b1a0e3a5f91056f5b977ef9c4c07ba6718)

* HDFS-15457. TestFsDatasetImpl fails intermittently (#2407)
(cherry picked from commit 98097b8f19789605b9697f6a959da57261e0fe19)

* HDFS-15818. Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig. Contributed by Leon Gao (#2679)
(cherry picked from commit 9434c1eccc255a25ea5e11f6d8c9e1f83996d6b4)

* HDFS-15160. amend to fix javac error supressing unchecked warning
Co-authored-by: Stephen O'Donnell <sodonnel@apache.org>
Co-authored-by: LeonGao <liangg@uber.com>
 2021-09-11 19:52:16 +08:00
Ahmed Hussein
1944e0d714 HDFS-16207. Remove NN logs stack trace for non-existent xattr query (#3375) 
Change-Id: Ibde523b20a6b8ac92991da52583e625a018d2ee6
 2021-09-09 05:28:29 +00:00
litao
ce100b5807
HDFS-16177. Bug fix for Util#receiveFile (#3310) 
Reviewed-by: Hui Fei <ferhui@apache.org>
(cherry picked from commit 07627ef19e2bf4c87f12b53e508edf8fee05856a)
(cherry picked from commit 0067c06606e8e3176148fa1856aeb3ae0930951c)
 2021-08-19 12:31:44 +08:00
Brahma Reddy Battula
3cf2479c04 Preparing for 3.2.4 development. 2021-08-08 18:35:16 +05:30
Akira Ajisaka
78c5fdd90d
Fix potential heap buffer overflow in hdfs.c. Contributed by Igor Chervatyuk. 
(cherry picked from commit 4972e7a246f4aab665fd04ce72d1848bc5da9d4e)
 2021-08-07 11:28:22 +09:00
wangzhaohui
b98a8d259b
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c4d1368e1195132bb32a815cb12934c)
(cherry picked from commit 7068e18842117f33b5caceee244e8b27618c121a)
 2021-08-06 18:25:28 +08:00
wangzhaohui
6b10154123 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b733177cf9b8b10088527e34099d73f)
 2021-08-06 09:52:51 +01:00
Giovanni Matteo Fumarola
2f3c0b8dd1
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 
(cherry picked from commit 1a08302897bac18baf0f56b7d41ad54f63ae191b)
 2021-08-04 15:54:08 +09:00
Siyao Meng
fe59fe6744 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00b5589fe610ff994d81feba86039297)
(cherry picked from commit 72508e6430b9fe5145120ac23d3cf714c16e7579)
 2021-08-03 20:45:17 +01:00
He Xiaoqiao
f46a9e229e
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4863499207733a9f4c8c3cd2ec9c9d2)
 2021-07-29 11:26:09 +08:00
Akira Ajisaka
7fe4f3cb89 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60ceed733bfacb791fc5ed06116720dd0.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488c8da0227b95548c1fe40823b16a44d)
 2021-07-26 01:54:57 +09:00
He Xiaoqiao
913dbf2537
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df4199791a9093324e089b4a5cc880be8d6ff)
 2021-07-19 19:13:55 +08:00
Kihwal Lee
70c5e9e780 HDFS-16127. Improper pipeline close recovery causes a permanent write failure or data loss. Contributed by Kihwal Lee. 
(cherry picked from commit 47002719f2aa4ff58378d528d38b0f0962a45c25)
 2021-07-16 14:24:52 -05:00
crossfire
88f3ce1127
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810beb7abb8540c051114488e69a128ea3)
(cherry picked from commit bd0b8a0bcf96fe66a93f59b74822f419cac7eb09)
 2021-07-16 11:01:03 +08:00
Jim Brennan
4fdeac1228 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 20:59:09 +00:00
Abhishek Das
f9832031c2 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b573270dc960117c3b6c74bb78215caa)
 2021-07-13 18:25:03 -07:00
Artem Smotrakov
6c7f192f0f HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada573291266c4d6447862072fdf88af5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 14:25:08 +09:00
S O'Donnell
e2eacbd089 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76ca5c206639e2ddea4f3bef393dbb11)
 2021-07-09 10:34:14 +01:00
bshashikant
844b8865c0 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9ab884015290876798a734cbe5c0038b)
 2021-07-09 09:35:36 +01:00
Viraj Jasani
4912e4c903
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf521dc9c9660b7dd836fce04d05bab5)
 2021-07-05 13:43:10 +09:00
Viraj Jasani
707d003eb6
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073d76bef5c6ad3819e7710317733bd7a)
(cherry picked from commit 8538881710df3dd33ab23670d50d1b0119930169)
 2021-06-29 18:14:02 +08:00
Jim Brennan
9731dfbf0d HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 17:57:15 +00:00
Ahmed Hussein
dc46cec277 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
litao
00bd7b2aa4 HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
Hui Fei
c14f3e4ed1
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3114) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 21:04:31 +08:00
Wei-Chiu Chuang
11034f2488
HDFS-16074. Remove an expensive debug string concatenation (#3107) 
Reviewed-by: Ayush Saxena <ayushsaxena@apache.org>
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Reviewed-by: Inigo Goiri <inigoiri@apache.org>
Reviewed-by: Hui Fei <ferhui@apache.org>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: litao <tomleescut@gmail.com>
(cherry picked from commit 1ad674e5489e6080f21c308673ecc73a54ba3752)
 2021-06-17 11:38:58 +08:00
Ayush Saxena
77f61df352 HDFS-16073. Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB. Contributed by lei w. 2021-06-16 15:55:06 +05:30
Takanobu Asanuma
b8fc956a4d HDFS-16068. WebHdfsFileSystem has a possible connection leak in connection with HttpFS (#3104) 
(cherry picked from commit 839fcf7682270bc6ba151875549d64d81917f425)
 2021-06-15 17:23:34 +09:00
Konstantin V Shvachko
549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f999c0a80db53a2a96b5be5505d4d151)
 2021-05-27 18:55:35 -07:00
Konstantin V Shvachko
14fd4776ae HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f4f236674ce929164cc460037730abb)
 2021-05-26 15:10:52 -07:00
litao
6df2dc8463
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f98967ced0cbcf32d3acf6dfd10712e66a3)
(cherry picked from commit 2f56e2b54d0080cd22d3d8461567aa0364f983d8)
 2021-05-17 12:29:38 +08:00