Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,664 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7993 Commits

Author SHA1 Message Date
Brahma Reddy Battula 182167a9fb Preparing for 3.2.3 release. 2021-08-08 19:17:14 +05:30
wangzhaohui b98a8d259b
HDFS-16154. TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely (#3270) 
Co-authored-by: wangzhaohui8 <wangzhaohui8@jd.com>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit e85c44657c)
(cherry picked from commit 7068e18842)
 2021-08-06 18:25:28 +08:00
wangzhaohui 6b10154123 HDFS-16153. Avoid evaluation of LOG.debug statement in QuorumJournalManager (#3269). Contributed by wangzhaohui. 
(cherry picked from commit a73b64f86b)
 2021-08-06 09:52:51 +01:00
Giovanni Matteo Fumarola 2f3c0b8dd1
HDFS-14189. Fix intermittent failure of TestNameNodeMetrics. Contributed by Ayush Saxena. 
(cherry picked from commit 1a08302897)
 2021-08-04 15:54:08 +09:00
Siyao Meng fe59fe6744 HDFS-16055. Quota is not preserved in snapshot INode (#3078) 
(cherry picked from commit ebee2aed00)
(cherry picked from commit 72508e6430)
 2021-08-03 20:45:17 +01:00
He Xiaoqiao f46a9e229e
HDFS-15175. Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog. Contributed by Wan Chang. 
Reviewed-by: He Xiaoqiao <hexiaoqiao@apache.org>
Reviewed-by: Stephen O'Donnell <sodonnel@apache.org>
(cherry picked from commit 683feaa1d4)
 2021-07-29 11:26:09 +08:00
Akira Ajisaka 7fe4f3cb89 HDFS-12920. HDFS default value change (with adding time unit) breaks old version MR tarball work with new version (3.0) of hadoop. (#3227) 
Revert "HDFS-10845. Change defaults in hdfs-site.xml to match timeunit type. Contributed by Yiqun Lin"

This reverts commit b6d839a60c.

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 2f2f822488)
 2021-07-26 01:54:57 +09:00
He Xiaoqiao 913dbf2537
HDFS-15942. Increase Quota initialization threads. Contributed by Stephen O'Donnell. 
(cherry picked from commit 459df41997)
 2021-07-19 19:13:55 +08:00
crossfire 88f3ce1127
HDFS-15650. Make the socket timeout for computing checksum of striped blocks configurable (#2414) 
(cherry picked from commit 4bb25c810b)
(cherry picked from commit bd0b8a0bcf96fe66a93f59b74822f419cac7eb09)
 2021-07-16 11:01:03 +08:00
Jim Brennan 4fdeac1228 HDFS-16042. DatanodeAdminMonitor scan should be delay based. Contributed by Ahmed Hussein. 2021-07-15 20:59:09 +00:00
Abhishek Das f9832031c2 HADOOP-17028. ViewFS should initialize mounted target filesystems lazily. Contributed by Abhishek Das (#2260) 
(cherry picked from commit 1dd03cc4b5)
 2021-07-13 18:25:03 -07:00
Artem Smotrakov 6c7f192f0f HADOOP-17793. Better token validation (#3189) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba325a8ada)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenSecretManager.java
 2021-07-10 14:25:08 +09:00
S O'Donnell e2eacbd089 HDFS-15796. ConcurrentModificationException error happens on NameNode occasionally. Contributed by Daniel Ma 
(cherry picked from commit b6c06c4b76)
 2021-07-09 10:34:14 +01:00
bshashikant 844b8865c0 HDFS-16121. Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames. (#3188) 
(cherry picked from commit 6450c1ea9a)
 2021-07-09 09:35:36 +01:00
Viraj Jasani 4912e4c903
HDFS-16108. Fix incorrect log placeholders used in JournalNodeSyncer (#3169) 
(cherry picked from commit da575a53cf)
 2021-07-05 13:43:10 +09:00
Viraj Jasani 707d003eb6
HDFS-16092. Avoid creating LayoutFlags redundant objects (#3150) 
(cherry picked from commit 93a1685073)
(cherry picked from commit 8538881710)
 2021-06-29 18:14:02 +08:00
Jim Brennan 9731dfbf0d HDFS-15659. MiniDFSCluster dfs.namenode.redundancy.considerLoad default to false (#2443). Contributed by Ahmed Hussein. 2021-06-25 17:57:15 +00:00
Ahmed Hussein dc46cec277 HDFS-16061. DFTestUtil.waitReplication can produce false positives (#3095). Contributed by Ahmed Hussein. 
Reviewed-by: Jim Brennan <jbrennan@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
litao 00bd7b2aa4 HDFS-16078. Remove unused parameters for DatanodeManager.handleLifeline() (#3119). Contributed by tomscut. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-06-20 10:37:27 +05:30
Hui Fei c14f3e4ed1
HDFS-13671. Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet (#3114) 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/ReplicaMap.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockInfo.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java
 2021-06-18 21:04:31 +08:00
Konstantin V Shvachko 549987ed87 HDFS-16040. RpcQueueTime metric counts requeued calls as unique events. Contributed by Simbarashe Dzinamarira. 
(cherry picked from commit 8ce30f51f9)
 2021-05-27 18:55:35 -07:00
Konstantin V Shvachko 14fd4776ae HDFS-15915. Race condition with async edits logging due to updating txId outside of the namesystem log. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 1abd03d68f)
 2021-05-26 15:10:52 -07:00
litao 6df2dc8463
HDFS-15873. Add namenode address in logs for block report (#2743) 
(cherry picked from commit 6a922f9896)
(cherry picked from commit 2f56e2b54d)
 2021-05-17 12:29:38 +08:00
Hui Fei ab4e90cc28 HDFS-15652. Make block size from NNThroughputBenchmark configurable (#2416) 
(cherry picked from commit 8a6d5b9151)
 2021-04-29 12:48:06 -07:00
Ayush Saxena 5d1127c8f3 HDFS-15378. TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk. Contributed by hemanthboyina. 
(cherry picked from commit 8db38c98a6)
 2021-04-27 23:19:22 +09:00
Stephen O'Donnell 7a81e50bd2 HDFS-15621. Datanode DirectoryScanner uses excessive memory (#2849). Contributed by Stephen O'Donnell 
(cherry picked from commit 605ed85c29)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit f6efb58b07)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDirectoryScanner.java
 2021-04-26 12:19:01 +01:00
Viraj Jasani 5c2b728516
HDFS-15989. Split TestBalancer and De-flake testMaxIterationTime() (#2943) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
 2021-04-23 12:58:16 +09:00
Wei-Chiu Chuang 8ff61f9b07
HADOOP-17627. Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39. (#2879) 
* HADOOP-17371. Bump Jetty to the latest version 9.4.34. Contributed by Wei-Chiu Chuang. (#2453)

(cherry picked from commit 66ee0a6df0)
(cherry picked from commit 6340ac857b)

 Conflicts:
	hadoop-client-modules/hadoop-client-minicluster/pom.xml

Change-Id: I673ac136922740cb1d426cb9593aa1bd3e9acd32

* HADOOP-17621. hadoop-auth to remove jetty-server dependency. (#2865)

Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit dac60b8282)
(cherry picked from commit 1110b03752)

* HADOOP-17625. Update to Jetty 9.4.39. (#2870)

Reviewed-by: cxorm <lianp964@gmail.com>
(cherry picked from commit 6040e86e99)
(cherry picked from commit 7f7535534d)
 2021-04-19 20:12:27 -07:00
Wei-Chiu Chuang f0331827e5
HDFS-15940. Fixing and refactoring tests specific to Block recovery. (#2902) 
* HDFS-15940. Fixing and refactoring tests specific to Block recovery.(#2844). Contributed by Viraj Jasani
(cherry picked from commit 2b5fd341b9)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockRecovery.java

* HDFS-15940. Fix TestBlockRecovery2#testRaceBetweenReplicaRecoveryAndFinalizeBlock (ADDENDUM) (#2874)

(cherry picked from commit 56bd968fb4)
(cherry picked from commit 4994b73eeb)

Co-authored-by: Viraj Jasani <vjasani@apache.org>
 2021-04-15 11:20:23 +08:00
Wei-Chiu Chuang e6df0fb84c HDFS-15815. if required storageType are unavailable, log the failed reason during choosing Datanode. Contributed by Yang Yun. (#2882) 
(cherry picked from commit e391844e8e)

Co-authored-by: Ayush Saxena <ayushsaxena@apache.org>
(cherry picked from commit bfba6f1f3c)
 2021-04-13 15:00:14 +08:00
Wei-Chiu Chuang 0faa626bd8
HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e6892675)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
 2021-04-12 19:45:49 -07:00
Stephen O'Donnell 3422691fb2 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7)
 2021-04-08 12:31:36 +01:00
litao d566e4e41d HDFS-15892. Add metric for editPendingQ in FSEditLogAsync (#2770) 
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
(cherry picked from commit 4bd04126d6)
 2021-04-02 11:00:33 +09:00
Viraj Jasani 5458ebf607
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 16:45:24 -07:00
He Xiaoqiao d456106dce
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:26:25 +08:00
Viraj Jasani 97b8992fd8
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2797) 2021-03-24 10:13:59 -07:00
Takanobu Asanuma 9060972172
HADOOP-17138. Fix spotbugs warnings surfaced after upgrade to 4.0.6. (#2155) (#2806) 
(cherry picked from commit 1b29c9bfee)

Co-authored-by: Masatake Iwasaki <iwasakims@apache.org>
 2021-03-24 17:09:43 +09:00
litao d05d15620e
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 2021-03-23 18:45:24 +09:00
Ayush Saxena 3533889884 HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58)
 2021-03-22 12:55:47 +00:00
Ayush Saxena 4f56caf61f HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad)
 2021-03-19 19:23:50 +00:00
Viraj Jasani c86a2fd6d5 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348)
 2021-03-17 15:14:03 +09:00
Takanobu Asanuma 13c52a7adc HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8)
 2021-03-17 14:18:02 +09:00
touchida 87015f073a
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7)
 2021-03-17 12:11:34 +09:00
Hui Fei 01f9d73dad HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee4)
 2021-03-11 10:22:34 +08:00
tomscut 03ac2e41c0 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106)
 2021-03-06 14:17:10 -08:00
litao 92e29d57d9 HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380)
 2021-03-03 22:37:23 +09:00
Konstantin V Shvachko 1d48f3b44c HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4)
 2021-03-01 17:24:21 -08:00
S O'Donnell 707806092f HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
 2021-03-01 12:24:31 +00:00
S O'Donnell f98bc24ab6 HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5)
 2021-02-24 11:22:13 +00:00
Wang Yuxuan fe822b525e HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
(cherry picked from commit 52a46f064a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
 2021-02-22 18:15:07 +08:00