Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,815 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7627 Commits

Author SHA1 Message Date
Wang Yuxuan 09bbbc8b86 HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f)
(cherry picked from commit 52a46f064a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
(cherry picked from commit fe822b525e)
 2021-02-22 18:16:12 +08:00
jianghuazhu cab9172d2e HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b)
(cherry picked from commit ee0e32abdc)
(cherry picked from commit efb796f45f)
 2021-02-22 15:25:08 +08:00
Zehao Chen e162ef84dc HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd2)
 2021-02-15 19:41:30 +00:00
Jim Brennan 0661977821 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 23:01:50 +00:00
fermi e4d5ac227c HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c89)
 2021-02-05 11:18:09 +08:00
Hui Fei 6e5cc45a16 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc)
 2021-02-04 10:01:55 +08:00
crossfire 69635e09d2 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e20)
 2021-02-02 10:44:59 +00:00
Jim Brennan dfe23ab6ba [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939)
 2021-01-28 18:41:09 +00:00
Jim Brennan 7943b31c29 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f6)
 2021-01-28 17:14:27 +00:00
He Xiaoqiao d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Akira Ajisaka 33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a84709)
 2021-01-21 10:12:03 +09:00
Masatake Iwasaki 9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c)
 2021-01-14 16:44:59 +09:00
Ahmed Hussein 640d8a1f75
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186c)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 2021-01-14 10:15:58 +09:00
touchida 669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba)
 2021-01-13 11:39:27 +09:00
Wei-Chiu Chuang 13768fe6f0 HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout (#2533) 
(cherry picked from commit 2b4febcf57)
(cherry picked from commit 94c126cc9e)
(cherry picked from commit 8af0fcf628)
 2021-01-04 21:00:29 -08:00
Ahmed Hussein a23faaec18 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234)
 2020-12-17 18:29:03 +00:00
Ayush Saxena 9e50a1082b HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 11:05:17 +05:30
Ayush Saxena a68ca4a292 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:35:01 +05:30
Ayush Saxena a09d636e2c HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aa)
(cherry picked from commit ed78df0f75)
 2020-12-14 20:41:36 -08:00
S O'Donnell e7df47c45c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 52adda89c4)
 2020-12-14 11:51:44 +00:00
Jim Brennan 777f426d6a HDFS-15711. Add Metrics to HttpFS Server. (#2521) Contributed by Ahmed Hussein and Kihwal Lee 2020-12-10 22:16:52 +00:00
Neil 6258405f10 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50)
(cherry picked from commit bc5458bbd4)
(cherry picked from commit 9123f9245c)
 2020-12-10 13:50:55 -08:00
Ahmed Hussein 8b8a685b65 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a2172)
 2020-12-08 19:54:12 +00:00
Hui Fei 2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
crossfire 173d298c63 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d)
(cherry picked from commit edd9b659ca)
(cherry picked from commit 31a6b45117)
 2020-12-07 15:50:58 -08:00
Jim Brennan 8df2902a53 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4a)
 2020-12-07 21:38:39 +00:00
Ahmed Hussein 5f5d7ecf59 HDFS-15706. HttpFS: Log more information on request failures. (#2515) 
(cherry picked from commit 07655a7040)
(cherry picked from commit a7dbd3b6f9)
(cherry picked from commit f547cd43d1)
 2020-12-03 15:15:47 -08:00
Jim Brennan 521d5b381a HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 22:25:12 +00:00
Ahmed Hussein a63a30ff99 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7)
 2020-12-02 20:54:32 +00:00
Ahmed Hussein 2331f86dfb HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82f)
 2020-12-01 21:43:37 +00:00
Takanobu Asanuma cbc352ea2d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d)
 2020-11-13 11:10:32 +09:00
Yizhong Zhang e2d7e72c1b HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794)
(cherry picked from commit 12bfd9194a)
(cherry picked from commit 52c2ce53c2)
 2020-11-10 09:14:37 -08:00
Ayush Saxena 76a7c0c0ef HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:57:13 +05:30
Konstantin V Shvachko 079f1bb2f9 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4)
 2020-11-03 12:34:01 -08:00
Ahmed Hussein 34915b0373
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4)
 2020-10-27 20:55:16 +09:00
Hui Fei eb909fd726 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827)
 2020-10-26 22:41:24 +08:00
Wei-Chiu Chuang 7f3645ec5b HDFS-15644. Failed volumes can cause DNs to stop block reporting. Contributed by Ahmed Hussein. 
(cherry picked from commit 74634eb002)
(cherry picked from commit c8eb86ee3d)
(cherry picked from commit 87666e2438)
 2020-10-23 12:06:47 -07:00
Inigo Goiri 5f4a12c8a3 HDFS-15459. TestBlockTokenWithDFSStriped fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit e7aa4da0a2)
 2020-10-23 09:50:36 -07:00
Kihwal Lee 7a3085d552 HDFS-15622. Deleted blocks linger in the replications queue. Contributed by Ahmed Hussein. 
(cherry picked from commit da1b6e3cc2)
 2020-10-22 21:01:09 -05:00
Kihwal Lee 33db7c140b HDFS-15618. Improve datanode shutdown latency. Contributed by Ahmed Hussein. 
(cherry picked from commit cf932a7e2d)
(cherry picked from commit f3b2d85690)
 2020-10-22 15:26:34 -05:00
Takanobu Asanuma c808a21550 HDFS-15639. [JDK 11] Fix Javadoc errors in hadoop-hdfs-client. (#2394) 
(cherry picked from commit 30f06e0c74)
 2020-10-20 19:13:59 +09:00
Kihwal Lee ffe6e39c96 HDFS-15628. HttpFS server throws NPE if a file is a symlink. Contributed by Ahmed Hussein. 
(cherry picked from commit e45407128d)
 2020-10-16 11:41:17 -05:00
Kihwal Lee 9aefc1759b HDFS-15627. Log delete audits before collecting blocks. 
(cherry picked from commit 740a2c4635)
 2020-10-16 11:18:42 -05:00
Konstantin V Shvachko 2667fbd085 HDFS-15567. [SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly. Contributed by Konstantin V Shvachko. 
(cherry picked from commit b3786d6c3c)
 2020-10-12 17:52:37 -07:00
Doroszlai, Attila dd17772b53 HADOOP-16990. Update Mockserver. Contributed by Attila Doroszlai. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 8675a0d283bcf1f6a0c1e5953d28f7497c63ba66)
 2020-10-08 23:44:43 -07:00
S O'Donnell 4a8b59df59 HDFS-15415. Reduce locking in Datanode DirectoryScanner. Contributed by Stephen O'Donnell 2020-09-30 09:39:34 +01:00
S O'Donnell 27b2c1ea7b HDFS-15583. Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1. Contributed by Stephen O'Donnell 
(cherry picked from commit 5f34e3214e)
 2020-09-22 10:51:16 +01:00
hemanthboyina 43b113de69 HDFS-15574. Remove unnecessary sort of block list in DirectoryScanner. Contributed by Stephen O'Donnell. 
(cherry picked from commit aa582ccc2a)
 2020-09-17 10:17:30 +05:30
Uma Maheswara Rao G 0808cb6355 HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false (#2205) 
* HDFS-15515: mkdirs on fallback should throw IOE out instead of suppressing and returning false

* Used LambdaTestUtils#intercept in test

(cherry picked from commit 99b120a06e)
 2020-08-25 12:16:47 -07:00
S O'Donnell 0f234667dd HDFS-14852. Removing from LowRedundancyBlocks does not remove the block from all queues. Contributed by Fei Hui. 
(cherry picked from commit 6a49bf9bff)
 2020-08-25 16:05:00 +01:00