Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,879 Commits 326 Branches 371 Tags
Commit Graph

7370 Commits

Author SHA1 Message Date
Wei-Chiu Chuang
8a865273db HDFS-15759. EC: Verify EC reconstruction correctness on DataNode (#2585) (#2868) 
(cherry picked from commit 95e68926750b55196cf9da53c25359c98ef58a4f)
(cherry picked from commit 6cfa01c905ca422c3c6b2d1d735dd16ee08b4fb4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/StripedReconstructor.java

Co-authored-by: touchida <56789230+touchida@users.noreply.github.com>
(cherry picked from commit 0faa626bd84a02688e0938de1320fc32ec550aea)
 2021-04-13 11:09:52 +08:00
Stephen O'Donnell
40ed9e1dc0 HDFS-15937. Reduce memory used during datanode layout upgrade. Contributed by Stephen O'Donnell (#2838) 
(cherry picked from commit 4c567fcff7af45c75117ee4a75c087aa454a89e5)
 2021-04-08 12:45:01 +01:00
Viraj Jasani
739f70b552
HDFS-15931 : Fix non-static inner classes for better memory management (#2830). Contributed by Viraj Jasani 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2021-04-01 17:05:07 -07:00
He Xiaoqiao
b33e3083ec
HDFS-15919. BlockPoolManager should log stack trace if unable to get Namenode addresses. Contributed by Stephen O'Donnell. 2021-03-25 12:32:34 +08:00
Viraj Jasani
a0dd4e0722
HDFS-15911 : Provide blocks moved count in Balancer iteration result (#2799) 
Contributed by Viraj Jasani.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Ayush Saxena <ayushsaxena@apache.org>
 2021-03-24 10:19:56 -07:00
litao
75f2d47ff5 HDFS-15906. Close FSImage and FSNamesystem after formatting is complete (#2800) 
(cherry picked from commit d05d15620e320d28bfbeeee2a4f5224e69062528)
 2021-03-23 18:46:19 +09:00
Ayush Saxena
c9c471e04e HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team. 
(cherry picked from commit e3d1966f58ad473b8e852aa2b11c8ed2b434d9e4)
 2021-03-22 13:10:02 +00:00
Ayush Saxena
30a1b17b22 HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team. 
(cherry picked from commit 2377df38ad2396f8b42d60031001a25d9d6fa32e)
 2021-03-19 19:50:19 +00:00
Viraj Jasani
503101c1f1 HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles (#2772) 
(cherry picked from commit 9c43b60348bea7f89f7c0fc86f2813fdf2d1929f)
 2021-03-17 15:14:29 +09:00
Takanobu Asanuma
b898058e45 HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel. 
(cherry picked from commit 9ba60c33d8e68766aec49e50ec4c86519c6f9889)
 2021-03-17 14:18:54 +09:00
touchida
b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically (#2726) 
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc74ab8a6bbc53305b7a49df2437406f2)
 2021-03-17 13:12:40 +09:00
Hui Fei
7250f0e327 HDFS-15875. Check whether file is being truncated before truncate (#2746) 
(cherry picked from commit 6a55baeee46f61686bd7fd8b62d141399e9af4dc)
 2021-03-11 10:45:57 +08:00
tomscut
7b925bc486 HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. (#2668) Contributed by tomscut. 
(cherry picked from commit 9cb51bf106802c78b1400fba9f1d1c7e772dd5e7)
 2021-03-06 14:18:20 -08:00
litao
c503130d8b HDFS-15870. Remove unused configuration dfs.namenode.stripe.min (#2739) 
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380bce2ce7d8918e0772a5762fdba3122)
 2021-03-03 22:38:04 +09:00
Konstantin V Shvachko
78bd68a0b8 HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu. 
(cherry picked from commit 9501c698f4789188f744f3a1fba7f1a9bb9b8aa4)
 2021-03-01 17:27:38 -08:00
S O'Donnell
53f46214e4 HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell 
(cherry picked from commit c3b3b36dee475e5f37f85946be1a42a1b1e622a5)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java

(cherry picked from commit 707806092f64f709814e18cb0d1b4f58731a5716)
 2021-03-01 12:26:41 +00:00
S O'Donnell
bf229135ca HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee 
(cherry picked from commit 9c583f36d5d0024ab052429ef2f0c0b20e1ea258)
 2021-02-24 11:33:43 +00:00
Wang Yuxuan
09bbbc8b86 HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED (#2559) 
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2fa83ad4119954b32a3a8daac9d08134)
(cherry picked from commit 52a46f064a70505034367cffc32dfa1fabc7134b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
(cherry picked from commit fe822b525e4b9d5ebe58eb5f6d81af717f6290f3)
 2021-02-22 18:16:12 +08:00
jianghuazhu
cab9172d2e HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. (#2686) 
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8bf96cf5a1ec75134fb67fdeb955e5b6)
(cherry picked from commit ee0e32abdcfd08582e4545c30e7dca120e5dafd3)
(cherry picked from commit efb796f45fb22a83587da941c5ba70efd34af1b9)
 2021-02-22 15:25:08 +08:00
Zehao Chen
e162ef84dc HDFS-15821. Add metrics for in-service datanodes (#2690). Contributed by Zehao Chen. 
(cherry picked from commit 07a4220cd27c69b86b837e8da320bad0031f7895)
 2021-02-15 19:41:30 +00:00
Jim Brennan
0661977821 [HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan 2021-02-05 23:01:50 +00:00
fermi
e4d5ac227c HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang 
(cherry picked from commit 7ef2875c8932c54ced1f0d4b892a54d22d7cfaf3)
 2021-02-05 11:18:09 +08:00
Hui Fei
6e5cc45a16 HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang 
(cherry picked from commit 15a1f7adfc02af81e89e9fb96691f6cdf19ceff5)
 2021-02-04 10:01:55 +08:00
crossfire
69635e09d2 HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka (#2657) 
(cherry picked from commit 18978f2e204d105fb05807d33387a048e9ddb762)
 2021-02-02 10:44:59 +00:00
Jim Brennan
dfe23ab6ba [HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan. 
(cherry picked from commit f565476939332c2b17a0f470ac650ab55c6c0f64)
 2021-01-28 18:41:09 +00:00
Jim Brennan
7943b31c29 [HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan. 
(cherry picked from commit 6184ab07f670c181d1fda99e76729e920fff287e)
 2021-01-28 17:14:27 +00:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang. 2021-01-22 00:45:17 +08:00
Akira Ajisaka
33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement (#2626) 
(cherry picked from commit 45b3a8470975e67cce0e3e4d8eb8c7b6e7b6992f)
 2021-01-21 10:12:03 +09:00
Masatake Iwasaki
9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. (#2475) 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c72c27b02032ed3cf9dde15745ed605)
 2021-01-14 16:44:59 +09:00
Ahmed Hussein
640d8a1f75
HDFS-15717. Improve fsck logging. (#2529) Contributed by Kihwal Lee and Ahmed Hussein 
(cherry picked from commit be35fa186cc0c5c1f90028a9383851d47791bf04)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
 2021-01-14 10:15:58 +09:00
touchida
669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently (#2598) 
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba462770c99f4fe3521f4448d5bc7b0a)
 2021-01-13 11:39:27 +09:00
Ahmed Hussein
a23faaec18 HDFS-15704. Mitigate lease monitor's rapid infinite loop. (#2511). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit c2672bb234256e342fd9b5ef6fe4455471009693)
 2020-12-17 18:29:03 +00:00
Ayush Saxena
9e50a1082b HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena. 2020-12-15 11:05:17 +05:30
Ayush Saxena
a68ca4a292 HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena. 2020-12-15 10:35:01 +05:30
Ayush Saxena
a09d636e2c HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aaa628b750ea5ad2eb0d68e0fdbc55d5)
(cherry picked from commit ed78df0f7596ee613db5ee9a591c05f03e6b3fc4)
 2020-12-14 20:41:36 -08:00
S O'Donnell
e7df47c45c HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell 
(cherry picked from commit 52adda89c422dbaa30ca48ea70c45693fdcb9df7)
 2020-12-14 11:51:44 +00:00
Neil
6258405f10 HDFS-15720 namenode audit async logger should add some log4j config (#2532) 
(cherry picked from commit 9bd3c9bc50672ebdd7f12b9a2235aec37b2a07cf)
(cherry picked from commit bc5458bbd4e6829bc02de57671c017fbdac06849)
(cherry picked from commit 9123f9245cc49764d93a5295544a17519191430f)
 2020-12-10 13:50:55 -08:00
Ahmed Hussein
8b8a685b65 HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy (#2528) 
(cherry picked from commit 01383a21722be706d7ab682323285e794f71cadf)
 2020-12-08 19:54:12 +00:00
Hui Fei
2fde1f8bbf HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao. 2020-12-08 11:03:02 +08:00
crossfire
173d298c63 HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. (#2518) 
(cherry picked from commit 40f7543a6d5765c98d41c78736124b7b7f078aa2)
(cherry picked from commit edd9b659caf77d14caf6df88bac3cedf025e1f25)
(cherry picked from commit 31a6b45117c7af2b045cc19461e1c232ce805b02)
 2020-12-07 15:50:58 -08:00
Jim Brennan
8df2902a53 HDFS-15707. NNTop counts don't add up as expected. (#2516) Contributed by Ahmed Hussein and Daryn Sharp 
(cherry picked from commit 6a5864ee4ac387a390642f56aff8581fb5ed8ffd)
 2020-12-07 21:38:39 +00:00
Jim Brennan
521d5b381a HDFS-15703. Don't generate edits for set operations that are no-op (#2508). Contributed by Daryn Sharp and Ahmed Hussein 2020-12-02 22:25:12 +00:00
Ahmed Hussein
a63a30ff99 HDFS-15695. NN should not let the balancer run in safemode (#2489). Contributed by Daryn Sharp and Ahmed Hussein 
(cherry picked from commit 2b5b556dd7f17f095548b4a4485e7b7218af74be)
 2020-12-02 20:54:32 +00:00
Ahmed Hussein
2331f86dfb HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. (#2487) Contributed by Kuhu Shukla and Ahmed Hussein 
(cherry picked from commit 918ba9e82fb0bc6f96301c7ca74a32018b60c49e)
 2020-12-01 21:43:37 +00:00
Takanobu Asanuma
cbc352ea2d HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li. 
(cherry picked from commit f56cd88d7d9381d13329d25bb4f367e5378b6b5b)
 2020-11-13 11:10:32 +09:00
Yizhong Zhang
e2d7e72c1b HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue. 
(cherry picked from commit 82de07c794fe119946bec2e563c1937f312e8ac9)
(cherry picked from commit 12bfd9194a1b6ff89fb31f1f49e69c727f844155)
(cherry picked from commit 52c2ce53c25738713c6bc2b710249dd708c3d9de)
 2020-11-10 09:14:37 -08:00
Ayush Saxena
76a7c0c0ef HDFS-15643. EC: Fix checksum computation in case of native encoders. (#2424). Contributed by Ayush Saxena. 2020-11-04 09:57:13 +05:30
Konstantin V Shvachko
079f1bb2f9 HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko. 
(cherry picked from commit d07dc7afb4aa0d6cc9f9be530802e54610776a4d)
 2020-11-03 12:34:01 -08:00
Ahmed Hussein
34915b0373
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky (#2410) 
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac43a480a010d40c182c1d4cc5346db8f)
 2020-10-27 20:55:16 +09:00
Hui Fei
eb909fd726 HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang. 
(cherry picked from commit 12c908c827c80ea37ed5207a65831d0b699c8381)
 2020-10-26 22:41:24 +08:00