litao
75f2d47ff5
HDFS-15906. Close FSImage and FSNamesystem after formatting is complete ( #2800 )
...
(cherry picked from commit d05d15620e
)
2021-03-23 18:46:19 +09:00
Ayush Saxena
c9c471e04e
HDFS-15439. Setting dfs.mover.retry.max.attempts to negative value will retry forever. Contributed by AMC-team.
...
(cherry picked from commit e3d1966f58
)
2021-03-22 13:10:02 +00:00
Ayush Saxena
30a1b17b22
HDFS-15438. Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy. Contributed by AMC-team.
...
(cherry picked from commit 2377df38ad
)
2021-03-19 19:50:19 +00:00
Ayush Saxena
8354ce680b
HDFS-15633. Avoid redundant RPC calls for getDiskStatus. ( #2386 ). Contributed by Ayush Saxena.
2021-03-19 23:14:37 +05:30
Viraj Jasani
503101c1f1
HDFS-15895 : Remove redundant String#format in DFSAdmin#printOpenFiles ( #2772 )
...
(cherry picked from commit 9c43b60348
)
2021-03-17 15:14:29 +09:00
Takanobu Asanuma
b898058e45
HDFS-15890. Improve the Logs for File Concat Operation. Contributed by Bhavik Patel.
...
(cherry picked from commit 9ba60c33d8
)
2021-03-17 14:18:54 +09:00
touchida
b4c07209fc
HDFS-13975. TestBalancer#testMaxIterationTime fails sporadically ( #2726 )
...
Co-authored-by: toshihiko.uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 25390babc7
)
2021-03-17 13:12:40 +09:00
Hui Fei
7250f0e327
HDFS-15875. Check whether file is being truncated before truncate ( #2746 )
...
(cherry picked from commit 6a55baeee4
)
2021-03-11 10:45:57 +08:00
tomscut
7b925bc486
HDFS-15808. Add metrics for FSNamesystem read/write lock hold long time. ( #2668 ) Contributed by tomscut.
...
(cherry picked from commit 9cb51bf106
)
2021-03-06 14:18:20 -08:00
litao
c503130d8b
HDFS-15870. Remove unused configuration dfs.namenode.stripe.min ( #2739 )
...
Co-authored-by: tomscut <litao@bigo.sg>
(cherry picked from commit cdba06e380
)
2021-03-03 22:38:04 +09:00
Konstantin V Shvachko
78bd68a0b8
HDFS-15849. ExpiredHeartbeats metric should be of Type.COUNTER. Contributed by Qi Zhu.
...
(cherry picked from commit 9501c698f4
)
2021-03-01 17:27:38 -08:00
S O'Donnell
53f46214e4
HDFS-14013. Skip any credentials stored in HDFS when starting ZKFC. Contributed by Stephen O'Donnell
...
(cherry picked from commit c3b3b36dee
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java
(cherry picked from commit 707806092f
)
2021-03-01 12:26:41 +00:00
Surendra Singh Lilhore
379a7a5726
HDFS-15210. EC : File write hanged when DN is shutdown by admin command. Contributed by Surendra Singh Lilhore.
...
(cherry picked from commit db6252b6c3
)
(cherry picked from commit 7856af2cc65377352db5ebc93a778f373f71d215)
(cherry picked from commit 5315de3d437f78d81dc5fa9f43849de6198220e6)
2021-02-26 10:49:59 +08:00
S O'Donnell
bf229135ca
HDFS-15422. Reported IBR is partially replaced with stored info when queuing. Contributed by Stephen O'Donnell and Kihwal Lee
...
(cherry picked from commit 9c583f36d5
)
2021-02-24 11:33:43 +00:00
Wang Yuxuan
09bbbc8b86
HDFS-15734. [READ] DirectoryScanner#scan need not check StorageType.PROVIDED ( #2559 )
...
Co-authored-by: wangyuxuan <wangyuxuan@xiaomi.com>
(cherry picked from commit 1e32ab7d2f
)
(cherry picked from commit 52a46f064a
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DirectoryScanner.java
(cherry picked from commit fe822b525e
)
2021-02-22 18:16:12 +08:00
jianghuazhu
cab9172d2e
HDFS-15826. Solve the problem of incorrect progress of delegation tokens when loading FsImage. ( #2686 )
...
Co-authored-by: zhujianghua <zhujianghua@zhujianghuadeMacBook-Pro.local>
(cherry picked from commit 98ffd65d8b
)
(cherry picked from commit ee0e32abdc
)
(cherry picked from commit efb796f45f
)
2021-02-22 15:25:08 +08:00
Zehao Chen
e162ef84dc
HDFS-15821. Add metrics for in-service datanodes ( #2690 ). Contributed by Zehao Chen.
...
(cherry picked from commit 07a4220cd2
)
2021-02-15 19:41:30 +00:00
Jim Brennan
0661977821
[HDFS-15813] DataStreamer: keep sending heartbeat packets during flush. Contributed by Daryn Sharp and Jim Brennan
2021-02-05 23:01:50 +00:00
fermi
e4d5ac227c
HDFS-15798. EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number. Contributed by huhaiyang
...
(cherry picked from commit 7ef2875c89
)
2021-02-05 11:18:09 +08:00
Hui Fei
6e5cc45a16
HDFS-15779. EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block. Contributed by Hongbing Wang
...
(cherry picked from commit 15a1f7adfc
)
2021-02-04 10:01:55 +08:00
crossfire
69635e09d2
HDFS-15795. EC: Wrong checksum when reconstruction was failed by exception. Contributed by Yushi Hayasaka ( #2657 )
...
(cherry picked from commit 18978f2e20
)
2021-02-02 10:44:59 +00:00
Jim Brennan
dfe23ab6ba
[HDFS-15789] Lease renewal does not require namesystem lock. Contributed by Daryn Sharp and Jim Brennan.
...
(cherry picked from commit f565476939
)
2021-01-28 18:41:09 +00:00
Jim Brennan
7943b31c29
[HDFS-10498] Intermittent test failure TestSnapshotFileLength.testSnapshotFileLength. Contributed by Jim Brennan.
...
(cherry picked from commit 6184ab07f6
)
2021-01-28 17:14:27 +00:00
He Xiaoqiao
d1d79488fd
HADOOP-16947. Stale record should be remove when MutableRollingAverages generating aggregate data. Contributed by Haibin Huang.
2021-01-22 00:45:17 +08:00
Akira Ajisaka
33a71c8b9e
HDFS-15783. Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement ( #2626 )
...
(cherry picked from commit 45b3a84709
)
2021-01-21 10:12:03 +09:00
Masatake Iwasaki
9d5144e66d
HDFS-15672. TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk. ( #2475 )
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 630f8ddd2c
)
2021-01-14 16:44:59 +09:00
Ahmed Hussein
640d8a1f75
HDFS-15717. Improve fsck logging. ( #2529 ) Contributed by Kihwal Lee and Ahmed Hussein
...
(cherry picked from commit be35fa186c
)
Conflicts:
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java
2021-01-14 10:15:58 +09:00
touchida
669c6a7cf4
HDFS-15762. TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently ( #2598 )
...
Co-authored-by: Toshihiko Uchida <toshihiko.uchida@linecorp.com>
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 8ec824f2ba
)
2021-01-13 11:39:27 +09:00
Wei-Chiu Chuang
13768fe6f0
HDFS-15719. [Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout ( #2533 )
...
(cherry picked from commit 2b4febcf57
)
(cherry picked from commit 94c126cc9e
)
(cherry picked from commit 8af0fcf628
)
2021-01-04 21:00:29 -08:00
Ahmed Hussein
a23faaec18
HDFS-15704. Mitigate lease monitor's rapid infinite loop. ( #2511 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit c2672bb234
)
2020-12-17 18:29:03 +00:00
Ayush Saxena
9e50a1082b
HDFS-15200. Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage. Contributed by Ayush Saxena.
2020-12-15 11:05:17 +05:30
Ayush Saxena
a68ca4a292
HDFS-15187. CORRUPT replica mismatch between namenodes after failover. Contributed by Ayush Saxena.
2020-12-15 10:35:01 +05:30
Ayush Saxena
a09d636e2c
HDFS-15170. EC: Block gets marked as CORRUPT in case of failover and pipeline recovery. Contributed by Ayush Saxena.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 5e5c53070f1946b86e49c039e1f5ae42e5d022cb)
(cherry picked from commit faf26f70aa
)
(cherry picked from commit ed78df0f75
)
2020-12-14 20:41:36 -08:00
S O'Donnell
e7df47c45c
HDFS-15725. Lease Recovery never completes for a committed block which the DNs never finalize. Contributed by Stephen O'Donnell
...
(cherry picked from commit 52adda89c4
)
2020-12-14 11:51:44 +00:00
Jim Brennan
777f426d6a
HDFS-15711. Add Metrics to HttpFS Server. ( #2521 ) Contributed by Ahmed Hussein and Kihwal Lee
2020-12-10 22:16:52 +00:00
Neil
6258405f10
HDFS-15720 namenode audit async logger should add some log4j config ( #2532 )
...
(cherry picked from commit 9bd3c9bc50
)
(cherry picked from commit bc5458bbd4
)
(cherry picked from commit 9123f9245c
)
2020-12-10 13:50:55 -08:00
Ahmed Hussein
8b8a685b65
HDFS-15716. WaitforReplication in TestUpgradeDomainBlockPlacementPolicy ( #2528 )
...
(cherry picked from commit 01383a2172
)
2020-12-08 19:54:12 +00:00
Hui Fei
2fde1f8bbf
HDFS-15240. Erasure Coding: dirty buffer causes reconstruction block error. Contributed by HuangTao.
2020-12-08 11:03:02 +08:00
crossfire
173d298c63
HDFS-15709. Socket file descriptor leak in StripedBlockChecksumReconstructor. ( #2518 )
...
(cherry picked from commit 40f7543a6d
)
(cherry picked from commit edd9b659ca
)
(cherry picked from commit 31a6b45117
)
2020-12-07 15:50:58 -08:00
Jim Brennan
8df2902a53
HDFS-15707. NNTop counts don't add up as expected. ( #2516 ) Contributed by Ahmed Hussein and Daryn Sharp
...
(cherry picked from commit 6a5864ee4a
)
2020-12-07 21:38:39 +00:00
Ahmed Hussein
5f5d7ecf59
HDFS-15706. HttpFS: Log more information on request failures. ( #2515 )
...
(cherry picked from commit 07655a7040
)
(cherry picked from commit a7dbd3b6f9
)
(cherry picked from commit f547cd43d1
)
2020-12-03 15:15:47 -08:00
Jim Brennan
521d5b381a
HDFS-15703. Don't generate edits for set operations that are no-op ( #2508 ). Contributed by Daryn Sharp and Ahmed Hussein
2020-12-02 22:25:12 +00:00
Ahmed Hussein
a63a30ff99
HDFS-15695. NN should not let the balancer run in safemode ( #2489 ). Contributed by Daryn Sharp and Ahmed Hussein
...
(cherry picked from commit 2b5b556dd7
)
2020-12-02 20:54:32 +00:00
Ahmed Hussein
2331f86dfb
HDFS-15694. Avoid calling UpdateHeartBeatState inside DataNodeDescriptor. ( #2487 ) Contributed by Kuhu Shukla and Ahmed Hussein
...
(cherry picked from commit 918ba9e82f
)
2020-12-01 21:43:37 +00:00
Takanobu Asanuma
cbc352ea2d
HDFS-15538. Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml. Contributed by Xieming Li.
...
(cherry picked from commit f56cd88d7d
)
2020-11-13 11:10:32 +09:00
Yizhong Zhang
e2d7e72c1b
HADOOP-17142. Fix outdated properties of JournalNode when performing rollback. Contributed by Deegue.
...
(cherry picked from commit 82de07c794
)
(cherry picked from commit 12bfd9194a
)
(cherry picked from commit 52c2ce53c2
)
2020-11-10 09:14:37 -08:00
Ayush Saxena
76a7c0c0ef
HDFS-15643. EC: Fix checksum computation in case of native encoders. ( #2424 ). Contributed by Ayush Saxena.
2020-11-04 09:57:13 +05:30
Konstantin V Shvachko
079f1bb2f9
HDFS-15665. Balancer logging improvements. Contributed by Konstantin V Shvachko.
...
(cherry picked from commit d07dc7afb4
)
2020-11-03 12:34:01 -08:00
Ahmed Hussein
34915b0373
HDFS-9776. testMultipleAppendsDuringCatchupTailing is flaky ( #2410 )
...
Co-authored-by: ahussein <ahmed.hussein@verizonmedia.com>
(cherry picked from commit ae74407ac4
)
2020-10-27 20:55:16 +09:00
Hui Fei
eb909fd726
HDFS-15641. DataNode could meet deadlock if invoke refreshNameNode. Contributed by Hongbing Wang.
...
(cherry picked from commit 12c908c827
)
2020-10-26 22:41:24 +08:00