Commit Graph

7251 Commits

Author SHA1 Message Date
Wei-Chiu Chuang 61915fbd83 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs.
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
(cherry picked from commit 096533c2dc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties

(cherry picked from commit 1d55e1e229)
2020-03-18 09:59:12 -07:00
S O'Donnell 4e1021975b HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:47:57 +00:00
Surendra Singh Lilhore 64eb5b2709 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:47:54 +05:30
Masatake Iwasaki abee7402e4 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago.
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1)
2020-02-27 16:37:23 -08:00
Konstantin V Shvachko 4af188587f HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko.
(cherry picked from commit ecbcb058b8)
2020-02-27 16:02:30 -08:00
Ayush Saxena 7bfa123d58 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li.
(cherry picked from commit b182dd809c)
2020-02-27 14:55:50 -08:00
Kihwal Lee b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed
by Ahmed Hussein.

(cherry picked from commit 27cfda708e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
2020-02-27 09:45:12 -06:00
Stephen O'Donnell 8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b990)
(cherry picked from commit 2377649cdb)
2020-02-25 13:29:15 -08:00
Wei-Chiu Chuang 268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun.
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817)
2020-02-24 13:37:59 -08:00
Ayush Saxena d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko.
(cherry picked from commit 6f84269bcd)
2020-02-21 10:56:31 -08:00
Masatake Iwasaki 8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758)
(cherry picked from commit 2338d25dc7)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec72)
2020-02-21 13:51:26 +09:00
Kihwal Lee c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
2020-02-17 16:07:03 -06:00
Ayush Saxena 2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore 6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Kihwal Lee bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default.
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97b)
(cherry picked from commit f1840669cf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
2020-02-11 14:57:49 -06:00
Ayush Saxena 547f1798aa HDFS-15157. Fix compilation failure for branch-3.1. Contributed by Ayush Saxena 2020-02-09 08:14:33 +05:30
Chen Liang 466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang.
(cherry picked from commit 226cd3977b)
2020-02-04 12:36:46 -08:00
Kihwal Lee a55a0a1f6d HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan.
(cherry picked from commit 10a60fbe20)
2020-02-04 12:24:58 -06:00
Stephen O'Donnell 8ea4787f9f HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d93)
(cherry picked from commit e89436b183)
2020-01-31 16:13:53 -08:00
Inigo Goiri ffaf0d3c82 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein.
(cherry picked from commit 1839c467f6)
2020-01-31 11:25:36 -08:00
Chen Liang 5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Kihwal Lee b8d87cfc32 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4)
2020-01-29 11:01:51 -06:00
Ayush Saxena 1293199532 HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:17:21 +05:30
Kihwal Lee 7dd43cec7f HDFS-15119. Allow expiration of cached locations in DFSInputStream.
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c9)
(cherry picked from commit 1bb9667137)
2020-01-24 09:53:26 -06:00
Kihwal Lee b4e9725955 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein.
(cherry picked from commit bd03053ea2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java

(cherry picked from commit 4842708320)
2020-01-22 09:28:01 -06:00
Ayush Saxena 12b258115b HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:34:37 +05:30
Chen Liang 5490061885 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang.
(cherry picked from commit 26a969ec73)
2020-01-10 17:19:05 -08:00
Kihwal Lee 82fa8d88ef HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein.
(cherry picked from commit 5fb901ac40)
2020-01-10 12:47:49 -06:00
Surendra Singh Lilhore 09d3707875 HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:18:58 +05:30
Rakesh Radhakrishnan b1eb47865c HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He.
(cherry picked from commit 7030722e5d)
2020-01-08 14:32:45 +05:30
Masatake Iwasaki c5580af1c4 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797)
(cherry picked from commit aba3f6c3e1)
2020-01-08 17:06:49 +09:00
Masatake Iwasaki 4c8b01abcb
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775)
(cherry picked from commit a43c177f1d)
2020-01-08 12:38:51 +09:00
Rakesh Radhakrishnan 8b748c0308 HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He.
(cherry picked from commit d79cce20ab)
2020-01-02 12:19:38 +05:30
Surendra Singh Lilhore 55510d32bf HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:25:28 +05:30
Takanobu Asanuma 454a0c2913 HDFS-14934. [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0. Contributed by Ayush Saxena.
(cherry picked from commit dc32f583af)
2019-12-28 21:35:22 +09:00
Konstantin V Shvachko 1a22e4036c HDFS-15076. Fix tests that hold FSDirectory lock, without holding FSNamesystem lock. Contributed by Konstantin V Shvachko.
(cherry picked from commit b98ac2a3af)
2019-12-24 12:50:13 -08:00
Eric Badger ef00db4ce9 Revert "HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui."
This reverts commit 27555620fa.
2019-12-20 19:16:22 +00:00
Inigo Goiri 27555620fa HDFS-15062. Add LOG when sendIBRs failed. Contributed by Fei Hui.
(cherry picked from commit 52d7b745c6)
2019-12-19 09:47:30 -08:00
Shashikant Banerjee 280cd6dd1e HDFS-15012. NN fails to parse Edit logs after applying HDFS-13101. Contributed by Shashikant Banerjee.
(cherry picked from commit fdd96e46d1)
2019-12-18 23:11:18 +05:30
Chen Liang ea0eeb8f1a HDFS-15036. Active NameNode should not silently fail the image transfer. Contributed by Chen Liang.
(cherry picked from commit 65c4660bcd)
(cherry picked from commit d4a6901c42)
2019-12-12 15:58:09 -08:00
Erik Krogen b06965ae32 HDFS-15032. Properly handle InvocationTargetExceptions in the proxy created by ProxyCombiner. This fixes a bug encountered by the HDFS balancer when used with Observer Nodes. Contributed by Erik Krogen.
(cherry picked from c174d50b30)
(cherry picked from 3402c87353)
2019-12-11 09:41:11 -08:00
Surendra Singh Lilhore 5eb21b83df HDFS-15010. BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method. Contributed by Surendra Singh Lilhore. 2019-11-29 00:07:45 +05:30
Ayush Saxena 35f0478042 HDFS-14519. NameQuota is not update after concat operation, so namequota is wrong. Contributed by Ranith Sardar. 2019-11-23 02:47:59 +05:30
Surendra Singh Lilhore 8551c04726 HDFS-14940. HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB. Contributed by hemanthboyina. 2019-11-22 00:45:55 +05:30
He Xiaoqiao d101b74814 HDFS-14952. Skip safemode if blockTotal is 0 in new NN. Contributed by Xiaoqiao He.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Mukul Kumar Singh <msingh@apache.org>
(cherry picked from commit 0b50aa29fd)
(cherry picked from commit 1d383436f09f8e490171c8f6af4605971ad2f904)
2019-11-19 12:22:54 -08:00
Siyao Meng df13f6d485 HADOOP-16703. Backport HADOOP-16152 to branch-3.1. Contributed by Siyao Meng.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-11-19 12:14:27 -08:00
Ayush Saxena b4f0c63a6a HDFS-14942. Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex. Contributed by Lisheng Sun.
(cherry picked from commit 9e287054a8)
(cherry picked from commit d16441460215155b25fa649a7ca733e579d97929)
2019-11-18 10:23:48 -08:00
Erik Krogen 17779adb32 HDFS-14973. More strictly enforce Balancer/Mover/SPS throttling of getBlocks RPCs to NameNodes. Contributed by Erik Krogen.
(cherry picked from b2cc8b6b4a)
(cherry picked from 60655bfe54)
2019-11-15 11:14:49 -08:00