Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,591 Commits 326 Branches 371 Tags
Commit Graph

7265 Commits

Author SHA1 Message Date
Ayush Saxena
69163f5e5d Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:18:04 +05:30
Gabor Bota
ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Akira Ajisaka
74ec3c001d
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f696aba7fc209874d232baf2a50d547)
 2020-04-28 06:30:38 +09:00
Mingliang Liu
75354285e6
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:49:29 -07:00
Dhiraj
68dcdad666
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:53:32 -07:00
Wei-Chiu Chuang
6f7a30fd71 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170a062265b7687b31b2a3735827958fb)
(cherry picked from commit d6c48f81009eaf9462cb6ad1ec1cbe609b223cb2)
(cherry picked from commit 9e3b4c9ebd926aa7a5766fcc22239a4951982fc7)
 2020-04-24 11:54:29 -07:00
Mingliang Liu
03ff1d3bd8
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 14:14:19 -07:00
Inigo Goiri
3938aa422e HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 
(cherry picked from commit efdfe679d64ce9de4ba6aaf2afa34e180f68d969)
(cherry picked from commit 7d64d44515045bb3dedd91a4a0351bc5a6178a14)
 2020-04-21 09:50:31 -07:00
Stephen O'Donnell
0ea8f3a19c HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a9ea9f6e217a0ba158c48da89b3faf2)
(cherry picked from commit aaad947c740c91c36139f3f0569ae78b53bca682)
(cherry picked from commit 041c93a26cd0ff705bea9d6283951702cf2cf1c2)
 2020-04-16 20:19:26 -07:00
Ayush Saxena
6a4c3fad66 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 
(cherry picked from commit 62deab17a33cef723d73f8d8b9e37e5bddbc1813)
(cherry picked from commit 6cd9290401735a2c33a0ff0ae7324876ef9615e9)
 2020-03-25 17:14:24 -07:00
Eric Yang
b837431a08 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928bb877f26b11b5b212a100a79941f35)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

(cherry picked from commit ba6b3a384863b57bc7eeeb736950f544e6ed8d6d)
 2020-03-25 17:14:18 -07:00
Giovanni Matteo Fumarola
4aa7734fc7 HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota. 
(cherry picked from commit 00d5e631b596f8712600879366e5283829e7ee5d)
(cherry picked from commit eca7bc7ac4292ff08b9e1ea2e22116d6e58e8b95)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 2020-03-25 17:13:58 -07:00
Ayush Saxena
4891f24a94 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 
(cherry picked from commit 6191d4b4a0919863fda78e549ab6c60022e3ebc2)
(cherry picked from commit 1e3b0df6abcc1252907c41aaedb3e7e257bce497)
 2020-03-25 16:42:09 -07:00
Yiqun Lin
f1a19b7a3f HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e6063072b2bec491edd3f412eb7ac21f3)
 2020-03-25 16:31:54 -07:00
Wei-Chiu Chuang
61915fbd83 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
(cherry picked from commit 096533c2dc0afd51367030725d797480a22ba7e2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties

(cherry picked from commit 1d55e1e229d835a6743523b9cf4c1e97b4259128)
 2020-03-18 09:59:12 -07:00
S O'Donnell
4e1021975b HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:47:57 +00:00
Surendra Singh Lilhore
64eb5b2709 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:47:54 +05:30
Masatake Iwasaki
abee7402e4 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1406ea3a8225a1b6306c2e78440353b)
 2020-02-27 16:37:23 -08:00
Konstantin V Shvachko
4af188587f HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ecbcb058b8bc0fbc3903acb56814c6d9608bc396)
 2020-02-27 16:02:30 -08:00
Ayush Saxena
7bfa123d58 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 
(cherry picked from commit b182dd809cf42aed66806901631ad6062ca0820c)
 2020-02-27 14:55:50 -08:00
Kihwal Lee
b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708ef66dfbe5f52a5f1e716298a294f3f7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Stephen O'Donnell
8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b9907b590ed2d73a0d68f079c7f4d754b1)
(cherry picked from commit 2377649cdb1307a72a946b95a9a04f316c6483ef)
 2020-02-25 13:29:15 -08:00
Wei-Chiu Chuang
268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817e09c5a3ee7b378a0f4faee9b853315)
 2020-02-24 13:37:59 -08:00
Ayush Saxena
d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko
cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd5cdb08ca68b2d8276f66d34a2a7a0d)
 2020-02-21 10:56:31 -08:00
Masatake Iwasaki
8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7150d75fbda84cc95422380b5622224)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec7245769aebb36e52725522c42cb3dd88)
 2020-02-21 13:51:26 +09:00
Kihwal Lee
c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71cba068e7abf1c76ad629bfeec10852)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
Ayush Saxena
2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena
cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore
6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Kihwal Lee
bf09bfa2db HDFS-14758. Make lease hard limit configurable and reduce the default. 
Contributed by hemanthboyina.

(cherry picked from commit 9b8a78d97bfd825ce840c6033371c7f10e49a5b8)
(cherry picked from commit f1840669cfc92c033521bf95989479523e2a649d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsClientConfigKeys.java
 2020-02-11 14:57:49 -06:00
Ayush Saxena
547f1798aa HDFS-15157. Fix compilation failure for branch-3.1. Contributed by Ayush Saxena 2020-02-09 08:14:33 +05:30
Chen Liang
466f77a5cf HDFS-15148. dfs.namenode.send.qop.enabled should not apply to primary NN port. Contributed by Chen Liang. 
(cherry picked from commit 226cd3977bfc5948afff6a2d38a41e341089d5fe)
 2020-02-04 12:36:46 -08:00
Kihwal Lee
a55a0a1f6d HDFS-12491. Support wildcard in CLASSPATH for libhdfs. Contributed by Muhammad Samir Khan. 
(cherry picked from commit 10a60fbe20bb08cdd71076ea9bf2ebb3a2f6226e)
 2020-02-04 12:24:58 -06:00
Stephen O'Donnell
8ea4787f9f HDFS-7175. Client-side SocketTimeoutException during Fsck. Contributed by Stephen O'Donnell, Akira Ajisaka. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Co-authored-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit 1e3a0b0d931676b191cb4813ed1a283ebb24d4eb)
(cherry picked from commit e89436b1839aa87b4bedfc7e2d558c7541e73540)
 2020-01-31 16:13:53 -08:00
Inigo Goiri
ffaf0d3c82 HDFS-13179. TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently. Contributed by Ahmed Hussein. 
(cherry picked from commit 1839c467f60cbb8592d446694ec3d7710cda5142)
 2020-01-31 11:25:36 -08:00
Chen Liang
5e1d89b74d [SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster. Conntributed by Chen Liang 2020-01-29 11:15:35 -08:00
Kihwal Lee
b8d87cfc32 HDFS-15146. TestBalancerRPCDelay.testBalancerRPCDelay fails 
intermittently. Contributed by Ahmed Hussein.

(cherry picked from commit 799d4c1cf4e8fe78eb9ab607a0449cdd075041fb)
 2020-01-29 11:01:51 -06:00
Ayush Saxena
1293199532 HDFS-14993. checkDiskError doesn't work during datanode startup. Contributed by Yang Yun. 2020-01-28 23:17:21 +05:30
Kihwal Lee
7dd43cec7f HDFS-15119. Allow expiration of cached locations in DFSInputStream. 
Contributed by Ahmed Hussein.

(cherry picked from commit d10f77e3c91225f86ed9c0f0e6a9adf2e1434674)
(cherry picked from commit 1bb9667137bc80e63dea3e53bb2746be011a2a9a)
 2020-01-24 09:53:26 -06:00
Kihwal Lee
b4e9725955 HDFS-14968. Add ability to log stale datanodes. Contributed by Ahmed Hussein. 
(cherry picked from commit bd03053ea2f32ef982e37fbf2ffd679cb7dda797)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java

(cherry picked from commit 484270832064c84122348e663aaf5927ed411ebb)
 2020-01-22 09:28:01 -06:00
Ayush Saxena
12b258115b HDFS-14578. AvailableSpaceBlockPlacementPolicy always prefers local node. Contributed by Ayush Saxena. 2020-01-11 10:34:37 +05:30
Chen Liang
5490061885 HDFS-15099. [SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException for ObserverNode. Contributed by Chen Liang. 
(cherry picked from commit 26a969ec734dbdbf1d544f486dfa33f15c291789)
 2020-01-10 17:19:05 -08:00
Kihwal Lee
82fa8d88ef HDFS-15095. Fix TestDecommissioningStatus. Contributed by Ahmed Hussein. 
(cherry picked from commit 5fb901ac4017b4f13b089ecd920e864cd53ad3a6)
 2020-01-10 12:47:49 -06:00
Surendra Singh Lilhore
09d3707875 HDFS-14957. INodeReference Space Consumed was not same in QuotaUsage and ContentSummary. Contributed by hemanthboyina. 2020-01-09 12:18:58 +05:30
Rakesh Radhakrishnan
b1eb47865c HDFS-15080. Fix the issue in reading persistent memory cached data with an offset. Contributed by Feilong He. 
(cherry picked from commit 7030722e5d9f376245a9ab0a6a883538b6c55f82)
 2020-01-08 14:32:45 +05:30
Masatake Iwasaki
c5580af1c4 HDFS-15077. Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout. (#1797) 
(cherry picked from commit aba3f6c3e1fbb150ea7ff0411c41ffd3a2796208)
 2020-01-08 17:06:49 +09:00
Masatake Iwasaki
4c8b01abcb
HDFS-15072. HDFS MiniCluster fails to start when run in directory path with a %. (#1775) 
(cherry picked from commit a43c177f1d4c2b6149a2680dd23d91103eca3be0)
 2020-01-08 12:38:51 +09:00
Rakesh Radhakrishnan
8b748c0308 HDFS-14740. Recover data blocks from persistent memory read cache during datanode restarts. Contributed by Feilong He. 
(cherry picked from commit d79cce20abbbf321f6dcce03f4087544124a7cd2)
 2020-01-02 12:19:38 +05:30
Surendra Singh Lilhore
55510d32bf HDFS-15074. DataNode.DataTransfer thread should catch all the expception and log it. Contributed by hemanthboyina. 2019-12-29 11:25:28 +05:30