Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
19,615 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

7539 Commits

Author SHA1 Message Date
Takanobu Asanuma 7168938223 HDFS-15363. BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroup. Contributed by hemanthboyina. 
(cherry picked from commit 4d22d1c58f)
 2020-05-23 17:35:30 +09:00
Chen Liang 8f53545f9d HDFS-15293. Relax the condition for accepting a fsimage when receiving a checkpoint. Contributed by Chen Liang 2020-05-18 12:08:08 -07:00
Akira Ajisaka de4ac41528
Revert "SPNEGO TLS verification" 
This reverts commit 4b7d6667fd.
 2020-05-13 17:16:15 +09:00
Uma Maheswara Rao G fda93334dc HDFS-15305. Extend ViewFS and provide ViewFileSystemOverloadScheme implementation with scheme configurable. Contributed by Uma Maheswara Rao G. 
(cherry picked from commit 9c8236d04d)
 2020-05-12 21:33:49 -07:00
Virajith Jalaparti bec5d7baa0 HADOOP-15565. Add an inner FS cache to ViewFileSystem, separate from the global cache, to avoid file system leaks. Contributed by Jinglun. 2020-05-12 19:37:40 -07:00
Ayush Saxena d24e1bc388 HDFS-14367. EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threads. Contributed by Guo Lei. 2020-05-12 18:45:52 +05:30
Babneet Singh 42997d14a9 HDFS-15270. Account for *env == NULL in hdfsThreadDestructor (#1951) 
OpenJ9 JVM properly terminates the thread before hdfsThreadDestructor is
invoked. JNIEnv is a mirror of J9VMThread in OpenJ9. After proper thread
termination, accessing JNIEnv in hdfsThreadDestructor (*env)->GetJavaVM,
yields a SIGSEGV since *env is NULL after thread cleanup is performed.

The main purpose of hdfsThreadDestructor is to invoke
DetachCurrentThread, which performs thread cleanup in OpenJ9. Since
OpenJ9 performs thread cleanup before hdfsThreadDestructor is invoked,
hdfsThreadDestructor should account for *env == NULL and skip
DetachCurrentThread.

Signed-off-by: Babneet Singh <sbabneet@ca.ibm.com>
(cherry picked from commit 1996351b0b)
(cherry picked from commit d1e5e393c3)
(cherry picked from commit 44fdee351d)
 2020-05-11 10:46:21 -07:00
Ayush Saxena 86c15b8287 HDFS-15250. Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressException. Contributed by Ctest. 2020-05-10 11:45:13 +05:30
Ayush Saxena 8988f33ac3 HDFS-15339. TestHDFSCLI fails for user names with the dot/dash character. Contributed by Yan Xiaole. 2020-05-08 00:57:41 +05:30
He Xiaoqiao dbb2809026 HDFS-15272. Backport HDFS-12862 to branch-3.1. Contributed by Xiaoqiao He, Wang XL. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-04 18:13:01 -07:00
Inigo Goiri 5612537df0 HDFS-14599. HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmpty. Contributed by He Xiaoqiao. 
(cherry picked from commit 4a212242d9)
 2020-05-04 18:03:57 -07:00
Konstantin V Shvachko 8358d7c175 HDFS-15323. StandbyNode fails transition to active due to insufficient transaction tailing. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ebb878bab9)
 2020-05-04 10:50:14 -07:00
Ayush Saxena 69163f5e5d Erasure Coding: metrics xmitsInProgress become to negative. Contributed by maobaolong and Toshihiko Uchida. 2020-05-03 19:18:04 +05:30
Eric Yang 4b7d6667fd
SPNEGO TLS verification 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit ba66f3b454)
 2020-05-02 21:52:06 +09:00
Gabor Bota ec6d2a8402 Preparing for 3.1.5 development 
Change-Id: Iabc64aba7392e3b6f9e4e18109fcaa2cfc01d1f9
 2020-04-29 11:18:18 +02:00
Akira Ajisaka 74ec3c001d
HDFS-15286. Concat on a same file deleting the file. Contributed by hemanthboyina. 
(cherry picked from commit 5e0eda5d5f)
 2020-04-28 06:30:38 +09:00
Mingliang Liu 75354285e6
HDFS-15297. TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently. Contributed by Ayush Saxena 2020-04-25 15:49:29 -07:00
Dhiraj 68dcdad666
HDFS-15281. Make sure ZKFC uses dfs.namenode.rpc-address to bind to host address (#1964) 
Contributed by Dhiraj Hegde.

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Inigo Goiri <inigoiri@apache.org>
 2020-04-25 13:53:32 -07:00
Wei-Chiu Chuang 6f7a30fd71 HDFS-15295. AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance. Contributed by Jinglun. 
(cherry picked from commit 453771f170)
(cherry picked from commit d6c48f8100)
(cherry picked from commit 9e3b4c9ebd)
 2020-04-24 11:54:29 -07:00
Mingliang Liu 03ff1d3bd8
HDFS-15276. Concat on INodeRefernce fails with illegal state exception. Contributed by hemanthboyina 2020-04-22 14:14:19 -07:00
Inigo Goiri 3938aa422e HDFS-14004. TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk. Contributed by Ayush Saxena. 
(cherry picked from commit efdfe679d6)
(cherry picked from commit 7d64d44515)
 2020-04-21 09:50:31 -07:00
Stephen O'Donnell 0ea8f3a19c HDFS-15283. Cache pool MAXTTL is not persisted and restored on cluster restart. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3481895f8a)
(cherry picked from commit aaad947c74)
(cherry picked from commit 041c93a26c)
 2020-04-16 20:19:26 -07:00
Ayush Saxena 6a4c3fad66 HDFS-14647. NPE during secure namenode startup. Contributed by Fengnan Li. 
(cherry picked from commit 62deab17a3)
(cherry picked from commit 6cd9290401)
 2020-03-25 17:14:24 -07:00
Eric Yang b837431a08 HDFS-14434. Ignore user.name query parameter in secure WebHDFS. 
Contributed by KWON BYUNGCHANG

(cherry picked from commit d78854b928)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java

(cherry picked from commit ba6b3a3848)
 2020-03-25 17:14:18 -07:00
Giovanni Matteo Fumarola 4aa7734fc7 HDFS-14006. Refactor name node to allow different token verification implementations. Contributed by CR Hota. 
(cherry picked from commit 00d5e631b5)
(cherry picked from commit eca7bc7ac4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
 2020-03-25 17:13:58 -07:00
Isa Hekmatizadeh d64f688198 HDFS-15219. DFS Client will stuck when ResponseProcessor.run throw Error (#1902). Contributed by zhengchenyu. 
(cherry picked from commit d9c4f1129c)
 2020-03-25 16:52:42 -07:00
Ayush Saxena 4891f24a94 HDFS-15158. The number of failed volumes mismatch with volumeFailures of Datanode metrics. Contributed by Yang Yun. 
(cherry picked from commit 6191d4b4a0)
(cherry picked from commit 1e3b0df6ab)
 2020-03-25 16:42:09 -07:00
Yiqun Lin f1a19b7a3f HDFS-14986. ReplicaCachingGetSpaceUsed throws ConcurrentModificationException. Contributed by Aiphago. 
(cherry picked from commit 2b452b4e60)
 2020-03-25 16:31:54 -07:00
Wei-Chiu Chuang 61915fbd83 HDFS-15208. Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs. 
Contributed by Wei-Chiu Chuang.

Change-Id: I7abeaf4b65e2a4758356aa69ab8930f7e993077d
(cherry picked from commit 096533c2dc)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/resources/log4j.properties

(cherry picked from commit 1d55e1e229)
 2020-03-18 09:59:12 -07:00
Surendra Singh Lilhore a3224ff799 HDFS-15211. EC: File write hangs during close in case of Exception during updatePipeline. Contributed by Ayush Saxena. 2020-03-15 20:59:00 +05:30
Ayush Saxena 8566c65917 HDFS-15197. [SBN read] Change ObserverRetryOnActiveException log to debug. Contributed by Chen Liang. 2020-03-15 16:31:58 +05:30
S O'Donnell 4e1021975b HDFS-15216. Wrong Use Case of -showprogress in fsck. Contributed by Ravuri Sushma sree. 2020-03-13 15:47:57 +00:00
Surendra Singh Lilhore 64eb5b2709 HDFS-15199. NPE in BlockSender. Contributed by Ayush Saxena. 2020-02-28 18:47:54 +05:30
Ayush Saxena 354e63b366 HDFS-12999. When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice. Contributed by lufei and Fei Hui. 
(cherry picked from commit df622cf4a3)
 2020-02-27 16:37:30 -08:00
Masatake Iwasaki abee7402e4 HDFS-15068. DataNode could meet deadlock if invoke refreshVolumes when register. Contributed by Aiphago. 
Signed-off-by: Masatake Iwasaki <iwasakims@apache.org>
(cherry picked from commit 037ec8cfb1)
 2020-02-27 16:37:23 -08:00
Konstantin V Shvachko 4af188587f HDFS-14731. [FGL] Remove redundant locking on NameNode. Contributed by Konstantin V Shvachko. 
(cherry picked from commit ecbcb058b8)
 2020-02-27 16:02:30 -08:00
Ayush Saxena 7bfa123d58 HDFS-15111. stopStandbyServices() should log which service state it is transitioning from. Contributed by Xieming Li. 
(cherry picked from commit b182dd809c)
 2020-02-27 14:55:50 -08:00
Istvan Fajth b5022b0515 HDFS-14668 Support Fuse with Users from multiple Security Realms (#1739) 
(cherry picked from commit 57aa048516)
(cherry picked from commit e42ac486e7)
 2020-02-27 07:49:45 -08:00
Kihwal Lee b92477c638 HDFS-15147. LazyPersistTestCase wait logic is error-prone. Contributed 
by Ahmed Hussein.

(cherry picked from commit 27cfda708e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/LazyPersistTestCase.java
 2020-02-27 09:45:12 -06:00
Stephen O'Donnell 8aaa8d1b71 HDFS-14861. Reset LowRedundancyBlocks Iterator periodically. Contributed by Stephen O'Donnell. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 900430b990)
(cherry picked from commit 2377649cdb)
 2020-02-25 13:29:15 -08:00
Wei-Chiu Chuang 268e35d3ae HDFS-15174. Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations. Contributed by Lisheng Sun. 
(cherry picked from commit cb4daeaad95470125c59ccb80b988261ae83d534)
(cherry picked from commit f9437b1817)
 2020-02-24 13:37:59 -08:00
Ayush Saxena d318b48e12 HDFS-15166. Remove redundant field fStream in ByteStringLog. Contributed by Xieming Li. 2020-02-24 20:17:52 +05:30
Konstantin V Shvachko cd0525054d HDFS-15185. StartupProgress reports edits segments until the entire startup completes. Contributed by Konstantin V Shvachko. 
(cherry picked from commit 6f84269bcd)
 2020-02-21 10:56:31 -08:00
Masatake Iwasaki 8095e3b149 HDFS-15052. WebHDFS getTrashRoot leads to OOM due to FileSystem object creation. (#1758) 
(cherry picked from commit 2338d25dc7)

 Conflicts:
        hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java

(cherry picked from commit 610805ec72)
 2020-02-21 13:51:26 +09:00
Kihwal Lee c54051def5 HDFS-12459. Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit 3ead525c71)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/WebHDFS.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2020-02-17 16:07:03 -06:00
xiaoxiaopan118 b59fa3e571
HDFS-15173. RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable' (#1849) 
(cherry picked from commit 439d935e1d)
 2020-02-17 15:58:35 +09:00
Ayush Saxena 2190fb1fcf HDFS-15115. Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug. Contributed by wangzhixiang 2020-02-16 10:09:40 +05:30
Ayush Saxena cf5a7186e5 HDFS-15164. Fix TestDelegationTokensWithHA. Contributed by Ayush Saxena. 2020-02-14 22:11:39 +05:30
Surendra Singh Lilhore 6a4297adc4 HDFS-15086. Block scheduled counter never get decremet if the block got deleted before replication. Contributed by hemanthboyina. 2020-02-13 23:04:28 +05:30
Ayush Saxena 83ab853733 HDFS-15161. When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close(). Contributed by Lisheng Sun 2020-02-12 20:30:13 +05:30