Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,673 Commits 326 Branches 371 Tags
Commit Graph

6912 Commits

Author SHA1 Message Date
Arpit Agarwal
41cf2b8e7c HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:16:48 -08:00
Yiqun Lin
38febc0ae2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin. 
(cherry picked from commit 543f3abbee79d7ec70353f0cdda6397ee001324e)
 2018-02-09 13:59:38 +08:00
Xiaoyu Yao
49ab09f00b HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao. 
(cherry picked from commit 8faf0b50d435039f69ea35f592856ca04d378809)
 2018-02-08 10:18:35 -08:00
Manoj Govindassamy
bef769e436 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 
(cherry picked from commit 73ff09b79a5cf9932edc21c58f3a730f7379086b)

(cherry picked from commit 821729905ed8439ee5f15372faaf6b40aff7f155)
 2018-02-07 19:49:27 -08:00
Yongjun Zhang
5105b60614 HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java

(cherry picked from commit 60d82ac61f2afa2a61c8eeabf20e45b5a89311d0)
 2018-02-07 16:49:26 -08:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Brahma Reddy Battula
c6349c3204 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 
(cherry picked from commit 01bd6ab18fa48f4c7cac1497905b52e547962599)
 2018-02-07 23:27:24 +05:30
Xiao Chen
49de91446f HDFS-12933. Improve logging when DFSStripedOutputStream failed to write some blocks. Contributed by chencan. 
(cherry picked from commit e5c2fdddd69ceca21d641de56904648acedc74bf)
 2018-02-06 22:57:15 -08:00
Wei-Chiu Chuang
29a3b64ec0 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2021f4bdce3b27c46edaad198f0007a26a8a1391)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java
 2018-02-06 22:57:15 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Yongjun Zhang
35247cca87 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang. 
(cherry picked from commit 4e9a59ce16e81b4bd6fae443a997ef24d588a6e8)

(cherry picked from commit 0524fac21ef8a8e440a2fe759936e6fb6a63cbf1)
 2018-02-04 22:53:09 -08:00
Yiqun Lin
712e9381d2 HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 17:34:12 +08:00
Akira Ajisaka
01661d4f90
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative 
(cherry picked from commit 4aef8bd2efd68bf96c077ddda1538dcd5691b437)
 2018-02-02 14:35:11 +09:00
Hanisha Koneru
992d7b5739 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham. 
(cherry picked from commit dd50f53997239bf9078481cf46592ca3e41520b5)
 2018-02-01 12:32:47 -08:00
Xiaoyu Yao
784b454122 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar. 
(cherry picked from commit b3ae11d59790bb08b81848e9f944db7d3afbbd8a)
 2018-02-01 12:21:29 -08:00
Inigo Goiri
7491aaeb8f HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 6ca7204cebf4b7060696a07b1777723a6dfa7d9d)
 2018-02-01 10:37:56 -08:00
Xiao Chen
4010262ae0 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge. 
(cherry picked from commit ae2177d296a322d13708b85aaa8a971b8dcce128)
 2018-01-31 22:50:52 -08:00
Xiaoyu Yao
e260a0af24 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar. 
(cherry picked from commit b3ac6f7ce3f5e014af7ad6bf882431a97716e020)
 2018-01-31 22:40:13 -08:00
Hanisha Koneru
b71d775f57 HDFS-13092. Reduce verbosity for ThrottledAsyncChecker#schedule. Contributed by Mukul Kumar Singh. 
(cherry picked from commit 3ce2190b581526ad2d49e8c3a47be1547037310c)
 2018-01-31 11:07:39 -08:00
Xiaoyu Yao
02790eb633 HDFS-13061. SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel. Contributed by Ajay Kumar. 
(cherry picked from commit f4bcae10d306ef2b0083e1de17253ec732b5b5c1)
 2018-01-31 10:47:46 -08:00
Inigo Goiri
961dcb693d HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 2018-01-31 09:08:47 -08:00
Inigo Goiri
0ce4851ee2 HDFS-13083. RBF: Fix doc error setting up client. Contributed by tartarus. 
(cherry picked from commit 5206b2c7ca479dd53f614d75bab594043a9866e1)
 2018-01-30 18:27:51 -08:00
Weiwei Yang
7f62d2a473 HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 2018-01-31 07:35:54 +08:00
Kihwal Lee
673200ac1e HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah 
(cherry picked from commit fde95d463c3123b315b3d07cb5b7b7dc19f7cb73)
 2018-01-29 17:25:30 -06:00
Xiao Chen
75be15cf02 HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi. 
(cherry picked from commit b63dcd583f0b98e785831004f41bd7c7de8b3c18)
 2018-01-28 22:24:00 -08:00
Xiao Chen
fd5081bd24 HDFS-13065. TestErasureCodingMultipleRacks#testSkewedRack3 is failing. Contributed by Gabor Bota. 
(cherry picked from commit 6bc2f7f4b4b8d4c36e92764d4c975c17f9fdd63b)
 2018-01-28 22:12:25 -08:00
Arpit Agarwal
480b69c03c HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:23 -08:00
Yiqun Lin
52ead8b90a HDFS-13042. RBF: Heartbeat Router State. Contributed by Inigo Goiri. 
(cherry picked from commit 7721fff74494eb7fbbbba7f8bb4b4692d880d301)
 2018-01-25 15:53:09 +08:00
Jianfei Jiang
2a36b4b1ed
HDFS-13063. Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit eb2dd0868864da994612f1dd1452ea38d31f5e85)
 2018-01-25 15:31:45 +09:00
Inigo Goiri
e4ad5f7f37 HDFS-12772. RBF: Federation Router State State Store internal API. Contributed by Inigo Goiri. 
(cherry picked from commit 95743c672e6b42b227a22dfa7cc16edc7bdb58bb)
 2018-01-23 19:16:27 -08:00
Lei Xu
d33050b8cd HDFS-13039. StripedBlockReader.createBlockReader leaks socket on IOException. (Lei (Eddy) Xu) 
(cherry picked from commit 2ed9d61aadaffbf56ae0ed124520edab97646b49)
 2018-01-19 16:42:22 -08:00
Inigo Goiri
6c10e0e12b HDFS-13028. RBF: Fix spurious TestRouterRpc#testProxyGetStats. Contributed by Inigo Goiri. 
(cherry picked from commit 130f8bce00e8cab59c3aa4d668e8bddd84544713)
 2018-01-19 09:48:45 -08:00
Vinayakumar B
2443692434 HDFS-9049. Make Datanode Netty reverse proxy port to be configurable. Contributed by Vinayakumar B. 
(cherry picked from commit 09efdfe9e13c9695867ce4034aa6ec970c2032f1)
 2018-01-17 13:01:52 -06:00
Lei Xu
ce394be809 HDFS-13004. TestLeaseRecoveryStriped.testLeaseRecovery is failing when safeLength is 0MB or larger than the test file. (Zsolt Venczel via lei) 
(cherry picked from commit 3bd9ea63df769345a9d02a404cfb61323a4cd7e3)
 2018-01-17 12:59:53 -06:00
Inigo Goiri
49e285ded5 HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-01-17 12:57:14 -06:00
Yiqun Lin
cb07172e21 HDFS-11848. Enhance dfsadmin listOpenFiles command to list files under a given path. Contributed by Yiqun Lin. 2018-01-17 12:50:06 -06:00
Manoj Govindassamy
710555e150 HDFS-11847. Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning. 2018-01-17 12:44:09 -06:00
Brahma Reddy Battula
d03115d028 HDFS-8693. refreshNamenodes does not support adding a new standby to a running DN. Contributed by Ajith S. 
(cherry picked from commit 880b9d24ff7b5f350ec99bac9b0862009460b486)
 2018-01-16 16:22:10 +05:30
Wei-Chiu Chuang
b830040766 HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. Contributed by Vinayakumar B. 
(cherry picked from commit d6c31a3e6b60c4b8af9ae4661f16614805654e59)
 2018-01-12 10:42:58 -08:00
Inigo Goiri
5ed1eff48b HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by Inigo Goiri. 
(cherry picked from commit d9006d8a4e34eae78dfa1cf3be50eeb81c2fc11f)
 2018-01-09 18:55:46 -08:00
Lei Xu
92f63d04a1 HDFS-12994. TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 47563d86fe6ba1a2de934c9ed740d0aafbf72d4e)
 2018-01-09 12:26:30 -08:00
Wei Yan
cbecf554c9 HDFS-12945. Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods. Contributed by Wei Yan. 
(cherry picked from commit 2ee0d64aceed876f57f09eb9efe1872b6de98d2e)
 2018-01-08 14:56:22 -08:00
Inigo Goiri
53a7bd186d HDFS-12988. RBF: Mount table entries not properly updated in the local cache. Contributed by Inigo Goiri. 
(cherry picked from commit 83b513ac6d5448f3771b0b95f91e7aa7961ae2cc)
 2018-01-05 09:12:43 -08:00
Lei Xu
47773d68b2 HDFS-12860. StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 85067eaa8e352afeed9c094da3815100873c84e4)
 2018-01-04 10:33:34 -08:00
Sean Mackrory
91b59b7c28 HDFS-12913. TestDNFencingWithReplication sometimes fails because of NameNode in standby state. Contributed by Zsolt Venczel. 
(cherry picked from commit 45a47198b43a96817679d5bf9e570d405555e3eb)
 2018-01-04 08:47:02 -07:00
Lei Xu
8f9603cb9d HDFS-12915. Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy. (Contributed by Chris Douglas) 
(cherry picked from commit 6e3e1b8cde737e4c03b0f5279cab0239e7069a72)
 2017-12-29 12:23:33 -08:00
Xiao Chen
b7701495f4 HDFS-9023. When NN is not able to identify DN for replication, reason behind it can be logged. 
(cherry picked from commit 5bf7e594d7d54e5295fe4240c3d60c08d4755ab7)
 2017-12-28 11:55:14 -08:00
Lei Xu
2fbfaac6c0 HDFS-12938. TestErasureCodigCLI testAll failing consistently. (Contributed by Ajay Kumar) 
(cherry picked from commit b318bed01affa150d70661f263efff9a5c9422f6)
 2017-12-21 10:30:40 -08:00
Arpit Agarwal
d85576aefd HDFS-12845. JournalNode Test failures. Contributed by Bharat Viswanadham. 2017-12-19 14:12:25 -08:00
Yiqun Lin
f7b3950f27 HDFS-12937. RBF: Add more unit tests for router admin commands. Contributed by Yiqun Lin. 
(cherry picked from commit e040c97b7743469f363eeae52c8abcf4fe7c65d5)
 2017-12-19 15:34:01 +08:00