Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,866 Commits 326 Branches 371 Tags
Commit Graph

6783 Commits

Author SHA1 Message Date
Kihwal Lee
1087b9af8c HDFS-12070. Failed block recovery leaves files open indefinitely and at risk for data loss. Contributed by Kihwal Lee. 
(cherry picked from commit 451265a83d8798624ae2a144bc58fa41db826704)
 2018-02-26 10:30:50 -06:00
Xiaoyu Yao
21d4b5fd24 HDFS-13052. WebHDFS: Add support for snasphot diff. Contributed by Lokesh Jain. 
(cherry picked from commit 1e84e46f1621fe694f806bfc41d3b2a06c9500b6)
 2018-02-23 19:41:28 -08:00
Xiao Chen
1fbd419957 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException. 
(cherry picked from commit 51088d323359587dca7831f74c9d065c2fccc60d)
 2018-02-23 14:00:25 -08:00
Xiaoyu Yao
ec1b445c62 HDFS-13136. Avoid taking FSN lock while doing group member lookup for FSD permission check. Contributed by Xiaoyu Yao. 
(cherry picked from commit e978e1d9064fc794ea83217d0af2315d4167211e)
 2018-02-22 20:18:02 -08:00
Steve Loughran
c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions. 
Contributed by Andras Bokor.
 2018-02-21 11:11:28 +00:00
Anu Engineer
c0d8103ba2 HDFS-13175. Add more information for checking argument in DiskBalancerVolume. 
Contributed by  Lei (Eddy) Xu.

(cherry picked from commit 121e1e1280c7b019f6d2cc3ba9eae1ead0dd8408)
 2018-02-20 19:59:31 -08:00
Yiqun Lin
7601489995 HDFS-13119. RBF: Manage unavailable clusters. Contributed by Yiqun Lin. 
(cherry picked from commit 8896d20b91520053a6bbfb680adb345cd24f4142)
 2018-02-20 13:06:26 -08:00
Kihwal Lee
7b83fca453 HDFS-13112. Token expiration edits may cause log corruption or deadlock. Contributed by Daryn Sharp. 
(cherry picked from commit 47473952e56b0380147d42f4110ad03c2276c961)
 2018-02-15 15:35:27 -06:00
Kihwal Lee
11ff6327af xattr api cleanup 
(cherry picked from commit da59acd8ca9ab5b49b988ffca64e8cce91c5f741)
 2018-02-15 11:14:42 -06:00
Brahma Reddy Battula
055d5e0b85 HDFS-8693. Addendum patch to execute the command using UGI. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 35c17351cab645dcc72e0d2ae1608507aa787ffb)
 2018-02-12 22:17:35 +05:30
Arpit Agarwal
41cf2b8e7c HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:16:48 -08:00
Yiqun Lin
38febc0ae2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin. 
(cherry picked from commit 543f3abbee79d7ec70353f0cdda6397ee001324e)
 2018-02-09 13:59:38 +08:00
Xiaoyu Yao
49ab09f00b HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao. 
(cherry picked from commit 8faf0b50d435039f69ea35f592856ca04d378809)
 2018-02-08 10:18:35 -08:00
Manoj Govindassamy
bef769e436 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted. 
(cherry picked from commit 73ff09b79a5cf9932edc21c58f3a730f7379086b)

(cherry picked from commit 821729905ed8439ee5f15372faaf6b40aff7f155)
 2018-02-07 19:49:27 -08:00
Yongjun Zhang
5105b60614 HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java

(cherry picked from commit 60d82ac61f2afa2a61c8eeabf20e45b5a89311d0)
 2018-02-07 16:49:26 -08:00
Anu Engineer
2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen. 
(cherry picked from commit 4304fcd5bdf9fb7aa9181e866eea383f89bf171f)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb35dbe30c75c648bf759e8aeb715834a)
 2018-02-07 11:16:53 -08:00
Brahma Reddy Battula
c6349c3204 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang. 
(cherry picked from commit 01bd6ab18fa48f4c7cac1497905b52e547962599)
 2018-02-07 23:27:24 +05:30
Wei-Chiu Chuang
29a3b64ec0 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang. 
(cherry picked from commit 2021f4bdce3b27c46edaad198f0007a26a8a1391)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java
 2018-02-06 22:57:15 -08:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Yiqun Lin
712e9381d2 HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 17:34:12 +08:00
Akira Ajisaka
01661d4f90
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative 
(cherry picked from commit 4aef8bd2efd68bf96c077ddda1538dcd5691b437)
 2018-02-02 14:35:11 +09:00
Hanisha Koneru
992d7b5739 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham. 
(cherry picked from commit dd50f53997239bf9078481cf46592ca3e41520b5)
 2018-02-01 12:32:47 -08:00
Xiaoyu Yao
784b454122 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar. 
(cherry picked from commit b3ae11d59790bb08b81848e9f944db7d3afbbd8a)
 2018-02-01 12:21:29 -08:00
Inigo Goiri
7491aaeb8f HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri. 
(cherry picked from commit 6ca7204cebf4b7060696a07b1777723a6dfa7d9d)
 2018-02-01 10:37:56 -08:00
Xiao Chen
4010262ae0 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge. 
(cherry picked from commit ae2177d296a322d13708b85aaa8a971b8dcce128)
 2018-01-31 22:50:52 -08:00
Xiaoyu Yao
e260a0af24 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar. 
(cherry picked from commit b3ac6f7ce3f5e014af7ad6bf882431a97716e020)
 2018-01-31 22:40:13 -08:00
Hanisha Koneru
b71d775f57 HDFS-13092. Reduce verbosity for ThrottledAsyncChecker#schedule. Contributed by Mukul Kumar Singh. 
(cherry picked from commit 3ce2190b581526ad2d49e8c3a47be1547037310c)
 2018-01-31 11:07:39 -08:00
Xiaoyu Yao
02790eb633 HDFS-13061. SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel. Contributed by Ajay Kumar. 
(cherry picked from commit f4bcae10d306ef2b0083e1de17253ec732b5b5c1)
 2018-01-31 10:47:46 -08:00
Inigo Goiri
961dcb693d HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 2018-01-31 09:08:47 -08:00
Inigo Goiri
0ce4851ee2 HDFS-13083. RBF: Fix doc error setting up client. Contributed by tartarus. 
(cherry picked from commit 5206b2c7ca479dd53f614d75bab594043a9866e1)
 2018-01-30 18:27:51 -08:00
Weiwei Yang
7f62d2a473 HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 2018-01-31 07:35:54 +08:00
Kihwal Lee
673200ac1e HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah 
(cherry picked from commit fde95d463c3123b315b3d07cb5b7b7dc19f7cb73)
 2018-01-29 17:25:30 -06:00
Xiao Chen
75be15cf02 HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi. 
(cherry picked from commit b63dcd583f0b98e785831004f41bd7c7de8b3c18)
 2018-01-28 22:24:00 -08:00
Xiao Chen
fd5081bd24 HDFS-13065. TestErasureCodingMultipleRacks#testSkewedRack3 is failing. Contributed by Gabor Bota. 
(cherry picked from commit 6bc2f7f4b4b8d4c36e92764d4c975c17f9fdd63b)
 2018-01-28 22:12:25 -08:00
Arpit Agarwal
480b69c03c HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:23 -08:00
Yiqun Lin
52ead8b90a HDFS-13042. RBF: Heartbeat Router State. Contributed by Inigo Goiri. 
(cherry picked from commit 7721fff74494eb7fbbbba7f8bb4b4692d880d301)
 2018-01-25 15:53:09 +08:00
Jianfei Jiang
2a36b4b1ed
HDFS-13063. Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md 
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit eb2dd0868864da994612f1dd1452ea38d31f5e85)
 2018-01-25 15:31:45 +09:00
Inigo Goiri
e4ad5f7f37 HDFS-12772. RBF: Federation Router State State Store internal API. Contributed by Inigo Goiri. 
(cherry picked from commit 95743c672e6b42b227a22dfa7cc16edc7bdb58bb)
 2018-01-23 19:16:27 -08:00
Lei Xu
d33050b8cd HDFS-13039. StripedBlockReader.createBlockReader leaks socket on IOException. (Lei (Eddy) Xu) 
(cherry picked from commit 2ed9d61aadaffbf56ae0ed124520edab97646b49)
 2018-01-19 16:42:22 -08:00
Inigo Goiri
6c10e0e12b HDFS-13028. RBF: Fix spurious TestRouterRpc#testProxyGetStats. Contributed by Inigo Goiri. 
(cherry picked from commit 130f8bce00e8cab59c3aa4d668e8bddd84544713)
 2018-01-19 09:48:45 -08:00
Vinayakumar B
2443692434 HDFS-9049. Make Datanode Netty reverse proxy port to be configurable. Contributed by Vinayakumar B. 
(cherry picked from commit 09efdfe9e13c9695867ce4034aa6ec970c2032f1)
 2018-01-17 13:01:52 -06:00
Lei Xu
ce394be809 HDFS-13004. TestLeaseRecoveryStriped.testLeaseRecovery is failing when safeLength is 0MB or larger than the test file. (Zsolt Venczel via lei) 
(cherry picked from commit 3bd9ea63df769345a9d02a404cfb61323a4cd7e3)
 2018-01-17 12:59:53 -06:00
Inigo Goiri
49e285ded5 HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-01-17 12:57:14 -06:00
Yiqun Lin
cb07172e21 HDFS-11848. Enhance dfsadmin listOpenFiles command to list files under a given path. Contributed by Yiqun Lin. 2018-01-17 12:50:06 -06:00
Manoj Govindassamy
710555e150 HDFS-11847. Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning. 2018-01-17 12:44:09 -06:00
Brahma Reddy Battula
d03115d028 HDFS-8693. refreshNamenodes does not support adding a new standby to a running DN. Contributed by Ajith S. 
(cherry picked from commit 880b9d24ff7b5f350ec99bac9b0862009460b486)
 2018-01-16 16:22:10 +05:30
Wei-Chiu Chuang
b830040766 HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. Contributed by Vinayakumar B. 
(cherry picked from commit d6c31a3e6b60c4b8af9ae4661f16614805654e59)
 2018-01-12 10:42:58 -08:00
Inigo Goiri
5ed1eff48b HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by Inigo Goiri. 
(cherry picked from commit d9006d8a4e34eae78dfa1cf3be50eeb81c2fc11f)
 2018-01-09 18:55:46 -08:00
Lei Xu
92f63d04a1 HDFS-12994. TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout. (Contributed by Lei (Eddy) Xu) 
(cherry picked from commit 47563d86fe6ba1a2de934c9ed740d0aafbf72d4e)
 2018-01-09 12:26:30 -08:00
Wei Yan
cbecf554c9 HDFS-12945. Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods. Contributed by Wei Yan. 
(cherry picked from commit 2ee0d64aceed876f57f09eb9efe1872b6de98d2e)
 2018-01-08 14:56:22 -08:00