Commit Graph

6795 Commits

Author SHA1 Message Date
Akira Ajisaka 89105a623b
HDFS-12156. TestFSImage fails without -Pnative
(cherry picked from commit 319defafc1)
2018-03-13 11:27:46 +09:00
Inigo Goiri 177dd6580d HDFS-13241. RBF: TestRouterSafemode failed if the port 8888 is in use. Contributed by maobaolong.
(cherry picked from commit 91c82c90f0)
2018-03-12 17:30:00 -07:00
Chris Douglas 2210593dad HADOOP-14742. Document multi-URI replication Inode for ViewFS. Contributed by Gera Shegalov
(cherry picked from commit ddb67ca707)
(cherry picked from commit c9364b3bce)
2018-03-12 13:55:10 -07:00
Inigo Goiri 29cf29a853 HDFS-11399. Many tests fails in Windows due to injecting disk failures. Contributed by Yiqun Lin.
(cherry picked from commit ac627f561f)
2018-03-12 10:05:43 -07:00
Yiqun Lin 2e607f2103 HDFS-13240. RBF: Update some inaccurate document descriptions. Contributed by Yiqun Lin.
(cherry picked from commit 4743d4a2c7)
2018-03-10 11:32:12 +08:00
Inigo Goiri 465c3f0e24 HDFS-13232. RBF: ConnectionPool should return first usable connection. Contributed by Ekanth S.
(cherry picked from commit 8133cd5305)
2018-03-09 18:27:17 -08:00
Inigo Goiri c453fcb23f HDFS-13212. RBF: Fix router location cache issue. Contributed by Weiwei Wu.
(cherry picked from commit afe1a3ccd5)
2018-03-09 17:20:29 -08:00
Hanisha Koneru bd2ebf031c HDFS-13244. Add stack, conf, metrics links to utilities dropdown in NN webUI. Contributed by Bharat Viswanadham.
(cherry picked from commit 4eeff62f69)
2018-03-09 15:49:09 -08:00
Lokesh Jain 80aabd72c2
HDFS-13190. Document WebHDFS support for snapshot diff
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
Signed-off-by: Xiaoyu Yao <xyao@apache.org>
(cherry picked from commit 7b0dc31020)
2018-03-09 15:07:34 -08:00
Hanisha Koneru dec53bb9e1 HDFS-11394. Support for getting erasure coding policy through WebHDFS#FileStatus.
(cherry picked from commit 9a082fbe6e)
2018-03-09 13:39:49 -08:00
Yiqun Lin f879504fe1 HDFS-13233. RBF: MountTableResolver doesn't return the correct mount point of the given path. Contributed by wangzhiyuan. 2018-03-09 15:48:36 +08:00
Inigo Goiri 68cff8c235 HDFS-13232. RBF: ConnectionManager's cleanup task will compare each pool's own active conns with its total conns. Contributed by Chao Sun.
(cherry picked from commit 0c2b969e01)
2018-03-08 09:33:24 -08:00
Hanisha Koneru 0b1e9665df HDFS-13225. StripeReader#checkMissingBlocks() 's IOException info is incomplete. Contributed by lufei.
(cherry picked from commit 19ae4429f9)
2018-03-07 14:22:01 -08:00
Yiqun Lin 253ef33f80 HDFS-13214. RBF: Complete document of Router configuration. Contributed by Yiqun Lin. 2018-03-07 16:07:05 +08:00
Xiaoyu Yao b74df5523c HDFS-13109. Support fully qualified hdfs path in EZ commands. Contributed by Hanisha Koneru.
(cherry picked from commit edf9445708)
2018-03-06 18:30:10 -08:00
Xiao Chen 283d48a4f7 HDFS-13170. Port webhdfs unmaskedpermission parameter to HTTPFS. Contributed by Stephen O'Donnell.
(cherry picked from commit 92c15e19272a8bb59f8704c44d4cd713d3a58bcb)
2018-03-06 09:58:09 -08:00
Xiao Chen d28db463c2 HDFS-13040. Kerberized inotify client fails despite kinit properly. Contributed by Istvan Fajth, Wei-Chiu Chuang, Xiao Chen.
(cherry picked from commit c75105f07b)
2018-03-02 16:47:36 -08:00
Xiaoyu Yao 908ca63c03 HDFS-13081. Datanode#checkSecureConfig should allow SASL and privileged HTTP. Contributed by Ajay Kumar.
(cherry picked from commit f20e10b2dd)
2018-02-28 09:48:12 -08:00
Xiaoyu Yao ce356dcbf6 HDFS-13114. CryptoAdmin#ReencryptZoneCommand should resolve Namespace info from path. Contributed by Hanisha Koneru.
(cherry picked from commit 90fbe5c6834a6685e65542fa69e396fad2678927)
2018-02-28 08:44:14 -08:00
Inigo Goiri 0eff245d3d HDFS-13199. RBF: Fix the hdfs router page missing label icon issue. Contributed by maobaolong.
(cherry picked from commit d86f301d46)
2018-02-27 17:41:42 -08:00
Yiqun Lin 09017aee90 HDFS-13184. RBF: Improve the unit test TestRouterRPCClientRetries. Contributed by Yiqun Lin.
(cherry picked from commit 1e85a995d1)
2018-02-27 10:52:49 +08:00
Chao Sun 26395aef67 HDFS-13145. SBN crash when transition to ANN with in-progress edit tailing enabled. Contributed by Chao Sun.
(cherry picked from commit ae290a4bb4)
2018-02-26 16:18:51 -08:00
Wei Yan 1fb87df87e HDFS-13187. RBF: Fix Routers information shown in the web UI. Contributed by Inigo Goiri and Wei Yan.
(cherry picked from commit 6ce9f79cc9)
2018-02-26 15:29:11 -08:00
Kihwal Lee 1087b9af8c HDFS-12070. Failed block recovery leaves files open indefinitely and at risk for data loss. Contributed by Kihwal Lee.
(cherry picked from commit 451265a83d)
2018-02-26 10:30:50 -06:00
Xiaoyu Yao 21d4b5fd24 HDFS-13052. WebHDFS: Add support for snasphot diff. Contributed by Lokesh Jain.
(cherry picked from commit 1e84e46f16)
2018-02-23 19:41:28 -08:00
Xiao Chen 1fbd419957 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException.
(cherry picked from commit 51088d3233)
2018-02-23 14:00:25 -08:00
Xiaoyu Yao ec1b445c62 HDFS-13136. Avoid taking FSN lock while doing group member lookup for FSD permission check. Contributed by Xiaoyu Yao.
(cherry picked from commit e978e1d9064fc794ea83217d0af2315d4167211e)
2018-02-22 20:18:02 -08:00
Steve Loughran c2bbe22c5a HADOOP-10571. Use Log.*(Object, Throwable) overload to log exceptions.
Contributed by Andras Bokor.
2018-02-21 11:11:28 +00:00
Anu Engineer c0d8103ba2 HDFS-13175. Add more information for checking argument in DiskBalancerVolume.
Contributed by  Lei (Eddy) Xu.

(cherry picked from commit 121e1e1280)
2018-02-20 19:59:31 -08:00
Yiqun Lin 7601489995 HDFS-13119. RBF: Manage unavailable clusters. Contributed by Yiqun Lin.
(cherry picked from commit 8896d20b91)
2018-02-20 13:06:26 -08:00
Kihwal Lee 7b83fca453 HDFS-13112. Token expiration edits may cause log corruption or deadlock. Contributed by Daryn Sharp.
(cherry picked from commit 47473952e5)
2018-02-15 15:35:27 -06:00
Kihwal Lee 11ff6327af xattr api cleanup
(cherry picked from commit da59acd8ca)
2018-02-15 11:14:42 -06:00
Brahma Reddy Battula 055d5e0b85 HDFS-8693. Addendum patch to execute the command using UGI. Contributed by Brahma Reddy Battula.
(cherry picked from commit 35c17351ca)
2018-02-12 22:17:35 +05:30
Arpit Agarwal 41cf2b8e7c HDFS-10453. ReplicationMonitor thread could stuck for long time due to the race between replication and delete of same file in a large cluster.. Contributed by He Xiaoqiao. 2018-02-12 07:16:48 -08:00
Yiqun Lin 38febc0ae2 HDFS-13099. RBF: Use the ZooKeeper as the default State Store. Contributed by Yiqun Lin.
(cherry picked from commit 543f3abbee)
2018-02-09 13:59:38 +08:00
Xiaoyu Yao 49ab09f00b HDFS-13120. Snapshot diff could be corrupted after concat. Contributed by Xiaoyu Yao.
(cherry picked from commit 8faf0b50d4)
2018-02-08 10:18:35 -08:00
Manoj Govindassamy bef769e436 HDFS-12985. NameNode crashes during restart after an OpenForWrite file present in the Snapshot got deleted.
(cherry picked from commit 73ff09b79a)

(cherry picked from commit 821729905e)
2018-02-07 19:49:27 -08:00
Yongjun Zhang 5105b60614 HDFS-13115. In getNumUnderConstructionBlocks(), ignore the inodeIds for which the inodes have been deleted. Contributed by Yongjun Zhang.
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/LeaseManager.java

(cherry picked from commit 60d82ac61f)
2018-02-07 16:49:26 -08:00
Anu Engineer 2d6ab20007 HDFS-12990. Change default NameNode RPC port back to 8020. Contributed by Xiao Chen.
(cherry picked from commit 4304fcd5bd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/ViewFs.md
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestGetConf.java

(cherry picked from commit 9264f10bb3)
2018-02-07 11:16:53 -08:00
Brahma Reddy Battula c6349c3204 HDFS-12935. Get ambiguous result for DFSAdmin command in HA mode when only one namenode is up. Contributed by Jianfei Jiang.
(cherry picked from commit 01bd6ab18f)
2018-02-07 23:27:24 +05:30
Xiao Chen 49de91446f HDFS-12933. Improve logging when DFSStripedOutputStream failed to write some blocks. Contributed by chencan.
(cherry picked from commit e5c2fdddd6)
2018-02-06 22:57:15 -08:00
Wei-Chiu Chuang 29a3b64ec0 HDFS-11187. Optimize disk access for last partial chunk checksum of Finalized replica. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 2021f4bdce)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/ReplicaBuilder.java
2018-02-06 22:57:15 -08:00
Lei Xu 7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Yongjun Zhang 35247cca87 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang.
(cherry picked from commit 4e9a59ce16)

(cherry picked from commit 0524fac21e)
2018-02-04 22:53:09 -08:00
Yiqun Lin 712e9381d2 HDFS-13068. RBF: Add router admin option to manage safe mode. Contributed by Yiqun Lin. 2018-02-02 17:34:12 +08:00
Akira Ajisaka 01661d4f90
HDFS-13048. LowRedundancyReplicatedBlocks metric can be negative
(cherry picked from commit 4aef8bd2ef)
2018-02-02 14:35:11 +09:00
Hanisha Koneru 992d7b5739 HDFS-13062. Provide support for JN to use separate journal disk per namespace. Contributed by Bharat Viswanadham.
(cherry picked from commit dd50f53997)
2018-02-01 12:32:47 -08:00
Xiaoyu Yao 784b454122 HDFS-12997. Move logging to slf4j in BlockPoolSliceStorage and Storage. Contributed by Ajay Kumar.
(cherry picked from commit b3ae11d597)
2018-02-01 12:21:29 -08:00
Inigo Goiri 7491aaeb8f HDFS-13043. RBF: Expose the state of the Routers in the federation. Contributed by Inigo Goiri.
(cherry picked from commit 6ca7204ceb)
2018-02-01 10:37:56 -08:00
Xiao Chen 4010262ae0 HDFS-12897. getErasureCodingPolicy should handle .snapshot dir better. Contributed by LiXin Ge.
(cherry picked from commit ae2177d296)
2018-01-31 22:50:52 -08:00
Xiaoyu Yao e260a0af24 HDFS-13060. Adding a BlacklistBasedTrustedChannelResolver for TrustedChannelResolver. Contributed by Ajay Kumar.
(cherry picked from commit b3ac6f7ce3f5e014af7ad6bf882431a97716e020)
2018-01-31 22:40:13 -08:00
Hanisha Koneru b71d775f57 HDFS-13092. Reduce verbosity for ThrottledAsyncChecker#schedule. Contributed by Mukul Kumar Singh.
(cherry picked from commit 3ce2190b58)
2018-01-31 11:07:39 -08:00
Xiaoyu Yao 02790eb633 HDFS-13061. SaslDataTransferClient#checkTrustAndSend should not trust a partially trusted channel. Contributed by Ajay Kumar.
(cherry picked from commit f4bcae10d306ef2b0083e1de17253ec732b5b5c1)
2018-01-31 10:47:46 -08:00
Inigo Goiri 961dcb693d HDFS-13044. RBF: Add a safe mode for the Router. Contributed by Inigo Goiri. 2018-01-31 09:08:47 -08:00
Inigo Goiri 0ce4851ee2 HDFS-13083. RBF: Fix doc error setting up client. Contributed by tartarus.
(cherry picked from commit 5206b2c7ca)
2018-01-30 18:27:51 -08:00
Weiwei Yang 7f62d2a473 HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen. 2018-01-31 07:35:54 +08:00
Kihwal Lee 673200ac1e HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah
(cherry picked from commit fde95d463c)
2018-01-29 17:25:30 -06:00
Xiao Chen 75be15cf02 HDFS-12974. Exception message is not printed when creating an encryption zone fails with AuthorizationException. Contributed by fang zhenyi.
(cherry picked from commit b63dcd583f)
2018-01-28 22:24:00 -08:00
Xiao Chen fd5081bd24 HDFS-13065. TestErasureCodingMultipleRacks#testSkewedRack3 is failing. Contributed by Gabor Bota.
(cherry picked from commit 6bc2f7f4b4)
2018-01-28 22:12:25 -08:00
Arpit Agarwal 480b69c03c HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:23 -08:00
Yiqun Lin 52ead8b90a HDFS-13042. RBF: Heartbeat Router State. Contributed by Inigo Goiri.
(cherry picked from commit 7721fff744)
2018-01-25 15:53:09 +08:00
Jianfei Jiang 2a36b4b1ed
HDFS-13063. Fix the incorrect spelling in HDFSHighAvailabilityWithQJM.md
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit eb2dd08688)
2018-01-25 15:31:45 +09:00
Inigo Goiri e4ad5f7f37 HDFS-12772. RBF: Federation Router State State Store internal API. Contributed by Inigo Goiri.
(cherry picked from commit 95743c672e)
2018-01-23 19:16:27 -08:00
Lei Xu d33050b8cd HDFS-13039. StripedBlockReader.createBlockReader leaks socket on IOException. (Lei (Eddy) Xu)
(cherry picked from commit 2ed9d61aad)
2018-01-19 16:42:22 -08:00
Inigo Goiri 6c10e0e12b HDFS-13028. RBF: Fix spurious TestRouterRpc#testProxyGetStats. Contributed by Inigo Goiri.
(cherry picked from commit 130f8bce00)
2018-01-19 09:48:45 -08:00
Vinayakumar B 2443692434 HDFS-9049. Make Datanode Netty reverse proxy port to be configurable. Contributed by Vinayakumar B.
(cherry picked from commit 09efdfe9e1)
2018-01-17 13:01:52 -06:00
Lei Xu ce394be809 HDFS-13004. TestLeaseRecoveryStriped.testLeaseRecovery is failing when safeLength is 0MB or larger than the test file. (Zsolt Venczel via lei)
(cherry picked from commit 3bd9ea63df)
2018-01-17 12:59:53 -06:00
Inigo Goiri 49e285ded5 HDFS-12919. RBF: Support erasure coding methods in RouterRpcServer. Contributed by Inigo Goiri. 2018-01-17 12:57:14 -06:00
Yiqun Lin cb07172e21 HDFS-11848. Enhance dfsadmin listOpenFiles command to list files under a given path. Contributed by Yiqun Lin. 2018-01-17 12:50:06 -06:00
Manoj Govindassamy 710555e150 HDFS-11847. Enhance dfsadmin listOpenFiles command to list files blocking datanode decommissioning. 2018-01-17 12:44:09 -06:00
Brahma Reddy Battula d03115d028 HDFS-8693. refreshNamenodes does not support adding a new standby to a running DN. Contributed by Ajith S.
(cherry picked from commit 880b9d24ff)
2018-01-16 16:22:10 +05:30
Wei-Chiu Chuang b830040766 HDFS-11915. Sync rbw dir on the first hsync() to avoid file lost on power failure. Contributed by Vinayakumar B.
(cherry picked from commit d6c31a3e6b)
2018-01-12 10:42:58 -08:00
Inigo Goiri 5ed1eff48b HDFS-12802. RBF: Control MountTableResolver cache size. Contrubuted by Inigo Goiri.
(cherry picked from commit d9006d8a4e)
2018-01-09 18:55:46 -08:00
Lei Xu 92f63d04a1 HDFS-12994. TestReconstructStripedFile.testNNSendsErasureCodingTasks fails due to socket timeout. (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 47563d86fe)
2018-01-09 12:26:30 -08:00
Wei Yan cbecf554c9 HDFS-12945. Switch to ClientProtocol instead of NamenodeProtocols in NamenodeWebHdfsMethods. Contributed by Wei Yan.
(cherry picked from commit 2ee0d64ace)
2018-01-08 14:56:22 -08:00
Inigo Goiri 53a7bd186d HDFS-12988. RBF: Mount table entries not properly updated in the local cache. Contributed by Inigo Goiri.
(cherry picked from commit 83b513ac6d)
2018-01-05 09:12:43 -08:00
Lei Xu 47773d68b2 HDFS-12860. StripedBlockUtil#getRangesInternalBlocks throws exception for the block group size larger than 2GB. (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 85067eaa8e352afeed9c094da3815100873c84e4)
2018-01-04 10:33:34 -08:00
Sean Mackrory 91b59b7c28 HDFS-12913. TestDNFencingWithReplication sometimes fails because of NameNode in standby state. Contributed by Zsolt Venczel.
(cherry picked from commit 45a47198b4)
2018-01-04 08:47:02 -07:00
Lei Xu 8f9603cb9d HDFS-12915. Fix findbugs warning in INodeFile$HeaderFormat.getBlockLayoutRedundancy. (Contributed by Chris Douglas)
(cherry picked from commit 6e3e1b8cde)
2017-12-29 12:23:33 -08:00
Xiao Chen b7701495f4 HDFS-9023. When NN is not able to identify DN for replication, reason behind it can be logged.
(cherry picked from commit 5bf7e594d7)
2017-12-28 11:55:14 -08:00
Lei Xu 2fbfaac6c0 HDFS-12938. TestErasureCodigCLI testAll failing consistently. (Contributed by Ajay Kumar)
(cherry picked from commit b318bed01a)
2017-12-21 10:30:40 -08:00
Arpit Agarwal d85576aefd HDFS-12845. JournalNode Test failures. Contributed by Bharat Viswanadham. 2017-12-19 14:12:25 -08:00
Yiqun Lin f7b3950f27 HDFS-12937. RBF: Add more unit tests for router admin commands. Contributed by Yiqun Lin.
(cherry picked from commit e040c97b77)
2017-12-19 15:34:01 +08:00
Junping Du 53b9d81f46 Add 2.8.3 release jdiff files.
(cherry picked from commit c89f99aade)
2017-12-18 22:40:04 -08:00
Yiqun Lin a56f4c2ca7 HDFS-12930. Remove the extra space in HdfsImageViewer.md. Contributed by Rahul Pathak.
(cherry picked from commit 25a36b7452)
2017-12-19 11:25:01 +08:00
Erik Krogen 7c648d3e72 HDFS-12818. Support multiple storages in DataNodeCluster / SimulatedFSDataset. Contributed by Erik Krogen.
(cherry picked from commit 94576b17fb)
2017-12-18 11:41:06 -08:00
Lei Xu 0191a862a3 HDFS-12927. Update erasure coding doc to address unsupported APIs. (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 949be14b08)
2017-12-15 10:29:08 -08:00
Lei Xu 2821edf6d0 HDFS-12819. Setting/Unsetting EC policy shows warning if the directory is not empty. (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 1c15b1751c)
2017-12-15 10:06:00 -08:00
Yiqun Lin 61e16bff63 HDFS-12895. RBF: Add ACL support for mount table. Contributed by Yiqun Lin.
(cherry picked from commit ee028bfdf1)
2017-12-15 14:12:00 +08:00
Xiao Chen cb2c5e1707 HDFS-12910. Secure Datanode Starter should log the port when it fails to bind. Contributed by Stephen O'Donnell and Nanda kumar.
(cherry picked from commit e1cb278cd0)
2017-12-14 19:23:18 -08:00
Kihwal Lee 52c36d6d4d HDFS-12907. Allow read-only access to reserved raw for non-superusers. Contributed by Rushabh S Shah.
(cherry picked from commit f5a72424c0)
2017-12-14 15:00:27 -06:00
Jason Lowe c661f30a78 HDFS-12881. Output streams closed with IOUtils suppressing write errors. Contributed by Ajay Kumar
(cherry picked from commit 80db744ee5)
2017-12-14 09:53:56 -06:00
Yiqun Lin 925f3e62db HDFS-12883. RBF: Document Router and State Store metrics. Contributed by Yiqun Lin.
(cherry picked from commit 91c96bdf8e)
2017-12-14 16:02:37 +08:00
Andrew Wang 58225e4030 Update CHANGES, RELEASENOTES, jdiff for 3.0.0 release.
(cherry picked from commit d447152d49)
2017-12-13 13:05:45 -08:00
Wei-Chiu Chuang 8062a7428d HDFS-12891. Do not invalidate blocks if toInvalidate is empty. Contributed by Zsolt Venczel.
(cherry picked from commit 55fc2d6485)
2017-12-11 16:43:42 -08:00
Inigo Goiri 6a43fbe724 HDFS-12875. RBF: Complete logic for -readonly option of dfsrouteradmin add command. Contributed by Inigo Goiri.
(cherry picked from commit 5cd1056ad7)
2017-12-11 15:16:19 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) ba874d6787 HADOOP-15059. Undoing the switch of Credentials to PB format as default - done via HADOOP-12563 for supporting 2.x to 3.x upgrades.
(cherry picked from commit f19638333b)
2017-12-08 08:03:23 -08:00
Xiao Chen 1d1401ad38 HADOOP-15056. Fix TestUnbuffer#testUnbufferException failure. Contributed by Jack Bearden.
(cherry picked from commit 19e0894209)
2017-12-07 21:06:25 -08:00
John Zhuge 9942952dd8 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge.
(cherry picked from commit bf6a660232)
2017-12-07 21:06:25 -08:00
Lei Xu f3143d225a HDFS-12840. Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog. Contributed by Lei (Eddy) Xu.
(cherry picked from commit 67662e2ac9)
2017-12-07 11:20:19 -08:00