Commit Graph

304 Commits

Author SHA1 Message Date
Xiao Chen 80f7165374 HDFS-13164. File not closed if streamer fail with DSQuotaExceededException.
(cherry picked from commit 51088d3233)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
2018-02-23 14:00:33 -08:00
Gabor Bota 696e8e0299
HADOOP-15236. Fix typo in RequestHedgingProxyProvider and RequestHedgingRMFailoverProxyProvider
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
(cherry picked from commit c36b4aa31c)
(cherry picked from commit b8942cb3bf)
2018-02-23 13:58:08 +09:00
Yongjun Zhang 6dfea05049 HDFS-13100. Handle IllegalArgumentException when GETSERVERDEFAULTS is not implemented in webhdfs. Contributed by Aaron T. Myers and Yongjun Zhang.
(cherry picked from commit 4e9a59ce16)
2018-02-02 23:33:10 -08:00
Weiwei Yang 0c2c4c20cb HDFS-12528. Add an option to not disable short-circuit reads on failures. Contributed by Xiao Chen.
(cherry picked from commit 2e7331ca26)
2018-01-30 21:40:07 -08:00
Kihwal Lee eda786ea12 HDFS-12574. Add CryptoInputStream to WebHdfsFileSystem read call. Contributed by Rushabh S Shah. 2018-01-29 17:52:04 -06:00
Arpit Agarwal 7be5a46a41 HDFS-13054. Handling PathIsNotEmptyDirectoryException in DFSClient delete call. Contributed by Nanda kumar. 2018-01-26 11:45:36 -08:00
Xiao Chen c9d48352c2 HADOOP-15012. Add readahead, dropbehind, and unbuffer to StreamCapabilities. Contributed by John Zhuge. 2017-12-07 21:20:33 -08:00
Xiao Chen 274bbb193f HDFS-12396. Webhdfs file system should get delegation token from kms provider. Contributed by Rushabh S Shah. 2017-12-04 10:40:22 -08:00
Kihwal Lee 6fd9d11173 HDFS-11754. Make FsServerDefaults cache configurable. Contributed by Mikhail Erofeev.
(cherry picked from commit 53509f295b)
2017-11-29 15:15:41 -06:00
Kihwal Lee 45f59bde60 HDFS-12754. Lease renewal can hit a deadlock. Contributed by Kuhu Shukla. 2017-11-27 16:52:18 -06:00
Tsz-Wo Nicholas Sze ed879d4a72 HDFS-12813. RequestHedgingProxyProvider can hide Exception thrown from the Namenode for proxy size of 1. Contributed by Mukul Kumar Singh 2017-11-20 17:11:18 -08:00
Arpit Agarwal f671c22e3e HDFS-12705. WebHdfsFileSystem exceptions should retain the caused by exception. Contributed by Hanisha Koneru. 2017-11-13 13:56:13 -08:00
John Zhuge d946307faa HDFS-12318. Fix IOException condition for openInfo in DFSInputStream. Contributed by legend.
(cherry picked from commit 3123895db0)
2017-11-09 08:58:03 -08:00
Weiwei Yang 0f20434e18 HDFS-12744. More logs when short-circuit read is failed and disabled. Contributed by Weiwei Yang.
(cherry picked from commit 56b88b0670)
2017-11-01 17:01:23 +08:00
Arun Suresh 71a55bebfb Preparing for 2.10.0 development 2017-10-31 12:51:14 -07:00
Arpit Agarwal 8f28af4fac HDFS-9914. Fix configurable WebhDFS connect/read timeout. Contributed by Xiaoyu Yao. 2017-10-27 07:45:11 -07:00
Wei-Chiu Chuang a596c6f5eb HDFS-12485. expunge may fail to remove trash from encryption zone. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 8dbc8909c9)
2017-10-16 12:59:41 -07:00
Daryn Sharp 7f3ed5f61e [HDFS-12386] Add fsserver defaults call to WebhdfsFileSystem. (Rushabh Shah via daryn) 2017-09-26 15:21:37 -05:00
Ming Ma 7dd662eafd HDFS-12473. Change hosts JSON file format. 2017-09-20 09:08:41 -07:00
Brahma Reddy Battula eddc581860 HDFS-11799. Introduce a config to allow setting up write pipeline with fewer nodes than replication factor. Contributed by Brahma Reddy Battula
(cherry picked from commit fda1221c55)
2017-09-19 11:31:22 +05:30
Kihwal Lee 5a83ffa396 HDFS-12299. Race Between update pipeline and DN Re-Registration. Contributed by Brahma Reddy Battula. 2017-08-25 14:53:53 -05:00
John Zhuge c54c3500ed HDFS-11738. Hedged pread takes more time when block moved from initial locations. Contributed by Vinayakumar B.
(cherry picked from commit b6bfb2fcb2)
2017-08-23 19:44:52 +05:30
John Zhuge ed4d6aa2c1 HDFS-11303. Hedged read might hang infinitely if read data from all DN failed . Contributed by Chen Zhang, Wei-chiu Chuang, and John Zhuge.
(cherry picked from commit 8b242f09a6)
2017-08-11 21:55:23 -07:00
Haohui Mai 21d76f710e HDFS-10326. Disable setting tcp socket send/receive buffers for write pipelines. Contributed by Daryn Sharp. 2017-08-08 14:58:44 -07:00
Manoj Govindassamy 70c511598f HDFS-11402. HDFS Snapshots should capture point-in-time copies of OPEN files. 2017-08-01 17:11:27 -07:00
Lei Xu f09d20cffb HADOOP-14397. Pull up the builder pattern to FileSystem and add AbstractContractCreateTest for it. (Lei (Eddy) Xu)
(cherry picked from commit 667ee003bf47e44beb3fdff8d06a7264a13dd22c)
2017-07-31 20:07:13 -07:00
Mingliang Liu b803641523 HDFS-12089. Fix ambiguous NN retry log message in WebHDFS. Contributed by Eric Badger
(cherry picked from commit 6436768baf)
2017-07-05 11:12:15 -07:00
Wei-Chiu Chuang 6917c19167 HDFS-11881. NameNode consumes a lot of memory for snapshot diff report generation. Contributed by Manoj Govindassamy.
(cherry picked from commit 16c8dbde57)
2017-06-29 06:40:24 -07:00
Ravi Prakash ef491f4637 HDFS-11993. Add log info when connect to datanode socket address failed. Contributed by chencan
(cherry picked from commit a9d3412b4c)
2017-06-26 13:25:32 -07:00
Andrew Wang 1ec1aebd35 HDFS-10480. Add an admin command to list currently open files. Contributed by Manoj Govindassamy. 2017-06-19 09:52:07 -07:00
Lei Xu 33afa1fdca HADOOP-14395. Provide Builder pattern for DistributedFileSystem.append. Contributed by Lei (Eddy) Xu.
(cherry picked from commit 6460df21a0)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
2017-06-16 17:32:29 -07:00
Xiao Chen 2afe9722af HADOOP-14524. Make CryptoCodec Closeable so it can be cleaned up proactively. 2017-06-16 09:49:41 -07:00
Lei Xu abac844c90 HADOOP-14394. Provide Builder pattern for DistributedFileSystem.create. (lei)
(cherry picked from commit 5fbec46525d6d49837d934556b59ba77bd2301a8)

    Conflicts:
    hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/NameNodeConnector.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDistributedFileSystem.java
    hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestErasureCodingPolicies.java
2017-06-15 10:59:11 -07:00
Vinayakumar B 9a26120809 HDFS-11708. Positional read will fail if replicas moved to different DNs after stream is opened. Contributed by Vinayakumar B. 2017-06-07 10:57:47 +05:30
Brahma Reddy Battula ac2af66e77 HDFS-11905. Fix license header inconsistency in hdfs. Contributed by Yeliang Cang. 2017-06-02 22:22:43 +08:00
Yiqun Lin 870b5e39d2 HDFS-11359. DFSAdmin report command supports displaying maintenance state datanodes. Contributed by Yiqun Lin. 2017-06-02 12:53:31 +08:00
Andrew Wang d0a4e7e95d HDFS-11383. Intern strings in BlockLocation and ExtendedBlock. Contributed by Misha Dmitriev.
(cherry picked from commit 7101477e47)
2017-06-01 15:20:22 -07:00
Kihwal Lee 212a566088 HDFS-11856. Ability to re-add Upgrading nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-31 12:45:33 -05:00
Xiao Chen 54971c4195 HDFS-11421. Make WebHDFS' ACLs RegEx configurable. Contributed by Harsh J. 2017-05-25 21:01:39 -07:00
Sunil G ec346d6377 YARN-6584. Correct license headers in hadoop-common, hdfs, yarn and mapreduce. Contributed by Yeliang Cang. 2017-05-23 07:57:36 +05:30
Andrew Wang 18b89e902b HDFS-11644. Support for querying outputstream capabilities. Contributed by Manoj Govindassamy. 2017-05-13 21:41:11 -07:00
Kihwal Lee 0d9322aa5b HDFS-11702. Remove indefinite caching of key provider uri in DFSClient. Contributed by Rushabh S Shah.
(cherry picked from commit cef2815cf4)
2017-05-08 08:29:03 -05:00
Lei Xu 1f30a8a105 HDFS-11687. Add new public encryption APIs required by Hive. (lei)
Change-Id: I4a23a00de63ad18022312ceb1f306a87d032d07c
(cherry picked from commit 25f5d9ad5e)
2017-05-04 12:07:29 -07:00
Akira Ajisaka cc66153e12
HDFS-11417. Add datanode admin command to get the storage info. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 93fa48fcf2)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocol/ClientDatanodeProtocol.java
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/ClientDatanodeProtocolServerSideTranslatorPB.java
2017-04-26 18:08:12 +09:00
Yongjun Zhang 31d7d47070 HDFS-11689. New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code. Contributed by Yongjun Zhang.
(cherry picked from commit 5078df7be3)
2017-04-21 14:01:47 -07:00
Arpit Agarwal 5c092097b6 HDFS-11603. Improve slow mirror/disk warnings in BlockReceiver. 2017-04-18 16:20:43 -07:00
Chris Nauroth d5e2bd4096 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 00ed21a6fedb45a7c8992b8d45adaa83f14af34c)
2017-04-11 21:18:47 -07:00
Hanisha Koneru f16bb79957 HDFS-11551. Support Timeout when checking single disk. Contributed by Hanisha Koneru. 2017-04-11 16:54:28 -07:00
Xiaoyu Yao 0391c92022 HDFS-11608. HDFS write crashed with block size greater than 2 GB. Contributed by Xiaobing Zhou.
(cherry picked from commit 0eacd4c13b)
2017-04-06 16:15:56 -07:00
Andrew Wang ed8f738be5 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-06 14:38:22 -07:00