Commit Graph

221 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 396de0fd8f Preparing for 2.8.3 development 2017-06-07 15:38:20 -07:00
Vinayakumar B 4a391c72d4 HDFS-11708. Positional read will fail if replicas moved to different DNs after stream is opened. Contributed by Vinayakumar B. 2017-06-07 10:59:04 +05:30
Kihwal Lee fe6de5e594 HDFS-11856. Ability to re-add upgrading nodes to pipeline for future pipeline updates. Contributed by Vinayakumar B. 2017-05-31 13:08:51 -05:00
Lei Xu 5b8564f986 HDFS-11687. Add new public encryption APIs required by Hive. (lei)
Change-Id: I4a23a00de63ad18022312ceb1f306a87d032d07c
(cherry picked from commit 25f5d9ad5e)
(cherry picked from commit 1f30a8a105)
2017-05-09 12:11:48 -07:00
Kihwal Lee 9d8d3a84bd HDFS-11702. Remove indefinite caching of key provider uri in DFSClient. Contributed by Rushabh S Shah.
(cherry picked from commit cef2815cf4)
2017-05-08 08:31:18 -05:00
Ming Ma c4c5533216 HDFS-9005. Provide configuration support for upgrade domain. 2017-05-02 06:53:32 -07:00
Yongjun Zhang c88e5e1836 HDFS-11689. New exception thrown by DFSClient%isHDFSEncryptionEnabled broke hacky hive code. Contributed by Yongjun Zhang.
(cherry picked from commit 5078df7be3)
2017-04-21 14:12:04 -07:00
Chris Nauroth c4bf504395 HDFS-11163. Mover should move the file blocks to default storage once policy is unset. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 00ed21a6fedb45a7c8992b8d45adaa83f14af34c)
(cherry picked from commit d5e2bd4096)
2017-04-11 21:55:28 -07:00
Junping Du dfac95a3a2 Preparing for 2.8.1 development 2017-04-11 11:21:49 -07:00
Xiaoyu Yao 6367bc1f09 HDFS-11608. HDFS write crashed with block size greater than 2 GB. Contributed by Xiaobing Zhou.
(cherry picked from commit 0eacd4c13b)
(cherry picked from commit 0391c92022)
2017-04-06 16:26:57 -07:00
Andrew Wang b1dfdea5b4 HADOOP-14104. Client should always ask namenode for kms provider path. Contributed by Rushabh S Shah. 2017-04-06 14:39:56 -07:00
Andrew Wang d12a0a25b4 HDFS-11538. Move ClientProtocol HA proxies into hadoop-hdfs-client. Contributed by Huafeng Wang.
(cherry picked from commit e154893d3b)
2017-04-05 16:36:07 -07:00
Andrew Wang 7b8f87eb78 HDFS-9705. Refine the behaviour of getFileChecksum when length = 0. Contributed by SammiChen and Kai Zheng.
(cherry picked from commit 82b4a9c3d0)
2017-03-28 21:41:10 -07:00
Zhe Zhang 8ca9915d0f HADOOP-9631. ViewFs should use underlying FileSystem's server side defaults. Contributed by Lohit Vijayarenu and Erik Krogen.
(cherry picked from commit 59d69257a8)
(cherry picked from commit f98f4bb6ca)
2017-03-22 22:32:41 -07:00
Wei-Chiu Chuang 295e8685dd HDFS-8498. Blocks can be committed with wrong size. Contributed by Jing Zhao.
(cherry picked from commit f3cdf29af4)
2017-02-25 21:15:07 -08:00
Kihwal Lee 22f8b66137 HDFS-11379. DFSInputStream may infinite loop requesting block locations. Contributed by Daryn Sharp.
(cherry picked from commit 33c62d2d19)
2017-02-10 12:56:44 -06:00
Andrew Wang b7b8671c0e HDFS-11312. Fix incompatible tag number change for nonDfsUsed in DatanodeInfoProto. Contributed by Sean Mackrory.
(cherry picked from commit d51f8ba808)
(cherry picked from commit b089e122b1)
2017-01-11 15:13:43 -08:00
Junping Du 7706a63fb4 Preparing for 2.8.1 development after cutting off 2.8.0 branch. 2017-01-10 18:03:51 -08:00
Haohui Mai e690cd5011 HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao. 2017-01-04 21:15:01 -08:00
Brahma Reddy Battula 7392a5a72c Revert "HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao."
This reverts commit 0ff9efcf37.
2017-01-03 19:30:45 +05:30
Haohui Mai 0ff9efcf37 HDFS-11280. Allow WebHDFS to reuse HTTP connections to NN. Contributed by Zheng Shao. 2016-12-31 01:07:41 -08:00
Andrew Wang 158292fd7e HDFS-10684. WebHDFS DataNode calls fail without parameter createparent. Contributed by John Zhuge.
(cherry picked from commit fbdbbd57cd)
(cherry picked from commit 6b795c34d0)
2016-12-13 18:01:40 -08:00
Vinayakumar B 69db9eecfa HDFS-10570. Remove classpath conflicts of netty-all jar in hadoop-hdfs-client (Contributed by Vinayakumar B)
(cherry picked from commit c6fb9f0d0b)
2016-12-12 12:12:30 +05:30
Andrew Wang e8b0ef0618 HDFS-8630. WebHDFS : Support get/set/unset StoragePolicy. Contributed by Surendra Singh Lilhore. 2016-12-09 14:56:49 -08:00
Vinayakumar B 7dc22dbe0c HDFS-9057. allow/disallow snapshots via webhdfs (Contributed by Brahma Reddy Battula)
(cherry picked from commit 6d2332ae37)
(cherry picked from commit 4b21b47f56)
2016-12-07 15:58:29 -08:00
Andrew Wang 1a7b2ab9bb Revert "HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang"
This reverts commit 3b80424d4f.
2016-12-05 23:09:30 -08:00
Mingliang Liu 3b80424d4f HDFS-11156. Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API. Contributed by Weiwei Yang
(cherry picked from commit c7ff34f8dc)
(cherry picked from commit be969e5918)
2016-12-02 11:38:36 -08:00
Kihwal Lee 15f08e9298 HDFS-8870. Lease is leaked on write failure. Contributed by Kuhu Shukla.
(cherry picked from commit 4fcea8a0c8)
2016-11-15 14:57:04 -06:00
Masatake Iwasaki 6d9b6fac86 HADOOP-13017. Implementations of InputStream.read(buffer, offset, bytes) to exit 0 if bytes==0. Contributed by Steve Loughran.
(cherry picked from commit 0bdd263d82)
2016-10-27 15:55:35 +09:00
Kihwal Lee 49adab66b8 HDFS-11050. Change log level to 'warn' when ssl initialization fails and defaults to DEFAULT_TIMEOUT_CONN_CONFIGURATOR. Contributed by Kuhu Shukla.
(cherry picked from commit ce6bbfb23c)
2016-10-26 08:11:51 -05:00
Xiao Chen d22a6a8b82 HDFS-11009. Add a tool to reconstruct block meta file from CLI.
(cherry picked from commit bc4a32aea50e86819730312e89315c0244ce64bf)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetUtil.java

(cherry picked from commit ad7d3c4db8)
2016-10-18 22:42:09 -07:00
Akira Ajisaka 65e6a1181c HDFS-11012. Unnecessary INFO logging on DFSClients for InvalidToken. Contributed by Harsh J.
(cherry picked from commit 5ad037df25)
(cherry picked from commit 35832901e8)
2016-10-15 22:19:14 +09:00
Andrew Wang f13b84b4eb HDFS-10883. 's behavior is not consistent in DFS after enabling EZ. Contributed by Yuanbo Liu.
(cherry picked from commit 0007360c33)
2016-10-14 11:41:42 -07:00
Benoy Antony ca20ffea46 HDFS-10735 Distcp using webhdfs on secure HA clusters fails with StandbyException
(cherry picked from commit 701c27a776)
2016-10-14 10:32:37 -07:00
Chris Nauroth e36b665f90 HADOOP-13150. Avoid use of toString() in output of HDFS ACL shell commands. Contributed by Chris Nauroth.
(cherry picked from commit 1d330fbaf6)
(cherry picked from commit a28ffd0fde)
2016-10-06 13:24:54 -07:00
Xiao Chen bf043e63f9 HDFS-10918. Add a tool to get FileEncryptionInfo from CLI. Contributed by Xiao Chen.
(cherry picked from commit 853d65a157)
2016-10-03 16:02:11 -07:00
Xiaoyu Yao b0b29de8ab HDFS-10690. Optimize insertion/removal of replica in ShortCircuitCache. Contributed by Fenghua Hu.
(cherry picked from commit 607705c488)
2016-10-03 11:05:58 -07:00
Kihwal Lee f5b193b482 HDFS-10850. getEZForPath should NOT throw FNF. Contributed by Andrew Wang.
(cherry picked from commit 0670149c88)
2016-09-30 09:16:13 -05:00
Andrew Wang c4f81f199f HDFS-10914. Move remnants of oah.hdfs.client to hadoop-hdfs-client.
(cherry picked from commit 92e5e91598)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java

(cherry picked from commit 9ddaa29301ea8ae2e3cb0f1e031343d5b91f92ee)
2016-09-28 16:03:02 -07:00
Wei-Chiu Chuang f0f3b6a66a HDFS-10609. Uncaught InvalidEncryptionKeyException during pipeline recovery may abort downstream applications. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 3ae652f821)
(cherry picked from commit bde787db23)
2016-09-26 14:46:02 -07:00
Xiao Chen 900dc70a01 HDFS-10489. Deprecate dfs.encryption.key.provider.uri for HDFS encryption zones. Contributed by Xiao Chen.
(cherry picked from commit ea839bd48e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/HdfsConfiguration.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/resources/hdfs-default.xml

(cherry picked from commit 20585ade1d9d7fbd8b9df8b1e3db5a74fd4441a6)
2016-09-19 16:38:00 -07:00
Andrew Wang 912631a226 HDFS-10831. Add log when URLConnectionFactory.openConnection failed. Contributed by yunjiong zhao.
(cherry picked from commit b07c266dca)
(cherry picked from commit 72ea641468)
2016-09-08 18:30:36 -07:00
Arpit Agarwal 162ee0f0a4 HDFS-9038. DFS reserved space is erroneously counted towards non-DFS used. (Brahma Reddy Battula) 2016-09-06 13:52:05 -07:00
Wei-Chiu Chuang bd729af8a8 HDFS-8986. Add option to -du to calculate directory space usage excluding snapshots. Contributed by Xiao Chen. 2016-08-24 08:53:43 -07:00
Xiao Chen b89d79ca1d HDFS-10549. Correctly revoke file leases when closing files. Contributed by Yiqun Lin.
(cherry picked from commit 85aacaadb5a3f8c78b191867c0bde09b3c4b3c3c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSStripedOutputStream.java

(cherry picked from commit a20b943cf9)
2016-08-17 16:00:33 -07:00
Kihwal Lee eb9af6ca34 HDFS-10717. Fix findbugs warnings of hadoop-hdfs-client in branch-2. Contributed by Akira Ajisaka.
(cherry picked from commit 6ead6f3bb8)
2016-08-04 12:36:51 -05:00
Kihwal Lee 58df27b873 HDFS-10662. Optimize UTF8 string/byte conversions. Contributed by Daryn Sharp.
(cherry picked from commit 6ae39199da)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirWriteFileOp.java

(cherry picked from commit 77b61d1f4e)
2016-08-04 09:45:03 -05:00
Kihwal Lee 804db5b277 HDFS-10707. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon.
(cherry picked from commit c0166b0743)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/webhdfs/WebHdfsHandler.java
2016-08-04 09:35:16 -05:00
Jing Zhao 54be36a6d6 HDFS-9894. Add unsetStoragePolicy API to FileContext/AbstractFileSystem and derivatives. Contributed by Xiaobing Zhou.
(cherry picked from commit 7149cdb3c2d9dd390cd8668883cbe5db94090e0a)
2016-07-25 09:51:59 -07:00
Arpit Agarwal bbf711c030 HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-07-25 09:51:59 -07:00