Commit Graph

4115 Commits

Author SHA1 Message Date
Arpit Agarwal b0ea50bb29 HDFS-1477. Support reconfiguring dfs.heartbeat.interval and dfs.namenode.heartbeat.recheck-interval without NN restart. (Contributed by Xiaobing Zhou) 2016-03-10 19:05:22 -08:00
Colin Patrick Mccabe 710811652c HDFS-9933. ReverseXML should be capitalized in oiv usage message (cmccabe)
(cherry picked from commit 79961ecea8)
2016-03-10 13:45:02 -08:00
Colin Patrick Mccabe c82d1e1187 HDFS-9934. ReverseXML oiv processor should bail out if the XML file's layoutVersion doesn't match oiv's (cmccabe)
(cherry picked from commit bd49354c6d)
2016-03-10 13:41:32 -08:00
Colin Patrick Mccabe b8b580e6b8 HDFS-9927. Document the new OIV ReverseXML processor (Wei-Chiu Chuang via cmccabe)
(cherry picked from commit 500875dfcc)
2016-03-10 13:39:20 -08:00
Allen Wittenauer 9f4b3eaac0 Revert "HADOOP-12366. expose calculated paths (aw)"
This reverts commit 45bb467918.
2016-03-09 09:59:19 -08:00
Chris Nauroth 38a031ce68 HADOOP-12905. Clean up CHANGES.txt RAT exclusions from pom.xml files. Contributed by Chris Nauroth.
(cherry picked from commit 89b16d27e2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/pom.xml
	hadoop-mapreduce-project/pom.xml
2016-03-08 21:22:57 -08:00
Arpit Agarwal 9d67f98e05 HDFS-9882. Add heartbeatsTotal in Datanode metrics. (Contributed by Hua Liu) 2016-03-07 21:10:34 -08:00
Akira Ajisaka fe0009a2bd HDFS-9812. Streamer threads leak if failure happens when closing DFSOutputStream. Contributed by Lin Yiqun.
(cherry picked from commit 352d299cf8)
2016-03-08 10:55:59 +09:00
Arpit Agarwal 3e987446e3 HDFS-9906. Remove spammy log spew when a datanode is restarted. (Contributed by Brahma Reddy Battula) 2016-03-07 12:20:07 -08:00
Masatake Iwasaki 135ceb6c7b HDFS-9865. TestBlockReplacement fails intermittently in trunk (Lin Yiqun via iwasakims)
(cherry picked from commit d718fc1ee5)
2016-03-08 02:11:07 +09:00
Harsh J ed421cb128 HDFS-9521. TransferFsImage.receiveFile should account and log separate times for image download and fsync to disk. Contributed by Wellington Chevreuil
(cherry picked from commit fd1c09be3e)
2016-03-07 17:23:11 +05:30
Chris Nauroth 592f161e13 HDFS-9239. DataNode Lifeline Protocol: an alternative protocol for reporting DataNode liveness. Contributed by Chris Nauroth.
(cherry picked from commit 2759689d7d)
2016-03-04 15:30:28 -08:00
Yongjun Zhang 1d6c9486a4 HDFS-9889. Update balancer/mover document about HDFS-6133 feature. Contributed by Yongjun Zhang.
(cherry picked from commit 8e08861a14)
2016-03-04 13:28:10 -08:00
Masatake Iwasaki 722182fdfb HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims)
(cherry picked from commit cbd31328a6)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
2016-03-04 14:12:44 +09:00
Andrew Wang 670944810c HADOOP-11792. Remove all of the CHANGES.txt files. 2016-03-03 21:02:58 -08:00
Masatake Iwasaki 8095c612a3 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims)
(cherry picked from commit 33a412e8a4)
2016-03-03 18:57:41 +09:00
Masatake Iwasaki 69e024f8fa HADOOP-12793. Write a new group mapping service guide (Wei-Chiu Chuang via iwasakims)
(cherry picked from commit f70e317ccc)
2016-03-03 17:14:53 +09:00
Arpit Agarwal 7b4a1001ea HDFS-9534. Add CLI command to clear storage policy from a path. (Contributed by Xiaobing Zhou) 2016-03-02 18:35:38 -08:00
Colin Patrick Mccabe 5998b20d63 HDFS-9835. OIV: add ReverseXML processor which reconstructs an fsimage from an XML file (cmccabe)
(cherry picked from commit 700b0e4019)
2016-03-02 18:18:45 -08:00
Zhe Zhang b97ac02c4b HDFS-9886. Configuration properties for hedged read is broken. Contributed by Akira AJISAKA. 2016-03-02 10:27:09 -08:00
Xiaoyu Yao 6bffe1a908 HDFS-9887. WebHdfs socket timeouts should be configurable. Contributed by Austin Donnelly and Chris Douglas.
(cherry picked from commit 5abf051249)
2016-03-02 09:24:46 -08:00
Akira Ajisaka 463ec6fa8c HDFS-9851. NameNode throws NPE when setPermission is called on a path that does not exist. Contributed by Brahma Reddy Battula.
(cherry picked from commit 27e0681f28)
2016-03-02 14:43:49 +09:00
Akira Ajisaka e840141726 HDFS-9766. TestDataNodeMetrics#testDataNodeTimeSpend fails intermittently. Contributed by Xiao Chen.
(cherry picked from commit 81fbc40f084eab85750cc7d4cc967d43d70d8333)
2016-03-02 14:14:27 +09:00
Andrew Wang 81b7b9de27 HDFS-9881. DistributedFileSystem#getTrashRoot returns incorrect path for encryption zones.
(cherry picked from commit 4abb2fa687)
2016-03-01 16:35:43 -08:00
Kihwal Lee bcbff207d6 HDFS-9880. TestDatanodeRegistration fails occasionally. Contributed by Kihwal Lee.
(cherry picked from commit e76b13c415)
2016-03-01 17:01:37 -06:00
Kihwal Lee a823c30a9b HDFS-8791. block ID-based DN storage layout can be very slow for datanode on ext4. Contributed by Chris Trezzo.
(cherry picked from commit b8dbffbef87c0ee0bee5e17649586fc9f0e1f55b)
2016-03-01 15:03:32 -06:00
Chris Nauroth 6659d13f12 HDFS-9870. Remove unused imports from DFSUtil. Contributed by Brahma Reddy Battula.
(cherry picked from commit 2137e8feeb)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2016-03-01 11:16:21 -08:00
Jing Zhao 3cb7ae11a8 HDFS-7964. Add support for async edit logging. Contributed by Daryn Sharp.
(cherry picked from commit 2151716832)
2016-02-29 15:45:23 -08:00
Akira Ajisaka 531a081b01 HDFS-9864. Correct reference for RENEWDELEGATIONTOKEN and CANCELDELEGATIONTOKEN in webhdfs doc. Contributed by Brahma Reddy Battula.
(cherry picked from commit 056f9013122dc3f0effc62d620261c312dd7f8ed)
2016-02-29 18:42:35 +09:00
Tsz-Wo Nicholas Sze 7b79567cf9 HDFS-9710. DN can be configured to send block receipt IBRs in batches. 2016-02-26 15:34:00 -08:00
Xiaoyu Yao 1513c0179e HDFS-9831. Document webhdfs retry configuration keys introduced by HDFS-5219/HDFS-5122. Contributed by Xiaobing Zhou. 2016-02-26 14:24:03 -08:00
Kihwal Lee 03afcdd39d HDFS-9855. Modify TestAuditLoggerWithCommands to workaround the absence of HDFS-8332. Contributed by Kuhu Shukla. 2016-02-26 10:22:27 -06:00
Karthik Kambatla 6722e17f6d HDFS-9858. RollingFileSystemSink can throw an NPE on non-secure clusters. (Daniel Templeton via kasha)
(cherry picked from commit c2460dad64)
2016-02-25 16:36:07 -08:00
Chris Nauroth 4892f6a0eb HDFS-9843. Document distcp options required for copying between encrypted locations. Contributed by Xiaoyu Yao.
(cherry picked from commit dbbfc58c33)
2016-02-24 15:16:13 -08:00
cnauroth 59cd23e279 HDFS-9854. Log cipher suite negotiation more verbosely. Contributed by Wei-Chiu Chuang.
(cherry picked from commit d1dd248b75)
2016-02-24 12:11:44 -08:00
Kihwal Lee 83f7f62be3 HDFS-9395. Make HDFS audit logging consistant. Contributed by Kuhu Shukla.
(cherry picked from commit d27d7fc72e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
2016-02-24 12:08:43 -06:00
Zhe Zhang 9ecb71f92c HDFS-9844. Correct path creation in getTrashRoot to handle root dir.
Change-Id: I5afe6c7176063892e3e632698c849bd9f4b92906
2016-02-23 13:30:21 -08:00
Colin Patrick Mccabe 440379a0d4 HDFS-9549. TestCacheDirectives#testExceedsCapacity is flaky (Xiao Chen via cmccabe)
(cherry picked from commit 211c78c090)
2016-02-23 12:05:53 -08:00
Tsuyoshi Ozawa 56b82de6e1 HDFS-9084. Pagination, sorting and filtering of files/directories in the HDFS Web UI. (Contributed by Ravi Prakash)
(cherry picked from commit 48c61cd6aa)
2016-02-23 19:31:22 +09:00
Tsz-Wo Nicholas Sze 9b10373b2b HDFS-8578. On upgrade, Datanode should process all storage/data dirs in parallel. Contributed by vinayakumarb and szetszwo 2016-02-22 15:16:14 -08:00
Zhe Zhang 56b7f2c2c1 HADOOP-12668. Support excluding weak Ciphers in HttpServer2 through ssl-server.conf. Contributed by Vijay Singh.
Change-Id: Ie46a5427d29188935427f67e55203c19fcd83335
2016-02-22 14:14:39 -08:00
Arpit Agarwal fef99bcae0 HDFS-9842. dfs.datanode.balance.bandwidthPerSec should accept friendly size units. (Contributed by Lin Yiqun) 2016-02-22 11:34:17 -08:00
Vinayakumar B 2e3c35a835 HDFS-9425. Expose number of blocks per volume as a metric (Contributed by Brahma Reddy Battula)
(cherry picked from commit 342c9572bf)
2016-02-22 09:29:47 +05:30
Vinayakumar B 3e76768f21 HDFS-7452. skip StandbyException log for getCorruptFiles() (Contributed by Bramha Reddy Battula)
(cherry picked from commit f313516731)
2016-02-22 09:22:12 +05:30
Arpit Agarwal 7660cfd91a HDFS-9839. Reduce verbosity of processReport logging. (Contributed by Arpit Agarwal)
This closes #78
2016-02-20 23:20:18 -08:00
Yongjun Zhang b144e0f15d HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit ba1c9d484a)
2016-02-19 10:29:27 -08:00
Tsz-Wo Nicholas Sze 5faa2934e8 Move HDFS-8923 to Release 2.8.0 in CHANGES.txt. 2016-02-18 18:44:40 -08:00
Tsz-Wo Nicholas Sze 2a09f00ac3 HDFS-8923. Add -source flag to balancer usage message. Contributed by Chris Trezzo 2016-02-18 18:39:09 -08:00
Tsz-Wo Nicholas Sze 38fc086940 HADOOP-10865. Add a Crc32 chunked verification benchmark for both directly and non-directly buffer cases. 2016-02-18 12:02:00 -08:00
Kihwal Lee b83348c2e3 HDFS-4946. Allow preferLocalNode in BlockPlacementPolicyDefault to be configurable. Contributed by James Kinley and Nathan Roberts.
(cherry picked from commit ba6d5ed955)
2016-02-18 13:27:39 -06:00
cnauroth 779a69e28d HDFS-9711. Integrate CSRF prevention filter in WebHDFS. Contributed by Chris Nauroth.
(cherry picked from commit 5d1889a66d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java
2016-02-18 10:09:33 -08:00
Akira Ajisaka c183f235f1 HDFS-6832. Fix the usage of 'hdfs namenode' command. Contributed by Manjunath Ballur.
(cherry picked from commit e78d8e66f7)
2016-02-19 01:35:43 +09:00
Xiaoyu Yao ad4fcd1f9f HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou.
(cherry picked from commit fe5624b85d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java
2016-02-17 16:05:07 -08:00
Zhe Zhang e226dbbc8a HDFS-9799. Reimplement getCurrentTrashDir to remove incompatibility. (zhz)
Change-Id: I7834bcebffed38cb384db5395ddb8b6dd9e79a0b
2016-02-17 13:31:48 -08:00
Andrew Wang c22aedac68 HDFS-9608. Disk IO imbalance in HDFS with heterogeneous storages. Contributed by Wei Zhou.
(cherry picked from commit 3a23dc683c)
2016-02-17 11:30:13 -08:00
Haohui Mai e7d1507e21 HDFS-9815. Move o.a.h.fs.Hdfs to hadoop-hdfs-client. Contributed by Vinayakumar B. 2016-02-17 10:56:57 -08:00
Junping Du 6874a142d8 HDFS-9456. BlockPlacementPolicyWithNodeGroup should override verifyBlockPlacement(). Contributed by Xiaobing Zhou.
(cherry picked from commit 77ba5add0d)
2016-02-16 19:00:52 -08:00
Akira Ajisaka 8dec72bfd7 HDFS-9765. TestBlockScanner#testVolumeIteratorWithCaching fails intermittently. (aajisaka)
(cherry picked from commit 8ed07bd3e6)
2016-02-15 19:08:35 +09:00
Akira Ajisaka 5e78264cde HDFS-9773. Remove dead code related to SimulatedFSDataset in tests. Contributed by Brahma Reddy Battula.
(cherry picked from commit b0738ae673)
2016-02-15 15:35:25 +09:00
Akira Ajisaka 75c5470165 HDFS-9691. TestBlockManagerSafeMode#testCheckSafeMode fails intermittently. Contributed by Mingliang Liu.
(cherry picked from commit ac5da11399)
2016-02-15 14:00:40 +09:00
Junping Du a28be9edc7 Set the release date for 2.6.4.
(cherry picked from commit ec12ce8f48)
2016-02-13 01:44:12 -08:00
Arpit Agarwal 63f51208a8 HDFS-9801. ReconfigurableBase should update the cached configuration. (Arpit Agarwal) 2016-02-12 12:41:12 -08:00
Jing Zhao f8c9c0ff0e HDFS-9754. Avoid unnecessary getBlockCollection calls in BlockManager. Contributed by Jing Zhao.
(cherry picked from commit 972782d956)
2016-02-12 11:40:28 -08:00
Arun Suresh f501c6dc49 HDFS-9797. Log Standby exceptions thrown by RequestHedgingProxyProvider at DEBUG Level (Inigo Goiri via asuresh)
(cherry picked from commit 9fdfb546fb)
2016-02-12 11:19:29 -08:00
Zhe Zhang 5e8be29ac9 HDFS-9644. Update encryption documentation to reflect nested EZs. (zhz)
Change-Id: I84ae96f4711a16ded46d501c51ca322bfca5151d
2016-02-12 10:46:51 -08:00
Masatake Iwasaki f71fa500f9 HDFS-9700. DFSClient and DFSOutputStream should set TCP_NODELAY on sockets for DataTransferProtocol (Gary Helmling via iwasakims)
(cherry picked from commit 372d1302c6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
2016-02-13 03:32:58 +09:00
Akira Ajisaka a714d04f5d HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. Contributed by Lin Yiqun.
(cherry picked from commit e6a7044b85)
2016-02-13 02:04:01 +09:00
Arpit Agarwal 7f1d18fec6 HDFS-9790. HDFS Balancer should exit with a proper message if upgrade is not finalized (Contributed by Xiaobing Zhou) 2016-02-11 21:18:45 -08:00
Karthik Kambatla 383f3ff28c HDFS-9780. RollingFileSystemSink doesn't work on secure clusters. (Daniel Templeton via kasha)
(cherry picked from commit 33ce7f6c07)
2016-02-11 20:45:23 -08:00
Karthik Kambatla d3a0bb6d32 HDFS-9637. Tests for RollingFileSystemSink. Addendum patch to add missing file. 2016-02-11 20:41:06 -08:00
Karthik Kambatla ee4ace3edf HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) 2016-02-11 16:45:31 -08:00
Andrew Wang d489570755 HDFS-9788. Incompatible tag renumbering in HeartbeatResponseProto.
(cherry picked from commit aeb13ef2ab)
2016-02-10 15:26:14 -08:00
Akira Ajisaka a60a0a7128 HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit 802a7edfae)
2016-02-10 23:50:04 +09:00
Akira Ajisaka 15ef3a85d9 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta."
This reverts commit e4c01b8b1c.
2016-02-10 23:45:15 +09:00
Uma Maheswara Rao G 04ecce1ca9 HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value. Contributed by Kuhu Shukla.
(cherry picked from commit a7fce9ab41)
2016-02-09 15:16:53 -08:00
Allen Wittenauer 73b195eccc HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler (Ryan Sasson via aw) 2016-02-09 14:16:12 -08:00
Akira Ajisaka cb53dfcc95 HDFS-9784. Example usage is not correct in Transparent Encryption document. Contributed by Ohnishi Takashi.
(cherry picked from commit 60d2011b7c)
2016-02-10 03:30:44 +09:00
Walter Su ac856893e7 Move HDFS-9347 from 2.8.0 to 2.6.5 in CHANGES.txt. 2016-02-09 10:53:45 +08:00
zhezhang ef9a81e875 HDFS-9244. Support nested encryption zones.
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
2016-02-08 16:31:41 -08:00
Uma Maheswara Rao G 44a020ed81 HDFS-9777. Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh)
(cherry picked from commit bffaa38a91)
2016-02-08 14:00:47 -08:00
Kihwal Lee 494cfd1dc4 HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su.
(cherry picked from commit 193d27de0a)
2016-02-08 12:17:07 -06:00
Uma Maheswara Rao G 82d4b2cf27 HDFS-9713. DataXceiver#copyBlock should return if block is pinned. Contributed by Uma Maheswara Rao G.
(cherry picked from commit aea31eee78)
2016-02-07 20:55:58 -08:00
Akira Ajisaka e4c01b8b1c HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit fe124da5ff)
2016-02-06 19:32:50 +09:00
cnauroth a7232f5e1d HDFS-9761. Rebalancer sleeps too long between iterations. Contributed by Mingliang Liu.
(cherry picked from commit c6497949e8)
2016-02-05 15:07:59 -08:00
Tsz-Wo Nicholas Sze 117397b89f HDFS-9726. Refactor IBR code to a new class. 2016-02-05 23:20:16 +08:00
Haohui Mai e62ca14856 HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse ObjectMapper. Contributed by Akira Ajisaka. 2016-02-04 11:35:34 -08:00
Kihwal Lee e26c58fb4e HDFS-9730. Storage ID update does not happen when there is a layout change. Contributed by Tsz Wo Nicholas Sze.
(cherry picked from commit 496f33de0c)
2016-02-04 13:21:30 -06:00
Vinayakumar B 6ee66be2fa HDFS-9629. Update the footer of Web UI to show year 2016 (Contributed by Xiao Chen and Vinayakumar B)
(cherry picked from commit ddcd6954d03d3caadc8b69f7b886a3a17c3c9d0c)
2016-02-04 13:38:26 +05:30
Konstantin V Shvachko 4a0550537c HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of -namenode. Contributed by Mingliang Liu. 2016-02-03 14:51:41 -08:00
Konstantin V Shvachko 6d1f9a6b4c Move HDFS-9601 to branch-2.8.0 in CHANGES.txt. (shv) 2016-02-03 14:36:25 -08:00
Jing Zhao 459c31de4a HDFS-9748. Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice. Contributed by Walter Su.
(cherry picked from commit 7badf15604)
2016-02-03 13:32:19 -08:00
Vinayakumar B 6941e2ccbf HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() (Contributed by Chang Li)
(cherry picked from commit eb2fb943fd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
2016-02-03 11:21:08 +05:30
Vinayakumar B e8a9a69a27 HDFS-9739. DatanodeStorage.isValidStorageId() is broken (Contributed by Mingliang Liu)
(cherry picked from commit d6b1acb940)
2016-02-03 10:33:55 +05:30
Vinayakumar B 8577c0b6a2 HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 04375756a5)
2016-02-03 07:35:47 +05:30
Junping Du 3583c77ee8 Adding release 2.6.5 to CHANGES.txt 2016-02-02 18:09:17 -08:00
Colin Patrick Mccabe 60d3a3c30b HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size (Elliot Clark via cmccabe)
(cherry picked from commit 2da03b48eb)
2016-02-02 11:04:33 -08:00
Lei Xu 2f1d1c67f0 HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference. (Xiao Chen via lei)
(cherry picked from commit 9d494f0c0e)
2016-02-02 10:42:24 -08:00
Vinayakumar B 80dd039b6e HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys before initialize (Contributed by DENG FEI)
(cherry picked from commit 4d7055e0c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAConfiguration.java
2016-02-02 12:56:31 +05:30
Yongjun Zhang fc8d9cc758 HDFS-9406. FSImage may get corrupted after deleting snapshot. (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang)
(cherry picked from commit 34ab50ea92)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
2016-02-01 14:09:52 -08:00
Lei Xu d2f8e6c269 HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei)
(cherry picked from commit 12a4ebb4471f78b642e12c2ec0784c20f62cca3c)
2016-02-01 12:56:00 -08:00
Xiaoyu Yao 7cc3b1c7a3 HDFS-9210. Fix some misuse of %n in VolumeScanner#printStats. Contributed by Xiaoyu Yao.
(cherry picked from commit b11008d54d)
2016-02-01 11:57:11 -08:00
Steve Loughran 7786e97df2 HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames (Mingliang Liu via stevel) 2016-02-01 16:01:32 +00:00
Vinayakumar B 71374cca9d HDFS-9566. Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method (Contributed by Daryn Sharp)
(cherry picked from commit e418bd1fb0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
2016-02-01 13:32:00 +05:30
Yongjun Zhang 2fb711bde9 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit eddd823cd6)
2016-01-29 12:13:03 -08:00
Yongjun Zhang a24898fd49 HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. (Xiao Chen via Yongjun Zhang)
(cherry picked from commit 8ee060311c)
2016-01-28 23:13:41 -08:00
Colin Patrick Mccabe dc65479fc0 HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had an error (Rakesh R via cmccabe)
(cherry picked from commit f67149ab08)
2016-01-28 20:12:18 -08:00
Tsz-Wo Nicholas Sze d37ecbf721 HDFS-9654. Code refactoring for HDFS-8578. 2016-01-28 10:56:39 +08:00
Tsz-Wo Nicholas Sze b10d8ced21 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-28 10:42:40 +08:00
Jing Zhao d7982222fa HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:51:10 -08:00
Zhe Zhang 2056dd7d3a HDFS-9541. Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes greater than 2 GB. Contributed by Colin Patrick McCabe.
Change-Id: Ifce1b9be534dc8f7e9d2634cd60e423921b9810f
2016-01-26 11:26:31 -08:00
Tsz-Wo Nicholas Sze c59c142f47 HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. 2016-01-26 14:56:23 +08:00
Jitendra Pandey 4334976187 HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. Contributed by Mingliang Liu. 2016-01-25 17:06:36 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 97f6cfa94b Release process for 2.7.2: Set the release date for 2.7.2
(cherry picked from commit ec4d2d9f40)
2016-01-25 15:47:02 -08:00
Arpit Agarwal 0dbaef4377 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:16:45 -08:00
Eric Yang da61953a0b HDFS-9653. Added blocks pending deletion report to dfsadmin.
(Weiwei Yang via eyang)
2016-01-24 14:26:46 -08:00
Kihwal Lee b5db1d4410 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:19:23 -06:00
Akira Ajisaka 4fe28a644e HDFS-9682. Fix a typo "aplication" in HttpFS document. Contributed by Weiwei yang.
(cherry picked from commit 6831634682)
2016-01-22 20:12:39 +09:00
Masatake Iwasaki 73ef77c12f HDFS-9618. Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks (iwasakims) 2016-01-22 14:12:06 +09:00
Masatake Iwasaki 245c3728ef HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block (iwasakims) 2016-01-22 12:39:39 +09:00
Zhe Zhang 22caeb1c0a HDFS-9674. The HTrace span for OpWriteBlock should record the maxWriteToDisk time. Contributed by Colin McCabe.
Change-Id: I9bf3f3bcd57f5880189ad7c160f3dd66f97d904b
2016-01-21 13:26:54 -08:00
Jing Zhao c19bdc1926 HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao.
(cherry picked from commit c304890c8c)
2016-01-21 11:37:24 -08:00
Kihwal Lee 3d8d8f1497 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne.
(cherry picked from commit 7b70500484574a565dd8cd5c7d8b5bc7c6d91154)
2016-01-21 09:33:30 -06:00
Vinayakumar B 845acfd96c HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI)
(cherry picked from commit b7372b7166)
2016-01-21 19:18:36 +05:30
Lei Xu c57f55223c HDFS-9655. NN should start JVM pause monitor before loading fsimage. (John Zhuge via Lei (Eddy) Xu)
(cherry picked from commit 2ec438e8f7)
2016-01-20 14:26:47 -08:00
Colin Patrick Mccabe c56044b9ee HDFS-9576: HTrace: collect position/length information on read operations (zhz via cmccabe)
(cherry picked from commit 7905788db9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
2016-01-20 11:27:51 -08:00
Vinayakumar B 252bf56e59 HDFS-9661. Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw (Contributed by ade)
(cherry picked from commit 1425578690)
2016-01-20 21:05:44 +05:30
Yongjun Zhang a5ee4c09de HDFS-6054. MiniQJMHACluster should not use static port to avoid binding failure in unit test. (Yongjun Zhang)
(cherry picked from commit 57d0a94305)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniQJMHACluster.java
2016-01-19 23:13:46 -08:00
Akira Ajisaka 3a8d372518 HDFS-9623. Update example configuration of block state change log in log4j.properties. Contributed by Masatake Iwasaki.
(cherry picked from commit 92c5f565fd)
2016-01-19 13:31:48 +09:00
Tsz-Wo Nicholas Sze 87779a4478 In CHANGES.txt, move HDFS-9294 to Release 2.6.4. 2016-01-16 23:46:19 +08:00
Andrew Wang 46323d7744 HDFS-9624. DataNode start slowly due to the initial DU command operations. (Lin Yiqun via wang)
(cherry picked from commit c07f7fa8ff)
2016-01-15 11:28:55 -08:00
Yongjun Zhang ee012c1f6f HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit a9c69ebeb7)
2016-01-15 10:10:28 -08:00
Arpit Agarwal aadcaf4f32 HDFS-9415. Document dfs.cluster.administrators and dfs.permissions.superusergroup. (Contributed by Xiaobing Zhou) 2016-01-14 14:18:00 -08:00
Yongjun Zhang cfeecf113b HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message change. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit 817cc1f02a)
2016-01-14 12:45:18 -08:00
Colin Patrick Mccabe cea1c71087 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe)
(cherry picked from commit b1ed28fa77)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
2016-01-14 11:13:13 -08:00
Junping Du 586efcf3c8 Move HDFS-8326 out of CHANGES.txt entry for branch-2.7.2. 2016-01-14 06:32:52 -08:00
Junping Du 62eabec549 Add missing CHANGES.txt entry for HDFS-8767.
(cherry picked from commit e50dd19a9c)

Conflicts:

	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2016-01-14 06:13:02 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 1fba4051f8 HADOOP-5323. Trash documentation should describe its directory structure and configurations. (Weiwei Yang via ozawa)
Moved entry from HDFS CHANGES.txt to that of common.

(cherry picked from commit cb73c753da)
2016-01-13 14:56:08 -08:00
Yongjun Zhang 5e2416b901 HDFS-9569. Log the name of the fsimage being loaded for better supportability. (Yongjun Zhang)
(cherry picked from commit 25051c3bd0)
2016-01-12 09:24:54 -08:00
Jing Zhao 313f03bfda HDFS-9621. Consolidate FSDirStatAndListingOp#createFileStatus to let its INodesInPath parameter always include the target INode. Contributed by Jing Zhao. 2016-01-11 22:48:36 -08:00
Xiaoyu Yao c5f63e637a Correct commit message for HDFS-9584
(cherry picked from commit 103d3cfc4e)
2016-01-11 19:00:17 -08:00
Xiaoyu Yao 521f60c094 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore.
(cherry picked from commit c2e2e13455)
2016-01-11 17:12:43 -08:00
Arpit Agarwal 68a32a79b5 HDFS-9639. Inconsistent Logging in BootstrapStandby. (Contributed by Xiaobing Zhou) 2016-01-11 15:09:43 -08:00
Zhe Zhang 17dfa04be6 HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng.
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2016-01-11 09:53:20 -08:00
Lei Xu 699d420497 HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (Tony Wu via lei)
(cherry picked from commit fd8065a763)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java
2016-01-08 15:10:41 -08:00
Zhe Zhang 40dabae236 HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. Contributed by Xiao Chen.
Change-Id: I5cee398dfc361121d8b666ffd1203cc14a4ace24
2016-01-08 13:19:14 -08:00
Kihwal Lee 8d04b7c272 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee.
(cherry picked from commit 38c4c14472)
2016-01-08 11:15:12 -06:00
Vinayakumar B 08f39bd5d1 HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the correct namenode (Contributed by Wei-Chiu Chuang)
(cherry picked from commit 6702e7d66a)
2016-01-07 14:20:39 +05:30
Vinayakumar B aa710bd461 HDFS-9600. do not check replication if the block is under construction (Contributed by Phil Yang)
(cherry picked from commit 34cd7cd765)
2016-01-07 11:30:57 +05:30
Haohui Mai 71006d5c22 HDFS-9047. Retire libwebhdfs. Contributed by Haohui Mai. 2016-01-06 16:13:43 -08:00
Arpit Agarwal 2e6990cdf2 HDFS-9498. Move code that tracks blocks with future generation stamps to BlockManagerSafeMode. (Contributed by Mingliang Liu) 2016-01-06 10:44:51 -08:00
Haohui Mai 150c9a80ee HDFS-9605. Add links to failed volumes to explorer.html in HDFS Web UI. Contributed by Archana T. 2016-01-05 13:56:08 -08:00
Haohui Mai 79dc184224 HDFS-7779. Support changing ownership, group and replication in HDFS Web UI. Contributed by Ravi Prakash. 2016-01-05 13:44:40 -08:00
Uma Mahesh cdd796d199 HADOOP-12658. Clear javadoc and check style issues around DomainSocket. Contributed by Kai Zheng
(cherry picked from commit 778146eaae)
2016-01-04 15:04:46 -08:00
Akira Ajisaka 247ab92464 Set release date for 2.6.3
(cherry picked from commit db99e30f67)
2016-01-04 16:12:52 +09:00
Akira Ajisaka d98cfe611f HADOOP-11252. RPC client does not time out by default. Contributed by Wilfred Spiegelenburg and Masatake Iwasaki.
(cherry picked from commit 64ae85fd2e)
2016-01-04 14:32:08 +09:00
cnauroth 1f8162b74e HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. Contributed by Xiao Chen. 2015-12-29 10:57:24 -08:00
Benoy Antony 42160d358a HDFS-9597. BaseReplicationPolicyTest should update data node stats after adding a data node. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 8cfd672397)
2015-12-25 08:57:34 -08:00
Benoy Antony 0d249a5e28 HDFS-9034. StorageTypeStats Metric should not count failed storage. Contributed by Surendra Singh Lilhore.
(cherry picked from commit df83230948)
2015-12-22 15:29:36 -08:00
Kihwal Lee 131260f0a7 HDFS-7163. WebHdfsFileSystem should retry reads according to the configured retry policy. Contributed by Eric Payne.
(cherry picked from commit 867048c3e4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
2015-12-22 14:19:12 -06:00
cnauroth 9b039a5004 HDFS-9458. TestBackupNode always binds to port 50070, which can cause bind failures. Contributed by Xiao Chen.
(cherry picked from commit 5c0ff69618)
2015-12-22 10:29:40 -08:00
Colin Patrick Mccabe a56c5ef636 HDFS-9589. Block files which have been hardlinked should be duplicated before the DataNode appends to the them (cmccabe)
(cherry picked from commit bb540ba85a)
2015-12-22 09:37:30 -08:00
Akira Ajisaka 12caea3274 Move HDFS-8914 from 2.8.0 to 2.7.3 in CHANGES.txt.
(cherry picked from commit 2cb5afffc4)
2015-12-21 21:53:44 +09:00
Akira Ajisaka 7526a2a005 HDFS-9505. HDFS Architecture documentation needs to be refreshed. Contributed by Masatake Iwasaki.
(cherry picked from commit fa544020f6)
2015-12-21 21:23:28 +09:00
Uma Mahesh fc9e130825 YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh)
(cherry picked from commit 0f82b5d878)
2015-12-19 23:17:33 -08:00
cnauroth dd91cf9fa5 HADOOP-12570. HDFS Secure Mode Documentation updates. Contributed by Arpit Agarwal.
(cherry picked from commit 932956458d)
2015-12-18 16:25:06 -08:00
Zhe Zhang 223b47b398 HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown() is wrong. Contributed by Wei-Chiu Chuang.
Change-Id: Idde8522e11c77f82b376e0c90a9bf33d163d5cb9
2015-12-18 16:05:21 -08:00
Colin Patrick Mccabe 31d2fb91fc HDFS-9350. Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe)
(cherry picked from commit e63388fdf2)
2015-12-18 10:05:12 -08:00
Andrew Wang 71fb205735 HDFS-8477. describe dfs.ha.zkfc.port in hdfs-default.xml. Contributed by Kanaka Kumar Avvaru.
(cherry picked from commit 058cfddebb)
2015-12-18 09:14:23 -08:00
Vinayakumar B c887bcd1f0 HDFS-9393. After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy (Contributed by J.Andreina)
(cherry picked from commit bfadf11b36)
2015-12-18 11:39:46 +05:30
cnauroth 7e5306e795 HDFS-9571. Fix ASF Licence warnings in Jenkins reports. Contributed by Brahma Reddy Battula.
(cherry picked from commit 6da9deec20)
2015-12-17 16:26:08 -08:00
Kihwal Lee 3c0ae5914c HDFS-9533. seen_txid in the shared edits directory is modified during bootstrapping. Contributed by Kihwal Lee.
(cherry picked from commit 5cb1e0118b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java
2015-12-17 17:23:06 -06:00
cnauroth 294d0f6f6a HDFS-9572. Prevent DataNode log spam if a client connects on the data transfer port but sends no data. Contributed by Chris Nauroth
(cherry picked from commit 03bab8dea1)
2015-12-17 14:05:05 -08:00
Arpit Agarwal c0733db768 HDFS-9515. NPE when MiniDFSCluster#shutdown is invoked on uninitialized reference. (Contributed by Wei-Chiu Chuang) 2015-12-17 13:57:12 -08:00
cnauroth a40ff6d4fb Revert "HDFS-9569. Log the name of the fsimage being loaded for better supportability. Contributed by Yongjun Zhang."
This reverts commit 03e10f1fb3.
2015-12-17 13:53:25 -08:00
Allen Wittenauer a5c74cd921 HDFS-9570. Minor typos, grammar, and case sensitivity cleanup in HdfsPermissionsGuide.md's (Travis Campbell via aw)
(cherry picked from commit 8d278d8d2e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-12-17 11:40:58 -08:00
cnauroth 940026fc6b HDFS-9552. Document types of permission checks performed for HDFS operations. Contributed by Chris Nauroth.
(cherry picked from commit 724e71f63b)
2015-12-17 10:43:54 -08:00
Arpit Agarwal 0c1d1e311c HDFS-9565. TestDistributedFileSystem.testLocatedFileStatusStorageIdsTypes is flaky due to race condition. (Contributed by Wei-Chiu Chuang) 2015-12-17 10:37:58 -08:00
Kihwal Lee 03e10f1fb3 HDFS-9569. Log the name of the fsimage being loaded for better supportability. Contributed by Yongjun Zhang.
(cherry picked from commit eb6939cea0)
2015-12-17 10:30:58 -06:00
Konstantin V Shvachko 3d0222eafd Update CHANGES.txt to move HDFS-9516 to 2.7.3 section. 2015-12-16 23:24:34 -08:00
Uma Mahesh 3be39d0229 HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh)
(cherry picked from commit f741476146)
2015-12-16 22:04:19 -08:00
Uma Mahesh e82135df87 Revert this commit as there is compilation issue with this patch in branch-2 "HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh)"
This reverts commit 991ce3d630.
2015-12-16 19:18:30 -08:00
Uma Mahesh 991ce3d630 HDFS-9198. Coalesce IBR processing in the NN. (Daryn Sharp via umamahesh)
(cherry picked from commit f741476146)
2015-12-16 18:25:29 -08:00
Andrew Wang c7cb29defe HDFS-9300. TestDirectoryScanner.testThrottle() is still a little flakey. Contributed by Daniel Templeton.
(cherry picked from commit 3c0adac88c)
2015-12-16 16:21:16 -08:00
cnauroth e8e3737ce8 HDFS-9557. Reduce object allocation in PB conversion. Contributed by Daryn Sharp.
(cherry picked from commit c470c8953d)
2015-12-16 11:11:27 -08:00
Andrew Wang 2796c1099f HDFS-8894. Set SO_KEEPALIVE on DN server sockets. Contributed by Kanaka Kumar Avvaru.
(cherry picked from commit 49949a4bb0)
2015-12-15 14:39:04 -08:00
Jing Zhao e0abb0a4e7 HDFS-9371. Code cleanup for DatanodeManager. Contributed by Jing Zhao.
(cherry picked from commit 8602692338)
2015-12-15 11:02:04 -08:00
Plamen Jeliazkov 15124c86d8 HDFS-9516. Truncate file fails with data dirs on multiple disks. Contributed by Plamen Jeliazkov. 2015-12-15 00:39:42 -08:00
Lei Xu 6df45118e5 Revert "Revert "HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe)""
This reverts commit fb909554a8.
2015-12-14 11:26:32 -08:00
Jing Zhao e02ad5a618 HDFS-9535. Newly completed blocks in IBR should not be considered under-replicated too quickly. Contributed by Mingliang Liu.
(cherry picked from commit e534569814)
2015-12-14 11:18:28 -08:00
Lei Xu 0852d3562a HDFS-9281. Change TestDeleteBlockPool to not explicitly use File to check block pool existence. (lei)
(cherry picked from commit db975af5fae506eb3d586bd9201f76adb3fd1281)
2015-12-14 11:01:48 -08:00
Yongjun Zhang be167d0154 HDFS-9514. TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; exception being swallowed. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit bf5295b118)
2015-12-11 20:56:58 -08:00
Yongjun Zhang 3fc9593318 HDFS-9519. Some coding improvement in SecondaryNameNode#main. (Xiao Chen via Yongjun Zhang)
(cherry picked from commit 2a4c7d4fac)
2015-12-11 20:56:46 -08:00
Xiaoyu Yao 08616b3654 HDFS-8785. TestDistributedFileSystem is failing in trunk. Contributed by Xiaoyu Yao.
(cherry picked from commit 2196e39e14)
2015-12-11 20:56:12 -08:00
Uma Mahesh 5b1e1dfb3f HDFS-9528. Cleanup namenode audit/log/exception messages. (szetszwo via umamahesh)
(cherry picked from commit 796a676d18)
2015-12-11 19:18:25 -08:00
Yongjun Zhang e00888d5cb HDFS-9532. Detailed exception info is lost in reportTo methods of ErrorReportAction and ReportBadBlockAction. (Yongjun Zhang)
(cherry picked from commit f5a911452f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-12-11 18:24:50 -08:00
Xiaoyu Yao 5595b912e4 HDFS-8326. Documentation about when checkpoints are run is out of date. (Misty Stanley-Jones via xyao)
(cherry picked from commit d0e75e60fb)
2015-12-11 08:41:30 -08:00
Kihwal Lee a72ef921f0 HDFS-9445. Datanode may deadlock while handling a bad volume. Contributed by Walter Su.
(cherry picked from commit a48301791e)
2015-12-11 08:46:03 -06:00
Uma Mahesh 7dc558b6a7 HDFS-9472. concat() API does not give proper exception messages on ./reserved relative path (Rakesh R via umamahesh)
(cherry picked from commit e363417e7b)
2015-12-11 00:01:02 -08:00
Lei Xu fb909554a8 Revert "HDFS-8860. Remove unused Replica copyOnWrite code (Lei (Eddy) Xu via Colin P. McCabe)"
This reverts commit b9bf1a9391.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/DataNodeTestUtils.java
2015-12-10 11:23:02 -10:00
Tsz-Wo Nicholas Sze c65a796f88 HDFS-9527. The return type of FSNamesystem.getBlockCollection should be changed to INodeFile. 2015-12-09 18:10:34 -08:00
Vinod Kumar Vavilapalli e81cb68508 HDFS-9273. Moving to 2.6.3 CHANGES section to reflect the backport. 2015-12-08 11:34:05 -08:00
cnauroth 1072be12b9 HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. Contributed by Xiaobing Zhou.
(cherry picked from commit 303f80e656)
2015-12-08 10:50:13 -08:00
Jing Zhao c993327483 HDFS-9129. Move the safemode block count into BlockManager. Contributed by Mingliang Liu. 2015-12-07 13:46:12 -08:00
Steve Loughran af14458de7 HADOOP-12321. Make JvmPauseMonitor an AbstractService. (Sunil G via Stevel) [includes HDFS-8947 MAPREDUCE-6462 and YARN-4072] 2015-12-06 17:42:56 +00:00
Arpit Agarwal 6999e764cb HDFS-9414. Add missing license header 2015-12-05 09:24:44 -08:00
Arpit Agarwal 71be31201a HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. (Contributed by Xiaobing Zhou) 2015-12-04 20:53:43 -08:00
Lei Xu ce49ba39d2 HDFS-9491. Tests should get the number of pending async deletes via FsDatasetTestUtils. (Tony Wu via lei)
(cherry picked from commit c92713dac56da0f836382005d84b577cfeb36a05)
2015-12-04 18:05:51 -08:00
Xiaoyu Yao eb7f9901b4 HDFS-8831. Trash Support for deletion in HDFS encryption zone. Contributed by Xiaoyu Yao.
(cherry picked from commit cbc7b6bf97)
2015-12-04 16:57:58 -08:00
Arpit Agarwal 496c8969d4 HDFS-9214. Add missing license header 2015-12-04 15:33:24 -08:00
Arpit Agarwal 9ddd7a95b9 HDFS-9214. Support reconfiguring dfs.datanode.balance.max.concurrent.moves without DN restart. (Contributed by Xiaobing Zhou) 2015-12-04 14:46:54 -08:00
Yongjun Zhang 78a21e773d HDFS-9474. TestPipelinesFailover should not fail when printing debug message. (John Zhuge via Yongjun Zhang)
(cherry picked from commit 59dbe8b3e9)
2015-12-04 13:50:03 -08:00
Colin P. McCabe 3beedc177d HDFS-9267. TestDiskError should get stored replicas through FsDatasetTestUtils. (Lei (Eddy) Xu via Colin P. McCabe)
(cherry picked from commit e02bbeb886)
2015-12-04 12:51:55 -08:00
Lei Xu 58714f3b73 HDFS-9490. MiniDFSCluster should change block generation stamp via FsDatasetTestUtils. (Tony Wu via lei)
(cherry picked from commit 0ac8fb4b33)
2015-12-04 10:26:06 -08:00
Ming Ma a68faf1a02 HDFS-9430 Remove waitForLoadingFSImage since checkNNStartup has ensured image loaded and namenode started. (Brahma Reddy Battula via mingma)
(cherry picked from commit 3fa33b5c2c)
2015-12-04 09:55:02 -08:00
Konstantin V Shvachko 33d4588a7c HDFS-9484. NNThroughputBenchmark$BlockReportStats should not send empty block reports. Contributed by Mingliang Liu. 2015-12-03 14:56:00 -08:00
Konstantin V Shvachko 3aa73a867a HDFS-9436. Make NNThroughputBenchmark$BlockReportStats run with 10 datanodes by default. Contributed by Mingliang. 2015-12-03 14:54:06 -08:00
Tsz-Wo Nicholas Sze bac127a4e2 HDFS-9294. DFSClient deadlock when close file and failed to renew lease. Contributed by Brahma Reddy Battula 2015-12-02 17:49:16 -08:00
Colin Patrick Mccabe 9b516a2a05 HDFS-9429. Tests in TestDFSAdminWithHA intermittently fail with EOFException (Xiao Chen via Colin P. McCabe)
(cherry picked from commit 53e3bf7e70)
2015-12-01 23:58:09 -08:00
Arpit Agarwal 0a1ef49b33 HDFS-6533. TestBPOfferService#testBasicFunctionalitytest fails intermittently. (Contributed by Wei-Chiu Chuang) 2015-12-01 13:32:37 -08:00
Jing Zhao 7e416aa70d HDFS-9485. Make BlockManager#removeFromExcessReplicateMap accept BlockInfo instead of Block. Contributed by Mingliang Liu.
(cherry picked from commit 830eb252aa)
2015-12-01 13:07:19 -08:00
Zhe Zhang 9db4b77ea5 HDFS-9269. Update the documentation and wrapper for fuse-dfs. Contributed by Wei-Chiu Chuang.
Change-Id: Ia9ec512de2464bf94725cc7c15c378c59d0f04c4
2015-12-01 10:25:32 -08:00
Allen Wittenauer 45bb467918 HADOOP-12366. expose calculated paths (aw) 2015-12-01 08:16:20 -08:00
Junping Du 092c469059 Creating 2.6.4 entries in CHANGES.txt files.
(cherry picked from commit 028894281b)
2015-12-01 08:18:57 -08:00
Akira Ajisaka 402a2f2d64 HDFS-9336. deleteSnapshot throws NPE when snapshotname is null. Contributed by Brahma Reddy Battula.
(cherry picked from commit 1c05393b51)
2015-12-01 11:30:11 +08:00
Andrew Wang ce1111ceea HDFS-9470. Encryption zone on root not loaded from fsimage after NN restart. Xiao Chen via wang.
(cherry picked from commit 9b8e50b424)
2015-11-30 14:36:06 -08:00
Vinayakumar B 57e0c735f8 HDFS-9426. Rollingupgrade finalization is not backward compatible (Contributed by Kihwal Lee)
(cherry picked from commit c62d42cd8b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolClientSideTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/DatanodeProtocolServerSideTranslatorPB.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/proto/DatanodeProtocol.proto

(cherry picked from commit 9f256d1d71)
2015-11-26 09:40:44 +05:30
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Vinod Kumar Vavilapalli 767815f317 Adding release 2.8.0 to CHANGES.txt
(cherry picked from commit d57fd181c7)
2015-11-25 17:37:47 -08:00
Jing Zhao 288cf8437b HDFS-9467. Fix data race accessing writeLockHeldTimeStamp in FSNamesystem. Contributed by Mingliang Liu.
(cherry picked from commit e556c35b05)
2015-11-25 14:23:06 -08:00
Konstantin V Shvachko 9c1ef648a9 HDFS-9407. TestFileTruncate should not use fixed NN port. Contributed by Brahma Reddy Battula. 2015-11-25 14:16:03 -08:00
Xiaoyu Yao 4f1f9f7eff HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info. Contributed by Xiaoyu Yao.
(cherry picked from commit e3d673901b)
2015-11-25 13:54:22 -08:00
Haohui Mai 15caaa7d0a HDFS-9459. hadoop-hdfs-native-client fails test build on Windows after transition to ctest. Contributed by Chris Nauroth. 2015-11-25 10:32:13 -08:00
Yongjun Zhang 48b294c58e HDFS-9438. TestPipelinesFailover assumes Linux ifconfig. (John Zhuge via Yongjun Zhang)
(cherry picked from commit 8176ea7dc6)
2015-11-25 08:50:02 -08:00
Yongjun Zhang 7ac8d8f510 HDFS-6694. Addendum. Update CHANGES.txt for cherry-picking to 2.8.
(cherry picked from commit 84d01ad7f4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-11-25 08:49:26 -08:00
arp 566ceac1ff HDFS-6694. TestPipelinesFailover.testPipelineRecoveryStress tests fail intermittently with various symptoms - debugging patch (Contributed by Yongjun Zhang)
(cherry picked from commit c5d9a4a91e)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-11-25 08:45:59 -08:00
Tom Zeng d6f7d22929 HADOOP-12415. Add io.netty dependency to hadoop-nfs and to hadoop-hdfs, needed to build Bigtop successfuly, see BIGTOP-2049
Signed-off-by: Konstantin Boudnik <cos@boudnik.org>
(cherry picked from commit b4c6b511e7)
2015-11-24 18:39:01 -08:00
Tsz-Wo Nicholas Sze 79e55fb47f HDFS-8807. dfs.datanode.data.dir does not handle spaces between storageType and URI correctly. Contributed by Anu Engineer 2015-11-24 16:05:21 -08:00
Haohui Mai b07c91a16b HDFS-8335. FSNamesystem should construct FSPermissionChecker only if permission is enabled. Contributed by Gabor Liptak. 2015-11-24 13:07:45 -08:00
Tsz-Wo Nicholas Sze 15ecbc95bf Move HDFS-9434 to 2.6.3 in CHANGES.txt. 2015-11-24 12:44:37 -08:00
Ming Ma 85d04dc464 HDFS-9314. Improve BlockPlacementPolicyDefault's picking of excess replicas. (Xiao Chen via mingma)
(cherry picked from commit 0e54b164a8)
2015-11-24 10:31:23 -08:00
cnauroth c79f017752 HDFS-6101. TestReplaceDatanodeOnFailure fails occasionally. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 1777608fa0)
2015-11-24 09:39:37 -08:00
Uma Mahesh 52c889ac14 HDFS-9433. DFS getEZForPath API on a non-existent file should throw FileNotFoundException (Rakesh R via umamahesh)
(cherry picked from commit 411e2b2e7c)
2015-11-23 18:46:48 -08:00
Haohui Mai d469e9e4ca HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. Contributed by Walter Su. 2015-11-23 10:50:23 -08:00
Steve Loughran 331ed6998b Revert "HDFS-9423. Fix intermittent failure of TestEditLogTailer. Contributed by Masatake Iwasaki."
This reverts commit de3b33e121.
2015-11-23 12:28:25 +00:00
Walter Su de3b33e121 HDFS-9423. Fix intermittent failure of TestEditLogTailer. Contributed by Masatake Iwasaki.
(cherry picked from commit 201f14e271)
2015-11-23 18:50:29 +08:00
Walter Su 040c582b99 HDFS-9435. TestBlockRecovery#testRBWReplicas is failing intermittently. Contributed by Rakesh R.
(cherry picked from commit 001ecf962c)
2015-11-23 15:50:57 +08:00
Walter Su 846de7d842 HDFS-9428. Fix intermittent failure of TestDNFencing.testQueueingWithAppend. Contributed by Masatake Iwasaki.
(cherry picked from commit 5aba093361)
2015-11-23 15:38:41 +08:00
Haohui Mai c07fab5f6f HDFS-9024. Deprecate the TotalFiles metric. Contributed by Akira Ahisaka. 2015-11-22 23:03:05 -08:00
Haohui Mai 5ce7e72145 HDFS-9356. Decommissioning node does not have Last Contact value in the UI. Contributed by Surendra Singh Lilhore. 2015-11-22 22:59:12 -08:00
Haohui Mai 529103c51f HDFS-7897. Shutdown metrics when stopping JournalNode. Contributed by zhouyingchao. 2015-11-22 19:12:32 -08:00
Haohui Mai 8aa12772d4 HDFS-6885. Fix wrong use of BytesWritable in FSEditLogOp#RenameOp. Contributed by Yi Liu. 2015-11-22 17:17:01 -08:00
Haohui Mai 53f6a9e134 HDFS-3302. Review and improve HDFS trash documentation. Contributed by Harsh J. 2015-11-22 16:30:46 -08:00
Haohui Mai 2a4cd27bdc HDFS-7796. Include X-editable for slick contenteditable fields in the web UI. Contributed by Ravi Prakash. 2015-11-22 16:22:14 -08:00
Haohui Mai 469fef2a4f HDFS-9153. Pretty-format the output for DFSIO. Contributed by Kai Zheng. 2015-11-22 16:18:16 -08:00
Haohui Mai 049ccf71ce HDFS-8914. Document HA support in the HDFS HdfsDesign.md. Contributed by Lars Francke. 2015-11-22 16:10:38 -08:00
Haohui Mai 64add87f5c HADOOP-11218. Add TLSv1.1,TLSv1.2 to KMS, HttpFS, SSLFactory. Contributed by Vijay Singh. 2015-11-22 16:00:44 -08:00
Haohui Mai f504d95e87 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:57:23 -08:00
Haohui Mai 2c185222f9 HDFS-9443. Disabling HDFS client socket cache causes logging message printed to console for CLI commands. Contributed by Chris Nauroth. 2015-11-22 15:06:36 -08:00
Tsz-Wo Nicholas Sze bf82a8de30 HDFS-9434. Recommission a datanode with 500k blocks may pause NN for 30 seconds for printing info log messags. 2015-11-19 14:54:59 -08:00
Yongjun Zhang 0664782025 HDFS-9439. Include status of closeAck into exception message in DataNode#run. (Xiao Chen via Yongjun Zhang)
(cherry picked from commit 31dc5c650e)
2015-11-19 14:22:44 -08:00
Ming Ma 5a3db21563 HDFS-8056. Decommissioned dead nodes should continue to be counted as dead after NN restart. (mingma)
(cherry picked from commit 1c4951a7a0)
2015-11-19 10:04:40 -08:00
Akira Ajisaka c74e42b4a2 HDFS-9431. DistributedFileSystem#concat fails if the target path is relative. Contributed by Kazuho Fujii.
(cherry picked from commit ac1aa6c819)
2015-11-19 20:55:36 +09:00
cnauroth 7b94ae17ab HDFS-9400. TestRollingUpgradeRollback fails on branch-2. Contributed by Brahma Reddy Battula.
(cherry picked from commit bad2afe39a)
2015-11-18 11:00:18 -08:00
Colin Patrick Mccabe 0f9f9ba3cb HDFS-9252. Change TestFileTruncate to use FsDatasetTestUtils to get block file size and genstamp. (Lei (Eddy) Xu via cmccabe)
(cherry picked from commit dfbde3fc51)
2015-11-17 10:54:32 -08:00
Akira Ajisaka 2babf12810 Move HDFS-8615 from 2.8.0 to 2.6.3 in CHANGES.txt.
(cherry picked from commit b82a6cee4b)
2015-11-17 16:34:04 +09:00
Arpit Agarwal efcaf9c28f HDFS-9397. Fix typo for readChecksum() LOG.warn in BlockSender.java. (Contributed by Enrique Flores) 2015-11-16 21:02:06 -08:00
Walter Su bff9d08a43 HDFS-9358. TestNodeCount#testNodeCount timed out. Contributed by Masatake Iwasaki.
(cherry picked from commit 621cbb4f69)
2015-11-17 10:53:50 +08:00
Xiaoyu Yao dceed6b0a8 HDFS-9421. NNThroughputBenchmark replication test NPE with -namenode option. Contributed by Mingliang Liu.
(cherry picked from commit a4f62a2d58)
2015-11-16 16:42:57 -08:00
Xiaoyu Yao 5ea3f03ecc HDFS-9387. Fix namenodeUri parameter parsing in NNThroughputBenchmark. Contributed by Mingliang Liu.
(cherry picked from commit 2701f2d255)
2015-11-16 16:31:06 -08:00
Ming Ma 42b55ff23e HDFS-9413. getContentSummary() on standby should throw StandbyException. (Brahma Reddy Battula via mingma)
(cherry picked from commit 02653add98)
2015-11-16 09:34:03 -08:00
Walter Su 0f5aae5fd8 HDFS-9410. Some tests should always reset sysout and syserr. Contributed by Xiao Chen.
(cherry picked from commit cccf88480b)
2015-11-13 14:29:25 +08:00
Kihwal Lee 8726a5e020 HDFS-9396. Total files and directories on jmx and web UI on standby is uninitialized. Contributed by Kihwal Lee.
(cherry picked from commit a2a104f3d3)
2015-11-11 15:37:11 -06:00
Xiaoyu Yao f3a075e053 HDFS-9245. Fix findbugs warnings in hdfs-nfs/WriteCtx. Contributed by Mingliang Liu.
(cherry picked from commit 6e4562b844)
2015-11-10 16:51:01 -08:00
Xiaoyu Yao 48a494cdc3 HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 41d3f8899d)
2015-11-10 16:48:06 -08:00
Haohui Mai aaa38581f5 HDFS-9369. Use ctest to run tests for hadoop-hdfs-native-client. Contributed by Haohui Mai. 2015-11-10 14:53:22 -08:00
Zhe Zhang edd4b2d458 HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. Contributed by Xiao Chen.
Change-Id: I9e42f724f27924cf73891425a832de37ce014a1e
2015-11-10 09:55:39 -08:00
Walter Su e5420feb09 HDFS-9401. Fix findbugs warnings in BlockRecoveryWorker. Contributed by Brahma Reddy Battula.
(cherry picked from commit 2fda45b9dc)
2015-11-10 14:54:04 +08:00
Haohui Mai c91bd3aabb HDFS-2261. AOP unit tests are not getting compiled or run. Contributed by Haohui Mai. 2015-11-09 15:26:01 -08:00
Yongjun Zhang 6bca317157 HDFS-9249. NPE is thrown if an IOException is thrown in NameNode constructor. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit 2741a2109b)
2015-11-09 14:12:20 -08:00
Kihwal Lee ceb062f093 HDFS-9383. TestByteArrayManager#testByteArrayManager fails. Contributed by Tsz Wo Nicholas Sze.
(cherry picked from commit ef926b2e38)
2015-11-09 08:57:25 -06:00
cnauroth 385dcc4064 HDFS-9394. branch-2 hadoop-hdfs-client fails during FileSystem ServiceLoader initialization, because HftpFileSystem is missing. Contributed by Mingliang Liu.
(cherry picked from commit 1ee2db5c41)
2015-11-07 07:39:45 -08:00
Tsz-Wo Nicholas Sze 9766fc988f HDFS-9398. Make ByteArraryManager log message in one-line format. Contributed by Mingliang Liu 2015-11-07 22:22:36 +08:00
Arpit Agarwal 1e0746e756 HDFS-9379. Make NNThroughputBenchmark support more than 10 datanodes. (Contributed by Mingliang Liu) 2015-11-06 18:59:03 -08:00
Kihwal Lee 481e7248de HDFS-9318. considerLoad factor can be improved. Contributed by Kuhu Shukla.
(cherry picked from commit bf6aa30a15)
2015-11-06 14:08:10 -06:00
Yongjun Zhang 41d0d9a32c HDFS-9236. Missing sanity check for block size during block recovery. (Tony Wu via Yongjun Zhang)
(cherry picked from commit b64242c0d2)
2015-11-06 11:26:41 -08:00
Arpit Agarwal 33908c6f52 HDFS-6481. DatanodeManager#getDatanodeStorageInfos() should check the length of storageIDs. (Contributed by szetszwo) 2015-11-06 10:19:00 -08:00
cnauroth 75bcc8bcd8 HDFS-9384. TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server. Contributed by Chris Nauroth.
(cherry picked from commit 66c0967310)
2015-11-05 22:07:34 -08:00
Yongjun Zhang 1bb89cbac7 HDFS-9377. Fix findbugs warnings in FSDirSnapshotOp. (Mingliang Liu via Yongjun Zhang)
(cherry picked from commit 286cc6483f)
2015-11-05 14:54:29 -08:00
Lei Xu 489c920c9a HDFS-9282. Make data directory count and storage raw capacity related tests FsDataset-agnostic. (Tony Wu via lei)
(cherry picked from commit 5a1a54c9f5d9a8f7585000fe33ef106e62dbb43a)
2015-11-05 10:46:46 -08:00
Lei Xu 7a2a17cb5f HDFS-9363. Add fetchReplica to FsDatasetTestUtils to return FsDataset-agnostic replica. (Tony Wu via lei)
(cherry picked from commit 4976a02841592b61f2d2b40d2aadd5146571322c)
2015-11-05 10:46:46 -08:00
Haohui Mai aa03030e4a HDFS-9372. Remove dead code in DataStorage.recoverTransitionRead. Contributed by Duo Zhang. 2015-11-05 10:00:18 -08:00
Kihwal Lee 37abd0e33b HDFS-4937. ReplicationMonitor can infinite-loop in BlockPlacementPolicyDefault#chooseRandom(). Contributed by Kihwal Lee.
(cherry picked from commit ff47f35dee)
2015-11-05 09:26:53 -06:00
cnauroth 6edd41ac69 HDFS-9378. hadoop-hdfs-client tests do not write logs. Contributed by Chris Nauroth.
(cherry picked from commit c563401796)
2015-11-05 06:48:32 -08:00
Xiaoyu Yao 891be39993 HDFS-9360. Storage type usage isn't updated properly after file deletion. Contributed by Ming Ma.
(cherry picked from commit ea5bb48326)
2015-11-05 01:40:05 -08:00
Lei Xu 8ed1fd2171 HDFS-9331. Modify TestNameNodeMXBean#testNameNodeMXBeanInfo() to account for filesystem entirely allocated for DFS use. (Tony Wu via lei)
(cherry picked from commit e2a5441b06)
2015-11-04 10:28:00 -08:00
Haohui Mai 6d92778191 Revert "HDFS-8855. Webhdfs client leaks active NameNode connections. Contributed by Xiaobing Zhou."
This reverts commit e7203b6954.
2015-11-04 10:22:21 -08:00
Lei Xu 3f2be7b18f HDFS-9007. Fix HDFS Balancer to honor upgrade domain policy. (Ming Ma via lei) 2015-11-04 10:21:58 -08:00
Kihwal Lee c8ffea3db7 fix up CHANGES.txt 2015-11-04 12:15:58 -06:00
Haohui Mai b61aa716e8 HDFS-9357. NN UI renders icons of decommissioned DN incorrectly. Contributed by Surendra Singh Lilhore. 2015-11-04 09:17:01 -08:00
Jason Lowe 8334d86472 Add 2.7.3 release to CHANGES.txt
(cherry picked from commit 0383a3973b)
2015-11-04 16:27:28 +00:00