Commit Graph

3845 Commits

Author SHA1 Message Date
Karthik Kambatla ee4ace3edf HDFS-9637. Tests for RollingFileSystemSink. (Daniel Templeton via kasha) 2016-02-11 16:45:31 -08:00
Andrew Wang d489570755 HDFS-9788. Incompatible tag renumbering in HeartbeatResponseProto.
(cherry picked from commit aeb13ef2ab)
2016-02-10 15:26:14 -08:00
Akira Ajisaka a60a0a7128 HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit 802a7edfae)
2016-02-10 23:50:04 +09:00
Akira Ajisaka 15ef3a85d9 Revert "HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta."
This reverts commit e4c01b8b1c.
2016-02-10 23:45:15 +09:00
Uma Maheswara Rao G 04ecce1ca9 HDFS-9779 . TestReplicationPolicyWithNodeGroup NODE variable picks wrong rack value. Contributed by Kuhu Shukla.
(cherry picked from commit a7fce9ab41)
2016-02-09 15:16:53 -08:00
Allen Wittenauer 73b195eccc HDFS-9760. WebHDFS AuthFilter cannot be configured with custom AltKerberos auth handler (Ryan Sasson via aw) 2016-02-09 14:16:12 -08:00
Akira Ajisaka cb53dfcc95 HDFS-9784. Example usage is not correct in Transparent Encryption document. Contributed by Ohnishi Takashi.
(cherry picked from commit 60d2011b7c)
2016-02-10 03:30:44 +09:00
Walter Su ac856893e7 Move HDFS-9347 from 2.8.0 to 2.6.5 in CHANGES.txt. 2016-02-09 10:53:45 +08:00
zhezhang ef9a81e875 HDFS-9244. Support nested encryption zones.
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
2016-02-08 16:31:41 -08:00
Uma Maheswara Rao G 44a020ed81 HDFS-9777. Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh)
(cherry picked from commit bffaa38a91)
2016-02-08 14:00:47 -08:00
Kihwal Lee 494cfd1dc4 HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su.
(cherry picked from commit 193d27de0a)
2016-02-08 12:17:07 -06:00
Uma Maheswara Rao G 82d4b2cf27 HDFS-9713. DataXceiver#copyBlock should return if block is pinned. Contributed by Uma Maheswara Rao G.
(cherry picked from commit aea31eee78)
2016-02-07 20:55:58 -08:00
Akira Ajisaka e4c01b8b1c HDFS-9686. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta.
(cherry picked from commit fe124da5ff)
2016-02-06 19:32:50 +09:00
cnauroth a7232f5e1d HDFS-9761. Rebalancer sleeps too long between iterations. Contributed by Mingliang Liu.
(cherry picked from commit c6497949e8)
2016-02-05 15:07:59 -08:00
Tsz-Wo Nicholas Sze 117397b89f HDFS-9726. Refactor IBR code to a new class. 2016-02-05 23:20:16 +08:00
Haohui Mai e62ca14856 HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse ObjectMapper. Contributed by Akira Ajisaka. 2016-02-04 11:35:34 -08:00
Kihwal Lee e26c58fb4e HDFS-9730. Storage ID update does not happen when there is a layout change. Contributed by Tsz Wo Nicholas Sze.
(cherry picked from commit 496f33de0c)
2016-02-04 13:21:30 -06:00
Vinayakumar B 6ee66be2fa HDFS-9629. Update the footer of Web UI to show year 2016 (Contributed by Xiao Chen and Vinayakumar B)
(cherry picked from commit ddcd6954d03d3caadc8b69f7b886a3a17c3c9d0c)
2016-02-04 13:38:26 +05:30
Konstantin V Shvachko 4a0550537c HDFS-9503. Use generic option -fs for NNThroughputBenchmark instead of -namenode. Contributed by Mingliang Liu. 2016-02-03 14:51:41 -08:00
Konstantin V Shvachko 6d1f9a6b4c Move HDFS-9601 to branch-2.8.0 in CHANGES.txt. (shv) 2016-02-03 14:36:25 -08:00
Jing Zhao 459c31de4a HDFS-9748. Avoid duplication in pendingReplications when addExpectedReplicasToPending is called twice. Contributed by Walter Su.
(cherry picked from commit 7badf15604)
2016-02-03 13:32:19 -08:00
Vinayakumar B 6941e2ccbf HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() (Contributed by Chang Li)
(cherry picked from commit eb2fb943fd)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/DFSTestUtil.java
2016-02-03 11:21:08 +05:30
Vinayakumar B e8a9a69a27 HDFS-9739. DatanodeStorage.isValidStorageId() is broken (Contributed by Mingliang Liu)
(cherry picked from commit d6b1acb940)
2016-02-03 10:33:55 +05:30
Vinayakumar B 8577c0b6a2 HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed by Lei (Eddy) Xu)
(cherry picked from commit 04375756a5)
2016-02-03 07:35:47 +05:30
Junping Du 3583c77ee8 Adding release 2.6.5 to CHANGES.txt 2016-02-02 18:09:17 -08:00
Colin Patrick Mccabe 60d3a3c30b HDFS-9669. TcpPeerServer should respect ipc.server.listen.queue.size (Elliot Clark via cmccabe)
(cherry picked from commit 2da03b48eb)
2016-02-02 11:04:33 -08:00
Lei Xu 2f1d1c67f0 HDFS-9721. Allow Delimited PB OIV tool to run upon fsimage that contains INodeReference. (Xiao Chen via lei)
(cherry picked from commit 9d494f0c0e)
2016-02-02 10:42:24 -08:00
Vinayakumar B 80dd039b6e HDFS-9718. HAUtil#getConfForOtherNodes should unset independent generic keys before initialize (Contributed by DENG FEI)
(cherry picked from commit 4d7055e0c8)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HAUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAConfiguration.java
2016-02-02 12:56:31 +05:30
Yongjun Zhang fc8d9cc758 HDFS-9406. FSImage may get corrupted after deleting snapshot. (Contributed by Jing Zhao, Stanislav Antic, Vinayakumar B, Yongjun Zhang)
(cherry picked from commit 34ab50ea92)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java
2016-02-01 14:09:52 -08:00
Lei Xu d2f8e6c269 HDFS-9701. DN may deadlock when hot-swapping under load. (Xiao Chen via lei)
(cherry picked from commit 12a4ebb4471f78b642e12c2ec0784c20f62cca3c)
2016-02-01 12:56:00 -08:00
Xiaoyu Yao 7cc3b1c7a3 HDFS-9210. Fix some misuse of %n in VolumeScanner#printStats. Contributed by Xiaoyu Yao.
(cherry picked from commit b11008d54d)
2016-02-01 11:57:11 -08:00
Steve Loughran 7786e97df2 HDFS-9708. FSNamesystem.initAuditLoggers() doesn't trim classnames (Mingliang Liu via stevel) 2016-02-01 16:01:32 +00:00
Vinayakumar B 71374cca9d HDFS-9566. Remove expensive 'BlocksMap#getStorages(Block b, final DatanodeStorage.State state)' method (Contributed by Daryn Sharp)
(cherry picked from commit e418bd1fb0)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
2016-02-01 13:32:00 +05:30
Yongjun Zhang 2fb711bde9 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit eddd823cd6)
2016-01-29 12:13:03 -08:00
Yongjun Zhang a24898fd49 HDFS-9706. Log more details in debug logs in BlockReceiver's constructor. (Xiao Chen via Yongjun Zhang)
(cherry picked from commit 8ee060311c)
2016-01-28 23:13:41 -08:00
Colin Patrick Mccabe dc65479fc0 HDFS-7764. DirectoryScanner shouldn't abort the scan if one directory had an error (Rakesh R via cmccabe)
(cherry picked from commit f67149ab08)
2016-01-28 20:12:18 -08:00
Tsz-Wo Nicholas Sze d37ecbf721 HDFS-9654. Code refactoring for HDFS-8578. 2016-01-28 10:56:39 +08:00
Tsz-Wo Nicholas Sze b10d8ced21 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-28 10:42:40 +08:00
Jing Zhao d7982222fa HDFS-9677. Rename generationStampV1/generationStampV2 to legacyGenerationStamp/generationStamp. Contributed by Mingliang Liu. 2016-01-27 15:51:10 -08:00
Zhe Zhang 2056dd7d3a HDFS-9541. Add hdfsStreamBuilder API to libhdfs to support defaultBlockSizes greater than 2 GB. Contributed by Colin Patrick McCabe.
Change-Id: Ifce1b9be534dc8f7e9d2634cd60e423921b9810f
2016-01-26 11:26:31 -08:00
Tsz-Wo Nicholas Sze c59c142f47 HDFS-9690. ClientProtocol.addBlock is not idempotent after HDFS-8071. 2016-01-26 14:56:23 +08:00
Jitendra Pandey 4334976187 HDFS-9672. o.a.h.hdfs.TestLeaseRecovery2 fails intermittently. Contributed by Mingliang Liu. 2016-01-25 17:06:36 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 97f6cfa94b Release process for 2.7.2: Set the release date for 2.7.2
(cherry picked from commit ec4d2d9f40)
2016-01-25 15:47:02 -08:00
Arpit Agarwal 0dbaef4377 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:16:45 -08:00
Eric Yang da61953a0b HDFS-9653. Added blocks pending deletion report to dfsadmin.
(Weiwei Yang via eyang)
2016-01-24 14:26:46 -08:00
Kihwal Lee b5db1d4410 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:19:23 -06:00
Akira Ajisaka 4fe28a644e HDFS-9682. Fix a typo "aplication" in HttpFS document. Contributed by Weiwei yang.
(cherry picked from commit 6831634682)
2016-01-22 20:12:39 +09:00
Masatake Iwasaki 73ef77c12f HDFS-9618. Fix mismatch between log level and guard in BlockManager#computeRecoveryWorkForBlocks (iwasakims) 2016-01-22 14:12:06 +09:00
Masatake Iwasaki 245c3728ef HDFS-9601. NNThroughputBenchmark.BlockReportStats should handle NotReplicatedYetException on adding block (iwasakims) 2016-01-22 12:39:39 +09:00
Zhe Zhang 22caeb1c0a HDFS-9674. The HTrace span for OpWriteBlock should record the maxWriteToDisk time. Contributed by Colin McCabe.
Change-Id: I9bf3f3bcd57f5880189ad7c160f3dd66f97d904b
2016-01-21 13:26:54 -08:00
Jing Zhao c19bdc1926 HDFS-9542. Move BlockIdManager from FSNamesystem to BlockManager. Contributed by Jing Zhao.
(cherry picked from commit c304890c8c)
2016-01-21 11:37:24 -08:00
Kihwal Lee 3d8d8f1497 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne.
(cherry picked from commit 7b70500484574a565dd8cd5c7d8b5bc7c6d91154)
2016-01-21 09:33:30 -06:00
Vinayakumar B 845acfd96c HDFS-9625. set replication for empty file failed when set storage policy (Contributed by DENG FEI)
(cherry picked from commit b7372b7166)
2016-01-21 19:18:36 +05:30
Lei Xu c57f55223c HDFS-9655. NN should start JVM pause monitor before loading fsimage. (John Zhuge via Lei (Eddy) Xu)
(cherry picked from commit 2ec438e8f7)
2016-01-20 14:26:47 -08:00
Colin Patrick Mccabe c56044b9ee HDFS-9576: HTrace: collect position/length information on read operations (zhz via cmccabe)
(cherry picked from commit 7905788db9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
2016-01-20 11:27:51 -08:00
Vinayakumar B 252bf56e59 HDFS-9661. Deadlock in DN.FsDatasetImpl between moveBlockAcrossStorage and createRbw (Contributed by ade)
(cherry picked from commit 1425578690)
2016-01-20 21:05:44 +05:30
Yongjun Zhang a5ee4c09de HDFS-6054. MiniQJMHACluster should not use static port to avoid binding failure in unit test. (Yongjun Zhang)
(cherry picked from commit 57d0a94305)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniQJMHACluster.java
2016-01-19 23:13:46 -08:00
Akira Ajisaka 3a8d372518 HDFS-9623. Update example configuration of block state change log in log4j.properties. Contributed by Masatake Iwasaki.
(cherry picked from commit 92c5f565fd)
2016-01-19 13:31:48 +09:00
Tsz-Wo Nicholas Sze 87779a4478 In CHANGES.txt, move HDFS-9294 to Release 2.6.4. 2016-01-16 23:46:19 +08:00
Andrew Wang 46323d7744 HDFS-9624. DataNode start slowly due to the initial DU command operations. (Lin Yiqun via wang)
(cherry picked from commit c07f7fa8ff)
2016-01-15 11:28:55 -08:00
Yongjun Zhang ee012c1f6f HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit a9c69ebeb7)
2016-01-15 10:10:28 -08:00
Arpit Agarwal aadcaf4f32 HDFS-9415. Document dfs.cluster.administrators and dfs.permissions.superusergroup. (Contributed by Xiaobing Zhou) 2016-01-14 14:18:00 -08:00
Yongjun Zhang cfeecf113b HDFS-9648. TestStartup.testImageChecksum is broken by HDFS-9569's message change. (Wei-Chiu Chuang via Yongjun Zhang)
(cherry picked from commit 817cc1f02a)
2016-01-14 12:45:18 -08:00
Colin Patrick Mccabe cea1c71087 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe)
(cherry picked from commit b1ed28fa77)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
2016-01-14 11:13:13 -08:00
Junping Du 586efcf3c8 Move HDFS-8326 out of CHANGES.txt entry for branch-2.7.2. 2016-01-14 06:32:52 -08:00
Junping Du 62eabec549 Add missing CHANGES.txt entry for HDFS-8767.
(cherry picked from commit e50dd19a9c)

Conflicts:

	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2016-01-14 06:13:02 -08:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 1fba4051f8 HADOOP-5323. Trash documentation should describe its directory structure and configurations. (Weiwei Yang via ozawa)
Moved entry from HDFS CHANGES.txt to that of common.

(cherry picked from commit cb73c753da)
2016-01-13 14:56:08 -08:00
Yongjun Zhang 5e2416b901 HDFS-9569. Log the name of the fsimage being loaded for better supportability. (Yongjun Zhang)
(cherry picked from commit 25051c3bd0)
2016-01-12 09:24:54 -08:00
Jing Zhao 313f03bfda HDFS-9621. Consolidate FSDirStatAndListingOp#createFileStatus to let its INodesInPath parameter always include the target INode. Contributed by Jing Zhao. 2016-01-11 22:48:36 -08:00
Xiaoyu Yao c5f63e637a Correct commit message for HDFS-9584
(cherry picked from commit 103d3cfc4e)
2016-01-11 19:00:17 -08:00
Xiaoyu Yao 521f60c094 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore.
(cherry picked from commit c2e2e13455)
2016-01-11 17:12:43 -08:00
Arpit Agarwal 68a32a79b5 HDFS-9639. Inconsistent Logging in BootstrapStandby. (Contributed by Xiaobing Zhou) 2016-01-11 15:09:43 -08:00
Zhe Zhang 17dfa04be6 HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng.
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2016-01-11 09:53:20 -08:00
Lei Xu 699d420497 HDFS-9493. Test o.a.h.hdfs.server.namenode.TestMetaSave fails in trunk. (Tony Wu via lei)
(cherry picked from commit fd8065a763)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetaSave.java
2016-01-08 15:10:41 -08:00
Zhe Zhang 40dabae236 HDFS-9626. TestBlockReplacement#testBlockReplacement fails occasionally. Contributed by Xiao Chen.
Change-Id: I5cee398dfc361121d8b666ffd1203cc14a4ace24
2016-01-08 13:19:14 -08:00
Kihwal Lee 8d04b7c272 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee.
(cherry picked from commit 38c4c14472)
2016-01-08 11:15:12 -06:00
Vinayakumar B 08f39bd5d1 HDFS-9619. SimulatedFSDataset sometimes can not find blockpool for the correct namenode (Contributed by Wei-Chiu Chuang)
(cherry picked from commit 6702e7d66a)
2016-01-07 14:20:39 +05:30
Vinayakumar B aa710bd461 HDFS-9600. do not check replication if the block is under construction (Contributed by Phil Yang)
(cherry picked from commit 34cd7cd765)
2016-01-07 11:30:57 +05:30
Haohui Mai 71006d5c22 HDFS-9047. Retire libwebhdfs. Contributed by Haohui Mai. 2016-01-06 16:13:43 -08:00
Arpit Agarwal 2e6990cdf2 HDFS-9498. Move code that tracks blocks with future generation stamps to BlockManagerSafeMode. (Contributed by Mingliang Liu) 2016-01-06 10:44:51 -08:00
Haohui Mai 150c9a80ee HDFS-9605. Add links to failed volumes to explorer.html in HDFS Web UI. Contributed by Archana T. 2016-01-05 13:56:08 -08:00
Haohui Mai 79dc184224 HDFS-7779. Support changing ownership, group and replication in HDFS Web UI. Contributed by Ravi Prakash. 2016-01-05 13:44:40 -08:00
Uma Mahesh cdd796d199 HADOOP-12658. Clear javadoc and check style issues around DomainSocket. Contributed by Kai Zheng
(cherry picked from commit 778146eaae)
2016-01-04 15:04:46 -08:00
Akira Ajisaka 247ab92464 Set release date for 2.6.3
(cherry picked from commit db99e30f67)
2016-01-04 16:12:52 +09:00
Akira Ajisaka d98cfe611f HADOOP-11252. RPC client does not time out by default. Contributed by Wilfred Spiegelenburg and Masatake Iwasaki.
(cherry picked from commit 64ae85fd2e)
2016-01-04 14:32:08 +09:00
cnauroth 1f8162b74e HDFS-7553. fix the TestDFSUpgradeWithHA due to BindException. Contributed by Xiao Chen. 2015-12-29 10:57:24 -08:00
Benoy Antony 42160d358a HDFS-9597. BaseReplicationPolicyTest should update data node stats after adding a data node. Contributed by Wei-Chiu Chuang.
(cherry picked from commit 8cfd672397)
2015-12-25 08:57:34 -08:00
Benoy Antony 0d249a5e28 HDFS-9034. StorageTypeStats Metric should not count failed storage. Contributed by Surendra Singh Lilhore.
(cherry picked from commit df83230948)
2015-12-22 15:29:36 -08:00
Kihwal Lee 131260f0a7 HDFS-7163. WebHdfsFileSystem should retry reads according to the configured retry policy. Contributed by Eric Payne.
(cherry picked from commit 867048c3e4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
2015-12-22 14:19:12 -06:00
cnauroth 9b039a5004 HDFS-9458. TestBackupNode always binds to port 50070, which can cause bind failures. Contributed by Xiao Chen.
(cherry picked from commit 5c0ff69618)
2015-12-22 10:29:40 -08:00
Colin Patrick Mccabe a56c5ef636 HDFS-9589. Block files which have been hardlinked should be duplicated before the DataNode appends to the them (cmccabe)
(cherry picked from commit bb540ba85a)
2015-12-22 09:37:30 -08:00
Akira Ajisaka 12caea3274 Move HDFS-8914 from 2.8.0 to 2.7.3 in CHANGES.txt.
(cherry picked from commit 2cb5afffc4)
2015-12-21 21:53:44 +09:00
Akira Ajisaka 7526a2a005 HDFS-9505. HDFS Architecture documentation needs to be refreshed. Contributed by Masatake Iwasaki.
(cherry picked from commit fa544020f6)
2015-12-21 21:23:28 +09:00
Uma Mahesh fc9e130825 YARN-4480. Clean up some inappropriate imports. (Kai Zheng via umamahesh)
(cherry picked from commit 0f82b5d878)
2015-12-19 23:17:33 -08:00
cnauroth dd91cf9fa5 HADOOP-12570. HDFS Secure Mode Documentation updates. Contributed by Arpit Agarwal.
(cherry picked from commit 932956458d)
2015-12-18 16:25:06 -08:00
Zhe Zhang 223b47b398 HDFS-9347. Invariant assumption in TestQuorumJournalManager.shutdown() is wrong. Contributed by Wei-Chiu Chuang.
Change-Id: Idde8522e11c77f82b376e0c90a9bf33d163d5cb9
2015-12-18 16:05:21 -08:00
Colin Patrick Mccabe 31d2fb91fc HDFS-9350. Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe)
(cherry picked from commit e63388fdf2)
2015-12-18 10:05:12 -08:00
Andrew Wang 71fb205735 HDFS-8477. describe dfs.ha.zkfc.port in hdfs-default.xml. Contributed by Kanaka Kumar Avvaru.
(cherry picked from commit 058cfddebb)
2015-12-18 09:14:23 -08:00
Vinayakumar B c887bcd1f0 HDFS-9393. After choosing favored nodes, choosing nodes for remaining replicas should go through BlockPlacementPolicy (Contributed by J.Andreina)
(cherry picked from commit bfadf11b36)
2015-12-18 11:39:46 +05:30
cnauroth 7e5306e795 HDFS-9571. Fix ASF Licence warnings in Jenkins reports. Contributed by Brahma Reddy Battula.
(cherry picked from commit 6da9deec20)
2015-12-17 16:26:08 -08:00