Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,877 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2528 Commits

Author SHA1 Message Date
Sangjin Lee 8f8f3b1027 Updated the 2.6.2 final release date. 
(cherry picked from commit a4a6b5b4b4)
 2015-10-30 18:51:50 -07:00
Jason Lowe 0af4f222a1 Creating 2.6.3 entries in CHANGES.txt files. 
(cherry picked from commit eadf7b3096)

Conflicts:

	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-mapreduce-project/CHANGES.txt
	hadoop-yarn-project/CHANGES.txt
 2015-10-30 14:58:42 +00:00
Sangjin Lee 1d3bed1f94 Set the release date for 2.6.2 2015-10-28 20:57:16 -07:00
Sangjin Lee 6466ead9e0 Preparing for 2.6.3 development 2015-10-21 10:58:06 -07:00
Vinod Kumar Vavilapalli b99a85139e Creating 2.6.2 entries in CHANGES.txt files. 
(cherry picked from commit 33474939a0)
 2015-09-25 16:00:06 -07:00
Sangjin Lee 4cb7dbaead Preparing for 2.6.2 development: mvn versions:set -DnewVersion=2.6.2 2015-09-25 15:51:13 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 41d19f4740 Release process for 2.6.1: Set the release date for 2.6.1 2015-09-23 15:15:36 -07:00
Vinod Kumar Vavilapalli b4d876d837 Set the release date for 2.6.1-RC1 2015-09-16 12:04:57 -07:00
Vinod Kumar Vavilapalli 075d899d10 Set the release date for 2.6.1-RC0 2015-09-09 15:39:02 -07:00
Vinod Kumar Vavilapalli 3462a00dd2 Preparing for release 2.6.1: mvn versions:set -DnewVersion=2.6.1 2015-09-09 15:29:57 -07:00
Colin Patrick Mccabe adc3405321 HDFS-7552. Change FsVolumeList toString() to fix TestDataNodeVolumeFailureToleration (Liang Xie via Colin P. McCabe) 
(cherry picked from commit a4876c130f)

(cherry picked from commit 22f72c7946)
 2015-09-09 14:59:31 -07:00
yliu 0adcfe6c7e HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) 
(cherry picked from commit 146db49f7f)
 2015-09-08 22:57:36 -07:00
cnauroth c3928b3068 HDFS-7570. SecondaryNameNode need twice memory when calling reloadFromImageFile. Contributed by zhaoyunjiong. 
(cherry picked from commit 85aec75ce5)
(cherry picked from commit 27eecaef10)
 2015-09-08 17:21:58 -07:00
Vinod Kumar Vavilapalli 33537078a8 HDFS-8384. Allow NN to startup if there are files having a lease but are not under construction. Contributed by Jing Zhao. 2015-09-08 16:45:38 -07:00
Colin Patrick Mccabe a976acc02d HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit a4d9acc51d)
(cherry picked from commit 9264b7e119)
 2015-09-08 16:18:30 -07:00
Colin Patrick Mccabe 2e86d1f254 HDFS-8480. Fix performance and timeout issues in HDFS-7929 by using hard-links to preserve old edit logs, instead of copying them. (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 7b424f938c)
(cherry picked from commit cbd11681ce)
 2015-09-08 16:12:45 -07:00
Colin Patrick Mccabe 4363145128 HDFS-7446. HDFS inotify should have the ability to determine what txid it has read up to (cmccabe) 
(cherry picked from commit 75a326aaff)

(cherry picked from commit 06552a15d5)
 2015-09-08 15:32:17 -07:00
Tsz-Wo Nicholas Sze 8ed162bcbf HDFS-7980. Incremental BlockReport will dramatically slow down namenode startup. Contributed by Walter Su 
(cherry picked from commit 4e1f2eb395)
 2015-09-07 11:46:33 -07:00
Vinayakumar B a2434dcf2a HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina) 
(cherry picked from commit 54f83d9bd9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/NameNodeProxies.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestFileCreation.java

(cherry picked from commit 8090a6ee63)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

(cherry picked from commit 04a7000c8d955a134d86967f4b4622ea920e1ea7)
 2015-09-03 14:06:09 -07:00
Jing Zhao 46b9393cab HDFS-7609. Avoid retry cache collision when Standby NameNode loading edits. Contributed by Ming Ma. 
(cherry picked from commit 7817674a3a)
(cherry picked from commit 17fb442a4c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java

(cherry picked from commit fad2a062ddbb955a42dd5a90d64781617287f8df)
 2015-09-03 13:40:45 -07:00
cnauroth 563dbd29eb HDFS-8431. hdfs crypto class not found in Windows. Contributed by Anu Engineer. 
(cherry picked from commit 50eeea1300)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 25db341278)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs.cmd

(cherry picked from commit 77a10e76e99c14cd26ebb3664304f6ed9cc7bf65)
 2015-09-03 12:59:35 -07:00
Kihwal Lee fb096b97cc HDFS-8404. Pending block replication can get stuck using older genstamp. Contributed by Nathan Roberts. 
(cherry picked from commit 8860e352c3)

(cherry picked from commit 536b9ee6d6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestPendingReplication.java

(cherry picked from commit 2d5e60fa12a62463cd54f1b6b0fcb2ccdbd82c42)
 2015-09-03 12:32:49 -07:00
Kihwal Lee b3ca4dfaff HDFS-8245. Standby namenode doesn't process DELETED_BLOCK if the addblock request is in edit log. Contributed by Rushabh S Shah. 
(cherry picked from commit 2d4ae3d18b)

(cherry picked from commit f264a5aeed)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestBlockReplacement.java

(cherry picked from commit 470019e9b88e0fcede926442b91d102b595c7ace)
 2015-09-03 12:19:21 -07:00
Colin Patrick Mccabe 51d5fc62c2 HDFS-7278. Add a command that allows sysadmins to manually trigger full block reports from a DN (cmccabe) 
(cherry picked from commit baf794dc40)

(cherry picked from commit 5f3d967aae)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HDFSCommands.apt.vm

(cherry picked from commit a776ef5ad2876b9acf6cf89824c306783f7759f1)
 2015-09-03 11:43:27 -07:00
Kihwal Lee bbed4e67a9 HDFS-7894. Rolling upgrade readiness is not updated in jmx until query command is issued. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 6f622672b6)

(cherry picked from commit 802a5775f3)
(cherry picked from commit 995382c5234ad6c07f327e5d1f2a1c7e391a0b60)
 2015-09-03 11:38:46 -07:00
Xiaoyu Yao 08006b84f8 HDFS-8219. setStoragePolicy with folder behavior is different after cluster restart. (surendra singh lilhore via Xiaoyu Yao) 
(cherry picked from commit 0100b15501)
(cherry picked from commit e68e8b3b5c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestBlockStoragePolicy.java

(cherry picked from commit b054cb68fa0fc6d1e9e77ac84575731e7d1ec0c7)
 2015-09-02 15:31:46 -07:00
Colin Patrick Mccabe 8770f1f039 HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm (cmccabe) 
(cherry picked from commit a0e0a63209)
(cherry picked from commit 788b76761d)
 2015-09-02 14:38:26 -07:00
Colin Patrick Mccabe 181281cfca HDFS-8070. Pre-HDFS-7915 DFSClient cannot use short circuit on post-HDFS-7915 DataNode (cmccabe) 
(cherry picked from commit a8898445dc)
(cherry picked from commit 3d0385c3c8)
 2015-09-02 14:37:51 -07:00
Colin Patrick Mccabe 9df4b7c10e HDFS-7915. The DataNode can sometimes allocate a ShortCircuitShm slot and fail to tell the DFSClient about it because of a network error (cmccabe) 
(cherry picked from commit bc9cb3e271)
(cherry picked from commit c71b54fa32)
 2015-09-02 14:20:17 -07:00
Jing Zhao b7e7896410 HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Contributed by Jing Zhao. 
(cherry picked from commit fddd55279d)
(cherry picked from commit 38b031d6ba)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java

(cherry picked from commit b4e227e65d3a81150f46aac5e987d759defc1452)
 2015-09-02 11:06:04 -07:00
Kihwal Lee 619f793846 HDFS-8046. Allow better control of getContentSummary. Contributed by Kihwal Lee. 
(cherry picked from commit 285b31e75e)

(cherry picked from commit 7e622076d4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ContentSummaryComputationContext.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirStatAndListingOp.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java

(cherry picked from commit 1ef5e0b18066ca949adcf4c55a41f186c47e7264)
 2015-09-01 21:24:53 -07:00
Arpit Agarwal 31d30e8111 HDFS-8072. Reserved RBW space is not released if client terminates while writing block. (Arpit Agarwal) 
(cherry picked from commit f0324738c9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/extdataset/ExternalReplicaInPipeline.java

(cherry picked from commit de21de7e2243ef8a89082121d838b88e3c10f05b)
 2015-09-01 20:40:09 -07:00
Colin Patrick Mccabe c3f5ea11ec HDFS-7999. FsDatasetImpl#createTemporary sometimes holds the FSDatasetImpl lock for a very long time (sinago via cmccabe) 
(cherry picked from commit 28bebc81db)
(cherry picked from commit a827089905)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

(cherry picked from commit c3a3092c37926eca75ea149c4c061742f6599b40)
 2015-09-01 20:15:20 -07:00
Kihwal Lee bc8728cd27 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
(cherry picked from commit 04ee18ed48)

(cherry picked from commit c4cedfc1d6)
(cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)
 2015-09-01 19:04:52 -07:00
Andrew Wang 0b1e66f01d HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 
(cherry picked from commit 50ee8f4e67)
(cherry picked from commit 2f46ee50bd)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)
 2015-09-01 18:30:18 -07:00
Tsz-Wo Nicholas Sze 9b105b78d4 HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the datanode. Contributed by Brahma Reddy Battula 
(cherry picked from commit fe693b72de)
(cherry picked from commit 4c648779d69a668f5147b183af2d40d45d1227d2)
 2015-09-01 18:25:02 -07:00
yliu 553efd7190 HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 
(cherry picked from commit 90164ffd84)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java

(cherry picked from commit 2c9a7461ec2ceba5885e95bc79f8dcbfd198df60)
 2015-09-01 17:06:37 -07:00
Colin Patrick Mccabe 81a445edf8 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 43b41f2241)
(cherry picked from commit 219eb22c15)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java

(cherry picked from commit 03798416bfe27383c52e4d9f632fe9fa168c6e95)
 2015-09-01 16:55:50 -07:00
Jing Zhao 49ef26b5ac HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 
Committed Ming Ma's 2.6 patch.

(cherry picked from commit 7f0bb5d3fe0db2e6b9354c8d8a1b603f2390184f)
 2015-09-01 16:33:30 -07:00
Colin Patrick Mccabe 9642a861e1 HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 5c1036d598)
(cherry picked from commit eefca23e8c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit c723f3b1bd9eab261ab5edca33c4dae5ce3d0d30)
 2015-09-01 15:35:08 -07:00
Colin Patrick Mccabe 5dbb0325df HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit a17584936c)
(cherry picked from commit 7779f38e68)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit 65ae3e2ff16ce1114a0115ff916837b0173b77f1)
 2015-09-01 15:29:59 -07:00
Jing Zhao b25491dc45 HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit 24db0812be)
(cherry picked from commit 994dadb9ba)
(cherry picked from commit 0bc5c6495a7feb4365af0ce5fe48fc87b7e1749f)
 2015-09-01 15:17:57 -07:00
Jing Zhao 6e090bc53d HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. 
(cherry picked from commit b442aeec95)
(cherry picked from commit 6090f51725)
(cherry picked from commit e1af1ac4e91d36b21df18ce5627e1f69f27f0776)
 2015-09-01 13:36:21 -07:00
Andrew Wang 406ec495bb HDFS-7763. fix zkfc hung issue due to not catching exception in a corner case. Contributed by Liang Xie. 
(cherry picked from commit 7105ebaa9f)
(cherry picked from commit efb7e287f4)
(cherry picked from commit fd70e4db105e140fc3d60042abb3f598c9afd13f)
 2015-09-01 11:30:04 -07:00
cnauroth be4d4c9cbf HDFS-7009. Active NN and standby NN have different live nodes. Contributed by Ming Ma. 
(cherry picked from commit 769507bd7a)
(cherry picked from commit 657a6e389b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java

(cherry picked from commit d5ddc3450f2f49ea411de590ff3de15b5ec4e17c)
 2015-09-01 11:23:26 -07:00
Kihwal Lee 02e0b6e306 HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah. 
(cherry picked from commit 7ae5255a16)

(cherry picked from commit b9157f92fc)

Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/site/resources/image-with-zero-block-size.tar.gz
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java

(cherry picked from commit 1faa44d8f4d7b944e99dd0470ea2638c7653a131)
 2015-08-31 18:32:00 -07:00
Vinayakumar B 6154abfeda HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.(Contributed by Vinayakumar B) 
(cherry picked from commit 3d15728ff5)
(cherry picked from commit a1bf7aecf7)
(cherry picked from commit c1e65de57e8ef760586e28cd37397ea9a7ac7944)
 2015-08-31 17:26:28 -07:00
Kihwal Lee a1c3e6a058 HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang 
(cherry picked from commit 843806d03a)

(cherry picked from commit c428d303f6)
(cherry picked from commit 21d8b22feef2fc9f6793d9c157391b142368e588)
 2015-08-31 15:55:07 -07:00
Arpit Agarwal f81f97eb7c HDFS-7575. Upgrade should generate a unique storage ID for each volume. (Contributed by Arpit Agarwal) 
(cherry picked from commit 1d9d166c0b)
(cherry picked from commit ca8e1b0739b6653833f9bc8990ab126420703f66)
 2015-08-31 15:08:42 -07:00
Kihwal Lee ae288f7291 HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade shutdown command. Contributed by Eric Payne. 
(cherry picked from commit 6bbf9fdd04)

(cherry picked from commit 33534a0c9a)
(cherry picked from commit e9a28251ee46e64e1b99b2dd54b0432bdc0b9578)
 2015-08-31 11:53:24 -07:00