Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,761 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

2514 Commits

Author SHA1 Message Date
Kihwal Lee bc8728cd27 HDFS-7742. Favoring decommissioning node for replication can cause a block to stay 
underreplicated for long periods. Contributed by Nathan Roberts.
(cherry picked from commit 04ee18ed48)

(cherry picked from commit c4cedfc1d6)
(cherry picked from commit c6b68a82adea8de488b255594d35db8e01f5fc8f)
 2015-09-01 19:04:52 -07:00
Andrew Wang 0b1e66f01d HDFS-7960. The full block report should prune zombie storages even if they're not empty. Contributed by Colin McCabe and Eddy Xu. 
(cherry picked from commit 50ee8f4e67)
(cherry picked from commit 2f46ee50bd)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/protocol/TestBlockListAsLongs.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestNameNodePrunesMissingStorages.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestTriggerBlockReport.java

(cherry picked from commit 03d4af39e794dc03d764122077b434d658b6405e)
 2015-09-01 18:30:18 -07:00
Tsz-Wo Nicholas Sze 9b105b78d4 HDFS-7884. Fix NullPointerException in BlockSender when the generation stamp provided by the client is larger than the one stored in the datanode. Contributed by Brahma Reddy Battula 
(cherry picked from commit fe693b72de)
(cherry picked from commit 4c648779d69a668f5147b183af2d40d45d1227d2)
 2015-09-01 18:25:02 -07:00
yliu 553efd7190 HDFS-7930. commitBlockSynchronization() does not remove locations. (yliu) 
(cherry picked from commit 90164ffd84)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFileTruncate.java

(cherry picked from commit 2c9a7461ec2ceba5885e95bc79f8dcbfd198df60)
 2015-09-01 17:06:37 -07:00
Colin Patrick Mccabe 81a445edf8 HDFS-7929. inotify unable fetch pre-upgrade edit log segments once upgrade starts (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit 43b41f2241)
(cherry picked from commit 219eb22c15)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NNUpgradeUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUpgrade.java

(cherry picked from commit 03798416bfe27383c52e4d9f632fe9fa168c6e95)
 2015-09-01 16:55:50 -07:00
Jing Zhao 49ef26b5ac HDFS-7587. Edit log corruption can happen if append fails with a quota violation. Contributed by Jing Zhao. 
Committed Ming Ma's 2.6 patch.

(cherry picked from commit 7f0bb5d3fe0db2e6b9354c8d8a1b603f2390184f)
 2015-09-01 16:33:30 -07:00
Colin Patrick Mccabe 9642a861e1 HDFS-7830. DataNode does not release the volume lock when adding a volume fails. (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 5c1036d598)
(cherry picked from commit eefca23e8c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetTestUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit c723f3b1bd9eab261ab5edca33c4dae5ce3d0d30)
 2015-09-01 15:35:08 -07:00
Colin Patrick Mccabe 5dbb0325df HDFS-7610. Fix removal of dynamically added DN volumes (Lei (Eddy) Xu via Colin P. McCabe) 
(cherry picked from commit a17584936c)
(cherry picked from commit 7779f38e68)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit 65ae3e2ff16ce1114a0115ff916837b0173b77f1)
 2015-09-01 15:29:59 -07:00
Jing Zhao b25491dc45 HDFS-7885. Datanode should not trust the generation stamp provided by client. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit 24db0812be)
(cherry picked from commit 994dadb9ba)
(cherry picked from commit 0bc5c6495a7feb4365af0ce5fe48fc87b7e1749f)
 2015-09-01 15:17:57 -07:00
Jing Zhao 6e090bc53d HDFS-7871. NameNodeEditLogRoller can keep printing 'Swallowing exception' message. Contributed by Jing Zhao. 
(cherry picked from commit b442aeec95)
(cherry picked from commit 6090f51725)
(cherry picked from commit e1af1ac4e91d36b21df18ce5627e1f69f27f0776)
 2015-09-01 13:36:21 -07:00
Andrew Wang 406ec495bb HDFS-7763. fix zkfc hung issue due to not catching exception in a corner case. Contributed by Liang Xie. 
(cherry picked from commit 7105ebaa9f)
(cherry picked from commit efb7e287f4)
(cherry picked from commit fd70e4db105e140fc3d60042abb3f598c9afd13f)
 2015-09-01 11:30:04 -07:00
cnauroth be4d4c9cbf HDFS-7009. Active NN and standby NN have different live nodes. Contributed by Ming Ma. 
(cherry picked from commit 769507bd7a)
(cherry picked from commit 657a6e389b)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestDatanodeProtocolRetryPolicy.java

(cherry picked from commit d5ddc3450f2f49ea411de590ff3de15b5ec4e17c)
 2015-09-01 11:23:26 -07:00
Kihwal Lee 02e0b6e306 HDFS-7788. Post-2.6 namenode may not start up with an image containing inodes created with an old release. Contributed by Rushabh Shah. 
(cherry picked from commit 7ae5255a16)

(cherry picked from commit b9157f92fc)

Conflicts:
  hadoop-hdfs-project/hadoop-hdfs/src/site/resources/image-with-zero-block-size.tar.gz
  hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSImage.java

(cherry picked from commit 1faa44d8f4d7b944e99dd0470ea2638c7653a131)
 2015-08-31 18:32:00 -07:00
Vinayakumar B 6154abfeda HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode.(Contributed by Vinayakumar B) 
(cherry picked from commit 3d15728ff5)
(cherry picked from commit a1bf7aecf7)
(cherry picked from commit c1e65de57e8ef760586e28cd37397ea9a7ac7944)
 2015-08-31 17:26:28 -07:00
Kihwal Lee a1c3e6a058 HDFS-7707. Edit log corruption due to delayed block removal again. Contributed by Yongjun Zhang 
(cherry picked from commit 843806d03a)

(cherry picked from commit c428d303f6)
(cherry picked from commit 21d8b22feef2fc9f6793d9c157391b142368e588)
 2015-08-31 15:55:07 -07:00
Arpit Agarwal f81f97eb7c HDFS-7575. Upgrade should generate a unique storage ID for each volume. (Contributed by Arpit Agarwal) 
(cherry picked from commit 1d9d166c0b)
(cherry picked from commit ca8e1b0739b6653833f9bc8990ab126420703f66)
 2015-08-31 15:08:42 -07:00
Kihwal Lee ae288f7291 HDFS-7533. Datanode sometimes does not shutdown on receiving upgrade shutdown command. Contributed by Eric Payne. 
(cherry picked from commit 6bbf9fdd04)

(cherry picked from commit 33534a0c9a)
(cherry picked from commit e9a28251ee46e64e1b99b2dd54b0432bdc0b9578)
 2015-08-31 11:53:24 -07:00
cnauroth 8382e52c94 HDFS-7596. NameNode should prune dead storages from storageMap. Contributed by Arpit Agarwal. 
(cherry picked from commit ef3c3a832c)
(cherry picked from commit 75e4e55e12)
(cherry picked from commit cc637d6ece64dfeb89e78c7e9766836149e098be)
 2015-08-31 11:26:48 -07:00
Jing Zhao c8ce11d067 HDFS-7182. JMX metrics aren't accessible when NN is busy. Contributed by Ming Ma. 
(cherry picked from commit 4b589e7cfa)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystemMBean.java

(cherry picked from commit 96f0813c5d6140aabe7b2837f30971936276e689)
 2015-08-30 21:21:05 -07:00
cnauroth e6b52fc3e9 HDFS-7579. Improve log reporting during block report rpc failure. Contributed by Charles Lamb. 
(cherry picked from commit 7e2d9a3242)
(cherry picked from commit f0acb7c2a2)
(cherry picked from commit 33fb7b45195a89b6464e4a1cb3fbf6bbad2bcecb)
 2015-08-30 21:17:34 -07:00
Colin Patrick Mccabe c2e00e0012 HDFS-7531. Improve the concurrent access on FsVolumeList (Lei Xu via Colin P. McCabe) 
(cherry picked from commit 3b173d9517)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java

(cherry picked from commit dda1fc169d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsVolumeList.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/TestFsDatasetImpl.java

(cherry picked from commit ba28192f9d5a8385283bd717bca494e6981d378f)
 2015-08-30 20:11:41 -07:00
Colin Patrick Mccabe 70f33f5eff HDFS-7035. Make adding a new data directory to the DataNode an atomic operation and improve error handling (Lei Xu via Colin P. McCabe) 
(cherry picked from commit a9331fe9b0)

(cherry picked from commit ec2621e907)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java

(cherry picked from commit d79a5849999cdb0bc315938b80ed71b4f2dcb720)
 2015-08-30 12:13:33 -07:00
Arpit Agarwal 46e9926ce3 HDFS-8486. Addendum patch to fix backporting bug. 
(cherry picked from commit 6c9e26609ceac5d16434cc7f4a1013e6faacd92f)
 2015-08-30 12:02:23 -07:00
Kihwal Lee 8549891cb3 HDFS-7213. processIncrementalBlockReport performance degradation. 
Contributed by Eric Payne.
(cherry picked from commit e226b5b40d)

(cherry picked from commit 946463efef)
(cherry picked from commit f94aa4d25c2f96faf5164e807c2c3eb031e9a1fe)
 2015-08-30 11:46:16 -07:00
Andrew Wang 0898c21e24 HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang) 
(cherry picked from commit 406c09ad11)
(cherry picked from commit 2e15754a92)
(cherry picked from commit 014d07de2e9b39be4b6793f0e09fcf8548570ad5)
 2015-08-27 18:48:17 -07:00
Tao Luo 6d2f80107f HDFS-7263. Snapshot read can reveal future bytes for appended files. Contributed by Tao Luo. 
(cherry picked from commit 8bfef59029)

(cherry picked from commit 3827a1acdbc4f9fec3179dcafa614734b5fa31bc)
 2015-08-27 17:53:06 -07:00
Colin Patrick Mccabe d3a7769bff HDFS-7235. DataNode#transferBlock should report blocks that don't exist using reportBadBlock (yzhang via cmccabe) 
(cherry picked from commit ac9ab037e9)

(cherry picked from commit 842a54a5f6)
(cherry picked from commit 1aa9e34c5106c496ffd390f6b2c822d387fb1908)
 2015-08-27 17:47:39 -07:00
Ming Ma dda8e0e328 HDFS-7314. When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient. (mingma) 
(cherry picked from commit fbd88f1062)
(cherry picked from commit 516bbf1c20)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSClientRetries.java

(cherry picked from commit fb1bf424bdad20fff7ab390ce75c4bec558e7e6d)
 2015-08-27 17:38:06 -07:00
Arpit Agarwal aa2303088f HDFS-8486. DN startup may cause severe data loss. (Contributed by Daryn Sharp) 2015-08-06 15:17:45 -07:00
Akira Ajisaka 0daf9dd552 Revert "Merge r1588258 from trunk, HDFS-6153. Document "fileId" and "childrenNum" fields in the FileStatus Json schema" 
This reverts commit 641cbc0feb.
 2015-04-16 16:40:31 -05:00
Arpit Agarwal ef212fbe7e HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. (Contributed by Arpit Agarwal) 2015-02-04 16:26:00 -08:00
Tsz-Wo Nicholas Sze 4f5d330623 HDFS-3443. Fix NPE when namenode transition to active during startup by adding checkNNStartup() in NameNodeRpcServer. Contributed by Vinayakumar B 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java
 2015-01-21 11:54:26 -08:00
Colin Patrick Mccabe 51d49ffffd HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume (cmccabe) 
(cherry picked from commit 8fa265a290)
(cherry picked from commit bcaf15e2fa)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataStorage.java
 2014-12-19 13:25:33 -08:00
arp 91042c950f HDFS-7503. Namenode restart after large deletions can cause slow processReport (Arpit Agarwal) 2014-12-10 23:38:55 -08:00
Colin Patrick Mccabe d93bfbace4 HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes (Noah Lorang via Colin P. McCabe) 
(cherry picked from commit d8352b9b2b)
(cherry picked from commit a037d6030b)
 2014-12-09 10:59:49 -08:00
Colin Patrick Mccabe 575d7bd5b0 HDFS-4882. Prevent the Namenode's LeaseManager from looping forever in checkLeases (Ravi Prakash via Colin P. McCabe) 
(cherry picked from commit daacbc18d7)
 2014-11-24 11:42:46 -08:00
cnauroth 2379932968 HDFS-7425. NameNode block deletion logging uses incorrect appender. Contributed by Chris Nauroth. 
(cherry picked from commit b8c094b075)
(cherry picked from commit 8d8eb8dcec)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-21 16:58:06 -08:00
Karthik Kambatla 5913c5143b Update release date for 2.5.2 
(cherry picked from commit 1e9a3f42ff)
 2014-11-21 14:35:06 -08:00
Arun C. Murthy 98a4538cf4 Preparing to release hadoop-2.6.0-rc1. 
(cherry picked from commit 9ff7315cbe)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-mapreduce-project/CHANGES.txt
	hadoop-yarn-project/CHANGES.txt
 2014-11-13 13:05:50 -08:00
cnauroth ed2a4db6d4 HDFS-7385. ThreadLocal used in FSEditLog class causes FSImage permission mess up. Contributed by jiangyu. 
(cherry picked from commit b0a41de68c)
(cherry picked from commit 4e9001530f)
 2014-11-13 12:37:48 -08:00
Haohui Mai 59b2a74ea9 Fix CHANGES.txt to keep it consistent with trunk. 2014-11-12 20:13:38 -08:00
Arun C. Murthy 797e62defa HDFS-7391. Renable SSLv2Hello in HttpFS. Contributed by Robert Kanter. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-12 19:00:50 -08:00
Arun C. Murthy 42849e5077 HDFS-7387. Merging to branch-2.6 for hadoop-2.6.0-rc1. 
(cherry picked from commit 09f36b1688)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-12 11:51:11 -08:00
Brandon Li d0ca88c5a1 HDFS-7387. NFS may only do partial commit due to a race between COMMIT and write. Contributed by Brandon Li 
(cherry picked from commit 99d9d0c2d1)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java

(cherry picked from commit 0c3b7887e6)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-12 11:50:04 -08:00
Karthik Kambatla ba01844260 Set the release date for 2.5.2 
(cherry picked from commit 68a0508728)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt

(cherry picked from commit 92de44c2c5)
 2014-11-10 15:24:14 -08:00
Arun C. Murthy 60584c732f Preparing to release hadoop-2.6.0: Set version in branch-2.6 to 2.6.1-SNAPSHOT. 2014-11-09 19:21:11 -08:00
Arun C. Murthy be31db82f7 Updated CHANGES.txt to reflect hadoop-2.6.0 release dates. 
(cherry picked from commit 770cc14442)
 2014-11-09 18:53:55 -08:00
Suresh Srinivas c98fcd1d3b HDFS-7383. DataNode.requestShortCircuitFdsForRead may throw NullPointerException. Contributed by Tsz Wo Nicholas Sze. 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-09 18:18:58 -08:00
Arun C. Murthy 85531834b3 HADOOP-11286. Copied LimitInputStream from guava-0.14 to hadoop to avoid issues with newer versions of guava in applications. Contributed by Christopher Tubbs. 
(cherry picked from commit 6caa8100d5)
 2014-11-09 05:25:01 -08:00
cnauroth 539ef7fc13 HDFS-7382. DataNode in secure mode may throw NullPointerException if client connects before DataNode registers itself with NameNode. Contributed by Chris Nauroth. 
(cherry picked from commit 9ba8d8c7eb)
(cherry picked from commit 98b81c6624)
 2014-11-08 22:31:22 -08:00