Commit Graph

2990 Commits

Author SHA1 Message Date
Tsz-Wo Nicholas Sze 9ab89a9266 HDFS-8278. When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size. 2015-08-17 18:42:41 -07:00
yliu 076c688780 HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) 2015-08-18 09:25:03 +08:00
Arpit Agarwal 85cb596e2e HDFS-8880. NameNode metrics logging. (Arpit Agarwal) 2015-08-17 16:55:24 -07:00
Colin Patrick Mccabe 39326ba71c HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via Colin P. McCabe) 2015-08-17 15:24:56 -07:00
Colin Patrick Mccabe 08508ca1c8 HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe)
(cherry picked from commit c77bd6af16)
2015-08-17 12:07:16 -07:00
Haohui Mai 55f7097900 HDFS-8801. Convert BlockInfoUnderConstruction as a feature. Contributed by Jing Zhao. 2015-08-17 11:28:51 -07:00
Haohui Mai 4e712faac2 HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. Contributed by Haohui Mai. 2015-08-17 11:04:17 -07:00
Xiaoyu Yao 27ccbd51f6 HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. Contributed by Anu Engineer.
(cherry picked from commit a7862d5fe4)
2015-08-17 10:23:56 -07:00
Andrew Wang 9dcb426819 HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging.
(cherry picked from commit 3fe758436041d55da1ccde4d5109da38d1f5110f)
2015-08-17 10:16:37 -07:00
Jing Zhao 2edfc3ce71 HDFS-8891. HDFS concat should keep srcs order. Contributed by Yong Zhang.
(cherry picked from commit dc7a061668)
2015-08-14 14:43:33 -07:00
Tsz-Wo Nicholas Sze 61b9e5f7ff HDFS-8824. Do not use small blocks for balancing the cluster. 2015-08-14 13:08:57 -07:00
Arpit Agarwal b094f8b589 HDFS-7649. Multihoming docs should emphasize using hostnames in configurations. (Contributed by Brahma Reddy Battula) 2015-08-14 13:08:19 -07:00
Xiaoyu Yao 17256e2ae8 HDFS-8565. Typo in dfshealth.html - Decomissioning. (nijel via xyao)
(cherry picked from commit 1569228ec9)
2015-08-14 12:35:17 -07:00
Vinayakumar B 0b6574380c HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina)
Moved to 2.6.1

(cherry picked from commit 84bf71295a)
2015-08-14 16:13:47 +05:30
Vinayakumar B 15760a148b HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang)
Moved to 2.6.1

(cherry picked from commit 08bd4edf40)
2015-08-14 12:38:23 +05:30
Vinayakumar B 9839abe1a8 HDFS-7263. Snapshot read can reveal future bytes for appended files. Contributed by Tao Luo.
Moved CHANGES.txt entry to 2.6.1

(cherry picked from commit fa2641143c)
2015-08-14 11:46:30 +05:30
Vinayakumar B c2a9c3929e HDFS-7235. DataNode#transferBlock should report blocks that don't exist using reportBadBlock (yzhang via cmccabe)
Moved CHANGES.txt entry to 2.6.1

(cherry picked from commit f2b4bc9b6a)
2015-08-14 11:38:03 +05:30
Vinayakumar B 2c0063d420 HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne.
Moved CHANGES.TXT entry to 2.6.1

(cherry picked from commit d25cb8fe12)
2015-08-14 11:24:12 +05:30
Akira Ajisaka 200fadedc0 HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer. Contributed by Jagadesh Kiran N.
(cherry picked from commit 40f815131e)
2015-08-13 12:23:03 +09:00
Xiaoyu Yao 458b684941 HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode restart. Contributed by Xiaoyu Yao.
(cherry picked from commit 3e715a4f4c)
2015-08-12 15:25:20 -07:00
Andrew Wang d53296281a HDFS-8887. Expose storage type and storage ID in BlockLocation.
(cherry picked from commit 1ea1a8334e)
2015-08-11 23:33:06 -07:00
Jing Zhao f0e13c5499 HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula.
(cherry picked from commit 1fc3c779a4)
2015-08-11 10:28:48 -07:00
Tsz-Wo Nicholas Sze ac8d153046 HDFS-8818. Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable. 2015-08-10 16:53:15 -07:00
Jakob Homan e771dbefe9 HDFS-8866. Typo in docs: Rumtime -> Runtime. Contributed by Gabor Liptak.
(cherry picked from commit 8f73bdd06b)
2015-08-07 11:39:54 -07:00
Andrew Wang 0c4c7b3b62 HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. Contributed by Walter Su. 2015-08-07 09:53:04 -07:00
Arpit Agarwal 42a05d29ee HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal) 2015-08-06 18:52:07 -07:00
Jing Zhao d7d804623e Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class. Contributed by Zhe Zhang."
This reverts commit c17439c2dd.

(cherry picked from commit f4c523b69b)
2015-08-06 10:26:09 -07:00
Jing Zhao 2c694286b4 Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang."
This reverts commit de480d6c89.

(cherry picked from commit 663eba0ab1)
2015-08-06 10:25:54 -07:00
Vinayakumar B f8932d478f HDFS-8815. DFS getStoragePolicy implementation using single RPC call (Contributed by Surendra Singh Lilhore)
(cherry picked from commit cc71ad80e1)
2015-08-06 11:11:59 +05:30
Colin Patrick Mccabe b64d2ac478 HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki via Colin P. McCabe)
(cherry picked from commit c95993cbaf)
2015-08-04 12:22:40 -07:00
yliu d2b941f94a HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) 2015-08-04 15:51:37 +08:00
Zhihai Xu c6e79178d4 HDFS-8847. change TestHDFSContractAppend to not override testRenameFileBeingAppended method. Contributed by Zhihai Xu
(cherry picked from commit 90b510405a)
2015-07-31 21:48:03 -07:00
Xiaoyu Yao 52ee3ecdf6 HDFS-6860. BlockStateChange logs are too noisy. Contributed by Chang Li and Xiaoyu Yao.
(cherry picked from commit d311a38a6b)
2015-07-31 16:19:11 -07:00
Harsh J 212b8b919d HDFS-8821. Explain message "Operation category X is not supported in state standby". Contributed by Gautam Gopalakrishnan.
(cherry picked from commit c5caa25b8f)
2015-07-31 08:59:22 +05:30
Arpit Agarwal 95e71772cf HDFS-7192. DN should ignore lazyPersist hint if the writer is not local. (Contributed by Arpit Agarwal) 2015-07-30 13:16:52 -07:00
Haohui Mai 3faf7bc3fe HDFS-8816. Improve visualization for the Datanode tab in the NN UI. Contributed by Haohui Mai. 2015-07-29 17:14:20 -07:00
Vinayakumar B a0238d85f6 HDFS-8670. Better to exclude decommissioned nodes for namenode NodeUsage JMX (Contributed by J.Andreina)
(cherry picked from commit 6374ee0db4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-07-29 14:49:16 +05:30
Akira Ajisaka dff49aee08 Revert "HDFS-6682. Add a metric to expose the timestamp of the oldest under-replicated block. (aajisaka)"
This reverts commit 17319c4a65.
2015-07-29 16:52:54 +09:00
Vinayakumar B dcf1f1fdf8 HDFS-8822. Add SSD storagepolicy tests in TestBlockStoragePolicy#testDefaultPolicies (Contributed by Vinayakumar B)
(cherry picked from commit 975e138df3)
2015-07-29 11:51:18 +05:30
Vinayakumar B 0c09597942 HDFS-8811. Move BlockStoragePolicy name's constants from HdfsServerConstants.java to HdfsConstants.java (Contributed by Vinayakumar B)
(cherry picked from commit 50887e5b07)
2015-07-29 11:46:19 +05:30
Jakob Homan 71a5aa267d HDFS-8180. AbstractFileSystem Implementation for WebHdfs. Contributed by Sathosh G Nayak.
(cherry picked from commit 0712a8103f)
2015-07-28 21:11:02 -07:00
Arun Suresh ffe6a34d5b HDFS-7858. Improve HA Namenode Failover detection on the client. (asuresh)
(cherry picked from commit 030fcfa99c)
2015-07-27 23:03:34 -07:00
Akira Ajisaka 1cf5e40838 HDFS-8810. Correct assertions in TestDFSInotifyEventInputStream class. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 1df78688c6)
2015-07-27 13:18:02 +09:00
Akira Ajisaka 7bdd32ce3c HDFS-8735. Inotify: All events classes should implement toString() API. Contributed by Surendra Singh Lilhore.
(cherry picked from commit f8f6091823)
2015-07-25 02:57:33 +09:00
Akira Ajisaka b2a532f61c HDFS-8806. Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared. Contributed by Zhe Zhang.
(cherry picked from commit 206d4933a5)
2015-07-24 18:29:24 +09:00
Akira Ajisaka 17319c4a65 HDFS-6682. Add a metric to expose the timestamp of the oldest under-replicated block. (aajisaka)
(cherry picked from commit 02c01815ec)
2015-07-24 11:39:16 +09:00
Haohui Mai d3e6b40b44 HDFS-8730. Clean up the import statements in ClientProtocol. Contributed by Takanobu Asanuma. 2015-07-23 10:30:43 -07:00
Jing Zhao 71764a92c6 HDFS-8797. WebHdfsFileSystem creates too many connections for pread. Contributed by Jing Zhao.
(cherry picked from commit e91ccfad07)
2015-07-22 18:03:45 -07:00
yliu 930e344d78 HDFS-8795. Improve InvalidateBlocks#node2blocks. (yliu) 2015-07-22 15:15:08 +08:00
Haohui Mai 53ed25c3d1 HDFS-8495. Consolidate append() related implementation into a single class. Contributed by Rakesh R. 2015-07-21 17:28:28 -07:00
cnauroth aab1c14df6 HDFS-8773. Few FSNamesystem metrics are not documented in the Metrics page. Contributed by Rakesh R.
(cherry picked from commit a26cc66f38)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-07-21 14:13:15 -07:00
cnauroth 57d55d40dd HDFS-8721. Add a metric for number of encryption zones. Contributed by Rakesh R.
(cherry picked from commit cb03768b1b)
2015-07-21 13:56:07 -07:00
Ravi Prakash b728edce0f Revert "HDFS-8344. NameNode doesn't recover lease for files with missing blocks (raviprak)"
This reverts commit 3d58c7a700.

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
2015-07-21 11:30:41 -07:00
Vinayakumar B cd3692fe4b HDFS-7582. Enforce maximum number of ACL entries separately per access and default. (Contributed by Vinayakumar B)
(cherry picked from commit 29cf887b22)
2015-07-21 15:17:26 +05:30
Haohui Mai 4283831348 HDFS-7483. Display information per tier on the Namenode UI. Contributed by Benoy Antony and Haohui Mai. 2015-07-20 20:13:16 -07:00
Akira Ajisaka e78ee18642 Move HDFS-6945 to 2.7.2 section in CHANGES.txt.
(cherry picked from commit a628f67590)
2015-07-21 11:45:53 +09:00
yliu cadd02ad1d HDFS-8794. Improve CorruptReplicasMap#corruptReplicasMap. (yliu) 2015-07-21 09:18:43 +08:00
Ravi Prakash 3d58c7a700 HDFS-8344. NameNode doesn't recover lease for files with missing blocks (raviprak)
(cherry picked from commit e4f756260f)
2015-07-20 14:04:28 -07:00
Ming Ma 516bbf1c20 HDFS-7314. When the DFSClient lease cannot be renewed, abort open-for-write files rather than the entire DFSClient. (mingma)
(cherry picked from commit fbd88f1062)
2015-07-16 12:55:29 -07:00
Arpit Agarwal dc211ca658 HDFS-8778. TestBlockReportRateLimiting#testLeaseExpiration can deadlock. (Contributed by Arpit Agarwal) 2015-07-15 14:09:06 -07:00
Akira Ajisaka bad5d288ac HDFS-8742. Inotify: Support event for OP_TRUNCATE. Contributed by Surendra Singh Lilhore.
(cherry picked from commit 979c9ca2ca)
2015-07-15 04:43:03 +09:00
Kihwal Lee b0a2dc9c84 HDFS-8722. Optimize datanode writes for small writes and flushes. Contributed by Kihwal Lee
(cherry picked from commit 59388a8015)
2015-07-14 14:07:38 -05:00
Colin Patrick Mccabe 471037883f HDFS-7608: hdfs dfsclient newConnectedPeer has no write timeout (Xiaoyu Yao via Colin P. McCabe)
(cherry picked from commit 1d74ccecec)
2015-07-14 11:00:45 -07:00
Tsz-Wo Nicholas Sze d8d6d69b09 HDFS-8541. Mover should exit with NO_MOVE_PROGRESS if there is no move progress. Contributed by Surendra Singh Lilhore 2015-07-13 15:13:14 -07:00
Tsz-Wo Nicholas Sze 2e455d4d8b Add HDFS-8143 to CHANGES.txt. 2015-07-13 15:07:28 -07:00
Jing Zhao 0e8c335e7c HDFS-8751. Remove setBlocks API from INodeFile and misc code cleanup. Contributed by Zhe Zhang
(cherry picked from commit 47f4c54106)
2015-07-10 14:18:09 -07:00
Vinayakumar B c1447e654d HDFS-2956. calling fetchdt without a --renewer argument throws NPE (Contributed by Vinayakumar B)HDFS-2956. calling fetchdt without a --renewer argument throws NPE (Contributed by Vinayakumar B)
(cherry picked from commit b48908033f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDelegationTokenFetcher.java
2015-07-10 15:52:35 +05:30
Akira Ajisaka 3a0318ea93 HDFS-8749. Fix findbugs warnings in BlockManager.java. Contributed by Brahma Reddy Battula.
(cherry picked from commit d66302ed9b)
2015-07-10 15:04:44 +09:00
Jing Zhao ae0d5d6917 HDFS-8729. Fix TestFileTruncate#testTruncateWithDataNodesRestartImmediately which occasionally failed. Contributed by Walter Su.
(cherry picked from commit f4ca530c1c)
2015-07-09 13:18:35 -07:00
Haohui Mai 981fe4cfaa HDFS-8726. Move protobuf files that define the client-sever protocols to hdfs-client. Contributed by Haohui Mai. 2015-07-08 10:37:31 -07:00
Arpit Agarwal e519df47b2 HDFS-8642. Make TestFileTruncate more reliable. (Contributed by Rakesh R) 2015-07-08 09:31:17 -07:00
Vinayakumar B 6fe2e7c3f3 HDFS-8712. Remove 'public' and 'abstract' modifiers in FsVolumeSpi and FsDatasetSpi (Contributed by Lei (Eddy) Xu)
(cherry picked from commit bd4e10900c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
2015-07-08 17:03:18 +05:30
Vinayakumar B 01e12b7d69 Revert "HDFS-8712. Remove 'public' and 'abstract' modifiers in FsVolumeSpi and FsDatasetSpi (Contributed by Lei (Eddy) Xu)"
This reverts commit 38ee9247cd.
2015-07-08 16:59:22 +05:30
Vinayakumar B 38ee9247cd HDFS-8712. Remove 'public' and 'abstract' modifiers in FsVolumeSpi and FsDatasetSpi (Contributed by Lei (Eddy) Xu)
(cherry picked from commit bd4e10900c)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsDatasetSpi.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/FsVolumeSpi.java
2015-07-08 16:36:12 +05:30
Xiaoyu Yao 907abdca73 HDFS-8711. setSpaceQuota command should print the available storage type when input storage type is wrong. Contributed by Brahma Reddy Battula.
(cherry picked from commit b68701b7b2)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestQuota.java
2015-07-08 16:25:47 +05:30
Haohui Mai b181c13cbf HDFS-8620. Clean up the checkstyle warinings about ClientProtocol. Contributed by Takanobu Asanuma. 2015-07-07 14:04:12 -07:00
Jing Zhao bed74b620a Revert "HDFS-8652. Track BlockInfo instead of Block in CorruptReplicasMap. Contributed by Jing Zhao."
This reverts commit d62b63d297.

(cherry picked from commit bc99aaffe7)
2015-07-07 10:14:17 -07:00
Vinod Kumar Vavilapalli (I am also known as @tshooter.) a8bf34cad8 Release process for 2.7.1: Set the release date for 2.7.1.
(cherry picked from commit bf89ddb9b8)
2015-07-06 16:41:00 -07:00
Jing Zhao 3d73d2027f HDFS-8652. Track BlockInfo instead of Block in CorruptReplicasMap. Contributed by Jing Zhao.
(cherry picked from commit d62b63d297)
2015-07-06 15:55:03 -07:00
Vinayakumar B 9a774dfae5 HDFS-8686. WebHdfsFileSystem#getXAttr(Path p, final String name) doesn't work if namespace is in capitals (Contributed by kanaka kumar avvaru)
(cherry picked from commit fc92d3e651)
2015-07-06 16:09:42 +05:30
Vinayakumar B b963ca84bc HDFS-8577. Avoid retrying to recover lease on a file which does not exist (Contributed by J.Andreina)
(cherry picked from commit 2eae130ab9)
2015-07-03 13:36:07 +05:30
Andrew Wang efe9ae9fc6 HDFS-8709. Clarify automatic sync in FSEditLog#logEdit.
(cherry picked from commit 5fddc5177d)
2015-07-02 10:26:43 -07:00
Vinayakumar B 55bf014a07 HDFS-8703. Merge refactor of DFSInputStream from ErasureCoding branch (Contributed by Vinayakumar B)
(cherry picked from commit bff5999d07)
2015-07-02 16:12:13 +05:30
Jing Zhao ff7562cc99 HDFS-8666. Speedup the TestMover test. Contributed by Walter Su.
(cherry picked from commit 152e5df3b6)
2015-07-01 15:00:16 -07:00
Arpit Agarwal d38f377cd3 HDFS-8706. Fix typo in datanode startup options in HDFSCommands.html. (Controibuted by Brahma Reddy Battula) 2015-07-01 10:36:10 -07:00
Colin Patrick Mccabe 604d1bbdb7 HDFS-8635. Migrate HDFS native build to new CMake framework (Alan Burlison via Colin P. McCabe)
(cherry picked from commit 29df73dc05)
2015-06-30 16:01:48 -07:00
Vinayakumar B 3c632c80a2 HDFS-8579. Update HDFS usage with missing options (Contributed by J.Andreina)
(cherry picked from commit 77588e1d32)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/main/bin/hdfs
2015-06-30 15:02:46 +05:30
Haohui Mai eccd4f273e HDFS-8493. Consolidate truncate() related implementation in a single class. Contributed by Rakesh R. 2015-06-29 16:44:58 -07:00
Arpit Agarwal ff78e611c0 HDFS-8687. Remove the duplicate usage message from Dfsck.java. (Contributed by Brahma Reddy Battula) 2015-06-29 14:56:49 -07:00
Yongjun Zhang 11bdcb754a HDFS-8659. Block scanner INFO message is spamming logs. Contributed by Yongjun Zhang.
(cherry picked from commit 43a1288fff)
2015-06-29 14:48:17 -07:00
Andrew Wang ada7f66e12 HDFS-8653. Code cleanup for DatanodeManager, DatanodeDescriptor and DatanodeStorageInfo. Contributed by Zhe Zhang.
(cherry picked from commit 2ffd84273a)
2015-06-29 12:12:54 -07:00
Vinayakumar B 438bb25d43 HDFS-8628. Update missing command option for fetchdt (Contributed by J.Andreina)
(cherry picked from commit fde20ffcef)
2015-06-29 16:51:15 +05:30
Vinayakumar B beab746177 HDFS-8586. Dead Datanode is allocated for write when client is from deadnode (Contributed by Brahma Reddy Battula)
(cherry picked from commit 88ceb382ef)
2015-06-29 15:25:54 +05:30
Vinod Kumar Vavilapalli 4155bb565e Adding release 2.7.2 to CHANGES.txt.
(cherry picked from commit aad6a7d5db)
2015-06-28 16:30:27 -07:00
Arpit Agarwal 0fd47fa809 HDFS-8681. BlockScanner is incorrectly disabled by default. (Contributed by Arpit Agarwal) 2015-06-28 14:51:30 -07:00
Andrew Wang dd7776b2fe HDFS-8656. Preserve compatibility of ClientProtocol#rollingUpgrade after finalization.
(cherry picked from commit 60b858bfa6)
2015-06-26 11:33:40 -07:00
Jing Zhao 83d76151e2 HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang.
(cherry picked from commit de480d6c89)
2015-06-26 11:04:05 -07:00
Colin Patrick Mccabe 8552af91f4 HDFS-8651. Make hadoop-hdfs-project Native code -Wall-clean (Alan Burlison via Colin P. McCabe)
(cherry picked from commit 1b764a01fd)
2015-06-26 10:28:14 -07:00
Andrew Wang 84fdd4a3a0 HDFS-8546. Use try with resources in DataStorage and Storage.
(cherry picked from commit 1403b84b12)
2015-06-25 17:50:39 -07:00
Andrew Wang ba406b723d HDFS-8665. Fix replication check in DFSTestUtils#waitForReplication.
(cherry picked from commit ff0e5e572f)
2015-06-25 17:29:30 -07:00