Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
9,942 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

3402 Commits

Author SHA1 Message Date
Jing Zhao f39c749f2a HDFS-8980. Remove unnecessary block replacement in INodeFile. Contributed by Jing Zhao. 
(cherry picked from commit caa04de149)
 2015-08-31 11:48:32 -07:00
Jakob Homan 94944ba4f6 HDFS-8155. Support OAuth2 in WebHDFS. 
(cherry picked from commit 837fb75e8e)
 2015-08-29 18:38:21 -07:00
Arpit Agarwal f1d5f1417e HDFS-8983. NameNode support for protected directories. (Contributed by Arpit Agarwal) 2015-08-29 09:57:53 -07:00
Haohui Mai a3d0534558 HDFS-8925. Move BlockReaderLocal to hdfs-client. Contributed by Mingliang Liu. 2015-08-28 14:38:23 -07:00
Colin Patrick Mccabe 159969f658 HDFS-8950. NameNode refresh doesn't remove DataNodes that are no longer in the allowed list (Daniel Templeton) 
(cherry picked from commit b94b56806d)
 2015-08-28 14:22:05 -07:00
Haohui Mai dea531a3d5 HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone classes from BlockManager. Contributed by Mingliang Liu. 2015-08-28 14:13:59 -07:00
Kihwal Lee 5184779418 HDFS-8865. Improve quota initialization performance. Contributed by Kihwal Lee. 
(cherry picked from commit b6ceee9bf4)
 2015-08-28 13:19:28 -05:00
yliu 5f9b323eb0 HDFS-8963. Fix incorrect sign extension of xattr length in HDFS-8900. (Colin Patrick McCabe via yliu) 2015-08-28 10:53:53 +08:00
Haohui Mai 07a19325c4 Revert "HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone classes from BlockManager. Contributed by Mingliang Liu." 
This reverts commit 0ae19f76f4.
 2015-08-27 16:09:51 -07:00
Haohui Mai 0ae19f76f4 HDFS-8938. Extract BlockToMarkCorrupt and ReplicationWork as standalone classes from BlockManager. Contributed by Mingliang Liu. 2015-08-27 15:49:26 -07:00
Haohui Mai 1b01d163a2 HDFS-8969. Clean up findbugs warnings for HDFS-8823 and HDFS-8932. Contributed by Anu Engineer. 2015-08-27 13:03:29 -07:00
Haohui Mai f4b2348503 HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. 2015-08-27 09:13:37 -07:00
Haohui Mai af1e2993c7 HDFS-8962. Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf. Contributed by Mingliang Liu. 2015-08-27 09:05:28 -07:00
Harsh J 574f3a98ce HDFS-2390. dfsadmin -setBalancerBandwidth does not validate -ve value. Contributed by Gautam Gopalakrishnan. 
(cherry picked from commit 0bf285413f)
 2015-08-27 16:23:22 +05:30
Vinayakumar B d63122979f HDFS-8682. Should not remove decommissioned node,while calculating the number of live/dead decommissioned node. (Contributed by J. Andreina) 
(cherry picked from commit fdb56f74f3)
 2015-08-27 13:07:35 +05:30
Haohui Mai 2f3faf39ef HDFS-8248. Store INodeId instead of the INodeFile object in BlockInfoContiguous. Contributed by Haohui Mai. 2015-08-26 18:14:49 -07:00
Jing Zhao 10c4d6df61 HDFS-8896. DataNode object isn't GCed when shutdown, because it has GC root in ShutdownHookManager. Contributed by Walter Su. 
(cherry picked from commit f44b599003)
 2015-08-26 16:28:38 -07:00
Haohui Mai f0f6f1c7e9 HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by Mingliang Liu. 2015-08-26 14:22:33 -07:00
Colin Patrick Mccabe 9264b7e119 HDFS-8846. Add a unit test for INotify functionality across a layout version upgrade (Zhe Zhang via Colin P. McCabe) 
(cherry picked from commit a4d9acc51d)
 2015-08-25 14:30:15 -07:00
yliu df5dbf317d HDFS-8900. Compact XAttrs to optimize memory footprint. (yliu) 2015-08-25 16:14:11 +08:00
Xiaoyu Yao 137bde0755 HDFS-8932. NPE thrown in NameNode when try to get TotalSyncCount metric before editLogStream initialization. Contributed by Surendra Singh Lilhore 
(cherry picked from commit 3b00eaea25)
 2015-08-24 16:57:02 -07:00
Haohui Mai ef8437a382 HDFS-8948. Use GenericTestUtils to set log levels in TestPread and TestReplaceDatanodeOnFailure. Contributed by Mingliang Liu. 2015-08-24 16:16:33 -07:00
Jing Zhao eefc1c174b HDFS-8928. Improvements for BlockUnderConstructionFeature: ReplicaUnderConstruction as a separate class and replicas as an array. Contributed by Jing Zhao. 
(cherry picked from commit bdd79388f3)
 2015-08-24 15:55:58 -07:00
Haohui Mai 95f8e93691 HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu. 2015-08-24 13:40:30 -07:00
Andrew Wang a727c6db05 Revert "HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu." 
This reverts commit 8e4afa3a67.
 2015-08-24 11:51:46 -07:00
Colin Patrick Mccabe 87d0133703 HDFS-8930. Block report lease may leak if the 2nd full block report comes when NN is still in safemode (Colin P. McCabe via Jing Zhao) 
(cherry picked from commit b5ce87f84d)
 2015-08-24 11:33:21 -07:00
Akira Ajisaka f7ee225052 HDFS-8942. Update hyperlink to rack awareness page in HDFS Architecture documentation. Contributed by Masatake Iwasaki. 
(cherry picked from commit bcaf83902a)
 2015-08-24 13:53:22 +09:00
Haohui Mai 8e4afa3a67 HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu. 2015-08-22 13:30:48 -07:00
Haohui Mai 8c205267c8 HDFS-8823. Move replication factor into individual blocks. Contributed by Haohui Mai. 2015-08-22 00:09:59 -07:00
Lei Xu 3a4d614f68 HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) 
(cherry picked from 7087e700e0)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
 2015-08-21 17:17:19 -07:00
Chris Douglas c8f1097989 HDFS-8891. HDFS concat should keep srcs order. Contributed by Yong Zhang. 
Moved CHANGES.txt entry to 2.7.2

(cherry picked from commit b0564c9f3c)
 2015-08-20 20:39:59 -07:00
Vinod Kumar Vavilapalli 45be1240d9 Creating 2.6.2 entries in CHANGES.txt files. 
(cherry picked from commit 33474939a0)
 2015-08-20 17:23:54 -07:00
Jing Zhao cd0c6b5789 HDFS-8809. HDFS fsck reports under construction blocks as CORRUPT. Contributed by Jing Zhao. 
(cherry picked from commit c8bca62718)
 2015-08-20 16:32:04 -07:00
Colin Patrick Mccabe a0450b96a2 HDFS-8922. Link the native_mini_dfs test library with libdl, since IBM Java requires it (Ayappan via Colin P. McCabe) 
(cherry picked from commit 7642f64c24)
 2015-08-20 14:40:37 -07:00
Yongjun Zhang b085c5ef6f HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 
(cherry picked from commit 0bc15cb6e6)
 2015-08-20 08:10:34 -07:00
yliu 146db49f7f HDFS-8863. The remaining space check in BlockPlacementPolicyDefault is flawed. (Kihwal Lee via yliu) 2015-08-20 20:46:45 +08:00
yliu 63bdbb7793 HDFS-8884. Fail-fast check in BlockPlacementPolicyDefault#chooseTarget. (yliu) 2015-08-20 20:08:20 +08:00
Jing Zhao 62dfe1b7e4 HDFS-8917. Cleanup BlockInfoUnderConstruction from comments and tests. Contributed by Zhe Zhang. 
(cherry picked from commit 4e14f7982a)
 2015-08-19 15:16:35 -07:00
Haohui Mai 535b6db74c HDFS-8803. Move DfsClientConf to hdfs-client. Contributed by Mingliang Liu. 2015-08-19 11:43:32 -07:00
Jing Zhao c0a4cd978a HDFS-8867. Enable optimized block reports. Contributed by Daryn Sharp. 
(cherry picked from commit f61120d964)
 2015-08-19 10:39:18 -07:00
Arpit Agarwal 10a0ddbac3 HDFS-8911. NameNode Metric : Add Editlog counters as a JMX metric. (Contributed by Anu Engineer) 2015-08-19 09:42:48 -07:00
yliu 0824fa1c4f HDFS-8908. TestAppendSnapshotTruncate may fail with IOException: Failed to replace a bad datanode. (Tsz Wo Nicholas Sze via yliu) 2015-08-19 21:56:56 +08:00
Tsz-Wo Nicholas Sze 7245f7fad2 HDFS-8826. In Balancer, add an option to specify the source node list so that balancer only selects blocks to move from those nodes. 2015-08-18 19:26:58 -07:00
Chris Douglas 79af15fd37 HDFS-8435. Support CreateFlag in WebHDFS. Contributed by Jakob Homan 2015-08-18 17:33:21 -07:00
Akira Ajisaka 38b344adfa HDFS-8852. HDFS architecture documentation of version 2.x is outdated about append write support. Contributed by Ajith S. 
(cherry picked from commit fc509f66d8)
 2015-08-18 23:32:57 +09:00
Tsz-Wo Nicholas Sze 9ab89a9266 HDFS-8278. When computing max-size-to-move in Balancer, count only the storage with remaining >= default block size. 2015-08-17 18:42:41 -07:00
yliu 076c688780 HDFS-8862. BlockManager#excessReplicateMap should use a HashMap. (yliu) 2015-08-18 09:25:03 +08:00
Arpit Agarwal 85cb596e2e HDFS-8880. NameNode metrics logging. (Arpit Agarwal) 2015-08-17 16:55:24 -07:00
Colin Patrick Mccabe 39326ba71c HDFS-8845. DiskChecker should not traverse the entire tree (Chang Li via Colin P. McCabe) 2015-08-17 15:24:56 -07:00
Colin Patrick Mccabe 08508ca1c8 HDFS-8792. BlockManager#postponedMisreplicatedBlocks should use a LightWeightHashSet to save memory (Yi Liu via Colin P. McCabe) 
(cherry picked from commit c77bd6af16)
 2015-08-17 12:07:16 -07:00
Haohui Mai 55f7097900 HDFS-8801. Convert BlockInfoUnderConstruction as a feature. Contributed by Jing Zhao. 2015-08-17 11:28:51 -07:00
Haohui Mai 4e712faac2 HDFS-6407. Add sorting and pagination in the datanode tab of the NN Web UI. Contributed by Haohui Mai. 2015-08-17 11:04:17 -07:00
Xiaoyu Yao 27ccbd51f6 HDFS-8883. NameNode Metrics : Add FSNameSystem lock Queue Length. Contributed by Anu Engineer. 
(cherry picked from commit a7862d5fe4)
 2015-08-17 10:23:56 -07:00
Andrew Wang 9dcb426819 HDFS-8713. Convert DatanodeDescriptor to use SLF4J logging. 
(cherry picked from commit 3fe758436041d55da1ccde4d5109da38d1f5110f)
 2015-08-17 10:16:37 -07:00
Jing Zhao 2edfc3ce71 HDFS-8891. HDFS concat should keep srcs order. Contributed by Yong Zhang. 
(cherry picked from commit dc7a061668)
 2015-08-14 14:43:33 -07:00
Tsz-Wo Nicholas Sze 61b9e5f7ff HDFS-8824. Do not use small blocks for balancing the cluster. 2015-08-14 13:08:57 -07:00
Arpit Agarwal b094f8b589 HDFS-7649. Multihoming docs should emphasize using hostnames in configurations. (Contributed by Brahma Reddy Battula) 2015-08-14 13:08:19 -07:00
Xiaoyu Yao 17256e2ae8 HDFS-8565. Typo in dfshealth.html - Decomissioning. (nijel via xyao) 
(cherry picked from commit 1569228ec9)
 2015-08-14 12:35:17 -07:00
Vinayakumar B 0b6574380c HDFS-8270. create() always retried with hardcoded timeout when file already exists with open lease (Contributed by J.Andreina) 
Moved to 2.6.1

(cherry picked from commit 84bf71295a)
 2015-08-14 16:13:47 +05:30
Vinayakumar B 15760a148b HDFS-7225. Remove stale block invalidation work when DN re-registers with different UUID. (Zhe Zhang and Andrew Wang) 
Moved to 2.6.1

(cherry picked from commit 08bd4edf40)
 2015-08-14 12:38:23 +05:30
Vinayakumar B 9839abe1a8 HDFS-7263. Snapshot read can reveal future bytes for appended files. Contributed by Tao Luo. 
Moved CHANGES.txt entry to 2.6.1

(cherry picked from commit fa2641143c)
 2015-08-14 11:46:30 +05:30
Vinayakumar B c2a9c3929e HDFS-7235. DataNode#transferBlock should report blocks that don't exist using reportBadBlock (yzhang via cmccabe) 
Moved CHANGES.txt entry to 2.6.1

(cherry picked from commit f2b4bc9b6a)
 2015-08-14 11:38:03 +05:30
Vinayakumar B 2c0063d420 HDFS-7213. processIncrementalBlockReport performance degradation. Contributed by Eric Payne. 
Moved CHANGES.TXT entry to 2.6.1

(cherry picked from commit d25cb8fe12)
 2015-08-14 11:24:12 +05:30
yliu 073cb16d81 HADOOP-12295. Improve NetworkTopology#InnerNode#remove logic. (yliu) 2015-08-13 16:42:40 +08:00
Akira Ajisaka 200fadedc0 HDFS-8622. Implement GETCONTENTSUMMARY operation for WebImageViewer. Contributed by Jagadesh Kiran N. 
(cherry picked from commit 40f815131e)
 2015-08-13 12:23:03 +09:00
cnauroth bf8b09d4b8 HADOOP-12258. Need translate java.nio.file.NoSuchFileException to FileNotFoundException to avoid regression. Contributed by Zhihai Xu. 
(cherry picked from commit 6cc8e38db5)
 2015-08-12 16:45:05 -07:00
Xiaoyu Yao 458b684941 HDFS-8879. Quota by storage type usage incorrectly initialized upon namenode restart. Contributed by Xiaoyu Yao. 
(cherry picked from commit 3e715a4f4c)
 2015-08-12 15:25:20 -07:00
Andrew Wang d53296281a HDFS-8887. Expose storage type and storage ID in BlockLocation. 
(cherry picked from commit 1ea1a8334e)
 2015-08-11 23:33:06 -07:00
Jing Zhao f0e13c5499 HDFS-8805. Archival Storage: getStoragePolicy should not need superuser privilege. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 1fc3c779a4)
 2015-08-11 10:28:48 -07:00
Tsz-Wo Nicholas Sze ac8d153046 HDFS-8818. Changes the global moveExecutor to per datanode executors and changes MAX_SIZE_TO_MOVE to be configurable. 2015-08-10 16:53:15 -07:00
Jakob Homan e771dbefe9 HDFS-8866. Typo in docs: Rumtime -> Runtime. Contributed by Gabor Liptak. 
(cherry picked from commit 8f73bdd06b)
 2015-08-07 11:39:54 -07:00
Andrew Wang 0c4c7b3b62 HDFS-8772. Fix TestStandbyIsHot#testDatanodeRestarts which occasionally fails. Contributed by Walter Su. 2015-08-07 09:53:04 -07:00
Arpit Agarwal 42a05d29ee HDFS-8856. Make LeaseManager#countPath O(1). (Contributed by Arpit Agarwal) 2015-08-06 18:52:07 -07:00
Jing Zhao d7d804623e Revert "HDFS-8499. Refactor BlockInfo class hierarchy with static helper class. Contributed by Zhe Zhang." 
This reverts commit c17439c2dd.

(cherry picked from commit f4c523b69b)
 2015-08-06 10:26:09 -07:00
Jing Zhao 2c694286b4 Revert "HDFS-8623. Refactor NameNode handling of invalid, corrupt, and under-recovery blocks. Contributed by Zhe Zhang." 
This reverts commit de480d6c89.

(cherry picked from commit 663eba0ab1)
 2015-08-06 10:25:54 -07:00
Vinayakumar B f8932d478f HDFS-8815. DFS getStoragePolicy implementation using single RPC call (Contributed by Surendra Singh Lilhore) 
(cherry picked from commit cc71ad80e1)
 2015-08-06 11:11:59 +05:30
Lei Xu f7b0f292e7 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 
(cherry picked from commit d5403747b5)
 2015-08-04 18:56:07 -07:00
Colin Patrick Mccabe b64d2ac478 HDFS-8844. TestHDFSCLI does not cleanup the test directory (Masatake Iwasaki via Colin P. McCabe) 
(cherry picked from commit c95993cbaf)
 2015-08-04 12:22:40 -07:00
yliu d2b941f94a HDFS-8850. VolumeScanner thread exits with exception if there is no block pool to be scanned but there are suspicious blocks. (Colin Patrick McCabe via yliu) 2015-08-04 15:51:37 +08:00
Zhihai Xu c6e79178d4 HDFS-8847. change TestHDFSContractAppend to not override testRenameFileBeingAppended method. Contributed by Zhihai Xu 
(cherry picked from commit 90b510405a)
 2015-07-31 21:48:03 -07:00
Xiaoyu Yao 52ee3ecdf6 HDFS-6860. BlockStateChange logs are too noisy. Contributed by Chang Li and Xiaoyu Yao. 
(cherry picked from commit d311a38a6b)
 2015-07-31 16:19:11 -07:00
Colin Patrick Mccabe 07a042aa3c HADOOP-7824. NativeIO.java flags and identifiers must be set correctly for each platform, not hardcoded to their Linux values (Martin Walsh via Colin P. McCabe) 
(cherry picked from commit 21d10ccc6e)
 2015-07-31 15:02:01 -07:00
Harsh J 212b8b919d HDFS-8821. Explain message "Operation category X is not supported in state standby". Contributed by Gautam Gopalakrishnan. 
(cherry picked from commit c5caa25b8f)
 2015-07-31 08:59:22 +05:30
Arpit Agarwal 95e71772cf HDFS-7192. DN should ignore lazyPersist hint if the writer is not local. (Contributed by Arpit Agarwal) 2015-07-30 13:16:52 -07:00
Haohui Mai 3faf7bc3fe HDFS-8816. Improve visualization for the Datanode tab in the NN UI. Contributed by Haohui Mai. 2015-07-29 17:14:20 -07:00
Jason Lowe e6b4714d0b HADOOP-10945. 4-digit octal umask permissions throws a parse error. Contributed by Chang Li 
(cherry picked from commit 6f0a35724f)
 2015-07-29 15:25:59 +00:00
Vinayakumar B a0238d85f6 HDFS-8670. Better to exclude decommissioned nodes for namenode NodeUsage JMX (Contributed by J.Andreina) 
(cherry picked from commit 6374ee0db4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2015-07-29 14:49:16 +05:30
Akira Ajisaka dff49aee08 Revert "HDFS-6682. Add a metric to expose the timestamp of the oldest under-replicated block. (aajisaka)" 
This reverts commit 17319c4a65.
 2015-07-29 16:52:54 +09:00
Lei Xu 9d81916040 HDFS-8834. TestReplication is not valid after HDFS-6482. (Contributed by Lei Xu) 
(cherry picked from commit f4f1b8b267)
 2015-07-28 23:41:44 -07:00
Vinayakumar B dcf1f1fdf8 HDFS-8822. Add SSD storagepolicy tests in TestBlockStoragePolicy#testDefaultPolicies (Contributed by Vinayakumar B) 
(cherry picked from commit 975e138df3)
 2015-07-29 11:51:18 +05:30
Vinayakumar B 0c09597942 HDFS-8811. Move BlockStoragePolicy name's constants from HdfsServerConstants.java to HdfsConstants.java (Contributed by Vinayakumar B) 
(cherry picked from commit 50887e5b07)
 2015-07-29 11:46:19 +05:30
Jakob Homan 71a5aa267d HDFS-8180. AbstractFileSystem Implementation for WebHdfs. Contributed by Sathosh G Nayak. 
(cherry picked from commit 0712a8103f)
 2015-07-28 21:11:02 -07:00
Arun Suresh ffe6a34d5b HDFS-7858. Improve HA Namenode Failover detection on the client. (asuresh) 
(cherry picked from commit 030fcfa99c)
 2015-07-27 23:03:34 -07:00
Akira Ajisaka 1cf5e40838 HDFS-8810. Correct assertions in TestDFSInotifyEventInputStream class. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 1df78688c6)
 2015-07-27 13:18:02 +09:00
Akira Ajisaka 7bdd32ce3c HDFS-8735. Inotify: All events classes should implement toString() API. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit f8f6091823)
 2015-07-25 02:57:33 +09:00
Akira Ajisaka b2a532f61c HDFS-8806. Inconsistent metrics: number of missing blocks with replication factor 1 not properly cleared. Contributed by Zhe Zhang. 
(cherry picked from commit 206d4933a5)
 2015-07-24 18:29:24 +09:00
Akira Ajisaka 17319c4a65 HDFS-6682. Add a metric to expose the timestamp of the oldest under-replicated block. (aajisaka) 
(cherry picked from commit 02c01815ec)
 2015-07-24 11:39:16 +09:00
Haohui Mai d3e6b40b44 HDFS-8730. Clean up the import statements in ClientProtocol. Contributed by Takanobu Asanuma. 2015-07-23 10:30:43 -07:00
Arpit Agarwal 601b503895 HADOOP-12161. Add getStoragePolicy API to the FileSystem interface. (Contributed by Brahma Reddy Battula) 2015-07-23 10:14:01 -07:00
Jing Zhao 71764a92c6 HDFS-8797. WebHdfsFileSystem creates too many connections for pread. Contributed by Jing Zhao. 
(cherry picked from commit e91ccfad07)
 2015-07-22 18:03:45 -07:00