Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,518 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

175 Commits

Author SHA1 Message Date
Akira Ajisaka a714d04f5d HDFS-9768. Reuse ObjectMapper instance in HDFS to improve the performance. Contributed by Lin Yiqun. 
(cherry picked from commit e6a7044b85)
 2016-02-13 02:04:01 +09:00
zhezhang ef9a81e875 HDFS-9244. Support nested encryption zones. 
Change-Id: I43a13035a8b27956e90967ee82058efb647c3415
 2016-02-08 16:31:41 -08:00
Uma Maheswara Rao G 44a020ed81 HDFS-9777. Fix typos in DFSAdmin command line and documentation.(Wei-Chiu Chuang via umamahesh) 
(cherry picked from commit bffaa38a91)
 2016-02-08 14:00:47 -08:00
Kihwal Lee 494cfd1dc4 HDFS-9752. Permanent write failures may happen to slow writers during datanode rolling upgrades. Contributed by Walter Su. 
(cherry picked from commit 193d27de0a)
 2016-02-08 12:17:07 -06:00
Haohui Mai e62ca14856 HDFS-9724. Degraded performance in WebHDFS listing as it does not reuse ObjectMapper. Contributed by Akira Ajisaka. 2016-02-04 11:35:34 -08:00
Vinayakumar B e8a9a69a27 HDFS-9739. DatanodeStorage.isValidStorageId() is broken (Contributed by Mingliang Liu) 
(cherry picked from commit d6b1acb940)
 2016-02-03 10:33:55 +05:30
Tsz-Wo Nicholas Sze b10d8ced21 HDFS-8999. Allow a file to be closed with COMMITTED but not yet COMPLETE blocks. 2016-01-28 10:42:40 +08:00
Arpit Agarwal 0dbaef4377 HDFS-9094. Add command line option to ask NameNode reload configuration. (Contributed by Xiaobing Zhou) 2016-01-25 12:16:45 -08:00
Eric Yang da61953a0b HDFS-9653. Added blocks pending deletion report to dfsadmin. 
(Weiwei Yang via eyang)
 2016-01-24 14:26:46 -08:00
Kihwal Lee b5db1d4410 HDFS-8898. Create API and command-line argument to get quota and quota usage without detailed content summary. Contributed by Ming Ma. 2016-01-22 09:19:23 -06:00
Kihwal Lee 3d8d8f1497 HDFS-9634. webhdfs client side exceptions don't provide enough details. Contributed by Eric Payne. 
(cherry picked from commit 7b70500484574a565dd8cd5c7d8b5bc7c6d91154)
 2016-01-21 09:33:30 -06:00
Colin Patrick Mccabe c56044b9ee HDFS-9576: HTrace: collect position/length information on read operations (zhz via cmccabe) 
(cherry picked from commit 7905788db9)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
 2016-01-20 11:27:51 -08:00
Kihwal Lee 8d04b7c272 HDFS-9574. Reduce client failures during datanode restart. Contributed by Kihwal Lee. 
(cherry picked from commit 38c4c14472)
 2016-01-08 11:15:12 -06:00
Uma Mahesh cdd796d199 HADOOP-12658. Clear javadoc and check style issues around DomainSocket. Contributed by Kai Zheng 
(cherry picked from commit 778146eaae)
 2016-01-04 15:04:46 -08:00
Kihwal Lee 131260f0a7 HDFS-7163. WebHdfsFileSystem should retry reads according to the configured retry policy. Contributed by Eric Payne. 
(cherry picked from commit 867048c3e4)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHDFS.java
 2015-12-22 14:19:12 -06:00
Colin Patrick Mccabe 31d2fb91fc HDFS-9350. Avoid creating temprorary strings in Block.toString() and getBlockName() (Staffan Friberg via cmccabe) 
(cherry picked from commit e63388fdf2)
 2015-12-18 10:05:12 -08:00
cnauroth e8e3737ce8 HDFS-9557. Reduce object allocation in PB conversion. Contributed by Daryn Sharp. 
(cherry picked from commit c470c8953d)
 2015-12-16 11:11:27 -08:00
Andrew Wang 2796c1099f HDFS-8894. Set SO_KEEPALIVE on DN server sockets. Contributed by Kanaka Kumar Avvaru. 
(cherry picked from commit 49949a4bb0)
 2015-12-15 14:39:04 -08:00
cnauroth 1072be12b9 HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. Contributed by Xiaobing Zhou. 
(cherry picked from commit 303f80e656)
 2015-12-08 10:50:13 -08:00
Arpit Agarwal 6999e764cb HDFS-9414. Add missing license header 2015-12-05 09:24:44 -08:00
Arpit Agarwal 71be31201a HDFS-9414. Refactor reconfiguration of ClientDatanodeProtocol for reusability. (Contributed by Xiaobing Zhou) 2015-12-04 20:53:43 -08:00
Xiaoyu Yao eb7f9901b4 HDFS-8831. Trash Support for deletion in HDFS encryption zone. Contributed by Xiaoyu Yao. 
(cherry picked from commit cbc7b6bf97)
 2015-12-04 16:57:58 -08:00
Tsz-Wo Nicholas Sze bac127a4e2 HDFS-9294. DFSClient deadlock when close file and failed to renew lease. Contributed by Brahma Reddy Battula 2015-12-02 17:49:16 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Xiaoyu Yao 4f1f9f7eff HDFS-8512. WebHDFS : GETFILESTATUS should return LocatedBlock with storage type info. Contributed by Xiaoyu Yao. 
(cherry picked from commit e3d673901b)
 2015-11-25 13:54:22 -08:00
Uma Mahesh 52c889ac14 HDFS-9433. DFS getEZForPath API on a non-existent file should throw FileNotFoundException (Rakesh R via umamahesh) 
(cherry picked from commit 411e2b2e7c)
 2015-11-23 18:46:48 -08:00
Haohui Mai d469e9e4ca HDFS-7988. Replace usage of ExactSizeInputStream with LimitInputStream. Contributed by Walter Su. 2015-11-23 10:50:23 -08:00
Haohui Mai 2c185222f9 HDFS-9443. Disabling HDFS client socket cache causes logging message printed to console for CLI commands. Contributed by Chris Nauroth. 2015-11-22 15:06:36 -08:00
Akira Ajisaka c74e42b4a2 HDFS-9431. DistributedFileSystem#concat fails if the target path is relative. Contributed by Kazuho Fujii. 
(cherry picked from commit ac1aa6c819)
 2015-11-19 20:55:36 +09:00
cnauroth 7b94ae17ab HDFS-9400. TestRollingUpgradeRollback fails on branch-2. Contributed by Brahma Reddy Battula. 
(cherry picked from commit bad2afe39a)
 2015-11-18 11:00:18 -08:00
Xiaoyu Yao 48a494cdc3 HDFS-9234. WebHdfs: getContentSummary() should give quota for storage types. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 41d3f8899d)
 2015-11-10 16:48:06 -08:00
Zhe Zhang edd4b2d458 HDFS-9364. Unnecessary DNS resolution attempts when creating NameNodeProxies. Contributed by Xiao Chen. 
Change-Id: I9e42f724f27924cf73891425a832de37ce014a1e
 2015-11-10 09:55:39 -08:00
Kihwal Lee ceb062f093 HDFS-9383. TestByteArrayManager#testByteArrayManager fails. Contributed by Tsz Wo Nicholas Sze. 
(cherry picked from commit ef926b2e38)
 2015-11-09 08:57:25 -06:00
cnauroth 385dcc4064 HDFS-9394. branch-2 hadoop-hdfs-client fails during FileSystem ServiceLoader initialization, because HftpFileSystem is missing. Contributed by Mingliang Liu. 
(cherry picked from commit 1ee2db5c41)
 2015-11-07 07:39:45 -08:00
Tsz-Wo Nicholas Sze 9766fc988f HDFS-9398. Make ByteArraryManager log message in one-line format. Contributed by Mingliang Liu 2015-11-07 22:22:36 +08:00
cnauroth 75bcc8bcd8 HDFS-9384. TestWebHdfsContentLength intermittently hangs and fails due to TCP conversation mismatch between client and server. Contributed by Chris Nauroth. 
(cherry picked from commit 66c0967310)
 2015-11-05 22:07:34 -08:00
cnauroth 6edd41ac69 HDFS-9378. hadoop-hdfs-client tests do not write logs. Contributed by Chris Nauroth. 
(cherry picked from commit c563401796)
 2015-11-05 06:48:32 -08:00
Zhe Zhang 4f29f5d192 HDFS-9289. Make DataStreamer#block thread safe and verify genStamp in commitBlock. Contributed by Chang Li. 
Change-Id: I0591b1f9c6a0a8b368408b195151de2e0aa440c4
 2015-11-03 14:23:02 -08:00
Haohui Mai 9e911a2439 HDFS-9168. Move client side unit test to hadoop-hdfs-client. Contributed by Haohui Mai. 2015-10-28 15:54:53 -07:00
Ming Ma 2c335a8434 HDFS-9259. Make SO_SNDBUF size configurable at DFSClient side for hdfs write scenario. (Mingliang Liu via mingma) 
(cherry picked from commit aa09880ab8)
 2015-10-27 09:29:37 -07:00
Yongjun Zhang b50af4e04a HDFS-7284. Add more debug info to BlockInfoUnderConstruction#setGenerationStampAndVerifyReplicas. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit 5e718de522)
 2015-10-26 13:36:25 -07:00
Arpit Agarwal b727dfd61c HDFS-4015. Safemode should count and report orphaned blocks. (Contributed by Anu Engineer) 2015-10-23 21:40:26 -07:00
Haohui Mai 558ae8c2df HDFS-9301. HDFS clients can't construct HdfsConfiguration instances. Contributed by Mingliang Liu. 2015-10-23 14:53:15 -07:00
Kihwal Lee d24d0fd981 HDFS-9290. DFSClient#callAppend() is not backward compatible for slightly older NameNodes. Contributed by Tony Wu. 
(cherry picked from commit b9e0417bdf)
 2015-10-23 16:38:47 -05:00
Andrew Wang c35771c46e HDFS-9250. Add Precondition check to LocatedBlock#addCachedLoc. Contributed by Xiao Chen. 
(cherry picked from commit 8175c4f6b9)
 2015-10-19 16:13:14 -07:00
Andrew Wang 12d3b69217 HDFS-8941. DistributedFileSystem listCorruptFileBlocks API should resolve relative path. Contributed by Rakesh R. 
(cherry picked from commit c32614f410)
 2015-10-09 11:57:07 -07:00
Jing Zhao a4b8dc48f1 HDFS-9196. Fix TestWebHdfsContentLength. Contributed by Masatake Iwasaki. 
(cherry picked from commit 239d119c67)
 2015-10-06 23:55:33 -07:00
Vinayakumar B 96fbe42bd7 HDFS-7899. Improve EOF error message (Contributed by Jagadesh Kiran N) 
(cherry picked from commit 874c8ed239)
 2015-10-06 15:55:12 +05:30
Haohui Mai 1257483ebf HDFS-8979. Clean up checkstyle warnings in hadoop-hdfs-client module. Contributed by Mingliang Liu. 2015-10-03 11:37:19 -07:00
cnauroth 7273ef0590 HDFS-9175. Change scope of 'AccessTokenProvider.getAccessToken()' and 'CredentialBasedAccessTokenProvider.getCredential()' abstract methods to public. Contributed by Santhosh Nayak. 
(cherry picked from commit c7e03c3a6c)
 2015-09-30 16:06:45 -07:00
Haohui Mai 37d42fcec1 HDFS-8971. Remove guards when calling LOG.debug() and LOG.trace() in client package. Contributed by Mingliang Liu. 2015-09-29 17:52:42 -07:00
Haohui Mai 58bdc9c672 Revert "HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. Contributed by Haohui Mai." 
This reverts commit cf873fff9f.
 2015-09-29 17:51:31 -07:00
Haohui Mai cf873fff9f HDFS-9170. Move libhdfs / fuse-dfs / libwebhdfs to hdfs-client. Contributed by Haohui Mai. 2015-09-29 17:48:35 -07:00
Haohui Mai cf6fb09162 HDFS-9166. Move hftp / hsftp filesystem to hfds-client. Contributed by Mingliang Liu. 2015-09-29 10:07:19 -07:00
Haohui Mai c705eab221 HDFS-9165. Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client. Contributed by Mingliang Liu. 2015-09-29 09:51:30 -07:00
Colin Patrick Mccabe 171bd1cb2b HADOOP-12447. Clean up some htrace integration issues (cmccabe) 
(cherry picked from commit 850d679acb)
 2015-09-29 09:38:10 -07:00
Kihwal Lee 95b8f1f3a6 HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 
(cherry picked from commit 3abbdc929b)
 2015-09-28 17:33:20 -05:00
Lei Xu 213ff984ee HDFS-9147. Fix the setting of visibleLength in ExternalBlockReader. (Colin P. McCabe via Lei (Eddy) Xu) 
(cherry picked from commit e5992ef4df)
 2015-09-28 13:43:52 -07:00
Kihwal Lee 468b330feb HDFS-9106. Transfer failure during pipeline recovery causes permanent write failures. Contributed by Kihwal Lee. 
(cherry picked from commit 4c9497cbf0)
 2015-09-28 14:30:06 -05:00
Colin Patrick Mccabe 3b9d32627f HDFS-9080. Update htrace version to 4.0.1 (cmccabe) 
(cherry picked from commit 892ade689f)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
        hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/BlockStorageLocationUtil.java
	hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java
 2015-09-28 08:15:00 -07:00
Haohui Mai 504d6fd950 HDFS-8740. Move DistributedFileSystem to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-27 10:54:53 -07:00
Haohui Mai 94cbb6d164 HDFS-8053. Move DFSIn/OutputStream and related classes to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-26 11:16:50 -07:00
Lei Xu 82d4c05b2d HDFS-9133. ExternalBlockReader and ReplicaAccessor need to return -1 on read when at EOF. (Colin Patrick McCabe via Lei (Eddy) Xu) 
(cherry picked from commit 67b0e967f0)
 2015-09-25 16:02:27 -07:00
Lei Xu 1a2870f6ff HDFS-9132. Pass genstamp to ReplicaAccessorBuilder. (Colin Patrick McCabe via Lei (Eddy) Xu) 2015-09-25 15:51:29 -07:00
Yongjun Zhang bc5ee80794 HDFS-5795. RemoteBlockReader2#checkSuccess() shoud print error status. (Xiao Chen via Yongjun Zhang) 
(cherry picked from commit 83e65c5fe8)
 2015-09-24 22:15:28 -07:00
Haohui Mai 85c73d159e HDFS-9134. Move LEASE_{SOFTLIMIT,HARDLIMIT}_PERIOD constants from HdfsServerConstants to HdfsConstants. Contributed by Mingliang Liu. 2015-09-24 20:29:52 -07:00
Haohui Mai d148b6a234 HDFS-9131. Move config keys used by hdfs-client to HdfsClientConfigKeys. Contributed by Mingliang Liu. 2015-09-24 00:30:17 -07:00
Haohui Mai 090b1f6398 HDFS-8733. Keep server related definition in hdfs.proto on server side. Contributed by Mingliang Liu. 2015-09-22 20:57:11 -07:00
Haohui Mai ed7211563c HDFS-9039. Separate client and server side methods of o.a.h.hdfs.NameNodeProxies. Contributed by Mingliang Liu. 2015-09-22 20:52:46 -07:00
Haohui Mai 4e752a37e7 HDFS-9111. Move hdfs-client protobuf convert methods from PBHelper to PBHelperClient. Contributed by Mingliang Liu. 2015-09-21 19:43:14 -07:00
Lei Xu 0095936a6a HDFS-9004. Add upgrade domain to DatanodeInfo. Contributed by Ming Ma (via Lei (Eddy) Xu). 
Change-Id: I887c66578eebd61acc34b94f18da6e6851c609f4
(cherry picked from commit 3a9c7076e8)
 2015-09-19 18:09:56 -07:00
Haohui Mai cadde8c1e5 HDFS-9022. Move NameNode.getAddress() and NameNode.getUri() to hadoop-hdfs-client. Contributed by Mingliang Liu. 2015-09-17 14:17:30 -07:00
cnauroth b297a9c95c HDFS-9082. Change the log level in WebHdfsFileSystem.initialize() from INFO to DEBUG. Contributed by Santhosh Nayak. 
(cherry picked from commit 559c09dc0e)
 2015-09-15 15:13:56 -07:00
Haohui Mai 323821b174 HDFS-9041. Move entries in META-INF/services/o.a.h.fs.FileSystem to hdfs-client. Contributed by Mingliang Liu. 2015-09-12 11:02:27 -07:00
Arpit Agarwal d732c7b9de HDFS-9027. Refactor o.a.h.hdfs.DataStreamer#isLazyPersist() method. (Contributed by Mingliang Liu) 2015-09-11 10:12:02 -07:00
Vinayakumar B 03f50de48d HDFS-7116. Add a command to get the balancer bandwidth (Contributed by Rakesh R) 
(cherry picked from commit 0f0e897bf1)
 2015-09-09 22:59:55 +05:30
Akira Ajisaka eb9e06f8e3 HDFS-9033. dfsadmin -metasave prints "NaN" for cache used%. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0113e4528d)
 2015-09-09 17:21:22 +09:00
Haohui Mai 8f712dd2f0 HDFS-9012. Move o.a.h.hdfs.protocol.datatransfer.PipelineAck class to hadoop-hdfs-client module. Contributed by Mingliang Liu. 2015-09-04 10:41:28 -07:00
Haohui Mai d99018d6bc HDFS-9002. Move o.a.h.hdfs.net/*Peer classes to hdfs-client. Contributed by Mingliang Liu. 2015-09-03 15:33:22 -07:00
Akira Ajisaka 772fe0f478 HDFS-8885. ByteRangeInputStream used in webhdfs does not override available(). Contributed by Shradha Revankar. 
(cherry picked from commit c92e31bd65)
 2015-09-03 19:41:36 +09:00
Haohui Mai 94ab3b9e8f HDFS-8990. Move RemoteBlockReader to hdfs-client module. Contributed by Mingliang Liu. 2015-08-31 13:54:42 -07:00
Jakob Homan 94944ba4f6 HDFS-8155. Support OAuth2 in WebHDFS. 
(cherry picked from commit 837fb75e8e)
 2015-08-29 18:38:21 -07:00
Haohui Mai a3d0534558 HDFS-8925. Move BlockReaderLocal to hdfs-client. Contributed by Mingliang Liu. 2015-08-28 14:38:23 -07:00
Haohui Mai f4b2348503 HDFS-8961. Investigate lock issue in o.a.h.hdfs.shortcircuit.DfsClientShmManager.EndpointShmManager. Contributed by Mingliang Liu. 2015-08-27 09:13:37 -07:00
Haohui Mai af1e2993c7 HDFS-8962. Clean up checkstyle warnings in o.a.h.hdfs.DfsClientConf. Contributed by Mingliang Liu. 2015-08-27 09:05:28 -07:00
Haohui Mai f0f6f1c7e9 HDFS-8951. Move the shortcircuit package to hdfs-client. Contributed by Mingliang Liu. 2015-08-26 14:22:33 -07:00
Haohui Mai 95f8e93691 HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu. 2015-08-24 13:40:30 -07:00
Andrew Wang a727c6db05 Revert "HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu." 
This reverts commit 8e4afa3a67.
 2015-08-24 11:51:46 -07:00
Haohui Mai 8e4afa3a67 HDFS-8934. Move ShortCircuitShm to hdfs-client. Contributed by Mingliang Liu. 2015-08-22 13:30:48 -07:00
Lei Xu 3a4d614f68 HDFS-8924. Add pluggable interface for reading replicas in DFSClient. (Colin Patrick McCabe via Lei Xu) 
(cherry picked from 7087e700e0)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/client/impl/DfsClientConf.java
 2015-08-21 17:17:19 -07:00
Haohui Mai 535b6db74c HDFS-8803. Move DfsClientConf to hdfs-client. Contributed by Mingliang Liu. 2015-08-19 11:43:32 -07:00
Chris Douglas 79af15fd37 HDFS-8435. Support CreateFlag in WebHDFS. Contributed by Jakob Homan 2015-08-18 17:33:21 -07:00
Andrew Wang d53296281a HDFS-8887. Expose storage type and storage ID in BlockLocation. 
(cherry picked from commit 1ea1a8334e)
 2015-08-11 23:33:06 -07:00
Vinayakumar B f8932d478f HDFS-8815. DFS getStoragePolicy implementation using single RPC call (Contributed by Surendra Singh Lilhore) 
(cherry picked from commit cc71ad80e1)
 2015-08-06 11:11:59 +05:30
Vinayakumar B 0c09597942 HDFS-8811. Move BlockStoragePolicy name's constants from HdfsServerConstants.java to HdfsConstants.java (Contributed by Vinayakumar B) 
(cherry picked from commit 50887e5b07)
 2015-07-29 11:46:19 +05:30
Akira Ajisaka 7bdd32ce3c HDFS-8735. Inotify: All events classes should implement toString() API. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit f8f6091823)
 2015-07-25 02:57:33 +09:00
Haohui Mai d3e6b40b44 HDFS-8730. Clean up the import statements in ClientProtocol. Contributed by Takanobu Asanuma. 2015-07-23 10:30:43 -07:00
Jing Zhao 71764a92c6 HDFS-8797. WebHdfsFileSystem creates too many connections for pread. Contributed by Jing Zhao. 
(cherry picked from commit e91ccfad07)
 2015-07-22 18:03:45 -07:00
Akira Ajisaka bad5d288ac HDFS-8742. Inotify: Support event for OP_TRUNCATE. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit 979c9ca2ca)
 2015-07-15 04:43:03 +09:00
Haohui Mai 981fe4cfaa HDFS-8726. Move protobuf files that define the client-sever protocols to hdfs-client. Contributed by Haohui Mai. 2015-07-08 10:37:31 -07:00