Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,237 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

497 Commits

Author SHA1 Message Date
Chris Nauroth 5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
 2016-10-24 21:22:46 -07:00
Steve Loughran 385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Steve Loughran f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb.
 2016-10-20 22:48:26 +01:00
Anu Engineer 42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran c640ba5fc9 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:30 +01:00
Steve Loughran bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Allen Wittenauer a5a56c3564 HADOOP-13034. Log message about input options in distcp lacks some items (Takashi Ohnishi via aw) 
(cherry picked from commit 422c73a865)
 2016-10-17 16:35:32 -07:00
Andrew Wang 4ed7cf3b36 HADOOP-13724. Fix a few typos in site markdown documents. Contributed by Ding Fei. 
(cherry picked from commit 987ee51141)
 2016-10-17 13:32:39 -07:00
Jing Zhao ad69baf6a9 HADOOP-13024. Distcp with -delete feature on raw data not implemented. Contributed by Mavin Martin. 
(cherry picked from commit 0a85d07983)
 2016-10-13 13:26:02 -07:00
Chris Nauroth 2de2d0df1d HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
 2016-10-07 11:41:31 -07:00
Steve Loughran a6bb21eec4 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:47:14 +01:00
Steve Loughran 6ed335a018 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:15 +01:00
Chris Nauroth 213f0ad7f0 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
 2016-10-04 10:37:04 -07:00
Steve Loughran 896df3f55a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:00:34 +01:00
Chris Nauroth 1f1e47e411 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
 2016-09-28 16:21:00 -07:00
Brahma Reddy Battula 5737d04c5f HDFS-9885. Correct the distcp counters name while displaying counters. Contributed by Surendra Singh Lilhore 
(cherry picked from commit e17a4970be)
 2016-09-27 10:47:07 +05:30
Steve Loughran 247cdbb86a HADOOP-13643. Math error in AbstractContractDistCpTest. Contributed by Aaron Fabbri. 2016-09-23 10:00:32 +01:00
Chris Nauroth e19f910245 HADOOP-13169. Randomize file list in SimpleCopyListing. Contributed by Rajesh Balamohan. 
(cherry picked from commit 98bdb51397)
 2016-09-19 15:16:56 -07:00
Steve Loughran 3b9268cfba HADOOP-13606 swift FS to add a service load metadata file. Contributed by Steve Loughran 2016-09-14 15:44:45 +01:00
Steve Loughran 6a016faf59 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:34 +01:00
Steve Loughran 5f8b6f0313 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:24:15 +01:00
Chris Nauroth e28930a38b HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
 2016-09-06 09:40:41 -07:00
Arun Suresh b279f42d79 YARN-5221. Expose UpdateResourceRequest API to allow AM to request for change in container properties. (asuresh) 
(cherry picked from commit d6d9cff21b)
 2016-08-30 16:33:31 -07:00
Chris Nauroth cda68de9b9 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:19:20 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Chris Nauroth 804a40720b HADOOP-13513. Java 1.7 support for org.apache.hadoop.fs.azure testcases. Contributed by Tibor Kiss. 
(cherry picked from commit ae4db25443)
 2016-08-18 10:45:03 -07:00
Junping Du d464483bf7 YARN-4676. Automatic and Asynchronous Decommissioning Nodes Status Tracking. Contributed by Diniel Zhi. 2016-08-18 07:23:29 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Chris Nauroth 986162f977 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
 2016-08-17 14:55:07 -07:00
Chris Nauroth 555a57adf2 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
 2016-08-16 17:09:28 -07:00
Chris Nauroth 287688d6ca HADOOP-13403. AzureNativeFileSystem rename/delete performance improvements. Contributed by Subramanyam Pattipaka. 
(cherry picked from commit 2ed58c40e5)
 2016-08-08 12:28:46 -07:00
Chris Nauroth 8ec3d702a3 HADOOP-13459. hadoop-azure runs several test cases repeatedly, causing unnecessarily long running time. Contributed by Chris Nauroth. 
(cherry picked from commit 4ef1bc0bdb)
 2016-08-02 22:32:37 -07:00
Chris Nauroth 59eec8248a HADOOP-13354. Update WASB driver to use the latest version (4.2.0) of SDK for Microsoft Azure Storage Clients. Contributed by Sivaguru Sankaridurg. 
(cherry picked from commit b43de80031)

Conflicts:
	hadoop-project/pom.xml
 2016-07-27 15:52:07 -07:00
Steve Loughran b757eff30c HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:49:54 +01:00
Steve Loughran 271e7bb00b HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:44:03 +01:00
Steve Loughran bdde7edba5 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 17:59:50 +01:00
mattf-horton ea10e1384f HADOOP-13382. Remove unneeded commons-httpclient dependencies from POM files in Hadoop and sub-projects, while adding it in to the only place it is still needed, hadoop-openstack. Contributed by Matt Foley. 2016-07-21 15:58:05 -07:00
Steve Loughran d3df810dce HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:30 +01:00
Steve Loughran 5490ed5f87 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:33 +01:00
Jitendra Pandey 328866905c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:29:59 -07:00
Jian He e94e6be772 YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:37:52 -07:00
Andrew Wang 38b1eafdbc HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8)
 2016-07-11 18:06:35 -07:00
Steve Loughran 89121745c0 HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:00 +01:00
Akira Ajisaka f4b63a391b Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux." 
This reverts commit 1d175f0f3f.
 2016-07-09 21:00:13 -07:00
Jitendra Pandey 63b64a20c2 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:00:34 -07:00
Jitendra Pandey 06b89f0d14 HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:35:50 -07:00
Steve Loughran 70a45e3122 HADOOP-13239 Deprecate s3:// in branch-2 2016-06-29 13:16:26 +01:00
Yongjun Zhang 8a121dc3b8 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3a)
 2016-06-28 23:29:48 -07:00
Jing Zhao 856bc4e28b HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 2016-06-22 15:15:47 -07:00
Steve Loughran fb9167fdef HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:42:30 +01:00