Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,734 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

119 Commits

Author SHA1 Message Date
Akira Ajisaka 25ac54a52e HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252)
 2017-02-15 01:46:59 +09:00
Lei Xu 6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Mingliang Liu 2dbd86bcf4 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:06:39 -08:00
Mingliang Liu c37e814fae HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:48:38 -08:00
Mingliang Liu a4e0539636 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:39:23 -08:00
Mingliang Liu 39b74b6f9e HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99f)
 2016-11-30 13:04:44 -08:00
Mingliang Liu 1b831d8e01 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8a)
 2016-11-28 16:35:18 -08:00
Steve Loughran 9701ab48e8 HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:34:11 +00:00
Steve Loughran f3fff5c2e1 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:04 +00:00
Mingliang Liu cae4a8c05d HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:05:58 -08:00
Steve Loughran c4ccafdaff HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:52:49 +00:00
Chris Nauroth 082d69ee66 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
 2016-10-26 08:57:29 -07:00
Chris Nauroth 67e01f7218 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
 2016-10-26 08:27:33 -07:00
Chris Nauroth 05d772e297 HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
 2016-10-25 09:03:11 -07:00
Chris Nauroth 5c2f67bdae HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e94030)
 2016-10-24 21:54:16 -07:00
Chris Nauroth 5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
 2016-10-24 21:22:46 -07:00
Steve Loughran 385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Steve Loughran f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb.
 2016-10-20 22:48:26 +01:00
Anu Engineer 42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran c640ba5fc9 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:30 +01:00
Steve Loughran bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Chris Nauroth 2de2d0df1d HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
 2016-10-07 11:41:31 -07:00
Steve Loughran a6bb21eec4 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:47:14 +01:00
Steve Loughran 6ed335a018 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:15 +01:00
Chris Nauroth 213f0ad7f0 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
 2016-10-04 10:37:04 -07:00
Steve Loughran 896df3f55a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:00:34 +01:00
Chris Nauroth 1f1e47e411 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
 2016-09-28 16:21:00 -07:00
Steve Loughran 6a016faf59 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:34 +01:00
Steve Loughran 5f8b6f0313 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:24:15 +01:00
Chris Nauroth e28930a38b HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
 2016-09-06 09:40:41 -07:00
Chris Nauroth cda68de9b9 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:19:20 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Chris Nauroth 986162f977 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
 2016-08-17 14:55:07 -07:00
Chris Nauroth 555a57adf2 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
 2016-08-16 17:09:28 -07:00
Steve Loughran b757eff30c HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:49:54 +01:00
Steve Loughran 271e7bb00b HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:44:03 +01:00
Steve Loughran bdde7edba5 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 17:59:50 +01:00
Steve Loughran d3df810dce HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:30 +01:00
Steve Loughran 5490ed5f87 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:33 +01:00
Jitendra Pandey 328866905c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:29:59 -07:00
Steve Loughran 89121745c0 HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:00 +01:00
Jitendra Pandey 63b64a20c2 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:00:34 -07:00
Jitendra Pandey 06b89f0d14 HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:35:50 -07:00
Steve Loughran 70a45e3122 HADOOP-13239 Deprecate s3:// in branch-2 2016-06-29 13:16:26 +01:00
Steve Loughran fb9167fdef HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:42:30 +01:00
Chris Nauroth e6ebe9ab4e HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 
(cherry picked from commit b2c596cdda)
 2016-06-21 11:32:11 -07:00
Jitendra Pandey 8f93bbcb11 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:28:55 -07:00
Chris Nauroth e4b63ed7b4 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 
(cherry picked from commit 127d2c7281)
 2016-06-16 11:18:40 -07:00