Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,808 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

129 Commits

Author SHA1 Message Date
Steve Loughran 51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran
 2017-03-02 21:41:49 +00:00
Steve Loughran 29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.
 2017-02-25 15:31:26 +00:00
Steve Loughran c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 
(cherry picked from commit ff87ca8441)
 2017-02-24 10:40:58 +00:00
Mingliang Liu db955f0844 HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu 
(cherry picked from commit 0013090fb4)
 2017-02-22 14:38:57 -08:00
Steve Loughran 171b18693a HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran. 2017-02-22 11:41:24 +00:00
Steve Loughran 6843659689 HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 2017-02-21 18:29:30 +00:00
Steve Loughran 274c02d2fa HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:00 +00:00
Steve Loughran 8a88e8ea57 HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 
(cherry picked from commit b1c1f05b1d)
 2017-02-18 18:17:54 +00:00
Mingliang Liu b23e93d465 HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory 
(cherry picked from commit 658702efff)
 2017-02-16 16:59:31 -08:00
Mingliang Liu 29e1007172 HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 
(cherry picked from commit bdad8b7b97)
 2017-02-16 16:42:56 -08:00
Akira Ajisaka 25ac54a52e HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252)
 2017-02-15 01:46:59 +09:00
Lei Xu 6d62d0ea87 HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei) 2017-02-11 15:02:19 -08:00
Steve Loughran ec2f5b18ab HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:24:25 +00:00
Mingliang Liu 2dbd86bcf4 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:06:39 -08:00
Mingliang Liu c37e814fae HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:48:38 -08:00
Mingliang Liu a4e0539636 HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:39:23 -08:00
Mingliang Liu 39b74b6f9e HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99f)
 2016-11-30 13:04:44 -08:00
Mingliang Liu 1b831d8e01 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8a)
 2016-11-28 16:35:18 -08:00
Steve Loughran 9701ab48e8 HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:34:11 +00:00
Steve Loughran f3fff5c2e1 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:04 +00:00
Mingliang Liu cae4a8c05d HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran 2016-11-22 14:05:58 -08:00
Steve Loughran c4ccafdaff HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:52:49 +00:00
Chris Nauroth 082d69ee66 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
 2016-10-26 08:57:29 -07:00
Chris Nauroth 67e01f7218 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
 2016-10-26 08:27:33 -07:00
Chris Nauroth 05d772e297 HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
 2016-10-25 09:03:11 -07:00
Chris Nauroth 5c2f67bdae HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e94030)
 2016-10-24 21:54:16 -07:00
Chris Nauroth 5b7cbb5a3c HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
 2016-10-24 21:22:46 -07:00
Steve Loughran 385c1daa46 HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:11:31 +01:00
Steve Loughran f0af3dee25 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 42f8a1d6eb.
 2016-10-20 22:48:26 +01:00
Anu Engineer 42f8a1d6eb HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:47:57 -07:00
Steve Loughran c640ba5fc9 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:30 +01:00
Steve Loughran bc176961e6 HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:33:38 +01:00
Chris Nauroth 2de2d0df1d HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
 2016-10-07 11:41:31 -07:00
Steve Loughran a6bb21eec4 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 12:47:14 +01:00
Steve Loughran 6ed335a018 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:15 +01:00
Chris Nauroth 213f0ad7f0 HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
 2016-10-04 10:37:04 -07:00
Steve Loughran 896df3f55a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-09-29 17:00:34 +01:00
Chris Nauroth 1f1e47e411 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
 2016-09-28 16:21:00 -07:00
Steve Loughran 6a016faf59 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:34 +01:00
Steve Loughran 5f8b6f0313 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:24:15 +01:00
Chris Nauroth e28930a38b HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
 2016-09-06 09:40:41 -07:00
Chris Nauroth cda68de9b9 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:19:20 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Chris Nauroth 986162f977 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
 2016-08-17 14:55:07 -07:00
Chris Nauroth 555a57adf2 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
 2016-08-16 17:09:28 -07:00
Steve Loughran b757eff30c HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:49:54 +01:00
Steve Loughran 271e7bb00b HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:44:03 +01:00
Steve Loughran bdde7edba5 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 17:59:50 +01:00
Steve Loughran d3df810dce HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:30 +01:00