Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,302 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

134 Commits

Author SHA1 Message Date
Vinod Kumar Vavilapalli (I am also known as @tshooter.) 396de0fd8f Preparing for 2.8.3 development 2017-06-07 15:38:20 -07:00
John Zhuge f8a3de64f8 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 
(cherry picked from commit 6c6a7a5962)
 2017-05-28 22:29:49 -07:00
Steve Loughran 952113ebfd
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran

(cherry picked from commit 1ae6be46b4)
 2017-04-19 15:37:27 +01:00
Junping Du dfac95a3a2 Preparing for 2.8.1 development 2017-04-11 11:21:49 -07:00
Mingliang Liu e47c95ffbf HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 2017-04-03 11:15:03 -07:00
Akira Ajisaka 6e005ad326
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 0d053eeb30)
(cherry picked from commit bb187e145a)
 2017-03-30 18:13:55 +09:00
Mingliang Liu 45956131b9 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 
(cherry picked from commit 2e30aa72e0)
 2017-03-23 11:42:23 -07:00
Steve Loughran 595501c9dd
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran
 2017-03-21 13:13:16 +00:00
Akira Ajisaka 20fa7664d1
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 
(cherry picked from commit f597f4c43e)
(cherry picked from commit d2d0872f4d)
 2017-03-07 15:17:42 +09:00
Steve Loughran 2acc4d913e
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38b)
 2017-03-02 21:43:05 +00:00
Steve Loughran 05ab09ca87
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures. 
Contributed by Steve Loughran.
 2017-02-25 16:10:15 +00:00
Steve Loughran ff87ca8441
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory. 2017-02-24 10:35:21 +00:00
Steve Loughran bb4b2f641c HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory. 
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
 2017-02-21 18:29:46 +00:00
Steve Loughran 8a05ea4aba HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan 2017-02-20 16:21:12 +00:00
Steve Loughran b1c1f05b1d HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge 2017-02-18 18:16:19 +00:00
Mingliang Liu 3b4f840ccb HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran 
(cherry picked from commit bdad8b7b97)
 2017-02-16 16:43:22 -08:00
Akira Ajisaka 4718f65545 HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin. 
(cherry picked from commit b9f8491252)
(cherry picked from commit 25ac54a52e)
 2017-02-15 01:47:21 +09:00
Steve Loughran e32cd52481 HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran 2017-01-11 17:15:01 +00:00
Junping Du 7706a63fb4 Preparing for 2.8.1 development after cutting off 2.8.0 branch. 2017-01-10 18:03:51 -08:00
Mingliang Liu 01b50b36b6 HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran 
(cherry picked from commit c6a3923245)
 2016-12-12 15:11:11 -08:00
Mingliang Liu 3232f5ffe6 HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran 
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
 2016-12-02 13:55:17 -08:00
Mingliang Liu 4a1b7d6dee HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran 
(cherry picked from commit 2ff84a0040)
 2016-12-02 13:42:58 -08:00
Mingliang Liu afad13f8d9 HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran 
(cherry picked from commit 3fd844b99f)
 2016-11-30 13:06:22 -08:00
Mingliang Liu 59f6067568 HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran 
(cherry picked from commit d60a60be8a)
 2016-11-28 16:42:41 -08:00
Steve Loughran db37ad0ebf HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay. 2016-11-25 13:33:32 +00:00
Steve Loughran 01e3b0dae8 HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran 2016-11-23 21:37:23 +00:00
Steve Loughran a83f10c90d HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi. 2016-10-31 20:53:22 +00:00
Chris Nauroth 4df1780991 HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth. 
(cherry picked from commit 1f8490a5ba)
(cherry picked from commit 082d69ee66)
 2016-10-26 08:57:46 -07:00
Chris Nauroth c940c68c79 HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran. 
(cherry picked from commit 9cad3e2350)
(cherry picked from commit 67e01f7218)
 2016-10-26 08:27:39 -07:00
Chris Nauroth 3d5f41544a HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth. 
(cherry picked from commit 309a43925c)
(cherry picked from commit 05d772e297)
 2016-10-25 09:03:19 -07:00
Chris Nauroth 02f0472d1e HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran. 
(cherry picked from commit 3372e94030)
(cherry picked from commit 5c2f67bdae)
 2016-10-24 21:54:25 -07:00
Chris Nauroth 78970e0db3 HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth. 
(cherry picked from commit d8fa1cfa67)
(cherry picked from commit 5b7cbb5a3c)
 2016-10-24 21:23:00 -07:00
Steve Loughran 4b56954fea HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-21 19:12:03 +01:00
Steve Loughran da7e2f08f1 Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran." 
This reverts commit 671d219c9c.
 2016-10-20 22:49:11 +01:00
Anu Engineer 671d219c9c HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran. 2016-10-20 12:50:49 -07:00
Steve Loughran 769babf470 HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse 2016-10-20 14:50:44 +01:00
Steve Loughran 1ecbf323ba HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran 2016-10-18 19:34:07 +01:00
Chris Nauroth 67d8301e55 HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
(cherry picked from commit e28930a38b)
 2016-10-10 14:01:19 -07:00
Chris Nauroth 027d76fa8c HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 
(cherry picked from commit cda68de9b9)
 2016-10-10 14:01:13 -07:00
Chris Nauroth d29dc39791 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
(cherry picked from commit 986162f977)
 2016-10-10 14:01:07 -07:00
Chris Nauroth c6e69f6fb5 HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth. 
(cherry picked from commit 69620f9559)
(cherry picked from commit 2de2d0df1d)
 2016-10-07 11:41:40 -07:00
Steve Loughran 96cc056cb7 HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran. 2016-10-07 13:22:20 +01:00
Steve Loughran d24933ad3a HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan. 2016-10-07 13:12:04 +01:00
Steve Loughran f66863f848 HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory 2016-10-05 15:01:52 +01:00
Chris Nauroth 4ea1e73e5b HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth. 
(cherry picked from commit 88b9444a81)
(cherry picked from commit 213f0ad7f0)
 2016-10-04 10:37:10 -07:00
Chris Nauroth b7b06a5f34 HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran. 
(cherry picked from commit 47f80922dc)
(cherry picked from commit 1f1e47e411)
 2016-09-28 16:21:10 -07:00
Steve Loughran 5706754f77 HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran 2016-09-09 18:54:56 +01:00
Steve Loughran c45f1ec858 HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran 2016-09-07 12:19:05 +01:00
Chris Nauroth dd6c149c4a HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
(cherry picked from commit 6dba38f08b)
 2016-08-19 12:55:57 -07:00
Steve Loughran c09797ce9f HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:35:26 +01:00