Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,000 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

89 Commits

Author SHA1 Message Date
Chris Nauroth e28930a38b HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth. 
(cherry picked from commit d152557cf7)
 2016-09-06 09:40:41 -07:00
Chris Nauroth cda68de9b9 HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth. 2016-08-23 07:19:20 -07:00
Chris Nauroth 6dba38f08b HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran. 
(cherry picked from commit 763f0497bb)
 2016-08-19 12:30:47 -07:00
Steve Loughran e39d81be80 HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie 2016-08-18 14:36:41 +01:00
Chris Nauroth 986162f977 HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran. 
(cherry picked from commit 822d661b8f)
 2016-08-17 14:55:07 -07:00
Chris Nauroth 555a57adf2 HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran. 
(cherry picked from commit 3808876c73)
 2016-08-16 17:09:28 -07:00
Steve Loughran b757eff30c HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran 2016-07-25 15:49:54 +01:00
Steve Loughran 271e7bb00b HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan 2016-07-25 14:44:03 +01:00
Steve Loughran bdde7edba5 HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong 2016-07-23 17:59:50 +01:00
Steve Loughran d3df810dce HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:30 +01:00
Steve Loughran 5490ed5f87 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:33 +01:00
Jitendra Pandey 328866905c HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:29:59 -07:00
Steve Loughran 89121745c0 HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:00 +01:00
Jitendra Pandey 63b64a20c2 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:00:34 -07:00
Jitendra Pandey 06b89f0d14 HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:35:50 -07:00
Steve Loughran 70a45e3122 HADOOP-13239 Deprecate s3:// in branch-2 2016-06-29 13:16:26 +01:00
Steve Loughran fb9167fdef HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:42:30 +01:00
Chris Nauroth e6ebe9ab4e HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 
(cherry picked from commit b2c596cdda)
 2016-06-21 11:32:11 -07:00
Jitendra Pandey 8f93bbcb11 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:28:55 -07:00
Chris Nauroth e4b63ed7b4 HADOOP-13241. document s3a better. Contributed by Steve Loughran. 
(cherry picked from commit 127d2c7281)
 2016-06-16 11:18:40 -07:00
Ravi Prakash b216dbfb55 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 
(cherry picked from commit 4aefe119a0)
 2016-06-16 11:14:38 -07:00
Steve Loughran 6350e4b1fb HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 20:59:54 +01:00
Steve Loughran df29f77846 HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 16:36:27 +01:00
Steve Loughran 0cbbc2f0db HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:41:54 +02:00
Chris Nauroth 043a0c2e6c HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:56:07 -07:00
Chris Nauroth e9942c5b34 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 
(cherry picked from commit 16b1cc7af9)
 2016-06-01 14:49:31 -07:00
Steve Loughran b29bf5d335 HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:17:50 +01:00
Steve Loughran 15f997cca6 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:19:56 +01:00
Steve Loughran eb160561e0 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:52:30 +01:00
Steve Loughran e29d7fb585 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:47 +01:00
Chris Nauroth 60e5397a9b HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459fa)
 2016-05-19 22:00:32 -07:00
Steve Loughran dd0a8201bc HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:19:08 +01:00
Steve Loughran f14208d362 HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:37 +01:00
Steve Loughran 1370dfc775 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:31 +01:00
Steve Loughran 389dd91dd4 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:23 +01:00
Steve Loughran 257b2f5beb HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:05 +01:00
Chris Nauroth ba48834431 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
 2016-05-10 15:14:36 -07:00
Steve Loughran f47ea0cc3f HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:13 +01:00
Steve Loughran edcb2a8410 HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:24:50 +01:00
Steve Loughran e124c3a2ae HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:10 +01:00
Steve Loughran 9acfde3a03 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:55 +01:00
Chris Nauroth 26a23eff82 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d)
 2016-04-08 13:37:46 -07:00
Steve Loughran 1533403a9a HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:09:47 +01:00
Steve Loughran caaadf9f3e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:01:37 +01:00
Steve Loughran 20e808d83e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:51 +01:00
Harsh J 7c5b55d4e5 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe29)
 2016-04-01 14:35:58 +05:30
Chris Nauroth e402371b6a HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415)
 2016-02-29 22:00:27 -08:00
cnauroth b4c4f30fa4 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025)
 2016-02-18 14:07:09 -08:00
cnauroth 6731bb468d HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
 2016-02-17 12:20:41 -08:00
Steve Loughran 60474a769e HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:20:24 +00:00