05d772e297
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
...
(cherry picked from commit 309a43925c
2016-10-25 09:03:11 -07:00
5c2f67bdae
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
...
(cherry picked from commit 3372e94030
2016-10-24 21:54:16 -07:00
5b7cbb5a3c
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
...
(cherry picked from commit d8fa1cfa67
2016-10-24 21:22:46 -07:00
385c1daa46
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-21 19:11:31 +01:00
f0af3dee25
Revert "HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran."
...
This reverts commit 42f8a1d6eb
2016-10-20 22:48:26 +01:00
42f8a1d6eb
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
2016-10-20 12:47:57 -07:00
c640ba5fc9
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
2016-10-20 14:50:30 +01:00
bc176961e6
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
2016-10-18 19:33:38 +01:00
2de2d0df1d
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
...
(cherry picked from commit 69620f9559
2016-10-07 11:41:31 -07:00
a6bb21eec4
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
2016-10-07 12:47:14 +01:00
6ed335a018
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
2016-10-05 15:01:15 +01:00
213f0ad7f0
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
...
(cherry picked from commit 88b9444a81
2016-10-04 10:37:04 -07:00
896df3f55a
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
2016-09-29 17:00:34 +01:00
1f1e47e411
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
...
(cherry picked from commit 47f80922dc
2016-09-28 16:21:00 -07:00
6a016faf59
HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran
2016-09-09 18:54:34 +01:00
5f8b6f0313
HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran
2016-09-07 12:24:15 +01:00
e28930a38b
HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d152557cf7
2016-09-06 09:40:41 -07:00
cda68de9b9
HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth.
2016-08-23 07:19:20 -07:00
6dba38f08b
HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran.
...
(cherry picked from commit 763f0497bb
2016-08-19 12:30:47 -07:00
e39d81be80
HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie
2016-08-18 14:36:41 +01:00
986162f977
HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran.
...
(cherry picked from commit 822d661b8f
2016-08-17 14:55:07 -07:00
555a57adf2
HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran.
...
(cherry picked from commit 3808876c73
2016-08-16 17:09:28 -07:00
b757eff30c
HADOOP-13188 S3A file-create should throw error rather than overwrite directories. Contributed by Steve Loughran
2016-07-25 15:49:54 +01:00
271e7bb00b
HADOOP-13406 S3AFileSystem: Consider reusing filestatus in delete() and mkdirs(). Contributed by Rajesh Balamohan
2016-07-25 14:44:03 +01:00
bdde7edba5
HADOOP-13389 TestS3ATemporaryCredentials.testSTS error when using IAM credentials. Contributed by Steven K. Wong
2016-07-23 17:59:50 +01:00
d3df810dce
HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel.
2016-07-20 14:01:30 +01:00
5490ed5f87
HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan)
2016-07-20 13:42:33 +01:00
328866905c
HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu.
2016-07-15 14:29:59 -07:00
89121745c0
HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse.
2016-07-11 17:21:00 +01:00
63b64a20c2
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 15:00:34 -07:00
06b89f0d14
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:35:50 -07:00
70a45e3122
HADOOP-13239 Deprecate s3:// in branch-2
2016-06-29 13:16:26 +01:00
fb9167fdef
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:42:30 +01:00
e6ebe9ab4e
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
...
(cherry picked from commit b2c596cdda
2016-06-21 11:32:11 -07:00
8f93bbcb11
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:28:55 -07:00
e4b63ed7b4
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
...
(cherry picked from commit 127d2c7281
2016-06-16 11:18:40 -07:00
b216dbfb55
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
...
(cherry picked from commit 4aefe119a0
2016-06-16 11:14:38 -07:00
6350e4b1fb
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 20:59:54 +01:00
df29f77846
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 16:36:27 +01:00
0cbbc2f0db
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:41:54 +02:00
043a0c2e6c
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:56:07 -07:00
e9942c5b34
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
...
(cherry picked from commit 16b1cc7af9
2016-06-01 14:49:31 -07:00
b29bf5d335
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:17:50 +01:00
15f997cca6
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 14:19:56 +01:00
eb160561e0
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:52:30 +01:00
e29d7fb585
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:47 +01:00
60e5397a9b
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
...
(cherry picked from commit 93258459fa
2016-05-19 22:00:32 -07:00
dd0a8201bc
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:19:08 +01:00
f14208d362
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:46:37 +01:00
1370dfc775
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:23:31 +01:00
389dd91dd4
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:36:23 +01:00
257b2f5beb
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:57:05 +01:00
ba48834431
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
2016-05-10 15:14:36 -07:00
f47ea0cc3f
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:40:13 +01:00
edcb2a8410
HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel)
2016-04-22 11:24:50 +01:00
e124c3a2ae
HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel)
2016-04-14 12:45:10 +01:00
9acfde3a03
HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel
2016-04-09 11:24:55 +01:00
26a23eff82
HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran.
...
(cherry picked from commit 843ee8d59d
2016-04-08 13:37:46 -07:00
1533403a9a
HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel)
2016-04-08 17:09:47 +01:00
caaadf9f3e
HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass
2016-04-04 17:01:37 +01:00
20e808d83e
HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse.
2016-04-03 16:39:51 +01:00
7c5b55d4e5
HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh.
...
(cherry picked from commit 256c82fe29
2016-04-01 14:35:58 +05:30
e402371b6a
HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay.
...
(cherry picked from commit d251e55415
2016-02-29 22:00:27 -08:00
b4c4f30fa4
HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay.
...
(cherry picked from commit 8ab7658025
2016-02-18 14:07:09 -08:00
6731bb468d
HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay.
...
(cherry picked from commit 76fab26c5c
2016-02-17 12:20:41 -08:00
60474a769e
HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel)
2016-02-06 18:20:24 +00:00
2cbb8bbd72
HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano
...
(cherry picked from commit 1acc509b45
2016-01-19 14:00:54 -08:00
63b8be149a
HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei)
2016-01-12 12:21:29 -08:00
6330683778
HADOOP-12689. S3 filesystem operations stopped working correctly
...
(cherry picked from commit 2d16f40dab
2016-01-05 23:43:17 -08:00
b20e77eba3
Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0
2015-11-25 17:47:22 -08:00
47941858af
Revert "HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei)"
...
This reverts commit 01ae30796d
2015-11-06 09:52:21 -08:00
01ae30796d
HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei)
...
(cherry picked from commit bff7c90a56
2015-11-05 18:36:55 -08:00
85c0cb0075
HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu.
...
(cherry picked from commit 7fe521b1dd
2015-09-25 22:34:10 -07:00
de42fce390
HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu)
2015-08-29 09:58:57 -07:00
f7b0f292e7
HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu)
...
(cherry picked from commit d5403747b5
2015-08-04 18:56:07 -07:00
83a97907cc
HADOOP-12059. S3Credentials should support use of CredentialProvider. Contributed by Sean Busbey.
...
(cherry picked from commit 2dbc40e608
2015-06-05 13:11:12 -07:00
14d88a85e4
HADOOP-11922. Misspelling of threshold in log4j.properties for tests in hadoop-tools (Contributed by Gabor Liptak)
...
(cherry picked from commit f33efbc827
2015-05-08 10:23:27 +05:30
a6a5d1d6b5
HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel)
2015-04-23 21:39:43 +01:00
77297017d8
HADOOP-11720. [JDK8] Fix javadoc errors caused by incorrect or illegal tags in hadoop-tools. Contributed by Akira AJISAKA.
...
(cherry picked from commit ef9946cd52
2015-03-17 16:09:38 +09:00
d9416317a9
Preparing for 2.8.0 development. Changing SNAPSHOT version to be 2.8 on branch-2.
2015-03-08 20:32:51 -07:00
b8a2087951
HADOOP-11670. Regression: s3a auth setup broken. (Adam Budde via stevel)
2015-03-08 11:22:01 -07:00
824c32de1a
HADOOP-11183. Memory-based S3AOutputstream. (Thomas Demoor via stevel)
2015-03-03 16:18:39 -08:00
e97613b429
HADOOP-11480. Typo in hadoop-aws/index.md uses wrong scheme for test.fs.s3.name. Contributed by Ted Yu.
...
(cherry picked from commit 1a625b8158
2015-02-24 17:13:35 -08:00
a36cad0522
HADOOP-11584 s3a file block size set to 0 in getFileStatus. (Brahma Reddy Battula via stevel)
2015-02-21 12:02:41 +00:00
4276f8b315
HADOOP-11607. Reduce log spew in S3AFileSystem. (Lei (Eddy) Xu via stevel)
2015-02-20 20:51:00 +00:00
411c8a7cda
HADOOP-11521. Make connection timeout configurable in s3a. (Thomas Demoor via stevel)
2015-02-17 20:00:00 +00:00
f52fcdc2e0
HADOOP-11522. Update S3A Documentation. (Thomas Demoor via stevel)
2015-02-17 18:14:31 +00:00
701b96ca8e
HADOOP-11570. S3AInputStream.close() downloads the remaining bytes of the object from S3. (Dan Hecht via stevel).
2015-02-17 16:36:32 +00:00
af7368cf03
HADOOP-11520. Clean incomplete multi-part uploads in S3A tests. (Thomas Demoor via stevel)
2015-02-06 20:59:56 +00:00
4cd9657b71
HADOOP-11463 Replace method-local TransferManager object with S3AFileSystem#transfers. (Ted Yu via stevel)
2015-02-05 12:19:49 +00:00
648510e031
HADOOP-11488. Difference in default connection timeout for S3A FS. Contributed by Daisuke Kobayashi.
...
(cherry picked from commit ffc75d6ebe
2015-02-01 00:17:24 +05:30
15865587ec
HADOOP-11171 Enable using a proxy server to connect to S3a. (Thomas Demoor via stevel)
2015-01-17 18:40:30 +00:00
ed2f8a1ba0
HADOOP-10542 Potential null pointer dereference in Jets3tFileSystemStore retrieveBlock(). (Ted Yu via stevel)
2015-01-17 18:26:03 +00:00
651c7debe4
HADOOP-11261 Set custom endpoint for S3A. (Thomas Demoor via stevel)
2015-01-16 10:15:22 +00:00
2800e73c27
HADOOP-11446 S3AOutputStream should use shared thread pool to avoid OutOfMemoryError
2015-01-05 12:59:48 +00:00
a1b8b9ca59
HADOOP-11412 POMs mention "The Apache Software License" rather than "Apache License". (Herve Boutemy via stevel)
2014-12-16 11:03:02 +00:00
21347a05f5
HADOOP-11394. hadoop-aws documentation missing. Contributed by Chris Nauroth.
...
(cherry picked from commit 9458cd5bce
2014-12-12 23:29:40 -08:00
a1e4a12dc0
HADOOP-11381. Fix findbugs warnings in hadoop-distcp, hadoop-aws, hadoop-azure, and hadoop-openstack. Contributed by Li Lu.
2014-12-09 20:46:44 -08:00
adfb830a2b
Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT.
2014-11-09 19:19:02 -08:00
9082fe4e20
HADOOP-10714. AmazonS3Client.deleteObjects() need to be limited to 1000 entries per call. Contributed by Juan Yu.
...
(cherry picked from commit 6ba52d88ec
2014-11-05 17:24:55 -08:00
Chris Nauroth
Steve Loughran
Anu Engineer
Jitendra Pandey
Ravi Prakash
Harsh J
Lei Xu
Ravi Prakash
Vinod Kumar Vavilapalli
Lei Xu
Andrew Wang
Vinayakumar B
Tsuyoshi Ozawa
Akira Ajisaka
Haohui Mai
Arun C. Murthy
Aaron T. Myers