a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
2019-05-09 21:13:29 +01:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests.
...
Contributed by Ben Roling.
2019-04-24 14:55:58 +01:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables.
...
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.
Contributed by Steve Loughran.
2019-04-11 17:12:12 -07:00
215ffc792e
HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException
...
Contributed by Steve Loughran.
Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
2019-04-04 21:14:18 +01:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
df578c07ec
HADOOP-16195 MarshalledCredentials toString
...
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
2019-03-28 17:01:57 +00:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren.
...
Author: Gabor Bota
2019-03-28 15:49:56 +00:00
ae2eb2dd42
HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists ( #636 )
2019-03-22 11:42:00 +00:00
9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
2019-03-18 19:13:13 +00:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
...
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
(Revised patch applied after stevel committed the wrong one; that has been reverted)
2019-02-19 11:33:02 +00:00
920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."
...
This reverts commit c4a00d1ad3
2019-02-18 14:57:22 +00:00
6c999fe4b0
HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki.
2019-02-12 06:07:47 +09:00
de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
2019-01-25 10:27:59 +09:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
2019-01-14 17:27:00 +00:00
04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set.
...
Contributed by Gabor Bota.
2019-01-14 13:12:05 +00:00
3420e26ae5
HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota.
2019-01-09 10:57:58 -07:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
d8f670ff28
HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal.
2018-12-27 08:19:25 -07:00
a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies.
...
Contributed by Steve Loughran.
2018-12-20 18:09:01 +00:00
82b798581d
HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota.
2018-12-12 09:30:13 -07:00
1a25bbe9ec
HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota.
2018-12-11 08:33:13 -07:00
c35de95a22
HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota.
2018-12-11 08:29:39 -07:00
3ff8580f22
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota)
2018-12-10 14:03:08 -07:00
7eb0d3a324
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
2018-11-29 09:36:39 -07:00
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
085f10e75d
HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
e148c3ff09
HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
5d96b74f33
HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
2018-11-27 20:39:54 +00:00
66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia.
2018-11-15 16:30:24 +09:00
d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong.
...
Contributed by Steve Loughran and Ewan Higgs.
2018-10-16 20:02:54 +01:00
ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
...
Contributed by Steve Loughran.
2018-10-11 14:57:38 +01:00
7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard.
...
Contributed by Gabor Bota
2018-10-09 10:46:41 +01:00
4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-03 00:24:29 -07:00
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-02 21:22:49 -07:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
c07715e378
HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran
2018-09-24 12:53:21 -07:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config.
...
Contributed by Gabor Bota.
2018-09-17 22:40:08 +01:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine)
2018-09-12 16:36:01 -07:00
1f6c4545cf
HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran
2018-09-12 10:58:39 -07:00
47b72c87eb
HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded.
...
Contributed by Gabor Bota.
2018-09-12 09:12:38 -06:00
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota)
2018-09-07 10:25:20 -07:00
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
...
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota)
2018-08-17 10:15:39 -07:00
0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375.
...
Contributed by Steve Loughran.
2018-08-16 09:58:46 -07:00
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
a13929ddcb
HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran.
2018-08-13 12:57:45 +02:00
da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
...
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
7862f1523f
HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota)
2018-08-07 20:13:09 -06:00
48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
...
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
59adeb8d7f
HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota.
2018-07-27 18:23:29 -06:00
a08812a1b1
HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota.
2018-07-12 17:24:01 +02:00
d503f65b66
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
...
connection in case of timeout.
2018-07-10 17:52:57 +02:00
93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota)
2018-07-03 13:50:11 -07:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
c687a6617d
HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota.
2018-06-25 14:59:41 -06:00
55fad6a3de
HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota.
2018-06-22 11:36:56 -06:00
b089a06793
HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota.
2018-06-20 16:45:08 -06:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota)
2018-05-29 19:20:22 -07:00
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota.
2018-05-29 13:38:15 -07:00
cc3600aabd
HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
...
Contributed by Steve Loughran.
(cherry picked from commit 170f1040d4
2018-05-17 19:08:55 +01:00
b670837469
HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name.
...
Contributed by Sean Mackrory.
2018-05-15 16:13:56 +01:00
8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota)
2018-05-08 18:58:42 -07:00
69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota)
2018-05-08 15:29:54 -07:00
96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri)
2018-05-04 11:34:45 -07:00
919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota.
2018-04-30 16:02:57 -07:00
7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota)
2018-04-26 20:41:29 -07:00
989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota)
2018-04-23 15:26:53 -07:00
1312f9ae4c
HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.
2018-04-18 16:04:00 +08:00
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.
2018-04-10 23:55:38 -07:00
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota.
2018-04-05 20:23:17 -07:00
5a174f8ac6
HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException.
...
Contributed by Gabor Bota.
2018-04-03 16:31:34 +01:00
7ce6b41509
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.
2018-03-20 21:12:20 -07:00
1976e0066e
HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
...
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
45d1b0fdcc
HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
2018-03-12 20:05:39 -07:00
dd05871b8b
HADOOP-15297. Make S3A etag => checksum feature optional.
...
Contributed by Steve Loughran.
2018-03-12 14:01:42 +00:00
e0307e53e2
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled.
...
Contributed by Anis Elleuch.
(cherry picked from commit 1dedc68f9d
2018-03-07 18:51:10 +00:00
8110d6a0d5
HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions.
...
Contributed by Aaron Fabbri.
2018-03-05 14:06:20 +00:00
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
7ac88244c5
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key.
...
Contributed by Steve Loughran.
2018-02-16 16:37:06 +00:00
9a013b255f
HADOOP-15176. Enhance IAM Assumed Role support in S3A client.
...
Contributed by Steve Loughran
(cherry picked from commit 96c047fbb9
2018-02-15 15:57:10 +00:00
b27ab7dd81
HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document.
...
Contributed by Steve Loughran.
(cherry picked from commit c761e658f6
2018-02-15 14:57:56 +00:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
1093a73689
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2018-01-18 13:13:58 +00:00
f274fe33ea
Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits."
...
This reverts commit 35ad9b1dd2
2018-01-18 12:35:57 +00:00
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran.
2018-01-17 00:05:24 -08:00
a0c71dcc33
HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing
...
after OutputCommitter patch.
Contributed by Steve Loughran
2018-01-15 11:33:32 +00:00
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-15 15:45:51 +09:00
1a09da7400
HADOOP-15163. Fix S3ACommitter documentation
...
Contributed by Alessandro Andrioni.
(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
2018-01-10 15:37:07 +00:00
b62a5ece95
HADOOP-15161. s3a: Stream and common statistics missing from metrics
...
Contributed by Sean Mackrory
2018-01-09 18:46:52 +00:00
c8ff0cc304
HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls.
...
Contributed by Steve Loughran
2017-12-21 14:58:58 +00:00
ef450df443
HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance.
...
Contributed by Steve Loughran.
2017-12-21 14:15:53 +00:00
1ba491ff90
HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive
2017-12-20 18:25:33 +00:00
35ad9b1dd2
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2017-12-18 21:19:06 +00:00
f86c81d923
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
2017-12-14 11:14:52 +08:00
28792b6b7f
Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu"
...
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
2017-12-14 11:05:11 +08:00
73b86979d6
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
2017-12-06 12:01:36 +08:00
6555af81a2
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory.
2017-12-05 11:06:32 -08:00
3150c019ae
HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes.
...
Contributed by Steve Loughran
2017-12-05 15:05:41 +00:00
3cd75845da
MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran.
2017-11-27 22:01:00 +05:30
de8b6ca5ef
HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
...
Contributed by Steve Loughran and Ryan Blue.
2017-11-22 15:28:12 +00:00
6903cf096e
HADOOP-13514. Upgrade maven surefire plugin to 2.20.1
...
Signed-off-by: Allen Wittenauer <aw@apache.org>
2017-11-19 12:39:37 -08:00
47011d7dd3
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran
2017-09-25 15:59:38 -07:00
49467165a5
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran.
2017-09-14 14:10:48 -07:00
5bbca80428
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A.
...
Contributed by Aaron Fabbri
2017-09-08 12:07:02 +01:00
83449abb40
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter.
...
Contributed by Yonger.
2017-09-07 19:55:56 +01:00
50506e90a8
HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge.
2017-09-05 23:26:57 -07:00
0d419c984f
Preparing for 3.1.0 development
2017-09-01 11:53:48 -07:00
621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A.
...
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
2017-09-01 14:13:41 +01:00
7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws.
...
Contributed by Steve Loughran
(cherry picked from commit ced547d5f0
2017-08-04 11:09:08 +01:00
af2773f609
Updating version for 3.0.0-beta1 development
2017-06-29 17:57:40 -07:00
72993b33b7
HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
...
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
2017-06-29 17:07:52 -07:00
ee243e5289
HADOOP-14190. Add more on S3 regions to the s3a documentation.
...
Contributed by Steve Loughran
2017-06-28 10:22:13 +01:00
bc4dfe9c9c
HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
2017-06-27 13:19:14 -07:00
f214a9961f
HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran
2017-06-15 11:16:16 -07:00
6aeda55bb8
HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu
2017-06-05 11:26:56 -07:00
6c6a7a5962
HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
2017-05-28 22:25:00 -07:00
16ad896d5c
Update maven version for 3.0.0-alpha4 development
2017-05-26 14:09:44 -07:00
6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu.
2017-05-25 15:36:44 +09:00
6672810eea
HADOOP-14432. S3A copyFromLocalFile to be robust, tested. Contributed by Steve Loughran
2017-05-19 11:51:43 -07:00
ba70225cf6
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
...
Contributed by Steve Loughran.
(cherry picked from commit 2ac5aab8d7
2017-05-18 15:44:39 +01:00
0d5c8ed8e0
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor.
2017-05-11 16:37:32 -05:00
5f934f8386
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
...
Contributed by Steve Moist.
2017-04-24 20:33:19 +01:00
667966c13c
HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran
2017-04-20 17:13:36 -07:00
6b015d00c9
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
...
Contributed by Steve Loughran
2017-04-19 10:21:44 +01:00
b053fdc547
HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu
2017-04-12 14:30:34 -07:00
b8305e6d06
HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu.
2017-04-12 10:02:13 -07:00
5faa949b78
HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka
2017-04-03 11:07:14 -07:00
0d053eeb30
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
2017-03-30 18:11:50 +09:00
4966a6e26e
HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy.
2017-03-29 14:37:21 -07:00
Ben Roling
Sahil Takiar
Rajat Khandelwal
Ben Roling
Steve Loughran
Gabor Bota
Lokesh Jain
Adam Antal
Abhishek Modi
Masatake Iwasaki
Andrew Olson
Akira Ajisaka
Sean Mackrory
Sean Mackrory
Ewan Higgs
Aaron Fabbri
Sunil G
Mingliang Liu
Ewan Higgs
Chris Douglas
Yiqun Lin
fang zhenyi
Wangda Tan
Mukul Kumar Singh
Kai Zheng
Rohith Sharma K S
John Zhuge
Andrew Wang
Chris Nauroth