bac234b0a8
HADOOP-17338 Intermittent S3AInputStream failures: Premature end of Content-Length delimited message body etc ( #2692 ))
...
Backported from commit ID 3d2193cd64
2021-02-09 12:04:14 +00:00
ceea43115b
HADOOP-16767 Handle non-IO exceptions in reopen() ( #2685 )
...
Contributed by Sergei Poganshev.
Catches Exception instead of IOException in closeStream()
and so handle exceptions such as SdkClientException by
aborting the wrapped stream. This will increase resilience
to failures, as any which occuring during stream closure
will be caught. Furthermore, because the
underlying HTTP connection is aborted, rather than closed,
it will not be recycled to cause problems on subsequent
operations.
2021-02-08 16:03:43 +00:00
a75f2915f9
HADOOP-17199. S3A Directory Marker HADOOP-13230 backport
...
This backports the listing-side changes of HADOOP-13230.
With this patch in, this branch of Hadoop is compatible with S3A clients
which do not delete directory markers when files are created underneath.
It does not allow this version to disable marker deletion; if the
fs.s3a.marker.retention option is changed to request this, a message
is printed at INFO and the policy remains at "delete"
The s3guard bucket-info command has been extended to support
probing for marker retention, as has the hasPathCapability method on
S3AFileSystem.
Read the documentation!
Change-Id: Ief23c39256b194abcc445231b5f7a018185ea9f6
2020-09-03 18:29:30 +01:00
2c321c3c30
HADOOP-14918. Remove the Local Dynamo DB test option (branch-2.10) ( #1864 ). Contributed by Jonathan Hung and Gabor Bota.
2020-03-23 16:22:26 +01:00
a8044c75ad
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml ( #1753 )
...
Contributed by Mingliang Liu
2019-12-11 11:56:46 -08:00
cdb0cafff5
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu
...
This closes #1733
2019-12-05 17:56:02 -08:00
9b8044d00b
HADOOP-16109. Parquet reading S3AFileSystem causes EOF ( #589 )
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
2019-03-13 15:33:58 +00:00
437bfa90c2
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
...
Contributed by Steve Loughran.
(cherry picked from commit f1fbc56371
2018-10-12 09:49:28 +01:00
11e7f2e584
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled.
...
Contributed by Anis Elleuch.
(cherry picked from commit 1dedc68f9dcc0f14c13c
2018-06-25 17:27:49 +01:00
312c571694
HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common
...
Contributed by Genmao Yu
2018-01-27 20:31:21 -08:00
28e05dc1da
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests.
...
Contributed by Steve Loughran
2017-10-03 20:06:45 +01:00
a1afc6aa59
HADOOP-13345 S3Guard: Improved Consistency for S3A.
...
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
(cherry-picked from/based on commit 621b43e254
2017-09-29 11:29:22 +01:00
27a41848ac
HADOOP-14890 Move up to AWS SDK 1.11.199.
...
Contributed by Steve Loughran.
2017-09-26 20:11:10 +01:00
6e20ae954c
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter.
...
Contributed by Yonger.
(cherry picked from commit 83449abb40
2017-09-07 19:58:40 +01:00
4c34c03b27
HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge.
2017-09-06 01:03:18 -07:00
2d69952925
HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
...
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
(cherry picked from commit 72993b33b7
2017-06-29 17:08:51 -07:00
44d43a8848
HADOOP-14190. Add more on S3 regions to the s3a documentation.
...
Contributed by Steve Loughran
(cherry picked from commit ee243e5289
2017-06-28 10:26:05 +01:00
ea402c364f
HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
...
(cherry picked from commit bc4dfe9c9c
2017-06-27 13:20:03 -07:00
867026d94e
HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu
...
(cherry picked from commit 6aeda55bb8
2017-06-05 11:31:13 -07:00
f72f194e24
HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
...
(cherry picked from commit 6c6a7a5962
2017-05-28 22:29:03 -07:00
2ac5aab8d7
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
...
Contributed by Steve Loughran.
2017-05-18 15:38:46 +01:00
d1da5ba3af
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
...
Contributed by Steve Moist.
2017-04-25 17:42:30 +01:00
3b7bb7b94b
HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran
2017-04-20 17:07:57 -07:00
508389ee54
HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu
...
(cherry picked from commit 507845074970b90330db86bc1ff0124def550bb5)
2017-04-12 14:30:54 -07:00
1bfb382296
HADOOP-14301. Deprecate SharedInstanceProfileCredentialsProvider in branch-2. Contributed by Mingliang Liu.
2017-04-12 10:07:00 -07:00
8c21b2a259
HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka
...
(cherry picked from commit 5faa949b78
2017-04-03 11:07:56 -07:00
bb187e145a
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
...
(cherry picked from commit 0d053eeb30
2017-03-30 18:13:01 +09:00
b24b416db5
HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu
...
(cherry picked from commit 2e30aa72e0
2017-03-23 11:37:14 -07:00
20878d052c
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject().
...
Contributed by Yuanbo Liu
2017-03-23 12:52:22 +00:00
b4481fef41
HADOOP-14204 S3A multipart commit failing,
...
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
Contributed by Steve Loughran
(cherry picked from commit 595501c9dd
2017-03-21 13:14:43 +00:00
90e3283a0a
HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran.
2017-03-20 11:29:30 -07:00
ed0d426a88
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
2017-03-14 15:00:59 -07:00
51fefbe38b
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
...
Contributed by Steve Loughran
2017-03-02 21:41:49 +00:00
29fe5af017
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
...
Contributed by Steve Loughran.
2017-02-25 15:31:26 +00:00
c6d7da2112
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
...
(cherry picked from commit ff87ca8441
2017-02-24 10:40:58 +00:00
db955f0844
HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu
...
(cherry picked from commit 0013090fb4
2017-02-22 14:38:57 -08:00
171b18693a
HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran.
2017-02-22 11:41:24 +00:00
6843659689
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
2017-02-21 18:29:30 +00:00
274c02d2fa
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
2017-02-20 16:21:00 +00:00
8a88e8ea57
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
...
(cherry picked from commit b1c1f05b1d
2017-02-18 18:17:54 +00:00
29e1007172
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
...
(cherry picked from commit bdad8b7b97
2017-02-16 16:42:56 -08:00
25ac54a52e
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
...
(cherry picked from commit b9f8491252
2017-02-15 01:46:59 +09:00
6d62d0ea87
HADOOP-13204. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
2017-02-11 15:02:19 -08:00
ec2f5b18ab
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
2017-01-11 17:24:25 +00:00
2dbd86bcf4
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
...
(cherry picked from commit c6a3923245
2016-12-12 15:06:39 -08:00
c37e814fae
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
...
(cherry picked from commit 6b6bc4705d2b608e810ae7d1912eac49bfd4c6ca)
2016-12-02 13:48:38 -08:00
a4e0539636
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
...
(cherry picked from commit 2ff84a0040
2016-12-02 13:39:23 -08:00
39b74b6f9e
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
...
(cherry picked from commit 3fd844b99f
2016-11-30 13:04:44 -08:00
1b831d8e01
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
...
(cherry picked from commit d60a60be8a
2016-11-28 16:35:18 -08:00
9701ab48e8
HADOOP-12804. Read Proxy Password from Credential Providers in S3 FileSystem. Contributed by Larry McCay.
2016-11-25 13:34:11 +00:00
yzhangal
Steve Loughran
Jonathan Hung
Mingliang Liu
Steve Loughran
John Zhuge
Chris Nauroth
Akira Ajisaka
Arpit Agarwal
Lei Xu