d19981fe48
HADOOP-16758. Refine testing.md to tell user better how to use auth-keys.xml ( #1753 )
...
Contributed by Mingliang Liu
2019-12-11 11:54:12 -08:00
8a60429e0b
HADOOP-16735. Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN. Contributed by Mingliang Liu
...
This closes #1733
2019-12-05 17:50:28 -08:00
7d5bb2ebb7
Preparing for 3.2.2-SNAPSHOT development.
2019-09-07 08:52:08 +05:30
a9a3450560
HADOOP-16331. Fix ASF License check in pom.xml. Contributed by Akira Ajisaka.
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:34:16 +09:00
855dc997d6
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:27:33 +09:00
12e0053932
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
(cherry picked from commit 591ca69823
2019-05-09 21:14:37 +01:00
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
2019-04-03 21:35:19 +01:00
60c9042286
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-29 15:25:45 +00:00
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
(cherry picked from commit c0427c84dd
2019-03-18 19:14:43 +00:00
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
36f3e775d4
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
(cherry picked from commit de804e53b9
2019-02-07 10:09:13 +00:00
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
(cherry picked from commit c1d24f8483
2019-02-04 17:10:19 +00:00
dc12754ab6
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
...
(cherry picked from commit 3c60303ac5
2019-01-25 10:28:46 +09:00
fa1d4ba7d4
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
(cherry picked from commit 4c106fca0c
2018-11-27 20:40:48 +00:00
8c9681d7f0
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia.
...
(cherry picked from commit 66b1335bb3
2018-11-15 16:31:05 +09:00
bde4fd5ed9
Preparing for 3.2.0 release
2018-10-18 17:07:45 +05:30
c19864d8c8
HADOOP-15848. ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error.
...
Contributed by Ewan Higgs.
2018-10-16 19:58:58 +01:00
70d39c74d9
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
...
Contributed by Steve Loughran.
(cherry picked from commit ee816f1fd7
2018-10-11 14:58:37 +01:00
c07715e378
HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran
2018-09-24 12:53:21 -07:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config.
...
Contributed by Gabor Bota.
2018-09-17 22:40:08 +01:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine)
2018-09-12 16:36:01 -07:00
1f6c4545cf
HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran
2018-09-12 10:58:39 -07:00
47b72c87eb
HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded.
...
Contributed by Gabor Bota.
2018-09-12 09:12:38 -06:00
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota)
2018-09-07 10:25:20 -07:00
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
...
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota)
2018-08-17 10:15:39 -07:00
0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375.
...
Contributed by Steve Loughran.
2018-08-16 09:58:46 -07:00
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
a13929ddcb
HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran.
2018-08-13 12:57:45 +02:00
da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
...
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
7862f1523f
HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota)
2018-08-07 20:13:09 -06:00
48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
...
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
59adeb8d7f
HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota.
2018-07-27 18:23:29 -06:00
a08812a1b1
HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota.
2018-07-12 17:24:01 +02:00
d503f65b66
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
...
connection in case of timeout.
2018-07-10 17:52:57 +02:00
93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota)
2018-07-03 13:50:11 -07:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
c687a6617d
HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota.
2018-06-25 14:59:41 -06:00
55fad6a3de
HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota.
2018-06-22 11:36:56 -06:00
b089a06793
HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota.
2018-06-20 16:45:08 -06:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota)
2018-05-29 19:20:22 -07:00
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota.
2018-05-29 13:38:15 -07:00
cc3600aabd
HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
...
Contributed by Steve Loughran.
(cherry picked from commit 170f1040d4
2018-05-17 19:08:55 +01:00
Mingliang Liu
Rohith Sharma K S
Takanobu Asanuma
Akira Ajisaka
Rajat Khandelwal
Steve Loughran
Adam Antal
Ben Roling
Andrew Olson
Sunil G
Aaron Fabbri
Sean Mackrory
Gabor Bota
Ewan Higgs
Chris Douglas