511df1e837
HADOOP-16430. S3AFilesystem.delete to incrementally update s3guard with deletions
...
Contributed by Steve Loughran.
This overlaps the scanning for directory entries with batched calls to S3 DELETE and updates of the S3Guard tables.
It also uses S3Guard to list the files to delete, so find newly created files even when S3 listings are not use consistent.
For path which the client considers S3Guard to be authoritative, we also do a recursive LIST of the store and delete files; this is to find unindexed files and do guarantee that the delete(path, true) call really does delete everything underneath.
Change-Id: Ice2f6e940c506e0b3a78fa534a99721b1698708e
2019-09-05 14:25:15 +01:00
6f068cf53f
HADOOP-16416. mark DynamoDBMetadataStore.deleteTrackingValueMap as final. Contributed by kevin su.
...
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2019-08-27 16:17:32 -07:00
23e532d739
Revert "HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran"
...
This reverts commit 69ddb36876
2019-08-26 12:37:26 +02:00
61b2df2331
HADOOP-16470. Make last AWS credential provider in default auth chain EC2ContainerCredentialsProviderWrapper.
...
Contributed by Steve Loughran.
Contains HADOOP-16471. Restore (documented) fs.s3a.SharedInstanceProfileCredentialsProvider.
Change-Id: I06b99b57459cac80bf743c5c54f04e59bb54c2f8
2019-08-22 17:27:56 +01:00
69ddb36876
HADOOP-16193. Add extra S3A MPU test to see what happens if a file is created during the MPU. Contributed by Steve Loughran
2019-08-22 13:56:47 +02:00
0e4b757955
HADOOP-16500 S3ADelegationTokens to only log at debug on startup ( #1269 ). Contributed by Steve Loughran.
...
Change-Id: Ifafc15f32791911976d7ebc36fb6e8853f59ed41
2019-08-14 10:50:26 +02:00
189dc10884
HADOOP-16481. ITestS3GuardDDBRootOperations.test_300_MetastorePrune needs to set region. ( #1209 ). Contributed by Steve Loughran.
2019-08-09 17:33:08 +02:00
e25a5c2eab
HADOOP-16499. S3A retry policy to be exponential ( #1246 ). Contributed by Steve Loughran.
2019-08-09 15:52:37 +02:00
b01efe5cf6
HADOOP-16472. findbugs warning on LocalMetadataStore.ttlTimeProvider sync
...
Contributed by Steve Loughran.
Moved the setter and addAncestors to synchronized
Change-Id: Ib362c66d1b8c9124eca7db9a44274ac08d0b3be6
2019-08-02 22:30:48 +01:00
bca86bd289
HADOOP-16469. Update committers.md
...
Contributed by Felipe Lopes.
Change-Id: I5c05b878bde073aeb45bf22340183893f85269e1
2019-07-30 12:47:55 +01:00
7b219778e0
HADOOP-16433. S3Guard: Filter expired entries and tombstones when listing with MetadataStore.listChildren().
...
Contributed by Gabor Bota.
This pulls the tracking of the lastUpdated timestamp of metadata entries up from the DDB metastore into all s3guard stores, and then uses this to filter out expired tombstones from listings.
Change-Id: I80f121236b49c75a024116f65a3ef29d3580b462
2019-07-24 18:11:43 +01:00
4317d33232
HADOOP-16380. S3Guard to determine empty directory status for all non-root directories.
...
Contributed by Steve Loughran and Gabor Bota.
This
* Asks S3Guard to determine the empty directory status.
* Has S3A's root directory rm("/") command to always return false (as abfs does)
* Documents that object stores MAY do this
* Overloads ContractTestUtils.assertDeleted to let assertions declare that the source directory does not need to exist. This stops inconsistencies in directory listings failing a root test.
It avoids a recent regression (HADOOP-16279) where if there was a tombstone above the first element found in a directory listing, the directory would be considered empty, when in fact there were child entries. That could downgrade an rm(path, recursive) to a no-op, while also confusing rename(src, dest), as dest could be mistaken for an empty directory and so permit the copy above it, rather than reject it "destination path exists and is not empty".
Change-Id: I136a3d1a5a48a67e6155d790a40ff558d0d2c108
2019-07-23 14:52:03 +01:00
7f1b76ca35
HADOOP-13868. [s3a] New default for S3A multi-part configuration ( #1125 )
2019-07-19 09:49:59 -06:00
cd967c75a7
HADOOP-15847. S3Guard testConcurrentTableCreations to set R/W capacity == 0
...
Contributed by lqjaclee
Change-Id: I4a4d5b29f2677c188799479e4db38f07fa0591d1
2019-07-19 14:46:55 +01:00
d545f9c290
HADOOP-16437 documentation typo fix: fs.s3a.experimental.input.fadvise
...
Fix fs.s3a.experimental.fadvise to fs.s3a.experimental.input.fadvise
Contributed by: Josh Rosen
2019-07-18 23:19:38 +01:00
c58e11bf52
HADOOP-16383. Pass ITtlTimeProvider instance in initialize method in MetadataStore interface. Contributed by Gabor Bota. ( #1009 )
2019-07-17 16:24:39 +02:00
5672efa5c7
HADOOP-15729. [s3a] Allow core threads to time out. ( #1075 )
2019-07-16 18:14:23 -06:00
b15ef7dc3d
HADOOP-16384: S3A: Avoid inconsistencies between DDB and S3.
...
Contributed by Steve Loughran
Contains
- HADOOP-16397. Hadoop S3Guard Prune command to support a -tombstone option.
- HADOOP-16406. ITestDynamoDBMetadataStore.testProvisionTable times out intermittently
This patch doesn't fix the underlying problem but it
* changes some tests to clean up better
* does a lot more in logging operations in against DDB, if enabled
* adds an entry point to dump the state of the metastore and s3 tables (precursor to fsck)
* adds a purge entry point to help clean up after a test run has got a store into a mess
* s3guard prune command adds -tombstone option to only clear tombstones
The outcome is that tests should pass consistently and if problems occur we have better diagnostics.
Change-Id: I3eca3f5529d7f6fec398c0ff0472919f08f054eb
2019-07-12 13:02:25 +01:00
6a3433bffd
HADOOP-16357. TeraSort Job failing on S3 DirectoryStagingCommitter: destination path exists.
...
Contributed by Steve Loughran.
This patch
* changes the default for the staging committer to append, as we get for the classic FileOutputFormat committer
* adds a check for the dest path being a file not a dir
* adds tests for this
* Changes AbstractCommitTerasortIT. to not use the simple parser, so fails if the file is present.
Change-Id: Id53742958ed1cf321ff96c9063505d64f3254f53
2019-07-11 18:15:34 +01:00
c7b5f858a0
HADOOP-16393. S3Guard init command uses global settings, not those of target bucket.
...
Contributed by Steve Loughran.
Change-Id: I226a91ab8d7758340f8d221aa80a7abf9a0d3e8f
2019-07-10 20:57:02 +01:00
de6b7bc67a
HADOOP-16409. Allow authoritative mode on non-qualified paths. Contributed by Sean Mackrory
2019-07-08 19:27:07 +02:00
34747c373f
HADOOP-16396. Allow authoritative mode on a subdirectory. ( #1043 )
2019-07-03 12:04:47 -06:00
366f3deec5
HADOOP-16390. escape javadoc in S3AUtils public methods
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-06-25 17:47:37 +09:00
e02eb24e0a
HADOOP-15183. S3Guard store becomes inconsistent after partial failure of rename.
...
Contributed by Steve Loughran.
Change-Id: I825b0bc36be960475d2d259b1cdab45ae1bb78eb
2019-06-20 09:56:40 +01:00
28291a9e8a
HADOOP-16379: S3AInputStream.unbuffer should merge input stream stats into fs-wide stats
...
Contributed by Sahil Takiar
Change-Id: I2bcfaaea00d12c633757069402dcd0b91a5f5c05
2019-06-20 09:42:27 +01:00
f9cc9e1621
HADOOP-16279. S3Guard: Implement time-based (TTL) expiry for entries (and tombstones).
...
Contributed by Gabor Bota.
Change-Id: I73a2d2861901dedfe7a0e783b310fbb95e7c1af9
2019-06-16 17:05:01 +01:00
4e38dafde4
HADOOP-15563. S3Guard to support creating on-demand DDB tables.
...
Contributed by Steve Loughran
Change-Id: I2262b5b9f52e42ded8ed6f50fd39756f96e77087
2019-06-07 18:26:10 +01:00
ec26c431f9
HADOOP-16117. Update AWS SDK to 1.11.563.
...
Contributed by Steve Loughran.
Change-Id: I7c46ed2a6378e1370f567acf4cdcfeb93e43fa13
2019-06-06 10:08:18 +01:00
309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
...
This reverts commit b067f8acaa
2019-06-05 13:54:55 +01:00
7724d8031b
Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
...
This reverts commit 5906268f0d
2019-06-05 13:54:42 +01:00
afd844059c
HADOOP-16331. Fix ASF License check in pom.xml
...
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
0c73dba3a6
HADOOP-16332. Remove S3A dependency on http core.
...
Contributed by Steve Loughran.
Change-Id: I53209c993a405fefdb5e1b692d5a56d027d3b845
2019-05-28 22:50:37 +01:00
9f933e6446
HADOOP-16323. https everywhere in Maven settings.
2019-05-27 15:24:59 +09:00
5906268f0d
HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors
2019-05-21 11:30:45 -06:00
a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
...
Contributed by Ben Roling.
S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.
You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.
This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.
For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
b067f8acaa
HADOOP-16050: s3a SSL connections should use OpenSSL
...
(cherry picked from commit aebf229c17
2019-05-16 08:57:54 -06:00
591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
2019-05-09 21:13:29 +01:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests.
...
Contributed by Ben Roling.
2019-04-24 14:55:58 +01:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables.
...
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.
Contributed by Steve Loughran.
2019-04-11 17:12:12 -07:00
215ffc792e
HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException
...
Contributed by Steve Loughran.
Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
2019-04-04 21:14:18 +01:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
df578c07ec
HADOOP-16195 MarshalledCredentials toString
...
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
2019-03-28 17:01:57 +00:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren.
...
Author: Gabor Bota
2019-03-28 15:49:56 +00:00
ae2eb2dd42
HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists ( #636 )
2019-03-22 11:42:00 +00:00
9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
2019-03-18 19:13:13 +00:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
...
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
(Revised patch applied after stevel committed the wrong one; that has been reverted)
2019-02-19 11:33:02 +00:00
920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."
...
This reverts commit c4a00d1ad3
2019-02-18 14:57:22 +00:00
6c999fe4b0
HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki.
2019-02-12 06:07:47 +09:00
de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
2019-01-25 10:27:59 +09:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
2019-01-14 17:27:00 +00:00
04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set.
...
Contributed by Gabor Bota.
2019-01-14 13:12:05 +00:00
3420e26ae5
HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota.
2019-01-09 10:57:58 -07:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
d8f670ff28
HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal.
2018-12-27 08:19:25 -07:00
a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies.
...
Contributed by Steve Loughran.
2018-12-20 18:09:01 +00:00
82b798581d
HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota.
2018-12-12 09:30:13 -07:00
1a25bbe9ec
HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota.
2018-12-11 08:33:13 -07:00
c35de95a22
HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota.
2018-12-11 08:29:39 -07:00
3ff8580f22
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota)
2018-12-10 14:03:08 -07:00
7eb0d3a324
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
2018-11-29 09:36:39 -07:00
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
085f10e75d
HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
e148c3ff09
HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
5d96b74f33
HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
2018-11-27 20:39:54 +00:00
66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia.
2018-11-15 16:30:24 +09:00
d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong.
...
Contributed by Steve Loughran and Ewan Higgs.
2018-10-16 20:02:54 +01:00
ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
...
Contributed by Steve Loughran.
2018-10-11 14:57:38 +01:00
7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard.
...
Contributed by Gabor Bota
2018-10-09 10:46:41 +01:00
4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-03 00:24:29 -07:00
046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota
2018-10-02 21:22:49 -07:00
58fa96b697
Changed version in trunk to 3.3.0-SNAPSHOT.
2018-10-02 22:41:41 +05:30
c07715e378
HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran
2018-09-24 12:53:21 -07:00
d060cbea48
HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran.
2018-09-24 21:19:47 +05:30
26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config.
...
Contributed by Gabor Bota.
2018-09-17 22:40:08 +01:00
d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran)
2018-09-12 21:04:49 -07:00
d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine)
2018-09-12 16:36:01 -07:00
1f6c4545cf
HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran
2018-09-12 10:58:39 -07:00
47b72c87eb
HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded.
...
Contributed by Gabor Bota.
2018-09-12 09:12:38 -06:00
87f63b6479
HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran
2018-09-11 17:18:42 -07:00
36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota)
2018-09-07 10:25:20 -07:00
5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
...
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
...
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota)
2018-08-17 10:15:39 -07:00
0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375.
...
Contributed by Steve Loughran.
2018-08-16 09:58:46 -07:00
3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering.
2018-08-16 00:31:59 +09:00
a13929ddcb
HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran.
2018-08-13 12:57:45 +02:00
da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
...
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
2ec97abb2e
HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran.
2018-08-08 13:50:23 +02:00
7862f1523f
HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota)
2018-08-07 20:13:09 -06:00
48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
...
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
59adeb8d7f
HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota.
2018-07-27 18:23:29 -06:00
a08812a1b1
HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota.
2018-07-12 17:24:01 +02:00
d503f65b66
HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
...
connection in case of timeout.
2018-07-10 17:52:57 +02:00
93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota)
2018-07-03 13:50:11 -07:00
2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma.
2018-06-28 14:37:22 +09:00
c687a6617d
HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota.
2018-06-25 14:59:41 -06:00
55fad6a3de
HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota.
2018-06-22 11:36:56 -06:00
b089a06793
HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota.
2018-06-20 16:45:08 -06:00
980031bb04
HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs
2018-06-17 11:54:26 -07:00
5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota)
2018-05-29 19:20:22 -07:00
30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota.
2018-05-29 13:38:15 -07:00
cc3600aabd
HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
...
Contributed by Steve Loughran.
(cherry picked from commit 170f1040d4
2018-05-17 19:08:55 +01:00
b670837469
HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name.
...
Contributed by Sean Mackrory.
2018-05-15 16:13:56 +01:00
8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota)
2018-05-08 18:58:42 -07:00
69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota)
2018-05-08 15:29:54 -07:00
96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri)
2018-05-04 11:34:45 -07:00
919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota.
2018-04-30 16:02:57 -07:00
7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota)
2018-04-26 20:41:29 -07:00
989a3929a9
HADOOP-14756 S3Guard: expose capability query in MetadataStore and add tests of authoritative mode (Gabor Bota)
2018-04-23 15:26:53 -07:00
1312f9ae4c
HADOOP-15391. Add missing css file in hadoop-aws, hadoop-aliyun, hadoop-azure and hadoop-azure-datalake modules.
2018-04-18 16:04:00 +08:00
b0aff8a962
HADOOP-15346 S3ARetryPolicy for 400/BadArgument to be "fail". Contributed by Steve Loughran.
2018-04-10 23:55:38 -07:00
ea3849f0cc
HADOOP-14759 S3GuardTool prune to prune specific bucket entries. Contributed by Gabor Bota.
2018-04-05 20:23:17 -07:00
5a174f8ac6
HADOOP-14758. S3GuardTool.prune to handle UnsupportedOperationException.
...
Contributed by Gabor Bota.
2018-04-03 16:31:34 +01:00
7ce6b41509
HADOOP-15332. Fix typos in hadoop-aws markdown docs. Contributed by Gabor Bota.
2018-03-20 21:12:20 -07:00
1976e0066e
HADOOP-15209. DistCp to eliminate needless deletion of files under already-deleted directories.
...
Contributed by Steve Loughran.
2018-03-15 18:05:14 +00:00
45d1b0fdcc
HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer
2018-03-12 20:05:39 -07:00
dd05871b8b
HADOOP-15297. Make S3A etag => checksum feature optional.
...
Contributed by Steve Loughran.
2018-03-12 14:01:42 +00:00
e0307e53e2
HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled.
...
Contributed by Anis Elleuch.
(cherry picked from commit 1dedc68f9d
2018-03-07 18:51:10 +00:00
8110d6a0d5
HADOOP-13761. S3Guard: implement retries for DDB failures and throttling; translate exceptions.
...
Contributed by Aaron Fabbri.
2018-03-05 14:06:20 +00:00
4d4dde5112
HADOOP-15223. Replace Collections.EMPTY* with empty* when available
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-02-18 22:19:39 +09:00
7ac88244c5
HADOOP-14507. Extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key.
...
Contributed by Steve Loughran.
2018-02-16 16:37:06 +00:00
9a013b255f
HADOOP-15176. Enhance IAM Assumed Role support in S3A client.
...
Contributed by Steve Loughran
(cherry picked from commit 96c047fbb9
2018-02-15 15:57:10 +00:00
b27ab7dd81
HADOOP-15076. Enhance S3A troubleshooting documents and add a performance document.
...
Contributed by Steve Loughran.
(cherry picked from commit c761e658f6
2018-02-15 14:57:56 +00:00
60f9e60b3b
Preparing for 3.2.0 development
...
Change-Id: I6d0e01f3d665d26573ef2b957add1cf0cddf7938
2018-02-11 11:17:38 +08:00
1093a73689
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2018-01-18 13:13:58 +00:00
f274fe33ea
Revert "HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits."
...
This reverts commit 35ad9b1dd2
2018-01-18 12:35:57 +00:00
268ab4e027
HADOOP-15141 Support IAM Assumed roles in S3A. Contributed by Steve Loughran.
2018-01-17 00:05:24 -08:00
a0c71dcc33
HADOOP-15079. ITestS3AFileOperationCost#testFakeDirectoryDeletion failing
...
after OutputCommitter patch.
Contributed by Steve Loughran
2018-01-15 11:33:32 +00:00
2e1e9017aa
HADOOP-15172. Fix the javadoc warning in WriteOperationHelper.java
...
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-01-15 15:45:51 +09:00
1a09da7400
HADOOP-15163. Fix S3ACommitter documentation
...
Contributed by Alessandro Andrioni.
(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
2018-01-10 15:37:07 +00:00
b62a5ece95
HADOOP-15161. s3a: Stream and common statistics missing from metrics
...
Contributed by Sean Mackrory
2018-01-09 18:46:52 +00:00
c8ff0cc304
HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls.
...
Contributed by Steve Loughran
2017-12-21 14:58:58 +00:00
ef450df443
HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance.
...
Contributed by Steve Loughran.
2017-12-21 14:15:53 +00:00
1ba491ff90
HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive
2017-12-20 18:25:33 +00:00
35ad9b1dd2
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
...
Contributed by Aaron Fabbri
2017-12-18 21:19:06 +00:00
f86c81d923
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
2017-12-14 11:14:52 +08:00
28792b6b7f
Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu"
...
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
2017-12-14 11:05:11 +08:00
Steve Loughran
HUAN-PING SU
Ewan Higgs
Ewan Higgs
Felipe Lopes
Gabor Bota
Sean Mackrory
lqjaclee
Josh Rosen
kkori
Sahil Takiar
Akira Ajisaka
Ben Roling
Rajat Khandelwal
Ben Roling
Steve Loughran
Lokesh Jain
Adam Antal
Abhishek Modi
Masatake Iwasaki
Andrew Olson
Sean Mackrory
Sean Mackrory
Aaron Fabbri
Sunil G
Mingliang Liu
Chris Douglas
Yiqun Lin
fang zhenyi
Wangda Tan
Mukul Kumar Singh
Kai Zheng