Steve Loughran
|
1a09da7400
|
HADOOP-15163. Fix S3ACommitter documentation
Contributed by Alessandro Andrioni.
(cherry picked from commit 100e8a1ae1d930dde084af7d1281e491c7f124ec)
|
2018-01-10 15:37:07 +00:00 |
|
Steve Loughran
|
b62a5ece95
|
HADOOP-15161. s3a: Stream and common statistics missing from metrics
Contributed by Sean Mackrory
|
2018-01-09 18:46:52 +00:00 |
|
Steve Loughran
|
c8ff0cc304
|
HADOOP-13282. S3 blob etags to be made visible in S3A status/getFileChecksum() calls.
Contributed by Steve Loughran
|
2017-12-21 14:58:58 +00:00 |
|
Steve Loughran
|
ef450df443
|
HADOOP-15113. NPE in S3A getFileStatus: null instrumentation on using closed instance.
Contributed by Steve Loughran.
|
2017-12-21 14:15:53 +00:00 |
|
Steve Loughran
|
1ba491ff90
|
HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive
|
2017-12-20 18:25:33 +00:00 |
|
Steve Loughran
|
35ad9b1dd2
|
HADOOP-13974. S3Guard CLI to support list/purge of pending multipart commits.
Contributed by Aaron Fabbri
|
2017-12-18 21:19:06 +00:00 |
Kai Zheng
|
f86c81d923
|
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
|
2017-12-14 11:14:52 +08:00 |
|
Kai Zheng
|
28792b6b7f
|
Revert "HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu"
This reverts commit 479d6a5792262c977025c26fd4960574b0db6847
|
2017-12-14 11:05:11 +08:00 |
|
Kai Zheng
|
73b86979d6
|
HADOOP-15039. Move SemaphoredDelegatingExecutor to hadoop-common. Contributed by Genmao Yu
|
2017-12-06 12:01:36 +08:00 |
Aaron Fabbri
|
6555af81a2
|
HADOOP-14475 Metrics of S3A don't print out when enabled. Contributed by Younger and Sean Mackrory.
|
2017-12-05 11:06:32 -08:00 |
|
Steve Loughran
|
3150c019ae
|
HADOOP-15071 S3a troubleshooting docs to add a couple more failure modes.
Contributed by Steve Loughran
|
2017-12-05 15:05:41 +00:00 |
Rohith Sharma K S
|
3cd75845da
|
MAPREDUCE-7014. Fix java doc errors in jdk1.8. Contributed by Steve Loughran.
|
2017-11-27 22:01:00 +05:30 |
|
Steve Loughran
|
de8b6ca5ef
|
HADOOP-13786 Add S3A committer for zero-rename commits to S3 endpoints.
Contributed by Steve Loughran and Ryan Blue.
|
2017-11-22 15:28:12 +00:00 |
Akira Ajisaka
|
6903cf096e
|
HADOOP-13514. Upgrade maven surefire plugin to 2.20.1
Signed-off-by: Allen Wittenauer <aw@apache.org>
|
2017-11-19 12:39:37 -08:00 |
|
Aaron Fabbri
|
47011d7dd3
|
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran
|
2017-09-25 15:59:38 -07:00 |
|
Aaron Fabbri
|
49467165a5
|
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran.
|
2017-09-14 14:10:48 -07:00 |
|
Steve Loughran
|
5bbca80428
|
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A.
Contributed by Aaron Fabbri
|
2017-09-08 12:07:02 +01:00 |
|
Steve Loughran
|
83449abb40
|
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter.
Contributed by Yonger.
|
2017-09-07 19:55:56 +01:00 |
John Zhuge
|
50506e90a8
|
HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge.
|
2017-09-05 23:26:57 -07:00 |
Andrew Wang
|
0d419c984f
|
Preparing for 3.1.0 development
|
2017-09-01 11:53:48 -07:00 |
|
Steve Loughran
|
621b43e254
|
HADOOP-13345 HS3Guard: Improved Consistency for S3A.
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
|
2017-09-01 14:13:41 +01:00 |
|
Steve Loughran
|
7fc324aabd
|
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws.
Contributed by Steve Loughran
(cherry picked from commit ced547d5f0)
|
2017-08-04 11:09:08 +01:00 |
|
Andrew Wang
|
af2773f609
|
Updating version for 3.0.0-beta1 development
|
2017-06-29 17:57:40 -07:00 |
Mingliang Liu
|
72993b33b7
|
HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
|
2017-06-29 17:07:52 -07:00 |
|
Steve Loughran
|
ee243e5289
|
HADOOP-14190. Add more on S3 regions to the s3a documentation.
Contributed by Steve Loughran
|
2017-06-28 10:22:13 +01:00 |
|
Mingliang Liu
|
bc4dfe9c9c
|
HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu
|
2017-06-27 13:19:14 -07:00 |
|
Mingliang Liu
|
f214a9961f
|
HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran
|
2017-06-15 11:16:16 -07:00 |
|
Mingliang Liu
|
6aeda55bb8
|
HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu
|
2017-06-05 11:26:56 -07:00 |
|
John Zhuge
|
6c6a7a5962
|
HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge.
|
2017-05-28 22:25:00 -07:00 |
|
Andrew Wang
|
16ad896d5c
|
Update maven version for 3.0.0-alpha4 development
|
2017-05-26 14:09:44 -07:00 |
|
Akira Ajisaka
|
6a52b5e144
|
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu.
|
2017-05-25 15:36:44 +09:00 |
|
Mingliang Liu
|
6672810eea
|
HADOOP-14432. S3A copyFromLocalFile to be robust, tested. Contributed by Steve Loughran
|
2017-05-19 11:51:43 -07:00 |
|
Steve Loughran
|
ba70225cf6
|
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries.
Contributed by Steve Loughran.
(cherry picked from commit 2ac5aab8d7)
|
2017-05-18 15:44:39 +01:00 |
|
Akira Ajisaka
|
0d5c8ed8e0
|
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor.
|
2017-05-11 16:37:32 -05:00 |
|
Steve Loughran
|
5f934f8386
|
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus.
Contributed by Steve Moist.
|
2017-04-24 20:33:19 +01:00 |
|
Mingliang Liu
|
667966c13c
|
HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran
|
2017-04-20 17:13:36 -07:00 |
|
Steve Loughran
|
6b015d00c9
|
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs.
Contributed by Steve Loughran
|
2017-04-19 10:21:44 +01:00 |
|
Mingliang Liu
|
b053fdc547
|
HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu
|
2017-04-12 14:30:34 -07:00 |
Chris Nauroth
|
b8305e6d06
|
HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu.
|
2017-04-12 10:02:13 -07:00 |
|
Mingliang Liu
|
5faa949b78
|
HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka
|
2017-04-03 11:07:14 -07:00 |
|
Akira Ajisaka
|
0d053eeb30
|
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula.
|
2017-03-30 18:11:50 +09:00 |
|
Andrew Wang
|
4966a6e26e
|
HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy.
|
2017-03-29 14:37:21 -07:00 |
|
Andrew Wang
|
52b00600df
|
HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy.
|
2017-03-24 11:44:46 -07:00 |
|
Mingliang Liu
|
2e30aa72e0
|
HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu
|
2017-03-23 11:33:29 -07:00 |
|
Steve Loughran
|
a5a4867f3b
|
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject().
Contributed by Yuanbo Liu
(cherry picked from commit 20878d052c)
|
2017-03-23 12:54:54 +00:00 |
|
Steve Loughran
|
2841666f1f
|
HADOOP-14204 S3A multipart commit failing,
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".
Contributed by Steve Loughran
(cherry picked from commit 595501c9dd)
|
2017-03-21 13:15:15 +00:00 |
Arpit Agarwal
|
6c399a88e9
|
HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran.
|
2017-03-20 11:29:23 -07:00 |
|
Mingliang Liu
|
fa67a96d7b
|
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
|
2017-03-14 14:38:21 -07:00 |
|
Mingliang Liu
|
e6cda5819b
|
Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu"
This reverts commit b8c69557b7.
|
2017-03-14 13:03:42 -07:00 |
|
Mingliang Liu
|
b8c69557b7
|
HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu
|
2017-03-13 14:15:02 -07:00 |
|
Steve Loughran
|
092ec39fb9
|
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials.
Contributed by Yuanbo Liu
|
2017-03-10 17:43:22 +00:00 |
|
Akira Ajisaka
|
f597f4c43e
|
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri.
|
2017-03-07 15:14:55 +09:00 |
|
Steve Loughran
|
a97833e0ed
|
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry.
Contributed by Steve Loughran
(cherry picked from commit 51fefbe38b)
|
2017-03-02 21:44:30 +00:00 |
|
Akira Ajisaka
|
258342e76c
|
HADOOP-14118. move jets3t into a dependency on hadoop-aws JAR.
|
2017-02-28 13:47:44 +09:00 |
|
Steve Loughran
|
dab00da19f
|
HADOOP-14028. S3A BlockOutputStreams doesn't delete temporary files in multipart uploads or handle part upload failures.
Contributed by Steve Loughran.
(cherry picked from commit 29fe5af017)
|
2017-02-25 15:35:19 +00:00 |
|
Steve Loughran
|
9c22a91662
|
HADOOP-14114 S3A can no longer handle unencoded + in URIs. Contributed by Sean Mackrory.
(cherry picked from commit ff87ca8441)
|
2017-02-24 10:41:36 +00:00 |
|
Mingliang Liu
|
a4d4a23785
|
HADOOP-13321. Deprecate FileSystem APIs that promote inefficient call patterns. Contributed by Chris Nauroth and Mingliang Liu
|
2017-02-23 12:55:40 -08:00 |
|
Mingliang Liu
|
0013090fb4
|
HADOOP-14102. Relax error message assertion in S3A test ITestS3AEncryptionSSEC. Contributed by Mingliang Liu
|
2017-02-22 13:34:20 -08:00 |
|
Steve Loughran
|
4f4250fbcc
|
HADOOP-14099 Split S3 testing documentation out into its own file. Contributed by Steve Loughran.
|
2017-02-22 11:43:48 +00:00 |
|
Steve Loughran
|
2158496f6b
|
HADOOP-13826. S3A Deadlock in multipart copy due to thread pool limits. Contributed by Sean Mackrory.
(cherry picked from commit e3a74e0369e6e2217d1280179b390227fe1b1684)
|
2017-02-21 18:28:49 +00:00 |
|
Steve Loughran
|
8035749c26
|
HADOOP-14081. S3A: Consider avoiding array copy in S3ABlockOutputStream (ByteArrayBlock). Contributed by Rajesh Balamohan
|
2017-02-20 16:21:46 +00:00 |
|
Steve Loughran
|
3a2e30fa9f
|
HADOOP-14092. Typo in hadoop-aws index.md. Contributed by John Zhuge
(cherry picked from commit b1c1f05b1d)
|
2017-02-18 18:17:11 +00:00 |
|
Mingliang Liu
|
658702efff
|
HADOOP-14040. Use shaded aws-sdk uber-JAR 1.11.86. Contributed by Steve Loughran and Sean Mackrory
|
2017-02-16 16:51:03 -08:00 |
|
Mingliang Liu
|
bdad8b7b97
|
HADOOP-14019. Fix some typos in the s3a docs. Contributed by Steve Loughran
|
2017-02-16 16:41:31 -08:00 |
|
Akira Ajisaka
|
b9f8491252
|
HADOOP-14058. Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks. Contributed by Yiqun Lin.
|
2017-02-15 01:45:56 +09:00 |
Lei Xu
|
839b690ed5
|
HADOOP-13075. Add support for SSE-KMS and SSE-C in s3a filesystem. (Steve Moist via lei)
|
2017-02-11 13:59:03 -08:00 |
|
Andrew Wang
|
5d8b80ea9b
|
Preparing for 3.0.0-alpha3 development
|
2017-01-19 15:50:07 -08:00 |
|
Steve Loughran
|
e648b6e138
|
HADOOP-13336 S3A to support per-bucket configuration. Contributed by Steve Loughran
|
2017-01-11 17:25:15 +00:00 |
|
Mingliang Liu
|
c6a3923245
|
HADOOP-13871. ITestS3AInputStreamPerformance.testTimeToOpenAndReadWholeFileBlocks performance awful. Contributed by Steve Loughran
|
2016-12-12 14:55:34 -08:00 |
|
Mingliang Liu
|
51211a7d7a
|
HADOOP-13855. Fix a couple of the s3a statistic names to be consistent with the rest. Contributed by Steve Loughran
|
2016-12-02 14:01:42 -08:00 |
|
Mingliang Liu
|
2ff84a0040
|
HADOOP-13857. S3AUtils.translateException to map (wrapped) InterruptedExceptions to InterruptedIOEs. Contributed by Steve Loughran
|
2016-12-02 13:36:04 -08:00 |
|
Mingliang Liu
|
3fd844b99f
|
HADOOP-13830. Intermittent failure of ITestS3NContractRootDir#testRecursiveRootListing: "Can not create a Path from an empty string". Contributed by Steve Loughran
|
2016-11-30 13:01:19 -08:00 |
|
Mingliang Liu
|
d60a60be8a
|
HADOOP-13823. s3a rename: fail if dest file exists. Contributed by Steve Loughran
|
2016-11-28 16:30:29 -08:00 |
|
Steve Loughran
|
0de0c32ddd
|
HADOOP-13801 regression: ITestS3AMiniYarnCluster failing. Contributed by Steve Loughran
|
2016-11-23 21:37:39 +00:00 |
|
Mingliang Liu
|
af791b774b
|
HADOOP-13050. Upgrade to AWS SDK 10.11+. Contributed by Chris Nauroth and Steve Loughran
|
2016-11-22 14:00:35 -08:00 |
|
Steve Loughran
|
a1761a841e
|
HADOOP-13680. fs.s3a.readahead.range to use getLongBytes. Contributed by Abhishek Modi.
|
2016-10-31 20:54:46 +00:00 |
|
Chris Nauroth
|
1f8490a5ba
|
HADOOP-13502. Split fs.contract.is-blobstore flag into more descriptive flags for use by contract tests. Contributed by Chris Nauroth.
|
2016-10-26 08:55:42 -07:00 |
|
Chris Nauroth
|
9cad3e2350
|
HADOOP-13614. Purge some superfluous/obsolete S3 FS tests that are slowing test runs down. Contributed by Steve Loughran.
|
2016-10-26 08:27:26 -07:00 |
|
Chris Nauroth
|
309a43925c
|
HADOOP-13309. Document S3A known limitations in file ownership and permission model. Contributed by Chris Nauroth.
|
2016-10-25 09:03:03 -07:00 |
|
Chris Nauroth
|
3372e94030
|
HADOOP-12774. s3a should use UGI.getCurrentUser.getShortname() for username. Contributed by Steve Loughran.
|
2016-10-24 21:54:06 -07:00 |
|
Chris Nauroth
|
d8fa1cfa67
|
HADOOP-13727. S3A: Reduce high number of connections to EC2 Instance Metadata Service caused by InstanceProfileCredentialsProvider. Contributed by Chris Nauroth.
|
2016-10-24 21:22:34 -07:00 |
Anu Engineer
|
3fbf4cd5da
|
HADOOP-13716. Add LambdaTestUtils class for tests; fix eventual consistency problem in contract test setup. Contributed by Steve Loughran.
|
2016-10-20 12:33:58 -07:00 |
|
Steve Loughran
|
9ae270af02
|
HADOOP-13735 ITestS3AFileContextStatistics.testStatistics() failing. Contributed by Pieter Reuse
|
2016-10-20 14:50:59 +01:00 |
|
Steve Loughran
|
6c348c5691
|
HADOOP-13560. S3ABlockOutputStream to support huge (many GB) file writes. Contributed by Steve Loughran
|
2016-10-18 21:16:02 +01:00 |
|
Chris Nauroth
|
69620f9559
|
HADOOP-13692. hadoop-aws should declare explicit dependency on Jackson 2 jars to prevent classpath conflicts. Contributed by Chris Nauroth.
|
2016-10-07 11:41:19 -07:00 |
|
Steve Loughran
|
ebd4f39a39
|
HADOOP-12977 s3a to handle delete("/", true) robustly. Contributed by Steve Loughran.
|
2016-10-07 12:51:40 +01:00 |
|
Steve Loughran
|
d6be1e75d8
|
HADOOP-12667 s3a to support createNonRecursive API. Contributed by Sean Mackrory
|
2016-10-05 15:02:11 +01:00 |
|
Chris Nauroth
|
88b9444a81
|
HADOOP-13674. S3A can provide a more detailed error message when accessing a bucket through an incorrect S3 endpoint. Contributed by Chris Nauroth.
|
2016-10-04 10:36:58 -07:00 |
|
Steve Loughran
|
ee0c722dc8
|
HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories. Contributed by Rajesh Balamohan.
|
2016-09-29 17:01:00 +01:00 |
|
Chris Nauroth
|
47f80922dc
|
HADOOP-13599. s3a close() to be non-synchronized, so avoid risk of deadlock on shutdown. Contributed by Steve Loughran.
|
2016-09-28 16:19:32 -07:00 |
|
Mingliang Liu
|
96142efa2d
|
HADOOP-13621. s3:// should have been fully cut off from trunk. Contributed by Mingliang Liu.
|
2016-09-17 22:07:46 -07:00 |
|
Steve Loughran
|
4b6d795f28
|
HADOOP-13540 improve section on troubleshooting s3a auth problems. Contributed by Steve Loughran
|
2016-09-09 18:55:32 +01:00 |
|
Steve Loughran
|
7fdfcd8a6c
|
HADOOP-13541 explicitly declare the Joda time version S3A depends on. Contributed by Stevel Loughran
|
2016-09-07 12:25:23 +01:00 |
|
Chris Nauroth
|
d152557cf7
|
HADOOP-13447. Refactor S3AFileSystem to support introduction of separate metadata repository and tests. Contributed by Chris Nauroth.
|
2016-09-06 09:36:21 -07:00 |
Chris Douglas
|
c37346d0e3
|
HADOOP-12726. Unsupported FS operations should throw UnsupportedOperationException. Contributed by Daniel Templeton.
|
2016-08-23 14:12:52 -07:00 |
|
Chris Nauroth
|
6f9c346e57
|
HADOOP-13446. Support running isolated unit tests separate from AWS integration tests. Contributed by Chris Nauroth.
|
2016-08-23 07:18:49 -07:00 |
|
Chris Nauroth
|
763f0497bb
|
HADOOP-13252. Tune S3A provider plugin mechanism. Contributed by Steve Loughran.
|
2016-08-19 10:48:10 -07:00 |
|
Steve Loughran
|
040c185d62
|
HADOOP-13405 doc for fs.s3a.acl.default indicates incorrect values. Contributed by Shen Yinjie
|
2016-08-18 14:36:55 +01:00 |
|
Chris Nauroth
|
822d661b8f
|
HADOOP-13208. S3A listFiles(recursive=true) to do a bulk listObjects instead of walking the pseudo-tree of directories. Contributed by Steve Loughran.
|
2016-08-17 14:54:54 -07:00 |
|
Chris Nauroth
|
3808876c73
|
HADOOP-13324. s3a tests don't authenticate with S3 frankfurt (or other V4 auth only endpoints). Contributed by Steve Loughran.
|
2016-08-16 17:05:52 -07:00 |