Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,903 Commits 326 Branches 371 Tags
Commit Graph

180 Commits

Author SHA1 Message Date
Steve Loughran
7f70a78ce3
HADOOP-16109. Parquet reading S3AFileSystem causes EOF (#589) 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran

Change-Id: I39b87f3d5daa98f65de2c0a44e348821a4930573
 2019-03-12 11:36:32 +00:00
Steve Loughran
7dac294110
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
 2018-11-27 20:57:16 +00:00
Steve Loughran
2abc4ab60a
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.

(cherry picked from commit f1fbc563719db88c50d57820b5052880cfd6b1aa)
 2018-10-12 09:49:02 +01:00
Yongjun Zhang
f0de11ba98 Preparing for 3.0.4 development 2018-05-29 23:40:26 -07:00
Lei Xu
3717df89ee Preparing for 3.0.3 development 2018-04-12 13:57:46 -07:00
Chris Douglas
a03c8ea61f HADOOP-14696. parallel tests don't work for Windows. Contributed by Allen Wittenauer 
(cherry picked from commit 45d1b0fdcc04a86be91a9b72073cdc30bec04d3b)
(cherry picked from commit 09940b1eb3b7ed764149f4a993c1857e9c6ad938)
 2018-03-12 20:06:22 -07:00
Steve Loughran
cc0f14c13c HADOOP-15267. S3A multipart upload fails when SSE-C encryption is enabled. 
Contributed by Anis Elleuch.

(cherry picked from commit 1dedc68f9d8d8544d715e67ee77cd3f017c21699)
 2018-03-07 19:20:53 +00:00
Lei Xu
7b377f43d0 Preparing for 3.0.2 development 2018-02-06 10:50:37 -08:00
Steve Loughran
ec4f5f0f0d HADOOP-15039/HADOOP-15189. Move SemaphoredDelegatingExecutor to hadoop-common 
Contributed by Genmao Yu
 2018-01-26 15:23:08 -08:00
Steve Loughran
11f577f3e6 HADOOP-14965. S3a input stream "normal" fadvise mode to be adaptive 
(cherry picked from commit 1ba491ff907fc5d2618add980734a3534e2be098)
 2017-12-22 11:41:45 +00:00
Andrew Wang
1c8d334287 Preparing for 3.0.1 development 2017-11-14 11:17:18 -08:00
Andrew Wang
5a4f37019a Preparing for 3.0.0 GA development 2017-09-28 11:32:01 -07:00
Aaron Fabbri
60b966cb45
HADOOP-14220 Enhance S3GuardTool with bucket-info and set-capacity commands, tests. Contributed by Steve Loughran 2017-09-25 16:01:15 -07:00
Aaron Fabbri
62e8a5ceed
HADOOP-14738 Remove S3N and obsolete bits of S3A; rework docs. Contributed by Steve Loughran. 2017-09-14 16:12:29 -07:00
Steve Loughran
70c95d15e3
HADOOP-13421. Switch to v2 of the S3 List Objects API in S3A. 
Contributed by Aaron Fabbri

(cherry picked from commit 5bbca80428ffbe776650652de86a3bba885edb31)
 2017-09-08 12:08:36 +01:00
Steve Loughran
5757600b8d
HADOOP-14774. S3A case "testRandomReadOverBuffer" failed due to improper range parameter. 
Contributed by Yonger.

(cherry picked from commit 83449abb40633a36b22de538cf82fca240af067e)
 2017-09-07 19:58:03 +01:00
John Zhuge
3dd5444b4d HADOOP-14103. Sort out hadoop-aws contract-test-options.xml. Contributed by John Zhuge. 
(cherry picked from commit 50506e90a843aaf1f771a32b872fdd7704b619f5)
 2017-09-06 11:33:43 -07:00
Steve Loughran
621b43e254
HADOOP-13345 HS3Guard: Improved Consistency for S3A. 
Contributed by: Chris Nauroth, Aaron Fabbri, Mingliang Liu, Lei (Eddy) Xu,
Sean Mackrory, Steve Loughran and others.
 2017-09-01 14:13:41 +01:00
Steve Loughran
7fc324aabd
HADOOP-14126. Remove jackson, joda and other transient aws SDK dependencies from hadoop-aws. 
Contributed by Steve Loughran

(cherry picked from commit ced547d5f0dbea571cbc472c5f55fe89d5900a6f)
 2017-08-04 11:09:08 +01:00
Andrew Wang
af2773f609 Updating version for 3.0.0-beta1 development 2017-06-29 17:57:40 -07:00
Mingliang Liu
72993b33b7 HADOOP-14596. AWS SDK 1.11+ aborts() on close() if > 0 bytes in stream; logs error. Contributed by Steve Loughran 
Change-Id: I49173bf6163796903d64594a8ca8a4bd26ad2bfc
 2017-06-29 17:07:52 -07:00
Steve Loughran
ee243e5289
HADOOP-14190. Add more on S3 regions to the s3a documentation. 
Contributed by Steve Loughran
 2017-06-28 10:22:13 +01:00
Mingliang Liu
bc4dfe9c9c HADOOP-14594. ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions. Contributed by Mingliang Liu 2017-06-27 13:19:14 -07:00
Mingliang Liu
f214a9961f HADOOP-14494. ITestJets3tNativeS3FileSystemContract tests NPEs in teardown if store undefined. Contributed by Steve Loughran 2017-06-15 11:16:16 -07:00
Mingliang Liu
6aeda55bb8 HADOOP-14428. s3a: mkdir appears to be broken. Contributed by Mingliang Liu 2017-06-05 11:26:56 -07:00
John Zhuge
6c6a7a5962 HADOOP-14464. hadoop-aws doc header warning #5 line wrapped. Contributed by John Zhuge. 2017-05-28 22:25:00 -07:00
Andrew Wang
16ad896d5c Update maven version for 3.0.0-alpha4 development 2017-05-26 14:09:44 -07:00
Akira Ajisaka
6a52b5e144
HADOOP-14180. FileSystem contract tests to replace JUnit 3 with 4. Contributed by Xiaobing Zhou and Mingliang Liu. 2017-05-25 15:36:44 +09:00
Mingliang Liu
6672810eea HADOOP-14432. S3A copyFromLocalFile to be robust, tested. Contributed by Steve Loughran 2017-05-19 11:51:43 -07:00
Steve Loughran
ba70225cf6
HADOOP-11572. s3a delete() operation fails during a concurrent delete of child entries. 
Contributed by Steve Loughran.

(cherry picked from commit 2ac5aab8d725f761a9f9723471a4426f6b5d78c4)
 2017-05-18 15:44:39 +01:00
Akira Ajisaka
0d5c8ed8e0
HADOOP-14401. maven-project-info-reports-plugin can be removed. Contributed by Andras Bokor. 2017-05-11 16:37:32 -05:00
Steve Loughran
5f934f8386
HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus. 
Contributed by Steve Moist.
 2017-04-24 20:33:19 +01:00
Mingliang Liu
667966c13c HADOOP-14324. Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics. Contributed by Steve Loughran 2017-04-20 17:13:36 -07:00
Steve Loughran
6b015d00c9
HADOOP-14321. explicitly exclude s3a root dir ITests from parallel runs. 
Contributed by Steve Loughran
 2017-04-19 10:21:44 +01:00
Mingliang Liu
b053fdc547 HADOOP-14255. S3A to delete unnecessary fake directory objects in mkdirs(). Contributed by Mingliang Liu 2017-04-12 14:30:34 -07:00
Chris Nauroth
b8305e6d06 HADOOP-14248. Retire SharedInstanceProfileCredentialsProvider in trunk. Contributed by Mingliang Liu. 2017-04-12 10:02:13 -07:00
Mingliang Liu
5faa949b78 HADOOP-14268. Fix markdown itemization in hadoop-aws documents. Contributed by Akira Ajisaka 2017-04-03 11:07:14 -07:00
Akira Ajisaka
0d053eeb30
HADOOP-14256. [S3A DOC] Correct the format for "Seoul" example. Contributed by Brahma Reddy Battula. 2017-03-30 18:11:50 +09:00
Andrew Wang
4966a6e26e HADOOP-14223. Extend FileStatus#toString() to include details like Erasure Coding and Encryption. Contributed by Manoj Govindassamy. 2017-03-29 14:37:21 -07:00
Andrew Wang
52b00600df HADOOP-13715. Add isErasureCoded() API to FileStatus class. Contributed by Manoj Govindassamy. 2017-03-24 11:44:46 -07:00
Mingliang Liu
2e30aa72e0 HADOOP-14135. Remove URI parameter in AWSCredentialProvider constructors. Contributed by Mingliang Liu 2017-03-23 11:33:29 -07:00
Steve Loughran
a5a4867f3b
HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject(). 
Contributed by Yuanbo Liu

(cherry picked from commit 20878d052cebc715c2494a97362fdff08885a77f)
 2017-03-23 12:54:54 +00:00
Steve Loughran
2841666f1f
HADOOP-14204 S3A multipart commit failing, 
"UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort".

Contributed by Steve Loughran

(cherry picked from commit 595501c9ddacd5fdb740ac1bf9bfeb0ffbbd6c0e)
 2017-03-21 13:15:15 +00:00
Arpit Agarwal
6c399a88e9 HADOOP-14059. typo in s3a rename(self, subdir) error message. Contributed by Steve Loughran. 2017-03-20 11:29:23 -07:00
Mingliang Liu
fa67a96d7b HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-14 14:38:21 -07:00
Mingliang Liu
e6cda5819b Revert "HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu" 
This reverts commit b8c69557b7a23ff9c4c0b2c9d595338a08b873f1.
 2017-03-14 13:03:42 -07:00
Mingliang Liu
b8c69557b7 HADOOP-14170. FileSystemContractBaseTest is not cleaning up test directory clearly. Contributed by Mingliang Liu 2017-03-13 14:15:02 -07:00
Steve Loughran
092ec39fb9
HADOOP-14111 cut some obsolete, ignored s3 tests in TestS3Credentials. 
Contributed by Yuanbo Liu
 2017-03-10 17:43:22 +00:00
Akira Ajisaka
f597f4c43e
HADOOP-14087. S3A typo in pom.xml test exclusions. Contributed by Aaron Fabbri. 2017-03-07 15:14:55 +09:00
Steve Loughran
a97833e0ed
HADOOP-14138. Remove S3A ref from META-INF service discovery, rely on existing core-default entry. 
Contributed by Steve Loughran

(cherry picked from commit 51fefbe38bf40373d2855163fa6444821b0f994c)
 2017-03-02 21:44:30 +00:00