Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
20,944 Commits 326 Branches 371 Tags
Commit Graph

1113 Commits

Author SHA1 Message Date
Steve Loughran
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f848345f6d34a2ac2d570d49e9787a0df6a)
 2019-02-04 17:10:19 +00:00
Akira Ajisaka
dc12754ab6
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document. 
(cherry picked from commit 3c60303ac59d3b6cc375e7ac10214fc36d330fa4)
 2019-01-25 10:28:46 +09:00
Da Zhou
29de303e0a
HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.

(cherry picked from commit 00ad9e23e88d1e1b1f887b6c9473a7a924a95a97)
 2019-01-22 16:41:33 +00:00
Da Zhou
1d4390e16b
HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.

(cherry picked from commit 30863c5ae3a7ad69b6b4853bad2e8f22c7c67639)
 2019-01-14 19:45:30 +00:00
Da Zhou
8b5fbe7a12
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.
 2019-01-11 11:12:39 +00:00
Da Zhou
9cb6000c8a
HADOOP-16036. WASB: Disable jetty logging configuration announcement. 
Contributed by Da Zhou.

(cherry picked from commit 852701f7935eb94eda6f44b74a2ce060e7c159a7)
 2019-01-10 12:08:27 +00:00
Da Zhou
6c2500d7ca
HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.

(cherry picked from commit 72112691421bedb66d887ba4e18d1b7d65a42f2c)
 2019-01-10 12:03:48 +00:00
Da Zhou
f7de630e85
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.

(cherry picked from commit e8d19003695e3bc76bfa8e1187a238eec0220def)
 2019-01-10 11:59:29 +00:00
Kai Xie
5dce9d75e6
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7eaec71433548dc47f5a9a995cbea15a)
 2019-01-08 13:34:51 +00:00
Weiwei Yang
977e0ff8b9 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c46704dcdb63089dd971e2a5ba1deaac)
 2019-01-07 16:06:03 +08:00
Sunil G
71bee05339
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae46a3c04fc56897986a4ab4e5aa98503.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9.

(cherry picked from commit 1f425271a73fff1fdbe3fbfdb71e906fd1ac0574)
 2019-01-05 17:36:15 +09:00
Weiwei Yang
38ef85171d HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 
(cherry picked from commit 040a202b202a37f3b922cd321eb0a8ded457d88b)
 2019-01-03 21:40:43 +08:00
Da Zhou
f122ae7279
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.

(cherry picked from commit 346c0c8aff0b206d45f34dbce4fcc81364115d95)
 2018-12-17 11:18:12 +00:00
Da Zhou
d09dbcc8fb
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.
 2018-12-17 11:08:17 +00:00
Da Zhou
87d9a54968
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.

(cherry picked from commit b2523d8100844338e073531c47666d744a101caf)
 2018-12-17 11:05:39 +00:00
Da Zhou
2d2212a508
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.

(cherry picked from commit a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b)
 2018-12-04 13:40:03 +00:00
Da Zhou
9bc1fd4721
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.

(cherry picked from commit 7ccb640a66bb5bb9f657a2db85bcc8ae0ded4892)
 2018-11-30 10:13:57 +00:00
Steve Loughran
fa1d4ba7d4
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
 2018-11-27 20:40:48 +00:00
Da Zhou
1a3a4960d9
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>
 2018-11-27 18:11:30 +00:00
Da Zhou
f5d2806c81
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.

(cherry picked from commit a8302e398c2ca4b0deff062c0e921053351b688e)
 2018-11-23 14:19:36 +00:00
Weiwei Yang
fea9d37ad5 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 
(cherry picked from commit 5ff0cf86a940fd83f1425794921cc075b19f1108)
 2018-11-23 14:10:30 +08:00
Weiwei Yang
0b2cfc8ab8 HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 
(cherry picked from commit be0708c6eb3c759e43639e0f8df7c9910308fb97)
 2018-11-19 14:21:38 +08:00
Arpit Agarwal
351bfa1bcf HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f14ddff581b63f0902b7760e4d2bd56)
 2018-11-15 13:58:29 -08:00
Akira Ajisaka
8c9681d7f0
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 66b1335bb3a9a6f3a3db455540c973d4a85bef73)
 2018-11-15 16:31:05 +09:00
Sammi Chen
37082a664a HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce84dcf68bcd7de5a5ed1c7d904796e9)
(cherry picked from commit 64cb97fb4467513f73fde18f96f391ad34e3bb0a)
(cherry picked from commit 5d532cfc6f23f942ed10edab55ed251eb99a0664)
 2018-11-14 13:48:51 +08:00
Da Zhou
4039840510
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.

(cherry picked from commit a13be203b7877ba56ef63aac4a2e65d4e1a4adbc)
 2018-11-13 21:48:05 +00:00
Da Zhou
7440bc5a9c
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit 9dbb2b67c6fa69a663c7c23fbb876bac4b6c09ce)
 2018-11-09 11:07:51 +00:00
Junhua Gu
66715005f9
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.
 2018-11-08 17:20:52 +00:00
Sammi Chen
ca22bf175f HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538e838f40581519080d7c8621c65e53b)
(cherry picked from commit 366541d834f70fd6f8d4c5296a9e844236c6fd74)
(cherry picked from commit c5a227062fae33ce3137161802d6f090a6af010e)
 2018-10-26 15:33:05 +08:00
Ted Yu
a7dd244a49 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
 2018-10-19 13:22:01 -07:00
Giovanni Matteo Fumarola
0b6a7b5f16 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:26:22 -07:00
Sunil G
bde4fd5ed9 Preparing for 3.2.0 release 2018-10-18 17:07:45 +05:30
Steve Loughran
c19864d8c8
HADOOP-15848. ITestS3AContractMultipartUploader#testMultipartUploadEmptyPart test error. 
Contributed by Ewan Higgs.
 2018-10-16 19:58:58 +01:00
bibinchundatt
aff21d7271 YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt. 
(cherry picked from commit b4a38e7b3e530756fb79d23dd4e218beeb5e3190)
 2018-10-15 18:11:01 +05:30
Steve Loughran
70d39c74d9
HADOOP-15837. DynamoDB table Update can fail S3A FS init. 
Contributed by Steve Loughran.

(cherry picked from commit ee816f1fd78b029b9efa567e8b1b62391563de14)
 2018-10-11 14:58:37 +01:00
Steve Loughran
83b9b25c51
HADOOP-15809. ABFS: better exception handling when making getAccessToken call. 
Contributed by Da Zhou

(cherry picked from commit 273cc2d4e972dc7cc371761a4ab2d1bc20dd5ca0)
 2018-10-05 11:29:43 +01:00
Steve Loughran
c6942a315b
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld. 
Contributed by Abhishek Modi.

(cherry picked from commit e8b8604314a2ea894b1f24939c42e782b83335aa)
 2018-10-03 12:59:16 +01:00
Steve Loughran
e5e9d7b595
HADOOP-15795. Make HTTPS the default protocol for ABFS. 
Contributed by Da Zhou.

(cherry picked from commit 7051bd78b17b2666c2fa0f61823920285a060a76)
 2018-10-03 12:53:56 +01:00
Steve Loughran
a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore. 
Contributed by Da Zhou
 2018-10-02 11:42:52 +01:00
Steve Loughran
43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization 
Contributed by Yuan Gao.
 2018-10-02 11:37:28 +01:00
Steve Loughran
a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies. 
Contributed by Da Zhou.
 2018-09-25 20:58:32 +01:00
Steve Loughran
d5da9928c9 HADOOP-15723. ABFS: Ranger Support. 
Contributed by Yuan Gao.
 2018-09-25 19:13:10 +01:00
Sammi Chen
2b635125fb HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 2018-09-25 19:48:30 +08:00
Mingliang Liu
c07715e378 HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran 2018-09-24 12:53:21 -07:00
Sunil G
d060cbea48 HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran. 2018-09-24 21:19:47 +05:30
Sean Mackrory
0def61482b Merge branch 'HADOOP-15407' into trunk 2018-09-22 21:19:12 -06:00
Steve Loughran
d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read. 
Contributed by Sneha Varma.
 2018-09-21 11:06:24 +01:00
Steve Loughran
a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable. 
Contributed by Steve Loughran.
 2018-09-20 17:36:18 +01:00
Sean Mackrory
8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran
a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped. 
Contributed by Steve Loughran
 2018-09-19 13:57:39 +01:00