Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
21,193 Commits 326 Branches 371 Tags
Commit Graph

1132 Commits

Author SHA1 Message Date
Akira Ajisaka
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 
(cherry picked from commit 8b6deebb1dda49e5e35180ed5c5fb5b5221c1516)
 2019-04-03 19:36:17 +09:00
Steve Loughran
60c9042286
HADOOP-16058. S3A tests to include Terasort. 
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
 2019-03-29 15:25:45 +00:00
Siyao Meng
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail. 
Contributed by Siyao Meng

(cherry picked from commit ce4bafdf442c004b6deb25eaa2fa7e947b8ad269)
 2019-03-26 18:43:43 +00:00
Andrew Olson
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized. 
Author:    Andrew Olson
(cherry picked from commit faba3591d32f2e4808c2faeb9472348d52619c8a)
 2019-03-22 10:36:34 +00:00
Weiwei Yang
39f60faa60 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 
(cherry picked from commit 568d3ab8b65d1348dec9c971feffe200e6cba2ef)
 2019-03-19 10:08:11 +08:00
Adam Antal
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants. 
Contributed by Adam Antal.

(cherry picked from commit c0427c84dddf942529dfdfc5cc7a3e25e3f12c5e)
 2019-03-18 19:14:43 +00:00
Ben Roling
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files. 
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
 2019-03-14 19:46:34 +00:00
Steve Loughran
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF 
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
 2019-03-11 11:15:25 +00:00
Da Zhou
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties. 
Author:    Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ade5568ae37dcc3753aea0d1e0fd701b)
 2019-03-08 13:53:44 +00:00
Da Zhou
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name 
Contributed by Da Zhou.

(cherry picked from commit 3988e75ca385aec31ca1fc49d6cffce1ea935825)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:47:58 +00:00
Da Zhou
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth. 
Contributed by Da Zhou and Junhua Gu.

(cherry picked from commit 1f1655028eede24197705a594b6ef19e6737db35)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:44:46 +00:00
Da Zhou
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. 
Contributed by Shweta Yakkali.

Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e7e8477c62e0f8bf1bac6de3e00a8a4)
Signed-off-by: Steve Loughran <stevel@apache.org>
 2019-03-05 10:39:37 +00:00
Steve Loughran
685a41f449
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS. 
Contributed by Steve Loughran.

(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
 2019-02-21 13:39:37 +00:00
Masatake Iwasaki
dc9c3ce30b HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki. 
(cherry picked from commit aa3ad3660506382884324c4b8997973f5a68e29a)
 2019-02-20 22:17:18 +09:00
bibinchundatt
bdfdf12178 YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 
(cherry picked from commit 779dae4de7e518938d58badcef065ea457be911c)
 2019-02-20 00:37:47 +05:30
bibinchundatt
f06ac51c37 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 
(cherry picked from commit 134ae8fc8045e2ae1ed7ca54df95f14ffc863d09)
 2019-02-14 22:58:37 +05:30
Ranith Sardar
c5eca3f7ee
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 
Contributed by Ranith Sardar.

(cherry picked from commit 546c5d70efebb828389f609a89b123c4ee51f867)
 2019-02-07 21:49:18 +00:00
Andrew Olson
36f3e775d4
HADOOP-15281. Distcp to add no-rename copy option. 
Contributed by Andrew Olson.

(cherry picked from commit de804e53b9d20a2df75a4c7252bf83ed52011488)
 2019-02-07 10:09:13 +00:00
Da Zhou
84ce0f1bfa
HADOOP-16074. WASB: Update container not found error code. 
Contributed by Da Zhou.

(cherry picked from commit ba9efe06fadb9631763f3e623d8748bbe59ff748)
 2019-02-05 14:41:15 +00:00
Steve Loughran
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests. 
Contributed by Ewan Higgs and Steve Loughran.

(cherry picked from commit c1d24f848345f6d34a2ac2d570d49e9787a0df6a)
 2019-02-04 17:10:19 +00:00
Akira Ajisaka
dc12754ab6
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document. 
(cherry picked from commit 3c60303ac59d3b6cc375e7ac10214fc36d330fa4)
 2019-01-25 10:28:46 +09:00
Da Zhou
29de303e0a
HADOOP-16048. ABFS: Fix Date format parser. 
Contributed by Da Zhou.

(cherry picked from commit 00ad9e23e88d1e1b1f887b6c9473a7a924a95a97)
 2019-01-22 16:41:33 +00:00
Da Zhou
1d4390e16b
HADOOP-16044. ABFS: Better exception handling of DNS errors followup 
Contributed by Da Zhou.

(cherry picked from commit 30863c5ae3a7ad69b6b4853bad2e8f22c7c67639)
 2019-01-14 19:45:30 +00:00
Da Zhou
8b5fbe7a12
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME. 
Contributed by Da Zhou.
 2019-01-11 11:12:39 +00:00
Da Zhou
9cb6000c8a
HADOOP-16036. WASB: Disable jetty logging configuration announcement. 
Contributed by Da Zhou.

(cherry picked from commit 852701f7935eb94eda6f44b74a2ce060e7c159a7)
 2019-01-10 12:08:27 +00:00
Da Zhou
6c2500d7ca
HADOOP-15662. Better exception handling of DNS errors. 
Contributed by Da Zhou.

(cherry picked from commit 72112691421bedb66d887ba4e18d1b7d65a42f2c)
 2019-01-10 12:03:48 +00:00
Da Zhou
f7de630e85
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration. 
Contributed by Da Zhou.

(cherry picked from commit e8d19003695e3bc76bfa8e1187a238eec0220def)
 2019-01-10 11:59:29 +00:00
Kai Xie
5dce9d75e6
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0. 
Contributed by Kai Xie.

(cherry picked from commit 188bebbe7eaec71433548dc47f5a9a995cbea15a)
 2019-01-08 13:34:51 +00:00
Weiwei Yang
977e0ff8b9 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 
(cherry picked from commit f87b3b11c46704dcdb63089dd971e2a5ba1deaac)
 2019-01-07 16:06:03 +08:00
Sunil G
71bee05339
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu." 
This reverts commit e4fca6aae46a3c04fc56897986a4ab4e5aa98503.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb059fc204ed35bc0e264c42dd3a9fe9.

(cherry picked from commit 1f425271a73fff1fdbe3fbfdb71e906fd1ac0574)
 2019-01-05 17:36:15 +09:00
Weiwei Yang
38ef85171d HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 
(cherry picked from commit 040a202b202a37f3b922cd321eb0a8ded457d88b)
 2019-01-03 21:40:43 +08:00
Da Zhou
f122ae7279
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException. 
Contributed by Da Zhou.

(cherry picked from commit 346c0c8aff0b206d45f34dbce4fcc81364115d95)
 2018-12-17 11:18:12 +00:00
Da Zhou
d09dbcc8fb
HADOOP-15972 ABFS: reduce list page size to to 500. 
Contributed by Da Zhou.
 2018-12-17 11:08:17 +00:00
Da Zhou
87d9a54968
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs. 
Contributed by Da Zhou.

(cherry picked from commit b2523d8100844338e073531c47666d744a101caf)
 2018-12-17 11:05:39 +00:00
Da Zhou
2d2212a508
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS. 
Contributed by Da Zhou.

(cherry picked from commit a8bbd818d5bc4762324bcdb7cf1fdd5c2f93891b)
 2018-12-04 13:40:03 +00:00
Da Zhou
9bc1fd4721
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet. 
Contributed by Da Zhou.

(cherry picked from commit 7ccb640a66bb5bb9f657a2db85bcc8ae0ded4892)
 2018-11-30 10:13:57 +00:00
Steve Loughran
fa1d4ba7d4
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem. 
Contributed by Steve Loughran.

(cherry picked from commit 4c106fca0ca91536e288f11052568406a0b84300)
 2018-11-27 20:40:48 +00:00
Da Zhou
1a3a4960d9
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename. 
Contributed by Da Zhou <da.zhou@microsoft.com>
 2018-11-27 18:11:30 +00:00
Da Zhou
f5d2806c81
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2. 
Contributed by Junhua Gu and Da Zhou.

(cherry picked from commit a8302e398c2ca4b0deff062c0e921053351b688e)
 2018-11-23 14:19:36 +00:00
Weiwei Yang
fea9d37ad5 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 
(cherry picked from commit 5ff0cf86a940fd83f1425794921cc075b19f1108)
 2018-11-23 14:10:30 +08:00
Weiwei Yang
0b2cfc8ab8 HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 
(cherry picked from commit be0708c6eb3c759e43639e0f8df7c9910308fb97)
 2018-11-19 14:21:38 +08:00
Arpit Agarwal
351bfa1bcf HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 914b0cf15f14ddff581b63f0902b7760e4d2bd56)
 2018-11-15 13:58:29 -08:00
Akira Ajisaka
8c9681d7f0
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 
(cherry picked from commit 66b1335bb3a9a6f3a3db455540c973d4a85bef73)
 2018-11-15 16:31:05 +09:00
Sammi Chen
37082a664a HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 
(cherry picked from commit 3fade865ce84dcf68bcd7de5a5ed1c7d904796e9)
(cherry picked from commit 64cb97fb4467513f73fde18f96f391ad34e3bb0a)
(cherry picked from commit 5d532cfc6f23f942ed10edab55ed251eb99a0664)
 2018-11-14 13:48:51 +08:00
Da Zhou
4039840510
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore 
Contributed by Da Zhou.

(cherry picked from commit a13be203b7877ba56ef63aac4a2e65d4e1a4adbc)
 2018-11-13 21:48:05 +00:00
Da Zhou
7440bc5a9c
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console. 
Author:    Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit 9dbb2b67c6fa69a663c7c23fbb876bac4b6c09ce)
 2018-11-09 11:07:51 +00:00
Junhua Gu
66715005f9
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries. 
Contributed by Junhua Gu.
 2018-11-08 17:20:52 +00:00
Sammi Chen
ca22bf175f HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 
(cherry picked from commit 7574d18538e838f40581519080d7c8621c65e53b)
(cherry picked from commit 366541d834f70fd6f8d4c5296a9e844236c6fd74)
(cherry picked from commit c5a227062fae33ce3137161802d6f090a6af010e)
 2018-10-26 15:33:05 +08:00
Ted Yu
a7dd244a49 HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu. 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
(cherry picked from commit e2cecb681e2aab8b7c5465719cac53dce407a64c)
 2018-10-19 13:22:01 -07:00
Giovanni Matteo Fumarola
0b6a7b5f16 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:26:22 -07:00