26eb9f52fb
HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu.
...
(cherry picked from commit 2d8282bb82
2019-05-14 14:06:42 -07:00
12e0053932
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
(cherry picked from commit 591ca69823
2019-05-09 21:14:37 +01:00
df5d8f05d9
HADOOP-16227. Upgrade checkstyle to 8.19
...
(cherry picked from commit 4b4fef2f0e
2019-04-15 10:47:02 +09:00
03079be707
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
...
(cherry picked from commit bbdbc7a9a1
2019-04-12 11:59:14 +09:00
b6ebe74526
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
(cherry picked from commit 366186d999
2019-04-03 21:35:19 +01:00
80a8d3310e
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
...
(cherry picked from commit 8b6deebb1d
2019-04-03 19:36:17 +09:00
60c9042286
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-29 15:25:45 +00:00
52cfbc39cc
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
...
Contributed by Siyao Meng
(cherry picked from commit ce4bafdf44
2019-03-26 18:43:43 +00:00
ade3af6ef2
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
...
Author: Andrew Olson
(cherry picked from commit faba3591d3
2019-03-22 10:36:34 +00:00
39f60faa60
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
...
(cherry picked from commit 568d3ab8b6
2019-03-19 10:08:11 +08:00
81a6ba1825
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
(cherry picked from commit c0427c84dd
2019-03-18 19:14:43 +00:00
43e8ac6097
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-14 19:46:34 +00:00
b6f6c34223
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-11 11:15:25 +00:00
cfaf21a4ba
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit e0260417ad
2019-03-08 13:53:44 +00:00
dc38fc598d
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
(cherry picked from commit 3988e75ca3
2019-03-05 10:47:58 +00:00
075f6b061c
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
(cherry picked from commit 1f1655028e
2019-03-05 10:44:46 +00:00
ae832ccffe
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS.
...
Contributed by Shweta Yakkali.
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
(cherry picked from commit 02eb91856e
2019-03-05 10:39:37 +00:00
685a41f449
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
(cherry picked from commit 9cb2f470b759bbe7609a00e8f8f72779e2daae80)
2019-02-21 13:39:37 +00:00
dc9c3ce30b
HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki.
...
(cherry picked from commit aa3ad36605
2019-02-20 22:17:18 +09:00
bdfdf12178
YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T.
...
(cherry picked from commit 779dae4de7
2019-02-20 00:37:47 +05:30
f06ac51c37
YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt.
...
(cherry picked from commit 134ae8fc80
2019-02-14 22:58:37 +05:30
c5eca3f7ee
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on.
...
Contributed by Ranith Sardar.
(cherry picked from commit 546c5d70ef
2019-02-07 21:49:18 +00:00
36f3e775d4
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
(cherry picked from commit de804e53b9
2019-02-07 10:09:13 +00:00
84ce0f1bfa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
(cherry picked from commit ba9efe06fa
2019-02-05 14:41:15 +00:00
bdd17be9ec
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
(cherry picked from commit c1d24f8483
2019-02-04 17:10:19 +00:00
dc12754ab6
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
...
(cherry picked from commit 3c60303ac5
2019-01-25 10:28:46 +09:00
29de303e0a
HADOOP-16048. ABFS: Fix Date format parser.
...
Contributed by Da Zhou.
(cherry picked from commit 00ad9e23e8
2019-01-22 16:41:33 +00:00
1d4390e16b
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
...
Contributed by Da Zhou.
(cherry picked from commit 30863c5ae3
2019-01-14 19:45:30 +00:00
8b5fbe7a12
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
...
Contributed by Da Zhou.
2019-01-11 11:12:39 +00:00
9cb6000c8a
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
...
Contributed by Da Zhou.
(cherry picked from commit 852701f793
2019-01-10 12:08:27 +00:00
6c2500d7ca
HADOOP-15662. Better exception handling of DNS errors.
...
Contributed by Da Zhou.
(cherry picked from commit 7211269142
2019-01-10 12:03:48 +00:00
f7de630e85
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
...
Contributed by Da Zhou.
(cherry picked from commit e8d1900369
2019-01-10 11:59:29 +00:00
5dce9d75e6
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
...
Contributed by Kai Xie.
(cherry picked from commit 188bebbe7e
2019-01-08 13:34:51 +00:00
977e0ff8b9
HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu.
...
(cherry picked from commit f87b3b11c4
2019-01-07 16:06:03 +08:00
71bee05339
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu."
...
This reverts commit e4fca6aae42b635125fb1f425271a7
2019-01-05 17:36:15 +09:00
38ef85171d
HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu.
...
(cherry picked from commit 040a202b20
2019-01-03 21:40:43 +08:00
f122ae7279
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
...
Contributed by Da Zhou.
(cherry picked from commit 346c0c8aff
2018-12-17 11:18:12 +00:00
d09dbcc8fb
HADOOP-15972 ABFS: reduce list page size to to 500.
...
Contributed by Da Zhou.
2018-12-17 11:08:17 +00:00
87d9a54968
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
...
Contributed by Da Zhou.
(cherry picked from commit b2523d8100
2018-12-17 11:05:39 +00:00
2d2212a508
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
...
Contributed by Da Zhou.
(cherry picked from commit a8bbd818d5
2018-12-04 13:40:03 +00:00
9bc1fd4721
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
...
Contributed by Da Zhou.
(cherry picked from commit 7ccb640a66
2018-11-30 10:13:57 +00:00
fa1d4ba7d4
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
(cherry picked from commit 4c106fca0c
2018-11-27 20:40:48 +00:00
1a3a4960d9
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename.
...
Contributed by Da Zhou <da.zhou@microsoft.com>
2018-11-27 18:11:30 +00:00
f5d2806c81
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2.
...
Contributed by Junhua Gu and Da Zhou.
(cherry picked from commit a8302e398c
2018-11-23 14:19:36 +00:00
fea9d37ad5
HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu.
...
(cherry picked from commit 5ff0cf86a9
2018-11-23 14:10:30 +08:00
0b2cfc8ab8
HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu.
...
(cherry picked from commit be0708c6eb
2018-11-19 14:21:38 +08:00
351bfa1bcf
HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia.
...
(cherry picked from commit 914b0cf15f
2018-11-15 13:58:29 -08:00
8c9681d7f0
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia.
...
(cherry picked from commit 66b1335bb3
2018-11-15 16:31:05 +09:00
37082a664a
HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu.
...
(cherry picked from commit 3fade865ce64cb97fb445d532cfc6f
2018-11-14 13:48:51 +08:00
4039840510
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
...
Contributed by Da Zhou.
(cherry picked from commit a13be203b7
2018-11-13 21:48:05 +00:00
Weiwei Yang
Rajat Khandelwal
Akira Ajisaka
Masatake Iwasaki
Steve Loughran
Siyao Meng
Andrew Olson
Adam Antal
Ben Roling
Da Zhou
bibinchundatt
Ranith Sardar
Kai Xie
Sunil G
Arpit Agarwal
Sammi Chen