de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
2019-02-05 14:39:45 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki.
2019-01-29 18:29:56 -08:00
02eb91856e
HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali.
...
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
2019-01-29 16:50:14 -08:00
3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document.
2019-01-25 10:27:59 +09:00
00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser.
...
Contributed by Da Zhou.
2019-01-22 16:40:42 +00:00
30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
...
Contributed by Da Zhou.
2019-01-14 19:32:56 +00:00
6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
2019-01-14 17:27:00 +00:00
04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set.
...
Contributed by Gabor Bota.
2019-01-14 13:12:05 +00:00
fb8932a727
HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena.
2019-01-11 10:54:49 -08:00
dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
...
Contributed by Da Zhou.
(cherry picked from commit 8b5fbe7a12
2019-01-11 11:13:41 +00:00
852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
...
Contributed by Da Zhou.
2019-01-10 12:07:09 +00:00
7211269142
HADOOP-15662. Better exception handling of DNS errors.
...
Contributed by Da Zhou.
2019-01-10 12:02:58 +00:00
e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
...
Contributed by Da Zhou.
2019-01-10 11:58:39 +00:00
3420e26ae5
HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota.
2019-01-09 10:57:58 -07:00
188bebbe7e
HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
...
Contributed by Kai Xie.
2019-01-08 11:57:57 +00:00
7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
...
This reverts commit d7152332b3
2019-01-08 14:51:30 +09:00
d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
...
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
f87b3b11c4
HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu.
2019-01-07 15:56:49 +08:00
1f425271a7
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu."
...
This reverts commit e4fca6aae42b635125fb
2019-01-05 17:35:29 +09:00
040a202b20
HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu.
2019-01-03 21:37:49 +08:00
13f0ee21f2
HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali.
2019-01-02 09:33:09 -07:00
d8f670ff28
HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal.
2018-12-27 08:19:25 -07:00
f82922dcfa
YARN-5168. Added exposed port information for Docker container.
...
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies.
...
Contributed by Steve Loughran.
2018-12-20 18:09:01 +00:00
784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress.
...
Contributed by Vishwajeet Dusane.
2018-12-20 18:00:05 +00:00
346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
...
Contributed by Da Zhou.
2018-12-17 11:15:20 +00:00
62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500.
...
Contributed by Da Zhou.
(cherry picked from commit d09dbcc8fb
2018-12-17 11:10:22 +00:00
b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
...
Contributed by Da Zhou.
2018-12-17 11:04:40 +00:00
82b798581d
HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota.
2018-12-12 09:30:13 -07:00
1a25bbe9ec
HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota.
2018-12-11 08:33:13 -07:00
c35de95a22
HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota.
2018-12-11 08:29:39 -07:00
3ff8580f22
HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota)
2018-12-10 14:03:08 -07:00
8fc0d04517
HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti.
2018-12-07 18:30:36 -08:00
a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
...
Contributed by Da Zhou.
2018-12-04 13:39:10 +00:00
7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
...
Contributed by Da Zhou.
2018-11-30 10:12:41 +00:00
7eb0d3a324
HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota.
2018-11-29 09:36:39 -07:00
c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
...
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
085f10e75d
HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
e148c3ff09
HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
5d96b74f33
HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota.
2018-11-28 10:45:09 -07:00
4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
...
Contributed by Steve Loughran.
2018-11-27 20:39:54 +00:00
300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename.
...
Contributed by Da Zhou <da.zhou@microsoft.com>
(cherry picked from commit 1a3a4960d9
2018-11-27 18:13:07 +00:00
a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2.
...
Contributed by Junhua Gu and Da Zhou.
2018-11-23 14:17:36 +00:00
5ff0cf86a9
HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu.
2018-11-23 13:56:54 +08:00
be0708c6eb
HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu.
2018-11-19 14:00:54 +08:00
914b0cf15f
HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia.
2018-11-15 13:58:13 -08:00
Andrew Olson
Da Zhou
Steve Loughran
Akira Ajisaka
Shweta Yakkali
Adam Antal
Gabor Bota
Giovanni Matteo Fumarola
Sean Mackrory
Kai Xie
Weiwei Yang
Sunil G
Sean Mackrory
Eric Yang
Vishwajeet Dusane
Inigo Goiri
Ewan Higgs
Arpit Agarwal