Commit Graph

1360 Commits

Author SHA1 Message Date
Akira Ajisaka 3ea4f41d9f
MAPREDUCE-6794. Remove unused properties from TTConfig.java 2019-06-07 10:27:41 +09:00
Steve Loughran ec26c431f9
HADOOP-16117. Update AWS SDK to 1.11.563.
Contributed by Steve Loughran.

Change-Id: I7c46ed2a6378e1370f567acf4cdcfeb93e43fa13
2019-06-06 10:08:18 +01:00
Steve Loughran 309501c6fa
Revert "HADOOP-16050: s3a SSL connections should use OpenSSL"
This reverts commit b067f8acaa.

Change-Id: I584b050a56c0e6f70b11fa3f7db00d5ac46e7dd8
2019-06-05 13:54:55 +01:00
Steve Loughran 7724d8031b Revert "HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors"
This reverts commit 5906268f0d.
2019-06-05 13:54:42 +01:00
Akira Ajisaka afd844059c HADOOP-16331. Fix ASF License check in pom.xml
Signed-off-by: Takanobu Asanuma <tasanuma@apache.org>
2019-05-29 17:25:13 +09:00
Steve Loughran 0c73dba3a6
HADOOP-16332. Remove S3A dependency on http core.
Contributed by Steve Loughran.

Change-Id: I53209c993a405fefdb5e1b692d5a56d027d3b845
2019-05-28 22:50:37 +01:00
Akira Ajisaka 9f933e6446
HADOOP-16323. https everywhere in Maven settings. 2019-05-27 15:24:59 +09:00
Sahil Takiar 5906268f0d HADOOP-16321: ITestS3ASSL+TestOpenSSLSocketFactory failing with java.lang.UnsatisfiedLinkErrors 2019-05-21 11:30:45 -06:00
Ben Roling a36274d699
HADOOP-16085. S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite.
Contributed by Ben Roling.

S3Guard will now track the etag of uploaded files and, if an S3
bucket is versioned, the object version.

You can then control how to react to a mismatch between the data
in the DynamoDB table and that in the store: warn, fail, or, when
using versions, return the original value.

This adds two new columns to the table: etag and version.
This is transparent to older S3A clients -but when such clients
add/update data to the S3Guard table, they will not add these values.
As a result, the etag/version checks will not work with files uploaded by older clients.

For a consistent experience, upgrade all clients to use the latest hadoop version.
2019-05-19 22:29:54 +01:00
Sahil Takiar b067f8acaa HADOOP-16050: s3a SSL connections should use OpenSSL
(cherry picked from commit aebf229c17)
2019-05-16 08:57:54 -06:00
Andrew Olson c15b3bca86
HADOOP-16294: Enable access to input options by DistCp subclasses.
Adding a protected-scope getter for the DistCpOptions, so that a subclass does
not need to save its own copy of the inputOptions supplied to its constructor,
if it wishes to override the createInputFileListing method with logic similar
to the original implementation, i.e. calling CopyListing#buildListing with a path and input options.

Author:    Andrew Olson
2019-05-16 16:11:12 +02:00
Weiwei Yang 2d8282bb82 HADOOP-16306. AliyunOSS: Remove temporary files when upload small files to OSS. Contributed by wujinhu. 2019-05-14 14:05:39 -07:00
DadanielZ ff27e8eabd HADOOP-16251. ABFS: add FSMainOperationsBaseTest. Re-commit to fix git metadata.
Author: Da Zhou
2019-05-10 13:57:30 -07:00
Aaron Fabbri 5f1a34649f Revert "HADOOP-16251. ABFS: add FSMainOperationsBaseTest"
This reverts commit 7c2d7c05a9.

Git Commit name and email were incorrect. Will re-commit.
2019-05-10 13:55:56 -07:00
DadanielZ 7c2d7c05a9 HADOOP-16251. ABFS: add FSMainOperationsBaseTest
Author: Da Zhou
2019-05-10 12:46:03 -07:00
Rajat Khandelwal 591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
Contributed by Rajat Khandelwal
2019-05-09 21:13:29 +01:00
Da Zhou 3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
Author:    Da Zhou
2019-05-08 17:20:46 +01:00
Giovanni Matteo Fumarola 7a3188d054 HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena. 2019-05-02 12:58:42 -07:00
Ben Roling 0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure. 2019-04-30 11:53:26 +01:00
Da Zhou 1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
Contributed by Da Zhou.
2019-04-29 13:27:28 +01:00
Ben Roling e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests.
Contributed by Ben Roling.
2019-04-24 14:55:58 +01:00
Takanobu Asanuma a5ceed26f3 HADOOP-16253. Update AssertJ to 3.12.2. 2019-04-16 12:53:52 +09:00
Akira Ajisaka 4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19 2019-04-15 10:46:10 +09:00
Sahil Takiar 2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
Author:    Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
Masatake Iwasaki bbdbc7a9a1 HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki. 2019-04-12 11:52:18 +09:00
Steve Loughran cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables.
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.

Contributed by Steve Loughran.
2019-04-11 17:12:12 -07:00
Giovanni Matteo Fumarola 358e928622 YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Addendum. Contributed by Inigo Goiri. 2019-04-09 14:18:05 -07:00
Steve Loughran 215ffc792e HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException
Contributed by Steve Loughran.

Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
2019-04-04 21:14:18 +01:00
Steve Loughran 366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true (#685)
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without  using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.

Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.

Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up

Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.

Contributed by Steve Loughran.

Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
Akira Ajisaka 8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji. 2019-04-03 19:35:02 +09:00
Steve Loughran df578c07ec HADOOP-16195 MarshalledCredentials toString
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
2019-03-28 17:01:57 +00:00
Gabor Bota b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
Author:    Gabor Bota
2019-03-28 15:59:25 +00:00
Gabor Bota cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren.
Author:    Gabor Bota
2019-03-28 15:49:56 +00:00
Siyao Meng ce4bafdf44
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
Contributed by Siyao Meng
2019-03-26 18:42:54 +00:00
Lokesh Jain ae2eb2dd42 HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists (#636) 2019-03-22 11:42:00 +00:00
Andrew Olson faba3591d3
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
Author:    Andrew Olson
2019-03-22 10:35:30 +00:00
Steve Loughran 9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
Contributed by Steve Loughran.

This includes
 - HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
 - MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
 - MAPREDUCE-7091. Terasort on S3A to switch to new committers
 - MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
Ajay Kumar 399563fec6 HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia. 2019-03-20 11:20:45 -07:00
Weiwei Yang 568d3ab8b6 HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu. 2019-03-19 09:55:32 +08:00
Adam Antal c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
Contributed by Adam Antal.
2019-03-18 19:13:13 +00:00
Vishwajeet Dusane 1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.
Contributed by Vishwajeet Dusane.
2019-03-18 13:18:08 +00:00
Ben Roling 6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
Author: Ben Roling <ben.roling@gmail.com>

Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
Denes Gerencser ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console.
Follow-on to HADOOP-15851.

Author:    Denes Gerencser <dgerencser@cloudera.com>
2019-03-09 18:45:17 +00:00
Shweta Yakkali bb8ad096e7 HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore
(Contributed by Shweta Yakkali via Daniel Templeton)

Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
2019-03-09 09:55:56 -08:00
Steve Loughran 0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
Nobody gets seek right. No matter how many times they think they have.

Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
Da Zhou e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties.
Author:    Da Zhou <da.zhou@microsoft.com>
2019-03-08 13:53:08 +00:00
Da Zhou 05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens.
Contributed by Da Zhou.

Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:02:34 +00:00
Steve Loughran 65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store.
Contributed by Steve Loughran.
2019-02-28 14:22:32 +00:00
Abhishek Modi 52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
Da Zhou 3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name
Contributed by Da Zhou.
2019-02-26 15:37:24 +00:00
Steve Loughran a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
Contributed by Steve Loughran.
2019-02-21 14:08:46 +00:00
Masatake Iwasaki aa3ad36605 HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki. 2019-02-20 22:00:57 +09:00
bibinchundatt 779dae4de7 YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T. 2019-02-20 00:36:57 +05:30
Adam Antal 1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
Contributed by Adam Antal.

(Revised patch applied after stevel committed the wrong one; that has been reverted)
2019-02-19 11:33:02 +00:00
Steve Loughran 920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."
This reverts commit c4a00d1ad3.
2019-02-18 14:57:22 +00:00
Akira Ajisaka 0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR. 2019-02-18 15:21:46 +09:00
bibinchundatt 134ae8fc80 YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt. 2019-02-14 22:56:52 +05:30
Masatake Iwasaki 6c999fe4b0 HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki. 2019-02-12 06:07:47 +09:00
Da Zhou 1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
Contributed by Da Zhou and Junhua Gu.
2019-02-07 21:58:21 +00:00
Steve Loughran 668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
(accidentally mixed in two patches)

This reverts commit fa8cd1bf28.
2019-02-07 21:57:22 +00:00
Ranith Sardar 546c5d70ef
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on. 2019-02-07 21:48:07 +00:00
Andrew Olson de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
Da Zhou fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
Da Zhou ba9efe06fa
HADOOP-16074. WASB: Update container not found error code.
Contributed by Da Zhou.
2019-02-05 14:39:45 +00:00
Steve Loughran f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
S3A to implement S3 Select through this API.

The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.

The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.

This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.

In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).

Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
Akira Ajisaka 1129288cf5
HADOOP-14178. Move Mockito up to version 2.23.4. Contributed by Akira Ajisaka and Masatake Iwasaki. 2019-01-29 18:29:56 -08:00
Shweta Yakkali 02eb91856e HADOOP-16041. Include Hadoop version in User-Agent string for ABFS. Contributed by Shweta Yakkali.
Signed-off-by: Sean Mackrory <mackrorysd@apache.org>
2019-01-29 16:50:14 -08:00
Akira Ajisaka 3c60303ac5
HADOOP-16065. -Ddynamodb should be -Ddynamo in AWS SDK testing document. 2019-01-25 10:27:59 +09:00
Da Zhou 00ad9e23e8
HADOOP-16048. ABFS: Fix Date format parser.
Contributed by Da Zhou.
2019-01-22 16:40:42 +00:00
Da Zhou 30863c5ae3
HADOOP-16044. ABFS: Better exception handling of DNS errors followup
Contributed by Da Zhou.
2019-01-14 19:32:56 +00:00
Steve Loughran 6d0bffe17e
HADOOP-14556. S3A to support Delegation Tokens.
Contributed by Steve Loughran and Daryn Sharp.
2019-01-14 17:59:27 +00:00
Adam Antal c4a00d1ad3
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
Contributed by Adam Antal.
2019-01-14 17:27:00 +00:00
Gabor Bota 04fcbef9c9
HADOOP-16043. NPE in ITestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is not set.
Contributed by Gabor Bota.
2019-01-14 13:12:05 +00:00
Giovanni Matteo Fumarola fb8932a727 HADOOP-16029. Consecutive StringBuilder.append can be reused. Contributed by Ayush Saxena. 2019-01-11 10:54:49 -08:00
Da Zhou dddad985d7
HADOOP-15975. ABFS: remove timeout check for DELETE and RENAME.
Contributed by Da Zhou.

(cherry picked from commit 8b5fbe7a12)
2019-01-11 11:13:41 +00:00
Da Zhou 852701f793
HADOOP-16036. WASB: Disable jetty logging configuration announcement.
Contributed by Da Zhou.
2019-01-10 12:07:09 +00:00
Da Zhou 7211269142
HADOOP-15662. Better exception handling of DNS errors.
Contributed by Da Zhou.
2019-01-10 12:02:58 +00:00
Da Zhou e8d1900369
HADOOP-16040. ABFS: Bug fix for tolerateOobAppends configuration.
Contributed by Da Zhou.
2019-01-10 11:58:39 +00:00
Sean Mackrory 3420e26ae5 HADOOP-16027. [DOC] Effective use of FS instances during S3A integration tests. Contributed by Gabor Bota. 2019-01-09 10:57:58 -07:00
Kai Xie 188bebbe7e HADOOP-16018. DistCp won't reassemble chunks when blocks per chunk > 0.
Contributed by Kai Xie.
2019-01-08 11:57:57 +00:00
Akira Ajisaka 7f78397036
Revert "HADOOP-14556. S3A to support Delegation Tokens."
This reverts commit d7152332b3.
2019-01-08 14:51:30 +09:00
Steve Loughran d7152332b3
HADOOP-14556. S3A to support Delegation Tokens.
Contributed by Steve Loughran.
2019-01-07 13:18:03 +00:00
Weiwei Yang f87b3b11c4 HADOOP-16030. AliyunOSS: bring fixes back from HADOOP-15671. Contributed by wujinhu. 2019-01-07 15:56:49 +08:00
Sunil G 1f425271a7
Revert "HADOOP-15759. AliyunOSS: Update oss-sdk version to 3.0.0. Contributed by Jinhu Wu."
This reverts commit e4fca6aae4.

Revert "HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu."

This reverts commit 2b635125fb.
2019-01-05 17:35:29 +09:00
Weiwei Yang 040a202b20 HADOOP-15323. AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore. Contributed wujinhu. 2019-01-03 21:37:49 +08:00
Sean Mackrory 13f0ee21f2 HADOOP-15860. ABFS: Throw exception when directory / file name ends with a period (.). Contributed by Shweta Yakkali. 2019-01-02 09:33:09 -07:00
Sean Mackrory d8f670ff28 HADOOP-15819. FileSystem cache misused in S3A integration tests. Contributed by Adam Antal. 2018-12-27 08:19:25 -07:00
Eric Yang f82922dcfa YARN-5168. Added exposed port information for Docker container.
Contributed by Xun Liu
2018-12-21 19:44:07 -05:00
Steve Loughran a668f8e6c6
HADOOP-16015. Add bouncycastle jars to hadoop-aws as test dependencies.
Contributed by Steve Loughran.
2018-12-20 18:09:01 +00:00
Vishwajeet Dusane 784ea3aff6
HADOOP-15965. Upgrade to ADLS SDK which has major performance improvement for ingress/egress.
Contributed by Vishwajeet Dusane.
2018-12-20 18:00:05 +00:00
Da Zhou 346c0c8aff
HADOOP-16004. ABFS: Convert 404 error response in AbfsInputStream and AbfsOutPutStream to FileNotFoundException.
Contributed by Da Zhou.
2018-12-17 11:15:20 +00:00
Da Zhou 62df60737c
HADOOP-15972 ABFS: reduce list page size to to 500.
Contributed by Da Zhou.

(cherry picked from commit d09dbcc8fb)
2018-12-17 11:10:22 +00:00
Da Zhou b2523d8100
HADOOP-15969. ABFS: getNamespaceEnabled can fail blocking user access thru ACLs.
Contributed by Da Zhou.
2018-12-17 11:04:40 +00:00
Sean Mackrory 82b798581d HADOOP-15988. DynamoDBMetadataStore#innerGet should support empty directory flag when using authoritative listings. Contributed by Gabor Bota. 2018-12-12 09:30:13 -07:00
Sean Mackrory 1a25bbe9ec HADOOP-15845. Require explicit URI on CLI for s3guard init and destroy. Contributed by Gabor Bota. 2018-12-11 08:33:13 -07:00
Sean Mackrory c35de95a22 HADOOP-15987. ITestDynamoDBMetadataStore should check if table configured properly. Contributed by Gabor Bota. 2018-12-11 08:29:39 -07:00
Sean Mackrory 3ff8580f22 HADOOP-15428. s3guard bucket-info will create s3guard table if FS is set to do this automatically. (Contributed by Gabor Bota) 2018-12-10 14:03:08 -07:00
Inigo Goiri 8fc0d04517 HDFS-14001. [PROVIDED Storage] bootstrapStandby should manage the InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-12-07 18:30:36 -08:00
Da Zhou a8bbd818d5
HADOOP-15968. ABFS: add try catch for UGI failure when initializing ABFS.
Contributed by Da Zhou.
2018-12-04 13:39:10 +00:00
Da Zhou 7ccb640a66
HADOOP-15957. WASB: Add asterisk wildcard support for PageBlobDirSet.
Contributed by Da Zhou.
2018-11-30 10:12:41 +00:00
Sean Mackrory 7eb0d3a324 HADOOP-14927. ITestS3GuardTool failures in testDestroyNoBucket(). Contributed by Gabor Bota. 2018-11-29 09:36:39 -07:00
Ewan Higgs c1d24f8483
HDFS-13713. Add specification of Multipart Upload API to FS specification, with contract tests.
Contributed by Ewan Higgs and Steve Loughran.
2018-11-29 15:12:17 +00:00
Sean Mackrory 085f10e75d HADOOP-15947. Fix ITestDynamoDBMetadataStore test error issues. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory e148c3ff09 HADOOP-15798. LocalMetadataStore put() does not retain isDeleted in parent listing. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Sean Mackrory 5d96b74f33 HADOOP-15370. S3A log message on rm s3a://bucket/ not intuitive. Contributed by Gabor Bota. 2018-11-28 10:45:09 -07:00
Steve Loughran 4c106fca0c
HADOOP-15932. Oozie unable to create sharelib in s3a filesystem.
Contributed by Steve Loughran.
2018-11-27 20:39:54 +00:00
Da Zhou 300f772c36
HADOOP-15940. ABFS: For HNS account, avoid unnecessary get call when doing Rename.
Contributed by Da Zhou <da.zhou@microsoft.com>

(cherry picked from commit 1a3a4960d9)
2018-11-27 18:13:07 +00:00
Da Zhou a8302e398c
HADOOP-15872. ABFS: Update to target 2018-11-09 REST version for ADLS Gen 2.
Contributed by Junhua Gu and Da Zhou.
2018-11-23 14:17:36 +00:00
Weiwei Yang 5ff0cf86a9 HADOOP-15943. AliyunOSS: add missing owner & group attributes for oss FileStatus. Contributed by wujinhu. 2018-11-23 13:56:54 +08:00
Weiwei Yang be0708c6eb HADOOP-15919. AliyunOSS: Enable Yarn to use OSS. Contributed by wujinhu. 2018-11-19 14:00:54 +08:00
Arpit Agarwal 914b0cf15f HADOOP-12558. distcp documentation is woefully out of date. Contributed by Dinesh Chitlangia. 2018-11-15 13:58:13 -08:00
Akira Ajisaka 66b1335bb3
HADOOP-15926. Document upgrading the section in NOTICE.txt when upgrading the version of AWS SDK. Contributed by Dinesh Chitlangia. 2018-11-15 16:30:24 +09:00
Sammi Chen 3fade865ce HADOOP-15917. AliyunOSS: fix incorrect ReadOps and WriteOps in statistics. Contributed by Jinhu Wu. 2018-11-14 12:58:57 +08:00
Da Zhou a13be203b7
HADOOP-15876. Use keySet().removeAll() to remove multiple keys from Map in AzureBlobFileSystemStore
Contributed by Da Zhou.
2018-11-13 21:46:18 +00:00
Da Zhou 9dbb2b67c6
HADOOP-15812. ABFS: Improve AbfsRestOperationException format to ensure full msg can be displayed on console.
Author:    Da Zhou <da.zhou@microsoft.com>
2018-11-09 11:06:23 +00:00
Junhua Gu 724c15007b
HADOOP-15846. ABFS: fix mask related bugs in setAcl, modifyAclEntries and removeAclEntries.
Contributed by Junhua Gu.

(cherry picked from commit 66715005f9)
2018-11-08 17:21:40 +00:00
Sammi Chen 7574d18538 HADOOP-15868. AliyunOSS: update document for properties of multiple part download, multiple part upload and directory copy. Contributed by Jinhu Wu. 2018-10-26 15:19:56 +08:00
Daniel Templeton e374584479 HADOOP-15823. ABFS: Stop requiring client ID and tenant ID for MSI
(Contributed by Da Zhou via Daniel Templeton)

Change-Id: I546ab3a1df1efec635c08c388148e718dc4a9843
2018-10-24 14:18:35 -07:00
Robert Kanter 5e294e950b YARN-8919. Some tests fail due to NoClassDefFoundError for OperatorCreationException (tasanuma0829 via rkanter) 2018-10-23 13:37:17 -07:00
Ted Yu e2cecb681e HADOOP-15850. CopyCommitter#concatFileChunks should check that the blocks per chunk is not 0. Contributed by Ted Yu.
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2018-10-19 13:21:06 -07:00
Giovanni Matteo Fumarola cc09b2b0c7 HDFS-14007. Incompatible layout when generating FSImage. Contributed by Inigo Goiri. 2018-10-18 17:24:23 -07:00
Steve Loughran d59ca43bff
HADOOP-15826. @Retries annotation of putObject() call & uses wrong.
Contributed by Steve Loughran and Ewan Higgs.
2018-10-16 20:02:54 +01:00
Steve Loughran ef9dc6c44c
HADOOP-15851. Disable wildfly logs to the console.
Contributed by Vishwajeet Dusane.
2018-10-15 22:53:55 +01:00
bibinchundatt b4a38e7b3e YARN-8830. SLS tool fix node addition. Contributed by Bibin A Chundatt. 2018-10-15 16:10:25 +05:30
Steve Loughran e36ae9639f
HADOOP-15831. Include modificationTime in the toString method of CopyListingFileStatus.
Contributed by Ted Yu.
2018-10-12 09:59:19 +01:00
Steve Loughran ee816f1fd7
HADOOP-15837. DynamoDB table Update can fail S3A FS init.
Contributed by Steve Loughran.
2018-10-11 14:57:38 +01:00
Steve Loughran bd50fa956b
HADOOP-15825. ABFS: Enable some tests for namespace not enabled account using OAuth.
Contributed by Da Zhou.
2018-10-09 20:02:12 +01:00
Jason Lowe 4b5b1ac3d1 MAPREDUCE-7130. Rumen crashes trying to handle MRAppMaster recovery events. Contributed by Peter Bacsko 2018-10-09 13:27:03 -05:00
Steve Loughran 7ba1cfdea7
HADOOP-15827. NPE in DynamoDBMetadataStore.lambda$listChildren for root + auth S3Guard.
Contributed by Gabor Bota
2018-10-09 10:46:41 +01:00
Steve Loughran 273cc2d4e9
HADOOP-15809. ABFS: better exception handling when making getAccessToken call.
Contributed by Da Zhou
2018-10-05 11:29:03 +01:00
Steve Loughran 7051bd78b1
HADOOP-15795. Make HTTPS the default protocol for ABFS.
Contributed by Da Zhou.
2018-10-03 12:52:53 +01:00
Aaron Fabbri 4f752d442b
HADOOP-15621 2/2 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-03 00:24:29 -07:00
Aaron Fabbri 046b8768af
HADOOP-15621 S3Guard: Implement time-based (TTL) expiry for Authoritative Directory Listing. Contributed by Gabor Bota 2018-10-02 21:22:49 -07:00
Steve Loughran e8b8604314
HADOOP-15792. typo in AzureBlobFileSystem.getIsNamespaceEnabeld.
Contributed by Abhishek Modi.
2018-10-02 18:57:20 +01:00
Sunil G 58fa96b697 Changed version in trunk to 3.3.0-SNAPSHOT. 2018-10-02 22:41:41 +05:30
Steve Loughran a383ac47ca
HADOOP-15801. ABFS: Fixing skipUserGroupMetadata in AzureBlobFileSystemStore.
Contributed by Da Zhou
2018-10-02 11:42:52 +01:00
Steve Loughran 43bc984891
HADOOP-15793. ABFS: Skip unsupported test cases when non namespace enabled in ITestAzureBlobFileSystemAuthorization
Contributed by Yuan Gao.
2018-10-02 11:37:28 +01:00
Steve Loughran a4abf02028
HADOOP-15739. ABFS: remove unused maven dependencies and add used undeclared dependencies.
Contributed by Da Zhou.
2018-09-25 20:58:32 +01:00
Steve Loughran d5da9928c9 HADOOP-15723. ABFS: Ranger Support.
Contributed by Yuan Gao.
2018-09-25 19:13:10 +01:00
Sammi Chen 2b635125fb HADOOP-15671. AliyunOSS: Support Assume Roles in AliyunOSS. Contributed by Jinhu Wu. 2018-09-25 19:48:30 +08:00
Mingliang Liu c07715e378 HADOOP-15781 S3A assumed role tests failing due to changed error text in AWS exceptions. Contributed by Steve Loughran 2018-09-24 12:53:21 -07:00
Sunil G d060cbea48 HDFS-13937. Multipart Uploader APIs to be marked as private/unstable in 3.2.0. Contributed by Steve Loughran. 2018-09-24 21:19:47 +05:30
Sean Mackrory 0def61482b Merge branch 'HADOOP-15407' into trunk 2018-09-22 21:19:12 -06:00
Steve Loughran d0b4624c88
HADOOP-15778. ABFS: Fix client side throttling for read.
Contributed by Sneha Varma.
2018-09-21 11:06:24 +01:00
Steve Loughran a5692c2da5 HADOOP-15704. Mark ABFS extension package and interfaces as LimitedPrivate/Unstable.
Contributed by Steve Loughran.
2018-09-20 17:36:18 +01:00
Sean Mackrory 8e831ba458 HADOOP-15773. Fixing checkstyle and other issues raised by Yetus. 2018-09-19 16:56:33 -06:00
Steve Loughran a55d26b23e
HADOOP-15769. ABFS: distcp tests are always skipped.
Contributed by Steve Loughran
2018-09-19 13:57:39 +01:00
Steve Loughran df2166a643
HADOOP-15719. Fail-fast when using OAuth over http.
Contributed by Da Zhou.
2018-09-18 12:20:52 +01:00
Steve Loughran 51d368982b
HADOOP-15714. Tune abfs/wasb parallel and sequential test execution.
Contributed by Da Zhou.
2018-09-18 12:09:25 +01:00
Steve Loughran 524776625d
HADOOP-15715. ITestAzureBlobFileSystemE2E timing out with non-scale timeout of 10 min.
Contributed by Da Zhou
2018-09-18 11:48:46 +01:00
Steve Loughran 1cf38a38da
HADOOP-15744. AbstractContractAppendTest fails against HDFS on HADOOP-15407 branch.
Contributed by Steve Loughran.
2018-09-18 10:56:56 +01:00
Steve Loughran 26d0c63a1e
HADOOP-15754. s3guard: testDynamoTableTagging should clear existing config.
Contributed by Gabor Bota.
2018-09-17 22:40:08 +01:00
Thomas Marquardt b4c23043d3 HADOOP-15757. ABFS: remove dependency on common-codec Base64.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 26211019c8 HADOOP-15753. ABFS: support path "abfs://mycluster/file/path"
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt e5593cbd83 HADOOP-15694. ABFS: Allow OAuth credentials to not be tied to accounts.
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 13c70e9ba3 HADOOP-15740. ABFS: Check variable names during initialization of AbfsClientThrottlingIntercept.
Contributed by Sneha Varma.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 6801b30733 HADOOP-15728. ABFS: Add backward compatibility to handle Unsupported Operation
for storage account with no namespace feature.

Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 347a52a866 Fixing findbugs and license issues related to:
HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 97f06b3fc7 HADOOP-15703. ABFS - Implement client-side throttling.
Contributed by Sneha Varma and Thomas Marquardt.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 4410eacba7 HADOOP-15664. ABFS: Reduce test run time via parallelization and grouping.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 81dc4a995c HADOOP-15663. ABFS: Simplify configuration.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt df57c6c3b1 HADOOP-15692. ABFS: extensible support for custom oauth.
Contributed by Junhua Gu and Rajeev Bansal.
2018-09-17 19:54:01 +00:00
Thomas Marquardt dd2b22fa31 HADOOP-15682. ABFS: Add support for StreamCapabilities. Fix javadoc and checkstyle.
Contributed by Thomas Marquardt.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 6b6f8cc2be HADOOP 15688. ABFS: InputStream wrapped in FSDataInputStream twice.
Contributed by Sean Mackrory.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 9c1e4e8139 HADOOP-15661. ABFS: Add support for ACL.
Contributed by Junhua Gu and Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt 9149b9703e HADOOP-15660. ABFS: Add support for OAuth
Contributed by Da Zhou, Rajeev Bansal, and Junhua Gu.
2018-09-17 19:54:01 +00:00
Thomas Marquardt d6a4f39bd5 HADOOP-15669. ABFS: Improve HTTPS Performance.
Contributed by Vishwajeet Dusane.
2018-09-17 19:54:01 +00:00
Thomas Marquardt cc5cc60c41 Fixing issue due to commit 2b2399d6 after rebase onto trunk. 2018-09-17 19:54:01 +00:00
Thomas Marquardt b54b0c1b67 HADOOP-15659. Code changes for bug fix and new tests.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Thomas Marquardt ce03a93f78 HADOOP-15446. ABFS: tune imports & javadocs; stabilise tests.
Contributed by Steve Loughran and Da Zhou.
2018-09-17 19:54:01 +00:00
Steve Loughran a271fd0eca HADOOP-15560. ABFS: removed dependency injection and unnecessary dependencies.
Contributed by Da Zhou.
2018-09-17 19:54:01 +00:00
Steve Loughran f044deedbb HADOOP-15407. HADOOP-15540. Support Windows Azure Storage - Blob file system "ABFS" in Hadoop: Core Commit.
Contributed by Shane Mainali, Thomas Marquardt, Zichen Sun, Georgi Chalakov, Esfandiar Manii, Amit Singh, Dana Kaban, Da Zhou, Junhua Gu, Saher Ahwal, Saurabh Pant, James Baker, Shaoyu Zhang, Lawrence Chen, Kevin Chen and Steve Loughran
2018-09-17 19:54:01 +00:00
Steve Loughran d7c0a08a1c
HADOOP-15426 Make S3guard client resilient to DDB throttle events and network failures (Contributed by Steve Loughran) 2018-09-12 21:04:49 -07:00
Aaron Fabbri d32a8d5d58
HADOOP-14734 add option to tag DDB table(s) created. (Contributed by Gabor Bota and Abe Fine) 2018-09-12 16:36:01 -07:00
Mingliang Liu 1f6c4545cf HADOOP-15750. Remove obsolete S3A test ITestS3ACredentialsInURL. Contributed by Steve Loughran 2018-09-12 10:58:39 -07:00
Sean Mackrory 47b72c87eb HADOOP-15635. s3guard set-capacity command to fail fast if bucket is unguarded.
Contributed by Gabor Bota.
2018-09-12 09:12:38 -06:00
bibinchundatt c44088ac19 YARN-8739. Fix jenkins issues for Node Attributes branch. Contributed by Sunil Govindan. 2018-09-12 16:01:01 +05:30
Naganarasimha eb08543c7a YARN-8103. Add CLI interface to query node attributes. Contributed by Bibin A Chundatt. 2018-09-12 16:01:01 +05:30
Sunil G 89b3ebd11e YARN-8092. Expose Node Attributes info via RM nodes REST API. Contributed by Weiwei Yang. 2018-09-12 16:01:00 +05:30
Mingliang Liu 87f63b6479 HADOOP-14833. Remove s3a user:secret authentication. Contributed by Steve Loughran 2018-09-11 17:18:42 -07:00
Steve Loughran 1eef0acfe5
HADOOP-15677. WASB: Add support for StreamCapabilities.
Contributed by Thomas Marquardt.
2018-09-10 14:24:41 +01:00
Gabor Bota 36c7c78260
HADOOP-15709 Move S3Guard LocalMetadataStore constants to org.apache.hadoop.fs.s3a.Constants (Contributed by Gabor Bota) 2018-09-07 10:25:20 -07:00
Steve Loughran 5a0babf765
HADOOP-15107. Stabilize/tune S3A committers; review correctness & docs.
Contributed by Steve Loughran.
2018-08-30 14:49:53 +01:00
Steve Loughran e8d138ca7c
HADOOP-15680. ITestNativeAzureFileSystemConcurrencyLive times out.
Contributed by Andras Bokor.
2018-08-30 14:36:00 +01:00
Steve Loughran 2e6c1109dc
HADOOP-15667. FileSystemMultipartUploader should verify that UploadHandle has non-0 length.
Contributed by Ewan Higgs
2018-08-30 14:33:16 +01:00
Yufei Gu 8563fd67be YARN-8632. Threads in SLS quit without logging exception. Contributed by Xianghao Lu. 2018-08-24 11:23:17 -07:00
Surendra Singh Lilhore 96c4575d73 HDFS-13805. Journal Nodes should allow to format non-empty directories with -force option. Contributed by Surendra Singh Lilhore. 2018-08-24 08:14:57 +05:30
Aaron Fabbri d7232857d8
HADOOP-14154 Persist isAuthoritative bit in DynamoDBMetaStore (Contributed by Gabor Bota) 2018-08-17 10:15:39 -07:00
Steve Loughran 0e832e7a74
HADOOP-15642. Update aws-sdk version to 1.11.375.
Contributed by Steve Loughran.
2018-08-16 09:58:46 -07:00
Akira Ajisaka 3e3963b035
HADOOP-15552. Move logging APIs over to slf4j in hadoop-tools - Part2. Contributed by Ian Pickering. 2018-08-16 00:31:59 +09:00
Ewan Higgs a13929ddcb HADOOP-15645. ITestS3GuardToolLocal.testDiffCommand fails if bucket has per-bucket binding to DDB. Contributed by Steve Loughran. 2018-08-13 12:57:45 +02:00
Steve Loughran da9a39eed1
HADOOP-15583. Stabilize S3A Assumed Role support.
Contributed by Steve Loughran.
2018-08-08 22:57:24 -07:00
Ewan Higgs 2ec97abb2e HADOOP-15576. S3A Multipart Uploader to work with S3Guard and encryption Originally contributed by Ewan Higgs with refinements by Steve Loughran. 2018-08-08 13:50:23 +02:00
Sean Mackrory 7862f1523f HADOOP-15400. Improve S3Guard documentation on Authoritative Mode implementation. (Contributed by Gabor Bota) 2018-08-07 20:13:09 -06:00
Steve Loughran 48673bc2a8
HADOOP-15626. FileContextMainOperationsBaseTest.testBuilderCreateAppendExistingFile fails on filesystems without append.
Contributed by Steve Loughran.
2018-08-03 16:06:00 -07:00
Inigo Goiri 9fea5c9ee7 YARN-8175. Add support for Node Labels in SLS. Contributed by Abhishek Modi. 2018-07-31 09:36:34 -07:00
Sammi Chen 0857f116b7 HADOOP-15607. AliyunOSS: fix duplicated partNumber issue in AliyunOSSBlockOutputStream. Contributed by Jinhu Wu. 2018-07-30 10:53:44 +08:00
Sean Mackrory 59adeb8d7f HADOOP-15636. Follow-up from HADOOP-14918; restoring test under new name. Contributed by Gabor Bota. 2018-07-27 18:23:29 -06:00
bibinchundatt 3d3158cea4 YARN-8577. Fix the broken anchor in SLS site-doc. Contributed by Weiwei Yang. 2018-07-25 18:59:05 +05:30
Steve Loughran 45d9568aaa
HADOOP-15547/ WASB: improve listStatus performance.
Contributed by Thomas Marquardt.

(cherry picked from commit 749fff577e)
2018-07-19 12:31:19 -07:00
Wangda Tan 752dcce5f4 YARN-8511. When AM releases a container, RM removes allocation tags before it is released by NM. (Weiwei Yang via wangda)
Change-Id: I6f9f409f2ef685b405cbff547dea9623bf3322d9
2018-07-16 10:54:41 -07:00
Sean Mackrory a08812a1b1 HADOOP-15349. S3Guard DDB retryBackoff to be more informative on limits exceeded. Contributed by Gabor Bota. 2018-07-12 17:24:01 +02:00
Sean Mackrory d503f65b66 HADOOP-15541. [s3a] Shouldn't try to drain stream before aborting
connection in case of timeout.
2018-07-10 17:52:57 +02:00
Steve Loughran ca8b80bf59
HADOOP-15384. distcp numListstatusThreads option doesn't get to -delete scan.
Contributed by Steve Loughran.
2018-07-10 10:43:59 +01:00
Aaron Fabbri 93ac01cb59
HADOOP-15215 s3guard set-capacity command to fail on read/write of 0 (Gabor Bota) 2018-07-03 13:50:11 -07:00
Inigo Goiri 1804a31515 HDFS-13536. [PROVIDED Storage] HA for InMemoryAliasMap. Contributed by Virajith Jalaparti. 2018-07-02 10:48:20 -07:00
Inigo Goiri 73746c5da7 HDFS-13707. [PROVIDED Storage] Fix failing integration tests in ITestProvidedImplementation. Contributed by Virajith Jalaparti. 2018-06-29 09:56:13 -07:00
Akira Ajisaka 2b2399d623
HADOOP-15495. Upgrade commons-lang version to 3.7 in hadoop-common-project and hadoop-tools. Contributed by Takanobu Asanuma. 2018-06-28 14:37:22 +09:00
Todd Lipcon 7a3c6e9c3c HADOOP-15550. Avoid static initialization of ObjectMappers 2018-06-25 15:36:45 -07:00
Sean Mackrory c687a6617d HADOOP-15423. Merge fileCache and dirCache into ine single cache in LocalMetadataStore. Contributed by Gabor Bota. 2018-06-25 14:59:41 -06:00
Anu Engineer e16e5b307d HDDS-177. Create a releasable ozonefs artifact
Contributed by Marton, Elek.
2018-06-24 01:05:04 -07:00
Anu Engineer ca14fec02f HDDS-184. Upgrade common-langs version to 3.7 in hadoop-tools/hadoop-ozone.
Contributed by Takanobu Asanuma.
2018-06-23 15:49:44 -07:00
Sean Mackrory 55fad6a3de HADOOP-15416. Clear error message in S3Guard diff if source not found. Contributed by Gabor Bota. 2018-06-22 11:36:56 -06:00
Sean Mackrory b089a06793 HADOOP-14918. Remove the Local Dynamo DB test option. Contributed by Gabor Bota. 2018-06-20 16:45:08 -06:00
Chris Douglas f34744603e HADOOP-15533. Make WASB listStatus messages consistent. Contributed by Esfandiar Manii 2018-06-17 23:12:18 -07:00
Chris Douglas 980031bb04 HADOOP-13186. Multipart Uploader API. Contributed by Ewan Higgs 2018-06-17 11:54:26 -07:00
Jitendra Pandey f4c7c91123 HADOOP-15483. Upgrade jquery to version 3.3.1. Contributed by Lokesh Jain, Mukul Kumar Singh and Sunil Govindan. 2018-06-13 00:36:02 -07:00
Inigo Goiri c35481594f YARN-8422. TestAMSimulator failing with NPE. Contributed by Giovanni Matteo Fumarola. 2018-06-12 10:59:50 -07:00
Steve Loughran d901be6795
HADOOP-15506. Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks.
Contributed by Esfandiar Manii.
2018-06-06 18:28:14 +01:00
Anu Engineer 3b34148c4f HDDS-88. Create separate message structure to represent ports in DatanodeDetails.
Contributed by Nanda Kumar.
2018-05-30 08:52:07 -07:00
Aaron Fabbri 5f6769f796
HADOOP-15480 AbstractS3GuardToolTestBase.testDiffCommand fails when using dynamo (Gabor Bota) 2018-05-29 19:20:22 -07:00
Aaron Fabbri 30284d020d
HADOOP-14946 S3Guard testPruneCommandCLI can fail. Contributed by Gabor Bota. 2018-05-29 13:38:15 -07:00
Inigo Goiri 8fdc993a99 HADOOP-15498. TestHadoopArchiveLogs (#testGenerateScript, #testPrepareWorkingDir) fails on Windows. Contributed by Anbang Hu. 2018-05-28 16:45:42 -07:00
Johan Gustavsson d14e26b31f
HADOOP-15477. Make unjar in RunJar overrideable
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-28 17:29:59 +09:00
Steve Loughran ba842847c9 HADOOP-15478. WASB: hflush() and hsync() regression.
Contributed by Thomas Marquardt.
2018-05-21 11:12:34 +01:00
Steve Loughran cc3600aabd HADOOP-15469. S3A directory committer commit job fails if _temporary directory created under dest.
Contributed by Steve Loughran.

(cherry picked from commit 170f1040d4)
2018-05-17 19:08:55 +01:00
Steve Loughran b670837469 HADOOP-15442. ITestS3AMetrics.testMetricsRegister can't know metrics source's name.
Contributed by Sean Mackrory.
2018-05-15 16:13:56 +01:00
Steve Loughran ba051b0686
HADOOP-15354. hadoop-aliyun & hadoop-azure modules to mark hadoop-common as provided
Signed-off-by: Akira Ajisaka <aajisaka@apache.org>
2018-05-10 18:38:08 +09:00
Anu Engineer 46e0f27862 HDDS-18. Ozone Shell should use RestClient and RpcClient. Contributed by Lokesh Jain. 2018-05-09 23:00:13 -07:00
Sean Mackrory 1cfe7506f7 HADOOP-15356. Make HTTP timeout configurable in ADLS connector. Contributed by Atul Sikaria and Sean Mackrory. 2018-05-09 15:15:51 -06:00
Mukul Kumar Singh 343b51dcf4 HDDS-28. Ozone:Duplicate declaration in hadoop-tools/hadoop-ozone/pom.xml. Contributed by Tsz Wo Nicholas Sze. 2018-05-09 20:20:21 +05:30
Mukul Kumar Singh 208b97e969 HDDS-20. Ozone: Add support for rename key within a bucket for rpc client. Contributed by Lokesh Jain. 2018-05-09 19:06:07 +05:30
Aaron Fabbri 8981674bbc
HADOOP-15420 s3guard ITestS3GuardToolLocal failures in diff tests (Gabor Bota) 2018-05-08 18:58:42 -07:00
Aaron Fabbri 69aac696d9
HADOOP-13649 s3guard: implement time-based (TTL) expiry for LocalMetadataStore (Gabor Bota) 2018-05-08 15:29:54 -07:00
Steve Loughran 5b11b9fd41 HADOOP-15446. WASB: PageBlobInputStream.skip breaks HBASE replication.
Contributed by Thomas Marquardt
2018-05-07 11:54:08 +01:00
Aaron Fabbri 96c843f64b
HADOOP-15444 ITestS3GuardToolDynamo should only run with -Ddynamo (Aaron Fabbri) 2018-05-04 11:34:45 -07:00
Sean Mackrory 85381c7b60 HADOOP-15434. Upgrade to ADLS SDK that exposes current timeout. 2018-05-02 21:30:31 -06:00
Aaron Fabbri 919865a34b
HADOOP-15239 S3ABlockOutputStream.flush() be no-op when stream closed. Contributed by Gabor Bota. 2018-04-30 16:02:57 -07:00
Weiwei Yang ba2db39c65 YARN-8152. Add chart in SLS to illustrate the throughput of the scheduler. Contributed by Tao Yang. 2018-04-27 15:36:21 +08:00
Aaron Fabbri 7d8bcf534a
HADOOP-13756 LocalMetadataStore#put(DirListingMetadata) should also put file metadata into fileHash. (Gabor Bota) 2018-04-26 20:41:29 -07:00
Owen O'Malley 2adda92de1
HDFS-7240 git commit fixup
Signed-off-by: Owen O'Malley <omalley@apache.org>
2018-04-26 13:52:17 -07:00
Mukul Kumar Singh ae8ac7f082 HDFS-13424. Ozone: Refactor MiniOzoneClassicCluster. Contributed by Nanda Kumar. 2018-04-26 05:36:04 -07:00
Nanda kumar 025058f251 HDFS-13446. Ozone: Fix OzoneFileSystem contract test failures. Contributed by Mukul Kumar Singh. 2018-04-26 05:36:04 -07:00
Anu Engineer 8b832f3c35 HDFS-13405. Ozone: Rename HDSL to HDDS.
Contributed by Ajay Kumar, Elek Marton, Mukul Kumar Singh, Shashikant Banerjee and Anu Engineer.
2018-04-26 05:36:04 -07:00
Nanda kumar 7ae9543a57 HDFS-13391. Ozone: Make dependency of internal sub-module scope as provided in maven. Contributed by Nanda kumar. 2018-04-26 05:36:04 -07:00
Anu Engineer 04e1287771 HDFS-13298. Ozone: Make ozone/hdsl/cblock modules turned off by default. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Anu Engineer 4e61bc431e HDFS-13258. Ozone: restructure Hdsl/Ozone code to separated maven subprojects.
Contributed by Elek Marton, Mukul Kumar Singh, Xiaoyu Yao, Ajay Kumar, Anu Engineer, Lokesh Jain, Nanda Kumar.
2018-04-26 05:36:04 -07:00
Anu Engineer 5e7164c614 HDFS-13137. Ozone: Ozonefs read fails because ChunkGroupInputStream#read does not iterate through all the blocks in the key.
Contributed by Mukul Kumar Singh.
2018-04-26 05:36:04 -07:00
Anu Engineer 61651dcf5c HDFS-13108. Ozone: OzoneFileSystem: Simplified url schema for Ozone File System. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00
Mukul Kumar Singh cd807c9890 HDFS-13133. Ozone: OzoneFileSystem: Calling delete with non-existing path shouldn't be logged on ERROR level. Contributed by Elek, Marton. 2018-04-26 05:36:04 -07:00