591ca69823
HADOOP-16278. With S3A Filesystem, Long Running services End up Doing lot of GC and eventually die.
...
Contributed by Rajat Khandelwal
2019-05-09 21:13:29 +01:00
3418bbbb59
HADOOP-16269. ABFS: add listFileStatus with StartFrom.
...
Author: Da Zhou
2019-05-08 17:20:46 +01:00
7a3188d054
HADOOP-16282. Avoid FileStream to improve performance. Contributed by Ayush Saxena.
2019-05-02 12:58:42 -07:00
0af4011580
HADOOP-16221. S3Guard: add option to fail operation on metadata write failure.
2019-04-30 11:53:26 +01:00
1cef194a28
HADOOP-16242. ABFS: add bufferpool to AbfsOutputStream.
...
Contributed by Da Zhou.
2019-04-29 13:27:28 +01:00
e1c5ddf2aa
HADOOP-16252. Add prefix to dynamo tables in tests.
...
Contributed by Ben Roling.
2019-04-24 14:55:58 +01:00
a5ceed26f3
HADOOP-16253. Update AssertJ to 3.12.2.
2019-04-16 12:53:52 +09:00
4b4fef2f0e
HADOOP-16227. Upgrade checkstyle to 8.19
2019-04-15 10:46:10 +09:00
2382f63fc0
HADOOP-14747. S3AInputStream to implement CanUnbuffer.
...
Author: Sahil Takiar <stakiar@cloudera.com>
2019-04-12 18:12:02 -07:00
bbdbc7a9a1
HADOOP-14544. DistCp documentation for command line options is misaligned. Contributed by Masatake Iwasaki.
2019-04-12 11:52:18 +09:00
cf4efcab3b
HADOOP-16118. S3Guard to support on-demand DDB tables.
...
This is the first step for on-demand operations: things recognize when they are using on-demand tables,
as do the tests.
Contributed by Steve Loughran.
2019-04-11 17:12:12 -07:00
358e928622
YARN-999. In case of long running tasks, reduce node resource should balloon out resource quickly by calling preemption API and suspending running task. Addendum. Contributed by Inigo Goiri.
2019-04-09 14:18:05 -07:00
215ffc792e
HADOOP-16197 S3AUtils.translateException to map CredentialInitializationException to AccessDeniedException
...
Contributed by Steve Loughran.
Change-Id: Ie98ca5210bf0009f297edbcacf1fc6dfe5ea70cd.
2019-04-04 21:14:18 +01:00
366186d999
HADOOP-16233. S3AFileStatus to declare that isEncrypted() is always true ( #685 )
...
This is needed to fix up some confusion about caching of job.addCache() handling of S3A paths; all parent dirs -the files are downloaded by the NM without using the DTs of the user submitting the job. This means that when you submit jobs to an EC2 cluster with lower IAM permissions than the user, cached resources don't get downloaded and the job doesn't start.
Production code changes:
* S3AFileStatus Adds "true" to the superclass's encrypted flag during construction.
Tests
* Base AbstractContractOpenTest can control whether zero byte files created in tests are encrypted. Not done via an XML attribute, just a subclass point. Thoughts?
* Verify that the filecache considers paths to not have the permissions which trigger reduce-privilege downloads
* And extend ITestDelegatedMRJob to test a completely different bucket (open street map), to verify that cached resources do get their tokens picked up
Docs:
* Advise FS developers to say all files are encrypted. It's otherwise harmless and it'll stop other people seeing impossible to debug error messages on app launch.
Contributed by Steve Loughran.
Change-Id: Ifaae4c9d735ccc5eafeebd2584b65daf2d4e5da3
2019-04-03 21:23:40 +01:00
8b6deebb1d
HADOOP-16232. Fix errors in the checkstyle configration xmls. Contributed by Wanqiang Ji.
2019-04-03 19:35:02 +09:00
df578c07ec
HADOOP-16195 MarshalledCredentials toString
...
Change-Id: I4f1bdd2be0d5760c5501dce6edb6122499108b53
2019-03-28 17:01:57 +00:00
b5db238383
HADOOP-15999. S3Guard: Better support for out-of-band operations.
...
Author: Gabor Bota
2019-03-28 15:59:25 +00:00
cfb0186903
HADOOP-16186. S3Guard: NPE in DynamoDBMetadataStore.lambda$listChildren.
...
Author: Gabor Bota
2019-03-28 15:49:56 +00:00
ce4bafdf44
HADOOP-16037. DistCp: Document usage of Sync (-diff option) in detail.
...
Contributed by Siyao Meng
2019-03-26 18:42:54 +00:00
ae2eb2dd42
HADOOP-16201: S3AFileSystem#innerMkdirs builds needless lists ( #636 )
2019-03-22 11:42:00 +00:00
faba3591d3
HADOOP-16147. Allow CopyListing sequence file keys and values to be more easily customized.
...
Author: Andrew Olson
2019-03-22 10:35:30 +00:00
9f1c017f44
HADOOP-16058. S3A tests to include Terasort.
...
Contributed by Steve Loughran.
This includes
- HADOOP-15890. Some S3A committer tests don't match ITest* pattern; don't run in maven
- MAPREDUCE-7090. BigMapOutput example doesn't work with paths off cluster fs
- MAPREDUCE-7091. Terasort on S3A to switch to new committers
- MAPREDUCE-7092. MR examples to work better against cloud stores
2019-03-21 11:15:37 +00:00
399563fec6
HDFS-14176. Replace incorrect use of system property user.name. Contributed by Dinesh Chitlangia.
2019-03-20 11:20:45 -07:00
568d3ab8b6
HADOOP-16191. AliyunOSS: improvements for copyFile/copyDirectory and logging. Contributed by wujinhu.
2019-03-19 09:55:32 +08:00
c0427c84dd
HADOOP-16124. Extend documentation in testing.md about S3 endpoint constants.
...
Contributed by Adam Antal.
2019-03-18 19:13:13 +00:00
1edf1914ac
HADOOP-16182. Update abfs storage back-end with "close" flag when application is done writing to a file.
...
Contributed by Vishwajeet Dusane.
2019-03-18 13:18:08 +00:00
6fa229891e
HADOOP-15625. S3A input stream to use etags/version number to detect changed source files.
...
Author: Ben Roling <ben.roling@gmail.com>
Initial patch from Brahma Reddy Battula.
2019-03-13 20:37:11 +00:00
ddede7ae6f
HADOOP-16174. Disable wildfly logs to the console.
...
Follow-on to HADOOP-15851.
Author: Denes Gerencser <dgerencser@cloudera.com>
2019-03-09 18:45:17 +00:00
bb8ad096e7
HADOOP-16157. [Clean-up] Remove NULL check before instanceof in AzureNativeFileSystemStore
...
(Contributed by Shweta Yakkali via Daniel Templeton)
Change-Id: I6269ae66378e46eed440a76f847ae1af1fa95450
2019-03-09 09:55:56 -08:00
0cbe9ad8c2
HADOOP-16109. Parquet reading S3AFileSystem causes EOF
...
Nobody gets seek right. No matter how many times they think they have.
Reproducible test from: Dave Christianson
Fixed seek() logic: Steve Loughran
2019-03-09 16:00:34 +00:00
e0260417ad
HADOOP-16169. ABFS: Bug fix for getPathProperties.
...
Author: Da Zhou <da.zhou@microsoft.com>
2019-03-08 13:53:08 +00:00
05df151d09
HADOOP-16163. NPE in setup/teardown of ITestAbfsDelegationTokens.
...
Contributed by Da Zhou.
Signed-off-by: Steve Loughran <stevel@apache.org>
2019-03-05 14:02:34 +00:00
65f60e56b0
HADOOP-16068. ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store.
...
Contributed by Steve Loughran.
2019-02-28 14:22:32 +00:00
52b2eab575
HADOOP-16093. Move DurationInfo from hadoop-aws to hadoop-common org.apache.hadoop.util.
...
Contributed by Abhishek Modi
2019-02-26 17:10:41 +00:00
3988e75ca3
HADOOP-16136. ABFS: Should only transform username to short name
...
Contributed by Da Zhou.
2019-02-26 15:37:24 +00:00
a868f59d52
HADOOP-16105. WASB in secure mode does not set connectingUsingSAS.
...
Contributed by Steve Loughran.
2019-02-21 14:08:46 +00:00
aa3ad36605
HADOOP-16104. Wasb tests to downgrade to skip when test a/c is namespace enabled. Contributed by Masatake Iwasaki.
2019-02-20 22:00:57 +09:00
779dae4de7
YARN-9309. Improve graph text in SLS to avoid overlapping. Contributed by Bilwa S T.
2019-02-20 00:36:57 +05:30
1e0ae6ed15
HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found.
...
Contributed by Adam Antal.
(Revised patch applied after stevel committed the wrong one; that has been reverted)
2019-02-19 11:33:02 +00:00
920a89627d
Revert "HADOOP-15843. s3guard bucket-info command to not print a stack trace on bucket-not-found."
...
This reverts commit c4a00d1ad3
2019-02-18 14:57:22 +00:00
0f2b65c3da
HADOOP-16116. Fix Spelling Mistakes - DECOMISSIONED. Contributed by BELUGA BEHR.
2019-02-18 15:21:46 +09:00
134ae8fc80
YARN-9293. Optimize MockAMLauncher event handling. Contributed by Bibin A Chundatt.
2019-02-14 22:56:52 +05:30
6c999fe4b0
HADOOP-16098. Fix javadoc warnings in hadoop-aws. Contributed by Masatake Iwasaki.
2019-02-12 06:07:47 +09:00
1f1655028e
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-07 21:58:21 +00:00
668817a6ce
Revert "HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth."
...
(accidentally mixed in two patches)
This reverts commit fa8cd1bf28
2019-02-07 21:57:22 +00:00
546c5d70ef
HADOOP-16032. Distcp It should clear sub directory ACL before applying new ACL on.
2019-02-07 21:48:07 +00:00
de804e53b9
HADOOP-15281. Distcp to add no-rename copy option.
...
Contributed by Andrew Olson.
2019-02-07 10:07:22 +00:00
fa8cd1bf28
HADOOP-15954. ABFS: Enable owner and group conversion for MSI and login user using OAuth.
...
Contributed by Da Zhou and Junhua Gu.
2019-02-05 19:23:15 +00:00
ba9efe06fa
HADOOP-16074. WASB: Update container not found error code.
...
Contributed by Da Zhou.
2019-02-05 14:39:45 +00:00
f365957c63
HADOOP-15229. Add FileSystem builder-based openFile() API to match createFile();
...
S3A to implement S3 Select through this API.
The new openFile() API is asynchronous, and implemented across FileSystem and FileContext.
The MapReduce V2 inputs are moved to this API, and you can actually set must/may
options to pass in.
This is more useful for setting things like s3a seek policy than for S3 select,
as the existing input format/record readers can't handle S3 select output where
the stream is shorter than the file length, and splitting plain text is suboptimal.
Future work is needed there.
In the meantime, any/all filesystem connectors are now free to add their own filesystem-specific
configuration parameters which can be set in jobs and used to set filesystem input stream
options (seek policy, retry, encryption secrets, etc).
Contributed by Steve Loughran
2019-02-05 11:51:02 +00:00
Rajat Khandelwal
Da Zhou
Giovanni Matteo Fumarola
Ben Roling
Takanobu Asanuma
Akira Ajisaka
Sahil Takiar
Masatake Iwasaki
Steve Loughran
Gabor Bota
Siyao Meng
Lokesh Jain
Andrew Olson
Ajay Kumar
Weiwei Yang
Adam Antal
Vishwajeet Dusane
Denes Gerencser
Shweta Yakkali
Abhishek Modi
bibinchundatt
Ranith Sardar