Jian He
23eb3c7ceb
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:38:35 -07:00
Andrew Wang
4e7e48cdc1
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
...
(cherry picked from commit f292624bd8
)
(cherry picked from commit 38b1eafdbc
)
2016-07-11 18:06:45 -07:00
Steve Loughran
310c08454f
HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse.
2016-07-11 17:21:14 +01:00
Akira Ajisaka
a8cfcce542
Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux."
...
This reverts commit 1d175f0f3f
.
2016-07-09 21:01:07 -07:00
Jitendra Pandey
9c7ce389c7
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 15:04:47 -07:00
Jitendra Pandey
bc7fd76a1f
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:37:32 -07:00
Steve Loughran
f4f0c5074d
HADOOP-13239 Deprecate s3:// in branch-2
2016-06-29 13:16:10 +01:00
Yongjun Zhang
4a57c03f30
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
...
(cherry picked from commit 8113855b3a
)
2016-06-28 23:34:44 -07:00
Jing Zhao
d17265f467
HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu.
...
(cherry picked from commit 856bc4e28b
)
2016-06-22 15:22:54 -07:00
Steve Loughran
1eb45180a6
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:41:52 +01:00
Chris Nauroth
e30b7112ed
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
...
(cherry picked from commit b2c596cdda
)
(cherry picked from commit e6ebe9ab4e
)
2016-06-21 11:34:46 -07:00
Jitendra Pandey
2f95d22b17
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:31:54 -07:00
Ravi Prakash
605443c913
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
2016-06-16 20:45:59 +01:00
Chris Nauroth
eddefbd01b
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
...
(cherry picked from commit 127d2c7281
)
(cherry picked from commit e4b63ed7b4
)
2016-06-16 11:19:00 -07:00
Wangda Tan
d838c6443d
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:21:41 -07:00
Steve Loughran
faf9a9f4e0
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 20:58:30 +01:00
Steve Loughran
7e09601a90
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 16:36:51 +01:00
Wangda Tan
19e578870d
YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. (wangda)
2016-06-07 12:41:50 -07:00
Steve Loughran
f9478c95bd
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:40:49 +02:00
Chris Nauroth
b8216c10d8
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:56:36 -07:00
Chris Nauroth
3ab0e8d500
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
...
(cherry picked from commit 16b1cc7af9
)
(cherry picked from commit e9942c5b34
)
2016-06-01 14:49:41 -07:00
Steve Loughran
809e359d2d
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:17:18 +01:00
Wangda Tan
2f3e1d965d
Revert "YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan."
...
This reverts commit 457884737f
.
2016-05-31 22:16:53 -07:00
Varun Vasudev
457884737f
YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 20:57:56 +05:30
Yongjun Zhang
3fc4d4a369
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
...
(cherry picked from commit cfb860dee7
)
2016-05-26 23:34:32 -07:00
Steve Loughran
5930e813c5
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-21 19:10:16 +01:00
Steve Loughran
f2aef95bd3
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 14:20:19 +01:00
Steve Loughran
ce17c9fd51
Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth."
...
This reverts commit edb912c571
.
2016-05-20 14:00:26 +01:00
Steve Loughran
f32b37ec89
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:51:56 +01:00
Steve Loughran
edb912c571
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:35 +01:00
Chris Nauroth
dbff8f2932
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
...
(cherry picked from commit 93258459fa
)
(cherry picked from commit 60e5397a9b
)
2016-05-19 22:00:42 -07:00
Jing Zhao
8755905e30
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
...
(cherry picked from commit 03788d3015
)
(cherry picked from commit dd7f5a3b88
)
2016-05-17 15:55:31 -07:00
Steve Loughran
42d9beaa1d
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:18:51 +01:00
Steve Loughran
b44ecfff12
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:02:22 +01:00
Chris Nauroth
28a2f28e42
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
...
(cherry picked from commit b9685e85d5
)
(cherry picked from commit bb41547fb4
)
2016-05-16 11:53:33 -07:00
Steve Loughran
a84850b4ca
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:46:15 +01:00
Andrew Wang
b5fd6dd4a8
HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki.
...
(cherry picked from commit fa440a39b7
)
2016-05-12 23:58:26 -07:00
Steve Loughran
8d74a58042
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:23:18 +01:00
Steve Loughran
9cbb942465
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:36:10 +01:00
Steve Loughran
af5e5d6d47
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:56:48 +01:00
Chris Nauroth
bbd463b3b2
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
(cherry picked from commit ba48834431
)
2016-05-10 15:14:42 -07:00
Steve Loughran
11b170adc8
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:40:23 +01:00
Jason Lowe
b452ecf4a0
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
...
(cherry picked from commit ee86cef2fe
)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
2016-05-05 14:40:35 +00:00
Masatake Iwasaki
50d06acb09
HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA.
...
(cherry picked from commit ef0870ad03
)
2016-04-29 20:40:47 +09:00
Yongjun Zhang
7ff4fb89ca
HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang)
...
(cherry picked from commit 959a28dd12
)
(cherry picked from commit 7955b6f66e
)
2016-04-26 16:30:54 -07:00
Jing Zhao
b98b057f45
HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi.
...
(cherry picked from commit 404f57f328
)
(cherry picked from commit 864baf23aa
)
2016-04-26 16:30:46 -07:00
Akira Ajisaka
6cb501e431
HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi.
...
(cherry picked from commit 02c51c27d9
)
(cherry picked from commit 58642d2244
)
2016-04-25 22:34:37 +09:00
Steve Loughran
5df89f9a8a
HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel)
2016-04-22 11:24:24 +01:00
Yongjun Zhang
a1beb5fe1e
HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang)
...
(cherry picked from commit a749ba0cea
)
2016-04-21 12:29:49 -07:00
Steve Loughran
4d90014560
HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel)
2016-04-14 12:45:26 +01:00