Jitendra Pandey
328866905c
HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu.
2016-07-15 14:29:59 -07:00
Jian He
e94e6be772
YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan
2016-07-11 22:37:52 -07:00
Andrew Wang
38b1eafdbc
HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge.
...
(cherry picked from commit f292624bd8
)
2016-07-11 18:06:35 -07:00
Steve Loughran
89121745c0
HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse.
2016-07-11 17:21:00 +01:00
Akira Ajisaka
f4b63a391b
Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux."
...
This reverts commit 1d175f0f3f
.
2016-07-09 21:00:13 -07:00
Jitendra Pandey
63b64a20c2
HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu.
2016-07-07 15:00:34 -07:00
Jitendra Pandey
06b89f0d14
HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu.
2016-07-01 15:35:50 -07:00
Steve Loughran
70a45e3122
HADOOP-13239 Deprecate s3:// in branch-2
2016-06-29 13:16:26 +01:00
Yongjun Zhang
8a121dc3b8
HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang.
...
(cherry picked from commit 8113855b3a
)
2016-06-28 23:29:48 -07:00
Jing Zhao
856bc4e28b
HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu.
2016-06-22 15:15:47 -07:00
Steve Loughran
fb9167fdef
HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel.
2016-06-22 15:42:30 +01:00
Chris Nauroth
e6ebe9ab4e
HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth.
...
(cherry picked from commit b2c596cdda
)
2016-06-21 11:32:11 -07:00
Jitendra Pandey
8f93bbcb11
HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu.
2016-06-20 16:28:55 -07:00
Chris Nauroth
e4b63ed7b4
HADOOP-13241. document s3a better. Contributed by Steve Loughran.
...
(cherry picked from commit 127d2c7281
)
2016-06-16 11:18:40 -07:00
Ravi Prakash
b216dbfb55
HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran.
...
(cherry picked from commit 4aefe119a0
)
2016-06-16 11:14:38 -07:00
Wangda Tan
863bfa4d6a
YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda)
2016-06-14 15:12:00 -07:00
Chris Nauroth
1925942294
Revert "HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane."
...
This reverts commit a8f03ef7ea
.
2016-06-09 15:17:18 -07:00
Chris Nauroth
a8f03ef7ea
HADOOP-12666. Support Microsoft Azure Data Lake - as a file system in Hadoop. Contributed by Vishwajeet Dusane.
...
(cherry picked from commit 9581fb715c
)
Conflicts:
hadoop-tools/hadoop-tools-dist/pom.xml
hadoop-tools/pom.xml
2016-06-09 14:49:05 -07:00
Steve Loughran
6350e4b1fb
HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory.
2016-06-09 20:59:54 +01:00
Steve Loughran
df29f77846
HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth
2016-06-09 16:36:27 +01:00
Steve Loughran
0cbbc2f0db
HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker.
2016-06-06 23:41:54 +02:00
Chris Nauroth
043a0c2e6c
HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran.
2016-06-03 08:56:07 -07:00
Chris Nauroth
e9942c5b34
HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran.
...
(cherry picked from commit 16b1cc7af9
)
2016-06-01 14:49:31 -07:00
Steve Loughran
b29bf5d335
HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel)
2016-06-01 14:17:50 +01:00
Varun Vasudev
4f36c3d214
YARN-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan.
2016-05-29 20:55:39 +05:30
Yongjun Zhang
82a88c7e67
HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang)
...
(cherry picked from commit cfb860dee7
)
2016-05-26 23:32:46 -07:00
Steve Loughran
15f997cca6
HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran)
2016-05-21 14:19:56 +01:00
Steve Loughran
eb160561e0
HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong.
2016-05-20 13:52:30 +01:00
Steve Loughran
e29d7fb585
HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth.
2016-05-20 12:21:47 +01:00
Chris Nauroth
60e5397a9b
HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran.
...
(cherry picked from commit 93258459fa
)
2016-05-19 22:00:32 -07:00
Arun Suresh
1b4600abcf
YARN-4412. Create ClusterMonitor to compute ordered list of preferred NMs for OPPORTUNITIC containers. (asuresh)
...
(cherry picked from commit 341888a0aa
)
2016-05-18 22:35:55 -07:00
Jing Zhao
dd7f5a3b88
HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu.
...
(cherry picked from commit 03788d3015
)
2016-05-17 15:53:39 -07:00
Steve Loughran
dd0a8201bc
HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel)
2016-05-17 13:19:08 +01:00
Steve Loughran
dfc062a092
HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel)
2016-05-17 13:01:50 +01:00
Chris Nauroth
bb41547fb4
HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran.
...
(cherry picked from commit b9685e85d5
)
2016-05-16 11:53:25 -07:00
Steve Loughran
f14208d362
HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel
2016-05-13 10:46:37 +01:00
Andrew Wang
f6f20616c6
HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki.
...
(cherry picked from commit fa440a39b7
)
2016-05-12 23:58:24 -07:00
Steve Loughran
1370dfc775
HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel
...
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
2016-05-12 19:23:31 +01:00
Steve Loughran
389dd91dd4
HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run.
2016-05-12 16:36:23 +01:00
Steve Loughran
257b2f5beb
HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel.
2016-05-12 13:57:05 +01:00
Chris Nauroth
ba48834431
HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth.
...
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
2016-05-10 15:14:36 -07:00
Steve Loughran
f47ea0cc3f
HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel)
2016-05-10 21:40:13 +01:00
Jian He
007d6d1d57
YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan
...
(cherry picked from commit 996a210ab0
)
2016-05-09 14:50:00 -07:00
Jason Lowe
ee86cef2fe
YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla
...
(cherry picked from commit d0da13229c
)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
2016-05-05 14:33:01 +00:00
Jing Zhao
d43d8a1bcd
HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu.
...
(cherry picked from commit af942585a1
)
2016-05-04 10:24:06 -07:00
Masatake Iwasaki
ff233dfca0
HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA.
...
(cherry picked from commit ef0870ad03
)
2016-04-29 20:27:45 +09:00
Yongjun Zhang
7955b6f66e
HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang)
...
(cherry picked from commit 959a28dd12
)
2016-04-26 16:27:59 -07:00
Akira Ajisaka
58642d2244
HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi.
...
(cherry picked from commit 02c51c27d9
)
2016-04-25 22:34:18 +09:00
Steve Loughran
edcb2a8410
HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel)
2016-04-22 11:24:50 +01:00
Yongjun Zhang
f98f4151ac
HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang)
...
(cherry picked from commit a749ba0cea
)
2016-04-21 12:23:42 -07:00