Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,188 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

425 Commits

Author SHA1 Message Date
Yongjun Zhang 3fc4d4a369 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee7)
 2016-05-26 23:34:32 -07:00
Steve Loughran 5930e813c5 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-21 19:10:16 +01:00
Steve Loughran f2aef95bd3 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:20:19 +01:00
Steve Loughran ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c571.
 2016-05-20 14:00:26 +01:00
Steve Loughran f32b37ec89 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:51:56 +01:00
Steve Loughran edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Chris Nauroth dbff8f2932 HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459fa)
(cherry picked from commit 60e5397a9b)
 2016-05-19 22:00:42 -07:00
Jing Zhao 8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015)
(cherry picked from commit dd7f5a3b88)
 2016-05-17 15:55:31 -07:00
Steve Loughran 42d9beaa1d HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:18:51 +01:00
Steve Loughran b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth 28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d5)
(cherry picked from commit bb41547fb4)
 2016-05-16 11:53:33 -07:00
Steve Loughran a84850b4ca HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:15 +01:00
Andrew Wang b5fd6dd4a8 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7)
 2016-05-12 23:58:26 -07:00
Steve Loughran 8d74a58042 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:18 +01:00
Steve Loughran 9cbb942465 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:10 +01:00
Steve Loughran af5e5d6d47 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:56:48 +01:00
Chris Nauroth bbd463b3b2 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
(cherry picked from commit ba48834431)
 2016-05-10 15:14:42 -07:00
Steve Loughran 11b170adc8 HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:23 +01:00
Jason Lowe b452ecf4a0 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit ee86cef2fe)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-05-05 14:40:35 +00:00
Masatake Iwasaki 50d06acb09 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 
(cherry picked from commit ef0870ad03)
 2016-04-29 20:40:47 +09:00
Yongjun Zhang 7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd12)
(cherry picked from commit 7955b6f66e)
 2016-04-26 16:30:54 -07:00
Jing Zhao b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328)
(cherry picked from commit 864baf23aa)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka 6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d9)
(cherry picked from commit 58642d2244)
 2016-04-25 22:34:37 +09:00
Steve Loughran 5df89f9a8a HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:24:24 +01:00
Yongjun Zhang a1beb5fe1e HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0cea)
 2016-04-21 12:29:49 -07:00
Steve Loughran 4d90014560 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:26 +01:00
Jason Lowe 3cf7856878 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 9d3c51eb5a.
 2016-04-11 15:57:56 +00:00
Steve Loughran f6351519b1 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:39 +01:00
Chris Nauroth 1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d)
(cherry picked from commit 26a23eff82)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Jason Lowe 9d3c51eb5a YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit 1cbcd4a491)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-04-05 13:47:05 +00:00
Steve Loughran 4d7162e934 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:00:51 +01:00
Steve Loughran ab67b50543 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:14 +01:00
Harsh J 719c131c4d HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe29)
(cherry picked from commit 7c5b55d4e5)
 2016-04-01 14:15:43 +01:00
Akira Ajisaka 68cd8112e7 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 7b2cf29f4f)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-29 18:24:22 +09:00
Akira Ajisaka 0286ac2576 Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 416ea158c2.
 2016-03-28 16:06:10 +09:00
Akira Ajisaka 416ea158c2 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc936)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 0dc1453626)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-25 15:33:01 +09:00
Chris Nauroth f3e4dac87d HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 
(cherry picked from commit 57a0e6543c)
 2016-03-23 12:17:39 -07:00
Chris Nauroth cb063d5699 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f85)
(cherry picked from commit 2e60387401)
 2016-03-04 22:58:04 -08:00
Masatake Iwasaki 9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a6)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfb)
 2016-03-04 14:13:30 +09:00
Masatake Iwasaki 55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Chris Nauroth f563c53535 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b)
(cherry picked from commit bce33b5179)
 2016-03-01 13:19:03 -08:00
Chris Nauroth f1236c5d7c HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415)
(cherry picked from commit e402371b6a)
 2016-02-29 22:00:41 -08:00
Masatake Iwasaki c7824e8e9e HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b14)
 2016-02-29 18:26:09 +09:00
Chris Nauroth 60611c9c75 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b7)
(cherry picked from commit 2e761dabd8)
 2016-02-28 11:23:09 -08:00
cnauroth 8ad907f06a HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f691)
(cherry picked from commit 9ea19a861f)
 2016-02-23 21:38:01 -08:00
cnauroth 606547dbd2 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1)
(cherry picked from commit a5511debdb)
 2016-02-22 17:30:53 -08:00
cnauroth 746d669847 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b2)
(cherry picked from commit 3f14eed580)
 2016-02-22 13:39:57 -08:00
Yongjun Zhang 5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a)
 2016-02-19 10:29:46 -08:00
cnauroth 49602ef580 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025)
(cherry picked from commit b4c4f30fa4)
 2016-02-18 14:07:15 -08:00
cnauroth 26ce37a1a6 HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

(cherry picked from commit 6731bb468d)
 2016-02-17 12:20:49 -08:00