Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,167 Commits 326 Branches 371 Tags
Commit Graph

423 Commits

Author SHA1 Message Date
Steve Loughran
f2aef95bd3 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:20:19 +01:00
Steve Loughran
ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c57123ce4cba109a41a7dc21a45da7d27c.
 2016-05-20 14:00:26 +01:00
Steve Loughran
f32b37ec89 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:51:56 +01:00
Steve Loughran
edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Chris Nauroth
dbff8f2932 HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459faf56bc84121ba99d20eaef95273329e)
(cherry picked from commit 60e5397a9bb6a1d8ca6a09c2893108af77dc96c3)
 2016-05-19 22:00:42 -07:00
Jing Zhao
8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015c962eac1a35fa5df39356e8b84731c)
(cherry picked from commit dd7f5a3b88cc37945bfc96ca91ed5f39d44ee394)
 2016-05-17 15:55:31 -07:00
Steve Loughran
42d9beaa1d HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:18:51 +01:00
Steve Loughran
b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth
28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d59e69e5dd64802fa9025dad1b315be5)
(cherry picked from commit bb41547fb42b9dd2b80ec2a7286db6d35fd251b5)
 2016-05-16 11:53:33 -07:00
Steve Loughran
a84850b4ca HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:15 +01:00
Andrew Wang
b5fd6dd4a8 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7883183b9cf43c056908ca7bf68e6a1)
 2016-05-12 23:58:26 -07:00
Steve Loughran
8d74a58042 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:18 +01:00
Steve Loughran
9cbb942465 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:10 +01:00
Steve Loughran
af5e5d6d47 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:56:48 +01:00
Chris Nauroth
bbd463b3b2 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
(cherry picked from commit ba4883443141354fa457887fb2501226f6a0fa1f)
 2016-05-10 15:14:42 -07:00
Steve Loughran
11b170adc8 HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:23 +01:00
Jason Lowe
b452ecf4a0 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit ee86cef2fee992fb773e87c895000ffc2ee788b0)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-05-05 14:40:35 +00:00
Masatake Iwasaki
50d06acb09 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 
(cherry picked from commit ef0870ad038a1b72ea14ce550e34139b81eb901b)
 2016-04-29 20:40:47 +09:00
Yongjun Zhang
7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd1216dfac78d05b438828e8503108d963)
(cherry picked from commit 7955b6f66ee670a57ccdeb7b6a9e6af25c0d67c7)
 2016-04-26 16:30:54 -07:00
Jing Zhao
b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328b00a42ec8b952ad08cd7a80207c7f2)
(cherry picked from commit 864baf23aa3df201f81b721852221c6475388295)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka
6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d92e74be7089083555e85da0e73f7fda)
(cherry picked from commit 58642d2244c3062672cc8a8b8a7cdbcd8929cd14)
 2016-04-25 22:34:37 +09:00
Steve Loughran
5df89f9a8a HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:24:24 +01:00
Yongjun Zhang
a1beb5fe1e HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0ceaa843aa83146b6bea19e031c8dc3296)
 2016-04-21 12:29:49 -07:00
Steve Loughran
4d90014560 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:26 +01:00
Jason Lowe
3cf7856878 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 9d3c51eb5a701ae4ba296e2fd3fba154fa0097c0.
 2016-04-11 15:57:56 +00:00
Steve Loughran
f6351519b1 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:39 +01:00
Chris Nauroth
1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d8bacbca0d87ccf0790772e39d16138)
(cherry picked from commit 26a23eff828a3f4fcbd4362761c329657603277d)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Jason Lowe
9d3c51eb5a YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit 1cbcd4a491e6a57d466c2897335614dc6770b475)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-04-05 13:47:05 +00:00
Steve Loughran
4d7162e934 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:00:51 +01:00
Steve Loughran
ab67b50543 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:14 +01:00
Harsh J
719c131c4d HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe2981748cd0befc5490d8118d139908f9)
(cherry picked from commit 7c5b55d4e5f4317abed0259909b89a32297836f8)
 2016-04-01 14:15:43 +01:00
Akira Ajisaka
68cd8112e7 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c20087009bd76a0bb2183a479f787f0)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 7b2cf29f4f673a955910b335d46687989cde478b)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-29 18:24:22 +09:00
Akira Ajisaka
0286ac2576 Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 416ea158c2aa197b4a997ed420842154a1294b1d.
 2016-03-28 16:06:10 +09:00
Akira Ajisaka
416ea158c2 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc9365851f5b02d967d13c8c0cbca850a86)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 0dc145362642f6e5f9def8f50775b5c16599f6e6)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-25 15:33:01 +09:00
Chris Nauroth
f3e4dac87d HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 
(cherry picked from commit 57a0e6543cd00c5b8a9c0ee5a9613596a7e8b78e)
 2016-03-23 12:17:39 -07:00
Chris Nauroth
cb063d5699 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f854b74ede9668e35db314b0a93be81b2)
(cherry picked from commit 2e60387401e1dff3fa36e32b267f2855c2e4ba49)
 2016-03-04 22:58:04 -08:00
Masatake Iwasaki
9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a61252f253aa2bcdb9cad70f7876d231)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfbb654be655fbca42f34bba5a05c2f6d)
 2016-03-04 14:13:30 +09:00
Masatake Iwasaki
55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Chris Nauroth
f563c53535 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b1f94c9f53022baf0209fc1a7aaf7c2)
(cherry picked from commit bce33b51790b90f0655e36df3a0394f5691f8d6e)
 2016-03-01 13:19:03 -08:00
Chris Nauroth
f1236c5d7c HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415f1fab085159b9eb3b43214d100b6a8)
(cherry picked from commit e402371b6a0854a89e7bd573b04e43fed5e3e9cf)
 2016-02-29 22:00:41 -08:00
Masatake Iwasaki
c7824e8e9e HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b146135d86433e9c7bfa17e294b7928f2)
 2016-02-29 18:26:09 +09:00
Chris Nauroth
60611c9c75 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b750eafa135a37fa275325d0398255fb)
(cherry picked from commit 2e761dabd8b0458bb324a32b28c4a8daf31a69b5)
 2016-02-28 11:23:09 -08:00
cnauroth
8ad907f06a HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f6916f12e9d6b97072badd1b176be443bd)
(cherry picked from commit 9ea19a861f10ca6281eaed2ee5032af21eaa53b3)
 2016-02-23 21:38:01 -08:00
cnauroth
606547dbd2 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1163ab4a1ce081a426e5190d1b8aff4)
(cherry picked from commit a5511debdb63770b58e406926343a1a189225d78)
 2016-02-22 17:30:53 -08:00
cnauroth
746d669847 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b208635e8f9a14c375d4101141aefa4a)
(cherry picked from commit 3f14eed58055d30b7c2dcb9240355920ff48af7e)
 2016-02-22 13:39:57 -08:00
Yongjun Zhang
5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a8d6a18b4b7365c49d09635d24e579a)
 2016-02-19 10:29:46 -08:00
cnauroth
49602ef580 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025616d154decd97578de3327ec6f9109)
(cherry picked from commit b4c4f30fa4fdb3bd9adcf669932b8633ab11fd25)
 2016-02-18 14:07:15 -08:00
cnauroth
26ce37a1a6 HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c02cef38924d04136407489fd9457d9)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

(cherry picked from commit 6731bb468dab743e0546e8e0fbbefaf46a1729c2)
 2016-02-17 12:20:49 -08:00
cnauroth
b846401307 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 91a96eaa534dbb27e81b6c24bbb8138200a80a83)
(cherry picked from commit ac5a7935901255e5fc56b88574e72f875942b906)
 2016-02-12 15:50:28 -08:00
Steve Loughran
f4bcc76df6 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:17:30 +00:00