Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,835 Commits 326 Branches 371 Tags
Commit Graph

558 Commits

Author SHA1 Message Date
Jing Zhao
864baf23aa HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328b00a42ec8b952ad08cd7a80207c7f2)
 2016-04-14 10:35:43 -07:00
Steve Loughran
e124c3a2ae HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:10 +01:00
Jason Lowe
f1a370ce8b Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 814ceeb4893ad9ed903ebcfd30fe3158ec1c71df.
 2016-04-11 15:56:29 +00:00
Steve Loughran
9acfde3a03 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:55 +01:00
Chris Nauroth
26a23eff82 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d8bacbca0d87ccf0790772e39d16138)
 2016-04-08 13:37:46 -07:00
Steve Loughran
1533403a9a HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:09:47 +01:00
Jason Lowe
814ceeb489 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit 1cbcd4a491e6a57d466c2897335614dc6770b475)
 2016-04-05 13:41:18 +00:00
Steve Loughran
caaadf9f3e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:01:37 +01:00
Steve Loughran
20e808d83e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:51 +01:00
Harsh J
7c5b55d4e5 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe2981748cd0befc5490d8118d139908f9)
 2016-04-01 14:35:58 +05:30
Akira Ajisaka
7b2cf29f4f MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c20087009bd76a0bb2183a479f787f0)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
 2016-03-29 18:20:01 +09:00
Akira Ajisaka
2b1f066d2d Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 0dc145362642f6e5f9def8f50775b5c16599f6e6.
 2016-03-28 16:05:55 +09:00
Akira Ajisaka
0dc1453626 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc9365851f5b02d967d13c8c0cbca850a86)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
 2016-03-25 15:23:23 +09:00
Chris Nauroth
57a0e6543c HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 2016-03-23 12:17:27 -07:00
Chris Nauroth
2e60387401 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f854b74ede9668e35db314b0a93be81b2)
 2016-03-04 22:57:56 -08:00
Masatake Iwasaki
722182fdfb HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a61252f253aa2bcdb9cad70f7876d231)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
 2016-03-04 14:12:44 +09:00
Masatake Iwasaki
8095c612a3 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383)
 2016-03-03 18:57:41 +09:00
Chris Nauroth
bce33b5179 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b1f94c9f53022baf0209fc1a7aaf7c2)
 2016-03-01 13:18:57 -08:00
Chris Nauroth
e402371b6a HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415f1fab085159b9eb3b43214d100b6a8)
 2016-02-29 22:00:27 -08:00
Masatake Iwasaki
32c7791a6c HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b146135d86433e9c7bfa17e294b7928f2)
 2016-02-29 18:23:33 +09:00
Chris Nauroth
2e761dabd8 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b750eafa135a37fa275325d0398255fb)
 2016-02-28 11:23:02 -08:00
cnauroth
9ea19a861f HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f6916f12e9d6b97072badd1b176be443bd)
 2016-02-23 21:37:56 -08:00
cnauroth
a5511debdb HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1163ab4a1ce081a426e5190d1b8aff4)
 2016-02-22 17:30:32 -08:00
cnauroth
3f14eed580 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b208635e8f9a14c375d4101141aefa4a)
 2016-02-22 13:39:45 -08:00
Yongjun Zhang
b144e0f15d HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a8d6a18b4b7365c49d09635d24e579a)
 2016-02-19 10:29:27 -08:00
cnauroth
b4c4f30fa4 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025616d154decd97578de3327ec6f9109)
 2016-02-18 14:07:09 -08:00
cnauroth
6731bb468d HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c02cef38924d04136407489fd9457d9)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
 2016-02-17 12:20:41 -08:00
cnauroth
ac5a793590 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 91a96eaa534dbb27e81b6c24bbb8138200a80a83)
 2016-02-12 15:50:19 -08:00
Steve Loughran
60474a769e HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:20:24 +00:00
Yongjun Zhang
2fb711bde9 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6246ddc66218eb01009c44b0236eaaa)
 2016-01-29 12:13:03 -08:00
cnauroth
bd01fdb64d MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 
(cherry picked from commit 41da9a07738c6d019e2467f139750b7e2f50a914)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-01-28 14:46:42 -08:00
Xuan
69f3fa71a2 YARN-4612. Fix rumen and scheduler load simulator handle killed tasks 
properly. Contributed by Ming Ma.

(cherry picked from commit 4efdf3a979c361348612f817a3253be6d0de58f7)
 2016-01-26 18:18:03 -08:00
Akira Ajisaka
4edd7bd820 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 
(cherry picked from commit 736eb17a796a1c1ad5f4db2c6a64f6752db7bec3)
 2016-01-25 13:48:28 +09:00
Akira Ajisaka
6368d102b9 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 
(cherry picked from commit 0bae506c221f9736d08a6fba00ce5a862ab43cce)
 2016-01-22 21:09:09 +09:00
Akira Ajisaka
7f215ffdd5 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 2016-01-22 18:37:17 +09:00
Xuan
66b030f608 YARN-4611. Fix scheduler load simulator to support multi-layer network 
location. Contributed by Ming Ma.

(cherry picked from commit 1708a4cd2377c56e2cb5738720b7eaf10baf13c8)
 2016-01-20 15:43:02 -08:00
Ravi Prakash
2cbb8bbd72 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 
(cherry picked from commit 1acc509b45d58c0eb7e83ea1ba13169410be0dbe)
 2016-01-19 14:00:54 -08:00
Wangda Tan
23deeb4230 HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 
(cherry picked from commit 89d1fd5dac4bccf42d82686e146b02eb60d14736)
 2016-01-19 21:28:51 +08:00
Wangda Tan
704b37d0e1 YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit a44ce3f14fd940601f984fbf7980aa6fdc8f23b7)
 2016-01-19 09:31:13 +08:00
Wangda Tan
1625281ccb Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe57285635e8058c34aa40a103845b49ca7d6ff.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 150f5ae0343e872ee8bef39c57008c1389f0ba9e)
 2016-01-19 09:31:07 +08:00
cnauroth
62d6166211 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 
(cherry picked from commit 8bc93db2e7c64830b6a662f28c8917a9eef4e7c9)
 2016-01-18 09:09:05 -08:00
Wangda Tan
6dab78c71a YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85eb34c8125cfb7d26d07cdfac12b3579)
 2016-01-18 11:15:59 +08:00
Yongjun Zhang
ee012c1f6f HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb707801071db3cc22bfcd14f87be443a)
 2016-01-15 10:10:28 -08:00
Colin Patrick Mccabe
cea1c71087 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe) 
(cherry picked from commit b1ed28fa77cb2fab80c54f9dfeb5d8b7139eca34)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
 2016-01-14 11:13:13 -08:00
Colin Patrick Mccabe
2baeaa20a3 HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8315582c4ff2951144b096c23a64e753f397572d)
 2016-01-13 16:33:02 -08:00
Lei Xu
63b8be149a HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 2016-01-12 12:21:29 -08:00
Xiaoyu Yao
521f60c094 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e134555010ec28da296bcfef4ba2613a5c6c)
 2016-01-11 17:12:43 -08:00
Zhe Zhang
17dfa04be6 HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-11 09:53:20 -08:00
cnauroth
813841e10b HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 
(cherry picked from commit 0e76f1fceaaaeb66bdb4818f43b9a55fc092bf79)
 2016-01-09 22:18:22 -08:00
cnauroth
debd13387d HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f0fa6d869b9abb5a900ea1c9eb4eb19ec9831dc4)
 2016-01-08 20:19:11 -08:00