Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
10,958 Commits 326 Branches 371 Tags
Commit Graph

393 Commits

Author SHA1 Message Date
Harsh J
719c131c4d HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe2981748cd0befc5490d8118d139908f9)
(cherry picked from commit 7c5b55d4e5f4317abed0259909b89a32297836f8)
 2016-04-01 14:15:43 +01:00
Akira Ajisaka
68cd8112e7 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c20087009bd76a0bb2183a479f787f0)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 7b2cf29f4f673a955910b335d46687989cde478b)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-29 18:24:22 +09:00
Akira Ajisaka
0286ac2576 Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 416ea158c2aa197b4a997ed420842154a1294b1d.
 2016-03-28 16:06:10 +09:00
Akira Ajisaka
416ea158c2 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc9365851f5b02d967d13c8c0cbca850a86)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 0dc145362642f6e5f9def8f50775b5c16599f6e6)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-25 15:33:01 +09:00
Chris Nauroth
f3e4dac87d HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 
(cherry picked from commit 57a0e6543cd00c5b8a9c0ee5a9613596a7e8b78e)
 2016-03-23 12:17:39 -07:00
Chris Nauroth
cb063d5699 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f854b74ede9668e35db314b0a93be81b2)
(cherry picked from commit 2e60387401e1dff3fa36e32b267f2855c2e4ba49)
 2016-03-04 22:58:04 -08:00
Masatake Iwasaki
9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a61252f253aa2bcdb9cad70f7876d231)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfbb654be655fbca42f34bba5a05c2f6d)
 2016-03-04 14:13:30 +09:00
Masatake Iwasaki
55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4ab729d588a9576fb7eb90239c6e383)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Chris Nauroth
f563c53535 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b1f94c9f53022baf0209fc1a7aaf7c2)
(cherry picked from commit bce33b51790b90f0655e36df3a0394f5691f8d6e)
 2016-03-01 13:19:03 -08:00
Chris Nauroth
f1236c5d7c HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415f1fab085159b9eb3b43214d100b6a8)
(cherry picked from commit e402371b6a0854a89e7bd573b04e43fed5e3e9cf)
 2016-02-29 22:00:41 -08:00
Masatake Iwasaki
c7824e8e9e HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b146135d86433e9c7bfa17e294b7928f2)
 2016-02-29 18:26:09 +09:00
Chris Nauroth
60611c9c75 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b750eafa135a37fa275325d0398255fb)
(cherry picked from commit 2e761dabd8b0458bb324a32b28c4a8daf31a69b5)
 2016-02-28 11:23:09 -08:00
cnauroth
8ad907f06a HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f6916f12e9d6b97072badd1b176be443bd)
(cherry picked from commit 9ea19a861f10ca6281eaed2ee5032af21eaa53b3)
 2016-02-23 21:38:01 -08:00
cnauroth
606547dbd2 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1163ab4a1ce081a426e5190d1b8aff4)
(cherry picked from commit a5511debdb63770b58e406926343a1a189225d78)
 2016-02-22 17:30:53 -08:00
cnauroth
746d669847 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b208635e8f9a14c375d4101141aefa4a)
(cherry picked from commit 3f14eed58055d30b7c2dcb9240355920ff48af7e)
 2016-02-22 13:39:57 -08:00
Yongjun Zhang
5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a8d6a18b4b7365c49d09635d24e579a)
 2016-02-19 10:29:46 -08:00
cnauroth
49602ef580 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025616d154decd97578de3327ec6f9109)
(cherry picked from commit b4c4f30fa4fdb3bd9adcf669932b8633ab11fd25)
 2016-02-18 14:07:15 -08:00
cnauroth
26ce37a1a6 HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c02cef38924d04136407489fd9457d9)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

(cherry picked from commit 6731bb468dab743e0546e8e0fbbefaf46a1729c2)
 2016-02-17 12:20:49 -08:00
cnauroth
b846401307 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 91a96eaa534dbb27e81b6c24bbb8138200a80a83)
(cherry picked from commit ac5a7935901255e5fc56b88574e72f875942b906)
 2016-02-12 15:50:28 -08:00
Steve Loughran
f4bcc76df6 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:17:30 +00:00
Yongjun Zhang
f028e3d7bc HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6246ddc66218eb01009c44b0236eaaa)
 2016-01-29 12:13:18 -08:00
cnauroth
ced22ca78f MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 
(cherry picked from commit 41da9a07738c6d019e2467f139750b7e2f50a914)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt

(cherry picked from commit bd01fdb64d17b6146fc3588040aba9ed53ca57f9)
 2016-01-28 14:47:03 -08:00
Akira Ajisaka
d228bf1de3 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 
(cherry picked from commit 736eb17a796a1c1ad5f4db2c6a64f6752db7bec3)
(cherry picked from commit 4edd7bd82094dd8c1474a912532e6373cedf79c9)
 2016-01-25 13:48:46 +09:00
Akira Ajisaka
2097016ab1 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 
(cherry picked from commit 0bae506c221f9736d08a6fba00ce5a862ab43cce)
(cherry picked from commit 6368d102b993e2bda08272ed368b9be912c0abbe)
 2016-01-22 21:09:23 +09:00
Akira Ajisaka
b85b8d90d1 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 
(cherry picked from commit 7f215ffdd5219fc585ec484073b317a940895a60)
 2016-01-22 18:38:08 +09:00
Ravi Prakash
a359dc87d4 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 
(cherry picked from commit 1acc509b45d58c0eb7e83ea1ba13169410be0dbe)
(cherry picked from commit 2cbb8bbd728aab6d438c3051c970e4b4475b8e68)
 2016-01-19 14:01:51 -08:00
Wangda Tan
5dc2e78c97 HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 
(cherry picked from commit 89d1fd5dac4bccf42d82686e146b02eb60d14736)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2016-01-19 21:31:35 +08:00
Wangda Tan
2a2ea38a9b YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit a44ce3f14fd940601f984fbf7980aa6fdc8f23b7)
 2016-01-19 09:33:52 +08:00
Wangda Tan
bacb38aaf3 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe57285635e8058c34aa40a103845b49ca7d6ff.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 150f5ae0343e872ee8bef39c57008c1389f0ba9e)
 2016-01-19 09:33:47 +08:00
cnauroth
ffc0d98886 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 
(cherry picked from commit 8bc93db2e7c64830b6a662f28c8917a9eef4e7c9)
(cherry picked from commit 62d616621134cc5e68f4d5fd32f49ea4d731417c)
 2016-01-18 09:13:31 -08:00
Wangda Tan
1cb0ad00bb YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85eb34c8125cfb7d26d07cdfac12b3579)
(cherry picked from commit 6dab78c71a739c626e8b78c3f1cf126387253a30)
 2016-01-18 17:08:42 +08:00
Ravi Prakash
5468c40fe7 HADOOP-12689. S3 filesystem operations stopped working correctly 
(cherry picked from commit 2d16f40dab291a29b3fc005221b12fd587615d4e)
(cherry picked from commit 6330683778d684d539ae756226f62728776cc1d7)
 2016-01-15 15:19:15 -08:00
Yongjun Zhang
4c8131b1bf HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb707801071db3cc22bfcd14f87be443a)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-15 10:12:15 -08:00
Lei Xu
39f07295fb HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 
(cherry picked from commit 126705f67eaa6d866f7572fbddf133c5d7552353)
 2016-01-12 12:21:00 -08:00
Xiaoyu Yao
a9cbfd717c HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e134555010ec28da296bcfef4ba2613a5c6c)
 2016-01-11 17:16:43 -08:00
Zhe Zhang
87d614c70b HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
 2016-01-11 09:57:01 -08:00
cnauroth
e26f702866 HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 
(cherry picked from commit 0e76f1fceaaaeb66bdb4818f43b9a55fc092bf79)
(cherry picked from commit 813841e10bf9ac30481852bb5aeacc98921bb41c)
 2016-01-09 22:18:30 -08:00
cnauroth
f0f8e3ee7c HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f0fa6d869b9abb5a900ea1c9eb4eb19ec9831dc4)
(cherry picked from commit debd13387daad868de466586370846c32040b7ad)
 2016-01-08 20:19:18 -08:00
cnauroth
506a517f9f HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade. 
(cherry picked from commit 978bbdfeb2d12efd6e750da6a14849e072fb814b)
(cherry picked from commit 4753676a34c4a3de29f9d20281de09533b73ce53)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2016-01-06 11:17:01 -08:00
Wangda Tan
8c3ad2b734 YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) 
(cherry picked from commit 6eefae1b33928486e7a8572e14eaf00b83f2acfc)
 2015-12-30 15:42:27 -08:00
Xiaoyu Yao
c16e126d16 HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth. 
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
 2015-12-30 11:16:06 -08:00
Wangda Tan
9ed9eb265f YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 
(cherry picked from commit 79c41b1d83e981ae74cb8b58ffcf7907b7612ad4)
(cherry picked from commit 9f4a6ec6630f3776859f8d677532370095a58d0c)
 2015-12-18 16:34:45 -08:00
Robert Kanter
7b4bf23b2a MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter) 
(cherry picked from commit 6d84cc16b3e0685fef01d0e3526b0f7556ceff51)
 2015-11-25 17:13:00 -08:00
Akira Ajisaka
4257991257 HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang. 
(cherry picked from commit b53f20135599cc7825b6213794e2f999bc43e2ad)
 2015-11-26 01:21:32 +09:00
Karthik Kambatla
49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 
(cherry picked from commit 52948bb20bd1446164df1d3920c46c96dad750ae)
 2015-11-24 13:52:29 +05:30
Haohui Mai
1d175f0f3f HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux. 2015-11-22 18:12:53 -08:00
Haohui Mai
ac0694eb6e HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena. 2015-11-22 17:25:29 -08:00
Haohui Mai
f504d95e87 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:57:23 -08:00
Karthik Kambatla
d3f9508096 YARN-4367. SLS webapp doesn't load. (kasha) 2015-11-18 13:45:00 -08:00
Akira Ajisaka
c5c663c05d HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 
(cherry picked from commit f94d89270464ea8e0d19e26e425835cd6a5dc5de)
 2015-11-13 21:53:23 +09:00