Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,334 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

552 Commits

Author SHA1 Message Date
Steve Loughran c177823ebe HADOOP-13387 users always get told off for using S3 —even when not using it. Contributed by stevel. 2016-07-20 14:01:17 +01:00
Steve Loughran 0f1e02a298 HADOOP-13212 Provide an option to set the socket buffers in S3AFileSystem (Rajesh Balamohan) 2016-07-20 13:42:19 +01:00
Jitendra Pandey a363277be5 HADOOP-13368. DFSOpsCountStatistics$OpType#fromSymbol and s3a.Statistic#fromSymbol should be O(1) operation. Contributed by Mingliang Liu. 2016-07-15 14:32:20 -07:00
Jian He 23eb3c7ceb YARN-5270. Solve miscellaneous issues caused by YARN-4844. Contributed by Wangda Tan 2016-07-11 22:38:35 -07:00
Andrew Wang 4e7e48cdc1 HDFS-10300. TestDistCpSystem should share MiniDFSCluster. Contributed by John Zhuge. 
(cherry picked from commit f292624bd8)
(cherry picked from commit 38b1eafdbc)
 2016-07-11 18:06:45 -07:00
Steve Loughran 310c08454f HADOOP-13139 Branch-2: S3a to use thread pool that blocks clients. Contributed by Pieter Reuse. 2016-07-11 17:21:14 +01:00
Akira Ajisaka a8cfcce542 Revert "HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux." 
This reverts commit 1d175f0f3f.
 2016-07-09 21:01:07 -07:00
Jitendra Pandey 9c7ce389c7 HADOOP-13283. Support reset operation for new global storage statistics and per FS storage stats. Contributed by Mingliang Liu. 2016-07-07 15:04:47 -07:00
Jitendra Pandey bc7fd76a1f HADOOP-13305. Define common statistics names across schemes. Contributed by Mingliang Liu. 2016-07-01 15:37:32 -07:00
Steve Loughran f4f0c5074d HADOOP-13239 Deprecate s3:// in branch-2 2016-06-29 13:16:10 +01:00
Yongjun Zhang 4a57c03f30 HDFS-10396. Using -diff option with DistCp may get "Comparison method violates its general contract" exception. Contributed by Yongjun Zhang. 
(cherry picked from commit 8113855b3a)
 2016-06-28 23:34:44 -07:00
Jing Zhao d17265f467 HDFS-10556. DistCpOptions should be validated automatically. Contributed by Mingliang Liu. 
(cherry picked from commit 856bc4e28b)
 2016-06-22 15:22:54 -07:00
Steve Loughran 1eb45180a6 HADOOP-13203 S3A: Support fadvise "random" mode for high performance readPositioned() reads. Contributed by Rajesh Balamohan and stevel. 2016-06-22 15:41:52 +01:00
Chris Nauroth e30b7112ed HADOOP-13287. TestS3ACredentials#testInstantiateFromURL fails if AWS secret key contains +. Contributed by Chris Nauroth. 
(cherry picked from commit b2c596cdda)
(cherry picked from commit e6ebe9ab4e)
 2016-06-21 11:34:46 -07:00
Jitendra Pandey 2f95d22b17 HADOOP-13291. Probing stats in DFSOpsCountStatistics/S3AStorageStatistics should be correctly implemented. Contributed by Mingliang Liu. 2016-06-20 16:31:54 -07:00
Ravi Prakash 605443c913 HADOOP-3733. "s3x:" URLs break when Secret Key contains a slash, even if encoded. Contributed by Steve Loughran. 2016-06-16 20:45:59 +01:00
Chris Nauroth eddefbd01b HADOOP-13241. document s3a better. Contributed by Steve Loughran. 
(cherry picked from commit 127d2c7281)
(cherry picked from commit e4b63ed7b4)
 2016-06-16 11:19:00 -07:00
Wangda Tan d838c6443d YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:21:41 -07:00
Steve Loughran faf9a9f4e0 HADOOP-12537 S3A to support Amazon STS temporary credentials. Contributed by Sean Mackrory. 2016-06-09 20:58:30 +01:00
Steve Loughran 7e09601a90 HADOOP-13237: s3a initialization against public bucket fails if caller lacks any credentials. Contributed by Chris Nauroth 2016-06-09 16:36:51 +01:00
Wangda Tan 19e578870d YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. (wangda) 2016-06-07 12:41:50 -07:00
Steve Loughran f9478c95bd HADOOP-12807 S3AFileSystem should read AWS credentials from environment variables. Contributed by Tobin Baker. 2016-06-06 23:40:49 +02:00
Chris Nauroth b8216c10d8 HADOOP-13171. Add StorageStatistics to S3A; instrument some more operations. Contributed by Steve Loughran. 2016-06-03 08:56:36 -07:00
Chris Nauroth 3ab0e8d500 HADOOP-13131. Add tests to verify that S3A supports SSE-S3 encryption. Contributed by Steve Loughran. 
(cherry picked from commit 16b1cc7af9)
(cherry picked from commit e9942c5b34)
 2016-06-01 14:49:41 -07:00
Steve Loughran 809e359d2d HADOOP-13162. Consider reducing number of getFileStatus calls in S3AFileSystem.mkdirs. (Rajesh Balamohan via stevel) 2016-06-01 14:17:18 +01:00
Wangda Tan 2f3e1d965d Revert "YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan." 
This reverts commit 457884737f.
 2016-05-31 22:16:53 -07:00
Varun Vasudev 457884737f YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:57:56 +05:30
Yongjun Zhang 3fc4d4a369 HADOOP-13199. Add doc for distcp -filters. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit cfb860dee7)
 2016-05-26 23:34:32 -07:00
Steve Loughran 5930e813c5 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-21 19:10:16 +01:00
Steve Loughran f2aef95bd3 HADOOP-13130. s3a failures can surface as RTEs, not IOEs. (Steve Loughran) 2016-05-21 14:20:19 +01:00
Steve Loughran ce17c9fd51 Revert "HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth." 
This reverts commit edb912c571.
 2016-05-20 14:00:26 +01:00
Steve Loughran f32b37ec89 HADOOP-12723 S3A: Add ability to plug in any AWSCredentialsProvider. Contributed by Steven Wong. 2016-05-20 13:51:56 +01:00
Steve Loughran edb912c571 HADOOP-13145 In DistCp, prevent unnecessary getFileStatus call when not preserving metadata. Contributed by Chris Nauroth. 2016-05-20 12:21:35 +01:00
Chris Nauroth dbff8f2932 HADOOP-13183. S3A proxy tests fail after httpclient/httpcore upgrade. Contributed by Steve Loughran. 
(cherry picked from commit 93258459fa)
(cherry picked from commit 60e5397a9b)
 2016-05-19 22:00:42 -07:00
Jing Zhao 8755905e30 HDFS-10397. Distcp should ignore -delete option if -diff option is provided instead of exiting. Contributed by Mingliang Liu. 
(cherry picked from commit 03788d3015)
(cherry picked from commit dd7f5a3b88)
 2016-05-17 15:55:31 -07:00
Steve Loughran 42d9beaa1d HADOOP-13158 S3AFileSystem#toString might throw NullPointerException due to null cannedACL. (Chris Nauroth via stevel) 2016-05-17 13:18:51 +01:00
Steve Loughran b44ecfff12 HADOOP-13163 Reuse pre-computed filestatus in Distcp-CopyMapper (Rajesh Balamohan via stevel) 2016-05-17 13:02:22 +01:00
Chris Nauroth 28a2f28e42 HADOOP-13148. TestDistCpViewFs to include IOExceptions in test error reports. Contributed by Steve Loughran. 
(cherry picked from commit b9685e85d5)
(cherry picked from commit bb41547fb4)
 2016-05-16 11:53:33 -07:00
Steve Loughran a84850b4ca HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:15 +01:00
Andrew Wang b5fd6dd4a8 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7)
 2016-05-12 23:58:26 -07:00
Steve Loughran 8d74a58042 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:18 +01:00
Steve Loughran 9cbb942465 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:10 +01:00
Steve Loughran af5e5d6d47 HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:56:48 +01:00
Chris Nauroth bbd463b3b2 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
(cherry picked from commit ba48834431)
 2016-05-10 15:14:42 -07:00
Steve Loughran 11b170adc8 HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:23 +01:00
Jason Lowe b452ecf4a0 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit ee86cef2fe)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-05-05 14:40:35 +00:00
Masatake Iwasaki 50d06acb09 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 
(cherry picked from commit ef0870ad03)
 2016-04-29 20:40:47 +09:00
Yongjun Zhang 7ff4fb89ca HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd12)
(cherry picked from commit 7955b6f66e)
 2016-04-26 16:30:54 -07:00
Jing Zhao b98b057f45 HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328)
(cherry picked from commit 864baf23aa)
 2016-04-26 16:30:46 -07:00
Akira Ajisaka 6cb501e431 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d9)
(cherry picked from commit 58642d2244)
 2016-04-25 22:34:37 +09:00
Steve Loughran 5df89f9a8a HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:24:24 +01:00
Yongjun Zhang a1beb5fe1e HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0cea)
 2016-04-21 12:29:49 -07:00
Steve Loughran 4d90014560 HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:26 +01:00
Jason Lowe 3cf7856878 Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 9d3c51eb5a.
 2016-04-11 15:57:56 +00:00
Steve Loughran f6351519b1 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:39 +01:00
Chris Nauroth 1b5b85e560 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d)
(cherry picked from commit 26a23eff82)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java
 2016-04-08 13:44:27 -07:00
Jason Lowe 9d3c51eb5a YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit 1cbcd4a491)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
 2016-04-05 13:47:05 +00:00
Steve Loughran 4d7162e934 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:00:51 +01:00
Steve Loughran ab67b50543 HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:14 +01:00
Harsh J 719c131c4d HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe29)
(cherry picked from commit 7c5b55d4e5)
 2016-04-01 14:15:43 +01:00
Akira Ajisaka 68cd8112e7 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 7b2cf29f4f)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-29 18:24:22 +09:00
Akira Ajisaka 0286ac2576 Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 416ea158c2.
 2016-03-28 16:06:10 +09:00
Akira Ajisaka 416ea158c2 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc936)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java

(cherry picked from commit 0dc1453626)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
 2016-03-25 15:33:01 +09:00
Chris Nauroth f3e4dac87d HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 
(cherry picked from commit 57a0e6543c)
 2016-03-23 12:17:39 -07:00
Chris Nauroth cb063d5699 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f85)
(cherry picked from commit 2e60387401)
 2016-03-04 22:58:04 -08:00
Masatake Iwasaki 9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a6)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfb)
 2016-03-04 14:13:30 +09:00
Masatake Iwasaki 55f7ceb0db HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4)

 Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-03-03 18:58:59 +09:00
Chris Nauroth f563c53535 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b)
(cherry picked from commit bce33b5179)
 2016-03-01 13:19:03 -08:00
Chris Nauroth f1236c5d7c HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415)
(cherry picked from commit e402371b6a)
 2016-02-29 22:00:41 -08:00
Masatake Iwasaki c7824e8e9e HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b14)
 2016-02-29 18:26:09 +09:00
Chris Nauroth 60611c9c75 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b7)
(cherry picked from commit 2e761dabd8)
 2016-02-28 11:23:09 -08:00
cnauroth 8ad907f06a HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f691)
(cherry picked from commit 9ea19a861f)
 2016-02-23 21:38:01 -08:00
cnauroth 606547dbd2 HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1)
(cherry picked from commit a5511debdb)
 2016-02-22 17:30:53 -08:00
cnauroth 746d669847 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b2)
(cherry picked from commit 3f14eed580)
 2016-02-22 13:39:57 -08:00
Yongjun Zhang 5a5f9ca1f0 HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a)
 2016-02-19 10:29:46 -08:00
cnauroth 49602ef580 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025)
(cherry picked from commit b4c4f30fa4)
 2016-02-18 14:07:15 -08:00
cnauroth 26ce37a1a6 HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java

(cherry picked from commit 6731bb468d)
 2016-02-17 12:20:49 -08:00
cnauroth b846401307 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 91a96eaa53)
(cherry picked from commit ac5a793590)
 2016-02-12 15:50:28 -08:00
Steve Loughran f4bcc76df6 HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:17:30 +00:00
Yongjun Zhang f028e3d7bc HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6)
 2016-01-29 12:13:18 -08:00
cnauroth ced22ca78f MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 
(cherry picked from commit 41da9a0773)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt

(cherry picked from commit bd01fdb64d)
 2016-01-28 14:47:03 -08:00
Akira Ajisaka d228bf1de3 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 
(cherry picked from commit 736eb17a79)
(cherry picked from commit 4edd7bd820)
 2016-01-25 13:48:46 +09:00
Akira Ajisaka 2097016ab1 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 
(cherry picked from commit 0bae506c22)
(cherry picked from commit 6368d102b9)
 2016-01-22 21:09:23 +09:00
Akira Ajisaka b85b8d90d1 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 
(cherry picked from commit 7f215ffdd5)
 2016-01-22 18:38:08 +09:00
Ravi Prakash a359dc87d4 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 
(cherry picked from commit 1acc509b45)
(cherry picked from commit 2cbb8bbd72)
 2016-01-19 14:01:51 -08:00
Wangda Tan 5dc2e78c97 HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 
(cherry picked from commit 89d1fd5dac)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2016-01-19 21:31:35 +08:00
Wangda Tan 2a2ea38a9b YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit a44ce3f14f)
 2016-01-19 09:33:52 +08:00
Wangda Tan bacb38aaf3 Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe5728563.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 150f5ae034)
 2016-01-19 09:33:47 +08:00
cnauroth ffc0d98886 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 
(cherry picked from commit 8bc93db2e7)
(cherry picked from commit 62d6166211)
 2016-01-18 09:13:31 -08:00
Wangda Tan 1cb0ad00bb YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85e)
(cherry picked from commit 6dab78c71a)
 2016-01-18 17:08:42 +08:00
Ravi Prakash 5468c40fe7 HADOOP-12689. S3 filesystem operations stopped working correctly 
(cherry picked from commit 2d16f40dab)
(cherry picked from commit 6330683778)
 2016-01-15 15:19:15 -08:00
Yongjun Zhang 4c8131b1bf HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb7)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-15 10:12:15 -08:00
Lei Xu 39f07295fb HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 
(cherry picked from commit 126705f67e)
 2016-01-12 12:21:00 -08:00
Xiaoyu Yao a9cbfd717c HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e13455)
 2016-01-11 17:16:43 -08:00
Zhe Zhang 87d614c70b HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0
 2016-01-11 09:57:01 -08:00
cnauroth e26f702866 HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 
(cherry picked from commit 0e76f1fcea)
(cherry picked from commit 813841e10b)
 2016-01-09 22:18:30 -08:00
cnauroth f0f8e3ee7c HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f0fa6d869b)
(cherry picked from commit debd13387d)
 2016-01-08 20:19:18 -08:00
cnauroth 506a517f9f HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade. 
(cherry picked from commit 978bbdfeb2)
(cherry picked from commit 4753676a34)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2016-01-06 11:17:01 -08:00
Wangda Tan 8c3ad2b734 YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) 
(cherry picked from commit 6eefae1b33)
 2015-12-30 15:42:27 -08:00
Xiaoyu Yao c16e126d16 HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth. 
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
 2015-12-30 11:16:06 -08:00
Wangda Tan 9ed9eb265f YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 
(cherry picked from commit 79c41b1d83)
(cherry picked from commit 9f4a6ec663)
 2015-12-18 16:34:45 -08:00
Robert Kanter 7b4bf23b2a MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter) 
(cherry picked from commit 6d84cc16b3)
 2015-11-25 17:13:00 -08:00
Akira Ajisaka 4257991257 HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang. 
(cherry picked from commit b53f20135599cc7825b6213794e2f999bc43e2ad)
 2015-11-26 01:21:32 +09:00
Karthik Kambatla 49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 
(cherry picked from commit 52948bb20b)
 2015-11-24 13:52:29 +05:30
Haohui Mai 1d175f0f3f HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux. 2015-11-22 18:12:53 -08:00
Haohui Mai ac0694eb6e HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena. 2015-11-22 17:25:29 -08:00
Haohui Mai f504d95e87 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:57:23 -08:00
Karthik Kambatla d3f9508096 YARN-4367. SLS webapp doesn't load. (kasha) 2015-11-18 13:45:00 -08:00
Akira Ajisaka c5c663c05d HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 
(cherry picked from commit f94d892704)
 2015-11-13 21:53:23 +09:00
Lei Xu 47941858af Revert "HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei)" 
This reverts commit 01ae30796d.
 2015-11-06 09:52:21 -08:00
Lei Xu 01ae30796d HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei) 
(cherry picked from commit bff7c90a56)
 2015-11-05 18:36:55 -08:00
cnauroth 81f7e8af22 HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown. Contributed by Gaurav Kanade. 
(cherry picked from commit 0fb1867fd6)
 2015-11-04 10:29:01 -08:00
cnauroth c1d7b26e94 HADOOP-12508. delete fails with exception when lease is held on blob. Contributed by Gaurav Kanade. 
(cherry picked from commit 9e7dcab185)
 2015-11-02 10:21:46 -08:00
cnauroth c260d7ec78 HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. Contributed by Dushyanth. 
(cherry picked from commit 28790692624177d89fb1e4f59e2f83a659fc3089)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2015-11-02 09:39:27 -08:00
Kihwal Lee dae9f0ef53 Addendum to MAPREDUCE-6451 
(cherry picked from commit b24fe06483)
 2015-10-30 16:06:50 -05:00
Kihwal Lee 52819fe4ef MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 
(cherry picked from commit 2868ca0328)
 2015-10-30 14:57:57 -05:00
cnauroth c7f87dc2d0 HADOOP-12519. hadoop-azure tests should avoid creating a metrics configuration fiale in the module root directory. Contributed by Chris Nauroth. 
(cherry picked from commit 2d10cb8e00)
 2015-10-28 09:44:21 -07:00
cnauroth 0377795e06 HADOOP-11685. StorageException complaining " no lease ID" during HBase distributed log splitting. Contributed by Duo Xu. 
(cherry picked from commit 1f7ecb0c84)
 2015-10-27 22:56:33 -07:00
cnauroth a23f79b92c HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Contributed by Chris Nauroth. 
(cherry picked from commit 73822de7c3)
 2015-10-27 22:49:04 -07:00
Varun Vasudev c5bf1cb7af YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt. Contributed by Bibin A Chundatt. 
(cherry picked from commit c28e16b40c)
 2015-10-27 20:56:45 +05:30
cnauroth b023539c19 HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race Condition Scenarios. Contributed by Gaurav Kanade. 
(cherry picked from commit cb282d5b89)
 2015-10-22 14:30:09 -07:00
cnauroth 10398daadf HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. Contributed by Gaurav Kanade. 
(cherry picked from commit 47641fcbc9)
 2015-10-22 12:21:44 -07:00
Robert Kanter 492142097b MAPREDUCE-6495. Docs for archive-logs tool (rkanter) 
(cherry picked from commit 0c4af0f998)
 2015-10-20 17:34:56 -07:00
Haohui Mai ee04add696 HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:29 -07:00
Haohui Mai f6b0fcdc75 Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu." 
This reverts commit a412edf5c4.
 2015-10-10 03:25:16 -07:00
Haohui Mai a412edf5c4 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:13 -07:00
Robert Kanter e5bf4594cc MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME (rkanter) 
(cherry picked from commit 1bca1bb517)
 2015-10-06 16:50:18 -07:00
cnauroth 535762fadb HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and writes. Contributed by Dushyanth. 
(cherry picked from commit 5f6edb30c2)
 2015-10-05 20:12:33 -07:00
Akira Ajisaka c6f30e5913 HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). Contributed by Rahul Palamuttam and Kanaka Kumar Avvaru. 
(cherry picked from commit ecbfd68974)
 2015-10-01 18:35:19 +02:00
Robert Kanter 46ef5aa8eb MAPREDUCE-6494. Permission issue when running archive-logs tool as different users (rkanter) 
(cherry picked from commit 5db371f52f)
 2015-09-30 17:34:03 -07:00
Kihwal Lee 95b8f1f3a6 HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 
(cherry picked from commit 3abbdc929b)
 2015-09-28 17:33:20 -05:00
Harsh J 03e370e306 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit 66dad854c0)
 2015-09-28 13:14:20 +05:30
cnauroth 85c0cb0075 HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 7fe521b1dd)
 2015-09-25 22:34:10 -07:00
Robert Kanter d8a5d2b2fc MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) 
(cherry picked from commit 8fec882cfaac047af1892615fe32d8c3ca02e7f9)
 2015-09-25 15:00:07 -07:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
 2015-09-24 10:30:04 -07:00
Harsh J 38420754f5 MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit de1322aa76007832698af682a3b76042116043b9)
 2015-09-22 21:38:31 +05:30
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab)
 2015-09-22 12:54:44 +09:00
Steve Loughran 988749e6ea HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:06 +01:00
Karthik Kambatla 999aa22509 MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) 2015-09-09 17:49:23 -07:00
Tsuyoshi Ozawa 7f1abf0b27 HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter error message. Contributed by Wei Yan. 
(cherry picked from commit 355eaaa33d)
 2015-09-03 22:49:12 +09:00
Lei Xu de42fce390 HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu) 2015-08-29 09:58:57 -07:00
Rohith Sharma K S 6dc732f2f7 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 
(cherry picked from commit feaf034994)
 2015-08-24 11:30:46 +05:30
Yongjun Zhang b085c5ef6f HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 
(cherry picked from commit 0bc15cb6e6)
 2015-08-20 08:10:34 -07:00
cnauroth b83d475252 HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Contributed by Chris Nauroth. 
(cherry picked from commit 3da0bedaec)
 2015-08-05 14:56:54 -07:00
Lei Xu f7b0f292e7 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 
(cherry picked from commit d5403747b5)
 2015-08-04 18:56:07 -07:00
cnauroth bb080b32d4 HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 
(cherry picked from commit efa97243ec)
 2015-07-22 11:17:00 -07:00
Vinayakumar B 03d68b5575 HADOOP-12017. Hadoop archives command should use configurable replication factor when closing (Contributed by Bibin A Chundatt) 
(cherry picked from commit 94c6a4aa85)
 2015-07-22 10:27:03 +05:30
Wangda Tan 3dd113fa7c YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit c39ca541f4)
 2015-07-21 09:57:53 -07:00
Vinayakumar B 8ef3821356 HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. (Contributed by Vinayakumar B) 
(cherry picked from commit 87f29c6b8a)
 2015-07-21 13:13:31 +05:30
Steve Loughran f3296a1984 HADOOP-12235 hadoop-openstack junit & mockito dependencies should be "provided". (Ted Yu via stevel) 2015-07-20 11:22:22 +01:00