Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,541 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

523 Commits

Author SHA1 Message Date
Steve Loughran f14208d362 HADOOP-13113 Enable parallel test execution for hadoop-aws. Chris Nauroth via stevel 2016-05-13 10:46:37 +01:00
Andrew Wang f6f20616c6 HADOOP-12868. Fix hadoop-openstack undeclared and unused dependencies. Contributed by Masatake Iwasaki. 
(cherry picked from commit fa440a39b7)
 2016-05-12 23:58:24 -07:00
Steve Loughran 1370dfc775 HADOOP-13028 add low level counter metrics for S3A; use in read performance tests. contributed by: stevel 
patch includes
HADOOP-12844 Recover when S3A fails on IOException in read()
HADOOP-13058 S3A FS fails during init against a read-only FS if multipart purge
HADOOP-13047 S3a Forward seek in stream length to be configurable
 2016-05-12 19:23:31 +01:00
Steve Loughran 389dd91dd4 HADOOP-13116 Jets3tNativeS3FileSystemContractTest does not run. 2016-05-12 16:36:23 +01:00
Steve Loughran 257b2f5beb HADOOP-13122 Customize User-Agent header sent in HTTP requests by S3A. Chris Nauroth via stevel. 2016-05-12 13:57:05 +01:00
Chris Nauroth ba48834431 HADOOP-12801. Suppress obsolete S3FileSystem tests. Contributed by Chris Nauroth. 
(cherry picked from commit d3dbfa7feee7922ddcd0b2e97898adf4e0c37de3)
 2016-05-10 15:14:36 -07:00
Steve Loughran f47ea0cc3f HADOOP-12982 Document missing S3A and S3 properties. (Wei-Chiu Chuang via stevel) 2016-05-10 21:40:13 +01:00
Jian He 007d6d1d57 YARN-4778. Support specifying resources for task containers in SLS. Contributed by Wangda Tan 
(cherry picked from commit 996a210ab0)
 2016-05-09 14:50:00 -07:00
Jason Lowe ee86cef2fe YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit d0da13229c)

Conflicts:

	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
 2016-05-05 14:33:01 +00:00
Jing Zhao d43d8a1bcd HADOOP-12469. distcp should not ignore the ignoreFailures option. Contributed by Mingliang Liu. 
(cherry picked from commit af942585a1)
 2016-05-04 10:24:06 -07:00
Masatake Iwasaki ff233dfca0 HADOOP-12378. Fix findbugs warnings in hadoop-tools module. Contributed by Akira AJISAKA. 
(cherry picked from commit ef0870ad03)
 2016-04-29 20:27:45 +09:00
Yongjun Zhang 7955b6f66e HDFS-10313. Distcp need to enforce the order of snapshot names passed to -diff. (Lin Yiqun via Yongjun Zhang) 
(cherry picked from commit 959a28dd12)
 2016-04-26 16:27:59 -07:00
Akira Ajisaka 58642d2244 HDFS-10298. Document the usage of distcp -diff option. Contributed by Takashi Ohnishi. 
(cherry picked from commit 02c51c27d9)
 2016-04-25 22:34:18 +09:00
Steve Loughran edcb2a8410 HADOOP-12891. S3AFileSystem should configure Multipart Copy threshold and chunk size. (Andrew Olson via stevel) 2016-04-22 11:24:50 +01:00
Yongjun Zhang f98f4151ac HDFS-9670. DistCp throws NPE when source is root. (John Zhuge via Yongjun Zhang) 
(cherry picked from commit a749ba0cea)
 2016-04-21 12:23:42 -07:00
Jing Zhao 864baf23aa HDFS-10216. Distcp -diff throws exception when handling relative path. Contributed by Takashi Ohnishi. 
(cherry picked from commit 404f57f328)
 2016-04-14 10:35:43 -07:00
Steve Loughran e124c3a2ae HADOOP-12963 Allow using path style addressing for accessing the s3 endpoint. (Stephen Montgomery via stevel) 2016-04-14 12:45:10 +01:00
Jason Lowe f1a370ce8b Revert "YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla" 
This reverts commit 814ceeb489.
 2016-04-11 15:56:29 +00:00
Steve Loughran 9acfde3a03 HADOOP-12444 Support lazy seek in S3AInputStream. Rajesh Balamohan via stevel 2016-04-09 11:24:55 +01:00
Chris Nauroth 26a23eff82 HADOOP-12994. Specify PositionedReadable, add contract tests, fix problems. Contributed by Steve Loughran. 
(cherry picked from commit 843ee8d59d)
 2016-04-08 13:37:46 -07:00
Steve Loughran 1533403a9a HADOOP-12753 S3A JUnit tests failing if using HTTP proxy. (Zoran Rajic via stevel) 2016-04-08 17:09:47 +01:00
Jason Lowe 814ceeb489 YARN-4311. Removing nodes from include and exclude lists will not remove them from decommissioned nodes list. Contributed by Kuhu Shukla 
(cherry picked from commit 1cbcd4a491)
 2016-04-05 13:41:18 +00:00
Steve Loughran caaadf9f3e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. - omitted new S3A subclass 2016-04-04 17:01:37 +01:00
Steve Loughran 20e808d83e HADOOP-12169 ListStatus on empty dir in S3A lists itself instead of returning an empty list. author: Pieter Reuse. 2016-04-03 16:39:51 +01:00
Harsh J 7c5b55d4e5 HADOOP-11687. Ignore x-* and response headers when copying an Amazon S3 object. Contributed by Aaron Peterson and harsh. 
(cherry picked from commit 256c82fe29)
 2016-04-01 14:35:58 +05:30
Akira Ajisaka 7b2cf29f4f MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 0050fa5f1c)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
 2016-03-29 18:20:01 +09:00
Akira Ajisaka 2b1f066d2d Revert "MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote." 
This reverts commit 0dc1453626.
 2016-03-28 16:05:55 +09:00
Akira Ajisaka 0dc1453626 MAPREDUCE-6543. Migrate MR client test cases part 2. Contributed by Dustin Cote. 
(cherry picked from commit 2c268cc936)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/TestMRJobClient.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/partition/TestTotalOrderPartitioner.java
 2016-03-25 15:23:23 +09:00
Chris Nauroth 57a0e6543c HDFS-10199. Unit tests TestCopyFiles, TestDistCh, TestLogalyzer under org.apache.hadoop.tools are failing. Contributed by Tibor Kiss. 2016-03-23 12:17:27 -07:00
Chris Nauroth 2e60387401 HADOOP-12717. NPE when trying to rename a directory in Windows Azure Storage FileSystem. Contributed by Robert Yokota and Gaurav Kanade. 
(cherry picked from commit c50aad0f85)
 2016-03-04 22:57:56 -08:00
Masatake Iwasaki 722182fdfb HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a6)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md
 2016-03-04 14:12:44 +09:00
Masatake Iwasaki 8095c612a3 HDFS-9048. DistCp documentation is out-of-dated (Daisuke Kobayashi via iwasakims) 
(cherry picked from commit 33a412e8a4)
 2016-03-03 18:57:41 +09:00
Chris Nauroth bce33b5179 HADOOP-12853. Change WASB documentation regarding page blob support. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f98dff329b)
 2016-03-01 13:18:57 -08:00
Chris Nauroth e402371b6a HADOOP-12851. S3AFileSystem Uptake of ProviderUtils.excludeIncompatibleCredentialProviders. Contributed by Larry McCay. 
(cherry picked from commit d251e55415)
 2016-02-29 22:00:27 -08:00
Masatake Iwasaki 32c7791a6c HADOOP-12552. Fix undeclared/unused dependency to httpclient (iwasakims) 
(cherry picked from commit 809ebc0b14)
 2016-02-29 18:23:33 +09:00
Chris Nauroth 2e761dabd8 HADOOP-12846. Credential Provider Recursive Dependencies. Contributed by Larry McCay. 
(cherry picked from commit 7634d404b7)
 2016-02-28 11:23:02 -08:00
cnauroth 9ea19a861f HADOOP-12535. Run FileSystem contract tests with hadoop-azure. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 3369a4f691)
 2016-02-23 21:37:56 -08:00
cnauroth a5511debdb HADOOP-12555. WASB to read credentials from a credential provider. Contributed by Larry McCay. 
(cherry picked from commit 27b77751c1)
 2016-02-22 17:30:32 -08:00
cnauroth 3f14eed580 HADOOP-11613. Remove commons-httpclient dependency from hadoop-azure. Contributed by Masatake Iwasaki. 
(cherry picked from commit d4f5fc23b2)
 2016-02-22 13:39:45 -08:00
Yongjun Zhang b144e0f15d HDFS-9764. DistCp doesn't print value for several arguments including -numListstatusThreads. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit ba1c9d484a)
 2016-02-19 10:29:27 -08:00
cnauroth b4c4f30fa4 HADOOP-11031. Design Document for Credential Provider API. Contributed by Larry McCay. 
(cherry picked from commit 8ab7658025)
 2016-02-18 14:07:09 -08:00
cnauroth 6731bb468d HADOOP-12548. Read s3a creds from a Credential Provider. Contributed by Larry McCay. 
(cherry picked from commit 76fab26c5c)

Conflicts:
	hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java
 2016-02-17 12:20:41 -08:00
cnauroth ac5a793590 HADOOP-12780. During WASB atomic rename handle crash when one directory has been renamed but not file under it. Contributed by Madhumita Chakraborty. 
(cherry picked from commit 91a96eaa53)
 2016-02-12 15:50:19 -08:00
Steve Loughran 60474a769e HADOOP-12292. Make use of DeleteObjects optional. (Thomas Demoor via stevel) 2016-02-06 18:20:24 +00:00
Yongjun Zhang 2fb711bde9 HDFS-9638. Improve DistCp Help and documentation. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit eddd823cd6)
 2016-01-29 12:13:03 -08:00
cnauroth bd01fdb64d MAPREDUCE-6563. Streaming documentation contains a stray % character. Contributed by Chris Nauroth. 
(cherry picked from commit 41da9a0773)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-01-28 14:46:42 -08:00
Xuan 69f3fa71a2 YARN-4612. Fix rumen and scheduler load simulator handle killed tasks 
properly. Contributed by Ming Ma.

(cherry picked from commit 4efdf3a979)
 2016-01-26 18:18:03 -08:00
Akira Ajisaka 4edd7bd820 HADOOP-12731. Remove useless boxing/unboxing code. Contributed by Kousuke Saruta. 
(cherry picked from commit 736eb17a79)
 2016-01-25 13:48:28 +09:00
Akira Ajisaka 6368d102b9 HADOOP-12730. Hadoop streaming -mapper and -reducer options are wrongly documented as required. Contributed by Kengo Seki. 
(cherry picked from commit 0bae506c22)
 2016-01-22 21:09:09 +09:00
Akira Ajisaka 7f215ffdd5 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 2016-01-22 18:37:17 +09:00
Xuan 66b030f608 YARN-4611. Fix scheduler load simulator to support multi-layer network 
location. Contributed by Ming Ma.

(cherry picked from commit 1708a4cd23)
 2016-01-20 15:43:02 -08:00
Ravi Prakash 2cbb8bbd72 HADOOP-12696. Add tests for S3FileSystem Contract. Contributed by Matt Paduano 
(cherry picked from commit 1acc509b45)
 2016-01-19 14:00:54 -08:00
Wangda Tan 23deeb4230 HADOOP-12356. Fix computing CPU usage statistics on Windows. (Inigo Goiri via wangda) 
(cherry picked from commit 89d1fd5dac)
 2016-01-19 21:28:51 +08:00
Wangda Tan 704b37d0e1 YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit a44ce3f14f)
 2016-01-19 09:31:13 +08:00
Wangda Tan 1625281ccb Revert "YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda)" 
This reverts commit 3fe5728563.

Conflicts:
	hadoop-yarn-project/CHANGES.txt

(cherry picked from commit 150f5ae034)
 2016-01-19 09:31:07 +08:00
cnauroth 62d6166211 HADOOP-12635. Adding Append API support for WASB. Contributed by Dushyanth. 
(cherry picked from commit 8bc93db2e7)
 2016-01-18 09:09:05 -08:00
Wangda Tan 6dab78c71a YARN-4502. Fix two AM containers get allocated when AM restart. (Vinod Kumar Vavilapalli via wangda) 
(cherry picked from commit 805a9ed85e)
 2016-01-18 11:15:59 +08:00
Yongjun Zhang ee012c1f6f HDFS-9612. DistCp worker threads are not terminated after jobs are done. (Wei-Chiu Chuang via Yongjun Zhang) 
(cherry picked from commit a9c69ebeb7)
 2016-01-15 10:10:28 -08:00
Colin Patrick Mccabe cea1c71087 HADOOP-8887. Use a Maven plugin to build the native code using CMake (cmccabe) 
(cherry picked from commit b1ed28fa77)

Conflicts:
	hadoop-common-project/hadoop-common/pom.xml
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/pom.xml
 2016-01-14 11:13:13 -08:00
Colin Patrick Mccabe 2baeaa20a3 HDFS-9517. Fix missing @Test annotation on TestDistCpUtils.testUnpackAttributes (Wei-Chiu Chuang via cmccabe) 
(cherry picked from commit 8315582c4f)
 2016-01-13 16:33:02 -08:00
Lei Xu 63b8be149a HADOOP-11262. Enable YARN to use S3A. (Pieter Reuse via lei) 2016-01-12 12:21:29 -08:00
Xiaoyu Yao 521f60c094 HDFS-8584. NPE in distcp when ssl configuration file does not exist in class path. Contributed by Surendra Singh Lilhore. 
(cherry picked from commit c2e2e13455)
 2016-01-11 17:12:43 -08:00
Zhe Zhang 17dfa04be6 HDFS-9630. DistCp minor refactoring and clean up. Contributed by Kai Zheng. 
Change-Id: I363c4ffcac32116ddcdc0a22fac3db92f14a0db0

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2016-01-11 09:53:20 -08:00
cnauroth 813841e10b HADOOP-12551. Introduce FileNotFoundException for WASB FileSystem API. Contributed by Dushyanth. 
(cherry picked from commit 0e76f1fcea)
 2016-01-09 22:18:22 -08:00
cnauroth debd13387d HADOOP-12678. Handle empty rename pending metadata file during atomic rename in redo path. Contributed by Madhumita Chakraborty. 
(cherry picked from commit f0fa6d869b)
 2016-01-08 20:19:11 -08:00
cnauroth 4753676a34 HADOOP-12634. Change Lazy Rename Pending Operation Completion of WASB to address case of potential data loss due to partial copy. Contributed by Gaurav Kanade. 
(cherry picked from commit 978bbdfeb2)
 2016-01-06 11:16:18 -08:00
Ravi Prakash 6330683778 HADOOP-12689. S3 filesystem operations stopped working correctly 
(cherry picked from commit 2d16f40dab)
 2016-01-05 23:43:17 -08:00
Wangda Tan 6eefae1b33 YARN-4510. Fix SLS startup failure caused by NPE. (Bibin A Chundatt via wangda) 2015-12-30 15:34:39 -08:00
Wangda Tan 9a7d5f69ec YARN-4522. Queue acl can be checked at app submission. (Jian He via wangda) 
(cherry picked from commit 8310b2e9ff)
 2015-12-30 15:31:11 -08:00
Xiaoyu Yao a65e159c59 HADOOP-12608. Fix exception message in WASB when connecting with anonymous credential. Contributed by Dushyanth. 
(cherry picked from commit 007a8decc61750720033e06b6d08861ac3788e41)
 2015-12-30 11:12:02 -08:00
Wangda Tan 9f4a6ec663 YARN-4293. ResourceUtilization should be a part of yarn node CLI. (Sunil G via wangda) 
(cherry picked from commit 79c41b1d83)
 2015-12-16 13:28:56 -08:00
Vinod Kumar Vavilapalli b20e77eba3 Preparing for 2.9.0 development: mvn versions:set -DnewVersion=2.9.0 2015-11-25 17:47:22 -08:00
Robert Kanter 7b4bf23b2a MAPREDUCE-6550. archive-logs tool changes log ownership to the Yarn user when using DefaultContainerExecutor (rkanter) 
(cherry picked from commit 6d84cc16b3)
 2015-11-25 17:13:00 -08:00
Akira Ajisaka 4257991257 HADOOP-12598. Add XML namespace declarations for some hadoop/tools modules. Contributed by Xin Wang. 
(cherry picked from commit b53f20135599cc7825b6213794e2f999bc43e2ad)
 2015-11-26 01:21:32 +09:00
Karthik Kambatla 49ed955c91 YARN-3980. Plumb resource-utilization info in node heartbeat through to the scheduler. (Inigo Goiri via kasha) 
(cherry picked from commit 52948bb20b)
 2015-11-24 13:52:29 +05:30
Haohui Mai 1d175f0f3f HADOOP-10465. Fix use of generics within SortedMapWritable. Contributed by Bertrand Dechoux. 2015-11-22 18:12:53 -08:00
Haohui Mai ac0694eb6e HADOOP-12098. Remove redundant test dependencies in Hadoop Archives. Contributed by Varun Saxena. 2015-11-22 17:25:29 -08:00
Haohui Mai f504d95e87 HDFS-9402. Switch DataNode.LOG to use slf4j. Contributed by Walter Su. 2015-11-22 15:57:23 -08:00
Karthik Kambatla d3f9508096 YARN-4367. SLS webapp doesn't load. (kasha) 2015-11-18 13:45:00 -08:00
Akira Ajisaka c5c663c05d HADOOP-12545. Hadoop javadoc has broken links for AccessControlList, ImpersonationProvider, DefaultImpersonationProvider, and DistCp. Contributed by Arshad Mohammad. 
(cherry picked from commit f94d892704)
 2015-11-13 21:53:23 +09:00
Lei Xu 47941858af Revert "HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei)" 
This reverts commit 01ae30796d.
 2015-11-06 09:52:21 -08:00
Lei Xu 01ae30796d HADOOP-11684. S3a to use thread pool that blocks clients. (Thomas Demoor and Aaron Fabbri via lei) 
(cherry picked from commit bff7c90a56)
 2015-11-05 18:36:55 -08:00
cnauroth 81f7e8af22 HADOOP-12540. TestAzureFileSystemInstrumentation#testClientErrorMetrics fails intermittently due to assumption that a lease error will be thrown. Contributed by Gaurav Kanade. 
(cherry picked from commit 0fb1867fd6)
 2015-11-04 10:29:01 -08:00
cnauroth c1d7b26e94 HADOOP-12508. delete fails with exception when lease is held on blob. Contributed by Gaurav Kanade. 
(cherry picked from commit 9e7dcab185)
 2015-11-02 10:21:46 -08:00
cnauroth c260d7ec78 HADOOP-12533. Introduce FileNotFoundException in WASB for read and seek API. Contributed by Dushyanth. 
(cherry picked from commit 28790692624177d89fb1e4f59e2f83a659fc3089)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
 2015-11-02 09:39:27 -08:00
Kihwal Lee dae9f0ef53 Addendum to MAPREDUCE-6451 
(cherry picked from commit b24fe06483)
 2015-10-30 16:06:50 -05:00
Kihwal Lee 52819fe4ef MAPREDUCE-6451. DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic. Contributed by Kuhu Shukla. 
(cherry picked from commit 2868ca0328)
 2015-10-30 14:57:57 -05:00
cnauroth c7f87dc2d0 HADOOP-12519. hadoop-azure tests should avoid creating a metrics configuration fiale in the module root directory. Contributed by Chris Nauroth. 
(cherry picked from commit 2d10cb8e00)
 2015-10-28 09:44:21 -07:00
cnauroth 0377795e06 HADOOP-11685. StorageException complaining " no lease ID" during HBase distributed log splitting. Contributed by Duo Xu. 
(cherry picked from commit 1f7ecb0c84)
 2015-10-27 22:56:33 -07:00
cnauroth a23f79b92c HADOOP-12520. Use XInclude in hadoop-azure test configuration to isolate Azure Storage account keys for service integration tests. Contributed by Chris Nauroth. 
(cherry picked from commit 73822de7c3)
 2015-10-27 22:49:04 -07:00
Varun Vasudev c5bf1cb7af YARN-4302. SLS not able start due to NPE in SchedulerApplicationAttempt. Contributed by Bibin A Chundatt. 
(cherry picked from commit c28e16b40c)
 2015-10-27 20:56:45 +05:30
cnauroth b023539c19 HADOOP-12484. Single File Rename Throws Incorrectly In Potential Race Condition Scenarios. Contributed by Gaurav Kanade. 
(cherry picked from commit cb282d5b89)
 2015-10-22 14:30:09 -07:00
cnauroth 10398daadf HADOOP-12334. Change Mode Of Copy Operation of HBase WAL Archiving to bypass Azure Storage Throttling after retries. Contributed by Gaurav Kanade. 
(cherry picked from commit 47641fcbc9)
 2015-10-22 12:21:44 -07:00
Robert Kanter 492142097b MAPREDUCE-6495. Docs for archive-logs tool (rkanter) 
(cherry picked from commit 0c4af0f998)
 2015-10-20 17:34:56 -07:00
Haohui Mai ee04add696 HDFS-9167. Update pom.xml in other modules to depend on hdfs-client instead of hdfs. Contributed by Mingliang Liu. 2015-10-13 15:24:29 -07:00
Haohui Mai f6b0fcdc75 Revert "HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu." 
This reverts commit a412edf5c4.
 2015-10-10 03:25:16 -07:00
Haohui Mai a412edf5c4 HADOOP-12469. distcp shout not ignore the ignoreFailures option. Contributed by Mingliang Liu. 2015-10-09 22:38:13 -07:00
Robert Kanter e5bf4594cc MAPREDUCE-6503. archive-logs tool should use HADOOP_PREFIX instead of HADOOP_HOME (rkanter) 
(cherry picked from commit 1bca1bb517)
 2015-10-06 16:50:18 -07:00
cnauroth 535762fadb HADOOP-12350. WASB Logging: Improve WASB Logging around deletes, reads and writes. Contributed by Dushyanth. 
(cherry picked from commit 5f6edb30c2)
 2015-10-05 20:12:33 -07:00
Akira Ajisaka c6f30e5913 HADOOP-10296. Incorrect null check in SwiftRestClient#buildException(). Contributed by Rahul Palamuttam and Kanaka Kumar Avvaru. 
(cherry picked from commit ecbfd68974)
 2015-10-01 18:35:19 +02:00
Robert Kanter 46ef5aa8eb MAPREDUCE-6494. Permission issue when running archive-logs tool as different users (rkanter) 
(cherry picked from commit 5db371f52f)
 2015-09-30 17:34:03 -07:00
Kihwal Lee 95b8f1f3a6 HADOOP-12446. Undeprecate createNonRecursive(). Contributed by Ted Yu. 
(cherry picked from commit 3abbdc929b)
 2015-09-28 17:33:20 -05:00
Harsh J 03e370e306 MAPREDUCE-6471. Document distcp incremental copy. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit 66dad854c0)
 2015-09-28 13:14:20 +05:30
cnauroth 85c0cb0075 HADOOP-11918. Listing an empty s3a root directory throws FileNotFound. Contributed by Lei (Eddy) Xu. 
(cherry picked from commit 7fe521b1dd)
 2015-09-25 22:34:10 -07:00
Robert Kanter d8a5d2b2fc MAPREDUCE-6480. archive-logs tool may miss applications (rkanter) 
(cherry picked from commit 8fec882cfaac047af1892615fe32d8c3ca02e7f9)
 2015-09-25 15:00:07 -07:00
Jian He d458805690 YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan 
(cherry picked from commit 89cab1ba5f)

Conflicts:
	hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java
 2015-09-24 10:30:04 -07:00
Harsh J 38420754f5 MAPREDUCE-5045. UtilTest#isCygwin method appears to be unused. Contributed by Neelesh Srinivas Salian. 
(cherry picked from commit de1322aa76007832698af682a3b76042116043b9)
 2015-09-22 21:38:31 +05:30
Tsuyoshi Ozawa 52cc2ca488 HADOOP-12428. Fix inconsistency between log-level guards and statements. Contributed by Jagadesh Kiran N and Jackie Chang. 
(cherry picked from commit dfd807afab)
 2015-09-22 12:54:44 +09:00
Steve Loughran 988749e6ea HADOOP-12087. [JDK8] Fix javadoc errors caused by incorrect or illegal tags. (Akira AJISAKA via stevel). 2015-09-13 14:25:06 +01:00
Karthik Kambatla 999aa22509 MAPREDUCE-6415. Create a tool to combine aggregated logs into HAR files. (Robert Kanter via kasha) 2015-09-09 17:49:23 -07:00
Tsuyoshi Ozawa 7f1abf0b27 HADOOP-10318. Incorrect reference to nodeFile in RumenToSLSConverter error message. Contributed by Wei Yan. 
(cherry picked from commit 355eaaa33d)
 2015-09-03 22:49:12 +09:00
Lei Xu de42fce390 HADOOP-12346. Increase some default timeouts / retries for S3a connector. (Sean Mackrory via Lei (Eddy) Xu) 2015-08-29 09:58:57 -07:00
Rohith Sharma K S 6dc732f2f7 YARN-3896. RMNode transitioned from RUNNING to REBOOTED because its response id has not been reset synchronously. (Jun Gong via rohithsharmaks) 
(cherry picked from commit feaf034994)
 2015-08-24 11:30:46 +05:30
Yongjun Zhang b085c5ef6f HDFS-8828. Utilize Snapshot diff report to build diff copy list in distcp. (Yufei Gu via Yongjun Zhang) 
(cherry picked from commit 0bc15cb6e6)
 2015-08-20 08:10:34 -07:00
cnauroth b83d475252 HADOOP-12304. Applications using FileContext fail with the default file system configured to be wasb/s3/etc. Contributed by Chris Nauroth. 
(cherry picked from commit 3da0bedaec)
 2015-08-05 14:56:54 -07:00
Lei Xu f7b0f292e7 HADOOP-12269. Update aws-sdk dependency to 1.10.6 (Thomas Demoor via Lei (Eddy) Xu) 
(cherry picked from commit d5403747b5)
 2015-08-04 18:56:07 -07:00
cnauroth bb080b32d4 HADOOP-12239. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 
(cherry picked from commit efa97243ec)
 2015-07-22 11:17:00 -07:00
Vinayakumar B 03d68b5575 HADOOP-12017. Hadoop archives command should use configurable replication factor when closing (Contributed by Bibin A Chundatt) 
(cherry picked from commit 94c6a4aa85)
 2015-07-22 10:27:03 +05:30
Wangda Tan 3dd113fa7c YARN-2003. Support for Application priority : Changes in RM and Capacity Scheduler. (Sunil G via wangda) 
(cherry picked from commit c39ca541f4)
 2015-07-21 09:57:53 -07:00
Vinayakumar B 8ef3821356 HADOOP-12214. Parse 'HadoopArchive' commandline using cli Options. (Contributed by Vinayakumar B) 
(cherry picked from commit 87f29c6b8a)
 2015-07-21 13:13:31 +05:30
Steve Loughran f3296a1984 HADOOP-12235 hadoop-openstack junit & mockito dependencies should be "provided". (Ted Yu via stevel) 2015-07-20 11:22:22 +01:00
Chris Douglas c6b0bbabc7 HADOOP-12211. Collect disk usage on the node. Contributed by Robert Grandl 
(cherry picked from commit a431ed9075)
 2015-07-13 15:36:25 -07:00
Ming Ma b169889f01 YARN-3445. Cache runningApps in RMNode for getting running apps on given NodeId. (Junping Du via mingma) 
(cherry picked from commit 08244264c0)
 2015-07-10 08:34:01 -07:00
Chris Douglas 66419d3672 HADOOP-12210. Collect network usage on the node. Contributed by Robert Grandl 
(cherry picked from commit 1a0752d85a)
 2015-07-09 17:48:56 -07:00
Robert Kanter 3f79ee7494 HADOOP-12159. Move DistCpUtils#compareFs() to org.apache.hadoop.fs.FileUtil and fix for HA namespaces (rchiang via rkanter) 
(cherry picked from commit aaafa0b2ee)
 2015-06-30 16:50:42 -07:00
cnauroth 399df10c58 HADOOP-12089. StorageException complaining " no lease ID" when updating FolderLastModifiedTime in WASB. Contributed by Duo Xu. 
(cherry picked from commit 460e98f7b3)
 2015-06-29 13:49:12 -07:00
Colin Patrick Mccabe df58fdaf21 HADOOP-12104. Migrate Hadoop Pipes native build to new CMake framework (Alan Burlison via Colin P. McCabe) 
(cherry picked from commit 5a27c3fd76)
 2015-06-29 12:31:43 -07:00
cnauroth f5b0cce7fa HADOOP-12073. Azure FileSystem PageBlobInputStream does not return -1 on EOF. Contributed by Ivan Mitic. 
(cherry picked from commit c45784bc90)
 2015-06-08 22:42:24 -07:00
Andrew Wang 83a97907cc HADOOP-12059. S3Credentials should support use of CredentialProvider. Contributed by Sean Busbey. 
(cherry picked from commit 2dbc40e608)
 2015-06-05 13:11:12 -07:00
Robert Kanter ee3002933a MAPREDUCE-5965. Hadoop streaming throws error if list of input files is high. Error is: "error=7, Argument list too long at if number of input file is high" (wilfreds via rkanter) 
(cherry picked from commit cc70df98e7)
 2015-06-03 18:42:16 -07:00
cnauroth 39f451e721 HADOOP-11959. WASB should configure client side socket timeout in storage client blob request options. Contributed by Ivan Mitic. 
(cherry picked from commit 94e7d49a6d)

Conflicts:
	hadoop-project/pom.xml
 2015-05-28 12:32:55 -07:00
Colin Patrick Mccabe 3dec58dd78 HADOOP-11969. ThreadLocal initialization in several classes is not thread safe (Sean Busbey via Colin P. McCabe) 
(cherry picked from commit 7dba7005b7)
 2015-05-26 12:40:41 -07:00
Vinod Kumar Vavilapalli a67cb4826b YARN-160. Enhanced NodeManager to automatically obtain cpu/memory values from underlying OS when configured to do so. Contributed by Varun Vasudev. 
(cherry picked from commit 500a1d9c76)
 2015-05-26 11:39:40 -07:00
Colin Patrick Mccabe 97ebe1b2cf HADOOP-11927. Fix "undefined reference to dlopen" error when compiling libhadooppipes (Xianyin Xin via Colin P. McCabe) 
(cherry picked from commit f34638399a)
 2015-05-22 12:57:29 -07:00
Jing Zhao 5caea4cd46 HADOOP-1540. Support file exclusion list in distcp. Contributed by Rich Haase. 
(cherry picked from commit 0790275f05)
 2015-05-18 13:26:09 -07:00
Akira Ajisaka b26d99c394 HADOOP-11939. Deprecate DistCpV1 and Logalyzer. Contributed by Brahma Reddy Battula. 
(cherry picked from commit f5c48238d5)
 2015-05-18 16:14:19 +09:00
cnauroth 91855c2340 HADOOP-11960. Enable Azure-Storage Client Side logging. Contributed by Dushyanth. 
(cherry picked from commit cb8e69a80c)
 2015-05-14 22:22:46 -07:00
Akira Ajisaka b9a92c4452 HADOOP-9723. Improve error message when hadoop archive output path already exists. Contributed by Jean-Baptiste Onofré and Yongjun Zhang. 
(cherry picked from commit 92c38e41e1)
 2015-05-13 17:30:49 +09:00
Vinayakumar B 14d88a85e4 HADOOP-11922. Misspelling of threshold in log4j.properties for tests in hadoop-tools (Contributed by Gabor Liptak) 
(cherry picked from commit f33efbc827)
 2015-05-08 10:23:27 +05:30
Jonathan Eagles 9ed6b77da1 HADOOP-11889. Make checkstyle runnable from root project (Gera Shegalov via jeagles) 
(cherry picked from commit 2d7363b273)

Conflicts:
	pom.xml
 2015-05-01 19:04:41 -07:00
Akira Ajisaka 47ba08a119 HADOOP-11821. Fix findbugs warnings in hadoop-sls. Contributed by Brahma Reddy Battula. 
(cherry picked from commit f384a063a6)
 2015-04-30 19:49:32 +09:00
Ravi Prakash 122262a1fb HADOOP-11876. Refactor code to make it more readable, minor maybePrintStats bug (Zoran Dimitrijevic via raviprak) 2015-04-24 13:39:48 -07:00
Steve Loughran a6a5d1d6b5 HADOOP-11730. Regression: s3n read failure recovery broken. (Takenori Sato via stevel) 2015-04-23 21:39:43 +01:00
Ravi Prakash aa34aa5caa HADOOP-11827. Speed-up distcp buildListing() using threadpool (Zoran Dimitrijevic via raviprak) 2015-04-21 16:50:40 -07:00
Jing Zhao d4dd97eabd HDFS-8151. Always use snapshot path as source when invalid snapshot names are used for diff based distcp. Contributed by Jing Zhao. 
(cherry picked from commit 4c097e473b)
 2015-04-15 12:39:09 -07:00
Akira Ajisaka f603b9ccca MAPREDUCE-6307. Remove property mapreduce.tasktracker.taskmemorymanager.monitoringinterval. Contributed by J.Andreina. 
(cherry picked from commit 83979e61a4)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2015-04-10 20:23:46 +09:00
Colin Patrick Mccabe 386b90a700 HADOOP-11785. Reduce the number of listStatus operation in distcp buildListing (Zoran Dimitrijevic via Colin P. McCabe) 
(cherry picked from commit 932730df7d)
 2015-04-03 14:08:53 -07:00
Haohui Mai 9e114ee607 HDFS-8036. Use snapshot path as source when using snapshot diff report in DistCp. Contributed by Jing Zhao. 2015-04-01 16:51:09 -07:00
Harsh J 274db918c3 MAPREDUCE-6291. Correct mapred queue usage command. Contributed by Brahma Reddy Battula. 
(cherry picked from commit f5432cb47dfacaa57faed32a29d00bbeff8a5385)
 2015-03-28 11:58:44 +05:30
Tsuyoshi Ozawa 24b60a7d1a HADOOP-11760. Fix typo of javadoc in DistCp. Contributed by Brahma Reddy Battula. 
(cherry picked from commit e074952bd6)
 2015-03-27 23:16:37 +09:00