Apache
/
hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
8,503 Commits 326 Branches 371 Tags 1.5 GiB
Commit Graph

1822 Commits

Author SHA1 Message Date
Jian He d2356a552a Revert "YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai." 
This reverts commit 51af8d367d.

Conflicts:
	hadoop-yarn-project/CHANGES.txt
(cherry picked from commit a4f2995b9e)
 2014-12-15 11:08:44 -08:00
Harsh J a3fa1f0761 MAPREDUCE-6194. Bubble up final exception in failures during creation of output collectors. Contributed by Varun Saxena. 
(cherry picked from commit 298d09c9b5)
 2014-12-15 14:27:49 +05:30
Gera Shegalov 6465931c19 MAPREDUCE-4879. TeraOutputFormat may overwrite an existing output directory. (gera) 
(cherry picked from commit 25a0440238)
 2014-12-13 18:12:02 -08:00
Devaraj K 25f4a0193b MAPREDUCE-6046. Change the class name for logs in RMCommunicator. 
Contributed by Sahil Takiar.

(cherry picked from commit 0bd0229110)
 2014-12-12 11:44:00 +05:30
Gera Shegalov 8437c31b67 HADOOP-11211. mapreduce.job.classloader.system.classes semantics should be order-independent. (Yitong Zhou via gera) 
(cherry picked from commit 0bcea111e5)
 2014-12-11 13:29:40 -08:00
Harsh J 6e06a51e30 MAPREDUCE-5420. Remove mapreduce.task.tmp.dir from mapred-default.xml. Contributed by James Carman. (harsh) 
(cherry picked from commit cb99f43305)
 2014-12-11 09:46:08 +05:30
Haohui Mai 49aacee2cb HADOOP-11372. Fix new findbugs warnings in mapreduce-examples. Contributed by Li Lu. 2014-12-09 10:49:11 -08:00
Haohui Mai 759179e631 HADOOP-11369. Fix new findbugs warnings in hadoop-mapreduce-client, non-core directories. Contributed by Li Lu. 2014-12-09 10:46:25 -08:00
Harsh J bb1fedfbc3 MAPREDUCE-6177. Minor typo in the EncryptedShuffle document about ssl-client.xml. Contributed by Yangping Wu. (harsh) 
(cherry picked from commit 8963515b88)
 2014-12-08 16:00:12 +05:30
Jason Lowe db723a8499 MAPREDUCE-5932. Provide an option to use a dedicated reduce-side shuffle log. Contributed by Gera Shegalov 
(cherry picked from commit 03ab24aa01)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
 2014-12-03 17:12:47 +00:00
Zhijie Shen d21ef79707 YARN-2890. MiniYARNCluster should start the timeline server based on the configuration. Contributed by Mit Desai. 
(cherry picked from commit 51af8d367d)
 2014-12-01 19:08:31 -08:00
Jason Lowe b6d7b789bd MAPREDUCE-6160. Potential NullPointerException in MRClientProtocol interface implementation. Contributed by Rohith 
(cherry picked from commit 0c588904f8)
 2014-12-01 22:40:38 +00:00
Jason Lowe 2bdef5697b MAPREDUCE-6172. TestDbClasses timeouts are too aggressive. Contributed by Varun Saxena 
(cherry picked from commit 2b30fb1053)
 2014-12-01 21:52:51 +00:00
Jian He 80e9be2d95 MAPREDUCE-5568. Fixed CompletedJob in JHS to show progress percentage correctly in case the number of mappers or reducers is zero. Contributed by MinJi Kim 
(cherry picked from commit 78f7cdbfd6)
 2014-11-25 13:03:40 -08:00
Karthik Kambatla e0df934ed1 Update release date for 2.5.2 
(cherry picked from commit 1e9a3f42ff)
 2014-11-21 14:34:47 -08:00
Jonathan Eagles 6570f7fd72 YARN-2375. Allow enabling/disabling timeline server per framework. (Mit Desai via jeagles) 
(cherry picked from commit c298a9a845)
 2014-11-20 23:40:26 -06:00
Karthik Kambatla 3a4211ec2c MAPREDUCE-6169. MergeQueue should release reference to the current item from key and value at the end of the iteration to save memory. (Zhihai Xu via kasha) 
(cherry picked from commit 90194ca1cb)
 2014-11-20 15:37:18 -08:00
Devaraj K 09df35e51d MAPREDUCE-6049. AM JVM does not exit if MRClientService graceful shutdown 
fails. Contributed by Rohith.
(cherry picked from commit 7250b0bf91)
 2014-11-18 22:07:26 +05:30
Jason Lowe 0b9a1a3e03 MAPREDUCE-6162. mapred hsadmin fails on a secure cluster. Contributed by Jason Lowe 
(cherry picked from commit 09b3dee122)
 2014-11-17 23:01:16 +00:00
Ravi Prakash 45692effe4 MAPREDUCE-5918. LineRecordReader can return the same decompressor to CodecPool multiple times (Sergey Murylev via raviprak) 2014-11-14 03:48:28 -08:00
Arun C. Murthy fc324d7062 Preparing to release hadoop-2.6.0-rc1. 
(cherry picked from commit 9ff7315cbe)
 2014-11-13 13:03:13 -08:00
Jason Lowe e62a5f9d0a MAPREDUCE-6156. Fetcher - connect() doesn't handle connection refused correctly. Contributed by Junping Du 
(cherry picked from commit 177e8090f5)
 2014-11-13 15:43:28 +00:00
cnauroth 69a7780ee5 HADOOP-7984. Add hadoop --loglevel option to change log level. Contributed by Aikira AJISAKA. 2014-11-12 21:44:05 -08:00
Karthik Kambatla 92de44c2c5 Set the release date for 2.5.2 
(cherry picked from commit 68a0508728)

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-11-10 15:23:43 -08:00
Arun C. Murthy adfb830a2b Preparing to release hadoop-2.6.0: Set version in branch-2 to 2.7.0-SNAPSHOT. 2014-11-09 19:19:02 -08:00
Arun C. Murthy cff08dab5e Updated CHANGES.txt to reflect hadoop-2.6.0 release dates. 
(cherry picked from commit 770cc14442)
 2014-11-09 18:48:01 -08:00
Arun C. Murthy 175d222bfc YARN-2830. Add backwords compatible ContainerId.newInstance constructor. Contributed by Jonathan Eagles. 
(cherry picked from commit 43cd07b408)
 2014-11-09 15:03:59 -08:00
Arun C. Murthy 1f5bad04ce HADOOP-11286. Copied LimitInputStream from guava-0.14 to hadoop to avoid issues with newer versions of guava in applications. Contributed by Christopher Tubbs. 
(cherry picked from commit 6caa8100d5)
 2014-11-09 05:23:10 -08:00
Kihwal Lee a19123df9a MAPREDUCE-5958. Wrong reduce task progress if map output is compressed. Contributed by Emilio Coppa and Jason Lowe. 
(cherry picked from commit 8f701ae07a)
 2014-11-06 15:56:10 -06:00
Jason Lowe c0bbb72912 MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path. Contributed by Gera Shegalov 
(cherry picked from commit 10f9f5101c)
 2014-11-06 15:12:06 +00:00
Jian He 944fb1c3a9 MAPREDUCE-6048. Fixed TestJavaSerialization failure. Contributed by Varun Vasudev 
(cherry picked from commit 73068f677b)
 2014-11-04 20:20:33 -08:00
Haohui Mai fa92b40f09 HADOOP-11230. Add missing dependency of bouncycastle for kms, httpfs, hdfs, MR and YARN. Contributed by Robert Kanter. 2014-11-04 17:53:00 -08:00
Karthik Kambatla 6f9484221e Adding release 2.5.2 to CHANGES.txt 
(cherry picked from commit 4de56d27fb)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-11-04 16:51:54 -08:00
Haohui Mai aa09c4c342 HADOOP-10563. Remove the dependency of jsp in trunk. Contributed by Haohui Mai. 2014-11-04 14:52:51 -08:00
Zhijie Shen c4aaa4db6c MAPREDUCE-6052. Supported overriding the default container-log4j.properties file per job. Contributed by Junping Du. 
(cherry picked from commit ed63b11646)
 2014-11-01 00:49:38 -07:00
Vinod Kumar Vavilapalli 0ba8d8c19c YARN-2698. Moved some node label APIs to be correctly placed in client protocol. Contributed by Wangda Tan. 
(cherry picked from commit e0233c16eb)
 2014-10-30 23:02:06 -07:00
Kihwal Lee 45b6dea95f MAPREDUCE-6022. map_input_file is missing from streaming job 
environment. Contributed by Jason Lowe.
(cherry picked from commit b056048114)
 2014-10-29 12:30:08 -05:00
Vinod Kumar Vavilapalli 063bb0508b MAPREDUCE-6142. Fixed test failures in TestJobHistoryEventHandler and TestMRTimelineEventHandling. Contributed by Zhijie Shen. 
(cherry picked from commit 3f1441245d)
 2014-10-29 09:58:50 -07:00
Karthik Kambatla 160aebcae2 HADOOP-11243. SSLFactory shouldn't allow SSLv3. (Wei Yan via kasha) 
(cherry picked from commit 3c5f5af118)
 2014-10-28 18:08:00 -07:00
Zhijie Shen 52695d38fc MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. 
(cherry picked from commit 971e91c8c0)
 2014-10-27 21:08:38 -07:00
Zhijie Shen 733380fa70 MAPREDUCE-5933. Enabled MR AM to post history events to the timeline server. Contributed by Robert Kanter. 
(cherry picked from commit 6b2f11b54b)
 2014-10-27 20:41:02 -07:00
Zhijie Shen 3b03ea6b50 YARN-2209. Replaced AM resync/shutdown command with corresponding exceptions and made related MR changes. Contributed by Jian He. 
(cherry picked from commit 0f3b6900be)
 2014-10-23 22:03:30 -07:00
Jian He 1c235a4448 YARN-2198. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor. Contributed by Remus Rusanu 
(cherry picked from commit 3b12fd6cfb)
 2014-10-22 15:58:26 -07:00
Jian He 9b973b7c83 MAPREDUCE-6126. Fixed Rumen JobBuilder to ignore NormalizedResourceEvent. Contributed by Junping Du 
Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2014-10-22 10:57:32 -07:00
Ivan Mitic b4668f41e8 Revert "MAPREDUCE-5911. Terasort TeraOutputFormat does not check for output directory existance. Contributed by Bruno P. Kinoshita." 
This reverts commit 767a3fd0b7.
 2014-10-19 23:50:42 -04:00
Ivan Mitic 767a3fd0b7 MAPREDUCE-5911. Terasort TeraOutputFormat does not check for output directory existance. Contributed by Bruno P. Kinoshita. 
(cherry picked from commit 7bbda6ef92)
 2014-10-19 19:10:31 -04:00
Jason Lowe 8885b75d78 MAPREDUCE-5542. Killing a job just as it finishes can generate an NPE in client. Contributed by Rohith 
(cherry picked from commit 209b1699fc)
 2014-10-17 19:53:17 +00:00
Jian He f93d2ea27e YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 2014-10-15 15:28:26 -07:00
Jason Lowe 025b6c13e9 MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling. Contributed by Gera Shegalov 
(cherry picked from commit f19771a24c)
 2014-10-15 17:51:53 +00:00
Jason Lowe 1179a0cb07 MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting for maps. Contributed by Siqi Li 
(cherry picked from commit b9edad6403)
 2014-10-15 15:54:57 +00:00
Jason Lowe 942fb67a04 MAPREDUCE-6115. TestPipeApplication#testSubmitter fails in trunk. Contributed by Binglin Chang 
(cherry picked from commit 793dbf2465)
 2014-10-13 16:05:29 +00:00
Jason Lowe e17e3bdbc4 MAPREDUCE-6125. TestContainerLauncherImpl sometimes fails. Contributed by Mit Desai 
(cherry picked from commit bbe80cdc7b)
 2014-10-13 15:09:26 +00:00
Karthik Kambatla 7bfd9e068d MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha) 
(cherry picked from commit ed0e0ef9748ce5b231de677ab41c3035137bbde4)
 2014-10-11 22:49:46 -07:00
cnauroth f6278cbac5 MAPREDUCE-6123. TestCombineFileInputFormat incorrectly starts 2 MiniDFSCluster instances. Contributed by Chris Nauroth. 
(cherry picked from commit 777877d079)
 2014-10-09 22:38:25 -07:00
cnauroth 61be28f501 MAPREDUCE-6122. Fix CHANGES.txt. 
(cherry picked from commit 2f0166db56)
 2014-10-09 22:32:38 -07:00
cnauroth 85a2b90740 MAPREDUCE-6122. TestLineRecordReader may fail due to test data files checked out of git with incorrect line endings. Contributed by Chris Nauroth. 
(cherry picked from commit 684170d5ef)
 2014-10-09 22:27:40 -07:00
Jason Lowe 979b915ecc MAPREDUCE-6029. TestCommitterEventHandler fails in trunk. Contributed by Mit Desai 
(cherry picked from commit 2e789eb226)
 2014-10-07 14:11:48 +00:00
subru 30a370e705 MAPREDUCE-6103.Adding reservation APIs to MR resource manager delegate. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit aa92dd45f2)
(cherry picked from commit 3f282762d1)
 2014-10-06 10:21:06 -07:00
Arun C. Murthy 6a60fa279a Preparing to branch hadoop-2.6 2014-09-30 10:37:03 -07:00
Jason Lowe 7b8f2f5b1d MAPREDUCE-6094. TestMRCJCFileInputFormat.testAddInputPath() fails on trunk. Contributed by Akira AJISAKA 
(cherry picked from commit 23fbfe36c3)
 2014-09-29 18:14:11 +00:00
Allen Wittenauer c65ef9ca1e MAPREDUCE-6072. Remove INSTALL document (Akira AJISAKA via aw) 2014-09-29 08:31:10 -07:00
Jason Lowe 0cfff28f57 HADOOP-11049. javax package system class default is too broad. Contributed by Sangjin Lee 
(cherry picked from commit cf4631d7c9)
 2014-09-29 14:37:28 +00:00
Jian He a6879b606e MAPREDUCE-6087. Fixed wrong config name of MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS. Contributed by Akira AJISAKA 
(cherry picked from commit 5f16c98ad6)
 2014-09-26 18:06:19 -07:00
Allen Wittenauer 67d6fc3f01 MAPREDUCE-5945. Update the description of GenericOptionsParser -jt option (Akira AJISAKA via aw) 2014-09-26 14:26:36 -07:00
Allen Wittenauer e5d284664b MAPREDUCE-5796. Use current version of the archive name in DistributedCacheDeploy document (Akira AJISAKA via aw) 2014-09-26 12:59:59 -07:00
Allen Wittenauer fe8222998f MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags (Tsuyoshi OZAWA via aw) 2014-09-26 11:17:11 -07:00
Zhijie Shen 7b3ecc547f MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. 
(cherry picked from commit 662fc11ae7)
 2014-09-25 23:46:57 -07:00
Allen Wittenauer 7ffb636a54 MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) 2014-09-25 11:33:01 -07:00
Allen Wittenauer afde00a0f5 MAPREDUCE-6109. Fix minor typo in distcp -p usage text (Charles Lamb via aw) 2014-09-25 11:28:50 -07:00
Jason Lowe d5d9fd3255 MAPREDUCE-6104. TestJobHistoryParsing.testPartialJob fails in branch-2. Contributed by Mit Desai 
(cherry picked from commit 7af4c3888b)
 2014-09-24 16:10:28 +00:00
Zhijie Shen b8d645539d MAPREDUCE-5279. Made MR headroom calculation honor cpu dimension when YARN scheduler resource type is memory plus cpu. Contributed by Peng Zhang and Varun Vasudev. 2014-09-22 10:41:55 -07:00
Jason Lowe f4534746fb MAPREDUCE-6095. Enable DistributedCache for uber-mode Jobs. Contributed by Gera Shegalov 
(cherry picked from commit 7039b98e1c)
 2014-09-22 15:22:58 +00:00
Allen Wittenauer 0037b8f67b YARN-2460. Remove obsolete entries from yarn-default.xml (Ray Chiang via aw) 2014-09-19 14:08:04 -07:00
Jason Lowe bdb864e5e1 MAPREDUCE-6091. YARNRunner.getJobStatus() fails with ApplicationNotFoundException if the job rolled off the RM view. Contributed by Sangjin Lee 
(cherry picked from commit 951847ba94)
 2014-09-19 20:17:44 +00:00
Zhijie Shen 9333ee3068 YARN-2568. Fixed the potential test failures due to race conditions when RM work-preserving recovery is enabled. Contributed by Jian He. 
(cherry picked from commit 6fe5c6b746)
 2014-09-18 22:02:53 -07:00
Karthik Kambatla 372bf54072 MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs. (Zhihai Xu via kasha) 
(cherry picked from commit 52945a33cc)
 2014-09-18 15:47:21 -07:00
Jason Lowe 44c7113133 MAPREDUCE-5891. Improved shuffle error handling across NM restarts. Contributed by Junping Du 
(cherry picked from commit 2c3da25fd7)
 2014-09-18 22:01:42 +00:00
Andrew Wang 5bc71071d1 HADOOP-11106. Document considerations of HAR and Encryption. (clamb via wang) 
(cherry picked from commit fe2f54d4cb)
 2014-09-18 14:51:27 -07:00
Jason Lowe 4b4e44a8aa MAPREDUCE-6090. mapred hsadmin getGroups fails to connect in some cases. Contributed by Robert Kanter 
(cherry picked from commit 1cf3198047)
 2014-09-18 21:02:13 +00:00
Vinod Kumar Vavilapalli 5cdb24d4b1 YARN-611. Added an API to let apps specify an interval beyond which AM failures should be ignored towards counting max-attempts. Contributed by Xuan Gong. 
(cherry picked from commit 14e2639fd0)
 2014-09-13 18:05:02 -07:00
Karthik Kambatla d22cd065ed Update CHANGES.txt to set the release date for 2.5.1 
(cherry picked from commit 17ffbe0864)
 2014-09-11 17:36:06 -07:00
Jason Lowe 6f8abe7efe MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 9e28e089c6)
 2014-09-11 16:20:51 +00:00
Jason Lowe e6f108c99b MAPREDUCE-6075. HistoryServerFileSystemStateStore can create zero-length files. Contributed by Jason Lowe 
(cherry picked from commit 17a025d9dda9949f191b650e3ef96090880f568a)
 2014-09-10 22:12:24 +00:00
Jason Lowe 0b3f918b3b MAPREDUCE-6071. JobImpl#makeUberDecision doesn't log that Uber mode is disabled because of too much CPUs. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 45efc966ee)
 2014-09-05 13:48:16 +00:00
Chris Douglas 69ee11dc93 MAPREDUCE-6063. Correct spill size calculation for spills wrapping the circular buffer. Contributed by zhihai xu. 2014-09-03 17:02:17 -07:00
Karthik Kambatla 9855e884cb HADOOP-10956. Fix create-release script to include docs and necessary txt files. (kasha) 
(cherry picked from commit ce0462129f)
 2014-09-03 09:39:32 -07:00
Andrew Wang 075bb9e380 HADOOP-11021. Configurable replication factor in the hadoop archive command. Contributed by Zhe Zhang. 
(cherry picked from commit ea1c6f31c2)
 2014-08-29 14:47:28 -07:00
Jason Lowe 27086f5942 MAPREDUCE-5931. Validate SleepJob command line parameters. Contributed by Gera Shegalov 
(cherry picked from commit 15366d9227)
 2014-08-29 19:52:10 +00:00
Chris Douglas 4f95017602 Fix typos in log messages. Contributed by Ray Chiang 2014-08-28 16:30:13 -07:00
Alejandro Abdelnur 1a65717ffa Removing CHANGES-fs-encryption.txt files 2014-08-28 15:13:24 -07:00
Alejandro Abdelnur 5c37f02e27 Fix up CHANGES.txt for HDFS-6134, HADOOP-10150 and related JIRAs following merge to branch-2 
Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
	hadoop-mapreduce-project/CHANGES.txt

Conflicts:
	hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
 2014-08-28 15:05:07 -07:00
Alejandro Abdelnur c77bd85b62 HDFS-6134 and HADOOP-10150 subtasks. 2014-08-28 15:03:08 -07:00
Jason Lowe 38df629367 MAPREDUCE-5885. build/test/test.mapred.spill causes release audit warnings. Contributed by Chen He 
(cherry picked from commit 812bd0c0e5)
 2014-08-27 15:22:06 +00:00
Karthik Kambatla ba95033cc4 Fix CHANGES.txt entry for MAPREDUCE-6033. 
(cherry picked from commit 2d9b77d617)
 2014-08-27 01:17:19 -07:00
Karthik Kambatla 2a3c8331fe Add a section for 2.5.1 in CHANGES.txt 
(cherry picked from commit dc154ab86d)
 2014-08-27 00:56:17 -07:00
Sanford Ryza c38e751bca MAPREDUCE-5130. Fix mapred-default.xml after bad merge. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619930 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 20:58:12 +00:00
Zhijie Shen 7cad25ec94 MAPREDUCE-6044. Fully qualified intermediate done dir path breaks per-user dir creation on Windows. Contributed by Zhijie Shen. 
svn merge --ignore-ancestry -c 1619863 ../../trunk/


git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619864 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 17:08:17 +00:00
Sanford Ryza 53da85efaf MAPREDUCE-5130. Add missing job config options to mapred-default.xml (Ray Chiang via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619631 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 23:54:38 +00:00
Jason Darrell Lowe 23b4c0b6c1 HADOOP-10893. isolated classloader on the client side. Contributed by Sangjin Lee 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619605 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 21:41:24 +00:00
Karthik Kambatla c9d7647826 MAPREDUCE-5974. Allow specifying multiple MapOutputCollectors with fallback. (Todd Lipcon via kasha) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619496 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 17:39:54 +00:00