Commit Graph

585 Commits

Author SHA1 Message Date
Jason Lowe 599146d10b MAPREDUCE-6741. Add MR support to redact job conf properties. Contributed by Haibo Chen
(cherry picked from commit f1b74a3d9f)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
2016-10-05 15:33:23 +00:00
Varun Saxena 30a4c0bb63 MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena) 2016-08-20 04:18:25 +05:30
Vinod Kumar Vavilapalli f4b58eae94 Preparing for 2.7.4 development: Updating snapshot version to 2.7.4 on branch-2.7. 2016-06-14 19:14:02 -07:00
Jason Lowe 7ded648aee MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg
(cherry picked from commit 9227dfc25f)
2016-05-13 14:37:55 +00:00
Eric Payne 2ec5bb0fe8 MAPREDUCE-6633. AM should retry map attempts if the reduce task encounters commpression related errors. Contributed by Rushabh Shah
(cherry picked from commit 1fec06e037)
2016-04-18 16:12:53 +00:00
Wangda Tan 801c95c1cc MAPREDUCE-6302. Preempt reducers after a configurable timeout irrespective of headroom. (kasha via wangda) 2016-04-08 14:35:28 -07:00
Robert Kanter 66e1e92ffd MAPREDUCE-6622. Add capability to set JHS job cache to a task-based limit (rchiang via rkanter)
(cherry picked from commit 0f72da7e28)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
2016-03-06 18:34:46 -08:00
Varun Vasudev 0edc764184 MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du.
(cherry picked from commit c6f2d761d5)
(cherry picked from commit f1999fe275)

 Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
2016-02-23 14:45:07 +05:30
Andrew Wang 403934c011 MAPREDUCE-6637. Testcase Failure : TestFileInputFormat.testSplitLocationInfo. Contributed by Brahma Reddy Battula.
(cherry picked from commit 37577852ba)
(cherry picked from commit 7f6737951a)
2016-02-19 16:37:28 -08:00
Jason Lowe 7f6d150c9a MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). Contributed by Xuan Gong
(cherry picked from commit 43e669b22d)
2016-02-02 19:32:37 +00:00
Jason Lowe 296ecbd5ec MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory. Contributed by Bibin A Chundatt 2016-01-15 16:59:44 +00:00
Sangjin Lee 28fd4c70ca MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee)
(cherry picked from commit f6f16118d3)
2016-01-05 16:01:09 -08:00
Akira Ajisaka e94a8aea57 MAPREDUCE-6583. Clarify confusing sentence in MapReduce tutorial document. Contributed by Kai Sasaki.
(cherry picked from commit 7995a6ea4d)
(cherry picked from commit 8607cb6074)
2015-12-21 00:19:11 +09:00
Vinod Kumar Vavilapalli cab5d86738 Revert "MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev"
This reverts commit ba02e8f6d1.
2015-12-16 15:17:01 -08:00
Xuan ba02e8f6d1 MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev
(cherry picked from commit fc470840a0)
(cherry picked from commit 3b10ff6a6f)
2015-12-07 14:18:26 -08:00
Jason Lowe e1b6a5413f MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter)
(cherry picked from commit 7fd00b3db4)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
2015-11-30 17:45:27 +00:00
Jason Lowe 4ba5667aed MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly misleading. Contributed by Nathan Roberts
(cherry picked from commit cab3c7c889)
2015-11-24 22:02:21 +00:00
Jian He f95f416fad MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API. Contributed by Junping Du 2015-11-16 17:13:49 -08:00
Vinod Kumar Vavilapalli 09348c4bce Preparing for 2.7.3 development 2015-11-03 15:18:10 -08:00
Jason Lowe 521196874c MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. Contributed by Zhihai Xu
(cherry picked from commit 58d1a02b8d)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
2015-09-18 15:10:38 +00:00
Jason Lowe 850d6a3fd3 MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well. Contributed by Vinayakumar B, Rushabh Shah, and Akira AJISAKA
(cherry picked from commit 077250d8d7)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
2015-09-18 15:04:02 +00:00
Jason Lowe 4ea42b84b7 MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov
(cherry picked from commit 7dc3c1203d)

(cherry picked from commit 87c2d915f1)
2015-09-10 11:00:00 -07:00
Tsuyoshi Ozawa e8a15495bc MAPREDUCE-6442. Stack trace is missing when error occurs in client protocol provider's constructor Contributed by Chang Li.
(cherry picked from commit 9b685773ec)
2015-09-05 11:24:30 +09:00
Vinod Kumar Vavilapalli a36f090c55 Preparing for 2.7.2 development: mvn versions:set -DnewVersion=2.7.2 2015-06-28 16:41:12 -07:00
Karthik Kambatla d404caaa43 MAPREDUCE-6387. Serialize the recently added Task#encryptedSpillKey field at the end. (Arun Suresh via kasha)
(cherry picked from commit 6786daab33)
(cherry picked from commit 1a2e6e8091)
2015-06-05 09:19:52 -07:00
Akira Ajisaka 718dca4859 HADOOP-12058. Fix dead links to DistCp and Hadoop Archives pages. Contributed by Kazuho Fujii.
(cherry picked from commit dbed757cba)
(cherry picked from commit 1cccd1eeb3)
2015-06-05 01:47:01 +09:00
Tsuyoshi Ozawa 6b03ec5137 MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. 2015-05-19 12:01:14 -07:00
Vinod Kumar Vavilapalli d9d7bbd99b Fixing MR intermediate spills. Contributed by Arun Suresh.
(cherry picked from commit 6b710a42e0)
(cherry picked from commit 87862970f1)
2015-05-14 17:00:56 -07:00
Vinod Kumar Vavilapalli 0e68e304f7 MAPREDUCE-6251. Added a new config for JobClient to retry JobStatus calls so that they don't fail on history-server backed by DFSes with not so strong guarantees. Contributed by Craig Welch.
(cherry picked from commit f24452d14e)
2015-05-12 12:16:16 -07:00
Jason Lowe e7b6a076db MAPREDUCE-6259. IllegalArgumentException due to missing job submit time. Contributed by zhihai xu
(cherry picked from commit bf70c5ae28)
2015-05-04 20:40:43 +00:00
Jason Lowe 30d0f10458 MAPREDUCE-6334. Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler. Contributed by Eric Payne
(cherry picked from commit bc1bd7e5c4)
2015-04-28 20:20:32 +00:00
Robert Kanter 0eae3bc0f6 MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter)
(cherry picked from commit d50e8f0928)
2015-04-23 17:47:00 -07:00
Vinod Kumar Vavilapalli a7fa46c2e0 Preparing for 2.7.1 development: mvn versions:set 2015-04-09 14:13:30 -07:00
Junping Du 86e8c9958a MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe.
(cherry picked from commit eccb7d46ef)
2015-04-02 12:16:22 -07:00
Vinod Kumar Vavilapalli ec123abdc9 Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)"
This reverts commit 7bfd9e068d.

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
2015-03-31 11:30:04 -07:00
Vinod Kumar Vavilapalli 1c050bd135 Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)"
This reverts commit f9341c1e2c.

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
2015-03-31 11:28:51 -07:00
Vinod Kumar Vavilapalli 02646ddba2 Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu."
This reverts commit 929b04ce3a.
2015-03-31 11:26:13 -07:00
Vinod Kumar Vavilapalli 35af6f1802 YARN-3304. Cleaning up ResourceCalculatorProcessTree APIs for public use and removing inconsistencies in the default values. Contributed by Junping Du and Karthik Kambatla.
(cherry picked from commit c358368f51)
2015-03-30 10:11:49 -07:00
Harsh J 929b04ce3a MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu. 2015-03-22 10:18:32 +05:30
Jason Lowe 0be2461df5 MAPREDUCE-6275. Race condition in FileOutputCommitter v2 for user-specified task output subdirs. Contributed by Gera Shegalov and Siqi Li
(cherry picked from commit 91baca145a)
2015-03-19 21:41:49 +00:00
Tsuyoshi Ozawa 9f4ec4c540 MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers. Contributed by Zhihai Xu
(cherry picked from commit 9d38520c8e)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml

(cherry picked from commit 10745fcecd)
2015-03-14 16:46:53 +09:00
Haohui Mai 245f7b2a77 HADOOP-11633. Convert remaining branch-2 .apt.vm files to markdown. Contributed by Masatake Iwasaki. 2015-03-11 14:31:16 -07:00
Gera Shegalov fca89922c2 MAPREDUCE-4815. Speed up FileOutputCommitter#commitJob for many output files. (Siqi Li via gera)
(cherry picked from commit aa92b764a7)
2015-03-10 11:45:03 -07:00
Junping Du 9d1f67f2f2 Configurable timeout between YARNRunner terminate the application and forcefully kill. Contributed by Eric Payne.
(cherry picked from commit d39bc903a0)
(cherry picked from commit dbcdcb0d3c)
2015-03-10 06:24:34 -07:00
Tsuyoshi Ozawa 28f4e6b222 HADOOP-11602. Backport 'Fix toUpperCase/toLowerCase to use Locale.ENGLISH.' (ozawa)
(cherry picked from commit b46f9e72db)
2015-03-09 19:54:45 +09:00
Tsuyoshi Ozawa ccde4aed18 MAPREDUCE-5657. Backport '[JDK8] Fix Javadoc errors caused by incorrect or illegal tags in doc comments.' Contributed by Akira AJISAKA. 2015-03-09 12:00:04 +09:00
Karthik Kambatla f4d6c5e337 MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha)
(cherry picked from commit c66c3ac6bf)
2015-03-04 14:47:06 -08:00
Junping Du a9613bcc94 Ability to limit running map and reduce tasks. Contributed by Jason Lowe.
(cherry picked from commit 4228de9402)

Conflicts:
    hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
2015-03-03 02:23:00 -08:00
Akira Ajisaka f7a724ca9e HADOOP-10976. Backport "moving the source code of hadoop-tools docs to the directory under hadoop-tools" to branch-2. Contributed by Masatake Iwasaki.
(cherry picked from commit 9112f093cd)

Conflicts:
	hadoop-common-project/hadoop-common/CHANGES.txt
2015-02-28 17:06:58 -08:00
Akira Ajisaka 8aa0c73294 MAPREDUCE-5612. Add javadoc for TaskCompletionEvent.Status. Contributed by Chris Palmer.
(cherry picked from commit bfbf076b7d)
2015-02-26 15:06:26 -08:00