Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,586 Commits 326 Branches 371 Tags
Commit Graph

661 Commits

Author SHA1 Message Date
Jason Lowe
00c8ab91ba MAPREDUCE-6761. Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure. Contributed by Peter Vary 
(cherry picked from commit 6fa9bf4407a0b09be8ce4e52d2fde89adba34f1a)
 2016-08-24 14:42:53 +00:00
Akira Ajisaka
854af9cee1 MAPREDUCE-6587. Remove unused params in connection-related methods of Fetcher. Contributed by Yiqun Lin. 
(cherry picked from commit 8cc4a67059e37b2083cd5468b35a64a403a3e3ae)
(cherry picked from commit 41fe85f8688e6edd2a94ffd831d776e7c7a3614f)
 2016-08-23 17:12:32 +09:00
Wangda Tan
e2bd490236 MAPREDUCE-6310. Add jdiff support to MapReduce. (Li Lu/vinodkv via wangda) 
(cherry picked from commit 3d937457ee60d617d71b36bb6a29344e771670b4)
(cherry picked from commit 65822e55fd99511ab0c39f802d3ec58290445f9d)
 2016-08-19 16:49:54 -07:00
Varun Saxena
6f1a9f1b6a MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena) 2016-08-20 04:08:55 +05:30
Akira Ajisaka
59693ebbd5 MAPREDUCE-6730. Use StandardCharsets instead of String overload in TextOutputFormat. Contributed by Sahil Kang. 
This closes #114

(cherry picked from commit 70c278115249898132490a89a548fd936c09f54b)
(cherry picked from commit e54de94e5ab40405d837ee7664e90921f5bc38e4)
 2016-08-04 20:18:37 +09:00
Akira Ajisaka
19062df92f MAPREDUCE-6746. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 
(cherry picked from commit 3f0bffea2d35083c45acee79338e10b608374d8e)
(cherry picked from commit 0fc838540ba74a12336191388661e41fc3b77928)
 2016-07-30 11:47:33 +09:00
Robert Kanter
ba83b62d36 MAPREDUCE-6625. TestCLI#testGetJob fails occasionally (haibochen via rkanter) 
(cherry picked from commit ad256beb50e8851d5e6e5e743b00e819e75e3549)
 2016-07-12 16:31:24 -07:00
Akira Ajisaka
5522a46f4c HADOOP-13320. Fix arguments check in documentation for WordCount v2.0. Contributed by niccolo becchi. 
(cherry picked from commit 9d46a49c746b9e1ef552dbb10d1e22f87db68c76)
(cherry picked from commit b3b81b4f0e1cd0b37ab2dfdf3e7301e0ef3e0676)
 2016-07-07 15:32:20 -07:00
Akira Ajisaka
a0f6109907 MAPREDUCE-6542. HistoryViewer uses SimpleDateFormat, but SimpleDateFormat is not threadsafe. Contributed by zhangyubiao. 2016-06-27 15:19:21 -07:00
Akira Ajisaka
ae5dc10ecd MAPREDUCE-6725. Javadoc for CLI#listEvents() contains no-existent param. Contributed by shenyinjie. 
(cherry picked from commit 0f23cd497b0baa0899c7565cd94e2f27dbc7c0ff)
(cherry picked from commit 2830ee5f0fdbdf8123d0dd0858baf2c059a3f96f)
 2016-06-25 00:27:35 +09:00
Wangda Tan
93b406b66d YARN-4844. Addendum patch, revert few changes to public and evolving APIs to avoid breaking downstream projects as much as possible. 
(cherry picked from commit 7eb524ddee4af80c87b134070945977bf73e950f)
 2016-06-16 11:10:46 -07:00
Wangda Tan
d838c6443d YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:21:41 -07:00
Akira Ajisaka
3bea925cea MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton. 
(cherry picked from commit 0b7b8a377611b2a3041a2995504a437c36dfa6e6)
(cherry picked from commit db2d96b378d6a4ba9a504b4daaca5d8ff6f5f227)
 2016-06-10 19:17:26 +09:00
Wangda Tan
19e578870d YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. (wangda) 2016-06-07 12:41:50 -07:00
Wangda Tan
2f3e1d965d Revert "YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan." 
This reverts commit 457884737f75c796413ce860b1859a31cc5292ca.
 2016-05-31 22:16:53 -07:00
Varun Vasudev
457884737f YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:57:56 +05:30
Jason Lowe
ff8caccc87 MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)
 2016-05-13 14:36:13 +00:00
Steve Loughran
9b8f7a8fb2 MAPREDUCE-6639 Process hangs in LocatedFileStatusFetcher if FileSystem.get throws. Ryan Blue via stevel 2016-05-12 18:59:56 +01:00
Eric Payne
3de0ff030c MAPREDUCE-6633. AM should retry map attempts if the reduce task encounters commpression related errors. Contributed by Rushabh Shah 
(cherry picked from commit 1fec06e037d2b22dafc64f33d4f1231bef4ceba8)
 2016-04-09 18:29:44 +00:00
Wangda Tan
2d9f42d685 MAPREDUCE-6579. JobStatus#getFailureInfo should not output diagnostic information when the job is running. (Akira AJISAKA via wangda) 
(cherry picked from commit 6529c87551e07e9da1e1b273eff4c6bad6b3e838)
 2016-03-15 17:32:32 -07:00
Harsh J
5f059e03f5 MAPREDUCE-6648. Add yarn.app.mapreduce.am.log.level to mapred-default.xml (harsh) 2016-03-07 13:13:43 +05:30
Robert Kanter
3d8ae9c577 MAPREDUCE-6622. Add capability to set JHS job cache to a task-based limit (rchiang via rkanter) 
(cherry picked from commit 0f72da7e281376f4fcbfbf3fb33f5d7fedcdb1aa)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-03-06 19:26:12 -08:00
Masatake Iwasaki
9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a61252f253aa2bcdb9cad70f7876d231)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfbb654be655fbca42f34bba5a05c2f6d)
 2016-03-04 14:13:30 +09:00
Varun Vasudev
cb4f5ebd3e MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du. 
(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)
(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)
 2016-02-23 13:07:40 +05:30
Andrew Wang
d31660ef4a MAPREDUCE-6637. Testcase Failure : TestFileInputFormat.testSplitLocationInfo. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 37577852ba54ba506b0b6c4db5e95d176f5e9b63)
(cherry picked from commit 7f6737951a124480af2f3fff9358865fff3ed091)
 2016-02-19 16:33:44 -08:00
Tsuyoshi Ozawa
7ccb0e0601 MAPREDUCE-6341. addendum patch to fix typos. Contributed by John Michael Luy. 
(cherry picked from commit 453e7e0d1091cf795152c8992c6c5cbc772d80d6)
 2016-02-17 14:18:24 +09:00
Akira Ajisaka
f943bf0e22 MAPREDUCE-6626. Reuse ObjectMapper instance in MapReduce. Contributed by Lin Yiqun. 
(cherry picked from commit 10393d9a666419a5d8dc0cfb48da96e9c690c11e)
(cherry picked from commit 8cbe65a07dd0deb150e91b13e3a1cec13e3716ee)
 2016-02-10 03:05:23 +09:00
Jason Lowe
1d322ca5cc MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). Contributed by Xuan Gong 
(cherry picked from commit 43e669b22d9ac0b81ab8cc397c5d6d92bac986c6)
 2016-02-02 19:32:04 +00:00
Akira Ajisaka
b85b8d90d1 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 
(cherry picked from commit 7f215ffdd5219fc585ec484073b317a940895a60)
 2016-01-22 18:38:08 +09:00
Jason Lowe
de31bff67d MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory. Contributed by Bibin A Chundatt 
(cherry picked from commit 9fbd579ab50f051c8ad63d316ca5a3e24822dcdd)
 2016-01-15 16:54:03 +00:00
Akira Ajisaka
6816481c72 MAPREDUCE-6601. Fix typo in Job#setUseNewAPI. Contributed by Kai Sasaki. 
(cherry picked from commit 5cc44d18aa946ce2a7a9ffbd7bcce57b07e9e1cb)
(cherry picked from commit 7c4be1599152b29d329b4e61c21a935aa234d5a9)
 2016-01-15 00:44:52 +09:00
Jason Lowe
a22a7f61e4 MAPREDUCE-6473. Job submission can take a long time during Cluster initialization. Contributed by Kuhu Shukla 
(cherry picked from commit f657b54281d517c69df0a23b97b47befa77d1dc2)
 2016-01-12 23:59:06 +00:00
Junping Du
d7e10f8174 MAPREDUCE-6068. Illegal progress value warnings in map tasks. Contributed by Binglin Chang. 
(cherry picked from commit 8c1adeaa26a7eaaca891dccdb71ff085e598a7de)
(cherry picked from commit 51073c910ccdfaa07f7b8dab290617b56e4d86b7)
 2016-01-11 06:14:58 -08:00
Sangjin Lee
ae535ec93e MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee) 
(cherry picked from commit f6f16118d38fcfe3b724f05fad752cb223f441ec)
(cherry picked from commit 28fd4c70cab27c2872dc6a63ddee223ccf53f21d)
 2016-01-06 08:44:37 -08:00
Akira Ajisaka
2139458a98 MAPREDUCE-6589. TestTaskLog outputs a log under directory other than target/test-dir. (aajisaka) 
(cherry picked from commit 2a59d01503ed547bac15443fc273b5e04a2839d9)
(cherry picked from commit a8314505d0c7fa4a3748bef4c156095bdfdd3fc0)
 2015-12-30 01:34:26 +09:00
Akira Ajisaka
7063d8d013 MAPREDUCE-6584. Remove trailing whitespaces from mapred-default.xml. (aajisaka) 
(cherry picked from commit 1a06949126df84ffed703547a88beee5ff0bec83)
 2015-12-30 01:04:40 +09:00
Akira Ajisaka
5f34cbff5c MAPREDUCE-6583. Clarify confusing sentence in MapReduce tutorial document. Contributed by Kai Sasaki. 
(cherry picked from commit 7995a6ea4dc524e5b17606359d09df72d771224a)
(cherry picked from commit 8607cb6074b40733d8990618a44c490f9f303ae3)
 2015-12-21 00:18:40 +09:00
Xuan
93c483889c MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev 
(cherry picked from commit d4e766de931c1cc478561a211215c517ba1f341c)
(cherry picked from commit 5891d6ed55fdc73da6afa8517ebe9f4d2b666473)
 2015-12-16 15:31:09 -08:00
Robert Kanter
8c6273d117 MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 
(cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)
 2015-11-25 17:03:52 -08:00
Jason Lowe
2f18218508 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G 
(cherry picked from commit f634505d48d97e4d461980d68a0cbdf87223646d)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
 2015-11-24 22:15:37 +00:00
Jason Lowe
ef3e01a1e4 MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly misleading. Contributed by Nathan Roberts 
(cherry picked from commit cab3c7c8892ad33a7eb0955b01e99872ab95e192)
 2015-11-24 22:01:53 +00:00
Jian He
f562c155d2 MAPREDUCE-5485. Allow repeating job commit by extending OutputCommitter API. Contributed by Junping Du 2015-11-16 17:08:22 -08:00
Jason Lowe
ee5c20ddef MAPREDUCE-6533. testDetermineCacheVisibilities of TestClientDistributedCacheManager is broken. Contributed by Chang Li 
(cherry picked from commit df68eac825c2b64b2b2dc2193134afaf2a1d9076)
 2015-11-11 17:27:03 +00:00
Jason Lowe
145058003a MAPREDUCE-6515. Update Application priority in AM side from AM-RM heartbeat. Contributed by Sunil G 
(cherry picked from commit cf953b6258b036fc482456b4591cfb98054f48f2)
 2015-10-29 18:05:56 +00:00
Jason Lowe
be72ed1c75 MAPREDUCE-6489. Fail fast rogue tasks that write too much to local disk. Contributed by Maysam Yabandeh 2015-10-21 14:11:48 +00:00
Robert Kanter
492142097b MAPREDUCE-6495. Docs for archive-logs tool (rkanter) 
(cherry picked from commit 0c4af0f99811a7138954391df3761aef9ff09155)
 2015-10-20 17:34:56 -07:00
Karthik Kambatla
2d8f9e3fd9 MAPREDUCE-6302. Incorrect headroom can lead to a deadlock between map and reduce allocations. (kasha) 
(cherry picked from commit 4aa9b3e75ca86917125e56e1b438668273a5d87f)

Conflicts:
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/rm/RMContainerAllocator.java
 2015-10-09 07:54:47 -07:00
Akira Ajisaka
22ff37f5ce MAPREDUCE-6479. Add missing mapred job command options in mapreduce document. Contributed by nijel. 
(cherry picked from commit 584cf3bb35b96749f711b9e7145cf3fb71897cd9)
 2015-10-09 10:58:15 +09:00
Zhihai Xu
e5ba1a0521 MAPREDUCE-6484. Yarn Client uses local address instead of RM address as token renewer in a secure cluster when RM HA is enabled. Contributed by Zhihai Xu 
(cherry picked from commit 97a08807ec328a6f455d941fe3d9a469d885caa6)
 2015-09-24 08:14:11 -07:00
Wangda Tan
6041fc75d8 MAPREDUCE-6478. Add an option to skip cleanupJob stage or ignore cleanup failure during commitJob. (Junping Du via wangda) 2015-09-18 10:19:32 -07:00