Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
11,947 Commits 326 Branches 371 Tags
Commit Graph

671 Commits

Author SHA1 Message Date
Junping Du
5f36b62265 Revert "MAPREDUCE-5875. Make Counter limits consistent across JobClient, MRAppMaster, and YarnChild. (Gera Shegalov via kasha)" 
This reverts commit 7bfd9e068d205f752c05d51778a4a4702329d31e.

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/JobSubmitter.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/jobhistory/HistoryViewer.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/main/java/org/apache/hadoop/mapreduce/v2/hs/CompletedJob.java
 2016-12-28 17:37:20 -08:00
Junping Du
8755a6f5ac Revert "MAPREDUCE-6199. AbstractCounters are not reset completely on deserialization (adhoot via rkanter)" 
This reverts commit f9341c1e2cffc4e3ac37cb7da4e535ca5c6d3913.

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-12-28 17:13:04 -08:00
Junping Du
e8d7801319 Revert "MAPREDUCE-6286. A typo in HistoryViewer makes some code useless, which causes counter limits are not reset correctly. Contributed by Zhihai Xu." 
This reverts commit 7c72c7f5290f0991f65102c2bd413f965a8ca409.

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-12-28 17:05:17 -08:00
Akira Ajisaka
0c58241b22 MAPREDUCE-6820. Fix dead links in Job relevant classes. Contributed by Yiqun Lin. 
(cherry picked from commit 92a8917ca5a25bf6bee75acbb554aae87766a7e7)
(cherry picked from commit 1e1ab9bc1bde48593eceafb7380d7501d4bbc200)
 2016-12-10 10:57:33 +09:00
Akira Ajisaka
61128c50de MAPREDUCE-6795. Update the document for JobConf#setNumReduceTasks. Contributed by Yiqun Lin. 
(cherry picked from commit 235f39b7c44c5907bd4de063892905577c35cd80)
(cherry picked from commit a3897f77aefaf0a9823b31a1d3e8f4d1eded8642)
 2016-11-01 20:52:52 +09:00
Akira Ajisaka
757b38ea43 MAPREDUCE-6799. Document mapreduce.jobhistory.webapp.https.address in mapred-default.xml. Contributed by Yiqun Lin. 
(cherry picked from commit f646fe34be704a22cbea322cce2826f8f201bd34)
(cherry picked from commit ef13a5ef9d7859cbb720ee46ebed8172358ad0d5)
 2016-10-31 15:19:37 +09:00
Jason Lowe
0a405c4f71 MAPREDUCE-6724. Single shuffle to memory must not exceed Integer#MAX_VALUE. (Haibo Chen via gera) 
(cherry picked from commit 6890d5b472320fa7592ed1b08b623c55a27089c6)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMergeManager.java
 2016-10-28 14:40:28 +00:00
Naganarasimha
8a664eba7d MAPREDUCE-2631. Potential resource leaks in BinaryProtocol$TeeOutputStream.java. Contributed by Sunil G. 
(cherry picked from commit 28660f51af161a9fa301523d96a6f8ae4ebd6edd)
 2016-10-28 06:04:03 +05:30
Karthik Kambatla
c955a59d63 MAPREDUCE-6740. Enforce mapreduce.task.timeout to be at least mapreduce.task.progress-report.interval. (Haibo Chen via kasha) 2016-10-05 23:12:45 -07:00
Jason Lowe
f1b74a3d9f MAPREDUCE-6741. Add MR support to redact job conf properties. Contributed by Haibo Chen 2016-10-05 15:24:10 +00:00
Jason Lowe
00c8ab91ba MAPREDUCE-6761. Regression when handling providers - invalid configuration ServiceConfiguration causes Cluster initialization failure. Contributed by Peter Vary 
(cherry picked from commit 6fa9bf4407a0b09be8ce4e52d2fde89adba34f1a)
 2016-08-24 14:42:53 +00:00
Akira Ajisaka
854af9cee1 MAPREDUCE-6587. Remove unused params in connection-related methods of Fetcher. Contributed by Yiqun Lin. 
(cherry picked from commit 8cc4a67059e37b2083cd5468b35a64a403a3e3ae)
(cherry picked from commit 41fe85f8688e6edd2a94ffd831d776e7c7a3614f)
 2016-08-23 17:12:32 +09:00
Wangda Tan
e2bd490236 MAPREDUCE-6310. Add jdiff support to MapReduce. (Li Lu/vinodkv via wangda) 
(cherry picked from commit 3d937457ee60d617d71b36bb6a29344e771670b4)
(cherry picked from commit 65822e55fd99511ab0c39f802d3ec58290445f9d)
 2016-08-19 16:49:54 -07:00
Varun Saxena
6f1a9f1b6a MAPREDUCE-6763. Shuffle server listen queue is too small (Jason Lowe via Varun Saxena) 2016-08-20 04:08:55 +05:30
Akira Ajisaka
59693ebbd5 MAPREDUCE-6730. Use StandardCharsets instead of String overload in TextOutputFormat. Contributed by Sahil Kang. 
This closes #114

(cherry picked from commit 70c278115249898132490a89a548fd936c09f54b)
(cherry picked from commit e54de94e5ab40405d837ee7664e90921f5bc38e4)
 2016-08-04 20:18:37 +09:00
Akira Ajisaka
19062df92f MAPREDUCE-6746. Replace org.apache.commons.io.Charsets with java.nio.charset.StandardCharsets. Contributed by Vincent Poon. 
(cherry picked from commit 3f0bffea2d35083c45acee79338e10b608374d8e)
(cherry picked from commit 0fc838540ba74a12336191388661e41fc3b77928)
 2016-07-30 11:47:33 +09:00
Robert Kanter
ba83b62d36 MAPREDUCE-6625. TestCLI#testGetJob fails occasionally (haibochen via rkanter) 
(cherry picked from commit ad256beb50e8851d5e6e5e743b00e819e75e3549)
 2016-07-12 16:31:24 -07:00
Akira Ajisaka
5522a46f4c HADOOP-13320. Fix arguments check in documentation for WordCount v2.0. Contributed by niccolo becchi. 
(cherry picked from commit 9d46a49c746b9e1ef552dbb10d1e22f87db68c76)
(cherry picked from commit b3b81b4f0e1cd0b37ab2dfdf3e7301e0ef3e0676)
 2016-07-07 15:32:20 -07:00
Akira Ajisaka
a0f6109907 MAPREDUCE-6542. HistoryViewer uses SimpleDateFormat, but SimpleDateFormat is not threadsafe. Contributed by zhangyubiao. 2016-06-27 15:19:21 -07:00
Akira Ajisaka
ae5dc10ecd MAPREDUCE-6725. Javadoc for CLI#listEvents() contains no-existent param. Contributed by shenyinjie. 
(cherry picked from commit 0f23cd497b0baa0899c7565cd94e2f27dbc7c0ff)
(cherry picked from commit 2830ee5f0fdbdf8123d0dd0858baf2c059a3f96f)
 2016-06-25 00:27:35 +09:00
Wangda Tan
93b406b66d YARN-4844. Addendum patch, revert few changes to public and evolving APIs to avoid breaking downstream projects as much as possible. 
(cherry picked from commit 7eb524ddee4af80c87b134070945977bf73e950f)
 2016-06-16 11:10:46 -07:00
Wangda Tan
d838c6443d YARN-1942. Deprecate toString/fromString methods from ConverterUtils and move them to records classes like ContainerId/ApplicationId, etc. (wangda) 2016-06-14 15:21:41 -07:00
Akira Ajisaka
3bea925cea MAPREDUCE-6741. Refactor UncompressedSplitLineReader.fillBuffer(). Contributed by Daniel Templeton. 
(cherry picked from commit 0b7b8a377611b2a3041a2995504a437c36dfa6e6)
(cherry picked from commit db2d96b378d6a4ba9a504b4daaca5d8ff6f5f227)
 2016-06-10 19:17:26 +09:00
Wangda Tan
19e578870d YARN-4844. Add getMemorySize/getVirtualCoresSize to o.a.h.y.api.records.Resource. (wangda) 2016-06-07 12:41:50 -07:00
Wangda Tan
2f3e1d965d Revert "YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan." 
This reverts commit 457884737f75c796413ce860b1859a31cc5292ca.
 2016-05-31 22:16:53 -07:00
Varun Vasudev
457884737f YARn-4844. Add getMemoryLong/getVirtualCoreLong to o.a.h.y.api.records.Resource. Contributed by Wangda Tan. 2016-05-29 20:57:56 +05:30
Jason Lowe
ff8caccc87 MAPREDUCE-6558. multibyte delimiters with compressed input files generate duplicate records. Contributed by Wilfred Spiegelenburg 
(cherry picked from commit 9227dfc25f373a99cb66ad7d6bacef8dcf336f77)
 2016-05-13 14:36:13 +00:00
Steve Loughran
9b8f7a8fb2 MAPREDUCE-6639 Process hangs in LocatedFileStatusFetcher if FileSystem.get throws. Ryan Blue via stevel 2016-05-12 18:59:56 +01:00
Eric Payne
3de0ff030c MAPREDUCE-6633. AM should retry map attempts if the reduce task encounters commpression related errors. Contributed by Rushabh Shah 
(cherry picked from commit 1fec06e037d2b22dafc64f33d4f1231bef4ceba8)
 2016-04-09 18:29:44 +00:00
Wangda Tan
2d9f42d685 MAPREDUCE-6579. JobStatus#getFailureInfo should not output diagnostic information when the job is running. (Akira AJISAKA via wangda) 
(cherry picked from commit 6529c87551e07e9da1e1b273eff4c6bad6b3e838)
 2016-03-15 17:32:32 -07:00
Harsh J
5f059e03f5 MAPREDUCE-6648. Add yarn.app.mapreduce.am.log.level to mapred-default.xml (harsh) 2016-03-07 13:13:43 +05:30
Robert Kanter
3d8ae9c577 MAPREDUCE-6622. Add capability to set JHS job cache to a task-based limit (rchiang via rkanter) 
(cherry picked from commit 0f72da7e281376f4fcbfbf3fb33f5d7fedcdb1aa)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-03-06 19:26:12 -08:00
Masatake Iwasaki
9d551a3adf HADOOP-12470. In-page TOC of documentation should be automatically generated by doxia macro (iwasakims) 
(cherry picked from commit cbd31328a61252f253aa2bcdb9cad70f7876d231)

 Conflicts:
	hadoop-common-project/hadoop-common/src/site/markdown/CommandsManual.md
	hadoop-common-project/hadoop-common/src/site/markdown/UnixShellGuide.md

(cherry picked from commit 722182fdfbb654be655fbca42f34bba5a05c2f6d)
 2016-03-04 14:13:30 +09:00
Varun Vasudev
cb4f5ebd3e MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du. 
(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)
(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)
 2016-02-23 13:07:40 +05:30
Andrew Wang
d31660ef4a MAPREDUCE-6637. Testcase Failure : TestFileInputFormat.testSplitLocationInfo. Contributed by Brahma Reddy Battula. 
(cherry picked from commit 37577852ba54ba506b0b6c4db5e95d176f5e9b63)
(cherry picked from commit 7f6737951a124480af2f3fff9358865fff3ed091)
 2016-02-19 16:33:44 -08:00
Tsuyoshi Ozawa
7ccb0e0601 MAPREDUCE-6341. addendum patch to fix typos. Contributed by John Michael Luy. 
(cherry picked from commit 453e7e0d1091cf795152c8992c6c5cbc772d80d6)
 2016-02-17 14:18:24 +09:00
Akira Ajisaka
f943bf0e22 MAPREDUCE-6626. Reuse ObjectMapper instance in MapReduce. Contributed by Lin Yiqun. 
(cherry picked from commit 10393d9a666419a5d8dc0cfb48da96e9c690c11e)
(cherry picked from commit 8cbe65a07dd0deb150e91b13e3a1cec13e3716ee)
 2016-02-10 03:05:23 +09:00
Jason Lowe
1d322ca5cc MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). Contributed by Xuan Gong 
(cherry picked from commit 43e669b22d9ac0b81ab8cc397c5d6d92bac986c6)
 2016-02-02 19:32:04 +00:00
Akira Ajisaka
b85b8d90d1 MAPREDUCE-6605. Fix typos mapreduce.map.skip.proc.count.autoincr and mapreduce.reduce.skip.proc.count.autoincr in mapred-default.xml. Contributed by Kai Sasaki. 
(cherry picked from commit 7f215ffdd5219fc585ec484073b317a940895a60)
 2016-01-22 18:38:08 +09:00
Jason Lowe
de31bff67d MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory. Contributed by Bibin A Chundatt 
(cherry picked from commit 9fbd579ab50f051c8ad63d316ca5a3e24822dcdd)
 2016-01-15 16:54:03 +00:00
Akira Ajisaka
6816481c72 MAPREDUCE-6601. Fix typo in Job#setUseNewAPI. Contributed by Kai Sasaki. 
(cherry picked from commit 5cc44d18aa946ce2a7a9ffbd7bcce57b07e9e1cb)
(cherry picked from commit 7c4be1599152b29d329b4e61c21a935aa234d5a9)
 2016-01-15 00:44:52 +09:00
Jason Lowe
a22a7f61e4 MAPREDUCE-6473. Job submission can take a long time during Cluster initialization. Contributed by Kuhu Shukla 
(cherry picked from commit f657b54281d517c69df0a23b97b47befa77d1dc2)
 2016-01-12 23:59:06 +00:00
Junping Du
d7e10f8174 MAPREDUCE-6068. Illegal progress value warnings in map tasks. Contributed by Binglin Chang. 
(cherry picked from commit 8c1adeaa26a7eaaca891dccdb71ff085e598a7de)
(cherry picked from commit 51073c910ccdfaa07f7b8dab290617b56e4d86b7)
 2016-01-11 06:14:58 -08:00
Sangjin Lee
ae535ec93e MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee) 
(cherry picked from commit f6f16118d38fcfe3b724f05fad752cb223f441ec)
(cherry picked from commit 28fd4c70cab27c2872dc6a63ddee223ccf53f21d)
 2016-01-06 08:44:37 -08:00
Akira Ajisaka
2139458a98 MAPREDUCE-6589. TestTaskLog outputs a log under directory other than target/test-dir. (aajisaka) 
(cherry picked from commit 2a59d01503ed547bac15443fc273b5e04a2839d9)
(cherry picked from commit a8314505d0c7fa4a3748bef4c156095bdfdd3fc0)
 2015-12-30 01:34:26 +09:00
Akira Ajisaka
7063d8d013 MAPREDUCE-6584. Remove trailing whitespaces from mapred-default.xml. (aajisaka) 
(cherry picked from commit 1a06949126df84ffed703547a88beee5ff0bec83)
 2015-12-30 01:04:40 +09:00
Akira Ajisaka
5f34cbff5c MAPREDUCE-6583. Clarify confusing sentence in MapReduce tutorial document. Contributed by Kai Sasaki. 
(cherry picked from commit 7995a6ea4dc524e5b17606359d09df72d771224a)
(cherry picked from commit 8607cb6074b40733d8990618a44c490f9f303ae3)
 2015-12-21 00:18:40 +09:00
Xuan
93c483889c MAPREDUCE-6566. Add retry support to mapreduce CLI tool. Contributed by Varun Vasudev 
(cherry picked from commit d4e766de931c1cc478561a211215c517ba1f341c)
(cherry picked from commit 5891d6ed55fdc73da6afa8517ebe9f4d2b666473)
 2015-12-16 15:31:09 -08:00
Robert Kanter
8c6273d117 MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 
(cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)
 2015-11-25 17:03:52 -08:00
Jason Lowe
2f18218508 MAPREDUCE-5870. Support for passing Job priority through Application Submission Context in Mapreduce Side. Contributed by Sunil G 
(cherry picked from commit f634505d48d97e4d461980d68a0cbdf87223646d)

Conflicts:

	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestJobConf.java
 2015-11-24 22:15:37 +00:00