Apache/hadoop
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7,994 Commits 326 Branches 371 Tags
Commit Graph

524 Commits

Author SHA1 Message Date
Robert Kanter
909c96f054 MAPREDUCE-6622. Add capability to set JHS job cache to a task-based limit (rchiang via rkanter) 
(cherry picked from commit 0f72da7e281376f4fcbfbf3fb33f5d7fedcdb1aa)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-03-06 18:25:32 -08:00
Varun Vasudev
d32100d7fc MAPREDUCE-6635. Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException. Contributed by Junping Du. 
(cherry picked from commit c6f2d761d5430eac6b9f07f137a7028de4e0660c)
(cherry picked from commit f1999fe2754cbf11b138fb048c7486cab9b02c97)

 Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2016-02-23 14:08:14 +05:30
Jason Lowe
fc01ec866a MAPREDUCE-6621. Memory Leak in JobClient#submitJobInternal(). Contributed by Xuan Gong 
(cherry picked from commit 43e669b22d9ac0b81ab8cc397c5d6d92bac986c6)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
 2016-02-02 19:33:32 +00:00
Jason Lowe
cbc3288b55 MAPREDUCE-6554. MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory. Contributed by Bibin A Chundatt 
(cherry picked from commit 296ecbd5ec9f7d2eed84217a3318b4a3bcd7bd99)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
 2016-01-15 17:04:34 +00:00
Sangjin Lee
b2d02863a1 MAPREDUCE-6577. MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set (sjlee) 
(cherry picked from commit f6f16118d38fcfe3b724f05fad752cb223f441ec)
(cherry picked from commit 28fd4c70cab27c2872dc6a63ddee223ccf53f21d)
 2016-01-05 16:04:28 -08:00
Sangjin Lee
51a2e6304a Preparing for 2.6.4 development 2016-01-05 15:40:31 -08:00
Jason Lowe
53dccddfdb MAPREDUCE-6549. multibyte delimiters with LineRecordReader cause duplicate records (wilfreds via rkanter) 
(cherry picked from commit 7fd00b3db4b7d73afd41276ba9a06ec06a0e1762)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
 2015-11-30 17:51:37 +00:00
Jason Lowe
5074d836b0 MAPREDUCE-5883. "Total megabyte-seconds" in job counters is slightly misleading. Contributed by Nathan Roberts 
(cherry picked from commit cab3c7c8892ad33a7eb0955b01e99872ab95e192)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
 2015-11-24 22:03:39 +00:00
Jason Lowe
16e3dc24c5 MAPREDUCE-6481. LineRecordReader may give incomplete record and wrong position/key information for uncompressed input sometimes. Contributed by Zhihai Xu 
(cherry picked from commit 58d1a02b8d66b1d2a6ac2158be32bd35ad2e69bd)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
 2015-11-23 21:15:14 +00:00
Jason Lowe
e307078e78 MAPREDUCE-5948. org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record delimiters well. Contributed by Vinayakumar B, Rushabh Shah, and Akira AJISAKA 
(cherry picked from commit 077250d8d7b4b757543a39a6ce8bb6e3be356c6f)

Conflicts:

	hadoop-mapreduce-project/CHANGES.txt
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapred/TestLineRecordReader.java
	hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestLineRecordReader.java
 2015-11-23 21:07:39 +00:00
Sangjin Lee
6466ead9e0 Preparing for 2.6.3 development 2015-10-21 10:58:06 -07:00
Jason Lowe
4104ab346f MAPREDUCE-6334. Fetcher#copyMapOutput is leaking usedMemory upon IOException during InMemoryMapOutput shuffle handler. Contributed by Eric Payne 
(cherry picked from commit bc1bd7e5c4047b374420683d36a8c30eda6d75b6)
(cherry picked from commit 30d0f10458fbe0e3a85ea2e22a1bb3d4454fd896)
 2015-09-26 10:32:27 -07:00
Sangjin Lee
4cb7dbaead Preparing for 2.6.2 development: mvn versions:set -DnewVersion=2.6.2 2015-09-25 15:51:13 -07:00
Vinod Kumar Vavilapalli
3462a00dd2 Preparing for release 2.6.1: mvn versions:set -DnewVersion=2.6.1 2015-09-09 15:29:57 -07:00
Tsuyoshi Ozawa
6c0ad99b9d MAPREDUCE-6361. NPE issue in shuffle caused by concurrent issue between copySucceeded() in one thread and copyFailed() in another thread on the same host. Contributed by Junping Du. 
(cherry picked from commit f4e2b3cc0b1f4e49c306bc09a9dddd0495225bb2)
(cherry picked from commit fb5b0ebb459cc8812084090a7ce7ac29e2ad147c)
(cherry picked from commit 3a3dcf0bb689d6aa8bc524a31799067702814674)
 2015-09-03 11:46:47 -07:00
Jason Lowe
41601eae6b MAPREDUCE-5649. Reduce cannot use more than 2G memory for the final merge. Contributed by Gera Shegalov 
(cherry picked from commit 7dc3c1203d1ab14c09d0aaf0869a5bcdfafb0a5a)

(cherry picked from commit 87c2d915f1cc799cb4020c945c04d3ecb82ee963)
(cherry picked from commit 1da62ba736f5f161a18a52b7ca0d212786f3848c)
 2015-09-02 15:27:47 -07:00
Robert Kanter
c253ac68eb MAPREDUCE-6238. MR2 can't run local jobs with -libjars command options which is a regression from MR1 (zxu via rkanter) 
(cherry picked from commit d50e8f09287deeb51012d08e326a2ed71a6da869)
(cherry picked from commit 756c2542930756fef1cbff82056b418070f8d55f)
(cherry picked from commit 4f0fd74880e58958d89ca1cd4b7f665520595b2c)
 2015-09-02 11:40:32 -07:00
Karthik Kambatla
2f66dff7ce MAPREDUCE-6267. Refactor JobSubmitter#copyAndConfigureFiles into it's own class. (Chris Trezzo via kasha) 
(cherry picked from commit c66c3ac6bf9f63177279feec3f2917e4b882e2bc)
(cherry picked from commit f4d6c5e337e76dc408c9c8f19e306c3f4ba80d8e)
(cherry picked from commit 2b9cac26ec5457ad4e18ec418ff8249c37861eb8)
 2015-09-02 11:32:08 -07:00
Junping Du
8a9665a586 MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer. Contributed by Jason Lowe. 
(cherry picked from commit eccb7d46efbf07abcc6a01bd5e7d682f6815b824)

(cherry picked from commit cacadea632f7ab6fe4fdb1432e1a2c48e8ebd55f)
(cherry picked from commit 2abd4f61075739514fb3e63b118448895be02a30)
 2015-09-01 20:10:20 -07:00
Gera Shegalov
1931fa5f4b MAPREDUCE-6166. Reducers do not validate checksum of map outputs when fetching directly to disk. (Eric Payne via gera) 
(cherry picked from commit af006937e8ba82f98f468dc7375fe89c2e0a7912)
(cherry picked from commit 7c7bccfc31ede6f4afc043d81c8204046148c02e)
(cherry picked from commit 2e9dca584ba25809d9c6269e5f9326e09f55ed99)
 2015-08-30 19:50:43 -07:00
Tsuyoshi Ozawa
5b3d9bf636 MAPREDUCE-6237. Multiple mappers with DBInputFormat don't work because of reusing conections. Contributed by Kannan Rajah. 
(cherry picked from commit 241336ca2b7cf97d7e0bd84dbe0542b72f304dc9)

Conflicts:
	hadoop-mapreduce-project/CHANGES.txt
 2015-02-10 03:54:54 +09:00
Jason Lowe
9b383f6813 MAPREDUCE-6156. Fetcher - connect() doesn't handle connection refused correctly. Contributed by Junping Du 
(cherry picked from commit 177e8090f5809beb3ebcb656cd0affbb3f487de8)
 2014-11-13 15:44:00 +00:00
Arun C. Murthy
60584c732f Preparing to release hadoop-2.6.0: Set version in branch-2.6 to 2.6.1-SNAPSHOT. 2014-11-09 19:21:11 -08:00
Arun C. Murthy
85531834b3 HADOOP-11286. Copied LimitInputStream from guava-0.14 to hadoop to avoid issues with newer versions of guava in applications. Contributed by Christopher Tubbs. 
(cherry picked from commit 6caa8100d5d2547e34356dc279fd5e65b81a925a)
 2014-11-09 05:25:01 -08:00
Kihwal Lee
c38c5752e4 MAPREDUCE-5958. Wrong reduce task progress if map output is compressed. Contributed by Emilio Coppa and Jason Lowe. 
(cherry picked from commit 8f701ae07a0b1dc70b8e1eb8d4a5c35c0a1e76da)
 2014-11-06 15:56:50 -06:00
Jason Lowe
7be74ea09a MAPREDUCE-5960. JobSubmitter's check whether job.jar is local is incorrect with no authority in job jar path. Contributed by Gera Shegalov 
(cherry picked from commit 10f9f5101c44be7c675a44ded4aad212627ecdee)
 2014-11-06 15:12:37 +00:00
Zhijie Shen
7180167d01 MAPREDUCE-6052. Supported overriding the default container-log4j.properties file per job. Contributed by Junping Du. 
(cherry picked from commit ed63b116465290fdb0acdf89170025f47b307599)
 2014-11-01 00:50:32 -07:00
Karthik Kambatla
d8212c0b7e HADOOP-11243. SSLFactory shouldn't allow SSLv3. (Wei Yan via kasha) 
(cherry picked from commit 3c5f5af1184e85158dec962df0b0bc2be8d0d1e3)
 2014-10-28 18:10:06 -07:00
Zhijie Shen
d92d4e32f2 MAPREDUCE-6018. Added an MR specific config to enable emitting job history data to the timeline server. Contributed by Robert Kanter. 
(cherry picked from commit 971e91c8c03a23e4613ed3f071b4f982ee5a1b63)
 2014-10-27 21:09:20 -07:00
Jian He
29d0164ee7 YARN-2198. Remove the need to run NodeManager as privileged account for Windows Secure Container Executor. Contributed by Remus Rusanu 
(cherry picked from commit 3b12fd6cfbf4cc91ef8e8616c7aafa9de006cde5)

Conflicts:
	hadoop-yarn-project/CHANGES.txt
 2014-10-22 16:22:08 -07:00
Jian He
bd45dba807 YARN-2312. Deprecated old ContainerId#getId API and updated MapReduce to use ContainerId#getContainerId instead. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit f93d2ea27e7554272845fb11d97b3564704834b3)
 2014-10-15 15:31:57 -07:00
Jason Lowe
ec38eec165 MAPREDUCE-5970. Provide a boolean switch to enable MR-AM profiling. Contributed by Gera Shegalov 
(cherry picked from commit f19771a24c2f90982cf6dec35889836a6146c968)
 2014-10-15 17:52:12 +00:00
Jason Lowe
9203ea7ea9 MAPREDUCE-5873. Shuffle bandwidth computation includes time spent waiting for maps. Contributed by Siqi Li 
(cherry picked from commit b9edad64034a9c8a121ec2b37792c190ba561e26)
 2014-10-15 15:59:56 +00:00
cnauroth
3fe4e0d3df MAPREDUCE-6122. TestLineRecordReader may fail due to test data files checked out of git with incorrect line endings. Contributed by Chris Nauroth. 
(cherry picked from commit 684170d5efdc11bcd9967168f44a428d1a72af3c)
(cherry picked from commit 85a2b907407c7719a31c64d3747810263b3ec57d)
 2014-10-09 22:30:25 -07:00
subru
256a951049 MAPREDUCE-6103.Adding reservation APIs to MR resource manager delegate. Contributed by Subru Krishnan and Carlo Curino. 
(cherry picked from commit aa92dd45f2d8c89a8a17ad2e4449aa3ff08bc53a)
(cherry picked from commit 3f282762d1afc916de9207d3adeda852ca344853)
(cherry picked from commit 30a370e70504a4cc3222da0dc706c871fcebfa78)
 2014-10-06 10:29:13 -07:00
Jian He
a6879b606e MAPREDUCE-6087. Fixed wrong config name of MRJobConfig#MR_CLIENT_TO_AM_IPC_MAX_RETRIES_ON_TIMEOUTS. Contributed by Akira AJISAKA 
(cherry picked from commit 5f16c98ad66b562760d5d947151a38f14875d21a)
 2014-09-26 18:06:19 -07:00
Allen Wittenauer
e5d284664b MAPREDUCE-5796. Use current version of the archive name in DistributedCacheDeploy document (Akira AJISAKA via aw) 2014-09-26 12:59:59 -07:00
Allen Wittenauer
fe8222998f MAPREDUCE-6073. Description of mapreduce.job.speculative.slowtaskthreshold in mapred-default should be moved into description tags (Tsuyoshi OZAWA via aw) 2014-09-26 11:17:11 -07:00
Zhijie Shen
7b3ecc547f MAPREDUCE-5831. Make MR client ignore unknown counters received from AM. Contributed by Junping Du. 
(cherry picked from commit 662fc11ae7ccea9d05dfe54b2a794bb08cfd70a4)
 2014-09-25 23:46:57 -07:00
Allen Wittenauer
7ffb636a54 MAPREDUCE-6093. minor distcp doc edits (Charles Lamb via aw) 2014-09-25 11:33:01 -07:00
Karthik Kambatla
372bf54072 MAPREDUCE-6086. mapreduce.job.credentials.binary should allow all URIs. (Zhihai Xu via kasha) 
(cherry picked from commit 52945a33cc3fcb3b961cce0e7e3ca01291f2a223)
 2014-09-18 15:47:21 -07:00
Jason Lowe
44c7113133 MAPREDUCE-5891. Improved shuffle error handling across NM restarts. Contributed by Junping Du 
(cherry picked from commit 2c3da25fd718b3a9c1ed67f05b577975ae613f4e)
 2014-09-18 22:01:42 +00:00
Andrew Wang
5bc71071d1 HADOOP-11106. Document considerations of HAR and Encryption. (clamb via wang) 
(cherry picked from commit fe2f54d4cbd4b02bc3b8f642d0689b17dbfe169b)
 2014-09-18 14:51:27 -07:00
Jason Lowe
6f8abe7efe MAPREDUCE-6070. yarn.app.am.resource.mb/cpu-vcores affects uber mode but is not documented. Contributed by Tsuyoshi OZAWA 
(cherry picked from commit 9e28e089c6bcb0b264751bdfa1f994ce002ef2de)
 2014-09-11 16:20:51 +00:00
Chris Douglas
69ee11dc93 MAPREDUCE-6063. Correct spill size calculation for spills wrapping the circular buffer. Contributed by zhihai xu. 2014-09-03 17:02:17 -07:00
Andrew Wang
075bb9e380 HADOOP-11021. Configurable replication factor in the hadoop archive command. Contributed by Zhe Zhang. 
(cherry picked from commit ea1c6f31c2d2ea5b38ed57e2aa241d122103a721)
 2014-08-29 14:47:28 -07:00
Chris Douglas
4f95017602 Fix typos in log messages. Contributed by Ray Chiang 2014-08-28 16:30:13 -07:00
Alejandro Abdelnur
c77bd85b62 HDFS-6134 and HADOOP-10150 subtasks. 2014-08-28 15:03:08 -07:00
Sanford Ryza
c38e751bca MAPREDUCE-5130. Fix mapred-default.xml after bad merge. 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619930 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-22 20:58:12 +00:00
Sanford Ryza
53da85efaf MAPREDUCE-5130. Add missing job config options to mapred-default.xml (Ray Chiang via Sandy Ryza) 
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/branches/branch-2@1619631 13f79535-47bb-0310-9956-ffa450edef68
 2014-08-21 23:54:38 +00:00