HADOOP-6812. Change documentation for correct placement of configuration variables: mapreduce.reduce.input.buffer.percent, mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb. Contributed by Chris Douglas.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1064917 13f79535-47bb-0310-9956-ffa450edef68
This commit is contained in:
parent
c60feead5e
commit
077bc4af3c
|
@ -268,6 +268,11 @@ Release 0.22.0 - Unreleased
|
||||||
|
|
||||||
HADOOP-7110. Implement chmod with JNI. (todd)
|
HADOOP-7110. Implement chmod with JNI. (todd)
|
||||||
|
|
||||||
|
HADOOP-6812. Change documentation for correct placement of configuration
|
||||||
|
variables: mapreduce.reduce.input.buffer.percent,
|
||||||
|
mapreduce.task.io.sort.factor, mapreduce.task.io.sort.mb
|
||||||
|
(Chris Douglas via shv)
|
||||||
|
|
||||||
OPTIMIZATIONS
|
OPTIMIZATIONS
|
||||||
|
|
||||||
HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
|
HADOOP-6884. Add LOG.isDebugEnabled() guard for each LOG.debug(..).
|
||||||
|
|
|
@ -666,22 +666,33 @@
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>conf/core-site.xml</td>
|
<td>conf/mapred-site.xml</td>
|
||||||
<td>fs.inmemory.size.mb</td>
|
<td>mapreduce.reduce.shuffle.input.buffer.percent</td>
|
||||||
<td>200</td>
|
<td>0.80</td>
|
||||||
<td>
|
<td>
|
||||||
Larger amount of memory allocated for the in-memory
|
Larger amount of memory allocated for merging map output
|
||||||
file-system used to merge map-outputs at the reduces.
|
in memory during the shuffle. Expressed as a fraction of
|
||||||
|
the total heap.
|
||||||
</td>
|
</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>conf/core-site.xml</td>
|
<td>conf/mapred-site.xml</td>
|
||||||
|
<td>mapreduce.reduce.input.buffer.percent</td>
|
||||||
|
<td>0.80</td>
|
||||||
|
<td>
|
||||||
|
Larger amount of memory allocated for retaining map output
|
||||||
|
in memory during the reduce. Expressed as a fraction of
|
||||||
|
the total heap.
|
||||||
|
</td>
|
||||||
|
</tr>
|
||||||
|
<tr>
|
||||||
|
<td>conf/mapred-site.xml</td>
|
||||||
<td>mapreduce.task.io.sort.factor</td>
|
<td>mapreduce.task.io.sort.factor</td>
|
||||||
<td>100</td>
|
<td>100</td>
|
||||||
<td>More streams merged at once while sorting files.</td>
|
<td>More streams merged at once while sorting files.</td>
|
||||||
</tr>
|
</tr>
|
||||||
<tr>
|
<tr>
|
||||||
<td>conf/core-site.xml</td>
|
<td>conf/mapred-site.xml</td>
|
||||||
<td>mapreduce.task.io.sort.mb</td>
|
<td>mapreduce.task.io.sort.mb</td>
|
||||||
<td>200</td>
|
<td>200</td>
|
||||||
<td>Higher memory-limit while sorting data.</td>
|
<td>Higher memory-limit while sorting data.</td>
|
||||||
|
|
|
@ -250,7 +250,7 @@ info_port = Port number of the HDFS NameNode web UI
|
||||||
<p>For example:</p>
|
<p>For example:</p>
|
||||||
<source>
|
<source>
|
||||||
server-params = mapred.reduce.parallel.copies=20,io.sort.factor=100,io.sort.mb=128,io.file.buffer.size=131072
|
server-params = mapred.reduce.parallel.copies=20,io.sort.factor=100,io.sort.mb=128,io.file.buffer.size=131072
|
||||||
final-server-params = mapred.child.java.opts=-Xmx512m,dfs.block.size=134217728,fs.inmemory.size.mb=128
|
final-server-params = mapred.child.java.opts=-Xmx512m,dfs.block.size=134217728
|
||||||
</source>
|
</source>
|
||||||
<p>In order to provide the options from command line, you can use the following syntax:</p>
|
<p>In order to provide the options from command line, you can use the following syntax:</p>
|
||||||
<p>For configuring the MapReduce daemons use:</p>
|
<p>For configuring the MapReduce daemons use:</p>
|
||||||
|
|
Loading…
Reference in New Issue