Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Link up Hadoop class loading docs better. (#3302)

This commit is contained in:
Gian Merlino 2016-07-29 10:19:54 -07:00 committed by Fangjin Yang
parent 1fa681934c
commit e5397ed316
2 changed files with 6 additions and 7 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
docs/content/ingestion/batch-ingestion.md
View File

@ -178,14 +178,11 @@ The tuningConfig is optional and default parameters will be used if no tuningCon
} }
``` ```
The following properties can be used to tune how the MapReduce job is configured by overriding default Hadoop/YARN/Mapreduce configurations: Hadoop's [MapReduce documentation](https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml) lists the possible configuration parameters.
|Property name|Type|Description| With some Hadoop distributions, it may be necessary to set `mapreduce.job.classpath` or `mapreduce.job.user.classpath.first`
|-------------|----|-----------| to avoid class loading issues. See the [working with different Hadoop versions documentation](../operations/other-hadoop.html)
|mapreduce.job.user.classpath.first|String|Use Druid classpath instead of Hadoop classpath for common libraries like [Jackson](https://github.com/FasterXML/jackson) (required with [Cloudera Hadoop distribution](../operations/other-hadoop.html)) when set to `"true"`.| for more details.
|...|String|See [Mapred configuration](https://hadoop.apache.org/docs/stable/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml) for more configuration parameters.|
**Please note that using `mapreduce.job.user.classpath.first` is an expert feature and should not be used without a deep understanding of Hadoop and Java class loading mechanism.**
#### IndexSpec #### IndexSpec

2
docs/content/operations/other-hadoop.md
View File

@ -30,6 +30,8 @@ If your version of Hadoop does not support this functionality, you can also try
`mapreduce.job.user.classpath.first = true`. This instructs Hadoop to prefer loading Druid's version of a library when `mapreduce.job.user.classpath.first = true`. This instructs Hadoop to prefer loading Druid's version of a library when
there is a conflict. there is a conflict.
Generally, you should only set one of these parameters, not both.
These properties can be set in either one of the following ways: These properties can be set in either one of the following ways:
- Using the task definition, e.g. add `"mapreduce.job.classloader": "true"` to the `jobProperties` of the `tuningConfig` of your indexing task (see the [batch ingestion documentation](../ingestion/batch-ingestion.html)). - Using the task definition, e.g. add `"mapreduce.job.classloader": "true"` to the `jobProperties` of the `tuningConfig` of your indexing task (see the [batch ingestion documentation](../ingestion/batch-ingestion.html)).