Add mapreduce.job.classloader.system.classes property to 'Other Hadoop Versions' docs (#3706)

This commit is contained in:
Jonathan Wei 2016-11-18 16:16:50 -08:00 committed by Fangjin Yang
parent 7b56cec3b9
commit 7c63bee7f5
1 changed files with 23 additions and 0 deletions

View File

@ -37,6 +37,29 @@ These properties can be set in either one of the following ways:
- Using the task definition, e.g. add `"mapreduce.job.classloader": "true"` to the `jobProperties` of the `tuningConfig` of your indexing task (see the [batch ingestion documentation](../ingestion/batch-ingestion.html)). - Using the task definition, e.g. add `"mapreduce.job.classloader": "true"` to the `jobProperties` of the `tuningConfig` of your indexing task (see the [batch ingestion documentation](../ingestion/batch-ingestion.html)).
- Using system properties, e.g. on the middleManager set `druid.indexer.runner.javaOpts=... -Dhadoop.mapreduce.job.classloader=true`. - Using system properties, e.g. on the middleManager set `druid.indexer.runner.javaOpts=... -Dhadoop.mapreduce.job.classloader=true`.
### Overriding specific classes
When `mapreduce.job.classloader = true`, it is also possible to specifically define which classes should be loaded from the hadoop system classpath and which should be loaded from job-supplied JARs.
This is controlled by defining class inclusion/exclusion patterns in the `mapreduce.job.classloader.system.classes` property in the `jobProperties` of `tuningConfig`.
For example, some community members have reported version incompatibility errors with the Validator class:
```
Error: java.lang.ClassNotFoundException: javax.validation.Validator
```
The following `jobProperties` excludes `javax.validation.` classes from being loaded from the system classpath, while including those from `java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop.`.
```
"jobProperties": {
"mapreduce.job.classloader": "true",
"mapreduce.job.classloader.system.classes": "-javax.validation.,java.,javax.,org.apache.commons.logging.,org.apache.log4j.,org.apache.hadoop."
}
```
[mapred-default.xml](https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml) documentation contains more information about this property.
## Tip #3: Use specific versions of Hadoop libraries ## Tip #3: Use specific versions of Hadoop libraries
Druid loads Hadoop client libraries from two different locations. Each set of libraries is loaded in an isolated Druid loads Hadoop client libraries from two different locations. Each set of libraries is loaded in an isolated