Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Shade hadoop compat classes so they can be loaded in hadoop jobs (#13)

This commit is contained in:
Bryan Beaudreault 2022-06-09 09:51:39 -04:00 committed by GitHub
parent 12f40b6d6f
commit 56edc758ec
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
1 changed files with 34 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

34
hubspot-client-bundles/pom.xml
View File

@ -74,6 +74,40 @@
<pattern>com.google.protobuf</pattern>
<shadedPattern>${shade.prefix}.com.google.protobuf</shadedPattern>
</relocation>
<!--
This is a little brittle, but as far as I can tell this class list hasn't
really changed much. The core problem this solves is that our hadoop jobs
are executed with mapreduce.job.classloader.system.classes set, including
org.apache.hadoop. As a result trying to load any of these classes will
throw ClassNotFoundException because they aren't actually system classes
but are covered by that property. Another option would be to exclude
org.apache.hadoop.metrics2 from the system classes list, but there are a
bunch of classes in there in hadoop and it's hard to know the implications
of that. So I decided to solve this for hbase by shading these classes
which are brought in by hbase-hadoop-compat and hbase-hadoop2-compat.
These classes are used in snapshot scan jobs, because the metrics are
initialized when opening an HFile for read.
-->
<relocation>
<pattern>org.apache.hadoop.metrics2</pattern>
<shadedPattern>${shade.prefix}.org.apache.hadoop.metrics2</shadedPattern>
<includes>
<include>org.apache.hadoop.metrics2.MetricHistogram</include>
<include>org.apache.hadoop.metrics2.MetricsExecutor</include>
<include>org.apache.hadoop.metrics2.impl.JmxCacheBuster*</include>
<include>org.apache.hadoop.metrics2.lib.DefaultMetricsSystemHelper</include>
<include>org.apache.hadoop.metrics2.lib.DynamicMetricsRegistry</include>
<include>org.apache.hadoop.metrics2.lib.MetricsExecutorImpl</include>
<include>org.apache.hadoop.metrics2.lib.MetricsExecutorImpl*</include>
<include>org.apache.hadoop.metrics2.lib.MutableFastCounter</include>
<include>org.apache.hadoop.metrics2.lib.MutableHistogram</include>
<include>org.apache.hadoop.metrics2.lib.MutableRangeHistogram</include>
<include>org.apache.hadoop.metrics2.lib.MutableSizeHistogram</include>
<include>org.apache.hadoop.metrics2.lib.MutableTimeHistogram</include>
<include>org.apache.hadoop.metrics2.util.MetricQuantile</include>
<include>org.apache.hadoop.metrics2.util.MetricSampleQuantiles*</include>
</includes>
</relocation>
</relocations>
<filters>
<filter>