From 968b938fdefeeaee1e829a03979d3117eabda5ea Mon Sep 17 00:00:00 2001 From: Mike Drob Date: Wed, 27 Dec 2017 10:58:47 -0600 Subject: [PATCH] HBASE-19552 Update hbase-thirdparty version Some manual cleanup of changing package names in pom files and getting rid of the no-longer-needed netty system property. This commit will break compilation, package renames in source code are done in follow-on commits using straightforward find and replace. 's/org.apache.hadoop.hbase.shaded.com.google/org.apache.hbase.thirdparty.com.google/' 's/org.apache.hadoop.hbase.shaded.io.netty/org.apache.hbase.thirdparty.io.netty/' --- bin/hbase | 13 ++---- .../src/main/resources/hbase/checkstyle.xml | 2 +- hbase-mapreduce/pom.xml | 3 -- hbase-protocol-shaded/pom.xml | 2 +- hbase-server/pom.xml | 1 - hbase-spark/pom.xml | 9 +---- pom.xml | 24 +++-------- src/main/asciidoc/_chapters/developer.adoc | 40 ++----------------- 8 files changed, 16 insertions(+), 78 deletions(-) diff --git a/bin/hbase b/bin/hbase index 1fc51021035..f2d42518759 100755 --- a/bin/hbase +++ b/bin/hbase @@ -474,16 +474,9 @@ fi HEAP_SETTINGS="$JAVA_HEAP_MAX $JAVA_OFFHEAP_MAX" # Exec unless HBASE_NOEXEC is set. export CLASSPATH -# Netty is shaded in hbase. The shaded netty is pulled in with org.apache.hbase.thirdparty:hbase-shaded-netty. -# It has a .so in it. Shading requires rename of the .so and then passing a system property so netty finds the -# renamed .so and associates it w/ the relocated netty files. -# So we can find the relocated .so, we need to add a system property pointing at new location. Trick is from: -# https://stackoverflow.com/questions/33825743/rename-files-inside-a-jar-using-some-maven-plugin -# See toward the end of this issue for how to pass config: https://github.com/netty/netty/issues/6665 -# -NETTY_PROP="-Dorg.apache.hadoop.hbase.shaded.io.netty.packagePrefix=org.apache.hadoop.hbase.shaded." + if [ "${HBASE_NOEXEC}" != "" ]; then - "$JAVA" -Dproc_$COMMAND "$NETTY_PROP" -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@" + "$JAVA" -Dproc_$COMMAND -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@" else - exec "$JAVA" -Dproc_$COMMAND "$NETTY_PROP" -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@" + exec "$JAVA" -Dproc_$COMMAND -XX:OnOutOfMemoryError="kill -9 %p" $HEAP_SETTINGS $HBASE_OPTS $CLASS "$@" fi diff --git a/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml b/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml index bdfea28a502..4643709b862 100644 --- a/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml +++ b/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml @@ -67,7 +67,7 @@ http://checkstyle.sourceforge.net/config_imports.html --> - + diff --git a/hbase-mapreduce/pom.xml b/hbase-mapreduce/pom.xml index 7ba9a7c1752..0073440471c 100644 --- a/hbase-mapreduce/pom.xml +++ b/hbase-mapreduce/pom.xml @@ -76,9 +76,6 @@ org.apache.hadoop.hbase.ServerResourceCheckerJUnitListener - - org.apache.hadoop.hbase.shaded. - diff --git a/hbase-protocol-shaded/pom.xml b/hbase-protocol-shaded/pom.xml index 70af5d5b669..a9258db9e3d 100644 --- a/hbase-protocol-shaded/pom.xml +++ b/hbase-protocol-shaded/pom.xml @@ -120,7 +120,7 @@ ([^\.])com.google.protobuf - $1org.apache.hadoop.hbase.shaded.com.google.protobuf + $1org.apache.hbase.thirdparty.com.google.protobuf diff --git a/hbase-server/pom.xml b/hbase-server/pom.xml index fc87cbdace4..b5045c20a58 100644 --- a/hbase-server/pom.xml +++ b/hbase-server/pom.xml @@ -245,7 +245,6 @@ target/test-classes/webapps - org.apache.hadoop.hbase.shaded. diff --git a/hbase-spark/pom.xml b/hbase-spark/pom.xml index 1fafd318e8f..874cf9d9d6e 100644 --- a/hbase-spark/pom.xml +++ b/hbase-spark/pom.xml @@ -608,9 +608,6 @@ . WDF TestSuite.txt false - - org.apache.hadoop.hbase.shaded. - @@ -620,11 +617,7 @@ test - - org.apache.hadoop.hbase.shaded. - - -Xmx1536m -XX:ReservedCodeCacheSize=512m - + -Xmx1536m -XX:ReservedCodeCacheSize=512m false diff --git a/pom.xml b/pom.xml index 6e8aabcb947..5e4bc27fb16 100755 --- a/pom.xml +++ b/pom.xml @@ -604,25 +604,13 @@ ${test.output.tofile} ${test.build.classes} - - org.apache.hadoop.hbase.shaded. - - - ${test.exclude.pattern} - + + ${test.exclude.pattern} + @@ -1407,7 +1395,7 @@ 3.1.0-RC3 2.12 1.0.1 - 1.0.1 + 2.0.0 /usr /etc/hbase diff --git a/src/main/asciidoc/_chapters/developer.adoc b/src/main/asciidoc/_chapters/developer.adoc index 95334bfcd05..5a9df6034d5 100644 --- a/src/main/asciidoc/_chapters/developer.adoc +++ b/src/main/asciidoc/_chapters/developer.adoc @@ -2187,57 +2187,25 @@ libraries such as guava, netty, and protobuf. The mainline HBase project relies on the relocated versions of these libraries gotten from hbase-thirdparty rather than on finding these classes in their usual locations. We do this so we can specify whatever the version we wish. If we don't relocate, we must -harmonize our version to match that which hadoop and/or spark uses. +harmonize our version to match that which hadoop, spark, and other projects use. For developers, this means you need to be careful referring to classes from netty, guava, protobuf, gson, etc. (see the hbase-thirdparty pom.xml for what it provides). Devs must refer to the hbase-thirdparty provided classes. In practice, this is usually not an issue (though it can be a bit of a pain). You will have to hunt for the relocated version of your particular class. You'll -find it by prepending the general relocation prefix of `org.apache.hadoop.hbase.shaded.`. +find it by prepending the general relocation prefix of `org.apache.hbase.thirdparty.`. For example if you are looking for `com.google.protobuf.Message`, the relocated version used by HBase internals can be found at -`org.apache.hadoop.hbase.shaded.com.google.protobuf.Message`. +`org.apache.hbase.thirdparty.com.google.protobuf.Message`. For a few thirdparty libs, like protobuf (see the protobuf chapter in this book for the why), your IDE may give you both options -- the `com.google.protobuf.*` -and the `org.apache.hadoop.hbase.shaded.com.google.protobuf.*` -- because both +and the `org.apache.hbase.thirdparty.com.google.protobuf.*` -- because both classes are on your CLASSPATH. Unless you are doing the particular juggling required in Coprocessor Endpoint development (again see above cited protobuf chapter), you'll want to use the shaded version, always. -Of note, the relocation of netty is particular. The netty folks have put in -place facility to aid relocation; it seems like shading netty is a popular project. -One case of this requires the setting of a peculiar system property on the JVM -so that classes out in the bundld shared library (.so) can be found in their -relocated location. Here is the property that needs to be set: - -`-Dorg.apache.hadoop.hbase.shaded.io.netty.packagePrefix=org.apache.hadoop.hbase.shaded.` - -(Note that the trailing '.' is required). Starting hbase normally or when running -test suites, the setting of this property is done for you. If you are doing something -out of the ordinary, starting hbase from your own context, you'll need to provide -this property on platforms that favor the bundled .so. See release notes on HBASE-18271 -for more. The complaint you see is something like the following: -`Cause: java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMasterorg.apache.hadoop.hbase.shaded.io.netty.channel.epoll.` - -If running unit tests and you run into the above message, add the system property -to your surefire configuration by doing like the below: - -[source,xml] ----- - - maven-surefire-plugin - - - org.apache.hadoop.hbase.shaded. - - - ----- - -Again the trailing period in the value above is intended. - The `hbase-thirdparty` project has groupid of `org.apache.hbase.thirdparty`. As of this writing, it provides three jars; one for netty with an artifactid of `hbase-thirdparty-netty`, one for protobuf at `hbase-thirdparty-protobuf` and then