hbase/hbase-protocol-shaded/pom.xml

368 lines
15 KiB
XML
Raw Normal View History

HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<?xml version="1.0"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<!--
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-->
<modelVersion>4.0.0</modelVersion>
<parent>
<artifactId>hbase</artifactId>
<groupId>org.apache.hbase</groupId>
<version>3.0.0-SNAPSHOT</version>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<relativePath>..</relativePath>
</parent>
<artifactId>hbase-protocol-shaded</artifactId>
<name>Apache HBase - Shaded Protocol</name>
<description>Shaded protobuf protocol classes used by HBase internally.</description>
<properties>
<maven.javadoc.skip>true</maven.javadoc.skip>
<!--Version of protobuf that hbase uses internally (we shade our pb)
-->
<internal.protobuf.version>3.2.0</internal.protobuf.version>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<!--The Default target dir-->
<classes.dir>${project.build.directory}/classes</classes.dir>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<!--The Default location for sources-->
<sources.dir>src/main/java</sources.dir>
</properties>
<build>
<!--I want to override these in profile so define them
with variables up here-->
<sourceDirectory>${sources.dir}</sourceDirectory>
<outputDirectory>${classes.dir}</outputDirectory>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-site-plugin</artifactId>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
<!-- Make a jar and put the sources in the jar -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-source-plugin</artifactId>
</plugin>
<plugin>
<!--Make it so assembly:single does nothing in here-->
<artifactId>maven-assembly-plugin</artifactId>
<version>${maven.assembly.version}</version>
<configuration>
<skipAssembly>true</skipAssembly>
</configuration>
</plugin>
<plugin>
<artifactId>maven-surefire-plugin</artifactId>
<!-- Always skip the second part executions, since we only run simple unit tests in this module -->
<executions>
<execution>
<id>secondPartTestsExecution</id>
<phase>test</phase>
<goals>
<goal>test</goal>
</goals>
<configuration>
<skip>true</skip>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
<pluginManagement>
<plugins>
<!--This plugin's configuration is used to store Eclipse m2e settings only. It has no influence on the Maven build itself.-->
<plugin>
<groupId>org.eclipse.m2e</groupId>
<artifactId>lifecycle-mapping</artifactId>
<version>1.0.0</version>
<configuration>
<lifecycleMappingMetadata>
<pluginExecutions>
<pluginExecution>
<pluginExecutionFilter>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<versionRange>[2.0.5-alpha,)</versionRange>
<goals>
<goal>protoc</goal>
</goals>
</pluginExecutionFilter>
<action>
<ignore/>
</action>
</pluginExecution>
</pluginExecutions>
</lifecycleMappingMetadata>
</configuration>
</plugin>
</plugins>
</pluginManagement>
</build>
<dependencies>
<!--BE CAREFUL! Any dependency added here needs to be
excluded above in the shade plugin else the dependency
will get bundled-->
<!-- Intra-project dependencies -->
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-annotations</artifactId>
<exclusions>
<exclusion>
<groupId>jdk.tools</groupId>
<artifactId>jdk.tools</artifactId>
</exclusion>
</exclusions>
</dependency>
<!-- General dependencies -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${internal.protobuf.version}</version>
</dependency>
<dependency>
<groupId>commons-logging</groupId>
<artifactId>commons-logging</artifactId>
</dependency>
</dependencies>
<profiles>
<!-- Skip the tests in this module -->
<profile>
<id>skip-protocol-shaded-tests</id>
<activation>
<property>
<name>skip-protocol-shaded-tests</name>
</property>
</activation>
<properties>
<surefire.skipFirstPart>true</surefire.skipFirstPart>
</properties>
</profile>
<profile>
<id>compile-protobuf</id>
<!--
Generate and shade proto files. Drops generated java files
under src/main/java when done. Check in the generated files so
available at build time. Run this profile/step everytime you change
proto files or update the protobuf version.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
The below does a bunch of ugly stuff. It purges current content
of the generated and shaded com.google.protobuf java files first.
Let me say that again. We do a remove of java files under src/main/java
in the shaded dirs. It does this because later we apply patches and
patches fail if they've already been applied. We remove too because we
overlay the shaded protobuf and if files have been removed or added,
it'll be more plain if we have first done this delete.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
Next up we generate protos, build a scratch jar that contains protos
only and stuff we want shaded (we have to do this because shading only
works at install time on a jar), run the shade on the jar, then
carefully STOP this scratch jar from being put into the local repository
(because it can mess up builds that come later... mvn automatically wants
to install artifact into repo per module). Finally, undo this shaded
jar over the src/main/java directory, and then apply patches atop this.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
The result needs to be checked in.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
-->
<activation>
<property>
<name>compile-protobuf</name>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
</property>
</activation>
<properties>
<profile.id>compile-protobuf</profile.id>
<!--Directory under target to hold generated protos files-->
<protoc.sources.dir>${project.build.directory}/protoc-generated-sources</protoc.sources.dir>
<!--When doing this step, the sources.dir is pointed at generated protos, NOT src/main/java-->
<sources.dir>${protoc.sources.dir}</sources.dir>
<!--Where to compile protos into-->
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<classes.dir>${project.build.directory}/protoc-generated-classes</classes.dir>
<!--When the compile for this profile runs, make sure it makes jars that
can be related back to this shading profile. Give them the shading profile
name as a prefix.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
-->
<jar.finalName>${profile.id}.${project.artifactId}-${project.version}</jar.finalName>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
</properties>
<build>
<plugins>
<plugin>
<artifactId>maven-clean-plugin</artifactId>
<executions>
<execution>
<id>pre-compile-protoc</id>
<phase>generate-sources</phase>
<goals>
<goal>clean</goal>
</goals>
<configuration>
<filesets>
<fileset>
<directory>${basedir}/src/main/java/org/apache/hadoop/hbase/shaded</directory>
<includes>
<include>ipc/protobuf/generated/**/*.java</include>
<include>protobuf/generated/**/*.java</include>
<include>com/google/protobuf/**/*.java</include>
</includes>
<followSymlinks>false</followSymlinks>
</fileset>
</filesets>
</configuration>
</execution>
</executions>
</plugin>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<plugin>
<groupId>org.xolstice.maven.plugins</groupId>
<artifactId>protobuf-maven-plugin</artifactId>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<executions>
<execution>
<id>compile-protoc</id>
<phase>generate-sources</phase>
<goals>
<goal>compile</goal>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
</goals>
<configuration>
<protocArtifact>com.google.protobuf:protoc:${internal.protobuf.version}:exe:${os.detected.classifier}</protocArtifact>
<outputDirectory>${protoc.sources.dir}</outputDirectory>
<attachProtoSources>false</attachProtoSources>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-jar-plugin</artifactId>
<version>3.0.2</version>
<configuration>
<finalName>${jar.finalName}</finalName>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.4.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadeSourcesContent>true</shadeSourcesContent>
<createSourcesJar>true</createSourcesJar>
<relocations>
<relocation>
<pattern>com.google.protobuf</pattern>
<shadedPattern>org.apache.hadoop.hbase.shaded.com.google.protobuf</shadedPattern>
</relocation>
</relocations>
<!-- What I got when I did a mvn dependency:list for this
module. Exclude all but the protobuf
[INFO] commons-logging:commons-logging:jar:1.2:compile
[INFO] com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] log4j:log4j:jar:1.2.17:compile
[INFO] com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] org.hamcrest:hamcrest-core:jar:1.3:test
[INFO] org.mockito:mockito-all:jar:1.10.8:test
[INFO] junit:junit:jar:4.12:compile
[INFO] org.apache.hbase:hbase-annotations:jar:2.0.0-SNAPSHOT:compile
The list below must exlude all of the above except protobuf.
-->
<artifactSet>
<excludes>
<exclude>commons-logging:commons-logging</exclude>
<exclude>com.github.stephenc.findbugs:findbugs-annotations</exclude>
<exclude>log4j:log4j</exclude>
<exclude>org.hamcrest:hamcrest-core</exclude>
<exclude>org.mockito:mockito-all</exclude>
<exclude>junit:junit</exclude>
<exclude>org.apache.hbase:hbase-annotations</exclude>
</excludes>
</artifactSet>
</configuration>
</execution>
</executions>
</plugin>
<!--Now unpack the shaded jar made above so the shaded classes
are available to subsequent modules-->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<version>2.10</version>
<executions>
<execution>
<id>unpack</id>
<phase>package</phase>
<goals>
<goal>unpack</goal>
</goals>
<configuration>
<artifactItems>
<artifactItem>
<groupId>${project.groupId}</groupId>
<artifactId>${project.artifactId}</artifactId>
<version>${project.version}</version>
<classifier>sources</classifier>
<type>jar</type>
<overWrite>true</overWrite>
<outputDirectory>${basedir}/src/main/java</outputDirectory>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
<includes>**/*.java</includes>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-patch-plugin</artifactId>
<version>1.2</version>
<configuration>
<!--Patches are made at top-level-->
<targetDirectory>${basedir}/..</targetDirectory>
<skipApplication>false</skipApplication>
</configuration>
<executions>
<execution>
<id>patch</id>
<configuration>
<strip>1</strip>
<patchDirectory>src/main/patches</patchDirectory>
<patchTrackingFile>${project.build.directory}/patches-applied.txt</patchTrackingFile>
<naturalOrderProcessing>true</naturalOrderProcessing>
</configuration>
<phase>package</phase>
<goals>
<!--This should run after the above unpack phase-->
<goal>apply</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-install-plugin</artifactId>
<version>2.5.2</version>
<configuration>
<skip>true</skip>
</configuration>
</plugin>
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
</plugins>
</build>
</profile>
</profiles>
</project>