hbase/hbase-protocol-shaded/src/main/protobuf/Admin.proto

348 lines
8.9 KiB
Protocol Buffer
Raw Normal View History

HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
syntax = "proto2";
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
// This file contains protocol buffers that are used for Admin service.
package hbase.pb;
option java_package = "org.apache.hadoop.hbase.shaded.protobuf.generated";
option java_outer_classname = "AdminProtos";
option java_generic_services = true;
option java_generate_equals_and_hash = true;
option optimize_for = SPEED;
import "ClusterStatus.proto";
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
import "HBase.proto";
import "WAL.proto";
import "Quota.proto";
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
message GetRegionInfoRequest {
required RegionSpecifier region = 1;
optional bool compaction_state = 2;
optional bool best_split_row = 3;
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
}
message GetRegionInfoResponse {
required RegionInfo region_info = 1;
optional CompactionState compaction_state = 2;
// optional bool DEPRECATED_isRecovering = 3;
HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility. This doc. keeps state on where we are at w/ the new AM: https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn Includes list of tests disabled by this patch with reasons why. Based on patches from Matteos' repository and then fix up to get it all to pass cluster tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc.. including: 1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM. The basis comes from Matteo's repo here: https://github.com/matteobertozzi/hbase/commit/689227fcbfe8e6588433dbcdabf4526e3d478b2e Patch replaces old AM with the new under subpackage master.assignment. Mostly just updating classes to use new AM -- import changes -- rather than the old. It also removes old AM and supporting classes. See below for more detail. 2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) https://github.com/matteobertozzi/hbase/commit/3622cba4e331d2fc7bfc1932abb4c9cbf5802efa Adds running of remote procedure. Adds batching of remote calls. Adds support for assign/unassign in procedures. Adds version info reporting in rpc. Adds start of an AMv2. 3. Reporting of remote RS version is from here: https://github.com/matteobertozzi/hbase/commit/ddb4df3964e8298c88c0210e83493aa91ac0942d.patch 4. And remote dispatch of procedures is from: https://github.com/matteobertozzi/hbase/commit/186b9e7c4dae61a79509a6c3aad7f80ec61345e5 5. The split merge patches from here are also melded in: https://github.com/matteobertozzi/hbase/commit/9a3a95a2c2974842a4849d1ad867e70764e7f707 and https://github.com/matteobertozzi/hbase/commit/d6289307a02a777299f65238238a2a8af3253067 We add testing util for new AM and new sets of tests. Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping procedure id. We spewed loads of log too on big transitions such as master fail; fixed. Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge. Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were splitting split Regions among other things (CJ would run but wasn't taking lock on Regions so havoc). Added a bunch of doc. on Procedure primitives. Added new region-based state machine base class. Moved region-based state machines on to it. Found bugs in the way procedure locking was doing in a few of the region-based Procedures. Having them all have same subclass helps here. Added isSplittable and isMergeable to the Region Interface. Master would split/merge even though the Regions still had references. Fixed it so Master asks RegionServer if Region is splittable. Messing more w/ logging. Made all procedures log the same and report the state the same; helps when logging is regular. Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure. Added more functionality to MockMasterServices so can use it doing standalone testing of Procedures (made TestCatalogTracker use it instead of its own version). Add to MasterServices ability to wait on Master being up -- makes it so can Mock Master and start to implement standalone split testing. Start in on a Split region standalone test in TestAM. Fix bug where a Split can fail because it comes in in the middle of a Move (by holding lock for duration of a Move). Breaks CPs that were watching merge/split. These are run by Master now so you need to observe on Master, not on RegionServer. Details: M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java Takes List of regionstates on construction rather than a Set. NOTE!!!!! This is a change in a public class. M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Add utility getShortNameToLog M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java Add support for dispatching assign, split and merge processes. M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java Lots of doc on its inner workings. Bug fixes. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java Log and doc on workings. Bug fixes. A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java Dispatch remote procedures every 150ms or 32 items -- which ever happens first (configurable). Runs a timeout thread. This facility is not on yet; will come in as part of a later fix. Currently works a region at a time. This class carries notion of a remote procedure and of a buffer full of these. "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128 "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32 M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java Add in support for merge. Remove no-longer used methods. M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto Add execute procedures call ExecuteProcedures. M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto Add assign and unassign state support for procedures. M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java Adds getting RS version out of RPC Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000) M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Remove periodic metrics chore. This is done over in new AM now. Replace AM with the new. Host the procedures executor. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java Have AMv2 handle assigning meta. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java Extract version number of the server making rpc. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java Add new assign procedure. Runs assign via Procedure Dispatch. There can only be one RegionTransitionProcedure per region running at the time, since each procedure takes a lock on the region. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java Remove these hacky classes that were never supposed to live longer than a month or so to be replaced with real assigners. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java A procedure-based AM (AMv2). TODO - handle region migration - handle meta assignment first - handle sys table assignment first (e.g. acl, namespace) - handle table priorities "hbase.assignment.bootstrap.thread.pool.size"; default size is 16. "hbase.assignment.dispatch.wait.msec"; default wait is 150 "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100 "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000; "hbase.assignment.maximum.attempts"; default is 10; A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java Procedure that runs subprocedure to unassign and then assign to new location A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Manage store of region state (in hbase:meta by default). A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java In-memory state of all regions. Used by AMv2. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java Base RIT procedure for Assign and Unassign. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java Unassign procedure. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java Run region assignement in a manner that pays attention to target server version. Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 20:49:11 -04:00
// True if region is splittable, false otherwise.
optional bool splittable = 4;
// True if region is mergeable, false otherwise.
optional bool mergeable = 5;
// Get bestSplitRow
optional bytes best_split_row = 6;
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
enum CompactionState {
NONE = 0;
MINOR = 1;
MAJOR = 2;
MAJOR_AND_MINOR = 3;
}
}
/**
* Get a list of store files for a set of column families in a particular region.
* If no column family is specified, get the store files for all column families.
*/
message GetStoreFileRequest {
required RegionSpecifier region = 1;
repeated bytes family = 2;
}
message GetStoreFileResponse {
repeated string store_file = 1;
}
message GetOnlineRegionRequest {
}
message GetOnlineRegionResponse {
repeated RegionInfo region_info = 1;
}
message OpenRegionRequest {
repeated RegionOpenInfo open_info = 1;
// the intended server for this RPC.
optional uint64 serverStartCode = 2;
// wall clock time from master
optional uint64 master_system_time = 5;
message RegionOpenInfo {
required RegionInfo region = 1;
optional uint32 version_of_offline_node = 2;
repeated ServerName favored_nodes = 3;
// open region for distributedLogReplay
// optional bool DEPRECATED_openForDistributedLogReplay = 4;
optional int64 open_proc_id = 5 [default = -1];
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
}
}
message OpenRegionResponse {
repeated RegionOpeningState opening_state = 1;
enum RegionOpeningState {
OPENED = 0;
ALREADY_OPENED = 1;
FAILED_OPENING = 2;
}
}
message WarmupRegionRequest {
required RegionInfo regionInfo = 1;
}
message WarmupRegionResponse {
}
/**
* Closes the specified region and will use or not use ZK during the close
* according to the specified flag.
*/
message CloseRegionRequest {
required RegionSpecifier region = 1;
optional uint32 version_of_closing_node = 2;
optional bool transition_in_ZK = 3 [default = true];
optional ServerName destination_server = 4;
// the intended server for this RPC.
optional uint64 serverStartCode = 5;
optional int64 close_proc_id = 6 [default = -1];
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
}
message CloseRegionResponse {
required bool closed = 1;
}
/**
* Flushes the MemStore of the specified region.
* <p>
* This method is synchronous.
*/
message FlushRegionRequest {
required RegionSpecifier region = 1;
optional uint64 if_older_than_ts = 2;
optional bool write_flush_wal_marker = 3; // whether to write a marker to WAL even if not flushed
}
message FlushRegionResponse {
required uint64 last_flush_time = 1;
optional bool flushed = 2;
optional bool wrote_flush_wal_marker = 3;
}
/**
* Compacts the specified region. Performs a major compaction if specified.
* <p>
* This method is asynchronous.
*/
message CompactRegionRequest {
required RegionSpecifier region = 1;
optional bool major = 2;
optional bytes family = 3;
}
message CompactRegionResponse {
}
message CompactionSwitchRequest {
required bool enabled = 1;
}
message CompactionSwitchResponse {
required bool prev_state = 1;
}
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
message UpdateFavoredNodesRequest {
repeated RegionUpdateInfo update_info = 1;
message RegionUpdateInfo {
required RegionInfo region = 1;
repeated ServerName favored_nodes = 2;
}
}
message UpdateFavoredNodesResponse {
optional uint32 response = 1;
}
// Protocol buffer version of WAL for replication
message WALEntry {
required WALKey key = 1;
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
// Following may be null if the KVs/Cells are carried along the side in a cellblock (See
// RPC for more on cellblocks). If Cells/KVs are in a cellblock, this next field is null
// and associated_cell_count has count of Cells associated w/ this WALEntry
repeated bytes key_value_bytes = 2;
// If Cell data is carried alongside in a cellblock, this is count of Cells in the cellblock.
optional int32 associated_cell_count = 3;
}
/**
* Replicates the given entries. The guarantee is that the given entries
* will be durable on the slave cluster if this method returns without
* any exception.
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
*/
message ReplicateWALEntryRequest {
repeated WALEntry entry = 1;
optional string replicationClusterId = 2;
optional string sourceBaseNamespaceDirPath = 3;
optional string sourceHFileArchiveDirPath = 4;
}
message ReplicateWALEntryResponse {
}
message RollWALWriterRequest {
}
/*
* Roll request responses no longer include regions to flush
* this list will always be empty when talking to a 1.0 server
*/
message RollWALWriterResponse {
// A list of encoded name of regions to flush
repeated bytes region_to_flush = 1;
}
message StopServerRequest {
required string reason = 1;
}
message StopServerResponse {
}
message GetServerInfoRequest {
}
message ServerInfo {
required ServerName server_name = 1;
optional uint32 webui_port = 2;
}
message GetServerInfoResponse {
required ServerInfo server_info = 1;
}
message UpdateConfigurationRequest {
}
message UpdateConfigurationResponse {
}
message GetRegionLoadRequest {
optional TableName table_name = 1;
}
message GetRegionLoadResponse {
repeated RegionLoad region_loads = 1;
}
message ClearCompactionQueuesRequest {
repeated string queue_name = 1;
}
message ClearCompactionQueuesResponse {
}
message ClearRegionBlockCacheRequest {
repeated RegionSpecifier region = 1;
}
message ClearRegionBlockCacheResponse {
required CacheEvictionStats stats = 1;
}
message RemoteProcedureRequest {
required uint64 proc_id = 1;
required string proc_class = 2;
optional bytes proc_data = 3;
}
HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility. This doc. keeps state on where we are at w/ the new AM: https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn Includes list of tests disabled by this patch with reasons why. Based on patches from Matteos' repository and then fix up to get it all to pass cluster tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc.. including: 1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM. The basis comes from Matteo's repo here: https://github.com/matteobertozzi/hbase/commit/689227fcbfe8e6588433dbcdabf4526e3d478b2e Patch replaces old AM with the new under subpackage master.assignment. Mostly just updating classes to use new AM -- import changes -- rather than the old. It also removes old AM and supporting classes. See below for more detail. 2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) https://github.com/matteobertozzi/hbase/commit/3622cba4e331d2fc7bfc1932abb4c9cbf5802efa Adds running of remote procedure. Adds batching of remote calls. Adds support for assign/unassign in procedures. Adds version info reporting in rpc. Adds start of an AMv2. 3. Reporting of remote RS version is from here: https://github.com/matteobertozzi/hbase/commit/ddb4df3964e8298c88c0210e83493aa91ac0942d.patch 4. And remote dispatch of procedures is from: https://github.com/matteobertozzi/hbase/commit/186b9e7c4dae61a79509a6c3aad7f80ec61345e5 5. The split merge patches from here are also melded in: https://github.com/matteobertozzi/hbase/commit/9a3a95a2c2974842a4849d1ad867e70764e7f707 and https://github.com/matteobertozzi/hbase/commit/d6289307a02a777299f65238238a2a8af3253067 We add testing util for new AM and new sets of tests. Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping procedure id. We spewed loads of log too on big transitions such as master fail; fixed. Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge. Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were splitting split Regions among other things (CJ would run but wasn't taking lock on Regions so havoc). Added a bunch of doc. on Procedure primitives. Added new region-based state machine base class. Moved region-based state machines on to it. Found bugs in the way procedure locking was doing in a few of the region-based Procedures. Having them all have same subclass helps here. Added isSplittable and isMergeable to the Region Interface. Master would split/merge even though the Regions still had references. Fixed it so Master asks RegionServer if Region is splittable. Messing more w/ logging. Made all procedures log the same and report the state the same; helps when logging is regular. Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure. Added more functionality to MockMasterServices so can use it doing standalone testing of Procedures (made TestCatalogTracker use it instead of its own version). Add to MasterServices ability to wait on Master being up -- makes it so can Mock Master and start to implement standalone split testing. Start in on a Split region standalone test in TestAM. Fix bug where a Split can fail because it comes in in the middle of a Move (by holding lock for duration of a Move). Breaks CPs that were watching merge/split. These are run by Master now so you need to observe on Master, not on RegionServer. Details: M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java Takes List of regionstates on construction rather than a Set. NOTE!!!!! This is a change in a public class. M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Add utility getShortNameToLog M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java Add support for dispatching assign, split and merge processes. M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java Lots of doc on its inner workings. Bug fixes. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java Log and doc on workings. Bug fixes. A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java Dispatch remote procedures every 150ms or 32 items -- which ever happens first (configurable). Runs a timeout thread. This facility is not on yet; will come in as part of a later fix. Currently works a region at a time. This class carries notion of a remote procedure and of a buffer full of these. "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128 "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32 M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java Add in support for merge. Remove no-longer used methods. M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto Add execute procedures call ExecuteProcedures. M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto Add assign and unassign state support for procedures. M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java Adds getting RS version out of RPC Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000) M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Remove periodic metrics chore. This is done over in new AM now. Replace AM with the new. Host the procedures executor. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java Have AMv2 handle assigning meta. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java Extract version number of the server making rpc. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java Add new assign procedure. Runs assign via Procedure Dispatch. There can only be one RegionTransitionProcedure per region running at the time, since each procedure takes a lock on the region. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java Remove these hacky classes that were never supposed to live longer than a month or so to be replaced with real assigners. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java A procedure-based AM (AMv2). TODO - handle region migration - handle meta assignment first - handle sys table assignment first (e.g. acl, namespace) - handle table priorities "hbase.assignment.bootstrap.thread.pool.size"; default size is 16. "hbase.assignment.dispatch.wait.msec"; default wait is 150 "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100 "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000; "hbase.assignment.maximum.attempts"; default is 10; A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java Procedure that runs subprocedure to unassign and then assign to new location A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Manage store of region state (in hbase:meta by default). A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java In-memory state of all regions. Used by AMv2. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java Base RIT procedure for Assign and Unassign. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java Unassign procedure. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java Run region assignement in a manner that pays attention to target server version. Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 20:49:11 -04:00
message ExecuteProceduresRequest {
repeated OpenRegionRequest open_region = 1;
repeated CloseRegionRequest close_region = 2;
repeated RemoteProcedureRequest proc = 3;
HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility. This doc. keeps state on where we are at w/ the new AM: https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn Includes list of tests disabled by this patch with reasons why. Based on patches from Matteos' repository and then fix up to get it all to pass cluster tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc.. including: 1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM. The basis comes from Matteo's repo here: https://github.com/matteobertozzi/hbase/commit/689227fcbfe8e6588433dbcdabf4526e3d478b2e Patch replaces old AM with the new under subpackage master.assignment. Mostly just updating classes to use new AM -- import changes -- rather than the old. It also removes old AM and supporting classes. See below for more detail. 2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) https://github.com/matteobertozzi/hbase/commit/3622cba4e331d2fc7bfc1932abb4c9cbf5802efa Adds running of remote procedure. Adds batching of remote calls. Adds support for assign/unassign in procedures. Adds version info reporting in rpc. Adds start of an AMv2. 3. Reporting of remote RS version is from here: https://github.com/matteobertozzi/hbase/commit/ddb4df3964e8298c88c0210e83493aa91ac0942d.patch 4. And remote dispatch of procedures is from: https://github.com/matteobertozzi/hbase/commit/186b9e7c4dae61a79509a6c3aad7f80ec61345e5 5. The split merge patches from here are also melded in: https://github.com/matteobertozzi/hbase/commit/9a3a95a2c2974842a4849d1ad867e70764e7f707 and https://github.com/matteobertozzi/hbase/commit/d6289307a02a777299f65238238a2a8af3253067 We add testing util for new AM and new sets of tests. Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping procedure id. We spewed loads of log too on big transitions such as master fail; fixed. Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge. Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were splitting split Regions among other things (CJ would run but wasn't taking lock on Regions so havoc). Added a bunch of doc. on Procedure primitives. Added new region-based state machine base class. Moved region-based state machines on to it. Found bugs in the way procedure locking was doing in a few of the region-based Procedures. Having them all have same subclass helps here. Added isSplittable and isMergeable to the Region Interface. Master would split/merge even though the Regions still had references. Fixed it so Master asks RegionServer if Region is splittable. Messing more w/ logging. Made all procedures log the same and report the state the same; helps when logging is regular. Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure. Added more functionality to MockMasterServices so can use it doing standalone testing of Procedures (made TestCatalogTracker use it instead of its own version). Add to MasterServices ability to wait on Master being up -- makes it so can Mock Master and start to implement standalone split testing. Start in on a Split region standalone test in TestAM. Fix bug where a Split can fail because it comes in in the middle of a Move (by holding lock for duration of a Move). Breaks CPs that were watching merge/split. These are run by Master now so you need to observe on Master, not on RegionServer. Details: M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java Takes List of regionstates on construction rather than a Set. NOTE!!!!! This is a change in a public class. M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Add utility getShortNameToLog M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java Add support for dispatching assign, split and merge processes. M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java Lots of doc on its inner workings. Bug fixes. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java Log and doc on workings. Bug fixes. A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java Dispatch remote procedures every 150ms or 32 items -- which ever happens first (configurable). Runs a timeout thread. This facility is not on yet; will come in as part of a later fix. Currently works a region at a time. This class carries notion of a remote procedure and of a buffer full of these. "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128 "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32 M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java Add in support for merge. Remove no-longer used methods. M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto Add execute procedures call ExecuteProcedures. M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto Add assign and unassign state support for procedures. M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java Adds getting RS version out of RPC Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000) M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Remove periodic metrics chore. This is done over in new AM now. Replace AM with the new. Host the procedures executor. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java Have AMv2 handle assigning meta. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java Extract version number of the server making rpc. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java Add new assign procedure. Runs assign via Procedure Dispatch. There can only be one RegionTransitionProcedure per region running at the time, since each procedure takes a lock on the region. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java Remove these hacky classes that were never supposed to live longer than a month or so to be replaced with real assigners. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java A procedure-based AM (AMv2). TODO - handle region migration - handle meta assignment first - handle sys table assignment first (e.g. acl, namespace) - handle table priorities "hbase.assignment.bootstrap.thread.pool.size"; default size is 16. "hbase.assignment.dispatch.wait.msec"; default wait is 150 "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100 "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000; "hbase.assignment.maximum.attempts"; default is 10; A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java Procedure that runs subprocedure to unassign and then assign to new location A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Manage store of region state (in hbase:meta by default). A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java In-memory state of all regions. Used by AMv2. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java Base RIT procedure for Assign and Unassign. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java Unassign procedure. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java Run region assignement in a manner that pays attention to target server version. Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 20:49:11 -04:00
}
message ExecuteProceduresResponse {
}
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
service AdminService {
rpc GetRegionInfo(GetRegionInfoRequest)
returns(GetRegionInfoResponse);
rpc GetStoreFile(GetStoreFileRequest)
returns(GetStoreFileResponse);
rpc GetOnlineRegion(GetOnlineRegionRequest)
returns(GetOnlineRegionResponse);
rpc OpenRegion(OpenRegionRequest)
returns(OpenRegionResponse);
rpc WarmupRegion(WarmupRegionRequest)
returns(WarmupRegionResponse);
rpc CloseRegion(CloseRegionRequest)
returns(CloseRegionResponse);
rpc FlushRegion(FlushRegionRequest)
returns(FlushRegionResponse);
rpc CompactionSwitch(CompactionSwitchRequest)
returns(CompactionSwitchResponse);
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
rpc CompactRegion(CompactRegionRequest)
returns(CompactRegionResponse);
rpc ReplicateWALEntry(ReplicateWALEntryRequest)
returns(ReplicateWALEntryResponse);
rpc Replay(ReplicateWALEntryRequest)
returns(ReplicateWALEntryResponse);
rpc RollWALWriter(RollWALWriterRequest)
returns(RollWALWriterResponse);
rpc GetServerInfo(GetServerInfoRequest)
returns(GetServerInfoResponse);
rpc StopServer(StopServerRequest)
returns(StopServerResponse);
rpc UpdateFavoredNodes(UpdateFavoredNodesRequest)
returns(UpdateFavoredNodesResponse);
rpc UpdateConfiguration(UpdateConfigurationRequest)
returns(UpdateConfigurationResponse);
rpc GetRegionLoad(GetRegionLoadRequest)
returns(GetRegionLoadResponse);
rpc ClearCompactionQueues(ClearCompactionQueuesRequest)
returns(ClearCompactionQueuesResponse);
rpc ClearRegionBlockCache(ClearRegionBlockCacheRequest)
returns(ClearRegionBlockCacheResponse);
/** Fetches the RegionServer's view of space quotas */
rpc GetSpaceQuotaSnapshots(GetSpaceQuotaSnapshotsRequest)
returns(GetSpaceQuotaSnapshotsResponse);
HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility. This doc. keeps state on where we are at w/ the new AM: https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn Includes list of tests disabled by this patch with reasons why. Based on patches from Matteos' repository and then fix up to get it all to pass cluster tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc.. including: 1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM. The basis comes from Matteo's repo here: https://github.com/matteobertozzi/hbase/commit/689227fcbfe8e6588433dbcdabf4526e3d478b2e Patch replaces old AM with the new under subpackage master.assignment. Mostly just updating classes to use new AM -- import changes -- rather than the old. It also removes old AM and supporting classes. See below for more detail. 2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) https://github.com/matteobertozzi/hbase/commit/3622cba4e331d2fc7bfc1932abb4c9cbf5802efa Adds running of remote procedure. Adds batching of remote calls. Adds support for assign/unassign in procedures. Adds version info reporting in rpc. Adds start of an AMv2. 3. Reporting of remote RS version is from here: https://github.com/matteobertozzi/hbase/commit/ddb4df3964e8298c88c0210e83493aa91ac0942d.patch 4. And remote dispatch of procedures is from: https://github.com/matteobertozzi/hbase/commit/186b9e7c4dae61a79509a6c3aad7f80ec61345e5 5. The split merge patches from here are also melded in: https://github.com/matteobertozzi/hbase/commit/9a3a95a2c2974842a4849d1ad867e70764e7f707 and https://github.com/matteobertozzi/hbase/commit/d6289307a02a777299f65238238a2a8af3253067 We add testing util for new AM and new sets of tests. Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping procedure id. We spewed loads of log too on big transitions such as master fail; fixed. Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge. Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were splitting split Regions among other things (CJ would run but wasn't taking lock on Regions so havoc). Added a bunch of doc. on Procedure primitives. Added new region-based state machine base class. Moved region-based state machines on to it. Found bugs in the way procedure locking was doing in a few of the region-based Procedures. Having them all have same subclass helps here. Added isSplittable and isMergeable to the Region Interface. Master would split/merge even though the Regions still had references. Fixed it so Master asks RegionServer if Region is splittable. Messing more w/ logging. Made all procedures log the same and report the state the same; helps when logging is regular. Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure. Added more functionality to MockMasterServices so can use it doing standalone testing of Procedures (made TestCatalogTracker use it instead of its own version). Add to MasterServices ability to wait on Master being up -- makes it so can Mock Master and start to implement standalone split testing. Start in on a Split region standalone test in TestAM. Fix bug where a Split can fail because it comes in in the middle of a Move (by holding lock for duration of a Move). Breaks CPs that were watching merge/split. These are run by Master now so you need to observe on Master, not on RegionServer. Details: M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java Takes List of regionstates on construction rather than a Set. NOTE!!!!! This is a change in a public class. M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java Add utility getShortNameToLog M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java Add support for dispatching assign, split and merge processes. M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java Lots of doc on its inner workings. Bug fixes. M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java Log and doc on workings. Bug fixes. A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java Dispatch remote procedures every 150ms or 32 items -- which ever happens first (configurable). Runs a timeout thread. This facility is not on yet; will come in as part of a later fix. Currently works a region at a time. This class carries notion of a remote procedure and of a buffer full of these. "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128 "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32 M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java Add in support for merge. Remove no-longer used methods. M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto Add execute procedures call ExecuteProcedures. M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto Add assign and unassign state support for procedures. M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java Adds getting RS version out of RPC Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000) M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java Remove periodic metrics chore. This is done over in new AM now. Replace AM with the new. Host the procedures executor. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java Have AMv2 handle assigning meta. M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java Extract version number of the server making rpc. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java Add new assign procedure. Runs assign via Procedure Dispatch. There can only be one RegionTransitionProcedure per region running at the time, since each procedure takes a lock on the region. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java Remove these hacky classes that were never supposed to live longer than a month or so to be replaced with real assigners. D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java A procedure-based AM (AMv2). TODO - handle region migration - handle meta assignment first - handle sys table assignment first (e.g. acl, namespace) - handle table priorities "hbase.assignment.bootstrap.thread.pool.size"; default size is 16. "hbase.assignment.dispatch.wait.msec"; default wait is 150 "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100 "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000; "hbase.assignment.maximum.attempts"; default is 10; A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java Procedure that runs subprocedure to unassign and then assign to new location A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java Manage store of region state (in hbase:meta by default). A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java In-memory state of all regions. Used by AMv2. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java Base RIT procedure for Assign and Unassign. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java Unassign procedure. A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java Run region assignement in a manner that pays attention to target server version. Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 20:49:11 -04:00
rpc ExecuteProcedures(ExecuteProceduresRequest)
returns(ExecuteProceduresResponse);
HBASE-15638 Shade protobuf Which includes HBASE-16742 Add chapter for devs on how we do protobufs going forward HBASE-16741 Amend the generate protobufs out-of-band build step to include shade, pulling in protobuf source and a hook for patching protobuf Removed ByteStringer from hbase-protocol-shaded. Use the protobuf-3.1.0 trick directly instead. Makes stuff cleaner. All under 'shaded' dir is now generated. HBASE-16567 Upgrade to protobuf-3.1.x Regenerate all protos in this module with protoc3. Redo ByteStringer to use new pb3.1.0 unsafebytesutil instead of HBaseZeroCopyByteString HBASE-16264 Figure how to deal with endpoints and shaded pb Shade our protobufs. Do it in a manner that makes it so we can still have in our API references to com.google.protobuf (and in REST). The c.g.p in API is for Coprocessor Endpoints (CPEP) This patch is Tactic #4 from Shading Doc attached to the referenced issue. Figuring an appoach took a while because we have Coprocessor Endpoints mixed in with the core of HBase that are tough to untangle (FIX). Tactic #4 (the fourth attempt at addressing this issue) is COPY all but the CPEP .proto files currently in hbase-protocol to a new module named hbase-protocol-shaded. Generate .protos again in the new location and then relocate/shade the generated files. Let CPEPs keep on with the old references at com.google.protobuf.* and org.apache.hadoop.hbase.protobuf.* but change the hbase core so all instead refer to the relocated files in their new location at org.apache.hadoop.hbase.shaded.com.google.protobuf.*. Let the new module also shade protobufs themselves and change hbase core to pick up this shaded protobuf rather than directly reference com.google.protobuf. This approach allows us to explicitly refer to either the shaded or non-shaded version of a protobuf class in any particular context (though usually context dictates one or the other). Core runs on shaded protobuf. CPEPs continue to use whatever is on the classpath with com.google.protobuf.* which is pb2.5.0 for the near future at least. See above cited doc for follow-ons and downsides. In short, IDEs will complain about not being able to find the shaded protobufs since shading happens at package time; will fix by checking in all generated classes and relocated protobuf in a follow-on. Also, CPEPs currently suffer an extra-copy as marshalled from non-shaded to shaded. To fix. Finally, our .protos are duplicated; once shaded, and once not. Pain, but how else to reveal our protos to CPEPs or C++ client that wants to talk with HBase AND shade protobuf. Details: Add a new hbase-protocol-shaded module. It is a copy of hbase-protocol i with all relocated offset from o.a.h.h. to o.a.h.h.shaded. The new module also includes the relocated pb. It does not include CPEPs. They stay in their old location. Add another module hbase-endpoint which has in it all the endpoints that ship as part of hbase -- at least the ones that are not entangled with core such as AccessControl and Auth. Move all protos for these CPEPs here as well as their unit tests (mostly moving a bunch of stuff out of hbase-server module) Much of the change looks like this: -import org.apache.hadoop.hbase.protobuf.ProtobufUtil; -import org.apache.hadoop.hbase.protobuf.generated.ClusterIdProtos; +import org.apache.hadoop.hbase.protobuf.shaded.ProtobufUtil; +import org.apache.hadoop.hbase.shaded.protobuf.generated.ClusterIdProtos; In HTable and in HBaseAdmin, regularize the way Callables are used and also hide protobuf usage as much as possible moving it up into Callable super classes or out to utility classes. Still TODO is adding in of retries, etc., but can wait on procedure which will redo all this. Also in HTable and HBaseAdmin as well as in HRegionServer and Server, be explicit when using non-shaded protobuf. Do the full-path so it is clear. This is around endpoint coprocessors registration of services and execution of CPEP methods. Shrunk ProtobufUtil by moving methods used by one CPEP only back to the CPEP either into Client class or as new Util class; e.g. AccessControlUtil. There are actually two versions of ProtobufUtil now; a shaded one and a subset that is used by CPEPs doing non-shaded work. Made it so hbase-common no longer depends on hbase-protocol (with Matteo's help) R*Converter classes got moved down under shaded package -- they are for internal use only. There are no non-shaded versions of these classes. D hbase-client/src/main/java/org/apache/hadoop/hbase/client/AbstractRegionServerCallable D RetryingCallableBase Not used anymore and we have too many tiers of Callables so removed/cleaned-up. A ClientServicecallable Had to add this one. RegionServerCallable was made generic so it could be used for a few Interfaces (Client and Admin). Then added ClientServiceCallable to implement RegionServerCallable with the Client Interface.
2016-10-04 00:37:32 -04:00
}