Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
13,482 Commits 95 Branches 866 Tags 550 MiB
Java 96.1%
Ruby 1.7%
Perl 0.8%
Shell 0.7%
Python 0.3%
Other 0.1%
Go to file
Michael Stack dc1065a85d HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 
Move to a new AssignmentManager, one that describes Assignment using
a State Machine built on top of ProcedureV2 facility.

This doc. keeps state on where we are at w/ the new AM:
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn
Includes list of tests disabled by this patch with reasons why.

Based on patches from Matteos' repository and then fix up to get it all to pass cluster
tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc..
including:

1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM.
The basis comes from Matteo's repo here:
689227fcbf

Patch replaces old AM with the new under subpackage master.assignment.
Mostly just updating classes to use new AM -- import changes -- rather
than the old. It also removes old AM and supporting classes.
See below for more detail.

2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
3622cba4e3

Adds running of remote procedure. Adds batching of remote calls.
Adds support for assign/unassign in procedures. Adds version info
reporting in rpc. Adds start of an AMv2.

3. Reporting of remote RS version is from here:
ddb4df3964.patch

4. And remote dispatch of procedures is from:
186b9e7c4d

5. The split merge patches from here are also melded in:
9a3a95a2c2
and d6289307a0

We add testing util for new AM and new sets of tests.

Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping
procedure id. We spewed loads of log too on big transitions such as master fail; fixed.

Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge.
Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were
splitting split Regions among other things (CJ would run but wasn't
taking lock on Regions so havoc).

Added a bunch of doc. on Procedure primitives.

Added new region-based state machine base class. Moved region-based
state machines on to it.

Found bugs in the way procedure locking was doing in a few of the
region-based Procedures. Having them all have same subclass helps here.

Added isSplittable and isMergeable to the Region Interface.

Master would split/merge even though the Regions still had
references. Fixed it so Master asks RegionServer if Region
is splittable.

Messing more w/ logging. Made all procedures log the same and report
the state the same; helps when logging is regular.

Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure.

Added more functionality to MockMasterServices so can use it doing
standalone testing of Procedures (made TestCatalogTracker use it
instead of its own version).

Add to MasterServices ability to wait on Master being up -- makes
it so can Mock Master and start to implement standalone split testing.
Start in on a Split region standalone test in TestAM.

Fix bug where a Split can fail because it comes in in the middle of
a Move (by holding lock for duration of a Move).

Breaks CPs that were watching merge/split. These are run by Master now
so you need to observe on Master, not on RegionServer.

Details:

M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
Takes List of regionstates on construction rather than a Set.
NOTE!!!!! This is a change in a public class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Add utility getShortNameToLog

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
Add support for dispatching assign, split and merge processes.

M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
Lots of doc on its inner workings. Bug fixes.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
Log and doc on workings. Bug fixes.

A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
Dispatch remote procedures every 150ms or 32 items -- which ever
happens first (configurable). Runs a timeout thread. This facility is
not on yet; will come in as part of a later fix. Currently works a
region at a time. This class carries notion of a remote procedure and of a buffer full of these.
"hbase.procedure.remote.dispatcher.threadpool.size" with default = 128
"hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms
"hbase.procedure.remote.dispatcher.max.queue.size" with default = 32

M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
Add in support for merge. Remove no-longer used methods.

M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
Add execute procedures call ExecuteProcedures.

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
Add assign and unassign state support for procedures.

M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
Adds getting RS version out of RPC
Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Remove periodic metrics chore. This is done over in new AM now.
Replace AM with the new. Host the procedures executor.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
Have AMv2 handle assigning meta.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
Extract version number of the server making rpc.

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
Add new assign procedure. Runs assign via Procedure Dispatch.
There can only be one RegionTransitionProcedure per region running at the time,
since each procedure takes a lock on the region.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java
Remove these hacky classes that were never supposed to live longer than
a month or so to be replaced with real assigners.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
A procedure-based AM (AMv2).

TODO
 - handle region migration
 - handle meta assignment first
 - handle sys table assignment first (e.g. acl, namespace)
 - handle table priorities
  "hbase.assignment.bootstrap.thread.pool.size"; default size is 16.
  "hbase.assignment.dispatch.wait.msec"; default wait is 150
  "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100
  "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000;
  "hbase.assignment.maximum.attempts"; default is 10;

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
 Procedure that runs subprocedure to unassign and then assign to new location

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
 Manage store of region state (in hbase:meta by default).

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 In-memory state of all regions. Used by AMv2.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Base RIT procedure for Assign and Unassign.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Unassign procedure.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Run region assignement in a manner that pays attention to target server version.
 Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
 		2017-05-24 20:47:25 -07:00
bin HBASE-15199 (addendum) - When JRUBY_HOME is specified, update CLASSPATH and HBASE_OPTS when jruby is needed 2017-05-09 16:12:45 -05:00
conf HBASE-17722 Metrics subsystem stop/start messages add a lot of useless bulk to operational logging 2017-03-03 12:40:06 -08:00
dev-support HBASE-18076 (addendum) Flaky dashboard improvement: Add status markers to show trends of failure/success 2017-05-19 00:29:37 -07:00
hbase-annotations HBASE-17857 Remove IS annotations from IA.Public classes 2017-04-05 15:34:06 +08:00
hbase-archetypes HBASE-17722 Metrics subsystem stop/start messages add a lot of useless bulk to operational logging 2017-03-03 12:40:06 -08:00
hbase-assembly HBASE-17286 Add goal to remote-resources plugin 2017-05-19 11:37:49 -04:00
hbase-checkstyle HBASE-16785 We are not running all tests 2017-01-26 21:49:18 -08:00
hbase-client HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-common HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-endpoint HBASE-17857 Remove IS annotations from IA.Public classes 2017-04-05 15:34:06 +08:00
hbase-examples HBASE-17947 Location of Examples.proto is wrong in comment of RowCountEndPoint.java 2017-04-25 01:48:57 -07:00
hbase-external-blockcache HBASE-17532 Replaced explicit type with diamond operator 2017-03-07 11:22:51 -08:00
hbase-hadoop-compat HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-hadoop2-compat HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-it HBASE-16469 Several log refactoring/improvement suggestions 2017-04-11 14:16:12 -05:00
hbase-metrics HBASE-17581 mvn clean test -PskipXXXTests does not work properly for some modules (Yi Liang) 2017-02-02 11:05:17 -08:00
hbase-metrics-api HBASE-17448 Export metrics from RecoverableZooKeeper 2017-04-26 18:30:13 -07:00
hbase-native-client HBASE-14087 Ensure correct ASF headers for docs/code 2015-07-29 14:25:43 -05:00
hbase-prefix-tree HBASE-17809 cleanup unused class 2017-03-23 14:15:28 +08:00
hbase-procedure HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-protocol HBASE-17981 Consolidate the space quota shell commands 2017-05-22 13:41:36 -04:00
hbase-protocol-shaded HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-resource-bundle HBASE-18077 Update JUnit licensing to use EPL 2017-05-23 10:12:15 -05:00
hbase-rest HBASE-16356 REST API scanner: row prefix filter and custom filter parameters are mutually exclusive (Ben Watson) 2017-05-09 07:58:20 -07:00
hbase-rsgroup HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-server HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) 2017-05-24 20:47:25 -07:00
hbase-shaded HBASE-17286 Add goal to remote-resources plugin 2017-05-19 11:37:49 -04:00
hbase-shell HBASE-18001 Extend the "count" shell command to support specified conditions 2017-05-24 17:06:36 +08:00
hbase-spark HBASE-13074 Cleaned up usage of hbase.master.lease.thread.wakefrequency 2017-05-02 23:12:02 +08:00
hbase-testing-util HBASE-16682 Fix Shell tests failure. NoClassDefFoundError for MiniKdc. 2016-09-26 02:05:21 -07:00
hbase-thrift HBASE-13074 Cleaned up usage of hbase.master.lease.thread.wakefrequency 2017-05-02 23:12:02 +08:00
src/main HBASE-17977 Enable the MasterSpaceQuotaObserver by default 2017-05-22 13:41:36 -04:00
.gitattributes HBASE-6816. [WINDOWS] line endings on checkout for .sh files 2013-01-23 19:30:14 +00:00
.gitignore HBASE-16518 Remove unused .arcconfig file 2016-08-29 15:05:46 -07:00
CHANGES.txt HBASE-5466 Opening a table also opens the metatable and never closes it 2012-02-24 00:55:58 +00:00
LICENSE.txt HBASE-14338 License notification misspells 'Asciidoctor' (Lars Francke) 2015-09-22 09:53:53 -07:00
NOTICE.txt HBASE-17968 Fix NOTICE.txt for src-release 2017-05-01 11:25:44 -04:00
README.txt Updated links in README 2014-09-18 11:58:14 -07:00
pom.xml HBASE-17940 HMaster can not start due to Jasper related classes conflict 2017-04-20 16:06:50 +08:00

README.txt 

Apache HBase [1] is an open-source, distributed, versioned, column-oriented
store modeled after Google' Bigtable: A Distributed Storage System for
Structured Data by Chang et al.[2]  Just as Bigtable leverages the distributed
data storage provided by the Google File System, HBase provides Bigtable-like
capabilities on top of Apache Hadoop [3].

To get started using HBase, the full documentation for this release can be
found under the doc/ directory that accompanies this README.  Using a browser,
open the docs/index.html to view the project home page (or browse to [1]).
The hbase 'book' at http://hbase.apache.org/book.html has a 'quick start'
section and is where you should being your exploration of the hbase project.

The latest HBase can be downloaded from an Apache Mirror [4].

The source code can be found at [5]

The HBase issue tracker is at [6]

Apache HBase is made available under the Apache License, version 2.0 [7]

The HBase mailing lists and archives are listed here [8].

The HBase distribution includes cryptographic software. See the export control
notice here [9].

1. http://hbase.apache.org
2. http://research.google.com/archive/bigtable.html
3. http://hadoop.apache.org
4. http://www.apache.org/dyn/closer.cgi/hbase/
5. https://hbase.apache.org/source-repository.html
6. https://hbase.apache.org/issue-tracking.html
7. http://hbase.apache.org/license.html
8. http://hbase.apache.org/mail-lists.html
9. https://hbase.apache.org/export_control.html