Commit Graph

222 Commits

Author SHA1 Message Date
Duo Zhang 9e9a1e0f0d HBASE-21254 Need to find a way to limit the number of proc wal files 2018-10-12 11:05:13 +08:00
zhangduo 118b074684 HBASE-21250 Refactor WALProcedureStore and add more comments for better understanding the implementation 2018-10-07 17:09:09 +08:00
meiyi 6bc7089f9e HBASE-21249 Add jitter for ProcedureUtil.getBackoffTimeMs
Signed-off-by: zhangduo <zhangduo@apache.org>
2018-09-28 21:27:45 +08:00
zhangduo 22ac655704 HBASE-21233 Allow the procedure implementation to skip persistence of the state after a execution 2018-09-28 10:50:29 +08:00
Umesh Agashe dc767c06d2
HBASE-21023 Added bypassProcedure() API to HbckService 2018-09-19 15:18:16 -07:00
Michael Stack 76199a0a29 HBASE-21190 Log files and count of entries in each as we load from the MasterProcWAL store 2018-09-12 10:21:26 -07:00
TAK LON WU 7ecb435d9d
HBASE-21181 Use the same filesystem for wal archive directory and wal directory
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2018-09-11 15:32:51 -07:00
Duo Zhang c59ecfb961 HBASE-21172 Reimplement the retry backoff logic for ReopenTableRegionsProcedure 2018-09-11 15:13:58 +08:00
Michael Stack b83613fdce HBASE-21171 [amv2] Tool to parse a directory of MasterProcWALs standalone
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-09-09 09:28:59 -07:00
Michael Stack 3ac3249423
HBASE-21155 Save on a few log strings and some churn in wal splitter by skipping out early if no logs in dir 2018-09-06 16:52:47 -07:00
Allan Yang 7c1fad4992
HBASE-21083 Introduce a mechanism to bypass the execution of a stuck procedure 2018-08-30 12:23:24 -07:00
zhangduo bb3494134e HBASE-20881 Introduce a region transition procedure to handle all the state transition for a region 2018-08-21 06:12:09 +08:00
Allan Yang 159435ee40 HBASE-21050 Exclusive lock may be held by a SUCCESS state procedure forever
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: zhangduo <zhangduo@apache.org>
2018-08-15 15:39:48 -07:00
subrat.mishra 49ae8549cf HBASE-21040 Replace call to printStackTrace() with proper logger call
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-08-15 08:29:54 -07:00
Allan Yang 1114a1a65e HBASE-20978 [amv2] Worker terminating UNNATURALLY during MoveRegionProcedure Signed-off-by: Michael Stack <stack@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> 2018-08-14 16:30:20 -07:00
Allan Yang a07e755625 HBASE-20975 Lock may not be taken or released while rolling back procedure 2018-08-13 20:23:04 +08:00
jackbearden 953e5aa88c HBASE-20981. Rollback stateCount accounting thrown-off when exception out of rollbackState
Signed-off-by: Michael Stack <stack@apache.org>
2018-08-11 11:58:38 -07:00
Michael Stack c365c4084e HBASE-20989 Minor, miscellaneous logging fixes
Signed-off-by: Zach York <zyork@amazon.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2018-08-01 11:20:59 -07:00
Alex Leblang e963694259
HBASE-19369 Switch to Builder Pattern In WAL
This patch switches to the builder pattern by adding a helper method.
It also checks to ensure that the pattern is available (i.e. that
HBase is running on a hadoop version that supports it).

Amending-Author: Mike Drob <mdrob@apache.org>
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: zhangduo <zhangduo@apache.org>
2018-07-27 23:42:33 -05:00
zhangduo 7178a98258 HBASE-20939 There will be race when we call suspendIfNotReady and then throw ProcedureSuspendedException 2018-07-27 17:27:12 +08:00
zhangduo d43e28dc82 Revert "HBASE-20949 Add logs for debugging"
This reverts commit 8b8de1f8a7.
2018-07-27 08:40:46 +08:00
zhangduo 8b8de1f8a7 HBASE-20949 Add logs for debugging 2018-07-26 22:43:14 +08:00
zhangduo f3f17fa111 HBASE-20846 Restore procedure locks when master restarts 2018-07-25 14:37:26 +08:00
Michael Stack 067388bfd9 HBASE-20914 Trim Master memory usage
Add (weak reference) interning of ServerNames.

Correct Balancer regions x racks matrix.

Make smaller defaults when creating ArrayDeques.
2018-07-20 10:08:55 -07:00
zhangduo a838f7631f HBASE-20847 The parent procedure of RegionTransitionProcedure may not have the table lock 2018-07-11 17:34:35 +08:00
Yu Li ec8947f226 HBASE-20691 Change the default WAL storage policy back to "NONE""
This reverts commit 564c193d61 and added more doc
about why we choose "NONE" as the default.
2018-07-04 13:43:48 +08:00
zhangduo 0c97cda2a9 HBASE-19990 Create remote wal directory when transitting to state S 2018-06-28 18:07:44 +08:00
Michael Stack 21684a32fa HBASE-20745 Log when master proc wal rolls 2018-06-19 19:53:51 -07:00
zhangduo 6dbbd78aa0 HBASE-20708 Remove the usage of RecoverMetaProcedure in master startup 2018-06-19 15:02:10 +08:00
Josh Elser 8710825a9a HBASE-19735 Create a client-tarball assembly
Provides an extra client descriptor to build a second
tarball with a reduced set of dependencies. Not of great
impact now, but will build the way for better in the future.

Signed-off-by: Sean Busbey <busbey@apache.org>

 Conflicts:
	hbase-assembly/pom.xml
2018-06-18 11:31:13 -07:00
zhangduo 573b57d437 HBASE-20700 Move meta region when server crash can cause the procedure to be stuck 2018-06-11 14:57:31 +08:00
zhangduo a472f24d17 HBASE-20634 Reopen region while server crash can cause the procedure to be stuck
A reattempt at fixing HBASE-20173 [AMv2] DisableTableProcedure concurrent to ServerCrashProcedure can deadlock

The scenario is a SCP after processing WALs, goes to assign regions that
were on the crashed server but a concurrent Procedure gets in there
first and tries to unassign a region that was on the crashed server
(could be part of a move procedure or a disable table, etc.). The
unassign happens to run AFTER SCP has released all RPCs that
were going against the crashed server. The unassign fails because the
server is crashed. The unassign used to suspend itself only it would
never be woken up because the server it was going against had already
been processed. Worse, the SCP could not make progress because the
unassign was suspended with the lock on a region that it wanted to
assign held making it so it could make no progress.

In here, we add to the unassign recognition of the state where it is
running post SCP cleanup of RPCs. If present, unassign moves to finish
instead of suspending itself.

Includes a nice unit test made by Duo Zhang that reproduces nicely the
hung scenario.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/FailedRemoteDispatchException.java
 Moved this class back to hbase-procedure where it belongs.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NoNodeDispatchException.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NoServerDispatchException.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NullTargetServerDispatchException.java
 Specializiations on FRDE so we can be more particular when we say there
 was a problem.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
 Change addOperationToNode so we throw exceptions that give more detail
 on issue rather than a mysterious true/false

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
 Undo SERVER_CRASH_HANDLE_RIT2. Bad idea (from HBASE-20173)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 Have expireServer return true if it actually queued an expiration. Used
 later in this patch.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Hide methods that shouldn't be public. Add a particular check used out
 in unassign procedure failure processing.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
 Check that server we're to move from is actually online (might
 catch a few silly move requests early).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 Add doc on ServerState. Wasn't being used really. Now we actually stamp
 a Server OFFLINE after its WAL has been split. Means its safe to assign
 since all WALs have been processed. Add methods to update SPLITTING
 and to set it to OFFLINE after splitting done.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Change logging to be new-style and less repetitive of info.
 Cater to new way in which .addOperationToNode returns info (exceptions
 rather than true/false).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Add looking for the case where we failed assign AND we should not
 suspend because we will never be woken up because SCP is beyond
 doing this for all stuck RPCs.

 Some cleanup of the failure processing grouping where we can proceed.

 TODOs have been handled in this refactor including the TODO that
 wonders if it possible that there are concurrent fails coming in
 (Yes).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
 Doc and removing the old HBASE-20173 'fix'.
 Also updating ServerStateNode post WAL splitting so it gets marked
 OFFLINE.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestServerCrashProcedureStuck.java
 Nice test by Duo Zhang.

Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Duo Zhang <palomino219@gmail.com>
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-04 09:26:56 -07:00
zhangduo 997747076d HBASE-20659 Implement a reopen table regions procedure 2018-05-30 20:03:25 +08:00
Sean Busbey 8ba2a7eeb9 HBASE-20544 Make HBTU default to random ports.
Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Josh Elser <elserj@apache.org>
2018-05-09 23:35:20 -07:00
Chia-Ping Tsai 4cb444e77b
HBASE-20169 NPE when calling HBTU.shutdownMiniCluster (TestAssignmentManagerMetrics is flakey); AMENDMENT 2018-05-02 16:14:58 -07:00
Michael Stack 5a071dbe2b HBASE-20492 UnassignProcedure is stuck in retry loop on region stuck in OPENING state
Add backoff when stuck in RegionTransitionProcedure, the subclass of
AssignProcedure and UnassignProcedure. Can happen when we go to
transition but the current Region state is not what we expect.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
 Add doc on being able to suspend and wait on a timeout.

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
 Add 'attempt' counter so we can do backoff when we get stuck.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Add persistence of new 'attempt' counter

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Doc data members that are persisted by subclasses given this is 'odd'.
 Add a counter for 'attempts' used when 'stuck' to implement backoff.
 Add suspend with timeout when 'stuck'. Add callback when timeout is
 exhausted which does wakeup of this procedure.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestUnexpectedStateException.java
 Test of backoff.
2018-04-30 20:40:22 -07:00
Wei-Chiu Chuang 17a29ac231 HBASE-20338 WALProcedureStore#recoverLease() should have fixed sleeps for retrying rollWriter()
Signed-off-by: Mike Drob <mdrob@apache.org>
Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2018-04-12 16:33:55 -05:00
Umesh Agashe e4b51bb27d HBASE-20330 ProcedureExecutor.start() gets stuck in recover lease on store
rollWriter() fails after creating the file and returns false. In next iteration of while loop in recoverLease() file list is refreshed.

Signed-off-by: Appy <appy@cloudera.com>
2018-04-11 16:07:42 -07:00
Josh Elser 15c398f7d2 HBASE-20223 Update to hbase-thirdparty 2.1.0
Remove commons-cli and commons-collections4 use. Account
for the newer internal protobuf version of 3.5.1.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-03-26 22:05:19 -04:00
Umesh Agashe c614b9f3e8 HBASE-20224 Web UI is broken in standalone mode
Changes for HBASE-20027 seem to cause UI not showing up on default port in standalone mode. For concurrent
unit test execution, individual tests can set hbase.localcluster.assign.random.ports to true or modify
test/resources/hbase-site.xml.
2018-03-22 20:27:39 -07:00
Michael Stack 5d1b2110d1
Revert "HBASE-20224 Web UI is broken in standalone mode"
Broke shell tests.

This reverts commit dd9fe813ec.
2018-03-22 10:57:42 -07:00
Umesh Agashe 4cb40e6d84 HBASE-20224 Web UI is broken in standalone mode
Changes for HBASE-20027 seem to cause UI not showing up on default port in standalone mode. For concurrent
unit test execution, individual tests can set hbase.localcluster.assign.random.ports to true or modify
test/resources/hbase-site.xml.
2018-03-22 06:52:20 -07:00
Chia-Ping Tsai a6eeb26cc0 HBASE-20212 Make all Public classes have InterfaceAudience category
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-22 18:10:23 +08:00
Michael Stack acbdb86bb4 HBASE-20169 NPE when calling HBTU.shutdownMiniCluster
Adds a prepare step to RecoverMetaProcedure in which we test for
cluster up and master being up. If not up, we fail the run.

Modified hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
Modified hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
 Minor log cleanup.

Modified hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RecoverMetaProcedure.java
 Add pepare step.

Modified hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManagerMetrics.java
 Debug for the failing test....

Added hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRecoverMetaProcedure.java
 Test the prepare step goes down if master or cluster are down.
2018-03-20 13:11:57 -07:00
Michael Stack bedf849d83 HBASE-20213 [LOGGING] Aligning formatting and logging less (compactions,
in-memory compactions)

Log less. Log using same format as used elsewhere in log.

Align logs in HFileArchiver with how we format elsewhere. Removed
redundant 'region' qualifiers, tried to tighten up the emissions so
easier to read the long lines.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ChunkCreator.java
 Add a label for each of the chunkcreators we make (I was confused by
two chunk creater stats emissions in log file -- didn't know that one
was for data and the other index).

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
 Formatting. Log less.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreCompactionStrategy.java
 Make the emissions in here trace-level. When more than a few regions,
log is filled with this stuff.
2018-03-16 13:16:49 -07:00
Umesh Agashe 974200fca1 HBASE-20024 Fixed flakyness of TestMergeTableRegionsProcedure
We assumed that we can run for loop from 0 to lastStep sequentially. MergeTableRegionProcedure skips step 2. So, when i is 0 the procedure is already at step 3.
Added a method StateMachineProcedure#getCurrentStateId that can be used from test code only.
2018-03-09 12:45:39 -08:00
Sean Busbey 2a65066b35 HBASE-20070 refactor website generation
* rely on git plumbing commands when checking if we've built the site for a particular commit already
* switch to forcing '-e' for bash
* add command line switches for: path to hbase, working directory, and publishing
* only export JAVA/MAVEN HOME if they aren't already set.
* add some docs about assumptions
* Update javadoc plugin to consistently be version 3.0.0
* avoid duplicative site invocations on reactor modules
* update use of cp command so it works both on linux and mac
* manually skip enforcer plugin during build
* still doing install of all jars due to MJAVADOC-490, but then skip rebuilding during aggregate reports.
* avoid the pager on git-diff by teeing to a log file, which also helps later reviewing in the case of big changesets.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
2018-03-02 09:25:10 -06:00
Michael Stack b11e506664 HBASE-20069 fix existing findbugs errors in hbase-server 2018-02-26 16:01:31 -08:00
Michael Stack 549a6d93d4 HBASE-20043 ITBLL fails against hadoop3
Fix MoveRandomRegionOfTableAction. It depended on old AM behavior.
Make it do explicit move as is required in AMv3; w/o it, it was just
closing region causing test to fail.

Fix pom so hadoop3 profile specifies a different netty3 version.

Bunch of logging format change that came of trying trying to read
the spew from this test.
2018-02-24 17:29:54 -08:00
Michael Stack 51cea3e2c3 HBASE-20024 TestMergeTableRegionsProcedure is STILL flakey 2018-02-20 11:08:27 -08:00