Commit Graph

6823 Commits

Author SHA1 Message Date
jingyuntian 557985ee47 HBASE-20194 Basic Replication WebUI - Master
Signed-off-by: zhangduo <zhangduo@apache.org>
2018-06-26 18:27:50 +08:00
zhangduo a1b297e673 HBASE-20777 RpcConnection could still remain opened after we shutdown the NettyRpcServer 2018-06-26 09:07:58 +08:00
Michael Stack 40cd76af8a HBASE-20780 ServerRpcConnection logging cleanup Get rid of one of the logging lines in ServerRpcConnection by amalgamating all into one new-style log line. 2018-06-25 16:43:43 -07:00
Michael Stack 4e16ecd7f4
HBASE-20770 WAL cleaner logs way too much; gets clogged when lots of work to do
General log cleanup; setting stuff that can flood the log to TRACE.
2018-06-25 12:12:43 -07:00
Todd Lipcon bf65e95466 HBASE-20403. Fix race between prefetch task and non-pread HFile reads
With prefetch-on-open enabled, the task doing the prefetching was using
non-positional (i.e. streaming) reads. If the main (non-prefetch) thread
was also using non-positional reads, these two would conflict, because
inputstreams are not thread-safe for non-positional reads.

In the case of an encrypted filesystem, this could cause JVM crashes,
etc, as underlying cipher buffers were freed underneath the racing
threads. In the case of a non-encrypted filesystem, less severe errors
would be thrown. The included unit test reproduces the latter case.

(cherry picked from commit 025ddce868)
Signed-off-by: Todd Lipcon <todd@cloudera.com>
2018-06-25 12:11:36 -07:00
Michael Stack e915024e3c HBASE-20778 Make it so WALPE runs on DFS 2018-06-23 23:34:17 -07:00
zhangduo 2f4b0a4a42 HBASE-20775 TestMultiParallel is flakey 2018-06-24 08:26:26 +08:00
zhangduo dde042cc93 HBASE-20776 Update branch-2 version to 2.2.0-SNAPSHOT 2018-06-22 22:15:18 +08:00
zhangduo 108e636247 HBASE-18569 Add prefetch support for async region locator 2018-06-22 18:20:08 +08:00
zhangduo a86141b625 HBASE-20752 Make sure the regions are truly reopened after ReopenTableRegionsProcedure 2018-06-22 14:06:29 +08:00
zhangduo 6cebe06225 HBASE-20767 Always close hbaseAdmin along with connection in HBTU 2018-06-22 10:20:06 +08:00
Ankit Singhal 28d0d8c5cc HBASE-20642 Clients should re-use the same nonce across DDL operations
Also changes modify table operations to help the case where a MTP spans
two master, avoiding the sanity-checks propagating back to the client
unnecessarily.

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2018-06-20 15:10:52 -07:00
Josh Elser c8b76eb3f1 HBASE-20706 Prevent MTP from trying to reopen non-OPEN regions
ModifyTableProcedure is using MoveRegionProcedure in a way
that was unintended from the original implementation. As such,
we have to guard against certain usages of it. We know we can
re-open OPEN regions, but regions in OPENING will similarly
soon be OPEN (thus, we want to reopen those regions too).

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: zhangduo <zhangduo@apache.org>
2018-06-20 14:30:36 -07:00
zhangduo 5c2cb15e0b HBASE-20739 Add priority for SCP 2018-06-20 15:18:47 +08:00
zhangduo 83969b0da1 HBASE-20742 Always create WAL directory for region server 2018-06-20 14:21:28 +08:00
Michael Stack 9eeb501825 HBASE-20745 Log when master proc wal rolls 2018-06-19 19:53:29 -07:00
zhangduo 3e33aecea2 HBASE-20708 Remove the usage of RecoverMetaProcedure in master startup 2018-06-19 15:09:11 +08:00
Sean Busbey ee84a8f243 HBASE-20332 shaded mapreduce module shouldn't include hadoop
* modify the jar checking script to take args; make hadoop stuff optional
* separate out checking the artifacts that have hadoop vs those that don't.
* * Unfortunately means we need two modules for checking things
* * put in a safety check that the support script for checking jar contents is maintained in both modules
* * have to carve out an exception for o.a.hadoop.metrics2. :(
* fix duplicated class warning
* clean up dependencies in hbase-server and some modules that depend on it.
* allow Hadoop to have its own htrace where it needs it
* add a precommit check to make sure we're not using old htrace imports

 Conflicts:
	hbase-backup/pom.xml
	hbase-checkstyle/src/main/resources/hbase/checkstyle-suppressions.xml

Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-18 14:02:48 -07:00
Mike Drob b04c976fe6 HBASE-20478 Update checkstyle to v8.2
Cannot go to latest (8.9) yet due to
  https://github.com/checkstyle/checkstyle/issues/5279

* move hbaseanti import checks to checkstyle
* implment a few missing equals checks, and ignore one
* fix lots of javadoc errors

Signed-off-by: Sean Busbey <busbey@apache.org>
2018-06-18 14:02:40 -07:00
taiynlee 8edd5d948a HBASE-20737 put collection into ArrayList instead of addAll function
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2018-06-17 11:16:16 +08:00
tedyu b2afba580b HBASE-20723 Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them 2018-06-16 01:34:53 -07:00
Xu Cang b68746c0b2 HBASE-20695 Implement table level RegionServer replication metrics
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2018-06-15 10:45:13 +08:00
jingyuntian bde9f08a83 HBASE-20625 refactor some WALCellCodec related code
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2018-06-14 19:46:33 +08:00
zhangduo 161dc7c7f3 HBASE-20722 Make RegionServerTracker only depend on children changed event 2018-06-14 08:38:53 +08:00
Guanghao Zhang 075523dd1e HBASE-20561 The way we stop a ReplicationSource may cause the RS down 2018-06-13 18:05:27 +08:00
Balazs Meszaros d44e8a7aff HBASE-20656 Validate pre-2.0 coprocessors against HBase 2.0+
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-11 10:32:40 -05:00
Mike Drob 4b0bbd839e HBASE-20707 Move MissingSwitchDefault case check
Perform this check using error-prone instead of checkstyle because the
former can handle enum switches somewhat more intelligently.
2018-06-11 10:13:29 -05:00
zhangduo 6befdc43ba HBASE-20700 Move meta region when server crash can cause the procedure to be stuck 2018-06-11 15:28:21 +08:00
Guanghao Zhang 4d971d0f48 HBASE-20698 (addendum) Master don't record right server version until new started region server call regionServerReport method 2018-06-10 08:32:01 +08:00
Guanghao Zhang 9d15e16946 HBASE-20698 Master don't record right server version until new started region server call regionServerReport method 2018-06-09 14:47:07 +08:00
Nihal Jain 4a5fe54d94 HBASE-20699 QuotaCache should cancel the QuotaRefresherChore service inside its stop()
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-06-08 07:22:52 -07:00
Michael Stack 858eee20ec HBASE-20702 Processing crash, skip ONLINE'ing empty rows
Signed-off-by: Josh Elser <elserj@apache.org>
2018-06-07 09:54:32 -07:00
eric-maynard 271d93dc73 HBASE-20665: Changed log level of HBASE-8547 warning to debug
Closes #77

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2018-06-07 11:34:52 -04:00
Peter Somogyi 00289b8ffa HBASE-20683 Incorrect return value for PreUpgradeValidator
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2018-06-06 20:04:22 +02:00
Andrew Purtell d7b09de854 HBASE-20670 NPE in HMaster#isInMaintenanceMode 2018-06-04 15:19:45 -07:00
Michael Stack 063eefe3b0 HBASE-20634 Reopen region while server crash can cause the procedure to be stuck; ADDENDUM 2018-06-04 12:38:56 -07:00
Michael Stack 27e2c8c86b HBASE-20628 SegmentScanner does over-comparing when one flushing
Signed-off-by: eshcar <eshcar@oath.com>
Signed-off-by: anoopsjohn <anoopsamjohn@gmail.com>
2018-06-04 09:50:13 -07:00
zhangduo d834859404 HBASE-20634 Reopen region while server crash can cause the procedure to be stuck
A reattempt at fixing HBASE-20173 [AMv2] DisableTableProcedure concurrent to ServerCrashProcedure can deadlock

The scenario is a SCP after processing WALs, goes to assign regions that
were on the crashed server but a concurrent Procedure gets in there
first and tries to unassign a region that was on the crashed server
(could be part of a move procedure or a disable table, etc.). The
unassign happens to run AFTER SCP has released all RPCs that
were going against the crashed server. The unassign fails because the
server is crashed. The unassign used to suspend itself only it would
never be woken up because the server it was going against had already
been processed. Worse, the SCP could not make progress because the
unassign was suspended with the lock on a region that it wanted to
assign held making it so it could make no progress.

In here, we add to the unassign recognition of the state where it is
running post SCP cleanup of RPCs. If present, unassign moves to finish
instead of suspending itself.

Includes a nice unit test made by Duo Zhang that reproduces nicely the
hung scenario.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/FailedRemoteDispatchException.java
 Moved this class back to hbase-procedure where it belongs.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NoNodeDispatchException.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NoServerDispatchException.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/NullTargetServerDispatchException.java
 Specializiations on FRDE so we can be more particular when we say there
 was a problem.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
 Change addOperationToNode so we throw exceptions that give more detail
 on issue rather than a mysterious true/false

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
 Undo SERVER_CRASH_HANDLE_RIT2. Bad idea (from HBASE-20173)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 Have expireServer return true if it actually queued an expiration. Used
 later in this patch.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Hide methods that shouldn't be public. Add a particular check used out
 in unassign procedure failure processing.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
 Check that server we're to move from is actually online (might
 catch a few silly move requests early).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 Add doc on ServerState. Wasn't being used really. Now we actually stamp
 a Server OFFLINE after its WAL has been split. Means its safe to assign
 since all WALs have been processed. Add methods to update SPLITTING
 and to set it to OFFLINE after splitting done.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Change logging to be new-style and less repetitive of info.
 Cater to new way in which .addOperationToNode returns info (exceptions
 rather than true/false).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Add looking for the case where we failed assign AND we should not
 suspend because we will never be woken up because SCP is beyond
 doing this for all stuck RPCs.

 Some cleanup of the failure processing grouping where we can proceed.

 TODOs have been handled in this refactor including the TODO that
 wonders if it possible that there are concurrent fails coming in
 (Yes).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
 Doc and removing the old HBASE-20173 'fix'.
 Also updating ServerStateNode post WAL splitting so it gets marked
 OFFLINE.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestServerCrashProcedureStuck.java
 Nice test by Duo Zhang.

Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Duo Zhang <palomino219@gmail.com>
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-04 09:26:36 -07:00
maoling 4c95b82b61 HBASE-19761:Fix Checkstyle errors in hbase-zookeeper
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2018-06-02 10:17:27 +02:00
Andrew Purtell f46569a742 HBASE-20667 Rename TestGlobalThrottler to TestReplicationGlobalThrottler 2018-06-01 17:01:14 -07:00
Xu Cang d3e2248f12 HBASE-18116 Replication source in-memory accounting should not include bulk transfer hfiles
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2018-06-01 11:16:16 -07:00
Peter Somogyi 53d29d53c4 HBASE-20592 Create a tool to verify tables do not have prefix tree encoding
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-01 19:22:49 +02:00
Andrew Purtell b22409d51d Revert "HBASE-18116 fix replication source in-memory calculation by excluding bulk load file"
This reverts commit 050fae501a.
2018-05-31 15:28:37 -07:00
Xu Cang 050fae501a HBASE-18116 fix replication source in-memory calculation by excluding bulk load file
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2018-05-31 14:22:12 -07:00
Sean Busbey fc9743c17a HBASE-20444 Addendum keep folks from looking at raw version component array.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2018-05-31 14:17:41 -05:00
Andrew Purtell aaec02e0f5 HBASE-20646 TestWALProcedureStoreOnHDFS failing on branch-1 2018-05-30 14:44:54 -07:00
Andrew Purtell 15bb234d51 Revert "TestWALProcedureStoreOnHDFS failing on branch-1"
This reverts commit 694e79a67e.
2018-05-30 14:44:49 -07:00
Andrew Purtell 694e79a67e TestWALProcedureStoreOnHDFS failing on branch-1 2018-05-30 13:46:08 -07:00
zhangduo b785896cbd HBASE-20659 Implement a reopen table regions procedure 2018-05-30 20:03:35 +08:00
tedyu 856a3ac154 HBASE-20639 Implement permission checking through AccessController instead of RSGroupAdminEndpoint - revert due to pending discussion 2018-05-29 19:58:32 -07:00