Commit Graph

17367 Commits

Author SHA1 Message Date
Duo Zhang 7386369fec HBASE-23276 Add admin methods to get tables within a group (#1118)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 5de0a5ef7e HBASE-23235 Re-enable TestRSGroupsKillRS.testLowerMetaGroupVersion (#1117)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang c8d892cdef HBASE-23253 Rewrite rsgroup related UTs with the new methods introduced in HBASE-22932 (#813)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 72cbb129a4 HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e8e9eecef1 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e12064b725 HBASE-23232 Remove rsgroup profile from pom.xml of hbase-assembly (#779) 2020-03-02 15:43:40 +08:00
Duo Zhang 0f4a87c24d HBASE-23081 Add an option to enable/disable rs group feature (#691)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b10b39ad03 HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0e29d62a2d HBASE-22987 Calculate the region servers in default group in foreground (#599)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Guanghao Zhang 2bf2781df9 HBASE-22729 Start RSGroupInfoManager as default (#555)
Amending-Author: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b3399ef365 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 2e0039c2a3 HBASE-22820 Do not need to persist default rs group now (#482)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 44ddb30914 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 3709f335a8 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang f0a13bb74d HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2020-03-02 15:43:40 +08:00
Duo Zhang 662149e193 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang fbdaa2113d HBASE-22664 Move protobuf stuff in hbase-rsgroup to hbase-protocol-shaded (#362)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Viraj Jasani 48a3ccf523
HBASE-23914 : Set hbase.client.retries.number for TestThriftHBaseServiceHandler (#1227)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-01 17:46:50 +05:30
stack a420f0482e HBASE-23904 Procedure updating meta and Master shutdown are incompatible: CODE-BUG
Restore behavior from before HBASE-21789 (hbase-2.2.0) where we convert
all exceptions to IOEs, even RuntimeExceptions. Actual fix is this change (in case
obscured by doc and lambda simplification):

     } catch (Throwable e) {
-      Throwables.propagateIfPossible(e, IOException.class);
+      // Throw if an IOE else wrap in an IOE EVEN IF IT IS a RuntimeException (e.g.
+      // a RejectedExecutionException because the hosting exception is shutting down.
+      // This is old behavior worth reexamining. Procedures doing merge or split
+      // currently don't handle RuntimeExceptions coming up out of meta table edits.
+      // Would have to work on this at least. See HBASE-23904.
+      Throwables.throwIfInstanceOf(e, IOException.class);
2020-02-28 13:52:30 -08:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
Wellington Ramos Chevreuil 00ef6c624a
HBASE-23892 SecureTestCluster should allow its subclasses to pass their Class reference on HBaseKerberosUtils.setSSLConfiguration (#1207)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-28 18:35:15 +00:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Pankaj 3c3aae9a2f
HBASE-23740 Invalid StoreFile WARN log message printed for recovered.… (#1198)
* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory

* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory
2020-02-26 17:52:21 +05:30
Toshihiro Suzuki ecbed33092
HBASE-23146 Support CheckAndMutate with multiple conditions (#1114)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-02-26 08:09:04 +09:00
Jan Hentschel 9f223c2236
HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-25 21:10:09 +01:00
Nick Dimiduk 3e21dc73de HBASE-23874 Move Jira-attached file precommit definition from script in Jenkins config to dev-support (#1191) (ADDENDUM)
Addendum -- set new script executable
2020-02-24 10:07:23 -08:00
Wei-Chiu Chuang 69d683afcb
HBASE-23755. [OpenTracing] Declare HTrace is unusable in the user doc (#1196)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-24 23:08:13 +05:30
Nick Dimiduk a5916c8e8e
HBASE-23874 Move Jira-attached file precommit definition from script in Jenkins config to dev-support (#1191)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-24 09:18:31 -08:00
Guanghao Zhang 418234356f HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (addendum) 2020-02-24 22:11:41 +08:00
Michael 400b7ce954 HBASE-23854 replaced deprecated code in Example Scala Code section
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-21 15:06:03 -06:00
Sean Busbey 0b2eb3594b Revert "replaced deprecated code in Example Scala Code section"
This reverts commit 4381b18b87.
2020-02-21 15:05:16 -06:00
Michael 4381b18b87 replaced deprecated code in Example Scala Code section
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-21 15:03:38 -06:00
Minwoo Kang 3ab0135c0e
HBASE-23561 Look up of Region in Master by encoded region name is O(n) (#1193)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-21 16:40:45 +00:00
Bharath Vissapragada fe0d49bc52 HBASE-23331: Documentation for HBASE-18095 2020-02-19 20:10:17 -08:00
Bharath Vissapragada c34dbc3c29 HBASE-23804: Fix default master addr hostname in master registry (#1137)
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 02bd0eca53 HBASE-23752: Fix remaining test failures from nightly runs (#1102)
TestFromClientSideWithCoprocessor: Initialization bug causing parameterized
runs to fail.
TestCustomSaslAuthenticationProvider: Test config had to be fixed because
it was written pre-master registry implementation.
TestSnapshotScannerHDFSAclController: Cluster restart did not reset the
cached connection state.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 5c88672d54 HBASE-23731: De-flake TestFromClientSide (#1091)
There were a couple of issues.

- There was a leak of a file descriptor for hbck lock file. This
was contributing to all the "ConnectionRefused" stack traces since
it was trying to renew lease for an already expired mini dfs cluster.
This issue was there for a while, just that we noticed it now.

- After upgrade to JUnit 4.13, it looks like the behavior for test
timeouts has changed. Earlier the timeout seems to have applied for
each parameterized run, but now it looks like it is applied across
all the runs.

This patch fixes both the issues.

Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 229b8aaaf3 HBASE-23647: Make MasterRegistry the default impl. (#1039)
Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada d8b3f55518 HBASE-23330: Fix delegation token fetch with MasterRegistry (#1084)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 1aa6a4efb9 HBASE-23305: Master based registry implementation (#954)
Implements a master based registry for clients.

 - Supports hedged RPCs (fan out configured via configs).
 - Parameterized existing client tests to run with multiple registry combinations.
 - Added unit-test coverage for the new registry implementation.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 7162c02c0d HBASE-23648: Re-use underlying connection registry in RawAsyncHBaseAdmin (#994)
* HBASE-23648: Re-use underlying connection registry in RawAsyncHBaseAdmin

No need to create and close a new registry on demand. Audited other
usages of getRegistry() and the code looks fine.

* Fix checkstyle issues in RawAsyncHBaseAdmin
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 0a1c3b2055 HBASE-23604: Clarify AsyncRegistry usage in the code. (#957)
* HBASE-23604: Cleanup AsyncRegistry interface

- Cleans up the method names to make more sense and adds a little
more javadocs for context. In future patches we can revisit
the name of the actual class to make it more self explanatory.

- Does AsyncRegistry -> ConnectionRegistry rename.
"async" ness of the registry is kind of implicit based on
the interface contents and need not be reflected in the name.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 023078ffa4 HBASE-23304: RPCs needed for client meta information lookup (#904)
* HBASE-23304: RPCs needed for client meta information lookup

This patch implements the RPCs needed for the meta information
lookup during connection init. New tests added to cover the RPC
code paths. HBASE-23305 builds on this to implement the client
side logic.

Fixed a bunch of checkstyle nits around the places the patch
touches.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 8126566b27 HBASE-23281: Track meta region locations in masters (#830)
* HBASE-23281: Track meta region changes on masters

This patch adds a simple cache that tracks the meta region replica
locations. It keeps an eye on the region movements so that the
cached locations are not stale.

This information is used for servicing client RPCs for connections
that use master based registry (HBASE-18095). The RPC end points
will be added in a separate patch.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 3d298b087a HBASE-23275: Track active master's address in ActiveMasterManager (#812)
* HBASE-23275: Track active master's address in ActiveMasterManager

Currently we just track whether an active master exists.
It helps to also track the address of the active master in
all the masters to help serve the client RPC requests to
know which master is active.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
stack 3168cf210c HBASE-23877 [Flakey Test] TestStochasticLoadBalancerRegionReplicaHighReplication "Two or more region replicas are hosted on the same host after balance" 2020-02-19 16:09:11 -08:00
stack 435b79335e HBASE-23872 [Flakey Test] TestGenerateDelegationToken; Master not initialized after 200000ms 2020-02-19 15:48:33 -08:00
Vladimir Rodionov b8194b4902 HBASE-22749 Distributed MOB compactions
- MOB compaction is now handled in-line with per-region compaction on region
  servers
- regions with mob data store per-hfile metadata about which mob hfiles are
  referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
  initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
  will rewrite MOB values to ensure it happens. controlled by
  'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
  with 'hbase.mob.major.compaction.region.batch.size'. default is as
  parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
  set of referenced mob hfiles and archive those that are no longer
  referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
  amplification by not rewriting values from mob hfiles over a certain size
  limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
  control threshold by 'hbase.mob.compactions.max.file.size'.
  default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
  will delay old MOB file cleanup until per-region compaction has managed
  to compact each region at least once so that used mob hfile metadata can
  be gathered.
2020-02-19 16:06:38 -06:00
Guanghao Zhang 53afc1dc10
HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (#1182)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-02-19 18:20:12 +08:00
stack 156ababa4e HBASE-23867 [Flakey Test] TestStochasticLoadBalancerRegionReplicaSameHosts#testRegionReplicationOnMidClusterSameHosts 2020-02-18 22:40:34 -08:00