Commit Graph

17494 Commits

Author SHA1 Message Date
Nick Dimiduk 5750876191
HBASE-23932 Minor improvements to Region Normalizer (#1258)
- consolidate checks made by master on behalf of balancer and
   normalizer: deciding if the master is in a healthy state for
   running any actions at all (skipRegionManagementAction). Normalizer
   now does as balancer did previously.
 - both balancer and normalizer make one final check on above
   conditions between calculating an action plan and executing the
   plan. should make the process more responsive to shutdown
   requests.
 - change normalizer to only consider acting on a region when it is in
   the OPEN state. previously we would normalizer attempt to merge a
   region that was already in a MERGING_NEW,MERGING,MERGED state.
 - fix some typos in variable names.

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-10 10:08:37 -07:00
Nick Dimiduk 31484f007f
HBASE-23946 [JDK11] Yetus should vote -0 for known JDK11 issues (#1246)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-09 09:10:33 -07:00
Nick Dimiduk 1f77b1ffa5 Revert "HBASE-23945 Dockerfiles showing hadolint check failures"
This reverts commit c61f95022d.
2020-03-07 10:41:54 -08:00
nyl3532016 819965a35a
HBASE-23944 The method setClusterLoad of SimpleLoadBalancer is incorrect when balance by table (#1243)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-07 16:18:41 +08:00
Guanghao Zhang 9b0d214b7b
HBASE-23895 STUCK Region-In-Transition when failed to insert procedure to procedure store (#1221)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-07 15:50:09 +08:00
Andrew Purtell d099f242e9
Replace 1.5.0 with 1.6.0 in downloads.xml 2020-03-06 14:48:27 -08:00
Nick Dimiduk c61f95022d HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-06 09:52:15 -08:00
Guanghao Zhang df62dde394
HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table descriptor directly (#1223)
Signed-off-by: Pankaj <pankajkumar@apache.org>
2020-03-06 19:06:59 +08:00
BukrosSzabolcs 4cb60327be
HBASE-23891: Add an option to Actions to filter out meta RS (#1217)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-03-06 10:44:39 +00:00
Wellington Ramos Chevreuil 9c67abef18
HBASE-23939 Remove unused variables from HBaseSaslRpcServer (#1239)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-06 10:05:04 +00:00
Viraj Jasani 4ef326be15
HBASE-23909 : list_regions raise correct error if it is being splitted/merged/transitioning (#1238)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-06 13:01:17 +05:30
Nick Dimiduk 4c9d14a788 HBASE-23876 Add JDK11 compilation and unit test support to nightly job
Builds on the Dockerfile changes provided by HBASE-23767.

closes #1195

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Nick Dimiduk 055c07c7b7 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Sean Busbey aff0ff5d97 HBASE-23198 Update ref guide for distributed MOB compaction.
* add design doc for original MOB changes as they were when HBase 2.0 came out
* add design doc for distributed MOB compaction
* remove configuration and commands no longer relevant after distributed MOB compaction
* add in discussion of configuration options
* allow asciimath formulas since we use them in the discussion

closes #1232

Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2020-03-05 11:40:42 -06:00
stack 9bd39786df HBASE-23926 [Flakey Tests] Down the flakies re-run ferocity; it makes for too many fails.
Change the forkcount from 1.0C to 0.25C
2020-03-04 16:21:27 -08:00
Abhinaba Sarkar b6eefcaeb7
HBASE-23788 ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking (#1219)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-03-04 23:14:43 +05:30
Wei-Chiu Chuang 6d9802fc2e
HBASE-23861. Reconcile Hadoop version. (#1179)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-04 08:01:40 -08:00
Duo Zhang 29ea96fad8 HBASE-23921 Findbugs is OOM on master 2020-03-04 12:23:36 +08:00
stack f411e395d3 HBASE-22978 : Online slow response log (#1228)
Addendum to fix errorprone ERROR
2020-03-03 09:19:21 -08:00
Peter Somogyi 72adfffd28
HBASE-23920 Pass --copy-to argument in ExportSnapshot tests (#1234)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-03 16:32:35 +01:00
Guanghao Zhang 5dcbe687ae
HBASE-23912 Resolve the TODO of FSTableDescriptor's construct method (#1225)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-03 22:13:16 +08:00
Viraj Jasani c0301e3fdf
HBASE-23868 : Replace usages of HColumnDescriptor(byte [] familyName)… (#1222)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-02 17:20:03 +05:30
Sakthi c21b28fc5c
HBASE-23743 Add HBase 1.4.13 to the downloads page (#1226)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-02 00:22:22 -08:00
Duo Zhang 346d087f40 HBASE-23911 Attach the new rsgroup implementation design doc to our code base (#1224)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 420e38083f HBASE-23890 Update the rsgroup section in our ref guide (#1206)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 7f2d823164 HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the code base (#1152)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 37e87aeca7 HBASE-23807 Make rsgroup related shell command to use the new admin methods (#1148)
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 7386369fec HBASE-23276 Add admin methods to get tables within a group (#1118)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 5de0a5ef7e HBASE-23235 Re-enable TestRSGroupsKillRS.testLowerMetaGroupVersion (#1117)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang c8d892cdef HBASE-23253 Rewrite rsgroup related UTs with the new methods introduced in HBASE-22932 (#813)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 72cbb129a4 HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e8e9eecef1 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e12064b725 HBASE-23232 Remove rsgroup profile from pom.xml of hbase-assembly (#779) 2020-03-02 15:43:40 +08:00
Duo Zhang 0f4a87c24d HBASE-23081 Add an option to enable/disable rs group feature (#691)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b10b39ad03 HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0e29d62a2d HBASE-22987 Calculate the region servers in default group in foreground (#599)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Guanghao Zhang 2bf2781df9 HBASE-22729 Start RSGroupInfoManager as default (#555)
Amending-Author: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b3399ef365 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 2e0039c2a3 HBASE-22820 Do not need to persist default rs group now (#482)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 44ddb30914 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 3709f335a8 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang f0a13bb74d HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2020-03-02 15:43:40 +08:00
Duo Zhang 662149e193 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang fbdaa2113d HBASE-22664 Move protobuf stuff in hbase-rsgroup to hbase-protocol-shaded (#362)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Viraj Jasani 48a3ccf523
HBASE-23914 : Set hbase.client.retries.number for TestThriftHBaseServiceHandler (#1227)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-01 17:46:50 +05:30
stack a420f0482e HBASE-23904 Procedure updating meta and Master shutdown are incompatible: CODE-BUG
Restore behavior from before HBASE-21789 (hbase-2.2.0) where we convert
all exceptions to IOEs, even RuntimeExceptions. Actual fix is this change (in case
obscured by doc and lambda simplification):

     } catch (Throwable e) {
-      Throwables.propagateIfPossible(e, IOException.class);
+      // Throw if an IOE else wrap in an IOE EVEN IF IT IS a RuntimeException (e.g.
+      // a RejectedExecutionException because the hosting exception is shutting down.
+      // This is old behavior worth reexamining. Procedures doing merge or split
+      // currently don't handle RuntimeExceptions coming up out of meta table edits.
+      // Would have to work on this at least. See HBASE-23904.
+      Throwables.throwIfInstanceOf(e, IOException.class);
2020-02-28 13:52:30 -08:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
Wellington Ramos Chevreuil 00ef6c624a
HBASE-23892 SecureTestCluster should allow its subclasses to pass their Class reference on HBaseKerberosUtils.setSSLConfiguration (#1207)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-28 18:35:15 +00:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Pankaj 3c3aae9a2f
HBASE-23740 Invalid StoreFile WARN log message printed for recovered.… (#1198)
* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory

* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory
2020-02-26 17:52:21 +05:30